This is me re-visiting something I made a few years back for a production workload using utilities “not ready for production”. What could go wrong?
The goal was thousands of redis processes with some isolation. Previously their was none and everyone was root. This isn’t the entire process, but some of the main configuration points.
## Unmess sh cd /bin sudo rm sh sudo ln -s bash sh ## The usual suspects sudo apt-get update sudo apt-get install -y \ build-essential \ git-core \ xfsprogs \ dbus \ debootstrap \ curl \ lsof
Here we are just making the container mount points and sources. I don’t know why logs are stored in /mnt, it wasn’t my decision.
## Build the container sudo mkdir -p /var/lib/machines/redis sudo mkdir /opt/null sudo debootstrap --arch=amd64 jessie /var/lib/machines/redis/ ## Create redis user sudo useradd -u 9000 -s /bin/false -m redis sudo useradd -u 9000 -s /bin/false -R /var/lib/machines/redis/ redis ## Modify /mnt permissions sudo chmod g+w /mnt sudo chown :redis /mnt ## Modify redis ulimits sudo tee /etc/security/limits.d/redistogo.conf <<'EOF' redis soft nofile 100000 redis hard nofile 100000 EOF
systemd-nspawn uses D-Bus. D-Bus has a problem where it limits the number of connections a user, including root, can make. Hitting this max causes the system to die and have to be restarted. I was unable to save it from this and I lost the link to the response from developers. It's a hardcoded config but it can be configured.
## Configure DBuS to not suck sudo tee /etc/dbus-1/system.d/redisprov.conf <<'EOF' <busconfig> <limit name="service_start_timeout">120000</limit> <limit name="auth_timeout">240000</limit> <limit name="pending_fd_timeout">150000</limit> <limit name="max_completed_connections">100000</limit> <limit name="max_incomplete_connections">10000</limit> <limit name="max_connections_per_user">100000000</limit> <limit name="max_pending_service_starts">10000</limit> <limit name="max_names_per_connection">50000</limit> <limit name="max_match_rules_per_connection">50000</limit> <limit name="max_replies_per_connection">50000</limit> </busconfig> EOF
Now we write a few unit files that build the container. The symlinking allows you to use a variable startup to launch different processes and use different ports.
## # Install Redis systemd unit files # # Note(crainte): Not certain why but a direct symlink to the service file # will not allow the dynamic version to start. We have to # do the include for it to pick up the correct variable. # # Examples: # systemctl status [email protected] # # systemctl start [email protected] # # systemctl stop [email protected] # # machinectl status redis-2.8.21-9000 ## sudo tee /etc/systemd/system/redisprov.include <<'EOF' .include /etc/systemd/system/redisprov.service EOF sudo tee /etc/systemd/system/redisprov.service <<'EOF' [Unit] Description=redisprov [Service] LimitNOFILE=100000 ExecStart=/usr/bin/systemd-nspawn --user=redis --keep-unit --machine=redis-%p-%i -j --directory=/var/lib/machines/redis/ --bind=/usr/local/bin/redis/%p/:/bin/ --bind=/mnt --bind=/opt/null:/sbin --bind=/home/redis/%i redis-server /home/redis/%i/rprov.conf Restart=always SuccessExitStatus=1 [Install] Also=dbus.service EOF
Now after "systemctl start [email protected]" & "systemctl start [email protected]" you will have two redis processes running under nspawn. As I mentioned, these are notes from a few years back. If revisiting I could most likely remove the debootstrap option since redis isn't using that. But if you want to run a more full featured application under nspawn, you might need it.