Redis + systemd-nspawn

This is me re-visiting something I made a few years back for a production workload using utilities “not ready for production”. What could go wrong?

The goal was thousands of redis processes with some isolation. Previously their was none and everyone was root. This isn’t the entire process, but some of the main configuration points.

## Unmess sh
cd /bin                                                                            
sudo rm sh                                                                         
sudo ln -s bash sh                                                                 
## The usual suspects                                                              
sudo apt-get update                                                                
sudo apt-get install -y \                                                          
    build-essential \                                                              
    git-core \                                                                     
    xfsprogs \                                                                     
    dbus \                                                                         
    debootstrap \                                                                  
    curl \                                                                         

Here we are just making the container mount points and sources. I don’t know why logs are stored in /mnt, it wasn’t my decision.

## Build the container                                                          
sudo mkdir -p /var/lib/machines/redis                                           
sudo mkdir /opt/null                                                            
sudo debootstrap --arch=amd64 jessie /var/lib/machines/redis/                   
## Create redis user                                                            
sudo useradd -u 9000 -s /bin/false -m redis                                     
sudo useradd -u 9000 -s /bin/false -R /var/lib/machines/redis/ redis            
## Modify /mnt permissions                                                      
sudo chmod g+w /mnt                                                             
sudo chown :redis /mnt                                                          
## Modify redis ulimits                                                         
sudo tee /etc/security/limits.d/redistogo.conf <<'EOF'                          
redis   soft    nofile  100000                                                  
redis   hard    nofile  100000                                                  

systemd-nspawn uses D-Bus. D-Bus has a problem where it limits the number of connections a user, including root, can make. Hitting this max causes the system to die and have to be restarted. I was unable to save it from this and I lost the link to the response from developers. It's a hardcoded config but it can be configured.

## Configure DBuS to not suck                                                          
sudo tee /etc/dbus-1/system.d/redisprov.conf <<'EOF'                               
    <limit name="service_start_timeout">120000</limit>                             
    <limit name="auth_timeout">240000</limit>                                      
    <limit name="pending_fd_timeout">150000</limit>                                
    <limit name="max_completed_connections">100000</limit>                      
    <limit name="max_incomplete_connections">10000</limit>                      
    <limit name="max_connections_per_user">100000000</limit>                    
    <limit name="max_pending_service_starts">10000</limit>                      
    <limit name="max_names_per_connection">50000</limit>                        
    <limit name="max_match_rules_per_connection">50000</limit>                  
    <limit name="max_replies_per_connection">50000</limit>                      

Now we write a few unit files that build the container. The symlinking allows you to use a variable startup to launch different processes and use different ports.

# Install Redis systemd unit files                                              
#   Note(crainte): Not certain why but a direct symlink to the service file     
#                  will not allow the dynamic version to start. We have to      
#                  do the include for it to pick up the correct variable.       
#   Examples:                                                                   
#       systemctl status 2.8.21@9000                                            
#       systemctl start 2.8.21@9000                                             
#       systemctl stop 2.8.21@9000                                              
#       machinectl status redis-2.8.21-9000                                     
sudo tee /etc/systemd/system/redisprov.include <<'EOF'                          
.include /etc/systemd/system/redisprov.service                                  
sudo tee /etc/systemd/system/redisprov.service <<'EOF'                          
ExecStart=/usr/bin/systemd-nspawn --user=redis --keep-unit --machine=redis-%p-%i -j --directory=/var/lib/machines/redis/ --bind=/usr/local/bin/redis/%p/:/bin/ --bind=/mnt --bind=/opt/null:/sbin --bind=/home/redis/%i redis-server /home/redis/%i/rprov.conf

Now after "systemctl start 2.8.21@9000" & "systemctl start 2.8.21@9001" you will have two redis processes running under nspawn. As I mentioned, these are notes from a few years back. If revisiting I could most likely remove the debootstrap option since redis isn't using that. But if you want to run a more full featured application under nspawn, you might need it.

Good luck.

Leave a Reply

Your email address will not be published. Required fields are marked *