Docker in dev and in production – a complete and DIY guide

Docker is an amazing Linux containerization tool. At Burke Software, we moved our development environment to Fig months ago and are now using Docker in production as well. This guide should give you ideas. I’m going to cover a lot of technologies not related to Docker to give you an idea how we do things. In my examples I’m using burkesoftware.com which is on GitHub for learning purposes. You should be able to follow along and run the burkesoftware.com website in Docker! Talk about self promotion – did I mention we are available for hire?

Docker in development

In development we use Fig, a tool that makes Docker a bit easier to use. It’s great whether you’re a Linux admin, software engineer, or graphic designer with minimal command line experience. The Fig documentation is pretty good so I won’t go into running it. With Fig everything gets standardized and mimics production. It would be a lot to ask for a designer to run solr, redis, postgres, and celery. Fig lets you do this. If your production environment runs a worker queue like celery, so should develop. The most differences between development and production, the more opportunities for bugs.

Current state of Docker in production

Docker itself is stable and past the 1.0 release. However the tools around it are not. For nontrivial deployments you need a little more than basic Docker commands or ‘fig up’. Flynn and Deis looks REALLY cool but are not stable yet. There is mesos and shipyard and lots more. Running Docker by hand can be a bit daunting. The guide will focus on the by hand approach – with some tools to support it.

Server

Let’s start with a basic server. I’m using DigitalOcean. If you like this post and start a DigitalOcean account please consider using this affiliate link. The cool thing about Docker is you aren’t tied to any one service as long as that service runs Linux. AWS? Microsoft? Your decade-old desktop lying around? Anything you want.

I use Ansible to provision my server. The idea here is it’s somewhat self-documenting and I can throw out my DigitalOcean account and start it up on EC2 on a whim. Here is my Ansible YML file. This is my file and not intended to just copy. It’s so that you to know how to use Ansible and get some ideas. I will refer to it often. Basically, any task I would normally do by hand I do via Ansible so it’s reproducible. I’m using a private Git repo so I am actually adding secrets here.

Docker in Production

Docker itself is installed via Ansible. I’ll follow the order that a incoming request to the server would take.

1. An incoming request hits nginx installed on the host. Nginx is using proxies to route to a port on localhost that a Docker instance is listening to. The following is my configuration for burkesoftware.com. Nginx has the task to route a request for burkesoftware.com to port 8002. Port 8002 was arbitrarily assigned by me.

   {
       server {
       listen 80;
       server_name burkesoftware.com;
       access_log  /var/log/nginx/access.log;
   
       location / {
           proxy_pass http://127.0.0.1:8002;
           proxy_set_header Host $host;
           proxy_set_header X-Real-IP $remote_addr;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       }
   
   }
   

2. Supervisor – We need docker to be running and answer port 8002. We need an init system for Docker to run on and to respawn, restart, etc. Here is my supervisor conf file. WTF Fig in production???

   [program:burkesoftware.com]
   command = fig -f /opt/fig/burkesoftware.com/fig.yml up
   stdout_logfile = /var/log/webapps/burkesoftware.com.log
   redirect_stderr = true
   

3. Fig in production – This Docker blog post provides a basic overview of using Fig in production. While I prefer the Fig YML syntax over writing plain Docker commands, I still recommend taking some time to get familiar with Docker. You should know how to build, create, and remove Docker containers before going forward, because Fig won’t help you if things blow up. Once you have an understanding of Docker, though, you’ll find that Fig does make it very easy to connect and run various Docker containers. Here is my Fig production file:

   redis:
     image: dockerfile/redis
   web:
     build: /opt/burkesoftware.com
     command: gunicorn bsc_website.wsgi --log-file - -b 0.0.0.0:8000 -n burkesoftware.com
     volumes:
       - /opt/burkesoftware.com:/code
     ports:
       - "8002:8000"
     environment:
       - DATABASE_ADDR=NOPE
       - DATABASE_NAME=NOPE
       - DATABASE_USER=NOPE
       - DATABASE_PASSWORD=NOPE
       - USE_S3=Yup
       - AWS_ACCESS_KEY_ID=NOPE
       - AWS_SECRET_ACCESS_KEY=NOPE
       - AWS_STORAGE_BUCKET_NAME=NOPE
     mem_limit: 1000m
     links:
       - redis
   

   I’m saving the environment variables in the Fig file, which is in a private Git repo. I like tracking them in Git over something like Heroku where you just enter them without version control. Notice that port number again. I’m redirecting port 8002 to the container’s port 8000 – which is just the port I always use in Fig. It could be anything but I want to change as little as possible from dev. The mem_limit will prevent the container from eating all system RAM. Look how easy it is to get redis up! I can just as easily run celery workers and other junk just like Fig in development.

4. Persistent data – At this point our request has hit the Docker Gunicorn server which will respond. Cool. However, what happens when the container is restarted or even destroyed? Fig can deal with this itself and make databases persist; however, I don’t trust it in production. I’d like to destroy the container fully and create a new one without losing my important data. You could use dockervolumes to mount the persistent data. I’m just going to run Postgres on my host. You could also use Amazon’s RDS or an isolated database server. I feed in the Postgres credentials via environment variables as seen in the fig file. I’m storing my user uploaded files in S3. In my Ansible YML file you can see I’m backing up my entire Postgres database to S3 using the python package S3-backups. The important thing here is that I can stop and rm all my docker containers, rebuild them, and it’s not a big deal.

5. Updating production – I’m using Git hooks to update the server. I have staging servers here too. It’s nice to give your developers easy access to push to staging and production with just Git. Notice I started a bare Git repo in the Ansible YML file. I’ll use a post-receive hook to checkout the master branch, Fig build, collectstatic (Django specific), and migrate my database (also Django specific). Finally it will restart Docker using supervisor. The set -x will ensure whoever does the Git push will see everything in their terminal window. It’s a lot like Heroku or, more accurately, Heroku is a lot like a Git hook because it is a Git hook. Unlike Heroku I can install packages and run anything I want. 🙂

   #!/bin/bash
   
   NAME=burkesoftware.com
   set -x
   git --work-tree=/opt/$NAME/ checkout -f master
   
   fig -f /opt/fig/$NAME/fig.yml build
   fig -f /opt/fig/$NAME/fig.yml run --rm web ./manage.py collectstatic --noinput
   fig -f /opt/fig/$NAME/fig.yml run --rm web ./manage.py migrate
   
   supervisorctl restart $NAME
   

Hopefully all that gives you some idea of how we run Docker in production. Feel free to comment with questions!