This guide attempts to document how to deploy a Django application with Kubernetes while using continuous integration It assumes basic knowledge of Docker and running Kubernetes and will instead focus on using helm with CI. Goals:

• Must be entirely automated and deploy on git pushes
• Must run database migrations once and only once per deploy
  • Must revert deployment if migrations fail
• Must allow easy management of secrets via environment variables

My need for this is to deploy GlitchTip staging builds automatically. GlitchTip is an open source error tracking platform that is compatible with Sentry. You can find the finished helm chart and gitlab CI script here. I’m using DigitalOcean and Gitlab CI but this guide will generally work for any Kubernetes provider or Docker based CI tool.

Building Docker

This guide assumes you have basic familiarity with running Django in Docker. If not, consider a local build first using docker compose. I prefer using compose for local development because it’s very simple and easy to install.

Build a Docker image and tag it with the git short hash. This will allow us to specify an exact image build later on and will ensure code builds are tied to specific helm deployments. If we used “latest” instead, we may end up accidentally upgrading the Docker image. Using Gitlab CI the script may look like this:

docker build -t ${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_NAME} -t ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA}

This uses -t to tag the new build with the Gitlab CI environment variables to specify the docker registry and tags. It uses “ref name” which is the tag or branch name. This will result in a tag such as “1.3” or branch such as “dev”. This tagging is intended for users who may just want a specific named version or branch. The second -t tags it with the git short hash. This tag will be referenced later on by helm.

Before moving on – make sure you can now docker pull your CI built image and run it. Make sure to set the Dockerfile CMD to use gunicorn, uwsgi, or another production ready server. We’ll deal with Django migrations later using Helm.

Setting up Kubernetes

This guide assumes you know how to set up Kubernetes. I chose DigitalOcean because they provide managed Kubernetes, it’s reasonably priced, and I like supporting smaller companies. DigitalOcean limits choice which makes it easier to use for average looking projects. It doesn’t offer the level of customization and services AWS does. If you decide to use DigitalOcean and want to help offset the cost of my open source projects, considering using this affiliate link. My goals for a hosting platform are:

• Easy to use
• Able to be managed via terraform
• Managed Postgres
• Managed Kubernetes
• Able to restrict network access for internal services such as the database

Whichever platform you are using, make sure you have a database and it’s connection string and can authenticate to Kubernetes. If you are new to Kubernetes, I suggest deploying any docker image manually (without tooling like helm) to get a little more familiar. Technically, you could also run your database in Kubernetes and Helm. However I prefer managed stateful services and will not cover running the database in Kubernetes in this guide.

Deploy to Kubernetes with Helm in Gitlab CI

Update Feb 2021
The GlitchTip Helm Chart is now a generic Django + Celery Helm chart. Read more here.

---

Now that you have a Docker image and Kubernetes infrastructure, it’s time to write a Helm chart and deploy your image automatically from CI. A Helm chart allows you to write Kubernetes yaml configuration templates using variables. The chart I use for GlitchTip should be a good starting point for most Django apps. At a minimum, read the getting started section for Helm’s documentation. The GlitchTip chart includes one web server deployment and a Django migration job with helm lifecycle hook. You may need to set up an additional deployment if you use a worker such as Celery. The steps are the same, just override the Docker RUN command to start celery instead of your web server.

Run the initial helm install locally. This is necessary to set initial variables such as the database connection that don’t need to be set in CI each deploy. Reference each value to override in your chart’s values.yaml. If following my GlitchTip example, that will be databaseURL and secretKey. databaseURL is the Database connection string. I use django-environ to set this. You could also define a separate databaseUser, databasePassword, etc if you like making more work for yourself. The key to make this work is to ensure one way or another the database credentials and other configuration get passed in as environment variables that are read by your settings.py file. Ensure your CI server has built at least one docker image. Place your chart files in the same git repo as your Django project in a directory “chart”

Run helm install your-app-name ./chart --set databaseURL=string --set secretKey=random_string --set image.tag=git_short_hash​

If you use GlitchTip’s chart – it will not set up a load balancer but it will show output that explains how to connect locally just to test that everything is working. The Django migration job should also run and migrate your database. This guide will not include the many options you have for load balancing. I choose to use DigitalOcean’s load balancer and having it directly select the deployment’s pods. Note that in Kubernetes, a service of type Load Balancer may run a service providers load balancer and allow you to configure it through kubernetes config yaml. This will vary between providers. Here’s a sample load balancer that can be applied with ​kubectl –namespace your-namespace apply -f load-balancer.yaml note that it uses selector to directly send traffic from the load balancer to pods. It also contains DigitalOcean specific annotations, which is why I can’t document a universal way to do this.

apiVersion: v1
kind: Service
metadata:
  name: your-app-staging
  annotations:
    service.beta.kubernetes.io/do-loadbalancer-certificate-id: long-id
    service.beta.kubernetes.io/do-loadbalancer-healthcheck-path: /
    service.beta.kubernetes.io/do-loadbalancer-protocol: http
    service.beta.kubernetes.io/do-loadbalancer-redirect-http-to-https: "true"
    service.beta.kubernetes.io/do-loadbalancer-tls-ports: "443"
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: https
    port: 443
    protocol: TCP
    targetPort: 8080
  selector:
    app.kubernetes.io/instance: your-app-staging
    app.kubernetes.io/name: your-app

At this point you should have a fully working Django application.

Updating in CI using Helm

Now set up CI to upgrade your app on git pushes (or other criteria). While technically optional, I suggest making separate namespaces and service accounts for each environment. Unfortunately this process can feel obtuse at first and I felt was the hardest part of this project. For each environment, we need the following:

• Service Account
• Role Binding
• Secret with CA Cert and token

For a rough analogy the service account is a “user” but for a bot instead of a human. A role binding defines the permissions that something (say a service account) has. The role binding should have the “edit” permission for the namespace. The secret is like the “password” but is actually a certificate and token. Read more from Kubernetes documentation.

Once this is set up locally, test it out. For example, use the new service account auth in your ~/.kube/config and run kubectrl get pods –namespace=your-namespace. The CA cert and token from your recently created secret should be what is in your kube’s config file. I found no sane manner of editing multiple kubernetes configurations and resorted to manually editing the config file.

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: big-long-base64 
    server: https://stuff.k8s.ondigitalocean.com
  name: some-name

...

users:
- name: default
  user:
    token big-long-token-from-secret

Notice I used certifate-authority-data so I could reference the cert inline as base64. Next save the entire config file in Gitlab CI under settings, CI, Variables.

There’s actually a lot happening in this little bit of configuration. File type in Gitlab CI will cause the value to save into a random tmp file. The key “KUBECONFIG” will be set to the file location. KUBECONFIG is also the environment variable helm will use to locate the kube config file. Protected will allow this only to be available to protected git branches/tags. If we didn’t set protected, someone with only limited git access could make their own branch that runs echo $KUBECONFIG and view the very confidential data! If set up right, you should now be able to run helm with the authentication that just works.

Finally add the deploy step to Gitlab CI’s yaml file.

deploy-staging:
  stage: deploy
  image: lwolf/helm-kubectl-docker
  script:
    - helm upgrade your-app-staging ./chart --set image.tag=${CI_COMMIT_SHORT_SHA} --reuse-values
  environment:
    name: staging
    url: https://staging.example.com
  only:
    - master

​stage ensures it runs after the docker build. For image, use lwolf/helm-kubectl-docker which has helm already installed. The script is amazingly just one line thanks to the previous authentication and Gitlab CI variable tricks done. It runs helm upgrade with –set image.tag to the new git short hash and –reuse-values allows it to set this new value without overriding previous values. Using helm this way allows you to keep database secrets outside of Gitlab. Do note however that anyone with helm access can read these values. If you need a more robust system then you’ll need something like Vault. But even without Vault, we can isolate basic git users who can create branches and admin users who have access to helm and the master branch.

The environment section is optional and let’s Gitlab track deploys. “only” causes the script to only run on the master branch. Alternatively it could be set for other branches or tags.

If you need to change an environment variable, run the same upgrade command locally and –set as many variables as needed. Keep the –reuse-values. Because the databaseURL value is marked as required, helm will error instead of erase previous values should you forget the important –reuse-values.

Conclusion

I like Kubernetes for it’s reliability but I find it creates a large amount of decision fatigue. I hope this guide provides one way to do things that I find works. If you have a better way – let me know by commenting here or even open an issue on GlitchTip. I’m sure there’s room for improvement. For example, I’d rather generate the django secret key automatically but helm’s random function doesn’t let you store it persistently.

I don’t like Kube’s, maddening at times, complexity. Kubernetes is almost never a solution by itself and requires additional tools to make it work for even very basic use cases. I found Openshift to handle a lot of common use cases like deploy hooks and user/service management much easier. Openshift “routes” are also defined in standard yaml config rather than forcing the user to deal with propreitary annotations on a Load Balancer. However, I’m leery of using Openshift Online considering it hasn’t been updated to version 4 and no roadmap seems to exist. It’s also quite a bit more expensive (not that it’s bad to pay more for good open source software).

Finally if you need error tracking for your Django app and prefer open source solutions – give GlitchTip a try. Contributors are preferred, but you can also support the project by using the DigitalOcean affiliate link or donating. Burke Software also offers paid consulting services for open source software hosting and software development.