Deploy Django with helm to Kubernetes

This guide attempts to document how to deploy a Django application with Kubernetes while using continuous integration It assumes basic knowledge of Docker and running Kubernetes and will instead focus on using helm with CI. Goals:

  • Must be entirely automated and deploy on git pushes
  • Must run database migrations once and only once per deploy
    • Must revert deployment if migrations fail
  • Must allow easy management of secrets via environment variables

My need for this is to deploy GlitchTip staging builds automatically. GlitchTip is an open source error tracking platform that is compatible with Sentry. You can find the finished helm chart and gitlab CI script here. I’m using DigitalOcean and Gitlab CI but this guide will generally work for any Kubernetes provider or Docker based CI tool.

Building Docker

This guide assumes you have basic familiarity with running Django in Docker. If not, consider a local build first using docker compose. I prefer using compose for local development because it’s very simple and easy to install.

Build a Docker image and tag it with the git short hash. This will allow us to specify an exact image build later on and will ensure code builds are tied to specific helm deployments. If we used “latest” instead, we may end up accidentally upgrading the Docker image. Using Gitlab CI the script may look like this:

docker build -t ${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_NAME} -t ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA}

This uses -t to tag the new build with the Gitlab CI environment variables to specify the docker registry and tags. It uses “ref name” which is the tag or branch name. This will result in a tag such as “1.3” or branch such as “dev”. This tagging is intended for users who may just want a specific named version or branch. The second -t tags it with the git short hash. This tag will be referenced later on by helm.

Before moving on – make sure you can now docker pull your CI built image and run it. Make sure to set the Dockerfile CMD to use gunicorn, uwsgi, or another production ready server. We’ll deal with Django migrations later using Helm.

Setting up Kubernetes

This guide assumes you know how to set up Kubernetes. I chose DigitalOcean because they provide managed Kubernetes, it’s reasonably priced, and I like supporting smaller companies. DigitalOcean limits choice which makes it easier to use for average looking projects. It doesn’t offer the level of customization and services AWS does. If you decide to use DigitalOcean and want to help offset the cost of my open source projects, considering using this affiliate link. My goals for a hosting platform are:

  • Easy to use
  • Able to be managed via terraform
  • Managed Postgres
  • Managed Kubernetes
  • Able to restrict network access for internal services such as the database

Whichever platform you are using, make sure you have a database and it’s connection string and can authenticate to Kubernetes. If you are new to Kubernetes, I suggest deploying any docker image manually (without tooling like helm) to get a little more familiar. Technically, you could also run your database in Kubernetes and Helm. However I prefer managed stateful services and will not cover running the database in Kubernetes in this guide.

Deploy to Kubernetes with Helm in Gitlab CI

Now that you have a Docker image and Kubernetes infrastructure, it’s time to write a Helm chart and deploy your image automatically from CI. A Helm chart allows you to write Kubernetes yaml configuration templates using variables. The chart I use for GlitchTip should be a good starting point for most Django apps. At a minimum, read the getting started section for Helm’s documentation. The GlitchTip chart includes one web server deployment and a Django migration job with helm lifecycle hook. You may need to set up an additional deployment if you use a worker such as Celery. The steps are the same, just override the Docker RUN command to start celery instead of your web server.

Run the initial helm install locally. This is necessary to set initial variables such as the database connection that don’t need to be set in CI each deploy. Reference each value to override in your chart’s values.yaml. If following my GlitchTip example, that will be databaseURL and secretKey. databaseURL is the Database connection string. I use django-environ to set this. You could also define a separate databaseUser, databasePassword, etc if you like making more work for yourself. The key to make this work is to ensure one way or another the database credentials and other configuration get passed in as environment variables that are read by your settings.py file. Ensure your CI server has built at least one docker image. Place your chart files in the same git repo as your Django project in a directory “chart”

Run helm install your-app-name ./chart --set databaseURL=string --set secretKey=random_string --set image.tag=git_short_hash

If you use GlitchTip’s chart – it will not set up a load balancer but it will show output that explains how to connect locally just to test that everything is working. The Django migration job should also run and migrate your database. This guide will not include the many options you have for load balancing. I choose to use DigitalOcean’s load balancer and having it directly select the deployment’s pods. Note that in Kubernetes, a service of type Load Balancer may run a service providers load balancer and allow you to configure it through kubernetes config yaml. This will vary between providers. Here’s a sample load balancer that can be applied with ​kubectl –namespace your-namespace apply -f load-balancer.yaml note that it uses selector to directly send traffic from the load balancer to pods. It also contains DigitalOcean specific annotations, which is why I can’t document a universal way to do this.

apiVersion: v1
kind: Service
metadata:
  name: your-app-staging
  annotations:
    service.beta.kubernetes.io/do-loadbalancer-certificate-id: long-id
    service.beta.kubernetes.io/do-loadbalancer-healthcheck-path: /
    service.beta.kubernetes.io/do-loadbalancer-protocol: http
    service.beta.kubernetes.io/do-loadbalancer-redirect-http-to-https: "true"
    service.beta.kubernetes.io/do-loadbalancer-tls-ports: "443"
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: https
    port: 443
    protocol: TCP
    targetPort: 8080
  selector:
    app.kubernetes.io/instance: your-app-staging
    app.kubernetes.io/name: your-app

At this point you should have a fully working Django application.

Updating in CI using Helm

Now set up CI to upgrade your app on git pushes (or other criteria). While technically optional, I suggest making separate namespaces and service accounts for each environment. Unfortunately this process can feel obtuse at first and I felt was the hardest part of this project. For each environment, we need the following:

  • Service Account
  • Role Binding
  • Secret with CA Cert and token

For a rough analogy the service account is a “user” but for a bot instead of a human. A role binding defines the permissions that something (say a service account) has. The role binding should have the “edit” permission for the namespace. The secret is like the “password” but is actually a certificate and token. Read more from Kubernetes documentation.

Once this is set up locally, test it out. For example, use the new service account auth in your ~/.kube/config and run kubectrl get pods –namespace=your-namespace. The CA cert and token from your recently created secret should be what is in your kube’s config file. I found no sane manner of editing multiple kubernetes configurations and resorted to manually editing the config file.

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: big-long-base64 
    server: https://stuff.k8s.ondigitalocean.com
  name: some-name

...

users:
- name: default
  user:
    token big-long-token-from-secret

Notice I used certifate-authority-data so I could reference the cert inline as base64. Next save the entire config file in Gitlab CI under settings, CI, Variables.

Screenshot from 2020-01-24 10-59-53

There’s actually a lot happening in this little bit of configuration. File type in Gitlab CI will cause the value to save into a random tmp file. The key “KUBECONFIG” will be set to the file location. KUBECONFIG is also the environment variable helm will use to locate the kube config file. Protected will allow this only to be available to protected git branches/tags. If we didn’t set protected, someone with only limited git access could make their own branch that runs echo $KUBECONFIG and view the very confidential data! If set up right, you should now be able to run helm with the authentication that just works.

Finally add the deploy step to Gitlab CI’s yaml file.

deploy-staging:
  stage: deploy
  image: lwolf/helm-kubectl-docker
  script:
    - helm upgrade your-app-staging ./chart --set image.tag=${CI_COMMIT_SHORT_SHA} --reuse-values
  environment:
    name: staging
    url: https://staging.example.com
  only:
    - master

​stage ensures it runs after the docker build. For image, use lwolf/helm-kubectl-docker which has helm already installed. The script is amazingly just one line thanks to the previous authentication and Gitlab CI variable tricks done. It runs helm upgrade with –set image.tag to the new git short hash and –reuse-values allows it to set this new value without overriding previous values. Using helm this way allows you to keep database secrets outside of Gitlab. Do note however that anyone with helm access can read these values. If you need a more robust system then you’ll need something like Vault. But even without Vault, we can isolate basic git users who can create branches and admin users who have access to helm and the master branch.

The environment section is optional and let’s Gitlab track deploys. “only” causes the script to only run on the master branch. Alternatively it could be set for other branches or tags.

If you need to change an environment variable, run the same upgrade command locally and –set as many variables as needed. Keep the –reuse-values. Because the databaseURL value is marked as required, helm will error instead of erase previous values should you forget the important –reuse-values.

Conclusion

I like Kubernetes for it’s reliability but I find it creates a large amount of decision fatigue. I hope this guide provides one way to do things that I find works. If you have a better way – let me know by commenting here or even open an issue on GlitchTip. I’m sure there’s room for improvement. For example, I’d rather generate the django secret key automatically but helm’s random function doesn’t let you store it persistently.

I don’t like Kube’s, maddening at times, complexity. Kubernetes is almost never a solution by itself and requires additional tools to make it work for even very basic use cases. I found Openshift to handle a lot of common use cases like deploy hooks and user/service management much easier. Openshift “routes” are also defined in standard yaml config rather than forcing the user to deal with propreitary annotations on a Load Balancer. However, I’m leery of using Openshift Online considering it hasn’t been updated to version 4 and no roadmap seems to exist. It’s also quite a bit more expensive (not that it’s bad to pay more for good open source software).

Finally if you need error tracking for your Django app and prefer open source solutions – give GlitchTip a try. Contributors are preferred, but you can also support the project by using the DigitalOcean affiliate link or donating. Burke Software also offers paid consulting services for open source software hosting and software development.

Server side tracking with piwik and Django

Business owners want to track usage to gain insights on how users actually use their sites and apps. However tracking can raise privacy concerns, lead to poor site performance, and raises security concerns by inviting third party javascript to run.

For Passit, an open source password manager, we wanted to track how people use our app and view our passit.io marketing site. However we serve a privacy sensitive market. Letting a company like Google snoop on your password manager feels very wrong. Our solution is to use the open source and self hosted piwik analytics application with server side tracking.

Traditional client side tracking for our marketing site

passit.io uses the piwik javascript tracker. It runs on the same domain (piwik.passit.io) and doesn’t get flagged by Privacy Badger as a tracking tool. It won’t track your entire web history like Google Analytics or Facebook like buttons do.

Nice green 0 from privacy badger!

To respect privacy we can keep on the default piwik settings to anonomize ip addresses and respect the do not track header.

Server side tracking for app.passit.io

We’d like to have some idea of how people use our app as well. Sign ups, log ins, groups usage, ect. However injecting client side code feels wrong here. It would be a waste of your computer’s resources to track your movements to our piwik server and provides an attack vector. What if someone hijacked our piwik server and tried to inject random js into the passit app?

We can track usage of the app.passit.io api on the server side instead. We can simply track how many people use different api endpoints to get a good indication of user activity.

Django and piwik

Presenting django-server-side-piwik – a drop in Django app that uses middleware and Celery to record server side analytics. Let’s talk about how it’s built.

server_side_piwik uses the python piwikapi package to track server side usage. Their quickstart section shows how. We can implement it as Django middleware. Every request will have some data serialized and sent to a celery task for further processing. This means our main request thread isn’t blocked and we don’t slow down the app just to run analytics.

class PiwikMiddleware(object):
  """ Record every request to piwik """
  def __init__(self, get_response):
  self.get_response = get_response

def __call__(self, request):
  response = self.get_response(request)

  SITE_ID = getattr(settings, 'PIWIK_SITE_ID', None)
  if SITE_ID:
    ip = get_ip(request)
    keys_to_serialize = [
      'HTTP_USER_AGENT',
      'REMOTE_ADDR',
      'HTTP_REFERER',
      'HTTP_ACCEPT_LANGUAGE',
      'SERVER_NAME',
      'PATH_INFO',
      'QUERY_STRING',
    ]
    data = {
      'HTTPS': request.is_secure() 
    }
    for key in keys_to_serialize:
      if key in request.META:
        data[key] = request.META[key]
    record_analytic.delay(data, ip)
  return response

 

Now you can track usage from the backend which better respects user privacy. No javascript and no Google Analytics involved!

Feel free to check out the project on gitlab and let me know any comments or issues. Passit’s source is also on gitlab.

Finding near locations with GeoDjango and Postgis Part I

With GeoDjango we can find places in proximity to other places – this is very useful for things like a store locator. Let’s use a store locater as an example. Our store locator needs to be able to read in messy user input (zip, address, city, some combination). Then, locate any stores we have nearby.

General concept and theory

Screenshot from 2015-10-01 17-16-31

We have two problems to solve. One is to turn messy address input into a point on the globe. Then we need a way to query this point against other known points and determine which locations are close.

Set up known locations

Before we can really begin we need to set up GeoDjango. You can read the docs or use docker-compose.

It’s still a good idea to read the tutorial even if you use docker.

Let’s add a location. Something like:

class Location(models.Model):
    name = models.CharField(max_length=70)
    point = models.PointField()

    objects = models.GeoManager()

A PointField stores a point on the map. Because Earth is not flat we can’t use simple X, Y coordinates. Luckily you can almost think of Latitude and Longitude as X, Y. GeoDjango defaults to this. It’s also easy to get Latitude and Longitude from places like Google Maps. So if we want – we can ignore the complexities of mapping coordinates on Earth. Or you can read up on SRID if you want to learn more.

At this point we can start creating locations with points – but for ease of use add GeoModelAdmin to Django Admin to use Open Street Maps to set points.

from django.contrib import admin
from django.contrib.gis.admin import GeoModelAdmin
from .models import Location

@admin.register(Location)
class LocationAdmin(GeoModelAdmin):
    pass

Screenshot from 2015-10-01 17-31-54

Wow! We’re doing GIS!

Add a few locations. If you want to get their coordinates just type location.point.x (or y).

Querying for distance.

Django has some docs for this. Basically make a new point. Then query distance. Like this:

from django.contrib.gis.geos import fromstr
from django.contrib.gis.measure import D
from .models import Location

geom = fromstr('POINT(-73 40)')
Location.objects.filter(point__distance_lte=(geom, D(m=10000)))

m is meters – you can pass all sorts of things though. The result should be a queryset of Locations that are near our “geom” location.

Already we can find locations near other locations or arbitrary points! In Part II I’ll explain how to use Open Street Maps to turn a fuzzy query like “New York” into a point. And from there we can make a store locator!

Building an api for django activity stream with Generic Foreign Keys

I wanted to build a django-rest-framework api for interacting with django-activity-stream. Activity stream uses Generic Foreign Keys heavily which aren’t naturally supported. We can however reuse existing serializers and nest the data conditionally.

Here is a ModelSerializer for activity steam’s Action model.

from rest_framework import serializers
from actstream.models import Action
from myapp.models import ThingA, ThingB
from myapp.serializers import ThingASerializer, ThingBSerializer

class GenericRelatedField(serializers.Field):
    def to_representation(self, value):
        if isinstance(value, ThingA):
            return ThingASerializer(value).data
        if isinstance(value, ThingB):
            return ThingBSerializer(value).data
        # Not found - return string.
        return str(value)

class ActionSerializer(serializers.ModelSerializer):
    actor = GenericRelatedField(read_only=True)
    target = GenericRelatedField(read_only=True)
    action_object = GenericRelatedField(read_only=True)

    class Meta:
        model = Action

GenericRelatedField will check if the value is an instance of a known Model and assign it the appropriate serializer.

Next we can use a viewset for displaying Actions. Since activity stream uses querysets it’s pretty simple to integrate with a ModelViewSet. In my case I’m checking for a get parameter to determine whether we want all actions, actions of people the logged in user follows, or actions of the user. I added some filters on action and target content type too.

from rest_framework import viewsets
from actstream.models import user_stream, Action
from .serializers import ActionSerializer


class ActivityViewSet(viewsets.ReadOnlyModelViewSet):
    serializer_class = ActionSerializer

    def get_queryset(self):
        following = self.request.GET.get('following')
        if following and following != 'false' and following != '0':
            if following == 'myself':
                qs = user_stream(self.request.user, with_user_activity=True)
                return qs.filter(actor_object_id=self.request.user.id)
            else:  # Everyone else but me
                return user_stream(self.request.user)
        return Action.objects.all()

    filter_fields = (
        'actor_content_type', 'actor_content_type__model',
        'target_content_type', 'target_content_type__model',
    )

Here’s the end result, lots of nested data.
Screenshot from 2015-07-08 17:44:59

Adding new form in a formset

Everything I read about adding a new form to a formset with javascript involves cloning an existing form. This is a terrible method, what if the initial forms are 0? What about initial data? Here’s IMO better way to do it that uses empty_form, a function Django gives you to create a form where i is __prefix__ so you can easily replace it.

Add this under you “Add new FOO” button. In my case I have a question_form with many answers (answers_formset).
[html]

var form_count_{{ question_form.prefix }} = {{ answers_formset.total_form_count }};
$(‘#add_more_{{ question_form.prefix }}’).click(function() {
var form = ‘{{answers_formset.empty_form.as_custom|escapejs}}’.replace(/__prefix__/g, form_count_{{ question_form.prefix }});
$(‘#answers_div_{{ question_form.prefix }}’).append(form);
form_count_{{ question_form.prefix }}++;
$(‘#id_{{ answers_formset.prefix }}-TOTAL_FORMS’).val(form_count_{{ question_form.prefix }});
});

[/html]

This creates you empty_form right in javascript, replaces the __prefix__ with the correct number and inserts it, in my case I made an answers_div. See empty_form.as_custom, you could just do empty_form but that would just give the you basic form html. I want custom html. Make a separate template for this. Here’s mine but this just an example.

[html]
{{ answer.non_field_errors }}
{% for hidden in answer.hidden_fields %} {{ hidden }} {% endfor %}
<table>
<tr>
<td>
<span class="answer_span">{{ answer.answer }} {{ answer.answer.errors }}</span>
</td>
……etc…….
</tr>
</table>
[/html]

In your original template you can add the forms like this {% include “omr/answer_form.html” with answer=answer %}
But for the as_custom you need to edit your form itself to add the function.

[python]
def as_custom(self):
t = template.loader.get_template(‘answer_form.html’)
return t.render(Context({‘answer’: self},))
[/python]

I find this method far more stable than trying to clone existing forms. It seems to play well with the javascript I have in some of my widgets. Clone on the other hand gave me tons of trouble and hacks needed to fix it.

Django get_or_default

Quick hack today. Often I find myself wanting to get some django object, but in the case it doesn’t exist default it to some value. Specially I keep my end user configurable settings in my database. Typically I set this up with initial data so all the settings are already there, but sometimes I’ll add a setting and forgot to add it on some site instance.

[python]class Callable:
def __init__(self, anycallable):
self.__call__ = anycallable

def get_or_default(name, default=None):
""" Get the config object or create it with a default. Always use this when gettings configs"""
object, created = Configuration.objects.get_or_create(name=name)
if created:
object.value = default
object.save()
return object
get_or_default = Callable(get_or_default)[/python]

Now I can safely call things like edit_all = Configuration.get_or_default(“Edit all fields”, “False”) which will return my configuration object with the value set as False if not specified. Much better than a 500 error. There are plenty of other uses for this type of logic. Get_or_return_none for example. The goal for me is to stop 500 errors from my own carelessness by having safe defaults.

Django admin: better export to XLS


The goal here is to make a slick gui for selecting exactly what the user wants to export from Django’s Change List view. It will be an global action, so lets start there.

[python]
def export_simple_selected_objects(modeladmin, request, queryset):
selected_int = queryset.values_list(‘id’, flat=True)
selected = []
for s in selected_int:
selected.append(str(s))
ct = ContentType.objects.get_for_model(queryset.model)
return HttpResponseRedirect("/export_to_xls/?ct=%s&ids=%s" % (ct.pk, ",".join(selected)))
export_simple_selected_objects.short_description = "Export selected items to XLS"
admin.site.add_action(export_simple_selected_objects)

[/python]

This adds a global action called Export selected items to XLS. I went with xls instead of ods because xlwt is very mature and LibreOffice can open xls just fine. It’s limited by the max length of get variables because it just lists each id. See this bug report. Next is the view.

[python]
import xlwt
def admin_export_xls(request):
model_class = ContentType.objects.get(id=request.GET[‘ct’]).model_class()
queryset = model_class.objects.filter(pk__in=request.GET[‘ids’].split(‘,’))
model_fields = model_class._meta.fields

if ‘xls’ in request.POST:
workbook = xlwt.Workbook()
worksheet = workbook.add_sheet(unicode(model_class._meta.verbose_name_plural))
fields = []
# Get selected fields from POST data
for field in model_fields:
if ‘field__’ + field.name in request.POST:
fields.append(field)
# Title
for i, field in enumerate(fields):
worksheet.write(0,i, field.verbose_name)
for ri, row in enumerate(queryset): # For Row iterable, data row in the queryset
for ci, field in enumerate(fields): # For Cell iterable, field, fields
worksheet.write(ri+1, ci, unicode(getattr(row, field.name)))
# Boring file handeling crap
fd, fn = tempfile.mkstemp()
os.close(fd)
workbook.save(fn)
fh = open(fn, ‘rb’)
resp = fh.read()
fh.close()
response = HttpResponse(resp, mimetype=’application/ms-excel’)
response[‘Content-Disposition’] = ‘attachment; filename=%s.xls’ %
(unicode(model_class._meta.verbose_name_plural),)
return response

return render_to_response(‘export_to_xls.html’, {
‘model_name’: model_class._meta.verbose_name,
‘fields’: model_fields,
}, RequestContext(request, {}),)

[/python]

Remember to set up your URLs. Next is the HTML. Maybe something like this

[html]

$(document).ready(function()
{
$(“#check_all”).click(function()
{
var checked_status = this.checked;
$(“.check_field”).each(function()
{
this.checked = checked_status;
});
});
});

<h2> Export {{ model_name }} </h2>
<form method="post" action="">
<table>
<tr>
<th>
<input type="checkbox" id="check_all" checked="checked" />
</th>
<th>
Field
</th>
</tr>
{% for field in fields %}
<tr>
<td>
<input type="checkbox" class="check_field" checked="checked" name="field__{{ field.name }}"/>
</td>
<td>
{{ field.verbose_name }}
</td>
</tr>
{% endfor %}
</table>
<input type="submit" name="xls" value="Submit"/>
</form>
[/html]

The javascript just makes the check all box work. Note I use jquery, if you don’t you will need to rewrite it. Very simple but it works. Now users won’t have to delete unwanted columns from xls reports. Notice how the user is left on the export screen and not happily back to the edit list. Some ajax can solve this. I’m overriding the global change_list.html which actually isn’t ideal if you use any plugins that also override it. Here’s what I added.

[html]
/static/js/jquery.tools.min.js

$(document).ready(function()
{
$(“.button”).click(function()
{
if (
$(“option[value=export_simple_selected_objects]:selected”).length
&& $(“input:checked”).length
) {
$.post(
“”,
$(“#changelist-form”).serialize(),
function(data){
$(“#export_xls_form”).html(data);
}
);
$(“#export_xls_form”).overlay({
top: 60
});
$(“#export_xls_form”).overlay().load();
return false;
}
});
});

<!– Overlay, when you edit CSS, make sure this display is set to none initially –>

[/html]

I use jquery tools overlay to make a nice overlay screen while keeping the user on the change list page. Basically I want a div to appear and then load some stuff from ajax. What’s cool is that I just post the data to “” so the regular Django admin functions work without editing them for AJAX. Well I did add to the submit button onclick=’$(“#export_xls_form”).overlay().close();’ to close the window when submitting. Ok I’m a complete liar I also added get_variables = request.META[‘QUERY_STRING’] to the view as a cheap way to keep those GET variables. But hey it’s still works as a non ajax admin action and that’s cool.

In the screenshot I added a CSS3 shadow and rounded corners to make it look better.

What’s next? Well it would be nice if we could access foreign key fields. If this had some type of advanced search and saving mechanism, we’d have a full generic Django query builder. Hmm.