Building an api for django activity stream with Generic Foreign Keys

I wanted to build a django-rest-framework api for interacting with django-activity-stream. Activity stream uses Generic Foreign Keys heavily which aren’t naturally supported. We can however reuse existing serializers and nest the data conditionally.

Here is a ModelSerializer for activity steam’s Action model.

from rest_framework import serializers
from actstream.models import Action
from myapp.models import ThingA, ThingB
from myapp.serializers import ThingASerializer, ThingBSerializer

class GenericRelatedField(serializers.Field):
    def to_representation(self, value):
        if isinstance(value, ThingA):
            return ThingASerializer(value).data
        if isinstance(value, ThingB):
            return ThingBSerializer(value).data
        # Not found - return string.
        return str(value)

class ActionSerializer(serializers.ModelSerializer):
    actor = GenericRelatedField(read_only=True)
    target = GenericRelatedField(read_only=True)
    action_object = GenericRelatedField(read_only=True)

    class Meta:
        model = Action

GenericRelatedField will check if the value is an instance of a known Model and assign it the appropriate serializer.

Next we can use a viewset for displaying Actions. Since activity stream uses querysets it’s pretty simple to integrate with a ModelViewSet. In my case I’m checking for a get parameter to determine whether we want all actions, actions of people the logged in user follows, or actions of the user. I added some filters on action and target content type too.

from rest_framework import viewsets
from actstream.models import user_stream, Action
from .serializers import ActionSerializer


class ActivityViewSet(viewsets.ReadOnlyModelViewSet):
    serializer_class = ActionSerializer

    def get_queryset(self):
        following = self.request.GET.get('following')
        if following and following != 'false' and following != '0':
            if following == 'myself':
                qs = user_stream(self.request.user, with_user_activity=True)
                return qs.filter(actor_object_id=self.request.user.id)
            else:  # Everyone else but me
                return user_stream(self.request.user)
        return Action.objects.all()

    filter_fields = (
        'actor_content_type', 'actor_content_type__model',
        'target_content_type', 'target_content_type__model',
    )

Here’s the end result, lots of nested data.
Screenshot from 2015-07-08 17:44:59

Open source chat

Web based chatting services like Hipchat and Slack are catching on these days. Such services are proprietary which severely limits your freedom to extend them and gives control over communication to a for profit company and probably the NSA. What free alternatives are out there? We have good old IRC and XMPP servers. But these are typically a no go for non technical folks and time consuming to set up. Let’s review web based offerings starting with my favorite.

Let’s Chat

Screenshot from 2015-06-03 23:13:31
Let’s chat is based on node and mongodb. It features a nice web interface and some very basic xmpp support. It is not federated sadly so you can’t talk with xmpp users outside your server. It has no hosted offering. Installing it was the easiest of the options I looked at yet very buggy and annoying. This is why Slack is winning folks. Anyway here’s how I ran it.

  • Install from git. Do not use docker due to this bug.
  • Enable xmpp in the settings. You must set the host setting due to this bug. Here is my settings. I also had to enable TLS to get xmpp to work.
  • I use the hubot connector for fun stuff like animated gifs and scripting – which makes it more comparable to slack which offers a lot of integrations.
  • For Android I use Conversations which is an amazing xmpp client.

While I really hated installing Let’s Chat due to all the bugs – I’m happy with it now that it’s running. While I speak a bit badly about all the bugs I really appreciate the work put into it.

Kaiwa

Kaiwa is a web front end for an XMPP server. The huge benefit of this is you get a full XMPP server like Prosody which supports many of the more modern XMPP conventions (XEP’s). Federation is REALLY amazing and awesome except you will never use it because no one uses XMPP. In theory it would let you talk with people on other servers.

Kaiwa has some heavy requirements of a web server, ldap server, database, and xmpp server. ldap is a deal breaker for me. I would prefer storing user data by marking rusty nails that I must arrange myself in a garden of broken glass – because that would be far more pleasant than writing ldif files. The default kaiwa would lose all data when restarting docker even after mounting (looks to be an issue with the ldap container). I just gave up. Kaiwa also has no way to search history which is pretty important to me.

RocketChat

RocketChat is worth keeping an eye on, but is not finished or near feature complete as of this writing.

On just xmpp

XMPP is a standard for interoperable chat. That’s a great goal but also a big challenge. There is no good xmpp desktop client. None support images (Animated gifs I think are a big draw on why people like slack). The default chat client on Ubuntu, Empathy, hasn’t seen development in years.

Google used to champion XMPP but hangouts dropped most support except very basic one to one non federated chat. You suck Google.

Thanks to websockets it’s really easy to make chat apps. You can learn in a weekend. I made a simple app here with Django and Swampdragon. Learning XMPP on the other hand is hard. There are few resources and existing resources are out of date. XMPP tackles much more like federation so it’s unfair to compare it directly with websockets. Still, people are going to use what’s easy and that’s hosted proprietary services built on websockets. Hopefully open source can catch up. It would be great to see some crowd funded efforts in this space – I’d contribute.

The only time I will spam you

I’m applying to this $150,000 grant. We need 250 votes to be considered. Voting for us is an easy way to support projects like django-report-builder and django-simple-import.

We spin off these third party apps whenever possible. Burke Software’s main focus is an open source school information system. If anything on this blog has helped you please consider giving us a vote. Thank you!

Be warned the website requires a Facebook login. I apologize for that, I don’t even have a Facebook account myself and had to use my good old Testy McTest account to vote.

Docker in dev and in production – a complete and DIY guide

Docker is an amazing Linux containerization tool. At Burke Software, we moved our development environment to Fig months ago and are now using Docker in production as well. This guide should give you ideas. I’m going to cover a lot of technologies not related to Docker to give you an idea how we do things. In my examples I’m using burkesoftware.com which is on GitHub for learning purposes. You should be able to follow along and run the burkesoftware.com website in Docker! Talk about self promotion – did I mention we are available for hire?

Docker in development

In development we use Fig, a tool that makes Docker a bit easier to use. It’s great whether you’re a Linux admin, software engineer, or graphic designer with minimal command line experience. The Fig documentation is pretty good so I won’t go into running it. With Fig everything gets standardized and mimics production. It would be a lot to ask for a designer to run solr, redis, postgres, and celery. Fig lets you do this. If your production environment runs a worker queue like celery, so should develop. The most differences between development and production, the more opportunities for bugs.

Current state of Docker in production

Docker itself is stable and past the 1.0 release. However the tools around it are not. For nontrivial deployments you need a little more than basic Docker commands or ‘fig up’. Flynn and Deis looks REALLY cool but are not stable yet. There is mesos and shipyard and lots more. Running Docker by hand can be a bit daunting. The guide will focus on the by hand approach – with some tools to support it.

Server

Let’s start with a basic server. I’m using DigitalOcean. If you like this post and start a DigitalOcean account please consider using this affiliate link. The cool thing about Docker is you aren’t tied to any one service as long as that service runs Linux. AWS? Microsoft? Your decade-old desktop lying around? Anything you want.

I use Ansible to provision my server. The idea here is it’s somewhat self-documenting and I can throw out my DigitalOcean account and start it up on EC2 on a whim. Here is my Ansible YML file. This is my file and not intended to just copy. It’s so that you to know how to use Ansible and get some ideas. I will refer to it often. Basically, any task I would normally do by hand I do via Ansible so it’s reproducible. I’m using a private Git repo so I am actually adding secrets here.

Docker in Production

Docker itself is installed via Ansible. I’ll follow the order that a incoming request to the server would take.

  1. An incoming request hits nginx installed on the host. Nginx is using proxies to route to a port on localhost that a Docker instance is listening to. The following is my configuration for burkesoftware.com. Nginx has the task to route a request for burkesoftware.com to port 8002. Port 8002 was arbitrarily assigned by me.

    {
        server {
        listen 80;
        server_name burkesoftware.com;
        access_log  /var/log/nginx/access.log;
    
        location / {
            proxy_pass http://127.0.0.1:8002;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    
    }
    
  2. Supervisor – We need docker to be running and answer port 8002. We need an init system for Docker to run on and to respawn, restart, etc. Here is my supervisor conf file. WTF Fig in production???

    [program:burkesoftware.com]
    command = fig -f /opt/fig/burkesoftware.com/fig.yml up
    stdout_logfile = /var/log/webapps/burkesoftware.com.log
    redirect_stderr = true
    
  3. Fig in production – This Docker blog post provides a basic overview of using Fig in production. While I prefer the Fig YML syntax over writing plain Docker commands, I still recommend taking some time to get familiar with Docker. You should know how to build, create, and remove Docker containers before going forward, because Fig won’t help you if things blow up. Once you have an understanding of Docker, though, you’ll find that Fig does make it very easy to connect and run various Docker containers. Here is my Fig production file:

    redis:
      image: dockerfile/redis
    web:
      build: /opt/burkesoftware.com
      command: gunicorn bsc_website.wsgi --log-file - -b 0.0.0.0:8000 -n burkesoftware.com
      volumes:
        - /opt/burkesoftware.com:/code
      ports:
        - "8002:8000"
      environment:
        - DATABASE_ADDR=NOPE
        - DATABASE_NAME=NOPE
        - DATABASE_USER=NOPE
        - DATABASE_PASSWORD=NOPE
        - USE_S3=Yup
        - AWS_ACCESS_KEY_ID=NOPE
        - AWS_SECRET_ACCESS_KEY=NOPE
        - AWS_STORAGE_BUCKET_NAME=NOPE
      mem_limit: 1000m
      links:
        - redis
    

    I’m saving the environment variables in the Fig file, which is in a private Git repo. I like tracking them in Git over something like Heroku where you just enter them without version control. Notice that port number again. I’m redirecting port 8002 to the container’s port 8000 – which is just the port I always use in Fig. It could be anything but I want to change as little as possible from dev. The mem_limit will prevent the container from eating all system RAM. Look how easy it is to get redis up! I can just as easily run celery workers and other junk just like Fig in development.

  4. Persistent data – At this point our request has hit the Docker Gunicorn server which will respond. Cool. However, what happens when the container is restarted or even destroyed? Fig can deal with this itself and make databases persist; however, I don’t trust it in production. I’d like to destroy the container fully and create a new one without losing my important data. You could use dockervolumes to mount the persistent data. I’m just going to run Postgres on my host. You could also use Amazon’s RDS or an isolated database server. I feed in the Postgres credentials via environment variables as seen in the fig file. I’m storing my user uploaded files in S3. In my Ansible YML file you can see I’m backing up my entire Postgres database to S3 using the python package S3-backups. The important thing here is that I can stop and rm all my docker containers, rebuild them, and it’s not a big deal.

  5. Updating production – I’m using Git hooks to update the server. I have staging servers here too. It’s nice to give your developers easy access to push to staging and production with just Git. Notice I started a bare Git repo in the Ansible YML file. I’ll use a post-receive hook to checkout the master branch, Fig build, collectstatic (Django specific), and migrate my database (also Django specific). Finally it will restart Docker using supervisor. The set -x will ensure whoever does the Git push will see everything in their terminal window. It’s a lot like Heroku or, more accurately, Heroku is a lot like a Git hook because it is a Git hook. Unlike Heroku I can install packages and run anything I want. :)

    #!/bin/bash
    
    NAME=burkesoftware.com
    set -x
    git --work-tree=/opt/$NAME/ checkout -f master
    
    fig -f /opt/fig/$NAME/fig.yml build
    fig -f /opt/fig/$NAME/fig.yml run --rm web ./manage.py collectstatic --noinput
    fig -f /opt/fig/$NAME/fig.yml run --rm web ./manage.py migrate
    
    supervisorctl restart $NAME
    

Hopefully all that gives you some idea of how we run Docker in production. Feel free to comment with questions!

django, rest, and angularjs – a Don’t Repeat Yourself approach

I’m a django developer. When I started working with angular I wanted to keep using DRY principles that I’m used to with Django Forms. For example defining validation, verbose_name, etc in your models. This guide should give you an overview of building a system with django-rest-framework (DRF) and angular. It should also give you some ideas on using the rest option method to pull in some data about your fields.

Disclaimer – I’m a angular noob, criticism of this approach is much appreciated.

The entire project is on github so please follow along. The guide assumes you understand angular basics.

Creating a rest api in django

http://www.django-rest-framework.org/

I won’t go into detail because their documentation is great. Run the project above and check out the options method if you haven’t already. We have some great info here like help_text, (some) validation, and verbose name is now called label.

Screenshot from 2014-06-13 14:43:40

Consuming the api in angular

Let’s create a django admin like form that will save on blur.

Screenshot from 2014-07-05 15:41:42

Most examples of angular forms I’ve seen are highly repetitive. This feels wrong when you’re used to the Django Forms framework. Luckily we can ask the options method for meta data about our forms. Here’s an interesting post about the “almost unused” options method. Our client side app still retains it’s decoupled nature from the server. The client doesn’t care whether DRF or typing monkeys are providing the options method. It just cares about what label to be using or whether the field is required.

<span class="help-block" ng-if="fieldOptions.help_text">{{ fieldOptions.help_text }}</span>

In this example we see a help_text span shows only when help text is available. Now our fields are becoming more generic looking. Generic and repetitive tasks can be automated. Let’s make a directive to automate what we can (Notice I’m using coffeescript, js is also provided in the github project).

app.directive "bscField", ->
scope:
    fieldOptions: "="
    fieldForm: "="

templateUrl: "/static/app/partials/field.html"
transclude: true

bscField can accept a few attributes and uses transclude to allow customization of input itself. It’s a great strategy for including css framework gunk too. Check out the field.html partial. We can use it like this.

<div bsc-field field-options="pollOptions.int_field" field-form="form.int_field">
  <input class="form-control" name="int_field" ng-required="pollOptions.int_field.required" type="text" ng-model="poll.int_field" ng-blur="savePoll('int_field')" />
</div>

Notice I am still repeating myself a good bit. Consider it a work in progress. The input itself actually can’t be done in the partial and still work with ng-forms. Details here.

The RestfulModel factory will handle all of our interactions with the rest api. It uses restangular. I choose restangular over ngResource because it seemed a little bit easier to work with. It supports patch of the box which will be nice for our edit one field at a time approach. I’ve also introduced a isSaving property to the Forms so we can indicate to the user when a form is being saved. You can use RestfulModel in a controller like this:

    pollModel = new RestfulModel.Instance("polls")
    pollModel.getOptions().then (options) ->
        $scope.pollOptions = options
    pollModel.getOne($routeParams.poll_id, $scope.form).then (poll) ->
        $scope.poll = poll
        $scope.savePoll = poll.saveForm

Notice we are really just tying a model to our scope so we can access our options (rest options method) and the poll itself. We’re also adding a save function to the scope that we can have trigger on blur. ngRoute is being used to determine the id of the poll we are on.

$routeProvider.when “/poll/:poll_id/”,
controller: “PollController”
templateUrl: ‘/static/app/partials/polls.html’,

It’s probably best to just play with the github project and ask any questions you have in comments. Or perhaps tell me why I’m insane for doing things this way.

Saving and Error Handling

DRF will return a 400 (BAD REQUEST) when you attempt to save something invalid. Best of all it returns a reason why!

{"int_field": ["Enter a whole number."]}

Our directive can show this (and other states) to the user.

Screenshot from 2014-07-05 16:07:21

Next Steps

This is an experiment of mine that I hope triggers some discussion on DRY principles and javascript frameworks. Hopefully it gives you some ideas and hope that you don’t have to redefine all your server side models again in javascript. I’ll be putting the concept into production in django sis this summer. If it goes well I may try releasing RestfulModel as a stand alone angular project.

Too many id’s in django admin actions

The Django docs suggest sending queryset id’s for admin actions.  This works until you get tens of thousands of id’s in a queryset. At this point your URL with GET variables becomes too large.

I’m working around this with sessions. It’s not quite as nice as your url is no longer copy/pastable. I decided to display such to the user.

Here’s my action to export data.

def export_simple_selected_objects(modeladmin, request, queryset):
    selected_int = queryset.values_list('id', flat=True)
    selected = []
    for s in selected_int:
        selected.append(str(s))
    ct = ContentType.objects.get_for_model(queryset.model)
    if len(selected) > 10000:
        request.session['selected_ids'] = selected
        return HttpResponseRedirect("/admin_export/export_to_xls/?ct=%s&ids=IN_SESSION" % (ct.pk,))
    else:
        return HttpResponseRedirect("/admin_export/export_to_xls/?ct=%s&ids=%s" % (ct.pk, ",".join(selected)))
export_simple_selected_objects.short_description = "Export selected items to XLS"
admin.site.add_action(export_simple_selected_objects)

Then in my action view

field_name = self.request.GET.get('field', '')
model_class = ContentType.objects.get(id=self.request.GET['ct']).model_class()
if self.request.GET['ids'] == "IN_SESSION":
    queryset = model_class.objects.filter(pk__in=self.request.session['selected_ids'])
else:
    queryset = model_class.objects.filter(pk__in=self.request.GET['ids'].split(','))

At least now the user will know the data is in the session. Note the 10,000 limit is just made up! If your users don’t know what sessions are, their action will just work instead of doing nothing :)

python – convert documents (doc, docx, odt, pdf) to plain text without Libreoffice

I recently needed to convert some resumes to plain text. There are any number of use cases for wanting to extract readable text from binary formats. So here is a code snippet to do just that. I’m using some non python Linux programs and python libs. Notably absent is Libreoffice which would take care of a ton of formats. Libreoffice is however heavyweight and clunky to use. These programs will convert much faster. First let’s get some dependencies.

Notice – New versions of python-docx removed this function. Make sure to pip install docx and not the new python-docx.

Fedora 20 on Chromebook Pixel

I wrote about the Pixel before. This review will be about using Fedora 20 on the Pixel.

Why?

ChromeOS is nice. I was using a lot of vim in a ubuntu chroot. It is of course limited. I was also having suspend issues which takes away from the stability of ChromeOS which is really it’s main feature. Might have been related to using crouton. Ubuntu installs on the Pixel but I can’t recommend it. Unity has poor high dpi support. I also had a nightmare of crashes and bugs with Ubuntu. Fedora 20 comes default with gnome 3.10 which I heard support high dpi. So let’s try it out.

Installing

The fedora installer is bad. An installer should target two groups of users – technical and casual users. Ubuntu does a great job of this giving you fast routes to what any user is looking for.

Install perfection – Do it for me or send me to the partition editor.

Fedora on the other hand is just odd. The partition editor is horrid. After finding it, one must type in the size of any created partition. No slider, really? One more complaint – why is there a five second delay in grub? I only have one OS.

Booting to gnome 3.10

The boot animation works, another +1 over Ubuntu whose boot animation is lots of flickering and random text. I’ve heard it described by teachers as “programming talk”. Ok now to hack on gnome to make this readable….holy fuck it detected my high dpi screen automatically. Wow…I’m speechless. Unity can’t resize the top bar. XFCE requires editing themes to make larger borders. KDE can work but requires tweaks (and still looks ugly IMO). Gnome 3.10 just works.

I like the option to add accounts right away on first boot. I added my Google account with 2 factor auth no problem. The mini browser like window was scaled perfectly too. Too bad I don’t like evolution as it’s ready to go without any more configuration. I did install gnome-tweak-tool to decrease the text scaling factor just a tiny bit. The only tweak I had to do was install a dpi support plugin for Firefox. For the most part things size correctly. Google Chrome has an notable lack of high dpi Linux support despite ChromeOS being Linux but can set the default zoom to compensate. Screenshot from 2013-12-18 23:39:19

Fedora vs Ubuntu

Fedora has been mostly stable so far. The bug report tool hides in the Gnome Messaging center unlike Ubuntu that displays it more prominently leading to much panic from some users I’ve encountered. Screenshot from 2013-12-18 23:45:59I like Gnome 3.10 over Unity in other ways. While I hate my defaults it’s a easy to install extensions. Here’s what I use. Unity has equally annoying defaults but doing something as simple as changing the default tab behavior to the way God intended is difficult and buggy even!

So far I would say I’ve experienced less bugs than Ubuntu but using Fedora for a few days is hardly a scientific study.

Touch Support

When I say touch support I mean gestures that are intuitive and functional. Sliding a finger to scroll for instance. Emulating a mouse is not touch support.

Gnome has the best touch support I’ve seen in (non-google) Linux  and it sucks. Some but not all of the gnome apps actually support touch out of box. I can scroll around in Nautilus to my heart’s content. I can launch applications too. But no browsers support it – making is 90% useless. Not even Gnome’s browser epiphany supports it.

Touchegg is a poor man’s touch support. It works sometimes…if you can manage to install it. I had to install from source and find outdated rpms on http://rpm.pbone.net/. If you aren’t an expert I would plan at least an hour or more to get it running. Anyway I can now scroll with two fingers in Firefox and Chrome. I disabled the other gestures because they don’t work.

Resizing windows using touch is still impossible.

Bad Stuff

  • Headphones autodetect is disabled by default. Fix – Run alsamixer, select sound card, HDA Intel, press right until selecting HP/Speaker Auto Detect, enable by pressing “m”
  • Suspend fails sometimes. I have no fix. In the 8 years of using Linux I’ve never had a laptop that works reliably with suspend. Sometimes the mouse gets stuck in a specific location. Sometimes it just turns off.
  • Wifi – Ubuntu and Fedora both have issues connecting to the tether wifi on my phone. ChromeOS connects fine. Sadness.
  • Most people who support Linux mean that they support Ubuntu and maybe it works elsewhere. Take Dropbox for example – download the rpm from their site and it will break updating because it installs a non existent repo. Easy fix for someone experienced. For the casual user I could see this just bricking updates.

Should you install it?

Are you a casual Chromebook user looking for extra functionality? Go crouton – it’s a lot easier to set up and you keep the more stable ChromeOS around.

If you’re an experienced Linux user and thinking about which distribution to try – Fedora makes a fine choice. Especially if you want the latest Gnome version.

Django 1.6 urls.defaults compatibility work-around

ImportError: No module named defaults

Getting this error on your under-maintained third party django apps? First go bug the maintainer with a pull request removing the defaults in

from django.conf.urls.defaults import *
# should be
from django.conf.urls import *

After that you can work around the issue if you don’t feel like forking the app. Just copy their urls.py file into your application then import that urls file instead.

For example let’s say you want to use django-ckeditor. Copy their urls.py file into your project. Maybe call it ckeditor_urls.py and remove “defaults”

Now in your project’s url.py change:

(r'^ckeditor/', include('ckeditor.urls')),
# change to
(r'^ckeditor/', include('mystuff.ckeditor_urls')),

Once the 3rd party app fixes their app you can just revert this change.

David Burke's blog