Technology Against You

Blog

  • Django and OpenOffice.org uno reports

    Openoffice.org’s python uno library is great for report writing in a web application. I wanted to create a report template system where users can create templates with a word processor of their choice. I created a simple template model which consists of a CharField and FileField. Users can upload templates here in .doc, .odt, or any format OpenOffice.org supports.

    Next I created some generic template report functions.


    from django.http import HttpResponse
    from django.core.servers.basehttp import FileWrapper

    import uno
    import os
    import string
    import tempfile

    def findandreplace(document, search, find, replace):
    """This function searches and replaces. Create search, call function findFirst, and finally replace what we found."""
    #What to search for
    search.SearchString = unicode(find)
    #search.SearchCaseSensitive = True
    #search.SearchWords = True
    found = document.findFirst( search )
    if found:
    print 'Found %s' % find
    while found:
    found.String = string.replace( found.String, unicode(find),unicode(replace))
    found = document.findNext( found.End, search)

    def replace_report(infile, outfile, data):
    """Replace words in a file use like this
    data={}
    data['$TEST']='worked yay'
    returns a django HttpResponse of the file"""
    # Boring uno stuff
    local = uno.getComponentContext()
    resolver = local.ServiceManager.createInstanceWithContext("com.sun.star.bridge.UnoUrlResolver", local)
    context = resolver.resolve("uno:socket,host=localhost,port=2002;urp;StarOffice.ComponentContext")
    desktop = context.ServiceManager.createInstanceWithContext("com.sun.star.frame.Desktop", context)

    # open document
    document = desktop.loadComponentFromURL("file://" + str(infile) ,"_blank", 0, ())
    cursor = document.Text.createTextCursor()

    search = document.createSearchDescriptor()
    #Do a loop of the data and replace the content.
    for find,replace in data.items():
    findandreplace(document,search,unicode(find),unicode(replace))
    print find,replace

    # create temporariy file to store document in
    tmp = tempfile.NamedTemporaryFile()
    document.storeToURL("file://" + str(tmp.name), ())
    document.dispose()

    # create http response out of temporariy file.
    wrapper = FileWrapper(file(tmp.name))
    response = HttpResponse(wrapper, content_type='application/odt')
    response['Content-Length'] = os.path.getsize(tmp.name)
    response['Content-Disposition'] = 'attachment; filename=' + outfile

    return response

    In this example I’m replacing text in a file. The result is returned as a http response. I can use it like this


    template = Template.objects.get_or_create(name="My Selected Template")[0]
    data={}
    data['$date'] = str(date.today())
    return replace_report(template.file.path, "file.odt", data)

    In this case I am taking out my template and replacing all instances of the word “$date” with the current date. I choose to use $ to mark it as a variable but it could really be anything. Now when a user wants to change a font they can do it themselves.

  • Linux and Active Directory round 2

    * edit Jan 12 2010. Better more reliable scripts.

    * edit Oct 27th The netbook deployment went well this time. Feedback is generally that they have less problems than our Windows machines. Using Impress instead of MS powerpoint seems to be the biggest issue as Impress has trouble importing some powerpoint files in the office open XML format. Next on my todo list is play around with Unison file sync, NFS, and update this guide to reflect some improvements a colleague made.

    The goal here is to create typical Windows Active Directory connect like Linux workstation. This includes being able to log in using Active Directory and mount various shared folders. It also must be idiot proof. We don’t want users saving on the desktop not realizing the desktop is not part of their smb share. It also must be cloneable. There should be absolutely no required interaction with the the computer after putting on this image.

    Active Directory Integration

    I choose to use Centrify for this. Likewise open is another option, but it seemed more buggy and I hate the way you have to configure it.

    Ok make sure partner repositories are enabled and install centrifydc. If you want to make a clone-able image set your host name as something generic like “stockimage”. Add this script to crontab’s @restart

    #!/bin/bash
hostCurrent=$(hostname)
hostOld='image'
commonauth='/etc/pam.d/common-auth'
if [ "$hostCurrent" == "$hostOld" ]
then
    (
    set -x
    host1=$(/usr/sbin/dmidecode | grep 'Serial Number: ' | sed 's/.*: (.*)/1/;q')
    host2=
    host=$host1$host2
    host=$(echo $host | sed 's/[ ]*//g')
    hostname $host
    echo $host > /etc/hostname
    # TODO: axe this ugly hack and have upstart call us when we're connected
    counter=0
    while [ $counter -lt 60 ] && [ ! `/sbin/route -n | sed -rn 's/^0.0.0.0[ ]+([0-9.]+)[ ]+0.0.0.0.*/1/p'` ]
    do
        sleep 1
        counter=`expr $counter + 1`
    done
    # Do NOT put regular administrator password here!
    # Use a special account and keep it DISABLED.
    /usr/sbin/adleave -u 'j' -p 'secret'
    /usr/share/centrifydc/bin/centrifydc stop
    /usr/sbin/adjoin -f -u 'j' -p 'secret' -w --name $host youradserver.com
set +x
    ) >& /opt/ad.log
    /sbin/reboot
elif [ "`sed 1q $commonauth | grep '^# lines inserted by Centrify'`" ]
then
    # Prevent double password prompt
    pammount=`sed -rn '/^auth[ ]+optional[ ]+pam_mount.so$/p' $commonauth`
    sed -ri '/^auth[ ]+optional[ ]+pam_mount.so$/d' $commonauth
    sed -ri 's/^(auth[ ]+sufficient[ ]+pam_centrifydc.so)$/1 try_first_pass/' $commonauth
    sed -i  "1i\$pammount" $commonauth
    /sbin/reboot
fi

    That script will rename the hostname to something unique and join your domain. Because it’s on @reboot it runs every time the computer is turned on thus when you image it, it runs! Maybe put this in crontab last because you don’t want it running on your image. If you’re only joining one machine to AD then just run the adjoin command.

    Ok bug work around time! I said Centrify is less buggy remember. This is only important for multi user machines, if there will only be one user you may skip this section. Ubuntu will boot well before you are online and thus a new user will get Authentication Error when trying to log in right away. To fix this first of all make sure if you use wireless that “enabled for all users” is checked in NetworkManager.

    Next we need GDM to wait for centrify to connect before starting. We can do this with upstart.

    Edit /etc/init/gdm.conf and look for the start on section. This basically tells GDM to only start once certain things have happened. Add centrify-connected to the bottom as seen here.

    start on (filesystem
and started dbus
and (graphics-device-added fb0 PRIMARY_DEVICE_FOR_DISPLAY=1
or drm-device-added card0 PRIMARY_DEVICE_FOR_DISPLAY=1
or stopped udevtrigger)
and centrify-connected)

    Now it we need to make the centrify-connected signal. Edit /etc/init.d/centrifydc and look for case “$CMD” in start). Add “initctl emit centify-connected” under wait_adclient  Like this.

    start)
adclient_check
echo -n "Starting $NAME: "
start-stop-daemon --start --quiet --exec $DAEMON --pidfile $PIDFILE 
-- $OPTIONS
RETVAL=$?
if [ $RETVAL -eq 0 ]; then
echo "OK"
wait_adclient
# upstart won't start gdm until we say we're connected
initctl emit centrify-connected  # added
else
echo "FAIL

    That emits the signal telling GDM that Centrify is connected. The script is part of Centrify, it just didn’t emit the signal. It’s like they thought of this problem but didn’t actually fix it.You could reboot now if you wanted and it should work, but there’s much more we can do.

    If your splash no longer shows up (happens to me 100% of the time on 10.04). You can try this command to fix it. I add this note because it can take a while for wifi to connect, then centrify to find AD, then GDM to start. If the users stares at the black screen for 1 minute they will probably assume the computer is just broken.

    sudo -i
    echo FRAMEBUFFER=y > /etc/initramfs-tools/conf.d/splash Code:
    update-initramfs -u

    Mount Windows Shares

    Now your users will fire up nautilus and start browsing windows shares. smb://something.company.com/user$/myuser/documents is so easy to remember right? 😛

    Lets face it users don’t know what a file share is usually. Lets mount them automatically for them like Windows does.

    sudo apt-get install libpam-mount smbfs

    Now edit /etc/security/pam_mount.conf.xml

    make it looks something like this in the section <pam_mount> Make sure to change it for your own purposes. In this example I’m mounting a user’s documents folder.


    <pam_mount>

    <!-- Volume definitions -->

    <volume user="*" fstype="cifs" server="server" path="users/%(DOMAIN_USER)" mountpoint="~/Documents" />

    Don’t try mounting anything as Desktop because gnome won’t like it. There is a workaround here if you really need Desktop to be part of the share. First turn off the auto-creation of these folders (this puts you in charge of making them! Nautilus will not start if it doesn’t have these folders! Yes Nautilus sucks.) Edit /etc/xdg/user-dirs.conf and set enabled to False. Now edit /etc/skel/.profile and add the lines

    mkdir "$HOME/Documents/Desktop" 2> /dev/null
ln -s "$HOME/Documents/Desktop" "$HOME/Desktop" 2> /dev/null
mkdir "$HOME/Documents/Downloads" 2> /dev/null
ln -s "$HOME/Documents/Downloads" "$HOME/Downloads" 2> /dev/null

    This would symlink the Desktop and Downloads folder to be inside of Documents. Documents in actually a smb share. This should keep users saving files in places that are in smb.

    There are two Ubuntu bugs related to smbfs. If your system hangs at logout look here

    https://bugs.launchpad.net/ubuntu/+source/libpam-mount/+bug/574329

    If it hangs at shutdown look here

    https://bugs.launchpad.net/ubuntu/+source/samba/+bug/211631

    Ok another Centrify bug work around is explained here Centrify goofs up pam_mount. I mean who would really want to connect to shares AND use active directory  😉  If you can’t handle clicking

    1) Open /etc/pam.d/common-auth file

    2) Check if “auth optional pam_mount.so” is the first line and “auth sufficient pam_centrifydc.so” second line. If “yes” then change the second line to:

    auth       sufficient     pam_centrifydc.so try_first_pass

    I also found that adding try_first_pass to the pam_unix.so line will allow non AD users to log in without entering the password twice.

    Odds and ends

    If you want to give some users sudo edit /etc/sudoers and add

    %ADMIN\UnixAdmins ALL = (ALL) ALL

    That command would give the UnixAdmins group in the domain ADMIN sudo access.

    If you want to edit the default desktop install a program called sabayon. It crashes sometimes but when not crashing it works pretty well. For some reason rebooting fixes it’s crashing…weird.

    Another problem is that non admin users can disable network manager for the entire system even after a reboot! I’m not sure what to do about this, it’s a major headache because users won’t be able to log in at all without networking. I can’t think of any fix other than disabling network manager, but this is not ideal as sometimes there are legitimate uses for network manager.

    As Mike pointed one could use the adjoin –selfserve command if AD knows the hostnames in advance. I choose to use an account that is usually disabled so if users see the password, they won’t be able to do much.

    Compared to Windows this is a huge pain the first time to get right but cloning is way easier. I love running one image on many different models of computers. I moved the image around to several computers fine tuning it for each one. That included installing specific drivers. In the end I have one image that can be deployed anywhere.

    I’m still missing a few must have features such as syncing the entire home folder to a share. NFS home folder is not enough, try leaving the building with your NFS home folder laptop. iFolder is an option I want to look into more. My initial experience is that it’s difficult to configure. It also doesn’t give you a CIFS interface which is often nice to have. There are any number of hosted solutions (Dropbox, JungleDisk, Ubuntu one, etc) that might work for you. It’s a shame Canonical won’t offer this as an on site solution like they do Landscape. Sparkle Share is an upcoming project that aims to meet this need but it’s not ready yet. You could also look into CMS software such as Alfresco.

  • Creating a custom Django DateTimeField model

    In my last post I ran into a problem. A legacy database stores values as BigInt in MySQL using Unix time. Django uses datetime. I want the nice validation and widgets from Django’s DateTimeField. Here’s how to solve in.

    Create a new field, I called it UnixTimestampField and overrode models.DateTimeField.

    from django.db import models
    from datetime import datetime
    from time import strftime, mktime
    import time

    class UnixTimestampField(models.DateTimeField):
    __metaclass__ = models.SubfieldBase

    def __init__(self, null=False, blank=False, **kwargs):
    super(UnixTimestampField, self).__init__(**kwargs)

    def db_type(self):
    typ=['bigint']
    if self.isnull:
    typ += ['NULL']
    return ' '.join(typ)

    def to_python(self, value):
    super(UnixTimestampField, self)
    try:
    return datetime.fromtimestamp(float(value))
    except:
    return value

    def get_db_prep_value(self, value):
    if value==None:
    return None
    return time.mktime(value.timetuple())

    def get_prep_value(self, value):
    if value==None:
    return None
    return time.mktime(value.timetuple())

    It seems to work. I’ll update though if I run into any trouble down the line or re-factor it. This is my first attempt at making a custom Django Model Field.

  • Django models from non Django applications

    I have an existing LAMP application that I want to write some quick code for. I want to easily manipulate the database without taking the time to understand how the application works fully and mess around with PHP. The solution, create a Django model to act as a wrapper for the sections of the database I need.

    First I created some very simple views in MySQL to my new Django created database. I could have just put both programs in the same database but this should make updates easier. Remember in MySQL views are updatable only when the view references one table. All my views look like
    create otherdb.my_view as select * from newdb.some_table;
    With this method I can easily drop and replace the other database when I update it.

    Next I run Django’s inspectdb command to get some generic Django model code. Edit this to one’s liking. For me that meant renaming fields, deleting the primary key (so Django automatically makes it. Only works if it is in fact a primary key auto incr integer). Turn foreign keys into foreign key fields. Make dates into DateFields.

    Setup my Django admin interface and all the sudden I can use the spiffy Django admin interface on a completely different application. YAY!

    However I ran into a small problem. I’ll post this separately for people who only care about Creating a custom Django DateTimeField

  • Recipe

    I’ve decided to vary things up and post a recipe I made up, expect more to come as well as the normal mostly Django and Linux related posts.

    Amazing Vegetarian Sandwich

    • Tofu
    • Fresh veggies (tomato, peppers, anything)
    • Bread
    • Curry
    • Flour
    • Cheese
    • Butter or Oil

    Usually non meat sandwich’s suck and aren’t filling, but this sandwich is bad ass and will make the meat eaters jealous. First dry the Tofu the best you can. Slice it, powder it with flour, then fry it in butter or oil as fast and hot as you can. Sprinkle some curry on it while it cooks. Fry up some veggies if desired too or put them on uncooked. Finally add some Italian dressing or olive oil and pepper. Swiss cheese is good for sandwiches but any kind would work. These sandwiches are great for a picnic and bringing leftovers to work the next day.

  • Introducing SWORD

    Student WOrker Relational Database (SWORD) is a student worker database for use in schools with work study programs. It’s the project I’ve been working on for the past 6 months or so.

    http://code.google.com/p/student-worker-relational-database/

    It’s a great example of leveraging Django to create a full website in very little time (Until recently it was just a side project I worked on occasionally) . I’m releasing it open source to help distribute it for use in other schools. Right now the installation process is a little ugly though. I’ll have to learn how to make Debian  and RPM packages sooner or later. Next steps will be to integrate it with some other programs, possibly Schooltool and  SugarCRM. It’s already working with CAS for Single Sign On. If you know any schools that might be able to use this please comment or email. I may be offering it as a hosted solution soon.

  • Thoughts on Centrify

    Centrify Express is a gratis but not open source client for integrating various platforms (such as Linux and Mac) to active directory. I wrote about Likewise Open before when trying to get Linux on Active Directory with some success. Today I deployed Centrify on a few machines to see if it could fare any better.

    Ubuntu 10.04 Server: Centrify worked very well for this. Likewise open has many bugs that annoy me in Ubuntu so that’s what really motivated me to try something new. I ran into a little trouble installing it, but after running apt-get -f install it worked fine. Likewise in Ubuntu has a default domain bug and it also doesn’t seem to like installing on my Proxmox template for some reason. Of course Centrify isn’t (and probably never will be) in Ubuntu’s repositories which make it slightly harder to install.

    Ubuntu 10.04 Desktop: Desktop support is always where I get problems in trying to make linux work in Active Directory. The biggest annoyance is getting it to work in a wifi environment. Turn on wireless laptop, new user logs in, authentication failure. It has to do with network-manager not really connecting as soon as you think it would (even if available to all users is checked off). This is what really kills Linux for me in a place where people need to use different computers (ie school). Removing network-manager fixes it somewhat, but I still have issues where I have to wait a few minutes before it allows domain logins. It does at least appear that Centrify works as soon as network is available while likewise seems to just sit for a couple minutes before it works. Centrify however is missing some features that are a great in a company network. Likewise will pass on login information to pam and when connecting to samba shares. Centrify does not (edit, it can be made to work with pam_mount with some work arounds). Also when I log in with GDM in centrify I have to enter the password twice for gnome-keyring to work then again every time I connect to a samba share. There is a “centrified” version of Samba that I wasn’t able to get installed. Likewise just works in this case.

    Centrify Pro’s:

    • Less buggy to set up
    • Assumes default domain
    • Mitigates wifi issue at least a little bit

    Centrify Con’s

    • Not open source
    • Not in Ubuntu repository
    • Must enter password twice on login for gnome-keyring when using GDM, though there is a work around.
    • Doesn’t pass credentials to Samba shares
    • Little documentation. * edit they do have a pdf manual but they don’t have the forums history likewise does. When I Google Likewise <description of problem> I get results, while with Centrify I really have to hunt.

    Once again Linux can integrate well with AD in server land but has issues for end user desktops. Workswithu did a article comparing the two programs too for those interested. Overall Centrify is an acceptable solution, but falls short of a Linux Active Directory integration that “just works.”

  • Django and Google Maps

    Last post I wrote about extending the Django Admin interface by adding a little dynamic data to the help_text field. For more advanced things you need to customize the html widget itself.

    One thing I want to do is have a Django model automatically get a Google Map of the address I have stored and save this file. For my purposes though I don’t want this to be automatic, I’d rather an admin user check the map first then decide to overwrite any previous map file. I’ll extend the a widget to get this effect.

    First I need a preview box. I want to insert some extra html, I’m actually going to put this into a checkbox form, you will see why later. I need to override the render def on the widget to add some code.


    class MapImageWidget(forms.CheckboxInput):
    def render(self, name, value, attrs=None):
    output = []
    output.append(super(MapImageWidget, self).render(name, value, attrs))
    output.append("If checked, the above map file will be overwritten with Google Maps.<table><tr><td><a href="javascript:get_map()">Preview Google Maps</a></td></tr><tr><td> </td></tr></table>")

    return mark_safe(u''.join(output))

    I’m using Google Maps static maps url to get my previews. I made a javascript function called get_map to look up the url based on the address field on my model. I want it to actually use the currently typed in address rather than some ajax solution which may get the currently saved address.


    function get_map(){
    var address = document.getElementById("id_address").value;
    var city = document.getElementById("id_city").value;
    var state = document.getElementById("id_state").value;
    var zip = document.getElementById("id_zip").value;
    parent.mapframe.location="http://maps.google.com/maps/api/staticmap?sensor=false&size=400x400&markers=size:mid|color:red|" + address + "," + city + "," + state + "," + zip;
    }

    I put the javascript into my change_form.html admin template. This function is only for previewing by the user and has nothing to do with actually saving my map image. For that I have a checkbox as in this screen shot

    Maps preview in admin edit page

    When the user checks off the box, Django will know to get that image and set it to the map ImageField. I do this by overriding the save function on my model.


    def save(self, *args, **kwargs):
    if (self.use_google_maps):
    self.use_google_maps = False;
    image = urllib.urlretrieve("http://maps.google.com/maps/api/staticmap? sensor=false&size=400x400&markers=size:mid|color:red|" +
    self.address + "," + self.city + "," + self.state + "," + self.zip)
    self.map.save(self.company_name + "_map.jpg", File(open(image[0])))
    super(Company, self).save(*args, **kwargs)

    I use urllib to “wget” the file. This process will replace the previous practice of manually looking up maps, saving them, and uploading them. Of course if I trusted Google maps (or my addresses to be correct) I could just have it dynamically get that image when needed instead of storing the image file.

  • Django Hack: adding extra data to admin interface

    A common need I have for Django’s admin interface is to show a little more data for convenience right on the edit page. For example showing a link to a foreign key’s edit page right there. The way I do this is by setting the help_text field in the render_change_form function. I create a new function in my admin class to override render_change_form

    class whateverAdmin(admin.modelAdmin):
     def render_change_form(self, request, context, *args, **kwargs):
      context['adminform'].form.fields['someField'].help_text = "Go to edit page " + str(context['original'].anyFunction()) + " (will discard changes)"
      return super(whateverAdmin, self).render_change_form(request, context, args, kwargs)

    the anyFunction() is just a function I made to display a URL in my model. Notice the allow_tags line to allow the function to return the html <a> tag

    def anyFunction(self):
     try:
      field = self.someField
      urlRes = urlresolvers.reverse('admin:appName_someField_change', args=(field.id,))
      return '</a><a href="http://example.com' + urlRes + '">' + str(field) + '</a>'
     except:
     return ""
    anyFunction.allow_tags = True

    This link is then very convenient when using the admin interface just to look up information. The render_change_from function is also useful to editing about the admin page. I use it to modify queryset’s for foreign key data as well.

  • new domain!

    I’ve decided to get a domain and move to thetspc.com for my blog.  I’m using thetspc.com for some other projects more exciting things to come soon with that.  thetspc.com can host a blog for $5/month which is useful for features like domain mapping. Goodbye wordpress.com

    The new blog is actually running wordpress mu (multi user) on webfaction.

    webfaction

    I’m fairly pleased with their hosting. It lets you do basically whatever, it’s as close as one can get to a virtual private server with just cheap old hosting. Of particular interest for me they have a Django installer and unlimited websites. If your looking for hosting use the link above so I get a little money.