A non-production function decorator

As most developers know, not every piece of code is meant to be run on a production server. Instead of using a lot of "ifs" here and there I suggest implementing a framework specific "non_production" decorator. A simple django-specific implementation could look like this:

def non_production(func):

    def is_production():
        #django specific 
        from django.conf import settings 
        return getattr(settings, "PRODUCTION", False) 

    def wrapped(*args, **kwargs):
        if is_production():
            raise Exception("%s is not meant to be run on a production server" % \
                    func.__name__)      
        else:
            return func(*args, **kwargs)
    return wrapped

Now all you have to do is to apply it to your dev/test only functions:

@non_production
def test_something(a, b):
     pass

Cheers!
KR

The forloop django tempalte variable

Django template for-loop iteration is usually executed on QuerySets (paginated or not). For front-end purposes it's sometimes good to provide row numbers (especially if the data is supposed to be presented in a specific order).  A first non-django solution that comes my mind is something like this:

def test_view(request):

    ctx = {}

    objects = MyModel.objects.all()

    ctx['objects'] = zip(range(1,objects.count()+1), objects)

    return render_to_response("app/test.html", ctx)

And for the template:

<table>

{% for el in objects %}

    <tr>

        <td>el.0</td>

        <td>el.1.some_field</td>

        {% comment %} Other fields.... {% endcomment %}

   </tr>

{% endfor %}

</table>

This is a great python solution, but it's not a django solution. First of all, by using zip function we select the records from the database, adding pagination now would be a bit complicated (using the PaginationMiddleware won't be effecient since it works well only on lazy QuerySets). Another Disadvantage is the need of addressing the counter and data by index. Of course we could make the counter an attribute of the data objects... but it's still not THE solution.

Django provides a better way of solving this problem. Inside each for-loop you may access a forloop template variable. According to the django docs the following attributes are available:

forloop.counter	The current iteration of the loop (1-indexed)
forloop.counter0	The current iteration of the loop (0-indexed)
forloop.revcounter	The number of iterations from the end of the loop (1-indexed)
forloop.revcounter0	The number of iterations from the end of the loop (0-indexed)
forloop.first	True if this is the first time through the loop
forloop.last	True if this is the last time through the loop
forloop.parentloop	For nested loops, this is the loop "above" the current one

Now the template could look like this:

<table>

{% for el in objects %}

    <tr>

        <td>{{forloop.counter}}</td>

        <td>{{el.some_field}}</td>

        {% comment %} Other fields.... {% endcomment %}

   </tr>

{% endfor %}

</table>

This is more elegant and practical. It's also easy to combine it with the PaginationMiddleware. Ale you need to do is add forloop.counter with each page start index.

Cheers!
KR

Generic django forms

My last post presented a way of generating dynamic classes in python. Today I would like to present a way of applying this mechanism to Django forms, making them a bit generic. An example may be a login form, that may be used with an "accept regulations" checkbox or without it, depending on the context.

class LoginForm(forms.Form):
    username = forms.CharField(max_length=20, label="Username")
    password = forms.CharField(widget=forms.\
            PasswordInput(render_value=False),max_length=20, label="Password")

    def clean(self):
        #form validation/clean implementation
        pass

This a simplest login form you can get, all there is to do is to implement a login mechanism (or simply use it, it's available at the django.contrib.auth module). Both provided fields are required, so the is_valid method will generate an appropriate result. Now let's make a possibility to generate this form with an "accept regulations" which is also required.

class LoginForm(forms.Form):
    username = forms.CharField(max_length=20, label="Username")
    password = forms.CharField(widget=forms.\
            PasswordInput(render_value=False),max_length=20, label="Password")


    def clean(self):
        #form validation/clean implementation
        pass


    @classmethod
    def with_accept_regulations(cls):
        return type(
            "%sWithAcceptRegulations" % cls.__name__,
            (cls,),
            {"accept_regulations" : forms.BooleanField(required=True, \
                    label="I accept the regulations")},

Now we may easily obtain a login form with accept regulations using the with_accept_regulations class method. Example usage is presented below:

if something:
    CurrentLoginForm = LoginForm
else:
    CurrentLoginForm = LoginForm.with_accept_regulations()


form = CurrentLoginForm(request.POST)
if form.is_valid():
    #and os on...
    pass

This example is trivial, and a similar outcome may be achieved by using simpler mechanisms, however examples are suppose to be simple. This method may be applied to more demanding applications, a huge feature is the ability to chain class creation methods, for example you could try implementing the following:

LoginForm.with_accept_regulations().with_skin_selection(), which would generate a class LoginFormWithAcceptRegulationsWithSkinSelection. I know this is starting to look like java, but the long names are meant to be descriptive :-)



Have fun exploring new possibilities.

Cheers!
KR

P.S.
I was in a hurry, so the code may have some simple mistakes - beware :-)

Django admin: handling relations with large tables

One of the reasons, that Django became so popular, is that it provides a set of flexible administrative tools. You may setup your data management dashboard in relatively short time using some predefined classess.

A typical object admin view consists of a ModelForm of an appropriate model, and some basic actions (save, add, delete).  As you probably know, foreign keys are often represented as HTML select inputs. This works cool, when the related table is not vertically long, for example: electronic equipment brands. In such cases the combobox works well.

Imagine a situation when, the foreign key points to a user record,  a purchase order, or any other relation that does not have a limited number of rows. According to the previous example, the combo box should have thousands of options. This is a real performance killer, the ORM has to fetch the whole user/purchase order table in order to display a single object in an editable admin view. Don't be surprised if your query gets killed with the "2006 - MySQL server has gone away" status.

There is a simple way to solve this: instead of selecting the whole table for presenting available options, we may present only the currently related object (its primary key). To achieve this, we must mark the foreign keys as raw_id_fields. Below sample usage:

 

class PurchaseReportAdmin(admin.ModelAdmin):                                                                   
    #(...) other fields                                                                                 
    raw_id_fields = ['purchase_order']
admin.site.register(PurchaseReport, PurchaseReportAdmin )

Yes, it's as simple as that. Keep in mind, that in many cases using raw id fields won't be necessary, but when it come to vertically huge tables - it's a good time/performance saving solution.

Cheers!
KR

Determine Django form field clean order

Django provides some excellent out-of-the-box form validation/processing mechanisms. These include casual forms, model based forms, and formsets. One of their features is the ease of implementing data validation procedures. According to the documentation, the validation process includes executing the following methods in the presented order:

1. to_python (field)
2. validate (field)
3. run_validators (field)
4. clean (field)
5. clean_<fieldname> (form)
6. clean (form)

Since there is usually no need to extend form fields, let's focus on the form-side validation. Let's suppose you have the following forms:

class MyBaseForm(forms.Form):
    base_field1 = forms.CharField()
    base_field2 = forms.CharField()

    def clean_base_field1(self):
        #(...)
        return self.cleaned_data['base_field1']
       
    def clean_base_field2(self):
        #(...)
        return self.cleaned_data['base_field2']

class MyForm(MyBaseForm):
    field1 = forms.CharField()

    def clean_field1(self):
        #(...)
        return self.cleaned_data['field1']

All methods implemented in this example refer to step 5: clean_<fieldname>. So what would be the execution order if we try validating MyForm? Django manual states:

"These methods are run in the order given above, one field at a time. That is, for each field in the form (in the order they are declared in the form definition), the Field.clean() method (or its override) is run, then clean_<fieldname>(). Finally, once those two methods are run for every field, the Form.clean() method, or its override, is executed."

According to this statement, the expected order would be: clean_base_field1, clean_base_field2, field1. What if we don't like this order, should we rearrange the form definition? No such thing! There is a way change this order in a more elegant way. We may use fields.keyOrder to achieve it:

class MyForm(MyBaseForm):
    field1 = forms.CharField()

    def clean_field1(self):
        #(...)
        return self.cleaned_data['field1']

    def __init__(self, *args, **kwargs):
        super(MyForm, self).__init__(*args, **kwargs)
        self.fields.keyOrder = ['base_field1', 'field1', 'base_field2']

You don't need to extend a form to use this. You may also specify a partial order (if there are to many fields to be named explicitly) by putting only a few field names in the array and extending it by a list comprehension generated from the remaining self.fields.keys(). 

Cheers! 

~KR
   

Passing django variables to javascript via templates

I can fairly say, that reasonable web applications can't exist without a bit of javascript. We all remember those static web pages that were popular in the late 90'. Well their time is up, professional websites frontend is built using HTML/XHTML/HTML5 + CSS + JS (sometimes Flash).

AJAX requests or newer synchronous pull/push methods enable achieving some dynamic connectivity between frontends and backends. It is also possible to pass some initialization data this way, however if your website architecture is not a real-time client-server (ex. WebSockets) this may be inefficient.

A good practice is initializing JS variables along with rendering pages, so additional initialization request are not required. A simple way of implementing this mechanism is passing JSON encoded data to the template and decoding it in JavaScript. Lets implement the appropriate django view:

def init_js_template(request):
    import json
    json_data = json.dumps({'test' : '12345', 'arr' : [1,2,3]})
    return render_to_respons('init_json.html', {'json_data':json_data})

And the corresponding javascript initialization in the init_json.html template:

This way you will save some resources by reducing the number AJAX calls. Just keep in mind that this will only work for arrays/dictionaries containing primitive types or other arrays.

~KR

Potential problems with custom primary keys in django ORM

I had a strange situation lately that encouraged me to do some research about how django really handles primary keys.

As for a mature web framework django's ORM generates automatic integer primary keys if none are present. This feature is cool, cause we are relieved from generating our own primary keys (if its problematic), but we loose some easily accessible semantic information - for example: a car's primary key may be it's registration number instead of a meaningless integer. Whats more, a primary key is used in the default object __eq__ / __cmp__ functions, which makes it easier to filter data (no need for subselects).


But let's look at this situation:

class Dude(models.Model):

    name = models.CharField(primary_key=True, max_length=100)

    #and so on

We have a 'Dude' that should have a unique name, primary keys are in general unique so there's no problem here. Suppose we have some form where you add 'dudes'. So we execute:

try:

    Dude(name='John', **some_fields).save()

    #some more code

    Dude(name='John', **some_other_fields).save()

    print "You don\'t expect to see this message"

except IntegrityError as ie:

    pass

The logic states, that if you try to create another entry with a same value of a unique field, django should raise an integrity error. Well it will not, because this is also a primary key, if a 'dude' with a specific name exists, django interprets it like this:

Dude.objects.filter(name='John').update(**some_other_fields)

For me this was a bit unintuitive, but when think about it - this approach makes sense. All in all, I would recommend using a custom primary_key field only if there is some semantic/logical background behind it.

~KR

Django "is substring" like filter (MySQL)

Recently I came by a problem of selecting rows from a table on a condition that one of the fields is a substring of a given phrase. It's a bit hard to explain, but anyway I wanted to achieve the equivalence of:

filter(lambda x: "some phrase".find(x.some_field) >= 0, MyModel.objects.all())

on the SQL/ORM level.


So I searched the django documentation again and again and I failed to find anything useful. If the problem cannot be solved on the ORM level it must be solved with raw SQL:

SELECT * FROM myapp_mymodel WHERE "some phrase" LIKE CONCAT('%',some_field,'%');

Since the presented where clause connot be generated using the QuerySet.filter method we have to use the extra method instead. A django equivalent would look like this:

MyModel.objects.extra(where=["%s LIKE CONCAT('%%', some_field, '%%')"], \
    params=["some phrase"])


We should remember that the SQL LIKE operator is case insensitive (in this case it is desirable), however if you want a case sensitive filter, try using LIKE BINARY  instead.


~KR

Interactive debugging in Django

A long time ago in a galaxy far far away... programmers were debugging PHP applications using the echo function. Well it actually wasn't that long ago nor was the galaxy far away - we're talking about the late 90'; planet Earth.

Nowadays it is rather unthinkable, time is money and spending whole days placing and removing echo's is expensive (not mention ineffective). So how to efficiently debug Django views?

The first thing you need to do is set the following variable in settings.py:
   
   DEBUG = True
 
This option is enabled by default. If it is set, each time a view returns a HTTP 500 Internal Server Error, a debug view will be presented. It contains your current settings, request parameters and finally a stack trace - usually this is enough to solve most problems.

If it is not enough you may want to try the second option: pdb the standard python debugger tool. To insert a break point you should put the following line in your code and start the application via runserver:

    import pdb ; pdb.set_trace()

If you master the short-cut commands this is a very pleasant tool. You can also check out the interactive version ipdb (requires ipython).

However if you do not feel like debugging in the console, or you want to have access to the whole stack trace without inserting hundreds of break-points - werkzeug is the tool for you! It is an awesome interactive JavaScript based in-browser debugger. You can get it with pip (along with dependencies):

~ pip install django-admin-tools
~ pip install werkzeug

Now, instead of using the runserver command, you use the following:

~ ./manage.py runserver_plus

After this, each time you encounter an exception debug view appears... however this is no ordinary Django debug view, it contains an in-browser debugger like pdb which is capable of jumping between every point of the stack trace. This is just great, if I could also integrate vim with FireFox aswell... *kidding* :-)

~KR

Droping a multi-column unique constraint in MySQL with Django South

South is a great tool for managing database migrations (compatible with Django). It generates migrations by analysing the difference between the current data model and the previous one (stored in migration script files). However strange things may occur if you try do drop a multi-column unique constraint ( the django model defines it as: unique_together).  For example we have:


  1 class MyModel(models.Model):
  2     classs Meta:     
  3         app_label = 'myapp'
  4         unique_together = (('field1', 'field2',),)

We remove line 4, and run schemamigration:

~ ./manage.py schemamigration myapp --auto

South generates the following forward migration:

 23 class Migration(SchemaMigration):

 24     def forwards(self, orm):

 25         db.delete_unique('myapp_mymodel', ['field1', 'field2']) 

Let's try to execute it:

~ ./manage migrate kantor

And we get something like this:

ValueError: Cannot find a UNIQUE constraint on table myapp_mymodel, columns ['field1', 'field2']

We can investigate it using the mysql client:

mysql> SHOW CREATE TABLE myapp_mymodel;

(...)

UNIQUE KEY `myapp_mymodel_field1_667dc28f4f7b310_uniq` (`field1`,`field2`),

(...)

So the unique constraint really exists, but south fails to drop it. To solve this problem we have to drop that index ourselves. We should modify the forward migration the following way:

 23 class Migration(SchemaMigration):

 24     def forwards(self, orm):

 25         import south

 26         south.db.db.execute('drop index \
 27             myapp_mymodel_field1_667dc28f4f7b310_uniq on myapp_mymodel;')

That does the trick! Farewell multi-column unique constraints! 

~KR

Managing Django transacion processing (autocommit vs performance)

Django has some great ORM tools embedded that enable safe and simple methods of managing your database. There are also modules responsible for transaction processing, mainly TransactionMiddleware. By default it is present in the settings file, and I see no reason why it shouldn't - this middleware provides a very simple, yet powerful mechanism that considers your view processing as a single transaction. This pseudo-code  presents the main idea:

1  try:

2     start_transaction()

3     (your view code)

4     commit()

5  except:

6     rollback()


This is great, but since its a middleware module its not applicable to background processing. Instead the autocommit on save is used. This may be ineffective when you are processing large amounts of data (using celery, or some cron scheduled script). Each time you commit some changes, the DBMS has to perform some lazy tasks that usually require some I/O operations (recreating indexes, checking constraints etc.). This greatly increases the total processing time, yet if the process dies you still have some data... which tend to be useless without the missing data. So why not apply the above pattern to this problem? Django supports manual transaction management, all you have to do is use the commit_on_success or commit_manually decorators:

  1 from django.db import transaction 
  2                                   
  3
  4 @transaction.commit_on_success    
  5 def sophisticated_background_processing():
  6     #your code come here :-)
  7     #(...)                        
  8     pass
  9
 10 @transaction.commit_manually
 11 def some_other_background_processing():
 12     try:
 13         #your code
 14         #(...)
 15         transaction.commit()      
 16     except SomeException as se:   
 17         #handle exception
 18         transaction.commit()      
 19     except:
 20         #unhandled exceptions     
 21         transaction.rollback()      

The commit_on_success acts just like TransactionMiddleware for view, in most cases it will do. If we need some more flexibility we can always try the commit_manually decorator. It enables commiting/rollbacking data whenever you want. Just make sure all execution paths end with an appropriate action or django will raise an exception. 

Using manual-commit instead of auto-commit increased my accounting script performance about 5-10x, depending on the instance size (the processing is specific, and the data model is rather horizontal). 

Django view serving dynamically generated PDF files.

Serving static files is cool. However, static files have a drawback - mainly they tend to be static. I'm not saying that you should avoid serving static content or anything similar - static files have variety of important applications , which are not the topic of this post.

So what can you do when you want to serve a dynamically generated PDF file to the user? First of all you have to provide a package capable of generating such files, so unless you want to spend quite a few days implementing your own tool, you should try using ReportLab. ReportLab is quite powerful, but it takes a lot of effort to create a good layout. If you also want your PDF file to look sexy - you should choose pisa (it requires ReportLab as a dependency). Pisa is a HTML/CSS to PDF converter - which is just what we need (be warned - not all CSS styles are supported, but that's another history).

Let's have a look at this code:


  1 import ho.pisa
  2 import cStringIO
  3
  4 from django.http import HttpResponse
  5 from django.template.loader import get_template
  6 from django.template import Context
  7
  8    
  9 def generate_pdf(template_file, context={}):
 10     #to avoid using a temporary file StringIO has to be used
 11     pdf = cStringIO.StringIO()
 12     template = get_template(template_file)
 13    
 14     html_response  = template.render(Context(context))
 15    
 16     pdf_status = ho.pisa.pisaDocument(cStringIO.StringIO(html_response), pdf)
 17    
 18     if pdf_status.err:
 19         #catch it in the invoking view
 20         raise Exception('Oops! Something went wrong!')
 21     return HttpResponse(pdf.getvalue(), mimetype='application/pdf')
 
 This is a generic function that can render any template with an apropriate context and return a HttpResponse, this function may be used the following way:

 24 def generate_report_view(request):
 25     '''
 26         (...) some code
 27     '''
 28     return generate_pdf('reports/sample_report.html',\
 29         {'owner' : request.user, 'some_param' : 'Hello PDF Generator'})

There are two tricks here, that enable downloading the generated PDF directly from the view. Firstly, the cStringIO (faster version of StringIO) is used instead of a file, so that we do not have to make any HDD IO operations. Secondly we set the response mimetype to application/pdf, which informs a browser that this is not a regular site.

~KR

Search and replace many files based on regular expressions

In the past weeks our team has put a lot of effort into scaling our system, so it can handle greater amounts of traffic. A good way to decrease the load of the primary server is to serve static files from another machine. Django supports such mechanisms, one can specify the MEDIA_URL parameter in the settings file, which acts as a prefix when it comes to loading media files, including images, css and js files... well at least it should. As it occurred the template files (HTML) contained absolute URI paths. I decided this would be a great opportunity to modify those templates. The number of files that should be checked was rather high, and each of those files contained many media file references.

Here is where bash comes with a helpful hand. Since I did not intend to spend the whole morning on copy-pasting through hundreds of entries I wrote a script that does the magic thing for me:

  1 for f in $(find . | grep html$ | xargs egrep '"/media[^""]+[a-z]"'  | cut -d ":" -f1 | sort | uniq)
  2 do
  3     echo $f
  4     cat $f |  sed -r 's/(src|href)=\"\/media\/([^""]*\.[a-z]+)\"/\1=\"{{MEDIA_URL}}\2\"/g' > $f.tmp
  5     mv $f.tmp $f
  6 done

So what does it do? The script iterates over all html files found in subdirectories that have an absolute path starting with /media surrounded by quotation marks (the pipes ensures that each file is processed at most once). Line no. 4 is responsible for replacing the absolute path with a template variable. For example it changes:

(...) src="/media/some_path/some_image.jpg" (...), to
(...) src="{{MEDIA_URL}}some_path/some_image.jpg" (...)

Using the back references (groups) is essential, without it the search/replace context would be insufficient, which could result in modifying parts that don't refer to static media content. The first group covers the attribute (href or src), while the second group covers the file name.

Cheers!

A django login_required decorator that preserves URI query parameters

The default django auth module provides some awesome function decorators - login_required and permission_required. These decorators provide a flexible authorisation mechanism, each time a user tries to access a resource he is not permitted to view (modify or do anything with it) he is redirected to the login page. The user has a chance authenticate himself or provide new credentials (in case he was already authenticated but was lacking permissions), and if the authentication process is completed successfully the resource may be accessed.

In most cases using these decorators solves the problem of protecting resources while keeping the code (and the user interface) clean.

A problem occurs when we try to access a protected resource while attaching some URI query parameter, ex. http://example.com/resource/?foo=1&bar=example. We get redirected to the login page, and after providing our credentials we get redirected back to http://example.com/resource/ ... and the query parameters are gone!

Sadly the default login_required decorator does not preserve them... we have to provide our own decorator:

1. def resource_login_required(some_view):

2.     def wrapper(request, *args, **kw):

3.         if not request.user.is_authenticated():

4.             params = map(lambda x: "%s=%s&" % \

                  (x[0],x[1]), request.GET.items())

5.             return HttpResponseRedirect( \

                  "/login/?%snext=%s" % \              

                 ("".join(params),request.path))

6.         else:

7.             return some_view(request, *args, **kw)

8.     return wrapper

Line 4 is the key instruction, the presented lambda expression maps the parameter key-value pairs to an URI scheme query parameter representation. Next we concatenate the parameters with the original request path - this is it, after performing a successful login we should be redirected to the requested resource along with the request query parameters.

This decorator may by used just like the prevoiusly mentioned ones:

1.  @resource_login_required

2.  def my_view(request):

3.      #your view code

4.      pass

Feel free to adjust this decorator to your needs.