Potential problems with custom primary keys in django ORM

I had a strange situation lately that encouraged me to do some research about how django really handles primary keys.

As for a mature web framework django's ORM generates automatic integer primary keys if none are present. This feature is cool, cause we are relieved from generating our own primary keys (if its problematic), but we loose some easily accessible semantic information - for example: a car's primary key may be it's registration number instead of a meaningless integer. Whats more, a primary key is used in the default object __eq__ / __cmp__ functions, which makes it easier to filter data (no need for subselects).


But let's look at this situation:

class Dude(models.Model):

    name = models.CharField(primary_key=True, max_length=100)

    #and so on

We have a 'Dude' that should have a unique name, primary keys are in general unique so there's no problem here. Suppose we have some form where you add 'dudes'. So we execute:

try:

    Dude(name='John', **some_fields).save()

    #some more code

    Dude(name='John', **some_other_fields).save()

    print "You don\'t expect to see this message"

except IntegrityError as ie:

    pass

The logic states, that if you try to create another entry with a same value of a unique field, django should raise an integrity error. Well it will not, because this is also a primary key, if a 'dude' with a specific name exists, django interprets it like this:

Dude.objects.filter(name='John').update(**some_other_fields)

For me this was a bit unintuitive, but when think about it - this approach makes sense. All in all, I would recommend using a custom primary_key field only if there is some semantic/logical background behind it.

~KR

Managing Django transacion processing (autocommit vs performance)

Django has some great ORM tools embedded that enable safe and simple methods of managing your database. There are also modules responsible for transaction processing, mainly TransactionMiddleware. By default it is present in the settings file, and I see no reason why it shouldn't - this middleware provides a very simple, yet powerful mechanism that considers your view processing as a single transaction. This pseudo-code  presents the main idea:

1  try:

2     start_transaction()

3     (your view code)

4     commit()

5  except:

6     rollback()


This is great, but since its a middleware module its not applicable to background processing. Instead the autocommit on save is used. This may be ineffective when you are processing large amounts of data (using celery, or some cron scheduled script). Each time you commit some changes, the DBMS has to perform some lazy tasks that usually require some I/O operations (recreating indexes, checking constraints etc.). This greatly increases the total processing time, yet if the process dies you still have some data... which tend to be useless without the missing data. So why not apply the above pattern to this problem? Django supports manual transaction management, all you have to do is use the commit_on_success or commit_manually decorators:

  1 from django.db import transaction 
  2                                   
  3
  4 @transaction.commit_on_success    
  5 def sophisticated_background_processing():
  6     #your code come here :-)
  7     #(...)                        
  8     pass
  9
 10 @transaction.commit_manually
 11 def some_other_background_processing():
 12     try:
 13         #your code
 14         #(...)
 15         transaction.commit()      
 16     except SomeException as se:   
 17         #handle exception
 18         transaction.commit()      
 19     except:
 20         #unhandled exceptions     
 21         transaction.rollback()      

The commit_on_success acts just like TransactionMiddleware for view, in most cases it will do. If we need some more flexibility we can always try the commit_manually decorator. It enables commiting/rollbacking data whenever you want. Just make sure all execution paths end with an appropriate action or django will raise an exception. 

Using manual-commit instead of auto-commit increased my accounting script performance about 5-10x, depending on the instance size (the processing is specific, and the data model is rather horizontal).