Submit Blog  RSS Feeds

Thursday, August 30, 2012

Passing django variables to javascript via templates

I can fairly say, that reasonable web applications can't exist without a bit of javascript. We all remember those static web pages that were popular in the late 90'. Well their time is up, professional websites frontend is built using HTML/XHTML/HTML5 + CSS + JS (sometimes Flash).

AJAX requests or newer synchronous pull/push methods enable achieving some dynamic connectivity between frontends and backends. It is also possible to pass some initialization data this way, however if your website architecture is not a real-time client-server (ex. WebSockets) this may be inefficient.

A good practice is initializing JS variables along with rendering pages, so additional initialization request are not required. A simple way of implementing this mechanism is passing JSON encoded data to the template and decoding it in JavaScript. Lets implement the appropriate django view:

def init_js_template(request):
    import json
    json_data = json.dumps({'test' : '12345', 'arr' : [1,2,3]})
    return render_to_respons('init_json.html', {'json_data':json_data})

And the corresponding javascript initialization in the init_json.html template:



This way you will save some resources by reducing the number AJAX calls. Just keep in mind that this will only work for arrays/dictionaries containing primitive types or other arrays.

~KR

Sorting digital photos according to the real creation time using EXIF tags.

Recently I was on a holiday with my friends. Since there were a lot thrilling places to see, almost everybody took a camera. Finally after 8 days of intensive hiking we had over 900 photos taken and we wanted to make a chronological album. 

The first problem that occurred to us is that the global image namespace was not continuous (5 different cameras / cellphones with the ability to take photos), and sorting images by their generated names did not solve the it. 

Next we tried sorting the images by the file modification date - this was a reasonable approach, but there a few photos which were rotated after fetching them from the memory card causing their modification date to be inadequate for the mentioned problem. Furthermore 2 cameras were configured with an incorrect date / time value. 

A few years ago I was implementing an EXIF tag viewer for images, and I remembered that there are tags that store the original image creation time (and quite a few other parameters). I made some research and located a library with python bindings that handles these tags well: pyexiv2. 

You may install it using apt-get:

~ $ apt-get install python-pyexiv2

Next I implemented a simple script updates file names so they can be sorted lexicographically  according to the real creation time, have a look at the following code:
import pyexiv2
import sys
import os
import shutil
import datetime
from dateutil.relativedelta import relativedelta

class ExifSort():
    @classmethod
    def _test_extenstion(cls, filename):
        return filename.split(".")[-1].lower() in ['jpg','jpeg','png']
    @classmethod
    def sort_images(cls, path, prefix=None):
        files = []
        for file in os.listdir(path):
            if not cls._test_extenstion(file):
                continue
            metadata = pyexiv2.ImageMetadata(os.path.join(path, file))
            metadata.read()
            try:
                d = metadata['Exif.Photo.DateTimeOriginal'].value
                if metadata['Exif.Image.Make'].value == 'Canon':
                    d+=datetime.timedelta(days=1)
                elif metadata['Exif.Image.Make'].value == 'FUJIFILM':
                    d += relativedelta(years=1)    
                    d -= datetime.timedelta(hours=39)
            
                files.append((file, d, )) 
            except:
                print file
                continue

        files.sort(key=lambda x: x[1])
        mag = len(str(len(files))) 
        counter = 1
        tmp_prefix = "tmp_"
        for data in files:
            shutil.move(os.path.join(path, data[0]), os.path.join(path, tmp_prefix + data[0]))
        for data in files:
            ext = data[0].split(".")[-1]
            shutil.move(os.path.join(path, tmp_prefix + data[0]),\
                    os.path.join(path, "%s%s.%s" % (prefix or "",\
                    str(counter).rjust(mag,'0'), ext)))
            counter += 1


if __name__ == '__main__':
    try:
        ExifSort.sort_images(sys.argv[1])
    except:
        print "Usage: %s [directory]" % sys.argv[0]
        sys.exit(1)


This script iterates all files in a specific directory ( sys.argv[1] ), filters jpg/png files, reads their exif date and stores pairs: filename, creation datetime in an array. Finally the array is sorted using the real creation datetime, and the images are renamed (with or withour a specific prefix).   

Lines 22-26 are a specific exception - we had 2 cameras with an incorrect date, so we have to correct the read value so it may be compared with images from other cameras (evaluated empirically) . 

~KR 

Friday, August 17, 2012

A quick peek into new django 1.4 feature

Its been a few months since a stable release of django 1.4 has been published. Many of my projects were based on django 1.3, and some even on 1.1. Since they worked there was no sense migrating to the latest version, but recently I started a new project and decided to try out the latest stable version.

The first visible difference is the directory structure, if you for example execute:

~ $ django-admin.py startproject test

a following project directory will be created:

test/
  manage.py
  test/
     settings.py
     urls.py
     wsgi.py

You may now add you apps under the test/test or test directories, this affects the package namespace, this is a nice improvement since it allows you to share your modules not only between your applications but other django projects.

Now lets go to some other major improvements, that have a greater impact on process of building/managing web applications.

The first feature that significantly improves the quality of your code is the ability to implement in-browser test cases. In the previous editions you had to setup curl or write dedicated scripts that simulate user behavior. Django 1.4 is integrated with selenium - one of the most popular browser automation systems, designed for website testing. Yeah, no more situations when somebody actually forgot to commit a template to the production repository.

Another important feature improves the ORM, as from django 1.4 QuerySet's  have a method select_for_update(). Now what does it do? I'm speaking rhetorically, of course it  implements the SQL: SELECT ... FOR UPDATE  which locks the selected rows to prevent other threads modifying  them. Now this greatly improves transactional processing.

More features can be found on the official release page https://docs.djangoproject.com/en/dev/releases/1.4/.

As for requirements, the first thing I need to mention is that django 1.4 drops support for python 2.4. Most projects nowadays run on python 2.6 or 2.7, so this shouldn't be a problem. Support for python 3.X is not included, and it probably won't be in the next release aswell.  So for now I guess python 2.7 and django 1.4 is the strongest choice.

~KR


Tuesday, August 7, 2012

Creating multiple isolated python environements on a single OS using virtualenv

If you're doing a lot of coding in python, you probably came to a situation where you had to maintain many projects concurrently. In such situations it is hard to provide apropriate dependant library versions    for each project (especially if by any chance those different versions of a particular library lack a compatible API, just like python 2.X and 3.X).

A naive solution would require creating a seperate user profile (or using a virtual machine) for each maintained project. Ok seriously - don't even consider it.

If you want to have mutliple python development/deplyoment envirements you should have a place to store each appropriate set of libraries and interpreters. Secondly you should implement some scripts that run your project with the right environment variables (using the right interpreter).

This approach is better, but it sounds like a lot of work. Happily someone made the job for you. All you need to do is install virtualenv:

~ $ sudo pip install virtualenv

Now to you can configure your isolated python environment:

~ $ virtualenv my_env --quiet --no-site-packages && find my_env -maxdepth 2

my_env/
my_env/local
my_env/local/lib
my_env/local/include
my_env/local/bin
my_env/lib
my_env/lib/python2.7
my_env/include
my_env/include/python2.7
my_env/bin
my_env/bin/activate_this.py
my_env/bin/pip-2.7
my_env/bin/easy_install
my_env/bin/pip
my_env/bin/activate
my_env/bin/activate.fish
my_env/bin/python
my_env/bin/activate.csh
my_env/bin/easy_install-2.7


This is a fully functional python environment, in order to activate  it you have to load it with source  command:

~ $ source my_env/bin/activate

You command prompt should now look similar to this:

(my_env) ~ $
  
The environment has now been activated, you may now install new  libraries using pip or easy install without root privileges. All files will be installed in an appropriate directory in my_env. Running the python command will run an interpreter from the currently active virtual environment.

When you're done having fun, and you want to resume your system environment, just run:

(my_env) ~ $ deactivate 
~ $ 

You can have as many virtual environments as you like, the current version should even support moving/copying the environment.

~KR

Tuesday, July 31, 2012

Potential problems with custom primary keys in django ORM

I had a strange situation lately that encouraged me to do some research about how django really handles primary keys.

As for a mature web framework django's ORM generates automatic integer primary keys if none are present. This feature is cool, cause we are relieved from generating our own primary keys (if its problematic), but we loose some easily accessible semantic information - for example: a car's primary key may be it's registration number instead of a meaningless integer. Whats more, a primary key is used in the default object __eq__ / __cmp__ functions, which makes it easier to filter data (no need for subselects).


But let's look at this situation:

class Dude(models.Model):
    name = models.CharField(primary_key=True, max_length=100)
    #and so on

We have a 'Dude' that should have a unique name, primary keys are in general unique so there's no problem here. Suppose we have some form where you add 'dudes'. So we execute:

try:
    Dude(name='John', **some_fields).save()
    #some more code
    Dude(name='John', **some_other_fields).save()
    print "You don\'t expect to see this message"
except IntegrityError as ie:
    pass

The logic states, that if you try to create another entry with a same value of a unique field, django should raise an integrity error. Well it will not, because this is also a primary key, if a 'dude' with a specific name exists, django interprets it like this:

Dude.objects.filter(name='John').update(**some_other_fields)

For me this was a bit unintuitive, but when think about it - this approach makes sense. All in all, I would recommend using a custom primary_key field only if there is some semantic/logical background behind it.

~KR



Sunday, July 29, 2012

A quick look into HTML5 canvas tag

Lately I've experimenting with the new HTML5 features. There are many new tags, and attributes that add new functionality to existing tags. I wouldn't want to get to much into a general preview of these elements, instead I want to focus on single tag: <canvas>. A canvas element/property frequently appears in high level GUI designing programs or interfaces, it allows to draw custom graphics/primitives on a panel. The HTML5 canvas semantic is similar: it allows drawing graphics on the fly! Yes, no more CSS tricks to get an animation running effectively, now all you need is a browser supporting HTML5 and some basic JS knowledge.


Canvas and 2D graphics usually are connected with games, so after getting familiar with some basic concepts I started implementing a simple game - tetris similar. Here's what I have so far:





The currently implemented mechanics include:
  • falling bricks (1 type)
  • brick stacking (if one lands on another)
  • moving bricks left/right (arrow keys, it may not work via iframe)
The current code is available at http://boty.vipserv.org/tetris/tetris/tetris.html

Feel free to have a peek at the code, it's far from being tidy - this was just a fast canvas experiment.

The most popular browsers support HTML5, so I don't see any disadvantages in using it in day to day programming.

~KR

Tuesday, July 24, 2012

Cheating online games /polls / contests by using anonumous HTTP proxies / python

This post is indeed about cheating. You know those browser-based game profile refs that provide you with some benefits each time a persons clicks it? Thats right, everybody spammed them here and there, some people had many visitors, some had not. I wanted to gain some extra funds too, but the thing I hate more than not-having-extra-funds is spamming... I just felt bad about posting that ref anywhere it was possible, like:

Check out these hot chicks <a href="#stupid_game_ref#">photos</a>

It's quite obvious, that clicking the link again and again yourself didn't have much effect. Only requests with unique IP numbers (daily) where generating profit.  So the question was how to access an URI from many IP from one PC (my PC that is). The answer is simple: by using anonymous HTTP proxies.

There are many sites that aggregate lists of free proxies, like proxy-list.org. It's best to find a site that enables fetching proxy IP/PORT data in a script processable way (many sites have captchas). 

Let's get to the fun part, the following script executes specific actions for each proxy in the provided list.


  1 import socket
  2 import urllib
  3 import time
  4    
  5 HEADERS = {"User-Agent" : "Mozilla/5.0" }
  6    
  7 proxies = ["10.10.1.1:8080", "192.168.1.1:80"]
  8    
  9 #timeout in case proxy does not respond
 10 socket.setdefaulttimeout(5)
 11
 12 for proxy in proxies:
 13     opener = urllib.FancyURLopener({"http" : proxy})
 14     opener.addheaders = HEADERS.items()
 15     try:
 16         res = opener.open("http://some.uri/?ref=123456")
 17         res.read()
 18         time.sleep(3)
 19     except:
 20         print "Proxy %s is probably dead :-(" % proxy
 21        

Line 5 contains a basic User-Agent header, you can find more about setting the appropriate headers here. Line 10 sets the default socket timeout on 5 seconds - many proxies tend not to work 24/7, so best catch those exceptions.  Finally we create an opener from each IP and request some resources (our ref link), you might replace this simple request with a set of actions, or even make bots that act via proxies. Just make sure you're proxy is truly anonymous (easy to verify by a simple PHP script) .


This may be called cheating, but at least it's not spamming :-)


~KR





free counters