Preventing python from generating *pyc files on runtime

Whenever You import a module, CPython compiles it to byte code, and saves it using the same path and filename, except for a *.pyc extension (valid for python 2.X). A python script converted to byte code does not run any faster, the only advantage is that pyc files are loaded faster.  Although this is generally desired it may cause problems during development, when our application imports existing pyc files instead of compiling our freshly modified source files. Such problems should not occur to often, but when they do, we usually don't have a clue what's going on (I just fixed it, and it still crashes!?).

Remove *pyc files!

Of course You can create a script that performs a search-and-destroy on all *pyc files located in all of your projects subdirectories. This is cool, but preventing python from generating *pyc files (in dev) is even better.

So there are basically three ways to achieve it in python 2.7:

1. During script run, use the -B parameter
python -B some_script.py

2. Project level, insert this line on top your application / script (*py)
import sys
sys.dont_write_bytecode = True

3. Environment level, set the following env. variable:
export PYTHONDONTWRITEBYTECODE=1

Have fun eradicating *pyc files in Your dev environments!

Cheers!
KR

Setting up a development tmux session

Lets face it, it takes some time to setup a development session. Besides the IDE there are usually lots of other scripts and tools that need to be run, and monitored throughout the development process. Usually you need to run each script/tool in a separate terminal. This is quite inconvenient since multiple tools could be aggregated (you usually do not need a full-screen version  of htop running, same goes for logfiles). This problem may be solved using a terminal multiplexer, in our case tmux. If you're missing it, installing it is a must:

~ $ sudo apt-get install tmux

Using tmux is a enables organizing your dev session in a better/tidier manner. The greatest advantage  of using tmux over screen or a tabbed gnome-terminal is the possibility to splitting a pane (both horizontally and vertically), this eventually enables to setup your scrips/tools in any way you can imagine. Term 'eventually' was not used accidentally, eventually because it usually takes some typing to achieve the intended results. We programmers are lazy and like automating things, so why not create a script that sets up tmux with our predefined panes. This can be achieved by using tmuxinator. You can install it using gem.

~ $ sudo gem install tmuxinator

Now we can create a new session definition:

~ $ tmuxinator new fxbot

And type in some basic instructions to ~/.tmuxinator/fxbot.yml:

# ~/.tmuxinator/fxbot.yml

project_name: FXBot
project_root: ~/prj/forex/forex_bot/
tabs:
    - editor: vim .
    - console:
        layout: main-vertical
        panes:
            - #bash
            - ipython
    - stats:
        layout: main-vertical
        panes:
            - htop
            - tail -f logs.txt


Now when you run:


~ $ tmuxinator start fxbot


You will end up running tmux with three tabs, the editor tab will contain a running instance of vim, the console  tab will be split vertically, the left pane will have a bash terminal, while the right will have ipython running. The final stats tab will be displaying htop and the last entries of a logfile. Well maybe its not a hard session to set up manually, but imagine setting up 5+ panes with split screens and various tasks running. 


This tool may also be configured to execute tasks before starting (like setting up the database server). More information is available on the projects site. 

Cheers!
KR


~KR

Incremental backups with rsync

There are two types of computer users in the world: those who backup their data, and those who eventually will backup their data. Making regular backups consumes some time, but saves a lot of nerves and time(money) the primary data source crashes. Let's face it, backups are important.

Program source codes usually do not make problems, there are distributed version control systems with remote repositories which are ideal for not only sharing but also backing up the data.

Usually there are gigabytes of data that you will want to backup besides your source code. A good place for such backups is a remote storage or an external drive. If you'd like to automate the backup process as much as possible I suggest using rsync. It enables making incremental backups, which save your Internet bandwith / makes it faster to synchronize external drives. The following makes an incremental copy of some directories located in the home directory.

declare -a SOURCE_DIRS=("a" "b" "img" )

BACKUP_DIR=/mnt/backup/


for source_dir in ${SOURCE_DIRS[@]}
do
    echo "Current directory: $HOME/$source_dir"
    rsync -a "$HOME/$source_dir" $BACKUP_DIR
done
sync
echo "Backup complete"

This works exceptionally well. The -a option stands for:

• -r, --recursive recurse into directories
• -l, --links copy symlinks as symlinks
• -p, --perms preserve permissions
• -t, --times preserve modification times
• -g, --group preserve group
• -o, --owner preserve owner
• --devices, preserve device files
• --specials, preserve special files

More available at the rsync manual.

Cheers!
KR

Laptop screen backlight brightness adjustment

Well I have to admit that not all of my Asus UL80V function keys work on Linux. Among them are the brightness adjustment buttons. Well it's not a problem to adjust the brightness on linux anyway, I remember there was a possibility to set it manually in:

/proc/acpi/video/VGA/LCD/brightness

Sadly I failed to find the file on my current distribution (Mint 13 / Maya, based on Ubuntu 12). I made some research and searching and managed to locate another file responsible for the backlight brightness setting:

/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/backlight/acpi_video0/brightness

The brightness may be set the following way:

sudo echo "15" > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/backlight/acpi_video0/brightness 

This works, but is a little bit inconvenient. Now I need to find a way do bind this with the brightness increase/decrease buttons, which may be tricky since they don't appear in /dev/input/ events (or at least I haven't found a suitable event yet).

Cheers!
KR 

Hard links vs. symbolic links

Everybody uses links in his everyday virtual life. I don't think anyone could imagine navigating between websites without using hyper-refs, but let's move to links that refer to our file-system. If You had some experience with MS Windows before, You're probably familiar with symbolic links, aka shortcuts.

In general symbolic links may be interpreted as pointers directed to our files logical layer. In a larger scope this may look like a pointer to the pointer of physical data. If this still looks confusing, have a look at an example:

~ $ echo "Test file" > f1.txt
~ $ ln -s f1.tx f2.txt

This may be visualized the following way ("Test file" is the physical data here):

Now you can access the physical data using both f1.txt and f2.txt. However, if You delete f1.txt, the physical data will be lost (no label will point to it). A different situation occurs when You use hard links instead. Each time You create a file, a label is hard linked to it. In the previous example the hard link was created by executing:

~ $ echo "Test file" > f1.txt

By default each chunk of physical data has only 1 hard link attached, but more may be attached. For example:

~ $ ln f1.txt f2.txt 

Will create a hard link with label f2.txt to the physical data of f1.txt, let's visualize it:

You can access the physical data both via f1.txt and f2.txt. Unlike symbolic links, each of the hard links works even if the other stops to exist. In order to delete the physical data you need to unlink all hard links that point to it (rm f1.txt alone will not do...).

To sum up, symbolic links are cool because they operate on the logical layer, and thus are not limited to a single file system. Furthermore symbolic links may point to directories which is an important feature.

Hard links also have some features that their symbolic cousins have not. Hard links are always bound the the physical source of data, thus making them move/rename proof (symbolic links are not updated if you move/rename a corresponding hard link).

Hope this overview helps you to choose a right link for each situation.

Cheers!

~KR

Renaming Mercurial branches

I believe there is no need to present Mercurial (since You got here, You should be familiar with it anyway). I'd like to present a way of renaming / replacing branches. By default, without using extensions, it's impossible ("branches are permanent and global...")... but there are other ways to deal with it. Let us suppose we want to rename branch A to B. We can achieve it the following way:

hg update A
last_head=$(hg id -i)
hg ci -m "close A" --close-branch
hg update $last_head

hg branch B
hg ci -m "branche renamed to B"

This is it, now our branch is named B. In practice we just closed branched A and created a new branch B from the last commit. This activity may be visualized the following way:

Replacing an existing branch with another is a bit more tricky, here's what you have to do:

hg update A 
hg ci -m "close A" --close-branch
hg update B
hg branch A  -f #force, branch A exists
hg ci "rename to A"
#optional close branch B

The general idea may be presented the following way:

In order to create a branch that previously existed we have to use the force switch (hg branch). Nothing should go wrong if the previous head of branch A was closed, else You'll just end up creating another head.


Experimenting with hg is cool, just remember - before You try anything experimental, commit your work! This may save You a lot of nerves.

Cheers!

~KR

Serve text file via HTTP protocol using netcat

Unix based system provide a lot of cool network tools, today I'd like to show the potential of netcat. Netcat is a utility that may be used for "just about anything under the sun involving TCP and UDP" (netcat BSD manual) . Thats a pretty description, but let's go on to some practical stuff.

If we want to make a useful script, we should make it work with a popular protocol, such as HTTP. This way we don't have to worry about potential client application, we may use an ordinary web browser to test our script. More formal information about the HTTP protocol can be found in RFC2616.

So a bash script that generates a simple HTTP response and serves a text file may look like this:

if [ -z "$1" ]
then
    echo "Usage: $0 <text file to serve>"
    exit 1
fi

filename=$1

echo "
HTTP/1.1 200 OK
Date: $(LANG=en_US date -u) 
Server: NetCatFileServe
Last-Modified: $(LANG=en_US date -u)
Accept-Ranges: bytes
Content-Length: $(cat $filename | wc -c  | cut -d " " -f 1) 
Content-Type: application/force-download
Content-Disposition: attachment; filename=\"$filename\"

$(cat $filename)
"

If a HTTP client receives such a response, it should try to save the attached data as a file (a typical save-file-as window in a web browser).

Since we have a HTTP response generate, we need to create a server that will serve it - that's where netcat comes in handy. To serve some data we need to run it in the listening mode. For example :

~ $ ./ncserve.sh test.txt | nc -l 8888

If you now enter http://127.0.0.1:8888 (or your other IP) in a webbrowser, you should be able to download text.txt  file. You may also  test it using curl:

~ $ curl -X GET 127.0.0.1:8888
HTTP/1.1 200 OK
Date: Sat Oct 13 10:40:27 UTC 2012
Server: NetCatFileServe
Last-Modified: Sat Oct 13 10:40:27 UTC 2012
Accept-Ranges: bytes
Content-Length: 71
Content-Type: application/force-download
Content-Disposition: attachment; filename="test.txt"

This is a simple text file
bla bla bla
downloaded via netcatserver
:-)

This script only serve a file once and dies, if you want it to act like a regular HTTP server you should run it in a infinite loop.

Cheers!

~KR

Regular expression based process termination in linux.

Even though Unix-based systems are generally stable, some processes (usually non-kernel) could benefit from a kill from time to time. GUI applications may hang, some batch programs may have memory leaks, there are many things that can go wrong. Fortunately Linux provides a set of tools that may help in such situations, you can use the ps command to list active processes. It reports a snapshot of a process granting you information about the process id (PID), parent process id (PPID), priority, memory usage, current state and more. There are also kill  and pkill commands which may terminate a process identified by a specific id or executable name respectively.

If you are running multiple instances of a program, and you want to terminate only a few of them its hard to apply the presented commands - pkill will terminate all instances (that is not desired), while kill will require you to obtain PIDs (this requires some work). It may be easier to locate the mentioned processes by their command line arguments, execution paths or other run parameters, literally or using regular expressions.

The following script does the job:

if [ "$1" ]
then
    ps lax  | tr -s ' ' | cut -d ' '  -f 3,13- | \
    egrep "$1" | cut -d ' ' -f 1 | \ 
    xargs kill -9 2>/dev/null
else
    echo "Usage: $0 <extended regular expresion>"
fi

First a process list is obtained and adjusted for further processing (the text output is so lousy...). Using cut preserves only the third column (PID) and everything beyond the 13-th (whitespace separated application name with additional parameters). Next we match the output with a provided extended regular expression (ERE), be warned though - the tested string starts with the process ID so starting the ERE with a "^" is a bad idea (starting with "^[0-9]+" may work, but you'll eventually end up with restarting your system :-)).

Cheers!

~KR

Setting up a custom bash prompt

If you spend a lot of time exploiting the terminal not only on a single PC, but also on other servers via ssh, it is a good practice to have your bash prompt properly configured. If all your command prompts looks like this (example):

~ chriss $ 

it's almost impossible to determine your current location (server, directory, only a user - which may be common for all machines). The whole magic behind bash prompt configuration is in the $PS1 environment variable. There are also variables $PS2, $PS3, $PS4, but $PS1 is used as the primary bash prompt string. 

So let's create two prompts, one for each server your are usually connecting via ssh, and one for your local machine. Lets look at the possibilities : 

• \d : string date representation
• \e : an escape character
• \h : hostname sub-domain
• \H :  full hostname domain
• \j : current job count
• \n : newline character
• \t : time  / 24h HH:MM:SS
• \T : time / 12h HH:MM:SS
• \@ : time / 12h HH:MM
• \A : time / 24h HH:MM
• \u : username
• \w : current directory relative to $HOME
• \W : current directory

 There are more options to configure the prompt, but I personally find these most useful. Let's start with the local machine. Well since it's the default place you'll be starting your adventure from, and you'll eventually return to this prompt after your job on remote machines is done I suggest skipping most information and keeping the prompt something like this:

export PS1="~ \u \w $"

Which results in:

~ kr ~/prj/python $

It's easy, if you don't see no hostname on the prompt, you're probably still on the local machine. Displaying the working directory may save you a lot of ls / pwd executions. 

If you're connected to a remote machine, it's best to have a greater context. A good option beside the username i the hostname (full if you're having problems differentiating remote machines using only the first subdomain), the current working directory has proven useful not only on remote machines, and finally - if you're connecting to different time zones, it's a good idea to display the system time. Summing up, we would end up with something like this:

export PS1="[\A] \u@\H \W $"

This will result in the following prompt message:

[21:45] kr@some.secret.server current_dir $

This way you will never get confused about your terminal session. Remember that this only sets the prompt for the current session, if you want your prompt to get configured every time you start a session, you should apply this code to your ~/.bashrc file.

 Cheers!

~KR

Generating sounds using the Open Sound System audio interface

Recently I started implementing a tool for learning music scales. I've done some research and found a way of utilising /dev/dsp in python. If you don't know what is that device file responsible I'll give you a hint. Make sure you have your speakers on, and type in:

~ cat /dev/urandom > /dev/dsp

If the device ain't busy or otherwise locked you should hear a hum... it ain't bad for a random input stream. But with a little help from python and some basic knowledge about digital sound processing we can get much more than that.

Let's start with some basics. To generate a random hum (just like in the example above) using python ossaudiodev module you could try the following:

  1 import ossaudiodev
  2 import os
  3
  4 dsp = ossaudiodev.open('/dev/dsp', 'w')
  5 dsp.write(os.urandom(5000))
  6 dsp.close()

Let's move on to some signal processing theory. In order to generate a specific tone we should pass a discrete approximation of a sine function for the analysed time ranged instead of a random array. In presented source I will assume 44.1k samples per second (why is that? check the Shanonn's Law), a frequency of 440Hz (also known as A440, it serves as a general instrument tuning standard), and tone duration of 5 seconds.

  1 import ossaudiodev
  2 import math
  3 import wave
  4
  5 freq, sr, t = 440.0, 44100.0, 5.0
  6 total_samples = sr*t
  7 period = sr / freq
  8 natural_freq = 2.0*math.pi/period
  9 #evaluate x-axis / time-axis positions
 10 time_axis = map(lambda x: float(x)*natural_freq, range(int(period)))
 11 #evaluate singal amplitudes for the period
 12 period_amp_data = map(lambda x: 16*math.sin(x), time_axis)
 13 #repeat the singal, and pack as short, 16 bit values
 14 output_signal = ''
 15 for i in range(int(total_samples/period)):
 16     for j in range(len(period_amp_data)):
 17         output_signal += wave.struct.pack('h', period_amp_data[j])
 18 dsp = ossaudiodev.open("/dev/dsp", "w")
 19 #16 bit big endian coding, 1 channel, 44.1kHz
 20 dsp.setparameters(ossaudiodev.AFMT_U16_BE, 1, sr)
 21 dsp.write(output_signal)
 22 dsp.close()

You can easily refactor this code making it possible to produce any tone you want (generating multi-tone sounds and effects is a bit more tricky). It may be a bit hard to get through without some background in digital signal processing, but having a working example programmers can do magic.

A good way of optimising this code is using numpy arrays instead of python lists, because they support many matrix like transformations (no mapping function would be needed).

P.S. I'm still using Mint 9 Isadora LTS version and I am happy to have a /dev/dsp device file, however Ubuntu users are not so lucky: /dev/dsp is not present in the kernel since v. 10.10... well... you can always try recompiling it :-)

~KR

Heterogeneous system administration issues

I know heterogeneous environments became popular lately, but hey - let's talk about the drawbacks of such systems. So for example let us visualise a process that is dependant on Windows, Linux and MacOS that run on three separate machines.

First of all each of these machines has to be properly configured (services, security, performance). System administrators are usually commited to a specific platform so setting up other configurations is more time consuming.

For example, it took me some time to get familiar with using Darwin's launchctl service management framework. Well all I wanted to do is run my task periodically... cron is a great and simple tool capable of achieving it... however all other services/tasks were configured via launchctl, for me writing a huge XML configuration file instead of a simple crontab entry is an overkill. 

But anyway, lets us presuppose that this Mac enables it's HDD as a network drive available via SFTP and AFP. This hard drive contains data that needs to be processed by a dedicated piece of software that runs only on Windows... in case you found a fast way of solving this problem before I actually stated it: no you can't change that piece of software.

So the user responsible for processing the data mounts the SFTP shares and proceeds with his task. However the user also needs access to his own resources located in another location, which requires providing other credentials. It seems like Windows7 is not capable of keeping multiple SFTP sessions with a single machine (in fact it caches it, thus making it difficult to re-log again). Since we don't want to restart Windows every ten minutes (clear the system cache) let's keep Windows on a virtual machine that may be accessed by rdesktop.

We execute a python script responsible for preprocessing the data and starting the OS specific processing software:

#some data preprocessing
os.system('''<some operations> &&
             <a full path to the`specific software`.exe> 
             <many parameters> %s ''' % a_long_list_of_arguments)


And what do we get:

> The input line is to long

I managed to google out that MS maximum command prompt  length varies from 2047 to 8191 characters, depending on the OS version. Now this is hilarious... 

After all these adventures I installed cygwin (a tool that makes Windows reassemble an operating system) on the virtual machine and configured the script to run under the linux-like environment (finally it worked).

 So next time, remember: think twice (bah, thrice) before you intend to configure a sophisticated process on a heterogeneous environment :-)

~KR

Find duplicate (redundant) files: bash / linux

Recently I was implementing MT940-extract parsers for a variety of banks. Thousands of files, each containing hundreds of entries. Sometimes the entries had unique identification numbers... sometimes they had not. 

Problems occurred when, due to some random events, the extract storage started to contain duplicate files. As a result many redundant entries were loaded by the parser (this had major consequences on the whole processing).

I have implemented a few mechanisms to prevent this situation, one of them is a linux shell script that locates duplicate files in a selected subdirectory (unlimited depth):

  1 if [ -z $1 ]
  2 then
  3     echo "This script finds duplicate files in the selected directory"
  4     echo "Usage: ./find_duplicate.sh <base dir>"
  5     exit
  6 fi 
  7
  8 all_duplicate=$(find $1 | \
  9     egrep "\.[a-zA-Z0-9]+$" | \
 10     xargs md5sum 2>/dev/null | sed 's/ $/\n/g' | \
 11     sed 's/  /;/g' | sort | uniq  -w32 -D)
 12
 13 last_hash=""
 14
 15 for file in $all_duplicate
 16 do
 17     cur_hash=$(echo $file | cut -d ";" -f1)
 18     if [ "$cur_hash" = "$last_hash" ]
 19     then
 20         echo $(echo $file | cut -d ";" -f2)
 21     fi 
 22     last_hash=$cur_hash
 23 done

So lines 8-11 produce a list of all duplicate files. Since we only want to locate the redundant files, further processing is needed. In the second phase we iterate over the sorted "hash;filename" array and print out file names that have a predecessor with the same hash value thus leaving only a single file name unprinted ( within a group of duplicates that is).

This script ain't perfect, for example it will not work on file names that contain white spaces... anyway, who uses white spaces to name files? :-)

Feel free to correct/modify/share this code!

Logging mercurial (hg) update/merge history

Greetings fellow readers!

I finally managed to get my blog running... and no, this ain't another fashion blog... and this ain't another blog about cooking... Is it about voyages? Nice try... but it's not. This blog is about old-school programming :-)


Considering that this is my first post I'll start with something simple, yet very helpful -- a hook for logging the hg udpate and hg merge commands.

Why should this feature be helpful? Imagine a production server dependant on a large code repository (many programmers, many branches, you can hardly commit some changes without merging changesets). Having an update/merge history on such a server could save a lot of time and nerves when something goes wrong -- we have a list of previous stable versions.

In order to make log the mentioned activities you have to insert two lines in the [hooks] section of the .hgrc file from your home directory (or .hg/hgrc in your project directory):

~ cat ~/.hgrc

[hooks]

preupdate.pre_up = echo $(date +%D\ %T) "update [$(hg branch)]" $(hg id -i) >> .hg_update.log

update.post_up =  if [ $HG_ERROR -eq 0 ] ; then echo "$(date +%D\ %T) success: [$(hg branch)]" $HG_PARENT1 $HG_PARENT2; else echo "Errors occured" ; fi >> .hg_update.log

 These hooks log a mesage containing the current date, time, branch and changeset before and after the update/merge is performed. The history is stored in .hg_update.log (current directory). And a test:

~ hg update -C new_feature

~ cat .hg_update.log

03/07/12 22:22:12 update [default] 19478f194c18
03/07/12 22:22:12 success: [new_feature] 2c050459a85f

You can now easily revert those changes by executing:

~ hg update -C  19478f194c18

 

Hope you find my solution useful.