Setting up a development tmux session

Lets face it, it takes some time to setup a development session. Besides the IDE there are usually lots of other scripts and tools that need to be run, and monitored throughout the development process. Usually you need to run each script/tool in a separate terminal. This is quite inconvenient since multiple tools could be aggregated (you usually do not need a full-screen version  of htop running, same goes for logfiles). This problem may be solved using a terminal multiplexer, in our case tmux. If you're missing it, installing it is a must:

~ $ sudo apt-get install tmux

Using tmux is a enables organizing your dev session in a better/tidier manner. The greatest advantage  of using tmux over screen or a tabbed gnome-terminal is the possibility to splitting a pane (both horizontally and vertically), this eventually enables to setup your scrips/tools in any way you can imagine. Term 'eventually' was not used accidentally, eventually because it usually takes some typing to achieve the intended results. We programmers are lazy and like automating things, so why not create a script that sets up tmux with our predefined panes. This can be achieved by using tmuxinator. You can install it using gem.

~ $ sudo gem install tmuxinator

Now we can create a new session definition:

~ $ tmuxinator new fxbot

And type in some basic instructions to ~/.tmuxinator/fxbot.yml:

# ~/.tmuxinator/fxbot.yml

project_name: FXBot
project_root: ~/prj/forex/forex_bot/
tabs:
    - editor: vim .
    - console:
        layout: main-vertical
        panes:
            - #bash
            - ipython
    - stats:
        layout: main-vertical
        panes:
            - htop
            - tail -f logs.txt


Now when you run:


~ $ tmuxinator start fxbot


You will end up running tmux with three tabs, the editor tab will contain a running instance of vim, the console  tab will be split vertically, the left pane will have a bash terminal, while the right will have ipython running. The final stats tab will be displaying htop and the last entries of a logfile. Well maybe its not a hard session to set up manually, but imagine setting up 5+ panes with split screens and various tasks running. 


This tool may also be configured to execute tasks before starting (like setting up the database server). More information is available on the projects site. 

Cheers!
KR


~KR

Incremental backups with rsync

There are two types of computer users in the world: those who backup their data, and those who eventually will backup their data. Making regular backups consumes some time, but saves a lot of nerves and time(money) the primary data source crashes. Let's face it, backups are important.

Program source codes usually do not make problems, there are distributed version control systems with remote repositories which are ideal for not only sharing but also backing up the data.

Usually there are gigabytes of data that you will want to backup besides your source code. A good place for such backups is a remote storage or an external drive. If you'd like to automate the backup process as much as possible I suggest using rsync. It enables making incremental backups, which save your Internet bandwith / makes it faster to synchronize external drives. The following makes an incremental copy of some directories located in the home directory.

declare -a SOURCE_DIRS=("a" "b" "img" )

BACKUP_DIR=/mnt/backup/


for source_dir in ${SOURCE_DIRS[@]}
do
    echo "Current directory: $HOME/$source_dir"
    rsync -a "$HOME/$source_dir" $BACKUP_DIR
done
sync
echo "Backup complete"

This works exceptionally well. The -a option stands for:

• -r, --recursive recurse into directories
• -l, --links copy symlinks as symlinks
• -p, --perms preserve permissions
• -t, --times preserve modification times
• -g, --group preserve group
• -o, --owner preserve owner
• --devices, preserve device files
• --specials, preserve special files

More available at the rsync manual.

Cheers!
KR

Exporting MySQL queries into a CSV file.

Every now and then I need to do some non-standard data processing things, which requires grabbing a bit of data from a database. Usually this "non-standard data processing things" are simple scripts, and enhancing them to get data directly from a database would kill their main advantages: simplicity and a fast setup time.

CSV files are way better for this kind of processing. Since the data is located in a database, not a CSV file, we'll have to export it. Now there are two basic ways:

a) The MySQL geek way.
b) The bash/sh geek way.

So let's start with the pure MySQL way. Usually when queries are made using the standard mysql console client, the retrieved data is formated into something that should reassemble a table... well maybe it does, but "broad" result tables aren't readable anyway.

select * from some_table into outfile "/tmp/output.csv" \
fields terminated by ";" lines terminated by "\n";

All we have to do is change the field and line terminator to match our needs. About the file, you should pass a path that is writable for the mysql user. If the file exists, or the MySQL server has no write permissions, this query will fail.

Let's move on to the bash geek way - personally I don't fancy MySQL trciks, therefore I usually rely on bash. So here we go:

echo "select * from some_table;" | \
mysql -u my_usr -p my_db | sed 's/\t/;/g' | \
tail -n +2 > /home/chriss/output.csv

We execute a query, passing it via an input stream to the MySQL client. The client returns a tab separated list of fields (one per line), sed replaces those with semicolons. Next we dispose of the first line (using tail), since it contains column names. Finally we save the output as a file. Unlike in the MySQL way, you can write the file according to your write permissions, not  mysql's users.

Both approaches have some advantages and disadvantages, so their usage should be dependent on the context.

Regular expression based process termination in linux.

Even though Unix-based systems are generally stable, some processes (usually non-kernel) could benefit from a kill from time to time. GUI applications may hang, some batch programs may have memory leaks, there are many things that can go wrong. Fortunately Linux provides a set of tools that may help in such situations, you can use the ps command to list active processes. It reports a snapshot of a process granting you information about the process id (PID), parent process id (PPID), priority, memory usage, current state and more. There are also kill  and pkill commands which may terminate a process identified by a specific id or executable name respectively.

If you are running multiple instances of a program, and you want to terminate only a few of them its hard to apply the presented commands - pkill will terminate all instances (that is not desired), while kill will require you to obtain PIDs (this requires some work). It may be easier to locate the mentioned processes by their command line arguments, execution paths or other run parameters, literally or using regular expressions.

The following script does the job:

if [ "$1" ]
then
    ps lax  | tr -s ' ' | cut -d ' '  -f 3,13- | \
    egrep "$1" | cut -d ' ' -f 1 | \ 
    xargs kill -9 2>/dev/null
else
    echo "Usage: $0 <extended regular expresion>"
fi

First a process list is obtained and adjusted for further processing (the text output is so lousy...). Using cut preserves only the third column (PID) and everything beyond the 13-th (whitespace separated application name with additional parameters). Next we match the output with a provided extended regular expression (ERE), be warned though - the tested string starts with the process ID so starting the ERE with a "^" is a bad idea (starting with "^[0-9]+" may work, but you'll eventually end up with restarting your system :-)).

Cheers!

~KR

Setting up a custom bash prompt

If you spend a lot of time exploiting the terminal not only on a single PC, but also on other servers via ssh, it is a good practice to have your bash prompt properly configured. If all your command prompts looks like this (example):

~ chriss $ 

it's almost impossible to determine your current location (server, directory, only a user - which may be common for all machines). The whole magic behind bash prompt configuration is in the $PS1 environment variable. There are also variables $PS2, $PS3, $PS4, but $PS1 is used as the primary bash prompt string. 

So let's create two prompts, one for each server your are usually connecting via ssh, and one for your local machine. Lets look at the possibilities : 

• \d : string date representation
• \e : an escape character
• \h : hostname sub-domain
• \H :  full hostname domain
• \j : current job count
• \n : newline character
• \t : time  / 24h HH:MM:SS
• \T : time / 12h HH:MM:SS
• \@ : time / 12h HH:MM
• \A : time / 24h HH:MM
• \u : username
• \w : current directory relative to $HOME
• \W : current directory

 There are more options to configure the prompt, but I personally find these most useful. Let's start with the local machine. Well since it's the default place you'll be starting your adventure from, and you'll eventually return to this prompt after your job on remote machines is done I suggest skipping most information and keeping the prompt something like this:

export PS1="~ \u \w $"

Which results in:

~ kr ~/prj/python $

It's easy, if you don't see no hostname on the prompt, you're probably still on the local machine. Displaying the working directory may save you a lot of ls / pwd executions. 

If you're connected to a remote machine, it's best to have a greater context. A good option beside the username i the hostname (full if you're having problems differentiating remote machines using only the first subdomain), the current working directory has proven useful not only on remote machines, and finally - if you're connecting to different time zones, it's a good idea to display the system time. Summing up, we would end up with something like this:

export PS1="[\A] \u@\H \W $"

This will result in the following prompt message:

[21:45] kr@some.secret.server current_dir $

This way you will never get confused about your terminal session. Remember that this only sets the prompt for the current session, if you want your prompt to get configured every time you start a session, you should apply this code to your ~/.bashrc file.

 Cheers!

~KR

Find duplicate (redundant) files: bash / linux

Recently I was implementing MT940-extract parsers for a variety of banks. Thousands of files, each containing hundreds of entries. Sometimes the entries had unique identification numbers... sometimes they had not. 

Problems occurred when, due to some random events, the extract storage started to contain duplicate files. As a result many redundant entries were loaded by the parser (this had major consequences on the whole processing).

I have implemented a few mechanisms to prevent this situation, one of them is a linux shell script that locates duplicate files in a selected subdirectory (unlimited depth):

  1 if [ -z $1 ]
  2 then
  3     echo "This script finds duplicate files in the selected directory"
  4     echo "Usage: ./find_duplicate.sh <base dir>"
  5     exit
  6 fi 
  7
  8 all_duplicate=$(find $1 | \
  9     egrep "\.[a-zA-Z0-9]+$" | \
 10     xargs md5sum 2>/dev/null | sed 's/ $/\n/g' | \
 11     sed 's/  /;/g' | sort | uniq  -w32 -D)
 12
 13 last_hash=""
 14
 15 for file in $all_duplicate
 16 do
 17     cur_hash=$(echo $file | cut -d ";" -f1)
 18     if [ "$cur_hash" = "$last_hash" ]
 19     then
 20         echo $(echo $file | cut -d ";" -f2)
 21     fi 
 22     last_hash=$cur_hash
 23 done

So lines 8-11 produce a list of all duplicate files. Since we only want to locate the redundant files, further processing is needed. In the second phase we iterate over the sorted "hash;filename" array and print out file names that have a predecessor with the same hash value thus leaving only a single file name unprinted ( within a group of duplicates that is).

This script ain't perfect, for example it will not work on file names that contain white spaces... anyway, who uses white spaces to name files? :-)

Feel free to correct/modify/share this code!

Logging mercurial (hg) update/merge history

Greetings fellow readers!

I finally managed to get my blog running... and no, this ain't another fashion blog... and this ain't another blog about cooking... Is it about voyages? Nice try... but it's not. This blog is about old-school programming :-)


Considering that this is my first post I'll start with something simple, yet very helpful -- a hook for logging the hg udpate and hg merge commands.

Why should this feature be helpful? Imagine a production server dependant on a large code repository (many programmers, many branches, you can hardly commit some changes without merging changesets). Having an update/merge history on such a server could save a lot of time and nerves when something goes wrong -- we have a list of previous stable versions.

In order to make log the mentioned activities you have to insert two lines in the [hooks] section of the .hgrc file from your home directory (or .hg/hgrc in your project directory):

~ cat ~/.hgrc

[hooks]

preupdate.pre_up = echo $(date +%D\ %T) "update [$(hg branch)]" $(hg id -i) >> .hg_update.log

update.post_up =  if [ $HG_ERROR -eq 0 ] ; then echo "$(date +%D\ %T) success: [$(hg branch)]" $HG_PARENT1 $HG_PARENT2; else echo "Errors occured" ; fi >> .hg_update.log

 These hooks log a mesage containing the current date, time, branch and changeset before and after the update/merge is performed. The history is stored in .hg_update.log (current directory). And a test:

~ hg update -C new_feature

~ cat .hg_update.log

03/07/12 22:22:12 update [default] 19478f194c18
03/07/12 22:22:12 success: [new_feature] 2c050459a85f

You can now easily revert those changes by executing:

~ hg update -C  19478f194c18

 

Hope you find my solution useful.