Sunday, April 21, 2013

bash scripting - running jobs in parallel - an easy trick

The idea is to write a waiting script and place it right before the jobs we want to run in parallel, inside the loop(as shown in second picture below). The script will sleep for infinite time until maximum number of jobs are running. The algorithm is like this (1) count the number of jobs running that match the string provided, then wait for 60 seconds in loop, if the count is equal or greater than the maximum number of jobs we want to run (2) count and wait repeatedly, until less jobs are running. Once the number of jobs running is less than the maximum number of jobs we want to run, the script terminates. The script I wrote is accessible at http://web.missouri.edu/~bap54/misc/wait4parallelrun.sh

Here is a screenshot of basic logic and also an implementation example: