Condor Examples

Sequential programs

The full code of this example is in this public Git repository at GitHub

Consider a simple program which generates just one output file. For simplicity we assume that the input parameters are changed inside the program by generating a random number for example.

Let the a working directory called, let say
benchmark
with all the required files and programs.

Creates the condor directory in gfif
```
$ mkdir condor_example
```
Copy the working directory there and tar and gzip this:
```
$ tar -zcvf benchmark.tar.gz benchmark
```

Creates a bash script to tell condor what to do with the compressed directory. The IMPORTANT point is that this script must have an argument which allows to differentiate the output file of each sequential run

run.sh

#!/bin/bash
Process=$1
tar -xzvf benchmark.tar.gz
cd benchmark/
#run either one system program,
#or one script inside benchmark directory which
#take at least the argument $Process to differentiate 
#the output program:
./main_program.py $Process
#The main program produces an output Output$Process.out
# which must be declared inside the condor submit file: 
# job.submit  as:
#transfer_output_files =  benchmark/Output$(Process).spc
# See job.submit for details

#Finally, sleep for while to be sure that a new random seed
# is generated
sleep 1

The next step is to prepare the condor submit file. For sequantial programming the vainilla condor universe is used. This means that condor will create an independent directory for each queue. Once each queue is finished the output file with their corresponding Process identification is transfered back to the master node:
job.submit
```
Universe        = vanilla
Executable      = run.sh
Output          = tmp/run.out
Error           = tmp/run.error
Log             = tmp/run.log
Arguments       = $(Process)
#requirements = SlotID == 6 (Machine == "particle.udea.edu.co")
#requirements = (Machine != "particle.udea.edu.co")
ShouldTransferFiles = Yes
WhenToTransferOutput = ON_EXIT
transfer_input_files = benchmark.tar.gz
transfer_output_files =  benchmark/Output$(Process).out
queue 10
```
The variable: transfer_output_files, must match the real output file name(s) of the main program in the working directory. Note that some tmp directory must be created for the log files
```
$ mkdir tmp
```
Once the full job is finished you can gather all the results with a command like:
```
$ cat Output* > FullOutput.out
```
Send your job with
```
$ condor_submit job.submit
```
Check the status of your job with
```
$ condor_q
```
and see the logs in the tmp directory if necessary.
If you need kill some queue with some ID, use
```
$ condor_rm ID
```
To kill the full job(s) use:
```
$ condor_rm -a
```
If you change something in the working directory you should need back to the step 2

Similar