Condor

Condor: Making best use of the computers in the teaching labs

Introduction

The Condor Project enables us to run batch jobs on the pool of desktop computers around the department, that would otherwise be standing idle. Condor is particularly useful for high 'throughput' computing, such as an ensemble of independent model simulations, used to evaluate explore parameter-space.

Basic commands

You can run the following commands from the submission host of your condor pool. For Geography, this is condor.ggy.bris.ac.uk (Note that this is a Linux server).

You can review the status of all the machines in the 'condor pool' using the command condor_status:

Name               OpSys      Arch   State     Activity LoadAv Mem   ActvtyTime

slot1@GEOG-B224.gg WINNT51    INTEL  Owner     Idle     0.000  1661  0+01:30:04
slot2@GEOG-B224.gg WINNT51    INTEL  Owner     Idle     0.010  1661  0+01:30:05
slot1@geog-a105.gg WINNT51    INTEL  Claimed   Busy     1.130  1661  0+01:02:57
slot2@geog-a105.gg WINNT51    INTEL  Claimed   Busy     1.130  1661  0+01:02:58
slot1@geog-c200.gg WINNT51    INTEL  Unclaimed Idle     0.000  1662  0+00:00:04
slot2@geog-c200.gg WINNT51    INTEL  Unclaimed Idle     0.040  1662  0+00:00:00
...
...
                     Total Owner Claimed Unclaimed Matched Preempting Backfill

       INTEL/WINNT51   191    78     109         4       0          0        0

               Total   191    78     109         4       0          0        0

Typically you will get several screen's full of output, so I've chopped out the middle part of the listing, leaving just a few at the top and the final summary, given at the end.

From the listing, you can see that:

• The PC called GEOG-B224 has someone logged into it, indicated by the keyword Owner, but it is not working hard, as it is Idle.
• In contrast, geog-a105 is marked as Claimed, indicating that it has been grabbed by condor, and it working hard, Busy.
• The third possible state is exemplified by geog-c200, which is neither claimed nor in interactive use.

The final summary tells us that at the time of writing, the pool contains 191 PCs, 78 of which has a user logged in, 109 are claimed by condor and 4 remain unclaimed.

Another view of the state-of-play is given by condor_q:

 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD               
 51.0   ggdagw          4/22 13:21   0+01:30:07 R  0   3.7  test.bat          
...
...
 51.15  ggdagw          4/22 13:21   0+01:33:46 I  0   3.4  test.bat   
...
...
150 jobs; 44 idle, 106 running, 0 held   

Here we see that:

• job 51.0 was submitted at 13:21 on the 22nd of April, has been running for just over an hour and a half, and that the job executable is called 'test.bat'
• We can also see that job 51.15 is idling, rather than running.
• In total 106 of the 150 jobs submitted to condor are running, and accordingly 44 are still waiting to run and so are idle.

Submitting a simple script job

OK, so much for other people's jobs. Let's submit some ourselves. I have prepared some examples to make this as easy as possible. To get these, you can cut & paste the following onto your command line (I assume linux here, but the same files will work for a windows submission host too):

svn co http://source.ggy.bris.ac.uk/subversion-open/condor/trunk ./condor
cd condor/examples/example1

Without further ado, let's submit our first job to the pool:

condor_submit win.submit

If you look inside win.submit, you will see that it is a short and reasonably self explanatory file:

Universe   = vanilla
Notification = never
requirements = OpSys == "WINNT51" && Arch == "INTEL"
Output = test.out
Error = test.err
Log = test.log
should_transfer_files = YES
when_to_transfer_output = ON_EXIT_OR_EVICT
Executable = test.bat
Queue

The key lines to note for the moment are:

• the output of the job will collect in test.out
• any errors will go to test.err
• test.log will record the mechanics of sending the job to a remote PC
• and that test.bat calls the shots!
• the Queue keyword causes the job to enter the queue of jobs to be run

The executable file test.bat is a short and simple batch file (aka 'shell script' in Linux-speak):

echo Hello from a Condor batch file running on:
hostname
echo The date is:
date /T
echo and the time is:
time /T

The upshot of all the electrickery is the contents of test.out. All being well, this should have been created by now (if not, consult condor_status and condor_q for more information on the state-of-play):

Hello from a Condor batch file running on:
geog-c211
The date is:
22/04/2009 
and the time is:
15:36

Congratulations! You've run your first condor job That's the hardest part out of the way. All we have now are a few more details. So far so good? OK, let's move on to the next example.

Submitting a compiled executable

cd ../example2

In this example, we'll run the time-honoured "hello, world" job. I've included a precompiled executable for you to run. Note that I am assuming that the jobs will be run on Windows based PCs Indeed, we are submitting from a Linux server, but running our jobs on Windows PCs. This is fine for Condor, but we must remember it and not get confused.

To this end, I compiled hello_world.exe on my PC, as a DOS (or console) program. I hope it will run on your PCs too. When running jobs that you have compiled yourself, you must remember to compile it on Windows and to make sure that it will run on another Windows PC--not just your own--and then use, for example, the ssh file transfer tool to bring it over to the submission host.

OK, now that's all straight, let's run the job. We can do this as before:

condor_submit win.submit

Note that I have slightly modified the contents of win.submit:

...
Output = hello_world.out
Error = hello_world.err
Log = hello_world.log
Executable = hello_world.exe
Queue

And, all being well, we have a friendly message as output:

Hello World!

Jobs which use inputs

Sometimes we need our batch job to read some inputs from file. We'll do this in the next example:

cd ../example3
condor_submit win.submit

In this case, we've added the file test.in to the bones of example1. This is used in test.bat:

...
type test.in

In DOS-speak, this command will just echo the contents of this file, which will end up and the bottom of test.out:

...
tada! I'm from the input..

and we reference the input file in win.submit:

...
transfer_input_files = test.in
...

Well, that wasn't too bad, eh?!

Using command line arguments

In addition to input files, we may find that we need to pass a program some command line arguments. This is easy enough too.

cd ../example4

An additional line to win.submit is:

arguments = 6

and the result of the program is:

8 is the 6th Fibonacci number

Running an ensemble of jobs

Some of the real benefits of Condor come when we want to submit a number of jobs at the same time. Such an 'ensemble' is often used to explore parameter-space.

cd ../example5

In this case, we're going to submit several jobs using the same Fortran program. Let's take a look at the contents of win.submit on this occasion:

Universe   = vanilla
Notification = never
requirements = OpSys == "WINNT51" && Arch == "INTEL"
transfer_input_files = input.nml
Output = test.out
Error = test.err
Log = test.log
should_transfer_files = YES
when_to_transfer_output = ON_EXIT_OR_EVICT
Executable = fort_nml_io.exe
initialdir = job.$(Process)
Queue 3

The executable program file called fort_nml_io.exe is expecting to read an input file with the name input.nml. Of course, we would like to feed the program with different versions of the input file (providing different values for some parameters) for the different runs. However, we can't have multiple files with the same name in the same directory. (Things would be a good deal easier if we were using C, since we could pass the name of the different input files as command line arguments--Fortran does not support command line arguments, as a rule).

To get around this, we create different subdirectories to hold the relevant files for each run. These will be called job.0, job.1, job.2 etc. A nifty piece of scripting allows us to refer to these in the submission file as job.$(Process). The number of jobs submitted is determined by the value given to Queue on the last line. In this case we have specified Queue 3 and we will get 3 jobs submitting, counting from 0 to 2, as a consequence.

Run the ensemble in the usual way:

condor_submit win.submit

and then take a look at the contents of each of the subdirectories. Notice also that any output files created by the executable will also be transferred back from the PC which ran it, and placed in the relevant subdirectory.

Further Examples

A few more examples of submission scripts are given by the authors of Condor in the following page: Condor quick start

Condor and energy saving measures

PCs that have been idle for a considerable length of time are set to shutdown (or hibernate). These PCs are periodically (once early in the morning and again at lunchtime) 'woken-up', so that they are ready to receive jobs again. If there are no calls on them for work the PCs will shutdown again after a few hours have elapsed.