Difference between revisions of "Condor"

From SourceWiki
Jump to navigation Jump to search
 
(20 intermediate revisions by the same user not shown)
Line 4: Line 4:
 
=Introduction=
 
=Introduction=
  
−
[http://www.cs.wisc.edu/condor The Condor Project] enables us to run batch jobs on the pool of desktop computers around the department, that would otherwise be standing idle.  Condor is particularly useful for high 'throughput' computing, such as an ensemble of independent model simulations, used to evaluate explore parameter-space.
+
[http://www.cs.wisc.edu/condor The Condor Project] enables us to run batch jobs on idle CPUs found in a pool of desktop PCs, such as those in offices and computer labs in the department.  Condor is particularly useful for high 'throughput' computing.  An is example of this is the use of an ensemble of independent model simulations to explore parameter-space.
  
 
=Basic commands=
 
=Basic commands=
  
−
You can run the following commands from the '''submission host''' of your condor pool.  For Geography, this is '''condor.ggy.bris.ac.uk''' ('''Note that this is a Linux server''').
+
You can run the following commands from the '''submission host''' of your condor pool.  In Geography, the submission hosts are '''condor''' and '''dylan''' ('''Note that these are Linux servers''').
  
−
You can review the status of all the machines in the 'condor pool' using the command '''condor_status''':  
+
You can review the status of all the computers in the 'condor pool' using the command '''condor_status''':  
  
 
<pre>
 
<pre>
Line 30: Line 30:
 
</pre>
 
</pre>
  
−
Typically you will get several screen's full of output, so I've chopped out the middle part of the listing, leaving just a few at the top and the final summary, given at the end.
+
Typically you will get several screen's full of output, so I've chopped out a large swathe of the listing.  This leaves us with just few examples from the top of the listing and the final summary, given at the end.
  
 
From the listing, you can see that:
 
From the listing, you can see that:
  
−
* The PC called '''GEOG-B224''' has someone logged into it, indicated by the keyword '''Owner''', but it is not working hard, as it is '''Idle'''.   
+
* The PC called '''GEOG-B224''' has someone logged into it, indicated by the keyword '''Owner''', but it is not working hard and so is marked as '''Idle'''.   
−
* In contrast, '''geog-a105''' is marked as '''Claimed''', indicating that it has been grabbed by condor, and it working hard, '''Busy'''.
+
* In contrast, '''geog-a105''' is marked as '''Claimed''', indicating that it has been grabbed by Condor, and it working hard; marked '''Busy'''.
 
* The third possible state is exemplified by '''geog-c200''', which is neither claimed nor in interactive use.
 
* The third possible state is exemplified by '''geog-c200''', which is neither claimed nor in interactive use.
  
Line 55: Line 55:
 
Here we see that:
 
Here we see that:
 
* job 51.0 was submitted at 13:21 on the 22nd of April, has been running for just over an hour and a half, and that the job executable is called 'test.bat'
 
* job 51.0 was submitted at 13:21 on the 22nd of April, has been running for just over an hour and a half, and that the job executable is called 'test.bat'
−
* We can also see that job 51.15 is idling, rather than running.
+
* We can also see that job 51.15 is idling in the queue, rather than running.
 
* In total 106 of the 150 jobs submitted to condor are running, and accordingly 44 are still waiting to run and so are idle.
 
* In total 106 of the 150 jobs submitted to condor are running, and accordingly 44 are still waiting to run and so are idle.
  
 
=Submitting a simple script job=
 
=Submitting a simple script job=
  
−
OK, so much for other people's jobs.  Let's submit one ourselves.
+
OK, so much for other people's jobs.  Let's submit some of our own!  I have prepared some examples to make this as easy as possible. To get these examples, you can cut & paste the following onto your (Linux) command line (If you have a Windows submission host, the same files will work for you, but you will need to use [http://tortoisesvn.tigris.org TortoiseSVN] to get them):
  
−
Note that our submission host,  '''condor.ggy.bris.ac.uk''', is a '''Linux''' machine.
+
<pre>
 +
svn co https://svn.ggy.bris.ac.uk/subversion-open/condor/trunk ./condor
 +
cd condor/examples/example1
 +
</pre>
  
 +
Without further ado, let's submit our first job to the pool:
 +
 +
<pre>
 +
condor_submit win.submit
 +
</pre>
 +
 +
If you look inside '''win.submit''', you will see that it is a short and reasonably self explanatory file:
 +
 +
<pre>
 +
Universe  = vanilla
 +
Notification = never
 +
requirements = OpSys == "WINNT51" && Arch == "INTEL"
 +
Output = test.out
 +
Error = test.err
 +
Log = test.log
 +
should_transfer_files = YES
 +
when_to_transfer_output = ON_EXIT_OR_EVICT
 +
Executable = test.bat
 +
Queue
 +
</pre>
 +
 +
The key lines to note for the moment are:
 +
 +
* the output of the job will collect in '''test.out'''
 +
* any errors will go to '''test.err'''
 +
* '''test.log''' will record the mechanics of sending the job to a remote PC
 +
* and that '''test.bat''' calls the shots!
 +
* the '''Queue''' keyword causes the job to enter the queue of jobs to be run 
 +
 +
The executable file '''test.bat''' is a short and simple batch file (aka 'shell script' in Linux-speak):
 +
 +
<pre>
 +
echo Hello from a Condor batch file running on:
 +
hostname
 +
echo The date is:
 +
date /T
 +
echo and the time is:
 +
time /T
 +
</pre>
 +
 +
The upshot of all this jiggery-pokery are the contents of '''test.out'''.  All being well, this should have been created by now (if not, consult condor_status and condor_q for more information on the state-of-play), and look something like:
  
 
<pre>
 
<pre>
Line 74: Line 118:
 
</pre>
 
</pre>
  
−
=Submitting an executable which you have compiled=
+
'''Congratulations!  You've run your first condor job'''  That's the hardest part out of the way.  Honest!  All that remain are a few more details.  So far so good?  OK, let's move on to the next example.
 +
 
 +
=Submitting a compiled executable=
 +
 
 +
Move to the next example directory by typing:
 +
 
 +
<pre>
 +
cd ../example2
 +
</pre>
 +
 
 +
In this example, we'll run a time-honoured "hello, world" job.  I've included a pre-compiled executable for you to run.  '''Note that I am assuming that your jobs will be run on Windows based PCs'''  Note that despite this, we are submitting from a Linux server.  This is fine since Condor can handle it, but we must remember it and not get our operating systems and compiled executables in a muddle.
 +
 
 +
To this end, I compiled '''hello_world.exe''' on my PC, as a DOS (or console) program.  I hope it will run on your PCs too.  When running jobs that you have compiled yourself, you must remember to compile it on Windows and to make sure that it will run on ''another'' Windows PC--not just your own--and then use, for example, the ssh file transfer tool to bring it over to the submission host.  If you are in Geography and do not have a Windows C/C++ or Fortran compiler, contact [mailto:gethin.williams@bristol.ac.uk me] and I can probably compile your code for you.
  
 +
OK, now that's all straight, let's run the job.  We can do this as before:
 +
 +
<pre>
 +
condor_submit win.submit
 +
</pre>
 +
 +
Note that I have slightly modified the contents of '''win.submit''':
 +
 +
<pre>
 +
...
 +
Output = hello_world.out
 +
Error = hello_world.err
 +
Log = hello_world.log
 +
Executable = hello_world.exe
 +
Queue
 +
</pre>
 +
 +
And, all being well, we have a friendly message in '''test.out''':
 +
 +
<pre>
 +
Hello World!
 +
</pre>
 +
 +
=Jobs which use inputs=
 +
 +
Sometimes we need our batch job to read some inputs from file.  We'll do this in the next example:
 +
 +
<pre>
 +
cd ../example3
 +
condor_submit win.submit
 +
</pre>
 +
 +
In this case, we've added the file '''test.in''' to the bones of example1.  This is used in '''test.bat''':
 +
 +
<pre>
 +
...
 +
type test.in
 +
</pre>
 +
 +
In DOS-speak, this command will just echo the contents of this file, which will end up and the bottom of '''test.out''':
 +
 +
<pre>
 +
...
 +
tada! I'm from the input..
 +
</pre>
 +
 +
and we reference the input file in '''win.submit''':
 +
 +
<pre>
 +
...
 +
transfer_input_files = test.in
 +
...
 +
</pre>
 +
 +
Well, that wasn't too bad, eh?!
 +
 +
=Using command line arguments=
 +
 +
In addition to input files, we may find that we need to pass a program some command line arguments.  This is easy enough also.
 +
 +
<pre>
 +
cd ../example4
 +
</pre>
 +
 +
An additional line to '''win.submit''' is:
 +
 +
<pre>
 +
arguments = 6
 +
</pre>
 +
 +
and the result of the program is:
 +
 +
<pre>
 +
8 is the 6th Fibonacci number
 +
</pre>
  
 
=Running an ensemble of jobs=
 
=Running an ensemble of jobs=
 +
 +
Some of the real benefits of Condor come when we want to submit a number of jobs at the same time.  Such an 'ensemble' is often used to explore parameter-space.  Note that such an ensemble is often referred to as a 'cluster' in Condor-speak.  This refers to a cluster of jobs.  Not to be confused with a cluster of servers used in some High Performance Computing setup.
 +
 +
<pre>
 +
cd ../example5
 +
</pre>
 +
 +
In this case, we're going to submit several jobs which all make use of the same Fortran program.  Let's take a look at the salient parts of '''win.submit''' on this occasion:
 +
 +
<pre>
 +
...
 +
transfer_input_files = input.nml
 +
Output = test.out
 +
Error = test.err
 +
Log = test.log
 +
Executable = fort_nml_io.exe
 +
initialdir = job.$(Process)
 +
Queue 3
 +
</pre>
 +
 +
The executable program file called '''fort_nml_io.exe''' is expecting to read an input file with the name '''input.nml'''.  We would like to feed the program with different versions of the input file (providing different values for some parameters) for the different runs in our ensemble.  However, we can't have multiple files with the same name in the same directory.  (Things are a good deal easier using C/C++, since we can pass the name of the relevant input file as a command line argument.  The Fortran language does not support command line arguments, as a rule).
 +
 +
To get around this, we can create different sub-directories to hold the relevant files for each run.  These will be called '''job.0, job.1, job.2 etc'''.  A nifty piece of scripting allows us to refer to these in the submission file as '''job.$(Process)'''.  The number of jobs submitted is determined by the value given to Queue on the last line.  In this case we have specified '''Queue 3''' and we will get 3 jobs submitting, with $(Process) taking the values 0, 1 and 2 in turn, as a consequence.
 +
 +
We run the ensemble in the usual way:
 +
 +
<pre>
 +
condor_submit win.submit
 +
</pre>
 +
 +
Take a look at the contents of each of the sub-directories.  Notice also that any output files created by the executable are also be transferred back from the PC which ran it, and placed in the relevant sub-directory.
 +
 +
=Further Examples=
 +
 +
Well, in 5 small steps you've covered the majority of the ground required to make effective use of a Condor pool.  However, if you would like some more information and Condor, the following pages are useful:
 +
 +
* [http://www.cs.wisc.edu/condor The Condor Project home page]
 +
 +
* A few more examples of submission scripts are given by the authors of Condor in the following page: [http://www.cs.wisc.edu/condor/quick-start.html Condor quick start]
  
 
=Condor and energy saving measures=
 
=Condor and energy saving measures=
 +
 +
In an effort to not to waste energy, we have set PCs which have been idle for several hours to shutdown (or hibernate).  These PCs are periodically (once early in the morning and again at lunchtime) 'woken-up', so that they are ready to receive jobs again.  If there are no calls on them for work the PCs will shutdown again after a few more hours have elapsed.
 +
 +
The [http://www.carbontrust.co.uk/resource/conversion_factors/default.htm Carbon Trust] tells us that to produce 1kWh of grid electricity, 0.537 Kg of CO2 is emitted.  We calculate that we can save up to 70 tons of CO2 emissions annually from the Geography department, through enabling the shutdown facility in Condor.  To put this in context, (as of 2009) the average UK household emits 5.3 tons annually.
 +
 +
=Appendices=
 +
 +
If you would like to send your job to a particular machine, you should augment the requirements line in your submission script as below:
 +
 +
<pre>
 +
requirements = ..... && TARGET.Machine=="geog-XXX.ggy.bris.ac.uk"
 +
</pre>
 +
 +
You should check that the machine is '''unclaimed''' prior to submitting, using '''condor_status'''.

Latest revision as of 16:00, 5 July 2010

Condor: Making best use of the computers in the teaching labs

Introduction

The Condor Project enables us to run batch jobs on idle CPUs found in a pool of desktop PCs, such as those in offices and computer labs in the department. Condor is particularly useful for high 'throughput' computing. An is example of this is the use of an ensemble of independent model simulations to explore parameter-space.

Basic commands

You can run the following commands from the submission host of your condor pool. In Geography, the submission hosts are condor and dylan (Note that these are Linux servers).

You can review the status of all the computers in the 'condor pool' using the command condor_status: 

Name               OpSys      Arch   State     Activity LoadAv Mem   ActvtyTime

slot1@GEOG-B224.gg WINNT51    INTEL  Owner     Idle     0.000  1661  0+01:30:04
slot2@GEOG-B224.gg WINNT51    INTEL  Owner     Idle     0.010  1661  0+01:30:05
slot1@geog-a105.gg WINNT51    INTEL  Claimed   Busy     1.130  1661  0+01:02:57
slot2@geog-a105.gg WINNT51    INTEL  Claimed   Busy     1.130  1661  0+01:02:58
slot1@geog-c200.gg WINNT51    INTEL  Unclaimed Idle     0.000  1662  0+00:00:04
slot2@geog-c200.gg WINNT51    INTEL  Unclaimed Idle     0.040  1662  0+00:00:00
...
...
                     Total Owner Claimed Unclaimed Matched Preempting Backfill

       INTEL/WINNT51   191    78     109         4       0          0        0

               Total   191    78     109         4       0          0        0

Typically you will get several screen's full of output, so I've chopped out a large swathe of the listing. This leaves us with just few examples from the top of the listing and the final summary, given at the end.

From the listing, you can see that:

  • The PC called GEOG-B224 has someone logged into it, indicated by the keyword Owner, but it is not working hard and so is marked as Idle.
  • In contrast, geog-a105 is marked as Claimed, indicating that it has been grabbed by Condor, and it working hard; marked Busy.
  • The third possible state is exemplified by geog-c200, which is neither claimed nor in interactive use.

The final summary tells us that at the time of writing, the pool contains 191 PCs, 78 of which has a user logged in, 109 are claimed by condor and 4 remain unclaimed.

Another view of the state-of-play is given by condor_q: 

 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD               
 51.0   ggdagw          4/22 13:21   0+01:30:07 R  0   3.7  test.bat          
...
...
 51.15  ggdagw          4/22 13:21   0+01:33:46 I  0   3.4  test.bat   
...
...
150 jobs; 44 idle, 106 running, 0 held

Here we see that:

  • job 51.0 was submitted at 13:21 on the 22nd of April, has been running for just over an hour and a half, and that the job executable is called 'test.bat'
  • We can also see that job 51.15 is idling in the queue, rather than running.
  • In total 106 of the 150 jobs submitted to condor are running, and accordingly 44 are still waiting to run and so are idle.

Submitting a simple script job

OK, so much for other people's jobs. Let's submit some of our own! I have prepared some examples to make this as easy as possible. To get these examples, you can cut & paste the following onto your (Linux) command line (If you have a Windows submission host, the same files will work for you, but you will need to use TortoiseSVN to get them): 

svn co https://svn.ggy.bris.ac.uk/subversion-open/condor/trunk ./condor
cd condor/examples/example1

Without further ado, let's submit our first job to the pool: 

condor_submit win.submit

If you look inside win.submit, you will see that it is a short and reasonably self explanatory file: 

Universe   = vanilla
Notification = never
requirements = OpSys == "WINNT51" && Arch == "INTEL"
Output = test.out
Error = test.err
Log = test.log
should_transfer_files = YES
when_to_transfer_output = ON_EXIT_OR_EVICT
Executable = test.bat
Queue

The key lines to note for the moment are:

  • the output of the job will collect in test.out
  • any errors will go to test.err
  • test.log will record the mechanics of sending the job to a remote PC
  • and that test.bat calls the shots!
  • the Queue keyword causes the job to enter the queue of jobs to be run

The executable file test.bat is a short and simple batch file (aka 'shell script' in Linux-speak): 

echo Hello from a Condor batch file running on:
hostname
echo The date is:
date /T
echo and the time is:
time /T

The upshot of all this jiggery-pokery are the contents of test.out. All being well, this should have been created by now (if not, consult condor_status and condor_q for more information on the state-of-play), and look something like: 

Hello from a Condor batch file running on:
geog-c211
The date is:
22/04/2009 
and the time is:
15:36

Congratulations! You've run your first condor job That's the hardest part out of the way. Honest! All that remain are a few more details. So far so good? OK, let's move on to the next example.

Submitting a compiled executable

Move to the next example directory by typing: 

cd ../example2

In this example, we'll run a time-honoured "hello, world" job. I've included a pre-compiled executable for you to run. Note that I am assuming that your jobs will be run on Windows based PCs Note that despite this, we are submitting from a Linux server. This is fine since Condor can handle it, but we must remember it and not get our operating systems and compiled executables in a muddle.

To this end, I compiled hello_world.exe on my PC, as a DOS (or console) program. I hope it will run on your PCs too. When running jobs that you have compiled yourself, you must remember to compile it on Windows and to make sure that it will run on another Windows PC--not just your own--and then use, for example, the ssh file transfer tool to bring it over to the submission host. If you are in Geography and do not have a Windows C/C++ or Fortran compiler, contact me and I can probably compile your code for you.

OK, now that's all straight, let's run the job. We can do this as before: 

condor_submit win.submit

Note that I have slightly modified the contents of win.submit: 

...
Output = hello_world.out
Error = hello_world.err
Log = hello_world.log
Executable = hello_world.exe
Queue

And, all being well, we have a friendly message in test.out: 

Hello World!

Jobs which use inputs

Sometimes we need our batch job to read some inputs from file. We'll do this in the next example: 

cd ../example3
condor_submit win.submit

In this case, we've added the file test.in to the bones of example1. This is used in test.bat: 

...
type test.in

In DOS-speak, this command will just echo the contents of this file, which will end up and the bottom of test.out: 

...
tada! I'm from the input..

and we reference the input file in win.submit: 

...
transfer_input_files = test.in
...

Well, that wasn't too bad, eh?!

Using command line arguments

In addition to input files, we may find that we need to pass a program some command line arguments. This is easy enough also. 

cd ../example4

An additional line to win.submit is: 

arguments = 6

and the result of the program is: 

8 is the 6th Fibonacci number

Running an ensemble of jobs

Some of the real benefits of Condor come when we want to submit a number of jobs at the same time. Such an 'ensemble' is often used to explore parameter-space. Note that such an ensemble is often referred to as a 'cluster' in Condor-speak. This refers to a cluster of jobs. Not to be confused with a cluster of servers used in some High Performance Computing setup. 

cd ../example5

In this case, we're going to submit several jobs which all make use of the same Fortran program. Let's take a look at the salient parts of win.submit on this occasion: 

...
transfer_input_files = input.nml
Output = test.out
Error = test.err
Log = test.log
Executable = fort_nml_io.exe
initialdir = job.$(Process)
Queue 3

The executable program file called fort_nml_io.exe is expecting to read an input file with the name input.nml. We would like to feed the program with different versions of the input file (providing different values for some parameters) for the different runs in our ensemble. However, we can't have multiple files with the same name in the same directory. (Things are a good deal easier using C/C++, since we can pass the name of the relevant input file as a command line argument. The Fortran language does not support command line arguments, as a rule).

To get around this, we can create different sub-directories to hold the relevant files for each run. These will be called job.0, job.1, job.2 etc. A nifty piece of scripting allows us to refer to these in the submission file as job.$(Process). The number of jobs submitted is determined by the value given to Queue on the last line. In this case we have specified Queue 3 and we will get 3 jobs submitting, with $(Process) taking the values 0, 1 and 2 in turn, as a consequence.

We run the ensemble in the usual way: 

condor_submit win.submit

Take a look at the contents of each of the sub-directories. Notice also that any output files created by the executable are also be transferred back from the PC which ran it, and placed in the relevant sub-directory.

Further Examples

Well, in 5 small steps you've covered the majority of the ground required to make effective use of a Condor pool. However, if you would like some more information and Condor, the following pages are useful:

  • A few more examples of submission scripts are given by the authors of Condor in the following page: Condor quick start

Condor and energy saving measures

In an effort to not to waste energy, we have set PCs which have been idle for several hours to shutdown (or hibernate). These PCs are periodically (once early in the morning and again at lunchtime) 'woken-up', so that they are ready to receive jobs again. If there are no calls on them for work the PCs will shutdown again after a few more hours have elapsed.

The Carbon Trust tells us that to produce 1kWh of grid electricity, 0.537 Kg of CO2 is emitted. We calculate that we can save up to 70 tons of CO2 emissions annually from the Geography department, through enabling the shutdown facility in Condor. To put this in context, (as of 2009) the average UK household emits 5.3 tons annually.

Appendices

If you would like to send your job to a particular machine, you should augment the requirements line in your submission script as below: 

requirements = ..... && TARGET.Machine=="geog-XXX.ggy.bris.ac.uk"

You should check that the machine is unclaimed prior to submitting, using condor_status.

Retrieved from "https://source.geography.bristol.ac.uk/mediawiki/index.php?title=Condor&oldid=7469"