Submitting jobs to the Oscar queue
Applications are run on the compute nodes of the Oscar cluster as batch jobs. To run a batch job, you must submit a batch script that specifies how your job will run. This may include how many nodes and cores the job will use, how much time the job needs, and the name of the application, the location of its executables and any parameters or input it needs. Once submitted, the job will advance in the queue until it has reached the top, when it will then be launched on the compute nodes. The output of the job can be viewed both during and after the job runs.
A sample batch script is available in every user's home directory under the name batch.script. You can copy and modify this sample script, or you can start a new script from scratch using the following as a guide.
A batch script begins by specifing the bash shell as its interpreter:
#!/bin/bash
Next, a series of lines starting with #PBS specify the properties for your job, including:
Name
#PBS -N JobName
The name will help you and other users identify your job when it is listed in the queue. It is also used as the name for job's output files.
Time
#PBS -l walltime=00:10:00
The 'walltime' is an estimate of how long you believe your job will run. If your job runs for longer, it will be terminated by the scheduler. However, the shorter your walltime, the faster your job will make its way throug the queue.
Nodes
#PBS -l nodes=2:ppn=8
This line specifies how many nodes your job requires, and how many "processors per node" (or CPU cores) you will use on each node. If you have an MPI application, the product nodes*ppn will be how many MPI tasks are started. If you have a serial or threaded application that will run on only one node, you do not have to specify ppn. If you try to request a ppn greater than 8 (the number of cores on an Oscar node), your job will become stalled in the queue.
Note: you should only use more than one node if your application uses MPI or an equivalent library for passing messages between nodes. Serial or threaded applications will not be able to run on more than one node.
Other Options
While your job is running, the scheduler will temporarily copy the output and any error messages to the files
~/<jobid>.OU
~/<jobid>.ER
When your job finishes, the final output is by default placed in the directory where you submitted the script in two files named JobName.o<jobid> and JobName.e<jobid>. If your script specifies
#PBS -j oe
then your output and errors will be combined into only the JobName.o<jobid> file. You can also specify a different name and path for the output file with:
#PBS -o /path/to/output
Next, the script change directories to where it was submitted from:
cd $PBS_O_WORKDIR
and outputs additional information about which Oscar compute nodes it runs on and what directory it is running from:
echo Master process running on `hostname` echo Directory is `pwd` echo PBS has allocated the following nodes: echo `cat $PBS_NODEFILE` echo Starting execution at `date` NPROCS=`wc -l < $PBS_NODEFILE` echo This job has allocated $NPROCS CPUs
Finally, you must specify what application to execute, along with any parameters:
MPI Applications
# execute an MPI program mpirun -np $NPROCS mpiprogram <args>
The "mpirun" wrapper is used to launch an MPI application, and will start up the MPI tasks across the compute nodes.
Threaded Applications
# execute a threaded program myprogram <args> <-nt 8>
Most threaded applications will have a parameter such as "-nt" that allows you to specify the number of threads to use. Choose 8, the number of cores on an Oscar compute node.
Serial Applications
If you have a serial application you need to run with different inputs or parameters, you can "pack" up to 8 instances of the application to run concurrently on a single Oscar node. To do this, list the separate invocations of your application like this:
# execute 8 serial programs, in order to utilize all cores on a single node # The "&" causes each program to be run as a background process # The "wait" causes the batch script interpreter to wait until all # background processes have completed before continuing. myprogram args1 & myprogram args2 & myprogram args3 & myprogram args4 & myprogram args5 & myprogram args6 & myprogram args7 & myprogram args8 & wait
The final wait command keeps the job running until the last program finishes.
If you have a serial application that you need to run over many parameters (sometimes called a "parameter sweep"), please submit a support ticket and we will help you setup a custom script to achieve better load balancing and more effective use of the system.
Submitting with qsub
Once you have saved your batch script (for instance in a file called my.job), you can submit it to the queue with the command
qsub my.job
on an Oscar login node.
The Oscar system uses the Moab Cluster Suite for batch workload management, which includes Torque as the batch queue system. Torque provides many options for controlling the behavior of the job on the system. Some of these are documented in the sample batch script found in your home directory (~/batch.script). The complete set can be found in section 2 of the Torque manual.