Oscar Queues

When submitting a job to the Oscar compute cluster, you can choose different queues depending on the nature of your job. You can specify one of the following queues either in your qsub command:

qsub -q <queue_name> <batch_script>

or as a specification within your batch script:

#PBS -q <queue_name>

default

If you do not specify a queue, your job will be submitted to the default queue. This queue provides access to the Oscar compute nodes. While your job is running, it will have exclusive access to the nodes that have been reserved for it.

debug

A small pool of 4 Oscar compute nodes has been set aside for short-running jobs for debugging or testing purposes, and should have a faster turn-around time than the default queue. The time limit is 40 node-minutes, meaning you can use 4 nodes for up to 10 minutes, or 1 node for up to 40 minutes. Users can have only one eligible debug job. You may also want to run jobs interactively through this queue (see below).

timeshare

If you have a serial or threaded program that requires large memory and does not need dedicated access to the node it runs on, you can submit it through the timeshare queue. This queue packs multiple jobs onto the SMP nodes. In addition to walltime, you must specify an upper limit on how much memory your job will use. You can also use this queue for interactive access.

Interactive Jobs

You can request an interactive job from the batch system by adding the -I parameter to the qsub command:

qsub -I -q <queue_name> -l nodes=<# of nodes>,walltime=<time requested>

The qsub command will wait until your job is ready to run, and then will forward you to the head node, where you can type commands interactively. Instead of using qsub directly, you can also use a special wrapper command we have written called interact.

Queue Priority

The scheduler considers many factors when determining the run order of jobs in the queue. These include the:

  • size of the job
  • requested walltime
  • amount of resources you have used recently (e.g., "fair sharing")
  • priority of your account type

The account priority has three tiers:

  • Low (Exploratory)
  • Medium (Premium)
  • High (Condo)
Both Exploratory and Premium accounts can be affiliated with a Condo, and the Condo priority only applies to a portion of the cluster equivalent in size to the Condo. Once the Condo affiliates have requested more nodes than available in the Condo, their priority drops down to either medium or low, depending on whether they are a Premium or Exploratory account.

Backfilling

When a large or long-running job is near the top of the queue, the scheduler begins reserving nodes for it. If you queue a smaller job with a walltime shorter than the time required for the scheduler to finish reserving resources, the scheduler can backfill the reserved resources with your job to better utilize the system. Here is an example:

  • User1 has a 64-node job with a 24 hour walltime waiting at the top of the queue.
  • The scheduler can't reserve all 64 nodes until other currently running jobs finish, but it has already reserved 38 nodes and will need another 10 hours to reserve the final 26 nodes.
  • User2 submits a 16-node job with an 8 hour walltime, which is backfilled into the pool of 38 reserved nodes and runs immediately.

By requesting a shorter walltime for your job, you increase its chances of being backfilled and running sooner. In general, the more accurately you can predict the walltime, the sooner your job will run and the better the system will be utilized for all users.