Moab Batch System Guide

User introduction

The commands for managing jobs only work on the nodes linx64 and wukong and ock.
The command qstat shows the running jobs.
qstat -q shows the available queues and their time and memory and nr of CPU limits.
qsub my.job is used to submit jobs. You can either supply flags to the command or include options in your script, which is the recommended way.
pbsnodes -a is used to check the status and properties of the nodes in the clusters.
In addition to these torque commands, there are the moab commands that do the same thing. Sometimes they provide more detailed information. The command showq show thejobs in the queue like qstat.
checkjob JOBID gives details about the job. This is most useful when the job does not run yet and may tell you why it is waiting, although the message can be very cryptic.
The command checknode NODENAME tells you things about the node, such as which jobs are running on it and which jobs in the queue have a reservation to run on it. The reservation will show the estimated start time.

Queue definitions

The queue definitions can be checked with qstat -q and qstat -Q -f.

1. quick Used for debugging and testing and short production jobs.
2. clever Used for all production work.
3. brute Use this queue for jobs that run very long, such as old serial programs that have not be parallelized.

#PBS -l pmem=600mb
	  

Property definitions

Torque allows each node to be assigned properties. These can be specified in the job scripts so that you can direct Moab to allocate the correct node(s) to your job. You can check the properties with pbsnodes -a.

1. The property arwen, haku, ra, surg, wukong, ock denotes the group of nodes belonging to the cluster with that name.
2. i686, x8664, amd, ia64 denotes the CPU architecture. The nodes in Arwen have IA32 architecture and the arch command returns i686. The nodes in Haku have EM64T architecture and the arch command returns x86_64; since Torque does not allow underscore in properties, we use x8664. The nodes in RA are AMD Opteron 64 bit processors and the arch command returns x86_64; we defined the additional property amd for the Ra nodes to distinguish them from Haku nodes. The Altix ock has IA64 architecture and the arch command returns ia64.

Use of switches for parallel jobs

Moab will try to schedule parallel jobs so that all nodes allocated to a job will come from a group of nodes on the same switch to reduce communication delays. This is automatic. The groups are internally called bc1, bc2, hs0, hs1, rs0, rs1, rs2, rs3, ws0. They denote the groups of 14 nodes in Arwen, of 16 nodes in Haku, of 19 nodes in RA, and of 16 nodes in wukong, that share the same switch.

Also note that by default Torque scripts start in your $HOME directory, not the directory where you are when you submit the job. The error and output files are created by default in the directory where you submit the job.

Moab scheduling principles

The Moab scheduler is programmed using the following principles. You can plan and organize your work and submit the jobs to get the best turnaround time for your jobs.

• Jobs in the queue from users who not have any running jobs at that time, have preference over jobs in the queue from users who also have running jobs. As a result, if you have no jobs running, the first cores that come free will be yours.
• Jobs requesting many nodes are considered first, and single node-jobs are scheduled to fill the "holes". The same is true for jobs requesting many CPUs. Because one can ask for many CPUs arbitrarily distributed -l ncpus=8 or for CPUs distributed over nodes in a special way -l nodes=2:ppn=2, the scheduler looks at the nodes first, because this requirement is harder to satisfy.
• Jobs requesting a shorter wall clock time are considered before those with longer wall clock time limits. CPU time limits are not considered.

Serial job example

A simple serial job (download the job script file)

#!/bin/sh
#PBS -N serial
#PBS -o serial.out
#PBS -e serial.err
#PBS -l nodes=1:ppn=1:wukong
#PBS -l walltime=12:00:00
#PBS -l pmem=7500mb
echo Testing...
hostname
echo run a serial program with 7.5 GB of RAM...
echo done
	  

The standard output and standard error files are specified with "-o" and "-e".

The flag "-m abe" specifies the a mail message should be sent to the address specified with "-M" when the job "b"egins, "e"nds, and "a"borts.

The queue in which to run the job must be specified and it must be consistent with the requested walltime "-l walltime=12:00:00". Note that Moab schedules jobs that request shorter walltimes sooner and delays jobs that ask for a very long walltime.

The "-l nodes=1:ppn=1:wukong" specifies that this job asks for 1 CPU resource, in the form 1 node and 1 processor per node. It also specifies the property "wukong", so that the job will run only on wukong nodes. You can also specify "-l ncpus=1". To specify the type of node, you then also need "-l nodes=wukong". Since the configuraqtion specifies that each node has 2, 4 or 8 CPUs, two jobs can be run on each physical node, if they each ask for one CPU resource. You can use qstat -f jubname to see what node/CPU combination has been allocated to your job(s). This is also shown graphically on the cluster status pages on this web site.

It is also a good thing to specify the memory required by your program with "-l pmem=900mb". The default is 900mb for QTP and 600 mb for HPC. For parallel jobs, this is the memory neede per processor.

Parallel jobs examples

The following is a parallel job to run an OpenMP or POSIX Threads shared memory, or OpenMPI distributed memory parallel program on a single wukong node. QTP clusters have 2, 4 or 8 cores per node so 8 is the maximum you can ask for this kind of parallel job. (download the job script file)

#!/bin/sh
#PBS -N para
#PBS -o para.out
#PBS -e para.err
#PBS -q quick
#PBS -l nodes=1:ppn=8:wukong
echo Testing single node parallel...
hostname
# some preparations here

# run your shared memory parallel program here
# for example g03 with 8 processors
g03 benzee.inp > benzene.log

# or run 8-way parallel MPI job with mpirun
mpirun vasp chickenwire.inp > chickenwire.log

# some cleanup here
echo done.

The following is a parallel job to run a OpenMPI program on multiple nodes, with 2 CPUs (or cores) per node. (download the job script file)

#!/bin/sh
#PBS -N para
#PBS -o para.out
#PBS -e para.err
#PBS -q quick
#PBS -l nodes=2:ppn=2:arwen
echo Testing parallel...
hostname
# simple way....
mpirun hello > para.log 2>&1

# hard way...
echo PBS_NODEFILE
cat $PBS_NODEFILE
N=`wc -l $PBS_NODEFILE | awk '{print $1}'`
echo Nr nodes $N
echo starting hello...
mpirun -np $N hello > para.log 2>&1
echo done.

The environment variable PBS_NODES is set by PBS and is the name of a file with the hosts that are assigned to you by PBS.

You can download the small program hello.cpp to try running this job.

Administrator introduction

1. Torque server daemon The server daemon pbs_server runs on wukong is the interface that points to the nodes of the cluster. The configuration of the daemon is done through commands. These are collected in the file server_config_cmds.

   All nodes are listed in a nodes file with a property of bc1 or bc2, indicating which BladeCenetr the node is part of and two CPUs, as follows:

   arwen00 bc1 arwen i686 np=2
   arwen01 bc1 arwen i686 np=2
   ...
   arwen10 bc2 arwen i686 np=2
   ...
   haku00 hs0 haku x8664 np=2
   ...
   haku10 hs1 haku x8664 np=2
   ...
   ra00 rs0 ra x8664 np=2
   ...
   ra10 rs1 ra x8664 np=2
   ...
   ra20 rs2 ra x8664 np=2
   ...
   ra30 rs3 ra x8664 np=2
   ...
   wukongf np=8 ws0 wukong x8664
   

   Jobs will be allocated to node/CPU slots.

   The properties defined for the nodes allow you to target specific groups of nodes easily.

2. Torque execution daemon Each node runs a pbs_mom daemon with configuration file

   arch x86_64
   logevent       511
   $clienthost     wukong
   $restricted     wukong
   $ideal_load	8.0
   $max_load	10.0
   $usecp	*:/	/
   $tmpdir /scr_1/tmp
   size[fs=/scr_1/tmp]
   $timeout 120
   $fatal_job_poll_failure false
   	      

3. Moab scheduler The scheduler Moab also runs on wukong. The scheduler configuration can be complex and is defined in the file moab.cfg. See the Cluster Resources website
   

>> top