PBSPro 7.0 Batch System Guide

User introduction

The commands for managing jobs only work on the node arwen.
The command qstat shows the running jobs.
qstat -q shows the available queues and their time and memory and nr of CPU limits.
qsub my.job is used to submit jobs. You can either supply flags to the command or include options in your script, which is the recommended way.
pbsnodes -a is used to check the status and properties of the nodes in the clusters.

Queue definitions

1. quick Time limit 2 hours. Used for debugging and testing and short production jobs.
2. clever Time limit between 2 and 96 hours and up to 12 node parallel (24 CPUs). This is how clever people use the system.
3. brute Time limit longer than 96 hours and up to 6 node parallel (12 CPUs). This is for brute force and not so clever use of the system.

Property definitions

1. arwen, haku, ra denotes the group of nodes belonging to the cluster with that name.
2. i686, x8664, amd denotes the CPU architecture. The nodes in Arwen have IA32 architecture and the arch command returns i686. The nodes in Haku have EM64T architecture and the arch command returns x86_64; since OpnePBS does not allow underscore in properties, we use x8664. The nodes in RA are AMD Opteron 64 bit processors and the arch command returns x86_64; we defined the additional property amd for the Ra nodes to distinguish them from Haku nodes.

Use of switches for parallel jobs

Also note that by default PBS scripts start in your $HOME directory, not the directory where you are when you submit the job. The error and output files do live in the directory where you submit the job by default.

A simple serial job (download the job script file)

#!/bin/sh
#PBS -N serial
#PBS -o serial.out
#PBS -e serial.err
#PBS -m abe
#PBS -M deumens@qtp.ufl.edu
#PBS -q quick
#PBS -l nodes=1
echo Testing...
hostname
echo run a serial program...
echo done

The following is a parallel job to run a LAM MPI program (download the job script file)

#!/bin/sh
#PBS -N para
#PBS -o para.out
#PBS -e para.err
#PBS -m abe
#PBS -M deumens@qtp.ufl.edu
#PBS -q quick
#PBS -l nodes=4:bc2:ppn=2
cleanup() {
  lamhalt -v
  echo job killed
  exit
}
echo Testing parallel...
hostname
echo PBS_NODEFILE
cat $PBS_NODEFILE
N=`wc -l $PBS_NODEFILE | awk '{print $1}'`
echo Nr nodes $N
lamboot -v $PBS_NODEFILE
# Catch TERM and KILL signals to shut down MPI
trap cleanup TERM KILL
echo starting hello...
mpirun -np $N hello > para.log 2>&1
lamhalt -v 
echo done.

Administrator introduction

1. server daemon The server daemon runs on arwen, arwengw is the interface that points to the nodes of teh cluster.
   All nodes are listed in a nodes file with a property of bc1 or bc2, indicating which BladeCenetr the node is part of and two CPUs, as follows:

   arwen00 bc1 arwen i686 np=2
   arwen01 bc1 arwen i686 np=2
   ...
   arwen10 bc2 arwen i686 np=2
   ...
   haku00 hs0 haku x8664 np=2
   ...
   haku10 hs1 haku x8664 np=2
   ...
   ra00 rs0 ra x8664 np=2
   ...
   ra10 rs1 ra x8664 np=2
   ...
   ra20 rs2 ra x8664 np=2
   ...
   ra30 rs3 ra x8664 np=2
   ...
   

   Jobs will be allocated to node/CPU slots. Using the nodes as timesharing nodes does not work very well for parallel jobs that need to run across different nodes; but this may be my misunderstanding.

   The properties defined for the nodes allow you to target specific groups of nodes easily.

2. execution daemon Each node runs a pbs_mom daemon with configuration file

   $logevent       511
   $clienthost     arwengw
   $restricted     arwengw
   $ideal_load     2.0
   $max_load       2.1
   $usecp  *:/     /
   

   There is no execution daemon on the management node arwen or on the file server arwensrv.
3. scheduler The scheduler daemon also runs on arwen. The scheduler used is the simple FIFO with load balancing turned on:
   load_balancing: true ALL
   

>> top