Portable Batch System (PBS)
Systems & Access :: Learning to Use :: Linux Computing Resource
The Portable Batch System (PBS) is used to allocate nodes for a job and run it. You must use PBS to run jobs on the compute nodes. If the user account is not active or does not have sufficient NWCH, the job will not run.
Documentation for PBS can be found at http://www.usc.edu/hpcc/pbsman/man1/pbs.html, which is a copy of the PBS manpage. The same document also is accessible on hpc-master with the command man pbs. Links to all other PBS commands can be found at http://www.usc.edu/hpcc/pbsman.
In order to use the uschpc compute nodes, you must log on to uschpc at hpc.usc.edu and then submit a PBS job. To interact with the PBS queue, you primarily use the qsub and qstat commands. The qsub command is used to submit a job to the PBS queue, and the qstat command is used to check on the status of a job already in the PBS queue.
Using qsub
In order to run a job through PBS, you first create a control file that contains the commands to be executed. Typically, this is a PBS script. This script is then submitted to PBS using the qsub command. The command
qsub myjob.pbs
is the simplest form of a submission, but typically you will want to ask PBS for additional resources. A job on the uschpc is allowed, by default, a maximum of 0.5 NWCH (30 minutes). If your job needs to run longer than that, the control file or the command line should specify how much time is required using the walltime resource. The command
qsub -l walltime=2:00:00 myjob.pbs
will ask PBS for a limit of 2 NWCH. If your job does not finish within the specified time, it will be terminated.
You also need to tell PBS how many nodes and processors you require. If you need four uschpc nodes and you are only going to use one of the dual-processors on each node (no one else can use the other processor while you are using the node, so you are wasting one processor), then you should use the following command
qsub -l walltime=2:00:00,nodes=4 myjob.pbs
If you want to use both processors on each uschpc node, you should use the following command
qsub -l walltime=2:00:00,nodes=4:ppn=2 myjob.pbs
The jobname defaults to the first 15 characters of the name of the script submitted (see -N option). XX is the job's sequence number assigned by PBS.
If you need 32-bit node nodes, then you specify:
qsub -l walltime=2:00:00,nodes=4:ppn=2,arch=i686 myjob.pbs
If you need 64-bit node nodes, then you specify:
qsub -l walltime=2:00:00,nodes=4:ppn=2,arch=x86_64 myjob.pbs
If you frequently use the same qsub options for a given script, PBS lets you put the options in the script itself so that you do not have to type them every time you submit a job.
For the above example, you would put the following lines at the beginning of your script (myjob.pbs):
#Run on 8 processors on uschpc
#PBS -l walltime=2:00:00
#PBS -l nodes=4:ppn=2
Then when you submit subsequent jobs, you only need to enter the following command
qsub myjob.pbs
If you have the myjob.pbs set up as explained above and you want to override the default options in then myjob.pbs file, then the -l parameters on the qsub command line will override the option specified in the file.
Remember that the script that you submit to the PBS queue may run on a system that requires a brand new login. Any setup that you require for your program to run will have to be duplicated in the PBS script. This includes changing your current directory from the home directory to the directory required for your job. PBS provides a variable PBS_O_WORKDIR, that tells you what your directory was when you executed the qsub command. If you want your job to start from that same directory, you can put the following line in your script right after and #PBS lines:
cd "$PBS_O_WORKDIR"
Access to uschpc is governed by the HPCC allocation policy. You must submit a request and be granted an allocation. If you have more than one allocation because you are involved in more than one project or because you are taking a class in parallel programming, you may have multiple allocations. If so, you must specify which account you are using either on your qsub command or in a #PBS line in your PBS script.
The following command will submit your job to PBS using the account lc_drs (note that the account is specified with an uppercase A):
qsub -A lc_drs
You can find out what accounts you may access and their balances with the qbalance command:
qbalance -h
