A Simple MPI Program Execution
Systems & Access :: Learning to Use :: Linux Computing Resource
Example of an MPI Program mpitest.c
/* program hello */
/* Adapted from mpihello.f by drs */
#include <mpi.h>
#include <stdio.h>
int main(int argc, char **argv)
{
int *buf, i, rank, nints, len;
char hostname[256];
MPI_Init(&argc,&argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
gethostname(hostname,255);
printf("Hello world! I am process number: %d on host %s\n", rank, hostname);
MPI_Finalize();
return 0;
}
Example of MPI Program Execution Using mpiexec
The preferred means of starting MPI programs on the Linux cluster is mpiexec. The main advantage of using mpiexec over mpirun is that there is no need for sourcing any setup file during the program execution. However, you still must source the setup file before compiling your program. It is a lot easier to write the PBS script file using mpiexec than mpirun. The example MPI program above could be executed with the following PBS script that uses mpiexec.
Sample mpitest.pbs Using mpiexec
#!/bin/csh
#*** The "#PBS" lines must come before any non-blank non-comment lines ***
#PBS -l walltime=30:00,nodes=4:myri:ppn=2
set mycommand = "./mpitest" # note "./" is used to not rely on $PATH
set myargs = "" # arguments for $mycommand
set infiles = "" # list of input files
set outfiles = "" # list of output files
if ($?PBS_NODEFILE && $?PBS_JOBID) then
#count the number of processors assigned by PBS
set NP = `wc -l < $PBS_NODEFILE`
echo "Running on $NP processors: "`cat $PBS_NODEFILE`
else
echo "This script must be submitted to PBS with 'qsub -l nodes=X'"
exit 1
endif
# set up the job and input files, exiting if anything went wrong
cd "$PBS_O_WORKDIR" || exit 1
if ($?infiles) then
foreach file ($infiles)
cp -v $file /scratch/ || exit 1
end
endif
cd /scratch || exit 1
# run the command, saving the exit value
mpiexec $PBS_O_WORKDIR/$mycommand $myargs
set ret = $?
# get the output files
cd "$PBS_O_WORKDIR"
if ($?outfiles) then
foreach file ($outfiles)
cp -vf /scratch/`basename $file` $file
end
endif
echo "Done " `date`
exit $ret
Sample Run
To run a job, the first step is to source the setup file:
hpc-master: source /usr/usc/mpich/1.2.5..12/gm-cdk/setup.csh
The second step is to compile the program:
hpc-master: mpicc -o mpitest mpitest.c
The third step is to submit the .pbs file to the queue. When the job submission is successful, the qsub command returns the job's process number:
hpc-master: qsub mpitest.pbs
12240.hpc-pbs.usc.edu
The output files for the above run are created in the same directory where the qsub command was executed. The output files are created using the same filename as your PBS script filename with an extension that includes "e" for "error" or "o" for "output" followed by the process number. In our example above the file mpitest.pbs.e12240 contains the error messages and the file mpitest.pbs.o12240 contains the output of the program. The output of the program can be viewed by listing the output files:
hpc-master: ls -l mpitest.pbs.*
-rw------- 1 nirmal rds 0 Jul 16 14:59 mpitest.pbs.e12240
-rw------- 1 nirmal rds 1442 Jul 16 14:59 mpitest.pbs.o12240
hpc-master: more mpitest.pbs.*
::::::::::::::
mpitest.pbs.e12240
::::::::::::::
::::::::::::::
mpitest.pbs.o12240
::::::::::::::
----------------------------------------
Begin PBS Prologue Wed Dec 8 16:49:24 PST 2004
Job ID: 12240.hpc-pbs.usc.edu
Username: nirmal
Group: rds
Nodes: hpc0632 hpc0633 hpc0634 hpc0636
PVFS: mounting at /scratch
End PBS Prologue Wed Dec 8 16:49:27 PST 2004
----------------------------------------
Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
Starting 12240.hpc-pbs.usc.edu Wed Dec 8 16:49:27 PST 2004
Initiated on hpc0636
Running on 8 processors: hpc0636 hpc0636 hpc0634 hpc0634 hpc0633 hpc0633 hpc0632 hpc0632
Hello world! I am process number: 4 on host hpc0632
Hello world! I am process number: 3 on host hpc0634
Hello world! I am process number: 0 on host hpc0635
Hello world! I am process number: 2 on host hpc0634
Hello world! I am process number: 6 on host hpc0631
Hello world! I am process number: 1 on host hpc0635
Hello world! I am process number: 5 on host hpc0632
Hello world! I am process number: 7 on host hpc0631
Done Wed Dec 8 16:49:28 PST 2004
--------------------------------------------------
Begin PBS Epilogue Wed Dec 8 16:49:33 PST 2004
Job ID: 12240.hpc-pbs.usc.edu
Username: nirmal
Group: rds
Job Name: mpitest.pbs
Session: 4020
Limits: neednodes=4:myri:ppn=2,nodes=4:myri:ppn=2,walltime=00:02:00
Resources: cput=00:00:00,mem=12648kb,vmem=25752kb,walltime=00:00:04
Queue: quick
Account:
Nodes: hpc0632 hpc0633 hpc0634 hpc0636
Killing leftovers...
End PBS Epilogue Wed Dec 8 16:49:35 PST 2004
--------------------------------------------------
