Message Passing Interface (MPI): Tutorials
Systems & Access
:: Learning to Use ::
Linux Computing Resource
Some general information about MPI and tutorials on how to use it are available from numerous sites, including the following places, among others:
Installation Details of MPICH
MPICH is an open-source, portable implementation of the Message Passing Interface Standard. It contains a complete implementation of version 1.2 of the MPI Standard and also significant parts of MPI-2, particularly in the area of parallel I/O. MPICH is both a runtime and compile-time environment for MPI-compliant code.
mpich versions:
Solaris:
mpich 1.2.6 - open source Message Passing Interface
Linux:
mpich 1.2.7..5 - mpich over mx
MPICH-MX is mpich ported by Myricom for use on its MX transport for its Myrinet networking architecture. For more information, Please visit: http://www.myrinet.com/scs/index.html.
Transport Device
MPICH provides an abstraction layer for interprocess communication regardless of the operating environment, be it a single massively parallelized shared-memory machine or a cluster of distributed machines. The different
environments use different devices for communication. Each device is an underlying transport layer for the message passing. Examples of devices are p4 for clusters of machines on normal TCP/IP networks, mx for clusters of machines on Myrinet networks, shmem for shared memory SMP machines, and p4shmem for clusters of SMP machines on TCP/IP networks.
The Linux builds have support for mx and p4 devices. The mx device is Myrinet's high-speed transport layer and is available on all nodes.
The Solaris builds support the shmem, p4, and p4shmem devices. Use shmem to run your job on single shared-memory SMP machines such as almaak (rcf-01). Use p4 to run on a cluster of machines, such as HPCC's ~120 node Condor flock. p4shmem is a combination of the two devices for clusters of SMP machines. The p4shmem is not particularly useful in almaak's PBS queue.
Compilers
MPICH is built with support for several different compiler sets, but the user must choose one.
MPICH supports C, C++, F77, and F90. However not all compiler sets support all languages. Portland Group's CDK and Intel support all four languages. KCC supports only C++. Absoft supports F77 and F90. GNU supports, C, C++, and F77. For KCC and Absoft, GNU 3.2.3 is used for other cases.
MPICH is tightly integrated with the operating system and the compiler set the user intends to use. This version of mpich was built with various sets of compilers:
Solaris:
GNU - Open source compiler from the GNU project (2.95.2)
SPAC - Native Solaris sparc compilers
Linux:
GNU32 - Open source compiler from the GNUproject (3.4.3)
PGI - Portland Group's "Cluster Development Kit" (6.0-2)
INTEL - Intel's C++ and Fortran compilers (8.1)
Bindings
Each device and compiler set is called a binding and each one is very distinct from the others. Only one binding can be used at time. Care must be taken to ensure that the same binding is used at both compile-time and runtime.
Each binding is installed into:
/usr/usc/mpich/version/device-compiler[-subarch]/
On Linux, the following bindings exist:
mx-gnu34 - mx with GNU 3.4
mx-gnu4 - mx with GNU 4
mx-intel - mx with Intel compilers
mx-pgi - mx with PGI compilers(default)
p4-gnu32 - p4 with GNU 3.4.3
p4-intel - p4 with Intel compilers
Setup
With so many device and compiler combinations on two operating systems, environment setup can seem complicated. HPCC has defined preferred bindings to simplify your environment initialization. The preferred bindings work well on almaak and the Linux cluster. However, you are free to choose the best setup for your needs.
For most ITS packages, it is generally recommended to use the default link, which points to a particular default version. For MPICH, this is not recommended. Changing the default link after installing a newer version of MPICH will always break currently running or waiting jobs. For this reason, it is recommended that you always use a path that uses a particular version. For compatibility and to inform users of ITS's intent, default, old, and new links are installed. See the mpiexec note below for a
work-around.
You should look at the links in /usr/usc/mpich and decide on a version to use, then look at the specific bindings available in that version and choose a binding. You should then use the most specific link in your .profile (Bourne), .bash_profile, or .login (csh).
To add the preferred device/compiler sets to your environment automatically (gm-pgi on Linux, and shmem-spac on Solaris), add the following to your .login:
if ( -e /usr/usc/mpich/1.2.6..14a/setup.csh) then
source /usr/usc/mpich/1.2.6..14a/setup.csh
endif
If you use the the Bourne shell, add the following lines to .bash_profile (or .profile in case you dont have a .bash_profile):
if [ -e /usr/usc/mpich/1.2.6..14a/setup.sh ]; then
source /usr/usc/mpich/1.2.6..14a/setup.sh
fi
Examples of other available setup files are:
/usr/usc/mpich/1.2.6..14a/gm-intel-P4/setup.csh
/usr/usc/mpich/1.2.6..14a/gm-cdk/setup.csh
/usr/usc/mpich/1.2.6..14a/p4-gnu/setup.csh
/usr/usc/mpich/1.2.6..14a/shmem-spac/setup.csh
etc.
Compiling
MPICH includes wrapper scripts designed to hide the details of compiling with any particular build or compiler. Simply source an mpich build into your shell environment and use mpicc, mpiCC, mpif77, or mpif90 to compile your code. The arguments to the script are passed to the compile and interpreted normally. The wrapper script will link in the correct libraries and add rpaths as necessary.
Running
MPICH includes a wrapper script called mpirun(1) to hide the details of getting your program to run. To use mpirun, it is critical that the exact same mpich build that was used to build the program is also used to run it (see the mpiexec note below to get around this restriction). After sourcing the correct mpich build, mpirun requires a few arguments to tell it which nodes to run on.
The two most common options are -machinefile and -np. Under PBS or TORQUE, simply pass $PBS_NODEFILE to -machinefile. The number of processors will need to be passed to -np. Under PBS or TORQUE, the easiest way to get the value for -np is:
cat $PBS_NODEFILE | wc -l
Mpiexec
Linux Cluster users are encouraged to use mpiexec(1) instead of mpirun. Using mpiexec is far simpler because it senses the correct transport layer and integrates the directory into TORQUE (the job manager on the cluster). By default, mpiexec will use $PBS_NODEFILE and run one process on every allocated processor (see the mpiexec(1) manpage for details).
With the 1.2.6..14a and later builds of mpich, you do not have to source mpich's setup files before running mpiexec. This greatly simplifies the job submission scripts and gets around the need to statically define an mpich build. Users can source any mpich build in their shell's initialization.
Math Libraries
Several math-related libraries are built with each MPICH build for easy linking. At this time, goto (an optimized BLAS), mpiblacs, and scalapack are available. Simply add -llibraryname to your compile commands to link them into your executable file. The various mpich builds know where to find the libraries.
If linking with goto, also link in xerblac if you are using C code or xerblaf when using Fortran code. Likewise, mpiblacs users should link in mpiblacsCinit or mpiblacsF77init.
Users of goto may have several versions to choose from. Options such as -lgoto, -lgoto_p4, -lgoto_p3, and -lgoto_p4_512 etc. may be available. Please look at /usr/usc/math/1.2.6..14a/<binding>/lib/ for specific options. See http://www.cs.utexas.edu/users/kgoto for additional information.
These libraries have not been throughly tested on every mpich build. Please notify hpcc@usc.edu if you encounter any problems with them.
Documentation
The official MPICH documentation is at http://www.mcs.anl.gov/mpi/mpich. The Myrinet documentation is at http://www.myri.com/scs/READMES/README-mpich-gm, and there is a Frequently Asked Questions page for Myrinet at http://www.myri.com/fom-serve/cache/13.html.
