USC - High Performance Computing & Communications

Using Globus Toolkit

The Globus Toolkit is the de facto standard for deploying and using grid services. It includes tools for authentication, scheduling, file transfer and resource description.

Authentication for Globus services at USC is done with a Kerberos to X.509 certificate proxy. This allows users to authenticate with grid services without having to maintain an unique user certificate.

Details for setting up the proxy can be found on the Setting up credentials Web page.

Scheduling jobs in globus can be done using Condor via the Globus universe. See the Condor - Using Globus Universe under the Condor area.

Another method for submitting jobs is using specific Globus commands: globus-job-submit, globus-job-run, and globusrun.

Globus Commands

globus-job-submit

The globus-job-submit command runs as a background process. Once the job is submitted, the connection to the remote host goes away, and control is returned to the user. This command returns a contact string, which allows you to monitor your job. Using globus-job-submit, you can submit a large batch job, exit the system, and return later to query for the job status. The basic syntax is:

 globus-job-submit <gatekeeper-host>[/<jobmanager>]

You may specify either jobmanager-pbs or jobmanager-fork; however, the jobmanager argument is optional. If not specified, the job manager, jobmanager-fork, will be used.

A simple example

[jmehring@wave jmehring]$ globus-job-submit hpc-master.usc.edu/jobmanager-pbs \
                                            /bin/echo "Hello Globus."

On successful submission you will receive a link in the following format, which can be used to check your job status:

https://servername:port/15675/1121292464/
[jmehring@wave jmehring]$ globus-job-status https://servername:port/15675/1121292464/
DONE
[jmehring@wave jmehring]$ globus-job-get-output https://servername:port/15675/1121292464/
Hello Globus.

globus-job-run

The globus-job-run command is intended for simple interactive jobs. It includes functionality for staging executables and input/output files. The basic syntax is similar to globus-job-submit:

 globus-job-run <gatekeeper-host>[/<jobmanager>]

Unlike the globus-job-submit command that runs in the background, the connection to the remote host is maintained until the job completes. Therefore, no link is provided to query the status of the job or to fetch job output.

A simple example

In the following example, the connection to the server is kept open until the job finishes. Any output from the job will be displayed on the screen directly.

[jmehring@wave jmehring]$ globus-job-run hpc-master.usc.edu/jobmanager-pbs \
                                         /bin/echo "Hello Globus."
Hello Globus.

globusrun

In globusrun, you describe your job in a script written in the Resource Specification Language (RSL). The globusrun command gives you access to the full power of the RSL, which provides a common language for specifying resource and execution parameters necessary for running remote jobs. The basic syntax is:

 globusrun <params> <your_rsl_script>

You can view a sample RSL script by running globus-job-run with the - dumprsl flag.

almaak.usc.edu(4): source /usr/usc/globus/default/setup.csh
almaak.usc.edu(5): globus-job-run -dumprsl hpc.usc.edu /bin/echo "Hello, Globus world."
  &(executable="/bin/echo")
   (arguments= "Hello, Globus world.")

A simple example

Creating a new RSL specification file

[jmehring@wave jmehring]$ cat test.rsl
&(executable=/bin/echo)
(arguments= Hello, Globus world.)
(stdout = /home/rcf-12/jmehring/test.out)
(stderr = /home/rcf-12/jmehring/test.err)

Send that file to the server to be run as a job.

[jmehring@wave jmehring]$ globusrun -r hpc-master.usc.edu/jobmanager-pbs -f test.rsl
globus_gram_client_callback_allow successful
GRAM Job submission successful
GLOBUS_GRAM_PROTOCOL_JOB_STATE_ACTIVE
GLOBUS_GRAM_PROTOCOL_JOB_STATE_DONE

Output from the job will be saved on the filesystem on the remote server. You can use globus-url-copy to copy them back to your local machine.

For more information on RSL parameters please refer to RSL Parameters Web page.

globus-url-copy

Transferring files between grid-enabled sites can be done using gridftp. HPC runs a gridftp server on hpc-master.usc.edu.

globus-url-copy -vb -tcp-bs 8388608 \
                file://<full path to file> \
                gsiftp://<server>/<path to destination>

    Note: The back slash (\) lets you break a single command into multiple lines.

The -vb flag puts the transfer into verbose mode and will give an estimated completion time. The -p flags refers to how many parallel streams will be opened for the transfer. The -tcp-bs flag specifies the tcp buffer size in bytes. Depending on the network connection between the two sites, different combinations of -p and -tcp-bs can affect how long it takes to transfer your data between them. Remember that the file names have to include the absolute path on both sides of the transfer.

For more information, run globus-url-copy with the -help flag.

A simple example

Transferring a local file from wave.usc.edu [desktop with a globus toolkit installation] to hpc-master.usc.edu:

[jmehring@wave jmehring]$ globus-url-copy -vb -p 1 -tcp-bs 8388608 \
	file:///home/jmehring/test-globus \
	gsiftp://hpc-master.usc.edu/home/rcf-12/jmehring/test-globus

Source: file:///home/jmehring/
Dest:   gsiftp://hpc-master.usc.edu/home/rcf-12/jmehring/test-globus

Transferring a remote file on hpc-master.usc.edu to wave.usc.edu:

[jmehring@wave jmehring]$ globus-url-copy -vb -p 1 -tcp-bs 8388608  \
	gsiftp://hpc-master.usc.edu/home/rcf-12/jmehring/test-globus \
	file:///home/jmehring/test-globus

Source: gsiftp://hpc-master.usc.edu/home/rcf-12/jmehring/
Dest:   file:///home/jmehring/test-globus