RATS Under UNIX

RATS software under UNIX is installed in the Research Computing Facility (RCF) and Student Computing Facility (SCF) clusters of ITS machines. Users must have an account on a machine in the RCF or SCF in order to use RATS or any of the other statistics packages. Accounts are available from the ITS Computer Accounts Office in LVL Lower Commons (213/740-5555).

To invoke RATS, type rats at the UNIX prompt.

In order for the rats command to function, your environment needs to be set up to find the command and any other files or libraries the software needs. In the RCF, this is done automatically when you login, allowing you to access all the statistics software by simply typing the invocation command.

RCF users whose .login or .cshrc files are non-standard or who once had setup files for one or more specific statistics packages sourced in .login or .cshrc may need to fix their .login or .cshrc files to eliminate redundancy and/or to allow the system's automatic setup to take effect. For example, users on the RCF may have modified their .login or .cshrc to such an extent that the system's automatic setup is bypassed. These users can either restore the system's automatic setups (recommended) or add these lines

if  (-e  /usr/usc/stats/setup.csh)  then
   source  /usr/usc/stats/setup.csh
endif

to their .login file. Please call consulting (see last page) for help with these files.

Documentation and Help

At the end of this document, there is a listing of RATS documentation that is available from Estima, Inc.

RATS uses commands which it refers to as 'instructions'. RATS programs may typed in upper or lower case, except for filenames. Since UNIX is case-sensitive, filenames must be typed exactly as they appear in the UNIX directory listing.

Many commands have what RATS calls "Options", and some also have "supplementary cards". Options appear immediately after a instruction, always in parentheses. The parentheses in which options are placed must begin immediately after the instruction name to which they apply. There must not be a space between the instruction and the open parenthesis containing options. Options are separated by commas. For example,

DATA(FORMAT=PRN,ORG=OBS)

Supplementary cards always begin with '#', and provide additional information to the instruction, usually lists of series or equations.

Instructions may be of any length, and do not need to begin in column 1. If an instruction is continued to a second (or further) line, place a '$' at the end of the unfinished line, and make sure it is separated from the last character in the line by at least one blank, as in:

SET BRG = INCOME - OUTGO + RATE1 + LIEN  +  $
    TOTAVG + HEIGHT

Comment lines may be added to RATS program by beginning the line with an asterisk (*). Comment blocks may be added by beginning with '/*' and ending with '*/'. For example,

* this is a comment line

/* this is a comment block, in which the next two
   instructions will be ignored
OPEN  DATA  MY.DATA
DATA(FORMAT=PRN) / VAR1 VAR2
*/

To exit RATS when in interactive mode, type end at the '?' prompt. In batch mode (i.e., when running RATS commands from a file), it is not necessary to type end at the end of the commands.

---

The RATS interactive session is begun by entering the command

rats

which gives you some startup messages followed by the prompt:

?

RATS commands are entered at this prompt, as shown in the RATS manual, and results are shown on the screen as commands are entered.

In interactive RATS programs or sessions, it is possible to use information saved in UNIX files, or to direct information to UNIX files. This is done with the OPEN instruction. Standard I/O units which can be OPENed in RATS are:

   INPUT -- source file (or device) for RATS instructions
   OUTPUT -- file (or device) for output from RATS
   DATA -- source file for data read into RATS
   COPY -- output file primarily for data being written from RATS

During a RATS program (either batch or interactive) data are read into RATS from external UNIX files. Data can be in raw (ascii) form, RATS format, RATS transport format, or certain spreadsheet formats. (See Reference 1. for details.) Data in raw form may be delimited by spaces, commas, tabs or carriage returns.

If you want RATS to read raw data from an external file, specify the name of the file on an OPEN instruction, then follow that with the DATA instruction with the appropriate parameters and options. For example,

OPEN  DATA  FOODDATA.DAT
DATA(FORMAT=FREE,ORG=OBS)  /  FOODCONS  PRRETAIL  $
   DISPINC  FOODPROD  PRFARM

Once the output from your program is stored in a file, you can print that file in any way you print other text files. You can download it to a Macintosh or PC and print it there, or you can submit it to a printer known to UNIX and print it here on campus. If the printer is controlled by ITS in a Public User Room there will be a per-page charge for picking up the printout.

To see a list of printers known to UNIX, type printers at your UNIX prompt. The printer "code" will be shown in the leftmost column.

To print a file on one of the UNIX printers, use the lpr command with the -P switch, as in the following example, which prints "my.file" to the printer in UCC 101 (the UCC Distribution Area):

lpr -Pps_ucc101 my.file

NOTE: ps_ucc101 is used only as an example; this probably is not the most convenient printer for most users.

Some printers are configured for PostScript only, and will not accept text-file input from the lpr command. In this case the enscript command (which converts text to PostScript) should be used instead, as shown here:

enscript -Pps_ucc101 my.file

To print output that is more than 80 characters wide, many printers will require that you rotate the orientation of the page to "landscape" (the default usually is "portrait") and often you will also need to reduce the font size to get all 132 columns on the page without wrapping. This can be accomplished with the enscript command followed by the -r (rotate) option and the -f (font specification) option. One font that has been tested for wide output is Courier9, though others may work as well. The following will print wide output (132 characters per line) without any wrapping or truncation:

enscript -Pps_ucc101 -r -fCourier9 file.name

You can also download your output (or any file) to a PC or Mac, and print on a local printer, if you prefer. The consultants have documentation on how to download files.

Since UNIX users have quotas, it is possible to run out of disk space when processing large data sets. To avoid running out of disk space, users may request quota increases when needs arise. If you believe your circumstances require more disk space, contact the authorized signer in your department for information and permission. If you don't know who your authorized signer is, the ITS Computer Accounts office in CAL 323-4 (213.740.5555) can help you.

/tmp Space

Quite often, large data sets are needed only for short periods of time while analyses are done or subsets are extracted. For these situations, the best solution often is to make use of the temporary space in RCF and SCF machines that is set aside for all users. SCF users will have quotas on /tmp space, similar to their own disk file quotas, so this may not solve SCF users' problems completely. RCF users also have /tmp space quotas, and can request increases from ITS as necessary. The space is in a subdirectory called /tmp (and, on some systems, /tmp1, and so forth; enter ls -ld /tm* to see them all). For descriptions and policies related to shared /tmp space, scroll to Section 7 in the following page:
http://www.usc.edu/its/policies/procedures/

Anyone can write in the shared /tmp spaces, create subdirectories in them, and in general manipulate files as you would in your home directory. You should be aware that files stored in /tmp are not permanent, and will be deleted by the system on the regular /tmp cleanup schedule. This schedule is different for different machines and /tmp areas, but the bottom line is that your files will go away.

Compressing Large Files to Save Space

Users with large amounts of data can make use of compression techniques when data are stored for periods of more than a few days without being used. UNIX allows users to compress their files, then uncompress them when they need to be used, allowing a substantial savings of space both in the user's own subdirectory structure and in the common areas (such as /tmp) if they are used. Users are urged to compress any files larger than 20 Mb if they are not being used for long periods of time. The amount of time it takes to compress or uncompress files is minuscule, and much space will be saved by this operation.

To compress a file in UNIX, use 'compress'; to uncompress a compressed file, use 'uncompress' or 'zcat'. In the following examples we compress a file called test1.bigfile, which results in a compressed file called test1.bigfile.Z:

compress test1.bigfile

Either of these commands results in an uncompressed file:

uncompress test1.bigfile.Z
zcat test1.bigfile.Z > test1.bigfile

The 'zcat' command, shown above, normally is used for viewing files, but with the use of the redirect, can re-create files from their compressed versions. While the 'compress' and 'uncompress' commands use the previous names (with the addition or deletion of the .Z extension), 'zcat' can redirect the compressed file into an uncompressed file of any name.

When large numbers of users are running jobs on RCF or SCF machines, it may be advisable not to try large jobs that are I/O or CPU intensive, since the portions of time allocated to each user are, obviously, much smaller than at times of low usage. The longer a job takes, the lower priority it often receives. This can result in jobs that may take hours of real time in the middle of the day, when they might only take minutes at a less busy time.

There are several useful techniques for dealing with large jobs that take a lot of time. You can run your jobs in batch mode, rather than interactive. This avoids using any memory that would be necessary for interactive mode. You can run your batch job in the background (by placing an ampersand [&] at the end of the command-line invocation), which allows you to logout while the job finishes, if you wish. (Running a job in the background also allows you to avoid timeout problems associated with long periods of terminal inactivity when you are connected via a micom line or modem.) And you can queue your job to run at a later time, when the system is likely not to be so busy.

Batch RATS

RATS can be run in batch mode, by entering the RATS program into a file (for example, "myratstest.prog"), and then typing the rats command with the filename specified as input:

rats < myratstest.prog

The output will be reported to your screen (but will go by very fast if it generates more than one screen of output). If you want to store the output in a file, specify an output file, as in this example;

rats < myratstest.prog > myratstest.output

If you prefer, you may use the RATS program itself to open an output file. This is done with the OPEN command followed by specification of the OUTPUT unit. See below, and consult the RATS manual (listed at the end of this document) for details.

Once your output is in a file, you can view it with more or any other appropriate command, edit it with emacs or another editor, and print it with lpr, enscript, or the print procedure of your choice.

Delayed Batch Job Processing -- the  at  and  qsub  Commands

Any user can queue a job to run at a later time. Users running particularly large, compute-intensive, and/or long-running jobs are required to queue their jobs using the Portable Batch System (PBS) that makes use of rcf-01. For more information, see below, as well as our online document "Portable Batch System (PBS)":
www.usc.edu/hpcc/systems/use-l-4.php

To queue your job to run at a later time, you can use the  at  UNIX command. This requires that you place your invocation command and all other specifications (such as the filename of the program and any options or switches) into an executable shellscript file, and then specify the executable in an  at  command, such as

at 2:30a < execthis

where  execthis  is an executable shellscript you have created. (Of course, the name  execthis  is arbitrary, and can be any filename you choose.)

Let's say you have a statistics package called THYSTAT (for those who take documentation literally, note that this is a mythical package name, for illustration only), and your THYSTAT program is contained in  myprog.thystat1  which you keep in your home directory (~/) in a subdirectory called  programs. Part of the peculiar syntax of THYSTAT is that before the batch file name, you use a switch spelled  -in. For whatever reason, you wish to run your program late at night. To make an executable shellscript to run your batch program, enter the following into a file called  execthis :

#!/bin/sh -f
thystat -in ~/programs/myprog.thystat1

Then mark this file as an executable shellscript by typing the following at your UNIX prompt:
chmod u+x execthis

You can then run the  at  command, as shown above, and specify any time you feel would be best to run your job. For more details on the  at  and  chmod  commands, see the man pages or call the consultants.

The  qsub  Command and the PBS System

Multiple batch programs can be entered into a queue and run at a later time, allowing the user to 'submit' any number of jobs and still meet the system's limitation of one compute-bound job running at any given time. This method is required for jobs that are compute-intensive and/or will need multiple processors or extensive system resources. This type of queueing is done with the  qsub  command, which sends a request to the Portable Batch System (PBS).

RCF users normally can use the  qsub  command automatically. If it is not immediately available, run

source /usr/usc/pbs/default/setup.csh

Once this is set up, you can type  man pbs  and  man qsub  for complete online help information. Another helpful man page is  man pbs_resources_linux  which details the resource options (e.g., walltime, cput [cpu time], etc.) available for the  qsub  command.

The  qsub  command itself runs a script file, and inside the script file is the batch invocation of the software you wish to run (along with any other appropriate UNIX commands.  qsub  also writes to a LOG file that you specify, so you can see the STDIN and STDOUT messages associated with the running of your queued job. The general form of the  qsub  command is

qsub <flags> SCRIPTFILE

Once again, as in the example above, let's say you have a statistics package called THYSTAT (this is just a mythical example, to be substituted with whatever real command name and syntax you want to queue) and your THYSTAT program is contained in  myprog.thystat1  which you keep in your home directory (~/) in a subdirectory called  programs. Part of the peculiar syntax of THYSTAT, let's say, is that before the batch file name, you use a switch spelled  -in. Normally, you would run it in batch as follows:

thystat -in ~/programs/myprog.thystat1

In order to queue it with  qsub  and have the system run it later, you would create a qsub-appropriate SCRIPTFILE containing at least one line with a pound symbol ( # ) followed by the invocation of your software program, as shown above. Let's say you decide to put this into a file called  my.qsub.script. The contents of that file would be

#!/bin/csh -- a line beginning with # must begin the script file 
thystat  -in  ~/programs/myprog.thystat1

(Once again, remember that thystat is a bogus name, for example purposes only, and needs to be substituted with a real command name and syntax of your choice.)

The following command would queue your THYSTAT program for execution at a later time chosen by the system.

qsub -m be -k eo my.qsub.script

where  -m be  asks NQS to broadcast a message (i.e., to tell you) when the request ends execution;  -k eo  tells qsub to keep the STDERR and STDOUT output. A number of other flags are available, as listed in  man qsub  (remember to  source  the pbs setup file as shown above).

Many flags can be incorporated into the script file itself. For example, if you want your program always to report the results to your home directory, the -k eo flag may be entered into the file. Also, directions as to how much time to allow (walltime), how many nodes to use (nodes), and how many processors to use (ppn). If you want your job to run on the main rcf-01 processors, you can specify that in a separate nodes instruction (as shown in the example). Following is a sample (and simple!) script file that includes these requests. (The ls command is used as an example of a simple program that returns a detailed listing of the files in your home directory.)

#!/bin/csh -- any text here

#PBS -k eo
#PBS -l walltime=2:00:00
#PBS -l nodes=rcf-01
#PBS -l nodes=1:ppn=1

# run the program
ls -la ~/
exit

IMPORTANT NOTES:

• The syntax for the walltime value is hh:mm:ss (hours, minutes, seconds)
• The value for nodes= must always be 1 when submitting jobs to rcf-01, as the facility is configured as a single node.
• If explicit paths are used for commands or file specifications in the script file, /export and /auto must not be used. Instead, use the form /home/rcf-nn/username, where nn is the number of the file system partition where the files are stored, and username is the username of the home directory where the files are stored.

Often the  qsub  command will execute within a few minutes, though in some cases it could be delayed until the system is relatively unbusy at night.

The RATS people have provided a number of sample programs which illustrate various operations of RATS. These are found in

/usr/usc/rats/default/examples

Some of the sample programs use other files in the examples directory, so you may need to browse the example file you wish to run. If you find any references to external files, copy the example file and all the other relevant files to a directory of your own, and run the sample from there.

RATS User's Manual, Version 4, by Thomas A. Doan

Estima
1800 Sherman Avenue, Suite 301
Evanston IL 60201

847/864-1910 (tech support)
847/864-6221 (fax)

<estima@estima.com> === http://www.estima.com/

This Reference can be found in the manual racks in the Public User Area in the basement of Waite Phillips Hall.

This Reference is not normally available at the USC Bookstore. RATS documentation must be obtained directly from the vendor.

USC Students, Faculty and Staff may use computers and documentation in the Public User Areas in Waite Phillips Hall, King Olympic Hall and Salvatori Computer Science Center, as well as the Leavey Library Information Commons (Ground Floor).

Consulting. The ITS Consultants may be familiar with the language and general operations of this software, but it may be necessary to make an appointment with a member of the full time staff in order to receive detailed help. Support of statistics software is the responsibility of the ITS Statistics Consultant with the participation of other full-time ITS staff. These people may be contacted through the ITS Customer Support Center as shown here.

Customer Support. USC students, staff or faculty who would like information about ITS Workshops or about obtaining site-licensed software or other computing-related questions should visit the Customer Support Center in Leavey Library Lower Commons, or call 213.740.5555, or send eMail to <consult@usc.edu>

Documentation. This document, and many others on a variety of topics, are available in the ITS Customer Support Help System, available on the World Wide Web at:

http://www.usc.edu/its/

You can find Statistics Software Help Documents through the search engine at this same URL, or go directly to them at:

http://www.usc.edu/its/doc/statistics/help/

Newsgroups. Another source of information, this one providing the opportunity to exchange thoughts with other users, is the newsgroup 'usc.comp.all.stat.users'. For more information about reading news, subscribing to newsgroups, and related topics, visit

  
http://www.usc.edu/its/doc/internet/news/