TSP Under UNIX

revision date: 11 June 2002

TSP (Time Series Processor) is a statistics package that offers procedures for analyzing time series and other econometric data using a number of methods, including ordinary or nonlinear least squares, nonlinear multivariate least squares, and nonlinear two- and three-stage least squares regression. TSP under UNIX is installed on specific hosts in the Research Computing Facility (RCF) and Student Computing Facility (SCF) clusters of ITS machines. Users must have an account on a machine in one of these clusters in order to use TSP or any of the other statistics packages. Such accounts are available from the ITS Computer Accounts Office in LVL Lower Commons, 213/740-5555.

You may wish to work from a "personal subdirectory", which means you would routinely "cd" to a subdirectory other than your home directory. TSP, however, can be invoked from any subdirectory. The examples below will assume that files are stored and/or written to the current subdirectory (i.e., the directory from which you invoked TSP). Files that are not stored in the current directory may be used or created, but must be specified with the complete home directory pathname or the tilde-slash home directory convention. If you have questions about file specification in UNIX, please contact the UNIX consultants at the number given at the end of this document.

You can receive additional online help by invoking TSP and typing "help" at the TSP prompt.

To invoke TSP under UNIX, type tsp at the UNIX prompt. (Unlike previous versions of TSP which required different commands such as tsp30, tsp150 and so forth, to increase available memory, the current version allows you to increase the default memory with an option command within the program. See "TSP jobs that need more memory", below.) In order for this command to function, your environment needs to be set up to find the command and any other files or libraries the software needs. In the RCF, this is done automatically when you login, allowing you to access all the statistics software by simply typing the invocation command.

In the SCF, users of this (or any) statistics software need to add the following lines to their .login file:

if  (-e  /usr/usc/stats/setup.csh)  then
    source  /usr/usc/stats/setup.csh
endif

Users whose .login or .cshrc files are non-standard or who once had setup files for one or more specific statistics packages sourced in .login or .cshrc may need to fix their .login or .cshrc files to eliminate redundancy and/or to allow the system's automatic setup to take effect. For example, users on the RCF may have modified their .login or .cshrc to such an extent that the system's automatic setup is bypassed. These users can either restore the system's automatic setups (recommended) or add the lines shown above (which are usually only necessary for SCF users). Please contact consulting (see last page) for help with these files.

The current version of TSP allows the use of a login.tsp file, which can contain startup settings for your TSP session or batch job. If a file called 'login.tsp' exists in the directory where tsp is invoked, TSP will read it and execute it automatically. One example of a command appropriate for this file is

options memory=6;

which increases the default of 4MB of main memory available to 6 MB. (See "TSP jobs that need more memory" below for details on this option.)

TSP provides regression, forecasting, and other econometric tools for use on series data, such as those found in a time series. It offers the standard econometric techniques such as ordinary or two-stage least squares, limited information maximum likelihood (LIML), autoregressive correction and weighted least squares. It also offers advanced techniques including nonlinear least squares, Box-Jenkins estimation, multivariate regression, three- stage least squares, solution of linear and nonlinear models, and others.

TSP runs in batch and interactive modes in most terminal emulations. In OpenWindows (X-Windows), it can be run in an xterm or shelltool window with the terminal type set to vt100. The output files are character data, so that they can be sent to virtually any printer.

In terms of command and subcommand syntax, TSP Under UNIX is similar to versions running on other platforms (e.g., PC-DOS). Most of the differences, of course, relate to differences in the way external files are specified (if and when they are used), a convention which is operating-system-specific.

TSP in batch or interactive mode can read free-format or FORTRAN-formatted data. Data can be read from external files, or entered in-stream. Files containing TSP programs to be run in batch must end with .tsp or .TSP as in foo.tsp or BAR.TSP .

The output from TSP jobs contains the original input statements, results of the calculations and plots requested, and any error messages or other notes that might be generated by your program. Output from batch jobs is saved into a file that is named according to the name of the file containing the program. For example, if you enter

tsp foo.tsp

you get an output file called foo.out. If you enter

tsp BAR.TSP

you get an output file called BAR.OUT .

The elements of TSP programs are fully documented in the References listed at the end of this document. These references also contain a fair amount of statistical background text relating to time series analysis and related issues.

To invoke interactive TSP under UNIX, the following command is used:

tsp

This displays the TSP introductory information, and gives you the first prompt

1 ?

where you can begin typing your TSP commands. You can enter more than one command on a line (as shown below in the first line of the sample program), or you can enter one command per line (as shown in the second line of the example). Here is a sample program that you can try:

1 ? FREQ A; SMPL 78,83;
3 ? LOAD SALES,GNP;
3 ? 11.7 1846
3 ? 12.6 1785
3 ? 13.5 1862
3 ? 14.2 1770
3 ? 13.6 1478
3 ? 12.8 1762
4 ? LSALES=LOG(SALES); LGNP=LOG(GNP);
6 ? MSD LSALES,LGNP;
7 ? OLSQ LSALES C,LGNP;

Notice that since the first line has two TSP commands, the next line is labeled "3 ?". Also, since line 3 says you're going to LOAD some data, the "3 ?" prompt repeats until the six observations (for the six years 78 through 83 given in the SMPL command) are finished. The data end with a semicolon (see the last line marked "3 ?" above), which can be on the same line as the data from the last observation, or on its own line below the last observation. When line 6 is entered, output appears on the screen immediately. The same happens when line 7 is entered.

To direct your output to a file (instead of the terminal), specify a file name as follows

34 ? output 'nowgo.here' ;

at the point where you want to begin sending output to a file. To return to viewing output at the terminal, type

47 ? terminal;

and subsequent commands will show output on your screen.

At the end of each interactive TSP session, a file called BKUP.TSP is saved, which contains the commands you submitted during your session. This file is overwritten (if it exists) when a new session is started.

To run TSP in batch mode, enter the commands (and data, if they are to be entered in-stream) into a file with the extension .tsp or .TSP , then specify that file name on the command line, as follows:

tsp program1.tsp

TSP will run the commands in the file program1.tsp, and will produce a file for your output called program1.out. To test this, enter the commands and data shown in the interactive example above (be sure to leave out the prompts, such as "1 ?", "3 ?", and so forth) into a file called program1.tsp and then run it with the command shown immediately above.

TSP has provided a series of examples that illustrate many of the operations of TSP. These are found in

/usr/usc/tsp/default/examples

Each file in this subdirectory is a comlete TSP example which you can use as an input file. Some of the examples use external data files, which are found in the same directory. Since TSP writes its output file in the same directory where the input file exists, it is necessary for you to copy the input file to a directory for which you have write access. Say you wanted to run 'median.tsp' in your directory called '~/tspwork'. First,

cp      /usr/usc/tsp/default/examples/median.tsp
~/tspwork/median.tsp

Then go to '~/tspwork' and type

tsp median.tsp

and you can examine and study the output which will be found in 'median.out'.

To view TSP output, use the UNIX command "more" against the file you specified for the output, or the file that TSP creates for you from batch jobs. To edit out portions of the output you don't need or don't want to print, use any UNIX editor (e.g., emacs, vi).

Once the output from your program is stored in a file, you can print that file in any way you print other text files. You can download it to a Macintosh or PC and print it there, or you can submit it to a printer known to UNIX and print it here on campus. If the printer is controlled by ITS in a Public User Room there will be a per-page charge for picking up the printout.

To see a list of printers known to UNIX, type printers at your UNIX prompt. The printer "code" will be shown in the leftmost column.

To print a file on one of the UNIX printers, use the lpr command with the -P switch, as in the following example, which prints "my.file" to the printer in UCC 101 (the UCC Distribution Area):

lpr -Pps_ucc101 my.file

NOTE: ps_ucc101 is used only as an example; this probably is not the most convenient printer for most users.

Some printers are configured for PostScript only, and will not accept text-file input from the lpr command. In this case the enscript command (which converts text to PostScript) should be used instead, as shown here:

enscript -Pps_ucc101 my.file

To print output that is more than 80 characters wide, many printers will require that you rotate the orientation of the page to "landscape" (the default usually is "portrait") and often you will also need to reduce the font size to get all 132 columns on the page without wrapping. This can be accomplished with the enscript command followed by the -r (rotate) option and the -f (font specification) option. One font that has been tested for wide output is Courier9, though others may work as well. The following will print wide output (132 characters per line) without any wrapping or truncation:

enscript -Pps_ucc101 -r -fCourier9 file.name

You can also download your output (or any file) to a PC or Mac, and print on a local printer, if you prefer. The consultants have documentation on how to download files.

TSP by default uses 4 MB of memory, has 500,000 words working space, and can handle up to 165,000 data points in a regression or observations in a series. Previous versions of TSP required the use of different invocation commands to give TSP more than the default main memory (e.g., tsp15, tsp30, tsp120 and so forth). In the current version there is only one command: tsp. Additional memory is requested through the use of an options command which can be specified in a login.tsp file or on the first line of a batch program. (Additional memory cannot be requested after an interactive session has been invoked.)

To run TSP in more than the 4 Mb default main memory, add this line

options memory=m;

where 'm' is the number of megabytes you wish. Try not to overestimate, as you may be interfering with the work of others using the same computer. For example, to invoke TSP with 10 Mb of main memory, create (or edit) a file called 'login.tsp' and put the following line at the top of that file:

options memory=10;

The 'login.tsp' file must be stored in the directory where you invoke tsp. If you run tsp in batch mode, you may add the options line at the top of the batch file. You may not use the options command (or any other technique) to increase memory once you have started an interactive TSP session.

When large numbers of users are running jobs on RCF or SCF machines, it may be advisable not to try large jobs that are I/O or CPU intensive, since the portions of time allocated to each user are, obviously, much smaller than at times of low usage. The longer a job takes, the lower priority it often receives. This can result in jobs that may take hours of real time in the middle of the day, when they might only take minutes at a less busy time.

There are several useful techniques for dealing with large jobs that take a lot of time. You can run your jobs in batch mode, rather than interactive. This avoids using any memory that would be necessary for interactive mode. You can run your batch job in the background (by placing an ampersand [&] at the end of the command-line invocation), which allows you to logout while the job finishes, if you wish. (Running a job in the background also allows you to avoid timeout problems associated with long periods of terminal inactivity when you are connected via a micom line or modem.) And you can queue your job to run at a later time, when the system is likely not to be so busy.

Batch TSP

As explained above, TSP jobs can be run in batch by issuing the invocation command (tsp) followed by the name of the file containing the TSP program. To run a batch program in the background, simply put an ampersand (&) after the file specification:

tsp myinputfile.tsp &

When you have done this, you can perform other tasks in the foreground, and if you wish, you can logout and your job will continue to completion.

Delayed Batch Job Processing -- the  at  and  qsub  Commands

Any user can queue a job to run at a later time. Users running particularly large, compute-intensive, and/or long-running jobs are required to queue their jobs using the Portable Batch System (PBS) that makes use of rcf-01. For more information, see below, as well as our online document "Portable Batch System (PBS)":
www.usc.edu/hpcc/systems/use-l-4.php

To queue your job to run at a later time, you can use the  at  UNIX command. This requires that you place your invocation command and all other specifications (such as the filename of the program and any options or switches) into an executable shellscript file, and then specify the executable in an  at  command, such as

at 2:30a < execthis

where  execthis  is an executable shellscript you have created. (Of course, the name  execthis  is arbitrary, and can be any filename you choose.)

Let's say you have a statistics package called THYSTAT (for those who take documentation literally, note that this is a mythical package name, for illustration only), and your THYSTAT program is contained in  myprog.thystat1  which you keep in your home directory (~/) in a subdirectory called  programs. Part of the peculiar syntax of THYSTAT is that before the batch file name, you use a switch spelled  -in. For whatever reason, you wish to run your program late at night. To make an executable shellscript to run your batch program, enter the following into a file called  execthis :

#!/bin/sh -f
thystat -in ~/programs/myprog.thystat1

Then mark this file as an executable shellscript by typing the following at your UNIX prompt:
chmod u+x execthis

You can then run the  at  command, as shown above, and specify any time you feel would be best to run your job. For more details on the  at  and  chmod  commands, see the man pages or call the consultants.

The  qsub  Command and the PBS System

Multiple batch programs can be entered into a queue and run at a later time, allowing the user to 'submit' any number of jobs and still meet the system's limitation of one compute-bound job running at any given time. This method is required for jobs that are compute-intensive and/or will need multiple processors or extensive system resources. This type of queueing is done with the  qsub  command, which sends a request to the Portable Batch System (PBS).

RCF users normally can use the  qsub  command automatically. If it is not immediately available, run

source /usr/usc/pbs/default/setup.csh

Once this is set up, you can type  man pbs  and  man qsub  for complete online help information. Another helpful man page is  man pbs_resources_linux  which details the resource options (e.g., walltime, cput [cpu time], etc.) available for the  qsub  command.

The  qsub  command itself runs a script file, and inside the script file is the batch invocation of the software you wish to run (along with any other appropriate UNIX commands.  qsub  also writes to a LOG file that you specify, so you can see the STDIN and STDOUT messages associated with the running of your queued job. The general form of the  qsub  command is

qsub <flags> SCRIPTFILE

Once again, as in the example above, let's say you have a statistics package called THYSTAT (this is just a mythical example, to be substituted with whatever real command name and syntax you want to queue) and your THYSTAT program is contained in  myprog.thystat1  which you keep in your home directory (~/) in a subdirectory called  programs. Part of the peculiar syntax of THYSTAT, let's say, is that before the batch file name, you use a switch spelled  -in. Normally, you would run it in batch as follows:

thystat -in ~/programs/myprog.thystat1

In order to queue it with  qsub  and have the system run it later, you would create a qsub-appropriate SCRIPTFILE containing at least one line with a pound symbol ( # ) followed by the invocation of your software program, as shown above. Let's say you decide to put this into a file called  my.qsub.script. The contents of that file would be

#!/bin/csh -- a line beginning with # must begin the script file 
thystat  -in  ~/programs/myprog.thystat1

(Once again, remember that thystat is a bogus name, for example purposes only, and needs to be substituted with a real command name and syntax of your choice.)

The following command would queue your THYSTAT program for execution at a later time chosen by the system.

qsub -m be -k eo my.qsub.script

where  -m be  asks NQS to broadcast a message (i.e., to tell you) when the request ends execution;  -k eo  tells qsub to keep the STDERR and STDOUT output. A number of other flags are available, as listed in  man qsub  (remember to  source  the pbs setup file as shown above).

Many flags can be incorporated into the script file itself. For example, if you want your program always to report the results to your home directory, the -k eo flag may be entered into the file. Also, directions as to how much time to allow (walltime), how many nodes to use (nodes), and how many processors to use (ppn). If you want your job to run on the main rcf-01 processors, you can specify that in a separate nodes instruction (as shown in the example). Following is a sample (and simple!) script file that includes these requests. (The ls command is used as an example of a simple program that returns a detailed listing of the files in your home directory.)

#!/bin/csh -- any text here

#PBS -k eo
#PBS -l walltime=2:00:00
#PBS -l nodes=rcf-01
#PBS -l nodes=1:ppn=1

# run the program
ls -la ~/
exit

IMPORTANT NOTES:

• The syntax for the walltime value is hh:mm:ss (hours, minutes, seconds)
• The value for nodes= must always be 1 when submitting jobs to rcf-01, as the facility is configured as a single node.
• If explicit paths are used for commands or file specifications in the script file, /export and /auto must not be used. Instead, use the form /home/rcf-nn/username, where nn is the number of the file system partition where the files are stored, and username is the username of the home directory where the files are stored.

Often the  qsub  command will execute within a few minutes, though in some cases it could be delayed until the system is relatively unbusy at night.

Since UNIX users have quotas, it is possible to run out of disk space when processing large data sets. To avoid running out of disk space, users may request quota increases when needs arise. If you believe your circumstances require more disk space, contact the authorized signer in your department for information and permission. If you don't know who your authorized signer is, the ITS Computer Accounts office in CAL 323-4 (213.740.5555) can help you.

/tmp Space

Quite often, large data sets are needed only for short periods of time while analyses are done or subsets are extracted. For these situations, the best solution often is to make use of the temporary space in RCF and SCF machines that is set aside for all users. SCF users will have quotas on /tmp space, similar to their own disk file quotas, so this may not solve SCF users' problems completely. RCF users also have /tmp space quotas, and can request increases from ITS as necessary. The space is in a subdirectory called /tmp (and, on some systems, /tmp1, and so forth; enter ls -ld /tm* to see them all). For descriptions and policies related to shared /tmp space, scroll to Section 7 in the following page:
http://www.usc.edu/its/policies/procedures/

Anyone can write in the shared /tmp spaces, create subdirectories in them, and in general manipulate files as you would in your home directory. You should be aware that files stored in /tmp are not permanent, and will be deleted by the system on the regular /tmp cleanup schedule. This schedule is different for different machines and /tmp areas, but the bottom line is that your files will go away.

Compressing Large Files to Save Space

Users with large amounts of data can make use of compression techniques when data are stored for periods of more than a few days without being used. UNIX allows users to compress their files, then uncompress them when they need to be used, allowing a substantial savings of space both in the user's own subdirectory structure and in the common areas (such as /tmp) if they are used. Users are urged to compress any files larger than 20 Mb if they are not being used for long periods of time. The amount of time it takes to compress or uncompress files is minuscule, and much space will be saved by this operation.

To compress a file in UNIX, use 'compress'; to uncompress a compressed file, use 'uncompress' or 'zcat'. In the following examples we compress a file called test1.bigfile, which results in a compressed file called test1.bigfile.Z:

compress test1.bigfile

Either of these commands results in an uncompressed file:

uncompress test1.bigfile.Z
zcat test1.bigfile.Z > test1.bigfile

The 'zcat' command, shown above, normally is used for viewing files, but with the use of the redirect, can re-create files from their compressed versions. While the 'compress' and 'uncompress' commands use the previous names (with the addition or deletion of the .Z extension), 'zcat' can redirect the compressed file into an uncompressed file of any name.

During a TSP program (either batch or interactive) you may specify an external file containing raw data, instead of typing it "instream" with the other TSP commands. The external file must be encoded in ascii and must not have variable names or labels, only data values (either numeric or alphanumeric).

To have TSP read data from an external file, specify the name of that file in the LOAD command, as in the following example:

LOAD (FILE='my.data') VAR1, VAR2 ;

Note that UNIX is case-sensitive, so that MY.DATA would not work to call a file that UNIX knows as my.data. Also, TSP allows the use of the tilde-slash (~/) convention to represent a home directory when you wish to name files that are not in the current directory.

To write out data in raw form from your TSP program, use the WRITE command and specify a file name, as in the following example:

WRITE (FILE='rawoutput.data') VAR1, VAR2 ;

This writes the data in the variables VAR1 and VAR2 into an ascii file called rawoutput.data .

It is possible, though infrequent, that you will leave a TSP process running after you leave your TSP session or even after you logout of UNIX. This can happen especially when you get forced out of TSP abruptly, with the system returning inexplicably to the UNIX prompt. If you have any questions as to whether or not you have old processes running, you can easily find out with the following command:

ps -ugx

(The ps command on various systems may need options different from -ugx to get the desired results; check with the consultants if you have questions.) This shows a list of your processes, like the one below (some columns have been left out to save space):

USER       PID %CPU STAT  START    TIME   COMMAND
janedoe   6393  7.9  R    18:05    0:00   ps -ugx
janedoe   6256  0.0  IW   Sep  2   0:02   -csh (csh)
janedoe   4659  0.4  S    Jan 3  210:00   /usr/usc/tsp/tsp

If you see something under COMMAND that has TSP in it, this means a process involving TSP has been running for the amount of time under TIME. If you don't want any TSP processes to be running, you should kill the process by issuing the following command:

kill -9 4659

which "kills" process number 4659. Please check your processes daily to guard against overuse of the machine by unintended processes. This is a courtesy to other users, besides being in your best interests.

To end an interactive TSP session, type "stop" at the TSP prompt. for example,

9? stop <return>

1. TSP User's Guide, Version 4.5 (April 1999)
2. TSP Reference Manual, Version 4.5 (April 1999)

TSP International
P.O. Box 61015 Station A
Palo Alto CA 94306

650/326-1927 (voice)
650/493-2912 (fax)

<info@tspintl.com> === http://www.tspintl.com/

Reference 1. is a general guide to TSP operations, and Reference 2. is a command-lookup reference.

These documents are available in the manual racks in the Public User Area in the basement of Waite Phillips Hall and on the Reference shelves in the Lower Level Information Commons in Leavey Library.

TSP manuals are not usually available at the USC bookstore. References must be obtained directly from the vendor.

USC Students, Faculty and Staff may use computers and documentation in the Public User Areas in Waite Phillips Hall, King Olympic Hall and Salvatori Computer Science Center, as well as the Leavey Library Information Commons (Ground Floor).

Consulting. The ITS Consultants may be familiar with the language and general operations of this software, but it may be necessary to make an appointment with a member of the full time staff in order to receive detailed help. Support of statistics software is the responsibility of the ITS Statistics Consultant with the participation of other full-time ITS staff. These people may be contacted through the ITS Customer Support Center as shown here.

Customer Support. USC students, staff or faculty who would like information about ITS Workshops or about obtaining site-licensed software or other computing-related questions should visit the Customer Support Center in Leavey Library Lower Commons, or call 213.740.5555, or send eMail to <consult@usc.edu>

Documentation. This document, and many others on a variety of topics, are available in the ITS Customer Support Help System, available on the World Wide Web at:

http://www.usc.edu/its/

You can find Statistics Software Help Documents through the search engine at this same URL, or go directly to them at:

http://www.usc.edu/its/doc/statistics/help/

Newsgroups. Another source of information, this one providing the opportunity to exchange thoughts with other users, is the newsgroup 'usc.comp.all.stat.users'. For more information about reading news, subscribing to newsgroups, and related topics, visit

  
http://www.usc.edu/its/doc/internet/news/