Home › Corvus user guide

Corvus user guide

Contents:

 

Hardware

Corvus is an SGI Altix XE 1300 cluster with 70 nodes. A Voltaire Infiniband DDR 4x (20Gbps) switch connects all of the nodes as well as the network attached storage (NAS) server. 68 of the nodes are provided by 34 SGI Altix XE310 chassis which comprise of two nodes with a shared power supply. Each of these nodes has two 2.66 GHz Intel Clovertown quad core processors, 8GB RAM and a 250GB SATA drive. The remaining two nodes are SGI Altix XE240, each with two 2.66 GHz Intel Clovertown quad core processors, 16GB RAM and 146GB SAS drive. One of these is the cluster head node.

Corvus has access to a pool of storage which is shared amongst several of eResearch SA's facilities. Some of this is provided from a dedicated storage node. Another node is used as the front end (or host) for the cluster, for compiling code, testing programs and the submission of jobs.

In addition, each of the compute nodes on Corvus has 200GB of local disk space for storing data and other files whilst a job is running. These files will be removed once a job has terminated.

Back to top

Software

System software

Compilers and parallel programming libraries

Libraries

Application software

Back to top

Access and accounts

Time on the machine is available to researchers at any of the South Australian universities through eResearch SA. Researchers at these universities who wish to use any of eResearch SA's facilities should complete the membership form.

Anyone else who is interested in using eResearch SA's facilities should consult the Conditions of Use page to determine how best to gain access to the machine.

Back to top

Getting started

To use the cluster, you need to login to Hydra first (refer to the Hydra User Guide for more details on how to do this) and then connect to Corvus by typing the Unix command ssh corvus. This will connect you to the front end (commonly called the head node) and should be used for compiling, debugging and testing your program code, and for submitting jobs to the cluster. You should rarely need to log on to a node of the cluster, only to the front end.

Please read all of this User Guide before you try to run any jobs on the cluster, particularly the sections on Compiling programs and Running jobs.

Login files (.cshrc) and environment variables

Every time you login to our systems, a default .cshrc system file is run. This file establishes some of your basic environment, setting your prompt and ensuring your $PATH variable gives access to basic system commands. In addition you can have a file in your home directory, .cshrc.corvus, that is invoked by this global .cshrc login file and can be easily configured to extend your environment when you are using Corvus (there are equivalent files for Hydra and Aquila).

Note: Do not attempt to change the default .cshrc file in your home directory. Any changes you make will be overwritten by the system the next time you login. Any changes you wish to make to your operating environment should be made in the .cshrc.corvus file in your home directory.

Below is an example of a typical .cshrc.corvus file:

module load intel
module load gaussian

This file makes the following additions to your environment:

  • Sets the default compiler suite to be used to the Intel compiler suite
  • Allows access to the Gaussian application package

Note: You should avoid putting commands in your .cshrc.corvus file that write output to the screen, since this may affect some non-interactive jobs.

If you are unsure as to how to make changes to your default environment, please contact the eResearch SA Service Desk.

Modules

eResearch SA has embarked on using "modules" as the primary way to configure the user environment to provide access to software packages rather than the environment variable method used previously on Hydra and Aquila. This provides much easier access to the packages on the system. Researchers who have used APAC's HPC systems will have already had some exposure to this more dynamic mechanism for gaining access to software.

To see what modules are available to be loaded (which applications are available on the cluster), type

module avail

at the command prompt.

You can also see which modules you currently have loaded by typing

module list

Similarly, you can unload modules using, for example, module unload gaussian to unload the Gaussian module, removing all references to the Gaussian executable and associated runtime libraries

If you do not see a module listed for the application that you wish to run please contact the eResearch SA Service Desk.

Back to top

Porting programs to the cluster

Sequential programs

Sequential programs should run without change on a single processor of the cluster. You can therefore use the cluster without knowing how to write parallel programs, simply by submitting (multiple) sequential jobs.

Parallel programming

Alternatively, you can port or develop your programs using a standard parallel programming language. Programs written using Message Passing Interface (MPI), or OpenMP (shared memory directives) can be compiled and run on the cluster. OpenMP programs can only be run on one node (up to 8 processors) since they use shared memory. MPI jobs can be run on any number of processors up to the maximum available in the cluster at the time.

MPI

You can use MPI to parallelize programs written in Fortran, C or C++. This is more difficult to program than HPF or OpenMP, but typically gives better performance. For more information on MPI, you can look at this list of materials for learning MPI. There is a good online MPI Programming Course from Edinburgh Parallel Computing Centre. A standard reference book is Using MPI: Portable Parallel Programming with the Message-Passing Interface, by William Gropp, Ewing Lusk and Anthony Skjellum, MIT Press, 1994. More information is available in the Documentation section of this User Guide.

Parallel scientific software libraries

For some programs, the majority of the time is taken up in standard routines such as matrix solve, FFT, or computing eigenvalues. In that case, it is possible to use libraries containing parallel versions of these routines, which should speed up your program without requiring you to write any parallel code.

Corvus has the Intel Cluster Math Kernel Library (which includes an optimised LAPACK library) installed, as well as FFTW. Other open source libraries may be installed on request.

Standard software packages

Many standard software packages have parallel versions of the software available. The Software section of this User Guide lists some parallel programs that have been installed. Please contact the eResearch SA Service Desk if you would like other packages installed.

Help with parallel program development

eResearch SA periodically runs training courses on parallel programming. For help with porting programs and optimizing performance on the cluster, contact the eResearch SA Service Desk.

Back to top

Compiling programs

The following compilers are available on Corvus. They are easily accessible once you have loaded the correct module (refer to earlier section for description of modules).

GNU compilers

  • gcc (also aliased to cc) for C and C++ programs.
  • gfortran for Fortran 95 programs (and a certain compatibility level for Fortran 77 and Fortran 90).

Intel compilers

  • icc and icpc for C and C++ programs respectively.
  • ifort for Fortran programs.

Note: You may find that some programs will only compile, or will run faster, using certain compilers, so you may want to try them all.

Check the man pages and the Documentation section of this User Guide for details on usage and options for each compiler.

MPI programs

MPI programs should be compiled using mpicc (for C programs), mpiCC (C++), mpif77 (Fortran 77) or mpif90 (Fortran 90). To enable these commands you need to load the OpenMPI module (module load openmpi). This will use the underlying Compiler Suite that has been loaded earlier (gnu or intel) to select the appropriate OpenMPI package, ie. you need to select your compiler suite before you load the OpenMPI module.

Use the which command to check you are getting the right version of the MPI compilers. For example, when using the Intel compiler Suite:

which mpicc

should return something like:

/opt/shared/openmpi/1.2.5/intel/bin/mpicc

If you have Fortran code that makes use of temporary arrays you may find that it exceeds the stack space available and will cause your job to fail. If this happens, try using the -heap-arrays option during the compile (Intel compiler specific).

OpenMP programs

OpenMP directives for shared memory parallel programming are supported. These programs will only be able to run on a single node of Corvus (i.e. up to 8 processors).

General tips and information

All of the compilers will produce much faster code if you use compiler optimisation flags. Check the Documentation or man pages of the compiler you are using to find the appropriate optimisation flags (normally -O1, -O2, etc.).

Note: You should NOT use these optimisation flags when developing and debugging programs. They should only be used once you have checked that the program works, and you want to make it run faster. This is because it may take substantially longer to compile the program at a higher optimisation level.

Also, there is a greater chance of finding compiler problems or bugs at higher optimisation levels. The compiler may not be able to compile the program, or the output of the program may be incorrect. It is a good idea to check that the results of your programs compiled with a high optimisation level are the same as those with the default optimisation. If you detect an error when using a high optimisation level, try compiling that routine or program again at a lower optimisation level.

Note: Programs should only be compiled on the front end (the head node) of the cluster.

Back to top

Running jobs

Jobs are run on Corvus by submitting a jobscript to the queuing system. Corvus uses the ANU implementation of the Torque queueing system.

Jobs are submitted to the queue by issuing the command:

qsub myscript

where myscript contains relevant Torque commands and shell script commands.

Below are some generic examples of scripts with brief descriptions of each of the various Torque components. These may be adapted to suit your needs. Please note that you only need change those bits shown in red [this markup is still to be done] in order to get a functioning jobscript for Torque:

Sample Torque jobscript for a sequential job

#!/bin/csh

### Job name
#PBS -N MyJobName

### Join queuing system output and error files into a single output file
#PBS -j oe

### Send email to user when job ends or aborts
#PBS -m ae

### email address for user
#PBS -M Your-email-Address

### Queue name that job is submitted to
#PBS -q corvus

### Request nodes, memory, walltime. NB THESE ARE REQUIRED
#PBS -l ncpus=1
#PBS -l mem=Xgb,vmem=Xgb
#PBS -l walltime=HH:MM:SS
### If local scratch space required, request the amount you require.
### The location of the space will be $PBS_JOBFS when the script runs.
#PBS -l jobfs=Ygb

# This job's working directory
echo Working directory is $PBS_O_WORKDIR
cd $PBS_O_WORKDIR
echo Running on host `hostname`
echo Time is `date`

# Load module(s) if required
module load application
# Run the executable
ApplicationExe+Arguments

Note:

  • All lines beginning with #PBS are interpreted as Torque commands directly to the queuing system.
  • Output and error messages will be joined into a file that will be called something like MyJobName.oXXXXX in the directory from which the job is submitted (XXXXX will be the number component of the job id which is allocated when you submit the job with qsub).
  • MyJobName should be a concise but identifiable alphanumeric name for the job (starting with a letter, NOT a number).
  • mem=Xgb,vmem=Xgb states that the program will use at most X GB of memory and virtual memory during its runtime.
  • module load application is required if you don't automatically load this module in this shell's environment.
  • ApplicationExe+Arguments is the name of the program you want to run and all of the command line arguments you need. It may also include redirection of input and output streams.

A copy of this sample script can be downloaded here.

Sample Torque jobscript for an OpenMPI job

#!/bin/csh

### Job name
#PBS -N MyMPIJobName

### Output files
#PBS -j oe

### Mail to user when job ends or aborts
#PBS -m ae

### Mail address for user
#PBS -M Your-email-Address

### Queue name
#PBS -q corvus

### Request nodes, memory, walltime. NB THESE ARE REQUIRED
#PBS -l ncpus=N
#PBS -l mem=Xgb,vmem=Xgb
#PBS -l walltime=HH:MM:SS
### If local scratch space required, request the amount you require.
### The location of the space will be $PBS_JOBFS when the script runs.
#PBS -l jobfs=Ygb

# This job's working directory
echo Working directory is $PBS_O_WORKDIR
cd $PBS_O_WORKDIR
echo Running on host `hostname`
echo Time is `date`
echo Using nodes
cat $PBS_NODEFILE

# Load module(s) if required
module load Compiler
module load openmpi
module load application
# Run the executable
mpirun ApplicationExe+Arguments

Note:

  • All lines beginning with #PBS are interpreted as Torque commands directly to the queuing system.
  • Output and error messages will be joined into a file that will be called something like MyMPIJobName.oXXXXX in the directory from which the job is submitted (XXXXX will be the number component of the job id which is allocated when you submit the job with qsub).
  • MyMPIJobName should be a concise but identifiable alphanumeric name for the job (starting with a letter, NOT a number).
  • ncpus=N requests the number of cpus. If greater than 8 then it must be a multiple of 8.
  • mem=Xgb,vmem=Xgb states that the program will use at most X GB of memory and virtual memory across all cpus during its runtime.
  • module load Compiler is required so that your job will use the correct openmpi. Set Compiler to the name of the compiler suite used to build the application (probably intel or gnu).
  • module load application is required if you don't automatically load this module in this shell's environment.
  • ApplicationExe+Arguments is the name of the program you want to run and all of the command line arguments you need. It may also include redirection of input and output streams.

A copy of this sample script can be downloaded here.

Checking a job's status in the queue

Once a job has been submitted to Torque using qsub, it will print out a Job ID of the form XXXXX.corvus.eresearchsa.edu.au where XXXXX is a decimal number. This number is helpful to make checks on the job's status using the qstat command. Here is some sample output:

qstat -a 109596
corvus.sapac.edu.au:
Req'd Req'd Elap
Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
109596.corvus.s a_user corvus myjobname -- 96 96 48gb 12:00 R 00:05

To see all running jobs (including other users'), run the command qstat -r (the S column is the state, R means running and S indicates the job has been suspended to allow another job to run). A qstat -a will show these and also the jobs that are queued (not running, signified by state Q).

Deleting a queued job

To delete a queued or running job type

qdel job.id

where the job.id is the numerical portion of the output of qstat

Note: You will only be able to delete your own jobs.

For further information on the Torque commands it is highly recommended that you read the manual pages for qsub, qstat, qdel.

Back to top

Data storage and back-up

Temporary storage during computation

If your job requires local temporary space during execution, it is recommended that you use the jobfs feature of the ANU Torque. This allows you to nominate a maximum amount you will need and the scheduler will make sure that your job runs on a node(s) with sufficient scratch space. Please see the previous section on running jobs for an example of how to request it. All but one node have a maximum of 200GB available for jobfs.

If your job only needs a small amount of space then you may get away with simply using /tmp.

Long term storage

You may be asked to clean up your data area periodically, especially if you start to store large amounts of data there. This is to ensure everybody gets a fair allocation of a limited resource.

In addition, some research groups having need for greater capacity have funded their own dedicated storage which is managed by eResearch SA on their behalf.

Individual researchers, or research groups can arrange similar access to further storage capacity as their needs increase.

Back to top

Documentation

Compilers

Message passing interface (MPI)

Fortran and High Performance Fortran (HPF)

Back to top

Contacts and help

For more information on eResearch SA's facilities, systems support, assistance with parallel programming and performance optimisation and to report any problems, contact the eResearch SA Service Desk.

When reporting problems, please give as much information as you can to help us in diagnosis, for example:

  • When the problem occurred
  • What commands or programs you were trying to execute at the time
  • A copy of any error messages
  • A pointer to the program you were trying to run or compile
  • What compiler or Makefile you were using

Back to top