A cluster here is a set of computers of a similar architecture which share common management and configuration and usually share the users' files transparently. The systems have good communications between themselves, and jobs can either be launched on one processor or multiple copies can be launched on several, allowing part of the calculation to be performed in parallel.

Computationally intensive tasks interfere with general purpose time-sharing activities and one function of offering processors on a cluster is to dedicate the processors fully to that task in hand.

To use the parallel computing possibilities it is necessary to invoke languages where this is a built-in feature, or to analyse the problem and use library functions to distribute data and program over selected processors. Typically groups who develop such programs optimise them for their own systems. These notes might help the installation of such programs for use at the ILL.

Machine list

There are four clusters at the ILL on which scientists may become registered users. Click on the cluster name to monitor its load (intranet only). Other ILL machines may be monitored using the NetInfo ILL/SI service

Intel Pentium Xeon Cluster brick.ill.fr
Contact: Jean-Francois Perrin, Mark Johnson, Ron Ghosh

• Nodes  8
  • Processors:  2x Xeon (smp)
  • Clock Speed:  2.0GHz
  • bogomips:  3948.54 (one node)
  • Memory : 1-2.0Gb
• Interconnect:   Fast ethernet switch
• System:  SuSE Linux 9.3 (i586)
• Parallel Software:  MPI-CH

COMPAQ-ALPHA Cluster dsc.ill.fr
Contact: Mark Johnson, Ron Ghosh

• Nodes            4
• Servers          2 Compaq-DS20e
• Interconnect  PCI-Memory channel
• each server:
  • Processors           2x Alpha (smp)
  • Clock Speed         660Mhz
  • Memory                2 Gb
• System                 Compaq TRU64 Unix
• Parallel Software   Compaq MPI (MPI-CH)
  

Comparing floating point intensive calculations the Alpha processor alone appears to be about 35% faster than the Xeon on brick. Using the DFT calculation VASP as a test task, where it is known that MPI-CH may not be the optimum solution, the Alpha (MPI-CH) performance advantage is less than this, compared with the Intel cluster using MPI-LAM. For optimum performance it is clear that modifying the default configuration of the inter-communications parameters for MPI can lead to rewarding improvements in performance.

AMD Cluster amd.ill.fr
Contact: Mark Johnson, Ron Ghosh

• Nodes           16
• Servers          8
• Interconnect  Fast ethernet switch
• each server:
  • Processors          2x AMD Opteron(tm)
  • Processor            248 (smp)
  • Clock Speed        2.2 GHz
  • bogomips            4325.37 (one node)
  • Memory               2 Gb
• System                SUSE LINUX 10.1 (X86-64)
• Parallel Software MPI-CH

Fujitsu Siemens Cluster fscs.ill.fr
Contact: Mark Johnson, Ron Ghosh

• Nodes            32
• Servers          8
• Interconnect  Fast ethernet switch
• each server:
  • Processors           2x Dual Core AMD Opteron(tm) Processor 280 (smp)
  • Clock Speed         2.4 GHz
  • bogomips             4796.57 (one node)
  • Memory                2 Gb
• System                 SUSE LINUX 10.1 (X86-64)
• Parallel Software  MPI-CH

Parallel Programs

Linux Cluster                     Alpha Cluster
High Performance Fortran Portland Group HPF
LAM-MPI                            lam-6.5.4
MPI-CH     mpich-1.2.2      Compaq-MPI
MPI-RPM                            Portland Group

High Performance Fortran, HPF

The aim is to produce a program with a structure of SPMD, single program, multiple data model. Each processor loads the same program but then operates on a distinct subset of the program's data, each using a local portion of distributed arrays. Special directives are added to this variant of the Fortran standard to control data and parallel program flow. The Portland Group HPF is available on brick.

Message Passing Interface, MPI

This comprises of utilities with a library of functions callable from C, Fortran etc., which allow tasks to be copied to other nodes, portions of data transferred, and then results reassembled after processing. This standard call interface allows the implementation of transfers and synchronisation to be left either to Open-source developers, or proprietary manufacturers (who often use concepts from the first group but optimise performance to their own hardware). Because the calls are standardised it is possible to build variants using any of the mpi implementations, and select the optimal performer for routine use. In each case there are a number of options on buffer sizes and transfer methods to explore.

MPI-CH
This is used where there are fast communications between cluster members, which must all share the same architecture.

LAM-MPI
This typically uses ethernet for communications, and even has the possibilities of reformatting data on transfer in heterogeneous clusters. Clearly the task has to be rebuilt for each type of machine.

Building Programs with MPI

In addition to a library to the organising routines there are several tools designed to simplify development.
mpirun      this launches an MPI task and distributes it amongst available nodes (DEC/Compaq dmpirun)
mpiclean    this resets the nodes should the mpirun fail and leave the nodes in an indeterminate state
mpif77       traditionally this expands to the commands necessary
mpicc        to build the task, identifying the libraries etc., yielding a standard method for building tasks, allowing      makefiles independent from the transport method.
Fuller description of these tools can be found in the man pages. Useful features include switches to launch dummy operations which show the nodes to be used and/or the files and libraries to be loaded without actually doing any processing. Variants of each of these utilities appropriate to the communication method are kept in:
    LAM-MPI  /usr/lib/lam-6.5.4/bin
    MPI-CH  /usr/lib/mpich-1.2.2/bin

Note
To show the path of the default program installed, use the command:

$ which mpif77

On brick this is /usr/lib/lam-6.5.4/bin/mpif77 showing that lam-mpi is the default MPI method. To change this to MPI-CH place /usr/lib/mpich-1.2.2/bin in your PATH environment variable ahead of the lam-mpi reference modifying your .cshrc file. The .cshrc file is re-used as each node starts your task anew.

In general MPI libraries and utilities are built using the GNU configure script. If the default installation does not match the imported source code it may be necessary to build a private library appropriate to that task, specifying installation in the builder's area using ./configure -prefix ... If the new mpif77 is used the correct libraries will be loaded. (It is probably simplest to modify the PATH for this.) Such programs should be linked as static entities (linker switch -Bstatic) to include copies of all libraries at link time, avoiding conflicts which might arise as default sharable libraries are sought on target nodes when run by mpirun or Alinka.

The Portland compiler package includes utilities for profiling programs to identify compute intensive code, and also debuggers to analyse problems in execution of parallel routines.

FAQ - Frequently Asked Questions

Please send questions and comments to R. Ghosh for inclusion here.