Appendix C. Debugging MPI Problems
**********************************

The Amsterdam Modelling Suite (AMS) is designed to work out of the box
on as many platforms as possible, but it could be that you are
experiencing problems with running our programs in parallel. This
mostly happens on linux cluster environments, but the information
below could also be useful for Windows or MacOS users. However,
anything that mentions cluster environment or batch system is linux
specific!


Technical introduction into AMS
===============================

Most of our compute programs use Message Passing Interface (MPI)
communication for parallel operation. To make AMS as easy to use as
possible we ship the MPI runtime libraries within the AMS package, and
take care of launching of the MPI programs in parallel in our start
script. Because of this setup it is not needed to "mpirun" our
programs yourself, so **never do something like "mpirun $AMSBIN/ams"**
!


Execution order
---------------

When starting one of our programs, for example *$AMSBIN/ams*, you
actually call our start script via a symbolic link. This start script
then:

* checks which program it should launch, based on the name it was
  called from or the value of the -x flag

* detects how many cores should be used: -n or the NSCM environment
  variable are read, when both are empty we detect the cpu core count
  (see below for cluster environments!)

* sources the *$AMSBIN/setenv.sh* script to set up environment
  variables

* Write the given input to a temporary file

* detect if and which batch system it is running on

* do the correct parallel MPI launch of the program

"Old pre-AMS2020 info:" When running *$AMSBIN/adf*, this sequence is
preceded by a program that first sets up and runs the "atomic create
runs" in serial.


The $AMSBIN/start script
------------------------

As described above, the start script takes care of selecting the
number of cores to use for the calculation, sourcing the setenv.sh
script, and launching the selected program in parallel. You should not
need to edit it if your AMS program fails to start, or does not run on
the correct number of cores. In such cases, set the **SCM_MPIRUN_EXE**
and **SCM_MPIRUN_OPTIONS** variables in your environment to override
the autodetect mechanism in the start script!

The start script tries to detect if it is running on a cluster
environment with a batch system such as PBS/Torque, SLURM, SGE or LSB.
If it detects one of the environment variables associated with these
batch systems, it will leave figuring out the MPI job configuration
(cores and nodes) to the MPI library. The MPI library should get the
number of cores to run on, and which compute nodes to use (in case of
a multi-node calculation) from the batch system. the **NSCM**
environment variable and the **-n** flag are ignored in this case!

If no batch system is detected, and both **NSCM** and **-n** are not
used to explicitly set the number of cores to use for running the AMS
program, then the start script attempts to detect the number of cores
on the machine, and uses all available cores. We only count real CPU
cores (no hyperthreading or modules with shared FPUs such as AMD
Bulldozer), because the AMS programs do not benefit from
hyperthreading.

Finally the start script launches the AMS program using the correct
mpi commands for the situation.


The $AMSBIN/setenv.sh script
----------------------------

The *$AMSBIN/setenv.sh* script takes care of setting up environment
variables needed for running the AMS programs. This includes among
others:

* the LD_LIBRARY_PATH variable with locations to the required
  libraries (and DYLD_LIBRARY_PATH)

* the PYTHONPATH variable

The setenv script also responds to a couple of environment variables,
such as **SCM_USE_LOCAL_IMPI**, **SCM_USE_IMPI_2021**, and
**SCM_USE_LOCAL_OMPI**. See Appendix A on Environment Variables for
details.


MPI runtimes
------------

The Amsterdam Modelling Suite ships with the required MPI runtime
inside the package. For MacOS this is OpenMPI, for Windows this is
MSMPI, and for Linux both IntelMPI and OpenMPI flavors exist. When
working on Linux we advise to use the IntelMPI library, as this
provides the best out-of-the-box experience. The OpenMPI runtime has
no batch system integration build into it, so it will most likely not
work under a batch system. The runtimes can be found in
*$AMSBIN/IntelMPI* or *$AMSBIN/openmpi*, and are loaded into the
environment by the setenv.sh script. See the Additional information
section for more details.


Debugging MPI issues
====================

For non-batch system (desktop/laptop usage) related MPI issues, please
contact support@scm.com as these should almost never happen. However,
linux users might want to try using a newer IntelMPI runtime by
setting **export SCM_USE_IMPI_2021=1** in their environment.

For cluster usage we always advise to use the IntelMPI version. AMS
ships with the IntelMPI runtime inside the package, you do not need to
install this separately. IntelMPI can be used on Cray systems as well
via the ABI compatibility, see the Additional information section for
more details. If you for whatever reason need to use OpenMPI instead,
make sure to compile and use your own OpenMPI version that matches the
version used to build AMS (currently 4.1.1), and set
SCM_USE_LOCAL_OMPI=true in the environment. Make sure to to read the
Additional information and known issues document before you dive into
debugging MPI! If you know how to launch MPI programs on your cluster
and the autodetect mechanism is not giving the correct results, you
can set the **SCM_MPIRUN_EXE** and **SCM_MPIRUN_OPTIONS** variables in
your environment directly (make sure to "export" them). The start
script will then use those variables to launch the mpi job. See
Appendix A on Environment Variables for details.

The generic scheme for debugging IntelMPI / batch system issues
consists of the following steps:

1. Start by setting up the IntelMPI runtime libaries in the
   environment: load it from a module, or load your AMS environment
   and execute ". $AMSBIN/setenv.sh" without the quotes. Keep in mind
   that in order to test the IntelMPI 2021 runtime shipped with
   AMS2021, you will need to execute "export SCM_USE_IMPI_2021=1"
   before sourcing the setenv.sh script.

2. get the intel mpirun to work. Environment variables that are useful
   for this are I_MPI_DEBUG=100 (or higher), I_MPI_OFI_PROVIDER_DUMP=1
   (for information about the interconnects detected by IntelMPI), the
   I_MPI_FABRICS (for selecting a specific fabric, options depend on
   the IntelMPI runtime version) and I_MPI_HYDRA_TOPOLIB. See the
   IntelMPI 2018u3 developer reference manual for details on these
   variables for the default IntelMPI runtime. If you are using the
   IntelMPI 2021 runtime, see IntelMPI current developer reference
   manual

3. get a small mpi test program that's compiled with IntelMPI and a
   Fortran compiler. To make things a bit easier you can download an
   example with both binary and source code here.

4. try to mpirun the test program (mpirun ./mpitest) on a single node
   via the batch system. This usually respects the batch system
   reservation. In case it doesn't you will need to dive into the
   manuals to see what kind of environment variables you need to set,
   or arguments you need to give to get it to integrate with the batch
   system. IntelMPI developer references and guides help out with
   this, but make sure to get the manual matching your IntelMPI
   version! The manual of your batch system might also hold some
   clues.

5. once single node jobs work, then it is time to see what needs to be
   done for multi-node jobs. Usually this can be set up via
   integration with the batch system, but if that fails you could
   always write a bit of code that generates a "machine file", and
   tell IntelMPI to use this file with the -machinefile flag (see this
   for how a machine file should look)

It might be that the default IntelMPI runtime shipped with AMS
(currently 2018 update 3) needs to be updated for a better integration
with your batch system. In such a case you can try the IntelMPI 2021
runtime (also shipped with AMS) by setting "export
SCM_USE_IMPI_2021=1" in your environment. Or you can download and
install a newer IntelMPI runtime package (those are free to use), and
repeat the previous steps. If the downloaded IntelMPI works better
than the 2018 or 2021 runtime shipped with AMS, then you can set the
"export SCM_USE_LOCAL_IMPI=true" environment variable. This tells AMS
not to load the distributed IntelMPI into the environment, but instead
use the one already available. (You do not need to recompile AMS for
this!)

Finally: when requesting support for MPI issues via support@scm.com,
make sure to include as much information as possible. For example:
which batch system are you using, what version is it, do you have a
special interconnect such as Infiniband, and of course always send us
all the input and output files produced by the failed job, including
the sdtout and stderr of the batch system. From our experience the
fastest way to resolve such issues is if somebody from SCM can work
directly on the machine. Therefore you might want to consider setting
up a form of temporary remote access for an SCM employee to help you
out.


IntelMPI on Linux Quirks
========================

AMS with IntelMPI on Linux uses the 2018u3 runtime by default, because
in our experience this is still the fastest and most reliable IntelMPI
runtime. However, for newer CPUs (both AMD and Intel Xeon), the 2018
runtime sometimes has startup problems. In such cases we suggest
trying the IntelMPI 2021.1 runtime, which we also distribute with AMS,
by setting "export SCM_USE_IMPI_2021=1" in your environment. If this
does not work for your system, you can download and install a newer
(or older) IntelMPI runtime yourself, and use this by setting "export
SCM_USE_LOCAL_IMPI=1" in your environment.

The IntelMPI 2021.2 and 2021.4 runtimes have been found to deadlock
for some operations on AMD Zen3 processors. They do seem to work fine
on Intel CPUs. The IntelMPI 2021.6 runtime seems to work fine for both
Snellius (the new Dutch supercomputer at SARA), and works without
problems on our Zen3 systems, as well as Intel-based CPUs.
