Dear ADF-list members,

It has appeared to us that many ADF users are insufficiently aware of
the importance that the SCM_IOBUFFERSIZE environment variable has, in
certain circumstances, for ADF performance. This is partially due to
the fact that its description in the Installation manual has been
rather short. We therefore wish to provide more background here.

Very briefly:
=============
Problem: you notice for your ADF job that there is a gap between the
          Elapsed time and CPU time. These are printed by ADF near the
          end of your output file. This delay may well be (partially)
          due to the fact that the program is waiting for I/O.
Solution: increase the value for the SCM_IOBUFFERSIZE environment variable.
          I/O can be avoided completely in this manner if you have enough
          memory available.

In more detail:
===============

SCM has recently performed some scaling and performance tests on a popular
modern RISC SMP machine, which is typical for what is found at computing
centers. We have found that the ADF performance for large parallel jobs,
and for jobs running together with other jobs sharing the IO system,
can be significantly improved. This is achieved by adjusting the size of the
memory buffer ADF uses to keep files in memory. You can do this by setting
just one environment variable in your ADF job file. The speed-up is especially
significant for jobs using a significant number of CPUs (say 16 or more).

The performance improvement can be obtained because of the large difference
in CPU and I/O speed on modern RISC supercomputers. This difference is likely
going to increase further in the future, making this environment variable
even more important. And typically the IO systems are shared by all tasks of
all jobs, and thus also jobs from other users may have a significant effect
on the IO performance.

The environment variable is SCM_IOBUFFERSIZE. It determines how much memory
ADF uses for buffering I/O. The default value, specified in $ADFHOME/settings
is rather small: 8 MB. This choice has been made because we want ADF to run
also on old UNIX workstations with limited memory. For modern parallel
supercomputers, with much memory per CPU, this value is clearly too small.

For large jobs, this small value will lead to a nonnegligible amount of I/O.
This is usually not a problem on a PC cluster configuration, where the CPU
speed is typically a bit lower than on modern RISC machines, and where each
CPU has its own local hard disk to write to. In other words, in such a
configuration, we observe that CPU time and Elapsed time are typically close
to each other, and no dramatic improvements are to be expected from increasing
SCM_IOBUFFERSIZE. However, if sufficient memory is available, it is still
of some advantage to increase the value, even on PC clusters.

Timing example
==============

The effects of choosing the correct value for SCM_IOBUFFERSIZE can be
illustrated with the following timing for one geometry step of a 100-atom
molecule in a DZP basis set, on 32 CPUs

Worst case: heavy I/O on system due to some other jobs, default value of
8 MB for SCM_IOBUFFERSIZE

CPU time : 1996.0 sec. Elapsed time : 3599.3

Best case: large value for SCM_IOBUFFERSIZE (256 MB, all files kept in memory)

CPU time : 1738.9 sec. Elapsed time : 1775.5

On typical modern RISC processors, 1 GB memory per CPU is typical. In such a
case, a default value of 256 for SCM_IOBUFFERSIZE seems reasonable. However,
you do have to take into account that this reduces the amount of available
memory in the system, which may have adverse effects for your jobs in case
these exhaust the available memory on the system. In such cases it will be
better to reduce the value of SCM_IOBUFFERSIZE.

How to tell if you have the right value for SCM_IOBUFFERSIZE?
=============================================================

If Elapsed time and User time are close together in your job, you do not
have to increase the value for SCM_IOBUFFERSIZE.

If they differ substantially and you have enough memory available, choose
a larger value. Some experimenting may be useful for determining a good
default value. A reasonable default seems to be 256 MB for a 1 GB/CPU
machine.

how to interpret Buffered I/O output
====================================

The output, as printed by ADF, that one would hope for is as follows. This
example is for the job above run on 64 CPUs with 400 MB as SCM_IOBUFFERSIZE
value.

 Buffered I/O statistics
 =======================
 Memory available: 419430400
 Number of records fitting in memory: 101409
 Input : 2.6% of 1033 *4k bytes
 Output: 14.7% of 801 *4k bytes
 Records from serial files evicted: 0
                    others evicted: 0
 Hash table lookups: 5090 with 0 conflicts ( 0.00%)

The value of 400 MB returns in the output on the first line
"Memory available:" Because each record uses 4 kB, this immediately
determines the number of records in the second line.

The % numbers in the 3rd and 4th lines indicate how many read and
write requests resulted in a read from or write to file. Even if
everything easily fit in memory, as in this case, these percentages
will not be zero. The reason is that ADF has to read fragment files
before it can start a calculation and it has to write at least a
TAPE21 file.

The number of evicted records in lines 5 and 6 are probably the most
important numbers. If these are zero, this means that all files fit
in memory. In that case, no further improvements are to be expected
from further increasing SCM_IOBUFFERSIZE.

The last line is not so important to the typical user as it does
not tell much about ADF performance. It is technical information
mainly of interest to SCM to determine if the caching mechanism
works as expected.

If the value for SCM_IOBUFFERSIZE is low and the job is large,
the statistics look as follows, for comparison:
In this case the percentages "Input" and "Output" are large and
many records are evicted.

 Buffered I/O statistics
 =======================
 Memory available: 16777216
 Number of records fitting in memory: 4056
 Input : 56.3% of 307069 *4k bytes
 Output: 77.0% of 118013 *4k bytes
 Records from serial files evicted: 94538
                    others evicted: 132762
 Hash table lookups: 1217472 with 414681 conflicts ( 34.06%)

We would like to emphasize that this does not have to be a problem
in case the I/O bandwidth is sufficient to keep up with the CPUs.

Where to specify SCM_IOBUFFERSIZE?
==================================

You can set a good default value in your login script, together with
environment variables like ADFHOME. You can overwrite such a default
inside your job script, by specifying SCM_IOBUFFERSIZE there

----
#! /bin/sh
export SCM_IOBUFFERSIZE=128
$ADFBIN/adf <<eor
----
Further Information
===================
The SCM_IOBUFFERSIZE variable is also mentioned very briefly in section
3.5 of the Installation manual. 
Should you have further questions on this topic, or comments based on
your own timings, do not hesitate to mail to support_at_scm.com. If you
think you have comments that are useful for many users, you can of course
send them to the ADF-LIST as well.  
Best regards,
Stan van Gisbergen,
on behalf of the SCM team.
------------------------------------------------------------------------
Dr. S.J.A. van Gisbergen       SCIENTIFIC COMPUTING & MODELLING NV
Phone:  +31-20-4447626         Vrije Universiteit, Theoretical Chemistry
Fax:    +31-20-4447629         De Boelelaan 1083
E-mail: vangisbergen_at_scm.com   1081 HV Amsterdam, The Netherlands
                               http://www.scm.com
------------------------------------------------------------------------