2.8 Set up scratch space
Most programs from the ADF package use disk for temporary data. This data often takes a significant amount of space and is used frequently. To avoid run-time errors and performance loss you may want to make sure that the file system used for temporary files is both big and fast. Use the SCM_TMPDIR environment variable to tell the programs where to create their scratch directories. Please note that SCM_TMPDIR should always be set. If it is not set then each process will create its own directory in the current working directory where is was started.
You can set the SCM_TMPDIR environment variable, for example, as follows:
SCM_TMPDIR=/local/scratch/$USER export SCM_TMPDIR
In the next few paragraphs we will explain exactly what this variable does.
Child processes
Every parallel job consists of one master and one or more slave tasks. Master and slaves behave a bit differently with respect to their scratch directories. Slave processes will always create a directory for their scratch files in $SCM_TMPDIR. For efficiency reasons, that directory should reside on a local disk unless you are using very, very fast shared file system for large files. You need write access to that directory, and the file system should have enough free space. Please note that the SCM_TMPDIR environment variable will be passed from the master to slaves. Besides, slave tasks also chdir to the directory to avoid any risk that shared files are updated by more that one process at the same time.
After the job is finished, slave processes will delete their scratch directories. This can disabled by setting the SCM_DEBUG environment variable to any text, for example, to "yes". In this case the scratch directory and all its contects will be left intact.
Each slave writes its text output to a file called KidOutput located in its scratch directory. In case of an error this file will likely contain some sensible error message. To avoid losing the file in such a case, ADF will copy it to the directory where the job was started as KidOutput__#, where # is the process' rank.
Master process or serial runs
The master process (which is the only process in a serial run) will also create its temporary files in its own sub-directory of $SCM_TMPDIR. There are some exceptions: Some files, such as logfile and TAPE13, will be created in the directory where ADF was started because they are not performance-critical but are convenient to have in the current directory for different reasons. For example, logfile is nice to have in the current directory in order to follow the calculation progress and the TAPE13 is an emergency restart file that can be used if ADF crashes or is killed.
At the end of a calculation, the master will copy all result files from its scratch directory to the directory where it was started.
