SCM: Installation Manual

4.5 Known HP-MPI/Linux issue: libibverbs error message

HP-MPI 2.2.7 stops with the following (or similar) error message

libibverbs: Fatal: couldn't open sysfs class 'infiniband_verbs'.

This occurs, for example, on GigE-connected ROCKS clusters with OpenMPI installed. The error is caused by the fact that there are Infiniband libraries installed without corresponding kernel drivers and/or hardware. In this case, one has to enforce the use of TCP by HP-MPI, which can be done using one of the methods below:

edit the $ADFBIN/start script and add a -tcp option to all mpirun commands:
```
$ADFBIN/hpmpi/bin/mpirun.mpich -TCP ...
```
or set the MPIRUN_OPTIONS environment variable in your shell resource file (for example, ~/.bash_profile):
```
export MPIRUN_OPTIONS=-TCP
```
or set the MPI_IC_ORDER environment variable in your shell resource file (for example, ~/.bash_profile):
```
export MPI_IC_ORDER="TCP"
```

Any of these options will make sure all other interconnects but TCP are ignored.