Package Manager: Installing Optional Components¶
This guide describes how to install optional components of the Amsterdam Modeling Suite. To reduce the initial download size, the Amsterdam Modeling Suite does not include all components by default. This includes features such as
If the appropriate license is present these modules can be installed though the GUI, or the command-line. The easiest way requires an active internet connection, but it is also possible to to use a local copy of our repository downloaded ahead of time. These instructions assume that you have already successfully installed AMS.
New in version AMS2024.101: Python packages may be pre-installed by admins, including the machine learning potentials.
Introducing AMSpackages¶
Starting with AMS2021.1, several optional components will be provided through a unified program called AMSpackages. AMSpackages helps you maintain the installation of optional components of the Amsterdam Modeling Suite. Features installed through AMSpackages integrate with both the GUI as well as command line programs, and can be managed through either as well. Users will be able to download and install packages, without the need for a password or writing permissions to the main installation directory of AMS. All that is required is a valid license.
Users of the graphical interface will be prompted with a suggestion to install them when components are selected in AMSinput. Alternatively, the package manager can be opened using SCM -> Packages to install components ahead of time, and to update or remove installed components. You can find more usage instructions at the AMSpackages GUI section of the documentation.
Managing packages from the command line¶
On the command line, users can use "${AMSBIN}/amspackages"
to access the package manager interface.
There are several subcommands that can be used to access several features such as installing, updating, or removing packages.
Below the most commonly used options are explained.
To see a full list of options you can use "${AMSBIN}/amspackages" --help
.
Any of the subcommands may also have specific options, which can be seen when using --help
behind the subcommand.
For instance, for "${AMSBIN}/amspackages" install --help
.
Finding available packages¶
New in version AMS2024.101.
The
display
command was renamed tolist
.yaml
was added as an available format.
The list
command will show available packages.
"${AMSBIN}/amspackages" list | less -R
If you wish to refer to a package, you can use the ID
to install, remove it, or check if it is installed.
By default, list provides a long list of packages, but a wide view is also available.
For example:
> "${AMSBIN}/amspackages" list --format wide
AMSpackages Overview
Date: Tue May 11 12:07:01 2021
Repository: https://downloads.scm.com/Downloads/packages/
Package directory: /home/user/.scm/packages/AMS2021.1.packages
Name [id]| Instl. Version | Disk usage | Avail. Version | Licensed? | Req. licenses
========================================================|========================|================|========================|===========|================
ADFCRS-2018 Database [adfcrs]| | | 2018: build 1 | YES | CRS 2021.1
LFDFT atomic database [lfdft]| 1.0: build 0 | 1.42 GiB | 1.0: build 0 | YES | ADF 2021.1
All ML Potential backends [mlpotentials]| | | 1.0.0: build 0 | YES | MLPOT 2021.1
PiNN ML backend [pinn]| | | 0.3.1: build 0 | YES | MLPOT 2021.1
Quantum ESPRESSO [qe]| | | 6.3: build 1 | YES |
SchNetPack ML backend [schnetpack]| | | 0.3.1: build 0 | YES | MLPOT 2021.1
sGDML ML backend [sgdml]| | | 0.4.4: build 0 | YES | MLPOT 2021.1
TorchANI ML backend [torchani]| | | 2.2.0: build 0 | YES | MLPOT 2021.1
Alternatively, machine parsable output can be obtained using --format=yaml
.
Installing a package¶
To install a package, use the install
command and specify the ID of the package.
"${AMSBIN}/amspackages" install lfdft
Multiple packages can be specified on the same line for installation. If you wish to install all available packages, all can be used as a shortcut.
"${AMSBIN}/amspackages" install all
Some packages such as the Machine Learning potentials are installed into your amspython environment. Others are installed into a default directory.
for Linux users this typically is
$HOME/.scm/packages
for MacOS users
$HOME/Library/Application Support/SCM/packages
for Windows users:
%LOCALAPPDATA%/SCM/packages
Installing a package from a script¶
If you want to install a package non-interactively, for instance inside of a batch script, use the --yes
option after the install command. This will answer yes to any interactive prompts.
"${AMSBIN}/amspackages" install --yes lfdft
As an alternative you can set an environment variable. Add export SCM_AMSPKGS_NONINTERACTIVE=True
to your script to answer yes to all prompts.
Check if a package is installed¶
If you want to make sure that a package is installed, you can use the check
command.
"${AMSBIN}/amspackages" check lfdft
The exit status will be 0 if a package is installed, and non-zero otherwise.
For detailed information, you can pass the verbose (-v
) flag. This will tell you what version is installed.
> amspackages -v check lfdft
04-20 17:26:28 lfdft is installed: Ligand Field DFT v[1.0] - build:0 [974661607e9f46c78b6d875e12edfb7a]
Removing a package¶
Removing a package is done by using the remove
command.
This command can take a list of packages as arguments.
For example, to remove the Ligand Field DFT package use:
"${AMSBIN}/amspackages" remove lfdft
While we advise against it, there may be times when a user has accidentally or intentionally made changes or saved new files inside the directories
of one of the packages.
When trying to remove such a package, to prevent data loss you will be warned that a package is modified.
In order to remove packages despite modifications, use the --force
flag.
"${AMSBIN}/amspackages" remove --force lfdft
Updating a package¶
At times a newer version or build of a package may be available.
To install an update, use the update
command.
"${AMSBIN}/amspackages" update lfdft
Note
The update mechanism usually removes user modifications.
Therefore just like the remove command, you will have to specify the --force
flag to update such packages.
Reinstalling packages from a previous version of AMS¶
New in version AMS2022.101: Compatible components from a previous installation of AMS can be copied over, so redownloading and reinstalling is not necessary for some packages.
The copy
command can be used to reuse packages from your previous installation by copying them to the new installation directory.
You can either use the package ID of packages you wish to copy over, or simply type all
and the package manager will copy all compatible packages.
Not all packages will be compatible by default, and the exact list of compatible packages will depend on the version of AMS.
Python packages such as the ML potentials are not compatible with the copy functionality, so they will need to be reinstalled using the install command.
$ "${AMSBIN}/amspackages" copy all
Going to copy packages:
Quantum ESPRESSO v[6.3] - build:1
ADFCRS-2018 Database v[2018] - build:1
LFDFT atomic database v[1.0] - build:0
Proceed with copying packages? [y/N
copied: 100%|##########| 3/3 [00:09<00:00, 3.27s/pkg]
Please note
AMSpackages will not delete the contents of the previous installation. If you wish to remove packages from the old installation, please use the remove command within the old installation.
Using the package manager offline¶
In some cases, it may be difficult to directly download packages on the system where one wishes to install them. This could be due to a firewall on the machine, or an unstable connection. It is possible to instead use a local copy of the package repository downloaded on a different machine. Below you will find instructions to generate a copy of our repository.
You can find a listing of the repository contents at
https://downloads.scm.com/Downloads/packages/listings/
Be sure to use the listing that matches your version of AMS, or you may have unexpected issues with installed packages. You will need to download these files and preserve their directory structure.
Warning
Never install packages from an untrusted source. Only use a copy directly downloaded from our website.
MacOS and Linux¶
Method 1: wget¶
On Unix systems, wget can be used to achieve this from the command line. To download the repository for AMS2024.1 you would use the following from the command line:
wget --user scmuser --password scmpasswd -xi https://downloads.scm.com/Downloads/packages/listings/AMS2024.1.txt
This will create the downloads.scm.com
folder in your current working directory, containing just the path to the repository.
Method 2: python¶
You can download the following python script
to download a copy of the repository.
If you have a local installation of AMS, then you can run it with amspython.
The script can resume after interruption, and update files that have changed in size.
It uses the following command line options
usage: download_repository.py [-h] [--strict-authentication] [--no-ssl] LISTING_URL DOWNLOAD_FOLDER USER PASSWORD
positional arguments:
LISTING_URL Url pointing to the file listings for the repository.
DOWNLOAD_FOLDER A local directory for storing your download.
USER Username for downloading from the website.
PASSWORD Password for downloading from the website.
optional arguments:
-h, --help show this help message and exit
--strict-authentication
If supplied this script will exit on 401 errors. By
default, this script will skip files on 401 errors.
--no-ssl Don't use SSL verification.
All positional arguments are REQUIRED.
For example, you can run it as
amspython download_repository.py https://downloads.scm.com/Downloads/packages/listings/AMS2024.1.txt ~/Downloads/my/repo scmuser scmpassword
If you want to use your own version of python, you need to install tqdm, and requests. Note that the script requires python 3.6 or greater.
python -m pip install tqdm requests
python download_repository.py --help
Windows¶
For Windows users, we provide this powershell script
.
Download this script, along with the listing file.
You can right-click the script file in the file browser and select Run with Powershell
to start it.
It will open a blue window displaying the progress, as well as some interactive prompts that will require you to provide the listing file, the location to store the download, and your SCM user account and password.
Warning
Don’t close the blue powershell window while the program is running.
The script will automatically start downloading the files to the folder of your choosing. It may take a while to download all the files. When it is done, the blue window will display a message saying that it is safe to close.
Note
Your computer may not allow you to run powershell scripts by default.
If you receive a warning about not being allowed to run scripts, first open a open a powershell
window using win + R
on your keyboard, and typing powershell.
In the powershell window that opens type powershell -ep Bypass C:\Path\To\Script
. Fill in the correct path to the script you downloaded.
Below, you can see a demonstration of how you can download and use the local copy on Windows.
You will need an internet connection to see the video.
Using the downloaded copy¶
On the command line you can point the package manager to the downloaded repository as follows
amspackages --repository "/full/path/to/Downloads/packages/AMS2024.1.yml" list
Or, you can set the SCM_AMSPKGS_REPO
environment variable in your shell, or in the configuration file.
This will allow other programs that use the package manager to use the same repository for checking and installing packages.
In your shell (e.g. in .bashrc):
export SCM_AMSPKGS_REPO="/full/path/to/Downloads/packages/AMS2024.1.yml"
In the configuration file:
SCM_AMSPKGS_REPO: /full/path/to/Downloads/packages/AMS2024.1.yml
Instructions for administrators¶
It is typical for several users to access and use the same installation of AMS.
On shared systems, such as a compute cluster, it may therefore be desirable to share installed optional components.
This will prevent redundant installations of the same feature.
If you are an administrator of such a shared installation, you may use the --admin
flag to indicate that you wish to
provide a shared package. For instance to install a shared package, use
"${AMSBIN}/amspackages" --admin install lfdft
Instead of in the user directory, packages will be installed in a shared path. Users will need read-only access to the shared installation directory.
Default paths for the shared installation are operating system dependent, and can be configured to point elsewhere.
for linux users this typically is
/opt/SCM/packages
for MacOS users
/Library/Application Support/SCM/packages
for Windows users:
C:\ProgramData\SCM\packages
AMSpackages supports a set of environment variables that can be set to override default options.
These paths can be overridden using the SCM_AMSPKGS_SHAREDDIR
environment variable.
You should add this to the persistent configurations, so that users can find the installation path that you used for
installing.
Installing python packages for users¶
Python packages may be pre-installed by admins, including the machine learning potentials. The environment variable SCM_PYTHONDIR
must be set manually and exported consistently for all users to do so.
On linux clusters you could set it to the same location as SCM_AMSPKGS_SHAREDDIR
, e.g. export SCM_PYTHONDIR="/opt/SCM/packages
.
Make sure the paths you choose are available to users on compute nodes, not just the login node.
After this you can install some packages directly using the package manager:
export SCM_AMSPKGS_SHAREDDIR="/opt/SCM/packages"
export SCM_PYTHONDIR="/opt/SCM/packages"
"${AMSBIN}/amspackages" --admin install pandas jupyterlab mlpotentials
This will make the packages available to all users. Additionally, you may install packages from pypi.org — at your own risk! — using pip.
Warning
SCM does not provide technical support for packages from pypi.org or other sources. Do not install python packages from untrusted developers.
"${AMSBIN}/amspython" -m pip install lxml # installs the lxml package from pypi.org
When using the paths above, the venv containing all python packages will be located in
/opt/SCM/packages/AMS2024.1.venv
Persistent configurations¶
The amspackages
command has a lot of different options that users or admins may need to change to suit their particular system.
The full list of configuration options can always be seen when running ${AMSBIN}/amspackages --help
.
The usage text contains the name of the environment variables that AMSpackages looks for between []
brackets.
A user can export these in their shell environment, or an administrator can add them to the configuration file to make them persistent throughout the usage of the package manager.
These options will also affect the operation of the package manager gui.
The configuration file will make settings persistent for all users of the installation of AMS.
The file is located inside AMSBIN, under
"${AMSBIN}/amspackageslib/config.yml"
You can open this file in a plain text editor to make changes. Inside you can find several variables that are set by default. Please take care not to accidentally remove the pre-existing variables, as they are necessary for the package manager to function. You can add your own variables to the file, and edit variables such as the repository, for instance to use a local copy.
Some example variables you may wish to set
SCM_AMSPKGS_REPO: /downloads/SCM/packages/AMS2024.1.yml # A local copy of the package repository
SCM_AMSPKGS_SHAREDDIR: /shared/SCM/packages/ # Admin provided packages will be installed here
SCM_AMSPKGS_USERDIR: ~/.scm/packages # User packages will be installed here.
Note that using variables (for example $HOME
) inside the names of paths is not supported, but you can use a ~
as a shortcut for home in path names.
If you require a different configuration per user, it is recommended to set an environment variable inside the user environment instead of using the configuration file.
Note
Variables not prefixed with “SCM_AMSPKGS”, for example SCM_PYTHONDIR can not be set in the config.yml file. They should always be set and exported in the shell environment or they might not take effect.
Environment variables¶
AMSpackages can be configured through the following environment variables.
SCM_AMSPKGS_CONFIG: If set, variables will be read from this file instead of the default configuration file.
SCM_AMSPKGS_REPO: The location of the repository root definition file. Can be either a URL or a path to a local yaml file.
SCM_AMSPKGS_USERDIR: The location for user package directories.
SCM_AMSPKGS_SHAREDDIR: The location for admin/shared package directories.
SCM_AMSPKGS_VERBOSITY: Increase output verbosity (integer value between 0-4).
SCM_AMSPKGS_ALTERNATIVES: Specify keys to alternative versions of packages, e.g. the GPU enabled versions of the machine learning potential packages.
SCM_AMSPKGS_SSL: Set this variable to False
to disable SSL, or point it to a path containing certificates.
SCM_AMSPKGS_NONINTERACTIVE: Set this variable to True
to run without user input (e.g. yes to all prompts).
SCM_AMSPKGS_ADMIN: Set this variable to True
to install packages as an admin.
SCM_AMSPKGS_NOSHARED: Set this variable to True
to ignore any packages installed in the shared installation..
SCM_AMSPKGS_NOCACHE: Set this variable to True
to ignore cached repository information.
SCM_AMSPKGS_PIPSTRICT: Set this variable to True
to always strictly match exact package versions for python packages.
SCM_AMSPKGS_LIST_FORMAT: Set to wide
, long
, or yaml
for default display mode in the amspackages list
output,
SCM_AMSPKGS_COPY_SOURCE: For the amspackages copy
command, the version of the old installation of AMS from which to copy packages, e.g. AMS2024.1
.
F.A.Q. & Troubleshooting¶
Note
If you face problems with the package manager, please include the log file in your support ticket. The log file is called “amspackages.log” and is located under
for Linux
$HOME/.scm/packages/amspackages.log
for MacOS users
$HOME/Library/Application Support/SCM/packages/amspackages.log
for Windows users:
%LOCALAPPDATA%/SCM/packages/amspackages.log
Alternatively, run the command below from the command line and send the debug_info.txt file:
$AMSBIN/amspackages debug > debug_info.txt
Where are packages are installed ?¶
AMSpackages uses the following default locations for the installation of packages:
Linux:
$HOME/.scm/packages
MacOS:
$HOME/Library/Application Support/SCM/packages
Windows:
%LOCALAPPDATA%/SCM/packages
Every version of AMS has its own dedicated folder inside the packages
directory, to allow different installations
to appear side-by-side.
Packages themselves will be located in a uniquely named subdirectory.
It is typically not recommended to make manual changes to installed packages.
If you do need to know the location of a specific package, the loc
command on the command line can help you find them.
$ amspackages loc lfdft
/home/user/.scm/packages/AMS2024.1.packages/kfh27hvj/content
The default location can be changed using the SCM_AMSPKGS_USERDIR
environment variable.
The package manager exits saying “Unhandled exception…”¶
It appears that the package manager has run into an unexpected error. More information about the error will be in the log file, see also the top of this F.A.Q.
To see more information, the command line version of AMSPackages has a --verbose
option.
Using this flag twice will result in large amounts of debug information (short-cut -vv
),
which may include more information about the error.
Some errors may disappear after running amspackages clean
on the command line.
If these errors persist, please contact SCM support and include the log file.
I want to use my own version of package ‘X’¶
There is usually an environment variable that the user can set that points towards their own version of the package. Please check out the documentation of the specific package for instructions.
COSMO-RS Database 2018¶
You can point the environment variable SCM_ADFCRSDIR
to the location of the ADFCRS-2018
folder.
Please see the COSMO-RS Database Tutorial for details.
QuantumESPRESSO¶
Please consult the AMS Quantum ESPRESSO Engine documentation for details.
Machine Learning potentials¶
For python packages such as the ML potential backends, you can use the --alt
option , or SCM_AMSPKGS_ALTERNATIVES
to install GPU-enabled backends.
You can also install packages from PyPI.org yourself, using pip. See the ML potential documentation for details.
I am having trouble downloading packages¶
If you have an unstable or slow connection or no internet connection, you can try to download a copy of our repository elsewhere and use that. You can find instructions here.
Alternatively, you can follow the instructions for using your own version of a package, I want to use my own version of package ‘X’
My problem isn’t listed here¶
You can find more information about getting support for your problem on our support page.