amspython: Python Stack
***********************

Tip:

  For copy-pasteable AMS Python workflows, see Python Examples.


General
=======

The Amsterdam Modeling Suite includes a Python stack based on the
Enthought Python Distribution.

+----------------------------------------------------+----------------------------------------------------+
| AMS version                                        | Python version                                     |
|====================================================|====================================================|
| AMS 2023.1 - AMS 2026.1                            | 3.8.12                                             |
+----------------------------------------------------+----------------------------------------------------+
| AMS 2020.1 - AMS 2022.1                            | 3.6.9                                              |
+----------------------------------------------------+----------------------------------------------------+

This Python stack is completely separate from any other Python
installations on the system.

All programs within the Amsterdam Modeling Suite launch Python via a
special command, "amspython". See Getting Started and Useful commands.


Python packages available in amspython
======================================

The Python stack comes **pre-installed** with, for example:

* PLAMS:  Python library for running and analyzing AMS calculations

* pyCRS: Python library for running and analyzing COSMO-RS
  calculations

* input-classes: Python library for preparing AMS input files

* SCM Base Library: Python library for chemical structures, units,
  parsing AMS input files

* AKFReader: Python library for reading and parsing the AMS KF output
  files

* ASE: Use external calculators with AMS

* ASE: Use AMS as an ASE calculator

* pyZacros: Python library for Zacros Kinetic Monte Carlo simulations

* FlexMD: advanced Python-based molecular dynamics and QM/MM

* AuToGraFS: Python Library for MOFs, Zeolites, COFs, ...

* RDKit

* Standard packages like numpy, scipy, matplotlib

**Optional addon packages** available through the AMS package manager:

* Jupyter lab (optional in AMS2025 and older; included by default in
  AMS2026 and later),

* pandas (optional in AMS2025 and older; included by default in
  AMS2026 and later),

* pymatgen,

* PyTorch,

* Tensorflow,

* and more

**External community packages** not developed by SCM:

* PyADF pursues the concept of automatizing the workflows required for
  multiscale simulations.

* tcutility: Python library containing many helper functions and
  classes for use in programs written in the TheoCheM group


Installing new modules in the AMS Python Stack
==============================================

1. **Pip**

   You can extend the AMS Python Stack by using "pip" to install
   additional modules if they are available on the Python Package
   Index (PyPI).  All "pip" commands need to be prefixed with
   "$AMSBIN/amspython -m":

      $AMSBIN/amspython -m pip list
      $AMSBIN/amspython -m pip show scipy
      $AMSBIN/amspython -m pip search rotate-backups
      $AMSBIN/amspython -m pip install rotate-backups

   This will install the packages into a Python virtual environment.

   Warning:

     When you manually install packages into the AMS Python
     environment, you may break the SCM-supported ML Potential
     packages, for example by installing incompatible versions of
     dependencies. If this happens, it is easiest to remove the AMS
     Python virtual environment completely and reinstall the ML
     Potential packages from the package manager.

2. **Modify SCM_PYTHONPATH**

   Alternatively, you can add the location of the source to the
   "SCM_PYTHONPATH" variable to make the module available in the AMS
   Python Stack. To avoid collisions with other Python installations
   on the system, we unload "PYTHONPATH" and "PYTHONHOME" from the
   environment when launching the ADF Python Stack and put the content
   of "SCM_PYTHONPATH" into "PYTHONPATH".

   Hint:

     If you for some reason have to use the PYTHONPATH variable and
     are unable to use SCM_PYTHONPATH, you can modify
     $AMSBIN/amspython and $AMSBIN/amsipython to not have it cleared
     when starting Python.


Running JupyterLab (Jupyter Notebooks)
======================================

To get started, we recommend following the GUI tutorial Run Jobs with
Jupyter Notebooks.

*New in AMS2025*: Select **SCM → Jupyterlab** in AMSjobs.

To launch JupyterLab from the command-line, run:

   "$AMSBIN/amspython" -m jupyterlab

This will open JupyterLab in your web browser.

To create a new notebook, click the **blue plus (+) button** at the
top left and select **Notebook → Python 3 (ipykernel)**.


Run Jupyter notebooks through VS Code
-------------------------------------


Method 1: Use an existing Jupyter server in VS Code
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To connect the VS Code Jupyter Notebook interface to amspython:

* In AMS2026 and later, JupyterLab is already available. On AMS2025
  and older, install it first. Then run JupyterLab from the command-
  line. This will print a URL in the console. You can close the web
  browser window.

* In VS Code, choose Select Kernel → Existing Jupyter Server → enter
  the URL from the previous step.


Method 2: Create a conda environment that runs amspython
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

VS Code can typically autodetect conda environments on your machine.
This lets you launch Jupyter notebooks directly from VS Code without
first having to launch the server from the command-line.

These instructions assume that you have conda installed on your
machine. If not, you can download and install it from for example
Miniforge.

See also:

  Get started with conda or mamba in the engine ASE documentation.

This has only been tested on Linux.

**First**, make sure JupyterLab is available in amspython. In AMS2026
and later it is included by default; on older AMS versions install it
as described above.

**Then**, create a small conda environment where the "python"
executable simply runs "amspython":

   # create the conda environment
   conda create -n amspython-env

   # create the python executable
   conda run -n amspython-env sh -c '
   mkdir -p "$CONDA_PREFIX/bin"
   PYTHONFILE="$CONDA_PREFIX/bin/python"
   [ -f "$PYTHONFILE" ] && echo "$PYTHONFILE already exists, I will not overwrite it." && exit 1
   echo "#!/bin/sh" > "$PYTHONFILE"
   echo "\"\$AMSBIN/amspython\" \"\$@\"" >> "$PYTHONFILE"
   chmod +x "$PYTHONFILE"
   '

   # test the python executable
   conda run -n amspython-env python -c '
   import sys
   print(f"amspython-env runs python from {sys.executable}")
   '

In VS Code, you can then Select Kernel → Select Other Kernel →
amspython-env.


Execute a notebook from the command-line
----------------------------------------

If you want to run a Jupyter notebook from the command-line, for
example when submitting to a cluster, you can do something like the
following

   cat > template.tpl <<EOF
   {%- block body %}
   {{ nb | json_dumps }}
   {% endblock body %}
   EOF


   "$AMSBIN/amspython" \
       -m nbconvert \
       --to custom \
       --template template.tpl \
       --RegexRemovePreprocessor.patterns='^!' \
       --execute \
       --output output.ipynb \
       notebook.ipynb

Above, the purpose of "template.tpl" is to allow the use of the
"RegexRemovePreprocessor". It will exclude cells starting with an
exclamation mark "!", for example commands used to launch GUI modules
like AMSmovie or AMSspectra.


Jupyter frequently asked questions
----------------------------------


Select kernel
~~~~~~~~~~~~~

When opening a notebook, you may be asked to select a kernel:

[image]

If unsure which is the correct kernel, create a new notebook in
Jupyter and find the correct kernel in the top right:

[image]

Note that Jupyter may detect also kernels unrelated to AMS on the
system, e.g. installed by another program or the user.


Notebook is not trusted
~~~~~~~~~~~~~~~~~~~~~~~

Notebooks not created by the user, including those downloaded from
scm.com, will not always be trusted by Jupyter. A warning may display
in the terminal saying "Notebook <path>/<filename>.ipynb is not
trusted". This means that Jupyter will not execute the contents
automatically, but it is still possible to run the contents manually,
e.g. through pressing shift-enter in a cell. To trust a notebook,
either use the command "jupyter trust <path>/<filename>.ipynb" in the
command line, or click "View > Active Command Palette" and type "Trust
Notebook" in the search bar.


--- Logging error ---
~~~~~~~~~~~~~~~~~~~~~

On Windows, a "--- Logging error ---" is sometimes displayed. This
happens when Unicode characters (e.g. powers or symbols related to
units) are displayed. The error can be ignored, the characters will be
displayed correctly in the notebook and the source in the affected
cell is fully executed.


Python virtual environment
==========================


Default Python virtual environment
----------------------------------

Starting with AMS2020, the "amspython" command by default checks for a
Python virtual environment inside the user's home directory. If it
does not find a virtual environment, it will create one in the
following location (these are the default values for the
"SCM_PYTHONDIR" environment variable):

* Windows: "$USERPROFILE/.scm/python"

* Mac: "$HOME/Library/Application Support/SCM/python"

* Linux: "$HOME/.scm/python"

"amspython" then launches the Python binary located inside the virtual
environment.

If you install additional packages via pip (see above), they will be
installed into the virtual environment's "site-packages".


Virtual environments for different AMS versions
-----------------------------------------------

The virtual environment is tied to the major release version of AMS,
which is reflected in the name of the virtual environment directory
(e.g. AMS2020.1.venv).

Thus, upgrading from AMS2020.101 to AMS2020.102 will automatically let
you use all Python packages that you installed into the AMS2020.101
virtual environment.

To use previously installed Python packages when upgrading from
AMS2020.101 to e.g. AMS2020.301 or AMS2021.101, simply copy or rename
AMS2020.1.venv to AMS2020.3.venv or AMS2021.1.venv in the same
directory.

If you have several installations of different major releases of the
Amsterdam Modeling Suite on the same computer, multiple virtual
environments will also be created, one for each installation.


Changing the location of the Python virtual environment
-------------------------------------------------------

Set the "SCM_PYTHONDIR" environment variable to a directory in which
the virtual environment will be installed. If you use the graphical
user interface (GUI), this environment variable can be changed in the
GUI preferences. If you use the command line, set it in your
"amsbashrc.sh".  If you use both the GUI and the command line, you
should change it in both places.


Disabling the virtual environment
---------------------------------

Follow the steps for changing the location of the Python virtual
environment, but set "SCM_PYTHONDIR" to be empty.


Uninstalling the virtual environment
------------------------------------

Just delete the directory containing the virtual environment. This
will also delete any packages that you have installed into it.


Reinstalling the virtual environment
------------------------------------

To force a reinstallation of the virtual environment, even if it
already exists, run "$AMSBIN/amspython --install_venv". This will not
remove any packages inside the virtual environment.


Useful commands
===============

* Start a Python shell:

     $AMSBIN/amspython

* Install new packages via pip:

     $AMSBIN/amspython -m pip install name_of_package

* Find out the location of an installed package (e.g. numpy):

     $AMSBIN/amspython -c 'import numpy; print(numpy.__file__)'

* Find out which python binary is launched by amspython:

     $AMSBIN/amspython -c 'import sys; print(sys.executable)'

* Disable the virtual environment (one-time):

     SCM_PYTHONDIR='' $AMSBIN/amspython
