Appendix E: Mirroring the SCM package repository

Overview

A local package mirror is a copy of the SCM package repository that you keep on your own machine, shared filesystem, or internal network. This is useful when target systems cannot access the internet directly, when compute nodes are behind a firewall, or when you want to prepare one repository copy and reuse it across multiple AMS installations.

This appendix describes how to create a local copy of the SCM package repository. For information on telling AMSpackages to use that mirror, see Appendix D: Advanced package manager usage.

Warning

Never install packages from an untrusted source. Only use a mirror copied directly from SCM.

Using amsrepomirror

The preferred way to prepare and update a local SCM package mirror is with amsrepomirror. This tool is distributed separately from AMS and is intended to simplify repository mirroring compared with the manual methods below.

Detailed usage documentation for amsrepomirror will be added here in a future update. Until then, you can continue to create mirrors manually using the workflows below.

Manual mirroring methods

If you are not using amsrepomirror, the repository can still be mirrored manually. You can find a listing of the repository contents on our website.

Use the listing that matches your version of AMS, or you may have unexpected issues with installed packages. You will need to download these files and preserve their directory structure.

MacOS and Linux

Method 1: wget

Note

On MacOS, you will need to install wget first. You can use brew, or macports to install it.

On Unix systems, wget can be used to achieve this from the command line. To download the repository for AMS2025.1 you would use the following from the command line:

wget --user scmuser --password scmpasswd -xi https://downloads.scm.com/Downloads/packages/listings/AMS2026.1.txt

This will create the downloads.scm.com folder in your current working directory, containing just the path to the repository.

Method 2: python

You can download the following python script to download a copy of the repository. If you have a local installation of AMS, then you can run it with amspython. The script can resume after interruption, and update files that have changed in size.

It uses the following command line options

usage: download_repository.py [-h] [--strict-authentication] [--no-ssl] [--local-platform-only] [--delete-unlisted] LISTING_URL DOWNLOAD_FOLDER USER PASSWORD

positional arguments:
  LISTING_URL           Url pointing to the file listings for the repository.
  DOWNLOAD_FOLDER       A local directory for storing your download.
  USER                  Username for downloading from the website.
  PASSWORD              Password for downloading from the website.

optional arguments:
  -h, --help            show this help message and exit
  --strict-authentication
                        If supplied this script will exit on 401 errors. By default, this script will skip files on 401 errors.
  --no-ssl              Don't use SSL verification.
  --local-platform-only
                        Skip downloading files for platforms other than the local one (e.g. skip MacOS & Linux packages on Windows).
  --delete-unlisted     Delete files that are not present in the listing from the destination (e.g. clean up old versions of packages).

All positional arguments are REQUIRED.

For example, you can run it as

amspython download_repository.py https://downloads.scm.com/Downloads/packages/listings/AMS2026.1.txt ~/Downloads/my/repo scmuser scmpassword

If you want to use your own version of python, you need to install tqdm, and requests. Note that the script requires python 3.6 or greater.

python -m pip install tqdm requests
python download_repository.py --help

Windows

For Windows users, we provide this powershell script.

Download this script, along with the listing file. You can right-click the script file in the file browser and select Run with Powershell to start it.

It will open a blue window displaying the progress, as well as some interactive prompts that will require you to provide the listing file, the location to store the download, and your SCM user account and password.

Warning

Don’t close the blue powershell window while the program is running.

The script will automatically start downloading the files to the folder of your choosing. It may take a while to download all the files. When it is done, the blue window will display a message saying that it is safe to close.

Note

Your computer may not allow you to run powershell scripts by default. If you receive a warning about not being allowed to run scripts, first open a open a powershell window using win + R on your keyboard, and typing powershell. In the powershell window that opens type powershell -ep Bypass C:\Path\To\Script. Fill in the correct path to the script you downloaded.

Below, you can see a demonstration of how you can download and use the local copy on Windows.

You will need an internet connection to see the video.

Using the mirror with AMSpackages

After creating a mirror, point AMSpackages to it as described in Appendix D: Advanced package manager usage. In many common cases, AMSpackages will also detect a local mirror automatically when a folder named repository is placed inside AMSHOME, and it contains the matching AMS2026.1.yml file.