Appendix E: Mirroring the SCM package repository
************************************************


Overview
========

A local package mirror is a copy of the SCM package repository that
you keep on your own machine, shared filesystem, or internal network.
This is useful when target systems cannot access the internet
directly, when compute nodes are behind a firewall, or when you want
to prepare one repository copy and reuse it across multiple AMS
installations.

This appendix describes how to create a local copy of the SCM package
repository. For information on telling AMSpackages to use that mirror,
see Appendix D: Advanced package manager usage.

Warning:

  Never install packages from an untrusted source. Only use a mirror
  copied directly from SCM.


Using "amsrepomirror"
=====================

The preferred way to prepare and update a local SCM package mirror is
with "amsrepomirror". This tool is distributed separately from AMS and
is intended to simplify repository mirroring compared with the manual
methods below.

Detailed usage documentation for "amsrepomirror" will be added here in
a future update. Until then, you can continue to create mirrors
manually using the workflows below.


Manual mirroring methods
========================

If you are not using "amsrepomirror", the repository can still be
mirrored manually. You can find a listing of the repository contents
on our website.

Use the listing that matches your version of AMS, or you may have
unexpected issues with installed packages. You will need to download
these files and preserve their directory structure.


MacOS and Linux
===============


Method 1: wget
--------------

Note:

  On MacOS, you will need to install wget first. You can use brew, or
  macports to install it.

On Unix systems, wget can be used to achieve this from the command
line. To download the repository for AMS2025.1 you would use the
following from the command line:

   wget --user scmuser --password scmpasswd -xi https://downloads.scm.com/Downloads/packages/listings/AMS2026.1.txt

This will create the "downloads.scm.com" folder in your current
working directory, containing just the path to the repository.


Method 2: python
----------------

You can download the following "python script" to download a copy of
the repository. If you have a local installation of AMS, then you can
run it with amspython. The script can resume after interruption, and
update files that have changed in size.

It uses the following command line options

   usage: download_repository.py [-h] [--strict-authentication] [--no-ssl] [--local-platform-only] [--delete-unlisted] LISTING_URL DOWNLOAD_FOLDER USER PASSWORD

   positional arguments:
     LISTING_URL           Url pointing to the file listings for the repository.
     DOWNLOAD_FOLDER       A local directory for storing your download.
     USER                  Username for downloading from the website.
     PASSWORD              Password for downloading from the website.

   optional arguments:
     -h, --help            show this help message and exit
     --strict-authentication
                           If supplied this script will exit on 401 errors. By default, this script will skip files on 401 errors.
     --no-ssl              Don't use SSL verification.
     --local-platform-only
                           Skip downloading files for platforms other than the local one (e.g. skip MacOS & Linux packages on Windows).
     --delete-unlisted     Delete files that are not present in the listing from the destination (e.g. clean up old versions of packages).

   All positional arguments are REQUIRED.

For example, you can run it as

   amspython download_repository.py https://downloads.scm.com/Downloads/packages/listings/AMS2026.1.txt ~/Downloads/my/repo scmuser scmpassword

If you want to use your own version of python, you need to install
tqdm, and requests. Note that the script requires python 3.6 or
greater.

   python -m pip install tqdm requests
   python download_repository.py --help


Windows
=======

For Windows users, we provide this "powershell script".

Download this script, along with the listing file. You can right-click
the script file in the file browser and select "Run with Powershell"
to start it.

It will open a blue window displaying the progress, as well as some
interactive prompts that will require you to provide the listing file,
the location to store the download, and your SCM user account and
password.

Warning:

  Don't close the blue powershell window while the program is running.

The script will automatically start downloading the files to the
folder of your choosing. It may take a while to download all the
files. When it is done, the blue window will display a message saying
that it is safe to close.

Note:

  Your computer may not allow you to run powershell scripts by
  default. If you receive a warning about not being allowed to run
  scripts, first open a open a "powershell" window using "win + R" on
  your keyboard, and typing powershell. In the powershell window that
  opens type "powershell -ep Bypass C:\Path\To\Script". Fill in the
  correct path to the script you downloaded.

Below, you can see a demonstration of how you can download and use the
local copy on Windows.

You will need an internet connection to see the video.


Using the mirror with AMSpackages
=================================

After creating a mirror, point AMSpackages to it as described in
Appendix D: Advanced package manager usage. In many common cases,
AMSpackages will also detect a local mirror automatically when a
folder named "repository" is placed inside "AMSHOME", and it contains
the matching "AMS2026.1.yml" file.
