> ## Documentation Index > Fetch the complete documentation index at: https://anaconda.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Mirroring channels and packages Anaconda Enterprise enables you to create a local copy of a repository so users can access the packages from a centralized, on-premises location. The mirror can be complete, partial, or include specific packages or types of packages. You can also create a mirror in an air gapped environment to help improve performance and security. It can take hours to mirror the full repository. Before you can use Anaconda Enterprise’s convenient syncing tools to configure local mirrors for channels and packages, you’ll need to [configure access to the source of the packages](./config-repo-access) to be mirrored, whether an online repository or a tarball (if an airgapped installation). **Prerequisites:** * [Install and configure Anaconda Enterprise](../../install/main) * [Install conda](../../install/conda) * [Install the Anaconda Enterprise CLI](../../install/ae-cli) **Types of mirroring:** * To create a complete mirror, see [Mirroring the Anaconda repository](#mirroring-the-anaconda-repository) or [Mirroring a PYPI repository](#mirroring-a-pypi-repository). * To create partial mirror, see [Mirroring specific packages](#mirroring-specific-packages). * To mirror a repository in a system without internet access, see [Mirroring in an air-gapped environment](#mirroring-in-an-air-gapped-environment). * To share mirrors, see [Configuring Anaconda Enterprise](#ssl-verification) and [Sharing channels](#sharing-channels). **Configuration options:** * [Configuring SSL verification](#ssl-verification) * [Mirroring in a proxied environment](#mirroring-in-a-proxied-environment) * [Mirroring specific platforms and versions](#platform-specific-mirroring) * [Using blacklisting to exclude packages](#blacklist-mirroring) * [Using whitelisting to include packages](#whitelist-mirroring) * [Combining multiple configuration arguments](#combining-multiple-mirror-configurations) Log in to Anaconda Enterprise as an existing user using the following command: ``` $ anaconda-enterprise-cli login Username: anaconda-enterprise Password: Logged anaconda-enterprise in! ``` If Anaconda Enterprise 5 is installed in a proxied environment, see [Mirroring in a proxied environment](#mirroring-in-a-proxied-environment) for information on setting the `NO_PROXY` variable. *** ## Mirroring the Anaconda repository Anaconda recommends the following process as a best practice for mirroring the Anaconda Enterprise Repository. 1. Instead of using the default `anaconda.yaml` file included in the mirror tool installation, create two `yaml` files, one for mirroring the `main` channel, and another for mirroring the `free` channel. Example `main.yaml` file: ``` dest_channel: main channels: - https://repo.anaconda.com/pkgs/main platforms: - linux-64 - noarch ``` Example `free.yaml` file: ``` dest_channel: free channels: - https://repo.anaconda.com/pkgs/free platforms: - linux-64 - noarch ``` 2. If you saved both of these files to the home directory, you can use the following commands to mirror these channels. Otherwise, amend the path so that it corresponds to where you saved the files: ``` cas-sync-api-v5 --file ~/main.yaml cas-sync-api-v5 --file ~/free.yaml ``` This mirrors all of the packages from these channels in the Anaconda repository. If the channel doesn’t already exist, it will be automatically created and shared with all authenticated users. You can customize the permissions on the mirrored packages by [sharing the channel](../../data-science-workflows/packages/share). If you plan to mirror these channels on a regular basis, consider adding the `-c` flag to get a clean mirror each time. This will automatically remove any packages that have been removed from the Anaconda repository between mirrors from your internal repository—excluding any packages your organization has blacklisted. 3. Verify that the mirror was successful by logging into your account and navigating to the **Packages** tab. You should see a list of the mirrored packages. *** ## Mirroring a PyPI repository The full PyPI mirror size is currently [close to 4TB](https://p.datadoghq.com/sb/7dc8b3250-85dcf667bd), so ensure that your file storage location has sufficient disk space before proceeding. Rather than mirror the entire PyPI repository, you can use a configuration file such as `$PREFIX/etc/anaconda-platform/mirrors/pypi.yaml` to customize the mirror behavior and specify the subset of packages you want to mirror. **To create a PyPI mirror**: ``` anaconda-enterprise-cli mirror pypi --config pypi.yaml ``` This command loads the packages on `https://pypi.org` into the `pypi` user account. Mirrored packages can be viewed at `/repository/pypi/pypi/simple/`, replacing `` with the actual URL to your installation of Anaconda Enterprise. (The second `pypi` in the url should match the `user` configuration value described below.) The following configuration options are available for you to customize your configuration file: | Name | Description | | :----------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `user` | The local user under which the PyPI packages are imported. Default: `pypi`. | | `pkg_list` | A list of packages to mirror. Only packages listed are mirrored. If this is set, `blacklist` and `whitelist` settings are ignored. **Default:** `[]`. | | `whitelist` | A list of packages to mirror. Only packages listed are mirrored. If the list is empty, all packages are checked. Default: `[]`. | | `blacklist` | A list of packages to skip. The packages listed are ignored. Default: `[]`. | | `latest_only` | Only download the latest versions of the packages. Default: `false`. | | `remote_url` | The URL of the PyPI mirror. `/pypi` is appended to build the XML RPC API URL, `/simple` for the simple index and `/pypi/{package}/{version}/json` for the JSON API. Default: `https://pypi.python.org/`. | | `xml_rpc_api_url` | A custom value for XML RPC URL. If this value is present, it takes precedence over the URL built using `remote_url`. Default: `null`. | | `simple_index_url` | A custom value for the simple index URL. If this value is present, it takes precedence over the URL built using `remote_url`. Default: `null`. | | `use_xml_rpc` | Whether to use the XML RPC API as specified by [PEP381](https://www.python.org/dev/peps/pep-0381/). If this is set to `true`, the XML RPC API is used to determine which packages to check. Otherwise the scripts falls back to the simple index. If the XML RPC fails, the simple index is used. Default: `true`. | | `use_serial` | Whether to use the serial number provided by the XML RPC API. Only packages updated since the last serial saved are checked. If this is set to false, all PyPI packages are checked for updates. Default: `true`. | | `create_org` | Create the mirror user as an organization instead of a regular user account. All superusers are added to the “Owners” group of the organization. Default: `false`. | Note that all mirrored PyPI-like channels are publicly available to pull packages from both inside and outside the cluster (no auth token required). EXAMPLE: ``` whitelist: - requests - six - numpy - simplejson latest_only: true remote_url: https://pypi.org/ use_xml_rpc: true ``` ### Configuring pip To configure pip to use this new mirror, create `pip.conf` as follows: ``` [global] index-url=/repository/pypi/pypi/simple/ ``` replacing `` with the actual URL to your Anaconda Enterprise. To configure Anaconda Enterprise sessions and deployments to automatically use the `pip.conf` run the following command. ``` anaconda-enterprise-cli spark-config --config /etc/pip.conf pip.conf ``` Alternately, if you can use the `--index-url` flag directly when invoking pip. For example, ``` pip install --index-url /repository/pypi/pypi/simple/ ``` replacing `` with the actual URL to your Anaconda Enterprise installation, and `` with the name of a package that is in your local mirror. In the example URL, the second `pypi` should match the `user` configuration value described above. For more specific information on configuring pip, refer to the official documentation at [https://pip.pypa.io/en/stable/user\_guide/#config-file](https://pip.pypa.io/en/stable/user_guide/#config-file). *** ## Mirroring specific packages Alternately, you may not want to mirror all packages. In this case, you can specify which platforms or specific packages you want to mirror —*or*— use the whitelist, blacklist or license\_blacklist functionality to control which packages are mirrored, by editing the provided mirror files. *You cannot combine these methods*. For more information, see [Mirror configuration options](#mirror-configuration-options). ``` cas-sync-api-v5 --file ~/my-custom-anaconda.yaml ``` *** ## Mirroring R packages An example configuration file for mirroring R packages is also provided: ``` dest_channel: r channels: - https://repo.anaconda.com/pkgs/r/ platforms: - linux-64 ``` ``` cas-sync-api-v5 --file ~/cas-mirror/etc/anaconda-platform/mirrors/r.yaml ``` *** ## Mirroring in an air-gapped environment To mirror the repository in a system with no internet access, create a local copy of the repository by extracting the airgapped tarball and point `cas-sync-api-v5` to the extracted tarball. In this example we will extract to `/tmp`: ``` # Replace ```` with the actual path to the mirror file. cd /tmp tar xvf /mirror.tar ``` Now you have a local file-system repository located at `/tmp/mirror/pkgs`. You can mirror this repository by editing `/etc/anaconda-platform/mirrors/anaconda.yaml` to contain: ``` channels: - /tmp/mirror/pkgs ``` And then run the command: ``` cas-sync-api-v5 --file etc/anaconda-platform/mirrors/conda.yaml ``` ## This mirrors the contents of the local file-system repository to your Anaconda Enterprise installation under the username `anaconda`. ## Configuring Anaconda Enterprise After creating the mirror, [edit your Anaconda Enterprise configuration](../advanced/settings) to add this new mirrored channel to the default Anaconda Enterprise channels and make the packages available to users. ``` conda: channels: - defaults default_channels: - main - free - r channel_alias: https:///repository/conda ``` Replacing `` with the actual URL to your installation of Anaconda Enterprise. The `ap-workspace` pod must be restarted for the configuration change to take effect on new project editor sessions. To update the Anaconda Enterprise server with your changes, you’ll need to do the following: 1. Run the following command in an interactive shell to identify the pod associated with the workspace services: ``` kubectl get pods ``` 2. Restart the workspace services by running the following command: ``` kubectl delete pod anaconda-enterprise-ap-workspace- ``` *** ## Sharing channels To make your new channels visible to your users in their **Channels** list, you need to share the channels with them. EXAMPLE: To share new channels `main`, `free`, and `r` with group `everyone` for read access: ``` anaconda-enterprise-cli channels share --group everyone --level r main anaconda-enterprise-cli channels share --group everyone --level r free anaconda-enterprise-cli channels share --group everyone --level r r ``` After running the `share` command, verify by logging onto the user interface and viewing the **Channels** list. For more information, see [Sharing channels and packages](../../data-science-workflows/packages/share) *** ### Mirror configuration options You can use the following options to configure your mirror: * [Configuring SSL verification](#ssl-verification). * [Mirroring in a proxied environment](#mirroring-in-a-proxied-environment). * [Mirroring specific platforms and versions](#platform-specific-mirroring). * [Using blacklisting to exclude packages](#blacklist-mirroring). * [Using whitelisting to include package](#whitelist-mirroring). * [Combining multiple configuration arguments](#combining-multiple-mirror-configurations). **remote\_url** Specifies the remote URL from which the conda packages and the Anaconda and Miniconda installers are downloaded. The default value is: `https://repo.continuum.io/`. **channels** Specifies the remote channels from which conda packages are downloaded. The default is a list of the channels `/pkgs/free/` and `/pkgs/pro/` All specification information should be included in the same file, and can be passed to the cas-sync-api-v5 command via the `--file` argument: ``` cas-sync-api-v5 --file ~/cas-mirror/etc/anaconda-platform/mirrors/anaconda.yaml ``` **destination channel** The configuration option `dest_channel` specifies where files will be uploaded. The default value is: `anaconda`. *** ## SSL verification The mirroring tool uses two different settings for configuring SSL verification. When the mirroring tool connects to its destination, it uses the `ssl_verify` setting from `anaconda-enterprise-cli` to determine how to validate certificates. For example, to use a custom certificate authority: ``` anaconda-enterprise-cli config set sites.master.ssl_verify /etc/ssl/certs/ca-certificates.crt ``` The mirroring tool uses conda’s configuration to determine how to validate certificates when connecting to the source that it is pulling packages from. For example, to disable certificate validation when connecting to the source: ``` conda config --set ssl_verify false ``` *** ## Mirroring in a proxied environment If Anaconda Enterprise 5 is installed in a proxied environment, set the `NO_PROXY` variable. This ensures the mirroring tool does not use the proxy when communicating with the repository service, and prevents errors such as `Max retries exceeded`, `Cannot connect to proxy`, and `Tunnel connection failed: 503 Service Unavailable`. ``` export NO_PROXY= ``` *** ## Platform-specific mirroring By default, the `cas-sync-api-v5` tool mirrors all platforms. If you do not need all platforms, edit the YAML file to specify the platform(s) you want mirrored: ``` platforms: - linux-64 - osx-64 - win-64 ``` The platform argument is evaluated before any other argument. *** ## Package-specific mirroring In some cases you might want to mirror only a small subset of the repository. Rather than blacklisting a long list of packages you do not want mirrored, you can instead simply enumerate the list of packages you DO want mirrored. This argument cannot be used with the `blacklist`, `whitelist` or `license_blacklist` arguments—it can only be combined with platform-specific and version-specific mirroring. EXAMPLE: ``` pkg_list: - accelerate - pyqt - zope ``` This example mirrors only the three packages: Accelerate, PyQt & Zope. All other packages will be completely ignored. *** ## Python version-specific mirroring Mirror the repository with a Python version or versions specified. EXAMPLE: ``` python_versions: - 3.3 ``` Mirrors only Anaconda packages built for Python 3.3. *** ## License blacklist mirroring The mirroring script supports license blacklisting for the following license families: ``` AGPL GPL2 GPL3 LGPL BSD MIT Apache PSF Public-Domain Proprietary Other ``` EXAMPLE: ``` license_blacklist: - GPL2 - GPL3 - BSD ``` This example mirrors all the packages in the repository EXCEPT those that are GPL2-, GPL3-, or BSD-licensed, because those three licenses have been blacklisted. *** ## Blacklist mirroring The blacklist allows access to all packages EXCEPT those explicitly listed. If the `license_blacklist` and `blacklist` arguments are combined, `license_blacklist` is evaluated first, and `blacklist` is a supplemental modifier. EXAMPLE: ``` blacklist: - bzip2 - tk - openssl ``` This example mirrors the entire repository EXCEPT the `bzip2`, `Tk`, and `OpenSSL` packages. *** ## Whitelist mirroring The `whitelist` argument adds or includes packages that would be otherwise excluded by the `blacklist` and/or `license_blacklist` functions. EXAMPLE: ``` license_blacklist: - GPL2 - GPL3 whitelist: - readline ``` This example mirrors the entire repository EXCEPT any GPL2- or GPL3-licenses packages, but includes `readline`, despite the fact that it is GPL3-licensed. ----------------------------------------------------------------------------- ## Combining multiple mirror configurations You might find that combining two or more of the arguments above is the easiest way to get the exact combination of packages that you want. The platform argument is evaluated before any other argument. EXAMPLE: This example mirrors only Linux-64 distributions of the dnspython, Shapely and GDAL packages: ``` platforms: - linux-64 pkg_list: - dnspython - shapely - gdal ``` If the `license_blacklist` and `blacklist` arguments are combined, `license_blacklist` is evaluated first, and `blacklist` is a supplemental modifier. EXAMPLE: In this example, the mirror configuration does not mirror GPL2-licensed packages. It does not mirror the GPL3 licensed package `pyqt` because it has been blacklisted. It does mirror all other packages in the repository: ``` license_blacklist: - GPL2 blacklist: - pyqt ``` If the `blacklist` and `whitelist` arguments are both employed, the blacklist is evaluated first, with the whitelist functioning as a modifier. EXAMPLE: This example mirrors all packages in the repository except `astropy` and `pygments`. Despite being listed on the blacklist, accelerate is mirrored because it is listed on the whitelist. ``` blacklist: - accelerate - astropy - pygments whitelist: - accelerate ```