The Conda Configuration Engine for Power Users

 

 

Released last fall, conda 4.2 brought with it configuration superpowers. The capabilities are extensive, and they’re designed with conda power users, devops engineers, and sysadmins in mind.

Configuration information comes from four basic sources:

  1. hard-coded defaults,
  2. configuration files,
  3. environment variables, and
  4. command-line arguments.

Each time a conda process initializes, an operating context is built that in a cascading fashion merges configuration sources. Command-line arguments hold the highest precedence, and hard-coded defaults the lowest.

The configuration file search path has been dramatically expanded. In order from lowest to highest priority, and directly from the conda code,

SEARCH_PATH = ( 
    '/etc/conda/.condarc', 
    '/etc/conda/condarc', 
    '/etc/conda/condarc.d/', 
    '/var/lib/conda/.condarc', 
    '/var/lib/conda/condarc', 
    '/var/lib/conda/condarc.d/', 
    '$CONDA_ROOT/.condarc', 
    '$CONDA_ROOT/condarc', 
    '$CONDA_ROOT/condarc.d/', 
    '~/.conda/.condarc', 
    '~/.conda/condarc', 
    '~/.conda/condarc.d/', 
    '~/.condarc', 
    '$CONDA_PREFIX/.condarc', 
    '$CONDA_PREFIX/condarc', 
    '$CONDA_PREFIX/condarc.d/', 
    '$CONDARC', 
)

where environment variables and user home directory are expanded on first use.

$CONDA_ROOT

is automatically set to the root environment prefix (and shouldn’t be set by users), and

$CONDA_PREFIX

is automatically set for activated conda environments. Thus, conda environments can have their own individualized and customized configurations. For the ”

.d

” directories in the search path, conda will read in sorted order any (and only) files ending with .yml or .yaml extensions. The

$CONDARC

environment variable can be any path to a file having a .yml or .yaml extension, or containing “condarc” in the file name; it can also be a directory.

Environment variables hold second-highest precedence, and all configuration parameters are able to be specified as environment variables. To convert from the condarc file-based configuration parameter name to the environment variable parameter name, make the name all uppercase and prepend

CONDA_

. For example, conda’s

always_yes

configuration parameter can be specified using a

CONDA_ALWAYS_YES

environment variable.

Configuration parameters in some cases have aliases. For example, setting

always_yes: true

or

yes: true

in a configuration file is equivalent to the command-line flag

--yes

. They’re all also equivalent to both

CONDA_ALWAYS_YES=true

and

CONDA_YES=true

environment variables. A validation error is thrown if multiple parameters aliased to each other are specified within a single configuration source.

There are three basic configuration parameter types: primitive, map, and sequence. Each follow a slightly different set of merge rules.

The primitive configuration parameter is the easiest to merge. Within the linearized chain of information sources, the last source that sets the parameter wins. There is one caveat: if the parameter is trailed by a

#!final

flag, the merge cascade stops for that parameter. (Indeed, the markup concept is borrowed from the

!important

rule in CSS.) While still giving end-users extreme flexibility in most cases, we also give sysadmins the ability to lock down as much configuration as needed by making files read-only.

Map configuration parameters have elements that are key-value pairs. Merges are at the per-key level. Given two files with the contents

# file: /etc/conda/condarc.d/proxies.yml
proxy_servers: https: 
  http://prod-proxy

# file: ~/.conda/condarc.d/proxies.yml
proxy_servers: 
  http: http://dev-proxy:1080 
  https: http://dev-proxy:1081

the merged

proxy_servers

configuration will be

proxy_servers: 
  http: http://dev-proxy:1080 
  https: http://dev-proxy:1081

However, by modifying the contents of the first file to be

# file: /etc/conda/condarc.d/proxies.yml
proxy_servers:
  https: http://prod-proxy #!final

the merged settings will be

proxy_servers: 
  http: http://dev-proxy:1080 
  https: http://prod-proxy

Note the use of the

!final

flag acts at the per-key level. A

!final

flag can also be set for the parameter as a whole. With the first file again changed to

# file: /etc/conda/condarc.d/proxies.yml
proxy_servers: #!final 
  https: http://prod-proxy

the merged settings will be

proxy_servers: 
  https: http://prod-proxy

with no

http

key defined.

The sequence parameter merges are the most involved. Consider contents of the three files

# file: /etc/conda/condarc
channels: 
  - one 
  - two

# file: ~/.condarc
channels: 
  - three 
  - four

# file: $CONDA_PREFIX/.condarc
channels: 
  - five 
  - six

the final merged configuration will be

channels: 
  - five 
  - six 
  - three 
  - four 
  - one 
  - two

Sequence order within each individual configuration source is preserved, while still respecting sources’ overall precedence. Just like map parameters, a

!final

flag can be used for a sequence parameter as a whole. However, the

!final

flag does not apply to individual elements of sequence parameters, and instead

!top

and

!bottom

flags are available. Modifying the sequence example to the following

# file: /etc/conda/condarc
channels: 
  - one #!top 
  - two

# file: ~/.condarc
channels: #!final 
  - three 
  - four #!bottom

# file: $CONDA_PREFIX/.condarc
channels: 
  - five 
  - six

will yield a final merged configuration

channels: 
  - one 
  - three 
  - two 
  - four

Managing all of these new sources of configuration could become difficult without some new tools. The most basic is

conda config --validate

, which simply exits

0

if conda’s configured state passes all validation tests. The command

conda config --describe

(recently added in 4.3.16) gives a detailed description of available configuration parameters.

We’ve also added the commands

conda config --show-sources

and

conda config --show

. The first displays all of the configuration information conda recognizes–in its non-merged form broken out per source. The second gives the final, merged values for all configuration parameters.

Conda’s configuration engine gives power users tools for ultimate control. If you’ve read to this point, that’s probably you. And as a conda power user, please consider participating in the conda canary program. Be on the cutting edge, and also help influence new conda features and behaviors before they’re solidified in general availability releases.


You May Also Like

Data Science Blog
Deep Learning with GPUs in Anaconda Enterprise
AI is a hot topic right now. While a lot of the conversation surrounding advanced AI techniques such as deep learning and machine learning can be chalked up to hype, the under...
Read More
Data Science Blog
Seven Things You Might Not Know About Numba
In this post, I want to dive deeper and demonstrate several aspects of using Numba on the GPU that are often overlooked. I’ll quickly breeze through a number of topics, but ...
Read More
Company Blog
What You Missed on Day Three of AnacondaCON 2018
And that’s a wrap! Yesterday was the third and final day of AnacondaCON 2018, and what a ride it’s been. Read some highlights from what you missed, and stay tuned for our ...
Read More