Conda is the package manager that comes with Continuum’s Anaconda distribution. Conda makes it easy to install and manage all your packages and environments on Windows, Mac OS X, and Linux.

If you are new to conda, I recommend starting at the documentation, and watching my presentation from SciPy 2014.

In this post, I will assume that you are already familiar with conda and its basic usage, both for installing and building packages. I will look at a few advanced features or tips that may not be well known, even among advanced users of conda. These features will help you with feature discovery, customization, and allow you to manage your packages and environments in more advanced ways.

In part 1, we looked at --help, configuration, conda update --all, conda list --export and conda create --file, conda clean, and pinning packages. In this post, we will look at some tools that make building packages easier and some more features that help manage your environments and use conda more effectively from the command line.

conda skeleton

Building a conda package for an existing package can feel like a lot of boilerplate, because it involves writing down all the metadata for the package, which is already written down. This is especially true for Python packages on PyPI.

The conda skeleton command will generate a skeleton recipe, which you can then edit to a full recipe. It is often good enough that no additional edits are necessary.

conda skeleton pypi <packagename> will create a recipe for <packagename> in the current directory. For example, conda skeleton pypi pyinstrument will create a skeleton for the pyinstrument package.

The --recursive flag will recursively create recipes for any dependencies of the package in the same directory. When conda build runs and a dependency in the recipe is missing, but it finds a recipe for the package in the same directory, it will recursively build the dependency automatically.

conda skeleton also has support for building Perl packages from CPAN with conda skeleton cpan <packagename>. For example, conda skeleton cpan Math::Prime::Util will generate a skeleton recipe for Math::Prime::Util.

The meta.yaml generated by the skeleton command has several comments to help you extend or fixup the generated recipe.

In the future, the skeleton command may grow the ability to generate skeleton recipes from other sources in addition to PyPI and CPAN.

 

conda convert

Conda is unique among system-level package managers in that it is cross-platform: it works in the same way on Windows, Mac OS X, and Linux. However, although the commands work the same way on all platforms, each system requires completely separate packages. For many packages, this is a necessity: a Python binary that runs on Linux will not run on Windows. But for packages that consist only of Python code, the exact same compiled package can, in theory, work on all systems. This is not entirely true, due to some subtle differences, e.g., on Windows the directory structure of Python packages is different than it is on Unix systems. But this translation is simple enough that it can be done automatically.

The conda convert command will take a pure Python package build for one platform and convert it to work on other platforms. The command is run like

$ conda convert sympy-0.7.5-py34_0.tar.bz2 –platform win-32 –platform win-64

This will create platform subdirectories win-32 and win-64 and put the converted packages in them. These subdirectories are created because the package will have the same file name on all platforms.

You can use --platform all to generate packages for the five major platforms that conda supports: win-64, win-32, osx-64, linux-64, and linux-32. The -o flag can be used to change the output directory.

The conda convert command currently only supports converting conda packages from one platform to conda packages of another platform, but it will grow ways of converting other kinds of packages to and/or from conda packages. Currently, conda convert also has the ability to convert .exe installers from Christoph Gohlke into conda packages. To do this, run

$ conda convert sympy-0.7.5.win32-py3.3.exe

 

conda metapackage

A metapackage is a package that contains no files, only metadata. Metapackages are useful if you need an abstract package whose dependencies are the actual packages that will be installed, or that will itself be an abstract dependency of other packages. For instance, in Anaconda, ipython-notebook is a metapackage. The package itself does not contain any files, but it has all the dependencies necessary to install the notebook, such as ipython, pyzmq, tornado, and jinja2.

A metapackage can be created by creating a recipe with the necessary metadata in the meta.yaml, but you can also create a metapackage entirely from the command line with the conda metapackage command.

The metapackage command is run like

$ conda metapackage packagename 1.0

The package name and version are required, but there are several options to add other metadata, such as dependencies.

$ conda metapackage –help
usage: conda-metapackage [-h] [–no-binstar-upload]
[–build-number BUILD_NUMBER]
[–build-string BUILD_STRING]
[–dependencies [DEPENDENCIES [DEPENDENCIES …]]]
[–home HOME] [–license LICENSE] [–summary SUMMARY]
[–entry-points [ENTRY_POINTS [ENTRY_POINTS …]]]
name version

tool for building conda metapackages. A metapackage is a package with no
files, only metadata

positional arguments:
name name of the created package
version version of the created package

optional arguments:
-h, –help show this help message and exit
–no-binstar-upload do not ask to upload the package to binstar
–build-number BUILD_NUMBER
build number for the package (default is 0)
–build-string BUILD_STRING
build string for the package (default is automatically
generated)
–dependencies [DEPENDENCIES [DEPENDENCIES …]], -d [DEPENDENCIES [DEPENDENCIES …]]
The dependencies of the package. To specify a version
restriction for a dependency, wrap the dependency in
quotes, like ‘package >=2.0’
–home HOME The homepage for the metapackage
–license LICENSE The license of the metapackage
–summary SUMMARY Summary of the package. Pass this in as a string on
the command line, like –summary ‘A metapackage for
X’. It is recommended to use single quotes if you are
not doing variable substitution to avoid
interpretation of special characters.
–entry-points [ENTRY_POINTS [ENTRY_POINTS …]]
Python entry points to create automatically. They
should use the same syntax as in the meta.yaml of a
recipe, e.g., –entry-points
bsdiff4=bsdiff4.cli:main_bsdiff4 will create an entry
point called bsdiff4 that calls
bsdiff4.cli.main_bsdiff4()

A metapackage command for ipython-notebook might look something like

$ conda metapackage ipython-notebook 2.2.0 –dependencies python ipython
‘tornado >=3.1’ ‘pyzmq >=2.1.11’ jinja2 –license BSD –home
http://ipython.org/ –summary “Metapackage for the IPython notebook”

 

bdist_conda

If you are a package maintainer for a Python package, building a conda recipe can feel redundant, as all the metadata in the recipe is already specified in your setup.py. An alternative to building and maintaining a separate recipe is to use setup.py bdist_conda. The bdist_conda distutils extension is installed as part of the conda-build package, so you will need to use a conda python in a root environment that has conda-build installed. To use it, run

$ python setup.py bdist_conda

This will run the build process as it would for a conda recipe, but it will use the metadata and install instructions from setup.py.

You can also set conda-specific options in your setup() function, such as the build number and build string. See the bdist_conda docs for more information.

 

Subcommands

Conda can be extended with custom subcommands. Any executable that starts with conda- on your PATH will be picked up as a subcommand of conda. For example, if you have an executable named conda-mycommand, then conda mycommand will call conda-mycommand. conda --help will also parse the help from this command (it assumes the help format of argparse). Many of the commands that come with conda are actually subcommands, like conda-build and conda-skeleton. This lets us keep them in a separate package, conda-build, which can be installed and removed independently of the conda package itself.

Note that if you wish to use conda functionality from your subcommand, it is recommended to use conda-api. See the section on that below.

Bash tab completion

If you use Bash, you can get tab completion by installing the argcomplete package (conda install argcomplete), and adding

eval “$(register-python-argcomplete conda)”

to your bash profile (usually ~/.profile or ~/.bashrc). You can test this by opening a new terminal window or tab and typing

$ conda ins<TAB>

It should complete to

$ conda install

Completion will also work for option flags, and completion for package names is coming soon.

 

Revisions

Conda keeps a history of all the commands run in an environment, and the package changes they resulted in. You can see a list of all changes with

$ conda list –revisions

Here is an example:

$ conda create -n myenv python
Fetching package metadata: ………….
Solving package specifications: .
Package plan for installation in environment /Users/aaronmeurer/anaconda/envs/myenv:

The following NEW packages will be INSTALLED:

openssl: 1.0.1h-1 defaults
python: 3.4.1-4 defaults
readline: 6.2-2 defaults
sqlite: 3.8.4.1-0 asmeurer
tk: 8.5.15-0 defaults
xz: 5.0.5-0 defaults
zlib: 1.2.7-1 defaults

Linking packages …
[ COMPLETE ] |#########################################################| 100%
#
# To activate this environment, use:
# $ source activate myenv
#
# To deactivate this environment, use:
# $ source deactivate
#
$ conda install -n myenv sympy
Fetching package metadata: ………….
Solving package specifications: .
Package plan for installation in environment /Users/aaronmeurer/anaconda/envs/myenv:

The following NEW packages will be INSTALLED:

sympy: 0.7.5-py34_0 asmeurer

Linking packages …
[ COMPLETE ] |#########################################################| 100%
$ conda remove -n myenv sympy
Fetching package metadata: ………….

Package plan for package removal in environment /Users/aaronmeurer/anaconda/envs/myenv:

The following packages will be REMOVED:

sympy: 0.7.5-py34_0 asmeurer

Unlinking packages …
[ COMPLETE ] |#########################################################| 100%
$ conda list -n myenv –revisions
2014-09-22 14:51:24 (rev 0)

2014-09-22 14:51:26 (rev 1)
+openssl-1.0.1h
+python-3.4.1
+readline-6.2
+sqlite-3.8.4.1
+tk-8.5.15
+xz-5.0.5
+zlib-1.2.7

2014-09-22 14:51:45 (rev 2)
+sympy-0.7.5

2014-09-22 14:52:01 (rev 3)
-sympy-0.7.5

Notice that each revision has a number. We can rollback to a previous revision using conda install --revision.

$ conda install -n myenv –revision=2
Fetching package metadata: ………….

Package plan for installation in environment /Users/aaronmeurer/anaconda/envs/myenv:

The following NEW packages will be INSTALLED:

sympy: 0.7.5-py34_0 defaults

Linking packages …
[ COMPLETE ] |########################################################|100%

The revision history is stored in a file conda-meta/history in the environment. This file also keeps track of what commands were used at each revision.

$ cat ~/anaconda/envs/myenv/conda-meta/history
==> 2014-09-22 14:51:24 <==
# cmd: /Users/aaronmeurer/anaconda/bin/conda create -n myenv python
==> 2014-09-22 14:51:26 <==
# cmd: /Users/aaronmeurer/anaconda/bin/conda create -n myenv python
+openssl-1.0.1h-1
+python-3.4.1-4
+readline-6.2-2
+sqlite-3.8.4.1-0
+tk-8.5.15-0
+xz-5.0.5-0
+zlib-1.2.7-1
==> 2014-09-22 14:51:45 <==
# cmd: /Users/aaronmeurer/anaconda/bin/conda install -n myenv sympy
+sympy-0.7.5-py34_0
==> 2014-09-22 14:52:01 <==
# cmd: /Users/aaronmeurer/anaconda/bin/conda remove -n myenv sympy
-sympy-0.7.5-py34_0
==> 2014-09-22 14:56:32 <==
# cmd: /Users/aaronmeurer/anaconda/bin/conda install -n myenv –revision=2
+sympy-0.7.5-py34_0

 

conda-api

If you want to write your own code that extends conda, it is recommended to use conda-api, which can be installed with conda install conda-api. This API calls conda as a subprocess and uses the --json flag to parse JSON output. You can also use the --json flags yourself, which is useful if you want to use conda from a non-Python project.

Using conda-api or the --json API is preferred to using the conda directly, as the format conda command line output is not guaranteed, but the --json output is structured and will not change. It is also in general easier to parse.

It is highly recommended to not import conda from within a Python project. There are two reasons. First, the conda Python API is not guaranteed to remain stable. Second, any code that imports conda as a Python library must be installed in the root environment with conda itself, so it can no longer be installed into an arbitrary conda environment. Code that uses conda-api on the other hand can be installed into any conda environment.

Conclusion

This concludes part 2 of the Advanced Features of Conda blog series. Conda is open source (BSD licensed), and I encourage everyone reading to contribute. The source code for conda and conda-build is on GitHub. I also encourage people to contribute conda recipes to the conda-recipes repo. Join the conda mailing list to discuss conda.


About the Author

Aaron has a M.S. in mathematics from New Mexico State University and a B.S. in mathematics from New Mexico Tech. He is passionate about software, Python, mathematics, and the intersection of the three. Aaron is the lead developer of SymP …

Read more

Join the Disucssion