In a prior blog post, we shared with the community that we’re deprecating support for the Anaconda Build system on Anaconda Cloud. To help navigate the transition, we’re providing some additional guidance on suggested next steps. The Anaconda Cloud team recommends using the community-based conda-forge for open source projects. Conda-forge leverages Github to create high-quality recipes, and Circle-CI, Travis-CI, and Appveyor to produce conda packages on Linux, MacOS and Windows with a strong focus on automation and reproducibility.

Summary

In a prior blog post, we shared with the community that we’re deprecating support for the Anaconda Build system on Anaconda Cloud. To help navigate the transition, we’re providing some additional guidance on suggested next steps.

What is Continuum’s Recommended Approach? 

The Anaconda Cloud team recommends using the community-based conda-forge for open source projects. Conda-forge leverages Github to create high-quality recipes, and Circle-CI, Travis-CI, and Appveyor to produce conda packages on Linux, MacOS and Windows with a strong focus on automation and reproducibility.

Why conda-forge?

Here are some of the benefits you’ll get from using conda-forge: 

  1. Free: Conda-forge is a solidly supported community effort to offer free conda package builds. 
  2. Easier for your users: By building your package with conda-forge, you will be distributing your package in the conda-forge channel. This makes it easier for your users to access your package via the next-most-common place to get packages after the Anaconda defaults channel.
  3. Easier for you: You no longer have to set up extra CI or servers just for building conda packages. Instead, you can take advantage of an existing set of build machinery through the simplicity of creating a Pull Request in Github.
  4. Cross-Platform: If you were using Anaconda Build’s free public queue, only Linux builds were available. With conda-forge, you’ll get support for building packages not just on Linux, but also Windows and MacOS, opening up huge new audiences to your package.
  5. Better feedback: With many more people looking at your recipe, you’ll likely get additional feedback to help improve your package.  
  6. Evolve with best practices: As best practices around package building evolve, these standards will automatically be applied to your recipe(s), so you will be growing and improving in step with the community. 

To read more about conda-forge, see this guest blog post

What is the High-Level Migration Approach? 

If you were using Anaconda Build’s public Queue (Linux worker) and you had a working recipe previously, the migration process should be straightforward for you:

  1. Create a fork of conda-forge’s staged recipes repo to your personal github account, and clone your fork to your computer.
  2. Copy over your existing recipe into the staged-recipes/recipes folder
  3. Take a look at the example recipe in the staged-recipes/recipes folder, and adapt your recipe to the suggested style there.
  4. Submit a PR for new recipe against conda-forge’s staged-recipes repo.
  5. After review of your PR, a conda-forge moderator will merge in the PR.  
  6. Automated scripts will create a new feedstock repo and set up permissions for you on that repo.

Migration Tasks

This section walks through in more detail what the migration tasks are. It is a slightly expanded version of what you’ll find in the conda-forge Staged Recipes Repo readme.

Previously with Anaconda Build, you had…

  1. conda.recipe/: containing at least, meta.yml, maybe build/test scripts and resources (patches) 
  2. .binstar.yml: no longer used going forward

To get started with conda-forge…

  1. Fork https://github.com/conda-forge/staged-recipes
    • git clone https://github.com/<USERNAME>/staged-recipes
    • cp -r <old repo>/conda.recipe staged-recipes/recipes/<PKG-NAME>
    • Edit your recipe, based on example recipe in staged-recipes/recipes
    • git add recipes/
    • git push origin
  2. Create a Pull Request to conda-forge/staged-recipes from your new branch
    • The conda-forge linter bot will post a message about any potential issues or style suggestions it has on your recipe.
    • CI scripts will automatically build against Linux, MacOS, and Windows.
      • You can skip platforms by adding lines with selectors in your meta.yaml such as:
        • build:
          • skip: True  # [win]
  3. Reviewers will suggest improvements to your recipe.
  4. When all builds succeed, it’s ready to be reviewed by a community moderator for merge.
  5. The moderator will review and merge your new staged-recipes.  From here, automated scripts create a new repo (called a feedstock) just for your package, with the github id’s from the recipe-maintainers section added as maintainers. 
  6. As mentioned above, as best practices evolve, automated processes will periodically submit PR’s to your feedstock.  It is up to you and any other administrators to review and merge these PRs.

Example Packages

The following list links to a few prototypical scenarios you may find helpful as a reference: 

  • You can turn off a specific platform with build: skip: true [<platform>], ex: windows:
  • Use a PyPi release upstream with a published sha256 checksum (avoid git branches for the sake of reproducibility):
  • If using post/pre-link scripts, make sure they don’t make lots of output, instead write to .messages.txt:
  • If you are building packages with compiled components (pure C libraries, Python packages with compiled extensions, etc.) use the toolchain package
  • If you need custom linux packages that are not available as conda packages, specify them with yum_requirements.txt:
  • Many Python-only recipes drawn from PyPI, as well as other simple recipes, can be as simple as a single meta.yaml file with script commands:

What to be Aware of 

We’re really excited about conda-forge and supportive of its efforts. However, these are some things to keep in mind as you explore if it’s a fit for your project: 

  • If you were using Anaconda-Build as your Continuous Integration service, you probably do have to have to set up a new workflow; the astropy ci-helpers are very helpful. Conda-forge is not your project’s CI!
  • The upshot of this is you should only be building in conda-forge when you are ready to do a release. If you do try to use conda-forge as CI, you will likely get frustrated because your builds will take longer than you want and it won’t feel like a CI system.
  • When you submit a new package, you (and possibly the people you recruit) are signing up to be a maintainer of this package. The community expectation is that you will continue to update your package with respect to the upstream dependencies. The implied message is that conda-forge is not a great place to add hobby projects that you have no intention of maintaining.
  • If your recipe doesn’t meet the community quality guidelines, you are likely to get community feedback with the expectation that you’ll make changes.

Alternatives to be Aware of

Creating reproducible, automated, cross-platform is easier today than in the past with virtualization and containerization, but still challenging due to proprietary licenses and compiler toolchains. We’ve got some examples of conda build environments in different environments that can make easier to build your own packages virtually with vagrant:

…and Docker…

If you are thinking about how to tackle this problem at scale for private/commercial projects, or internally behind the firewall, please contact us to find out how we can help. 

Getting Started and Getting Involved

  1. General info: Read more about conda-forge: https://conda-forge.github.io/ 
  2. Create your receipe: start with the readme here: https://github.com/conda-forge/staged-recipes 
  3. For general Getting Started issues, try the FAQ on the staged-recipes wiki.
  4. Getting help: Non-package-specific issues, i.e. best practices, guidance, are best raised on the conda-forge.github.io issue tracker.  There is also a gitter chat channel: https://gitter.im/conda-forge/conda-forge.github.io.
  5. News: To find out how the community is evolving, stay tuned to meeting minutes on hackpad, and feel free to join meetings there.  Advanced notice is important, though – meetings are often hosted on Google Hangouts, which has a 10 person cap.  We have alternate avenues, but need to set them up.

What’s Your Experience?

If you’ve been using conda-forge or completed a migration, let us know what your experience has been in the comments section below, or on Twitter @ContinuumIO