conda-forge is a community led collection of recipes, build infrastructure and packages for the conda package manager.

conda-forge is an effort towards unification of these fragmented users and communities. The conda-forge organization was created to be a transparent, open and community-led organization to centralize and standardize package building and recipe hosting, while improving distribution of the maintenance burden.

 

conda-forge is a community led collection of recipes, build infrastructure and packages for the conda package manager.

The Problem

Historically, the scientific Python community has always wanted a cross-platform package manager that does not require elevated privileges, handles all types of packages, including compiled Python packages and non-Python packages, and generally lets Python be the awesome scientific toolbox of choice.

The conda package manager solved that problem, but in doing so has created new ones:

  • How to get niche tools that are not packaged by Continuum in the “default” channel for Anaconda and Miniconda?
  • Where should built packages be hosted?
  • How should binaries be built to ensure they are compatible with other systems and with the packages from the default channel?

Continuum Analytics does its best to produce reliable conda packages on the default channel, but it can be difficult to keep pace with the many highly specialized communities and their often complex build requirements. The default channel is therefore occasionally out of date, built without particular features or, in some situations, even broken. In response, Continuum provided Anaconda Cloud as a platform for hosting conda packages. Many communities have created their own channels on Anaconda Cloud to provide a collection of reliable packages that they know will work for their users. This has improved the situation within these communities but has also led to duplication of effort, recipe fragmentation and some unstable environments when combining packages from different channels.

The Solution

conda-forge is an effort towards unification of these fragmented users and communities. The conda-forge organization was created to be a transparent, open and community-led organization to centralize and standardize package building and recipe hosting, while improving distribution of the maintenance burden.

What Exactly is conda-forge?

In a nutshell, conda-forge is a GitHub organization containing repositories of conda recipes and a framework of automation scripts for facilitating CI setup for these recipes. In its current implementation, free services from AppVeyor, CircleCI and Travis CI power the continuous build service on Windows, Linux and OS X, respectively. Each recipe is treated as its own repository. referred to as a feedstock, and is automatically built in a clean and repeatable way on each platform.

The built distributions are uploaded to the central repository at anaconda.org/conda-forge and can be installed with conda. For example, to install a conda-forge package into an existing conda environment:

$ conda install –channel conda-forge <package-name>

Or, to add conda-forge to your channels so that it is always searched:

$ conda config –add channels conda-forge
 

How Can I be Part of the conda-forge Community?

Users can contribute in a number of ways. These include reporting issues (as can be seen by this sample issue), updating any of our long list of existing recipes (as can be seen by this sample PR) or by adding new recipes to the collection.

Adding new recipes starts with a a PR to staged-recipes. The recipe will be built on Windows, Linux and OS X to ensure the package’s builds and the recipe’s tests pass.The PR will also be reviewed by the community to assert the recipes are written in a clear and maintainable way. Once the recipe is ready it will be merged and a new feedstock repository will automatically be created for the recipe by the staged-recipes automation scripts. The feedstock repository has a team with commit rights automatically created using the GitHub handles listed in the recipe extra/recipe-maintainers field. The build and upload processes takes place in the feedstock and, once completed, the package will be available on the conda-forge channel.

A full example of this process can be seen with the “colorlog” package. A PR was raised at staged-recipes proposing the new recipe. It was then built and tested automatically, and, after some iteration, it was merged. Subsequently, the colorlog-feedstock repository was automatically generated with full write access for everybody listed in the recipe-maintainers section.

Feedstock Model vs Single Repository Model

Many communities are familiar with the “single repository” model – repositories like github.com/conda/conda-recipes that contain many folders of conda recipes.  This model is not ideal for community maintenance, as it lacks granularity of permissions and struggles to scale beyond tens of recipes. With the feedstock model, in which there is one repo per recipe, each recipe has its own team of maintainers and its own CI. The conda-forge/feedstocks repository puts the recipes back into the more familiar single repository model for those workflows which require it.

How to Find Us

Technical Build Details

The build centralization of conda-forge has provided an opportunity to standardize the build tools used in the ecosystem. By itself, Anaconda Cloud imposes no constraints on build tools. This results in some packages working with only a subset of user systems due to platform incompatibilities. For example, packages built on newer Linux systems will often not run on older Linux systems due to glibc compatibility issues. By unifying and solving these problems, together we are improving the likelihood that any package from the conda-forge channel will be compatible with most user systems. Additionally, pooling knowledge has led to better centralized build tools and documentation than any single community had before. Some of this documentation is at https://github.com/conda-forge/staged-recipes/wiki/Helpful-conda-links

What’s Next?

Conda forge is growing rapidly (~60 contributors, ~400 packages, and >118000 downloads). With more community involvement, everyone benefits: package compatibility is improved, packages stay current and we have a larger pool of knowledge to tackle more difficult issues. We can all go get work done, instead of fighting packaging!

conda-forge is open, transparent, and growing quickly. We would love to see more communities joining the effort to solve improve software packaging for the scientific Python community.

 

About the Authors

Filipe Fernandes

Guest Blogger

Filipe is an independent contractor for the South East Coastal Ocean Observing Regional Association (SECOORA). Filipe is a physical oceanographer turned developer. He is an advocate for the scientific Python stack and developer/maintaine …

Read more


Philip Elson

Guest Blogger

​Philip Elson is a technical lead at the Met Office, the UK’s national weather service. As a global provider of weather and climate services the Met Office has a strong emphasis on partnerships and collaboration, making open source con …

Read more


Q. What is your superpower(s)?

A. Analytics, Developer

Q. What is your technical specialty or area of research?

A. My skills developed out of need as an electron microscop …

Read more


John Kirkham

Guest Blogger

John Kirkham is a software developer that works at Howard Hughes Medical Institute’s Janelia Research Campus in Ashburn, VA. Some of his interests include packaging, image processing, and designing easy-to-use software for da …

Read more


Jonathan Helmus

Guest Blogger

Jonathan Helmus is an scientist and software engineer at Argonne National Laboratory near Chicago, IL.  He is the lead developer of the Python ARM Radar Toolkit (Py-ART), an open source library for working with weather radar …

Read more


Sam Gillingham

Guest Blogger

Sam has been interested in conda for a while as a way to get scientific packages out to Python programmers who might not have the skills to compile from source. He publishes packages on the “rios” channel.  Sam is particularly excited abou …

Read more

Join the Disucssion