New Improvements and Features
This 22.11 release of conda brings a suite of new improvements and features that have been highly anticipated by the community. Conda runs faster now—much faster. Here is an abbreviated list of notable changes:
-
The faster conda-libmamba-solver is no longer marked as experimental.
-
Conda package downloads are now parallelized.
-
Conda is now plugin ready, with a new architecture.
If you have not yet installed conda, you may do so by installing the Anaconda Distribution or Miniconda. If you have conda installed already (you can check in your terminal by entering “conda info”), all you need to do is update conda:
conda info
conda update -n base conda
The New conda-libmamba-solver, Explained
Back in March of this year, the conda team introduced a new experimental solver, conda-libmamba-solver, as part of our collaborative effort with Quansight to make conda faster. Speed improvement has been one of conda’s most highly requested updates, and after almost a year of joint development work, we are incredibly excited to share that the conda-libmamba-solver has now joined the default solver club.
To install and set the new solver, run the following commands:
conda install -n base conda-libmamba-solver
conda config --set solver libmamba
You may also refer to this comprehensive Getting Started guide. The classic solver is still available.
The conda-libmamba-solver greatly increases the speed with which conda solves an environment when installing packages. In our experimental release announcement, we compared the performance of the solvers by installing popular Python packages SciPy and TensorFlow on the same Windows machine.
In that performance test, the classic solver downloaded the two large libraries in 91 seconds, whereas libmamba, the libsolv-based solver, took only 15 seconds—an ~80% speed improvement. The exact speed improvements of the new solver will vary based on the environment and packages users are installing, but you can expect to see improvements in the 50-80% range across use cases.
Implementing a new solver for conda while prioritizing maximized backwards compatibility was a non-trivial challenge. Conda is used by over 30 million users around the world, and it didn’t have the underlying infrastructure to support experimental tools.
Directly updating the default solver itself without causing disruptions to existing workflows would require a high dose of optimistic hope. So, instead of doing that, we decided to develop a plugin architecture proof-of-concept with Quansight to empower users with the choice of an alternate solver.
If you are unsure whether you should use the new solver, you may refer to “Should I use conda-libmamba-solver?,” where we have detailed the differences between the classic and libmamba solvers.
TL;DR: The conda-libmamba-solver is now available as a fully supported solver option. This was made possible by a new plugin architecture. More about the plugin architecture below.
Install conda Packages Faster With Parallelized Downloads
In addition to increasing conda’s solving speeds, we have also greatly increased the speed of downloading and extracting conda packages.
Parallel extraction can approximately double the speed of extracting .conda and .tar.bz2 files. The more packages you are downloading and the faster your connection to the server, the greater the speed improvements of parallel downloads will be. Depending on your latency to the package server and the number of packages being downloaded, parallel downloads can cut the download process time in half, approximately.
Previously, when conda had to install multiple packages, conda would assess and extract each package sequentially. In other words, conda would download packages one by one. Now, conda will utilize up to five parallel download threads to download multiple packages simultaneously. See parallel download in action.
To demonstrate only the speed improvement of the download process, the performance test compares the installation of a conda .lock file on the same machine (a .lock file contains package, environment, and platform specifics). This performance test demonstrates a ~38% speed improvement over the deprecated download process (5.8 seconds vs. 3.6 seconds).
If you take a closer look at comparing conda’s behavior during the download process, you’ll notice that instead of a successive, top-down cascade of package downloads, it now looks more like a left-to-right shotgun blast.
New plugin architecture, who this?
We are thrilled to announce that the plugin architecture that brought the experimental conda-libmamba-solver to your local conda environment is now fully operationalized. This new plugin architecture for conda is a substantial change and we are hopeful that it will serve as a major addition to the conda community and ecosystem.
Conda’s new plugin architecture allows users to contribute to conda with a significantly lower barrier to entry. We’ve made many aspects of conda (such as the solver, virtual packages, and custom subcommands) to be plugin compatible by refactoring many sizable chunks of conda’s codebase to include “hooks.” Users can now create entirely custom plugins to the conda ecosystem to fit their specific needs without having to go through the entire peer review process.
Additionally, because of conda’s language and platform agnosticism, conda can now more easily fold in programmatic solutions written in any language for any platform.
Other Notable Changes
-
Environment variable LD_PRELOAD is now reported with ‘conda info –all’.
-
The –experimental-solver flag and the conda_experimental_solver environment variable are now pending deprecation, to be replaced by –solver and SolverChoice, respectively.
-
Removed reliance on the toolz library: Both toolz.functoolz.excepts and toolz.itertoolz.concatv have been removed.
Read the full release notes here.
Looking Forward
We are deeply passionate about making conda work for everyone—changes and improvements made to the conda project will always be oriented and committed to making conda more performant, flexible, and extensible. Stay tuned for a comprehensive breakdown of the new plugin architecture, its implications for the conda ecosystem, and a completely-written-in-Rust plugin showcase from the conda team.
We would also like to thank all the contributors (including nine that contributed for the first time!) that contributed to this conda release (these are GitHub handles):
@arq0017 made their first contribution in #11810
@beeankha
@conda-bot
@dbast
@dholth
@erykoff
@consideRatio made their first contribution in #12028
@jaimergp
@jezdez
@kathatherine
@kenodegard
@ForgottenProgramme made their first contribution in #11926
@hmaarrfk made their first contribution in #9946
@NikhilRaverkar made their first contribution in #11842
@pavelzw made their first contribution in #11849
@pkmooreanaconda made their first contribution in #12014
@fragmede made their first contribution in #11818
@SatyamVyas04 made their first contribution in #11870
@timhoffm
@travishathaway
@dependabot made their first contribution in #11965
@pre-commit-ci[bot]
