Announcing General Availability of conda 4.3

 

We’re excited to announce that conda 4.3 has been released for general availability. The 4.3 release series has several new features and substantial improvements. Below is a summary. 

To get the latest, just run

conda update conda

.

New Features

  • Unlink and Link Packages in a Single Transaction: In the past, conda hasn’t always been safe and defensive with its disk-mutating actions. It has gleefully clobbered existing files; mid-operation failures left environments completely broken. In some of the most severe examples, conda can appear to “uninstall itself.” With this release, the unlinking and linking of packages for an executed command is done in a single transaction. If a failure occurs for any reason while conda is mutating files on disk, the environment will be returned its previous state. While we’ve implemented some pre-transaction checks (verifying package integrity for example), it’s impossible to anticipate every failure mechanism. In some circumstances, OS file permissions cannot be fully known until an operation is attempted and fails, and conda itself is not without bugs. Moving forward, unforeseeable failures won’t be catastrophic.

  • Progressive Fetch and Extract Transactions: Like package unlinking and linking, the download and extract phases of package handling have also been given transaction-like behavior. The distinction is that the rollback on error is limited to a single package. Rather than rolling back the download and extract operation for all packages, the single-package rollback prevents the need for having to re-download every package if an error is encountered.

  • Generic- and Python-Type Noarch/Universal Packages: Along with conda-build 2.1, a noarch/universal type for Python packages is officially supported. These are much like universal Python wheels. Files in a Python noarch package are linked into a prefix just like any other conda package, with the following additional features:

    1. conda maps the
      site-packages

      directory to the correct location for the Python version in the environment,

    2. conda maps the
      python-scripts

      directory to either

      /bin

      or

      /Scripts

      depending on platform,

    3. conda creates the Python entry points specified in the conda-build recipe, and
    4. conda compiles pyc files at install time when prefix write permissions are guaranteed.

    Python noarch packages must be “fully universal.” They cannot have OS- or Python version-specific dependencies. They cannot have OS- or Python version-specific “scripts” files. If these features are needed, traditional conda packages must be used.

  • Multi-User Package Caches: While the on-disk package cache structure has been preserved, the core logic implementing package cache handling has had a complete overhaul. Writable and read-only package caches are fully supported.

  • Python API Module: An oft requested feature is the ability to use conda as a Python library, obviating the need to “shell out” to another Python process. Conda 4.3 includes a

    conda.cli.python_api

    module that facilitates this use case. While we maintain the user-facing command-line interface, conda commands can be executed in-process. There is also a

    conda.exports

    module to facilitate longer-term usage of conda as a library across conda releases. However, conda’s Python code is considered internal and private, subject to change at any time across releases. At the moment, conda will not install itself into environments other than its original install environment.

  • Remove All Locks: Locking has never been fully effective in conda, and it often created a false sense of security. In this release, multi-user package cache support has been implemented for improved safety by hard-linking packages in read-only caches to the user’s primary user package cache. Still, users are cautioned that undefined behavior can result when conda is running in multiple process and operating on the same package caches and/or environments.

Deprecations/Breaking Changes

  • Conda now has the ability to refuse to clobber existing files that are not within the unlink instructions of the transaction. This behavior is configurable via the

    path_conflict

    configuration option, which has three possible values:

    clobber

    ,

    warn

    , and

    prevent

    . In 4.3, the default value is 

    clobber

    . This preserves existing behaviors, and it gives package maintainers time to correct current incompatibilities within their package ecosystem. In 4.4, the default will switch to

    warn

    , which means these operations continue to clobber, but the warning messages are displayed. In 4.5, the default value will switch to

    prevent

    . As we tighten up the

    path_conflict

    constraint, a new command line flag

    --clobber

    will loosen it back up on an ad hoc basis. Using

    --clobber

    overrides the setting for

    path_conflict

    to effectively be

    clobber

    for that operation.

  • Conda signed packages have been removed in 4.3. Vulnerabilities existed, and an illusion of security is worse than not having the feature at all. We will be incorporating The Update Framework (TUF) into conda in a future feature release.

  • Conda 4.4 will drop support for older versions of conda-build.

Other Notable Improvements

  • A new “trace” log level is added, with output that is extremely verbose. To enable it, use

    -v -v -v

    or

    -vvv

    as a command-line flag, set a

    verbose: 3

    configuration parameter, or set a

    CONDA_VERBOSE=3

    environment variable.

  • The

    r

    channel is now part of the default channels.

  • Package resolution/solver hints have been improved with better messaging.


You May Also Like

Data Science Blog
Database Trends & Applications: Machine Learning and Data Science are Top Trends at Strata Data
Data professionals and vendors converged at Strata Data in New York to trade tips and tricks for handling big data. Top of mind for most was the impact of machine learning and...
Read More
Company Blog
Two Peas in a Pod: Anaconda + IBM Cognitive Systems
This week we are pleased to announce the availability of Anaconda on IBM’s Cognitive Systems, the company’s high performance deep learning platform, highlighting the fact ...
Read More
Company Blog
Introducing Anaconda Enterprise 5
Anaconda Enterprise 5 is built for your entire organization and serves the needs of Data Scientists, IT Administrators, Business Analysts & Managers, Developers and Machine Le...
Read More