Compute Anything with Metaflow
November 6, 2023
Today, we are releasing a major new capability in Metaflow: You can now use Metaflow for distributed high-performance computing in general, large-scale training for AI in particular. We do this by providing infrastructure primitives for easy to use, cost-efficient compute in various clouds, which we have integrated to popular frameworks like PyTorch, Ray, Tensorflow, Deepspeed, and MPI.
As every leader of a data organization knows, data sitting in a data warehouse or a lake has rather limited value. Data becomes valuable when you do something with it: Transform, query, analyze, and expose it to power products and processes.
The Next Era of Data and Compute
With the advent of large language models, for the first time in decades, we have a whole new paradigm for leveraging data. A downside is that ML, and the latest AI techniques in particular, are extremely compute-hungry.
Compute Everything with Metaflow
Since its inception, a killer feature of Metaflow has been its straightforward attitude towards compute. It does not require you to learn a new paradigm like Hadoop or Spark. Rather, it makes it trivially easy to run Python functions, first locally on your laptop, and then in the cloud without fuss.
As a result, you can architect and operate a new class of compute and data-intensive applications leveraging the massive Python ecosystem of AI, ML, and data tools without headaches. You can stay focused on data, models, and business logic, avoiding finicky abstractions and constrained environments.
New in Metaflow: Compute clusters on the fly
Autodesk is a great example of a company that sees massive opportunities in applying generative AI techniques in real-world problems at a very large scale. We collaborated with them to enable support for a new class of compute in Metaflow: Distributed compute involving interconnected tasks, based on an ephemeral, tightly connected compute cluster, created on the fly.
Today, we release new Metaflow extensions for Ray, Deepspeed, PyTorch, Tensorflow, and MPI, so data scientists and ML engineers can use their favorite tools for the job without having to focus on infrastructure.
Freedom to compute
Due to the heterogeneous nature of the next era of compute, sometimes it is cost-efficient to leverage on-premise compute resources, sometimes Azure, AWS or GCP, and sometimes specialized GPU clouds like CoreWeave. One size does not fit all. Importantly, flexibility does not mean that you have to compromise user experience or enterprise concerns of governance and security.
Get started right away
- For more technical details about the new integrations for distributed compute, see the second part of this announcement.
- If you want to start building production-ready ML and AI systems quickly with the freedom to compute without markup, get started with Outerbounds for free!