Agentic Metaflow

There’s no shortage of hype around agents. If you’ve read too many articles like that, don’t worry – this article won’t add hot air. More interestingly, the just released Metaflow 2.18 comes with a major new feature, the long-awaited support for conditional and recursive steps, which come in handy when building agentic applications.

Before diving into the details and a fun, illustrative example (with complete, try-it-at-home source code), let’s start with a quick overview of how agentic systems are built – and how Metaflow fits into the picture.

Building agentic systems with Metaflow

When talking about agents, the discussion tends to focus on agent frameworks, of which there are plenty – LangChain, AutoGen by Microsoft, Strands by AWS, Agent Development Kit by Google, OpenAI Agent SDK, amongst others. The primary function of these frameworks is to orchestrate the agent loop: accepting prompts, invoking LLMs, interpreting their outputs, and executing external tools whose results are fed back into the model. On top of this core loop, frameworks add guardrails, tracing, and support for evaluations.

A key insight is that while an agent framework may be necessary, it is not sufficient for building a complete agentic system. This diagram illustrates the bigger picture:

All the agent frameworks need to be deployed and executed in an agent environment. In the simplest case, execution could happen on your laptop but serious systems are more likely to run in a cloud environment such as Kubernetes. Notably, many agentic applications rely on entire auto-scaling fleets of agents, adding to requirements.

Besides an execution environment, agents require mechanisms for state and memory management. As with any deployed container, agents may crash or be interrupted. This is especially necessary for agents that perform long-running, compute-intensive tasks. Durable state enables them to recover from failures and continue execution from the last checkpoint.

Agents interact with the outside world by nature. To do so, they require access to LLMs, frequently more than one, allowing them to route requests to the most suitable model. For instance, Claude Code uses smaller models eagerly for practical tasks for which a large frontier model would be overkill. To reduce the impact of rate limits and costs, and to support domain-specific fine-tuning, agents may leverage open-weight models alongside proprietary ones.

Also, a defining characteristic of agents is tool use: they can invoke APIs, query databases, and interact with microservices through the Model Context Protocol (MCP). This, in turn, adds a layer of complexity as agents mustn’t compromise the stability and security of external APIs and databases. And crucially, agents cannot produce reliable, non-hallucinated results without access to accurate, domain-specific data.

Enter Metaflow

To summarize, a production-ready agentic system requires ways to

• Deploy agents in an auto-scaling cloud environment.
• Store agent state and recover from failures.
• Provide access to LLMs, including custom, domain-specific ones.
• Provide a fleet of stable and secure services, exposed via MCP.
• Expose agents to accurate data that’s often continuously updating.

As illustrated above, Metaflow serves as the baseplate for agentic systems, helping you address all of the above requirements. It does not replace or compete with agent frameworks, but it complements them by delivering foundational services – data, compute, orchestration, and versioning – which are needed to complete the system around the core agent loop.

Metaflow shines when powering autonomous agents, that is, scenarios where fleets of agents operate without continuous human oversight. Beyond chatbots or support assistants, think of agents built for deep research tasks, such as analyzing a decade of S-1 filings or detecting fraud across financial transactions. Such workloads often involve large numbers of long-running agents, massive datasets, and significant compute requirements – precisely the challenges Metaflow is designed to address.

New in Metaflow: Recursive and conditional steps

Effective state management is a critical requirement for long-running agents. Metaflow addresses this with efficient snapshotting of workflow state through artifacts. At Netflix alone, Metaflow manages more than 12PB of artifacts with no limits in sight, making it hard to imagine a more proven foundation for agent state management.

Unlike typical ML and data DAGs, agents are recursive by nature: Agent frameworks keep running them in a loop until some stopping condition is hit. To make artifact-based state management work for this pattern, we added support for recursive steps in Metaflow, now available in 2.18. And since loops need an exit, the new release also comes with conditional step transitions. Besides agentic use cases, recursion and conditionals have been long-awaited features which have many uses in traditional workflows as well.

With Metaflow 2.18, you are now able to construct flows where, within a foreach, the think step repeatedly calls itself until a stopping condition is met. Once that condition is satisfied, the flow transitions either to a done step or a fail step, wrapping up the processing. Naturally this is just an example flow – you can use conditionals and recursion almost anywhere in your flows, and nest them freely.

While recursion and conditionals introduce additional elements of runtime dynamism in Metaflow, the intent is to keep flows understandable and predictable, rather than fully emergent at runtime. With agentic systems in particular, which are inherently stochastic, it is crucial that the system remains debuggable and maintainable.

Switching @steps

As always in Metaflow, the syntax is straightforward: Instead of specifying a single step transition in self.next, you specify a dictionary of options – essentially a switch statement – keyed by a condition artifact.

This snippet shows a typical integration between an agent_framework – pick your favorite – and Metaflow. The framework runs the agent as usual, with its native session and memory management keeping the conversation state in a Python object. Periodically, the loop exits, for example, after reaching a maximum number of turns or a time limit, and the session state is returned and stored as a Metaflow artifact. When the task resumes, the agent continues from the previously stored state which Metaflow manages safely (and inexpensively!) in a cloud-native object storage.

While this looks deceptively simple – by design – the power of this setup comes from the fact that you can enrich the agent_loop with all the functionality of Metaflow, including managing and versioning prompts, handling failures with @retry, adding real-time observability with @card, containerizing your agent code, executing any number of agents in parallel, triggering agents in real-time with events, and provisioning ephemeral, sandboxed services and MCP servers on the compute layer.

Crucially, Metaflow lets you develop and test everything locally, offering a smoother and faster development experience than most cloud-based deployment options.

Example: Autonomous deep research agents

Let’s walk though a complete and realistic system demonstrating the power of autonomous agents, running on Metaflow. Find the source code here.

We want to conduct deep research on a topic, in this case, compiling biographies of music artists. To ensure accuracy, we can’t just ask an LLM to spin up imaginative life stories. Instead, we equip an agent with the complete MusicBrainz database, consisting of over 200M rows of facts about artists and releases. The agent is to use the database to retrieve facts about releases as well as validate results obtained from web searches, which it will use autonomously to compile a timeline of life events for an artist.