What is a workflow?

​

How a workflow is structured

from metaflow import FlowSpec, step

class MyFlow(FlowSpec):

    @step
    def start(self):
        self.data = load_something()
        self.next(self.process)

    @step
    def process(self):
        self.result = transform(self.data)
        self.next(self.end)

    @step
    def end(self):
        pass

if __name__ == "__main__":
    MyFlow()

class MyFlow(FlowSpec):

    @step(start=True)
    def begin(self):
        self.next(self.process)

    @step
    def process(self):
        self.next(self.finish)

    @step(end=True)
    def finish(self):
        pass

​

What makes workflows different from scripts

• Each step is isolated. Steps can run on different machines with different resources. A preprocessing step might run on a small CPU instance while a training step runs on a GPU node.
• Data flows between steps automatically. Values you assign to self in one step are available in all downstream steps, even across machines. The platform serializes, stores, and retrieves them transparently.
• Every run is versioned. The platform records every run’s code, data, and results. You can inspect, compare, and reproduce past runs without manual bookkeeping.
• Failures are recoverable. If a step fails, you can resume the flow from the point of failure without re-running the steps that already succeeded.

​

Workflows vs. deployments

• A workflow runs to completion. It starts, executes its steps, and finishes. Use workflows for training models, processing data, running evaluations, and any work with a defined end.
• A deployment stays running. It serves requests continuously until you stop it. Use deployments for model endpoints, APIs, dashboards, and services that need to be available on demand.

​

Building workflows