> ## Documentation Index
> Fetch the complete documentation index at: https://anaconda.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Structuring projects
Packaging code and its dependencies for reproducible execution can be [a complex task](https://xkcd.com/1987/). Metaflow and Anaconda Platform simplify this process. By following the best practices established on this page, you can ensure your projects are reproducible, ready for production, and structured for rapid iteration during development.
## Packaging software
A typical Metaflow project consists of five layers of software:
1. A Metaflow flow defined in a Python file.
2. Custom Python modules and packages that contain the project-specific code.
3. The Metaflow library itself and any installed Metaflow extensions.
4. Third-party libraries and frameworks used by the project.
5. The underlying operating system and hardware drivers, especially CUDA for GPUs.
Metaflow automatically packages layers 1 through 3, the parts that change most frequently during development. Layers 4 and 5, external libraries and low-level concerns such as device drivers, are managed separately. For details, see [Managing dependencies](/docs/platform/guides/compute/managing-dependencies).
## Structuring a project
To demonstrate a typical project structure, the following example creates a flow that visualizes a fractal using two off-the-shelf libraries, `pyfracgen` and `matplotlib`.
Follow Python best practices when designing your project: use [Python modules and packages](https://realpython.com/python-modules-packages/) to modularize your code into logical components.
For instance, it makes sense to create a dedicated module for all the logic related to fractal generation. Save the following in a file named `myfractal.py`:
```python theme={null}
def make_fractal():
import pyfracgen as pf
from matplotlib import pyplot as plt
string = "AAAAAABBBBBB"
xbound = (2.5, 3.4)
ybound = (3.4, 4.0)
res = pf.lyapunov(
string, xbound, ybound, width=4, height=3, dpi=300, ninit=2000, niter=2000
)
pf.images.markus_lyapunov_image(res, plt.cm.bone, plt.cm.bone_r, gammas=(8, 1))
return plt.gcf()
```
### Why separate modules and packages?
Creating a separate module, or a package composed of multiple modules, offers several benefits:
* You can develop each module independently. For example, you can test `make_fractal` in a notebook by adding a cell:
```python theme={null}
import myfractal
myfractal.make_fractal()
```
* You can use standard Python testing tools, such as [`pytest`](https://docs.pytest.org/en/stable/), to unit test the module.
* You can share the module between multiple flows and other systems, encouraging reusability and consistent business logic across projects.
### Using a custom module in a flow
Save this flow in `fractalflow.py`, in the same directory as `myfractal.py`:
```python highlight={9,13-14} expandable theme={null}
from metaflow import FlowSpec, card, pypi, step, current
from metaflow.cards import Image
class FractalFlow(FlowSpec):
@step
def start(self):
self.next(self.plot)
@pypi(python="3.11.9", packages={"pyfracgen": "0.0.11", "matplotlib": "3.9.0"})
@card(type="blank")
@step
def plot(self):
import myfractal
img = myfractal.make_fractal()
current.card.append(Image.from_matplotlib(img))
self.next(self.end)
@step
def end(self):
pass
if __name__ == "__main__":
FractalFlow()
```
The highlighted lines show the two things that make this work:
* **The `@pypi` decorator** declares the environment for the `plot` step: the Python version and the packages it needs.
* **The `myfractal` import** works because Metaflow packages the flow file and everything in its directory (and subdirectories) automatically.
For details about this packaging logic, see [Structuring projects in the Metaflow documentation](https://docs.metaflow.org/scaling/dependencies/project-structure).
To see exactly which files Metaflow includes in its [code package](https://docs.metaflow.org/api/client#MetaflowCode), run:
```sh theme={null}
python fractalflow.py --environment=pypi package list
```
To guarantee consistent execution across environments, Metaflow includes the `metaflow` library itself and all installed extensions.
### Including libraries and frameworks
The `myfractal.py` module only works if it can import the `pyfracgen` and `matplotlib` packages. You could install them manually with `pip install pyfracgen matplotlib`, but that approach has problems:
* Nothing in the code records which packages it needs. A colleague, or you on a new laptop, cannot reproduce the project without outside knowledge.
* Cloud executions cannot rely on your local installation. Each run needs its own environment.
* Production deployments are exposed to upstream changes. A new `matplotlib` release can break the code at any time.
The `@pypi` decorator on the `plot` step addresses all three: the dependencies are declared in the code, built into an isolated environment for every run, and pinned to specific versions. For the full range of dependency management options, see [Managing dependencies](/docs/platform/guides/compute/managing-dependencies).
To run the flow locally:
```sh theme={null}
python fractalflow.py --environment=pypi run
```
The first run takes a few minutes while Metaflow builds and caches the environment. When the run completes, open it in the platform's Runs view to see the fractal rendered on the `@card`.
To run the same flow on cloud compute, no code changes are needed:
```sh theme={null}
python fractalflow.py --environment=pypi run --with kubernetes
```
The `@pypi` decorator does not `pip install` the packages individually at run time. It creates and caches [a stable, production-ready execution environment](https://docs.metaflow.org/scaling/dependencies/internals), isolated from any changes in the upstream libraries.