New: Faster Cloud Compute

April 11, 2025

Most cloud compute environments today use a generic stack – standard container images, off-the-shelf registries, and unoptimized Kubernetes setups – leading to suboptimal performance for ML/AI workloads. In this post, we share benchmarks from our latest optimizations that cut task startup latency in your cloud account by 4x or more.

To illustrate a typical use case – running a model built with PyTorch and comparing its startup time across three different environments:

  1. In the first case, we use Outerbounds’ automatic containerization mechanism, Fast Bakery, to bake a custom image including PyTorch, optimized for the workload at hand. The task completes in 23 seconds.
  2. In the second case, we use Outerbounds’ optimized compute pools to pull a large, off-the-shelf image from ECR which includes PyTorch and other ML/AI libraries. The task completes in 33 seconds.
  3. The last case represents the baseline of using open-source Metaflow with AWS Batch. The task completes in 83 seconds.

Crucially, all workloads run entirely within your own cloud account, ensuring that no data or compute resources ever leave your environment.

Running cloud tasks fast (without idling instances)

We want to make tasks executing in the cloud feel as responsive as running code on your laptop or in a local notebook. Since Outerbounds is always deployed within your own cloud environment, maintaining pools of pre-warmed instances – while effective at reducing latency – is often impractical due to significantly increased cloud costs.

To optimize the system end-to-end, it is beneficial to look at the big picture. As of today, practically all container-based compute environments, such as AWS Batch and Kubernetes, involve three core components – an image builder, a container registry, and a container runtime – which prepare and serve standard container images for execution. These components are typically operated as independent services that you can mix and match, depicted as gray boxes below. While flexible, this makes it harder to optimize performance across the full pipeline.

Fast container pipeline

Let’s walk through the container lifecycle:

  1. First, your task and its execution environment need to be packaged as a container image. This image building step is typically handled through a CI/CD pipeline. Outerbounds integrates the step seamlessly into the platform with Fast Bakery, so developers don’t have to interact with separate systems to execute their code in the cloud.
  2. The resulting image is pushed to a container registry, either to a cloud-provided one such as Amazon ECR, or a separate service such as Docker Hub. Outerbounds ships with a built-in Fast Registry that is tightly integrated with our new high-throughput container runtime.
  3. The image is pulled for execution in a compute environment such as Kubernetes or AWS Batch. Outerbounds introduces its own latency optimizations – such as our new Fast Container Runtime.

New: Fast Container Runtime

We had started our optimization efforts with Fast Bakery that makes it trivial for developers to build and push images to a container registry on the fly – without having to write Dockerfiles or much think about it at all. It is magical to see giant images – often 10GB or larger – bake in seconds.

As our optimization efforts progressed, it became clear that the container runtime had emerged as a major remaining bottleneck. Our new Fast Container Runtime replaces containerd. The new runtime is fully transparent by design – you can use it with any existing container image, from any registry, without changing your workloads.

How fast is Fast Container Runtime?

Let’s take a look at the impact of the new container runtime in real-world workloads. We benchmark four typical ML/AI workloads:

  1. AWS Deep Learning Image (9.93GB)
  2. The official PyTorch image (3.1GB)
  3. A minimal PyTorch environment created on the fly with Metaflow’s @pypi, including numpy and pytorch (2.7GB)
  4. A typical ML environment created on the fly with @pypi, including altair, xgboost, and scikit-learn (0.65GB)

Task startup times chart

Across all workloads, the typical speedup on Outerbounds is 4x-10x, powered by Fast Container Runtime. The relative speedup is highest – about 10x – with the smallest image. Consider a nightly flow running 2000 small tasks like this – a typical Metaflow workload. Shaving off 30 seconds from each task saves up to 500 instance-hours monthly, or $4,500 annually on tasks occupying an m5.4xlarge.

Much time is wasted in downloading images

A key culprit for slowness on standard setups, e.g. on vanilla EKS, are container registries and containerd, which are not optimized for maximum throughput. Below, we compare three images hosted on ECR, DockerHub, or Outerbounds:

Registry latency comparison

In our earlier Fast Bakery article, we highlighted how traditional registries can be sluggish when uploading images. This benchmark completes the other side of the equation: they are also slow when it comes to downloading images and setting up a container runtime, which is arguably an even bigger issue, since the cost compounds with every new instance.

Outerbounds has the advantage of managing the entire task lifecycle – from image building to execution. This end-to-end control has allowed us to systematically eliminate bottlenecks, accelerating both existing images and newly baked environments. And we’re not done yet – stay tuned for more optimizations.

Seeing is believing

Don’t take our word for it. You can deploy Outerbounds in your cloud account with a few clicks (with a 14 day free trial), so you can test performance with your own images, packages, and workloads.