Modal Serverless GPU Guide: Run ML Workloads Without Infrastructure

Modal is a serverless GPU cloud that lets you run machine learning workloads without provisioning or managing infrastructure. You define your compute in Python, and Modal handles the rest: spinning up GPUs, scaling to zero when idle, and giving you a pay-per-second bill. This guide walks through the core concepts, from a first hello-world to deploying inference endpoints, and covers the practical details like GPU selection, container images, and persistent storage.

What it does

Modal turns your Python functions into serverless, GPU-accelerated tasks. Instead of renting a VM and babysitting it, you write a function, decorate it with the compute specs you need, and call it. Modal takes care of containerization, scheduling, and autoscaling. It is a good fit for GPU-intensive jobs that are bursty or intermittent, because you only pay for the seconds you actually use. It also shines for deploying models as auto-scaling APIs, running batch processing, and prototyping quickly. The platform is Python-native: you define your infrastructure in code, not in YAML, which keeps everything in one place and makes it version-controllable.

Before you start

You need Python installed on your machine. The Modal client is a pip package, and authentication happens through a browser-based flow. The skill is marked as optional and installed on demand, so you will need to install it into your Hermes Agent environment first. It supports Linux, macOS, and Windows, so most development machines are covered. You will also need a Modal account, which you create through the setup process.

Quick start

Installation

Install the client and authenticate. The setup command opens a browser for you to log in and link your account.

pip install modal
modal setup  # Opens browser for authentication

Hello World with GPU

This first script defines a Modal app, attaches a T4 GPU to a function, and runs nvidia-smi inside the container to confirm the GPU is visible. The @app.local_entrypoint() marks the function that runs when you execute the script locally, and it calls the remote function with .remote().

import modal

app = modal.App("hello-gpu")

@app.function(gpu="T4")
def gpu_info():
    import subprocess
    return subprocess.run(["nvidia-smi"], capture_output=True, text=True).stdout

@app.local_entrypoint()
def main():
    print(gpu_info.remote())

Run: modal run hello_gpu.py

You should see the output of nvidia-smi printed to your terminal, confirming that the T4 is available inside the container.

Basic inference endpoint

This example shows a class-based function that loads a model once at container start and then serves predictions. The @modal.enter() decorator runs the load_model method when the container starts, so the model is loaded into memory before any requests arrive. The @modal.method() decorator exposes generate as a callable remote method.

import modal

app = modal.App("text-generation")
image = modal.Image.debian_slim().pip_install("transformers", "torch", "accelerate")

@app.cls(gpu="A10G", image=image)
class TextGenerator:
    @modal.enter()
    def load_model(self):
        from transformers import pipeline
        self.pipe = pipeline("text-generation", model="gpt2", device=0)

    @modal.method()
    def generate(self, prompt: str) -> str:
        return self.pipe(prompt, max_length=100)[0]["generated_text"]

@app.local_entrypoint()
def main():
    print(TextGenerator().generate.remote("Hello, world"))

Core concepts

Key components

Component	Purpose
`App`	Container for functions and resources
`Function`	Serverless function with compute specs
`Cls`	Class-based functions with lifecycle hooks
`Image`	Container image definition
`Volume`	Persistent storage for models/data
`Secret`	Secure credential storage

Execution modes

Command	Description
`modal run script.py`	Execute and exit
`modal serve script.py`	Development with live reload
`modal deploy script.py`	Persistent cloud deployment

GPU configuration

Available GPUs

GPU	VRAM	Best For
`T4`	16GB	Budget inference, small models
`L4`	24GB	Inference, Ada Lovelace arch
`A10G`	24GB	Training/inference, 3.3x faster than T4
`L40S`	48GB	Recommended for inference (best cost/perf)
`A100-40GB`	40GB	Large model training
`A100-80GB`	80GB	Very large models
`H100`	80GB	Fastest, FP8 + Transformer Engine
`H200`	141GB	Auto-upgrade from H100, 4.8TB/s bandwidth
`B200`	Latest	Blackwell architecture

GPU specification patterns

You can specify a single GPU, a specific memory variant, multiple GPUs with a colon syntax, a list of fallbacks, or let Modal pick any available GPU.

# Single GPU
@app.function(gpu="A100")

# Specific memory variant
@app.function(gpu="A100-80GB")

# Multiple GPUs (up to 8)
@app.function(gpu="H100:4")

# GPU with fallbacks
@app.function(gpu=["H100", "A100", "L40S"])

# Any available GPU
@app.function(gpu="any")

Container images

You define the container image for your function. The examples show a basic Debian image with pip packages, a CUDA base image from a registry, and a Debian image with system packages installed via apt.

# Basic image with pip
image = modal.Image.debian_slim(python_version="3.11").pip_install(
    "torch==2.1.0", "transformers==4.36.0", "accelerate"
)

# From CUDA base
image = modal.Image.from_registry(
    "nvidia/cuda:12.1.0-cudnn8-devel-ubuntu22.04",
    add_python="3.11"
).pip_install("torch", "transformers")

# With system packages
image = modal.Image.debian_slim().apt_install("git", "ffmpeg").pip_install("whisper")

Persistent storage

Modal Volumes provide persistent storage that survives container restarts. This example mounts a volume at /models, checks if a model already exists, downloads it if not, and commits the changes so they persist.

volume = modal.Volume.from_name("model-cache", create_if_missing=True)

@app.function(gpu="A10G", volumes={"/models": volume})
def load_model():
    import os
    model_path = "/models/llama-7b"
    if not os.path.exists(model_path):
        model = download_model()
        model.save_pretrained(model_path)
        volume.commit()  # Persist changes
    return load_from_path(model_path)

Web endpoints

FastAPI endpoint decorator

Turn a single function into a REST API endpoint with @modal.fastapi_endpoint().

@app.function()
@modal.fastapi_endpoint(method="POST")
def predict(text: str) -> dict:
    return {"result": model.predict(text)}

Full ASGI app

For more complex applications, you can wrap a full FastAPI app and serve it through Modal.

from fastapi import FastAPI
web_app = FastAPI()

@web_app.post("/predict")
async def predict(text: str):
    return {"result": await model.predict.remote.aio(text)}

@app.function()
@modal.asgi_app()
def fastapi_app():
    return web_app

Web endpoint types

Decorator	Use Case
`@modal.fastapi_endpoint()`	Simple function → API
`@modal.asgi_app()`	Full FastAPI/Starlette apps
`@modal.wsgi_app()`	Django/Flask apps
`@modal.web_server(port)`	Arbitrary HTTP servers

Dynamic batching

If you have many small requests, dynamic batching can group them into larger batches to improve throughput. The @modal.batched decorator collects inputs for up to wait_ms milliseconds or until max_batch_size is reached, then calls the function once with the whole list.

@app.function()
@modal.batched(max_batch_size=32, wait_ms=100)
async def batch_predict(inputs: list[str]) -> list[dict]:
    # Inputs automatically batched
    return model.batch_predict(inputs)

Secrets management

Store sensitive values like API tokens as Modal Secrets, then reference them in your functions. The secret name is used to inject environment variables.

# Create secret
modal secret create huggingface HF_TOKEN=hf_xxx

@app.function(secrets=[modal.Secret.from_name("huggingface")])
def download_model():
    import os
    token = os.environ["HF_TOKEN"]

Scheduling

Run functions on a cron schedule or at a fixed interval.

@app.function(schedule=modal.Cron("0 0 * * *"))  # Daily midnight
def daily_job():
    pass

@app.function(schedule=modal.Period(hours=1))
def hourly_job():
    pass

Performance optimization

Cold start mitigation

Cold starts happen when a container spins up from scratch. You can keep containers warm with scaledown_window and allow them to handle multiple concurrent requests with the @modal.concurrent decorator. Note that in Modal 1.0, the autoscaler parameters were renamed; the comment in the code block reflects that.

# Modal 1.0 autoscaler params: scaledown_window (was container_idle_timeout).
# Input concurrency moved to the @modal.concurrent decorator.
@app.function(scaledown_window=300)  # Keep warm 5 min
@modal.concurrent(max_inputs=10)     # Handle concurrent requests per container
def inference():
    pass

Model loading best practices

Load heavy models once at container start using @modal.enter() rather than on every request. This moves the cost to the warm-up phase and keeps inference fast.

@app.cls(gpu="A100")
class Model:
    @modal.enter()  # Run once at container start
    def load(self):
        self.model = load_model()  # Load during warm-up

    @modal.method()
    def predict(self, x):
        return self.model(x)

Parallel processing

Fan out work across many containers with the .map() method. This example processes 1000 items in parallel, with Modal distributing the work across containers.

@app.function()
def process_item(item):
    return expensive_computation(item)

@app.function()
def run_parallel():
    items = list(range(1000))
    # Fan out to parallel containers
    results = list(process_item.map(items))
    return results

Common configuration

This block shows the most common function parameters: memory, CPU, timeout, scaling, and retries. The comments note the Modal 1.0 renames for the autoscaler parameters.

@app.function(
    gpu="A100",
    memory=32768,              # 32GB RAM
    cpu=4,                     # 4 CPU cores
    timeout=3600,              # 1 hour max
    scaledown_window=120,      # Keep warm 2 min (was container_idle_timeout)
    retries=3,                 # Retry on failure
    max_containers=10,         # Max concurrent containers (was concurrency_limit)
    min_containers=1,          # Keep N containers warm (was keep_warm)
)
def my_function():
    pass

Modal 1.0 autoscaler renames (see the migration guide):

container_idle_timeout → scaledown_window

concurrency_limit → max_containers

keep_warm → min_containers

allow_concurrent_inputs=N → the @modal.concurrent(max_inputs=N) decorator

Debugging

You can test a function locally with .local() instead of .remote(). For logs, use the modal app logs command.

# Test locally
if __name__ == "__main__":
    result = my_function.local()

# View logs
# modal app logs my-app

Common issues

Issue	Solution
Cold start latency	Increase `scaledown_window`, use `@modal.enter()`
GPU OOM	Use larger GPU (`A100-80GB`), enable gradient checkpointing
Image build fails	Pin dependency versions, check CUDA compatibility
Timeout errors	Increase `timeout`, add checkpointing

When not to use it

Modal is not the right fit for every workload. If you need long-running pods with persistent state, consider RunPod. For reserved GPU instances, Lambda Labs is a better option. If you need multi-cloud orchestration and cost optimization, SkyPilot handles that. And for complex multi-service architectures, a Kubernetes cluster gives you more control. Modal shines when you want to avoid infrastructure management and pay only for what you use, but if you need persistent, always-on resources or fine-grained control over the underlying infrastructure, those alternatives are worth evaluating.

Limits and gotchas

Cold starts: Even with optimizations, a cold start can add latency. Use scaledown_window and @modal.enter() to mitigate.
GPU memory: If you run out of GPU memory, you need to move to a larger GPU or enable gradient checkpointing. There is no automatic spillover.
Image builds: Builds can fail if you do not pin dependency versions or if there is a CUDA compatibility mismatch. Always pin versions and check the base image.
Timeouts: Functions have a maximum timeout. If your job runs longer, you need to increase the timeout or add checkpointing to resume.
Autoscaler renames: If you are migrating from an older Modal version, be aware that container_idle_timeout, concurrency_limit, keep_warm, and allow_concurrent_inputs have been renamed. The migration guide linked above has the full details.

References and further reading

For deeper dives, the skill includes two reference documents: an advanced usage guide covering multi-GPU, distributed training, and cost optimization, and a troubleshooting guide for common issues. The official Modal documentation, examples repository, pricing page, and Discord community are all linked from the source page and are good next stops.

Advanced Usage - Multi-GPU, distributed training, cost optimization
Troubleshooting - Common issues and solutions

Resources

Documentation: https://modal.com/docs
Examples: https://github.com/modal-labs/modal-examples
Pricing: https://modal.com/pricing
Discord: https://discord.gg/modal

Modal Serverless GPU: Run ML Workloads Without Managing Infrastructure