Neura Market Logo
OptionalMLOpsVersion 1.0.1

HuggingFace Accelerate: Unified Distributed Training with Hermes Agent

Run PyTorch training across GPUs with minimal changes.

Written by Neura Market from the official Hermes Agent documentation for Huggingface Accelerate. Commands, paths, and version numbers are reproduced from the source unchanged.

Read the official documentation

HuggingFace Accelerate is a lightweight library that wraps PyTorch's distributed training primitives behind a single, consistent API. If you have a PyTorch training script that runs on one GPU and you want to run it on multiple GPUs, across nodes, with mixed precision, or with advanced sharding strategies like DeepSpeed ZeRO or FSDP, Accelerate lets you do it by adding four lines of code and changing one more. It is the recommended distributed training layer for the HuggingFace ecosystem and ships as an optional skill in Hermes Agent.

What it does

Accelerate handles device placement, gradient scaling, data sharding, and launcher configuration so you do not have to. You write one training script, and Accelerate adapts it to the hardware you point it at. The same script that runs on a laptop with a single GPU can run on an eight-GPU workstation or a multi-node cluster without edits. You configure the hardware once through an interactive prompt or a YAML file, then launch with accelerate launch.

Before you start

This skill is optional and installed on demand. It runs on Linux, macOS, and Windows. You need a working PyTorch installation. For multi-GPU or multi-node setups, you need the appropriate hardware and networking (for example, NCCL for GPU communication). For DeepSpeed, install the deepspeed package separately. For FSDP, PyTorch 1.12 or later is required. For FP8 support, you need an H100 or newer GPU.

Install Accelerate with pip:

pip install accelerate

Quick start: convert a PyTorch script in four lines

The core idea is to add an Accelerator object, call prepare() on your model, optimizer, and dataloader, and replace loss.backward() with accelerator.backward(loss). Here is the diff:

import torch
+ from accelerate import Accelerator

+ accelerator = Accelerator()

  model = torch.nn.Transformer()
  optimizer = torch.optim.Adam(model.parameters())
  dataloader = torch.utils.data.DataLoader(dataset)

+ model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)

  for batch in dataloader:
      optimizer.zero_grad()
      loss = model(batch)
-     loss.backward()
+     accelerator.backward(loss)
      optimizer.step()

Run it with a single command:

accelerate launch train.py

Common workflows

Workflow 1: From single GPU to multi-GPU

Start with a script that hard-codes a single GPU:

# train.py
import torch

model = torch.nn.Linear(10, 2).to('cuda')
optimizer = torch.optim.Adam(model.parameters())
dataloader = torch.utils.data.DataLoader(dataset, batch_size=32)

for epoch in range(10):
    for batch in dataloader:
        batch = batch.to('cuda')
        optimizer.zero_grad()
        loss = model(batch).mean()
        loss.backward()
        optimizer.step()

Add the four Accelerate lines and remove the manual .to('cuda') calls:

# train.py
import torch
from accelerate import Accelerator  # +1

accelerator = Accelerator()  # +2

model = torch.nn.Linear(10, 2)
optimizer = torch.optim.Adam(model.parameters())
dataloader = torch.utils.data.DataLoader(dataset, batch_size=32)

model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)  # +3

for epoch in range(10):
    for batch in dataloader:
        # No .to('cuda') needed - automatic!
        optimizer.zero_grad()
        loss = model(batch).mean()
        accelerator.backward(loss)  # +4
        optimizer.step()

Configure the hardware interactively:

accelerate config

The prompt asks:

  • Which machine? (single/multi GPU/TPU/CPU)
  • How many machines? (1)
  • Mixed precision? (no/fp16/bf16/fp8)
  • DeepSpeed? (no/yes)

Then launch on any setup:

# Single GPU
accelerate launch train.py

# Multi-GPU (8 GPUs)
accelerate launch --multi_gpu --num_processes 8 train.py

# Multi-node
accelerate launch --multi_gpu --num_processes 16 \
  --num_machines 2 --machine_rank 0 \
  --main_process_ip $MASTER_ADDR \
  train.py

Workflow 2: Mixed precision training

Enable FP16, BF16, or FP8 by passing the mixed_precision argument to the Accelerator constructor:

from accelerate import Accelerator

# FP16 (with gradient scaling)
accelerator = Accelerator(mixed_precision='fp16')

# BF16 (no scaling, more stable)
accelerator = Accelerator(mixed_precision='bf16')

# FP8 (H100+)
accelerator = Accelerator(mixed_precision='fp8')

model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)

# Everything else is automatic!
for batch in dataloader:
    with accelerator.autocast():  # Optional, done automatically
        loss = model(batch)
    accelerator.backward(loss)

Workflow 3: DeepSpeed ZeRO integration

Pass a DeepSpeedPlugin instance to the Accelerator constructor. Do not pass a raw dictionary.

from accelerate import Accelerator, DeepSpeedPlugin

deepspeed_plugin = DeepSpeedPlugin(
    zero_stage=2,                     # ZeRO-2
    offload_optimizer_device="none",  # or "cpu" to offload
    gradient_accumulation_steps=4,
)

accelerator = Accelerator(
    mixed_precision='bf16',
    deepspeed_plugin=deepspeed_plugin,  # DeepSpeedPlugin instance (or dict[str, DeepSpeedPlugin])
)

# Same code as before!
model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)

You can also point at a full DeepSpeed JSON config file via the plugin:

from accelerate import Accelerator, DeepSpeedPlugin

# hf_ds_config accepts a path to a DeepSpeed config JSON (or a dict)
deepspeed_plugin = DeepSpeedPlugin(hf_ds_config="ds_config.json")
accelerator = Accelerator(mixed_precision='bf16', deepspeed_plugin=deepspeed_plugin)

Example ds_config.json:

{
    "fp16": {"enabled": false},
    "bf16": {"enabled": true},
    "zero_optimization": {
        "stage": 2,
        "offload_optimizer": {"device": "cpu"},
        "allgather_bucket_size": 5e8,
        "reduce_bucket_size": 5e8
    }
}

Or configure DeepSpeed through the interactive prompt:

accelerate config
# Select: DeepSpeed → ZeRO-2
# This writes an accelerate YAML config (default: ~/.cache/huggingface/accelerate/default_config.yaml)

Launch using the accelerate YAML config (not the raw DeepSpeed JSON):

# Uses the default accelerate config written by `accelerate config`
accelerate launch train.py

# Or point at a specific accelerate YAML
accelerate launch --config_file accelerate_deepspeed.yaml train.py

Workflow 4: FSDP (Fully Sharded Data Parallel)

Enable FSDP by passing a FullyShardedDataParallelPlugin:

from accelerate import Accelerator, FullyShardedDataParallelPlugin

fsdp_plugin = FullyShardedDataParallelPlugin(
    sharding_strategy="FULL_SHARD",  # ZeRO-3 equivalent
    auto_wrap_policy="transformer_based_wrap",  # valid: transformer_based_wrap | size_based_wrap | no_wrap
    cpu_offload=False
)

accelerator = Accelerator(
    mixed_precision='bf16',
    fsdp_plugin=fsdp_plugin
)

model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)

Or configure through the interactive prompt:

accelerate config
# Select: FSDP → Full Shard → No CPU Offload

Workflow 5: Gradient accumulation

Set gradient_accumulation_steps on the Accelerator and wrap the training step in accelerator.accumulate(model):

from accelerate import Accelerator

accelerator = Accelerator(gradient_accumulation_steps=4)

model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)

for batch in dataloader:
    with accelerator.accumulate(model):  # Handles accumulation
        optimizer.zero_grad()
        loss = model(batch)
        accelerator.backward(loss)
        optimizer.step()

The effective batch size is batch_size * num_gpus * gradient_accumulation_steps.

When to use vs alternatives

Use Accelerate when:

  • You want the simplest path to distributed training.
  • You need a single script that works on any hardware.
  • You are already in the HuggingFace ecosystem.
  • You want flexibility to switch between DDP, DeepSpeed, FSDP, or Megatron without rewriting code.
  • You are prototyping and want to move fast.

Key advantages:

  • 4 lines: Minimal code changes.
  • Unified API: Same code for DDP, DeepSpeed, FSDP, Megatron.
  • Automatic: Device placement, mixed precision, sharding.
  • Interactive config: No manual launcher setup.
  • Single launch: Works everywhere.

Use alternatives instead:

  • PyTorch Lightning: If you need callbacks and high-level abstractions.
  • Ray Train: For multi-node orchestration and hyperparameter tuning.
  • DeepSpeed: If you need direct API control and advanced features.
  • Raw DDP: If you want maximum control and minimal abstraction.

Common issues

Issue: Wrong device placement

Do not manually move tensors to device after prepare():

# WRONG
batch = batch.to('cuda')

# CORRECT
# Accelerate handles it automatically after prepare()

Issue: Gradient accumulation not working

Use the context manager:

# CORRECT
with accelerator.accumulate(model):
    optimizer.zero_grad()
    accelerator.backward(loss)
    optimizer.step()

Issue: Checkpointing in distributed

Use the accelerator's save and load methods:

# Save only on main process
if accelerator.is_main_process:
    accelerator.save_state('checkpoint/')

# Load on all processes
accelerator.load_state('checkpoint/')

Issue: Different results with FSDP

Ensure the same random seed on all processes:

from accelerate.utils import set_seed
set_seed(42)

Advanced topics

Hardware requirements

  • CPU: Works (slow)
  • Single GPU: Works
  • Multi-GPU: DDP (default), DeepSpeed, or FSDP
  • Multi-node: DDP, DeepSpeed, FSDP, Megatron
  • TPU: Supported
  • Apple MPS: Supported

Launcher requirements:

  • DDP: torch.distributed.run (built-in)
  • DeepSpeed: deepspeed (pip install deepspeed)
  • FSDP: PyTorch 1.12+ (built-in)
  • Megatron: Custom setup

Resources

More MLOps skills

ChromaEmbedding database for RAG and semantic search.ClipZero-shot image classification and image-text search.Evaluating Llms Harnesslm-eval-harness: benchmark LLMs (MMLU, GSM8K, etc.).Weights And BiasesW&B: log ML experiments, sweeps, model registry, dashboards.FaissFast vector similarity search at billion scale.Optimizing Attention FlashSpeed up long-sequence transformer training and inference.