PyTorch FSDP Skill: Distributed Training Reference for Hermes Agent

Fully sharded data-parallel training is how large models fit across multiple GPUs without duplicating every parameter on each device. This skill, installed on demand into Hermes Agent, turns the official PyTorch FSDP documentation into a reference you can query while you work. Reach for it when you are wrapping a model, choosing a sharding strategy, saving checkpoints, or mixing precision, and you want the canonical pattern without leaving your terminal.

What it does

The skill packages the official PyTorch FSDP guidance into reference files that Hermes Agent can read on request. Instead of reconstructing FSDP incantations from memory, you ask the agent to pull the relevant snippet. The bundled quick-reference catalog holds roughly 157k characters of runnable FSDP examples, covering wrapping, sharding-strategy selection, checkpointing, and mixed precision. That catalog lives in references/common-patterns.md and is the first place to look when you need a concrete starting point.

Before you start

This is an optional skill, so it is not present in a default Hermes Agent install. It is installed on demand, and it runs on Linux and macOS. The skill itself is a documentation bundle, not a Python package, so there is no separate dependency to install; you do need a working PyTorch FSDP environment to apply the examples. The skill is authored by Orchestra Research, released under the MIT license, and is at version 1.0.0.

Working with the skill

For beginners

If you are new to FSDP, start with the getting_started or tutorials reference files. They walk through foundational concepts before you attempt a full training loop.

For specific features

When you need details on a particular API or guide, use the appropriate category reference file, such as api or guides. The files are organized so you can jump straight to the topic instead of scanning everything.

For code examples

The quick reference section is the fastest route to working code. It contains common patterns extracted from the official docs, so you can adapt them rather than write from scratch.

Reference files

The skill includes comprehensive documentation in references/:

• other.md - Other documentation

Use view to read specific reference files when detailed information is needed.

Resources

references/

Organized documentation extracted from official sources. These files contain:

• Detailed explanations
• Code examples with language annotations
• Links to original documentation
• Table of contents for quick navigation

scripts/

Add helper scripts here for common automation tasks.

assets/

Add templates, boilerplate, or example projects here.

When to use this skill

The skill is designed to trigger whenever you are working with pytorch-fsdp, asking about its features or APIs, implementing a solution, debugging code, or learning best practices. If your question is about distributed training in general but not FSDP specifically, a different skill will serve you better.

Limits and gotchas

This skill was automatically generated from official documentation. That means the reference files preserve the structure and examples from the source docs, including their language detection for syntax highlighting. The quick-reference patterns are extracted from common usage examples, so they cover typical cases rather than every possible configuration. If the official docs change, the skill does not update itself; you must re-run the scraper with the same configuration to rebuild it with the latest information.

What pairs with this

The skill points to the official PyTorch FSDP documentation as its upstream source. For a wider MLOps workflow, combine this with other optional skills in the same category, such as those covering distributed training or model optimization, to move from a single-GPU script to a sharded multi-GPU run.