About **karpathy/autoresearch**
Autoresearch is an open-source project by Andrej Karpathy that implements an autonomous AI agent for conducting research on single-GPU nanochat training. The agent operates by modifying a training script (train.py), executing a fixed 5-minute training run, and evaluating the result using validation bits per byte (val_bpb) as the metric. It repeats this cycle overnight, keeping or discarding changes based on performance improvements. The human researcher provides high-level guidance through a markdown file (program.md), while the agent handles low-level code modifications. The setup is minimal, requiring only a single NVIDIA GPU (tested on H100), Python 3.10+, and the uv package manager.
Key Features
Autonomous agent modifies train.py to experiment with architectures, hyperparameters, optimizers, and batch sizes
Fixed 5-minute training time budget per experiment (wall clock, excluding startup/compilation)
Evaluates using validation bits per byte (val_bpb) metric, which is vocabulary-size-independent
Human defines research context and instructions via program.md file
Fixed prepare.py script handles data download, tokenizer training, and runtime utilities
Single NVIDIA GPU required (tested on H100)
Pros & Cons
Pros
- Frees human researchers from manual iterative experimentation
- Open-source code with minimal dependencies (Python, uv, a single GPU)
- Transparent logging and reproducibility via program.md and fixed setup
- Simple design allows easy extension to multi-agent or more complex research orgs
Cons
- Limited to single-GPU setups (tested on H100, may not scale to multi-GPU)
- 5-minute training runs may be insufficient for deeper convergence or large models
- Agent’s modifications can produce unpredictable results or fail to improve
- No built-in safety or constraint mechanisms to prevent runaway changes
Best For
Overnight automated LLM training experimentsHyperparameter and architecture search for small language modelsRapid prototyping of training pipelines on consumer-grade GPUsEducational tool for understanding AI agent-driven research workflows
FAQ
What GPU is required to run autoresearch?
A single NVIDIA GPU is required; the project was tested on an H100.
How long does each training experiment run?
Each experiment runs for exactly 5 minutes of wall clock time, excluding startup and compilation.
Which files can the AI agent modify?
The agent is only allowed to edit train.py. The prepare.py file is fixed, and program.md is authored by the human researcher.
What is val_bpb?
Val_bpb stands for validation bits per byte, the metric used to evaluate model performance. Lower values are better, and it is independent of vocabulary size, allowing fair comparison across architectural changes.