karpathy/llama2.c
FreeInference Llama 2 in one file of pure C
FreeFree tier
Outputs: text
About karpathy/llama2.c
llama2.c is a minimal, single-file C implementation for running inference on Llama 2 large language models. It allows training the Llama 2 architecture in PyTorch and then performing inference using a simple 700-line C file (run.c) with no dependencies. The project focuses on simplicity and education, supporting both custom tiny models (e.g., trained on TinyStories) and Meta's Llama 2 models in fp32 (up to 7B parameters). It runs entirely on CPU and achieves fast inference speeds—for example, ~110 tokens per second on an M1 MacBook Air for a 15M parameter model. The repository is open source and designed as a learning resource for understanding transformer-based LLM inference.
Key Features
Inference in a single 700-line C file (run.c) with no external dependencies
Train Llama 2 models using PyTorch, then deploy with pure C inference
Supports both custom trained tiny models and Meta's Llama 2 models (fp32, up to 7B parameters)
Fast CPU inference (~110 tokens/s for 15M model on M1 MacBook Air)
Educational and minimal design, ideal for learning LLM internals
Open source and easy to compile and run on any system with a C compiler
Pros & Cons
Pros
- Extremely simple and minimal codebase; easy to understand and modify
- No dependencies beyond a C compiler; portable across platforms
- Fast inference on CPU for small models
- Supports both custom trained models and Meta's Llama 2 checkpoints
- Open source with permissive license
- Great educational resource for learning about LLM inference
Cons
- Only supports fp32 inference; lacks quantization support (work in progress)
- Limited to models that fit in fp32 on available RAM; practically up to 7B parameters
- Only supports Llama 2 architecture, not other transformer variants
- No built-in GPU acceleration; CPU-only
- Relatively young project; may have limited community or documentation
Best For
Learning how transformer-based LLM inference works under the hoodRunning small language models on edge devices or CPUs without GPU dependenciesEducational projects and demonstrations in AI/ML coursesPrototyping and experimenting with custom tiny LLMsQuick and simple local text generation for narrow domains (e.g., children's stories)
FAQ
How do I run a model with llama2.c?
Download a model checkpoint (e.g., stories15M.bin), compile the C code with `make run`, and then run `./run stories15M.bin`. The program will generate text output.
What models are compatible?
The tool supports custom models trained on the Llama 2 architecture using the provided PyTorch training code, as well as Meta's official Llama 2 models. However, only fp32 weights are currently supported, limiting practical use to models up to 7B parameters.
What is the inference speed?
On an M1 MacBook Air, the 15M parameter model runs at approximately 110 tokens per second. Larger models will be slower but may still run at interactive rates.