Workshop to Train GPT Model from Scratch on Laptop
A GitHub repository introduces a practical workshop where participants code each element of a GPT training system. This approach helps users understand the function and purpose of every part.
Andrej Karpathy's nanoGPT first introduced the creator to large language models and transformers. That project showed how to assemble a functional language model using just a few hundred lines of PyTorch code. It shifted the creator's view of artificial intelligence and encouraged further study.
This workshop aims to deliver a similar introduction to others. While nanoGPT recreates GPT-2 with 124 million parameters and handles many topics, this effort reduces it to core elements. It uses a model around 10 million parameters that completes training on a typical laptop in less than one hour. The design fits into one workshop session.
The process avoids pre-packaged libraries. There is no use of commands like model = AutoModel.from_pretrained(). Participants construct the entire system.
Components Built in the Workshop
Users end up with a complete GPT model trained from the beginning on their MacBook. The model generates text similar to Shakespeare. The workshop guides writing these parts:
- Tokenizer: Converts text into numerical form for model processing.
- Model structure: Transformer with embeddings, attention mechanisms, and feed-forward layers.
- Training process: Forward pass, loss calculation, backpropagation, optimizer, and learning rate adjustments.
- Text output: Sampling methods from the trained model.
Requirements and Setup Instructions
The workshop runs on any laptop or desktop, including Mac, Linux, or Windows machines. It requires Python 3.12 or newer. Users need to feel comfortable reading Python code, but machine learning background is not necessary.
Training adapts automatically to Apple Silicon GPU via MPS, NVIDIA GPU through CUDA, or CPU. It also operates on Google Colab by uploading files and running with !python train.py.
For local setup, first install uv if absent:
On macOS or Linux:
curl -LsSf https://astral.sh/uv/install.sh | sh
On Windows:
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
Next, run uv sync, then create a scratchpad directory and enter it with mkdir scratchpad && cd scratchpad.
For Google Colab, install packages with !pip install torch numpy tqdm tiktoken. Upload data/shakespeare.txt to Colab files. Code goes in notebook cells, or upload .py files and execute with !python train.py.
Follow the documentation sequentially. Each section explains and guides coding one pipeline piece, covering its operation and rationale. At the end, users have their own model.py, train.py, and generate.py files.
Workshop Sections Overview
The workshop divides into six parts:
Stay ahead of the AI curve
The most important updates, news, and content — delivered weekly.
No spam. Unsubscribe anytime.
| Part | Content | Key Ideas | |, , , |, , , , -|, , , , , -| | Part 1: Tokenization | Character-level tokenizer | Character encoding, vocabulary size, why BPE struggles with small data | | Part 2: The Transformer | Complete GPT model structure | Embeddings, self-attention, layer normalization, MLP blocks | | Part 3: The Training Loop | Full training system | Loss functions, AdamW optimizer, gradient clipping, learning rate scheduling | | Part 4: Text Generation | Inference and sampling | Temperature control, top-k sampling, autoregressive decoding | | Part 5: Putting It All Together | Train on actual data, run tests | Loss curves, scaling tests, further steps | | Part 6: Competition | Train top AI poet | Source datasets, increase scale, submit best poem |
Model Architecture Summary
The system processes input text through these steps:
Text enters the tokenizer, for example "hello" becomes [20, 43, 50, 50, 53] using character-level encoding.
Token embeddings add positional details to create vectors of n_embd dimensions.
Transformer blocks repeat n_layer times. Each block applies layer normalization and self-attention with n_head parallel heads, plus a residual connection. Then layer normalization and MLP with 4x expansion, GELU activation, and projection, also with residual.
Final layer normalization leads to a linear layer outputting logits over vocab_size for the next token probability.
Available Model Configurations
| Config | Parameters | n_layer | n_head | n_embd | Train Time (M3 Pro) | |, , , , |, , , , , , |, , , , -|, , , , |, , , , |, , , , , , , , , , -| | Tiny | ~0.5M | 2 | 2 | 128 | ~5 min | | Small | ~4M | 4 | 4 | 256 | ~20 min | | Medium (default) | ~10M | 6 | 6 | 384 | ~45 min |
All use character-level tokenization with vocab_size=65 and block_size=256.
Tokenization Choices
Character-level tokenization suits Shakespeare data. BPE with GPT-2's 50,257 vocabulary requires larger datasets, as rare bigrams hinder pattern learning on small sets.
| Tokenizer | Vocab Size | Dataset Size Needed | |, , , , , -|, , , , , , |, , , , , , , , , , -| | Character-level | ~65 | Small (Shakespeare, ~1MB) | | BPE (tiktoken) | 50,257 | Large (TinyStories+, 100MB+) |
Part 5 explains adapting to BPE for bigger datasets.
Key Resources
- nanoGPT: Basis for this workshop, minimal GPT training in about 300 PyTorch lines.
- build-nanogpt video lecture: Four-hour session constructing GPT-2 from nothing.
- Karpathy's microgpt: Complete GPT in 200 lines of pure Python without external dependencies.
- nanochat: Pipeline to train a full ChatGPT replica.
- Attention Is All You Need (2017): Transformer paper.
- GPT-2 paper (2019): Language models for unsupervised learning.
- TinyStories paper: Small models on curated data perform strongly.
Andrej Karpathy, known for roles at OpenAI and Tesla leading AI efforts, released nanoGPT in 2022 to clarify language model training. The 2017 transformer paper by Vaswani and others shifted sequence modeling. GPT-2 from OpenAI in 2019 advanced generative text.