Large Language Models — Structured Notes

What is an LLM?

• A mathematical function that takes text as input and returns a probability distribution over possible next tokens
• Does not produce words directly; produces tokens

Making an LLM: Overview

Two stages:

1. Pretraining
2. Post-training

Pretraining

1. Data Collection

Three goals:

• Quantity — large volume so the model learns language rules
• Diversity — broad knowledge coverage
• Quality — no harmful content, no PII, no noise, no duplicates

Pipeline steps (see FineWeb):

• URL filtering
• Text extraction
• Cleaning / noise removal
• Deduplication
• PII removal

2. Tokenizer Training

What is a Token?

• The single unit of text an LLM produces per pass
• Not a word, not a character — a subword unit
• LLMs charge per token (see OpenRouter)

Why not words?

• Over 1 million English words; can't enumerate them all
• Can't handle novel/coined words

Why not characters?

• Some next characters are trivially obvious (e.g. "tokenizatio" → "n"); wasteful to run full forward pass
• Characters carry no meaning individually; tokens do ("token", "ization" have meaning; "t", "o", "k" don't)
• Would make inference far more expensive

Tokenization demo: Tiktokenizer

Famous consequence: LLMs never see "strawberry" — they see "st", "raw", "berry" (as token IDs). This explains apparent counting failures.

BPE (Byte Pair Encoding)

• Algorithm used by GPT
• Start with every character as its own token
• Repeatedly merge the most frequent adjacent pair into a new token
• After thousands of merges: common words become single tokens; rare/novel words split into known sub-tokens
• Trained independently, before the LLM itself
• BPE demo

3. Embeddings

What are Embeddings?

• Each token ID is mapped to a high-dimensional vector
• GPT-2: ~1000 dimensions; GPT-3: ~12,000 dimensions; Qwen3-8B: 4,096 dimensions
• TensorFlow Embedding Projector

Why vectors and not token IDs?

• A neural network cannot do arithmetic on a token ID
• Vectors allow encoding and deriving meaning through arithmetic

Examples:

king - man + woman ≈ queen
father - mother = man - woman
Hitler + Italy - Germany ≈ Mussolini

Why high dimensionality?

• Each dimension encodes some latent property (e.g. "Italian-ness", "dictator-ness")
• No human can pre-program these; they emerge from training

Polysemy Problem

• Many tokens have multiple meanings (e.g. "bank", "mole")
• Solution: embeddings are not static — the transformer updates them using surrounding context

Similarity Measurement

• Similarity between two embedding vectors = dot product
  • Dot product = 0 → no correlation
  • Dot product > 0 → similar direction
  • Dot product < 0 → opposite
• Semantle demo

4. Transformer Architecture

Origin paper: Attention Is All You Need (2017)

• Originally designed for machine translation
• Now powers nearly all text-based AI
• "T" in GPT = Transformer

Visualizer: Transformer Explainer

The Core Problem

"The cat sat on the mat because it was tired."

• "it" cannot be resolved by position alone
• The model must attend to other tokens to update the meaning of "it"

Attention Mechanism

Three vectors per token (derived by multiplying the token embedding by trained weight matrices):

• Query (Q): "What am I looking for?"
• Key (K): "What do I contain?"
• Value (V): "What information do I give if matched?"

Intuition: k[q] = v — attention is a soft dictionary lookup

Formula: $$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}} + M\right)V$$

• $QK^T$: dot product of Q and K (measures match)
• $\sqrt{d_k}$: normalisation — prevents dot product from inflating with more dimensions
• $M$: mask — set to $-\infty$ for future tokens, so softmax zeroes them out
• Result is multiplied by V to extract the matched value

Masked Self-Attention:

• "Masked" = future tokens are hidden during training
• "Self" = the sentence attends to itself

Attention formula demo

Multi-Head Attention

• Attention is run many times in parallel (multiple "heads")
• Outputs are combined, then passed through layer normalisation and a feedforward neural network

Full Forward Pass Summary

Input tokens
→ Token embeddings + Positional embeddings
→ Q, K, V vectors (via weight matrices)
→ QKᵀ → mask → softmax → × V
→ Feedforward neural network
→ Logits vector
→ Softmax → probability distribution over next token

LLM visualisation (BBycroft)

5. Training Objective: Cross-Entropy Loss

Cross-entropy explainer

• At the start, all weights are random → outputs are random
• Loss function: Cross-entropy $$L = -\log(p_{\text{correct token}})$$
• Low probability assigned to correct token → high loss
• Model updates weights via backpropagation to reduce loss

Perplexity:

• Interpretability wrapper around cross-entropy
• Perplexity demo

Post-Training

Base Model Problem

• The pretrained model is a base model / foundation model
• It only predicts next tokens
• Ask "What is the capital of France?" → it outputs more quiz questions, not an answer
• It is a stochastic parrot: repeats patterns from training data without following instructions

Stage 1: Supervised Fine-Tuning (SFT)

• Same cross-entropy loss, different data
• Data = (prompt, ideal response) pairs
• Teaches the model what assistant-style output looks like

Limitation: SFT has no concept of "wrong"

• It only increases probability of the demonstration response
• Does not decrease probability of harmful completions
• Rephrasing a prompt can bypass SFT refusals entirely

Stage 2: Preference Fine-Tuning (RLHF / DPO)

Why needed:

• SFT alone does not align the model's behaviour robustly
• Example of unaligned model: Microsoft Bing "Sydney" — LessWrong post

Data collection steps:

1. Create a set of prompts
2. Run SFT model to get multiple outputs per prompt
3. Have human annotators rank the outputs
4. Store the ranked preferences as a dataset

Training methods:

• RLHF (Reinforcement Learning from Human Feedback) — e.g. PPO
• DPO (Direct Preference Optimization) — e.g. IPO
• Both can be combined with PEFT / LoRA for efficiency

Key difference from SFT:

• Preference training with negative examples directly decreases probability of harmful completions across the reward landscape, not just at a single prompt

Stage 3: Reasoning Fine-Tuning

• "Think step by step" prompting used to improve results
• LLM providers have now baked this in automatically via fine-tuning
• Model is trained on step-by-step reasoning from textbooks
• Reinforcement learning is used to reinforce correct reasoning chains
• This is what produces "thinking" / reasoning models (vs instruct models)

Arguing LLMs can't "reason" is like arguing submarines can't "swim."

Evaluation (Evals)

• ARC Prize Leaderboard — reasoning benchmark; humans ~100%, best models ~84%
• METR — coding/agent benchmark (somewhat controversial)
• Chatbot Arena — human preference-based ranking (no fixed test set)

Quantization

Problem: Running large models requires storing all parameters in VRAM

Example calculation:

• Qwen3-8B: 8 billion parameters × 16 bits (BF16) = 16 GB required
• A typical laptop GPU has 6 GB VRAM

Solution — Quantization:

• Round each 16-bit weight value down to fewer bits
• Common: 4-bit quantization → ~4 GB for an 8B model

GGUF format — used by llama.cpp

Q4_K_M explained:

Component	Meaning
Q4	4 bits per weight
_K	K-quants: different layers get different precision (attention layers kept higher)
_M	Medium quality tier (balanced size vs quality)

Other tiers: _S (small/aggressive), _L (large/higher precision)

Reference: Quantization explainer | Bartowski's quantized models

Running Models Locally

• Ollama or llama.cpp to run GGUF models
• Hugging Face — repository for open-source models and datasets

Full Training Pipeline Summary

1. Collect data
2. Train tokenizer (BPE)
3. Train base model (cross-entropy on raw text)
4. Train SFT model (cross-entropy on demonstration data)
5. Train aligned model (RLHF / DPO with human preference data)
6. Optional: fine-tune for specific use cases
7. Quantize for deployment

Resources for Building Your Own Model

Resource	Purpose
Neural Networks: Zero to Hero — Karpathy	Full from-scratch training course
modded-nanogpt	GPT-2 level model, trains in ~2 min
Unsloth	Efficient LoRA fine-tuning of existing models
Hugging Face Datasets	Fine-tuning datasets