GPT

Analysis

Why This Paper Matters

Before GPT, most NLP models were trained from scratch for each task, requiring large labeled datasets. GPT introduced a paradigm shift: first learn a powerful language model from unlabeled text, then adapt it to specific tasks with minimal extra data. This approach dramatically improved performance across a wide range of natural language understanding benchmarks, setting the stage for the transformer-based pretraining revolution.

How It Works

GPT uses a decoder-only transformer architecture with 12 layers, 768-dimensional hidden states, and 12 attention heads. The model is first pre-trained on the BooksCorpus (over 7,000 books) using a standard autoregressive language modeling objective: given a context window of k tokens, predict the next token.

The training objective maximizes the likelihood of the next token given the previous k tokens, where the conditional probability is modeled by the transformer with parameters Θ.

During fine-tuning, the pre-trained model is adapted to a supervised task by adding a linear output layer on top of the final transformer activation. The input tokens are passed through the model, and the final hidden state is fed into a softmax classifier to predict the label. Importantly, GPT includes the language modeling objective as an auxiliary loss during fine-tuning, which improves generalization and speeds up convergence.

To handle diverse tasks with minimal architectural changes, GPT introduces task-aware input transformations. For textual entailment, premise and hypothesis are concatenated with a delimiter token. For similarity tasks, both possible orderings of the two sentences are processed independently and their representations are added element-wise. For question answering, the context, question, and each answer are concatenated with delimiters, and the model produces a distribution over answers via softmax.

Results

GPT achieves state-of-the-art results on 9 of 12 evaluated tasks. On natural language inference benchmarks, it outperforms previous methods by significant margins (e.g., 5.8% on RTE, 1.5% on QNLI). On question answering, it achieves 8.9% improvement on RACE and strong results on Story Cloze Test. On semantic similarity tasks (STS-B, MRPC) and classification tasks (CoLA, SST-2), GPT also sets new records.

Significance

GPT demonstrated that generative pretraining on unlabeled text is a powerful and general method for improving natural language understanding. Its two-stage framework—unsupervised pretraining followed by supervised fine-tuning—became the dominant paradigm in NLP, directly inspiring GPT-2, GPT-3, and many other models. The task-aware input transformation approach allowed a single pretrained model to be applied to diverse tasks without architectural changes, highlighting the flexibility of transformer-based language models. This work was a key milestone in the shift toward large-scale pretraining that has since revolutionized the field.