Gopher

Gopher is a 280 billion parameter transformer language model developed by DeepMind (now Google DeepMind) as part of a study into the effects of scale on language model performance. Published in December 2021, the research trained a series of models from 44 million to 280 billion parameters, with Gopher being the largest. The model excels in reading comprehension, fact-checking, and identification of toxic language, and achieves significant advancement on the Massive Multitask Language Understanding (MMLU) benchmark, approaching human expert performance. When prompted for dialogue, Gopher can exhibit surprising coherence, correctly discussing topics like cell biology with citations despite no dialogue-specific fine-tuning. However, the research also identifies persistent failure modes including repetition, stereotypical biases, and confident propagation of incorrect information. Two accompanying papers address ethical and social risks of large language models and a new architecture for improved training efficiency. Gopher is open source and free to use.