Neura Market
Language Models are Few-Shot Learners logo

Language Models are Few-Shot Learners

Free

GPT-3 capabilities.

FreeFree tier
Type
Open Source
Visit Website GitHub (40 stars)

About Language Models are Few-Shot Learners

This paper introduces GPT-3, an autoregressive language model with 175 billion parameters—10 times larger than any previous non-sparse language model. GPT-3 demonstrates that scaling up language models greatly improves task-agnostic, few-shot performance, often matching or exceeding prior state-of-the-art fine-tuning approaches. Without any gradient updates or fine-tuning, GPT-3 performs tasks specified purely via text interaction, achieving strong results on translation, question-answering, cloze tasks, unscrambling words, using novel words, 3-digit arithmetic, and generating news articles that human evaluators struggle to distinguish from human-written text. The paper also identifies datasets where few-shot learning still struggles and discusses methodological issues related to training on large web corpora, as well as broader societal impacts.

Key Features

175 billion parameters, 10x more than previous non-sparse language models
Autoregressive architecture trained on a large text corpus
Few-shot learning without gradient updates or fine-tuning
Tasks specified purely via text interaction
Strong performance on translation, question-answering, cloze tasks, arithmetic, and more
Generates news articles difficult to distinguish from human-written ones

Pros & Cons

Pros
  • Achieves competitive few-shot performance without fine-tuning
  • Scales to a huge model (175B parameters) yielding strong results
  • Capable of diverse NLP tasks with simple text prompts
  • High-quality text generation, sometimes indistinguishable from human writing
Cons
  • Still struggles on some NLP datasets in the few-shot setting
  • Methodological issues from training on large web corpora (biases, contamination)
  • Extremely large model requires immense computational resources
  • Not openly available as a deployable model (research paper only)

Best For

Translation between languagesQuestion answeringCloze/completion tasksOn-the-fly reasoning (e.g., unscrambling words, using novel words)Arithmetic reasoning (e.g., 3-digit arithmetic)Domain adaptation from few examplesGenerating realistic news articles

FAQ

What is GPT-3?
GPT-3 is a 175 billion parameter autoregressive language model introduced in the paper 'Language Models are Few-Shot Learners.' It is designed to perform a wide range of NLP tasks in a few-shot setting without fine-tuning.
How does GPT-3 perform tasks?
GPT-3 performs tasks purely via text interaction, using a few examples (few-shot) or instructions, without any gradient updates or fine-tuning. The task and demonstrations are specified in the input text.
What tasks can GPT-3 perform?
Translation, question-answering, cloze tasks, unscrambling words, using a novel word in a sentence, 3-digit arithmetic, and generating news articles that humans find hard to distinguish from human-written ones.
What are the limitations of GPT-3?
GPT-3 still struggles on some datasets in few-shot learning and faces methodological issues related to training on large web corpora, including potential biases. The paper discusses broader societal impacts.