Multitask Prompted Training Enables Zero-Shot Task Generalization
FreeZero-shot task generalization via multitask prompted training
FreeFree tier
Inputs: textOutputs: text
About Multitask Prompted Training Enables Zero-Shot Task Generalization
Multitask Prompted Training Enables Zero-Shot Task Generalization is a research paper (ICLR 2022 Spotlight) that introduces a system for mapping natural language tasks into a human-readable prompted form and fine-tuning a pretrained encoder-decoder model on a multitask mixture of diverse prompted datasets. The resulting model achieves strong zero-shot generalization on held-out tasks, often outperforming models up to 16x its size, and also performs well on a subset of BIG-bench tasks. All trained models and prompts are publicly available.
Key Features
System for mapping natural language tasks into human-readable prompted form
Multitask mixture training on diverse supervised datasets with multiple prompts per task
Fine-tunes a pretrained encoder-decoder model (T5-based) for zero-shot generalization
Demonstrates strong zero-shot performance on standard datasets (e.g., outperforming models 16x its size)
Evaluated on held-out tasks from the BIG-bench benchmark (outperforms models 6x its size)
All trained models and prompts released open source
Pros & Cons
Pros
- State-of-the-art zero-shot performance at time of publication
- Open-source models and prompts available for reproducibility
- Outperforms much larger models despite smaller size
- Peer-reviewed and spotlighted at ICLR 2022
- Employs diverse prompt wording to improve robustness
Cons
- Requires a pretrained encoder-decoder architecture as starting point
- Zero-shot performance may degrade on tasks very different from training distribution
- Needs careful design of prompt templates for each task
- Limited to tasks that can be formulated as text-to-text
Best For
Zero-shot natural language understanding and generationBenchmarking model generalization on held-out tasksRapid adaptation to new NLP tasks without task-specific fine-tuningResearch into multitask learning and prompting strategies
FAQ
What is the main contribution of this paper?
The paper demonstrates that explicit multitask prompted training on diverse datasets can directly induce zero-shot task generalization in language models, achieving strong performance on held-out tasks even with models smaller than those used in implicit multitask learning approaches.
What model architecture is used?
The authors fine-tune a pretrained encoder-decoder model based on T5 (Raffel et al., 2020) and Lester et al. (2021) on a multitask mixture of prompted datasets.
Are the models publicly available?
Yes, all trained models and prompts are released open source. Links are provided in the paper.
How does this approach compare to GPT-3's zero-shot performance?
The approach attains strong zero-shot performance on several standard datasets, often outperforming models up to 16x its size, which includes comparisons to GPT-3.
What is the BIG-bench result?
The model attains strong performance on a subset of tasks from the BIG-bench benchmark, outperforming models up to 6x its size.