Massive Multitask Language Understanding Pro — harder variant with 10 answer choices across 57 subjects

Instruction Following Evaluation — measures how well models follow complex multi-step instructions

LMSYS Chatbot Arena — crowdsourced blind comparison Elo rating from human preferences

Short factual questions testing factuality and calibration of language models.

Multilingual Massive Multitask Language Understanding across many languages.

Multi-Round Coreference Resolution — long-context needle-in-haystack retrieval.

Code generation benchmark — 164 Python programming problems measuring functional correctness

Terminal/CLI task completion — executing shell commands to achieve goals.

Scientific code generation — implementing research algorithms from papers.

Competition-level mathematics problems spanning algebra, geometry, number theory, and more

Grade School Math — 8,500 linguistically diverse grade school math word problems

AI2 Reasoning Challenge — grade-school science questions requiring multi-step reasoning

Subset of BIG-Bench tasks that are challenging for language models — tests reasoning

Commonsense natural language inference — predict the most plausible continuation of a scenario

Large-scale Winograd schema challenge for commonsense reasoning

Measures whether models generate truthful answers — tests resistance to common misconceptions

Harder version of MMMU with augmented candidate options and vision-only input.

Massive Multi-discipline Multimodal Understanding — college-level multimodal questions.

GUI element localization on professional software screenshots.

Real-world computer task completion in operating system environments.

Multi-tool orchestration — using multiple tools to solve complex tasks.

Web browsing comprehension — finding specific information across complex web pages.

Tool-Agent-User benchmark for retail customer service scenarios.