One Token Away from Collapse: Fragility of Instruction-Tuned Helpfulness (April 2026)

Free

Trivial lexical constraints (banning one punctuation mark) cause 14–48% response collapse in instruction-tuned LLMs — identified as planning failure via mechanistic analysis; base models show no collapse

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About One Token Away from Collapse: Fragility of Instruction-Tuned Helpfulness (April 2026)

This research paper investigates the fragility of instruction-tuned large language models under trivial lexical constraints, such as banning a single punctuation character or common word. Across seven models spanning five families (7B–70B, open- and closed-weight), such constraints cause a 14–48% loss in response comprehensiveness, confirmed by blinded human evaluation and over 4,100 automated pairwise comparisons. The study identifies the root cause as a planning failure: linear probes on prompt representations predict response length with high accuracy before generation begins, and two-pass generation recovers 59–96% of response length. Base models show no systematic degradation, indicating that instruction tuning couples task competence to narrow surface-form templates. The effect extends to realistic deployment constraints (e.g., preamble suppression, corporate tone guidelines) and exposes a methodological blind spot in standard LLM-as-judge evaluation, which detects only a 3.5% quality drop where pairwise evaluation reveals 23%.

Key Features

Identifies planning failure as root cause of response collapse under trivial lexical constraints

Demonstrates that instruction tuning introduces fragility to surface-form templates absent in base models

Covers seven models (7B–70B) across five families, including open- and closed-weight

Validates findings with blinded human evaluation by 10 STEM-trained evaluators and 4,100 automated pairwise comparisons

Extends analysis to realistic deployment constraints causing comparable degradation (-22% to -34%)

Reveals methodological blind spot: standard LLM-as-judge detects only 3.5% drop vs. 23% in pairwise evaluation

Pros & Cons

Pros

Provides rigorous mechanistic analysis of collapse via linear probes
Includes both human and automated evaluations for robust evidence
Covers a broad range of model scales and families (7B–70B)
Exposes a critical blind spot in current LLM evaluation practices

Cons

Only examines lexical constraints; non-lexical constraints not studied
Generalization to all instruction-tuned models not fully established
Focus solely on helpfulness metric; other quality aspects not evaluated

Best For

Evaluating robustness of instruction-tuned LLMs to lexical constraintsImproving evaluation methodologies for LLM helpfulnessUnderstanding and mitigating planning failures in instruction-tuned modelsInforming deployment guidelines to avoid unintended response degradation

FAQ

What causes the response collapse in instruction-tuned LLMs?

The study identifies it as a planning failure: linear probes on prompt representations predict response length with high accuracy before generation begins, and base models show no such structure, indicating instruction tuning couples task competence to narrow surface-form templates.

Do base models exhibit similar collapse under lexical constraints?

No, base models show no systematic degradation under identical constraints, demonstrating that instruction tuning introduces the fragility.

How much does response comprehensiveness drop?

Across seven models, banning a single punctuation or common word caused 14–48% loss in comprehensiveness.

Does standard LLM-as-judge evaluation detect this collapse?

Standard LLM-as-judge evaluation detects only a 3.5% quality drop, whereas pairwise evaluation reveals 23%, exposing a methodological blind spot.