Do Large Language Models Latently Perform Multi-Hop Reasoning?
FreeInvestigates latent multi-hop reasoning pathways in LLMs
About Do Large Language Models Latently Perform Multi-Hop Reasoning?
This research paper, accepted at ACL 2024, investigates whether large language models spontaneously perform multi-hop reasoning when given complex prompts like 'The mother of the singer of Superstition is'. The authors analyze two latent reasoning hops: first, the model must identify the bridge entity (e.g., Stevie Wonder) from an indirect mention, and second, it must use knowledge about that entity to answer. Through controlled experiments, they find strong evidence for the first hop across many relation types, but only moderate evidence for the full two-hop traversal. The study also reveals a clear scaling trend with model size for the first hop, but not for the second, suggesting challenges and opportunities for improving LLM reasoning.
Key Features
Pros & Cons
- Provides empirical evidence for latent multi-hop reasoning in LLMs
- Highlights contextual variability based on relation types
- Identifies scaling trends for the first reasoning hop
- Second-hop and full traversal evidence is moderate on average
- Scaling trend does not extend to the second reasoning hop
- Highly contextual performance across different prompt types