From Atomic Evidence to Logical Composition: Structured Compositional Reasoning over Compound Answer Options

Analysis

Why This Paper Matters

Large language models (LLMs) have shown remarkable capabilities in many natural language tasks, yet they often stumble when faced with compound answer options that require combining multiple atomic judgments under logical operators like AND, OR, and NEITHER/NOR. This paper identifies a fundamental limitation: even when models correctly judge individual atoms, they fail to compose them logically. This is a significant problem because real-world reasoning often involves such compound statements, and the inability to handle them undermines the reliability of LLMs in decision-making and question-answering.

The authors propose a novel framework that sidesteps the issue by never presenting the compound option to the model. Instead, they decompose each option into atomic answers, score each atom independently, and then use an operator-constrained integer linear program (ILP) to compose the scores into a final prediction. This approach is both intuitive and effective, as demonstrated by substantial improvements on two benchmarks. The introduction of LOGICAL-SATA, a new benchmark derived from SATA-Bench, provides a valuable resource for future research on logical composition in reading comprehension.

Technical Contributions

• Decomposition into Atomic Judgments: The framework breaks down compound options into atomic statements, allowing the model to focus on simpler judgments without being confused by the logical structure.
• Contrastive Hypothesis Scoring: For each atomic answer, the model scores contrastive hypotheses (e.g., true vs. false), which helps calibrate the confidence of each judgment.
• Operator-Constrained ILP: An integer linear program is used to combine the atomic scores, with constraints that enforce the logical operator semantics (AND, OR, NEITHER/NOR). This ensures that the final prediction respects the logical structure.
• New Benchmark: LOGICAL-SATA is introduced, derived from SATA-Bench, to evaluate reading comprehension with compound answer options, filling a gap in existing benchmarks.

Results

The proposed framework achieves dramatic improvements over baseline LLMs. On the human-validated LOGICAL-COMMONSENSEQA split, Macro-F1 jumps from 48.3 to 77.0, a relative improvement of nearly 60%. On LOGICAL-SATA, the improvement is from 47.0 to 75.6. Notably, the largest gains are observed on NEITHER/NOR options, which are often the most challenging because they require negating both atoms. These results underscore the effectiveness of the decomposition and composition approach.

Significance

This work has significant implications for the AI field. By demonstrating that structured compositional reasoning can be achieved through decomposition and ILP-based composition, it opens new avenues for improving LLM reasoning without requiring architectural changes or extensive fine-tuning. The approach is model-agnostic and can be applied to any LLM, making it a practical tool for enhancing logical reasoning in real-world applications. Moreover, the introduction of LOGICAL-SATA provides a benchmark that can drive further research in this area. The success on NEITHER/NOR suggests that the method is particularly adept at handling negation, a known weakness of LLMs. Overall, this paper contributes both a novel methodology and a new benchmark, advancing the state of the art in compositional reasoning.