CHallenging AI with Synthetic Evaluations (CHASE)

Analysis

Why This Paper Matters

Large language models (LLMs) are increasingly evaluated on static benchmarks that risk contamination—where the model may have seen the test data during training. This makes it hard to measure genuine reasoning ability. CHASE (CHallenging AI with Synthetic Evaluations) offers a way to generate fresh, challenging evaluation problems on demand, without human effort. By building problems from the ground up and hiding solution elements within context, CHASE aims to create tasks that are hard even for the model that generates them. This could help the AI community move beyond fixed benchmarks and toward more dynamic, robust evaluation.

How It Works

The core insight of CHASE is to start with a simple problem-solution pair and then systematically complicate the context. Instead of generating a hard problem and then solving it (which would make it trivial for the generating model), CHASE builds difficulty by hiding parts of the solution within a larger, irrelevant context. The framework uses two separate LLMs: a generator (G) and a verifier (V). Each step in the generation pipeline is a simple, specific task, allowing fine-grained control and verification.

For the CHASE-QA benchmark, the process begins with generating diverse scenarios (persona + document collection). Given a scenario, G creates a question-answer pair where the answer is composed of multiple points distributed across several documents. To increase difficulty, G also generates adversarial questions with answers that are similar in style but irrelevant to the original question. Finally, G produces full-length documents that contain the assigned answer points along with unrelated information. V checks that all answer points are present (completeness) and that no extra relevant information leaks into other documents (no leakage).

For CHASE-CODE, the framework generates repository-level code completion problems. Given a set of Python functions, the task is to implement a new function based on objectives. The generation process similarly decomposes into sub-tasks: creating function signatures, generating test cases, and verifying correctness. CHASE-MATH focuses on grade-school math word problems involving basic arithmetic, again using a bottom-up approach to build complexity.

Results

The paper introduces three benchmarks: CHASE-QA (671 problems), CHASE-CODE (500 problems), and CHASE-MATH (500 problems). However, the paper does not present any evaluation results—no model scores, no comparisons to existing benchmarks, and no analysis of problem difficulty. The contribution is purely methodological: a framework for generating the benchmarks, not the benchmarks themselves.

Significance

CHASE addresses a real need in the LLM evaluation landscape: the ability to generate fresh, challenging problems without human annotation. Its bottom-up approach and use of verifier LLMs to ensure quality are sensible design choices. However, the lack of empirical validation is a major gap. Without showing that the generated problems are actually hard for current models, or that they measure distinct capabilities, the framework remains an interesting idea rather than a proven tool. Future work should evaluate CHASE-generated benchmarks against existing ones and demonstrate their utility in detecting model weaknesses.