ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

Analysis

Why This Paper Matters

Enterprise workflows are increasingly relying on AI agents to extract structured data from unstructured documents, guided by user-defined schemas. However, existing benchmarks often focus on simple question-answering or generic extraction, failing to capture the complexities of real-world enterprise documents—such as long records, varied layouts, and the need for source grounding. ExtractBench addresses this gap by introducing a comprehensive benchmark that evaluates not only extraction accuracy but also record completeness, grounding, and cost, which are critical for practical deployment.

The paper's emphasis on measuring cost alongside accuracy is particularly significant. In enterprise settings, the trade-off between accuracy and cost directly impacts ROI. By providing a standardized evaluation, ExtractBench enables practitioners to make informed decisions about which models or agents to deploy, balancing performance with budget constraints. This is a step forward from purely accuracy-centric benchmarks, aligning with the needs of AI practitioners who must consider operational efficiency.

Technical Contributions

• Holistic Evaluation Metrics: Introduces order-insensitive value F1 for value accuracy, plus word- and page-level F1 for grounding, and measures cost per extraction task. This multi-faceted approach captures both correctness and traceability.
• Diverse and Tagged Dataset: 4,869 pages across 370 documents, 8 domains, and 67 types, with tags for challenge scenarios (e.g., long documents, complex tables) to enable targeted analysis.
• Scalable Ground-Truth Curation: Combines independent-system agreement for real documents, known values for synthetic lists, and human verification for forms, balancing quality and scalability.
• Benchmarking of Multiple Agent Types: Compares commercial VLMs, coding agents, and a specialized agent (LlamaExtract Agentic Plus), revealing distinct performance profiles.

Results

The paper reports that commercial VLMs perform well on short documents but often truncate record lists on long ones, leading to incomplete extractions. Coding agents, while more accurate, incur significantly higher costs. LlamaExtract Agentic Plus achieves the highest scores on all three metrics (value F1, word-level grounding F1, page-level grounding F1), with accuracy comparable to coding agents but at a fraction of the cost. This suggests that specialized agents can offer a cost-effective solution without sacrificing accuracy.

Significance

ExtractBench sets a new standard for evaluating schema-guided extraction systems, pushing the field beyond simple accuracy to include grounding and cost. This will likely influence future benchmark designs and encourage the development of more efficient and reliable extraction agents. For AI practitioners, the benchmark provides a practical tool for model selection and deployment, potentially accelerating the adoption of AI in enterprise document processing. The public availability of the dataset and code further facilitates reproducibility and community progress.