FinanceHarness: Autonomous Financial Deep Research Framework

Analysis

Why This Paper Matters

FinanceHarness addresses a critical gap in deep research systems: most are general-purpose and fail to meet the specialized needs of financial analysis. Financial research requires domain-specific knowledge to interpret historical patterns and forecast events, and it demands rigorous verification to avoid future information leakage. This paper introduces a comprehensive framework that automates the entire financial deep research pipeline, from data construction to reward modeling, and pairs it with a benchmark designed to be point-in-time and verifiable.

The significance is twofold. First, it provides a practical harness that can drive finance-oriented tools and workflows, making autonomous financial research more accessible. Second, it introduces FinanceGym, a benchmark that challenges even leading LLMs and agents, with scores below 40%, highlighting the difficulty of the task and the headroom for improvement. This combination of a robust framework and a rigorous benchmark is a step forward for applied AI in finance.

Technical Contributions

• Layered Harness Architecture: FinanceHarness integrates environment and data construction, agent execution loop, and reward modeling into a cohesive system, enabling end-to-end automation.
• FinanceGym Benchmark: Introduces thesis-driven research questions and rubrics that combine pre-cutoff and post-cutoff criteria, ensuring that models are evaluated on both historical analysis and forward-looking predictions without leakage.
• Point-in-Time Verification: The benchmark is designed to prevent future information leakage, a critical requirement for trustworthy financial research.
• Practitioner-Guided Workflows: The harness incorporates workflows guided by financial practitioners, aligning automation with real-world research practices.
• Open-Weight Backbone Integration: Demonstrates that the harness can be used with open-weight models, making it accessible for further research and development.

Results

The paper reports that professional expert validation yields an 82% pass rate on FinanceGym, indicating that the benchmark is solvable by human experts. In contrast, leading LLMs and agents score below 40%, showing that current AI systems have substantial room for improvement. With the same open-weight backbone, FinanceHarness improves the overall rubric score from 25.3% to 32.4%, a relative improvement of about 28%. This improvement is notable but still far from expert performance, underscoring the challenge of financial deep research.

Significance

FinanceHarness and FinanceGym have broader implications for the AI field. They set a new standard for domain-specific deep research benchmarks, emphasizing the importance of point-in-time evaluation to prevent data leakage. The framework's modular design could be adapted to other specialized domains, such as legal or medical research, where similar verification needs exist. By providing an open-source implementation, the authors enable further research and collaboration, potentially accelerating progress in autonomous financial analysis and beyond.