Architecturally Significant Requirements — Financial Document Analyst

ASR Catalog

ASR-01: Query Response Time

• Category: Performance
• Priority: High
• Requirement: End-to-end query response ≤ 5 seconds for single-turn Q&A
• Source: Assumption — interactive CLI use
• Architectural impact: Chunk size capped at 512 tokens; images described at ingest (zero query-time overhead); BM25 index built lazily and cached per company set; one structured-query LLM call covers HyDE + company/year filter + section routing simultaneously

ASR-02: Corpus Update Without Rebuild

• Category: Reliability / Maintainability
• Priority: High
• Requirement: Adding or removing a document must not require reprocessing the entire corpus
• Source: Task requirement (Ninja challenge)
• Architectural impact: ChromaDB supports per-document upsert/delete by doc_id; blue-green version swap via data/versions.json eliminates mixed-version windows; scripts/ingest.py --gc defers stale-chunk deletion to an explicit maintenance step

ASR-03: Access Control at Retrieval

• Category: Security
• Priority: High
• Requirement: Users may only retrieve chunks from documents they are authorized to access
• Source: Task requirement (Ninja challenge)
• Architectural impact: Every chunk indexed with metadata={"company", "role"}; retrieval applies $in where-filter on company scoped to the caller's allowed companies; pre-filter at vector DB level — no unauthorized content reaches the LLM

ASR-04: Answer Faithfulness

• Category: Quality
• Priority: High
• Requirement: Generated answers must be grounded in retrieved chunks; hallucinated claims are a failure mode
• Source: Task requirement; financial domain — incorrect numbers cause material harm
• Architectural impact: Prompt requires inline citation (DOC_NAME p.N); structured output enforces {answer, sources[]} schema; faithfulness measured via DeepEval LLM-judge (current: 0.956)

ASR-05: Retrieval Recall

• Category: Quality
• Priority: High
• Requirement: Recall@5 ≥ 0.65 on FinanceBench golden set — evidence page in top-5 retrieved chunks
• Source: Evaluation requirement; FinanceBench paper standard (k=5); evidence page numbers in dataset ground truth
• Architectural impact: Hybrid BM25+semantic search with RRF fusion; LLM section routing (income_statement / md_a / notes / etc.) as ChromaDB pre-filter; topical table summaries make dense numeric tables retrievable; LLM TOC classification assigns accurate section metadata at ingest; chunk overlap 50 tokens prevents boundary orphaning. Current result: 0.632 (target ≥ 0.65)

ASR-06: Multi-Modal Content Extraction

• Category: Quality
• Priority: Medium
• Requirement: Charts, graphs, and financial tables embedded in PDFs must be interpretable by the system
• Source: Task requirement (Ninja challenge)
• Architectural impact: PyMuPDF extracts images and tables at ingest; Azure OpenAI describes images as text chunks (content_type: image); Azure OpenAI generates topical prose summaries of tables without embedding numbers — enables semantic retrieval while raw table markdown stored in metadata.raw_table for answer generation

ASR-07: Evaluation Traceability

• Category: Observability
• Priority: Medium
• Requirement: Each response must include source citations (doc_name, page number); retrieved chunks must be attributable for quality measurement
• Source: Enables faithfulness scoring, retrieval recall measurement, and user trust
• Architectural impact: Structured output schema includes sources: list[Source]; ChainResult carries both the response and the raw retrieved docs; eval pipeline uses page citations for Recall@5 and passes chunk content to DeepEval LLM-judge metrics

ASR → Decision Mapping

ASR	Design Decisions
ASR-01	D-04 (image at ingest), D-06 (512-token chunks), D-07 (HyDE single call), D-08 (hybrid search: BM25 cached)
ASR-02	D-01 (ChromaDB), D-02 (blue-green version swap)
ASR-03	D-03 (metadata pre-filter)
ASR-04	D-05 (structured output + citations)
ASR-05	D-08 (hybrid BM25+semantic+RRF), D-09 (LLM section routing), D-10 (table summarization), D-11 (LLM TOC classification)
ASR-06	D-04 (image at ingest), D-10 (table summarization)
ASR-07	D-05 (structured output)