Evaluation Benchmarking Framework for Token-Optimized RAG

# Evaluation  Benchmarking Framework for Token-Optimized RAG

Evaluation & Benchmarking Framework for TokenOptimized RAG
Executive Summary
This document proposes a comprehensive evaluation and benchmarking framework for LLMC’s RetrievalAugmented Generation (RAG) pipeline, focusing on measuring the cost vs. quality trade-offs introduced
by recent token optimization techniques (compression, caching, tiered LLM routing). The goal is to provide
an automated harness that can quantitatively validate improvements (or catch regressions) in retrieval
accuracy and token efficiency. Key components of the design include a golden dataset of queries with
known relevant code spans and answers, a suite of retrieval quality metrics (Precision@K, Recall@K,
NDCG, MRR) 1 2 , and token efficiency metrics (e.g. tokens per answer, cost per query, context waste).
The framework will be built around pytest-based test suites and a command-line tool ( llmc eval ) to
run controlled experiments on different model tiers (local Qwen 7B vs 14B vs GPT-5-nano) under identical
conditions. Automated nightly evaluation runs (via GitHub Actions) will generate reports with
visualizations and flag any significant performance regressions (e.g. >10% drop in precision or >10%
increase in token usage). By combining rigorous information retrieval metrics with cost accounting, this
evaluation harness enables data-driven decisions on routing strategies and ensures that token
optimizations deliver real improvements in RAG quality per dollar.

Architecture Design
The evaluation framework is designed as a modular pipeline that interfaces with LLMC’s existing RAG
system and logging. It consists of several coordinated components:
• Test Harness & Runner: A Python-based harness (invoked via CLI or pytest) that orchestrates
running a set of standardized queries through the RAG pipeline. It ensures each query’s execution is
deterministic and instrumented – forcing specific model tiers when needed, capturing metrics
from logs, and controlling for randomness. The harness supports running in different modes (e.g.
local-only, full tiered routing) to compare strategies A vs B under identical conditions.
• Metrics Computation Engine: A library of metric functions to compute retrieval accuracy and
efficiency stats from the pipeline outputs. This engine parses the JSONL logs (e.g.
planner_metrics.jsonl and enrichment_metrics.jsonl ) to extract relevant fields
(retrieved span IDs, planner confidence, tokens used, etc.), then computes metrics like Precision@K
and Recall@K by comparing retrieved spans against the golden truth 3 4 . It also computes
aggregate token usage and cost metrics per query (using known token pricing for each tier).
• Golden Dataset Manager: A module responsible for loading and validating the curated set of test
queries (“golden” queries) along with their expected relevant spans and answers. It ensures the
dataset conforms to a fixed schema (see Golden Dataset Schema below) and that any code
references (file paths, symbol names) are up-to-date with the repository.
• Result Analyzer & Regressions: A comparison module that takes metrics from current runs and
optionally from a baseline (previous run or known good commit) to identify regressions. It can

1

perform statistical significance checks or simple threshold comparisons to decide if a difference in
metrics is likely noise or a real degradation. For example, if Precision@5 drops by >10% or average
tokens per query increases by >10%, the harness will mark it as a regression requiring attention.
• Reporting & Visualization: After computing metrics, the framework generates a human-readable
report (as Markdown or HTML). This report includes tabular summaries of metrics, trend graphs
over time (if historical data is stored), and specialized charts like cost vs. quality Pareto frontiers
(illustrating how different model tiers trade off answer quality for token cost) and tier usage
breakdowns (what percentage of queries were handled by local vs API vs Codex). These help
stakeholders quickly understand the impact of any changes.
Overall, the architecture cleanly separates data collection (via logs and test runs) from analysis (metrics
calculations) and presentation (reports), making it easy to extend. The harness will integrate with LLMC’s
existing CLI and logging: it uses the same pipeline invocation ( llmc CLI or underlying Python calls) and
reads the same JSONL log format to remain fully compatible with existing telemetry (append-only logs).
This modular design also allows re-running the evaluation on historical log files if needed, or plugging in
different metric computations (for example, integrating emerging RAG metrics like “faithfulness” from
RAGAS or “groundedness” from TruLens in the future 5 6 ).

Test Harness and Workflow
The test harness can be invoked either as a command-line tool ( llmc eval ) or as a pytest suite for CI
automation. Internally, it uses pytest fixtures to spin up any required services (e.g. ensuring the local
Chroma vector DB is populated, loading the Ollama Qwen models into memory) and to initialize log
capture. Each query from the golden set runs as a sub-test. The harness ensures that for each test query: 1.
The RAG planner is run to select a route/tier (unless a specific tier is being forced for an experiment), and
the retriever obtains top-K code spans from the knowledge base. We capture the planner’s chosen route
and confidence from planner_metrics.jsonl (fields: confidence , span_count , top_span etc.).
2. The enrichment step is executed – i.e., an LLM is prompted with the query and retrieved spans.
Depending on the mode, this could be forced to a particular model (e.g. always local 7B for a baseline run)
or use the normal smart routing. The enrichment_metrics.jsonl log captures which tier was used
( tier_used ), how many input tokens were sent ( tokens_in ), output tokens ( tokens_out ), and the
result status. 3. After each query, the harness collects the relevant log entries and resets any state for the
next query (to avoid cross-talk or cache interference).
Using pytest for the harness provides rich reporting and easy integration with CI. The harness will mark
tests as failed if certain quality criteria are not met (for instance, if a relevant span was missed entirely – low
recall – or if an answer is empty), and will record metrics for all queries regardless of pass/fail for aggregate
analysis.

Metrics Computation Pipeline
Once a batch of queries is executed, the framework computes metrics in two categories: RAG Quality and
Token Efficiency. All metric calculations are implemented in a standalone metrics.py module for clarity.
• Precision@K and Recall@K: Using the golden dataset’s list of expected relevant spans, the top-K spans
returned by the system are evaluated. Precision@K is the fraction of the retrieved top-K that are
relevant 3 . Recall@K is the fraction of all relevant spans that were retrieved in the top-K 4 . These

2

metrics are computed per query and averaged across the dataset. For example, if a query had 3
relevant code snippets and the system’s top-5 retrieval included 2 of them, Recall@5 = 0.67 while
Precision@5 = 0.4 for that query. We will typically use K corresponding to the number of spans the
system injects (e.g. K=3 or 5).
• Mean Reciprocal Rank (MRR): This is an order-aware metric focusing on the rank of the first relevant
result 2 . It computes the reciprocal of the rank position of the first correct span, averaged over all
queries. An MRR of 1.0 means the top result was always relevant 7 . This helps measure how well
the system prioritizes the most relevant context at rank 1.
• Normalized Discounted Cumulative Gain (NDCG@K): NDCG measures the quality of the ranking of
results, taking into account graded relevance and position 8 9 . In our case, we can treat each
span as either relevant or not (binary relevance), or assign a relevance score from the golden dataset
if available (e.g. if some spans are more relevant than others). NDCG@K will be computed to gauge
how well the system orders the spans – a higher NDCG means relevant spans are ranked high, close
to the ideal ordering 10 11 . We normalize DCG to account for varying number of results per query,
so scores are comparable across queries 12 11 .
• Tokens per Accurate Answer: For token efficiency, we define a metric of tokens consumed per correct
result. If a query’s answer is deemed accurate (e.g. the expected span was retrieved and used), we
record the total tokens (input + output) spent to get that answer. Averaged across the test set, this
tells us the token cost of each successful Q&A. A lower value is better – meaning the system uses
fewer tokens to get to a correct answer.
• Cost per Query: Using a cost model for each tier (e.g. $0.00 for local, $0.002 per 1K tokens for API,
$0.01 per 1K for Codex – actual rates to be defined), the harness estimates the dollar cost of each
query’s processing. It sums (tokens_in + tokens_out) * cost_per_token for the tier
actually used. We then report average cost per query and the distribution. This metric, combined
with quality metrics, allows plotting a Pareto frontier of cost vs quality. For instance, we might find
that using only local models yields $0 cost but lower recall, whereas using the API tier increases
recall by 5% at a marginal cost of $0.001 per query – such trade-offs will be made explicit by our eval
reports.
• Context Waste Ratio: This measures how much retrieved context was unnecessary. We can define it as
irrelevant tokens / total context tokens in the prompt. Using the golden dataset’s expected spans
as ground truth, any retrieved span that is not in the expected set is considered irrelevant
(potentially a false positive). We know how many tokens each span contributed (we can count
characters or tokens of those files). So for each query, if the system retrieved N spans totaling
T_tokens, and out of those, R_tokens were relevant (from spans that should have been retrieved), the
waste ratio = (T_tokens - R_tokens) / T_tokens. This helps quantify whether compression or better
retrieval filtering is reducing the inclusion of useless context. A lower waste ratio is better (more of
the prompt context was actually useful).
• Tier Efficiency & Escalation Rate: We track what percentage of queries were answered by each tier. Tier
efficiency is basically the routing distribution – e.g. 80% handled by local 7B, 15% escalated to API,
5% to Codex. A healthy trend might show more queries staying on cheaper tiers if optimizations are
working. Escalation rate specifically measures how often the system had to “retry” a query on a
higher tier after a lower-tier attempt failed or was deemed low-confidence. We can detect this by
seeing if a query ID appears multiple times in enrichment_metrics.jsonl with increasing
tier_used, or via a flag in the log result field (e.g. result: escalated ). This metric directly reflects
the planner’s accuracy: a high escalation rate means the router often underestimates difficulty
(tries local but then needs API, etc.).

3

• Confidence Calibration: The planner’s confidence score (as logged in planner_metrics) should
correlate with success. We will measure calibration by grouping queries into confidence buckets (e.g.
0-0.3, 0.3-0.6, 0.6-1.0) and computing the actual success rate in each bucket. If the planner is wellcalibrated, a group of queries with ~80% confidence should indeed have about 80% success (e.g.
retrieved the needed spans or answered correctly). We can use plots or a simple calibration error
metric. Additionally, we report the overall correlation between confidence and outcome. This helps
identify if the confidence scoring needs adjustment.
The metrics engine will output a structured summary (JSON or Python dict) of all these computed metrics
for use in reports. Notably, many of these metrics align with those used in industry frameworks: e.g. Arize’s
RAG monitoring focuses on Precision/Recall/F1 13 , ARES emphasizes MRR/NDCG 14 , and RAGAS includes
context relevancy and answer faithfulness 5 . By implementing Precision, Recall, MRR, NDCG, etc., we
ensure our evaluations are in line with accepted standards while tailoring additional metrics (token usage,
cost) to LLMC’s needs.

Golden Dataset Creation Methodology
Building a golden dataset is critical, as it serves as the ground truth for evaluation. Our approach to
curating and maintaining this dataset involves both expert knowledge and automation:
1. Representative Query Selection: We will gather a set of queries that reflect the real usage patterns
of LLMC. This includes common developer questions (e.g. “How does authentication work?”), edge
cases, and recent incidents where the system’s performance was suboptimal. Logs of actual user
queries can be mined for frequent or challenging queries. We aim for a dataset size that is large
enough to be representative (perhaps 50-100 queries) but small enough for manual verification.
2. Manual Annotation of Relevant Spans: For each query, domain experts (e.g. developers familiar
with the codebase) will identify which code snippets or documentation sections are relevant to
answering the question. This yields the list of expected_spans for that query. Each span entry
includes a stable identifier (could be a hash of the content or a reference like file path + function
name) and an optional relevance score. Initially, we can use binary relevance (relevant or not), but if
some spans are “highly relevant” vs “somewhat relevant,” we can encode that in
relevance_score . Important: We also record the context that should answer the question – if
the question is fact-based, the ground_truth_answer is provided for reference (this can be used for
evaluating answer correctness qualitatively, though our automated metrics focus on retrieval).
3. Diversity and Difficulty Tags: Each query in the golden set is tagged with metadata like topic area
( auth , database , UI etc.), difficulty ( easy , medium , hard ), and any special characteristics
(e.g. requires multi-hop reasoning, requires combining multiple files). This allows segmenting
results. For example, we might find that “hard” queries have lower precision, indicating room for
improvement in those cases.
4. Validation of Dataset: We will create a small script to verify that all expected_spans actually
exist in the current codebase (to avoid stale references). This script can be part of a pre-evaluation
step, warning if any span content changed significantly (in which case the expected answer might
change). The dataset file (likely a JSON or YAML like golden_queries.json ) will be under version
control so that updates are tracked via pull requests. Changes to the golden set (like adding a new
tricky query) should undergo review, since they effectively change the evaluation criteria.
5. Ongoing Updates: As the codebase evolves and new features are added, we’ll periodically update
the golden set. One methodology is to require that any major feature PR must contribute at least

4

one new golden query relevant to that feature. This way, the evaluation stays current. We will also
use the evaluation results themselves to identify gaps: for instance, if a certain type of question
consistently has poor recall, we may add more queries of that type (or refine expected spans) to
better cover that failure mode in testing.
The golden dataset schema will follow the structure given in the prompt, for example:

{
"query_id": "123e4567-e89b-12d3-a456-426614174000",
"query": "How does authentication work?",
"expected_spans": [
{
"span_hash": "sha256:abcd1234...",
"path": "auth/login.py",
"symbol": "authenticate_user",
"relevance_score": 1.0
},
{
"span_hash": "sha256:efgh5678...",
"path": "auth/token_service.py",
"symbol": "generate_jwt",
"relevance_score": 0.8
}
],
"ground_truth_answer": "Authentication uses JWT tokens issued at login. The
process is: ...",
"tags": ["auth", "security", "critical"],
"difficulty": "medium"
}
Each query gets a UUID for tracking. The expected spans list can include multiple relevant code pieces; in
the above example two functions are relevant, with one being absolutely critical ( relevance_score=1.0 )
and another somewhat relevant. The harness will treat any span with relevance_score > 0 as a true
positive for binary Precision/Recall, and could use the scores for weighted metrics like DCG. The
ground_truth_answer provides context for human evaluators (and could be used in future for automatic
answer checking via LLMs, though that’s outside our current scope due to avoiding external dependencies).
By having this curated dataset, we can perform consistent, reproducible evaluations. It effectively acts as
unit tests for the RAG system – if a code change or optimization causes a drop in performance on these
known queries, the harness will catch it. Over time, we’ll expand and refine the golden set to improve
coverage of LLMC’s capabilities.

Evaluation Harness Implementation
This section details how the framework will be implemented in code and integrated with the existing
system.

5

Language & Tools: We will use pure Python (for metrics and orchestration) and pytest for the test harness.
No external evaluation libraries are required beyond standard Python and possibly matplotlib or similar
for plotting in reports. The choice of pytest is motivated by simplicity and familiarity – developers can run
pytest -k eval locally to execute the eval suite, and CI can run it nightly. We will also implement a CLI
wrapper (as part of the llmc CLI tool) so that non-dev users can run llmc eval to generate a report
without dealing with pytest options.
Integration with LLMC Pipeline: The harness will likely call into existing scripts or modules to execute
queries. Since LLMC has a CLI ( llm_gateway.sh and possibly Python APIs in scripts/rag/* ), we have
options: - Use the LLM Gateway interface: e.g., call llm_gateway.js or codex_wrap.sh with flags to
simulate user queries. However, for fine-grained control (like forcing a particular tier or capturing outputs
programmatically), it’s better to call lower-level Python functions if available. - Leverage internal classes: If
the RAG system has classes like ContextQuerier or functions to retrieve spans (as seen in
query_context.py ), we can import and use them directly. Similarly, if there is a function to generate an
answer given context (the LLM call), that can be invoked directly with the model specified. - Alternatively,
use subprocess calls to the CLI for realism. But that complicates capturing intermediate data. Instead, we
might modify the code to log more details (e.g. log all retrieved span IDs in planner_metrics.jsonl ) so
we can rely on logs.
Given the need to measure retrieval metrics, we will ensure that during an eval run, all top-K retrieved
span identifiers are recorded. If the current logging only stores the top span and a count, we might extend
it to store a list of top span hashes. This could be added in a backward-compatible way (e.g. a new key
span_ids: [list,…] in planner_metrics.jsonl ). The harness can then read this after each query
to know exactly what was retrieved. If code changes are undesirable for now, an alternate approach is to
instrument the retrieval function in the harness (monkey-patch it to capture outputs). For the initial
implementation, a simple route is to parse the vector DB results directly by calling the ContextQuerier –
essentially duplicating what the system does, but in the test harness context where we can intercept the
results.
Running Evaluations: By default, llmc eval will run the entire golden suite with smart routing
enabled (to mimic real behavior). However, to support A/B testing and multi-model comparisons, the CLI
will allow parameters: - --tier local or --tier api or --tier codex to force all queries to use a
single tier model. This is useful to gather baseline metrics for each model (e.g. run all queries with Qwen-7B
vs with Qwen-14B vs with GPT-5-nano) for comparison. - --no-cache to disable any caching mechanism
(when caching is implemented) to ensure we measure fresh LLM calls. - --compare <config1.json>
<config2.json> to run two different configurations back-to-back and produce a comparative report. A
config JSON might specify settings like model, compression on/off, number of retrieved spans, etc., which
allows defining two experiment conditions. The harness would run the golden queries under each and then
use statistical tests to compare (details in A/B section). - --subset <tag> to run only queries matching a
certain tag or pattern (for faster debugging or focusing on a category of queries).
The harness will utilize existing logging as much as possible. After running, it reads

logs/

planner_metrics.jsonl and logs/enrichment_metrics.jsonl (which will have appended entries
for each query run) to extract the data. To isolate evaluation runs (so as not to mix with production logs),
the harness could direct the system to log to a separate location (e.g. set an env var to use logs/

6

eval_planner_metrics.jsonl for that run). Another approach is to timestamp each run and only read
log lines within that time window.
Performance and Cost Control: The implementation is mindful that evaluation should be efficient and
cheap. By using local Qwen models for most runs, we avoid incurring external API costs. We will likely
configure the router’s LLM (the one that makes routing decisions) to use a very cheap model or a local
heuristic for eval – or even bypass it entirely by deterministically routing in experiments (e.g. in a forced
local run, skip calling the router LLM). This not only saves cost but also makes eval results more
deterministic. If GPT-5-nano (or “Gemini 2.0 Flash” API) must be included, we will keep the query set
relatively small to cap the cost; moreover, the nightly CI could exclude the expensive model by default (only
run GPT-5-nano experiments on demand or weekly). In all cases, the cost per eval run is expected to be a
few cents at most (and <$1/month even with daily runs), satisfying the budget constraint.
Validation: We will write unit tests for the metrics functions (e.g., feed a known set of retrieved vs expected
spans and verify Precision@K is calculated correctly), and for the golden dataset loader (to ensure schema
compliance). These tests will reside in tests/eval/ and can be run as part of CI to ensure the evaluation
framework itself is reliable.
Finally, the harness will integrate with the existing workflow by adding a GitHub Actions job (see below)
and by including documentation (in README or a DOCS/ page) so developers know how to run it. The overall
implementation plan is greenfield (built from scratch) but informed by existing tools and patterns. For
instance, frameworks like LangSmith and RAGAS emphasize the importance of having both retrieval
metrics and answer quality metrics 15 5 ; while we are not using those frameworks directly (to avoid
external dependencies), our design takes inspiration from them and could integrate with them in the future
if desired. In summary, the harness will slot into LLMC’s toolkit as a natural extension, allowing continuous,
automated evaluation of the RAG pipeline’s efficacy.

A/B Testing Framework
A core objective is not only to measure absolute performance but also to facilitate A/B comparisons – for
example, comparing the current system with and without a new compression algorithm, or evaluating two
routing strategies. The evaluation framework supports this through both configuration and methodology:
• Configurable Experiments: As mentioned, the CLI can accept parameters or a config file to define a
run. We can represent an experiment as a set of system settings (e.g. {"compression": true,
"cache": false, "max_spans": 5, "router_model": "GPT-5-nano"} ) and a label. The
harness can run two (or more) configurations sequentially on the same golden queries and collect
metrics for each. All else being equal, differences in metrics can be attributed to the changed
setting.
• Isolation and Repeatability: Each experiment run will be self-contained. For instance, if comparing
caching vs no caching, we ensure that in the “no cache” condition, the system isn’t accidentally
benefiting from cache warmed in the other run. This might mean running experiments in separate
processes or clearing any in-memory cache between them. The harness will handle that (maybe by
restarting the context or using separate output log files).
• Statistical Significance: When comparing two strategies, we will compute the paired difference in
metrics per query where applicable. For example, for each query we can look at whether config A

7

returned the correct span and config B did, giving a set of outcomes. Statistical tests like McNemar’s
test (for binary success/failure comparisons) or a paired t-test (for metrics like tokens used per
query) can be applied to assess if the differences are significant at, say, p<0.05. Given our sample
size may be modest, we will be cautious and also rely on magnitude of improvement. For instance,
an increase of +5% Precision might not be significant on 50 queries, but +20% likely is. We will
highlight any differences beyond a certain threshold (like ±10%) as potentially meaningful even if
statistically borderline.
• Experiment Tracking: Each time an A/B test is run (especially in CI), we will record the results. This
could be as simple as committing a JSON summary of metrics for each config to a results directory,
or storing it as an artifact in CI. Over time, this builds a history of experiments. We can automate
comparison against the last main-branch run as the “B” in an A/B test for every pull request: i.e.,
treat the new code as A, baseline as B, and report if A is better or worse.
• Router Strategy Comparison: One specific use-case is to evaluate different routing LLM prompts
or thresholds. For example, maybe we have a new prompt for the routing LLM that is supposed to
better classify tasks. We can run one set where the router uses the old prompt, and one with the new
(this might require a slight interface to plug in different router behaviors – possibly by mocking the
router’s output or by actually using two different routing models). We then compare tier usage,
success rates, and cost. If the new strategy successfully keeps more queries on the cheaper local tier
without loss of quality, the metrics will show improved cost per query and similar precision – an ideal
outcome. If it under-routes (too many locals leading to missed answers), the recall/precision will
drop, which the harness would flag.
• Multiple Configs (A/B/C): While A/B usually implies two, our framework can generalize to multiple
variants if needed (though we must be careful to control variables). For example, we could compare
Qwen-7B vs Qwen-14B vs GPT-5-nano in one go (that’s effectively multi-model benchmarking). The
output would list metrics for each. In terms of significance, we’d likely do pairwise comparisons or
simply present the numbers and perhaps recommend an optimal choice (e.g. “GPT-5-nano gave
highest recall but at 2x cost of Qwen-14B – depending on priorities, one might choose differently”).
To implement A/B in code, the harness.py could have a function like run_experiment(config) ->
metrics and then a small loop to run for each config in a list. The results can then be passed to a
reports.compare_results(metrics_A, metrics_B) function which produces a diff report or overlay
charts.
The framework will also integrate with version control or CI in a basic way: for example, if a developer
pushes a branch with a proposed optimization, the nightly eval on that branch (or a PR-triggered eval) could
automatically compare its metrics to the last nightly on the main branch, and perhaps comment on the PR
with the comparison. This encourages data-driven merges (if the new code regresses metrics, we’d think
twice unless there’s a known trade-off).
In summary, the A/B testing capabilities ensure that we can experimentally validate each token optimization
technique’s impact. Instead of guessing, we can concretely say e.g. “Cache X reduced average tokens by
20% with no loss in Precision@3 13 ” or “The new compression scheme improved context waste ratio from
30% to 10%, boosting recall by 5 points.” Such statements will be backed by the automated eval data.

8

Regression Detection
One of the primary motivations for this framework is to catch regressions early, ideally before they hit
production. The design includes multiple layers of regression detection:
• Threshold-based Alerts: We will define clear thresholds for critical metrics. For example:
Precision@5 should not drop more than 5% relative to the baseline; average tokens per query should
not increase more than 10%; planner confidence calibration error should not worsen significantly.
These thresholds can be encoded as assertions in the test suite. For instance, after computing
metrics, a pytest test could assert assert current_metrics['Precision@5'] >= 0.95 *
baseline_metrics['Precision@5'] (if baseline is known). If this assertion fails, the CI run fails,
immediately flagging the regression.
• Baseline Management: We need a point of reference for regression checks. Initially, we might use a
static baseline (like metrics from the last release or a manually curated ideal). However, since the
system will hopefully improve over time, we likely treat the latest main branch metrics as the
baseline for new changes. We can store the last nightly main metrics in a JSON file (committed to the
repo or accessible to CI) and have the test harness load it when running on a feature branch. This
way, each run knows what “previous performance” was. In cases where improvements occur, we
update the baseline accordingly.
• Adaptive Gates: Some metrics might improve and we want to lock in that improvement. For
instance, if a new feature raises Precision@5 from 60% to 70%, we don’t want later changes to
silently slip back to 60%. We would adjust the baseline or threshold upward. A process will be
defined for updating these guardrails – likely via code review: e.g., when a PR intentionally improves
a metric, the PR can also update the expected baseline values in the test assertions.
• Automated Notifications: Failing the CI test is one mechanism, but we can make it more visible.
The GitHub Actions workflow can be configured to post a comment on the pull request or send a
Slack notification if an eval failure occurs on main. For nightly runs, if a regression is found, we could
have the action create a GitHub Issue or at least output a summary in the logs. Because we parse
metrics in a structured way, the workflow can even include a snippet like “ Regression detected:
Recall@10 dropped from 85% to seventy 75% (threshold 77%) on commit abc123” for quick visibility.
• Historical Trend Tracking: In addition to point-in-time regression checks, we’ll maintain a history of
metrics over time (could be simply a CSV or JSON appended each night). This allows trend analysis –
e.g., if answer latency or token usage has been creeping up over two weeks, we might want to
address it before it crosses a hard threshold. A simple script can plot metric values over date; more
advanced, we could integrate with a dashboard or monitoring tool if available. But even embedding
a chart of the last N runs in the report can help visually catch slow regressions that single-run
thresholds might miss.
• Zero-Tolerance Checks: Some failures are binary – e.g., if any golden query that used to succeed
now fails to retrieve the needed span, that’s a regression. We will treat any previously passing query
now failing as a high-severity regression (unless the golden set was updated). The harness can track
per-query outcomes and compare with previous outcomes. If a query’s Precision@K was 1 (it found
the answer) and now it’s 0, that’s a clear red flag. This is essentially a unit-test-like behavior: each
golden query ideally always passes unless intentionally changed.
• Controlled Environment: To avoid false alarms, we ensure the eval environment is consistent. The
vector DB should be the same snapshot of code for each run (the CI can rebuild the index nightly
from the repo). Randomness in LLM outputs could cause variance, but since we use local
deterministic models or set random seeds, the output should be stable. If using any non-

9

deterministic component (like an external model), we might run multiple trials or use moderate
thresholds to account for variance.
By combining these strategies, the framework will act as an early warning system. A developer will be
alerted within 24 hours (or immediately on PR CI) if their change negatively impacted the RAG pipeline’s
performance. This encourages quick iteration to fix the issue or to adjust the system. Moreover, having a
documented baseline of expected performance means we’re effectively putting quality gates in place –
LLMC’s pipeline cannot drift beyond certain bounds without detection.

Reporting & Visualization
The final output of the evaluation harness is a report that synthesizes all the findings for easy consumption
by engineers and product stakeholders. We outline the reporting features and format:
• Metrics Summary Tables: The report will start with a table of key metrics for the latest run (and
possibly a comparison to baseline). For example:
Metric

Current Run

Baseline (Last week)

Change

Precision@5

0.68

0.65

+0.03

Recall@5

0.50

0.48

+0.02

NDCG@5

0.72

0.70

+0.02

MRR

0.60

0.58

+0.02

Avg Tokens per Query

1200

1300

-100

Cost per Query (USD)

$0.00050

$0.00055

-9%

Context Waste Ratio

15%

25%

-10%

Tier1/2/3 Usage (% Qs)

80 / 15 / 5

75 / 20 / 5

-

Escalation Rate (% Qs)

5%

8%

-3pp

The table highlights improvements or regressions (e.g. bold for notable changes). This gives a quick at-aglance view.
• Detailed Breakdowns: Subsequent sections of the report dive deeper. For instance, a section on
“Retrieval Quality” will list the Precision and Recall at various K (maybe K=1,3,5,10) and could even list
query-by-query results (or at least the hardest queries). If certain queries had zero recall (missed
completely), we can list them here under a “Failures” subsection for inspection.
• Visual Plots: We will include a few key charts:
• Precision/Recall Curve: A line chart showing Precision@K vs Recall@K as K increases (like an IR
trade-off curve). This can illustrate how performance changes if we allowed more context (e.g.,
Precision drops as we include more spans, but recall increases). It helps decide an optimal K for
retrieval.
• Cost-Quality Frontier: Perhaps the most insightful – we can plot points representing different
configurations (7B vs 14B vs GPT-5-nano, or routing on vs off) on a graph of Cost (x-axis) vs Quality

10

(y-axis, say Precision or Recall). Ideally, our optimizations move the point towards bottom-right
(lower cost, higher quality). The Pareto frontier would connect the non-dominated configurations
14 . For example, one point might be (cost $0, Precision 0.6) for all-local, another (cost $0.0005,
Precision 0.7) for hybrid, another (cost $0.002, Precision 0.75) for all-codex. This visualization helps
justify if a more expensive model is worth the marginal gain in quality.
• Tier Usage Pie/Bar: A pie chart or bar graph showing what fraction of queries went to each tier
(Local, API, Codex). If we run multi-model comparisons, perhaps a stacked bar per model showing
how many queries were answered correctly vs needed escalation. This ties to the “Tier efficiency”
metric but in visual form.
• Confidence Calibration Plot: If feasible, a plot with confidence on x-axis and actual success
probability on y-axis (perhaps with error bars). A perfectly calibrated planner would lie on the
diagonal line y=x. This can reveal if the planner is overconfident or underconfident on average.
• Trend Graphs: If we have historical data, a time series line chart of a few metrics (Precision, cost,
etc.) over past weeks. This can be placed in an appendix or the online dashboard, but even a simple
sparkline in the report can be useful to see trajectory.
• Findings and Recommendations: The report will also include short textual analysis highlighting
notable results. E.g., “Finding: The new compression algorithm reduced context waste from 25% to
15%, leading to a slight increase in recall. However, precision remained roughly constant, indicating
the extra context was mostly relevant. Recommendation: Keep compression enabled and consider
increasing K to leverage freed token budget for possibly more recall.” This kind of commentary can
guide the team on next steps.
• Drill-down for Errors: For any queries where the system failed (didn’t retrieve something it should
have, or gave a wrong answer where we expected a certain answer), we provide a brief case study.
For example: Query: “How are JWT tokens generated?” – Expected span: auth/
token_service.py::generate_jwt – Retrieved: None in top 5. This indicates a recall failure;
maybe the embedding missed it. Such granular detail can be extremely useful for debugging (maybe
it turns out the code had a different terminology, etc.). We might limit this to the top N failures to
keep report concise.
• Format and Access: The primary format will be Markdown (for easy viewing in GitHub) and possibly
an HTML artifact for the CI. If using Markdown, we can include the charts as embedded images (the
GitHub Action can upload the chart images and embed links). Since the instructions mention
embedding images only if we have them, we will generate charts as PNGs and reference them
appropriately in the Markdown report. The report could be committed to a
DOCS/eval_report.md or posted as a PR comment.
• Automation: The nightly GitHub Action ( nightly_eval.yml ) will run llmc eval , collect the
output report and metrics, and perhaps push the report to a branch or store as an artifact. It can
also use the GitHub API to comment on any recent commits if regressions are found. Over time, we
might integrate this with Slack or email for critical alerts, but that’s optional.
The emphasis is on making the results easily interpretable. We want to lower the barrier for developers to
understand how a code change affected the RAG system. The combination of numeric metrics and visual
aids will cater to both detailed analysis and quick high-level overview.
In conclusion, this evaluation and benchmarking framework will enable continuous improvement of LLMC’s
RAG pipeline by providing clear, quantifiable feedback on every optimization. It ensures that token usage
optimizations truly pay off in either reduced costs or improved quality (or ideally both), and that any
degradation in performance is caught and addressed promptly. By drawing inspiration from existing best-

11

in-class RAG evaluation tools while tailoring to our specific system, we get the best of both worlds: a
custom, lightweight solution with rigorous metrics 5 6 .

Code Structure & Workflow Integration
Following the design above, below is the proposed code layout for implementing the evaluation harness
within the repository:

scripts/eval/
├── __init__.py
├── golden_dataset.py

# Loading and validating the golden dataset JSON/YAML

├── metrics.py
MRR, etc.)

# Metric computations (Precision@K, Recall@K, NDCG,

├── harness.py

# Core harness logic: running queries, controlling

tiers, collecting logs
├── regression.py
detecting regressions

# Functions for comparing metrics to baseline and

├── reports.py
assembly)

# Report generation (tables, charts, Markdown

└── cli.py
or argparse

# `llmc eval` CLI entry point, integrates with Click

tests/eval/
├── test_metrics.py
dummy data)

# Unit tests for metrics calculations (using small

├── test_golden_dataset.py

# Tests for golden dataset schema and loading

├── test_harness.py
# Tests for harness end-to-end on a tiny sample
dataset (possibly using a stub model)
└── test_confidence_calibration.py # Tests specifically for confidence
calibration logic (could simulate planner outputs)
.github/workflows/
└── nightly_eval.yml
# GitHub Actions workflow that triggers nightly (or
on demand) to run the eval and publish results
Module Descriptions: - golden_dataset.py : Contains the GoldenDataset class or methods to read
the golden dataset file. It will verify required fields for each query and possibly provide helper methods to
filter queries by tag or difficulty. It may also hold the path to the dataset file (e.g. data/
golden_queries.json ).

-

metrics.py :

Implements

functions

like

precision_at_k(retrieved_list, expected_list, k) , recall_at_k(...) , ndcg_at_k(...) ,
etc., as well as token efficiency calculations. Likely also has a
golden_data)

compute_all_metrics(run_logs,

that orchestrates computing all metrics from a run’s logs or results in memory. -

harness.py : This is the heart of the evaluation runner. It might define a class EvalHarness that takes
configuration (which model tiers to allow, etc.) and has a method run_all(golden_data) -> results .
Internally it will: - Loop through each query, call the RAG system (perhaps via a helper function that

12

encapsulates the query workflow). - Capture outputs and log data for that query in memory (we might
structure it as a dictionary per query with fields like retrieved_spans , used_tier , tokens_used ,
success etc.). - Optionally, it could also directly compute per-query metrics on the fly. - Ensure any
needed cleanup (clear caches, etc.) per iteration. - regression.py : Provides utilities to load baseline
metrics (e.g. from a file or previous run) and compare with new metrics. This could include functions for
statistical tests or simple threshold checks. It could return a summary of any regressions found (to be
consumed by reports or CLI for exit codes). - reports.py : Handles assembling the final report. It might
use templates for Markdown. For plotting, it could use matplotlib or plotly to generate charts saved to
logs/ or a temp folder, then reference them in the Markdown. This module focuses on presentation:
formatting percentages, highlighting changes, sorting outputs by worst cases, etc. -

cli.py : Ties

everything together. Likely uses Python’s argparse to parse options (or integrates into an existing CLI
framework if llmc has one). It will load the golden dataset, instantiate the harness with given options, run
the evaluation, get metrics, perform regression checks, generate the report, and finally print a summary to
console (and write report to a file). It will also set the process exit code to indicate success or failure (which
CI can use to mark the run pass/fail).
Testing: Under tests/eval/ , we will write tests for each piece: - test_metrics.py : e.g., test that
precision_at_k yields 1.0 when all retrieved are relevant, or yields correct fractions in known scenarios.
Also test NDCG with a toy relevance list. - test_golden_dataset.py : create a sample JSON in a temp file
and ensure our loader catches missing fields or bad formats (like non-unique IDs). - test_harness.py :
This one might use a mock RAG pipeline to simulate responses, because running the actual LLMs in a unit
test isn’t feasible. We can monkey-patch the call that queries the LLM to instead return a canned answer
and likewise stub the vector search to return predetermined spans. This way we can simulate a scenario
and ensure the harness correctly logs and captures it. For example, simulate a query where the retriever
returns spans A and B, expected answer uses A – then verify the harness’s results indicate a success. test_confidence_calibration.py : Could test our bucketing logic or a synthetic scenario to ensure
the calibration metric is computed as expected.
Workflow (nightly_eval.yml): This GitHub Action will likely: - Check out the repo, set up environment
(perhaps needs to install dependencies like sentence-transformers for embedding if not vendorized, and
ensure the Qwen model weights are present or can be pulled). - Possibly start up any services (if vector DB
is needed, e.g., ensure Chroma index is built – though we might use an in-memory index for eval). - Run
python -m scripts.eval.cli (or the equivalent llmc eval command). - Archive the resulting
report (as an artifact or commit to a branch, depending on the desired approach). - If failures, mark the job
as failed. Also, use regression.py output to possibly add annotations or messages.
Because the evaluation should run in <5 minutes locally (as per success criteria), we ensure efficiency by
limiting to, say, 50 queries and using local models. On CI, we might run on self-hosted runners with a GPU
for Qwen, or use CPU with quantized models if needed (Qwen-7B might run on CPU moderately okay for
small usage). We can also utilize the smaller “Qwen 2.5B” if available for faster run, since it’s mainly retrieval
we care about; however, for fidelity, using the same models as production is ideal.
In essence, this code structure cleanly isolates the evaluation code from production code, while still
leveraging the production pipeline where possible. The introduction of this scripts/eval module does
not interfere with runtime logic (it’s an additive test suite), thus maintaining the integrity of the system. By
placing it under scripts/ , we signify it’s a first-class tool for the project (not just tests).

13

With this framework in place, LLMC’s team will gain a powerful feedback loop: every day, they’ll know
exactly how their RAG pipeline is performing and where to focus improvements, supported by empirical
data and clear visualizations. This closes the loop between development and validation, ensuring the RAG
system continually moves toward an optimal cost-quality equilibrium. 14 5

1

2

3

4

5

7

8

9

10

11

12

13

14

RAG Evaluation: Don’t let customers tell you first | Pinecone

https://www.pinecone.io/learn/series/vector-databases-in-production-for-busy-engineers/rag-evaluation/
6

How do RAG evaluators like Trulens actually work? : r/LangChain

https://www.reddit.com/r/LangChain/comments/1lvevo2/how_do_rag_evaluators_like_trulens_actually_work/
15

The 5 best RAG evaluation tools in 2025 - Articles - Braintrust

https://www.braintrust.dev/articles/best-rag-evaluation-tools

14