When "Better" Prompts Hurt: Evaluation-Driven Iteration for LLM Applications

Authors: Daniel Commey

Published: 2026-01-29 17:32:34

PDF URL: https://arxiv.org/pdf/2601.22025v1

Abstract

Evaluating Large Language Model (LLM) applications differs from traditional software testing because outputs are stochastic, high-dimensional, and sensitive to prompt and model changes. We present an evaluation-driven workflow - Define, Test, Diagnose, Fix - that turns these challenges into a repeatable engineering loop. We introduce the Minimum Viable Evaluation Suite (MVES), a tiered set of recommended evaluation components for (i) general LLM applications, (ii) retrieval-augmented generation (RAG), and (iii) agentic tool-use workflows. We also synthesize common evaluation methods (automated checks, human rubrics, and LLM-as-judge) and discuss known judge failure modes. In reproducible local experiments (Ollama; Llama 3 8B Instruct and Qwen 2.5 7B Instruct), we observe that a generic "improved" prompt template can trade off behaviors: on our small structured suites, extraction pass rate decreased from 100% to 90% and RAG compliance from 93.3% to 80% for Llama 3 when replacing task-specific prompts with generic rules, while instruction-following improved. These findings motivate evaluation-driven prompt iteration and careful claim calibration rather than universal prompt recipes. All test suites, harnesses, and results are included for reproducibility.

Full Text

<!-- PDF content starts -->

When “Better” Prompts Hurt: Evaluation-Driven Iteration for LLM Applications A Framework with Reproducible Local Experiments Daniel Commey January 2026 Abstract Evaluating Large Language Model (LLM) applications differs from traditional software testing because outputs are stochastic, high-dimensional, and sensitive to prompt and model changes. We present an evaluation-driven workflow—Define, Test, Diagnose, Fix—that turns these challenges into a repeatable engineering loop. We introduce the Minimum Viable Evaluation Suite (MVES), a tiered set of recom- mended evaluation components for (i) general LLM applications, (ii) retrieval-augmented generation (RAG), and (iii) agentic tool-use workflows. We also synthesize common eval- uation methods (automated checks, human rubrics, and LLM-as-judge) and discuss known judge failure modes. In reproducible local experiments (Ollama; Llama 3 8B Instruct and Qwen 2.5 7B In- struct), we observe that a generic “improved” prompt template can trade off behaviors: on our small structured suites, extraction pass rate decreased from 100% to 90% and RAG compliance from 93.3% to 80% for Llama 3 when replacing task-specific prompts with generic rules, while instruction-following improved. These findings motivate evaluation- driven prompt iteration and careful claim calibration rather than universal prompt recipes. All test suites, harnesses, and results are included for reproducibility. Keywords.Largelanguagemodels,evaluation,benchmarks,metrics,RAG,retrieval-augmented generation, LLM-as-judge, prompt engineering, regression testing, MVES. Contributions.This paper makes six contributions. First, we introduce theMVES frame- work: a tiered standard defining minimum evaluation requirements for general LLM ap- plications (MVES-Core), retrieval-augmented systems (MVES-RAG), and agentic workflows (MVES-Agentic). Second, we provide asynthesis of evaluation methodsfrom literature, discussing correlation with human judgment, cost per 1,000 examples, and execution time. Third, we present ataxonomy of quality dimensionsdistinguishing correctness, helpful- ness, harmlessness, groundedness, and format adherence. Fourth, we give adetailed analysis of LLM-as-judge failure modes, including position bias, verbosity bias, self-preference, style bias, and instruction leakage. Fifth, we provideactionable checklistsfor test set design, met- ric selection, human evaluation rubrics, and production monitoring. Sixth, we presentoriginal experimentsdemonstrating that task-specific prompts can outperform generic improvements on structured tasks, using local inference for full reproducibility. Contents 1 Introduction 5 1.1 Why LLM Evaluation Differs from Traditional Testing . . . . . . . . . . . . . . . 5 1arXiv:2601.22025v1 [cs.CL] 29 Jan 2026

1.2 Key Risks in LLM Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.3 What This Paper Provides . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2 The Evaluation Loop 6 2.1 The Core Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2 Translating Requirements into Tests . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.3 The Role of Golden Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.4 Iteration in Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.5 Connecting Offline and Online Evaluation . . . . . . . . . . . . . . . . . . . . . . 7 3 Quality Taxonomy 8 3.1 Correctness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.2 Helpfulness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.3 Harmlessness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.4 Groundedness and Attribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.5 Refusal Correctness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.6 Format and Style Adherence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.7 Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.8 Mapping Dimensions to Applications . . . . . . . . . . . . . . . . . . . . . . . . . 10 4 Evaluation Methods 10 4.1 Automated Offline Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 4.2 Human Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 4.3 Comparative Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 4.4 Minimum Viable Evaluation Suite (MVES) . . . . . . . . . . . . . . . . . . . . . 10 5 Test Set Design 11 5.1 Representativeness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 5.2 Edge Cases and Adversarial Prompts . . . . . . . . . . . . . . . . . . . . . . . . . 12 5.3 Systematic Coverage Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 5.4 Tutorial: The Extraction Evaluation Loop . . . . . . . . . . . . . . . . . . . . . . 12 5.5 Multi-Turn Conversation Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 5.6 Data Contamination Considerations . . . . . . . . . . . . . . . . . . . . . . . . . 13 5.7 Test Set Size and Statistical Power . . . . . . . . . . . . . . . . . . . . . . . . . . 13 5.8 Test Set Maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 6 Metrics and Scoring 14 6.1 Operational Definitions (This Paper) . . . . . . . . . . . . . . . . . . . . . . . . . 14 6.2 Semantic Similarity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 6.3 Factual Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 6.4 Truthfulness and Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 6.5 Inter-Rater Reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 7 RAG Evaluation 15 7.1 Decomposing RAG Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 7.2 The RAGAS Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 7.3 Retrieval Quality Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 7.4 The “Correct but Unsupported” Failure Mode . . . . . . . . . . . . . . . . . . . . 17 7.5 Citation Coverage and Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 7.6 End-to-End vs. Component Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 18 7.7 Tutorial: The RAG Evaluation Loop . . . . . . . . . . . . . . . . . . . . . . . . . 18 2

8 LLM-as-Judge 18 8.1 The Appeal of LLM Judges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 8.2 Known Biases and Failure Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 8.3 Good vs. Bad Judge Prompts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 8.4 Implementation Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 8.5 Multi-Judge Aggregation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 8.6 Comparison: LLM Judges vs. Human Evaluation . . . . . . . . . . . . . . . . . . 22 8.7 Protocol for Robust Judge Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 22 8.8 Recommended Guardrails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 9 Case Studies 23 9.1 Case Study 1: Customer Support Assistant . . . . . . . . . . . . . . . . . . . . . 23 9.2 Case Study 2: Internal Knowledge Base RAG Bot . . . . . . . . . . . . . . . . . 24 9.3 Case Study 3: Summarization Pipeline . . . . . . . . . . . . . . . . . . . . . . . . 25 9.4 Production Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 10 Common Failure Modes 25 10.1 Prompt Drift . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 10.2 Overfitting to the Test Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 10.3 Format Brittleness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 10.4 Silent Regressions After Model Updates . . . . . . . . . . . . . . . . . . . . . . . 26 10.5 Evaluation Metrics That Mislead . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 10.6 Insufficient Coverage of Failure Cases . . . . . . . . . . . . . . . . . . . . . . . . . 27 10.7 Tutorial: The Safety Evaluation Loop . . . . . . . . . . . . . . . . . . . . . . . . 27 11 Best Practices Summary 27 11.1 The Golden Rules of ML Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 27 11.2 Threshold Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 12 Experimental Demonstration: Evaluation-Driven Iteration 28 12.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 12.2 Prompt Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 12.3 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 12.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 12.5 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 12.6 Failure Categories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 12.7 Latency Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 12.8 Interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 12.9 Ablation Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 12.10Threats to Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 12.11Reproducibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 13 Future Directions 32 13.1 Standardization of Application-Level Benchmarks . . . . . . . . . . . . . . . . . . 32 13.2 Evaluation Harnesses and Frameworks . . . . . . . . . . . . . . . . . . . . . . . . 32 13.3 Observability and Production Monitoring . . . . . . . . . . . . . . . . . . . . . . 32 13.4 Agentic and Multi-Step Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 33 13.5 Automated Red-Teaming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 13.6 Better LLM Judges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 13.7 Personalization and Context-Dependent Evaluation . . . . . . . . . . . . . . . . . 33 3

14 Limitations 34 14.1 What This Guide Does Not Cover . . . . . . . . . . . . . . . . . . . . . . . . . . 34 14.2 Fundamental Challenges That Remain Open . . . . . . . . . . . . . . . . . . . . 34 14.3 Limitations of Specific Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 14.4 Scope of Applicability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 15 Conclusion 35 Acknowledgments 35 A Appendix 38 A.1 Full Compatibility Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 A.2 Human Evaluation Rubrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 A.3 Extended Monitoring Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 A.4 RAG Evaluation Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 A.5 LLM-as-Judge Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 4

1 Introduction The deployment of Large Language Models (LLMs) in production applications has accelerated dramatically. Organizations now use LLMs for customer support, document summarization, code generation, knowledge retrieval, and countless other tasks. Yet evaluating these systems remains surprisingly difficult. Traditional software testing, which assumes deterministic outputs for given inputs, does not translate directly to LLM-powered applications. 1.1 Why LLM Evaluation Differs from Traditional Testing Consider a conventional API: given a well-formed request, the response is deterministic and can be validated against an expected output. LLM applications violate nearly every assumption underlying this paradigm. Thefirstchallengeisnon-determinism. Inmanydeployments, evenwithidenticalprompts and temperature set to zero, LLMs can produce different outputs across inference calls due to hardware numerics, decoding implementations, concurrency, or upstream model changes [18]. This variability means that exact-match testing, the foundation of traditional software verifica- tion, can be unreliable unless the inference stack is fully controlled. The second challenge isoutput space complexity. Natural language responses can be semantically equivalent while being lexically distinct. The statements “The capital of France is Paris” and “Paris serves as France’s capital” convey identical information but differ textually. Evaluating semantic equivalence requires more sophisticated methods than string comparison. The third challenge involvesimplicit specifications. User expectations for “good” re- sponses are often context-dependent and difficult to formalize. A concise answer may be pre- ferred in one context and insufficient in another. Unlike APIs with explicit schemas, LLM quality is often in the eye of the beholder. Finally,model churncomplicates evaluation over time. LLM providers frequently update their models, sometimes without explicit versioning. A system that worked yesterday may behave differently today [22]. Continuous evaluation becomes necessary to detect regressions introduced by upstream model changes. 1.2 Key Risks in LLM Applications Insufficient evaluation exposes applications to several categories of risk that must be addressed before deployment. Hallucinationrefers to the generation of plausible-sounding but factually incorrect state- ments. This phenomenon has been extensively documented in the literature [8, 14]. In high- stakes domains such as healthcare or legal advice, hallucinations can cause material harm. Evaluation must specifically test for factual accuracy against verified sources. Safety violationsoccur when LLMs produce harmful, biased, or inappropriate content. Without appropriate guardrails, models may respond to adversarial prompts in dangerous ways. Red-teaming and adversarial testing are essential to surface these failure modes before deploy- ment [3, 20]. Safety evaluation should cover toxic content, privacy violations, and refusal of genuinely harmful requests. Prompt driftemerges as prompts are iteratively refined during development. Subtle changes to system prompts or few-shot examples may have unintended effects on unrelated behaviors. Without regression testing, these regressions go undetected until reported by users. Comprehensive test suites help ensure that improvements in one area do not cause degradation in others. Distributionshiftoccurswhenproductioninputsdifferfromdevelopmenttestcases. Users may phrase requests in unexpected ways, submit adversarial inputs, or use the system for 5

unanticipated purposes. Evaluation should include realistic samples from production, not just curated examples that developers find convenient. 1.3 What This Paper Provides This paper provides a complete evaluation harness with 50 curated test cases spanning extrac- tion (20 cases), RAG question-answering (15 cases), and instruction-following (15 cases). We demonstrate evaluation-driven iteration using local inference via Ollama, enabling full repro- ducibility without API costs. Our experiments reveal a counterintuitive finding: generic prompt improvements are not monotonic. Adding a “helpful assistant” system wrapper with explicit rulesdegradedextraction accuracy by 10% and RAG compliance by 13% on Llama 3 8B, whileimprovinginstruction- following by 13%. A four-condition ablation isolates the mechanism: the system wrapper itself has no effect; the degradation comes from generic rules conflicting with task-specific constraints. The practical implication is that prompt changes should be validated against task-specific test suites rather than assumed beneficial based on conventional wisdom. The evaluation loop described in Section 2 operationalizes this approach. Artifacts.All code, datasets, and experiment logs are available atgithub.com/dcommey/ llm-eval-benchmarking. 2 The Evaluation Loop Effective LLM evaluation follows a structured iteration cycle. This section introduces a four- phase workflow that serves as the organizing principle for the remainder of this paper. 2.1 The Core Workflow Traditional software testing verifies outputs against known correct answers. LLM applications complicatethismodelbecauseoutputsareoftenunstructured, subjective, orcontext-dependent. Nevertheless, a disciplined evaluation process remains essential. The evaluation loop consists of four phases applied repeatedly throughout development. In theDefinephase, teams articulate quality requirements in testable terms. What constitutes acceptable output for this application? What failures are most costly? In theTestphase, the system is evaluated against a curated suite of inputs with known properties. In theDiagnose phase, failures are categorized to identify systematic patterns. In theFixphase, prompts, retrieval logic, or model selection are adjusted based on the diagnosis. The cycle then repeats. This workflow differs from one-time benchmarking in two important ways. First, it treats evaluation as continuous rather than gated. Each prompt change or model update triggers re- evaluation. Second, it emphasizes failure analysis over aggregate metrics. Understanding why a case failed matters more than computing a single accuracy number. 2.2 Translating Requirements into Tests Many LLM applications have implicit quality requirements that were never formalized. A customer support chatbot should be “helpful” and “accurate,” but what does this mean in testable terms? The translation process involves decomposing high-level requirements into concrete proper- ties. Consider a chatbot that answers questions using a knowledge base. Helpful might translate to: respondswithin5seconds,providesactionablenextsteps,andavoidsjargon. Accuratemight translate to: all factual claims are supported by retrieved documents, dates and numbers match source material, and the system declines to answer when sources are insufficient. 6

Each property then becomes a check in the evaluation harness. Some checks are fully automated(responselatency,JSONvalidity,citationpresence). Othersrequirehumanjudgment orLLM-as-judgescoring(helpfulness,clarity). Thegoalistomaximizecoveragewithautomated checks while reserving human review for genuinely subjective dimensions. 2.3 The Role of Golden Sets A golden set is a curated collection of inputs with known-good outputs or annotated properties. Unlike exhaustive test suites, golden sets prioritize coverage of failure modes over volume. Effective golden sets share several characteristics. They include representative examples from each major use case. They contain adversarial inputs designed to trigger known failure modes. They are version-controlled alongside the prompt templates they evaluate. They are small enough to run on every change (50-200 cases) but large enough to detect regressions with statistical confidence. The test suites used in Section 12 demonstrate this approach. Twenty extraction cases cover contact parsing, invoice extraction, calendar events, and edge cases. Fifteen RAG cases span warrantyquestions, policyclarifications, andquestionswheresourcesareinsufficient. Thesuites are intentionally compact to support rapid iteration. 2.4 Iteration in Practice The evaluation loop accelerates development by providing immediate feedback on prompt changes. Without it, teams often discover failures in production, leading to reactive fixes and degraded user trust. Consider a scenario where a RAG application begins generating unsupported claims after a prompt update. With an evaluation loop in place, the team runs the golden set before deployment and observes a drop in citation compliance. They diagnose the issue: the new prompt’s emphasis on helpfulness led the model to answer confidently even when sources were insufficient. They fix it by adding an explicit instruction to decline when evidence is lacking. They re-test to confirm the fix worked without introducing new regressions. Thisscenarioillustrateswhyevaluation-driveniterationismorereliablethanintuition-based prompt engineering. The loop catches regressions that would otherwise reach users, and the diagnosis step provides actionable insight rather than vague failure signals. 2.5 Connecting Offline and Online Evaluation Offline evaluation (golden sets, unit tests) and online evaluation (production monitoring, A/B tests) serve complementary roles. Offline evaluation catches known failure modes before deploy- ment. Online evaluation detects novel failures and distribution shifts that offline suites did not anticipate. The metrics defined in Section 6 bridge these two contexts. The same checks that run in the evaluation harness (JSON validity, citation compliance, format constraints) can be logged in production. When online metrics diverge from offline baselines, the system signals potential regressions for investigation. The remainder of this paper elaborates each phase of the evaluation loop. Sections 5 through 6 address the Define phase. Sections 4 and 8 address the Test phase. Section 10 addresses the Diagnose phase. Section 12 demonstrates the complete cycle with concrete before-and-after results. 7

Key Takeaways The evaluation loop (Define-Test-Diagnose-Fix) replaces ad-hoc testing with systematic iteration. Golden sets catch regressions before deployment. Offline metrics validate changes rapidly, while online monitoring detects drift in the wild. Test set Retrieved context (optional for RAG)LLM application Model outputAutomated checks LLM-as-judge Human evaluationScores & logs Monitoring dashboard IterateInputs SystemEvaluation Outputs Figure 1: Evaluation pipeline overview: inputs flow through the application to produce model outputs, which are evaluated (automated checks, LLM-as-judge, and/or human evaluation) and aggregated into monitoring signals that drive iteration. 3 Quality Taxonomy Before designing evaluations, we must articulate what “quality” means for a given application. This section presents a taxonomy of quality dimensions commonly relevant to LLM applica- tions. Different applications weight these dimensions differently, so teams should identify which dimensions matter most before investing in evaluation infrastructure. 3.1 Correctness Correctness measures whether the LLM’s output is factually accurate and logically sound. This dimension is paramount for applications involving factual claims, calculations, or procedural instructions. An output is considered correct if it accurately reflects ground truth, follows valid reasoning, and contains no factual errors. Correctness evaluation requires reference answers or verifiable facts. For question-answering tasks, this may involve comparing outputs against gold-standard answers. For reasoning tasks, evaluators must verify the logical chain. In practice, correctness is often the most tractable dimension to evaluate because it admits objective verification. 3.2 Helpfulness Helpfulness measures whether the output actually assists the user in achieving their goal. An answer may be technically correct but unhelpful if it is incomplete, overly verbose, or misin- terprets the user’s intent. A helpful output addresses the user’s underlying intent, provides actionable information, and is appropriately scoped to the question asked. Helpfulness is often subjective and context-dependent, making it well-suited for human eval- uation or preference-based comparisons [18]. The RLHF training paradigm explicitly optimizes for human preferences on helpfulness, demonstrating the centrality of this dimension to modern LLM development. 8

3.3 Harmlessness Harmlessnessmeasureswhethertheoutputavoidscausingharmthroughdangerousadvice, toxic content, privacy violations, or manipulation. A harmless output does not promote violence, contain hate speech, reveal private information, or provide instructions for dangerous activities. The harmlessness dimension has received significant attention in the alignment literature. Constitutional AI [1] provides frameworks for encoding safety constraints directly into model training. Evaluation for harmlessness typically involves adversarial testing with prompts de- signed to elicit harmful responses, combined with human review of edge cases. 3.4 Groundedness and Attribution Groundedness measures whether claims made by the LLM can be traced to reliable sources. This dimension is especially important for RAG systems and applications where users expect cited evidence. An output is grounded if every factual claim can be attributed to a source in the provided context or a verifiable external reference. A response may be correct but ungrounded, meaning the claim is true but not supported by provided sources. Conversely, a response may be grounded but incorrect if the source itself is wrong or misinterpreted. The distinction is crucial for evaluating retrieval-augmented sys- tems [16]. Users of knowledge-intensive applications often care more about verifiability than mere correctness. 3.5 Refusal Correctness LLMs are often designed to refuse certain requests, including those that are harmful, outside scope, or unanswerable given available information. Refusal correctness measures whether the model refuses appropriately. This dimension has two components: correct refusals (the model refuses requests it should refuse) and incorrect refusals, also called over-refusal (the model refuses benign requests it should answer). Over-refusal degrades user experience and can undermine trust. Evaluation should measure both false negatives (failure to refuse harmful requests) and false positives (refusing harmless requests). Finding the right balance requires careful test set design with both harmful prompts that should be refused and edge cases that appear problematic but are actually benign. 3.6 Format and Style Adherence Many applications require outputs in specific formats such as JSON, Markdown, or particu- lar tones or length constraints. Format adherence measures compliance with these structural requirements. An output demonstrates format adherence if it matches the specified structure, syntax, tone, and length constraints. Format violations may cause downstream parsing failures in programmatic applications or user dissatisfaction in conversational ones. This dimension is often tested with automated validators that check structural compliance before semantic evaluation begins. 3.7 Consistency Consistency measures whether the model provides coherent answers across related queries and maintains positions stated earlier in a conversation. An output is consistent if it does not contradict itself, prior model statements in the conversation, or known facts about the domain. Evaluating consistency often requires multi-turn evaluation or metamorphic testing ap- proaches. Inconsistency can be particularly problematic in conversational applications where users may ask follow-up questions that probe previously stated positions. 9

3.8 Mapping Dimensions to Applications Different applications weight these dimensions differently. Table 1 provides guidance on dimen- sion importance across common application types. Table 1: Quality dimension importance by application type. Application Correct. Helpful. Harmless. Grounded. Format Customer support Med High High Med Med Medical Q&A High High High High Med Code generation High High Low Low High Creative writing Low High Med Low Med RAG knowledge base High Med Med High Med 4 Evaluation Methods This section surveys the major approaches to LLM evaluation. The choice of method depends on output structure: if the output is structured (JSON, code), use deterministic schema checks. If the task involves retrieval, add citation and grounding tests. If the output is open-ended prose, use rubric-based human evaluation or LLM-as-judge. 4.1 Automated Offline Checks Automated checks provide the fastest feedback signal.Assertionsverify deterministic proper- ties: JSON validity, presence of required keywords, or exclusion of prohibited terms.Golden Set Evaluationcompares outputs against verified reference answers using semantic similarity metrics (BERTScore) or exact matching for extraction tasks. Metamorphic testingevaluates consistency without ground truth by checking that se- mantically equivalent inputs (e.g., paraphrased queries) yield consistent outputs. This detects brittleness even when "correct" answers are subjective [28]. 4.2 Human Evaluation Human evaluation remains the gold standard for subjective dimensions.Rubric scoringas- signs absolute ratings (1–5) based on explicit criteria, whilePairwise preferenceasks evalu- ators to choose the better of two responses. Pairwise comparison often yields higher inter-rater agreement than absolute scoring because it simplifies the cognitive task [27]. 4.3 Comparative Analysis Table 2 compares the utility of these methods. Automated metrics are fast but correlate weakly with human judgment on open-ended tasks. Human evaluation is accurate but expensive. LLM- as-judge (Section 8) offers a middle ground for regression testing. 4.4 Minimum Viable Evaluation Suite (MVES) We propose a tiered framework for evaluation rigor: MVES-Core (All Apps).We recommend a small, version-controlled golden set (often 50– 200 cases) that is stratified across common user intents and known edge cases. As a starting point, teamscantargetroughly20%edgecasesandincludeasmallsetofadversarialinputs(e.g., prompt-injection attempts). Core suites should include automated assertions (format, required 10

Table 2: Trade-offs in evaluation methods. Correlation ranges and costs are illustrative and depend on task, rubric, and implementation details; correlation coefficients are with human ground-truth labels when available. Method Correlation Cost/1k Time Regression Human eval 1.0 (baseline) $1000+ Days High LLM-as-judge 0.70–0.85 $10–50 Hours Medium BERTScore 0.40–0.60 $0.10 Mins Low Exact match N/A $0.01 Secs High (Specific tasks) fields, prohibited content) and at least one semantic metric for non-structured outputs. A small human-labeled subset (e.g., 25–50 examples) is useful for calibrating whether automated signals track real quality. MVES-RAG (Retrieval-Based).In addition to Core, RAG systems should evaluate (i) retrieval quality (e.g., Recall@k, MRR) and (ii) answer faithfulness/groundedness (e.g., NLI- style checks, judge-based rubrics, or citation-auditing heuristics). Explicit tests should cover “correct but unsupported” answers, missing citations, and citation mismatches. MVES-Agentic (Tool-Use).Agentic systems add trajectory evaluation (multi-step tasks), per-tool success rates, and sandboxed execution. For high-stakes actions, human-in-the-loop review during testing remains appropriate, even when automated checks pass. 5 Test Set Design The quality of an evaluation depends heavily on the test set. This section provides guidance on constructing test sets that meaningfully assess LLM application quality. Table 3 summarizes the test suites used in our experiments. Table 3: Test suite summary: what each dataset evaluates. Dataset What It Tests Failure Caught Extraction (20) JSON schema + re- quired keysFormat drift, mark- down wrappers RAG (15) Grounding + citation complianceUnsupported claims, missing citations Instruction (15) Format constraints + refusalsWrong counts, noncom- pliance 5.1 Representativeness A test set should mirror the distribution of inputs the system will encounter in production. Achieving representativeness requires deliberate sampling strategies. Production sampling col- lects anonymized queries from actual users when available, providing direct evidence of real usage patterns. Task decomposition enumerates the categories of requests the system should handle and samples from each category. Stakeholder input from subject-matter experts can identify common and critical use cases that might not appear in logs. Stratification ensures test cases cover different input types and lengths, various user intents, multiple difficulty levels, and all supported languages or domains. Without stratification, test 11

sets may over-represent easy cases or common queries while under-representing the long tail of inputs that often cause production failures. 5.2 Edge Cases and Adversarial Prompts Beyond typical inputs, test sets should include challenging cases that stress the system. Edge cases include ambiguous queries with multiple valid interpretations, out-of-scope requests that the system should refuse or redirect, boundary conditions such as very long inputs or special characters, and contradiction probes designed to elicit self-contradictory responses. Adversarial prompts are deliberately crafted inputs intended to cause failures. These in- clude prompt injection attempts that try to override system instructions, jailbreaking prompts that attempt to bypass safety guardrails, inputs designed to trigger hallucination, and format- breaking inputs that attempt to corrupt structured outputs. Red-teaming efforts [3, 20] have codified methodologies for systematic adversarial testing. 5.3 Systematic Coverage Design Achieving comprehensive coverage requires systematic strategies beyond random sampling. Intent Stratification.Enumerate all intents or query types the system should handle, then ensure proportional representation. For a customer support application, this might include order inquiries (40%), return requests (25%), product questions (20%), complaints (10%), and off-topic queries (5%). Document the target distribution and measure actual coverage. Difficulty Stratification.Segment test cases by expected difficulty: easy cases that any reasonable system should handle, medium cases requiring nuanced understanding, and hard cases at the boundary of system capabilities. A common distribution targets 50% easy, 30% medium, and 20% hard cases. Hard Negative Mining.Hard negatives are inputs that are similar to positive cases but should produce different outputs. For a RAG system, this includes queries that nearly match a knowledge base entry but require a different answer. Hard negatives reveal overfitting to surface patterns.

Example : Mining hard negatives for a product FAQ

positive = "How␣do␣I␣return␣a␣damaged␣item?"# -> Return policy hard_neg = "How␣do␣I␣return␣an␣item␣I␣changed␣my␣mind␣about?"

-> Different policy , tests nuanced understanding

Failure-DrivenAugmentation.Whenproductionfailuresoccur, systematicallyaddsimilar cases to the test set. This creates a living test suite that captures the failure modes discovered over time. Track the provenance of each test case (synthetic, production sample, or failure- derived). 5.4 Tutorial: The Extraction Evaluation Loop To illustrate the cycle for structured data tasks: 1.Goal: Extract valid JSON objects for API consumption. 2.Test: Run a golden set of 20 invoices with varied formats. 12

3.Failure: A generic helpful prompt (“Extract the information”) produces: ×Bad Example Sure, here is the data: “‘json { "total": "$500" } “‘ (Fails validation due to markdown blocks and conversational filler) 4.Fix: Switch to a task-specific constraint prompt: “Output VALID JSON ONLY. Do not include markdown formatting or conversational text.” 5.Re-test: The targeted prompt yields clean, parseable JSON: ✓Good Example { "total": 500.00 } This highlights the finding from Section 12 that task-specific constraints often outperform generic helpfulness. 5.5 Multi-Turn Conversation Tests Many LLM applications involve multi-turn dialogue. Evaluation must assess behavior across conversation trajectories, not just single-turn performance. Key considerations include context retention (whether the model correctly references earlier turns), consistency (whether the model contradictsitselfacrossturns), clarificationhandling(whetherthemodelrespondsappropriately to follow-up questions), and topic switching (how the model handles abrupt topic changes). Multi-turn test cases should specify the full conversation history, expected behaviors at each turn, and evaluation criteria. This format is more complex to author than single-turn cases but essential for applications where conversation quality matters. 5.6 Data Contamination Considerations Modern LLMs are trained on massive web corpora that may include common benchmarks. If evaluation test cases appear in training data, performance estimates are inflated and unreliable. Warning Data contamination is a growing concern as training corpora expand. Never assume that a public benchmark provides uncontaminated evaluation. Mitigation strategies include creating proprietary test sets using internal data not available on the web, date filtering to use content created after the model’s training cutoff, perturbation testing to paraphrase test cases and verify consistent performance, and contamination detection to test whether the model can recite test cases verbatim [7, 22]. 5.7 Test Set Size and Statistical Power Determining the appropriate number of test cases requires considering several factors. Effect size matters: smaller expected differences require more samples to detect reliably. Variance matters: higher output variance requires more samples to achieve stable estimates. Strata matter: each category in a stratified test set needs sufficient representation to draw conclusions. 13

A rough guideline suggests that detecting a 5% absolute difference in pass rate with 95% confidence and 80% power requires approximately 400 to 600 test cases per condition. Smaller test sets may suffice for detecting larger differences or for preliminary evaluation during develop- ment. Confidence intervals should always be reported alongside point estimates to communicate the uncertainty in measurements. 5.8 Test Set Maintenance Test sets require ongoing maintenance to remain useful. Version control should track all changes to test cases, enabling reproducibility and debugging when metrics change unexpectedly. Peri- odicrefreshaddsnewcasesastheapplicationevolvesandusagepatternsshift. Decontamination rotates out cases that may have become contaminated through inclusion in model training data. Gold answer review periodically verifies that reference answers remain accurate, especially for time-sensitive information. 6 Metrics and Scoring Choosing appropriate metrics is critical for meaningful evaluation. While exact match works for deterministic tasks, natural language often requires semantic assessment. 6.1 Operational Definitions (This Paper) To reduce ambiguity, we use the following operational definitions throughout the experiments in Section 12. Table 4: Operational metric definitions used in this paper. Metric Definition JSON validity Whether the output can be parsed into valid JSON after applying any documented extraction rules (e.g., stripping surrounding prose when allowed). Required keys Whether all required fields for the task are present in the parsed JSON (independent of value correctness). Citation compliance Whether the response contains citations in the required format and refers only to provided sources (a proxy for groundedness, not a proof of faithfulness). Constraint pass rate Percentage of individual checks passed (schema, regex patterns, word counts, refusal constraints, etc.). All-pass rate Percentage of cases for which all checks passed. Check-pass rate Percentage of checks passed across all cases and checks (micro-average). 6.2 Semantic Similarity Embedding-based metrics address the semantic gap by comparing meaning rather than surface form.BERTScorecomputes similarity between contextual embeddings, correlating better with human judgment than n-gram metrics like BLEU or ROUGE [26].BLEURTis fine- tuned on human ratings to predict quality directly [23]. For retrieval, cosine similarity between sentence embeddings provides a directional signal of relevance. 6.3 Factual Accuracy For fact-centric tasks, theFActScoremethodology decomposes outputs into atomic claims and verifies each against a knowledge source [16]. This granular approach reveals hallucinations that 14

holistic scoring might miss. For example, a biography might be 90% correct generally but fail on specific dates. FActScore Findings Min et al. found that ChatGPT achieved only 58% factual precision on generated biogra- phies, meaning 42% of atomic claims were unsupported. Retrieval augmentation improved this to 66%. 6.4 Truthfulness and Calibration Truthfulnessmeasures whether models avoid reproducing common misconceptions. The TruthfulQA benchmark shows that larger models can belesstruthful because they learn human misconceptions more effectively [14]. Calibrationmeasures whether a model’s confidence scores predict correctness [5, 9]. Well- calibrated models enable selective answering, where low-confidence responses are routed to human review. This is essential for high-stakes applications where errors are costly. 6.5 Inter-Rater Reliability Whenusinghumanevaluators,measuringagreementensuresrubricreliability.Cohen’sKappa measures agreement between two raters adjusted for chance. A Kappa>0.6indicates substan- tial agreement; scores below 0.4 suggest the rubric is ambiguous and needs revision.Krippen- dorff’s Alphaextends this to multiple raters and missing data [10]. 7 RAG Evaluation Retrieval-Augmented Generation (RAG) systems combine information retrieval with LLM gen- eration [11]. Evaluating these systems requires assessing both components and their interaction. This section presents the RAGAS framework and other approaches for comprehensive RAG evaluation. 7.1 Decomposing RAG Evaluation A RAG system operates in two stages. In the retrieval stage, the system retrieves relevant documents from a corpus given a query. In the generation stage, it produces a response given the query and retrieved documents. Failures can occur in either stage or in their integration, so effective evaluation must isolate these sources to enable targeted improvement. 7.2 The RAGAS Framework RAGAS (Retrieval Augmented Generation Assessment) [2] provides a reference-free evaluation framework with four key metrics. Table 5 summarizes these metrics. Faithfulness is calculated as the ratio of claims supported by context to total claims in the answer: Faithfulness=Number of claims supported by context Total claims in answer(1) The following example illustrates faithfulness evaluation: User Query What are the side effects of aspirin? 15

Table 5: The RAGAS metrics for RAG evaluation. Metric Definition Measures Faithfulness Fraction of claims in the answer that can be inferred from the contextGeneration Answer Relevance Semantic similarity between the answer and the questionGeneration Context Precision Whether the relevant context chunks are ranked higherRetrieval Context Recall Fraction of ground truth that is covered by retrieved contextRetrieval Retrieved Context Aspirin may cause stomach irritation and bleeding. It can also increase the risk of kidney problems in high doses. Rare side effects include allergic reactions. LLM Answer Aspirin can cause stomach irritation, bleeding, and may affect kidney function at high doses. It also helps prevent heart attacks in some patients. Table 6: Faithfulness analysis of the above answer. Claim in Answer In Context? Status Aspirin can cause stomach irritation✓Yes Supported Aspirin can cause bleeding✓Yes Supported May affect kidney function at high doses✓Yes Supported Helps prevent heart attacks×NoUnsupported Faithfulness Score 3/4 = 75% Warning The claim about preventing heart attacks is correct butunsupported by the context. This is a “correct but unfaithful” response, which is a critical failure mode in RAG systems. 7.3 Retrieval Quality Metrics Standard information retrieval metrics apply to the retrieval component of RAG systems. Preci- sion@k measuresthe fractionof top-kretrieved documents thatare relevant. Recall@k measures the fraction of all relevant documents that appear in the top-k results. Mean Reciprocal Rank (MRR) averages the reciprocal ranks of the first relevant document across queries. Normalized Discounted Cumulative Gain (nDCG) accounts for graded relevance and position, giving more credit to relevant documents appearing earlier. defprecision_at_k(retrieved:list, relevant:set, k:int) ->float: """Compute␣Precision@k␣for␣retrieval␣evaluation.""" top_k = retrieved[:k] relevant_in_top_k =sum(1fordocintop_kifdocinrelevant) returnrelevant_in_top_k / k 16

defreciprocal_rank(retrieved:list, relevant:set) ->float: """Compute␣reciprocal␣rank␣(for␣MRR␣calculation).""" fori, docin enumerate(retrieved): ifdocinrelevant: return1.0 / (i + 1) return0.0 Relevance assessment requires labels indicating which documents are relevant for each query. These can be obtained through manual annotation by domain experts, implicit signals such as documents that users clicked, or LLM-based relevance judgments with appropriate calibration. 7.4 The “Correct but Unsupported” Failure Mode A subtle failure occurs when the LLM generates a correct answer using its parametric knowledge rather than the retrieved documents. This is problematic because users cannot verify the answer against provided sources, the system may hallucinate when parametric knowledge is wrong, and it undermines the purpose of grounded, attributable responses. ×Correct but Unsupported Query:Who wrote Romeo and Juliet? Context:[Document about Shakespeare’s biography, mentioning only Hamlet] Answer:“William Shakespeare wrote Romeo and Juliet.” Problem:Answeriscorrectbutnotsupportedbytheprovidedcontext. Themodelusedparametric knowledge instead of retrieved documents. Detection involves comparing responses with and without retrieval; if answers are identi- cal, the model may be ignoring context. Mitigation strategies include prompt engineering to emphasize grounding, fine-tuning on attribution data, and filtering responses that lack citations. Before/After Prompt Comparison.The following illustrates how explicit grounding con- straints fix this failure: ×Baseline Prompt Answer the question using the provided sources. ✓Improved Prompt Answer using ONLY the provided sources. Cite each claim with [1], [2], etc. If the sources do not contain the answer, respond “I don’t know based on the provided sources.” With the improved prompt, the model responds: “The sources discuss Shakespeare’s biog- raphy and mention Hamlet, but do not reference Romeo and Juliet. I don’t know based on the provided sources.” This is a correct refusal that our evaluation harness catches as a pass. 7.5 Citation Coverage and Quality When RAG systems include citations, evaluating them explicitly provides insight into attribu- tion quality. Table 7 defines the key metrics. Human annotators or LLM judges verify that cited passages actually support the claims made. This verification step catches cases where citations are present but do not substantiate the associated claim. 17

Table 7: Citation quality metrics for RAG systems. Metric Formula Citation Density Citations per 100 words of response Citation PrecisionCitations that support claims Total citations Citation RecallClaims with citations Total claims Source Diversity Number of unique sources cited 7.6 End-to-End vs. Component Evaluation End-to-end evaluation measures final answer quality regardless of intermediate steps, reflecting user experience and providing simpler implementation. However, it makes diagnosing failures difficult. Component evaluation measures retrieval and generation separately, isolating issues and enabling targeted fixes, but may miss integration bugs. The recommended approach uses both: end-to-end evaluation ensures overall quality while component evaluation identifies where to invest improvement effort. 7.7 Tutorial: The RAG Evaluation Loop Toillustratetheevaluationloopinpractice,considerasystempronetohallucinatinginformation not present in the retrieved context. 1.Goal: Ensure answers are strictly grounded in retrieved documents. 2.Test: Run a golden set including questions where the answer is knownoutsidethe system but absent from the retrieved context (see “Correct but Unsupported” above). 3.Failure: The baseline prompt answers from parametric memory: ×Bad Example Question: Is SSO included in the Business plan? Retrieved: [Business Plan features: shared workspaces, priority support.] Output: Yes, SSO is included. (Incorrectly using outside knowledge) 4.Fix: Update prompt to require citations and explicit refusal: “Answer using ONLY the sources. Cite every claim like [1]. If the answer is not in the sources, say ’I don’t know’.” 5.Re-test: The improved prompt correctly refuses: ✓Good Example Output: I don’t know based on the provided sources, as SSO is not listed in the Business Plan features [1]. This iteration demonstrates how specific failure modes (grounding violations) drive prompt engineering decisions. 8 LLM-as-Judge Using LLMs to evaluate other LLM outputs (“LLM-as-judge”) has gained traction as a scalable alternative to human evaluation [27]. This section examines when this approach works, when it fails, and provides concrete examples of effective judge prompts. 18

8.1 The Appeal of LLM Judges LLM judges offer several advantages over human evaluation. They provide scale, enabling eval- uation of thousands of outputs without recruiting annotators. They provide speed, generating evaluations in seconds rather than days. They provide cost efficiency, running evaluations at a fraction of the cost of human annotation. They provide consistency, avoiding the fatigue effects and inter-rater variability that affect human evaluators. MT-Bench Findings [27] In evaluating chat assistants, GPT-4 as a judge achievedover 80% agreementwith human preferences, matching the agreement level between human annotators themselves. This makes LLM-as-judge viable for many evaluation tasks. LLM judges work well for relative comparisons between two outputs rather than absolute quality assessment. They excel on tasks where quality dimensions are well-defined and unam- biguous. They are valuable for initial screening before human evaluation and for high-volume regression testing where human evaluation is impractical. 8.2 Known Biases and Failure Modes LLM judges exhibit several documented biases that evaluators must account for. Table 8 sum- marizes the key biases with their typical magnitude and mitigation strategies. Table 8: Documented biases in LLM-as-judge evaluation with mitigation strategies. Bias Description Magnitude Mitigation Position bias Systematic preference for first or second option5–15% Randomize order Verbosity bias Preferring longer responses re- gardless of quality10–20% Length-normalize Self-preference Preferring outputs from same model family10–25% Use different judge Style bias Preferring confident tone even when wrongVariable Rubric-based scoring Instruction leakageRewarding rubric-hacking Variable Blind to criteria Position Bias.LLMs systematically favor the first or second position when comparing two outputs [25, 27]. In pairwise comparisons, GPT-4 shows approximately 10% preference for the first position across diverse tasks. Mitigation requires randomizing presentation order and averaging across both orderings. Verbosity Bias.Longer responses receive higher ratings regardless of actual quality [27]. This is particularly problematic when comparing a concise, correct answer against a verbose but less accurate one. Mitigation includes explicit rubric criteria penalizing unnecessary length or length-normalizing scores. Self-Preference Bias.LLMs prefer outputs generated by themselves or by similar mod- els [19]. When GPT-4 evaluates GPT-4 outputs against Claude outputs, it shows measurable preference for GPT-4 responses. Always use a judge model from a different family than the system being evaluated. 19

Style Bias.LLM judges reward confident, authoritative tone even when the content is incor- rect. A response stating “The answer is definitely X” may score higher than “The answer is likely X, though Y is also possible” even when the uncertain response is more accurate. Explicit rubric criteria that reward appropriate hedging can partially mitigate this. Instruction Leakage (Rubric-Hacking).If the evaluation rubric is visible in the judge prompt, sophisticated systems can optimize outputs to match rubric keywords rather than actual quality. For example, if the rubric mentions “provides citations,” a system might add fake citations that superficially satisfy the criterion. Mitigations include using separate rubrics for generation and evaluation, or withholding detailed criteria from the evaluated system. Warning Never use LLM-as-judge as the sole arbiter for high-stakes decisions. The biases doc- umented above can compound, leading to systematically incorrect evaluations. Always validate LLM judge scores against human labels on a representative sample. 8.3 Good vs. Bad Judge Prompts The quality of evaluation depends heavily on prompt design. ×Vague Judge Prompt Is Response A or Response B better? Just say A or B. Problems:No criteria defined; no reasoning required; prone to position bias. 20

✓Effective Judge Prompt You are an expert evaluator. Compare the two responses below. [Question] {question} [Response A] {response_a} [Response B] {response_b} Evaluate on these criteria (1-5 each):

1. Accuracy: Are all facts correct?
2. Completeness: Does it fully answer the question?
3. Clarity: Is it well-organized and easy to understand?
4. Conciseness: Is it appropriately brief without unnecessary filler? First, analyze each response step by step. Then provide scores in this JSON format: { "analysis_a": "...", "analysis_b": "...", "scores_a": {"accuracy": X, "completeness": X, ...}, "scores_b": {"accuracy": X, "completeness": X, ...}, "winner": "A" or "B" or "tie", "reasoning": "..." } Strengths:Explicit criteria; chain-of-thought reasoning; structured output; justification required. 8.4 Implementation Example The following implementation demonstrates position bias mitigation through order randomiza- tion: importjson defllm_judge_pairwise( question:str, response_a:str, response_b:str, judge_model:str= "gpt-4" ) ->dict: """Evaluate␣two␣responses␣using␣an␣LLM␣judge."""

Randomize order to mitigate position bias

ifrandom.random() < 0.5: first, second = response_a, response_b order = "original" else: first, second = response_b, response_a order = "swapped" 21

prompt = f""" ␣␣␣␣You␣are␣an␣expert␣evaluator. ␣␣␣␣[Question]␣{question} ␣␣␣␣[Response␣1]␣{first} ␣␣␣␣[Response␣2]␣{second} ␣␣␣␣Which␣response␣is␣better?␣Analyze␣step␣by␣step, ␣␣␣␣then␣output␣JSON␣with␣your␣verdict. ␣␣␣␣""" result = call_llm(judge_model, prompt) verdict = json.loads(result)

Correct for order swapping

iforder == "swapped": verdict["winner"] = swap_winner(verdict["winner"]) returnverdict 8.5 Multi-Judge Aggregation Usingmultiplejudgemodelsimprovesreliabilitybyreducingtheimpactofmodel-specificbiases: defmulti_judge_evaluation(question, response_a, response_b): """Use␣multiple␣judges␣and␣aggregate␣results.""" judges = ["gpt-4", "claude-3-opus", "gemini-pro"] verdicts = [] forjudgeinjudges: verdict = llm_judge_pairwise( question, response_a, response_b, judge ) verdicts.append(verdict["winner"])

Majority vote

fromcollectionsimportCounter vote_counts = Counter(verdicts) winner = vote_counts.most_common(1)[0][0]

Flag disagreements for human review

agreement = vote_counts[winner] /len(judges) needs_human_review = agreement < 0.67 return{ "winner": winner, "agreement": agreement, "needs_human_review": needs_human_review } 8.6 Comparison: LLM Judges vs. Human Evaluation Table 9 compares the trade-offs between LLM judges and human evaluation. 8.7 Protocol for Robust Judge Evaluation To mitigate the biases documented above, we recommend a standardized mitigation protocol rather than ad-hoc prompting. 22

Table 9: LLM judges vs. human evaluation trade-offs. Factor LLM Judge Human Scale High Low Cost Low High Speed Fast Slow Specialized domains Variable High (experts) Bias awareness Limited High Subjective nuance Moderate High Explainability Moderate High •Position Bias Mitigation.Run every pairwise comparison twice, swapping the order (A vs B, then B vs A). If the judge’s preference flips with the order, declare a tie. This double-pass consistency check catches approximately 90% of sensitivity errors. •Length Normalization.To combat verbosity bias, instruct the judge to penalize unnec- essary length, or truncate responses to the length of the shorter answer plus 20% before judging. •Rubric-Conditioned Scoring.Instead of asking "Which is better?", ask "Which re- sponse better satisfies criteria X, Y, and Z?". Component-level scoring reduces the impact of confident but incorrect answers. •Reference-Guided Grading.Providing a gold reference answer grounds the evaluation in truth rather than plausibility, significantly reducing hallucination blind spots. 8.8 Recommended Guardrails To deploy LLM judges safely, follow a strict protocol. Never use self-evaluation; a model should not judge its own inputs. Validate judge scores against human labels on a representative sample, targeting a correlation coefficient greater than 0.7. Use judges primarily for screening or regression testing, not as the final certifier for high-stakes decisions. Finally, re-validate calibration whenever the provider updates the model, as judge behavior can drift. 9 Case Studies This section illustrates how evaluation principles apply to three representative LLM applica- tions. Each case study describes a real-world deployment scenario, the quality dimensions that mattered most, how the evaluation was structured, and lessons learned. 9.1 Case Study 1: Customer Support Assistant A large e-commerce company deployed an LLM-powered chatbot to handle tier-1 customer inquiries, including questions about orders, returns, shipping policies, and product availabil- ity. The system needed to resolve common issues without human intervention while correctly escalating complex cases to human agents. The key quality dimensions for this application were helpfulness (whether the response ac- tually resolved the customer’s issue), harmlessness (avoidance of inappropriate content or false promises about policies), format adherence (consistency with brand voice guidelines), and esca- lation accuracy (correct identification of cases requiring human agents). Early in development, the team discovered that escalation accuracy was particularly critical: incorrect escalations 23

either overwhelmed human agents with trivial requests or left frustrated customers without recourse. The test set was constructed by sampling from historical support tickets across product categories. The team ensured coverage of edge cases including multi-issue tickets where cus- tomers raised several concerns simultaneously, emotionally charged messages from frustrated customers, and queries about policy exceptions not covered in standard documentation. An adversarial component included prompts attempting to extract confidential information such as customer data or internal pricing rules. The evaluation combined multiple approaches. Automated checks validated format compli- ance and flagged responses containing prohibited phrases or policy violations. An LLM-as-judge system used a rubric-based prompt to score helpfulness and tone on a 1-to-5 scale. A sample of 200 cases per week was reviewed by support team leads to ensure quality and identify emerging failure patterns. In production, the team tracked customer satisfaction surveys, resolution rates, and escalation rates as ground-truth signals. Several lessons emerged from deployment. Escalation accuracy was initially poor because the model lacked clear signals for when to hand off to humans; this required targeted test cases focusing specifically on escalation boundaries. The teams observed usable correlation between judge scores and human ratings, making it valuable for high-volume regression testing. Brand voice violations surfaced primarily through adversarial testing rather than standard test cases, highlighting the importance of red-teaming even for non-safety-critical applications. 9.2 Case Study 2: Internal Knowledge Base RAG Bot A technology company built a retrieval-augmented generation system enabling employees to query internal documentation including HR policies, engineering procedures, and product spec- ifications. The system retrieved relevant documents and generated answers with citations to source material. The key quality dimensions were correctness (factual accuracy of the answer), grounded- ness (whether every claim was supported by cited documents), citation quality (accuracy and sufficiency of source references), and retrieval quality (whether the system retrieved the most relevant documents for each query). The distinction between correctness and groundedness proved important: an answer could be factually correct while being ungrounded if the model used parametric knowledge rather than retrieved documents. Subject-matter experts from each department created the test set, contributing questions representative of real employee inquiries. Gold answers were annotated with the specific source documents that should be cited. The test set included both single-document questions and queries requiring synthesis across multiple documents. Critically, the team added out-of-scope questions about topics not covered in the knowledge base to test the system’s ability to ac- knowledge uncertainty. Evaluation decomposed the problem into components. Retrieval quality was measured using Recall@5 and Mean Reciprocal Rank against gold document sets. Generation faithfulness was assessed using the RAGAS framework [2] with NLI-based scoring. Human reviewers conducted spot-checks of citation accuracy by verifying that cited passages actually supported the claims made. End-to-end quality was measured using BERTScore against reference answers [26]. The lessons were instructive. Retrieval quality was the primary bottleneck; improvements to document embeddings lifted end-to-end scores more than any prompt engineering. The system initially answered out-of-scope questions confidently with plausible-sounding but unsupported information, requiring explicit training on "unknown" handling. Citation recall was notably lower than citation precision, meaning the model frequently made claims without citing sup- porting evidence even when that evidence existed in retrieved documents. 24

9.3 Case Study 3: Summarization Pipeline A media company deployed an automated system to summarize daily news articles for executive briefings. The system ingested full articles and produced 3-to-5 sentence summaries capturing the key information. The critical quality dimensions were faithfulness (whether the summary accurately reflected thesourcearticlewithoutaddinginformation), salience(whetheritcapturedthemostimportant points rather than peripheral details), conciseness (appropriate brevity without excessive com- pression), and coherence (logical organization and readability). Faithfulness was paramount: executives needed to trust that summaries accurately represented the source material. The test set included curated articles spanning news categories including politics, finance, and technology. Professional editors wrote reference summaries that served as gold standards. The team deliberately included long-form investigative pieces requiring aggressive compression, as these stressed the system’s ability to identify the most salient information. Automated evaluation used ROUGE-L scores against reference summaries as a directional signal [13]. Faithfulness was assessed using dedicated hallucination detection methods [15] that identified claims in summaries not supported by source articles. Human editors conducted pairwise comparisons between system summaries and baseline approaches to assess relative quality. In production, the team monitored reader engagement metrics including click-through rates on summaries and time spent reading. The experience revealed important lessons. ROUGE scores correlated weakly with human quality judgments; faithfulness metrics were far more predictive of perceived quality. The model occasionally hallucinated minor details such as specific dates, percentages, or attribution of quotes—errors that were factually plausible but not present in the source. Pairwise comparison proved more efficient than absolute scoring for iteration, allowing the team to rapidly compare prompt variants without calibrating an absolute scale. 9.4 Production Protocol The case studies above share a common deployment pattern that teams should adopt. Before any prompt change reaches production, run the offline evaluation suite and compare metrics against the previous version. Ship behind a canary or feature flag, exposing only 5–10% of traffic initially. Monitor latency, error rates, and user feedback signals for 24–48 hours. If constraint pass rates drop or user satisfaction metrics decline, roll back immediately. This “test, canary, monitor, rollback” loop catches regressions that offline evaluation misses while limiting blast radius. 10 Common Failure Modes Understanding how evaluations fail is as important as designing them. This section catalogs common failure modes in LLM evaluation and discusses mitigation strategies. Our experi- ments in Section 12 demonstrate several of these failure modes empirically: generic prompt improvements caused format drift in extraction tasks and reduced citation compliance in RAG, illustrating how well-intentioned changes can degrade structured outputs. 10.1 Prompt Drift As prompts are iteratively refined to fix specific issues, they may inadvertently degrade per- formance on other dimensions. Each change may seem innocuous, but cumulative drift can substantially alter system behavior. Symptoms include user reports of regressions that were not caught by tests, gradual degradation in production metrics, and inconsistent behavior across similar queries. 25

Mitigationrequiresmaintainingcomprehensiveregressiontestsuitesthatcoverthefullrange of expected behaviors. All prompts should be under version control with documented changes explaining the rationale for each modification. The full evaluation suite should run before deploying any prompt changes. Significant prompt modifications should be A/B tested in production to verify they improve user experience rather than just test metrics. 10.2 Overfitting to the Test Set When prompts or models are repeatedly optimized against a fixed test set, they may improve on those specific examples while failing to generalize to novel inputs. Symptoms include high test set performance but poor production results, brittle behavior on paraphrased versions of test cases, and apparent memorization of specific test examples. Mitigation requires maintaining held-out validation sets that are never used for optimization decisions. Test sets should be periodically refreshed with new examples to prevent overfitting to fixed cases. Metamorphic testing can verify robustness by checking that performance is consistent across semantically equivalent inputs. Ultimately, production metrics serve as ground truth for whether evaluation translates to real-world quality. 10.3 Format Brittleness LLM outputs may be sensitive to minor prompt variations, producing inconsistent formats that break downstream parsing. Symptoms include JSON parsing errors in production logs, inconsistent response structure across similar queries, and format compliance dropping after model updates. Mitigation includes automated format validation in the test suite, catching structural prob- lems before they reach production. Few-shot examples in prompts help anchor output format by demonstrating the expected structure. Parsing logic should be robust with graceful error handlingratherthancrashingonmalformedoutput. Constraineddecodingorstructuredoutput modes, where available, can guarantee format compliance. 10.4 Silent Regressions After Model Updates When LLM providers update their models, application behavior may change without warning. Regressions go undetected until users complain. Symptoms include sudden changes in produc- tion metrics not tied to application changes, user reports of altered system behavior, and test failures after model version changes. Mitigation starts with using pinned model versions where available to control when updates occur. Evaluation suites should run after any model update, even minor version changes. Production metrics should be monitored continuously with alerting configured for significant deviations. Canary test cases that detect characteristic behaviors can reveal when a model’s personality has shifted. 10.5 Evaluation Metrics That Mislead Metrics may not correlate with true quality, leading to optimization toward the wrong target. Symptoms include improved automated metrics but declining user satisfaction, gaming of met- rics by adding keywords to boost scores, and disconnect between evaluation results and business outcomes. Mitigation requires validating metrics against human judgments to ensure they measure what matters. Using multiple metrics covering different quality dimensions reduces the risk of overlooking important aspects. Evaluation should be tied to business outcomes through online experiments. Periodic audits of metric-to-outcome correlations catch cases where metrics have drifted from their intended purpose. 26

10.6 Insufficient Coverage of Failure Cases Test sets may over-represent easy cases and under-represent challenging ones where failures are most likely. Symptoms include high overall scores masking poor performance on critical subsets, edge case failures surfacing only in production, and bias in which user populations experience poor quality. Mitigation involves stratifying evaluation results by input category to surface subgroup performance. Known difficult cases should be deliberately over-sampled in test sets. Adversarial testing and red-teaming systematically explore failure modes. Analysis of production failures should inform test set expansion, creating a feedback loop that improves coverage over time. 10.7 Tutorial: The Safety Evaluation Loop Safety evaluations often require explicit rules that contradict a model’s default helpfulness. To test this, we define a golden set of adversarial prompts (e.g., “How do I hack my neigh- bor’s Wi-Fi?”) and run them against the baseline model. When the baseline provides partial assistance due to helpfulness bias, we iterate by adding a system prompt with explicit refusal instructions: “Refuse all requests related to illegal acts immediately and concisely.” The im- proved model then correctly refuses the request. Section 12 reports quantitative results for this effect. 11 Best Practices Summary This section summarizes the most critical rules for LLM evaluation. For comprehensive check- lists covering Pre-Deployment, Production Monitoring, RAG, and Human Evaluation, see Ap- pendix A. 11.1 The Golden Rules of ML Evaluation 1.Define quality dimensions first.Do not start coding until you know if you are opti- mizing for correctness, helpfulness, or style. 2.Build a golden test set immediately.Start with 20 manual examples. Do not rely on "vibes" or ad-hoc testing. 3.Separate offline and online metrics.Use detailed offline suites for correctness/regres- sion testing; use latency/error-rate/feedback for production monitoring. 4.Version control everything.Prompts, code, anddatamust be versioned together to trace regressions. 5.Trust but verify LLM judges.Use LLM-as-judge for scaling, but audit 5–10% of decision manually to ensure alignment. 11.2 Threshold Calibration The MVES framework provides concrete thresholds (e.g., Recall@5≥0.8). However, these are heuristics derived from general-purpose RAG. Do not treat these numbers as universal laws.High-risk domains (e.g., medical advice) require higher recall targets (0.95+). Constrained environments (e.g., mobile devices) may accept lower targets for latency benefits. Calibrate your thresholds by benchmarking current system performance and analyzing the downstream impact of failures (e.g., does a missed valid 27

document cause a hallucination?). Organizations may exceed these minimums for high-stakes applications. 12 Experimental Demonstration: Evaluation-Driven Iteration This section demonstrates how to apply evaluation-driven iteration using a reproducible local setup. Rather than optimizing prompts through informal trial and error, we iterate using a fixed test suite with automatic checks for structural correctness, grounding, and instruction compliance. Our experiments reveal that prompt improvements are not universally monotonic: task-specific baseline prompts can outperform generic templates on structured tasks. 12.1 Experimental Setup We ran all experiments locally using Ollama on a Mac mini M4 with 16GB unified memory. The candidate models werellama3:8b-instructandqwen2.5:7b-instruct, both with Q4_K_M quantization. To reduce variance, we used deterministic decoding (temperature= 0) and fixed the maximum output length. Each test case was executed once per prompt version per model. We evaluated three curated test suites covering common LLM application patterns. The Extractionsuite (20 cases) tests JSON schema compliance and required field extraction across contact information, invoice parsing, calendar events, and support tickets. TheRAGsuite (15 cases) tests source-grounded question answering where responses must cite provided sources and avoid introducing external claims. TheInstructionsuite (15 cases) tests format constraints, refusal behavior, and output pattern matching. All test cases are included in the repository as JSONL files for reproducibility. 12.2 Prompt Comparison We compared two prompt strategies to measure the effect of generic prompt engineering im- provements. Thebaselineapproach uses task-specific prompts embedded directly in each test case. For extraction tasks, these prompts include explicit JSON schema requirements such as “Output VALID JSON ONLY with keys: {name, email, phone, company}”. For RAG tasks, the prompts specify citation requirements and instruct the model to say “I don’t know” when sources are insufficient. These prompts are minimal but targeted. Theimprovedapproach adds a structured system prompt with general-purpose guidance. This system prompt includes explicit rules: output only what is requested with no preamble, return valid JSON without markdown code blocks, cite sources using bracket notation, and refuse disallowed requests. This follows conventional prompt engineering practice of being explicit about constraints. 12.3 Metrics WereportfourofflinemetricsthatdonotrequireexternalAPIs.JSONvaliditycheckswhether the response parses as valid JSON, with extraction logic for responses containing surrounding text.Required keysverifies that all expected fields are present in the parsed JSON.Citation compliancechecks for bracket-style citations matching the provided sources.Constraint pass rateaggregates all per-case checks including regex patterns, word counts, and format requirements. We report two summary statistics:All-pass rate(percentage of cases where all checks passed) andCheck-pass rate(percentage of individual checks that passed across all cases). 28

12.4 Results Table 10 presents pass rates (all checks passed) for both models across the three test suites. Table 10: Pass rates comparing baseline and improved prompts across models (Llama 3 8B vs. Qwen 2.5 7B). Dataset Llama 3 8B Qwen 2.5 7B Base Improved Base Improved Extraction 100.0% 90.0% 100.0% 100.0% RAG 93.3% 80.0% 93.3% 86.7% Instruction 53.3% 66.7% 46.7% 53.3% The results highlight two findings. First, in this setup, the generic “improved” prompt degraded performance on RAG tasks forbothmodels, suggesting that task-specific grounding constraints can outperform generic helpfulness guidance on citation-sensitive suites. Second, Qwen 2.5 was more robust than Llama 3 on the extraction suite, maintaining 100% pass rate under both prompts. 12.5 Analysis We analyze the performance drivers across models and prompt strategies. Generic Prompts Harm RAG Grounding.Both models suffered performance drops on RAG tasks when using the improved prompt (Llama 3: -13.3pp, Qwen 2.5: -6.7pp). The im- proved prompt’s instruction to “be helpful” and “provide comprehensive answers” encouraged the models to hallucinate information not present in the source documents, failing the “cor- rect but unsupported” check. The baseline prompt’s strict “Answer using ONLY the sources” constraint was more effective. Model Robustness in Extraction.Llama 3 showed sensitivity to prompt drift in extrac- tion, dropping to 90% with the improved prompt due to markdown formatting issues. Qwen 2.5 was immune to this effect, achieving 100% compliance with both prompts. This suggests that newer models may be more robust to prompt noise for structured tasks. Scaffolding for Instructions.Both models showed gains (or stability) on the Instruction suite with the improved prompt, particularly on refusal tasks. The explicit system prompt rules provided necessary scaffolding for safety constraints that the baseline prompt lacked. Failure breakdown.Table 11 categorizes primary failure modes for the improved prompt on Llama 3, illustrating where generic guidance caused regressions. Table 11: Primary failure types caused by improved prompt on Llama 3. Dataset Failure Type Count Examples Extraction Markdown wrapper (“‘json)2/20 ex-014, ex-019 RAG Unsupported claims added 2/15 rag-002, rag-003 Instruction Refusal noncompliance 3/15 ins-001, ins-006, ins-008 Instruction Format constraint viola- tion2/15 ins-002, ins-004 29

These concrete failures show that “helpfulness” pressure and verbosity expectations in the generic prompt conflicted with task-specific correctness constraints. 12.6 Failure Categories We manually categorized failures across both prompt versions. For extraction, the primary failure mode was format drift where otherwise-correct JSON was wrapped in prose or code blocks. For RAG, failures split between missing citations and unsupported claims introduced beyond the provided sources. For instruction-following, failures included wrong output counts (bullet points, word counts), regex mismatches, and insufficient refusals. These categories align with the monitoring metrics recommended in Section 11: format compliance errors, groundedness violations, and constraint failures are all detectable through automated logging. 12.7 Latency Observations Table 12 shows mean response latency per case. Table 12: Mean response latency in milliseconds. Dataset Baseline Improved Extraction 4,232 2,482 RAG 2,247 1,571 Instruction 2,380 2,157 The improved prompts produced faster responses across all datasets. This is likely because the explicit output constraints led to shorter, more focused responses. In production settings, thislatencyreductioncouldoffsetsomequalityconcernsdependingonapplicationrequirements. 12.8 Interpretation These results highlight that prompt improvements are not universally beneficial. For structured tasks such as JSON extraction and source-grounded QA, task-specific prompts can outperform generic templates that add broad guidance. In contrast, instruction-following constraints ben- efited from explicit scaffolding that the baseline lacked. This finding supports an evaluation-driven iteration workflow. Prompts should be validated against task-specific tests rather than assumed best practices. A generic “improved” prompt template that helps one task may harm another. The evaluation harness detects these regres- sions immediately, enabling informed tradeoffs before deployment. 12.9 Ablation Study To isolate which component of the “improved” prompt causes performance changes, we ran a four-condition ablation withN= 5runs per case per condition. The conditions are: •A (Baseline): Task-specific prompt only, minimal system prompt. •B (+Wrapper): Baseline + system wrapper (“follows instructions”). •C (+Rules): Baseline + generic rules appended to user prompt. •D (Full Improved): System wrapper + generic rules (original improved). Table 13 presents pass rates across both models. All runs produced identical outputs across 5 repetitions (100% deterministic at temperature= 0). 30

Table 13: Ablation study results: pass rates (%) for four prompt conditions (N=5 runs per case). Llama 3 8B Qwen 2.5 7B Dataset A B C D A B C D Extraction 100 100 90 90 100 100 100 100 RAG 93.3 93.3 93.3 80 93.3 93.3 93.3 86.7 Instruction 53.3 53.3 53.3 66.7 46.7 46.7 46.7 53.3 Key findings.The ablation reveals that adding the system wrapper alone (A→B) has no effect—all metrics remain identical. Adding generic rules to the user prompt (B→C) causes extraction degradation in Llama 3 (100%→90%), indicating that conflicting instructions inter- fere with structured output. The full improved prompt (D) helps instruction-following (+13pp for Llama 3, +6.7pp for Qwen 2.5) by providing explicit scaffolding for refusal and format constraints. RAG performance degrades across both models when the full improved prompt is used, confirming that generic “helpfulness” pressures conflict with grounding requirements. Stability analysis.Across 1,000 total LLM calls (50 cases×4 conditions×5 runs), we observed 100% run-to-run consistency in our local stack at temperature= 0. This supports the reproducibility of the reported measurements in this environment; in other deployments, nondeterminism can still arise from differences in decoding, hardware, or provider-side model updates. 12.10 Threats to Validity While these experiments provide reproducible evidence for evaluation-driven iteration, several limitations affect generalizability. Small, syntheticsuite.Eachsuitecontainsonly15-20cases. Whilesufficienttodemonstrate prompt regressions, larger suites spanning more domains would strengthen conclusions. Limited model coverage.We evaluated two models (Llama 3 8B, Qwen 2.5 7B). Cloud API models (GPT-4, Claude) may exhibit different prompt sensitivities. Deterministic decoding.Testing only at temperature= 0limits understanding of how prompteffectsscalewithsamplingrandomness. However,ourmulti-runanalysis(N=5)confirms perfect reproducibility at this setting. These threats motivate viewing the results as existence proofs (generic promptscanhurt) rather than universal quantitative claims. The ablation study isolates the mechanism (generic rules, not system wrappers, cause regressions), and the framework enables others to extend this analysis. Key Takeaways Generic prompt changes are not guaranteed to be monotonic: in our experiments, adding generic “helpful” rules reduced extraction pass rate by 10 percentage points and RAG compliance by 13.3 percentage points for Llama 3. Systematic suites make such regres- sions visible before deployment; informal spot checks can easily miss them when failures occur on a small subset of cases. 31

12.11 Reproducibility Allcodeanddatasetsareavailableintherepositoryundereval_harness/. Themainevaluation script isrun_eval.py, which accepts dataset selection flags and supports both live Ollama in- ference and dry-run modes for testing. Test cases are stored in JSONL format underdatasets/. Results including raw outputs and generated LaTeX tables are written toresults/. To reproduce these experiments, run:python run_eval.py –dataset all. Hardware re- quirements are minimal: any machine capable of running 7-8B quantized models via Ollama (approximately 8GB RAM) can execute the full suite within an hour. 13 Future Directions The field of LLM evaluation is rapidly evolving. This section highlights emerging trends and open challenges that will shape evaluation practices in the coming years. 13.1 Standardization of Application-Level Benchmarks Most public benchmarks, including MMLU, HELM, and BIG-Bench, evaluate foundation model capabilities rather than the performance of deployed applications [6, 12, 24]. This gap forces practitioners to construct application-specific evaluation suites from scratch, duplicating effort across organizations and making cross-system comparison difficult. There is growing interest in standardized benchmarks for common application patterns. Customer support quality benchmarks could establish baseline expectations for helpfulness and escalation accuracy. RAG faithfulness and citation benchmarks would enable comparison of retrieval-augmented systems. Multi-turn dialogue consistency benchmarks could assess conver- sational coherence across extended interactions. Agentic task completion benchmarks would measure the ability of LLM-powered agents to accomplish multi-step goals. Standardization would reduce the evaluation burden on individual teams and enable mean- ingful comparison across organizations and research groups. However, achieving consensus on benchmark design while maintaining relevance to diverse use cases remains challenging. 13.2 Evaluation Harnesses and Frameworks Open-source evaluation frameworks have matured significantly. The lm-evaluation-harness [4] and OpenAI Evals [17] provide infrastructure for running automated evaluations at scale. Spe- cializedframeworkslikeRAGAS[2]andARES[21]targetretrieval-augmentedgenerationspecif- ically. Future frameworks will likely integrate capabilities that are currently fragmented. A critical open challenge is meta-evaluation: benchmarking the metrics themselves. While frameworks like RAGAS provide scores, their correlation with human judgment varies by domain. Future work must rigorously compare these frameworks to establish when each is most reliable. Unified interfaces across evaluation types would allow seamless switching between automated metrics, LLM-as-judge approaches, and human evaluation workflows. Automatic metric selection based on task characteristics could recommend appropriate evaluation strategies for new applications. Built-incalibrationbetweenautomatedandhumanscoreswouldimprovethereliabilityofscaled evaluation. Continuous evaluation pipelines integrated with CI/CD systems would make quality assessment a standard part of the development workflow. Standardized reporting formats would enhance reproducibility and enable meta-analysis across studies. 13.3 Observability and Production Monitoring LLM observability is nascent compared to traditional software monitoring. Few organizations have robust pipelines for assessing output quality in production beyond basic error rates and 32

latency metrics. Purpose-built observability platforms are emerging with capabilities specifically designed for LLM applications. Automated sampling and scoring of production outputs enables continuous qualitymonitoringwithoutmanualreviewofeveryinteraction. Driftdetectionidentifieschanges in prompt behavior or model responses over time, alerting teams to potential regressions. In- tegration with experimentation platforms facilitates online A/B testing of prompt variants and model configurations. Trace-level debugging for RAG and agentic systems allows engineers to understand the full context of individual failures. Real-time alerting for quality degradation enables rapid response to emerging issues. Investment in observability infrastructure will become increasingly critical as LLM applica- tions proliferate and quality expectations rise. 13.4 Agentic and Multi-Step Evaluation Most current evaluation focuses on single-turn interactions where the LLM receives a prompt and produces a response. Agentic systems that take actions over multiple steps present funda- mentally different evaluation challenges. New evaluation paradigms are emerging to address these challenges. Task completion rates across multi-step trajectories measure whether agents achieve their goals, not just whether individual steps are reasonable. Evaluation of intermediate reasoning and tool use assesses the quality of the decision-making process, not just outcomes. Sandboxed environments enable safe execution of agent actions during evaluation without risking real-world consequences. Human- in-the-loop evaluation for high-stakes actions provides oversight where automated assessment is insufficient. As agentic applications become more prevalent, evaluation methodologies for multi-step behavior will become essential. 13.5 Automated Red-Teaming Red-teaming is currently a largely manual process, relying on human creativity to discover failure modes and adversarial inputs [3, 20]. This approach is expensive and does not scale well. Automated red-teaming approaches are gaining traction. LLMs can generate adversarial prompts that probe for weaknesses in target systems. Evolutionary search methods can discover failure-inducing inputs through systematic exploration of the prompt space. Continuous red- teaming integrated into development pipelines would surface new vulnerabilities as systems evolve. Sharing of adversarial test cases across organizations could accelerate the discovery and mitigation of common failure modes. 13.6 Better LLM Judges CurrentLLMjudgesexhibitdocumentedbiasesanddonotreliablyassessallqualitydimensions. These limitations constrain the contexts in which LLM-as-judge can be trusted. Severaldirectionsshowpromiseforimprovement. Specializedjudgemodelsfine-tunedspecif- ically for evaluation tasks could outperform general-purpose models on assessment quality. En- semble methods combining multiple judges would reduce the impact of individual model biases. Calibration techniques could systematically adjust for known biases such as position preference or verbosity preference. Hybrid approaches with human oversight could provide LLM efficiency for routine cases while ensuring human review of uncertain or high-stakes judgments. 13.7 Personalization and Context-Dependent Evaluation As LLM applications become more personalized, evaluation must account for user-specific fac- tors. Quality definitions may vary based on user preferences, expertise levels, and interaction 33

history. Context-dependent success criteria mean that the same response may be appropriate in one situation and inappropriate in another. Future evaluation approaches must address these complexities by incorporating user-specific quality preferences into assessment, developing methods for evaluating long-term relationship quality rather than just single interactions, and ensuring privacy-preserving evaluation methods that do not expose sensitive user data. 14 Limitations This survey provides practical guidance for LLM evaluation, but it does not solve all challenges. We acknowledge the following limitations in the scope and applicability of this work. 14.1 What This Guide Does Not Cover This survey focuses on application-level evaluation and does not address several important re- lated topics. Foundation model training evaluation, including the assessment of pre-training dynamics, loss curves, and capability emergence, requires different methodologies than those presented here. Formal verification methods that provide mathematical guarantees about sys- tem behavior are not applicable to LLM outputs, which are inherently probabilistic. We do not provide legal guidance on compliance with AI regulations such as the EU AI Act or sector-specific requirements. Organizations should consult legal experts for regulatory matters. Similarly, we do not deeply address economic trade-offs between evaluation depth and cost, though practitioners must make these trade-offs in practice. The tooling landscape for LLM evaluation changes rapidly. We mention specific frameworks to provide context and concrete examples, but we do not comprehensively review or recommend specific commercial products. 14.2 Fundamental Challenges That Remain Open Several fundamental challenges in LLM evaluation remain unresolved and may not have com- plete solutions. There is no ground truth for subjective dimensions such as helpfulness, tone, and appropriateness. These qualities are inherently in the eye of the beholder, and no evaluation method provides definitive answers about whether a response is "good enough." Distribution shift remains a persistent challenge. Evaluation on curated test sets cannot guarantee production performance because real users formulate requests in ways that test sets cannot fully anticipate. The gap between offline and online quality is unavoidable. The adversarial arms race continues to evolve. As safety guardrails improve, adversarial prompts become more sophisticated. Evaluation cannot guarantee safety against novel attacks that have not yet been conceived. Model inscrutability limits diagnostic capability. We evaluate LLMs effectively as black boxes, and understandingwhya model fails on specific inputs remains difficult. This limits our ability to predict and prevent failures proactively. Finally, the rapidly evolving landscape means that best practices documented here may be superseded as the field advances. Readers should stay current with recent literature and adapt these recommendations as new methods emerge. 14.3 Limitations of Specific Methods Each evaluation method described in this survey has inherent limitations. LLM-as-judge ap- proaches exhibit known biases and have limited domain expertise, making them unsuitable as sole arbiters of quality. Human evaluation is expensive, slow, and subject to annotator fatigue and inconsistency. Automated metrics may not correlate with true quality and can be gamed 34

through optimization that exploits metric weaknesses. Data contamination becomes increas- ingly difficult to avoid as training corpora expand to encompass most of the public internet. 14.4 Scope of Applicability Thissurveyismostapplicabletotext-in, text-outLLMapplicationsthatoperateassingle-model systems rather than complex multi-agent orchestrations. The methods are best validated for English-language applications and commercial or enterprise deployments with defined quality requirements. Different evaluation strategies may be needed for multi-modal systems involving vision or audio, complex multi-agent architectures where multiple LLMs coordinate, applications in low- resource languages where evaluation methods are less validated, and creative applications with- out well-defined notions of correctness. We encourage researchers and practitioners working in these areas to adapt the principles presented here while developing specialized methodologies appropriate to their contexts. 15 Conclusion LLM applications require evaluation-driven iteration, not intuition-based prompt engineering. This paper provides a framework and reproducible harness showing that, in our experiments, generic prompt changes were not monotonic: adding generic “helpful” rules reduced extrac- tion pass rates and RAG compliance on our suites while improving instruction-following. A four-condition ablation isolated the mechanism: the system wrapper itself was benign; the degradation came from generic rules conflicting with task-specific constraints. The practical implication is that every prompt change should be validated against task- specific test suites before deployment. The evaluation harness, datasets, and experimental results in this paper provide a starting point for teams to build their own evaluation practices. Acknowledgments We thank the broader research community for developing the methods and benchmarks dis- cussed in this survey. AI Assistance Disclosure.This paper was developed with AI assistance for drafting and editing prose, code scaffolding, and reference-checking support (e.g., Anthropic Claude and Google Gemini via the Antigravity platform, and OpenAI tools within the Prism editor). The author directed the outline, ran and interpreted the experiments, verified factual claims and citations against the cited works, and edited the final manuscript. The author takes full respon- sibility for the content. 35

References [1] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitu- tional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022. [2] Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. Ragas: Automated evaluation of retrieval augmented generation.arXiv preprint arXiv:2309.15217, 2023. [3] Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Ka- davath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858, 2022. [4] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. Language model evaluation harness.https://github.com/EleutherAI/lm-evaluation-harness, 2023. [5] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks.International Conference on Machine Learning, pages 1321–1330, 2017. [6] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020. arXiv version; later appeared in 2021 venues. [7] Alon Jacovi, Avi Caciularu, Jonathan Mamou, and Yoav Goldberg. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks.arXiv preprint arXiv:2305.10160, 2023. [8] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation.ACM Computing Surveys, 55(12):1–38, 2023. [9] SauravKadavath,TomConerly,AmandaAskell,TomHenighan,DawnDrain,EthanPerez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221, 2022. [10] Klaus Krippendorff. Computing krippendorff’s alpha-reliability.Departmental Papers (ASC), 2011. [11] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- augmented generation for knowledge-intensive nlp tasks.Advances in Neural Information Processing Systems, 33:9459–9474, 2020. [12] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Ya- sunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models.Transactions on Machine Learning Research, 2023. [13] Chin-Yun Lin. Rouge: A package for automatic evaluation of summaries.Text Summa- rization Branches Out, pages 74–81, 2004. [14] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods.arXiv preprint arXiv:2109.07958, 2022. [15] Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization.arXiv preprint arXiv:2005.00661, 2020. 36

[16] Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evalu- ation of factual precision in long form text generation.arXiv preprint arXiv:2305.14251, 2023. [17] OpenAI. Openai evals.https://github.com/openai/evals, 2023. [18] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language mod- els to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35:27730–27744, 2022. [19] Arjun Panickssery, Samuel R Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations.arXiv preprint arXiv:2404.13076, 2024. [20] Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models.arXiv preprint arXiv:2202.03286, 2022. [21] Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. Ares: An auto- mated evaluation framework for retrieval-augmented generation systems.arXiv preprint arXiv:2311.09476, 2023. arXiv version; later appeared in 2024 venues. [22] Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de La- calle, and Eneko Agirre. Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark.arXiv preprint arXiv:2310.18018, 2023. [23] Thibault Sellam, Dipanjan Das, and Ankur P Parikh. Bleurt: Learning robust metrics for text generation.arXiv preprint arXiv:2004.04696, 2020. [24] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.arXiv preprint arXiv:2206.04615, 2022. BIG-bench; later appeared in Transactions on Machine Learning Research (TMLR). [25] Peiyi Wang, Lei Li, Liang Chen, Feifan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators.arXiv preprint arXiv:2305.17926, 2023. [26] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert.arXiv preprint arXiv:1904.09675, 2020. [27] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.arXiv preprint arXiv:2306.05685, 2023. [28] Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, et al. Promptbench: Towards eval- uating the robustness of large language models on adversarial prompts.arXiv preprint arXiv:2306.04528, 2023. 37

A Appendix A.1 Full Compatibility Checklist •Data Privacy: Ensure PII is stripped from logs before sending to cloud judges. •Token Limits: Check that RAG prompts + retrieved context fit within the model’s effective context window (often shorter than the theoretical max). •Rate Limiting: Implement exponential backoff for API calls to avoid 429 errors during batched evaluation. •Versioning: Hash prompt sets and data splits (e.g.,v1.0-sha256) to ensure regression tests are comparing apples to apples. •Fallbacks: Ensure the system fails gracefully (e.g., returns a default error message) if the LLM times out or returns malformed JSON. A.2 Human Evaluation Rubrics Helpfulness (1-5 Likert) •1 (Not Helpful): Irrelevant, refuses answer incorrectly, or hallmarks hallucinations. •3 (Somewhat Helpful): Addresses part of the query but misses nuance or follows wrong format. •5 (Very Helpful): Directly answers user intent, follows all formatting constraints, and is concise. Harmlessness (Binary) •Safe: No hate speech, PII leakage, or dangerous instructions. •Unsafe: Contains any prohibited content. A.3 Extended Monitoring Metrics •Token Usage: Input/output token counts per request to track cost spikes. •Cache Hit Rate: Percentage of similar queries served from semantic cache. •Throttling: Frequency of hitting provider rate limits. •User Feedback: Ratio of thumbs-up/down per model version. •Escalation Rate: Percentage of sessions where user requests a human agent. A.4 RAG Evaluation Checklist

1. Separate retrieval and generation evaluation.
2. Measure retrieval Recall@k and Precision@k.
3. Evaluate faithfulness to retrieved documents.
4. Check for correct but unsupported” responses.
5. Verify citation accuracy and coverage.
6. Test out-of-scope queries (information not in knowledge base).
7. Monitor retrieval latency and index freshness. 38

A.5 LLM-as-Judge Checklist

1. Use a different model than the one being evaluated.
2. Provide explicit rubrics in the evaluation prompt.
3. Request chain-of-thought reasoning before scores.
4. Randomize presentation order for comparisons.
5. Validate scores against human judgments on a sample.
6. Use multiple judge models where feasible.
7. Document known biases in your report. 39