5_Evaluation

---

Table of Contents

1. Importance and Challenges of Evaluation
2. Metrics Used in Evaluation
3. Understanding nDCG
4. Understanding Recall@K
5. Golden Test Sets (Annotated Data)
6. Internal Demos and User Feedback
7. A/B Testing in Production
8. Eyeballing Results (Initial Step)
9. Domain-Specific Evaluation Data

---

---

Why is evaluation so critical when developing search and RAG systems with embeddings and rerankers, and what are the main challenges involved?

Summary (E-commerce Example):

• Evaluation is crucial to know if changes actually improve search on sites like Samsung.com, e.g., does a new reranker better surface relevant Samsung accessories for a specific phone model?
• It helps justify adding components like rerankers, weighing the performance gain (finding better Samsung TV deals) against potential latency increases on Samsung.com searches.
• It's essential for checking if general models work well on specific Samsung product data (out-of-domain testing) and comparing different search setups.
• Key challenges include the difficulty and cost of creating labeled datasets (e.g., pairing queries like "best Samsung camera phone" with truly relevant user reviews) and the subjectivity of relevance, especially for complex product comparisons on Samsung.com.
• Standard benchmarks often don't reflect real-world performance on specific Samsung product niches or user search behaviors.

Answer:

Based on the sources and our conversation history, Evaluation is a critical and often challenging aspect when working with Embeddings and Rerankers within the larger context of Retrieval Augmented Generation (RAG) and search systems. It serves to measure the effectiveness of these components and the overall system.

Why Evaluation is Important:

• Assess Impact: Evaluation is necessary to determine whether a new embedding adds additional signal or information to the retrieval process (e.g., finding more relevant Samsung products). It is crucial for figuring out if reranking actually helps improve performance.
• Justify Complexity: It helps justify adding potentially slower layers like rerankers on top of fast vector search by demonstrating tangible benefits.
• Domain Specificity: Evaluation is needed to see how embedding models perform on data that is out of the domain they were trained on (e.g., how a general web model performs on Samsung.com's specific product catalog), which is a significant limitation. You need to evaluate models specifically on your task and your domain.
• Compare Approaches: Evaluation allows you to collect evidence to compare different implementations or interventions in your search application (e.g., comparing two different embedding models for searching Samsung support articles).
• Measure Performance: It helps assess the performance of rerankers, especially in specific use cases and with challenges like long context (e.g., understanding relevance within long Samsung product reviews).
• Data Quality: Evaluation helps measure the quality of your training data and check if it is in the expected distribution.

Challenges in Evaluation:

• Data Scarcity & Cost: A major challenge is the lack of good evaluation data; it's described as something that "sucks" and nobody wants to do. Annotating retrieval data is very difficult and expensive, often requiring domain-specific background knowledge (e.g., understanding Samsung product features to label relevance). Creating a "good" data set is the most difficult part.
• Subjectivity: Subjectivity is a problem, as "people suck at saying what is similar and what it's not". It is very hard to explicitly state why two pieces of text (like product descriptions on Samsung.com) are similar.
• Misleading Benchmarks: Leaderboards and benchmarks can be misleading as the newest models might be heavily overfitted on the test data, not reflecting true out-of-domain performance relevant to a specific site like Samsung.com.
• Long Tail Issues: It's hard for models to perform well on "long tail" or new named entities (e.g., a brand new Samsung accessory model) they haven't seen during training, making evaluation difficult.
• Black Box Nature: Deep learning systems can feel like black boxes, making it hard to pinpoint the exact cause of an issue, though it's often related to the training data.

In summary, evaluation in RAG and search systems using embeddings and rerankers is essential but difficult, primarily due to the challenge of creating good labeled data specific to the domain (like Samsung's e-commerce environment). It involves a mix of quantitative metrics and qualitative human assessment and is deeply connected to understanding and improving data quality and model performance.

---

What kind of metrics are typically used to evaluate search and ranking systems, and how are they applied in practice?

Summary (E-commerce Example):

• Standard Information Retrieval (IR) metrics like nDCG and Recall@K are used to quantify search performance, measuring how well relevant Samsung products appear at the top of search results on Samsung.com.
• Rerankers provide relevance scores for items like Samsung user manuals or product reviews, often used with a threshold to decide what to show a user searching on Samsung.com.
• These semantic scores can be combined with other factors in a post-processing step, such as product recency (prioritizing the latest Samsung Galaxy phones) or popularity scores derived from user interactions (clicks, purchases) on Samsung.com.
• Setting appropriate thresholds for these scores to filter relevant Samsung warranty documents or product recommendations remains a practical challenge.

Answer:

Based on the sources, evaluation and metrics are fundamental aspects of building and refining retrieval systems, particularly within RAG and search applications.

Evaluation Context:

• Data Focus: Looking at your data is a massive part of building good models, especially when integrating a ranker. Manual checking is crucial as tooling for basic checks (e.g., data distribution) is lacking.
• Data Generation Challenge: A significant challenge is the lack of good evaluation data. When using synthetic data for training, evaluation data must not come from the same synthetic distribution to reflect real users. A mix is preferred.
• Evaluation Methods:
  • When labels are unavailable: A/B testing or eyeballing results from common queries.
  • For formal evaluation: Creating a golden test set (e.g., 50-100 carefully annotated queries) is key, though difficult.
• Evaluation Cycles: Often cross-functional, involving initial engineer checks ("VIP checks"), then tests with experts or test customers, collecting diverse feedback.
• Model Selection: Crucial to evaluate embeddings on your specific task and domain using real queries and labeled documents, focusing on out-of-domain performance, not just possibly overfitted benchmarks.

Metrics Used:

Metrics are used to measure the performance of models and systems:

• Similarity/Distance Metrics:
  • In semantic search, a distance metric like cosine similarity compares query and document embeddings. This is just one dimension of relevance.
  • Hamming distance is mentioned as fast for quantized embeddings on modern CPUs.
• Relevance Scores:
  • Rerankers output a score indicating relevance/similarity between a query and document.
  • Cross-encoders output a relevance score or classify pairs.
  • BM25 provides scores in systems like OpenSearch.
  • LLMs can also be used to score chunks (e.g., true/false for relevance).
  • Scores are often used with a threshold to filter results, but setting this threshold is challenging and often requires trial and error.
• Information Retrieval (IR) Ranking Metrics:
  • Common metrics used by the IR community include:
    • nDCG (Normalized Discounted Cumulative Gain): A rich metric considering the actual ranking position of relevant items.
    • Recall@K: Measures how many ground truth relevant documents are in the top K retrieved results.
    • MRR (Mean Reciprocal Rank).
    • (D)CG (Discounted Cumulative Gain).
• Combining Factors in Scoring:
  • Beyond semantic similarity, other factors ("mates") like recency, trustworthiness (PageRank, citations, popularity/clicks) can be added to scoring during reranking, often combined using weights in post-processing.
  • Alternatively, structured metadata (dates, locations) can be encoded into the document text for the ranker to consider directly (requires specific model training).
• Multimodal Weighting:
  • In combined multimodal embeddings (e.g., text + image), different modalities can be weighted differently, offering explainability and tuning options.

---

Could you explain what nDCG is and why it's considered a useful metric for evaluating search ranking quality?

Summary (E-commerce Example):

• nDCG (Normalized Discounted Cumulative Gain) is a key metric because it evaluates the quality of the ranking order in search results, not just the presence of relevant items.
• A high nDCG score on Samsung.com indicates that the most relevant products (e.g., the specific Samsung TV model searched for, or highly compatible Galaxy phone accessories) are correctly placed at the very top of the results list.
• It's considered a "rich metric" useful for comparing different ranking algorithms for Samsung product searches, as it rewards placing highly relevant items higher than less relevant ones.
• While valuable, aggregated scores like nDCG might not expose all specific ranking errors (e.g., why a completely irrelevant Samsung appliance part sometimes appears), requiring supplementary manual checks.

Answer:

Based on the sources, nDCG (Normalized Discounted Cumulative Gain) is discussed as a key metric for performance evaluation, especially within the Information Retrieval (IR) community.

• Definition and Purpose:
  • nDCG is explicitly named as one of the common metrics used to measure the performance of components like rerankers.
  • It is described as a "rich metric".
  • Its value lies in the fact that it takes into account the actual ranking of documents. Unlike simple binary labels (relevant/irrelevant) or metrics like Recall (which only checks presence in the top K), nDCG assigns higher scores when more relevant items are placed higher in the ranking. It discounts the value of relevant items found lower down the list.
• Usage Context:
  • It's used alongside other IR metrics like Recall and MRR.
  • When labeled data (a "golden test set") is available, nDCG (or its variant DCG) can be calculated to evaluate if a change, like adding a new embedding, has improved the ranking quality ("lifted DCG").
• Limitations:
  • While useful, nDCG provides an aggregated score.
  • Sometimes "really strange failed cases" might occur that are not apparent when evaluating solely based on nDCG or other common metrics derived from academic benchmarks. This highlights the need for supplementary evaluation methods.
• Dependency on Data:
  • Calculating nDCG effectively requires a well-annotated "golden test set" where the relevance level or ideal ranking of documents for given queries is defined. Creating this high-quality labeled data is noted as being the most difficult part of the evaluation process.

In essence, nDCG is favored because it provides a nuanced measure of ranking quality that reflects the user experience – finding the best results at the very top is better than finding them buried lower down.

---

What does Recall@K measure, and how is it used alongside other metrics like nDCG in evaluation?

Summary (E-commerce Example):

• Recall@K measures the proportion of truly relevant items (e.g., all compatible Samsung smartwatches for a specific Galaxy phone model) that appear within the top K search results displayed on a site like Samsung.com.
• It helps ensure that users are likely to see the relevant options quickly among the first few results, answering "Did we find most of the right Samsung accessories?"
• It complements metrics like nDCG, which focuses on the exact ranking order of those Samsung products ("Are the most relevant ones at the very top?").
• Calculating Recall@K requires a "golden set" defining which Samsung products are considered relevant for specific queries. Like other aggregated metrics, it should be used alongside manual checks.

Answer:

Based on the sources, Recall@K is a metric used in the information retrieval community for evaluation.

• Definition:
  • Recall@K specifically answers the question: how many ground truth documents are in those top K documents retrieved?
  • It measures the fraction of relevant documents that are successfully retrieved within the top K results.
• Purpose and Value:
  • It is considered a really informative metric for evaluating components like rerankers.
  • It helps assess whether the system is effective at bringing relevant items into the initial set of results shown to the user.
• Usage with Other Metrics:
  • Recall@K is often used alongside other common IR metrics like nDCG (Normalized Discounted Cumulative Gain).
  • While Recall@K checks for the presence of relevant items within the top K, nDCG provides a richer measure by considering their exact ranking position, rewarding systems that place more relevant items higher up.
  • Other metrics like MRR (Mean Reciprocal Rank) are also used in conjunction.
• Dependency on Data:
  • Like nDCG and MRR, calculating Recall@K requires a "golden test set" – a dataset that has been carefully annotated with the ground truth relevance labels for each query-document pair. Creating this high-quality dataset is noted as being the most difficult part of performance monitoring.
• Limitations:
  • As an aggregated metric, Recall@K might not reveal specific failure modes or unusual ranking behavior that could be observed through manual checks or demos.

In essence, Recall@K focuses on the completeness of relevant results within the top K positions, making it a valuable metric for ensuring users are presented with a good selection of relevant items early on.

---

What are 'golden test sets' in the context of search evaluation, and why are they so important despite being difficult to create?

Summary (E-commerce Example):

• Golden test sets are crucial collections of representative user queries (e.g., typical searches on Samsung.com like "compare Galaxy S24 models") paired with manually annotated relevant/irrelevant documents (like specific Samsung product pages, comparison tables, or help articles).
• They are essential for accurately evaluating model performance specifically within the target domain (e.g., Samsung's complex e-commerce environment), as general models often fail on niche Samsung product features or terminology.
• Creating these sets is challenging and costly, requiring domain expertise (e.g., understanding Samsung product lines and compatibility) and significant annotation effort, often lacking good tooling support.
• These sets form the basis for calculating key metrics like nDCG and Recall@K when assessing improvements to Samsung.com's search relevance and ranking. Despite the difficulty, they are fundamental for building robust and reliable search.

Answer:

Based on the sources, golden test sets, often composed of annotated data, are highlighted as a crucial component for effective evaluation, particularly within retrieval and reranking systems.

• Definition and Purpose:
  • A golden test set for retrieval/reranking typically consists of a collection of representative queries (ideally mirroring external customer queries) paired with labeled documents.
  • These labels indicate which documents are relevant (positives) and irrelevant (negatives) for each specific query within a particular domain (e.g., for searches on Samsung.com).
  • The primary purpose is to evaluate the performance of models (like embeddings, rerankers) and the overall system on a specific task and domain, providing a benchmark for improvement.
• Importance:
  • Evaluating Out-of-Domain Performance: Embeddings notoriously perform poorly "out of domain." Public benchmarks can be misleading due to overfitting. A custom golden test set is essential to accurately test model performance on your specific data and retrieval problem (e.g., how well a model finds relevant Samsung support documents, not just generic web pages).
  • Calculating Metrics: Once available, a golden test set enables the calculation of standard Information Retrieval (IR) metrics like nDCG, Recall@K, and MRR, providing quantitative measures of system performance.
  • Evaluating Component Contribution: They can be used to evaluate if adding new components, like a specific type of embedding, provides "additional signal or information".
• Creation Challenges:
  • Difficulty and Cost: Creating good evaluation data is acknowledged as difficult, expensive, undesirable work ("it sucks"), and a key missing piece in the current tooling landscape.
  • Annotation Effort: It requires significant time annotating and reviewing data to ensure label quality. This is particularly hard for long, complex documents requiring domain expertise (e.g., annotating technical manuals for Samsung products).
  • Subjectivity: Defining "similarity" or "relevance" consistently can be inherently difficult.
  • Tooling: Lack of good tools for generating evaluation data and measuring data quality is a major hurdle. Manual checking and reading through data are often necessary.
  • Size: While crucial, they don't necessarily need to be enormous; a carefully curated set of 50-100 queries against a relevant corpus can be effective.
• Usage Limitations:
  • Metrics derived from golden sets provide aggregated scores and may not reveal all "strange failed cases." Human-driven evaluation (demos, user feedback) remains necessary.

In summary, golden test sets are meticulously annotated datasets mirroring real-world queries and domain-specific documents. They are fundamental for reliable evaluation, calculating key metrics, and overcoming the limitations of generic benchmarks, despite the significant challenges associated with their creation and maintenance.

---

Beyond quantitative metrics, how are internal demos and user feedback used to evaluate and improve search systems?

Summary (E-commerce Example):

• Quantitative metrics like nDCG don't tell the whole story. Internal demos at Samsung might visually showcase how a new search algorithm handles ambiguous queries like "big screen phone" versus the old one, allowing for qualitative assessment.
• Direct user feedback from Samsung.com is vital. Real, often messy queries ("Galaxy buds keep disconnecting from my Samsung TV") reveal critical issues and user pain points that standard tests or metrics might miss.
• Analyzing this feedback helps identify weaknesses (e.g., difficulty finding software update instructions for a specific Samsung washing machine) and prioritize improvements for future updates to the Samsung website or customer support portals.
• Feedback can also uncover unexpected successful use cases or inspire new features for the Samsung ecosystem.

Answer:

Based on the sources, internal demos and user feedback are crucial qualitative components of the evaluation process, complementing quantitative metrics.

Internal Demos and Testing:

• Purpose: Demos serve as a key step to get a sense of how the model is actually doing and to showcase capabilities. Developers build demos to manually compare results (e.g., initial retrieval vs. reranked results for specific Samsung product queries) and illustrate value (e.g., showing how reranking improves relevance in a RAG setup for Samsung support documents).
• Process: Companies like Cohere build "a lot of demos" and use internal users for testing before release. This involves manually checking results for specific queries, examining relevance scores, testing different query formulations, and sometimes even using LLMs to verify code functionality.
• Identifying Issues: This hands-on approach helps catch "really strange failed cases" that aggregated metrics from benchmarks might miss.

User and Customer Feedback:

• Importance: Considered a "big learning" and crucial for understanding real-world performance. Real user queries are often messy (spelling mistakes, poor grammar) and differ significantly from clean training data. Feedback helps bridge this gap.
• Driving Development: Customer requests and feedback directly influence development focus. Analyzing user inputs helps identify weaknesses in models (e.g., problems with chunking long Samsung manuals, noted as a user "complaint"). This informs improvements in subsequent releases.
• Discovering Use Cases: Hearing how customers use models can reveal surprising and effective new applications (e.g., using a reranker for API classification) that developers hadn't anticipated.
• Production Assessment: In production, user feedback is invaluable. While golden test sets are recommended, manual checks and feedback are often the quickest ways to assess how a model performs in its specific use case (e.g., how well search works on the live Samsung.com site).
• Data for Fine-tuning: Collecting representative queries and documents from users is essential for fine-tuning models to specific use cases, especially given the difficulty of obtaining clean real-world training data.

Overall Role in Evaluation:

• Internal demos provide visual inspection and qualitative assessment, catching issues metrics might hide.
• User feedback provides real-world performance data, highlights practical limitations, reveals unexpected applications, and directly guides improvements.
• This iterative loop (build demo -> test internally -> gather user feedback -> identify issues -> improve model/data) is central to developing robust and effective search and RAG systems.

In summary, while metrics offer quantitative scores, internal demos and user feedback provide essential qualitative insights into how systems perform with real queries, real data (like the diverse content on Samsung.com), and real user expectations, driving targeted improvements.

---

When and how is A/B testing used for evaluating changes in search systems, especially when labeled data isn't available?

Summary (E-commerce Example):

• A/B testing is a key evaluation method used in production environments like Samsung.com, particularly when comprehensive labeled datasets ("golden sets") covering all possible user queries are unavailable or impractical to create.
• It involves deploying two versions of the search system simultaneously (e.g., the existing search logic vs. one with a new embedding model for Samsung TVs) to different, comparable segments of Samsung.com users.
• Performance is compared based on real user behavior and key business metrics, such as differences in click-through rates on Samsung product links, add-to-cart actions, session duration, or ultimately, conversion rates. This provides direct evidence of the change's impact in a live setting.

Answer:

Drawing on the sources, A/B testing is discussed as a valuable evaluation method, particularly useful in specific circumstances:

• When Labels Are Unavailable: A/B testing is explicitly recommended as an approach if you don't have labels (i.e., a pre-annotated golden test set). When labels are available, you can directly calculate IR metrics like MRR or DCG. Without labels, A/B testing provides a way to measure impact using live user interactions.
• How It Works:
  • It involves trying the system with the new change (e.g., a new embedding, a different reranker configuration) and comparing it against the baseline (without the change).
  • This comparison is run for a limited time in a production or near-production environment.
  • The core idea is to observe the results – specifically, how the change impacts user behavior or relevant business/system metrics.
• Preliminary Step: Before committing to a full A/B test, a simpler, quicker check involves "eyeballing" the results for common queries to get a subjective sense of whether the change seems preferable. This can help justify the effort of setting up an A/B test.
• Context within Production Evaluation:
  • A/B testing fits into the broader challenge of evaluating systems with real-world user data, which is often messy and differs from training data.
  • It complements other evaluation methods used in production, such as monitoring based on golden test sets (if available), collecting user feedback, and performing manual checks.
  • It directly measures the impact on the end-user experience and business goals, providing evidence beyond offline metric calculations.

In essence, A/B testing serves as a practical method for evaluating the real-world impact of changes to a search system by observing user behavior, especially when constructing comprehensive labeled datasets for offline evaluation is not feasible.

---

What role does 'eyeballing' or manually checking results play in the search evaluation process?

Summary (E-commerce Example):

• "Eyeballing" results refers to the practice of manually reviewing the search output for a set of common or important queries as an initial, qualitative evaluation step.
• For instance, before deploying a change on Samsung.com, a developer might quickly check the top results for queries like "Samsung OLED TVs", "Galaxy Watch bands", or "refrigerator water filter replacement".
• This quick, subjective assessment helps get an early "gut feeling" whether changes seem promising or potentially problematic, often serving to justify the effort required for more formal evaluation like A/B testing or detailed metric analysis on a golden set.
• It's a practical first-pass check before diving into more resource-intensive, quantitative evaluation methods.

Answer:

Based on the sources, "eyeballing" results plays a role as a practical, initial step in the evaluation process.

• Definition: Eyeballing involves manually checking and subjectively assessing the search results, typically for a set of common or important queries.
• Purpose and Timing:
  • It is described as a technique used particularly when traditional labels might not be available or before setting up more formal tests.
  • It serves as a first step or a quick way to get a sense of how a model or system change is performing.
  • You would "try your most common queries" and assess if you "like it more than before."
  • The goal is often to justify proceeding with more rigorous evaluation methods, such as running an A/B test or a more formal evaluation script.
• Place in Evaluation Spectrum:
  • Eyeballing falls under the umbrella of manual checks and is related to building demos or using internal tools for assessment.
  • It complements more quantitative approaches like calculating IR metrics (nDCG, Recall@K, MRR) based on golden test sets.
  • It acknowledges that aggregated metrics sometimes miss "strange failed cases," making direct observation valuable.
• Pragmatism: It's presented as a pragmatic approach, especially given the difficulty and cost associated with creating comprehensive, perfectly annotated evaluation datasets.

In summary, eyeballing results is an informal, qualitative check used early in the evaluation process. It provides a quick, preliminary assessment of whether a change appears beneficial, helping to decide if further, more costly evaluation efforts are warranted.

---

Why is using domain-specific data so critical for evaluating search models, and what are the challenges in creating it?

Summary (E-commerce Example):

• Using domain-specific evaluation data is critical because general models, often trained on broad web data, perform poorly when applied to specialized contexts like Samsung.com. They fail to understand specific Samsung product jargon, features (e.g., "AI EcoBubble"), or the nuances of the product catalog.
• Evaluation must use data reflecting the target environment – e.g., actual user queries submitted on Samsung.com paired with correctly labeled relevant Samsung product pages, manuals, or support documents. Relying on generic benchmarks is misleading.
• Creating this data is a major challenge:
  • It requires deep domain expertise (understanding Samsung's vast product catalog and technical specifications).
  • Annotation is costly and time-consuming.
  • Defining "relevance" for complex Samsung product comparisons can be subjective.
  • There's a lack of specialized tooling to aid the creation and quality control process.

Answer:

Based on the sources, using domain-specific evaluation data is highlighted as critically important due to model limitations, while its creation presents significant challenges.

Why Domain-Specific Evaluation Data is Critical:

• Poor Out-of-Domain Performance: Text embedding models are explicitly described as performing "very terrible" or "way worse" out of domain. A model trained on Wikipedia will likely fail on community forum data, news data, scientific papers, or a specific e-commerce catalog like Samsung.com. This is considered a "massive limitation" of current embeddings.
• Meaningful Evaluation: To truly understand how a model will perform for your specific task (e.g., searching Samsung product documentation), you must evaluate it on data from that domain.
• Misleading Benchmarks: Public leaderboards or benchmarks often feature models "overfitted on them", potentially even trained on the test sets. Relying solely on these benchmarks doesn't accurately predict performance on your unique data. You must check out-of-domain performance.
• Capturing Nuance: Domain-specific data is needed to evaluate if models handle specific terminology, entities (like Samsung product names), and user query styles prevalent in that domain.
• Real-World Relevance: Even aggregated metrics on standard benchmarks might miss failure cases specific to your domain; evaluation needs to reflect real-world usage patterns.

Challenges in Creating Domain-Specific Evaluation Data:

• Difficulty and Cost: Creating good evaluation data is repeatedly described as difficult, expensive, time-consuming, and undesirable work ("it sucks"). It's identified as a major missing piece in the current ecosystem.
• Annotation Burden: The process requires manual annotation, labeling documents as relevant/irrelevant for given queries. This is hard, especially for long/complex documents, and requires domain-specific background knowledge (e.g., understanding Samsung product features to judge relevance accurately). Ensuring annotation quality requires significant time and review.
• Subjectivity: Defining "similarity" or "relevance" consistently, especially for longer texts or complex topics (like comparing different Samsung phone models), is inherently difficult.
• Lack of Tooling: There's a noted lack of proper tooling to facilitate the creation of evaluation data or even perform basic quality checks like verifying data distribution.
• Manual Effort: Currently, creating evaluation data often relies heavily on manual checking, reading through inputs/outputs – described as "pretty boring" but the "most relevant part."
• Synthetic Data Pitfalls: While synthetic data generation is explored, using synthetically generated data for evaluation that comes from the same distribution as synthetic training data is cautioned against, as it doesn't reflect real user distributions.

In summary, domain-specific evaluation data is non-negotiable for accurately assessing and improving retrieval systems due to the significant performance drop-off of models outside their training domain. However, creating this vital data remains a major bottleneck due to the associated cost, effort, required expertise, and lack of adequate tooling.

---