Evaluator Rag Recall

You will be given a GROUND_TRUTH_PASSAGE (the ideal response) and an LLM_RETURNED_PASSAGE returned by an LLM as part of a Retrieval-Augmented Generation (RAG) process. Your task is to evaluate the LLM_RETURNED_PASSAGE for RECALL to assess how well it captures the information from the GROUND_TRUTH_PASSAGE. Please approach this evaluation with careful consideration and strive for precision in your assessments.

Evaluation Steps:

1. Thoroughly analyze the given GROUND_TRUTH_PASSAGE and LLM_RETURNED_PASSAGE.

2. Assess the LLM_RETURNED_PASSAGE for RECALL: Evaluate the extent to which the LLM_RETURNED_PASSAGE contains the key information present in each passage of the GROUND_TRUTH_PASSAGE. The GROUND_TRUTH_PASSAGE is structured as follows:

   [ESSENTIAL PASSAGE]
   ... some text ...
   
   [ESSENTIAL PASSAGE]
   ... some text ...
   
   [BONUS PASSAGE]
   ... some text ...
   
   [BONUS PASSAGE]
   ... some text ...
   

   • After carefully analyzing the contents of [ESSENTIAL PASSAGE] and [BONUS PASSAGE], evaluate the extent to which the LLM_RETURNED_PASSAGE contains the contents of each [ESSENTIAL PASSAGE] and [BONUS PASSAGE].
   • Score each ESSENTIAL passage on a scale of 1-5.
   • Score each BONUS passage on a scale of 0-5.
   • Provide a detailed, step-by-step analysis of your reasoning process, explicitly explaining how you arrived at your score for each passage.

3. Calculate the final RECALL score using the following formulas:

   RECALL_Essential_Formula: (sum of ESSENTIAL scores) / E
   
   RECALL_Bonus_Formula: (sum of BONUS scores) / (E * B * 2)
   
   RECALL_Combined_Formula: RECALL_Essential_Formula + RECALL_Bonus_Formula
   
   RECALL_Final_Score_Calculation: min(5, RECALL_Combined_Formula)
   
   RECALL_Final_Score: This is the final result of RECALL_Final_Score_Calculation
   

   Where:

   • Let E be the number of ESSENTIAL passages and B be the number of BONUS passages.
   • Only express decimals to a maximum of 2 decimal places (round up from 3).
   • If the result exceeds 5, cap it at 5.

4. Format your evaluation as shown in the Example Outputs below.

Example Outputs:

Example 1:

• RECALL_Reasoning: [ESSENTIAL PASSAGE] 1: ... [Provide detailed analysis and reasoning for the score] ... Score: 4 [ESSENTIAL PASSAGE] 2: ... [Provide detailed analysis and reasoning for the score] ... Score: 3 [BONUS PASSAGE] 1: ... [Provide detailed analysis and reasoning for the score] ... Score: 4 [BONUS PASSAGE] 2: ... [Provide detailed analysis and reasoning for the score] ... Score: 0

• RECALL_Essential_Formula: (4 + 3) / 2 = 3.5

• RECALL_Bonus_Formula: (4 + 0) / (2 * 2 * 2) = 0.5

• RECALL_Combined_Formula: 3.5 + 0.5 = 4

• RECALL_Final_Score_Calculation: min(5, 4) = 4

• RECALL_Final_Score: 4

Example 2:

• RECALL_Reasoning: [ESSENTIAL PASSAGE] 1: ... [Provide detailed analysis and reasoning for the score] ... Score: 4 [ESSENTIAL PASSAGE] 2: ... [Provide detailed analysis and reasoning for the score] ... Score: 4 [ESSENTIAL PASSAGE] 3: ... [Provide detailed analysis and reasoning for the score] ... Score: 3 [BONUS PASSAGE] 1: ... [Provide detailed analysis and reasoning for the score] ... Score: 0

• RECALL_Essential_Formula: (4 + 4 + 3) / 3 = 3.67

• RECALL_Bonus_Formula: (0) / (3 * 1 * 2) = 0

• RECALL_Combined_Formula: 3.67 + 0 = 3.67

• RECALL_Final_Score_Calculation: min(5, 3.67) = 3.67

• RECALL_Final_Score: 3.67

Example 3:

• RECALL_Reasoning: [ESSENTIAL PASSAGE] 1: ... [Provide detailed analysis and reasoning for the score] ... Score: 5 [ESSENTIAL PASSAGE] 2: ... [Provide detailed analysis and reasoning for the score] ... Score: 5 [BONUS PASSAGE] 1: ... [Provide detailed analysis and reasoning for the score] ... Score: 5

• RECALL_Essential_Formula: (5 + 5) / 2 = 5

• RECALL_Bonus_Formula: (5) / (2 * 1 * 2) = 1.25

• RECALL_Combined_Formula: 5 + 1.25 = 6.25

• RECALL_Final_Score_Calculation: min(5, 6.25) = 5

• RECALL_Final_Score: 5

Example 4:

• RECALL_Reasoning: [ESSENTIAL PASSAGE] 1: ... [Provide detailed analysis and reasoning for the score] ... Score: 1 [ESSENTIAL PASSAGE] 2: ... [Provide detailed analysis and reasoning for the score] ... Score: 2

• RECALL_Essential_Formula: (1 + 2) / 2 = 1.5

• RECALL_Bonus_Formula: (0) / (2 * 0 * 2) = 0

• RECALL_Combined_Formula: 1.5 + 0 = 1.5

• RECALL_Final_Score_Calculation: min(5, 1.5) = 1.5

• RECALL_Final_Score: 1.5

GROUND_TRUTH_PASSAGE: {passage_gt}

LLM_RETURNED_PASSAGE: {passage_llm}