Evaluator Rag Precision

You will be given a USER_QUERY and an LLM_RETURNED_PASSAGE returned by an LLM as part of a Retrieval-Augmented Generation (RAG) process. Your task is to evaluate the LLM_RETURNED_PASSAGE for PRECISION to assess its relevance to the USER_QUERY. Please approach this evaluation with careful consideration and strive for precision in your assessments.

Evaluation Steps:

1. Thoroughly analyze the given USER_QUERY and LLM_RETURNED_PASSAGE.

2. For each passage within the LLM_RETURNED_PASSAGE (separated by "--- PASSAGE_DIVIDER ---"), assess PRECISION (1-5) as follows:

   • Analyze the passage thoroughly, identifying information that directly answers the USER_QUERY and any content that may be off-topic or unnecessary.
   • Evaluate how focused and relevant the passage is in addressing the specific USER_QUERY. Consider the proportion of information that directly answers the query versus any unnecessary content.
   • Assign a score from 1 to 5 based on the relevance of the passage to the USER_QUERY. A higher score indicates that the passage contains predominantly relevant information that directly addresses the USER_QUERY, with minimal or no extraneous details. Lower scores should be given to passages that include significant amounts of off-topic or unnecessary details.
   • Provide a detailed, step-by-step analysis of your reasoning process, explicitly explaining how you arrived at your score for each passage.

3. Calculate the PRECISION_Score using the following formula:

   • Sum up all individual passage scores and divide by the total number of passages (count the number of "--- PASSAGE_DIVIDER ---" separators and add 1 to get the total number of passages).
   • PRECISION_Score = (Sum of all passage scores) / (Total number of passages)
   • Round the PRECISION_Score to two decimal places.

4. Format your evaluation as shown in the Example Output below.

Example Output:

• PRECISION_Reasoning: Passage 1: ... [Provide detailed analysis and reasoning for the score] ... Score: 4 Passage 2: ... [Provide detailed analysis and reasoning for the score] ... Score: 3 Passage 3: ... [Provide detailed analysis and reasoning for the score] ... Score: 4

• PRECISION_Formula: (4 + 3 + 4) / 3 = 3.67

• PRECISION_Score: 3.67

USER_QUERY: {query}

LLM_RETURNED_PASSAGE: {passage_llm}