Evaluate AI Agent Response Correctness with OpenAI & RAGAS

This n8n workflow implements the 'Correctness' evaluation metric from the RAGAS open-source project to assess AI agent responses. It compares the agent's verbose, conversational output against multiple ground truths, classifying extracted statements into True Positives (present in both), False Positives (in agent response but not ground truth), and False Negatives (in ground truth but missing from response). An average semantic similarity score is also computed using OpenAI embeddings.

The final score combines classification accuracy and similarity, providing a robust measure of response fidelity. High scores indicate precise and comprehensive answers, while low scores highlight hallucinations, omissions, or inadequate training data. This is ideal for iterative AI agent improvement in production environments.

Benefits include automated, scalable evaluation without manual review, integration with n8n's ecosystem for chaining into larger pipelines, and adaptability to domains like finance where accuracy is critical. Use cases span validating chatbots for budgeting advice, financial analysis agents, or compliance checkers in Finance & Accounting workflows.

Requirements: n8n 1.94+, OpenAI API key. Sample data available in linked Google Sheet for quick testing.