Evaluate RAG Response Groundedness with OpenAI

This workflow automates the evaluation of Retrieval-Augmented Generation (RAG) response accuracy by computing the 'document groundedness' metric. It uses an OpenAI LLM to analyze whether the generated response contains information exclusively from provided retrieved documents, flagging potential hallucinations or deviations. Adapted from Google Vertex AI's pointwise groundedness template, it processes inputs like user questions, agent responses, and context documents to output a precise score.

The process begins with a manual trigger or data input (e.g., from a Google Sheet with sample RAG evaluations), fetches relevant documents via HTTP (demo uses Bitcoin datasheet), and structures the data for LLM assessment. OpenAI is prompted to identify ungrounded claims in the response by cross-referencing against documents, producing a score from 0 (poor grounding) to 1 (fully grounded). This is ideal for RAG pipelines requiring vector store retrieval.

Benefits include quick detection of model alignment issues, reduced manual review, and scalable evaluation for AI agents. Use cases span LLM fine-tuning, production monitoring of chatbots with document search, QA for enterprise RAG apps, and benchmarking retrieval quality. Requires n8n 1.94+ and OpenAI credentials; sample data provided via Google Sheet for instant testing.