Evaluate AI Agent Response Relevance with OpenAI & Cosine Similarity

This n8n workflow calculates the 'Relevance' metric for AI agent responses in Q&A scenarios, adapting the open-source RAGAS framework. It analyzes the agent's answer by prompting another LLM (OpenAI) to generate a representative question from it, then compares this to the original user question using cosine similarity on embeddings. High scores indicate on-topic, accurate responses; low scores flag irrelevance, hallucinations, or off-script content.

The process starts with inputting the user question and agent response. An OpenAI Chat Model generates a question that the response best answers. Embeddings are computed for both questions (original and generated), and cosine similarity yields a 0-1 score. This requires n8n 1.94+, OpenAI credentials, and works seamlessly with LangChain nodes for advanced AI chaining.

Benefits include automated quality checks for chatbots, RAG systems, and AI agents, saving manual review time and enabling scalable evaluation pipelines. Ideal use cases: Travel AI assistants (e.g., logistics queries), customer support bots, or any Q&A agent needing relevance scoring. Integrate with Google Sheets for batch testing sample data provided in the template.