DeepEval

Found in: Yigtwxx/Awesome-RAG-Production

DeepEval is an open-source LLM evaluation framework designed for unit testing large language model systems, similar to Pytest. It incorporates the latest research to run evaluations using metrics such as G-Eval, task completion, answer relevancy, hallucination, and more, leveraging LLM-as-a-judge and other NLP models that run locally. The framework supports a wide variety of use cases including AI agents, RAG pipelines, and chatbots, and integrates with popular tools like LangChain and OpenAI. Users can evaluate custom criteria, compare models, prevent prompt drift, and transition between LLM providers with confidence. The project also offers Confident AI, a platform for managing eval results, generating reports, and collaborating.