Neura Market
TruLens logo

TruLens

Free

Evaluation and Tracking for LLM Experiments and AI Agents

FreeFree tier
Inputs: text
Type
Open Source
Company
TruEra
Visit Website GitHub (38 stars)

About TruLens

TruLens is an open-source tool for systematically evaluating and tracking LLM experiments and AI agents. It provides fine-grained, stack-agnostic instrumentation built on OpenTelemetry, enabling comprehensive tracing of every function call, LLM generation, retrieval, and tool invocation. TruLens includes a library of feedback functions (e.g., the RAG Triad and Honest, Harmless, Helpful evals) and supports multiple LLM providers (OpenAI, LiteLLM, Google Gemini, AWS Bedrock, Snowflake Cortex, HuggingFace, LangChain) as well as app framework integrations with LangChain/LangGraph and LlamaIndex. Users can instrument their applications, define feedbacks, and compare versions in an intuitive UI, making it easier to identify failure modes and iteratively improve performance.

Key Features

OpenTelemetry-based tracing capturing every function call, LLM generation, retrieval, and tool invocation
Pre-built Feedback Functions including RAG Triad and Honest, Harmless, Helpful evals
Support for multiple LLM providers: OpenAI, LiteLLM, Google, AWS Bedrock, Snowflake Cortex, HuggingFace, LangChain
App framework integrations for LangChain/LangGraph and LlamaIndex
Stack-agnostic instrumentation that works with any Python LLM app
User interface for comparing versions of your application
Export traces to Jaeger, Grafana Tempo, Datadog, or any OTLP-compatible backend

Pros & Cons

Pros
  • Open source with a permissive license, free to use
  • Deep, OpenTelemetry-based instrumentation provides granular tracing
  • Comprehensive feedback function library covers common evaluation needs
  • Works with a wide range of LLM providers and frameworks
  • Integrates with existing observability infrastructure via OTLP export

Best For

Evaluating RAG applications for retrieval quality and generation faithfulnessIteratively improving prompts, models, retrievers, and knowledge sourcesIdentifying failure modes in LLM-based applicationsTracking and comparing LLM experiments across different versionsBuilding and testing feedback functions for custom evaluation criteria

FAQ

What is TruLens?
TruLens is an open-source evaluation and tracking tool for LLM experiments and AI agents. It provides fine-grained instrumentation and comprehensive feedback functions to help systematically measure and improve LLM application performance.
How do I install TruLens?
Install TruLens via pip: 'pip install trulens-core'. For feedback evaluations with specific LLM providers, install additional packages such as 'trulens-providers-openai' or 'trulens trulens-providers-litellm'.
What LLM providers does TruLens support?
TruLens supports OpenAI, Azure OpenAI, LiteLLM (Anthropic, Cohere, Mistral), Google Gemini, AWS Bedrock, Snowflake Cortex, HuggingFace, and LangChain models.
Can I use TruLens with LangChain or LlamaIndex?
Yes, TruLens has dedicated app framework integrations for LangChain/LangGraph and LlamaIndex.
Does TruLens support exporting traces to external monitoring tools?
Yes, because TruLens instrumentation is built on OpenTelemetry, traces can be exported to any OTLP-compatible backend such as Jaeger, Grafana Tempo, or Datadog.