AcruxCore
Open-source LLMOps: version prompts, route LLM calls, trace and evaluate
AcruxCore is an open-source (Apache 2.0) LLMOps platform for engineering teams that build features and agents on large language models. Prompts live in one registry with version history, so a team can change and roll back a prompt without redeploying the app. An OpenAI-compatible gateway sits in front of many model providers and adds budgets, rate limits and caching. Every call is recorded as a trace with tokens, cost and latency, and a multi-step agent run becomes one trace. Evaluations score prompt versions against datasets with an LLM judge. SDKs for Python and TypeScript. Free to use, or self-host it with Docker.
Openllmetry
Open-source observability for your LLM application, based on OpenTelemetry 
Open-source observability for your LLM application, based on OpenTelemetry !GitHub Repo stars Found in: kyrolabs/awesome-langchain
parea.ai
Auto-create domain-specific evals and test AI systems from experiment to production.
Parea AI is an experimentation and human annotation platform designed for AI teams. It provides tools for experiment tracking, observability, and human annotation, helping teams confidently ship LLM applications to production. Parea AI offers features such as auto-creating domain-specific evals, performance testing and tracking, debugging failures, human review, prompt playground, deployment tools, observability, and dataset management.
Arize Phoenix
Open-source AI observability for LLM apps
Found in: Yigtwxx/Awesome-RAG-Production
Honeycomb AI
Observability platform with AI-powered debugging for LLM apps
Honeycomb extends its observability platform to LLM applications with OpenTelemetry-based tracing, natural language querying, and AI-powered root cause analysis for understanding complex AI system behavior.
LangSmith
LangSmith is an all-in-one platform to develop, debug, test, and monitor LLM applications, ensuring high performance, accuracy, and reliability.
LangSmith is an all-in-one platform to develop, debug, test, and monitor LLM applications, ensuring high performance, accuracy, and reliability.
Weights & Biases Weave
Trace, evaluate, and improve your LLM applications
Weave by Weights & Biases is an LLM observability toolkit that provides tracing, evaluation, and dataset management for AI applications. It integrates with the broader W&B experiment tracking ecosystem.
Helicone
Self-host / Cloud
Self-host / Cloud Found in: Yigtwxx/Awesome-RAG-Production
Langfuse
An open-source LLM engineering platform for tracing, evaluation, prompt management, and metrics. [#opensource](https://github.com/langfuse/langfuse)
An open-source LLM engineering platform for tracing, evaluation, prompt management, and metrics. #opensource Found in: steven2358/awesome-generative-ai