AcruxCore
Open-source LLMOps: version prompts, route LLM calls, trace and evaluate
About
AcruxCore is an open-source (Apache 2.0) LLMOps platform for engineering teams that build features and agents on large language models. Prompts live in one registry with version history, so a team can change and roll back a prompt without redeploying the app. An OpenAI-compatible gateway sits in front of many model providers and adds budgets, rate limits and caching. Every call is recorded as a trace with tokens, cost and latency, and a multi-step agent run becomes one trace. Evaluations score prompt versions against datasets with an LLM judge. SDKs for Python and TypeScript. Free to use, or self-host it with Docker.
Details
AcruxCore is an open-source LLMOps platform that provides a comprehensive solution for engineering teams developing features and agents based on large language models. It allows users to manage prompts in a versioned registry, enabling easy changes and rollbacks without the need to redeploy applications. The platform includes an OpenAI-compatible gateway that interfaces with multiple model providers, offering features such as budget management, rate limiting, and caching. Each request is meticulously traced, recording essential metrics like tokens used, cost, and latency, which helps teams monitor performance and optimize their LLM interactions.
The platform also features evaluation capabilities, allowing teams to score different prompt versions against datasets using an LLM judge. This functionality is crucial for ensuring high-quality outputs and continuous improvement of the prompts used. AcruxCore supports SDKs for both Python and TypeScript, making it accessible for developers across different programming environments. Users can choose to self-host the platform using Docker or utilize the hosted version available during the beta phase, which appears to be free to use.
Key Features
- Versioned prompt management
- OpenAI-compatible gateway
- Request tracing with metrics
- Evaluation scoring for prompt versions
- Audit trail of changes
- Self-hosting capability with Docker
- SDKs for Python and TypeScript
Use Cases
- Managing and deploying LLM prompts without redeploying applications
- Monitoring and optimizing LLM performance through detailed tracing
- Evaluating prompt effectiveness against specific datasets
- Implementing budget controls and rate limits for LLM usage
- Collaborating within teams with a comprehensive audit trail
Pros & Cons
Pros
- Open-source and self-hostable under Apache 2.0
- Facilitates rapid prompt changes without application redeployment
- Comprehensive tracing and evaluation features enhance performance monitoring
- Supports multiple programming environments with SDKs
- Free to use during beta, with self-hosting options available
Cons
- Free tier limits should be verified as the platform is in beta
- Self-hosting may require additional technical expertise
- Performance may vary based on the model providers connected
- Documentation and support may be limited as it is an open-source project
Frequently Asked Questions
Is AcruxCore free to use?
AcruxCore appears to be free during its beta phase, but this should be verified.
Can I self-host AcruxCore?
Yes, AcruxCore can be self-hosted using Docker.
What programming languages does AcruxCore support?
AcruxCore provides SDKs for Python and TypeScript.
How does the tracing feature work?
Tracing records metrics like tokens, cost, and latency for every request, helping teams monitor performance.
What is the evaluation feature?
The evaluation feature allows scoring of prompt versions against datasets using an LLM judge.