AutoArena

AutoArena is an open-source tool designed to automate head-to-head evaluations of GenAI systems using LLM judges. It allows users to quickly and accurately generate leaderboards comparing different LLMs, RAG setups, or prompt variations. Users can fine-tune custom judges to fit their specific needs. AutoArena facilitates trustworthy evaluation of LLMs, RAG systems, and generative AI applications through automated head-to-head judgement.

How to Use

Install AutoArena locally using pip install autoarena. Define your inputs (user prompts) and outputs (model responses) from your Generative AI system. Then, use the tool to run head-to-head evaluations with LLM judges to rank your systems. Collaborate with team members on AutoArena Cloud at autoarena.app.

Key Features

• Automated head-to-head evaluation using LLM judges
• Leaderboard generation for comparing LLMs, RAG setups, and prompt variations
• Fine-tuning of custom judges
• Elo score and Confidence Interval computation
• Integration with GitHub for CI/CD
• Parallelization, randomization, and rate limiting handling

Use Cases

• Evaluate different LLMs to determine the best performing model for a specific task.
• Compare various RAG setups to optimize retrieval and generation performance.
• Test different prompt variations to identify the most effective prompts.
• Block bad prompt changes, preprocessing or postprocessing updates, or RAG system updates in CI.
• Track performance improvements of new system versions against previous versions.