Introduction to Evalite v1 Preview

Evalite v1 is a streamlined evaluation toolkit designed specifically for AI agents. It prioritizes simplicity, speed, and adaptability, allowing developers to assess agent performance without wrestling with complex setups or bloated libraries. Whether you're testing language models, autonomous agents, or custom AI workflows, Evalite handles the heavy lifting so you can focus on insights.

This framework shines in real-world scenarios like benchmarking RAG systems, agentic chains, or multi-step reasoning tasks. By keeping dependencies minimal—relying only on essentials like Pydantic and basic data tools—Evalite runs efficiently even on resource-constrained environments. Check out the full source and contribute via the GitHub repository.

Getting Started: Installation and Basics

For beginners, jumping in is straightforward. Install Evalite using pip:

pip install evalite-ai

That's it—no conda environments, no Docker, no fuss. Once installed, you can start defining evaluations right in your Python scripts.

Your First Evaluation

Evalite revolves around four pillars: Tests, Suites, Metrics, and Datasets. A Test is a single unit of evaluation, like checking if an agent correctly summarizes a document. Suites group Tests for broader runs. Metrics score outputs (e.g., exact match or similarity). Datasets provide inputs and expected results.

Here's a practical beginner example: evaluating a simple Q&A agent.

from evalite import Test, Suite, ExactMatchMetric, Dataset
from evalite.metrics import semantic_similarity

# Define a dataset with questions and answers
dataset = Dataset([
    {"input": "What is Python?", "expected": "A programming language."},
    {"input": "Capital of France?", "expected": "Paris"}
])

# Create a metric
metric = ExactMatchMetric()

# Define a test
def qa_agent(input_text: str) -> str:
    # Simulate your agent here
    return "Python is great!"  # Placeholder

test = Test(
    name="Basic QA",
    fn=qa_agent,
    metric=metric
)

# Bundle into a suite
suite = Suite("QA Suite", tests=[test], dataset=dataset)

# Run it
results = suite.run()
print(results.summary())

This code runs two evaluations, computes exact matches, and outputs a summary table showing pass rates and scores. Expect output like:

Core Components: From Beginner Building Blocks to Advanced Customization

Tests: The Atomic Units

Tests encapsulate a single evaluation scenario. Each Test requires:

• A name for identification.
• An fn (function) that takes input and produces output—your agent's entry point.
• A metric to judge the output.

Advanced tip: Tests support async functions for high-throughput evals.

import asyncio

async def async_agent(input: str) -> str:
    await asyncio.sleep(0.1)  # Simulate API call
    return "Processed: " + input

test = Test(name="Async Test", fn=async_agent, metric=metric)

Suites: Organizing Evaluations

Suites aggregate Tests and pair them with Datasets. Key parameters:

• name: Descriptive label.
• tests: List of Test objects.
• dataset: Optional Dataset for batching.
• max_workers: Parallelism control (default: 4).

Run with suite.run() to get a Results object packed with stats, breakdowns, and JSON exports.

Real-world application: In agent development, create a Suite per capability (e.g., "Tool Use Suite", "Memory Suite").

Metrics: Quantifying Performance

Metrics turn outputs into numbers. Evalite includes built-ins:

• ExactMatchMetric: Binary match or 0/1 score.
• RegexMatchMetric(pattern): Flexible pattern matching.
• SemanticSimilarityMetric(model='all-MiniLM-L6-v2'): Cosine similarity via sentence-transformers.

Custom metrics? Subclass Metric:

from evalite import Metric

class CustomScore(Metric):
    def score(self, prediction: str, expected: str) -> float:
        # Your logic, e.g., keyword overlap
        return len(set(prediction.split()) & set(expected.split())) / len(expected.split())

Combine metrics in lists for multi-dimensional scoring:

metrics = [ExactMatchMetric(), SemanticSimilarityMetric()]
test = Test(..., metrics=metrics)

Pro tip: For RAG evals, pair semantic similarity with retrieval metrics.

Datasets: Feeding Realistic Inputs

Datasets are lists of dicts with input and optional expected, context, or metadata keys. Load from JSON/CSV:

from evalite.datasets import load_json_dataset

dataset = load_json_dataset("path/to/qa.jsonl")

Generate synthetic data? Integrate with libraries like DSPy or use loops for augmentation.

Running Evaluations: CLI and Python API

Python API is core, but CLI speeds up workflows:

# Initialize a project
evalite init my-eval
cd my-eval

# Run a suite
evalite run qa_suite.py

# List suites
evalite list

CLI outputs JSON logs for CI/CD pipelines. Example config in evalite.toml for reproducibility.

Advanced Features: Scaling and Integration

Parallelism and Batching

Leverage max_workers and async for 10x speedups on multi-core machines. For massive datasets, shard Suites.

Observability

Results include traces per run. Export to JSON, CSV, or integrate with Weights & Biases:

results.to_wandb(project="agent-evals")

Custom Workflows

Hook into agent frameworks like LangChain or LlamaIndex by wrapping their invoke() as Test fns.

Example for a LangChain agent:

from langchain.agents import AgentExecutor

test_fn = lambda input: agent_executor.invoke({"input": input})["output"]

Best Practices and Real-World Tips

• Start Small: One Test, one Metric—iterate.
• Version Datasets: Pin to Git for reproducible evals.
• Thresholds: Use results.filter(score > 0.8) for pass/fail gates.
• Edge Cases: Augment datasets with adversarial inputs.

In production, run evals pre-deploy: GitHub Actions script:

- name: Run Evals
  run: evalite run suite.py --json > results.json
  if: success()

Limitations and Roadmap

v1 Preview is lightweight by design—no built-in LLM judging (add via custom Metrics). Future: Native support for judge models, UI dashboard, more integrations.

Evalite empowers rapid iteration. Fork it on GitHub, build your evals, and ship better agents.