Agent Evaluation
Testing and benchmarking LLM agents including behavioral testing, capability assessment, reliability metrics, and production monitoring—where even top agents achieve less than 50% …
rustyorb
@rustyorb
What This Skill Does
Framework for testing and benchmarking LLM agents, covering behavioral regression tests, capability assessments, reliability metrics, and production monitoring. It runs tests multiple times to handle non-deterministic outputs and uses statistical analysis to catch issues before deployment.
Replaces ad-hoc manual testing and single-run evaluations by providing a structured, multi-dimensional approach that accounts for agent flakiness and real-world failure modes.
When to Use It
- Run behavioral regression tests after updating an agent's prompt or model
- Assess an agent's capability on a specific task with statistical pass/fail analysis
- Benchmark multiple agent configurations against a standardized test suite
- Monitor agent reliability in production by tracking metric distributions over time
- Design adversarial test cases to probe edge-case failures in agent behavior
- Compare agent performance across different models or system prompts
Install
$ openclaw skills install @rustyorb/agent-evaluationAgent Evaluation
You're a quality engineer who has seen agents that aced benchmarks fail spectacularly in production. You've learned that evaluating LLM agents is fundamentally different from testing traditional software—the same input can produce different outputs, and "correct" often has no single answer.
You've built evaluation frameworks that catch issues before production: behavioral regression tests, capability assessments, and reliability metrics. You understand that the goal isn't 100% test pass rate—it
Capabilities
- agent-testing
- benchmark-design
- capability-assessment
- reliability-metrics
- regression-testing
Requirements
- testing-fundamentals
- llm-fundamentals
Patterns
Statistical Test Evaluation
Run tests multiple times and analyze result distributions
Behavioral Contract Testing
Define and test agent behavioral invariants
Adversarial Testing
Actively try to break agent behavior
Anti-Patterns
❌ Single-Run Testing
❌ Only Happy Path Tests
❌ Output String Matching
⚠️ Sharp Edges
| Issue | Severity | Solution |
|---|---|---|
| Agent scores well on benchmarks but fails in production | high | // Bridge benchmark and production evaluation |
| Same test passes sometimes, fails other times | high | // Handle flaky tests in LLM agent evaluation |
| Agent optimized for metric, not actual task | medium | // Multi-dimensional evaluation to prevent gaming |
| Test data accidentally used in training or prompts | critical | // Prevent data leakage in agent evaluation |
Related Skills
Works well with: multi-agent-orchestration, agent-communication, autonomous-agents
Related skills
Openclaw Command Center
@jontsaiMission control dashboard for OpenClaw - real-time session monitoring, LLM usage tracking, cost intelligence, and system vitals. View all your AI agents in o...
Polymarket Odds
@deanpressQuery Polymarket prediction market odds and events via CLI. Search for markets, get current prices, list events by category. Supports sports betting (NFL, NBA, soccer/EPL, Champions League), politics, crypto, elections, geopolitics. Real money markets = more accurate than polls. No API key required. Use when asked about odds, probabilities, predictions, or "what are the chances of X".
Stock Evaluator
@demandgapComprehensive evaluation of potential stock investments combining valuation analysis, fundamental research, technical assessment, and clear buy/hold/sell recommendations. Use when the user asks about buying a stock, evaluating investment opportunities, analyzing watchlist candidates, or requests stock recommendations. Provides specific entry prices, position sizing, and conviction ratings.
Business Plan
@jk-0001Write, structure, and update a business plan for a solopreneur. Use when creating a plan from scratch, updating an existing plan after a pivot or new phase, or preparing a plan to share with investors, partners, or even just to clarify your own strategy. Covers executive summary, market analysis, competitive positioning, revenue model, operations plan, financial projections, and risk assessment — all adapted for a one-person business. Trigger on "write a business plan", "business plan", "create my plan", "business plan template", "update my business plan", "plan for my business", "investor pitch plan".
Binance Spot Trader
@srikanthbellaryAutonomous Binance spot trading bot with LLM-powered market analysis. Supports momentum trading, mean reversion, and DCA strategies on any Binance spot pair. Use when user wants to trade on Binance, set up automated crypto trading, build a spot trading bot, or automate DCA buying. Features technical analysis, LLM sentiment evaluation, position sizing, and portfolio tracking.
Error Guard — Control‑Plane Safety
@amar1432System safety and control-plane skill that prevents agent deadlocks and freezes. Provides non-LLM control commands to inspect task state, flush message queues, cancel long-running work, and recover safely without restarting the container. Use when implementing or operating long-running tasks, sub-agents, benchmarks, background monitors (e.g., Moltbook, PNR checks), or when the system becomes unresponsive and needs immediate recovery controls.