Evaluating Agent Skill Output Quality with Eval-Driven Iteration

Grading outputs

Grading means evaluating each assertion against the actual outputs and recording PASS or FAIL with specific evidence. The evidence should quote or reference the output, not just state an opinion.

The simplest approach is to give the outputs and assertions to an LLM and ask it to evaluate each one. For assertions that can be checked by code (valid JSON, correct row count, file exists with expected dimensions), use a verification script. Scripts are more reliable than LLM judgment for mechanical checks and reusable across iterations.

{
  "assertion_results": [
    {
      "text": "The output includes a bar chart image file",
      "passed": true,
      "evidence": "Found chart.png (45KB) in outputs directory"
    },
    {
      "text": "The chart shows exactly 3 months",
      "passed": true,
      "evidence": "Chart displays bars for March, July, and November"
    },
    {
      "text": "Both axes are labeled",
      "passed": false,
      "evidence": "Y-axis is labeled 'Revenue ($)' but X-axis has no label"
    },
    {
      "text": "The chart title or caption mentions revenue",
      "passed": true,
      "evidence": "Chart title reads 'Top 3 Months by Revenue'"
    }
  ],
  "summary": {
    "passed": 3,
    "failed": 1,
    "total": 4,
    "pass_rate": 0.75
  }
}

Grading principles

• Require concrete evidence for a PASS. Don't give the benefit of the doubt. If an assertion says "includes a summary" and the output has a section titled "Summary" with one vague sentence, that is a FAIL. The label is there but the substance is not.

• Review the assertions themselves, not just the results. While grading, notice when assertions are too easy (always pass regardless of skill quality), too hard (always fail even when the output is good), or unverifiable (cannot be checked from the output alone). Fix these for the next iteration.

  For comparing two skill versions, try blind comparison: present both outputs to an LLM judge without revealing which came from which version. The judge scores holistic qualities, like organization, formatting, usability, and polish, on its own rubric, free from bias about which version "should" be better. This complements assertion grading: two outputs might both pass all assertions but differ significantly in overall quality.

Aggregating results

Once every run in the iteration is graded, compute summary statistics per configuration and save them to benchmark.json alongside the eval directories (e.g., csv-analyzer-workspace/iteration-1/benchmark.json):

{
  "run_summary": {
    "with_skill": {
      "pass_rate": { "mean": 0.83, "stddev": 0.06 },
      "time_seconds": { "mean": 45.0, "stddev": 12.0 },
      "tokens": { "mean": 3800, "stddev": 400 }
    },
    "without_skill": {
      "pass_rate": { "mean": 0.33, "stddev": 0.10 },
      "time_seconds": { "mean": 32.0, "stddev": 8.0 },
      "tokens": { "mean": 2100, "stddev": 300 }
    },
    "delta": {
      "pass_rate": 0.50,
      "time_seconds": 13.0,
      "tokens": 1700
    }
  }
}

The delta tells you what the skill costs (more time, more tokens) and what it buys (higher pass rate). A skill that adds 13 seconds but improves pass rate by 50 percentage points is probably worth it. A skill that doubles token usage for a 2-point improvement might not be.

Standard deviation (stddev) is only meaningful with multiple runs per eval. In early iterations with just 2-3 test cases and single runs, focus on the raw pass counts and the delta. The statistical measures become useful as you expand the test set and run each eval multiple times.

Analyzing patterns

Aggregate statistics can hide important patterns. After computing the benchmarks:

• Remove or replace assertions that always pass in both configurations. These do not tell you anything useful. The model handles them fine without the skill. They inflate the with-skill pass rate without reflecting actual skill value.
• Investigate assertions that always fail in both configurations. Either the assertion is broken (asking for something the model cannot do), the test case is too hard, or the assertion is checking for the wrong thing. Fix these before the next iteration.
• Study assertions that pass with the skill but fail without. This is where the skill is clearly adding value. Understand why, which instructions or scripts made the difference.
• Tighten instructions when results are inconsistent across runs. If the same eval passes sometimes and fails others (reflected as high stddev in the benchmark), the eval may be flaky (sensitive to model randomness), or the skill's instructions may be ambiguous enough that the model interprets them differently each time. Add examples or more specific guidance to reduce ambiguity.
• Check time and token outliers. If one eval takes 3x longer than the others, read its execution transcript (the full log of what the model did during the run) to find the bottleneck.

Reviewing results with a human

Assertion grading and pattern analysis catch a lot, but they only check what you thought to write assertions for. A human reviewer brings a fresh perspective, catching issues you did not anticipate, noticing when the output is technically correct but misses the point, or spotting problems that are hard to express as pass/fail checks. For each test case, review the actual outputs alongside the grades.

Record specific feedback for each test case and save it in the workspace (e.g., as a feedback.json alongside the eval directories):

{
  "eval-top-months-chart": "The chart is missing axis labels and the months are in alphabetical order instead of chronological.",
  "eval-clean-missing-emails": ""
}

"The chart is missing axis labels" is actionable; "looks bad" is not. Empty feedback means the output looked fine, that test case passed your review. During the iteration step, focus your improvements on the test cases where you had specific complaints.

Iterating on the skill

After grading and reviewing, you have three sources of signal:

• Failed assertions point to specific gaps, a missing step, an unclear instruction, or a case the skill does not handle.
• Human feedback points to broader quality issues, the approach was wrong, the output was poorly structured, or the skill produced a technically correct but unhelpful result.
• Execution transcripts reveal why things went wrong. If the agent ignored an instruction, the instruction may be ambiguous. If the agent spent time on unproductive steps, those instructions may need to be simplified or removed.

The most effective way to turn these signals into skill improvements is to give all three, along with the current SKILL.md, to an LLM and ask it to propose changes. The LLM can synthesize patterns across failed assertions, reviewer complaints, and transcript behavior that would be tedious to connect manually. When prompting the LLM, include these guidelines:

• Generalize from feedback. The skill will be used across many different prompts, not just the test cases. Fixes should address underlying issues broadly rather than adding narrow patches for specific examples.
• Keep the skill lean. Fewer, better instructions often outperform exhaustive rules. If transcripts show wasted work (unnecessary validation, unneeded intermediate outputs), remove those instructions. If pass rates plateau despite adding more rules, the skill may be over-constrained. Try removing instructions and see if results hold or improve.
• Explain the why. Reasoning-based instructions ("Do X because Y tends to cause Z") work better than rigid directives ("ALWAYS do X, NEVER do Y"). Models follow instructions more reliably when they understand the purpose.
• Bundle repeated work. If every test run independently wrote a similar helper script (a chart builder, a data parser), that is a signal to bundle the script into the skill's scripts/ directory. See Using scripts for how to do this.

The loop

1. Give the eval signals and current SKILL.md to an LLM and ask it to propose improvements.
2. Review and apply the changes.
3. Rerun all test cases in a new iteration-<N+1>/ directory.
4. Grade and aggregate the new results.
5. Review with a human. Repeat.

Stop when you are satisfied with the results, feedback is consistently empty, or you are no longer seeing meaningful improvement between iterations.

The skill-creator Skill automates much of this workflow, running evals, grading assertions, aggregating benchmarks, and presenting results for human review.

When not to use it

If you only need to verify a skill works once, or if the skill is trivial (a single instruction with no branching), a full eval loop is overkill. The process is designed for skills where reliability across varied inputs matters and where the cost of a failure is high enough to justify the setup time.

Limits and gotchas

• The evals/evals.json file is the only file you author by hand. The other JSON files are produced during the eval process, not by you.
• Standard deviation is only meaningful with multiple runs per eval. In early iterations with just 2-3 test cases and single runs, focus on raw pass counts and the delta.
• Assertions that always pass or always fail in both configurations are noise. Remove or fix them.
• Human review catches things assertions miss, but the feedback must be specific to be actionable.

What pairs with this

• Using scripts for bundling repeated work into the skill's scripts/ directory.
• The skill-creator Skill automates much of this workflow.