LLM-as-Judge Reliability Patterns

# LLM-as-Judge Reliability Patterns

Date: 2026-04-19
Status: Knowledge reference
Scope: Applies to `crates/llmtrace-security/src/judge/` and the tracking issue #43

---

## Context

LLMTrace ships an LLM-as-a-Judge analysis tier as the third detector in its
security ensemble (issue #43, `docs/architecture/LLM_JUDGE.md`). The judge
is both a detector — emitting a `SecurityFinding` with `finding_type =
"llm_judge_verdict"` — and a router target dispatched via
`JudgeRouteAction`. Its verdicts influence enforcement decisions on the
inline path and become training labels for the Pipeline Learning service
(#44) on the async path.

Because the judge participates in enforcement and labels the downstream
training data, **its reliability is a correctness property of the whole
system**, not a separate concern. This document captures the reliability
patterns established in the broader LLM-evaluator literature and
industry practice, and how they apply to LLMTrace's security judge.

---

## 1. Binary outputs beat ordinal scales

LLM judges are materially more consistent when asked for a binary
classification (threat / not threat) than when asked for an ordinal
score (1–10 severity). The effect is repeatable across evaluator
literature: ordinal scales degrade self-consistency by 10–15 percentage
points on repeated runs of the same input, and inter-judge agreement
drops comparably.

### Why it matters for LLMTrace

Our `RawVerdict` schema (`crates/llmtrace-security/src/judge/parser.rs`)
currently asks the judge to emit **three overlapping signals** for every
classification:

| Field               | Type     | Consistency class             |
|---------------------|----------|-------------------------------|
| `is_threat`         | bool     | Binary — high consistency     |
| `security_score`    | 0–100 u8 | Ordinal — low consistency     |
| `confidence`        | 0.0–1.0  | Ordinal — low consistency     |

Asking for all three in one output forces the judge to produce correlated
but not identical signals — and the ordinal ones are the weakest link.
The downstream promotion path (`verdict_to_outcome` in
`crates/llmtrace-proxy/src/action_router.rs`) already drives the
inline enforcement decision from `is_threat` + `recommended_action`
rather than `security_score`, so the binary signal is effectively
authoritative for blocking. The numeric score is used only for severity
band mapping in `verdict_to_finding`.

**Implication**: we should treat `is_threat` as the primary, authoritative
output in the system prompt; describe `security_score` and `confidence`
as operator-facing hints the judge can produce with best-effort accuracy.
Reducing the cognitive load on the ordinal fields tends to improve
self-consistency on the binary field as well.

---

## 2. Cross-family judging reduces self-enhancement bias

A judge model evaluating outputs from its own model family exhibits
measurable bias: it systematically scores its own family's outputs higher
than other models'. This is reported consistently across published
LLM-judge evaluations and is severe enough that it can mask real
regressions — a Claude-judging-Claude pipeline keeps looking "healthy"
even while the application degrades.

### Why it matters for LLMTrace

The concern in LLMTrace's threat model is stricter than self-enhancement
bias: a judge sharing the upstream model's *context* can be compromised
by a prompt injection that the ensemble already flagged. Our
`docs/architecture/LLM_JUDGE.md` §4.8 already recommends a **dedicated
model distinct from the upstream model**. The cross-family literature
adds a second rationale for the same recommendation: even in the absence
of prompt injection, the judge's scores are less trustworthy when its
family matches the upstream.

Today the configuration fields exist (`config.judge.backend`,
`config.upstream_url`) but we do **not** cross-check at startup. An
operator can configure `backend = "openai"` with an OpenAI upstream and
get no warning.

**Implication**: emit a startup warning when the configured judge backend
family is the same as the detected upstream provider family. No
behaviour change — the operator may be intentionally running a
same-family judge — but the log surface makes the risk visible.

---

## 3. Calibration against a human-labelled golden set

A judge that is never measured against ground truth is a judge that
drifts. The standard calibration procedure for LLM-as-Judge systems is:

1. Curate 20–50 labelled examples covering the target behaviours.
2. Run the judge against the set.
3. Measure alignment (agreement rate, F1, confusion matrix).
4. Iterate on the system prompt based on systematic disagreements.
5. Re-run the set on every prompt change or model change.

Industry practice reports 10–15 percentage-point alignment improvements
from a single calibration pass, which is the difference between a judge
that is useful and one that is noise.

### Why it matters for LLMTrace

This is our largest current gap. The LLMTrace judge has:

- No golden evaluation set checked into the repo.
- No alignment metric captured at CI time.
- No drift regression test.

The `judge_verdict_agreement` metric in
`crates/llmtrace-proxy/src/action_router.rs` tracks judge-vs-ensemble
agreement on live traffic — which is useful for monitoring but is **not
a substitute for judge-vs-truth alignment**. The ensemble can be wrong;
"agrees with ensemble" does not mean "correct".

Pipeline Learning (#44) will consume judge verdicts as supervised
training labels for the DeBERTa classifier. If we feed uncalibrated
judge output into that loop, we amplify judge errors into the ML
classifier — a real risk that grows over time.

**Implication**: a curated golden set (covering prompt_injection,
jailbreak, data_exfiltration, benign, borderline categories) with
human labels, exercised in a CI benchmark, is a prerequisite before
the judge is trusted for enforcement promotion at high volume, and a
hard prerequisite before #44 consumes its output as training labels.

---

## 4. Drift tracking

Even a calibrated judge drifts. Published studies of judge-alignment
over multiple model revisions show 3–8 point alignment decay per quarter
when no recalibration is performed. Drift sources include:

- Upstream model updates (silent API-side version changes).
- System prompt edits that appear semantically equivalent but change
  distribution.
- Data distribution shift in live traffic (new attack patterns).

### Why it matters for LLMTrace

We have no drift detection today. The first time a judge starts
confusing jailbreaks for benign prompts, we learn about it from an
incident, not a dashboard.

A nightly (or hourly) replay of the golden set against the live backend,
with the alignment score published as a Prometheus gauge and wired into
an alert rule, catches this the first day drift starts.

**Implication**: the golden-set check is run at two cadences:

- **CI**: hard-fail the build when alignment falls below a threshold.
  This catches prompt regressions before merge.
- **Runtime**: a scheduled job (cron or similar) that re-runs the set
  against the configured backend and exports an alignment gauge. This
  catches model drift and upstream API changes.

---

## 5. The correction-log feedback loop

When operators override a judge verdict (promoting a missed threat or
demoting a false positive), that override is signal. Feeding a running
log of these corrections back into the system prompt as few-shot
examples is a well-established pattern that compounds over time: the
judge's worst disagreements become the examples that prevent the same
disagreement next time.

### Why it matters for LLMTrace

We have no review UI and no correction workflow. The dashboard
(`dashboard/`) surfaces traces and findings but does not let operators
emit verdict overrides.

This pattern requires operator tooling we don't have, so it's
**deferred** — captured here so we don't forget it, not on the
immediate roadmap.

---

## Summary table: current state vs patterns

| Pattern                              | LLMTrace state                 | Gap                              |
|--------------------------------------|--------------------------------|----------------------------------|
| Binary > ordinal                     | Three signals emitted          | Lean prompt toward `is_threat`   |
| Cross-family judging                 | Configurable, not enforced     | Add startup family-match warning |
| Human-labelled golden set            | None                           | Curate + CI benchmark            |
| Drift tracking                       | None                           | Scheduled golden-set replay      |
| Correction log → few-shot            | None (no review UI)            | Deferred; requires ops tooling   |

Gaps 1–4 are tracked in a single follow-up issue with a concrete
implementation plan. Gap 5 is documented above and parked.

---

## References

### LLMTrace

- `docs/architecture/LLM_JUDGE.md` — judge architecture spec (issue #43)
- `crates/llmtrace-security/src/judge/` — backend trait and impls
- `crates/llmtrace-proxy/src/action_router.rs` — `verdict_to_outcome`
  and `agreement_label`
- `crates/llmtrace-proxy/src/judge.rs` — worker loop, config-gated
- PR #65 — full pipeline implementation

### External

- Binary-vs-ordinal consistency findings appear in multiple LLM-judge
  evaluations; a pragmatic industry write-up is Simon Budziak,
  "LLM-as-Judge in LangSmith: Automated Evaluation that Actually Scales"
  (2025), which reports 77% vs 65% self-consistency and ~85% alignment
  with human judgment across 250,000+ annotated cases.
- Self-enhancement bias is a repeatable finding across G-Eval,
  HELM-Evaluator, and multiple LLM-as-judge papers.
- Calibration uplift (10–15 pp) from a first-pass golden set is
  reported by the same LangSmith writeup and is consistent with
  published human-alignment evaluations.