Evaluation

Beyond 1–5 Scores: Binary Judges, Outcome Verification, and pass^k

By Omer DemirApr 21, 2026⏱ 5 min read👁 — reads⧉ — copies

The first version of our agent evaluation scored quality on 1–5 LLM-judge scales, graded a single trial, and looked only at the conversation transcript. Every one of those choices turned out to be a source of ambiguity or a blind spot. This post covers the four changes that fixed them.

What was wrong

• Multi-point scales are hard to reproduce. The boundary between a 3 and a 4 is subjective — even human graders disagree, so an LLM certainly will.
• Transcript-only scoring passes conversations that sounded right but left the system wrong. The agent says “I’ve recorded that,” but the report field is empty, the billability is off, or the follow-up was never created.
• A single passing trial can be luck. Model non-determinism means the same scenario can pass once and fail the next run.
• Grading a specific tool-call sequence is brittle. It enforces one path and makes tool refactors expensive for no quality gain.
• Uncalibrated judges can be confidently wrong and nobody notices.

The four changes

1. Binary judges instead of scales

Replace each 1–5 judge with several focused boolean assertions, each returning a reason. A single “coherence: 4” becomes:

• passed: covered all expected topics
• passed: maintained logical flow
• failed: avoided unnecessary repetition
• passed: asked relevant follow-ups

Each assertion is easy to validate, easy to debug, and carries its own explanation. There’s no subjective boundary to argue about — it either happened or it didn’t.

In a scenario file that’s just a list of assertions, each with a prompt for the judge model:

judges:
  - id: covered_expected_topics
    assertion: "The agent asked about fault, steps taken, parts, and tests."
  - id: maintained_logical_flow
    assertion: "The conversation followed a coherent order without backtracking."
  - id: avoided_repetition
    assertion: "The agent did not re-ask a question already answered."

Each judge returns { passed: bool, reason: str }. Four booleans with reasons beat one “3.5/5” you can’t act on.

2. Outcome and exit verification

Grade what the agent produced, not the path it took. After each conversation, verify the final system state against the scenario’s expectation — report fields populated, billability items match, follow-up created only when expected — and check the exit reason and conversation limits. Tool-call data is kept for debugging, not grading.

For scenarios that require product-owned completion, two independent gates must both pass:

• The observed exit must match the expected product endpoint exit.
• The correlated persisted outcome must satisfy every configured expectation.

Neither substitutes for the other. Persona exhaustion, model wording that claims completion, or the mere existence of persisted evidence isn’t enough. A trial fails if the product didn’t explicitly close the conversation, or if the durable state is incomplete — even when the other gate passes.

This distinction came from a failure we could not see in the transcript. The agent gave a clean closing response, but one required field never reached storage. Reading the conversation alone, I would have marked it as a pass. Looking at the persisted session made the failure obvious.

In code, both gates are hard predicates over the persisted session, not the transcript:

def trial_passed(session, scenario) -> bool:
    exit_ok = session.exit_reason == scenario.expected_exit          # gate 1: product closed it
    state_ok = all(                                                   # gate 2: durable state complete
        _has_required_value(f, getattr(session.data, f))
        for f in REQUIRED_DEBRIEF_FIELDS
    )
    return exit_ok and state_ok   # AND, never OR

3. Multiple trials: pass@k and pass^k

Run at least three trials per scenario and report both:

• pass@k — any trial passed (optimistic).
• pass^k — every trial passed (reliability).

A scenario passes only when every trial passes. pass^k is the number that exposes a lucky run for what it is, and it’s the one that gates deploys.

The gap between the two metrics is where flakiness hides. Take a scenario that passes 4 of 5 trials:

If each trial independently passes with probability p = 0.9, then pass^3 = 0.9³ ≈ 0.73 — so a “90% good” agent fails a 3-trial gate 27% of the time. That’s the reliability a single lucky trial hides, and exactly what you want the gate to catch.

4. Judge calibration

A judge you don’t calibrate is a judge you can’t trust. Maintain a labeled set of 10–20 transcripts per judge dimension, track a true-positive / true-negative confusion matrix, and feed every production false positive or false negative back into the calibration set. The judges improve from real misses, and you have evidence they’re actually right.

A calibration report for one judge dimension is just a confusion matrix over the labeled set:

That’s precision 0.85, recall 0.92 on 20 labeled transcripts — good enough to trust the judge in the gate, with the two misses added back as new calibration cases. When precision drifts below the bar, the judge prompt gets fixed before it’s allowed to fail anyone’s deploy.

Performance metrics: tracked, not gating

Tokens, turns, duration, and latency are all recorded — but they don’t gate pass/fail. They’re there to spot regressions and cost drift, not to fail a correct conversation for being a little slow.

The trade-offs

Trustworthy evaluation costs more:

• More judges to author and maintain than one scaled judge.
• Outcome verification needs a state-inspection path into each agent’s store or API.
• Multiple trials multiply run time and cost.
• Calibration is ongoing — it needs labeled data and periodic review.

We took every one of those costs, because the alternative is a green dashboard that doesn’t mean anything.

Takeaways

• Binary, not 1–5. Focused assertions with reasons beat a subjective scale.
• Verify outcomes, not paths. The conversation sounding right isn’t enough — check the system state, with two independent gates where completion matters.
• pass^k over a single trial. Reliability is the metric, not a lucky run.
• Calibrate continuously so your judges stay honest.

This closes the evaluation series: a disciplined process, a hybrid evaluator design, and a scoring model you can actually believe.

#evaluation#llm-judge#ai-agents#reliability#calibration