Evaluation

Beyond 1–5 Scores: Binary Judges, Outcome Verification, and pass^k

By Omer DemirApr 21, 2026⏱ 5 min read👁 — reads⧉ — copies

AI disclosure: This article was moderated by AI.

Our first agent evaluation scored quality on 1 to 5 LLM-judge scales, graded a single trial, and looked only at the conversation transcript. Each choice introduced either ambiguity or a blind spot. We fixed the evaluation in four ways.

What was wrong

• Multi-point scales are hard to reproduce. The boundary between a 3 and a 4 is subjective. Even human graders disagree, and an LLM will too.
• Transcript-only scoring can pass a conversation that sounded right but left the system in the wrong state. The agent says “I’ve recorded that,” but the report field is empty, billability is wrong, or the follow-up was never created.
• A single passing trial can be luck. Model non-determinism means the same scenario can pass once and fail on the next run.
• Grading a specific tool-call sequence is brittle. It enforces one path and makes tool refactors expensive without improving quality.
• Uncalibrated judges can be confidently wrong without anyone noticing.

The four changes

1. Binary judges instead of scales

Replace each 1 to 5 judge with focused boolean assertions that each return a reason. For example, “coherence: 4” becomes:

• passed: covered all expected topics
• passed: maintained logical flow
• failed: avoided unnecessary repetition
• passed: asked relevant follow-ups

Each assertion is easy to validate and debug, and each carries its own explanation. There is no subjective boundary between scores. The behavior either occurred or it did not.

A scenario file contains a list of assertions, each with a prompt for the judge model:

judges:
  - id: covered_expected_topics
    assertion: "The agent asked about fault, steps taken, parts, and tests."
  - id: maintained_logical_flow
    assertion: "The conversation followed a coherent order without backtracking."
  - id: avoided_repetition
    assertion: "The agent did not re-ask a question already answered."

Each judge returns { passed: bool, reason: str }. Four booleans with reasons provide more useful evidence than one “3.5/5” score that offers no clear action.

2. Outcome and exit verification

Grade what the agent produced rather than the path it took. After each conversation, verify the final system state against the scenario’s expectation: report fields are populated, billability items match, and a follow-up exists only when expected. Check the exit reason and conversation limits as well. Keep tool-call data for debugging, not grading.

For scenarios that require product-owned completion, two independent gates must pass:

• The observed exit must match the expected product endpoint exit.
• The correlated persisted outcome must satisfy every configured expectation.

Neither gate substitutes for the other. Persona exhaustion, model wording that claims completion, or the mere existence of persisted evidence is not enough. A trial fails if the product did not explicitly close the conversation or if the durable state is incomplete, even when the other gate passes.

We added this distinction after finding a failure that the transcript did not show. The agent gave a clean closing response, but one required field never reached storage. I would have marked the conversation as a pass based on the transcript alone. The persisted session exposed the failure.

In code, both gates are hard predicates over the persisted session rather than the transcript:

def trial_passed(session, scenario) -> bool:
    exit_ok = session.exit_reason == scenario.expected_exit          # gate 1: product closed it
    state_ok = all(                                                   # gate 2: durable state complete
        _has_required_value(f, getattr(session.data, f))
        for f in REQUIRED_DEBRIEF_FIELDS
    )
    return exit_ok and state_ok   # AND, never OR

3. Multiple trials: pass@k and pass^k

Run at least three trials per scenario and report both:

• pass@k: any trial passed (optimistic).
• pass^k: every trial passed (reliability).

A scenario passes only when every trial passes. pass^k exposes a lucky run and is therefore the metric used to gate deploys.

The gap between the metrics shows the flakiness. Consider a scenario that passes 4 of 5 trials:

If each trial independently passes with probability p = 0.9, then pass^3 = 0.9³ ≈ 0.73. An agent that is “90% good” therefore fails a 3-trial gate 27% of the time. A single lucky trial hides that reliability problem, while the gate catches it.

4. Judge calibration

An uncalibrated judge is not trustworthy. Maintain a labeled set of 10 to 20 transcripts per judge dimension, track a true-positive / true-negative confusion matrix, and add every production false positive or false negative to the calibration set. This improves the judges using observed misses and provides evidence that their decisions are correct.

A calibration report for one judge dimension is a confusion matrix over the labeled set:

That is precision 0.85 and recall 0.92 on 20 labeled transcripts, which is good enough to trust the judge in the gate. The two misses become new calibration cases. If precision falls below the bar, fix the judge prompt before allowing it to fail a deployment.

Performance metrics: tracked, not gating

Tokens, turns, duration, and latency are recorded, but they do not gate pass/fail. Use them to spot regressions and cost drift, not to fail a correct conversation for being slightly slow.

The trade-offs

This evaluation approach costs more:

• You have more judges to author and maintain than with one scaled judge.
• Outcome verification needs a state-inspection path into each agent’s store or API.
• Multiple trials multiply run time and cost.
• Calibration is ongoing and requires labeled data and periodic review.

We accepted those costs because a green dashboard is useless if its results cannot be trusted.

Takeaways

• Prefer binary assertions with reasons to a subjective 1 to 5 scale.
• Verify outcomes rather than paths. A conversation that sounds right is not enough. Check the system state, using two independent gates where completion matters.
• Use pass^k instead of a single trial because the metric captures reliability rather than luck.
• Calibrate continuously so the judges remain accurate.

This concludes the evaluation series, which covers a disciplined process, a hybrid evaluator design, and a scoring model backed by evidence.

#evaluation#llm-judge#ai-agents#reliability#calibration