Evaluation

Eval-Driven Development: Red-Green-Refactor for AI Agents

By Omer DemirApr 10, 2026⏱ 4 min read👁 — reads⧉ — copies

The fastest way to make an AI agent worse is to “improve” it speculatively. You tweak a prompt to fix one anecdote, ship it, and quietly break five things nobody was watching. Borrowing from test-driven development — and lessons from teams at Microsoft, Anthropic, and OpenAI — we adopted eval-driven development: every agent change must be justified by a failing evaluation.

The core rule

If it ain’t broke, don’t fix it. Every modification originates from a failing evaluation that reproduced a real problem.

That’s the whole philosophy. No speculative prompt edits, no “this feels better” refactors. If you can’t write an evaluation that fails, you don’t have a problem worth changing the agent for.

The process below runs on a loop: the offline harness gates deploys, the online monitor watches production, and confirmed failures become new scenarios — with one shared judge library keeping both lanes calibrated. Source: diagrams/agent-evaluation-architecture.drawio (editable in draw.io).

The process

It’s Red-Green-Refactor, adapted for agents:

1. Failure detected

A failure surfaces from one of three channels:

• A previous evaluation run (nightly CI or manual).
• Manual testing or red teaming.
• Production observations or user feedback.

2. Open a ticket

Capture three things so the failure is reproducible and reviewable:

• Expected behavior — what the agent should have done.
• Observed behavior — what it actually did.
• Transcript link — the trace or recording.

3. Write a scenario

Turn the failure into a YAML scenario that reproduces it systematically — not a one-off script, but a repeatable case that lives in the suite forever. Say the agent forgot to flag a billable emergency repair; the scenario encodes the conversation and the expected end state:

name: emergency_repair_billable
scripted_turns:
  - role: user
    text: "Emergency call — compressor failed. I replaced the compressor and contactor."
  - role: user
    text: "Contract only covers scheduled maintenance, so this should be billable."
success_criteria:
  required_topics_covered: [repair, parts, billable]
  must_complete: true
metadata:
  expected_classification: billable      # <- the outcome the bug got wrong
  parts_replaced: [compressor, contactor]

4. Validate the failure is detected (Red)

Run the new scenario. Something must fail — an assertion, a judge, or an outcome check. If nothing fails, your suite is blind to the problem, so you add a judge that catches it. This is the red step: you don’t get to fix anything until you can measure the failure.

5. Iterate the agent (Green)

Now, and only now, change the prompt, tools, or configuration — until the failing scenario passes.

6. Validate nothing broke

Run the full suite to confirm no regressions. This is the step speculative changes skip, and it’s the one that saves you.

7. Merge with linked runs

A PR isn’t complete without two linked evaluation runs:

• One showing the issue is detected (step 4).
• One showing it’s resolved with no regressions (step 6).

The reviewer sees the red and the green. The change is self-justifying.

A worked example

The two linked runs make the fix legible at a glance. For the billable-repair bug above, three trials per scenario before and after the prompt change:

The left column proves the scenario reproduces the bug; the right column proves the fix didn’t regress the other 23 scenarios. No dashboard, no anecdote — just red → green with the suite as witness.

The supporting rules

• Add judges reactively. Don’t write judges for problems you haven’t observed. Speculative judges rot and generate false alarms.
• Add failure examples to calibration. Every false positive or false negative from a judge goes back into the calibration dataset, so the judges themselves improve from real misses.

Running it

# Run all scenarios with 3 trials each
dotnet run -- --scenario-dir=scenarios --trials=3

# Run a single scenario while iterating a fix
dotnet run -- --scenario scenarios/new-scenario.yaml --trials=1

# Run judge calibration
dotnet test --filter "Category=Calibration"

Why it works

Agents are non-deterministic, and their failure modes are long-tailed. You cannot brute-force quality by staring at prompts. What you can do is make every regression reproducible, every fix measurable, and every merge guarded by the same suite. Over time the scenario suite becomes an executable memory of every real problem the agent has ever had — which is exactly the asset you want when the next “small prompt tweak” comes along.

Takeaways

• No change without a failing eval. Reproduce first, fix second.
• Two linked runs per PR: one red, one green.
• Grow judges and calibration reactively from real failures, never speculatively.

The next posts in the series cover the machinery this process runs on: a hybrid evaluator design and binary judges with outcome scoring.

#evaluation#ai-agents#testing#llm-judge#process