Evaluation

Eval-Driven Development: Red-Green-Refactor for AI Agents

By Omer DemirApr 10, 2026⏱ 4 min read👁 — reads⧉ — copies

AI disclosure: This article was moderated by AI.

Speculative “improvements” are a fast way to make an AI agent worse. A prompt tweak may fix one anecdote while breaking five behaviors that nobody was watching. Drawing on test-driven development and lessons from teams at Microsoft, Anthropic, and OpenAI, we adopted eval-driven development: every agent change must be justified by a failing evaluation.

The core rule

If it ain’t broke, don’t fix it. Every modification starts with a failing evaluation that reproduces a real problem.

That rule excludes speculative prompt edits and “this feels better” refactors. If you cannot write an evaluation that fails, there is not enough evidence to change the agent.

The process runs as a loop. The offline harness gates deploys, the online monitor watches production, and confirmed failures become new scenarios. One shared judge library keeps both lanes calibrated. Source: diagrams/agent-evaluation-architecture.drawio (editable in draw.io).

The process

This is Red-Green-Refactor adapted for agents:

1. Failure detected

A failure can surface through three channels:

• A previous evaluation run (nightly CI or manual).
• Manual testing or red teaming.
• Production observations or user feedback.

2. Open a ticket

Record enough information to reproduce and review the failure:

• The expected behavior: what the agent should have done.
• The observed behavior: what it actually did.
• A transcript link to the trace or recording.

3. Write a scenario

Turn the failure into a YAML scenario that reproduces it systematically. The scenario is a repeatable case that remains in the suite, not a one-off script. If the agent failed to flag a billable emergency repair, for example, encode both the conversation and the expected end state:

name: emergency_repair_billable
scripted_turns:
  - role: user
    text: "Emergency call — compressor failed. I replaced the compressor and contactor."
  - role: user
    text: "Contract only covers scheduled maintenance, so this should be billable."
success_criteria:
  required_topics_covered: [repair, parts, billable]
  must_complete: true
metadata:
  expected_classification: billable      # <- the outcome the bug got wrong
  parts_replaced: [compressor, contactor]

4. Validate that the failure is detected (red)

Run the new scenario. An assertion, judge, or outcome check must fail. If nothing fails, the suite cannot detect the problem, so add a judge that catches it. Do not start the fix until the failure is measurable.

5. Iterate the agent (green)

Once the scenario fails for the right reason, change the prompt, tools, or configuration until it passes.

6. Validate nothing broke

Run the full suite to check for regressions. A targeted scenario proves the fix, while the full suite checks that the change did not break other behavior.

7. Merge with linked runs

A PR requires two linked evaluation runs:

• One showing the issue is detected (step 4).
• One showing it’s resolved with no regressions (step 6).

The reviewer can inspect both the red and green runs, so the evidence for the change stays with the PR.

A worked example

The two linked runs make the fix easy to review. For the billable-repair bug above, run three trials per scenario before and after the prompt change:

The left column proves that the scenario reproduces the bug. The right column shows that the fix did not regress the other 23 scenarios. The red and green runs provide the evidence directly, without relying on an anecdote or an aggregate dashboard.

The supporting rules

• Add judges in response to observed problems. Judges written for hypothetical failures tend to become stale and generate false alarms.
• Add failure examples to calibration. Every false positive or false negative from a judge goes into the calibration dataset, allowing the judges to improve from observed misses.

Running it

# Run all scenarios with 3 trials each
dotnet run -- --scenario-dir=scenarios --trials=3

# Run a single scenario while iterating a fix
dotnet run -- --scenario scenarios/new-scenario.yaml --trials=1

# Run judge calibration
dotnet test --filter "Category=Calibration"

Why it works

Agents are non-deterministic, and their failure modes are long-tailed. Staring at prompts will not brute-force quality. A shared suite can make each regression reproducible, each fix measurable, and each merge subject to the same checks. Over time, the scenario suite becomes an executable record of every observed problem. That record is especially useful when someone proposes the next “small prompt tweak.”

Takeaways

• Require a failing evaluation before changing the agent. Reproduce first, then fix.
• Link two runs to each PR: one red and one green.
• Grow judges and calibration data from observed failures rather than speculation.

The next posts in the series describe the supporting machinery: a hybrid evaluator design and binary judges with outcome scoring.

#evaluation#ai-agents#testing#llm-judge#process