Evaluation

Evaluating a gpt-realtime Voice Agent with Simulation

By Omer DemirAug 17, 2026⏱ 9 min read👁 — reads⧉ — copies

How do you write a regression test for a conversation? Our debriefing agent is a speech-to-speech model (Azure Voice Live / gpt-realtime) that talks a field technician through an 8-step post-job report over audio. It’s non-deterministic, stateful, real-time, and it holds a live WebSocket — none of which fits a normal assertion. This post is the in-depth version of how we evaluate it: we simulate the technician.

A persona-driven simulated user drives the real agent over a persistent WebSocket, we capture the transcript, tool calls, and final system state, and binary judges plus an outcome verifier grade the run. Here’s the whole loop.

Everything on the left is the evaluator; the only thing on the right is the real agent. Source: diagrams/voice-agent-simulation-eval.drawio (editable in draw.io).

Why simulation, not scripted asserts

You can’t unit-test a voice agent by feeding it strings and diffing the output. Three properties break that model:

• It’s audio. The agent consumes and produces PCM16 speech, not text. A realistic test has to actually speak to it.
• It’s stateful. Voice-activity detection, turn-taking, and conversational context all live in a single session. Open a fresh socket per turn and you’ve tested a different system.
• It’s non-deterministic. The same prompt yields different wording — and sometimes a different outcome — run to run.

So instead of scripting the agent, we script the other side of the conversation: a simulated technician with a personality, a job to report, and a noisy environment to report it in.

The realtime endpoint contract

The agent exposes a dedicated evaluator WebSocket that mirrors the production realtime endpoint but runs a dry-run session so nothing persists to the backing store during a test:

wss://…/api/evaluator/debriefing/ws?accessToken=<entra-jwt>&language=en

The first frame must be evaluator.init carrying the full session document and an optional agent config override. After that it’s a standard gpt-realtime stream:

{
  "type": "evaluator.init",
  "request": {
    "session": { "id": "…", "unitId": "HMI_TEST_01", "BriefingData": { "…": "…" } },
    "agentConfig": {
      "targetName": "debriefingSessionAgent",
      "enabledTools": ["RecordBillabilityItems", "DebriefingReportCollected", "…"],
      "instructions": "UNIT: {{unitId}} | EVENT CODE: {{eventCode}} | …"
    }
  }
}

Once initialized, the evaluator sends binary PCM16 audio frames for the technician’s speech (or {"type":"message","text":"…"} for a text turn) and receives the realtime event stream:

Injecting the whole session via evaluator.init is what lets us test any work order — including edge cases that never occurred in production — against the exact production agent config.

Generating the technician’s voice

The simulated technician has to actually speak. We put speech generation behind one interface with two implementations, chosen by config, because scripted regression and realistic conversation want opposite things:

public interface ISpeechGenerator : IAsyncDisposable
{
    string ProviderName { get; }
    Task InitializeAsync(CancellationToken ct = default);
    Task<byte[]> GenerateSpeechAsync(SpeechRequest request, CancellationToken ct = default);
}

public sealed class SpeechRequest
{
    public required string Text { get; init; }
    public required string Voice { get; init; }
    public SsmlProsody? Prosody { get; init; }   // ignored by Voice Live, honored by Azure Speech
}

The trick is that SsmlBuilder.EnrichWithContext() derives prosody from the persona and the environment — a senior technician speaks faster and flatter; a noisy machine shop raises volume — so a scenario author never hand-writes SSML. For deterministic regression we pick Azure Speech (the same SSML always produces byte-identical audio); for realism we pick Voice Live. One interface, two strategies.

The simulation inputs: persona × scenario × environment × data

Coverage comes from combining four independent asset types, exactly the pattern from the voice-debriefing post. A persona is a resolved JSON profile — this is a real one the harness produced:

{
  "name": "Alex Morgan",
  "experience_level": "senior",
  "communication_style": "precise, diagnostic, and comfortable with technical terminology",
  "verbosity": "moderate",
  "tech_affinity": "high",
  "voice": "en-US-AvaNeural",
  "traits": [
    "identifies components and fault codes precisely",
    "reports test counts and observed results",
    "does not invent measurements or work not present in the session data"
  ],
  "session_behavior": {
    "conclude_quickly": false,
    "instructions": "Answer as a technically fluent field technician. Use exact observed facts from the session data…"
  }
}

That last trait — does not invent facts — matters: the simulated user is itself an LLM, and without that guardrail it would hallucinate repairs and make every scenario un-gradeable. The persona is grounded in the decoupled data set (unit, equipment, parts, narrative), so the scenario file only carries conversation and evaluation logic. The effective matrix is scenarios × personas × data sets, and the suite spans happy paths, missing information, contradictions, escalation, multi-fault, emotional states, and billable edge cases — 20+ scenarios, each testing something meaningfully different rather than an exhaustive permutation.

The loop and its invariants

The driver runs a fixed loop per conversation, and the invariants are what make a non-deterministic system reproducible enough to gate on:

1. Drive the conversation with a persona plus scenario plus environment.
2. Keep a single persistent WebSocket session for the whole conversation.
3. Inject environment audio on the evaluator side only — never into the agent’s audio.
4. Capture the full transcript, the final outcome, and the exit reason.
5. Persist artifacts for traceability and run comparison.

The non-negotiables: deterministic environment injection with a fixed seed, clean agent audio, and a stateful session that preserves voice-activity detection and context. The anti-patterns that quietly invalidate results:

• Don’t prompt the model to “add background noise.” Noise is mixed by the evaluator, deterministically.
• Don’t mix audio server-side. The agent must hear a clean signal or you’re testing your mixer.
• Don’t open a stateless socket per turn. That destroys VAD and turn-taking — the very things you’re testing.

What a trial produces

Every trial writes an immutable artifact bundle — resolved persona, scenario, environment, transcript, tool calls, and a report. Here’s the shape of a real trial-report.json, lightly trimmed:

{
  "audio_bytes": { "mixed": 16621800, "technician": 12469800, "product": 4152000 },
  "evidence": {
    "persona_name": "Alex Morgan",
    "scenario_id": "wo-001-contract-debrief",
    "duration_seconds": 420.55,
    "tool_calls": [
      { "name": "complete_debrief", "arguments": "{}",
        "result": "{\"completed\":true,\"state\":\"report_collected\"}" }
    ],
    "turns": [
      { "role": "assistant", "content": "Hi there! Let's start the debrief. Can you confirm the job at Westbrook Tower was completed?" },
      { "role": "technician", "content": "Yes — elevator ELEV-4210 stopped between floors due to door sensor fault F-12. I reseated the sensor, reset the controller, ran two cycle tests, both passed. No parts replaced." },
      { "role": "assistant", "content": "Acknowledged. First, what fault did you find on arrival?" }
    ]
  }
}

Everything you need to grade — and to debug a failure — is in that bundle. Note the tool_calls entry: complete_debrief returned state: report_collected. That’s not transcript text; it’s the system-state signal the outcome verifier keys on. The audio_bytes split (technician vs product vs mixed) lets us account for who spoke and confirm the noise was mixed only on the evaluator side.

Scoring: judges, outcomes, exits — never the path

Grading combines four families, all reused across the offline gate and the online monitor so calibration effort is shared:

• Binary judges — focused pass/fail assertions with a reason each: all 8 steps covered, no hallucinated parts or measurements, technician input summarized accurately, off-topic input excluded from the report. (Why binary and not 1–5: see binary judges & pass^k.)
• LLM judges — groundedness and coherence, expressed as binary sub-judges, not a scale.
• Outcome verification — the debrief record has the required fields populated, the billability items match the scenario (down to the code-prefixed reason convention), and a follow-up is created only when expected. We grade what the agent produced, not the tool sequence it took.
• Deterministic scorers — topic coverage, turn/time limits, format compliance.

For scenarios that require product-owned completion, two gates must both pass: the observed exit matches the expected product exit and the persisted outcome satisfies every expectation. A model claiming “all done” is not completion; complete_debrief returning report_collected with a fully populated record is.

Then we run it at least three times. A scenario passes only when every trial passes:

• pass@k — any trial passed (is the behavior reachable at all?).
• pass^k — every trial passed (is it reliable? — the production signal).

Performance metrics (the duration_seconds, turns, tokens, latency in the bundle) are tracked but don’t gate — they surface regressions without adding fragile thresholds.

For standardized agent-behavior metrics on top of our domain judges, the same transcripts feed Azure Foundry’s evaluators over REST — the hybrid setup from earlier in the series.

Determinism where it counts

The whole design threads one needle: a live LLM voice agent is inherently non-deterministic, but an evaluation you gate deploys on must be reproducible. We get both by making the evaluator side deterministic — fixed-seed noise, SSML-scripted technician speech for regression, injected sessions — while letting the agent side be exactly as non-deterministic as production. Then pass^k over multiple trials measures the reliability that remains.

Takeaways

• Simulate the user, exercise the real agent. A persona-driven technician over a persistent WebSocket tests the actual gpt-realtime system, not a mock.
• Inject the session, dry-run the store. evaluator.init lets you test any work order — including edge cases — with zero production side effects.
• Split speech generation behind an interface: Azure Speech SSML for deterministic regression, Voice Live for realism.
• Keep the invariants: single stateful socket, clean agent audio, evaluator-side deterministic noise.
• Grade outcomes and exits, run pass^k. The conversation sounding right isn’t enough — the persisted record has to be right, every trial.

#evaluation#voice#gpt-realtime#simulation#azure