Evaluation

Evaluating a gpt-realtime Voice Agent with Simulation

By Omer DemirAug 17, 2026⏱ 9 min read👁 — reads⧉ — copies

AI disclosure: This article was moderated by AI.

Our debriefing agent is a speech-to-speech model (Azure Voice Live / gpt-realtime) that guides a field technician through an 8-step post-job report over audio. It is non-deterministic and stateful, runs in real time, and keeps a WebSocket open for the session. Normal input-output assertions do not cover that behavior, so our regression tests simulate the technician instead.

A persona-driven user simulator talks to the real agent over a persistent WebSocket. During the run, the evaluator captures the transcript, tool calls, and final system state. Binary judges and an outcome verifier then grade the result.

Everything on the left is the evaluator; the only thing on the right is the real agent. Source: diagrams/voice-agent-simulation-eval.drawio (editable in draw.io).

Why simulation, not scripted asserts

Feeding strings to a voice agent and diffing the output misses three properties of the system:

• The agent consumes and produces PCM16 speech, not text. A realistic test must speak to it.
• Voice-activity detection, turn-taking, and conversational context belong to one stateful session. Opening a fresh socket for each turn tests a different system.
• The same prompt can produce different wording and, in some runs, a different outcome.

Rather than script the agent, we script the other side of the conversation. The simulated technician has a personality, a job to report, and a noisy environment in which to report it.

The realtime endpoint contract

The agent exposes a dedicated evaluator WebSocket. It mirrors the production realtime endpoint but runs a dry-run session, so a test writes nothing to the backing store:

wss://…/api/evaluator/debriefing/ws?accessToken=<entra-jwt>&language=en

The first frame must be evaluator.init. It carries the full session document and an optional agent configuration override. The connection then continues as a standard gpt-realtime stream:

{
  "type": "evaluator.init",
  "request": {
    "session": { "id": "…", "unitId": "HMI_TEST_01", "BriefingData": { "…": "…" } },
    "agentConfig": {
      "targetName": "debriefingSessionAgent",
      "enabledTools": ["RecordBillabilityItems", "DebriefingReportCollected", "…"],
      "instructions": "UNIT: {{unitId}} | EVENT CODE: {{eventCode}} | …"
    }
  }
}

After initialization, the evaluator sends binary PCM16 audio frames for the technician’s speech (or {"type":"message","text":"…"} for a text turn) and receives the realtime event stream:

Injecting the complete session through evaluator.init lets us test any work order, including edge cases that never occurred in production, against the exact production agent configuration.

Generating the technician’s voice

The simulated technician must produce speech. We put speech generation behind one interface and select one of two implementations through configuration. Scripted regression needs repeatability; realistic conversation needs adaptive, low-latency speech.

public interface ISpeechGenerator : IAsyncDisposable
{
    string ProviderName { get; }
    Task InitializeAsync(CancellationToken ct = default);
    Task<byte[]> GenerateSpeechAsync(SpeechRequest request, CancellationToken ct = default);
}

public sealed class SpeechRequest
{
    public required string Text { get; init; }
    public required string Voice { get; init; }
    public SsmlProsody? Prosody { get; init; }   // ignored by Voice Live, honored by Azure Speech
}

SsmlBuilder.EnrichWithContext() derives prosody from the persona and environment. A senior technician speaks faster and flatter, while a noisy machine shop raises the volume. Scenario authors therefore do not have to write SSML by hand. We use Azure Speech for deterministic regression because the same SSML always produces byte-identical audio. We use Voice Live when realism matters. Both strategies implement the same interface.

The simulation inputs: persona × scenario × environment × data

Coverage comes from four independent asset types, following the pattern in the voice-debriefing post. A persona is a resolved JSON profile. The following example came from the harness:

{
  "name": "Alex Morgan",
  "experience_level": "senior",
  "communication_style": "precise, diagnostic, and comfortable with technical terminology",
  "verbosity": "moderate",
  "tech_affinity": "high",
  "voice": "en-US-AvaNeural",
  "traits": [
    "identifies components and fault codes precisely",
    "reports test counts and observed results",
    "does not invent measurements or work not present in the session data"
  ],
  "session_behavior": {
    "conclude_quickly": false,
    "instructions": "Answer as a technically fluent field technician. Use exact observed facts from the session data…"
  }
}

The final trait, does not invent facts, is important because the simulated user is also an LLM. Without that guardrail, it can hallucinate repairs and make a scenario impossible to grade. The persona is grounded in a decoupled data set containing the unit, equipment, parts, and narrative, so the scenario file contains only conversation and evaluation logic. This produces a matrix of scenarios × personas × data sets. The suite has more than 20 scenarios covering happy paths, missing information, contradictions, escalation, multiple faults, emotional states, and billable edge cases. Each scenario tests a meaningful difference rather than filling every possible permutation.

The loop and its invariants

The driver follows a fixed loop for each conversation. A small set of invariants makes the non-deterministic system reproducible enough to use as a gate:

1. Drive the conversation with a persona, scenario, and environment.
2. Keep a single persistent WebSocket session for the whole conversation.
3. Inject environment audio on the evaluator side only, never into the agent’s audio.
4. Capture the full transcript, the final outcome, and the exit reason.
5. Persist artifacts for traceability and run comparison.

Environment injection must use a fixed seed, agent audio must stay clean, and the session must remain stateful to preserve voice-activity detection and context. Three implementation mistakes invalidate the results:

• Do not prompt the model to “add background noise.” The evaluator mixes noise deterministically.
• Do not mix audio on the server. The agent must hear a clean signal, or the test measures the mixer.
• Do not open a stateless socket for each turn. Doing so destroys the VAD and turn-taking behavior under test.

What a trial produces

Every trial writes an immutable artifact bundle containing the resolved persona, scenario, environment, transcript, tool calls, and report. This lightly trimmed trial-report.json came from a real run:

{
  "audio_bytes": { "mixed": 16621800, "technician": 12469800, "product": 4152000 },
  "evidence": {
    "persona_name": "Alex Morgan",
    "scenario_id": "wo-001-contract-debrief",
    "duration_seconds": 420.55,
    "tool_calls": [
      { "name": "complete_debrief", "arguments": "{}",
        "result": "{\"completed\":true,\"state\":\"report_collected\"}" }
    ],
    "turns": [
      { "role": "assistant", "content": "Hi there! Let's start the debrief. Can you confirm the job at Westbrook Tower was completed?" },
      { "role": "technician", "content": "Yes — elevator ELEV-4210 stopped between floors due to door sensor fault F-12. I reseated the sensor, reset the controller, ran two cycle tests, both passed. No parts replaced." },
      { "role": "assistant", "content": "Acknowledged. First, what fault did you find on arrival?" }
    ]
  }
}

The bundle contains the evidence needed both to grade a run and to debug a failure. In tool_calls, complete_debrief returned state: report_collected. This is not transcript text. It is the system state used by the outcome verifier. The audio_bytes split among technician, product, and mixed audio accounts for who spoke and confirms that noise was mixed only on the evaluator side.

Scoring judges, outcomes, and exits

Grading uses four families of checks. We reuse all four in the offline gate and online monitor, which also lets them share the same calibration work:

• Binary judges make focused pass/fail assertions and provide a reason for each result. They check that all 8 steps were covered, no parts or measurements were hallucinated, technician input was summarized accurately, and off-topic input was excluded from the report. The case for binary scoring rather than a 1-to-5 scale is covered in binary judges & pass^k.
• LLM judges assess groundedness and coherence through binary sub-judges rather than a scale.
• Outcome verification checks that the debrief record has all required fields, billability items match the scenario down to the code-prefixed reason convention, and a follow-up is created only when expected. We grade the result the agent produced, not its sequence of tool calls.
• Deterministic scorers check topic coverage, turn and time limits, and format compliance.

For scenarios that require product-owned completion, two gates must pass. The observed exit must match the expected product exit, and the persisted outcome must satisfy every expectation. A model claiming “all done” does not count as completion. Completion requires complete_debrief to return report_collected with a fully populated record.

We run each scenario at least three times. It passes only if every trial passes:

• pass@k: any trial passed, showing that the behavior is reachable at all.
• pass^k: every trial passed, which is the reliability signal used for production.

Performance metrics (the duration_seconds, turns, tokens, latency in the bundle) are tracked but do not gate the run. They surface regressions without introducing fragile thresholds.

For standardized agent-behavior metrics on top of our domain judges, the same transcripts feed Azure Foundry’s evaluators over REST, using the hybrid setup described earlier in the series.

Determinism where it counts

The live LLM voice agent is inherently non-deterministic, but an evaluation used to gate deployments must be reproducible. We make the evaluator deterministic with fixed-seed noise, SSML-scripted technician speech for regression, and injected sessions. The agent remains as non-deterministic as it is in production. Running multiple trials and measuring pass^k tells us how reliable the remaining behavior is.

What the gate depends on

• A persona-driven technician using a persistent WebSocket tests the real gpt-realtime system, rather than a mock.
• evaluator.init can inject any work order, including edge cases, while the dry-run store prevents production side effects.
• The speech-generation interface supports Azure Speech SSML for deterministic regression and Voice Live for realism.
• Each run keeps one stateful socket, clean agent audio, and deterministic noise on the evaluator side.
• The gate checks outcomes and exits with pass^k. A plausible conversation is insufficient unless every trial also produces the correct persisted record.

#evaluation#voice#gpt-realtime#simulation#azure