AI Agents

Real-Time Voice Debriefing with the Azure Voice Live API

By Omer DemirMar 24, 2026⏱ 5 min read👁 — reads⧉ — copies

Typing a job report on a phone in a noisy machine shop is nobody’s idea of a good time. The field service assistant instead runs the post-job debrief by voice, in real time, over the Azure Voice Live API. This post covers how that loop works and — just as importantly — how we make a real-time voice agent testable.

The real-time loop

Debriefing uses a persistent WebSocket to a speech-to-speech model rather than a request/response call per turn. Config selects the backend: mock TTS for local development, and the Voice Live API whenever an endpoint is set.

voice:
  # mode selects the briefing TTS provider: mock (local) or azure (Azure Speech).
  # Real-time debriefing uses the Voice Live API whenever `endpoint` is set.
  mode: ${VOICE_MODE:mock}
  endpoint: ${AZURE_VOICELIVE_ENDPOINT}
  model: ${AZURE_VOICELIVE_MODEL:gpt-realtime}
  voice: ${AZURE_VOICELIVE_VOICE:alloy}

Over that socket, the agent drives the same 8-step debrief — fault observed, steps taken, parts replaced, tests performed, operational status, images, close — but conversationally. The technician talks; the agent listens, confirms, and records structured data through tool calls as the conversation unfolds.

The hard part: testing a voice agent

A real-time, non-deterministic voice loop is exactly the kind of thing that’s tempting to leave untested. We didn’t. The evaluation framework is persona + scenario + environment driven:

voice/
├── personas/       # Technician behavior profiles
├── scenarios/      # Conversation scripts with success criteria
├── environments/   # Background audio environment definitions
└── scorers/        # Custom scoring functions

Each axis varies independently:

• Personas model how different technicians talk — a chatty veteran who volunteers detail, a stealth technician who answers in monosyllables.
• Scenarios define the job (preventive maintenance, emergency repair) and the success criteria the debrief must satisfy.
• Environments inject realistic background audio — a quiet office versus a machine shop — so we know the agent holds up when it’s noisy.

Each is a small YAML file. A persona is a behavior profile:

name: chatty_veteran
verbosity: verbose
tech_affinity: high
traits: [detailed, mentions past experience, uses technical jargon]
utterance_style: conversational

A scenario carries the scripted technician turns plus machine-checkable success criteria — note the billable_edge_case flag and the expected end state in metadata:

name: emergency_repair_billable
billable_edge_case: true
scripted_turns:
  - role: user
    text: "Had an emergency call. The compressor failed on the rooftop unit."
  - role: user
    text: "I replaced the compressor and the contactor. Both were burned out."
  - role: user
    text: "The contract only covers scheduled maintenance so this should be billable."
success_criteria:
  required_topics_covered: [repair, parts, billable]
  min_turns: 6
  max_turns: 16
  must_complete: true
metadata:
  expected_classification: billable
  parts_replaced: [compressor, contactor]

An environment defines the acoustic conditions, down to the dB level and timed disruptions:

name: machine_shop
base_noise: machine_hum
background_voices: distant_chatter
noise_level_db: -20.0
transient_events:
  - { name: warning_beep, inject_at_turn: 2 }
  - { name: door_close, inject_at_turn: -1 }   # -1 = last turn

Crossing these gives a scenarios × personas × environments matrix from a small number of fixtures. Even a modest 6 scenarios × 4 personas × 3 environments is 72 distinct test conditions — real coverage without hand-writing 72 scripts.

Deterministic evaluation of a non-deterministic agent

The trick to evaluating a live agent reliably is evaluator-side environment injection. The eval driver runs its own persistent WebSocket loop and controls the technician side deterministically — it plays the persona, injects the environment, and drives the scenario — so the only source of variation is the agent under test. The agent is exercised as the real system, not a mock, which means tool refactors don’t require rewriting expectations.

You can run it with or without Azure:

# Mock mode — no Azure services required
python -m pytest tests/evals/ -v

# Via the eval runner (connects the offline suite to the live loop)
python -m evals.runner --implementation field_service_assistant --eval voice

Scoring adaptation, not just words

A voice agent has to adapt to the persona in front of it — terse with the stealth technician, patient with the chatty veteran. That’s directly scorable. The persona-adherence scorer checks the agent’s average response length against the band the persona expects, and penalizes empty replies:

def score_persona_adherence(predicted, expected):
    responses = predicted["agent_responses"]
    avg_words = sum(len(r.split()) for r in responses) / len(responses)

    max_avg = expected.get("max_avg_response_words", 200)
    min_avg = expected.get("min_avg_response_words", 5)

    score = 0.0
    # in-band = full marks; out-of-band = proportional penalty
    if min_avg <= avg_words <= max_avg:
        score += 1.0
    elif avg_words < min_avg:
        score += avg_words / min_avg
    else:
        score += max(0, 1.0 - (avg_words - max_avg) / max_avg)

    non_empty = sum(1 for r in responses if r.strip())
    return (score + non_empty / len(responses)) / 2   # 0.0 - 1.0

Alongside it run turn_quality and transcript_completeness scorers, so a passing run means the agent covered the required topics, matched the persona’s register, and produced a clean transcript.

For the full mechanics — the realtime WebSocket protocol, SSML speech generation, and how binary judges plus outcome verification grade each run — see the deep dive on evaluating a gpt-realtime voice agent with simulation.

Why this matters

Speech-to-speech agents fail in ways text agents don’t: they talk over the user, miss a quiet answer, or lose the thread when the shop floor gets loud. By making persona, scenario, and environment first-class, testable inputs, we can reproduce those failures on demand and gate deploys on them — instead of discovering them in the field.

Takeaways

• Use a persistent WebSocket and a speech-to-speech model for a natural, hands-free debrief.
• Make persona, scenario, and environment independent fixtures to get a large coverage matrix cheaply.
• Evaluate the real agent with deterministic, evaluator-side injection so the agent under test is the only variable.

#voice#azure#realtime#multi-agent#testing