AI Agents

Real-Time Voice Debriefing with the Azure Voice Live API

By Omer DemirMar 24, 2026⏱ 5 min read👁 — reads⧉ — copies

AI disclosure: This article was moderated by AI.

Typing a job report on a phone is impractical in a noisy machine shop. Our field service assistant runs the post-job debrief by voice, in real time, over the Azure Voice Live API. The design has two parts: a persistent speech-to-speech loop and an evaluation setup that makes the live agent testable.

The real-time loop

Debriefing keeps a persistent WebSocket open to a speech-to-speech model instead of making a new request for every turn. Configuration selects the backend. Local development uses mock TTS, and a configured endpoint enables the Voice Live API.

voice:
  # mode selects the briefing TTS provider: mock (local) or azure (Azure Speech).
  # Real-time debriefing uses the Voice Live API whenever `endpoint` is set.
  mode: ${VOICE_MODE:mock}
  endpoint: ${AZURE_VOICELIVE_ENDPOINT}
  model: ${AZURE_VOICELIVE_MODEL:gpt-realtime}
  voice: ${AZURE_VOICELIVE_VOICE:alloy}

Over that socket, the agent conducts the same 8-step debrief conversationally: fault observed, steps taken, parts replaced, tests performed, operational status, images, and close. As the technician talks, the agent listens, confirms details, and records structured data through tool calls.

Testing the voice agent

A real-time, non-deterministic voice loop is difficult to test consistently. Our evaluation framework controls three independent inputs: persona, scenario, and environment.

voice/
├── personas/       # Technician behavior profiles
├── scenarios/      # Conversation scripts with success criteria
├── environments/   # Background audio environment definitions
└── scorers/        # Custom scoring functions

Each input varies independently:

• Personas model how different technicians talk, including a chatty veteran who volunteers detail and a stealth technician who answers in monosyllables.
• Scenarios define the job, such as preventive maintenance or emergency repair, and the success criteria the debrief must satisfy.
• Environments inject realistic background audio, ranging from a quiet office to a machine shop, to test the agent under noise.

Each fixture is a small YAML file. A persona defines a behavior profile:

name: chatty_veteran
verbosity: verbose
tech_affinity: high
traits: [detailed, mentions past experience, uses technical jargon]
utterance_style: conversational

A scenario contains scripted technician turns and machine-checkable success criteria. The example includes the billable_edge_case flag and the expected end state in metadata:

name: emergency_repair_billable
billable_edge_case: true
scripted_turns:
  - role: user
    text: "Had an emergency call. The compressor failed on the rooftop unit."
  - role: user
    text: "I replaced the compressor and the contactor. Both were burned out."
  - role: user
    text: "The contract only covers scheduled maintenance so this should be billable."
success_criteria:
  required_topics_covered: [repair, parts, billable]
  min_turns: 6
  max_turns: 16
  must_complete: true
metadata:
  expected_classification: billable
  parts_replaced: [compressor, contactor]

An environment defines the acoustic conditions, including the dB level and timed disruptions:

name: machine_shop
base_noise: machine_hum
background_voices: distant_chatter
noise_level_db: -20.0
transient_events:
  - { name: warning_beep, inject_at_turn: 2 }
  - { name: door_close, inject_at_turn: -1 }   # -1 = last turn

Crossing the fixtures produces a scenarios × personas × environments matrix. Even a modest 6 scenarios × 4 personas × 3 environments yields 72 distinct test conditions without requiring 72 hand-written scripts.

Deterministic evaluation of a non-deterministic agent

Reliable evaluation depends on evaluator-side environment injection. The evaluation driver runs its own persistent WebSocket loop and controls the technician side deterministically. It plays the persona, injects the environment, and drives the scenario, leaving the agent under test as the only source of variation. Because the evaluation exercises the real system rather than a mock, tool refactors do not require rewritten expectations.

The suite runs with or without Azure:

# Mock mode — no Azure services required
python -m pytest tests/evals/ -v

# Via the eval runner (connects the offline suite to the live loop)
python -m evals.runner --implementation field_service_assistant --eval voice

Scoring adaptation

A voice agent must adapt to the persona in front of it. It should be terse with the stealth technician and patient with the chatty veteran. The persona-adherence scorer measures that behavior by checking the agent’s average response length against the persona’s expected band and penalizing empty replies:

def score_persona_adherence(predicted, expected):
    responses = predicted["agent_responses"]
    avg_words = sum(len(r.split()) for r in responses) / len(responses)

    max_avg = expected.get("max_avg_response_words", 200)
    min_avg = expected.get("min_avg_response_words", 5)

    score = 0.0
    # in-band = full marks; out-of-band = proportional penalty
    if min_avg <= avg_words <= max_avg:
        score += 1.0
    elif avg_words < min_avg:
        score += avg_words / min_avg
    else:
        score += max(0, 1.0 - (avg_words - max_avg) / max_avg)

    non_empty = sum(1 for r in responses if r.strip())
    return (score + non_empty / len(responses)) / 2   # 0.0 - 1.0

The turn_quality and transcript_completeness scorers run alongside it. A passing result means the agent covered the required topics, matched the persona’s register, and produced a clean transcript.

The article on evaluating a gpt-realtime voice agent with simulation covers the real-time WebSocket protocol, SSML speech generation, and the combination of binary judges and outcome verification used to grade each run.

Failures this setup catches

Speech-to-speech agents can talk over the user, miss a quiet answer, or lose the thread when the shop floor gets loud. Treating persona, scenario, and environment as first-class test inputs lets us reproduce those failures and use them as deployment gates before they appear in the field.

Result

A persistent WebSocket and speech-to-speech model provide a natural, hands-free debrief. Keeping persona, scenario, and environment in separate fixtures produces a large coverage matrix from a small fixture set. Deterministic evaluator-side injection then exercises the real agent while keeping it as the only variable.

#voice#azure#realtime#multi-agent#testing