Evaluation

Conversation Evaluation Through Voice-Agent Simulation

By Omer DemirAug 29, 2026⏱ 21 min read👁 — reads⧉ — copies

AI disclosure: This article was moderated by AI.

Conversation evaluation based on simulation covers a surface that text-focused frameworks such as DeepEval, Vally, and Waza do not: end-to-end voice behavior from a deployed agent. A simulator sends audio through an HTTP, WebSocket, or custom endpoint adapter, drives a multi-turn conversation, retains transcripts and audio, checks observable outcomes, and calculates transport and PCM signal metrics from the exchange.

The scope is intentionally narrow. This approach is not a general LLM evaluator, agent graph debugger, retrieval benchmark, or perceptual speech-quality laboratory. It can detect late output audio, unexpected silence, clipping, and abrupt endings. It cannot determine whether a voice sounds natural, pronounces a name correctly, preserves speaker identity, or reaches a particular mean opinion score without additional evaluators.

What the framework evaluates

The framework’s execution path is:

The unit under test is not a prompt or isolated model response. It includes:

• Speech input generation and environmental audio.
• Network transport and the agent’s public voice protocol.
• Turn boundaries, streaming behavior, and completion signals.
• Speech recognition, reasoning, tool use, and speech generation inside the product.
• Persisted business state observed after the conversation.
• Repeated-trial stability under controlled seeds.

This catches failures that text-only tests cannot. A correct answer is still a failed voice interaction if it starts after six seconds, arrives as clipped audio, ignores an interruption, or never saves the requested state.

Reference architecture and dependency model

A simulation runner needs a small local core for scenario loading, transport, audio processing, artifact serialization, and deterministic scoring. Cloud SDKs should remain optional so protocol and artifact tests can run without credentials. The operational dependency graph depends on the selected mode:

A Python implementation can use Pydantic, YAML parsing, HTTP and WebSocket transport, deterministic tone generation, WAV processing, and JSON serialization in its local core. Load cloud SDKs only when the selected persona, speech provider, or judge requires them, and return a targeted configuration error when a provider is unavailable.

A credential-free end-to-end implementation

Start with the checked-in workload-neutral fixture. It uses a scripted persona, deterministic tone audio, and a temporary local WebSocket server. The server is fake, but the client still passes through the production WebSocket adapter and strict evaluator.init protocol. This is a transport-level integration test rather than a direct in-memory method call.

Run configuration

Create examples/run-workload-neutral-voice.yaml:

scenario_path: scenario-workload-neutral-voice.yaml
persona_path: persona-workload-neutral-voice.yaml
data_path: data-workload-neutral-voice.yaml
script_path: script-workload-neutral-voice.yaml
persona_mode: scripted
speech_provider: tone
endpoint:
  adapter: fake
artifacts_dir: ../artifacts/workload-neutral-voice
trial_seed: 42
pace_audio: false

The fake endpoint accepts no cloud configuration. pace_audio: false avoids wall-clock sleeping in deterministic CI. For a real endpoint with server-side voice activity detection, audio normally must be paced in realtime.

Scenario

Create examples/scenario-workload-neutral-voice.yaml:

id: workload-neutral-voice-exchange
description: Confirm a short daily plan through a voice assistant
expected_topics:
  - plan recorded
  - reminder saved
conversation:
  max_turns: 4
  required_capabilities:
    - streaming_input
    - streaming_output
eval_spec:
  expected_exit: endpoint_completion
  limits:
    max_turns: 4
    max_time_seconds: 30

required_capabilities fails configuration before execution when an adapter cannot provide the requested interaction surface. The built-in HTTP adapter, for example, declares non-streaming input and output. It should not silently run a streaming scenario and produce a misleading result.

Persona and opaque workload data

Create examples/persona-workload-neutral-voice.yaml:

name: Jordan
experience_level: familiar
communication_style: concise
specialization: everyday planning
session_behavior:
  conclude_quickly: true
  decline_extras: true

Create examples/data-workload-neutral-voice.yaml:

session_id: planning-session-001
context:
  topic: daily planning
  requested_reminder: Submit the weekly summary at 16:00

The data model deliberately remains opaque. The runner preserves arbitrary mappings and sends them to the endpoint adapter without imposing field-service, scheduling, or customer-support fields. Product-specific coupling belongs in the adapter and outcome probe, not the core runner.

Scripted turn

Create examples/script-workload-neutral-voice.yaml:

persona_voice: en-US-AvaNeural
post_script_completion_wait_seconds: 0
turns:
  - trigger: planning prompt
    response: Please record the plan and save the reminder.
    keywords:
      - daily plan
      - reminder

The scripted persona searches unused turns for a case-insensitive keyword match. When none match, it asks the assistant to repeat itself. Scripted mode is deterministic but can break when the product changes its opening wording. Synthesized and realtime personas provide broader conversational coverage at the cost of determinism and additional spend.

Validate and run

Validation should resolve every referenced asset, normalize environment WAV files, check custom imports, and verify structure without opening cloud or product connections:

voice-evaluator validate examples/run-workload-neutral-voice.yaml

Then run three seeded trials:

voice-evaluator run examples/run-workload-neutral-voice.yaml \
  --trials 3 \
  --artifacts-dir artifacts/workload-neutral-voice

The CLI returns:

• Exit code 0 when all trials execute and pass pass^k.
• Exit code 2 for CLI, configuration, asset, or override validation errors.
• Exit code 3 when execution finishes but a critical quality verdict fails.
• Exit code 4 when a trial or provider fails to execute.

Expected output contract

A two-trial fake WebSocket test can assert this result contract:

{
  "process_exit_code": 0,
  "trial_count": 2,
  "pass_power_k": true,
  "trial_reports_present": true,
  "artifact_bundle_path_recorded": true
}

The end-to-end runner test should also record the protocol order:

[
  "evaluator.init",
  "binary_audio",
  "binary_audio",
  "input_audio_buffer.commit",
  "response.create"
]

It can assert the following execution properties:

execution_status: succeeded
exit_reason: endpoint_completion
passed: true
outcome_status: available
voice_event_schema_versions: ["1.0"]
interoperability_schema_version: "1.0"
audio_format: 24000 Hz, mono, 16-bit PCM

The deterministic verdict names in this example are:

[
  "expected_topics",
  "expected_exit",
  "concise",
  "outcome_follow_up_created",
  "outcome_report_fields_complete"
]

The test should confirm that the output bundle contains:

trial-001/
  transcript.json
  transcript.md
  tool-calls.json
  raw-endpoint-events.json
  voice-events.jsonl
  voice-metrics.json
  interoperability.json
  technician.wav
  product.wav
  mixed.wav
  resolved-scenario.json
  resolved-persona.json
  resolved-data.json
  resolved-environment.json
  resolved-script.json
  resolved-settings.json
  trial-report.json
run-report.json
evaluator.log

Passing this test shows that the code path writes a complete artifact bundle and satisfies its deterministic fake scenario. It does not establish live-provider reliability, cross-region latency, speech-recognition accuracy, naturalness, or production scale.

Endpoint integration contracts

The adapter layer defines the framework’s main architectural boundary. The runner depends on a ConversationEndpoint protocol rather than a particular agent SDK. The built-in choices are:

The WebSocket flow sends initialization data once and then streams audio:

{
  "type": "evaluator.init",
  "request": {
    "session_id": "planning-session-001",
    "context": {
      "topic": "daily planning"
    }
  }
}

After binary PCM frames, the evaluator commits the buffer and requests a response:

{"type":"input_audio_buffer.commit"}
{"type":"response.create"}

An adapter normalizes product events into framework events: audio deltas, final transcripts, tool calls, tool results, interruptions, turn completion, conversation completion, errors, and connection closure. As a result, product-specific conditionals stay out of the runner.

Custom adapter shape

An existing service can be integrated through a dotted class path. A simplified adapter follows this protocol:

from collections.abc import AsyncIterator


class ExistingVoiceServiceAdapter:
    capabilities = {
        "streaming_input": True,
        "streaming_output": True,
        "interruption": True,
        "tool_events": True,
        "input_transcripts": True,
    }

    async def connect(self) -> None:
        """Authenticate and open the product connection."""

    async def send_audio(self, audio: bytes) -> None:
        """Translate canonical PCM into the product wire format."""

    async def commit_turn(self) -> None:
        """Signal the end of evaluator input."""

    async def events(self) -> AsyncIterator[dict[str, object]]:
        """Normalize product messages into evaluator events."""
        if False:
            yield

    async def interrupt(self) -> None:
        """Request cancellation of current product output."""

    async def close(self) -> None:
        """Release network resources."""

Configure the adapter and its settings:

endpoint:
  adapter: my_voice_eval.adapters.ExistingVoiceServiceAdapter
  settings:
    uri: ${VOICE_AGENT_URL}
    access_token: ${VOICE_AGENT_ACCESS_TOKEN}
    connect_timeout_seconds: 10
    receive_timeout_seconds: 45
    max_message_bytes: 4194304

The exact method names and constructor behavior depend on the runner’s endpoint protocol and loader contract. Authentication, initialization, product event parsing, and wire-level retries belong to the adapter.

Outcome probes

Conversation evidence alone cannot prove that the application persisted the correct state. A separate OutcomeProbe queries the product after execution:

from collections.abc import Mapping
from typing import Any


class ReminderOutcomeProbe:
    async def probe(
        self,
    context: Mapping[str, Any],
    ) -> Mapping[str, Any] | None:
    session_id = str(context["session_id"])
    reminder = await self._client.get_reminder(session_id)
        return {
            "reminder_created": reminder is not None,
            "reminder_time": reminder.time if reminder else None,
      "session_id": session_id,
        }

The scenario can then require persisted values:

eval_spec:
  expected_exit: endpoint_completion
  outcome_spec:
    reminder_created: true
    reminder_time: "16:00"

Keep these checks separate. Transcript keyword matching is not a substitute for a database or API assertion.

Persona modes and realism

A simulation-based evaluator can provide three persona modes, each with a different reproducibility and cost profile.

Scripted mode

Scripted mode chooses a predefined response and converts it to audio. Use deterministic tone speech for transport tests or Azure Speech and Voice Live exact-text synthesis for realistic audio.

Scripted mode provides repeatable text, controlled coverage, low local cost, and straightforward regression diagnosis. Its conversational adaptability is limited, and keyword triggers can be brittle.

Synthesizer mode

Synthesizer mode asks Azure OpenAI for the next simulated-user reply using the persona, scenario, opaque run data, conversation history, and current product turn. A separate speech provider converts that exact response to PCM.

This mode supports varied paraphrases and longer conversations while retaining a text boundary before speech generation. With natural speech enabled, it introduces two nondeterministic systems: the persona model and the speech synthesis service.

Realtime mode

Realtime mode opens one Voice Live persona session per trial. Product audio is forwarded to the simulated user, and the returned audio and transcript become the next user turn. The evaluator controls append, commit, and response boundaries rather than delegating them to server voice activity detection.

This is the highest-fidelity built-in persona path and usually the most expensive. Failures are also harder to attribute because user simulation, product behavior, and two realtime audio paths can all vary.

Audio contract and environment simulation

The canonical audio format is raw pcm_s16le, 24,000 Hz, one channel. Environment WAV files are normalized into that format before execution. Compressed WAV, MP3, microphone capture, and speaker playback are not supported by the core runner.

Environment assets can mix continuous base noise, background voices, and transient events into simulated-user audio:

impact: 0.7
base_noise: audio/office-hum.wav
base_noise_volume: 0.2
background_voices: audio/distant-conversation.wav
background_voices_volume: 0.15
transient_events:
  - type: audio/door-close.wav
    volume: 0.5
    trigger_mode: on_turn
    on_turn_number: 2

The mixer uses the trial seed, so random placements can be reproduced. Continuous inputs loop as needed, transient events are scheduled or turn-triggered, and the mixer clips samples to the PCM16 range.

This setup tests degradation under controlled signal conditions. It does not model microphone frequency response, acoustic echo cancellation, packet loss, codec artifacts, room impulse response, or physical speaker-to-microphone coupling unless supplied audio or a custom adapter encodes those effects.

Voice timeline and metrics

Every trial can emit ordered voice-events.jsonl records with:

• Contract version, event ID, sequence, session ID, and turn ID.
• UTC timestamp and monotonic offset.
• Source, direction, event type, and speaker role.
• Optional PCM metadata.
• Optional W3C trace ID, span ID, and parent span ID.
• Event-specific normalized attributes.

From that timeline, the framework derives per-turn measurements:

Signal analysis produces these input and output fields:

{
  "encoding": "pcm_s16le",
  "sample_rate_hz": 24000,
  "channels": 1,
  "byte_count": 24000,
  "duration_seconds": 0.5,
  "silence_ratio": 0.08,
  "peak_amplitude": 12000,
  "rms_amplitude": 2140.7,
  "clipped_sample_ratio": 0.0,
  "possible_truncation": false
}

This JSON is an illustrative shape. The values depend on the trial audio.

The formulas are deterministic signal calculations. For a PCM sample sequence $x_1, x_2, \ldots, x_N$, duration and RMS amplitude are:

Silence ratio counts samples at or below an absolute threshold of 256. Clipping ratio counts samples at the PCM boundary. Possible truncation examines the final 10 milliseconds for a non-silent tail.

What these metrics do not measure

Transport and amplitude statistics do not measure perceptual voice quality. The current implementation does not provide:

• Word error rate or character error rate against a reference transcript.
• PESQ, POLQA, STOI, or another standardized intelligibility or quality score.
• Mean opinion score prediction or human-listener study support.
• Pronunciation, prosody, pacing, naturalness, emotion, or speaker-similarity scoring.
• Transcript-to-audio completeness based on speech recognition of rendered output.
• Acoustic echo, codec, packet-loss, jitter-buffer, or room-response analysis.

The shipping design discusses optional speech-quality evaluators for these concerns, but the inspected runtime implements only measurable transport and PCM characteristics. Mature deployments should export audio to a specialized speech evaluator or human review workflow. silence_ratio cannot support a broader quality verdict.

Deterministic and semantic scoring

The runner separates deterministic checks from model-based quality evaluation.

Deterministic verdicts

Built-in deterministic gates cover:

• Required topics in normalized assistant transcripts.
• Expected exit reason, such as endpoint_completion or script_exhaustion.
• Persisted outcome key-value expectations returned by an outcome probe.
• Successful execution without provider, adapter, timeout, or cleanup failures.

These checks are explainable, inexpensive, and suitable for critical CI gates. Expected-topic matching is token based, so semantically equivalent wording may fail. Use it for stable contractual phrases rather than broad meaning.

Optional Azure AI evaluation

Semantic evaluation uses the Azure AI Evaluation SDK after successful conversation execution. Supported built-in evaluator types are task adherence, groundedness, and coherence. Custom binary criteria use an Azure OpenAI label grader.

providers:
  azure_openai:
    endpoint: ${AZURE_OPENAI_ENDPOINT}
    model: voice-eval-judge
    auth: default_credential

quality_evaluation:
  timeout_seconds: 60
  azure_api_version: 2025-04-01-preview
  custom_judges:
    - name: no_unsupported_commitments
      criteria: >
        The assistant must not claim that a reminder was saved unless the
        recorded tool evidence supports that claim.
      is_critical: true
  azure_evaluators:
    - type: task_adherence
      is_critical: false
    - type: groundedness
      threshold: 4
      is_critical: false
    - type: coherence
      threshold: 3
      is_critical: false

Advisory evaluators remain visible without failing the deterministic gate. A critical judge failure or judge error fails closed. An unavailable required judge must not quietly become a pass.

Semantic evaluation still operates on normalized text and tool evidence, not perceptual audio. It adds model cost, latency, credential management, prompt sensitivity, and judge-version drift. Before making these checks release blockers, pin the judge deployment, retain rationales, and calibrate thresholds against human labels.

Multi-trial gates

Model responses, recognition, synthesis, timing, and networks all make voice systems nondeterministic. One passing conversation provides weak evidence. A simulation runner should record both common repeated-trial aggregates:

pass@k is true when at least one trial passes, which is useful for discovery or capability questions. pass^k is true only when every trial passes, and the CLI uses it as the quality gate.

For example, trial results [true, false, true] produce:

{
  "trial_count": 3,
  "passed_trials": 2,
  "pass_at_k": true,
  "pass_power_k": false
}

This example applies the aggregation rule to an illustrative run.

Seeds make evaluator-controlled choices reproducible, including tone audio and environment mixing. They cannot force deterministic output from remote speech, model, agent, or network services.

Artifact and interoperability design

The artifact bundle is one of the framework’s most useful features. It lets you investigate a failure through synchronized machine-readable and listenable evidence instead of relying on one aggregate score.

Use artifacts in this order:

1. Inspect trial-report.json for execution status, exit reason, critical verdicts, judge errors, and outcome state.
2. Read transcript.md and tool-calls.json to understand conversational and action behavior.
3. Inspect voice-metrics.json and voice-events.jsonl for latency, silence, interruption, and boundary timing.
4. Listen to technician.wav, product.wav, and mixed.wav to distinguish signal statistics from perceptual behavior.
5. Read raw endpoint events when normalization or product protocol behavior is suspect.
6. Correlate trace IDs with product observability systems.

interoperability.json packages schema version, session, scenario, trial, trace IDs, metrics, transcript, tool calls, and imported scores. External evaluators can enrich it without becoming part of the voice runner:

import json
from pathlib import Path

trial_path = Path("artifacts/run/trial-001/interoperability.json")
trial = json.loads(trial_path.read_text(encoding="utf-8"))
scores_path = Path("langchain-scores.json")
trial["imported_scores"] = json.loads(scores_path.read_text(encoding="utf-8"))

This division keeps ownership clear. DeepEval, Vally, Waza, LangChain, Agent Framework, or an observability platform can handle model and trajectory evaluation while the simulation runner handles the deployed audio exchange.

Cost model

Live suites incur provider and infrastructure costs. Per-trial cost is approximately:

For $S$ scenarios, $P$ personas, $E$ environments, and $K$ trials, the run count grows as:

A modest matrix of 20 scenarios, 3 personas, 4 environments, and 5 trials creates 1,200 conversations. Realtime persona simulation can effectively place a paid model on both sides of each conversation before judge calls are counted.

A staged pipeline keeps those costs under control:

Store WAV artifacts selectively. Across thousands of trials, three raw audio channels plus event logs can create a material retention cost.

Privacy and security

Voice artifacts are sensitive. Audio, transcripts, tool arguments, product outcomes, and opaque run data can contain personal, customer, operational, or regulated information.

The framework sanitizes secret-bearing configuration fields before writing resolved settings and calculating suite hashes. Custom adapters are still responsible for removing credentials from raw event payloads. Test that boundary explicitly because raw product events may include headers, signed URLs, or tool arguments the core sanitizer does not understand.

A production evaluation program should also:

• Use synthetic personas and records unless real data is specifically approved.
• Encrypt artifact storage and restrict data-plane access.
• Define retention separately for reports, transcripts, and WAV files.
• Avoid placing credentials or personal data directly in YAML.
• Review cloud-service data residency and logging behavior.
• Redact transcripts and tool payloads before external judge calls.
• Treat trace URLs and identifiers as potentially sensitive correlation data.
• Separate evaluator identities from production application identities.

DefaultAzureCredential avoids hard-coded keys, but it does not enforce least privilege. Assign only the speech, model inference, product endpoint, and artifact permissions required for the selected run.

Strengths

• It evaluates the deployed voice boundary rather than an isolated text response.
• HTTP, WebSocket, fake, and custom adapters keep product protocols outside the runner.
• Scripted, synthesized, and realtime personas support progressive realism.
• Outcome probes distinguish spoken claims from persisted application state.
• Ordered voice events, normalized transcripts, raw events, traces, and WAV files provide strong failure evidence.
• The fake WebSocket endpoint creates a credential-free CI path through the production adapter.
• Deterministic verdicts and optional semantic judges have separate failure semantics.
• Versioned JSON schemas support external artifact consumers.
• pass@k and pass^k make repeated-trial behavior explicit.

Weaknesses and risks

• Current metrics cover transport timing and PCM characteristics, not perceptual speech quality.
• Built-in interruption capability is incomplete for the standard HTTP and WebSocket adapters.
• Expected-topic scoring is lexical and can miss semantic equivalents.
• Natural persona modes add cloud cost and nondeterminism.
• Product-specific adapters and datasets can leak domain assumptions into otherwise generic runs.
• WAV-only environment input excludes common compressed audio formats.
• The framework does not capture microphones, play speakers, or reproduce physical acoustic loops.
• Custom adapters own security-sensitive authentication and raw-event sanitation.

Best fit and poor fit

Simulation-based conversation evaluation fits teams that need regression tests for a deployed voice endpoint across scripted audio, simulated users, network protocols, tool behavior, persisted outcomes, latency, and retained audio evidence. It is particularly relevant when the same agent already has separate text, trajectory, and RAG evaluation.

It is a poor fit for prompt quality, retrieval quality, agent graph state, skill selection, tool-call precision without audio, or standardized speech quality. DeepEval, Vally, Waza, framework-native tracing, and dedicated speech-quality systems address those needs more directly.

Final assessment

The framework makes the right architectural choice by exercising a real endpoint and preserving enough evidence to attribute failures across input audio, turn handling, agent behavior, tools, output audio, and persisted state. Its adapter boundary, fake transport fixture, artifact bundle, deterministic gates, and interoperability schema provide a credible technical foundation.

This is not a general conversation evaluator. Its distinct value is deployed voice-agent regression testing. PCM and latency metrics cover only part of that problem; perceptual quality still requires specialized evaluation or human review.

Why I created convo-eval

Text-based evaluation could tell me whether an answer was relevant or grounded, but it could not tell me whether the deployed voice experience actually worked. I created convo-eval to run repeatable, persona-driven conversations through a real endpoint and keep the audio, timing, transcript, tool calls, and persisted outcomes together. That evidence makes failures easier to reproduce and gives release gates something more concrete than a single model score.

I added several evaluator layers for different kinds of failure:

• Deterministic evaluators check execution success, required transcript topics, and the expected conversation exit.
• Outcome evaluators compare values returned by a custom probe with the state the scenario expects the application to persist.
• Custom binary judges use Azure OpenAI label grading for criteria such as unsupported commitments.
• Azure AI evaluators measure task adherence, groundedness, and coherence from the normalized conversation and its recorded tool evidence.

I also added voice evidence metrics for first-audio latency, dead air, turn and interruption timing, audio duration, silence, peak and RMS amplitude, clipping, and possible truncation. These are deterministic transport and PCM measurements rather than semantic evaluators, but they catch voice failures that a transcript-only score cannot see.

#AI Evaluation#Voice Agents#Conversational AI#Azure AI#Python