Evaluation

DeepEval vs Vally vs Waza vs convo-eval: A Technical Decision Guide

By Omer DemirAug 30, 2026⏱ 26 min read👁 — reads⧉ — copies

AI disclosure: This article was moderated by AI.

DeepEval, Microsoft Vally, Microsoft Waza, and convo-eval often appear in the same tool search, but each evaluates a different object. Metric and feature counts are a poor basis for choosing among them.

Start with the evidence boundary:

• DeepEval starts from LLM test cases, conversations, and instrumented traces.
• Vally starts from an agent task, isolated workspace, and normalized trajectory.
• Waza starts from an agent skill or custom-agent definition and asks whether it routes and behaves correctly.
• convo-eval starts from a deployed voice endpoint and retains transport, audio, transcript, tool, and persisted-outcome evidence.

The recommendation depends on the scenario:

This comparison covers DeepEval 4.2.0, Vally 0.14.0, Waza 0.38.7 with evaluation schema 1.2, and the authenticated convo-eval source contract 0.1.0. The review snapshot is August 30, 2026.

Executive verdict

DeepEval is the default for Python-first teams that need broad coverage of responses, RAG, conversations, safety, and traced agents. It has the largest metric catalog and strongest public community in this group. In exchange, teams take on a large dependency graph, frequent judge calls, default telemetry, and a collaboration path tied to Confident AI.

Vally fits coding and tool-using agents whose correctness depends on both the trajectory and final workspace. Its declarative experiments, isolated environments, deterministic graders, offline re-grading, golden patches, and reporters make it the strongest general trajectory pipeline reviewed here. Adoption should remain cautious because Vally is pre-1.0 and its referenced public source repository was unavailable during review.

Waza fits agent skills. It has the strongest support for positive and negative trigger tests, SKILL.md specification coverage, skill-invocation evidence, tool ordering, filesystem checks, no-skill baselines, and cross-model comparison. It is neither a RAG evaluator nor a complete multi-agent handoff evaluator.

The convo-eval architecture fits deployed voice endpoints that need repeatable audio conversations, outcome probes, voice timelines, and retained WAV evidence. The inspected source is not yet suitable as an external dependency because no public license or release exists.

Compare the unit under test first

Their native test objects make the differences easier to see.

Consider a support assistant that retrieves policy, invokes a refund tool, and speaks to the user:

1. DeepEval can test answer faithfulness against retrieved policy.
2. Vally can test the agent trajectory and resulting case workspace.
3. Waza can test whether the refund skill activates only for eligible requests.
4. convo-eval can test whether the deployed voice route understands audio, invokes the workflow, persists the refund state, and returns untruncated speech within an acceptable latency envelope.

These tests complement one another. Forcing one framework to own all four produces shallow coverage and awkward custom code.

Eligibility gates before scoring

Apply legal, runtime, security, and evidence gates before assigning feature scores. A weighted score cannot compensate for a failed eligibility requirement.

License and source gate

Source without a license is not “free but unsupported.” Copyright applies by default. Without permission, external users should not assume rights to copy, modify, redistribute, or operate the code.

Runtime gate

A Python 3.10-only platform excludes the current convo-eval matrix, and a Node 20 build fleet excludes Vally. Under a source-build-only policy, Waza’s embedded Git LFS dependency matters even though most users can install a prebuilt binary.

Evidence gate

No grader can recover evidence that was never observed. A rubric cannot determine whether audio clipped if the framework did not retain the audio.

Architecture comparison

The four execution models can be summarized as:

DeepEval
golden -> app invocation -> LLMTestCase / trace -> metric -> pytest verdict

Vally
stimulus -> isolated workspace -> executor -> trajectory -> graders -> report

Waza
skill + task -> workspace -> mock or Copilot executor -> behavior evidence -> graders

convo-eval
scenario + simulated user -> PCM -> deployed endpoint -> voice evidence + outcome -> gates

DeepEval architecture

DeepEval leaves application invocation under the test author’s control. The suite generates a current actual_output, builds an LLMTestCase, and applies one or more metrics. Instrumentation can add nested model, retriever, tool, and agent spans.

That flexibility works naturally with Pytest, but the team must design clean application fixtures and tracing boundaries. Storing a stale actual_output as test data grades a fixture instead of the current system.

Vally architecture

Vally owns more of the execution lifecycle. It prepares a workspace, invokes an executor, records a normalized trajectory, grades files and behavior, and saves results for offline re-grading.

Workspace state is first-class, which suits coding agents. The model is heavier than a row-based response test and depends on the selected executor capturing enough evidence.

Waza architecture

Waza combines task execution with skill discovery and injection. It records whether a skill was selected, whether that selection was correct, and whether execution improved over a baseline without the skill.

The baseline exposes regressions that absolute scoring misses. An answer can be acceptable while the skill makes the agent slower, less accurate, or more expensive.

convo-eval architecture

convo-eval hides product wire formats behind HTTP, WebSocket, fake, or custom endpoint adapters. It generates canonical 24 kHz mono PCM16 input, drives turns, normalizes endpoint events, derives a monotonic voice timeline, probes persisted state, and writes synchronized JSON and WAV artifacts.

Its boundary is the deployed endpoint. The product may use Agent Framework, LangGraph, an OpenAI SDK, custom orchestration, or no explicit agent framework at all.

Grader and metric comparison

Deterministic grading

Vally and Waza provide the deepest built-in deterministic coverage for filesystem-changing agents. DeepEval supports deterministic metrics, though semantic evaluation dominates its catalog and product identity. convo-eval derives deterministic voice metrics, but the inspected configuration does not expose every signal as a direct threshold gate.

Model-based grading

DeepEval has the broadest semantic metric catalog. Vally and Waza treat model judges as one grader class among many. In convo-eval, they are optional post-execution semantic checks over transcript and tool evidence, not perceptual voice quality.

Repetition and flakiness

All four can participate in repeated evaluation, but their semantics differ:

• DeepEval repeats through Pytest, metric configuration, or external orchestration and can aggregate test runs in Confident AI.
• Vally records pass@k, pass^k, and flakiness across agent executions.
• Waza supports trials, attempts, multi-model execution, baseline comparison, and caching where safe.
• convo-eval computes pass@k and strict pass^k over complete voice trials with deterministic seeds for evaluator-controlled choices.

Use pass@k to ask whether any attempt solved the task. Use pass^k to ask whether every required attempt passed. Reporting only the best attempt in a release gate hides instability.

Output provenance across this series

Terminal output can look authoritative even when it has been reconstructed. The articles therefore use three explicit evidence classes:

These provenance differences affect confidence. Source tests can prove deterministic contracts, but they cannot replace live provider trials, latency measurements, or perceptual listening.

Artifact comparison

The most useful diagnostic artifact is the one closest to the native risk:

• DeepEval needs metric reasons and trace spans.
• Vally needs the trajectory, diff, and final workspace evidence.
• Waza needs skill-invocation records, tool events, baseline results, and snapshots.
• convo-eval needs normalized and raw events, latency timing, persisted outcomes, and listenable audio.

Set retention limits. Prompts, retrieved context, source files, tool arguments, transcripts, audio, and business outcomes can all contain sensitive data.

Dependency and operational burden

DeepEval

The mandatory Python graph includes Pytest, Pydantic, OpenAI, OpenTelemetry, PostHog, HTTP clients, gRPC, Rich, Typer, and concurrency-related plugins. This supports an integrated evaluation and tracing platform at the cost of longer installs, more lockfile movement, and a wider security review.

Vally

Vally requires a modern Node runtime and includes workspace lifecycle, Git integration, executor support, result persistence, OpenTelemetry, and reporting. It is a larger operational component than a metric library. Its default execution path also introduces GitHub Copilot SDK credentials and behavior.

Waza

The prebuilt Waza binary keeps runtime installation compact. Source builds are less conventional: they require Go 1.26 or newer, Git, Git LFS, and a repository clone containing embedded Copilot CLI artifacts. Live execution still depends on Copilot authentication or a configured compatible provider.

convo-eval

The base Python package supports local fake runs without cloud credentials. Extras add Azure OpenAI, Azure AI Evaluation, Azure Speech, or Voice Live. Live product tests also need endpoint adapters, authentication, service availability, and enough storage for PCM audio artifacts.

Cost comparison

An open-source license with no fee does not make evaluation free. Application execution, judge execution, infrastructure, artifact storage, and human review all contribute to cost.

DeepEval hosted pricing snapshot

The DeepEval open-source package uses the Apache-2.0 license. At review time, Confident AI listed a free hosted tier, Starter at $200 per month, Team at$2,000 per month, and custom Enterprise pricing. Provider model calls are billed separately.

Vally and Waza

Vally’s package metadata and Waza’s repository report MIT licensing with no listed framework fee. Agent execution is their main cost. Deterministic re-grading avoids paying for another trajectory while tuning a grader.

Voice evaluation

A live voice trial may incur charges for a simulated persona model, persona speech, product speech recognition, product reasoning, product speech generation, semantic judges, network use, and three audio artifacts. The test matrix grows multiplicatively:

where $S$ is scenarios, $P$ is personas, $E$ is audio environments, and $K$ is repeated trials. A 20 by 3 by 4 by 5 matrix already creates 1,200 conversations.

Privacy and data egress

Disabling usage telemetry does not create a zero-egress evaluation. Judge calls still transmit evidence, and agent execution may transmit prompts, files, or tool context. Product endpoint tests intentionally send audio and data. OTLP exporters and hosted result systems add more boundaries.

For restricted workloads, map each field to every receiver:

prompt / scenario
  -> application model
  -> evaluator judge
  -> result artifact
  -> telemetry or trace backend

retrieved document / source file / opaque data
  -> agent context
  -> tool arguments
  -> grader context
  -> report or snapshot

audio
  -> speech provider
  -> deployed endpoint
  -> WAV artifact store
  -> optional perceptual evaluator

Apply synthetic data, least-privilege identities, encryption, redaction, and differentiated retention at each boundary.

Ownership, community, and support

Repository stars are neither a quality metric nor an SLA. They indicate only how likely users are to find examples, prior issues, integrations, and upgrade experience beyond the maintainer team.

Technical decision matrix

The scores below use 5 for strong native support, 3 for support that requires meaningful configuration or custom work, 1 for a weak fit, and 0 for unavailable or disqualified. This is an architectural aid, not a benchmark result.

Do not sum the table without weights and eligibility gates. Voice transport timing has no value in a text-only RAG service, while RAG faithfulness has limited value when the main requirement is whether a SKILL.md file triggers.

Reproducible selection policy

The following Python policy applies eligibility rules before ranking native fit:

from dataclasses import dataclass


@dataclass(frozen=True)
class Requirement:
    name: str
    runtime: str
    public_license_required: bool = True
    rag: bool = False
    trajectory: bool = False
    skill_routing: bool = False
    deployed_voice: bool = False


def select(requirement: Requirement) -> str:
    if requirement.deployed_voice:
        if requirement.public_license_required:
            return "BLOCKED: convo-eval fits technically but is not licensed"
        return "convo-eval"
    if requirement.skill_routing:
        return "Waza"
    if requirement.trajectory and requirement.runtime == "node22":
        return "Vally pilot"
    if requirement.rag or requirement.runtime.startswith("python"):
        return "DeepEval"
    return "Run a focused proof of concept"


requirements = [
    Requirement("rag-platform", "python312", rag=True),
    Requirement("coding-agent", "node22", trajectory=True),
    Requirement("skill-authoring", "native", skill_routing=True),
    Requirement("voice-regression", "python312", deployed_voice=True),
]

for requirement in requirements:
    print(f"{requirement.name}: {select(requirement)}")

The deterministic worked output is:

rag-platform: DeepEval
coding-agent: Vally pilot
skill-authoring: Waza
voice-regression: BLOCKED: convo-eval fits technically but is not licensed

The policy is intentionally simple. Before assigning weighted scores, an organization should add gates for source availability, approved providers, data residency, artifact retention, support, and budget.

Scenario decisions

Python RAG service

Choose DeepEval. Its native test-case fields distinguish context from retrieval context, and its metric catalog covers faithfulness, relevancy, precision, recall, safety, and custom criteria. Add deterministic assertions for citations, schemas, and business rules before using judge metrics.

Vally and Waza add unnecessary execution machinery unless the RAG system includes an agent that modifies files or depends on skill behavior. convo-eval matters only when the service exposes a voice path that requires separate testing.

Coding agent in a repository

Choose Vally when both the agent trajectory and final workspace matter. Its file, diff, tool-order, program, token-budget, and wall-time graders fit that problem directly. Offline re-grading and golden patches also help with evaluator development.

Choose Waza when the central question is whether one skill was selected and followed. When both concerns matter, Waza can cover skill routing and Vally can cover broader repository-task experiments. Confirm that the duplicate execution cost is justified.

Agent skill marketplace

Choose Waza. Each skill should have:

• Positive trigger prompts.
• Adjacent anti-trigger prompts.
• Specification coverage for instructions promised in SKILL.md.
• Required and forbidden tool assertions.
• Output and filesystem assertions.
• A no-skill baseline.
• Repeated trials across supported execution models.

DeepEval can judge the final answer, but it does not natively determine whether the skill activated correctly. Vally is a reasonable second choice when broader trajectory experiments matter more than Waza’s specialist workflow.

Deployed realtime voice agent

The convo-eval architecture is the only reviewed design that directly handles streaming audio, endpoint adapters, PCM normalization, voice events, latency derivation, environmental mixing, outcome probes, and WAV evidence.

The inspected package is not publicly adoptable. An external team has three choices:

1. Wait for an approved license and immutable release.
2. Build an internal harness from independently designed, licensed components using the same endpoint and evidence principles.
3. Use a licensed commercial or open-source voice-evaluation product and export transcript or trace evidence to the general evaluator.

Whatever implementation you choose, add human listening or a specialized speech-quality system. Silence, clipping, and latency do not measure naturalness, pronunciation, intelligibility, or speaker similarity.

TypeScript-only organization

Vally has the clearest native fit if the organization accepts Node 22.12 or newer and the package’s source and support boundary. DeepEval’s TypeScript package showed registry and dependency inconsistency during review, so documentation parity is not enough; require a clean installation test before selection.

Waza can run as a native binary in a TypeScript repository. That language independence helps when the target is a skill rather than application code.

Air-gapped or highly restricted environment

Running locally does not exempt any framework from review.

• DeepEval needs local deterministic metrics or an approved local judge, telemetry disabled, and no Confident AI upload.
• Vally needs a model-free executor path or approved local provider, telemetry disabled, and local artifact retention.
• Waza needs pre-archived binaries, disabled update checks, mock execution or an approved provider, and no remote graders.
• convo-eval needs the fake endpoint or approved internal services and remains blocked by licensing outside authorized use.

Deterministic checks should dominate strict release gates in all four designs.

Layered evaluation architecture

Many production agents need more than one evaluator. Organize evaluation by failure layer rather than forcing every test into one schema:

Layer 1: deterministic unit and contract tests
  schemas, pure functions, tool contracts, persisted-state assertions
                           |
                           v
Layer 2: response and retrieval evaluation
  DeepEval test cases, RAG metrics, calibrated semantic checks
                           |
                           v
Layer 3: agent execution evaluation
  Vally trajectories and workspace graders
                           |
                           v
Layer 4: skill routing evaluation
  Waza trigger, anti-trigger, baseline, and tool-behavior tests
                           |
                           v
Layer 5: deployed channel evaluation
  licensed voice harness, HTTP/WebSocket audio, latency, outcome, WAV
                           |
                           v
Layer 6: production monitoring and human review
  traces, sampled online evals, incidents, perceptual listening

Few applications need every layer. A simple RAG API may stop after layer 2 plus production monitoring. A skill-driven coding agent may emphasize layers 3 and 4, while a regulated voice workflow may require all six.

Shared result envelope

Keep native artifacts and publish a small cross-framework summary for dashboards and release policy:

{
  "schema_version": "1.0",
  "suite_id": "refund-agent-release-2026-08-30",
  "framework": "waza",
  "framework_version": "0.38.7",
  "scenario_id": "refund-skill-eligible-request",
  "trial_id": "trial-003",
  "execution_status": "succeeded",
  "gate_status": "passed",
  "scores": [
    {
      "name": "required_skill_invocation",
      "value": true,
      "critical": true
    }
  ],
  "artifact_uri": "artifact://agent-evals/waza/trial-003",
  "trace_ids": ["4bf92f3577b34da6a3ce929d0e0e4736"]
}

Do not flatten native evidence into this envelope. It should point to DeepEval traces, Vally trajectories, Waza snapshots, or voice artifacts rather than replacing them.

CI staging

A practical pipeline separates inexpensive deterministic checks from costly live suites:

name: evaluation-layers

on:
  pull_request:
  schedule:
    - cron: '0 2 * * *'

jobs:
  response-evals:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: pip install 'deepeval==4.2.0'
      - run: deepeval test run tests/evals -n 1
        env:
          DEEPEVAL_TELEMETRY_OPT_OUT: '1'
          OPENAI_API_KEY: ${{ secrets.EVAL_OPENAI_API_KEY }}

  skill-evals:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ./tools/waza-0.38.7 run evals/skill.yaml --reporter junit:results.xml
        env:
          WAZA_NO_UPDATE_CHECK: '1'
          GITHUB_TOKEN: ${{ secrets.COPILOT_EVAL_TOKEN }}
      - if: always()
        uses: actions/upload-artifact@v4
        with:
          name: skill-eval-results
          path: results.xml

  nightly-agent-evals:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '22'
      - run: npm ci
      - run: npx vally@0.14.0 eval --eval-spec evals/agent.yaml --junit --output-dir results
        env:
          VALLY_TELEMETRY_OPTOUT: '1'
          COPILOT_GITHUB_TOKEN: ${{ secrets.COPILOT_EVAL_TOKEN }}
      - if: always()
        uses: actions/upload-artifact@v4
        with:
          name: nightly-agent-results
          path: results/

The example omits convo-eval deliberately. An unlicensed source package should not appear as a normal public CI recommendation. Add a licensed voice harness in a separate scheduled or pre-release job.

Common selection mistakes

Counting metrics instead of mapping risks

Fifty text metrics do not cover audio truncation, and a rich file grader does not cover retrieval faithfulness. Start with the failure modes and evidence you need.

Making every judge a critical gate

Model judges vary with model versions, prompts, sampling, and provider behavior. Calibrate them against human labels, repeat borderline cases, and use deterministic gates for contractual requirements.

Treating pre-1.0 versions as stable APIs

Vally, Waza, and convo-eval all expose pre-1.0 surfaces. Pin versions, archive schemas and binaries, test upgrades, and preserve rollback paths.

Confusing local execution with privacy

Local orchestration may still send prompts, files, retrieved context, transcripts, audio, and tool evidence to providers. Document the complete data flow.

Ignoring evaluator validation

Validate the evaluators themselves. Vally supports golden trajectories and patches. Waza supports mock execution, MCP fixtures, and baseline comparisons. DeepEval custom metrics need ordinary unit tests. Voice adapters need conformance fixtures and known PCM signals.

Using one framework because procurement is easier

Standardization has value, but forcing all evidence into one evaluator may create more maintenance than supporting two focused tools. Standardize artifact retention, identity, telemetry, and summary reporting while allowing specialized execution engines.

Final recommendation by team

Python application and ML platform teams

Start with DeepEval. Build deterministic custom metrics for schemas and business invariants before adding calibrated semantic metrics. Disable telemetry explicitly and make a deliberate choice between local and hosted artifacts.

Agent platform teams

Pilot Vally for trajectory and workspace evaluation. Pin 0.14.0, validate the required executor and plugins, and preserve native artifacts. Resolve the public source and support model before making it organization-wide infrastructure.

Skill authors and coding-agent teams

Adopt Waza for SKILL.md routing and behavior. Test triggers and anti-triggers, compare against no-skill baselines, and start with deterministic graders. Keep custom-agent handoff expectations outside the gate until support matures.

Voice-agent teams

Use the convo-eval design as a requirements reference for deployed endpoint adapters, canonical audio, repeated simulated users, outcome probes, ordered timing evidence, trace correlation, and retained WAV files. Do not adopt the inspected package externally until it has an approved license and release. Add perceptual speech evaluation independently.

Closing verdict

DeepEval has the broadest coverage. Vally is strongest for general trajectory engineering, and Waza for skill evaluation. convo-eval has the best technical fit for deployed voice regression but fails the public-adoptability gate.

A defensible default is:

1. Use DeepEval for Python LLM, RAG, response, and trace evaluation.
2. Add Vally when coding-agent trajectories and final workspaces become first-class evidence.
3. Add Waza when skills, trigger boundaries, and skill-specific behavior are first-class release risks.
4. Add a licensed voice harness when the deployed audio experience is part of the product.
5. Keep deterministic contracts in the critical path and semantic judges calibrated, sparse, and evidence-backed.

Framework choice alone does not produce a sound evaluation. The work lies in choosing the correct evidence boundary, retaining artifacts that explain failures, validating graders, controlling cost and data egress, and refusing to turn unavailable evidence into confident scores.

Series references

• DeepEval: Pytest-Native Evaluation for LLM Applications and Agents
• Microsoft Vally: Declarative Trajectory Evaluation for AI Agents
• Microsoft Waza: Evaluation for Agent Skills, Triggers, and Tool Behavior
• convo-eval: End-to-End Evaluation for Deployed Voice Agents

#AI Evaluation#AI Agents#DeepEval#Microsoft Vally#Microsoft Waza#Voice Agents