Evaluation
Artwork: The Judgment of Paris by Lucas Cranach the Elder. The Metropolitan Museum of Art · Public domain
EvaluationDeepEval vs Vally vs Waza vs convo-eval: A Technical Decision Guide
Match the evaluator to the evidence boundary: DeepEval for responses and traces, Vally for trajectories and workspaces, Waza for skills, and a licensed voice harness for deployed audio.
Artwork: The Spanish Singer by Edouard Manet. The Metropolitan Museum of Art · Public domain
EvaluationConversation Evaluation Through Voice-Agent Simulation
Persona-driven simulations catch deployed voice failures that text tests miss by retaining audio, timing, transcripts, tools, and outcomes. Keep deterministic gates primary and evaluate perceptual speech quality separately.
Artwork: Wheat Field with Cypresses by Vincent van Gogh. The Metropolitan Museum of Art · Public domain
EvaluationDeepEval: Pytest-Native Evaluation for LLM Applications and Agents
DeepEval is the strongest general-purpose starting point for Python LLM, RAG, and agent evaluation. Start with deterministic outcomes, then add calibrated semantic metrics where code falls short.
Artwork: Wheat Field with Cypresses by Vincent van Gogh. The Metropolitan Museum of Art · Public domain
EvaluationMicrosoft Vally: Declarative Trajectory Evaluation for AI Agents
Vally offers the strongest trajectory pipeline for coding and tool-using agents, but its pre-1.0 contracts and unresolved source and support model make a version-pinned pilot safer than an organization-wide gate.
Artwork: The Fortune-Teller by Georges de La Tour. The Metropolitan Museum of Art · Public domain
EvaluationMicrosoft Waza: Evaluation for Agent Skills, Triggers, and Tool Behavior
Waza is a strong specialist for testing skill selection, anti-triggers, tools, and workspace effects. Pair it with other evaluators for RAG, voice, multimodal, or multi-agent behavior.
Artwork: La Berceuse by Vincent van Gogh. The Metropolitan Museum of Art · Public domain
EvaluationEvaluating a gpt-realtime Voice Agent with Simulation
How we test a speech-to-speech debriefing agent end-to-end: a persona-driven simulated technician talks to the real gpt-realtime agent over a WebSocket, and binary judges plus outcome verification grade the result.
Artwork: Saint Jerome by Jusepe de Ribera. Cleveland Museum of Art · Public domain
EvaluationSkill Evaluation with Microsoft Waza
Use Waza to test skill routing and output relevance, then separate implementation evaluations to gate code quality. Mock early, run live trials selectively, and enforce both tiers in CI.
Artwork: The Crucifixion of Saint Andrew by Caravaggio. Cleveland Museum of Art · Public domain
EvaluationBeyond 1–5 Scores: Binary Judges, Outcome Verification, and pass^k
Multi-point judge scales are subjective, transcript-only scoring misses system state, and a single trial can be a lucky run. Four changes that make agent evaluation trustworthy.
Artwork: Mäda Primavesi by Gustav Klimt. The Metropolitan Museum of Art · Public domain
EvaluationHybrid Agent Evaluation: Custom Judges + Azure Foundry Evaluators
Custom judges test scenario facts while Azure Foundry evaluators score general agent behavior. A REST boundary gives C# both views without adding Python to the application.
Artwork: Wheat Field with Cypresses by Vincent van Gogh. The Metropolitan Museum of Art · Public domain
EvaluationEval-Driven Development: Red-Green-Refactor for AI Agents
Every agent change should be justified by a failing evaluation. A disciplined, TDD-style process for evolving AI agents without regressions.