Applied Notes

Series

Agent Evaluation

3 of 3 parts published

  1. 1
    Eval-Driven Development: Red-Green-Refactor for AI Agents

    · Every agent change should be justified by a failing evaluation. A disciplined, TDD-style process for evolving AI agents without regressions.

  2. 2
    Hybrid Agent Evaluation: Custom Judges + Azure Foundry Evaluators

    · Custom judges test scenario facts while Azure Foundry evaluators score general agent behavior. A REST boundary gives C# both views without adding Python to the application.

  3. 3
    Beyond 1–5 Scores: Binary Judges, Outcome Verification, and pass^k

    · Multi-point judge scales are subjective, transcript-only scoring misses system state, and a single trial can be a lucky run. Four changes that make agent evaluation trustworthy.

AI Evaluation Frameworks in Practice

5 of 5 parts published

  1. 1
    DeepEval: Pytest-Native Evaluation for LLM Applications and Agents

    · DeepEval is the strongest general-purpose starting point for Python LLM, RAG, and agent evaluation. Start with deterministic outcomes, then add calibrated semantic metrics where code falls short.

  2. 2
    Microsoft Vally: Declarative Trajectory Evaluation for AI Agents

    · Vally offers the strongest trajectory pipeline for coding and tool-using agents, but its pre-1.0 contracts and unresolved source and support model make a version-pinned pilot safer than an organization-wide gate.

  3. 3
    Microsoft Waza: Evaluation for Agent Skills, Triggers, and Tool Behavior

    · Waza is a strong specialist for testing skill selection, anti-triggers, tools, and workspace effects. Pair it with other evaluators for RAG, voice, multimodal, or multi-agent behavior.

  4. 4
    Conversation Evaluation Through Voice-Agent Simulation

    · Persona-driven simulations catch deployed voice failures that text tests miss by retaining audio, timing, transcripts, tools, and outcomes. Keep deterministic gates primary and evaluate perceptual speech quality separately.

  5. 5
    DeepEval vs Vally vs Waza vs convo-eval: A Technical Decision Guide

    · Match the evaluator to the evidence boundary: DeepEval for responses and traces, Vally for trajectories and workspaces, Waza for skills, and a licensed voice harness for deployed audio.

Building a Governed Agentic SDLC

2 of 4 parts published

  1. 1
    What Changes When Coding Agents Can Act?

    · Governed Agentic SDLC is an integration problem: execution, identity, policy, evidence, evaluation, and human approval must reinforce one another while authority stays outside the model.

  2. 2
    Isolated Execution for Coding Agents: Where the Sandbox Ends

    · Choose an agent sandbox by its complete boundary, not its runtime label. Kernel isolation, network paths, credentials, lifecycle controls, evidence, and broker privilege can each decide whether containment holds.

On-Prem SLM Inference with ONNX Runtime GenAI

2 of 2 parts published

  1. 1
    Choosing an On-Prem SLM Inference Engine for Windows

    · ONNX Runtime GenAI was the practical default for offline Phi-3 inference on Windows: it led throughput and latency, stayed under five seconds, and avoided custom builds or Docker.

  2. 2
    Wiring ONNX Runtime GenAI into a LangChain RAG Pipeline

    · A small LangChain adapter can run ONNX Runtime GenAI locally, but deployment performance depends on building and shipping the optimized Phi-3 graph instead of loading a raw export.