Evaluation

Skill Evaluation with Microsoft Waza

By Omer DemirMay 5, 2026⏱ 5 min read👁 — reads⧉ — copies

Before a reusable AI skill ships to a customer engagement, you want proof it does the right thing: fires on the prompts it should, stays quiet on the ones it shouldn’t, and produces relevant output. We standardized that proof on Microsoft Waza, a YAML-first evaluation framework, and wired it into a two-tier pipeline that runs in CI.

Why Waza over a custom framework

An earlier project used a bespoke MLflow-based evaluation harness. It worked, but it meant a lot of custom scorer code tightly coupled to the Databricks/MLflow stack. Waza replaces that with a standard, portable format:

• Eval spec format — YAML evaluation definitions with graders and task patterns.
• Execution engines — mock for local/CI testing, copilot-sdk for production evaluation.
• Scaffold command — generates eval suites directly from SKILL.md files.
• Check command — validates skill structure and frontmatter compliance.
• Cross-model comparison — benchmarks a skill across GPT-4o, Claude Sonnet, and others.
• CI integration — GitHub Actions support for automated evaluation.

The payoff: one evaluation format shared across every skill, scaffolded from the skill’s own SKILL.md so setup is minimal.

The eval spec

A Waza eval is a YAML file declaring the skill under test, config, metrics, graders, and tasks:

name: conversational-assistant-eval
description: Tests that the skill triggers on technician-assistant requests,
  stays silent on unrelated prompts, and produces domain-relevant output.
skill: conversational-assistant
config:
  trials_per_task: 1
  executor: mock
  model: claude-sonnet-4.6
metrics:
  - name: accuracy
    weight: 0.6
    threshold: 0.8
    description: Correctness of trigger routing and output relevance
  - name: robustness
    weight: 0.4
    threshold: 0.7
    description: Consistent behavior across edge cases and negative triggers
graders:
  - type: behavior
    name: efficiency
    config:
      max_tokens: 50000
tasks:
  - "tasks/*.yaml"

Each task is its own file, which keeps the suite readable and lets you add a case per failure (exactly the eval-driven loop). A positive-trigger task looks like this:

id: positive-trigger-001
name: Build Technician Briefing Agent
inputs:
  prompt: Build a conversational assistant that briefs field technicians before
    each job with equipment history, contract terms, and IoT alerts
expected:
  output_contains: [briefing, technician]
graders:
  - type: trigger
    name: should-trigger
    config:
      skill_path: .github/skills/field-service/conversational-assistant
      mode: positive
  - type: text
    name: keyword-match
    config:
      regex_match: ["(?i)agent|equipment|contract"]

Grader types

Waza combines several grader kinds so you can assert on different aspects of behavior:

Negative-trigger tasks are as important as positive ones: a skill that fires on everything is as broken as one that never fires. The suite deliberately includes unrelated prompts and asserts the skill stays quiet.

Two tiers: skill evals and implementation evals

Skill-level evals prove routing and relevance. They don’t prove the generated code is any good. So a runner connects two tiers:

• Tier 1 — skill evals (waza check, waza run): validate that a skill produces correct routing and domain-relevant output.
• Tier 2 — implementation evals: validate that the generated code meets quality thresholds — extraction accuracy, classification, report quality, voice debriefing.

The runner orchestrates the end-to-end pipeline: discover implementation eval configs, load golden datasets and scorers, execute runs (mock or live), aggregate across implementations, and report a pass / warn / fail status.

@dataclass
class PipelineResult:
    eval_results: list[EvalResult] = field(default_factory=list)
    passed: int = 0
    warned: int = 0
    failed: int = 0

    @property
    def all_passed(self) -> bool:
        return self.failed == 0

That all_passed property is what a CI gate keys off: any failing eval fails the build.

Mock first, live when it matters

The mock executor runs the whole suite without API calls or cloud dependencies, which makes local iteration and CI fast and free. The copilot-sdk executor runs the same specs against real models for pre-deployment validation and cross-model comparison. Same YAML, two backends.

The day-to-day loop is three commands:

waza check   .github/skills/field-service/conversational-assistant   # validate structure/frontmatter
waza scaffold .github/skills/field-service/conversational-assistant  # generate an eval suite from SKILL.md
waza run     evals/conversational-assistant/eval.yaml --executor mock # run it

Because the same spec runs against multiple models, cross-model comparison is one flag away — and it’s how customers pick a model per deployment rather than by reputation:

Gating CI

The all_passed property maps straight onto a GitHub Actions gate — mock executor, no secrets, runs on every PR:

name: skill-evals
on: [pull_request]
jobs:
  waza:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: go install github.com/microsoft/waza/cmd/waza@latest
      - run: waza run evals/**/eval.yaml --executor mock --fail-on any

The trade-offs

• Waza is relatively new; specs may need updates as the tool evolves.
• The Go CLI is an extra dependency to install.
• Teams already invested in MLflow must adopt Waza or run parallel systems.

We accepted these because a standardized, scaffoldable, CI-friendly format across all skills is worth more than a bespoke harness that only one team understands.

Takeaways

• Declare evals as YAML, one task per file, scaffolded from SKILL.md.
• Grade triggers both ways — positive and negative — plus text and behavior.
• Separate skill evals from implementation evals, and gate CI on an aggregate pass/warn/fail.
• Mock for speed, live for truth.

#evaluation#waza#skills#ci#ai-agents