Evaluation

Skill Evaluation with Microsoft Waza

By Omer DemirMay 5, 2026⏱ 5 min read👁 — reads⧉ — copies

AI disclosure: This article was moderated by AI.

Before a reusable AI skill reaches a customer engagement, we test whether it activates for the right prompts, remains inactive for unrelated ones, and produces relevant output. We standardized those checks with Microsoft Waza, a YAML-first evaluation framework, and run them through a two-tier CI pipeline.

Why Waza over a custom framework

An earlier project used a custom MLflow-based evaluation harness. Although it worked, the harness required substantial scorer code coupled to the Databricks and MLflow stack. Waza provides a standard, portable format with the following capabilities:

• YAML evaluation specifications define graders and task patterns.
• The mock execution engine supports local and CI tests, while copilot-sdk runs production evaluations.
• The scaffold command generates evaluation suites from SKILL.md files.
• The check command validates skill structure and frontmatter compliance.
• Cross-model comparisons benchmark a skill across GPT-4o, Claude Sonnet, and other models.
• GitHub Actions support runs evaluations in CI.

Every skill can use the same evaluation format, and scaffolding it from the skill’s own SKILL.md keeps setup small.

The eval spec

A Waza evaluation is a YAML file that declares the skill under test along with its configuration, metrics, graders, and tasks:

name: conversational-assistant-eval
description: Tests that the skill triggers on technician-assistant requests,
  stays silent on unrelated prompts, and produces domain-relevant output.
skill: conversational-assistant
config:
  trials_per_task: 1
  executor: mock
  model: claude-sonnet-4.6
metrics:
  - name: accuracy
    weight: 0.6
    threshold: 0.8
    description: Correctness of trigger routing and output relevance
  - name: robustness
    weight: 0.4
    threshold: 0.7
    description: Consistent behavior across edge cases and negative triggers
graders:
  - type: behavior
    name: efficiency
    config:
      max_tokens: 50000
tasks:
  - "tasks/*.yaml"

Each task has its own file. This keeps the suite readable and supports adding one case for each failure, following the eval-driven loop. A positive-trigger task looks like this:

id: positive-trigger-001
name: Build Technician Briefing Agent
inputs:
  prompt: Build a conversational assistant that briefs field technicians before
    each job with equipment history, contract terms, and IoT alerts
expected:
  output_contains: [briefing, technician]
graders:
  - type: trigger
    name: should-trigger
    config:
      skill_path: .github/skills/field-service/conversational-assistant
      mode: positive
  - type: text
    name: keyword-match
    config:
      regex_match: ["(?i)agent|equipment|contract"]

Grader types

Waza combines several grader types to check different aspects of behavior:

Negative-trigger tasks matter as much as positive ones. A skill that activates for every prompt is as broken as one that never activates. The suite includes unrelated prompts and verifies that the skill remains inactive.

Two tiers: skill evals and implementation evals

Skill-level evaluations cover routing and relevance, but they do not measure the quality of generated code. The runner therefore connects two tiers:

• Tier 1 runs skill evaluations through waza check and waza run to validate routing and domain-relevant output.
• Tier 2 runs implementation evaluations to check generated code against quality thresholds for extraction accuracy, classification, report quality, and voice debriefing.

The runner discovers implementation evaluation configurations, loads golden datasets and scorers, executes mock or live runs, aggregates results across implementations, and reports a pass, warn, or fail status.

@dataclass
class PipelineResult:
    eval_results: list[EvalResult] = field(default_factory=list)
    passed: int = 0
    warned: int = 0
    failed: int = 0

    @property
    def all_passed(self) -> bool:
        return self.failed == 0

The CI gate uses all_passed, so any failed evaluation fails the build.

Mock first, live when it matters

The mock executor runs the complete suite without API calls or cloud dependencies. This keeps local iteration and CI fast and free. For pre-deployment validation and cross-model comparisons, the copilot-sdk executor runs the same specifications against real models. Both backends use the same YAML.

The routine workflow uses three commands:

waza check   .github/skills/field-service/conversational-assistant   # validate structure/frontmatter
waza scaffold .github/skills/field-service/conversational-assistant  # generate an eval suite from SKILL.md
waza run     evals/conversational-assistant/eval.yaml --executor mock # run it

Because one specification can run against several models, a flag enables cross-model comparison. Customers can then select a model for each deployment from measured results rather than reputation:

Gating CI

The all_passed property maps directly to a GitHub Actions gate. It uses the mock executor, requires no secrets, and runs on every pull request:

name: skill-evals
on: [pull_request]
jobs:
  waza:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: go install github.com/microsoft/waza/cmd/waza@latest
      - run: waza run evals/**/eval.yaml --executor mock --fail-on any

The trade-offs

• Waza is relatively new, so specifications may need updates as the tool evolves.
• The Go CLI adds a dependency to the toolchain.
• Teams already invested in MLflow must adopt Waza or run parallel systems.

We accepted those costs because a standardized format that supports scaffolding and CI can be shared across all skills. That provides more value than a custom harness understood by only one team.

Evaluation model

Each evaluation is declared in YAML with one task per file and can be scaffolded from SKILL.md. The suite tests both positive and negative triggers, along with text and behavior. Skill evaluations remain separate from implementation evaluations, and CI gates their aggregate pass, warn, or fail result. Mock runs support fast iteration; live runs provide the production-model result.

#evaluation#waza#skills#ci#ai-agents