Evaluation

Microsoft Waza: Evaluation for Agent Skills, Triggers, and Tool Behavior

By Omer DemirAug 28, 2026⏱ 20 min read👁 — reads⧉ — copies

AI disclosure: This article was moderated by AI.

Microsoft Waza has a narrower target than a general LLM evaluation framework: the agent skill. It tests whether a SKILL.md file is selected for the right request, ignored for the wrong one, and followed correctly after selection. Waza can also evaluate VS Code custom agents in .agent.md files, though that support has important limitations covered later.

A response-level evaluator may tell you whether an answer is relevant. It usually cannot tell you whether the agent selected the intended skill, called required tools in order, changed the expected files, stayed within a token budget, or improved because the skill was available. Waza records and grades those execution signals.

The review uses the standalone Waza v0.38.7 release and evaluation schema 1.2 as they stood on August 28, 2026. Waza changes quickly, so reproducible suites should pin both the binary and schema.

The evaluation object

Waza’s execution path is:

SKILL.md or .agent.md
        |
        v
evaluation spec + task + isolated workspace
        |
        v
mock executor or Copilot SDK executor
        |
        v
output + files + transcript + tool events + usage
        |
        v
deterministic graders and optional LLM judge
        |
        v
results.json + reports + snapshots + telemetry

The unit under test can include:

• The skill description and its USE FOR and DO NOT USE FOR routing contract.
• The instructions injected from the body of SKILL.md.
• The final text returned by the agent.
• Files created or changed in the task workspace.
• Tools and skills invoked during the session.
• Tool arguments, ordering, count, duration, and token usage.
• Repeated-trial stability and improvement over a no-skill baseline.

This model fits coding agents, operational agents, and skill ecosystems. Waza is not designed around RAG-specific metrics such as context precision, answer faithfulness, or retrieval recall. A prompt judge or remote grader can approximate them, but they are outside its main abstraction.

Installation and dependencies

Prebuilt binary

The recommended installation downloads a checksum-verified binary for macOS, Linux, or Windows on amd64 or arm64:

curl -fsSL https://raw.githubusercontent.com/microsoft/waza/main/install.sh | bash
waza --version

For native Windows PowerShell:

irm https://raw.githubusercontent.com/microsoft/waza/main/install.ps1 | iex
waza --version

For reproducible CI, download the v0.38.7 asset and checksum directly from the release rather than following main through the latest-version installer.

The prebuilt CLI does not require a local Go toolchain. Waza bundles the GitHub Copilot CLI used by the copilot-sdk executor and extracts it to the user’s cache on first use. Teams that manage the binary themselves can override its location with COPILOT_CLI_PATH.

Building from source

Source installation has a less ordinary dependency chain:

git clone --branch v0.38.7 https://github.com/microsoft/waza.git
cd waza
git lfs install
git lfs pull
go build -o waza ./cmd/waza
./waza --version

The current source tree requires Go 1.26 or later, Git, and Git LFS. go install github.com/microsoft/waza/... is explicitly unsupported because the build embeds Copilot CLI artifacts stored through Git LFS. That makes source builds less convenient than those of a conventional Go CLI, but it does not affect users of the prebuilt binary.

The legacy Python implementation ended at v0.3.2. Current releases are Go binaries. Do not combine older Python installation instructions or the historical schema with schema 1.2 examples.

Runtime services

Dependencies depend on the execution mode:

Waza also performs a cached, non-blocking update check. Disable it in sealed CI environments with:

export WAZA_NO_UPDATE_CHECK=1

A complete skill-selection evaluation

The following implementation evaluates a deployment skill. Positive prompts must invoke it, and negative prompts must not. Once selected, the skill must inspect the repository, write infrastructure, and validate the result without using a destructive shell command.

The skill under test

Create skills/azure-deploy-lite/SKILL.md:

---
name: azure-deploy-lite
description: >
  Prepare deployment-ready Bicep for a small Azure web application.
  USE FOR: create Bicep, prepare Azure infrastructure, validate a web app deployment.
  DO NOT USE FOR: investigate an existing production incident, analyze an Azure bill,
  or deploy resources immediately without review.
license: MIT
metadata:
  version: 1.0.0
---

## Workflow

1. Inspect the application and its deployment requirements.
2. Create or update Bicep under `infra/`.
3. Run a Bicep build or equivalent static validation.
4. Report changed files, validation results, and unresolved assumptions.

Never execute a live deployment. Never delete existing infrastructure files.

The description is part of the routing contract inspected by trigger evaluation and specification verification. It is not decorative metadata.

Evaluation specification

Create evals/azure-deploy-lite/eval.yaml:

name: azure-deploy-lite-eval
description: Evaluate routing and behavior of the deployment skill
skill: azure-deploy-lite
schemaVersion: "1.2"
version: "1.0"

config:
  executor: copilot-sdk
  model: claude-sonnet-4.6
  judge_model: gpt-4o-mini
  inject_skill_body: false
  timeout_seconds: 300
  trials_per_task: 3
  max_attempts: 1
  parallel: false

graders:
  - type: behavior
    name: bounded_execution
    config:
      max_tool_calls: 20
      max_tokens: 50000
      max_duration_ms: 300000

tasks:
  - id: prepare-bicep
    name: Prepare Bicep for a web application
    inputs:
      prompt: >
        Inspect this web application and prepare reviewable Bicep for deployment
        to Azure. Validate the templates, but do not deploy anything.
      context:
        fixture: fixtures/web-app
    expected:
      graders:
        - type: skill_invocation
          name: selected_deployment_skill
          config:
            required_skills: [azure-deploy-lite]
            mode: exact_match
            allow_extra: false
        - type: file
          name: created_infrastructure
          config:
            must_exist: [infra/main.bicep]
        - type: action_sequence
          name: inspected_edited_validated
          config:
            matching_mode: in_order_match
            expected_actions: [grep_search, apply_patch, run_in_terminal]
        - type: tool_calls
          name: no_destructive_shell
          config:
            forbidden_tools: [rm, sudo]

  - id: production-incident
    name: Do not select for production diagnostics
    inputs:
      prompt: >
        Our existing App Service is returning HTTP 503 in production.
        Investigate the incident and identify the root cause.
    expected:
      graders:
        - type: skill_invocation
          name: rejected_out_of_scope_request
          config:
            forbidden_skills: [azure-deploy-lite]
            allow_extra: true

  - id: azure-cost-analysis
    name: Do not select for cost analysis
    inputs:
      prompt: >
        Analyze last month's Azure bill and identify the three largest savings.
    expected:
      graders:
        - type: skill_invocation
          name: rejected_cost_request
          config:
            forbidden_skills: [azure-deploy-lite]
            allow_extra: true

For a routing test, inject_skill_body: false is essential. Waza keeps the skill discoverable by name and description without forcing its full body into the system prompt. Injecting the body before selection would test instruction following after forced activation rather than whether the agent chose the skill.

Keep task fixtures small and deterministic:

evals/azure-deploy-lite/
├── eval.yaml
└── fixtures/
    └── web-app/
        ├── package.json
        ├── src/
        │   └── server.ts
        └── deployment-requirements.md

Run the suite and retain machine-readable evidence:

waza spec verify \
  skills/azure-deploy-lite \
  evals/azure-deploy-lite/eval.yaml \
  --fail \
  --format github-actions

waza run evals/azure-deploy-lite/eval.yaml \
  --context-dir evals/azure-deploy-lite/fixtures \
  --output results.json \
  --transcript-dir transcripts \
  --session-log \
  --session-dir sessions \
  --reporter junit:results.xml \
  --snapshot snapshots \
  --verbose

The suite combines several evidence layers:

• skill_invocation tests routing behavior.
• file tests the post-execution workspace.
• action_sequence tests an ordered trajectory while allowing intermediate calls.
• tool_calls rejects dangerous tools.
• behavior imposes run-wide resource budgets.

Tool names in a trajectory grader must exactly match those emitted by the executor. Calibrate them from a captured transcript before making them a hard gate. A model may produce the correct file through another valid tool path, so reserve exact trajectories for required safety or process controls.

Trigger and anti-trigger testing

Selection quality has two distinct error classes:

For skills with expensive or privileged tools, false positives are often more dangerous than false negatives. Test USE FOR and DO NOT USE FOR language, near-boundary prompts, ambiguous prompts, and lexical collisions. An Azure deployment skill, for example, should not activate merely because an incident or cost question contains the word “Azure.”

Waza can scaffold positive and negative trigger cases from the skill description:

waza new eval azure-deploy-lite
waza suggest skills/azure-deploy-lite --focus triggers --count 10 --dry-run
waza suggest skills/azure-deploy-lite --focus negative-triggers --count 10 --dry-run

Generated cases are hypotheses rather than ground truth. Review them before --apply, then add domain-specific confounders that cannot be inferred from the skill description alone.

waza spec verify provides a second, deterministic check. It parses promises, use cases, exclusions, and parameter blocks from SKILL.md, maps them to task IDs, and reports uncovered requirements. The optional --semantic mode adds an LLM-assisted matching pass, which improves fuzzy matching but introduces judge cost and variance.

Deterministic and model-based graders

Waza’s current grader surface is broader than the short list in older releases. The principal categories are:

Every grader returns a score from 0.0 to 1.0, a pass flag, feedback, and structured details. Multiple graders run independently, and a task passes only if all of them pass. Weights affect the composite score but do not turn a failed required grader into a passing task:

This policy prevents a high style score from compensating for a failed security gate.

LLM-as-judge

Use a prompt grader only for properties that are difficult to encode deterministically:

- type: prompt
  name: deployment_plan_quality
  weight: 1.0
  config:
    rubric: instruction-following
    model: gpt-4o-mini

The judge calls one of two Waza-provided tools, set_waza_grade_pass or set_waza_grade_fail, to produce a binary result. A custom inline prompt or versioned local rubric can replace the built-in rubric. continue_session: true allows the judge to inspect the execution session and workspace, but it also gives the judge more context and increases both cost and coupling.

Prompt grading is flexible, with three significant costs:

1. It adds at least one model interaction per graded run.
2. Its verdict can drift when the judge model changes.
3. Agent output can contain content that attempts to influence the judge.

Use file, diff, program, tool_calls, or json_schema for hard requirements. Pin the judge model and rubric for subjective checks that remain.

Remote graders and supply-chain controls

Schema 1.2 supports Go-module-style grader references:

graders:
  - ref: github.com/waza-evals/fact#factuality@v1.0.0
    name: factuality_strict
    config:
      threshold: 0.9

Resolve references before running:

waza get evals/azure-deploy-lite/eval.yaml

Waza records the resolved commit and SHA-256 content digest in waza.lock. If the lock or cached digest is missing, runs fail instead of silently resolving another version. Config-only presets expand to built-in graders. Remote executable graders require explicit trust through interactive confirmation or --allow-exec.

The registry should still be treated as emerging infrastructure. As of this review, waza registry search returns bundled sample metadata while live index integration is pending.

Repetition, retries, and baselines

One successful run is weak evidence for an agent. trials_per_task repeats each task independently, while max_attempts retries failed execution within a trial and records the number of attempts required.

Retries are not additional independent trials. A task that passes after three attempts differs operationally from one that passes immediately. Report first-attempt success, eventual success, and the attempt distribution separately.

For $k$ independent trials with per-trial success probability $p$:

pass@k asks whether at least one run succeeds. pass^k asks whether every run succeeds. The first may matter for coding assistants; unattended deployment and security workflows should put much more weight on the second.

Waza’s baseline mode directly tests whether the skill helps:

waza run evals/azure-deploy-lite/eval.yaml \
  --baseline \
  --trials 5 \
  --output baseline-results.json

Each task runs once without the skill and once with it, after which Waza computes the improvement. Absolute scoring misses an important failure mode: a skill can produce acceptable output while making the base agent slower, less accurate, or more verbose.

Output: what an actual run looks like

The following output was recorded in the repository, not observed on the machine used to write this article. It comes from Waza’s current custom-agent guide and shows the task-level reporting contract:

Running: security-reviewer eval
Task: injection-detection ... PASS
Task: crypto-review ... PASS
Task: data-handling ... PARTIAL (4/5 validators)
Task: infrastructure-scan ... PASS

Results saved to results.json

The guide says these tasks cover injection, cryptography, sensitive-data handling, and infrastructure scanning. The PARTIAL result shows that Waza retains individual validator outcomes instead of collapsing all evidence into one opaque judge score.

This is not a locally reproduced benchmark or evidence that a particular model will obtain the same scores. Model, fixtures, credentials, and repository state all affect live runs. For the implementation in this article, run the commands in the previous section and archive results.json, results.xml, transcripts, session logs, and snapshots together.

Waza’s run command uses CI-friendly exit codes:

The adversarial command has a separate contract: 0 for pass, 2 for an unsafe outcome under fail policy, and 3 for configuration failure.

Artifacts, replay, and observability

Waza can preserve substantially more evidence than terminal output:

• Schema 1.2 JSON results, including normalized tool events and snapshot paths.
• JUnit XML for CI test interfaces.
• Per-task transcript JSON.
• Session event logs in NDJSON.
• Timestamped structured output directories.
• Self-contained snapshots with prompt, fixture digests, ordered tool events, redacted grader outcomes, and allow-listed environment data.
• OpenTelemetry spans sent to stdout, a file, or an OTLP endpoint.
• Local dashboard views for task status, trial distributions, model comparison, and trends.
• Optional Azure Blob Storage for shared run history.

Execution and grading can be separated:

waza run eval.yaml --skip-graders --output ungraded.json
waza grade eval.yaml --results ungraded.json --output regraded.json
waza compare ungraded.json regraded.json

Offline re-grading is valuable when deterministic graders or rubrics change. File-based graders still need the original agent workspace, supplied through --workspace; a JSON result cannot reconstruct arbitrary file state by itself.

Snapshots support consistency checks without another model call:

waza replay snapshots/prepare-bicep-run1.json
waza replay snapshots/run1.json --bisect snapshots/run2.json --json

Default replay checks the internal consistency of recorded evidence. Live replay is documented as planned, so snapshot replay is not a fresh end-to-end model reproduction.

CI implementation

A minimal GitHub Actions job can pin the release, disable update checks, execute the suite, and upload evidence:

name: Skill evaluation

on:
  pull_request:
    paths:
      - "skills/**"
      - "evals/**"

jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    env:
      WAZA_NO_UPDATE_CHECK: "1"
      GITHUB_TOKEN: ${{ secrets.COPILOT_GITHUB_TOKEN }}
    steps:
      - uses: actions/checkout@v4

      - name: Install Waza v0.38.7
        run: |
          curl -fL -o waza.tar.gz \
            https://github.com/microsoft/waza/releases/download/v0.38.7/waza-linux-amd64.tar.gz
          tar -xzf waza.tar.gz
          sudo install waza /usr/local/bin/waza

      - name: Verify specification coverage
        run: |
          waza spec verify skills/azure-deploy-lite \
            evals/azure-deploy-lite/eval.yaml \
            --fail --format github-actions

      - name: Run evaluation
        run: |
          waza run evals/azure-deploy-lite/eval.yaml \
            --output results.json \
            --reporter junit:results.xml \
            --transcript-dir transcripts \
            --snapshot snapshots

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: waza-evidence
          path: |
            results.json
            results.xml
            transcripts/
            snapshots/

Before adopting this example, confirm release asset filenames and checksums against the pinned release. For hermetic pull-request checks without model credentials, use the mock executor and MCP mock fixtures. Keep live-model suites in protected workflows or scheduled regression runs.

Model providers and isolation

The copilot-sdk executor uses GitHub Copilot by default. Waza can route the SDK to a compatible custom provider:

COPILOT_BASE_URL=https://example-resource.openai.azure.com \
COPILOT_PROVIDER=azure \
COPILOT_WIRE_API=responses \
COPILOT_API_KEY="$AZURE_OPENAI_API_KEY" \
waza run eval.yaml --executor copilot-sdk --model deployment-name

Provider settings can also use the COPILOT_PROVIDER_* aliases. When a custom provider is active, result metadata records usage.provider: "custom" and a sanitized provider host, not the complete URL.

Each task runs in an isolated temporary workspace. A fixture file or directory can be copied into it. A task can also materialize a detached worktree from a local Git repository. Current repository resources do not clone over HTTPS or SSH, initialize submodules, materialize Git LFS content, or automatically infer the source repository. Those constraints matter for realistic coding-agent benchmarks.

MCP mocks provide deterministic local stdio tools for Copilot SDK tests. Exact argument matching, JSON Schema matching, and regex matching let tests fail loudly on an unknown or unmodeled call instead of touching a live service.

Privacy and telemetry

Local execution does not automatically mean no data leaves the machine.

Data can cross a network boundary through:

• Copilot or a configured custom model provider during agent execution.
• A second model provider call for each prompt grader.
• An OTLP exporter.
• Configured Azure Blob result storage.
• Waza’s optional update check.
• Remote grader resolution during waza get.

OpenTelemetry export is off by default. When enabled, prompt content, tool arguments, tool results, and completions are represented by SHA-256 digest and length by default. --otel-include-payloads explicitly opts into raw content. Snapshot environment capture is default-deny and requires --snapshot-env-allow; built-in and optional YAML redaction policies protect snapshot output.

These defaults reduce exposure, but transcripts and results.json can still contain prompts, model outputs, file paths, tool arguments, or generated content. Treat evaluation artifacts as sensitive and define retention, access, and redaction policies before uploading them.

Cost model

The Waza binary is free under the MIT license, but running evaluations can still incur costs.

The direct cost centers are:

1. Agent model execution.
2. Prompt-grader model execution.
3. Responder-model execution for interactive skills.
4. CI compute and artifact retention.
5. Optional OpenTelemetry ingestion and retention.
6. Optional Azure Blob transactions, capacity, and egress.

For $N$ tasks, $T$ trials, up to $A$ attempts, $M$ execution models, and baseline flag $B \in \{0,1\}$, the maximum number of agent executions is approximately:

If every execution reaches $J$ prompt graders, the judge-call upper bound is approximately:

Interactive responders add one request per follow-up turn, up to each task’s max_followups. Token-based providers then produce an approximate model charge:

where $I_r$ and $O_r$ are request input and output tokens, and $P_{in,r}$ and $P_{out,r}$ are the provider’s per-million-token prices. GitHub Copilot plans can instead meter model usage through plan-specific premium requests. Waza reports usage, but it cannot determine every organization’s negotiated price, included quota, or downstream observability charge.

Control cost with deterministic graders, mock plus MCP fixtures in fast CI, small calibrated fixture repositories, cached deterministic runs, and task filters during development. Schedule live multi-trial model evaluation separately. Waza disables caching automatically when behavior or prompt graders make reuse unsafe.

Custom .agent.md support

Waza can evaluate a .agent.md file by injecting its Markdown body as the system message. When the frontmatter declares tools, Waza adds an implicit tool_constraint grader. Support remains incomplete in v0.38.7:

• model in agent frontmatter is informational; config.model controls the eval run.
• handoffs, mcp-servers, and coordinating agents are parsed but not evaluated.
• Handoff behavior cannot yet be tested directly.
• When a directory contains multiple .agent.md files, only the first is discovered.
• When SKILL.md and .agent.md coexist in one directory, SKILL.md takes priority and the agent is ignored.

Use separate directories and explicit evaluation targets to avoid discovery ambiguity. Do not claim multi-agent handoff coverage merely because Waza accepts the frontmatter fields.

Ownership, license, and community

Waza is hosted in the microsoft GitHub organization under the MIT license. At the time of review, the public repository showed approximately 1,300 stars, 79 forks, 15 listed contributors, 45 releases, four open issues, and three open pull requests. The v0.38.7 tag is attached to a verified commit.

Those figures suggest healthy early adoption for a specialized tool. They do not amount to a support contract or long-term API stability. The pre-1.0 version, rapid release cadence, pending registry integration, and expanding schema all point to a product surface that may still change substantially.

Microsoft repository ownership provides a security policy, code owners, automated dependency updates, and visible engineering activity. It does not by itself promise an Azure or Microsoft commercial support SLA. Production adopters should pin versions, archive binaries and checksums, validate schema changes, and establish an internal owner for evaluation infrastructure.

Strengths

• Purpose-built evaluation of SKILL.md selection and execution behavior.
• Positive and negative routing tests, specification coverage, and baseline-without-skill comparisons.
• Rich deterministic graders across text, files, diffs, JSON, programs, tools, actions, and skill invocations.
• Isolated fixture workspaces, local Git worktrees, and deterministic MCP mocks.
• Repeated trials, retries, multi-model runs, snapshots, offline re-grading, and comparison.
• CI-oriented exit codes, JUnit, JSON, transcripts, NDJSON, OpenTelemetry, and a dashboard.
• Supply-chain pinning for remote grader references through commits and content digests.
• MIT license, cross-platform binaries, and active development under Microsoft.

Limitations

• It is a specialist skill and agent-behavior evaluator, not a complete RAG or model-quality metric library.
• Live execution is coupled to the Copilot SDK route, even when using a custom compatible provider.
• Source builds require Go 1.26+, Git LFS, and a repository clone; go install is unavailable.
• Prompt graders add variance, latency, provider dependency, and cost.
• Custom-agent handoffs, MCP declarations, and agent coordination are parsed but not evaluated.
• Local Git worktrees are supported, but remote clone strategies, submodules, and LFS materialization are not.
• The public grader registry remains incomplete.
• Pre-1.0 releases and schema evolution require explicit version pinning and migration discipline.

Good fit

Choose Waza when you need to answer questions such as:

• Does this skill activate for the intended prompts and remain silent for adjacent requests?
• Does the skill improve the base agent rather than merely produce an acceptable answer?
• Did the coding agent inspect, edit, validate, and report in the required order?
• Did it call a forbidden tool, exceed a token budget, or modify an unexpected file?
• Is every promise in SKILL.md represented by an evaluation task?
• Can the same skill be compared across Copilot-accessible models with auditable artifacts?

Poor fit

Use another framework, or combine Waza with one, when the primary need is:

• RAG retrieval metrics, embedding evaluation, context precision, or faithfulness datasets.
• High-volume model benchmarking independent of an agent skill.
• Native Python test integration and custom metric classes.
• Browser, mobile, voice, or multimodal endpoint simulation.
• Fully evaluated multi-agent handoffs and MCP topology.
• A commercially supported evaluation service with a contractual SLA.

Verdict

Waza is the strongest specialist in this series for testing whether an agent skill is selected correctly and changes observable behavior as intended. Anti-trigger tests, skill-invocation evidence, tool trajectories, workspace graders, no-skill baselines, and CI artifacts cover failure modes that response-only evaluation misses.

That specialization also defines its limit. Waza should not replace a general LLM/RAG evaluator, a production observability platform, or a domain simulator. For teams that publish SKILL.md files or operate coding-agent workflows, the narrow scope is useful. Pin v0.38.7, prefer deterministic gates, and use prompt judges only where necessary. Custom-agent coordination and the public grader registry should be treated as developing features, not finished foundations.

Sources

• Waza repository and README
• Waza v0.38.7 release
• Waza documentation
• Grader reference
• Evaluating custom agents

#AI Evaluation#AI Agents#Microsoft Waza#GitHub Copilot#Agent Skills