Evaluation

Microsoft Vally: Declarative Trajectory Evaluation for AI Agents

By Omer DemirAug 28, 2026⏱ 15 min read👁 — reads⧉ — copies

AI disclosure: This article was moderated by AI.

Vally evaluates an agent by running an experiment, not by scoring a single response. It prepares an isolated workspace, gives the agent a stimulus, captures the trajectory and filesystem changes, and grades that evidence. It also saves enough detail to grade the run again without paying for another agent execution.

This model suits coding agents, tool-using agents, skills, and multi-turn workflows. It is heavier than necessary for a question-answer dataset whose rows already contain input-output pairs. For that case, DeepEval’s LLMTestCase is the smaller abstraction. Vally earns its place when correctness depends on what the agent did: the tools it called, the files it changed, whether it looped, whether it stayed within budget, and whether the final workspace works.

This assessment covers Vally 0.14.0 using public information available on August 28, 2026. Vally is pre-1.0. Microsoft publishes its documentation and npm packages, but the public source repository returned 404 during this review. That unresolved support boundary affects adoption.

Execution and evidence model

Vally’s pipeline is:

Stimulus -> Executor -> Trajectory -> Graders -> Score -> Report

Each layer has a distinct responsibility:

The trajectory is the defining boundary. Vally can evaluate both the final workspace and the path used to produce it. One grader can pass because hello.json exists, while another fails because the agent used a forbidden tool or repeated the same action six times.

Vally can also consume ATIF trajectories for offline and oracle workflows. A golden trajectory can represent expected tool calls and observations without running an agent, while a golden patch can represent the expected final file state. Those are different references: a patch proves what the workspace should become; a trajectory proves how a passing agent may behave.

Installation and dependency surface

Vally requires Node.js 22.12.0 or newer. Install the CLI globally, or pin it in a project for reproducible CI:

npm install --global @microsoft/vally-cli@0.14.0
vally --version

npm install --save-dev @microsoft/vally-cli@0.14.0
npx vally --version

Programmatic integrations and custom graders use the core package:

npm install --save-dev @microsoft/vally@0.14.0

The stable package set at the review snapshot was:

All three reported version 0.14.0; 0.15.0-next.2 was available as a prerelease. Pin the stable version because pre-1.0 minor releases may change schemas or plugin contracts.

The Node dependency tree is larger than that of a standalone assertion library. Vally handles the workspace lifecycle, Git integration, Copilot SDK execution, OpenTelemetry, result persistence, CLI rendering, YAML validation, and plugin loading. A basic local run does not require Python, Go, Docker, or a database. Your own stimulus environment commands may still require Git, package managers, language toolchains, or containers.

A deterministic coding-agent evaluation

The generated quickstart asks an agent to create a JSON file and checks facts in the final workspace. The following spec adds trajectory limits, so it tests both outcome and behavior:

name: greeting-file
description: Create and validate a small structured artifact
type: capability

defaults:
  runs: 3
  timeout: 2m
  model: gpt-5.5
  executor: copilot-sdk

stimuli:
  - name: greeting-json
    prompt: |
      Create hello.json with exactly two fields:
      "message", containing a friendly greeting, and
      "language", containing the value "en".
    graders:
      - type: file-exists
        config:
          path: hello.json
      - type: file-matches
        config:
          path: hello.json
          pattern: '"language"\s*:\s*"en"'
      - type: run-command
        config:
          command: node -e "JSON.parse(require('fs').readFileSync('hello.json'))"
          expected_exit_code: 0
      - type: tool-calls
        config:
          disallowed: [delete_file]
      - type: max-repeat
        config:
          max: 3
    constraints:
      max_turns: 8
      max_tokens: 4000

scoring:
  threshold: 1.0

Validate before spending model tokens:

VALLY_TELEMETRY_OPTOUT=1 vally lint --eval-spec eval.yaml --strict

Then execute and preserve machine-readable results:

VALLY_TELEMETRY_OPTOUT=1 vally eval \
  --eval-spec eval.yaml \
  --output-dir ./vally-results \
  --junit \
  --verbose

Once the trajectory and workspace exist, the built-in graders above are deterministic. They do not call a judge model. The agent under test still consumes model tokens unless you use a mock executor.

Observed output and what it proves

During this review, I ran a credential-free local smoke test with Vally 0.14.0 and its mock executor. The generated hello spec validated, but the mock agent did not create hello.json:

0.14.0
✔ evals/hello/eval.yaml is valid

Metrics
Tokens      0
Turns       0
Tool calls  0

Graders (0/3)
✘ file-exists hello.json
✘ file-matches hello.json /"language"\s*:.../
✘ file-matches hello.json /"message"\s*:.../

Summary
hello  score: 0.0%

The command exited successfully only because I supplied --threshold 0 to inspect the harness. That override suppresses verdict failures, but not execution or tooling errors. It is useful while building a benchmark and does not belong in a release gate.

The version, lint success, zero trajectory metrics, and three failed graders were observed locally on August 28, 2026. I shortened the formatting to remove terminal decoration. A mock run says nothing about production agent quality. It verifies the CLI, schema validation, grader dispatch, and threshold behavior, but not Copilot SDK execution.

The documentation’s successful quickstart is an official example, not a measurement from this review. It reports 1,204 tokens, two turns, one tool call, three passing graders, and a 100% score. The contrast shows that deterministic graders require real evidence. They do not invent success when an executor produces no file or trajectory.

Grader taxonomy

Vally 0.14.0 documents 34 built-in graders. Only prompt and panel require judge-model calls. The rest derive results from recorded evidence or execute local checks.

This taxonomy suggests a practical order. Prove file state, process exit codes, tool contracts, and budgets first. Add a model judge only for qualities that cannot be expressed as programmatic facts.

Add an LLM judge deliberately

The prompt grader reads a rubric and selected evidence. This example gives the judge the final repository snapshot and cumulative diff rather than limiting it to the assistant’s prose:

defaults:
  judge_model: gpt-5.5

stimuli:
  - name: refactor-parser
    prompt: Refactor the parser without changing public behavior.
    rubric:
      - Public function signatures remain compatible
      - Error messages retain actionable context
      - The change does not introduce unrelated refactoring
    graders:
      - type: run-command
        config:
          command: npm test
          expected_exit_code: 0
      - type: prompt
        config:
          prompt: Evaluate the implementation against every rubric item.
          scoring: scale_1_5
          evidence: [trajectory, diff, repo]

Vally omits sensitive file contents such as .env files and private keys from repo evidence. Trajectory logs may still contain secrets echoed by the agent or its tools, so redaction at one evidence boundary does not sanitize every artifact.

Write a custom grader in TypeScript

Custom graders implement the public Grader interface. A minimal policy grader can inspect the normalized trajectory without another model call:

import type {
  Grader,
  GraderInput,
  GraderResult,
} from '@microsoft/vally/graders';

export const noForcePush: Grader = {
  metadata: {
    name: 'no-force-push',
    description: 'Rejects destructive force-push commands',
  },
  async grade(input: GraderInput): Promise<GraderResult> {
    const serialized = JSON.stringify(input.trajectory);
    const passed = !/git push[^\n]*(--force|-f)\b/i.test(serialized);

    return {
      score: passed ? 1 : 0,
      passed,
      reason: passed
        ? 'No force-push command was observed.'
        : 'The trajectory contains a force-push command.',
    };
  },
};

Register the grader through a local or npm plugin and load it with --grader-plugin. Treat grader plugins as executable code with the same host access as the Vally process.

Re-grading, oracle checks, and experiments

Agent execution is usually the most expensive and variable part. Vally separates capture from grading:

vally eval --eval-spec eval.yaml --skip-grade --output jsonl > outcomes.jsonl
vally grade --eval-spec eval.yaml < outcomes.jsonl

This allows rubric and threshold changes without rerunning the agent. It cannot add evidence to the original trajectory. If the executor failed to capture tool results or diffs, an offline grader cannot recover them.

vally oracle validates the harness against a golden patch, golden trajectory, or custom metrics. Run it before publishing an evaluation. If a known-good solution fails, the benchmark is measuring a harness defect. Add a negative control where possible so permissive graders do not pass every candidate.

Experiments vary models, prompts, executors, or environment fields across declared variants. vally compare evaluates baseline and treatment runs, while the analytics server stores run data in SQLite for local exploration. These features make Vally closer to an evaluation workbench than a single test runner.

Multi-trial reliability

For $k \ge 2$, Vally records two different questions:

where $X_i$ is one when trial $i$ passes. pass@k asks whether the agent succeeds at least once, while pass^k asks whether it succeeds every time. Four passes in five trials can look strong under pass@5 and still fail the reliability requirement expressed by pass^5.

Vally also reports flakiness. Trials for one stimulus run sequentially, while --workers controls parallelism across stimuli. Non-Git workspaces may be prepared once and restored with copy-on-write clones; Git-backed environments are materialized fresh for each trial.

Providers and execution boundaries

The default executor is copilot-sdk. In CI it can authenticate with a fine-grained GitHub token that has Copilot Requests permission. The normal workflow token does not grant that permission.

The executor can instead use an OpenAI-compatible provider block:

defaults:
  model: gpt-4o
  executor:
    name: copilot-sdk
    config:
      provider:
        type: azure
        baseUrl: https://example.openai.azure.com/openai/v1/
        wireApi: responses
        apiKeyEnv: AZURE_OPENAI_API_KEY
        modelId: gpt-4o
        wireModel: agent-deployment
        azure:
          apiVersion: '2024-10-21'

Provider types cover OpenAI, Azure, and Anthropic wire protocols. Ollama, vLLM, LiteLLM, Foundry Local, and similar servers use the OpenAI-compatible path. The agent provider and judge provider are independent, so a local agent can be judged remotely or the reverse.

Vally stores credential environment-variable names rather than literal values in declarative provider blocks. It masks custom headers and resolved credentials in persisted metadata, but a tool or model can still echo a secret into a trajectory. Use short-lived credentials and inspect the artifact retention policy.

Cost model

Vally is MIT-licensed according to the published package metadata and has no listed framework fee. Operational cost has four layers:

For $S$ stimuli, $M$ agent models, and $R$ trials, the agent execution floor is $S \times M \times R$. If each result has one prompt judge and an $N$-member panel, the judge-evaluation floor is roughly $S \times M \times R \times (1 + N)$. Retries, multi-turn sessions, and provider-internal calls add to that floor.

Use vally grade while tuning graders to avoid paying for agent execution again. Static graders fit PR gates. Panel judges and broad multi-model matrices belong in scheduled runs unless their latency and cost are explicitly budgeted.

Privacy, telemetry, and artifacts

Vally usage telemetry is enabled by default. It includes command name, version, outcome, duration, aggregate counts, coarse runtime information, and a persistent pseudonymous device identifier. It does not include prompts, datasets, paths, arguments, model names, scores, or grader results.

Disable the stream with either variable:

export VALLY_TELEMETRY_OPTOUT=1
# or
export DO_NOT_TRACK=1

Evaluation tracing is separate. Detailed OpenTelemetry spans remain in the local otel-spans.jsonl by default. Passing --otlp-endpoint sends them to the selected collector. Agent and judge providers receive the prompts and evidence required for their calls.

The run directory can contain results.jsonl, Markdown reports, JUnit XML, metadata.json, events.jsonl, copied artifacts, diffs, prompts, model output, and tool payloads. Treat the result tree as production data, even when usage telemetry is disabled.

CI integration

Keep static validation and paid execution as distinct steps:

name: agent-evals
on:
  pull_request:
    paths:
      - '**/SKILL.md'
      - '**/eval.yaml'
      - 'src/**'

jobs:
  vally:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '22'
          cache: npm
      - run: npm ci
      - run: npx vally lint --eval-spec eval.yaml --strict
      - run: npx vally eval --eval-spec eval.yaml --junit --output-dir results
        env:
          VALLY_TELEMETRY_OPTOUT: '1'
          COPILOT_GITHUB_TOKEN: ${{ secrets.COPILOT_CLI_TOKEN }}
      - if: always()
        uses: actions/upload-artifact@v4
        with:
          name: vally-results
          path: results/

vally eval exits 1 for a failed verdict or execution/tooling failure. A configured threshold changes the verdict calculation; it does not suppress execution errors. Never use --threshold 0 in a gate unless the explicit intent is observation-only.

Ownership, support, and community

Microsoft publishes the documentation and npm packages. Package metadata reports the MIT license. The project is still pre-1.0, so compatibility guarantees are narrower than those of a mature major release.

The usual source signals were unavailable for assessing community maturity. The GitHub repository URL in the documentation returned 404, npm metadata had no usable public repository link, and some documented extension packages were not publicly installable during the review. I therefore could not independently assess contributor count, issue response time, source-level security review, release notes, or maintainer bus factor.

This does not invalidate the published packages, but it changes the support model. If Vally becomes a critical release gate, adoption should include an internal package mirror, locked versions, a rollback test, and a named Microsoft support or product contact.

Pros and cons

Good fit and poor fit

Choose Vally when:

• the unit under test is a coding or tool-using agent;
• success depends on filesystem state, tool ordering, budgets, or trajectory behavior;
• evaluation authors prefer declarative YAML over a Pytest suite;
• offline re-grading and baseline-treatment experiments are important;
• the organization already supports Node 22 and GitHub Copilot SDK execution;
• deterministic graders should carry most CI decisions.

Look elsewhere when:

• the workload is primarily a Python RAG or question-answer dataset;
• the team requires a fully public source repository before adoption;
• a stable 1.x API and long-term plugin compatibility are mandatory;
• the target is a deployed realtime voice endpoint with audio evidence;
• the runtime cannot preserve sensitive trajectories securely;
• the evaluation should be embedded directly in an existing Pytest test suite.

Verdict

Vally has the strongest general trajectory pipeline in this comparison. Isolated workspaces, deterministic graders, offline re-grading, golden evidence, experiments, and multiple artifact formats make it a coherent system for agent engineering. It covers more than a skill evaluator and observes more behavior than an output-metric library.

The recommendation is to adopt Vally behind a version-pinned pilot, not as an organization-wide release gate yet. Validate the exact executor and plugins you need. Keep deterministic graders in the critical path, archive enough trajectory evidence to explain failures, and resolve source availability and support before the tool becomes operationally indispensable.

The next article examines Microsoft Waza, a narrower Go-based evaluator whose deepest specialization is testing SKILL.md instructions, trigger precision, and tool behavior.

References

• Vally documentation
• Installation
• Run your first evaluation
• Evaluation specification
• Grader catalog
• Evaluation CLI reference
• CI integration
• Telemetry

#evaluation#microsoft-vally#ai-agents#trajectory#typescript