Evaluation

Hybrid Agent Evaluation: Custom Judges + Azure Foundry Evaluators

By Omer DemirApr 14, 2026⏱ 5 min read👁 — reads⧉ — copies

Evaluating a speech-to-speech debriefing agent means answering two very different questions. Did the conversation stay grounded in the facts of this specific job? And did the agent behave like a competent agent — complete the task, follow the protocol, call tools correctly? No single evaluator answers both well, so we went hybrid: custom LLM judges for domain quality, plus Azure Foundry’s built-in evaluators for agent behavior.

Two dimensions, two kinds of evaluator

• Domain quality is specific to this job: were the right parts recorded, do the test results match, is the follow-up correct? Generic evaluators can’t check facts they’ve never seen.
• Agent behavior is general: task completion, task adherence, tool-call accuracy. These are well-studied dimensions that Microsoft has already tuned and validated.

The options we weighed

We chose C. The deciding constraint was practical: this is a .NET 10 solution, and as of early 2026 the Azure.AI.Projects C# package doesn’t expose the evaluation API surface. A Python sidecar (Option D) would have solved that but poisoned the build with a cross-language dependency and all the CI/CD pain that brings.

How the split works

Custom evaluators own domain quality:

• GroundednessEvaluator — validates responses against scenario-specific facts.
• CoherenceEvaluator — assesses logical flow and consistency.

Foundry evaluators own agent behavior, reached over REST:

• Task Completion — did the agent finish the debrief?
• Task Adherence — did it follow the protocol?
• Tool Call Accuracy — did it invoke tools correctly?

Foundry ships nine built-in agent evaluators in total, spanning system-level outcomes and process-level tool use:

The integration seam

Foundry evaluators expect the OpenAI message schema. Our existing TranscriptDocument already captures role, content, and tool calls, so the mapping is mechanical:

{
  "query": [
    {"role": "user", "content": "I replaced the roller guides..."}
  ],
  "response": [
    {"role": "assistant", "content": [
      {"type": "tool_call", "name": "save_part", "arguments": {"partNumber": "RG-2450-A"}}
    ]},
    {"role": "tool", "content": [{"type": "tool_result", "tool_result": {"status": "saved"}}]},
    {"role": "assistant", "content": "Got it, I've recorded the roller guide replacement."}
  ]
}

Authentication uses DefaultAzureCredential against the OpenAI Evals REST API, with a create-run-poll lifecycle:

• POST /openai/evals — create the evaluation with azure_ai_evaluator testing criteria (evaluator_name, data_mapping, initialization_parameters).
• POST /openai/evals/{eval_id}/runs — execute a run.
• GET /openai/evals/{eval_id}/runs/{run_id} — poll for completion.
• GET .../output_items — pull per-row scores, labels, and reasoning.

An EvaluationRunner orchestrates both custom and Foundry evaluators in one pipeline, so a single run produces both scenario-fact groundedness and standardized behavior metrics.

The C# client is a thin wrapper over that lifecycle — authenticate, create, run, then poll until the run leaves the queued/in-progress states:

var cred = new DefaultAzureCredential();
var client = new AzureFoundryEvaluatorClient(endpoint, cred);

// 1. Create the eval with typed azure_ai_evaluator criteria
var evalId = await client.CreateEvalAsync(new AzureAiEvaluator {
    EvaluatorName = "builtin.tool_call_accuracy",
    DataMapping = new { query = "{{item.query}}", response = "{{item.response}}",
                        tool_definitions = "{{item.tool_definitions}}" },
});

// 2. Kick off a run over the transcript data source
var runId = await client.CreateRunAsync(evalId, transcripts);

// 3. Poll to completion
EvalRun run;
do {
    await Task.Delay(TimeSpan.FromSeconds(5));
    run = await client.GetRunAsync(evalId, runId);
} while (run.Status is "queued" or "in_progress");

var rows = await client.GetOutputItemsAsync(evalId, runId);   // scores, labels, reasoning

A single run then yields both lanes side by side — domain facts from the custom judges, behavior from Foundry:

The trade-offs, honestly

The hybrid isn’t free:

• You maintain two evaluation systems — custom judge prompts and a REST integration.
• The Foundry REST API is less mature than the Python SDK and has undocumented edges.
• The REST client needs real work: auth, polling, and error handling.

We accepted those because the alternative — either weaker coverage or a Python dependency in a .NET solution — was worse. Results flow into Azure DevOps pipeline summaries via the agent evaluation extension, so the whole thing lives in CI.

Takeaways

• Match the evaluator to the question. Custom judges for facts you own; standardized evaluators for general agent behavior.
• Let constraints pick the integration. A C#-only requirement made REST the right seam, despite its rough edges.
• Unify the pipeline so one run yields both domain and behavior signals.

Next: why we replaced 1–5 judge scales with binary judges and outcome scoring.

#evaluation#azure-foundry#llm-judge#ai-agents#dotnet