Series
Agent Evaluation
3 parts
- 1Eval-Driven Development: Red-Green-Refactor for AI Agents
· Every agent change should be justified by a failing evaluation. A disciplined, TDD-style process for evolving AI agents without regressions.
- 2Hybrid Agent Evaluation: Custom Judges + Azure Foundry Evaluators
· Why we combined domain-specific custom LLM judges with Azure Foundry's built-in agent evaluators over a REST API — getting behavior metrics and scenario-fact groundedness from C# without a Python dependency.
- 3Beyond 1–5 Scores: Binary Judges, Outcome Verification, and pass^k
· Multi-point judge scales are subjective, transcript-only scoring misses system state, and a single trial can be a lucky run. Four changes that make agent evaluation trustworthy.
On-Prem SLM Inference with ONNX Runtime GenAI
2 parts
- 1Choosing an On-Prem SLM Inference Engine for Windows
· How we benchmarked ONNX Runtime GenAI, LlamaCPP, Hugging Face Optimum, and Triton to run a sub-5-second RAG pipeline offline on Windows.
- 2Wiring ONNX Runtime GenAI into a LangChain RAG Pipeline
· Part two: a custom LangChain LLM wrapper over ONNX Runtime GenAI, streaming token generation, and building an optimized Phi-3 ONNX graph for offline Windows RAG.