Applied Notes

Independent notes by Omer Demir

Personal Notes and Explorations

While exploring, some parts caught my attention more than others. Here, I share personal notes and explorations about technical topics.

La Berceuse by Vincent van Gogh

Artwork: La Berceuse by Vincent van Gogh. The Metropolitan Museum of Art · Public domain

Latest note

Evaluation

Evaluating a gpt-realtime Voice Agent with Simulation

How we test a speech-to-speech debriefing agent end-to-end: a persona-driven simulated technician talks to the real gpt-realtime agent over a WebSocket, and binary judges plus outcome verification grade the result.

Read article

From the notebook

Latest notes

View all

Longer threads

Series

View all
Agent Evaluation3-part series On-Prem SLM Inference with ONNX Runtime GenAI2-part series

The full notebook

All writing

12 articles
  1. Evaluating a gpt-realtime Voice Agent with SimulationHow we test a speech-to-speech debriefing agent end-to-end: a persona-driven simulated technician talks to the real gpt-realtime agent over a WebSocket, and binary judges plus outcome verification grade the result.Evaluation
  2. Skill Evaluation with Microsoft WazaA YAML-first, two-tier approach to evaluating AI skills: Waza validates trigger routing and output relevance at the skill layer, then implementation evals gate code quality — all in CI.Evaluation
  3. Beyond 1–5 Scores: Binary Judges, Outcome Verification, and pass^kMulti-point judge scales are subjective, transcript-only scoring misses system state, and a single trial can be a lucky run. Four changes that make agent evaluation trustworthy.Evaluation
  4. Hybrid Agent Evaluation: Custom Judges + Azure Foundry EvaluatorsWhy we combined domain-specific custom LLM judges with Azure Foundry's built-in agent evaluators over a REST API — getting behavior metrics and scenario-fact groundedness from C# without a Python dependency.Evaluation
  5. Eval-Driven Development: Red-Green-Refactor for AI AgentsEvery agent change should be justified by a failing evaluation. A disciplined, TDD-style process for evolving AI agents without regressions.Evaluation
  6. Hybrid Activity Classification: Keyword Rules First, LLM FallbackA two-tier classifier that answers the easy cases with deterministic keyword rules and escalates only the ambiguous ones to an LLM — cutting cost and latency while keeping a confidence threshold in control.AI Agents
  7. Real-Time Voice Debriefing with the Azure Voice Live APIHow the field service assistant runs a hands-free, 8-step voice debrief over a persistent WebSocket loop — and how persona, scenario, and environment fixtures make it testable.AI Agents
  8. Designing a Multi-Agent Field Service AssistantThe architecture of a field service assistant built with FastAPI and Pydantic v2: specialized briefing, debriefing, and classification agents behind an orchestrator, with swappable connectors and persistence.AI Agents
  9. Four Chunking Strategies for Structured Documents in RAGA pluggable chunker design — simple, markdown, section-aware, and semantic — for splitting structured documents before vector indexing, with metadata that survives retrieval.RAG
  10. Wiring ONNX Runtime GenAI into a LangChain RAG PipelinePart two: a custom LangChain LLM wrapper over ONNX Runtime GenAI, streaming token generation, and building an optimized Phi-3 ONNX graph for offline Windows RAG.Machine Learning
  11. Choosing an On-Prem SLM Inference Engine for WindowsHow we benchmarked ONNX Runtime GenAI, LlamaCPP, Hugging Face Optimum, and Triton to run a sub-5-second RAG pipeline offline on Windows.Machine Learning
  12. Managed Identity on Logic App Consumption with TerraformTerraform does not natively wire Managed Identity API connections for Logic App Consumption. Here is how azapi_resource plus an ARM template deployment closes the gap.Azure