Applied Notes

Independent notes by Omer Demir

Personal Notes and Explorations

While exploring, some parts caught my attention more than others. Here, I share personal notes and explorations about technical topics.

The Chess Players by Thomas Eakins

Artwork: The Chess Players by Thomas Eakins. The Metropolitan Museum of Art · Public domain

Latest note

Agents

What Changes When Coding Agents Can Act?

Governed Agentic SDLC is an integration problem: execution, identity, policy, evidence, evaluation, and human approval must reinforce one another while authority stays outside the model.

Read article

From the notebook

Latest notes

View all

Longer threads

Series

View all
Agent Evaluation3-part series AI Evaluation Frameworks in Practice5-part series Building a Governed Agentic SDLC4-part series On-Prem SLM Inference with ONNX Runtime GenAI2-part series

The full notebook

All writing

19 articles
  1. What Changes When Coding Agents Can Act?Governed Agentic SDLC is an integration problem: execution, identity, policy, evidence, evaluation, and human approval must reinforce one another while authority stays outside the model.Agents
  2. Isolated Execution for Coding Agents: Where the Sandbox EndsChoose an agent sandbox by its complete boundary, not its runtime label. Kernel isolation, network paths, credentials, lifecycle controls, evidence, and broker privilege can each decide whether containment holds.Agents
  3. DeepEval vs Vally vs Waza vs convo-eval: A Technical Decision GuideMatch the evaluator to the evidence boundary: DeepEval for responses and traces, Vally for trajectories and workspaces, Waza for skills, and a licensed voice harness for deployed audio.Evaluation
  4. Conversation Evaluation Through Voice-Agent SimulationPersona-driven simulations catch deployed voice failures that text tests miss by retaining audio, timing, transcripts, tools, and outcomes. Keep deterministic gates primary and evaluate perceptual speech quality separately.Evaluation
  5. DeepEval: Pytest-Native Evaluation for LLM Applications and AgentsDeepEval is the strongest general-purpose starting point for Python LLM, RAG, and agent evaluation. Start with deterministic outcomes, then add calibrated semantic metrics where code falls short.Evaluation
  6. Microsoft Vally: Declarative Trajectory Evaluation for AI AgentsVally offers the strongest trajectory pipeline for coding and tool-using agents, but its pre-1.0 contracts and unresolved source and support model make a version-pinned pilot safer than an organization-wide gate.Evaluation
  7. Microsoft Waza: Evaluation for Agent Skills, Triggers, and Tool BehaviorWaza is a strong specialist for testing skill selection, anti-triggers, tools, and workspace effects. Pair it with other evaluators for RAG, voice, multimodal, or multi-agent behavior.Evaluation
  8. Evaluating a gpt-realtime Voice Agent with SimulationHow we test a speech-to-speech debriefing agent end-to-end: a persona-driven simulated technician talks to the real gpt-realtime agent over a WebSocket, and binary judges plus outcome verification grade the result.Evaluation
  9. Skill Evaluation with Microsoft WazaUse Waza to test skill routing and output relevance, then separate implementation evaluations to gate code quality. Mock early, run live trials selectively, and enforce both tiers in CI.Evaluation
  10. Beyond 1–5 Scores: Binary Judges, Outcome Verification, and pass^kMulti-point judge scales are subjective, transcript-only scoring misses system state, and a single trial can be a lucky run. Four changes that make agent evaluation trustworthy.Evaluation
  11. Hybrid Agent Evaluation: Custom Judges + Azure Foundry EvaluatorsCustom judges test scenario facts while Azure Foundry evaluators score general agent behavior. A REST boundary gives C# both views without adding Python to the application.Evaluation
  12. Eval-Driven Development: Red-Green-Refactor for AI AgentsEvery agent change should be justified by a failing evaluation. A disciplined, TDD-style process for evolving AI agents without regressions.Evaluation
  13. Hybrid Activity Classification: Keyword Rules First, LLM FallbackKeyword rules handle clear classifications, while an LLM receives only ambiguous cases. A confidence threshold controls escalation, reducing cost and latency without surrendering deterministic decisions.AI Agents
  14. Real-Time Voice Debriefing with the Azure Voice Live APIA persistent WebSocket supports an eight-step hands-free debrief, while persona, scenario, and environment fixtures make the nondeterministic voice agent repeatable enough to evaluate.AI Agents
  15. Designing a Multi-Agent Field Service AssistantSpecialized briefing, debriefing, and classification agents behind an orchestrator are easier to prompt, test, and evaluate than one large agent, while connectors keep local and cloud modes replaceable.AI Agents
  16. Four Chunking Strategies for Structured Documents in RAGNo chunking strategy wins every RAG workload. Put simple, Markdown, section-aware, and semantic chunkers behind one interface, preserve metadata, and choose by measured retrieval quality.RAG
  17. Wiring ONNX Runtime GenAI into a LangChain RAG PipelineA small LangChain adapter can run ONNX Runtime GenAI locally, but deployment performance depends on building and shipping the optimized Phi-3 graph instead of loading a raw export.Machine Learning
  18. Choosing an On-Prem SLM Inference Engine for WindowsONNX Runtime GenAI was the practical default for offline Phi-3 inference on Windows: it led throughput and latency, stayed under five seconds, and avoided custom builds or Docker.Machine Learning
  19. Managed Identity on Logic App Consumption with TerraformTerraform does not natively wire Managed Identity API connections for Logic App Consumption. Here is how azapi_resource plus an ARM template deployment closes the gap.Azure