Machine Learning

Choosing an On-Prem SLM Inference Engine for Windows

By Omer DemirMay 30, 2025⏱ 5 min read👁 — reads⧉ — copies

Running a Retrieval-Augmented Generation (RAG) system efficiently on Windows, without internet access, and under strict latency budgets comes down to one decision that dwarfs the rest: which inference engine you pick. After evaluating ONNX Runtime GenAI, LlamaCPP, Hugging Face Optimum, and Triton, ONNX Runtime GenAI won on token throughput, latency, and wall-clock efficiency for our scenario. This is the first of a two-part series; here we cover the evaluation and the numbers.

The constraints

We needed a RAG system built with LangChain, an SLM, and guardrails that could run in an air-gapped Windows environment with hard requirements:

• All computation local — no cloud calls.
• No internet connection at runtime.
• End-to-end inference under 5 seconds.
• Efficient deployment on Windows on-prem hardware.

Those constraints immediately rule some things out. No Docker on the target box means Triton is off the table. Windows-first means anything requiring extensive custom builds (looking at you, LlamaCPP) is a liability.

The candidates

We used Microsoft’s Phi-3 Mini 4k Instruct ONNX model, deployed on an Azure ND96amsr_A100_v4 instance (1× A100 80GB). Benchmarks used a fixed generation length of 256 tokens, 5 warmup runs, and 10 repetitions, driven by the benchmark script that ships in the onnxruntime-genai package.

Reproducing the benchmark

The numbers below come from the package’s own benchmark_e2e.py, swept across prompt lengths at batch size 1, targeting ~99% GPU utilization:

python benchmark_e2e.py \
  -i ./phi3-mini-4k-instruct-onnx \
  -g 256 \            # generation length (tokens)
  -p 16,64,256,1024,2048 \  # prompt lengths swept
  -b 1 \              # batch size
  -w 5 -r 10          # 5 warmup runs, 10 repetitions

We captured five metrics per run: token-generation throughput (tps), token-generation latency, wall-clock throughput, wall-clock latency, and GPU utilization. The last one matters as a sanity check — an engine that can’t saturate the A100 is leaving throughput on the table regardless of its per-token numbers.

Results

Token generation throughput (tokens/sec)

ONNX Runtime GenAI led at every prompt length, and its lead widened as prompts grew — exactly what you want for RAG, where retrieved context inflates the prompt.

Wall-clock latency (seconds)

The number that actually matters for the 5-second budget. ORT GenAI stayed comfortably under it even at 2048-token prompts.

Optimum’s weaker graph optimization — and its lack of Phi-3 support — shows up as roughly a full second of extra latency at long prompts. At a 2048-token prompt that’s 4.279s vs 3.073s — a 1.2-second gap, or ~39% slower, which is the difference between hitting and blowing the budget.

Why ONNX Runtime GenAI is faster

The gap isn’t luck; it’s what the engine does to the graph:

• Operator fusion and a purpose-built runtime. ORT GenAI runs an optimized graph with fused attention and a managed KV cache, so per-token work stays low as the sequence grows. That’s why its throughput lead widens with prompt length (1.20× → 1.59× vs Optimum from 16 → 2048 tokens) instead of shrinking.
• INT4 block quantization. The Phi-3 ONNX build is weight-quantized, cutting memory bandwidth — the actual bottleneck in autoregressive decoding — without the accuracy hit of naive INT8.
• First-class Phi-3 support. Optimum can’t optimize Phi-3 at all, so it falls back to a generic path; ORT GenAI’s builder emits a model tuned for exactly this architecture.

LlamaCPP is competitive on throughput but still trails by ~20%, and on Windows it demands custom builds we didn’t want to own.

Windows vs Linux

Interestingly, ORT GenAI’s wall-clock throughput on Windows matched or beat Linux across the board, which put to rest the assumption that Windows would cost us performance.

Decision

ONNX Runtime GenAI won on the merits:

• Performance — highest throughput and lowest latency at every prompt length.
• Compatibility — full optimization for Phi-3 ONNX models, which Optimum cannot do.
• Windows support — official binaries, no custom build gymnastics.
• Deployment fit — no Docker requirement, unlike Triton.

If you’re targeting offline, Windows-based SLM inference, ONNX Runtime GenAI is the pragmatic default.

In part two we wire the winning engine into a LangChain RAG pipeline — including a custom BaseLLM and the model-building step that produces the optimized ONNX graph.

References

• ONNX Runtime GenAI documentation
• Phi-3 Mini 4k Instruct ONNX
• onnxruntime-genai benchmark scripts

#slm#onnx#rag#inference#windows