ONNX Runtime and DirectML Explained: Running Vision Models on Windows

What ONNX is

ONNX (Open Neural Network Exchange) is an open format for machine-learning models. It defines two things: a common set of operators, the building blocks such as Conv, Relu, MatMul and Resize, and a common file format. A .onnx file stores the model’s computation graph, meaning which operators run in which order, together with its trained weights, serialized with Protocol Buffers.

The point is portability. You train in PyTorch, TensorFlow/Keras or scikit-learn, export once and run the file anywhere an ONNX runtime exists, without shipping the training framework. That is why so many computer-vision pipelines end in a .onnx file. ONNX is developed under open governance and is a graduate project of the Linux Foundation’s LF AI & Data.

Every ONNX model also declares an opset (operator set version). Newer opsets add operators or change their behavior, so the runtime must support the opset your model was exported with. Exporting with a slightly older, widely supported opset avoids many compatibility errors.

What ONNX Runtime does

ONNX Runtime (ORT) is a cross-platform model accelerator maintained by Microsoft. When you create an inference session, it loads the .onnx file, applies graph optimizations such as constant folding and operator fusion, then partitions the graph among the hardware back ends you allowed. Those back ends are called execution providers (EPs).

You pass EPs as a priority list. In Python, ort.InferenceSession('model.onnx', providers=['DmlExecutionProvider', 'CPUExecutionProvider']) runs every node DirectML can handle on the GPU and everything else on the CPU. Call sess.get_providers() afterwards to see which EPs actually loaded. ORT has APIs for Python, C, C++, C#, Java, JavaScript and Objective-C, so a model prototyped in Python can ship inside a C# or C++ app.

Execution providers compared: CPU, DirectML, CUDA, TensorRT

ONNX Runtime lists roughly 20 execution providers, but four matter for a Windows desktop PC running vision models.

Two newer names are worth knowing. TensorRT RTX is an NVIDIA provider for GeForce RTX 30-series and newer cards that compiles engines on the user’s machine in seconds. Windows ML is Microsoft’s Windows-maintained copy of ONNX Runtime with the same APIs; it can fetch hardware-specific EPs through Windows Update. Neither changes the basic idea: same .onnx file, different back end.

DirectML on Windows: NVIDIA, AMD and Intel GPUs

DirectML is Microsoft’s low-level machine-learning API in the style of DirectX 12. It arrived in Windows 10, version 1903, and runs on all DirectX 12-capable hardware. For ONNX Runtime that means one package covers:

• NVIDIA: Kepler (GTX 600 series) and newer
• AMD: GCN 1st gen (Radeon HD 7000 series) and newer
• Intel: Haswell (4th-gen Core) HD integrated graphics and newer
• Qualcomm: Adreno 600 and newer

That breadth is DirectML’s main strength. One build of an app runs on an AMD Radeon desktop, an Intel laptop and an NVIDIA gaming PC without installing CUDA.

DirectML settings that trip people up

• Memory patterns off, sequential execution. The DirectML EP does not support memory-pattern optimizations or parallel execution, so set enable_mem_pattern = False and execution_mode = ORT_SEQUENTIAL in the session options.
• No concurrent `Run` calls on one session. If two threads need inference, give each its own session or put a lock around Run.
• Fixed input shapes run best. DirectML is most efficient when tensor shapes are known when the session is created. Export with a fixed batch size and image size, or pin free dimensions with the dimension-override options.
• Pick the right GPU. On laptops with integrated and dedicated graphics, the device_id provider option selects the adapter; 0 is the default adapter.

FP16 vs FP32: precision, size and speed

Model weights and activations are normally stored as 32-bit floats (FP32). Converting to 16-bit floats (FP16, half precision) cuts the model size by up to half and can run faster on GPUs with fast FP16 hardware, such as NVIDIA’s Tensor Cores. Unlike INT8 quantization, FP16 conversion needs no calibration data.

The ONNX Runtime docs show the conversion with the onnxconverter-common package: load the model, call float16.convert_float_to_float16(model) and save it. Pass keep_io_types=True to keep inputs and outputs in FP32 so your preprocessing code does not change. If accuracy drops too far, the auto mixed-precision tool keeps sensitive nodes in FP32. With TensorRT you can leave the file in FP32 and set the provider option trt_fp16_enable instead.

Exporting a model from PyTorch to ONNX

Most vision models start life in PyTorch. A clean export takes six steps:

1. Install the exporter dependencies: pip install torch onnx onnxscript.
2. Switch the model to inference mode with model.eval() and create a dummy input of the exact shape you will use, for example torch.rand(1, 3, 640, 640).
3. Export with torch.onnx.export(model, (dummy,), 'model.onnx', input_names=['images'], dynamo=True). Since PyTorch 2.9, the torch.export-based exporter (dynamo=True) is the default.
4. Validate the file with onnx.checker.check_model and open it in a graph viewer such as Netron to confirm input and output names and shapes.
5. Run the same input through PyTorch and ONNX Runtime and check that the outputs match within a small tolerance.
6. Optionally convert to FP16, then benchmark every execution provider you plan to support.

For YOLO models, Ultralytics wraps all of this in one command: yolo export model=yolo26n.pt format=onnx, with arguments for image size, dynamic shapes, opset and NMS handling. If you plan to run on DirectML, keep the batch and image size fixed.

How fast is it? Typical latency and honest caveats

There is no single speed for ONNX Runtime. It depends on the model, input size, precision, provider, GPU, driver and whatever else the GPU is doing. Published reference numbers give a sense of scale:

• These are model-only times. Capture, color conversion, resizing, the copy to the GPU, NMS and drawing come on top.
• The T4 is a data-center GPU running TensorRT. DirectML on a consumer card will land somewhere else, and there are no official DirectML figures for these models, so measure your own.
• First runs are slower. Sessions warm up, and TensorRT builds an engine on first start: in one ONNX Runtime benchmark, session creation fell from 384 s without a cache to 9 s with an engine cache. Discard warm-up runs when timing.
• Report the median and the 99th percentile, not the best run. A pipeline that averages 8 ms but spikes to 25 ms still stutters at 60 fps.

Compare the total against your frame budget: 16.7 ms per frame at 60 fps, 8.3 ms at 120 fps. Setting enable_profiling = True in the session options writes a per-operator JSON trace that opens in Perfetto or chrome://tracing. For the rest of the chain, see where latency comes from in a capture and vision setup and the PC requirements for real-time computer vision.

Troubleshooting common ONNX Runtime errors

If nothing matches, set log_severity_level = 0 in the session options. The verbose log shows which provider each part of the graph was assigned to. Unfamiliar terms are defined in our console-vision glossary.

Frequently asked questions

Sources

• ONNX — Open Neural Network Exchange
• ONNX Runtime — Documentation overview
• ONNX Runtime — Execution providers
• ONNX Runtime — DirectML execution provider
• ONNX Runtime — CUDA execution provider
• ONNX Runtime — TensorRT execution provider
• ONNX Runtime — TensorRT RTX execution provider
• ONNX Runtime — Install
• ONNX Runtime — Float16 and mixed precision models
• ONNX Runtime — Profiling tools
• ONNX Runtime — Performance troubleshooting
• Microsoft Learn — Introduction to DirectML
• Microsoft Learn — What is Windows ML?
• PyTorch — torch.onnx documentation
• Ultralytics — YOLO11 model documentation
• Ultralytics — YOLO26 model documentation
• Ultralytics — Model export