Running a trained model on a server GPU is one problem. Running it fast, on whatever chip happens to be in the device in front of a user, is a different one. That’s the job of an inference runtime: a layer that takes a trained model and executes it efficiently on specific hardware, whether that’s a data-center GPU, a laptop CPU, or the Neural Engine inside an iPhone. Three names dominate that layer in 2026: Microsoft’s ONNX Runtime, NVIDIA’s TensorRT, and Apple’s Core ML (now folded into the broader Core AI framework for generative models). They solve overlapping problems in very different ways, and picking the wrong one means either leaving performance on the table or locking a product to hardware it doesn’t need.
At a Glance
| Runtime | Maintainer | License | Target hardware | Best for |
| ONNX Runtime | Microsoft (open governance) | MIT, free and open source | CPU, GPU, NPU — Windows, Linux, macOS, iOS, Android, browser | Portable inference across many devices and frameworks |
| TensorRT | NVIDIA | Free SDK, source not fully open (TensorRT-LLM parts are) | NVIDIA GPUs only (data center and Jetson/DRIVE lines) | Maximum throughput/latency on NVIDIA hardware |
| Core ML / Core AI | Apple | Ships free with Xcode | Apple Silicon only — CPU, GPU, Neural Engine | On-device inference inside iOS/macOS/visionOS apps |
Short Verdict
These three aren’t really competing for the same deployment. ONNX Runtime is the portability layer — it runs nearly anywhere and defers to whatever accelerator is present. TensorRT is the performance layer for NVIDIA GPUs specifically, squeezing out the last few milliseconds of latency at the cost of hardware lock-in. Core ML (and its 2026 successor, Core AI) is Apple’s walled garden for shipping models inside consumer apps without a network round-trip. Most real pipelines end up using more than one: export to ONNX for portability and CI testing, then compile a NVIDIA-specific build through TensorRT for the production GPU server, and a separate Core ML/Core AI build for the iOS app.
What Each One Actually Is
ONNX Runtime is a cross-platform inference and training engine built around the Open Neural Network Exchange (ONNX) format — a standardized way to represent a trained model so it isn’t tied to the framework (PyTorch, TensorFlow, scikit-learn) that produced it. Microsoft maintains it and uses it internally across Windows, Office, and Azure services. It runs on Linux, Windows, macOS, iOS, Android, and even inside a browser via WebAssembly, and it dispatches work to different “execution providers” depending on what’s available — CPU, a CUDA/TensorRT provider for NVIDIA GPUs, DirectML on Windows, CoreML as an execution provider on Apple hardware, or a dedicated NPU provider. It’s MIT-licensed and free.
TensorRT is NVIDIA’s inference optimizer and runtime, built specifically for NVIDIA GPUs. It takes a model (typically via an ONNX import) and compiles it into a GPU-specific execution plan: fusing layers, choosing the fastest kernel implementations for the exact GPU generation, and applying quantization (FP16, INT8, FP8, and FP4 on the newest Blackwell/Vera Rubin-class hardware) to cut memory and latency. The current release line is TensorRT 11.x, built against CUDA 13, and it explicitly supports the latest data-center GPUs on Linux x86, Linux SBSA (Arm server CPUs), and Windows x64. NVIDIA dropped DLA (Deep Learning Accelerator) support after TensorRT 10.7, so Jetson edge deployments that rely on DLA have to stay on a 10.x release rather than jumping to 11.x. TensorRT is free to download but is not a fully open-source project in the way ONNX Runtime is, and it only runs on NVIDIA silicon.
Core ML is Apple’s on-device ML framework: you convert a trained model (commonly with the coremltools Python package) into an .mlpackage, and the OS decides at runtime whether to run each layer on the CPU, GPU, or Neural Engine via an MLComputeUnits setting. It ships free with Xcode and has shipped in iOS since 2017. At WWDC 2026, Apple introduced Core AI as the explicit successor framework for neural networks and transformer-based models — covering everything from small vision models to on-device LLMs up to roughly 70 billion parameters across iPhone, iPad, Mac, and Vision Pro. Core AI adds a memory-safe Swift API, ahead-of-time compilation to cut load times, and automatic specialization to the exact chip and OS version at first launch. Apple now positions Core ML for classic, non-neural models (decision trees, simple regressions) and points developers toward Core AI — or MLX for custom model weights — for anything transformer-based. Both run exclusively on Apple Silicon.
How They Differ in Practice
| Dimension | ONNX Runtime | TensorRT | Core ML / Core AI |
| Hardware lock-in | None — portable by design | NVIDIA GPUs only | Apple Silicon only |
| Typical deployment | Cross-platform apps, Windows ML, server fallback | GPU-backed inference servers, robotics/Jetson edge boxes | iOS/macOS/visionOS apps, offline-capable features |
| Peak performance on its target hardware | Good, but generally behind a vendor-tuned runtime | Typically the fastest option on NVIDIA GPUs | Best possible use of the Neural Engine for Apple devices |
| Model source formats | Any ONNX-exportable framework | ONNX import, or NVIDIA’s own model-optimizer toolchains | PyTorch/TensorFlow via coremltools, or pre-optimized open models |
| Open source | Yes, MIT license | No (free SDK, source partially closed) | No (free, bundled with Apple’s developer tools) |
Pros and Cons
- ONNX Runtime pros: genuinely portable, free, strong language bindings, works as a fallback when a vendor-specific runtime isn’t available.
- ONNX Runtime cons: rarely the absolute fastest option on any single piece of hardware — it trades peak performance for reach.
- TensorRT pros: the fastest practical option for inference on NVIDIA GPUs, with mature INT8/FP8/FP4 quantization tooling.
- TensorRT cons: hard NVIDIA lock-in, a steeper build/compile workflow, and a release cadence that can drop support for older hardware paths (like DLA) between major versions.
- Core ML/Core AI pros: the only realistic way to get full Neural Engine acceleration, tightly integrated with iOS/macOS privacy and offline requirements.
- Core ML/Core AI cons: Apple Silicon only, and the Core ML → Core AI transition means some existing conversion pipelines and documentation are mid-migration in 2026.
Who Should Use Which
Teams shipping a model across many unknown customer environments — a desktop app, a Windows ML feature, a browser-based demo — should default to ONNX Runtime and treat it as the baseline deployment target. Teams running inference at scale on their own or a cloud provider’s NVIDIA GPUs, where every millisecond of latency or every point of GPU utilization has a direct cost, should compile a TensorRT path for production even if ONNX Runtime is used for development and testing. Teams building native iOS, macOS, or visionOS apps that need a model to run fully offline, with no data leaving the device, should use Core ML for classic models and start evaluating Core AI now for anything transformer-based, since it’s where Apple’s roadmap and new on-device LLM capabilities are heading.
Bottom Line
There’s no single winner because these tools don’t compete on the same axis. ONNX Runtime wins on portability, TensorRT wins on raw NVIDIA GPU performance, and Core ML/Core AI wins on anything that has to run privately on an Apple device. The practical pattern most teams land on is exporting a model to ONNX as the common intermediate format, then compiling platform-specific builds — TensorRT for GPU servers, Core ML/Core AI for Apple apps — from that same source model.
FAQ
Can I use TensorRT and ONNX Runtime together? Yes — ONNX Runtime has a TensorRT execution provider, so you can run inside the ONNX Runtime API while NVIDIA’s TensorRT does the actual GPU execution underneath.
Is Core ML being discontinued? No. Apple is keeping Core ML for classical, non-neural-network models, while routing neural network and transformer workloads to the newer Core AI framework introduced at WWDC 2026.
Do any of these cost money? All three are free to use. The cost in practice is engineering time for conversion and the hardware itself (NVIDIA GPUs for TensorRT, Apple devices for Core ML/Core AI).
What about AMD or Intel hardware? ONNX Runtime is the practical choice there — it has execution providers for DirectML, OpenVINO (Intel), and ROCm (AMD), none of which TensorRT or Core ML support.
