Nvidia builds the chips. AWS builds the plumbing. When Amazon talks about AI infrastructure, the conversation usually turns to Trainium chips or Bedrock’s managed models — but underneath both sits a less-discussed layer that is arguably just as important: the networking and virtualization stack that lets AWS wire tens of thousands of GPUs together into something that behaves like one giant computer. That stack has two main pieces — the Nitro System and the Elastic Fabric Adapter (EFA) — and this piece looks specifically at how they work and what they actually do.
At a Glance
| Component | What It Is | Key Fact |
|---|---|---|
| Nitro System | Custom hypervisor and offload hardware underlying modern EC2 instances | Moves virtualization, networking, and storage I/O off the host CPU onto dedicated cards |
| Elastic Fabric Adapter (EFA) | OS-bypass network interface for HPC/AI workloads | Free add-on on supported instances; EFA on P5 delivers 3,200 Gbps per instance — 8× the prior generation |
| EC2 UltraClusters | Petabit-scale, non-blocking network connecting large GPU fleets | AWS and Nvidia described scaling to 20,000 H100 GPUs and 20 exaFLOPS (announced March 2023) |
Short Verdict
AWS doesn’t win AI infrastructure on chips alone. The Nitro System and EFA are the plumbing that let it treat a rack — or tens of thousands of GPUs across many racks — as a single machine for training. Most of this is invisible unless you’re running distributed training or HPC workloads at scale, but it’s a real part of why AWS can credibly host frontier-scale training jobs at all.
What the Nitro System Actually Does
Every modern EC2 instance type is built on the Nitro System, AWS’s combination of dedicated offload hardware and a lightweight hypervisor. Instead of running networking, storage, and management functions as software on the same CPU that runs customer workloads, Nitro moves them onto dedicated cards:
- Nitro Cards handle I/O offload for VPC networking, EBS storage, and instance storage, plus a Nitro Card Controller that coordinates them.
- Nitro Security Chip moves virtualization and security functions to dedicated hardware and enforces a locked-down model that blocks administrative access — including, AWS states, by Amazon’s own employees.
- Nitro Hypervisor is a lightweight hypervisor that only manages memory and CPU allocation; AWS describes its performance as indistinguishable from bare metal.
- Nitro Isolation Engine enforces isolation between instances and between instances and AWS operators.
- AWS Nitro Enclaves and NitroTPM provide isolated compute environments and a TPM 2.0-conformant module for cryptographic attestation, respectively — used for sensitive workloads handling PII, healthcare, or financial data.
The practical effect, per AWS’s own description, is that instances get “practically all” of the host’s compute and memory resources, because the host no longer has to reserve capacity for hypervisor or management overhead. That matters for GPU instances specifically: a Nitro-based host spends less CPU and memory on virtualization bookkeeping and more on feeding data to the GPUs.
What EFA Adds on Top
Nitro handles a single instance. Elastic Fabric Adapter (EFA) is what lets many instances act together. EFA is an optional network interface, available at no additional cost on supported instance types, that gives applications a custom OS-bypass hardware interface for inter-instance communication.
Normally, network traffic goes through the operating system’s TCP/IP stack and a standard network driver (AWS’s ENA). EFA skips that path: applications talk to the EFA device through the Libfabric API — the user-space API of the OpenFabrics Interfaces framework — bypassing the kernel entirely. Underneath, EFA uses AWS’s Scalable Reliable Datagram (SRD) protocol, which provides built-in congestion control and low-latency, reliable transport tuned for HPC and ML traffic patterns.
Because most HPC programming models (MPI) and ML collective-communication libraries (notably Nvidia’s NCCL, used by PyTorch, TensorFlow, MXNet, and others) already support Libfabric, AWS says existing distributed training code can usually move to EFA with little or no modification. On GPU instances, EFA also integrates with Nvidia GPUDirect RDMA, letting data move between GPU memory on different instances without an extra trip through host memory.
EFA Has Generations, and Support Varies by Instance
EFA isn’t one fixed spec — it has shipped in multiple versions tied to the underlying Nitro hardware generation, and not every EC2 instance family supports it. AWS’s own EC2 User Guide lists support roughly as follows:
| Nitro Generation | EFA Version | Representative Instance Families |
|---|---|---|
| Nitro v3 | EFA v1 | p3dn, p4d/p4de, g4dn, g5, c5n, m5n/m5dn |
| Nitro v4 | EFA v2 | p5/p5e, trn1/trn1n, g6/g6e, c6i/c7i, hpc6a/hpc7a |
| Nitro v5 | EFA v3 | p5en, trn2/trn2u, c7gn, hpc7g |
| Nitro v6 | EFA v4 | p6-b200, p6e-gb200, g7/g7e, hpc8a |
RDMA support also varies: older Nitro v3 instances mostly lack RDMA write support, while Nitro v4 and later support RDMA read broadly and RDMA write on most types. EFA traffic cannot cross Availability Zones or VPCs and is not routable — it’s built for tightly-coupled, low-latency communication within a cluster placement group, not general-purpose networking.
How It Comes Together in EC2 UltraClusters
Nitro and EFA combine most visibly in EC2 UltraClusters — AWS’s term for large-scale deployments of GPU instances connected by a petabit-scale, non-blocking network and backed by Amazon FSx for Lustre for high-throughput storage. When AWS and Nvidia announced EC2 P5 instances in March 2023, each P5 instance packed eight Nvidia H100 GPUs, 640 GB of high-bandwidth GPU memory, and 16 petaFLOPs of mixed-precision performance — and used AWS’s second-generation EFA to deliver 3,200 Gbps of networking per instance, eight times the prior generation, integrated with GPUDirect RDMA. AWS and Nvidia stated that UltraClusters built from P5 instances could scale to 20,000 H100 GPUs working together, delivering up to 20 exaFLOPS of aggregate performance, and that some customers were already training models across more than 10,000 GPUs.
This is a different layer from the one covered in our look at AWS’s Trainium and Inferentia chips or Bedrock and SageMaker’s managed AI services. Nitro and EFA are infrastructure that underpins both Nvidia-GPU-based instances (P5, P6) and AWS’s own Trainium-based instances (Trn1, Trn2) — the networking fabric doesn’t care which chip is doing the math.
Trade-offs and Limitations
- Not every instance qualifies. EFA is tied to specific instance families and Nitro generations; older or smaller instance types don’t support it at all.
- It’s AWS-specific plumbing. EFA competes conceptually with Nvidia’s own InfiniBand and NVLink/Spectrum-X networking (covered in our piece on Nvidia’s networking stack), so workloads tuned tightly for one fabric may need adjustment to move to another cloud or on-prem InfiniBand cluster.
- Documented benchmarks are vendor-published. AWS’s own pages cite internal figures (the 4x CFD scaling improvement over its standard ENA networking, for instance) rather than independent third-party benchmarks; treat vendor performance claims as directional, not guaranteed for every workload.
- EFA is not routable. It only works within a single cluster placement group in one Availability Zone, so it solves tightly-coupled cluster communication, not general cross-region networking.
Who Should Care
ML infrastructure engineers and platform teams evaluating cloud GPU providers for large-scale distributed training are the main audience — if you’re comparing cloud GPU platforms, the networking fabric underneath the GPU count is often the deciding factor for how well training actually scales past a handful of nodes. If your workloads are single-instance, inference-only, or small-batch, Nitro’s general virtualization benefits still apply, but EFA’s benefits are mostly invisible to you.
Bottom Line
The Nitro System and EFA are the unglamorous but essential layer that turns a pile of GPU instances into a cluster that can actually train large models efficiently. Nitro lowers virtualization overhead on every instance; EFA, where supported, removes the kernel networking bottleneck between instances and lets AWS wire tens of thousands of GPUs into UltraClusters with petabit-scale, non-blocking networking. None of it replaces chip choice or managed-service convenience — it’s the plumbing that makes both of those actually scale.
FAQ
Is EFA free to use?
Yes — AWS states EFA is available at no additional cost on supported instance types; you pay the normal EC2 instance price.
Does EFA work with Nvidia GPUs?
Yes. On GPU instances like the P5 family, EFA integrates with Nvidia GPUDirect RDMA to move data between GPU memory on different instances with lower latency.
What is SRD?
Scalable Reliable Datagram is AWS’s custom transport protocol underlying EFA. AWS describes it as providing built-in congestion control and low-latency, reliable delivery suited to HPC and ML traffic patterns, as an alternative to standard TCP.
How large can an EC2 UltraCluster get?
In their March 2023 joint announcement, AWS and Nvidia described UltraClusters built on P5 instances scaling to 20,000 H100 GPUs, delivering up to 20 exaFLOPS of aggregate compute performance.
Sources
- AWS — The Nitro System
- AWS — Elastic Fabric Adapter
- AWS EC2 User Guide — Elastic Fabric Adapter for AI/ML and HPC workloads
- AWS and Nvidia — Next Generation Infrastructure for Training Large ML Models (March 2023)
Discover more from Software Online Review
Subscribe to get the latest posts sent to your email.
