Oracle Cloud Infrastructure Supercluster Networking: RDMA and GPU Scale-Out Explained

Training a frontier AI model isn’t really a chip problem anymore — it’s an interconnect problem. Once a job spans more than a few hundred GPUs, how fast those GPUs can read and write each other’s memory matters as much as how fast any single GPU can compute. Oracle’s answer is what it markets as the OCI Supercluster, and underneath the marketing language is a genuinely detailed RDMA networking design. This piece breaks down what Oracle has actually published about how that fabric works, how big it scales, and what happens when a cable flickers in the middle of a six-week training run.

At a Glance

Aspect	What Oracle Publishes
Largest single supercluster (Clos fabric)	Up to 131,072 NVIDIA Blackwell (B200) GPUs
Peak performance of that cluster	2.4 zettaFLOPS
Larger multi-datacenter system	“Zettascale10”: up to 800,000 GPUs, 16 zettaFLOPS, multigigawatt campus, first deployment in Abilene, TX
Network protocol	Custom RDMA over Converged Ethernet v2 (RoCE v2), via Oracle’s “Acceleron” NIC
Topology	Three-tier Clos fabric, 400 Gbps nonblocking per GPU
Aggregate bandwidth (131,072-GPU cluster)	52 Pbps nonblocking, at 400 Gbps per port
Cluster-network bandwidth (per OCI product page)	Up to 3,200 Gb/sec, plus a separate front-end network up to 400 Gb/sec
Latency	As low as ~2 microseconds (innermost tier) up to ~8–9.1 microseconds across the full cluster
Compute building block	Bare-metal “cluster networks” — instance pools (homogeneous) or compute clusters (mixed)
GPU shapes on the platform	H100, H200, B200 / GB200 NVL72, A100, L40S, plus AMD MI300X as an alternative

What a “Cluster Network” Actually Is on OCI

Oracle’s documentation describes a cluster network as a group of bare-metal HPC, GPU, or optimized instances placed in close physical proximity and connected by a Remote Direct Memory Access (RDMA) network, with latency Oracle says is comparable to an on-premises HPC cluster. There are two ways to build one. “Cluster networks with instance pools” manage a group of identical instances together, which suits predictable, fixed-size capacity. “Compute clusters” instead start as an empty RDMA network group that you populate and depopulate instance by instance — useful when you want to mix instance types or manage nodes independently.

Layered on top of (not instead of) a compute cluster is what Oracle calls a GPU memory cluster: a group of instances that share a single GPU memory fabric domain, running in a hardware-isolated environment, which Oracle describes as turning a set of GPUs into “a single AI supercomputer.” Getting this working on a given bare-metal shape — Oracle lists shapes including BM.GPU.H100.8, BM.GPU.A100, BM.GPU4.8, BM.HPC2.36, and BM.Optimized3.36 — depends on the Oracle Cloud Agent’s HPC plugin, which handles three jobs automatically: configuring the Mellanox ConnectX network adapters correctly, authenticating and tuning RDMA interfaces (including QoS and MTU settings), and emitting the additional RDMA and GPU performance metrics engineers need to see what the fabric is actually doing.

Inside the Fabric: RoCE v2 and the Acceleron NIC

RDMA’s core trick is letting one server read or write another server’s memory directly, bypassing the operating system kernel and most of the CPU overhead that ordinary TCP/IP networking requires — which is exactly why HPC and AI training clusters use it instead of standard data-center networking. The traditional way to deliver RDMA at scale is InfiniBand, and Oracle’s literature on its largest, multi-datacenter Zettascale10 system does reference InfiniBand-based networking as part of the design. But the documented fabric behind Oracle’s mainstream Supercluster product — the one that scales to 131,072 GPUs — is Oracle’s own custom implementation of RDMA over Converged Ethernet v2 (RoCE v2), running over NVIDIA ConnectX network interface cards and data processing units (DPUs) for hardware-accelerated packet handling.

Oracle markets this specific implementation as the Acceleron RoCE Network Fabric. Its distinguishing design choice is that each GPU’s network interface card carries its own onboard four-port Ethernet switch, letting a single GPU fan out to several upstream switches simultaneously rather than relying on one uplink. The result, per Oracle, is a flatter network built from multiple isolated “planes” — any one of which can be taken offline for maintenance without disrupting the others — and fewer switch hops between any two GPUs, which directly lowers latency. Oracle also says Acceleron carries line-rate encryption and enforces what it calls Zero Trust Packet Routing (ZPR): a security policy enforced at every NIC that can, for example, block a running workload from reaching object storage over the public internet, as a guard against data exfiltration from a compromised training job. Oracle’s own comparison claims up to 2x more networking capacity and up to 2x more storage IOPS versus its prior-generation fabric — a vendor-stated figure that Oracle does not benchmark against a named competitor.

The Three-Tier Clos Network and Its Latency Budget

The fabric behind Oracle’s largest documented single supercluster is a three-tier Clos topology — a multi-stage, leaf-and-spine-style design common in hyperscale networks specifically because it can be built nonblocking, meaning no two GPUs have to compete for the same internal link just because of where they happen to sit. Oracle scales this design to up to 131,072 NVIDIA Blackwell GPUs, each with its own dedicated, nonblocking 400 Gbps port. The three tiers have distinct latency and reach:

• Tier 1: up to 256 GPUs, roughly 2 microseconds of unidirectional latency
• Tier 2: up to 2,048 GPUs, roughly 5 microseconds
• Tier 3 (the full cluster): up to 131,072 GPUs, roughly 8 microseconds

Two engineering details explain how Oracle holds those numbers. First, the physical cable run from a GPU to its first-hop switch is capped at 40 meters, which keeps fiber-propagation delay — about 5 nanoseconds per meter — small relative to switching time. Second, the switch ASICs themselves are tuned to a latency budget under 1 microsecond, typically around 900 nanoseconds, by keeping routing and switching state in fast SRAM and TCAM memory rather than falling back to slower DRAM or HBM lookups. Aggregated across the full 131,072-GPU cluster, Oracle states the fabric delivers 52 petabits per second of nonblocking bandwidth at 400 Gbps per port. Separately, Oracle’s general Supercluster product page cites up to 3,200 Gb/sec of cluster-network bandwidth per node, plus an independent front-end network of up to 400 Gb/sec reserved for storage and management traffic, so ordinary I/O doesn’t compete with GPU-to-GPU chatter. Oracle also claims this adds up to 5x the network bandwidth and up to 5x lower latency than unnamed competing clouds — a marketing claim worth flagging as unverified rather than an independently benchmarked figure.

Keeping the Fabric Alive at 131,000-GPU Scale

Raw bandwidth and topology are only half the story; the other half is keeping six-figure-GPU jobs from stalling on congestion or a single bad cable. Classic RoCE deployments lean on Priority Flow Control (PFC) to avoid dropped packets, but PFC can cascade into head-of-line blocking across an entire fabric when one slow receiver backs up traffic for everyone behind it. Oracle’s network instead treats congestion control as the primary mechanism at every layer of the fabric, and separates traffic into distinct classes — so GenAI collective-communication traffic and HPC-MPI traffic each get their own quality of service rather than starving one another.

On top of that, Oracle runs what it calls intelligent GPU placement, which keeps a given customer’s leased GPUs physically close together in the topology to reduce latency and the odds of competing flows colliding, backed by a “network locality service” that exposes topology information to job schedulers so bandwidth-heavy work stays within a faster, lower tier instead of spilling across the entire Tier-3 fabric. Standard equal-cost multi-path (ECMP) routing — the default way most networks spread traffic across redundant links — has no awareness of how AI collective operations like all-reduce actually move data, so it can route several heavy flows onto the same path and collide. Oracle says it built, together with an unnamed switch vendor, “collectives-aware load balancing” that understands these ML traffic patterns and spreads them deliberately to avoid that.

Failure handling is its own discipline at this scale. On a 400G link, as few as three uncorrectable forward-error-correction block errors within about 20 nanoseconds can force the link down, and recovering from a flap typically costs 10 to 15 seconds while every layer — PMD, PMA, FEC, PCS, MAC, and IP — retrains and routing protocols like BGP or OSPF reconverge. At cluster scale, that’s meaningful lost training time, repeated across thousands of links. Oracle’s mitigations include “link debounce,” which keeps upper network layers up through brief optical interruptions without forcing a reroute; predictive monitoring that flags “repeat offender” cables and optics for automatic traffic rerouting and physical repair; and a GPU-hosted Oracle Cloud Agent that watches for host-side link and environmental anomalies.

Beyond One Cluster: Zettascale10

A single 131,072-GPU, 2.4-zettaFLOPS supercluster is itself only one building block of what Oracle, at Oracle AI World, branded Zettascale10: a system spanning multiple data centers, tied together by Oracle’s RoCE- and InfiniBand-based networks plus high-capacity fiber links between sites, rated at 16 zettaFLOPS and scaling to as many as 800,000 Nvidia GPUs inside a “multigigawatt” campus. The first such deployment is sited in Abilene, Texas. Oracle EVP Mahesh Thiagarajan framed the goal around efficiency rather than raw GPU count, stating that customers can “build, train, and deploy their largest AI models into production using less power per unit of performance.” Oracle has not published how the 131,072-GPU Clos fabric nests inside that larger multi-datacenter figure, nor site-level power specifics beyond the “multigigawatt” description — a real gap in an otherwise detailed set of disclosures.

Supercluster Size by GPU Generation

GPU / Accelerator	Max GPUs per Single OCI Supercluster (Oracle-published)
NVIDIA L40S	3,840
NVIDIA H100	16,384
AMD MI300X	16,384
NVIDIA A100	32,768
NVIDIA H200	65,536
NVIDIA GB200 NVL72 Superchips	More than 100,000
NVIDIA B200 (Blackwell)	Up to 131,072

Bottom Line

Oracle’s Supercluster pitch is ultimately a networking story dressed up as a GPU-count story. The headline numbers — 131,072 Blackwell GPUs, 800,000 GPUs under Zettascale10 — are the visible tip of a fabric built from custom Ethernet NICs with onboard switches, a carefully latency-budgeted three-tier Clos topology, and a fairly unusual amount of published detail on how Oracle keeps that fabric from falling over under real-world link failures. Whether it actually beats InfiniBand-based alternatives from other clouds in practice is something Oracle’s own marketing claims rather than something this piece can independently verify — but the underlying engineering choices are real, documented, and more specific than most cloud providers publish about their own AI networking.

FAQ

Is OCI’s Supercluster fabric InfiniBand or Ethernet?
Oracle’s documented, Acceleron-based Clos fabric that scales to 131,072 GPUs runs RoCE v2 over Ethernet. Oracle’s larger, multi-datacenter Zettascale10 system references both RoCE- and InfiniBand-based networking, but Oracle hasn’t published a diagram reconciling exactly which links use which protocol.

What does RDMA actually buy you over ordinary networking?
It lets one server read or write another’s memory directly, skipping most OS kernel and CPU overhead — which is why single-digit-microsecond, kernel-bypass communication is achievable at all between tens of thousands of GPUs.

Does every OCI customer get a full 131,072-GPU cluster?
No — these are Oracle’s published maximum platform-scale figures, not a guaranteed allocation size for any individual customer.

Does Oracle charge extra for RDMA bandwidth?
Oracle states that RDMA capability, block storage, and the published cluster-network bandwidth carry no additional charge beyond the compute shape’s price, and that the first 10 TB of monthly egress is free.

How does this compare to Nvidia’s own networking stack?
Oracle builds its superclusters on Nvidia GPU and Superchip hardware (including GB200 NVL72), but layers its own custom RoCE v2 fabric — Acceleron — on top, rather than adopting Nvidia’s InfiniBand/Spectrum-X networking stack end-to-end.