AWS runs two separate custom-silicon product lines for machine learning — Trainium and Inferentia — and it’s easy to assume Inferentia is just “last year’s Trainium” repurposed for a cheaper job. It isn’t. Inferentia chips were purpose-built for inference from their first generation in 2018, years before Trainium existed, and the two families have stayed architecturally distinct ever since: different core counts per chip, different memory configurations, and different instance-scaling models. Here’s what AWS has actually published about Inferentia and Inferentia2, how that compares with what AWS says about Trainium, and — importantly — what AWS has not yet announced.
At a Glance
| First Inferentia chip | Announced November 2018 |
| Inferentia (gen 1) cores | 4 NeuronCores per chip |
| Inferentia (gen 1) memory | 8 GB DDR4 per chip |
| Inferentia2 cores | 2 (second-generation) NeuronCores per chip |
| Inferentia2 compute | Up to 190 TFLOPS FP16 per chip |
| Inferentia2 memory | 32 GB HBM per chip (4x capacity, 10x bandwidth vs. gen 1) |
| Instance families | Inf1 (up to 16 chips), Inf2 (up to 12 chips) |
| Sibling chip for training | Trainium / Trainium2 / Trainium3 |
| Inferentia3 | Not officially announced as of this writing |
What It Is
Inferentia is AWS’s custom ASIC family dedicated to running trained models cheaply at scale — not training them. The first-generation chip, announced in 2018, packs four first-generation NeuronCores and 8 GB of DDR4 memory per chip (plus additional on-chip memory), supports FP16, BF16, and INT8, and powers Inf1 instances that scale up to 16 chips. AWS’s own figures put Inf1 at up to 2.3x higher throughput and up to 70% lower cost per inference than comparable general-purpose EC2 instances.
Inferentia2 is the current generation, and AWS describes it on its own product page as built for “large-scale generative AI applications.” Each Inferentia2 chip has two second-generation NeuronCores delivering up to 190 TFLOPS of FP16 performance, paired with 32 GB of HBM per chip — AWS states this is 4x the total memory and 10x the memory bandwidth of the first-generation chip. Inferentia2 also adds FP32, TF32, and a configurable FP8 format (cFP8) to the supported data types, along with hardware support for dynamic input sizes, custom C++ operators, and stochastic rounding — all features aimed specifically at serving large transformer-based models efficiently rather than training them from scratch. Inf2 instances scale up to 12 chips and, per AWS, deliver up to 4x higher throughput and up to 10x lower latency than Inf1, plus up to 50% better performance per watt than comparable EC2 instances. AWS calls Inf2 “the first inference-optimized instances in Amazon EC2 to support scale-out distributed inference.”
Inferentia vs. Trainium: Two Different Chips, One SDK
AWS positions these as genuinely separate silicon lines sharing a software stack (the Neuron SDK), not one chip wearing two hats. Trainium — now on its third generation — is built around large-scale distributed training: AWS’s Trainium2 press materials describe a Trn2 instance packing 16 chips for 20.8 peak petaflops of compute, and a Trn2 UltraServer connecting four Trn2 servers (64 chips total, via NeuronLink) for 83.2 peak petaflops, explicitly aimed at training and deploying very large models — AWS cites Project Rainier, an UltraCluster built with Anthropic using hundreds of thousands of Trainium2 chips, as the extreme end of that scale. Notably, AWS’s own Trainium3 messaging describes the chip as delivering “the best economics for high performance AI training and inference at scale,” meaning the newest Trainium generations are increasingly capable inference platforms too — the line between “training chip” and “inference chip” has blurred somewhat as Trainium has matured, even though Inferentia remains the dedicated, inference-only product.
| Inferentia2 | Trainium2 | |
|---|---|---|
| Primary design target | Inference serving at low cost/latency | Large-scale distributed training (increasingly also inference) |
| Scale unit | Up to 12 chips per Inf2 instance | 16 chips per Trn2 instance; 64 chips per UltraServer |
| Published compute figure | Up to 190 TFLOPS FP16 per chip | 20.8 peak petaflops per Trn2 instance (16 chips) |
| Published memory figure | 32 GB HBM per chip | Not published by AWS at the per-chip level |
| AWS price-performance claim | Up to 70% lower cost per inference (Inf1) vs. comparable EC2 | 30–40% better price-performance vs. EC2 P5e/P5en GPU instances |
One gap worth flagging plainly: AWS’s own Trainium2 materials do not publish a per-chip HBM memory capacity figure, so this article leaves that cell blank rather than repeating a third-party estimate that isn’t AWS’s own number.
The Cost Claims — And Whose Numbers They Are
AWS’s own benchmarked claims (70% lower cost per inference for Inf1, 50% better performance per watt for Inf2) are AWS’s figures, run against AWS’s own comparable EC2 instances. Beyond those, AWS’s Inferentia page cites a long list of customer-reported results that are not AWS benchmarks and should be read as such: Leonardo.ai and Finch AI each report roughly 80% cost reduction moving to Inferentia-based inference; NTT PC reports 90% lower cost versus current-generation GPU instances; Dataminr reports up to 9x better throughput per dollar; Tomofun reports an 83% lower deployment cost for a specific model (BLIP); Screening Eagle reports a 50% cost reduction; and Metagenomi reports up to 56% lower cost for protein design workloads. These are real customer statements AWS chose to publish, but they reflect specific workloads and migration paths, not a guaranteed savings rate for any given deployment.
Who Should Care
If your workload is primarily serving already-trained models — especially large transformer or generative AI models at high request volume — Inferentia2 and Inf2 instances are built specifically for that cost/latency profile, and AWS’s own throughput-per-dollar claims are a reasonable starting point for a proof-of-concept comparison against GPU instances. If you’re training models from scratch or doing heavy fine-tuning at scale, Trainium is AWS’s intended chip, not Inferentia — Inferentia’s smaller per-chip memory and core count reflect that it was never meant to hold the full state needed for large-scale backpropagation across huge models. Teams already committed to the Neuron SDK get to move between Inferentia and Trainium instance types with a shared tooling stack, which lowers the switching cost of trying both for a given workload.
Bottom Line
Inferentia and Trainium are genuinely different chips solving different problems, not a training/inference relabeling of the same silicon — Inferentia2’s smaller, cheaper-per-chip design with 32 GB of HBM and a focus on scale-out distributed inference is purpose-built for serving cost, while Trainium’s larger per-instance compute pools target training throughput (and, increasingly, inference at extreme scale). As of this writing, AWS has not announced an “Inferentia3,” so Inferentia2 remains the company’s current dedicated inference chip even as newer Trainium generations pick up more of the inference workload.
FAQ
Is there an Inferentia3?
Not as of this writing. AWS’s most recent custom-silicon announcements (Trainium3) are for the training line; AWS has not officially announced a third-generation Inferentia chip.
What’s the core difference between Inferentia2 and Trainium2?
Inferentia2 is built for low-cost, low-latency inference serving with 32 GB of HBM and up to 190 TFLOPS of FP16 per chip; Trainium2 is built for large-scale distributed training, with AWS citing 20.8 peak petaflops across 16 chips in a single Trn2 instance.
Can Trainium chips be used for inference?
Yes — AWS’s own messaging for Trainium3 describes it as delivering “the best economics for high performance AI training and inference at scale,” so the newer Trainium generations aren’t training-only in practice, even though Inferentia remains AWS’s dedicated inference product.
How many Inferentia2 chips fit in one instance?
Inf2 instances scale up to 12 Inferentia2 chips per instance, according to AWS.
Are AWS’s cost-savings claims independently verified?
The headline throughput and performance-per-watt figures are AWS’s own benchmarks against comparable EC2 instances. The larger percentage savings figures (80-90%) come from individual customer case studies that AWS publishes, not from AWS’s internal benchmarking methodology, and reflect specific workloads rather than a universal guarantee.
Do Inferentia and Trainium share the same software?
Yes — both use AWS’s Neuron SDK, which is part of why AWS can position them as a coherent custom-silicon family even though the chips themselves are architecturally distinct.
Sources
- AWS Inferentia — Official Product Page
- AWS Trainium — Official Product Page
- AWS Trainium2 Instances Now Generally Available (Amazon Press Release)
- Amazon EC2 Inf2 Instances GA Announcement (AWS Blog)
Discover more from Software Online Review
Subscribe to get the latest posts sent to your email.
