The Inference Stack

From the overarching software to the physical connections, every layer of AI infrastructure is being redesigned for performance.

NVIDIA’s GTC 2026 produced the expected headline: the Vera Rubin platform, seven new chips, and a $1 trillion demand forecast through 2027. A set of quieter announcements across software, silicon, physical interconnects, and cloud partnerships shared a single architectural driver.

Inference has displaced training as the primary force reshaping AI infrastructure. The systems being built to serve it at scale require different orchestration software, different silicon, different physical wiring, and a different market structure to deliver them. All four are now under construction.

A dedicated OS

A training run processes a known model against a known dataset for a predetermined duration, and the operator configures the cluster once before the job begins. Inference requests arrive in unpredictable bursts that vary in size, modality, and latency sensitivity. Agentic systems compound the variability by chaining multiple inference calls in rapid succession, each dependent on the output of the last. The infrastructure built for training treated GPU clusters as batch-processing farms. Inference requires those clusters to function as responsive services that allocate resources dynamically across competing demands.

NVIDIA released Dynamo 1.0 at GTC last week as open-source inference orchestration software. Jensen Huang called it the first operating system for AI factories. Dynamo splits inference into its two constituent phases and routes each to hardware that matches its demands. Prefill, which processes the incoming prompt in parallel, runs on GPUs optimized for compute throughput. Decode, which generates output tokens one at a time, runs on hardware optimized for memory bandwidth. Beyond phase-level routing, Dynamo manages the cluster as a unified resource, directing requests to GPUs that already hold relevant context in memory, offloading cached data to cheaper storage tiers during idle periods, and reallocating GPUs dynamically as request volumes shift.

Every major cloud provider has integrated Dynamo. AWS, Azure, Google Cloud, and OCI run it in production, alongside AI-native companies like Cursor and Perplexity and enterprises including PayPal, Pinterest, and ByteDance. NVIDIA claims that Dynamo boosts Blackwell inference performance by up to 7x, a gain derived entirely from open-source software extracting more throughput from GPUs already deployed in the field.

$20 billion in bandwidth

Dynamo’s disaggregation of prefill and decode revealed a hardware constraint. Once the two phases run on separate GPUs, each phase’s resource demands become visible in isolation. Prefill is compute-bound and massively parallel, the kind of workload for which GPUs were designed. Decode generates tokens sequentially, streaming the model’s full active parameters through memory on every token. Memory bandwidth constrains this phase. A Rubin GPU delivers 22 TB/s of HBM bandwidth, and decode-heavy agentic workloads can still saturate it.

The Groq 3 LPU, unveiled at GTC, addresses the decode constraint with a different class of memory. Each LPU carries 500 MB of on-chip SRAM instead of external HBM. That capacity amounts to a fraction of a Rubin GPU’s 288 GB, but SRAM delivers 150 TB/s of memory bandwidth, seven times the Rubin’s 22 TB/s. Decode performance hinges on how quickly parameters stream through the processor, and by that measure, the LPU outperforms the GPU sevenfold. The LPU pairs this bandwidth with a deterministic execution model. A compiler schedules every data movement and computation in advance, eliminating the latency jitter that arises from dynamic scheduling in conventional GPU hardware.

NVIDIA acquired Groq’s intellectual property and senior engineering staff for $20 billion on Christmas Eve 2025, its largest deal on record. The Groq 3 LPU emerged barely three months later, built on Groq’s second-generation architecture with minor modifications and manufactured by Samsung. The Register’s analysis confirmed the time-to-market logic. The chip lacks NVLink support, has no NVFP4 hardware, and ships without CUDA compatibility. These absences suggest that NVIDIA purchased an existing design rather than waiting to develop equivalent silicon in-house. The acquisition follows the template that NVIDIA’s 2019 Mellanox deal established, in which an acquired company’s technology became a permanent architectural layer of the platform. Groq’s LPU appears to have displaced NVIDIA’s own Rubin CPX inference accelerator, which received no mention during Huang’s keynote.

The Groq 3 LPX rack assembles 256 LPUs into a liquid-cooled system with 128 GB of aggregate SRAM and 40 PB/s of aggregate bandwidth. The rack sits physically beside a Vera Rubin NVL72 GPU rack in the data center. Dynamo orchestrates the division of labor between them. Rubin GPUs handle prefill and attention computation during decode, while the LPX rack handles the latency-sensitive feed-forward and mixture-of-experts operations, with each processor type serving the phase that matches its architectural strengths. NVIDIA claims that the combination delivers 35x higher inference throughput per megawatt than Blackwell NVL72 alone. The inference rack now contains two fundamentally different processor architectures, coordinated by software, serving a single stream of requests.

The optical floor

GPUs, LPUs, CPUs, and high-radix switches share a rack and communicate constantly within it. That communication runs over copper. At 200+ Gb/s per lane, copper’s reach collapses to under a meter, the power that signal amplification and retiming require grows punitive, and the limited physical space on chip edges for electrical pins caps how much I/O a processor can carry. Engineers call this last constraint the shoreline bottleneck. Interconnects consume a growing share of total data center energy as operators push copper beyond its comfortable range, and the heterogeneous inference racks now taking shape demand more bandwidth than copper can deliver.

AMD, Broadcom, Meta, Microsoft, NVIDIA, and OpenAI announced the Optical Compute Interconnect Multi-Source Agreement earlier this month, forming a consortium to define an open optical specification for scale-up interconnects inside AI racks. The specification’s central architectural choice separates the communication protocol from the physical transmission medium. NVIDIA’s NVLink and AMD’s UALink can both run over the same optical infrastructure without requiring distinct physical layers. The consortium’s founding members are the hyperscalers and chip companies that build the world’s largest AI clusters. They defined the specification around their own deployment constraints, lending the effort an urgency that traditional vendor-led standards bodies rarely match.

The OCI roadmap starts at 200 Gb/s per direction in Gen1, using four wavelengths of NRZ modulation, and scales to 800 Gb/s per fiber in Gen2, with a target of 3.2 Tb/s per fiber in later generations. The deeper architectural shift moves optical conversion from external pluggable modules to co-packaged optics, in which the optical transceiver sits inside the processor package itself. This change eliminates the electrical-to-optical boundary as a separate bottleneck. Early estimates suggest that power consumption per link drops from roughly 30 watts to roughly 9 watts through tighter silicon integration, a reduction that could let optical interconnects match copper’s energy profile while providing bandwidth and reach that copper cannot.

Specialized workloads

The specialization cascading through software, silicon, and physical interconnects has a market consequence. Two years ago, choosing an AI cloud provider meant picking among three hyperscalers. In 2026, The New Stack identifies at least six distinct categories of AI cloud provider, each serving a different workload profile.

Hyperscalers offer ecosystem depth. If an organization’s identity management runs through Entra ID or its analytics live in BigQuery, the friction of moving GPU workloads elsewhere is substantial. That depth comes at a premium in per-GPU pricing and provisioning time. Neoclouds like CoreWeave have emerged as the breakout category, offering pure GPU density with faster deployment. CoreWeave’s contracted revenue backlog reached $66.8 billion by the end of 2025, and NVIDIA invested $2 billion in the company in January 2026 to accelerate its buildout. Below the hyperscalers and neoclouds sit inference endpoint providers, sovereign cloud operators, and further specialized categories, each viable because inference demands differ enough in latency tolerance, batch size, context length, and cost sensitivity to sustain distinct infrastructure configurations.

Training workloads, for all their scale, impose relatively uniform infrastructure demands. Large batches run for extended periods at maximum throughput, and a provider that handles one training run well handles most of them. Inference workloads span a range from high-throughput batch processing to sub-millisecond agentic interactions, and the hardware configurations that serve each end of that spectrum differ architecturally. The cloud providers that serve them will remain distinct as well. Deloitte’s 2026 TMT Predictions estimated that inference will account for roughly two-thirds of AI compute spending. As that share grows, the infrastructure diversity required to serve it will widen the market fragmentation rather than resolve it.

A new era

Read as a single system, the GTC announcements describe a stack redesigned from first principles around inference economics. Dynamo provides the orchestration layer. The Groq LPU adds decode-optimized silicon beside GPUs in the same rack. OCI MSA defines the optical interconnect that these heterogeneous racks demand. The cloud market has fragmented into specialized categories to serve the workload diversity that inference creates, and each layer depends on the others.

Training-era infrastructure rewarded a single design goal: maximize throughput on large, predictable batch jobs, and the companies that scaled fastest won. Inference-era infrastructure rewards latency, efficiency, heterogeneity, and dynamic resource allocation simultaneously. The competitive landscape will increasingly reflect that broader set of demands.