Compute

A developer’s guide to architecting reliable GPU infrastructure at scale

Thu, 09 Apr 2026 22:00:00 +0000

Editor’s note: This blog post outlines Google Cloud’s GPU AI/ML infrastructure reliability strategy, and will be updated with links to new community articles as they appear.

As we enter the era of multi-trillion parameter models, computational power has transitioned from a utility to a mission-critical strategic asset. To meet relentless training demand, organizations are no longer just building clusters — they are engineering massive, integrated compute ecosystems comprising hundreds of thousands of high-performance accelerators that are interconnected with an ultra-high-bandwidth networking backplane. At this unprecedented scale, raw performance thrives when it is built upon a foundation of systemic resilience.

In "always-on" mission-critical environments, the statistical probability of hardware variance becomes a primary constraint for reliability. When thousands of GPUs are operating at peak utilization for months at a time, a 0.01% performance fluctuation can trigger a systemic failure. The cost of training interruptions now measured in millions of dollars and weeks of lost progress, the industry's focus has shifted. The true frontier of training isn't just about the size of the cluster — it’s about the resilient system architecture that is able to power the next generation of AI workloads.

The core challenge for the industry goes beyond simple hardware fixes; it requires the creation of holistic software and infrastructure frameworks designed to withstand the inevitable disruptions of massive-scale computing. In an environment where AI/ML infrastructure represents a major capital expenditure on a company's balance sheet, partnering with a cloud provider that places a premium on infrastructure reliability is paramount.

Operational realities of AI at scale

The construction of a supercomputer utilizing hundreds of thousands of advanced GPUs involves significant operational complexity. Maintaining optimal utilization over several months to train a single large language Model (LLM) subjects the hardware to high levels of sustained performance that exceed the design parameters of conventional data center equipment. The advent of rackscale GPU architectures, such as the NVIDIA GB200 NVL72 and NVIDIA GB300 NVL72, has shifted the landscape. Considerations now extend beyond individual machines to encompass entire domains, impacting multiple interconnected trays with the potential to require coordinated management for AI/ML workloads to avoid disruptions.

The business implications of infrastructure instability

For organizations at the forefront of AI innovation, infrastructure reliability poses a significant commercial risk with substantial economic consequences.

High cost of failure: A single failure in a massive training job requires restarting from the last checkpoint, wiping out days or even weeks of progress. When infrastructure spend is a big capex, every failure counts.
Delayed time-to-market: In the fast-moving AI space, being first matters. Every day spent debugging hardware failures is a day delaying releasing new models while competitors are getting ahead. Reliability issues can directly slow down model iteration cycles, delaying product launches and feature updates.
Operational complexities: Manually managing a large GPU cluster is a resource-intensive task. Companies come to the cloud to reduce the cost of managing the infrastructure. Without systemic reliability investments, operations teams can get overwhelmed by a constant stream of alerts, forced to play "whack-a-mole" to identify, isolate, and replace faulty nodes thus affecting their time spent on planning for the future capacity and model demands.
Expensive workarounds to mitigate failure impact: To achieve a certain level of performance and Goodput, companies can end up needing to buy 10-20% more hardware than they actually need as a buffer.

Quantitative assessment: Key reliability metrics

Beyond traditional uptime measurements, the primary metrics Google Cloud uses to measure AI infrastructure health and stability are MTBI and Goodput.

Mean Time Between Interruption (MTBI): The average time a system runs before encountering an interruption. Includes instance terminations as well as every customer workload interruption that our systems can observe (example GPU XIDs).
Goodput: The amount of useful computational work completed per unit time.

Google Cloud’s methodology: Engineering systemic resilience

The objective has shifted from expecting total hardware perfection to engineering systems that demonstrate inherent resilience. We understand that trust in our infrastructure begins with reliability. Our approach is based on four principles:

Proactive prevention: We’ve integrated hardware validation, real-time telemetry, and automated remediation throughout the infrastructure lifecycle. This systemic approach to shift from reactive troubleshooting to proactive management optimizes the reliability of mission-critical GPUs systems at scale.
Continuous monitoring and intelligent detection: We have transformed raw data into actionable insights by synthesizing multi-layered telemetry through automated analysis, to proactively identify and resolve anomalies. This data-driven approach shifts our infrastructure from reactive maintenance to an intelligent, self-healing system that helps ensure continuous workload stability.
Transparency and control: We empower users with full visibility and control over GPU infrastructure health. We provide a comprehensive suite of observability metrics and direct tools, allowing customers to correlate hardware status with their workload Goodput and report faults.
Minimizing disruptions: Our control plane integrates smart scheduling with predictive health signals to enable improved workload migration via maintenance notifications. If unexpected issues arise, customers can enable automated remediations and fast recovery mechanisms to initiate rapid restoration of service.

We have covered an in-depth journey into these principles in our technical deep-dive post linked below. We are launching a comprehensive technical deep dive series to explore Google’s approach towards AI/ML infrastructure reliability for Google Cloud GPUs further. Check back here as we add links to learn about:

Proactive prevention: Inside Google Cloud's multi-layered GPU qualification process
Transparency and Control : Providing Operational Transparency and Management tools to Mitigate GPU Workload Impact (Coming Soon)
Continuous monitoring and intelligent detection: Using ML to predict and prevent GPU downtime (coming soon)
Minimizing disruptions: Smart scheduling and fast recovery systems for mission-critical GPU clusters (coming soon)

AI infrastructure efficiency: Ironwood TPUs deliver 3.7x carbon efficiency gains

Mon, 06 Apr 2026 16:00:00 +0000

At Google, we are committed to being transparent about the environmental impact of our AI infrastructure, publishing metrics on the lifetime emissions of our chips — from manufacturing to powering these chips in the data center. Today, we are updating these metrics for our seventh-generation TPU, Ironwood, which demonstrates an approximately 3.7x improvement in Compute Carbon Intensity (CCI) compared to TPU v5p, the previous generation of performance-optimized TPUs.¹

In other words, despite the fact that AI is driving demand for additional compute resources, our ongoing work to optimize AI hardware is helping to improve the energy consumption and emissions of AI workloads.

Measuring AI accelerator efficiency: Compute Carbon Intensity (CCI)

To help manage the environmental impact of AI workloads, we monitor the Compute Carbon Intensity (CCI) of our AI accelerator hardware. CCI is defined in An Introduction to Life-Cycle Emissions of Artificial Intelligence Hardware²as the estimated amount of CO2 equivalent emitted for every utilized floating-point operation (CO2e/FLOP). This metric provides a holistic, chip-level view by including both the embodied emissions associated with manufacturing, transportation, and data center construction (Scope 3), as well as the operational emissions associated with running these chips in data centers (Scope 1 and 2).

The Ironwood advantage: high performance, low footprint

Google’s TPU CCI continues to improve with each chip generation. Drawing from empirical data measured in January 2026, Ironwood demonstrates a remarkable 3.7x improvement in CCI relative to TPU v5p. This accelerates efficiency gains from the 1.2x CCI improvement of TPU v5p relative to TPU v4, and demonstrates continued carbon efficiency optimization of Google’s performance-optimized TPU architecture.

These efficiency gains are driven by outsized compute performance increases between TPU generations relative to growth in machine energy consumption and manufacturing emissions. In fact, fleetwide measurements demonstrate a 5x improvement in utilized FLOPs across generations, from TPU v5p to Ironwood.³ Because the performance denominator in our CCI equation (CO2e/FLOP) is scaling faster than emissions, the net carbon cost per operation drops significantly with every new chip.

^{Figure 1: Ironwood’s accelerating CCI improvement measured on Google’s performance-optimized TPU cohort, considering January 2026 workloads.4}

Operating Google’s TPU fleet more efficiently

Updated TPU CCI metrics also offer a direct comparison to the measurement we published in 2025. Specifically, from October 2024 to January 2026, Google’s versatile TPU cohort ran more efficiently than what we reported previously:

TPU v5e achieved a 43% reduction in total CCI over 15 months, dropping to 228 gCO2e/EFLOP. This was driven by a 72% increase in average utilization.
Trillium, the sixth-generation TPU, saw a 20% reduction in total CCI over the same time period, bringing its emissions intensity down to 125 gCO2e/EFLOP.

^{Figure 2: Google’s versatile TPU cohort demonstrates deployment efficiency gains for the same TPU generations between October 2024 and January 2026.5}

These results demonstrate that Google continues to improve the carbon-efficiency of our AI infrastructure. While the massive scale of AI demand requires a significant and growing amount of power, our innovations allow us to deliver substantially more compute performance for every unit of energy consumed.

Decoupling energy and emissions from performance

To what can we attribute these improvements? Beyond Ironwood’s raw hardware capabilities, these CCI gains are further enabled by deep software and system-level optimizations across our infrastructure:

Software efficiency (MoE): The widespread adoption of sparse architectures, such as Mixture of Experts (MoE), routes computation only to necessary parameters. This drastically reduces the active FLOPs required per inference or training step without sacrificing model capacity or quality.
Lower precision math (FP8): By heavily leveraging 8-bit floating-point (FP8) formats, we effectively double compute throughput and halve memory bandwidth requirements compared to 16-bit formats. This shows that we can maintain output quality while exponentially decreasing the energy cost per mathematical operation.
Workload mix and intelligent scheduling: Advanced fleet orchestration continuously balances the workload mix across our infrastructure. By intelligently scheduling tasks, we ensure high continuous utilization rates, optimize duty cycles, and minimize the carbon penalty of idle power draw.

Scale sustainably with Google Cloud

AI’s trajectory requires infrastructure that can scale exponentially without an equivalent surge in carbon emissions. The 3.7x carbon efficiency improvement from TPU v5p to Ironwood demonstrates that we can achieve greater compute density while minimizing the growth of our energy and environmental footprint through deliberate hardware and software codesign. To learn more and get started with Ironwood, register your interest with this form.

_{1. Following the methodology published in an August 2025 technical report, we quantified the full lifecycle emissions of TPU hardware as a point-in-time snapshot across Google’s generations of TPUs as of January 2026. The functional unit for this study is one AI computer deployed in the data center, which includes one or more accelerator trays (containing TPUs) connected to one host tray (i.e., a computing server). Peripheral components beyond the tray (e.g., rack, shelf, and network equipment) and auxiliary computing and storage resources are excluded from the calculation of embodied and operational emissions. We include the electricity used in data center cooling in operational emissions. To estimate operational emissions from electricity consumption of running workloads, we used a one month sample of observed machine power data from our entire TPU fleet, applying Google’s 2024 average fleetwide carbon intensity. To estimate embodied emissions from manufacturing, transportation, and retirement, we performed a life-cycle assessment of the hardware. Data center construction emissions were estimated based on Google’s disclosed 2024 carbon footprint. These findings do not represent model-level emissions, nor are they a complete quantification of Google’s AI emissions. Based on the TPU location of a specific workload, CCI results of specific workloads may vary.
2. The authors would like to thank and acknowledge the co-authors of this paper for their important contributions to enable these results: Ian Schneider, Hui Xu, Stephan Benecke, Parthasarathy Ranganathan, and Cooper Elsworth.
3. This comparison considers the utilized FLOPS (BF16) between deployed TPU v5p and Ironwood chips in Google’s fleet in January 2026. This trend is consistent with the improvement in peak FLOPS (BF16) between v5p (459 FLOPS) and Ironwood (2,307 FLOPS).
4.The GHG protocol offers two accounting standards for operational emissions. Results presented here consider market-based emissions, which includes the impact of carbon-free energy purchases. Location-based accounting, which excludes carbon-free energy purchases, would raise operational CCI to 793, 712, and 195 gCO2e/EFLOP, respectively. The ratio of CCI improvements would be at a similar level, and Ironwood’s embodied CCI would drop from 23% to 8% of its total CCI.
5. To ensure a fair comparison across varying TPU utilizations, this analysis replicates the propensity score weighting methodology from the August 2025 technical report and compares January 2026 results to the results published in 2025. This statistical technique adjusts for duty cycle variations to balance the comparison of TPUs during a given time period. This empirical methodology results in small variations in calculated CCI between temporal periods, reflecting fluctuations in real-world energy consumption and hardware utilization across the global infrastructure.}

A developer’s guide to training with Ironwood TPUs

Mon, 23 Mar 2026 16:00:00 +0000

The transition toward trillion-parameter AI models has created an exponential demand for computational resources, testing the limits of traditional infrastructure. The seventh-generation Ironwood TPU features Google’s custom-designed AI infrastructure: It is engineered to scale as a holistic system supporting pods of up to 9,216 chips by combining Inter-Chip Interconnect (ICI), Optical Circuit Switch (OCS), Data Center Network (DCN) and massive aggregated High Bandwidth Memory (HBM) capacity. In addition, Ironwood features an integrated co-design between hardware architecture and software, introducing innovations such as compiler-centric XLA and Python-native kernels via Pallas. Together, these features significantly scale organizations’ capacity to train and serve sophisticated frontier models, optimize the entire AI lifecycle and enable sustained high performance.

This technical overview explores the specific methods and tools within the JAX and MaxText ecosystems designed to refine training efficiency and reach peak performance on Ironwood hardware.

Key optimization strategies for Ironwood

1. Leverage native FP8 with MaxText

Ironwood is the first TPU generation with native 8-bit floating point (FP8) support in its Matrix Multiply Units (MXUs). By utilizing FP8 precision for weights, activations, and gradients, users can theoretically double throughput compared to Brain Floating Point 16 (BF16). When FP8 recipes are configured correctly, increased efficiency is achievable without compromising model quality.

To implement these FP8 training recipes, users can start with the Qwix library. This functionality is enabled by specifying the relevant flags within the MaxText configuration.

See our blog post, Inside the optimization of FP8 training on Ironwood, in the Google Developer forums for more details.

2. Accelerate with Tokamax kernels

Tokamax is a library of high-performance JAX kernels optimized for TPUs. These kernels are designed to mitigate specific bottlenecks through the following mechanisms:

Splash Attention: This mechanism addresses the I/O limitations inherent in standard attention processes. By maintaining computations within on-chip SRAM, it is particularly effective for processing long context lengths where memory bandwidth typically becomes a constraint.
Megablox Grouped Matrix Multiplication (GMM): This manages the “ragged” tensors (data structures with inconsistent row lengths that typically create hardware idle time) often found in Mixture of Experts (MoE) models. By utilizing GMM, the system avoids inefficient padding and ensures higher utilization of the MXU.
Kernel tuning: The Tokamax library includes Utilities for hyperparameter optimization. These tools allow for the adjustment of tile sizes and other configurations to align with the specific memory hierarchy of the Ironwood TPU.

3. Offload collectives to SparseCore

The fourth-generation SparseCores in Ironwood are processors specifically designed to manage irregular memory access patterns. By using specific XLA flags, users can offload collective communication operations—such as All-Gather and Reduce-Scatter—directly to the SparseCore.

This offloading mechanism allows the TensorCores to remain dedicated to primary model computations while communication tasks execute in parallel. This functional overlap is a critical strategy for hiding communication latency and ensuring consistent data throughput to the MXUs.

4. Fine-tune the memory pipeline on VMEM

VMEM, a critical part of the TPU memory architecture, is a fast on-chip SRAM that is designed to optimize kernel performance. You can improve the overall speed of execution by tuning the allocation of VMEM between current operation and future weight prefetch. For example, increasing the VMEM reserved for the current scope allows increasing the tile sizes used by the kernel, which can increase kernel performance by removing potential memory stalls.

Refer to TPU Pipelining for more on TPU memory architecture.

5. Choose optimal sharding strategies

Lastly, MaxText supports various parallelism techniques which are available on all TPUs. The best choice depends on model size, architecture (Dense vs. MoE), and sequence length. Selecting a proper sharding strategy can improve the performance of the model:

Fully Sharded Data Parallelism (FSDP): This is the preferred strategy for training large models that exceed the memory capacity of a single chip. FSDP shards model weights, gradients, and optimizer states across multiple chips. Increasing the per-device batch size and introducing more compute can hide the latency of the All-Gather operations and improve efficiency.
Tensor Parallelism (TP): Shards individual tensors. Given Ironwood's high arithmetic intensity, TP is most effective for very large model dimensions. Leveraging TP with a dimension of 2 can take advantage of the fast die-to-die interconnect on Ironwood's dual-chiplet design.
Expert Parallelism (EP): Helpful for MoE models to distribute experts across devices.
Context Parallelism (CP): Necessary for very long sequences, sharding activations along the sequence dimension.
Hybrid approaches: Combining strategies is often required to balance compute, memory, and communication on large-scale runs.

See the Optimizing Frontier Model Training on TPU v7x Ironwood post in the Developer forums for more detail on techniques 2-5 above.

The Ironwood advantage: System-level performance

These optimization techniques, coupled with Ironwood's architectural strengths like the high-speed 3D Torus Inter-Chip Interconnect (ICI) and massive HBM capacity, create a highly performant platform for training frontier models. The tight co-design across hardware, compilers (XLA), and frameworks (JAX, MaxText) ensures you can extract maximum performance from your AI Infrastructure.

Ready to accelerate your AI journey? Explore the resources below to dive deeper into each optimization method.

Further reading

_{A special thanks to Hina Jajoo and Amanda Liang for their contributions to this blog post.}

Google Cloud and NVIDIA expand AI innovation across industries at GTC 2026

Mon, 16 Mar 2026 16:00:00 +0000

The era of agentic AI is fundamentally changing enterprise infrastructure needs. As organizations build systems capable of dynamic reasoning and autonomous execution, the underlying infrastructure must evolve as well. Scaling these agentic workloads alongside massive mixture-of-experts (MoE) architectures demands a deeply optimized co-engineered stack.

To meet these demands, we’ve built the Google Cloud AI Hypercomputer, an AI-optimized infrastructure as a service, that integrates performance-optimized hardware, leading software, open frameworks, and flexible consumption models into a single, cohesive system to deliver ultra-low latency, high-throughput, and cost-effective inference. To give our customers even more options within this integrated architecture, we are expanding our partnership with NVIDIA.

This week at NVIDIA GTC 2026, Google Cloud and NVIDIA are expanding our partnership with a wave of new announcements, showcasing a co-engineered AI infrastructure foundation:

Infrastructure and hardware

Strong momentum for Google Cloud G4 VMs, powered by NVIDIA RTX PRO 6000 Blackwell Server Edition
Preview of flexible, fractional G4 VMs using NVIDIA vGPU technology — a first in the industry for NVIDIA RTX PRO 6000 Blackwell Server Edition
Upcoming support for NVIDIA Vera Rubin NVL72 Platform

Software and platform

NVIDIA Dynamo integration with GKE Inference Gateway
Enhanced NVIDIA support across Vertex AI Training and Model Garden

Ecosystem

Kaggle competition for NVIDIA Nemotron on G4 VMs
Launch of a dedicated public sector AI startup accelerator program

Let’s take a closer look at the announcements.

Accelerating AI workloads with G4 VMs

G4 VMs, powered by NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs, are built to power a diverse spectrum of high-performance workloads — from advanced spatial computing to complete AI development lifecycles. For instance, companies like Otto Group One.O and WPP use the G4 to run physically accurate simulations and real-time 3D rendering at scale.

Beyond simulation, the G4 also shines in model fine-tuning and inference, particularly for models ranging from 30B to more than 100B parameters. By leveraging 4-bit floating point (FP4) precision and Google’s peer-to-peer (P2P) communication, customers are achieving higher throughput for model serving and considerable latency reductions, enabling a new class of real-time, multimodal AI agents and highly responsive generative AI applications.

Here are some examples of how customers are already leveraging the performance and efficiency of G4 VMs to accelerate their most demanding workloads:

“Google Cloud’s G4 VMs give us the scalable GPU backbone we need to push billions of miles of photorealistic simulation through our pipeline. The 4x lift in throughput means our ML teams can iterate faster, train on richer data, and validate edge cases long before our models ever see the real world.” – Sony Mohapatra, Director, AI/ML Engineering, General Motors

“Now with G4 VMs powered by NVIDIA Blackwell, we're pushing our multimodal models even further — faster inference, better reliability, instant replies across languages. The goal stays the same: making voice agents that work at enterprise scale without compromise. We are excited to keep building together and see what our customers deploy with this.” – Mati Staniszewski, Cofounder, ElevenLabs

“Google Cloud G4 VMs provide the computational backbone for our Robotic Coordination Layer, allowing us to synchronize autonomous fleets across our logistics centers with millisecond precision. By simulating complex warehouse environments in a high-fidelity digital twin, we can optimize our entire supply chain virtually before a single robot moves on the floor.” – Dr. Stefan Borsutzky, CEO of Otto Group One.O

“After transitioning to G4 VMs, we achieved a 50% reduction in processing latency and 6x increase in throughput just by updating our Terraform scripts. It’s rare to get that kind of performance boost for our core workloads without adding any operational overhead.” – Alfonso Acosta, Head of Engineering, Imgix

Introducing fractional G4 VMs

We are excited to announce the preview of fractional G4 VMs, providing a highly efficient and cost-effective entry point for AI and graphics workloads. These new configurations, using NVIDIA virtual GPU (vGPU) technology, allow you to leverage the power of the NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs in flexible, smaller increments, so you can right-size your infrastructure to match the specific demands of your applications.

“Enterprises need unprecedented flexibility to scale complex, agentic AI workloads. With Google Cloud, we’re introducing fractional G4 VMs powered by NVIDIA RTX PRO 6000 to let customers right‑size GPU capacity and maximize ROI. Together with our co‑engineered stack – from NVIDIA NeMo on Vertex AI to NVIDIA Dynamo with GKE – we’re delivering an open, high‑performance platform for next‑generation reasoning and MoE models.” – Ian Buck, VP / General Manager, Hyperscale and HPC, NVIDIA

By providing more granular access to advanced hardware, fractional G4 VMs let you optimize resource allocation and reduce overhead without sacrificing performance. You can now select from additional GPU slice sizes for your specific needs:

1/2 GPU: Ideal for more intensive tasks such as LLM inference, robotics sensor simulation, and high-fidelity 3D rendering.
1/4 GPU: Optimized for mainstream workloads, including mid-range creative design, video transcoding, and real-time data visualization.
1/8 GPU: Great for lightweight applications such as remote desktops, productivity tools, and entry-level streaming services.

These flexible G4 size portfolio let you:

Right-size infrastructure: Precisely match GPU capacity to application demands, ranging from lightweight remote desktops to intensive data processing.
Maximize cost efficiency: Lower operational overhead by utilizing — and paying for — only the fractional GPU resources you need for specific tasks.
Scale diverse workloads: Power a broad spectrum of innovation, from high-fidelity creative design and streaming to complex robotics simulations and real-time inference.

These fractional G4 VMs can be managed by Google Kubernetes Engine (GKE), allowing developers to use advanced container binpacking to achieve even higher price-performance and resource utilization. When managed through Dynamic Workload Scheduler, you can set fallback priorities for fractional slices. This significantly improves obtainability by allowing the scheduler to automatically find available GPU configurations for each workload.

“The G4 vGPU’s flexible sizing allows us to precisely tailor compute resources to the scale of each molecular simulation, ensuring maximum efficiency across our drug discovery pipeline. This granular control means our researchers can seamlessly pivot between smaller workflows and massive parallel processing without being constrained by fixed hardware configurations.” – Shane Brauner, EVP, CIO, Schrödinger

Scaling AI Hypercomputer with NVIDIA Vera Rubin NVL72

Building on our deep engineering partnership with NVIDIA, we’re proud to support the successor to NVIDIA Blackwell architecture, the recently announced NVIDIA Vera Rubin platform. We plan to be among the first cloud providers to offer NVIDIA Vera Rubin NVL72 rack-scale systems in the second half of 2026, integrating them into our AI Hypercomputer architecture to empower the next generation of reasoning and agentic AI.

Delivering efficiency across the AI infrastructure stack

As part of our commitment to a fully open ecosystem, we are excited to announce the integration of Dynamo and GKE Inference Gateway. This integration provides a modular, open-source control plane across the application layer and the hardware. By combining Dynamo with Inference Gateway on GKE, teams can tailor their infrastructure to their exact needs, allowing them to extract the maximum ROI from accelerators, accelerate time-to-market for new AI models, and future-proof their deployments.

You can learn to maximize performance for massive MoE architectures through new advanced scaling recipes for A4X VMs (powered by NVIDIA GB200 NVL72 and Dynamo). These configurations show how to overcome memory and interconnect bottlenecks when running AI inference workloads on AI Hypercomputer.

We are also enhancing resource obtainability through the Dynamic Workload Scheduler, with Calendar Mode and Flex Start for A4X and A4X Max (powered by NVIDIA GB300 NVL72), as well as new Flex Start support for G4 VMs. Dynamic Workload Scheduler lets you reserve the precise capacity that you need, or use flexible start windows.

Snap, a long-time Google Cloud customer, achieved significant cost savings by migrating two of its primary data processing pipelines to Google Cloud G2 VMs powered by NVIDIA L4 Tensor Core GPUs. This was made possible by leveraging Spark on GKE alongside NVIDIA’s new cuDF libraries, which automated the optimization of its shuffle-heavy workloads for optimal GPU efficiency. Learn more at GTC session S81678.

Advancing Vertex AI training and Model Garden

We are meeting the demands of next-generation AI with two major infrastructure advancements to Vertex AI training clusters. First, support for A4X VM domains lets you leverage Vertex AI’s managed infrastructure and framework capabilities for massive-scale training on NVIDIA GB200 NVL72 rack-scale systems. To ensure these intensive workloads remain uninterrupted, new hardware resiliency capabilities let you apply configurable, proactive fault detection scans, which identify and mitigate potential hardware issues before they can disrupt critical “hero” training runs. These capabilities enable higher goodput and helps ensure that multi-week training jobs stay on track without costly restarts.

“We are setting a new standard for the agentic enterprise — delivering highly capable, consistent, accurate, and responsive AI agents with Google and NVIDIA. By leveraging Vertex AI training clusters on NVIDIA GB200 NVL72 to power our Agentforce 360 Platform, we’ve eliminated infrastructure bottlenecks to keep our GPUs fully saturated. This high-performance, resilient architecture allows our researchers to focus on innovation at scale, driving substantial gains for our most complex reasoning workloads.” - Silvio Savarese, Chief Scientist, Salesforce

At the same time, we continue to broaden Vertex AI Model Garden with support for NVIDIA’s Nemotron 3 family of open models. These include the Nemotron 3 Nano, featuring one-click deployment to simplify integration into private VPCs. We’ve also expanded our catalog to include the NVIDIA Nemotron 3 Super 120B model for immediate access to high-performance, large-scale reasoning. To maximize the value of these models, we’ve integrated NVIDIA’s latest performance libraries directly into Vertex AI to optimize popular open-source models on NVIDIA TensorRT-LLM.

To enable the community to get hands-on with NVIDIA Nemotron on Google Cloud, we are also launching the NVIDIA Nemotron model reasoning challenge on Kaggle, powered by G4 VMs. The competition invites the community to improve Nemotron 3 Nano’s reasoning accuracy on a new benchmark using techniques such as prompting, synthetic data generation, data curation, and fine-tuning – all running on cost-efficient G4 infrastructure so participants can iterate quickly and share their methods with the broader ecosystem. To learn more and register, visit the Kaggle competition page.

Empowering public sector AI startups

To foster continued innovation within the ecosystem, Google Public Sector and NVIDIA are launching an AI startup accelerator program. This year-long initiative will support a select cohort of AI-focused Independent Software Vendors (ISVs) building solutions for the public sector.

Participants gain dual access to both NVIDIA Inception and Google Cloud’s ISV accelerator resources. Kicking off at GTC and continuing through Google Cloud Next, this joint program will equip emerging technology leaders with the co-engineered infrastructure, technical guidance, and go-to-market support required to scale mission-critical public sector applications. To learn more about the program, please complete the interest form. Additional cohorts will be selected and announced in the future.

Co-engineering collaboration powers every layer of the AI stack

The transition to complex, agentic AI demands more than just raw compute. It requires a fully optimized, co-engineered stack. By integrating flexible hardware like fractional G4 instances and the upcoming Vera Rubin platform into our AI Hypercomputer architecture, and pairing it with deep software co-engineering, we provide the scale, resilience, and efficiency you need to turn your most ambitious AI visions into reality.

Coming to GTC? Stop by booth #513 to learn more and talk to our team. And you can always learn more about our collaboration with NVIDIA at cloud.google.com/NVIDIA.

H4D VMs, now GA, deliver exceptional performance and scaling for HPC workloads

Wed, 04 Mar 2026 17:00:00 +0000

Today, we’re announcing the general availability of H4D VMs, our latest high performance computing (HPC)-optimized VM, powered by the 5th Generation AMD EPYC™ processors. H4D VMs deliver exceptional performance, scalability, and value for industries like manufacturing, health care and life sciences, weather forecasting, and electronic design automation (EDA). H4D supports orchestration via Cluster Toolkit with Slurm and via Google Kubernetes Engine (GKE). Each approach allows for near-instant deployment and scaling of demanding workloads.

For the first time, the Google Cloud CPU portfolio features a VM family with Cloud Remote Direct Memory Access (RDMA). H4D’s RDMA is on the Titanium network adapter and lets you scale single-node H4D performance to multiple nodes, accelerating large production workloads.

Faster time to solution across domains and scales

Powered by the high core density of the 5th Gen AMD EPYC CPU and Google’s innovative, low-latency Falcon hardware transport, H4D VMs enable you to iterate and discover faster than ever before.

We demonstrated H4D performance through a series of industry-standard benchmarks, showing its capabilities across diverse domains and problem sizes.

Healthcare and life sciences
For researchers in healthcare and life sciences (HCLS), H4D VMs accelerate complex molecular simulations critical to scientific discovery. Compared to our previous C2D VMs, H4D VMs deliver up to a 4.3X speedup running LAMMPs (LJ benchmark) at 96 VMs, delivering 95% parallel efficiency on 18k cores. For drug discovery, we demonstrated a 5.8X speed-up using GROMACS (water_33m) at 32 VMs delivering 72% parallel efficiency on 6k cores. H4D also delivers further scalability, which we demonstrated by running the LAMMPS LJ benchmark on 192 VMs (~37k cores) while maintaining 92% parallel efficiency (see Figure 3).

Manufacturing
For manufacturing, H4D VMs help engineers shorten design cycles, run larger simulations, and iterate faster by delivering a strong performance boost for mission-critical Computer-Aided Engineering (CAE) workflows. Compared to our previous C2D VMs when running complex Computational Fluid Dynamics (CFD) simulations, H4D VMs deliver a 4.1X speedup running Ansys Fluent (F1_RaceCar_140m benchmark) on 32 VMs with 85% parallel efficiency. When running open-source OpenFOAM (Motorbike_100m), we demonstrated a 5.2X speedup over C2D using 16 VMs and achieving superlinear parallel efficiency of 122%.

A new standard for HPC price/performance

H4D VMs are designed to deliver the best price-performance for HPC workloads on Google Cloud by pairing superior performance with flexible consumption models. H4D supports Dynamic Workload Scheduler (DWS), which adapts to your workflow with Flex Start mode for just-in-time capacity and Calendar mode for guaranteed reservations. This allows you to access compute for as low as 3 cents per core-hour without long-term commitments. The resulting performance and cost efficiencies over previous generation VMs are detailed in Figures 6 and 7.

Comprehensive HPC management

To manage and deploy large, dense clusters of H4D VMs, you can leverage Google Cloud’s Cluster Director, which offers advanced maintenance capabilities (you can sign up for the preview here) alongside the Cluster Toolkit for rapid cluster deployment via turnkey system blueprints. For job and workload management, H4D VMs integrate with Batch, Google Cloud’s fully managed, cloud-native service that handles queuing, scheduling, and resource provisioning. Additionally, there’s support for DWS, which can be used in both Calendar mode for future reservations and Flex Start mode for time-limited, on-demand usage.

What customers and partners are saying

“We were able to test the H4D platform in early access at Jump Trading, and were extremely impressed with the results. The successful testing process demonstrated that H4D offers the performance, stability, and efficiency we require for demanding, high-volume operations. We see up to 50% better price/performance compared to prior generation machines and are now accelerating integration with our critical grid workloads on Google Cloud." - Alex Davies, Chief Technology Officer & Benjamin Stromski, HPC Linux Engineering, Jump Trading

“There lingers, especially in large-scale and compute-intensive domains, the idea that the fastest systems can only be built on premises and run on bare metal hardware. Terms such as ‘hypervisor tax” are often thrown around as justification for operating with bare metal. Our testing paints a different picture. The Google H4D VM performs better on our financial risk benchmark than the bare metal top of stack AMD CPU of the same generation." - Hamza Mian/CEO, HMxLabs

"As a leading provider of managed HPC solutions for the demanding CAE and manufacturing sectors, our evaluation of the H4D platform was focused heavily on its ability to handle our clients' largest, most tightly-coupled simulation workloads. We are extremely impressed with the results. The testing confirmed that the underlying RDMA fabric exhibits the outstanding low-latency and high-bandwidth performance required for massive parallel processing. This level of interconnect efficiency is non-negotiable for speeding up critical manufacturing simulations like crash testing and CFD. H4D has proven itself to be a true accelerator for high-throughput engineering workloads, and we are excited about its potential to redefine the performance ceiling for HPC in the engineering world." - Rodney Mach/President, TotalCAE

“The new H4D instances are a significant step forward for our demanding next-generation TPU simulation workloads. We've seen a 30% performance improvement across a variety of EDA benchmarks compared to C2D, demonstrating the strong single core performance of H4D. This directly translates to faster development cycles and allows our engineering teams to iterate more quickly” - Trevor Switkowski, Technical Lead of Chip Design Methodology, Google Cloud

Experience H4D today

H4D is now available in us-central1-a (Iowa), europe-west4-b (Netherlands) and asia-southeast1-a (Singapore) with additional regions coming soon. Check regional availability on our Regions and Zones page and deploy your most demanding HPC workloads by leveraging Cloud RDMA.

_{The following configurations were run for the above benchmarks:LAMMPS version 20250722, GROMACS: version 2023.1, OpenFOAM version 2312, Ansys Fluent version 2024R1. All runs used IntelMPI 2021.17.2. C2D/C3D/C4D used TCP, H4D used RDMA with RXM & SAR_LIMIT=2G. All runs used full ppn (processes-per-node) available on each platform (56, 180, 192 for C2D, C3D and C4D/H4D respectively). Ansys Fluent runs used 168ppn on H4D and variable ppn for C4D. SMT off for all. Cost comparision across single nodes of H4D-highmem-192 with DWS Flex Start price, c3d-standard-360 and c2d-standard-112 OD price.}

_{Parallel efficiency and optimal node count depend on input size and communication patterns, and therefore vary across workloads.}

Simpler billing, clearer savings: A FinOps guide to updated spend-based CUDs

Thu, 12 Feb 2026 17:00:00 +0000

Optimizing cloud spend is one of the most rewarding aspects of FinOps — and committed use discounts (CUDs) remain one of the most effective levers to pull.

In July 2025, we began rolling out updates to the spend-based CUD model to make it easier to understand your costs and savings, expand coverage to new SKUs (including Cloud Run and H3/M-series VMs), and offer increased flexibility. These changes are now available to all customers. Let’s dive into how this new model simplifies your FinOps practice.

1. What is the spend-based CUD data change all about?

The most important shift is the move from a credit-based system to a direct discounted price model using consumption models.

Under the old credits model, you committed to an hourly on-demand amount. To find your savings (the actual cost reduction realized), you had to use three different numbers: the full on-demand cost, the commitment fee, and the offsetting credit.

1. The old math:

$10.00 (On-demand) + $5.50 (Commitment fee) - $10.00 (Credit) = $5.50 (Net Cost)
Savings = $10.00 (On-demand) - $5.50 (Net costs) = $4.50

With the new direct discount model, you don’t need to do that math to calculate your net costs. You commit directly to the net, discounted spend amount. Your usage is simply billed at that discounted rate.

2. The new math:

$5.50 (Discounted costs)
Savings = $10.00 (On-demand) - $5.50 (Discounted costs) = $4.50

You can now see your net cost at a glance, and calculating the savings only requires comparing the on-demand price ($10.00) to your new discounted cost ($5.50), which equals $4.50/hr.

2. How do I validate my savings before and after the changes?

The unified CUD Analysis tool is your best resource for auditing the migration or performing deep-dives on your spend. CUD Analysis for the new spend-based CUD model allows you to quickly verify the savings you are getting with the new model, and you can use this tool to compare that the savings didn’t change between the old and the new model.

You can validate your savings by following these steps:

1. Identify the date when the migration took place; you can see the migration date in the billing overview page.

2. Go to CUD Analysis to validate the savings before and after the migration.

3. To quantify costs from before the migration:

Filter the view for one day before the migration, in this case Oct. 26, 2025.
Select a CUD Product, for example Cloud SQL CUD.
In our example, we paid a $50.35 CUD fee to get a $69.12 credit. When you subtract that fee from the credit, your actual take-home savings were $18.77.

4. To validate costs after the migration

Change the date to Oct. 28, 2025
Under the new model, you pay the discounted rates upfront. Your dashboard will reflect a Net Cost of $50.35, compared to the $69.12 on-demand cost, clearly showing your $18.77 in savings.

In addition, this release also includes an update to Cost Reports to include “Savings Programs,” which accurately reflects your actual net savings ($18.77 in our example above), rather than gross credit. When comparing pre- and post-migration data in Cost Reports, ensure you include both usage SKUs and commitment fee SKUs to capture the full scope of the commitment.

3. What other capabilities are in the new CUD Analysis?

Beyond support for the new model, the new CUD Analysis tool offers deeper visibility into your CUD coverage and CUD utilization. You can now analyze your CUDs with hourly data granularity for up to 30 days. This is a major improvement for FinOps teams, as daily averages often hide underutilization spikes that occur during specific hours.

CUD Analysis: Compute Flexible CUD coverage analysis

CUD Analysis: Per CUD purchase utilization visibility

If you want to use your own data analysis tools, we offer a new spend-based CUD metadata export that lets you manage your spend-based CUDs programmatically. You can use this export to join with the Billing BigQuery Export datasets to run in-depth, programmatic analysis on all your commitment data. You can also export a CSV from the CUD Analysis view to see the raw data for every resource and its price without needing the full BigQuery export.

4. How much commitment should I buy?

Our CUD recommendations are the primary tool for determining how much of a commitment to purchase. We recently enhanced our Compute Flexible CUD commitment recommendations to provide greater accuracy by including data from GKE, Cloud Run, Cloud Run Functions, and Compute Engine. Additionally, CUD scenario modeling allows you to adjust these suggestions in real-time. You can adjust coverage thresholds, filter out specific dates with irregular usage, or extend the lookback analysis window up to 180 days to identify the exact commitment level that aligns with your specific risk profile.

CUD scenario modeling: experiment with multiple options to identify your ideal CUD strategy

5. Is there anything else I should know about Flex CUDs?

With the release of the new spend-based model, we’ve addressed the reporting limitation affecting customers who use a combination of Flex CUDs and GKE/Cloud Run CUDs. Previously, our analysis tools were unable to accurately identify the source of specific credits, leading to discrepancies in KPI metrics like savings, coverage, and utilization. Under the new spend-based CUD model, this limitation has been corrected, so your CUD analysis now provides an accurate, granular view of your savings per Google Cloud service.

To begin navigating the updated spend-based model, visit the Billing console. You can learn more in our documentation:

High-performance inference meets serverless compute with NVIDIA RTX PRO 6000 on Cloud Run

Mon, 02 Feb 2026 17:00:00 +0000

Running large-scale inference models can involve significant operational toil, including cluster management and manual VM maintenance. One solution is to leverage a serverless compute platform to abstract away the underlying infrastructure. Today, we’re bringing the serverless experience to high-end inference with support for NVIDIA RTX PRO™ 6000 Blackwell Server Edition GPUs on Cloud Run. Now in preview, you can deploy massive models like Gemma 3 27B or Llama 3.1 70B with the 'deploy and forget' experience you’ve come to expect from Cloud Run. No reservations. No cluster management. Just code.

A powerful GPU platform

The NVIDIA RTX PRO 6000 Blackwell GPU provides a huge leap in performance compared to the NVIDIA L4 GPU, bringing 96GB vGPU memory, 1.6 TB/s of bandwidth and support for FP4 and FP6. This means you can serve up to 70B+ parameter models without having to manage any underlying infrastructure. Cloud Run lets you attach a NVIDIA RTX PRO 6000 Blackwell GPU to your Cloud Run service, job, or worker pools, on demand, with no reservations required. Here are some ways you can use the NVIDIA RTX PRO 6000 Blackwell GPU to accelerate your business:

Generative AI and inference: With its FP4 precision support, the NVIDIA RTX PRO 6000 Blackwell GPU’s high-efficiency compute accelerates LLM fine-tuning and inference, letting you create real-time generative AI applications such as multi-modal and text-to-image creation models. By running your model on Cloud Run services, you can also take advantage of rapid startup and scaling, going from zero instances to having a GPU with drivers installed under 5 seconds. When traffic eventually scales down zero and no more requests are being received, Cloud Run automatically scales your GPU instances down to zero.
Fine-tuning and offline inference: NVIDIA RTX PRO 6000 Blackwell GPUs can be used in conjunction with Cloud Run jobs to fine-tune your model. The fifth-generation NVIDIA Tensor Cores can be used in conjunction with AI models to help accelerate rendering pipelines and enhance content creation.
Tailored scaling for specialized workloads: Use GPU-enabled worker pools to apply granular control over your GPU workers, whether you need to dynamically scale based on custom external metrics or manually provision "always-on" instances for complex, stateful processing.

We built Cloud Run to be the simplest way to run production-ready, GPU-accelerated tasks. Some highlights of Cloud Run include:

Managed GPUs with flexible compute: Cloud Run pre-installs the necessary NVIDIA drivers so you can focus on your code. Cloud Run instances using NVIDIA RTX PRO 6000 Blackwell GPUs can configure up to 44 vCPU and 176GB of RAM.
Production-grade reliability: By default, Cloud Run offers zonal redundancy, helping to ensure enough capacity for your service to be resilient to a zonal outage; this also applies to Cloud Run with GPUs. Alternatively, you can turn off zonal redundancy and benefit from a lower price for best-effort failover of your GPU workloads in case of a zonal outage.
Tight integration: Cloud Run works natively with the rest of Google Cloud. You can load massive model weights by mounting Cloud Storage buckets as local volumes, or use Identity-Aware Proxy (IAP) to secure traffic that’s bound for a Cloud Run service.

Get started

The NVIDIA RTX PRO 6000 Blackwell GPU is available in preview on demand with availability in us-central1 and europe-west4, and limited availability in asia-south2 and asia-southeast1. You can deploy your first service using Ollama, one of the easiest way to run open models, on Cloud Run with NVIDIA RTX PRO 6000 GPUs enabled:

code_block: <ListValue: [StructValue([('code', 'gcloud beta run deploy my-service \\\r\n--image ollama/ollama --port 11434 \\\r\n--cpu 20 --memory 80Gi \\\r\n--gpu-type nvidia-rtx-pro-6000 \\\r\n--no-gpu-zonal-redundancy \\\r\n--region us-central1'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fa0a1b63070>)])]>

For more details, check out our updated Cloud Run documentation and AI inference best practices.

Unlock 2x better price-performance with Axion-based N4A VMs, now generally available

Tue, 27 Jan 2026 17:00:00 +0000

January 27, 2026: The N4A is now generally available. You can get started by deploying N4A from the Google Cloud console.

Decision makers and builders today face a constant challenge: managing rising cloud costs while delivering the performance their customers demand. As applications evolve to use scale-out microservices and handle ever-growing data volumes, organizations need maximum efficiency from their underlying infrastructure to support their growing general-purpose workloads.

To meet this need, we’re excited to announce our latest Axion-based virtual machine series: N4A, available in preview on Compute Engine, Google Kubernetes Engine (GKE), Dataproc, and Batch, with support in Dataflow and other services coming soon.

N4A is the most cost-effective N-series VM to date, delivering up to 2x better price-performance and 80% better performance-per-watt than comparable current-generation x86-based VMs. This makes it easier for customers to further optimize the Total Cost of Ownership (TCO) for a broad range of general-purpose workloads. We see this with cloud-native businesses running scale-out web servers and microservices on GKE, enterprise teams managing backend application servers and mid-sized databases, and engineering organizations operating large CI/CD build farms.

At Google Cloud, we co-design our compute offerings with storage, networking and software at every layer of the stack, from orchestrators to runtimes, to deliver exceptional system-level performance and cost-efficiency. N4A’s breakthrough price-performance is powered by our latest-generation Google Axion Processors, built on the Arm® Neoverse® N3 compute core, Google Dynamic Resource Management (DRM) technology, and Titanium, Google Cloud’s custom-designed hardware and software system that offloads networking and storage processing to free up the CPU. Titanium is part of Google Cloud’s vertically integrated software stack — from the custom silicon in our servers to our planet-scale network traversing 7.75 million kilometers of terrestrial and subsea fiber across 42 regions — that is engineered to maximize efficiency and provide the ultra-low latency and high bandwidth to customers at global scale.

Redefining general-purpose compute and enabling AI inference

N4A is engineered for versatility, with a feature set to support your general-purpose and CPU-based AI workloads. It comes in predefined and custom shapes, with up to 64 vCPUs and 512GB of DDR5 in high-cpu (2GB of memory per vCPU), standard (4GB per vCPU), and high-memory (8GB per vCPU) configurations, with instance networking up to 50 Gbps of bandwidth. N4A VMs feature support for our latest generation Hyperdisk storage options, including Hyperdisk Balanced, Hyperdisk Throughput, and Hyperdisk ML (coming later), providing up to 160K IOPS, 2.4GB/s of throughput per instance.

N4A performs well across a range of industry-standard benchmarks that represent the key workloads our customers run every day. For example, relative to comparable current-generation x86-based VM offerings, N4A delivers up to 105% better price-performance for compute-bound workloads, up to 90% better price-performance for scale-out web servers, up to 85% better price-performance for Java applications, and up to 20% better price-performance for general-purpose databases.

Footnote: As of October 2025. Performance based on the estimated SPECrate®2017_int_base, estimated SPECjbb2015, MySQL Transactions/minute (RO), and Google internal Nginx Reverse Proxy benchmark scores run in production on comparable latest-generation generally-available VMs with general purpose storage types. Price-performance claims based on published and upcoming list prices for Google Cloud.

In the real world, early adopters are seeing dramatic price-performance improvements from the new N4A instances.

"At ZoomInfo, we operate a massive data intelligence platform where efficiency is paramount. Our core data processing pipelines, which are critical for delivering timely insights to our customers, run extensively on Dataflow and Java services in GKE. In our preview of the new N4A instances, we measured a 60% improvement in price-performance for these key workloads compared to their x86-based counterparts. This allows us to scale our platform more efficiently and deliver more value to our customers, faster." - Sergei Koren, Chief Infrastructure Architect, ZoomInfo

“Organizations today need performance, efficiency, flexibility, and scale to meet the computing demands of the AI era; this requires the close collaboration and co-design that is at the heart of our partnership with Google Cloud. As N4A redefines cost-efficiency, customers gain a new level of infrastructure optimization, enabling enterprises to choose the right infrastructure for their workload requirements with Arm and Google Cloud.” - Bhumik Patel, Director, Server Ecosystem Development, Infrastructure Business, Arm

Granular control with Custom Machine Types and Hyperdisk

A key advantage of our N-series VMs has always been flexibility, and with N4A, we are bringing one of our most popular features to the Axion family for the first time: Custom Machine Types (CMT). Instead of fitting your workload into a predefined shape, CMTs on N4A lets you independently configure the amount of vCPU and memory to meet your application's unique needs. This ability to right-size your instances means you pay only for the resources you use, minimizing waste and optimizing your total cost of ownership.

This same principle of matching resources to your specific workload applies to storage. N4A VMs feature support for our latest generation of Hyperdisk, allowing you to select the perfect storage profile for your application's needs:

Hyperdisk Balanced: Offers an optimal mix of performance and cost for the majority of general-purpose workloads, with up to 160K IOPs per N4A VM.
Hyperdisk Throughput: Delivers up to 2.4GiBps of max throughput for bandwidth-intensive analytics workloads like Hadoop or Kafka, providing high-capacity storage at an excellent value.
Hyperdisk ML (post GA): Purpose-built for AI/ML workloads, allows you to attach a single disk containing your model weights or datasets to up to 32 N4A instances simultaneously for large-scale inference or training tasks.
Hyperdisk Storage Pools: Instead of provisioning capacity and performance on a per-volume basis, allows you to provision performance and capacity in aggregate, further optimizing costs by up to 50% and simplifying management.

"At Vimeo, we have long relied on Custom Machine Types to efficiently manage our massive video transcoding platform. Our initial tests on the new Axion-based N4A instances have been very compelling, unlocking a new level of efficiency. We've observed a 30% improvement in performance for our core transcoding workload compared to comparable x86 VMs. This points to a clear path for improving our unit economics and scaling our services more profitably, without changing our operational model." - Joe Peled, Sr. Director of Hosting & Delivery Ops, Vimeo

A growing Arm-based Axion portfolio for customer choice

C-series VMs are designed for workloads that require consistently high performance, e.g., medium-to-large-scale databases and in-memory caches. Alongside them, N-series VMs have been a key Compute Engine pillar, offering a balance of price-performance and flexibility, lowering the cost of running workloads with variable resource needs such as scale-out Java/GKE workloads. We released our first Axion-based machine series, C4A, in October 2024, and the introduction of N4A complements C4A, providing a range of Google Axion instances suited to your workloads’ precise needs.

On top of that, GKE unlocks significant price-performance advantages by orchestrating Axion-based C4A and N4A machine types. GKE leverages Custom Compute Classes to provision and mix these machine types, matching workloads to the right hardware. This automated, heterogeneous cluster management allows teams to optimize their total cost of ownership across their entire application stack.

Also joining the Axion family is C4A.metal, Google Cloud’s first Axion bare metal instance that helps builders meet use cases that require access to the underlying physical server to run specialized applications in a non-virtualized environment, such as automotive systems development, workloads with strict licensing requirements, and Android software development. C4A.metal will be available in preview soon.

Supported by the broad and mature Arm ecosystem, adopting Axion is easier than ever, and the combination of C4A and N4A can help you lower the total cost of running your business, without compromising on performance or workload-specific requirements:

N4A for cost optimization and flexibility. Deliberately engineered for general-purpose workloads that need a balance of price and performance, including scale-out web servers, microservices, containerized applications, open-source databases, batch, data analytics, development environments, data preparation and AI/ML experimentation.
C4A for consistently high performance, predictability, and control. Powering workloads where every microsecond counts, such as medium- to large-scale databases, in-memory caches, cost-effective AI/ML inference, and high-traffic gaming servers. C4A delivers consistent performance, offering a controlled maintenance experience for mission-critical workloads, networking bandwidth up to 100 Gbps, and next-generation Titanium Local SSD storage.

"Migrating to Google Cloud's Axion portfolio gave us a critical competitive advantage. We slashed our compute consumption by 20% while maintaining low and stable latency with C4A instances, such as our Supply-Side Platform (SSP) backend service. Additionally, C4A enabled us to leverage Hyperdisk with precisely the IOPS we need for our stateful workloads, regardless of instance size. This flexibility gives us the best of both worlds - allowing us to win more ad auctions for our clients while significantly improving our margins. We're now testing the N4A family by running some of our key workloads that require the most flexibility, such as our API relay service. We are happy to share that several applications running in production are consuming 15% less CPU compared to our previous infrastructure, reducing our costs further, while ensuring that the right instance backs the workload characteristics required.” - Or Ben Dahan, Cloud & Software Architect at Rise

Get started with N4A today

N4A is available in the following Google Cloud regions: us-central1 (Iowa), us-east4 (Virginia), us-east1 (South Carolina), us-west1 (Oregon), asia-southeast1 (Singapore), europe-west1 (Belgium), europe-west2 (London), europe-west3 (Frankfurt) and europe-west4 (Netherlands) with more regions to follow. Learn more about N4A here in documentation; deploy N4A here in the console.

Scaling WideEP Mixture-of-Experts inference with Google Cloud A4X (GB200) and NVIDIA Dynamo

Thu, 22 Jan 2026 17:00:00 +0000

As organizations transition from standard LLMs to massive Mixture-of-Experts (MoE) architectures like DeepSeek-R1, the primary constraint has shifted from raw compute density to communication latency and memory bandwidth. Today, we’re releasing two new validated recipes designed to help customers overcome the infrastructure bottlenecks of the agentic AI era. These new recipes provide clear steps to optimize both throughput and latency built on the A4X machine series powered by NVIDIA GB200 NVL72 and NVIDIA Dynamo, which extend the reference architecture we published in September 2025 for disaggregated inference on A3 Ultra (NVIDIA H200) VMs.

We’re bringing the best of both worlds to AI infrastructure by combining the multi-layered scalability of Google Cloud’s AI infrastructure with the rack-scale acceleration of the A4X. These recipes are part of a broader collaboration between our organizations that includes investments in important inference infrastructure like Dynamic Resource Allocation (DRA) and Inference Gateway.

Highlights of the updated reference architecture include:

Infrastructure: Google Cloud’s A4X machine series, powered by NVIDIA GB200 NVL72, creating a single 72-GPU compute domain connected with fifth-generation NVIDIA NVLink.
Serving architecture: NVIDIA Dynamo functions as the distributed runtime, managing KV cache state and kernel scheduling across the rack-scale fabric.
Performance: For 8K/1K input sequence length (ISL)/ output sequence length (OSL) , we achieved over 6K total tokens/sec/GPU in throughput-optimized configurations and 10ms inter-token latency (ITL) in latency-optimized configurations.
Deployment: Two new recipes are available today in this repo for deploying this stack on Google Cloud using Google Kubernetes Engine (GKE) for orchestration.

The modern inference stack

To achieve exascale performance, inference cannot be treated as a monolithic workload. It requires a modular architecture where every layer is optimized for specific throughput and latency targets. The AI Hypercomputer inference stack consists of three distinct layers:

Infrastructure layer: The physical compute, networking, and storage fabric (e.g., A4X).
Serving layer: The specific model architecture and the optimized execution kernels (e.g., NVIDIA Dynamo, NVIDIA TensorRT-LLM, Pax) and runtime environment managing request scheduling, KV cache state, and distributed coordination.
Orchestration layer: The control plane for resource lifecycle management, scaling, and fault tolerance (e.g., Kubernetes).

In the reference architecture detailed below, we focus on a specific, high-performance instantiation of this stack designed for the NVIDIA ecosystem. We combine the A4X at the infrastructure layer with NVIDIA Dynamo at the model serving Layer, orchestrated by GKE.

Infrastructure layer: The A4X rack-scale architecture

In our A4X launch announcement in February 2025 we referenced how the A4X VM addressed bandwidth constraints by implementing the GB200 NVL72 architecture, which fundamentally alters the topology available to the scheduler.

Unlike previous generations where NVLink domains were bound by the server chassis (typically 8 GPUs), the A4X exposes a unified fabric, with:

72 NVIDIA Blackwell GPUs interconnected via the NVLink Switch System that enables the 72 GPUs to operate as one giant GPU with unified shared memory
130TB/s aggregate bandwidth, enabling all-to-all communication with latency profiles comparable to on-board memory access (72 GPUs x 1.8 TB/s/GPU)
Native NVFP4 support: Blackwell Tensor Cores support 4-bit floating point precision, effectively doubling throughput relative to FP8 for compatible model layers. We used FP8 Precision Scaling for this benchmark to support configuration and comparison with previously published results.

Serving layer: NVIDIA Dynamo

Hardware of this scale requires a runtime capable of managing distributed state without introducing synchronization overhead. NVIDIA Dynamo serves as this distributed inference runtime. It moves beyond simple model serving to coordinate the complex lifecycle of inference requests across the underlying infrastructure.

The serving layer optimizes utilization on the A4X through these specific mechanisms:

Wide Expert Parallelism (WideEP): Traditional MoE serving shards experts within a single node (typically 8 GPUs), leading to load imbalances when specific experts become "hot." We use the A4X's unified fabric to distribute experts across the full 72-GPU rack. This WideEP configuration absorbs bursty expert activation patterns by balancing the load across a massive compute pool, helping to ensure that no single GPU becomes a straggler.
Deep Expert Parallelism (DeepEP): While WideEP distributes the experts, DeepEP optimizes the critical "dispatch" and "combine" communication phases. DeepEP accelerates the high-bandwidth all-to-all operations required to route tokens to their assigned experts. This approach minimizes the synchronization overhead that typically bottlenecks MoE inference at scale.
Disaggregated request processing: Dynamo decouples the compute-bound prefill phase from the memory-bound decode phase. On the A4X, this allows the scheduler to allocate specific GPU groups within the rack to prefill (maximizing tensor core saturation) while other GPUs handle decode (maximizing memory bandwidth utilization), preventing resource contention.
Global KV cache management: Dynamo maintains a global view of the KV cache state. Its routing logic directs requests to the specific GPU holding the relevant context, minimizing redundant computation and cache migration.
JIT kernel optimization: The runtime leverages NVIDIA Blackwell-specific kernels, performing just-in-time fusion of operations to reduce memory-access overhead during the generation phase.

Orchestration layer: Mapping software to hardware

While the A4X provides the physical fabric and Dynamo provides the runtime logic, the orchestration layer is responsible for mapping the software requirements to the hardware topology. For rack-scale architectures like the GB200 NVL72, container orchestration needs to evolve beyond standard scheduling. By making the orchestrator explicitly aware of the physical NVLink domains, we can fully unlock the platform’s performance and help ensure optimal workload placement.

GKE enforces this hardware-software alignment through these specific mechanisms:

1. Rack-level atomic scheduling: With GB200 NVL72, the "unit of compute" is no longer a single GPU or a single node — the entire rack is the new fundamental building block of accelerated computing. We use GKE capacity reservations with specific affinity settings. This targets a reserved block of A4X infrastructure that guarantees dense deployment. By consuming this reservation, GKE helps ensure that all pods comprising a Dynamo instance land on the specific, physically contiguous rack hardware required to establish the NVLink domain, providing the hard topology guarantee needed for WideEP and DeepEP.

2. Low-latency model loading via GCS FUSE: Serving massive MoE models requires loading terabytes of weights into high-bandwidth memory (HBM). Traditional approaches that download weights to local disk incur unacceptable "cold start" latencies. We leverage the GCS FUSE CSI Driver to mount model weights directly from Google Cloud Storage as a local file system. This allows the Dynamo runtime to "lazy load" the model, streaming data chunks directly into GPU memory on demand. This approach eliminates the pre-download phase, significantly reducing the time-to-ready for new inference replicas and enabling faster auto-scaling in response to traffic bursts.

3. Kernel-bypass networking (GPUDirect RDMA): To maximize the aggregate 130 TB/s bandwidth of the A4X, the networking stack must minimize CPU and I/O involvement. We configure the GKE cluster to enable GPUDirect RDMA over the Titanium network adapter. By injecting specific NCCL topology configurations and enabling IPC_LOCK capabilities in the container, we allow the application to bypass the OS kernel and perform Direct Memory Access (DMA) operations between the GPU and the network interface. This configuration offloads the NVIDIA Grace CPU from data path management, so that networking I/O does not become a bottleneck during high-throughput token generation.

Performance validation

We observed the following when assessing the scaling characteristics of an 8K/1K workload on DeepSeek-R1 (FP8) with SGLang for two distinct optimization targets.

1. Throughput-optimized configuration

Setup: All 72 GPUs utilizing DeepEP. 10 prefill nodes with 5 workers (TP8) and 8 decode nodes with 1 worker (TP32).
Result: We sustained over 6K total tokens/sec/GPU (1.5K output tokens/sec/GPU), which matches the performance published by InferenceMAX (source).

2. Latency-optimized configuration

Setup: 8 GPUs (two nodes) without DeepEP. 1 prefill node with 1 prefill worker (TP4) and 1 decode node with 1 decode worker (TP4).
Result: We sustained a median Inter-Token Latency (ITL) of 10ms at a concurrency of 4, which matches the performance published by InferenceMAX (source).

Looking ahead

As models evolve from static chat interfaces to complex, multi-turn reasoning agents, the requirements for inference infrastructure will continue to shift. We are actively updating and releasing benchmarks and recipes as we invest across all three layers of the AI inference stack to meet these demands:

Infrastructure layer: The recently released A4X Max is based on the NVIDIA GB300 NVL72 in a single 72 GPU rack configuration, bringing 1.5X more NVFP4 FLOPs, 1.5X more GPU memory, and 2X higher network bandwidth compared to A4X.
Serving layer: We are actively exploring deeper integrations with components of NVIDIA Dynamo, e.g., pairing KV Block Manager with Google Cloud remote storage, funneling Dynamo metrics into our Cloud Monitoring dashboards for enhanced observability, and leveraging GKE Custom Compute Classes (CCC) for better capacity and obtainability, as well as setting a new baseline with FP4 precision.
Orchestration: We plan to incorporate additional optimizations into these tests, e.g. Inference Gateway as the intelligent inference scheduling component, following the design patterns established in the llm-d well-lit paths. We aim to provide a centralized mechanism for sophisticated traffic orchestration — handling request prioritization, queuing, and multi-model routing before the workload ever reaches the serving-layer runtime.

Whether you are deploying massive MoE models or architecting the next generation of reasoning agents, this stack provides the exascale foundation required to turn frontier research into production reality.

Get started today

At Google Cloud, we’re committed to providing the most open, flexible, and performant infrastructure for your AI workloads. With full support for the NVIDIA Dynamo suite — from intelligent routing and scaling to the latest NVIDIA AI infrastructure — we provide a complete, production-ready solution for serving LLMs at scale.

We updated our deployment repository with two specific recipes for the A4X machine class:

Recipe for throughput optimized - 72 GPUs with DeepEP
Recipe for latency optimized - 8 GPUs without DeepEP

We look forward to seeing what you build!

Simplify VM OS agent management at scale: Introducing VM Extensions Manager

Mon, 05 Jan 2026 17:00:00 +0000

If you're an IT administrator, you know that managing Operating System (OS) agents (Google calls them extensions) across a large fleet of VM instances can be complex and frustrating. Indeed, this operational overhead can be a major barrier to adopting extension-based services on VM fleets, despite the fact that they unlock powerful application-level capabilities.

To solve this problem, we’re excited to announce the preview of VM Extensions Manager, a new capability integrated directly into the Compute Engine API that simplifies installing and managing these Google-provided extensions.

VM Extensions Manager provides a centralized, policy-driven framework for managing the entire lifecycle of Google Cloud extensions on your VM instances. Instead of relying on manual scripts, startup scripts, or other bespoke solutions, you can now define a policy to ensure all your VM instances — both existing and new — conform to that state, reducing operational overhead from months to hours.

How to get started with VM Extensions Manager

VM Extensions Manager is integrated directly into the compute.googleapis.com API, meaning there are no new APIs to discover or enable. You can get started in minutes.

1. Define your extension policy
First, define a policy that specifies the desired state of your extensions.

For the preview, you can create zonal policies at the Project level. This policy targets VM instances within a single, specific zone.

Over the coming months, we’ll expand support to include global policies, as well as policies at the Organization and Folder levels. This will allow you to build a flexible hierarchy of policies (using priorities) to manage your extension on your enterprise fleet from a single control plane.

You can create this policy directly from the Google Cloud console:

Demo of Creating VM Extension policy using Cloud Console

2. Select your extensions
In the policy, you select the Google Cloud extensions you want to manage. For the preview, VM Extensions Manager supports several critical Google Cloud extensions, including:

Cloud Ops Agent (ops-agent): The Ops Agent is the primary agent for collecting telemetry from your Compute Engine instances.
Agent for SAP (sap-extension): Google Cloud's Agent for SAP is provided by Google Cloud for the support and monitoring of SAP workloads running on Compute Engine instances and Bare Metal Solution servers.
Agent for Compute Workload (workload-extension): The Agent for Compute Workloads lets you monitor and evaluate workloads running on Compute Engine.

We'll be adding support for more extension-based services in the coming months.

You can choose to pin a specific extension version or, keep it empty (the default) to get the latest extension installed. If you choose the default, VM Extensions Manager automatically handles the rollout of new versions as they are released — no more waiting to access new features and improvements.

3. Roll out global policy with more control
VM Extensions Manager gives you control over how global policy changes are deployed across many zones with rollout speeds. Zonal policies don't offer rollout speeds; they are enforced instantaneously when the VMs are online.

In coming weeks, we will expand support for global policy via gcloud first and update the documentation with relevant information. UI updates will follow in coming months.

At preview, however, global policy lets you select two distinct rollout speeds:

SLOW (Recommended): This is the default option, designed for safety. It orchestrates a zone-by-zone rollout (within the scope of the policy) with a built-in wait time between waves, minimizing the potential blast radius of a problematic change over a period of time, by default 5 days. This is perfect for standard maintenance and updates.
FAST: This option eliminates the wait time between waves, executing the change across the entire fleet across zones as quickly as possible. It is intended for urgent use cases, such as deploying a critical security patch in a "break-glass" emergency scenario across all VMs in all zones.

Once you save the policy, VM Extensions Manager takes over. The underlying progressive rollout engine manages the complex orchestration, and you can monitor its progress.

A flexible system for standardization and control

VM Extensions Manager is designed to bring standardization and control to extensions on your VM fleets. You can start today by applying zonal policies to your projects to ensure extensions are correctly installed on VM instances in the correct zones.

To get started defining Extension policies for your Compute Engine VM instances, read the documentation to create your first policy. We're excited to see how you use VM Extensions Manager to standardize, secure, and simplify the management of your VM fleet.

Automate AI and HPC clusters with Cluster Director, now generally available

Wed, 17 Dec 2025 18:00:00 +0000

The complexity of the infrastructure behind AI training and high performance computing (HPC) workloads can really slow teams down. At Google Cloud, where we work with some of the world’s largest AI research teams, we see it everywhere we go: researchers hampered by complex configuration files, platform teams struggling to manage GPUs with home-grown scripts, and operational leads battling the constant, unpredictable hardware failures that derail multi-week training runs. Access to raw compute isn't enough. To operate at the cutting edge, you need reliability that survives hardware failures, orchestration that respects topology, and a lifecycle management strategy that adapts to evolving needs.

Today, we are delivering on those requirements with the General Availability (GA) of Cluster Director and the Preview of Cluster Director support for Slurm on Google Kubernetes Engine (GKE).

Cluster Director (GA) is a managed infrastructure service designed to meet the rigorous demands of modern supercomputing. It replaces fragile DIY tooling with a robust topology-aware control plane that handles the entire lifecycle of Slurm clusters, from the first deployment to the thousandth training run.
We are expanding Cluster Director to support Slurm on GKE (Preview), designed to give you the best of both worlds: the familiar precision of high-performance scheduling and the automated scale of Kubernetes. It achieves this by treating GKE node pools as a direct compute resource for your Slurm cluster, allowing you to scale your workloads with Kubernetes' power without changing your existing Slurm workflows.

Cluster Director, now GA

Cluster Director offers advanced capabilities at each phase of the cluster lifecycle, spanning preparation (Day 0), where infrastructure design and capacity are determined; deployment (Day 1), where the cluster is automatically deployed and configured; and monitoring (Day 2), where performance, health, and optimization are continuously tracked.

This holistic approach ensures that you get the benefits of fully configurable infrastructure while automating lower-level operations so your compute resources are always optimized, reliable, and available.

So, what does all this cost? That’s the best part. There's no extra charge to use Cluster Director. You only pay for the underlying Google Cloud resources — your compute, storage, and networking.

How Cluster Director supports each phase of deployment

Day 0: Preparation

Standing up a cluster typically involves weeks of planning, wrangling Terraform, and debugging the network. Cluster Director changes the ‘Day 0’ experience entirely, with tools for designing infrastructure topology that’s optimized for your workload requirements.

To streamline your Day 0 setup, Cluster Director provides:

Reference architectures: We’ve codified Google’s internal best practices into reusable cluster templates, enabling you to spin up standardized, validated clusters in minutes. This helps ensure that every team in your organization is using the same security standards for their deployments and deploying on infrastructure that is configured correctly by default — right down to the network topology and storage mounting.
Guided configuration: We know that having too many options can lead to configuration paralysis. The Cluster Director control plane guides you through a streamlined setup flow. You select your resources, and our system handles the complex backend mapping, ensuring that storage tiers, network fabrics, and compute shapes are compatible and optimized before you deploy.
Broad hardware support: Cluster Director offers full support for large-scale AI systems, including Google Cloud’s A4X and A4X Max VMs powered by NVIDIA GB200 and GB300 GPUs, and versatile CPUs such as N2 VMs for cost-effective login nodes and debugging partitions.
Flexible consumption options: Cluster Director integrates with your preferred procurement strategy, with support for Reservations for guaranteed capacity during critical training runs, Dynamic Workload Scheduler Flex-start for dynamic scaling, or Spot VMs for opportunistic low-cost runs.

"Google Cloud's Cluster Director is optimized for managing large-scale AI and HPC environments. It complements the power and performance of NVIDIA's accelerated computing platform. Together, we're providing customers with a simplified, powerful, and scalable solution to tackle the next generation of computing challenges." - Dave Salvator, Director of Accelerated Computing Products, NVIDIA

Day 1: Deployment

Deploying hardware is one thing, but maximizing performance is another thing entirely. Day 1 is the execution phase, where your configuration transforms into a fully operational cluster. The good news is that Cluster Director doesn't just provision VMs, it validates that your software and hardware components are healthy, properly networked, and ready to accept the first workload.

To ensure a high-performance deployment, Cluster Director automates:

Getting a clean "bill of health": Before your job ever touches a GPU, Cluster Director runs a rigorous suite of health checks, including DCGMI diagnostics and NCCL performance validation, to verify the integrity of the network, storage, and accelerators.
Keeping accelerators fed with data: Storage throughput is often the silent killer of training efficiency. That’s why Cluster Director fully supports Google Cloud Managed Lustre with selectable performance tiers, allowing you to attach high-throughput parallel storage directly to your compute nodes, so your GPUs are never starved for data.
Maximizing Interconnect Performance: To achieve peak scaling, Cluster Director implements topology-aware scheduling and compact placement policies. By utilizing dense reservations on Google’s non-blocking fabric, the system ensures that your distributed workloads are placed on the shortest physical path possible, minimizing tail latency and maximizing collective communication (NCCL) speeds from the get-go.

“Cluster Director is an amazing product, which has enabled me to spin up a ready to use Nvidia GPU cluster with Slurm, including all networking, routing, and high performance network file-system for large-scale distributed model training within less than an hour. The cluster was immediately ready to run our containerizedAI training workloads with excellent throughput with only minimal customization effort." - Dr. Florian Eyben, Head of AI Foundation Models & Speech Technology, Agile Robots SE, Munich, Germany

Day 2: Monitoring

The reality of AI and HPC infrastructure is that hardware fails and requirements change. A rigid cluster is an inefficient cluster. As you move into the ongoing “Day 2” operational phase, you need to maintain cluster health, maximize utilization and performance. Cluster Director provides a control plane equipped for the complexities of long-term operations. Today we are introducing new active cluster management capabilities to handle the messy reality of Day 2 operations.

New active cluster management capabilities include:

Topology-level visibility: You can’t orchestrate what you can’t see. Cluster Director’s observability graphs and topology grids let you visualize your entire fleet, spot thermal throttles or interconnect issues, and optimize job placement based on physical proximity.
One-click remediation: When a node degrades, you shouldn't have to SSH in to debug it. Cluster Director allows you to replace faulty nodes with a single click directly from the Google Cloud console. The system handles the draining, teardown, and replacement, returning your cluster to full capacity in minutes.
Adaptive infrastructure: When your research needs change, so should your cluster. You can now modify active clusters, with activities such as adding or removing storage filesystems, on the fly, without tearing down the cluster or interrupting ongoing work.

Cluster Director support for Slurm on GKE, now in preview

Innovation thrives in the open. Google, the creator of Kubernetes, and SchedMD, the developers behind Slurm, have long championed the open-source technologies that power the world's most advanced computing. For years, NVIDIA and SchedMD have worked in lockstep to optimize GPU scheduling, introducing foundational features like the Generic Resource (GRES) framework and Multi-Instance GPU (MIG) support that are essential for modern AI. By acquiring SchedMD, NVIDIA is doubling down on its commitment to Slurm as a vendor-neutral standard, ensuring that the software powering the world's fastest supercomputers remains open, performant, and perfectly tuned for the future of accelerated computing.

Building on this foundation of accelerated computing, Google is deepening its collaboration with SchedMD to answer a fundamental industry challenge: how to bridge the gap between cloud-native orchestration and high-performance scheduling. We are excited to announce the Preview of Cluster Director support for Slurm on GKE, utilizing SchedMD’s Slinky offering.

This initiative brings together the two standards of the infrastructure world. By running a native Slurm cluster directly on top of GKE, we are amplifying the strengths of both communities:

Researchers get the uncompromised Slurm interface and batch capabilities, such as sbatch and squeue, that have defined HPC for decades.
Platform teams gain the operational velocity that GKE, with its auto-scaling, self-healing, and bin-packing, brings to the table.

Slurm on GKE is strengthened by our long-standing partnership with SchedMD, which helps create a unified, open, and powerful foundation for the next generation of AI and HPC workloads. Request preview access now.

Try Cluster Director today

Ready to start using Cluster Director for your AI and HPC cluster automation?

Learn more about the end-to-end capabilities in documentation.
Activate Cluster Director in the console.

Google named a Leader in The Forrester Wave™: AI Infrastructure Solutions, Q4 2025

Wed, 17 Dec 2025 17:00:00 +0000

For most organizations, the question is no longer if they will use AI, but how to scale it from a promising prototype into a production-grade service that drives business outcomes. In this age of inference, competitive advantage is defined by your ability to serve useful information to users around the world at the lowest possible cost. As you move from demos to production deployments at scale, you need to simplify infrastructure operations with integrated systems that provide the latest AI software and accelerator hardware platforms, while keeping costs and architectural complexity low.

Yesterday, Forrester released The Forrester Wave™: AI Infrastructure Solutions, Q4 2025 report, evaluating 13 vendors, and we believe their findings validate our commitment to solving these core challenges. Google received the highest score of all vendors in the Current Offering category and received the highest possible score in 16 out of 19 evaluation criteria, including, but not limited to: Vision, Architecture, Training, Inferencing, Efficiency, and Security.

Access the full report: The Forrester Wave™: AI Infrastructure Solutions, Q4 2025

Accelerating time-to-value with an integrated system

Enterprises don’t run AI in a vacuum. They need to integrate it with a diverse range of applications and databases while adhering to stringent security protocols. Forrester recognized Google Cloud’s strategy of co-design by giving us the highest possible score in the Efficiency and Scalability criteria:

“Google pursues a strategy of silicon-infrastructure co-design. It develops TPUs to improve inference efficiency and NVIDIA GPUs for access to broader ecosystem compatibility. Google designs TPUs to integrate tightly with its networking fabric, giving customers high bandwidth and low latency for inference at scale.”

For over two decades, we have operated some of the world's largest services, from Google Search and YouTube to Maps, where their unprecedented scale required us to solve problems that no one else had. We couldn't simply buy the platform and infrastructure we needed; we had to invent it. This led to a decade-long journey of deep, system-level co-design, building everything from our custom network fabric and specialized accelerators to frontier models, all under one roof.

The result was an integrated supercomputing system, AI Hypercomputer, which has paid significant dividends for our customers. It supports a wide range of AI-optimized hardware, allowing you to optimize for granular, workload-level objectives — whether that's higher throughput, lower latency, faster time-to-results, or lower TCO. That means you can use our custom Tensor Processing Units (TPUs), the latest NVIDIA GPUs, or both, backed by a system that tightly integrates accelerators with networking and storage for exceptional performance and efficiency. It’s also why today, leading generative AI companies such as Anthropic, Lightricks, and LG AI Research trust Google Cloud to power their most demanding AI workloads.¹

This system-level integration lays the foundation for speed, but operational complexity could still slow you down. To accelerate your time-to-market, we provide multiple ways to deploy and manage AI infrastructure, abstracting away the heavy lifting regardless of your preferred workflow. Google Kubernetes Engine (GKE) Autopilot automates management for containerized applications, helping customers like LiveX.AI reduce operational costs by 66%. Similarly, Cluster Director simplifies deployment for Slurm-based environments, enabling customers like LG AI Research to slash setup time from 10 days to under one day.

Managing AI cost and complexity

Forrester gave Google Cloud the highest scores possible in the Pricing Flexibility and Transparency criterion. The price of compute is only one part of the AI infrastructure cost equation. A complete view should also account for development costs, downtime and inefficient resource utilization. We offer optionality at every layer of the stack to provide the flexibility businesses demand.

Flexible consumption: Dynamic Workload Scheduler allows you to secure compute at up to 50% savings, by ensuring you only pay for the capacity you need, when you need it.
Load balancing: GKE Inference Gateway improves throughput by using AI-aware routing to balance requests across models, preventing bottlenecks and ensuring servers aren't sitting idle.
Eliminating data bottlenecks: Anywhere Cache co-locates data with compute, reducing read latency by up to 96% and eliminating the "integration tax" of moving data. By using Anywhere Cache together with our unified data platform BigQuery, you can avoid latency and egress fees while keeping your accelerators fed with data.

Mitigating strategic risk through flexibility and choice

We are also committed to enabling customer choice across accelerators, frameworks and multicloud environments. This isn’t new for us. Our deep experience with Kubernetes, which we developed then open-sourced, taught us that open ecosystems are the fastest path to innovation and provide our customers with the most flexibility. We are bringing that same ethos to the AI era by actively contributing to the tools you already use.

Open source frameworks and hardware portability: We continue to support open frameworks such as PyTorch, JAX, and Keras. We’ve also directly addressed concerns about workload portability on custom silicon by investing in TPU support for vLLM, allowing developers to easily switch between TPUs and GPUs (or use both) with only minimal configuration changes.
Hybrid and multicloud flexibility: Our commitment to choice extends to where you run your applications. Google Distributed Cloud brings our services to on-premises, edge and cloud locations, while Cross-Cloud Network securely connects applications and users with high-speed connectivity between your environments and other clouds. This powerful combination means you're no longer locked into a specific environment; you can easily migrate workloads and apply uniform management practices, streamlining operations, and mitigating the risk of lock-in.

Systems you can rely on

When your entire business model depends on the availability of AI services, infrastructure uptime is critical. Google Cloud's global infrastructure is engineered for enterprise-grade reliability, an approach rooted in our history as the birthplace of Site Reliability Engineering (SRE).

We operate one of the world's largest private software-defined networks, handling approximately 25% of global internet egress traffic. Unlike providers that rely on the public internet, we keep your traffic on Google’s own fiber to improve speed, reliability, and latency. This global backbone is powered by our Jupiter data center fabric, which scales to 13 Petabits/sec of bandwidth, delivering 50x greater reliability than previous generations — to say nothing of other providers. Finally, to improve cluster-level fault tolerance, we employ capabilities like elastic training and multi-tier checkpointing, which allow jobs to continue uninterrupted, by dynamically resizing the cluster around failed nodes while minimizing the time to recovery.

Building on a secure foundation

Our approach is to secure AI from the ground up. In fact, Google Cloud maintains a leading track record for cloud security. Independent analysis from cloudvulndb.org (2024-2025) shows that our platform has up to 70% fewer critical and high vulnerabilities compared to the other two leading cloud providers. We were also the first in the industry to publish an AI/ML Privacy Commitment, which guarantees that we do not use your data to train our models. With those safeguards in place, security is integrated into the foundation of Google Cloud, based on the zero-trust principles that protect Google’s own services:

A hardware root of trust: Our custom Titan chips, as part of our Titanium architecture, create a verifiable hardware root of trust. We recently extended this with Titanium Intelligence Enclaves for Private AI Compute, allowing you to process sensitive data in a hardened, isolated, and encrypted environment.
Built-in AI security: Security Command Center (SCC) natively integrates with our infrastructure, providing AI Protection by automatically discovering assets, preventing security issues, detecting active threats with frontline Google Threat Intelligence, and discovering known and unknown risks before attackers can exploit them.
Sovereign solutions: We enable you to meet stringent data residency, operational control, and software sovereignty requirements through solutions like Data Boundary. This is complemented by flexible options like partner-operated sovereign controls and Google Distributed Cloud for air-gapped needs.
Platform controls for AI and agent governance: Vertex AI provides the essential governance layer for the enterprise builder to deploy models and agents at scale. This trust is anchored in Google Cloud’s secure-by-default infrastructure, utilizing platform controls like VPC Service Controls (VPC-SC) and Customer-Managed Encryption Keys (CMEK) to sandbox environments and protect sensitive data, and Agent Identity for granular IAM permissions. At the platform level, Vertex AI and Agent Builder integrate Model Armor to provide runtime protection against emergent agentic threats, such as prompt injection and data exfiltration.

Delivering continuous AI innovation

We are honored to be recognized as a Leader in The Forrester Wave™ report, which we believe validates decades of R&D and our approach to building ultra-scale AI infrastructure. Look to us to continue on this path of system-level innovation as we help you convert the promise of AI into a reality.

Access the full report: The Forrester Wave™: AI Infrastructure Solutions, Q4 2025

^{1. IDC Business Value Snapshot, Sponsored by Google Cloud, The Business Value of Google Cloud AI Hypercomputer, US53855425, October 2025}

AI agents are here. Is your infrastructure ready?

Thu, 11 Dec 2025 17:00:00 +0000

Editor’s note: Today we hear from Dave McCarthy of IDC about a total cost of ownership crisis for AI infrastructure — and what you can do about it. Read on for his insights.

The AI landscape is undergoing a seismic shift. For the past few years, the industry has been focused on the massive, resource-intensive process of training generative AI models. But the focus is now rapidly pivoting to a new, even larger challenge: inference.

Inference — the process of using a trained model to make real-time predictions — is no longer just one part of the AI lifecycle; it is quickly becoming the dominant workload. In a recent IDC global survey of over 1,300 AI decision-makers, inference was already cited as the largest AI workload segment, accounting for 47% of all AI operations.

This dominance is driven by the sheer volume of real-world applications. While a model is trained periodically, it is used for inference non-stop, with every user query, API call, and recommendation. It is also critical to recognize that this inference surge will be distributed across hybrid environments. According to IDC survey respondents, 63% of workloads will reside in the cloud, which remains the standard for scalable applications like content creation and chatbots. In contrast, 37% will be deployed on on-premises infrastructure, usually related to use cases such as robotics and other systems that interact directly with the physical world.

Now, a new factor is set to multiply this demand: the rise of autonomous and semi-autonomous AI agents.

These "agentic workflows" represent the next logical step in AI, where models don't just respond to a single prompt but execute complex, multi-step tasks. An AI agent might be asked to "plan a trip to Paris," requiring it to perform dozens of interconnected operations: browsing for flights, checking hotel availability, comparing reviews, and mapping locations. Each of these steps is an inference operation, creating a cascade of requests that must be orchestrated across different systems.

This surge in demand is exposing a critical vulnerability for many organizations: the AI efficiency gap.

The TCO crisis in an age of agents

The AI efficiency gap is the difference between the theoretical performance of an AI stack and the actual, real-world performance achieved. This gap is the source of a Total Cost of Ownership (TCO) crisis, and it’s driven by system-wide inefficiencies.

Our research shows that more than half (54.3%) of organizations use multiple AI frameworks and hardware platforms. While this flexibility seems beneficial, it has a staggering downside: 92% of these organizations report a negative effect on efficiency.

This fragmented "patchwork" approach, stitched together from disparate and non-optimized services, creates a ripple effect of problems:

41.6% reported increased compute costs: Redundant processes and poor utilization drive up spending.
40.4% reported increased engineering complexity: Teams spend more time managing the fragmented stack than delivering value.
40.0% reported increased latency: Bottlenecks in one part of the system (like storage or networking) degrade the overall performance of an application.

The core problem is that organizations are paying for expensive, high-performance accelerators, but are failing to keep them busy. Our data shows that 29% of all AI budget waste is tied to inference. This waste is a direct result of idle GPU time (cited by 29.4% of respondents) and inefficient use of resources (22.3%).

When an expensive accelerator is idle, it’s often waiting for data from a slow storage system or for the application server to prepare the next request. This is a system-level failure, not a component failure.

This failure is often compounded by significant hurdles in data management, which serves as the fuel for these AI engines. Survey respondents highlighted three primary challenges contributing to this gap: 47.7% struggle with ensuring data quality and governance, 45.6% grapple with data storage management and related costs, and 44.1% cite the complexity and time required for data cleaning and preparation. When data pipelines cannot keep pace with high-speed accelerators, the entire infrastructure becomes inefficient.

Closing the gap: From fragmented stacks to integrated systems

To scale cost-effectively in the age of AI agents, we must stop thinking about individual components and start focusing on system-level design.

An agentic workflow, for example, requires tight coordination between two distinct types of compute:

General-purpose compute: This is the operational backbone. It runs the application servers, orchestrates the workflow, pre-processes data, and handles all the logic around the model.
Specialized accelerators: This is the high-performance engine that runs the AI model itself.

In a fragmented environment, these two sides are inefficiently connected, and latency skyrockets. The path forward is an optimized architecture where the software, networking, storage, and compute — both general-purpose and specialized — are designed to work as a single, cohesive system.

This holistic approach is the only sustainable way to manage the TCO of AI. It redefines the goal away from simply buying faster accelerators and toward improving the overall "price-performance" and "unit economics" of the entire end-to-end workflow. By eliminating bottlenecks and maximizing the utilization of every resource, organizations can finally close the efficiency gap. Organizations are actively shifting strategies to capture this value. Our survey indicates that 28.9% of respondents are prioritizing model optimization techniques, while 26.3% are partnering with AI service providers to navigate this complexity. Additionally, 25% are investing in training to upskill their teams, ensuring they can increase the value of their AI investments.

The age of inference is here, and the age of agents is right behind it. This next wave of innovation will be won not by the organizations with the most powerful accelerators, but by those who build the most efficient, integrated, and cost-effective systems to power them.

A message from Google Cloud

We sponsored this IDC research to help IT leaders navigate the critical shift to the "Age of Inference." We recognize that the "efficiency gap" identified here — driven by fragmented stacks and idle resources — is the primary barrier to sustainable ROI. That is why we created AI Hypercomputer: an integrated supercomputer system designed to deliver exceptional performance and efficiency for demanding AI workloads.

IDC surveyed 1,300 global IT leaders to uncover how they are designing their stack for maximum efficiency and ROI. Get your free copy of the whitepaper to learn more: The AI Efficiency Gap: From TCO Crisis to Optimized Cost and Performance.

Nutanix NC2 is now officially supported on Google Cloud

Tue, 09 Dec 2025 14:00:00 +0000

Today, we are thrilled to announce Nutanix Cloud Clusters (NC2) is generally available on Google Cloud.

NC2 on Google Cloud is designed to migrate and modernize specialized, regulated, and mission-critical applications without refactoring your workloads or compromising on performance. This partnership brings the power of Google Cloud’s infrastructure and advanced AI models to your hybrid cloud, without compromising on data residency, connectivity, or operational consistency. You can now run your Nutanix Hybrid Cloud directly on Google Compute Engine.

"The General Availability of Nutanix Cloud Clusters (NC2) on Google Cloud is a significant milestone empowering our joint customers to become AI-ready. We are excited to extend the simplicity and resilience of Nutanix NC2 onto Google Cloud's high-performance workload-optimized compute. Nutanix on Google Cloud enables our customers to migrate and modernize their critical workloads while unlocking the full power of Google’s industry-leading data and AI capabilities." - Saveen Pakala, VP, Product Management, Hybrid Cloud, Nutanix

Nutanix and Google Cloud allow you to maximize agility and minimize disruption for your critical applications. By combining NC2's enterprise flexibility with Google Cloud's power, you gain access to three core advantages. First, your workloads run on Compute Engine’s dynamically scalable workload-optimized infrastructure powering all machine families. Nutanix NC2 supports Compute Engine bare metal instances in the Z3 and C4 families. These are powered by the Titanium offload system and leverage Titanium SSDs for low-latency, high-throughput storage performance, hosted in Google Cloud with global reach, enterprise-grade security, and commitment to sustainability. Second, you accelerate AI innovation by co-locating data and machine learning services like Gemini Enterprise and Vertex AI. Finally, you can save costs by dynamically scaling capacity and utilizing committed use discounts (CUDs) and Flex CUDs.

Key use cases to accelerate your cloud journey

The integration of NC2 on Google Cloud offers flexible, strategic options for hybrid cloud operations. Beyond consolidation and cost control, these capabilities set the stage for true modernization:

Seamless workload migration: Move entire applications between your on-premises Nutanix environment and Google Cloud without re-factoring or re-architecting. This capability saves significant time during data center consolidation.
Consistent operations: Maintain the same management plane, security policies, and automation across your private data center and Google Cloud, which dramatically reduces operational complexity and training costs.
Disaster recovery (DR): Leverage Google Cloud as a robust and cost-efficient recovery target. Usage of a minimal “pilot light” cluster reduces compute costs, so you scale up only when a disaster event occurs.
Capacity bursting: Instantly add capacity in the cloud to handle seasonal demands, VDI workloads, development/test cycles, or requirements from mergers and acquisitions (M&A).
License portability: Protect your software investments by easily moving your existing Nutanix software licenses to Google Cloud as your business needs evolve.

“Like many others, we are always on a journey to modernize and shift to achieve the best outcomes for our customers. Nutanix Cloud Clusters (NC2) on Google Cloud brings us a solid platform to continue our hybrid cloud expansion. Our ability to seamlessly run workloads on-premises and on NC2 on Google Cloud without having to re-factor is increasingly valuable as we continue our modernization journey. We look forward to continuing our strong partnership with Google Cloud and Nutanix.” - VP of IT at a global oil & gas company based in Oklahoma

The architecture

NC2 on Compute Engine simplifies building a hybrid cloud by deploying the Nutanix Cloud Infrastructure (NCI) software stack, including the Acropolis Hypervisor (AHV), directly onto high-performance Compute Engine infrastructure.

The key components of the solution include:

Compute Engine instances: NC2 runs on Google Compute Engine bare metal instances in the recently introduced C4 and Z3 machine families. These powerful instances provide the foundation with high-density compute, memory, local NVMe storage, and high network bandwidth.

Machine Family	GCE Machine Type	vCPUs	Memory	Storage	Processor
Z3, Storage Optimized	z3-highmem-192-highlssd-metal	192	1536GB	72TB of NVMe Local SSD	Intel, Sapphire Rapid
C4, General Purpose	c4-highmem-288-lssd-metal	288	1080GB	18TB of NVMe Local SSD	Intel, Granite Rapid
C4, General Purpose	c4-standard-288-lssd-metal	288	2232GB	18TB of NVMe Local SSD	Intel, Granite Rapid

Simplified networking : NC2 runs entirely within your existing Google Cloud Virtual Private Cloud (VPC). Built-in Nutanix Flow Virtual Networking for overlay is integrated to reduce hybrid cloud complexity.
Unified management: The entire environment, both on-premises and in Google Cloud, is managed through the familiar Prism Central console, simplifying day-to-day operations and skill requirements for your IT teams.
Easy procurement: Later this month, you’ll be able to purchase Nutanix NC2 licensing directly from Google Cloud Marketplace . This offers a single, unified billing experience for both your Google Cloud infrastructure and Nutanix NC2, in one simple process. A key benefit is the ability to use your existing Google Cloud spend commitments for Nutanix NC2 software. This helps you maximize your investment and streamline your financial operations, providing more value from your cloud budget.

Connect your data to Google Cloud AI and analytics

A significant modernization opportunity comes from connecting your stable, trusted Nutanix workloads with Google Cloud's powerful data and AI tools. Your applications running on NC2 can tap directly into services like BigQuery and Vertex AI with low latency, enabling you to:

Derive deeper business value: Easily send application log data, transactional records, and other operational data from your Nutanix VMs to BigQuery for real-time, scalable data warehousing and complex analysis.
Build custom machine learning models: Use Vertex AI to create, deploy, and manage custom ML models that analyze data generated by your core applications (e.g., predictive maintenance or fraud detection).
Use conversational AI: Quickly build and deploy conversational agents using technologies like Dialogflow that interact directly with the application data residing on your NC2 cluster.

Ready to simplify your cloud operations?

NC2 on Google Cloud is currently available across 17 Google Cloud regions, with a planned expansion continuing through 2026. For precise details on regional and zonal availability, please check the official Google Compute Engine bare metal regional availability documentation, and reference the Compute Engine pricing page for infrastructure costs. To learn more about the solution, try taking a test drive or visit Nutanix partner page. Available later this month, you will be able to explore NC2 on Google Cloud licensing through the Google Cloud Marketplace.

Running high-scale reinforcement learning (RL) for LLMs on GKE

Mon, 10 Nov 2025 17:00:00 +0000

As Large Language Models (LLMs) evolve, Reinforcement Learning (RL) is becoming the crucial technique for aligning powerful models with human preferences and complex task objectives.

However, enterprises that need to implement and scale RL for LLMs are facing infrastructure challenges. The primary hurdles include the memory contention from concurrently hosting multiple large models (such as the actor, critic, reward, and reference models), iterative switching between high latency inference generation, and high throughput training phases.

This blog details Google Cloud's full-stack, integrated approach, from custom TPU hardware to the GKE orchestration layer — and shares how you can solve the hybrid, high-stakes demands of RL at scale.

A quick primer: Reinforcement Learning (RL) for LLMs

RL is a continuous feedback loop that combines elements of both training and inference. At a high level, the RL loop for LLMs functions as follows:

The LLM generates a response to a given prompt.
A "reward model" (often trained on human preferences) assigns a quantitative score, or reward, to the output.
An RL algorithm (e.g., DPO, GRPO) uses this reward signal to update the LLM's parameters, adjusting its policy to generate higher-rewarding outputs in subsequent interactions.

This generation, evaluation, and optimization continually improves the LLM's performance based on predefined objectives.

RL workloads are hybrid and cyclical. The main goal of RL is not to minimize error (training) or fast prediction (inference), but to maximize reward through iterative interaction. The primary constraint for the RL workload is not just the computational power, but also system-wide efficiency, specifically minimizing aggregate sampler latency and maximizing the speed of weight copying for efficient end-to-end step time.

Google Cloud's full-stack approach to RL

Solving these system-wide challenges requires an integrated approach. You can't just have fast hardware or a good orchestrator; you need every layer of the stack to work together. Here is how our full-stack approach is built to solve the specific demands of RL:

1. Flexible, high-performance compute (TPUs and GPUs): Instead of locking customers into one path, we provide two high-performance options. Our TPU stack is a vertically integrated, JAX-native solution where our custom hardware (excelling at matrix operations) is co-designed with our post-training libraries (MaxText and Tunix). In parallel, we fully support the NVIDIA GPU ecosystem, partnering with NVIDIA on optimized NeMo RL recipes so customers can leverage their existing expertise directly on GKE.

2. Holistic, full-stack optimization: We integrate optimization from the bare metal up. This includes our custom TPU accelerators, high-throughput storage (Managed Lustre, Google Cloud Storage), and — critically — the orchestration and scheduling that GKE provides. By optimizing the entire stack, we can attack the system-wide latencies that bottleneck hybrid RL workloads.

3. Leadership in open-source: RL infrastructure is complex and built on a wide range of tools. Our leadership starts with open-sourcing Kubernetes and extends to active partnerships with orchestrators like Ray. We contribute to key projects like vLLM, develop open-source solutions like llm-d for cost-effective serving, and open-source our own high-performance MaxText and Tunix libraries. This helps ensure you can integrate the best tools for the job, not just the ones from a single vendor.

4. Proven, mega-scale orchestration: Post-training RL can require compute resources that rival pre-training. This requires an orchestration layer that can manage massive, distributed jobs as a single unit. GKE AI mega-clusters support up to 65,000 nodes today, and we are heavily investing in multi-cluster solutions like MultiKueue to scale RL workloads beyond the limits of a single cluster.

Running RL workloads on GKE

Existing GKE infrastructure is well-suited for demanding RL workloads and provides several infrastructure-level efficiencies.

The image below outlines the architecture and key recommendations for implementing RL at scale.

Figure : GKE infrastructure for running RL

At the base, the infrastructure layer provides the foundational hardware, including supported compute types (CPUs, GPUs, and TPUs). You can use the Run:ai model streamer to accelerate the model streaming for all three compute types. High performance storage (Managed Lustre, Cloud Storage) can be used for storage needs for RL.

The middle layer is the managed K8s layer powered by GKE, which handles the resource orchestration, resource obtainability using Spot or Dynamic Workload Scheduler, autoscaling, placement, job queuing and job scheduling and more at mega scale.

Finally, the open frameworks layer runs on top of GKE, providing the application and execution environment. This includes the managed support for open-source tools such as KubeRay, Slurm and gVisor sandbox for secure isolated task execution.

Building RL workflow

Before creating an RL workload, you must first identify a clear use case. With that objective defined, you then architect the core components: selecting the algorithm (e.g, DPO, GRPO), the model server (like vLLM or SGLang), the target GPU/TPU hardware, and other critical configurations.

Next, you can provision a GKE cluster configured with Workload Identity, GCS Fuse, and DGCM metrics. For robust batch processing, install the Kueue and JobSet APIs. We recommend deploying Ray as the orchestrator on top of this GKE stack. From there, you can launch the Nemo RL container, configure it for your GRPO job, and begin monitoring its execution. For the detailed implementation steps and source code, please refer to this repository.

Getting started with RL

Run RL on GPUs: Try the RL recipe on TPUs using MaxText and Pathways for GRPO algorithm, or if you use GPUs, try the NemoRL recipes.
Partner with the open-source ecosystem: Our leadership in AI is built on open standards like Kubernetes, llm-d, Ray, MaxText or Tunix. We invite you to partner with us to build the future of AI together. Come contribute to llm-d! Join the llm-d community, check out the repository on GitHub, and help us define the future of open-source LLM serving.

N4D now GA: Gain up to 3.5x price-performance for scale-out workloads

Mon, 10 Nov 2025 17:00:00 +0000

In today's competitive environment, IT leaders are faced with supporting application scale, rolling out more features, and enabling high-bar customer experiences. This creates a direct and complex challenge: finding the right balance between performance and total cost of ownership (TCO) for the general-purpose workloads that power everyday business operations.

Today, we are announcing the general availability of the N4D machine series, the latest addition to Google Compute Engine’s cost-optimized, general-purpose portfolio. Addressing a wide range of workloads, such as web and application servers, data analytics platforms, and containerized microservices, N4D provides a flexible and price-performant solution.

The N4D machine series combines Google's Titanium infrastructure with 5th Gen AMD EPYC™ “Turin” processors, delivering up to 3.5x the throughput for web-serving workloads vs. the previous-generation N2D. N4D offers predefined shapes of up to 96 vCPUs and 768 GB of DDR5 memory, up to 50 Gbps of networking bandwidth, and Hyperdisk Balanced and Throughput storage. To deliver a blended cost savings, N4D allows you to move beyond rigid instance sizing for both compute and storage, with Custom Machine Types to independently configure the exact number of vCPUs and amount of memory, complemented with Hyperdisk, for tuning disk storage performance and capacity. For the most demanding general purpose workloads, pair N4D together with consistently high performance of C4D.

Google Cloud provides workload-optimized infrastructure to ensure the right resources are available for every task. Titanium in particular, with its multi-tier offloads and security capabilities, is foundational to that infrastructure. Titanium offloads networking and storage processing to free up the CPU, and its dedicated SmartNIC manages all I/O, ensuring the AMD EPYC cores are reserved exclusively for your application. Titanium is part of Google Cloud’s vertically integrated stack — from the custom silicon in our servers to our planet-scale network traversing 7.75 million kilometers of terrestrial and subsea fiber across 42 regions — that is engineered to maximize efficiency and provide the ultra-low latency and high bandwidth to customers at global scale.

A new standard for price-performance

N4D machine series doesn't just inch past the previous N2D generation; it sprints, delivering up to 50% higher price-performance for general computing workloads and up to 70% better price-performance for Java workloads. For web-serving workloads, N4D leverages Titanium and AMD’s Turin processors to drive incredible throughput. This results in up to 3.5x the price-performance vs N2D, driving faster response times and a better overall experience for your end-users.

As of October 2025. Performance based on the estimated SPECrate®2017_int_base, estimated SPECjbb2015, and Google internal Nginx Reverse Proxy benchmark scores run in production. Price-performance claims based on published and estimated list prices for Google Cloud.

“Our edge proxy fleet and internal data pipelines observed a 3-4x performance improvement on Google Cloud's N4D instances compared to N2D. Our benchmarks also show N4D processes the same workload with significantly greater consistency while using just a fraction of the CPU. This leap in price-performance allows us to efficiently scale our general-purpose workloads, and fits neatly in our fleet alongside more specific Google compute products we leverage.” - Matt Schallert, Member of Technical Staff, Chronosphere

“A 10% increase in throughput while cutting costs by up to 50% is a massive win for TCO optimization. That's what we achieved on Google Cloud's N4D machine series. For MediaGo, this efficiency is critical. It allows our AI-driven advertising platform to scale more cost-effectively, directly supporting our mission to maximize ROI for our global partners.” - MediaGo

"The move from N2D to N4D is a significant generational leap. This 144.14% performance uplift over 152 tests is a testament to Google's Titanium, unlocking the full potential of the new AMD EPYC 'Turin' processors. For those looking for the best possible price-performance in Google Cloud, the N4D instances are a clear winner." - Michael Larabel, Founder and Principal Author, Phoronix (Read the full study here.)

"With the launch of the new N4D instances, Google Cloud now offers the most comprehensive portfolio based on our 5th Gen AMD EPYC processors, marking a significant milestone in our strategic partnership. N4D machine series combines the leading performance of AMD CPUs with the uniqueness of Google's Custom Machine Types to deliver a remarkable uplift in price-performance, flexibility, and cost-optimization for everyday workloads. Our benchmark tests confirm this, showing measured performance gains of up to 75% over the previous generation N2D machine series for media encode and transcode workloads." – Ryan Rodman, Sr Director, Cloud Business Group, AMD

Complementing C4D machine series

Earlier this year, we introduced our general-purpose C4D machine series built on the same underlying processor as N4D. Its consistently high performance and enterprise features like advanced maintenance support, larger shapes, and our next-gen Titanium Local SSDs, make C4D a great fit for critical workloads. In fact, customers such as Silk and Chess.com report greater than 40% improvement in performance with C4D over prior generations.

But critical applications are only part of the story. A modern cloud architecture must also run countless general-purpose workloads where flexibility and price-performance are key. That’s why we designed N4D — as a complement to C4D. By leveraging C4D and N4D in tandem, you unlock the full spectrum of enterprise features, performance, flexibility, and cost-optimization, choosing:

C4D for consistent performance: This is your solution for the most demanding, latency-sensitive applications. With up to 200 Gbps networking, Local SSD support along with larger shapes up to 384 vCPUs and bare metal options, C4D delivers predictable, high-end performance for large databases, high-traffic ad and game servers, and demanding AI/ML inference workloads.
N4D for flexible cost-optimization: This is the engine for the vast majority of your general-purpose workloads. N4D’s leading price-performance, low cost, and flexibility allow you to slash TCO for applications like web servers, microservices, and development environments.

This approach is already delivering real-world results, allowing customers like Verve to optimize their business from both ends.

"With Google's Gen4 AMD portfolio, we can optimize for both revenue and cost simultaneously. C4D provides the consistent peak performance we need for our core ad servers — 81% faster than C3D — which directly translates to more revenue from higher fill-rates (successful bid/ask matching). Meanwhile, N4D delivers an incredible 2x performance and price-performance over N2D for everyday workloads, including scale-out microservices with GKE, enabling us to grow while slashing our overall TCO. This 'Better Together' strategy allows us to use the consistently peak performance of C4D for our mission-critical services and the flexible, cost-efficient N4D to aggressively reduce TCO everywhere else — a level of optimization that simply isn't possible with a single VM type elsewhere.” - Pablo Loschi, Principal Systems Engineer at Verve

The Custom Machine Type and Hyperdisk advantage

Custom Machine Types are a key differentiator for Google Cloud, letting you go beyond predefined "T-shirt sizes". Instead of forcing your workload into a box, you can tailor the infrastructure to fit your workload's needs, saving on cost. For instance, a memory-intensive workload requiring 16 vCPUs and 70 GB of RAM might typically be placed on a predefined N4D-highmem-16 shape, forcing you to pay for unused resources. With CMTs, you provision the exact 16 vCPU and 70 GB configuration, eliminating that waste and achieving up to 17% cost savings.

With shapes of up to 96 vCPUs and 768 GB of DDR5 memory, the combination of Custom Machine Types and N4D lets you dial in the exact resources you need with flexible vCPU-to-memory ratios along with extended memory support.

“At Symbotic, our vision is to revolutionize the global supply chain with an AI-powered robotics platform built for scale and efficiency. This demands an infrastructure that is both powerful and scalable. Google Cloud's N4D VMs, powered by AMD's latest EPYC processors, delivered exactly that. We observed a significant 40% performance uplift compared to the previous N2D generation, allowing us to cut our CPU footprint in half with no change in simulation speed or fidelity. The ability to pair these gains with Custom Machine Types — a capability unique to Google Cloud — is a game-changer. It allows us to precisely sculpt our infrastructure to our workloads and gain a significant TCO advantage versus other cloud offerings.” - Dan Inbar, Chief Information Officer, Symbotic

This granular control and TCO advantage extends beyond compute to your storage. Just as Custom Machine Types let you break free from fixed vCPU-to-memory ratios, Hyperdisk unbundles storage performance from capacity, letting you independently tune capacity and performance to precisely match your workload’s block storage requirements.

This is further enhanced by Hyperdisk Storage Pools for Hyperdisk Balanced volumes, which let you provision performance and capacity in aggregate, rather than managing each volume individually. The result is simpler management, higher efficiency, an easier path for modernizing SAN workloads — all this while helping you lower your storage TCO by as much as 30-50%.

Get started with N4D today

Adopting the latest N4D VM series is easy, particularly if you use Google Kubernetes Engine (GKE), where our custom compute classes remove the operational hurdles of migrating workloads to new hardware. Just add N4D to your prioritized list of VM types to ensure your workloads have the performance and flexibility they need to scale.

N4D is now available in us-central1 (Iowa), us-east1 (South Carolina), us-west1 (Oregon), us-west4 (Las Vegas), europe-west1 (Belgium), and europe-west4 (Netherlands).

Check for the latest availability on our Regions and Zones page and deploy your first instance today in the Google Cloud console or with GKE. Learn more about N4D details here in documentation.

^{1. 9xx5C-044 - Testing by AMD Performance Labs as of 10/21/2025. N4D-standard-16 score comparison to N2D-standard-16 running FFmpeg v6.1.1 benchmark (average of 2x encode and 2x transcode) on Ubuntu24.04LTS OS with 6.8.0-1021-gcp kernel, SMT On.}

^{Performance uplift (normalized to N2D):}

^{Ffmpeg_raw_vp9 1.76}^{Ffmpeg_h264_vp9 1.76}^{Ffmpeg_raw_h264 1.71}^{Ffmpeg_vp9_h264 1.76}^{FFmpeg average 1.75}

^{Cloud performance results presented are based on the test date in the configuration. Results may vary due to changes to the underlying configuration, and other conditions such as the placement of the VM and its resources, optimizations by the cloud service provider, accessed cloud regions, co-tenants, and the types of other workloads exercised at the same time on the system}

Announcing Ironwood TPUs General Availability and new Axion VMs to power the age of inference

Thu, 06 Nov 2025 13:00:00 +0000

Today’s frontier models, including Google’s Gemini, Veo, Imagen, and Anthropic’s Claude train and serve on Tensor Processing Units (TPUs). For many organizations, the focus is shifting from training these models to powering useful, responsive interactions with them. Constantly shifting model architectures, the rise of agentic workflows, plus near-exponential growth in demand for compute, define this new age of inference. In particular, agentic workflows that require orchestration and tight coordination between general-purpose compute and ML acceleration are creating new opportunities for custom silicon and vertically co-optimized system architectures.

We have been preparing for this transition for some time and today, we are announcing the availability of three new products built on custom silicon that deliver exceptional performance, lower costs, and enable new capabilities for inference and agentic workloads:

Ironwood, our seventh generation TPU, will be generally available in the coming weeks. Ironwood is purpose-built for the most demanding workloads: from large-scale model training and complex reinforcement learning (RL) to high-volume, low-latency AI inference and model serving. It offers a 10X peak performance improvement over TPU v5p and more than 4X better performance per chip for both training and inference workloads compared to TPU v6e (Trillium), making Ironwood our most powerful and energy-efficient custom silicon to date.
New Arm®-based Axion instances. N4A, our most cost-effective N series virtual machine to date, is now in preview. N4A offers up to 2x better price-performance than comparable current-generation x86-based VMs. We are also pleased to announce C4A metal, our first Arm-based bare metal instance, will be coming soon in preview.

Ironwood and these new Axion instances are just the latest in a long history of custom silicon innovation at Google, including TPUs, Video Coding Units (VCU) for YouTube, and five generations of Tensor chips for mobile. In each case, we build these processors to enable breakthroughs in performance that are only possible through deep, system-level co-design, with model research, software, and hardware development under one roof. This is how we built the first TPU ten years ago, which in turn unlocked the invention of the Transformer eight years ago — the very architecture that powers most of modern AI. It has also influenced more recent advancements like our Titanium architecture, and advanced liquid cooling that we’ve deployed at GigaWatt scale with fleet-wide uptime of ~99.999% since 2020.

Pictured: An Ironwood board showing three Ironwood TPUs connected to liquid cooling.

Pictured: Third-generation Cooling Distribution Units, providing liquid cooling to an Ironwood superpod.

Ironwood: The fastest path from model training to planet-scale inference

The early response to Ironwood is overwhelmingly enthusiastic. Anthropic is compelled by the impressive price-performance gains that accelerate their path from training massive Claude models to serving them to millions of users. In fact, Anthropic plans to access up to 1 million TPUs:

"Our customers, from Fortune 500 companies to startups, depend on Claude for their most critical work. As demand continues to grow exponentially, we're increasing our compute resources as we push the boundaries of AI research and product development. Ironwood’s improvements in both inference performance and training scalability will help us scale efficiently while maintaining the speed and reliability our customers expect." – James Bradbury, Head of Compute, Anthropic

Ironwood is being used by organizations of all sizes and across industries:

“Our mission at Lightricks is to define the cutting edge of open creativity, and that demands AI infrastructure that eliminates friction and cost at scale. We relied on Google Cloud TPUs and its massive ICI domain to achieve our breakthrough training efficiency for LTX-2, our leading open-source multimodal generative model. Now, as we enter the age of inference, our early testing makes us highly enthusiastic about Ironwood. We believe that Ironwood will enable us to create more nuanced, precise, and higher-fidelity image and video generation for our millions of global customers." - Yoav HaCohen, PhD, Director of Foundational Generative AI Research, Lightricks

“At Essential AI, our mission is to build powerful, open frontier models. We need massive, efficient scale, and Google Cloud's Ironwood TPUs deliver exactly that. The platform was incredibly easy to onboard, allowing our engineers to immediately leverage its power and focus on accelerating AI breakthroughs." - Philip Monk, Infrastructure Lead, Essential AI

System-level design maximizes inference performance, reliability, and cost

TPUs are a key component of AI Hypercomputer, our integrated supercomputing system that brings together compute, networking, storage, and software to improve system-level performance and efficiency. At the macro level, according to a recent IDC report, AI Hypercomputer customers achieved on average 353% three-year ROI, 28% lower IT costs, and 55% more efficient IT teams.

Ironwood TPUs will help customers push the limits of scale and efficiency even further. When you deploy TPUs, the system connects each individual chip to each other, creating a pod — allowing the interconnected TPUs to work as a single unit. With Ironwood, we can scale up to 9,216 chips in a superpod linked with breakthrough Inter-Chip Interconnect (ICI) networking at 9.6 Tb/s. This massive connectivity allows thousands of chips to quickly communicate with each other and access a staggering 1.77 Petabytes of shared High Bandwidth Memory (HBM), overcoming data bottlenecks for even the most demanding models.

Pictured: Part of an Ironwood superpod, directly connecting 9,216 Ironwood TPUs in a single domain.

At that scale, services demand uninterrupted availability. That’s why our Optical Circuit Switching (OCS) technology acts as a dynamic, reconfigurable fabric, instantly routing around interruptions to restore the workload while your services keep running. And when you need more power, Ironwood scales across pods into clusters of hundreds of thousands of TPUs.

Pictured: Jupiter data center network enables the connection of multiple Ironwood superpods into clusters of hundreds of thousands of TPUs.

The AI Hypercomputer advantage: Hardware and software co-designed for faster, more efficient outcomes

On top of this hardware is a co-designed software layer, where our goal is to maximize Ironwood’s massive processing power and memory, and make it easy to use throughout the AI lifecycle.

To improve fleet efficiency and operations, we’re excited to announce that TPU customers can now benefit from Cluster Director capabilities in Google Kubernetes Engine. This includes advanced maintenance and topology awareness for intelligent scheduling and highly resilient clusters.
For pre-training and post-training, we’re also sharing new enhancements to MaxText, a high-performance, open source LLM framework, to make it easier to implement training and reinforcement learning optimization techniques, such as Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO).
For inference, we recently announced enhanced support for TPUs in vLLM, allowing developers to switch between GPUs and TPUs, or run both, with only a few minor configuration changes, and GKE Inference Gateway, which intelligently load balances across TPU servers to reduce time-to-first-token (TTFT) latency by up to 96% and serving costs by up to 30%.

Our software layer is what enables AI Hypercomputer’s high performance and reliability for training, tuning, and serving demanding AI workloads at scale. Thanks to deep integrations across the stack — from data-center-wide hardware optimizations to open software and managed services— Ironwood TPUs are our most powerful and energy-efficient TPUs to date. Learn more about our approach to hardware and software co-design here.

Axion: Redefining general-purpose compute

Building and serving modern applications requires both highly specialized accelerators and powerful, efficient general-purpose compute. This was our vision for Axion, our custom Arm Neoverse®-based CPUs, which we designed to deliver compelling performance, cost and energy efficiency for everyday workloads.

Today, we are expanding our Axion portfolio with:

N4A (preview), our second general-purpose Axion VM, which is ideal for microservices, containerized applications, open-source databases, batch, data analytics, development environments, experimentation, data preparation and web serving jobs that make AI applications possible. Learn more about N4A here.
C4A metal (in preview soon), our first Arm-based bare-metal instance, which provides dedicated physical servers for specialized workloads such Android development, automotive in-car systems, software with strict licensing requirements, scale test farms, or running complex simulations. Learn more about C4A metal here.

With today's announcements, the Axion portfolio now includes three powerful options, N4A, C4A and C4A metal. Together, the C and N series allow you to lower the total cost of running your business without compromising on performance or workload-specific requirements.

Axion-based Instance	Optimized for	Key Features
N4A (preview)	Price-performance and flexibility	Up to 64 vCPUs, 512GB of DDR5 Memory, and 50 Gbps networking, with support for Custom Machine Types, Hyperdisk Balanced and Throughput storage.
C4A Metal (in preview soon)	Specialized workloads, such as Hypervisors and native Arm development	Up to 96 vCPUs, 768GB of DDR5 Memory, Hyperdisk storage and up to 100Gbps of networking
C4A	Consistently high performance	Up to 72 vCPUs, 576GB of DDR5 Memory, 100Gbps of Tier 1 networking, Titanium SSD with up to 6TB of local capacity, advanced maintenance controls and support for Hyperdisk Balanced, Throughput, and Extreme.

Axion’s inherent efficiency also makes it a valuable option for modern AI workflows. While specialized accelerators like Ironwood handle the complex task of model serving, Axion excels at the operational backbone: supporting high-volume data preparation, ingestion, and running application servers that host your intelligent applications. Axion is already translating into customer impact:

A powerful combination for AI and everyday computing

To thrive in an era with constantly shifting model architectures, software, and techniques, you need a combination of purpose-built AI accelerators for model training and serving, alongside efficient, general-purpose CPUs for the everyday workloads, including the workloads that support those AI applications.

Ultimately, whether you use Ironwood and Axion together or mix and match them with the other compute options available on AI Hypercomputer, this system-level approach gives you the ultimate flexibility and capability for the most demanding workloads. Sign up to test Ironwood, Axion N4A, or C4A metal today.

Announcing Axion C4A metal: Arm-based Axion instances for specialized use cases

Thu, 06 Nov 2025 13:00:00 +0000

Today, we are thrilled to announce C4A metal, our first bare metal instance running on Google Axion processors, available in preview soon. C4A metal is designed for specialized workloads that require direct hardware access and Arm®-native compatibility.

Now, organizations running environments such as Android development, automotive simulation, CI/CD pipelines, security workloads, and custom hypervisors can run them on Google Cloud, without the performance overheads and complexity of nested virtualization.

C4A metal instances, like other Axion instances, are built on the standard Arm architecture, so your applications and operating systems compiled for Arm remain portable across your cloud, on-premises, and edge environments, protecting your development investment. C4A metal offers 96 vCPUs, 768GB of DDR5 memory, up to 100Gbps of networking bandwidth, with full support for Google Cloud Hyperdisk including Hyperdisk Balanced, Extreme, Throughput, and ML block storage options.

Google Cloud provides workload-optimized infrastructure to ensure the right resources are available for every task. C4A metal, like the Google Cloud Axion virtual machine family, is powered by Titanium, a key component for multi-tier offloads and security that is foundational to our infrastructure. Titanium's custom-designed silicon offloads networking and storage processing to free up the CPU, and its dedicated SmartNIC manages all I/O, ensuring that Axion cores are reserved exclusively for your application's performance. Titanium is part of Google Cloud’s vertically integrated software stack — from the custom silicon in our servers to our planet-scale network traversing 7.75 million kilometers of terrestrial and subsea fiber across 42 regions — that is engineered to maximize efficiency and provide the ultra-low latency and high bandwidth to customers at global scale.

Architectural parity for automotive workloads

Automotive customers can benefit from the Arm architecture’s performance, efficiency, and flexible design for in-vehicle systems such as infotainment and Advanced Driver Assistance Systems (ADAS). Axion C4A metal instances enable architectural parity between test environments and production silicon, allowing automotive technology providers to validate their software on the same Arm Neoverse instruction set architecture (ISA) used in production electronic control units (ECUs). This significantly reduces the risk of late-stage integration failures. For performance-sensitive tasks, these customers can execute demanding virtual hardware-in-the-loop (vHIL) simulations with the consistent, low-latency performance of physical hardware, ensuring test results are reliable and accurate. Finally, C4A metal lets providers move beyond the constraints of a physical lab, by dynamically scaling entire test farms and transforming them from fixed capital expenses into flexible operational ones.

“In the era of AI-defined vehicles, the accelerating pace and complexity of technology are pushing us to rethink traditional linear approaches to software development. Google Cloud’s introduction of Axion C4A metal is a major step forward in this journey. By offering full architectural parity on Arm between test environments and physical silicon, customers can benefit from accelerated development cycles, enabling continuous integration and compliance for a variety of specialized use cases." - Dipti Vachani, Senior Vice President and General Manager, Automotive Business, Arm

“Our partners and customers rely on QNX to deliver the safety, security, reliability, and real-time performance required for their most mission-critical systems — from advanced driver assistance to digital cockpits. As the Software-Defined Vehicle era continues to gain momentum, decoupling software development from physical hardware is no longer optional — it’s essential for innovation at scale. The launch of Google Cloud’s C4A-metal instances on Axion introduces a powerful ARM-based bare metal platform that we are eager to test and support as this will enable transformative cloud infrastructure benefits for our automotive ecosystem.” - Grant Courville, Senior Vice President, Products and Strategy, QNX

“The future of automotive mobility demands unprecedented speed and precision in practice and development. For automakers and suppliers leveraging the Snapdragon Digital Chassis platform, aligning their cloud development and testing environments to ensure parity with the Snapdragon SoCs in the vehicle is absolutely crucial for efficiency and quality. We are excited about Google Cloud’s commitment to this segment — offering C4A-metal instances with Axion is a massive leap forward, giving the automotive ecosystem a true 1:1 physical to virtual environment in the cloud. This breakthrough significantly reduces integration challenges, slashes validation time, and allows our partners to unleash AI-driven features to market faster at scale.” - Laxmi Rayapudi, VP, Product Management, Qualcomm Technologies, Inc.

Align test and production for Android development

The Android platform was built for Arm-based processors, the standard for virtually all mobile devices. By running development and testing pipelines on the bare-metal instances of Axion processors with C4A metal, Android developers can benefit from native performance, eliminating the overhead of emulation management, such as slow instruction-by-instruction translation layers. In addition, they can significantly reduce latency for Android build toolchains and automated test systems, leading to faster feedback cycles. C4A metal also solves the performance challenges of nested virtualization, making it a great platform for scalable Cuttlefish (Cloud Android) environments.

Once available, developers can deploy scalable Cuttlefish environment farms on top C4A metal instances with an upcoming release of Horizon or by directly leveraging Cloud Android Orchestration. C4A metal allows these virtual devices to run directly on the physical hardware, providing the performance needed to build and manage large, high-fidelity test farms for true continuous testing.

Bare metal access without compromise

As a cloud offering, C4A metal enables a lower total cost of ownership by replacing the entire lifecycle of physical hardware procurement and management with a predictable operational expense. This eliminates the direct capital expenditures of purchasing servers, along with the associated operational costs of hardware maintenance contracts, power, cooling, and physical data center space. You can programmatically provision and de-provision instances to match your exact testing demands, ensuring you are not paying for an over-provisioned fleet of servers sitting idle waiting for peak development cycles.

Operating as standard compute resources within your Virtual Private Cloud (VPC), C4A metal instances inherit and leverage the same security policies, audit logging, and network controls as virtual machines. Instances are designed to appear as physical servers to your toolchain and support common monitoring and security agents, allowing for straightforward integration with your existing Google Cloud environments. This integration extends to storage, where network-attached Hyperdisk allows you to manage persistent disks using the same snapshot and resizing tools your teams already use for your virtual machine fleet.

“For our build system, true isolation is paramount. Running on Google Cloud’s new C4A metal instance on Axion enables us to isolate our package builds with a strong hypervisor security boundary without compromising on build performance." - Matthew Moore, Founder and CTO, Chainguard, Inc

Better together: the Axion C and N series

The addition of C4A metal to the Arm-based Axion portfolio allows customers to lower TCO by matching the right infrastructure to every workload. While Axion C4A virtual machines optimize for consistently high performance and N4A virtual machines (now in preview) optimize for price-performance and flexibility, C4A metal addresses the critical need for direct hardware access by specialized applications that require a non-virtualized Arm environment.

For example, an Android development company could create a highly efficient CI/CD pipeline by using C4A virtual machines for the build farm. For large-scale testing, they could use C4A metal to run Cuttlefish virtual devices directly on the physical hardware, eliminating nested virtualization overhead. To enable even higher fidelity, they can run Cuttlefish hybrid devices on C4A metal, reusing the system images from their physical hardware. Concurrently, supporting infrastructure such as CI/CD orchestrators and artifact repositories could run on cost-effective N4A instances, using Custom Machine Types to right-size resources and minimize operational expenses.

Coming soon to preview

C4A metal is scheduled for preview soon. Please fill this form to sign up for early access and additional updates.

From silicon to softmax: Inside the Ironwood AI stack

Thu, 06 Nov 2025 13:00:00 +0000

As machine learning models continue to scale, a specialized, co-designed hardware and software stack is no longer optional, it’s critical. Ironwood, our latest generation Tensor Processing Unit (TPU), is the cutting-edge hardware behind advanced models like Gemini and Nano Banana, from massive-scale training to high-throughput, low-latency inference. This blog details the core components of Google's AI software stack that are woven into Ironwood, demonstrating how this deep co-design unlocks performance, efficiency, and scale. We cover the JAX and PyTorch ecosystems, the XLA compiler, and the high-level frameworks that make this power accessible.

1. The co-designed foundation

Foundation models today have trillions of parameters that require computation at ultra-large scale. We designed the Ironwood stack from the silicon up to meet this challenge.

The core philosophy behind the Ironwood stack is system-level co-design, treating the entire TPU pod not as a collection of discrete accelerators, but as a single, cohesive supercomputer. This architecture is built on a custom interconnect that enables massive-scale Remote Direct Memory Access (RDMA), allowing thousands of chips to exchange data directly at high bandwidth and low latency, bypassing the host CPU. Ironwood has a total of 1.77 PB of directly accessible HBM capacity, where each chip has eight stacks of HBM3E, with a peak HBM bandwidth of 7.4 TB/s and capacity of 192 GiB.

Unlike general-purpose parallel processors,TPUs are Application-Specific Integrated Circuits (ASICs) built for one purpose: accelerating large-scale AI workloads. The deep integration of compute, memory, and networking is the foundation of their performance. At a high level, the TPU consists of two parts:

Hardware core: The TPU core is centered around a dense Matrix Multiply Unit (MXU) for matrix operations, complemented by a powerful Vector Processing Unit (VPU) for element-wise operations (activations, normalizations) and SparseCores for scalable embedding lookups. This specialized hardware design is what delivers Ironwood's 42.5 Exaflops of FP8 compute.

Software target: This hardware design is explicitly targeted by the Accelerated Linear Algebra (XLA) compiler, using a software co-design philosophy that combines the broad benefits of whole-program optimization with the precision of hand-crafted custom kernels. XLA's compiler-centric approach provides a powerful performance baseline by fusing operations into optimized kernels that saturate the MXU and VPU. This approach delivers good "out of the box" performance with broad framework and model support. This general-purpose optimization is then complemented by custom kernels (detailed below in the Pallas section) to achieve peak performance on specific model-hardware combinations. This dual-pronged strategy is a fundamental tenet of the co-design.

The figure below shows the layout of the Ironwood chip:

This specialized design extends to the connectivity between TPU chips for massive scale-up and scale-out for a total of 88473.6 Tbps (11059.2TB/s) for a complete Ironwood superpod.

The building block: Cubes and ICI. Each physical Ironwood host has four TPU chips. A single rack of these hosts has 64 Ironwood chips and forms a “cube”. Within this cube, every chip is connected via multiple high-speed Inter-Chip Interconnect (ICI) links that form a direct 3D Torus topology. This creates an extremely dense, all-to-all network fabric, enabling massive bandwidth and low latency for distributed operations within the cube.

Scaling with OCS: Pods and Superpods To scale beyond a single cube, multiple cubes are connected using an Optical Circuit Switch (OCS) network. This is a dynamic, reconfigurable optical network that connects entire cubes, allowing the system to scale from a small "pod" (e.g., a 256-chip Ironwood pod with four cubes) to a massive "superpod" (e.g., a 9,216-chip system with 144 cubes). This OCS-based topology is key to fault tolerance. If a cube or link fails, the OCS fabric manager instructs the OCS to optically bypass the unhealthy unit and establish new, complete optical circuits connecting only the healthy cubes, swapping in a designated spare. This dynamic reconfigurability allows for both resilient operation and the provisioning of efficient "slices" of any size. For the largest-scale systems, into the hundreds of thousands of chips, multiple superpods can then be connected via a standard Data-Center Network (DCN).

Chips can be configured in different “slices” with different OCS topologies as shown below.

Each chip is connected to 6 other chips in the 3D torus and provides 3 distinct axes for parallelism.

Ironwood delivers this performance while focusing on power efficiency, allowing AI workloads to run more cost-effectively. Ironwood perf/watt is 2x relative to Trillium, our previous-generation TPU. Our advanced liquid cooling solutions and optimized chip design can reliably sustain up to twice the performance of standard air cooling even under continuous, heavy AI workloads. Ironwood is nearly 30x more power efficient than our first Cloud TPU from 2018 and is our most power-efficient chip to date.

It’s the software stack's job to translate high-level code into optimized instructions that leverage the full power of the hardware. The stack supports two primary frameworks: the JAX ecosystem, which offers maximum performance and flexibility, as well as PyTorch on TPUs, which provides a native experience for the PyTorch community.

2. Optimizing the entire AI lifecycle

We use the principle of a co-designed Ironwood hardware and software stack to deliver maximum performance and efficiency across every phase of model development, with specific hardware and software capabilities tuned for each stage.

Pre-training: This phase demands sustained, massive-scale computation. A full 9,216-chip Ironwood superpod leverages the OCS and ICI fabric to operate as a single, massive parallel processor, achieving maximum sustained FLOPS utilization through different data formats. Running a job of this magnitude also requires resilience, which is managed by high-level software frameworks like MaxText, detailed in Section 3.3, that handle fault tolerance and checkpointing transparently.

Post-training (Fine-tuning and alignment): This stage includes diverse, FLOPS-intensive tasks like supervised fine-tuning (SFT) and Reinforcement Learning (RL), all requiring rapid iteration. RL, in particular, introduces complex, heterogeneous compute patterns. This stage often requires two distinct types of jobs to run concurrently: high-throughput, inference-like sampling to generate new data (often called 'actor rollouts'), and compute-intensive, training-like 'learner' steps that perform the gradient-based updates. Ironwood’s high-throughput, low-latency network and flexible OCS-based slicing are ideal for this type of rapid experimentation, efficiently managing the different hardware demands of both sampling and gradient-based updates. In Section 3.3, we discuss how we provide optimized software on Ironwood — including reference implementations and libraries — to make these complex fine-tuning and alignment workflows easier to manage and execute efficiently.

Inference (serving): In production, models must deliver low-latency predictions with high throughput and cost-efficiency. Ironwood is specifically engineered for this, with its large on-chip memory and compute power optimized for both the large-batch "prefill" phase and the memory-bandwidth-intensive "decode" phase of large generative models. To make this power easily accessible, we’ve optimized state-of-the-art serving engines. At launch, we’ve enabled vLLM, detailed in Section 3.3, providing the community with a top-tier, open-source solution that maximizes inference throughput on Ironwood.

3. The software ecosystem for TPUs

The TPU stack, and Ironwood’s stack in particular, is designed to be modular, allowing developers to operate at the level of abstraction they need. In this section, we focus on the compiler/runtime, framework, and AI stack libraries.

3.1 The JAX path: Performance and composability

JAX is a high-performance numerical computing system co-designed with the TPU architecture. It provides a familiar NumPy-like API backed by powerful function transformations:

jit (Just-in-Time compilation): Uses the XLA compiler to fuse operations into a single, optimized kernel for efficient TPU execution.
grad (automatic differentiation): Automatically computes gradients of Python functions, the fundamental mechanism for model training.
shard_map (parallelism): The primitive for expressing distributed computations, allowing explicit control over how functions and data are sharded across a mesh of TPU devices, directly mapping to the ICI/OCS topology.

This compositional approach allows developers to write clean, Pythonic code that JAX and XLA transform into highly parallelized programs optimized for TPU hardware. JAX is what Google Deepmind and other Google teams use to build, train, and service their variety of models.

For most developers, these primitives are abstracted by high-level frameworks, like MaxText, built upon a foundation of composable, production-proven libraries:

Optax: A flexible gradient processing and optimization library (e.g., AdamW)

Orbax: A library for asynchronous checkpointing of distributed arrays across large TPU slices

Qwix: A JAX quantization library supporting Quantization Aware Training (QAT) and Post-Training Quantization (PTQ)

Metrax: A library for collecting and processing evaluation metrics in a distributed setting

Tunix: A high-level library for orchestrating post-training jobs

Goodput: A library for measuring and monitoring real-time ML training efficiency, providing a detailed breakdown of badput (e.g., initialization, data loading, checkpointing)

3.2 The PyTorch path: A native eager experience

To bring Ironwood's power to the PyTorch community, we are developing a new, native PyTorch experience complete with support for a “native eager mode”, which executes operations immediately as they are called. Our goal is to provide a more natural and developer-friendly way to access Ironwood's scale, minimizing the code changes and level of effort required to adapt models for TPUs. This approach is designed to make the transition from local experimentation to large-scale training more straightforward.

This new framework is built on three core principles to ensure a truly PyTorch-native environment:

Full eager mode: Enables the rapid prototyping, debugging, and research workflows that developers expect from PyTorch.
Standard distributed APIs: Leverages the familiar torch.distributed API, built on DTensor, for scaling training workloads across TPU slices.
Idiomatic compilation: Uses torch.compile as the single, unified path to JIT compilation, utilizing XLA as its backend to trace the graph and compile it into efficient TPU machine code.

This ensures the transition from local experimentation to large-scale distributed training is a natural extension of the standard PyTorch workflow.

3.3 Frameworks: MaxText, PyTorch on TPU, and vLLM

While JAX and PyTorch provide the computational primitives, scaling to thousands of chips is a supercomputer management problem. High-level frameworks handle the complexities of resilience, fault tolerance, and infrastructure orchestration.

MaxText (JAX): MaxText is an open-source, high-performance LLM pre-training and post-training solution written in pure Python and JAX. MaxText demonstrates optimized training on its library of popular OSS models like DeepSeek, Qwen, gpt-oss, Gemma, and more. Whether users are pre-training large Mixture-of-Experts (MoE) models from scratch, or leveraging the latest Reinforcement Learning (RL) techniques on an OSS model, MaxText provides tutorials and APIs to make things easy. For scalability and resiliency, MaxText leverages Pathways, which was originally developed by Google DeepMind and now provides TPU users with differentiated capabilities like elastic training and multi-host inference during RL.

PyTorch on TPU: We recently shared our proposal about our PyTorch native experience on TPUs at Pytorch Conference 2025, including an early preview of training on TPU with minimal code changes. In addition to the framework itself, we are working with the community (RFC), investing in reproducible recipes, reference implementations, and migration tools to enable PyTorch users to use their favorite frameworks on TPUs. Expect further updates as this work matures.

vLLM TPU (Serving): vLLM TPU is now powered by tpu-inference, an expressive and powerful new hardware plugin that unifies JAX and PyTorch under a single lowering path – meaning both frameworks are translated to optimized TPU code through one common, shared backend. This new unified backend is not only faster than the previous generation of vLLM TPU but also offers broader model coverage. This integration provides more flexibility to JAX and PyTorch users, running PyTorch models performantly with no code changes while also extending native JAX support, all while retaining the standard vLLM user experience and interface.

3.4 Extreme performance: Custom kernels via Pallas

While XLA is powerful, cutting-edge research often requires novel algorithms e.g. new attention mechanisms, custom padding to handle dynamic ragged tensors and other optimizations for custom MoE models that the XLA compiler cannot yet optimize.

The JAX ecosystem solves this with Pallas, a JAX-native kernel programming language embedded directly in Python. Pallas presents a unified, Python-first experience, dramatically reducing cognitive load and accelerating the iteration cycle. Other platforms lack this unified, in-Python approach, forcing developers to fragment their workflow. To optimize these operations, they must drop into a disparate ecosystem of lower-level tools—from DSLs like Triton and cuTE to raw CUDA C++ and PTX. This introduces significant mental overhead by forcing developers to manually manage memory, streams, and kernel launches, pulling them out of their Python-based environment

This is a clear example of co-design. Developers use Pallas to explicitly manage the accelerator's memory hierarchy, defining how "tiles" of data are staged from HBM into the extremely fast on-chip SRAM to be operated on by the MXUs. Pallas has two main parts to it.

Pallas: The developer defines the high-level algorithmic structure and memory logistics in Python.
Mosaic: This compiler backend translates the Pallas definition into optimized TPU machine code. It handles operator fusion, determines optimal tiling strategies, and generates software pipelines to perfectly overlap data transfers (HBM-to-SRAM) with computation (on the MXUs), with the sole objective of saturating the compute units.

Because Pallas kernels are JAX-traceable, they are fully compatible with jit, vmap, and grad. This stack provides Python-native extensibility for both JAX and PyTorch, as PyTorch users can consume Pallas-optimized kernels without ever leaving the native PyTorch API. Pallas kernels for PyTorch and JAX models, on both TPU and GPU, are available via Tokamax, the ML ecosystem’s first multi-framework, multi-hardware kernel library.

3.5 Performance engineering: Observability and debugging

The Ironwood stack includes a full suite of tools for performance analysis, bottleneck detection, and debugging, allowing developers to fully optimize their workloads and operate large scale clusters reliably,

Cloud TPU metrics: Exposes key system-level counters (FLOPS, HBM bandwidth, ICI traffic) to Google Cloud Monitoring that can then be exported to popular monitoring tools like Prometheus.

TensorBoard: Visualizes training metrics (loss, accuracy) and hosts the XProf profiler UI.

XProf (OpenXLA Profiler): The essential toolset for deep performance analysis. It captures detailed execution data from both the host-CPU and all TPU devices, providing:

- Trace Viewer: A microsecond-level timeline of all operations, showing execution, collectives, and "bubbles" (idle time).
- Input Pipeline Analyzer: Diagnoses host-bound vs. compute-bound bottlenecks.
- Op Profile: Ranks all XLA/HLO operations by execution time to identify expensive kernels.
- Memory Profiler: Visualizes HBM usage over time to debug peak memory and fragmentation.

Debugging Tools:

- JAX Debugger (jax.debug): Enables print and breakpoints from within jit-compiled functions.
- TPU Monitoring Library: A real-time diagnostic dashboard (analogous to nvidia-smi) for live debugging of HBM utilization, MXU activity, and running processes.

Beyond performance optimization, developers and infra admins can view fleet efficiency and goodput metrics at various levels (e.g., job, reservation) to ensure maximum utilization of their TPU infrastructure.

4. Conclusion

The Ironwood stack is a complete, system-level co-design, from the silicon to the software. It delivers performance through a dual-pronged strategy: the XLA compiler provides broad, "out-of-the-box" optimization, while the Pallas and Mosaic stack enables hand-tuned kernel performance.

This entire co-designed platform is accessible to all developers, providing first-class, native support for both the JAX and the PyTorch ecosystem. Whether you are pre-training a massive model, running complex RL alignment, or serving at scale, Ironwood provides a direct, resilient, and high-performance path from idea to supercomputer.

Get started today with vLLM on TPU for inference and MaxText for pre-training and post-training.

Expanding our NVIDIA partnership: Now shipping A4X Max, Vertex AI Training, and more

Tue, 28 Oct 2025 18:30:00 +0000

Today's AI models are moving from billions to trillions of parameters, and are capable of complex, multi-modal reasoning. This leap in sophistication demands a new class of purpose-built infrastructure and software to handle the immense computational and memory requirements of these next-generation models.

At Google Cloud, we’re committed to empowering developers and organizations to build and deploy what's next in AI. Today, we are excited to deepen our partnership with NVIDIA with a suite of new capabilities that strengthens our platform for the entire AI lifecycle:

New A4X Max instances powered by NVIDIA’s GB300 NVL72, purpose-built for multimodal AI reasoning tasks
Google Kubernetes Engine (GKE), now supporting Dynamic Resource Allocation Kubernetes Network Driver (DRANET), boosting bandwidth in distributed AI/ML workloads
GKE Inference Gateway, now integrating with NVIDIA NeMo Guardrails
Vertex AI Model Garden to feature NVIDIA Nemotron models
Vertex AI Training recipes on top of the NVIDIA NeMo Framework and NeMo-RL

Let’s take a closer look at these developments.

A4X Max with NVIDIA GB300 GPUs

A4X Max is now shipping in production. These new instances, powered by NVIDIA GB300 NVL72, are optimized for the most demanding, multimodal AI reasoning workloads. A4X Max includes 72 Blackwell Ultra GPUs and 36 NVIDIA Grace CPUs connected with NVIDIA’s fifth-generation high-speed GPU interconnect NVIDIA NVLink to function as a single, unified compute platform with shared memory and high-bandwidth communication. Together with Google's Titanium ML adapter and Google Cloud's Jupiter network fabric, A4X Max is purpose-built to scale to tens of thousands of GPUs in non-blocking, rail-optimized clusters. Compared to A4X powered by NVIDIA GB200 NVL72, A4X Max delivers 2x the network bandwidth on each system.

A4X Max leverages Google Cloud’s Cluster Director, letting you combine optimized compute, networking, and Google’s storage offerings into a cohesive, performant, and easily managed environment. Cluster Director manages the complete lifecycle of A4X Max clusters — from provisioning and topology-aware placement across the NVL72 domains, to providing powerful observability and resiliency capabilities. It integrates with optimized storage solutions like Managed Lustre, while a managed pre-configured Slurm environment offers fault-tolerant scalable job scheduling for A4X Max. Cluster Director also provides deep observability into job and system performance across the GPUs, NVLink and DC networking fabrics. To maximize throughput, Cluster Director helps ensure high reliability with features like automatic straggler detection and in-job recovery. Cluster Director capabilities like topology aware scheduling, maintenance management, and faulty node reporting are also available transparently through Google Kubernetes Engine (GKE), enabling customers to stay in the GKE environment while running A4X Max.

What all this this means for your workloads:

Optimized reasoning and inference: With its 72-GPU NVLink domain, delivering 1.5x FP4 FLOPs, 1.5x HBM memory capacity, and 2x the network bandwidth compared to A4X, A4X Max is specifically designed for low-latency inference, especially for the largest reasoning models. When integrated with GKE Inference Gateway, you benefit from prefix-aware load balancing, improving Time to First Token latency for prefix-heavy workloads. Disaggregated serving can also be enabled to further optimize performance. This is achieved by leveraging Inference Gateway, llm-d, and vLLM together, resulting in significant throughput improvements.
Enhanced training and serving performance: With more than 1.4 exaflops per GB300 NVL72 system, A4X Max offers a 4x increase in LLM training and serving performance compared to A3 VMs powered by NVIDIA H100 GPUs.
Maximum scalability and parallelization: Based on RDMA over Converged Ethernet (RoCE), A4X Max’s networking fabric delivers low-latency high-performance GPU-to-GPU collectives for distributed training and disaggregated serving workloads. By leveraging a new data-center-scaling design, A4X Max clusters can be 2x larger compared to A4X clusters.

The preview of A4X Max instances comes on the heels of our new G4 VMs powered by NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs, and support for NVIDIA Omniverse libraries. Taken together, these offerings underscore our commitment to delivering an end-to-end platform for every AI workload, while our deepening partnership with NVIDIA provides you with a powerful, comprehensive ecosystem to build what's next in AI.

Increased RDMA performance with GKE DRANET

Today, we’re deploying managed DRANET into production, starting with A4X Max. By enabling topology-aware scheduling of GPUs and RDMA network interface cards, DRANET boosts bus bandwidth for all-gather and all-reduce operations in distributed AI/ML workloads. This translates to improved cost efficiency due to better VM utilization. It does this by scheduling GKE Pods on nodes where the RDMA device and the GPU have the best possible connectivity. DRANET also simplifies RDMA management by making RDMA devices first-class, native resources within GKE. Learn more about DRANET for GKE here.

GKE and NVIDIA NeMo Guardrails

As organizations deploy their AI models into production, they must ensure their safety, security, and responsible behavior. Today, we are announcing the integration of NVIDIA NeMo Guardrails with GKE Inference Gateway, an extension to GKE Gateway for serving generative AI applications.

GKE Inference Gateway optimizes model serving with features like model-aware routing and autoscaling, while NeMo Guardrails add a critical layer of safety, preventing models from engaging in undesirable topics or responding to malicious prompts. Together, they offer a secure, scalable, and manageable inference solution to speed up your AI initiatives.

Vertex AI Model Garden to feature NVIDIA Nemotron models

To give developers greater choice and performance, Vertex AI Model Garden will soon have support for NVIDIA’s Nemotron family of open models as NVIDIA NIM microservices. This integration — starting with the upcoming availability of the NVIDIA Llama Nemotron Super v1.5 model — will give developers and organizations access to the NVIDIA’s latest open-weight models directly within Vertex AI. With a Vertex AI managed deployment, you can rapidly develop and deploy custom AI agents powered by Nemotron models, all while maintaining control over performance, cost, and compliance.

Models deployed through Vertex AI offer the following benefits :

Granular control over your deployments, with the ability to optimize for performance or cost by selecting from a wide range of machine types and Google Cloud regions.
Robust security by deploying models entirely within your own VPC and adhering to your VPC-SC policies.
Incredible ease of use — you can discover, license, and deploy these cutting-edge models in just a few clicks.

Vertex AI Training with NVIDIA NeMo Integration

Vertex AI Training provides the essential control and flexibility enterprises need to adapt foundation models to their proprietary data. To accelerate the creation of highly accurate, proprietary models, we are announcing expanded capabilities in Vertex AI Training that simplify and accelerate the path to developing large-scale models.

Customers benefit from a fully managed and resilient Slurm environment that simplifies large-scale training. Automated resiliency features improve cluster uptime. Our comprehensive data-science tooling removes much of the guesswork from complex model development. Finally, curated and optimized pre-training and post-training recipes built on top of standardized frameworks like NVIDIA NeMo and NeMo-RL empower builders to move from a novel idea to a production-ready, domain-specialized model with greater speed and efficiency.

Take the next steps

These updates enhance the capabilities and flexibility of our Google Cloud platform for running AI workloads. You can choose between the flexibility and control of infrastructure as a service (IaaS) with Google Compute Engine or GKE with Cluster Director; or the fully managed, end-to-end experience of Vertex AI, which provides a secure, scalable, and simplified workflow to train, tune, and manage models.

Together, these infrastructure innovations represent a significant step forward in our mission to provide a complete platform for AI development and deployment. The combination of Google Cloud’s infrastructure and NVIDIA’s latest technology provides a solid foundation for building the next generation of AI applications.

To get started with the A4X Max preview, please contact your Google Cloud sales representative. Vertex AI Training, meanwhile, has everything you need to transform your models into proprietary assets that define your business advantage. To deploy and manage AI models at scale with enterprise-grade security and efficiency, learn how GKE Inference Gateway can help you serve inference workloads. We are excited to see what you will build.

Compute

A developer’s guide to architecting reliable GPU infrastructure at scale

Operational realities of AI at scale

The business implications of infrastructure instability

Quantitative assessment: Key reliability metrics

Google Cloud’s methodology: Engineering systemic resilience

AI infrastructure efficiency: Ironwood TPUs deliver 3.7x carbon efficiency gains

Measuring AI accelerator efficiency: Compute Carbon Intensity (CCI)

The Ironwood advantage: high performance, low footprint

Operating Google’s TPU fleet more efficiently

Decoupling energy and emissions from performance

Scale sustainably with Google Cloud

A developer’s guide to training with Ironwood TPUs

Key optimization strategies for Ironwood

1. Leverage native FP8 with MaxText

2. Accelerate with Tokamax kernels

3. Offload collectives to SparseCore

4. Fine-tune the memory pipeline on VMEM

5. Choose optimal sharding strategies

The Ironwood advantage: System-level performance

Google Cloud and NVIDIA expand AI innovation across industries at GTC 2026

Accelerating AI workloads with G4 VMs

Introducing fractional G4 VMs

Scaling AI Hypercomputer with NVIDIA Vera Rubin NVL72

Delivering efficiency across the AI infrastructure stack

Advancing Vertex AI training and Model Garden

Empowering public sector AI startups

Co-engineering collaboration powers every layer of the AI stack

H4D VMs, now GA, deliver exceptional performance and scaling for HPC workloads

Faster time to solution across domains and scales

A new standard for HPC price/performance

Comprehensive HPC management

What customers and partners are saying

Experience H4D today

Simpler billing, clearer savings: A FinOps guide to updated spend-based CUDs

1. What is the spend-based CUD data change all about?

2. How do I validate my savings before and after the changes?

3. What other capabilities are in the new CUD Analysis?

4. How much commitment should I buy?

5. Is there anything else I should know about Flex CUDs?

Save more with expanded coverage for Compute Flex CUDs

High-performance inference meets serverless compute with NVIDIA RTX PRO 6000 on Cloud Run

A powerful GPU platform

Get started

Unlock 2x better price-performance with Axion-based N4A VMs, now generally available

Redefining general-purpose compute and enabling AI inference

Granular control with Custom Machine Types and Hyperdisk

A growing Arm-based Axion portfolio for customer choice

Get started with N4A today

Scaling WideEP Mixture-of-Experts inference with Google Cloud A4X (GB200) and NVIDIA Dynamo

The modern inference stack

Infrastructure layer: The A4X rack-scale architecture

Serving layer: NVIDIA Dynamo

Orchestration layer: Mapping software to hardware

Performance validation

Looking ahead

Get started today

Simplify VM OS agent management at scale: Introducing VM Extensions Manager

How to get started with VM Extensions Manager

A flexible system for standardization and control

Automate AI and HPC clusters with Cluster Director, now generally available

Cluster Director, now GA

How Cluster Director supports each phase of deployment

Cluster Director support for Slurm on GKE, now in preview

Try Cluster Director today

Google named a Leader in The Forrester Wave™: AI Infrastructure Solutions, Q4 2025

Accelerating time-to-value with an integrated system

Managing AI cost and complexity

Mitigating strategic risk through flexibility and choice

Systems you can rely on

Building on a secure foundation

Delivering continuous AI innovation

AI agents are here. Is your infrastructure ready?

The TCO crisis in an age of agents

Closing the gap: From fragmented stacks to integrated systems

A message from Google Cloud

Nutanix NC2 is now officially supported on Google Cloud

Key use cases to accelerate your cloud journey

The architecture

Connect your data to Google Cloud AI and analytics