AI infrastructure

Minimize idle accelerators: Native RL job interleaving with co-operative time-slicing in llm-d

Thu, 23 Jul 2026 17:00:00 +0000

The math behind reinforcement learning (RL) post-training for large language models (LLMs) is notoriously unforgiving. As frontier AI labs push the boundaries of reasoning and coding models using RL post-training algorithms like Group Relative Policy Optimization (GRPO), they routinely hit hard architectural and infrastructure constraints. While much of the industry's focus remains on acquiring raw accelerator capacity, infrastructure efficiency is equally critical for achieving the high velocity needed to run multiple RL jobs and drive models to higher levels of intelligence. At scale, distributed RL suffers from severe resource bottlenecks because synchronous sampling and training run as strictly sequential phases, causing trainer and sampler resources to alternate sitting idle. Meanwhile, asynchronous architectures attempt to overlap these phases, but trainers still experience frequent idle gaps while waiting for specific trajectory batches to finish before starting the next cycle.

Today, we are introducing a solution to this structural waste: co-operative time-slicing through the llm-d project. By treating discrete RL steps — such as sampling rollouts and gradient training — as dynamic, schedulable entities, we can interleave independent RL jobs onto shared physical hardware. Our initial benchmarks show that this platform-level multiplexing increases aggregate accelerator duty cycles from a ~40% baseline up to 70% without impacting model convergence or accuracy. This improves price-performance and lowers TCO significantly by eliminating wasted compute accrued over time.

For synchronous setups, the platform interleaves both samplers and trainers to minimize alternating idle windows, while asynchronous workloads leverage time-slicing to dynamically reclaim and utilize the fragmented idle gaps between RL-trainer iterations.

Throughout this blog, we will describe the time-slicing solution, detailing the technical flows, current release and future roadmap.

llm-d for RL infrastructure efficiency (the bigger picture)

From the get-go, we anticipated the severe infrastructure bottlenecks of large-scale RL post-training and invested in addressing infrastructure inefficiency for RL workloads.

We have built llm-d into a highly composable infrastructure stack for inference, agentic and RL workloads focused on eliminating accelerator idle time. The llm-d stack for RL features:

Throughput-driven inference (llm-d-router): A mature, production-tested engine deployed across RL workloads and focused on maximizing rollout generation throughput to continuously saturate the pipeline.
High-velocity Agent Sandbox (recipe): Tested for scale and density, and helping deliver secure, sub-second tool-use and isolated code execution during rollout generation and evals. Agent Sandbox serves as the high-speed intake manifold for reward signal generation, helping ensure the Sandbox never becomes the latency bottleneck that starves your time-sliced NVIDIA GPUs.
Core pipeline primitives: To combat reliability and speed in weight transfer, we are building Weight Propagation Interface (WPI), as well as focusing on improving overall observability and reliability for RL.

The efficiency problem with RL loops

Distributed RL post-training operates as a fragmented, continuous cycle alternating between generation (sampling rollouts) and optimization (gradient updates). Because traditional cloud infrastructure is designed for continuous, steady-state workloads, standard Kubernetes clusters can’t adapt to this alternating cadence.

At scale, this structural cadence introduces two massive systemic inefficiencies:

Idle accelerators: Because these phases occur sequentially, GPU clusters sit completely idle (0% utilization) for 40% to 60% of their lifecycle. Trainers sit idle waiting for sampling rollouts to finish; samplers sit idle during gradient updates and weights distribution. This could represent millions in wasted capital annually.
Locked-in context: RL training and samplers hold their accelerator allocations for the entirety of their runtime even during idle phases because the NVIDIA CUDA context and all device memory needs to remain resident. Standard schedulers treat these pods as static, siloed allocations rather than aligning them to the alternating, phase-level states of the live RL loop, leaving valuable hardware locked up even during inactive phases.

Importantly, this is not just a synchronous RL problem. Asynchronous variants overlap generation and training, but they do not fully mitigate idle time. Generation remains the inherent bottleneck of the RL loop, meaning trainer accelerators still starve while waiting for rollout data to accumulate. The closer an asynchronous job runs to on-policy, the larger those idle windows become — bounded staleness limits how far generation and training can drift apart, stalling the pipeline whenever fresh rollouts are not ready.

How co-operative time-slicing (RL job interleaving) helps

To eliminate idle accelerators during RL jobs, co-operative time-slicing under the llm-d project allows the infrastructure to dynamically interleave independent RL jobs onto shared hardware blocks rather than forcing hardware to wait on upstream phases. This helps drive aggregate accelerator utilization up without altering the underlying model convergence or accuracy.

When Job A goes idle at a phase boundary in synchronous RL (or stalls on fresh rollout data in asynchronous RL), the infrastructure time-slices the physical accelerators, swapping in the active sampling or training phase of Job B. Under the hood, a swap is a checkpoint/restore: Job A's entire device state is checkpointed out of accelerator memory into host DRAM, and Job B's previously saved state is restored in its place. Because only one job's state ever occupies the accelerator at a time, steps alternate safely without framework-level interference or out-of-memory (OOM) faults.

Time-slicing: High-level architecture

The time-slicing system architecture is organized into three layers: workload-scoped (application logic), cluster-scoped (coordination), and node-scoped (hardware management).

Workload-scoped layer (application runtime)
This is where the user's code runs — training loops, inference servers, and RL frameworks. The new addition is the time-slice client library, which exposes two gRPC APIs on the time-slice orchestrator: acquire() to request exclusive accelerator access, and yield() to release it. The user wraps any accelerator-touching phase with these calls to signal phase boundaries to the orchestrator. Everything else — the ML framework (PyTorch FSDP, vLLM, etc.), the CUDA context, the accelerator memory allocations — runs unmodified.

Cluster-scoped layer (control and orchestration plane)
This layer decides which job gets accelerator access, and when. Jobs that share the same physical accelerators — for example, two RL jobs interleaving on the same set of GPU nodes — are placed into a group. For each group, the time-slice orchestrator maintains a lock queue — an ordered list of jobs waiting for exclusive access to that group's accelerators. Only the job at the head of the queue holds the lock and runs on the hardware; all the other jobs wait, blocked on their acquire() call. When the running job calls yield(), the orchestrator passes the lock to the next job in the queue and triggers a coordinated context switch across every node in the group. In the future, a workload placement optimizer will be able to profile workload phase patterns and automatically pair jobs with complementary idle phases, removing the need for the user to explicitly indicate job groupings.

Node-scoped layer (hardware and data plane isolation)
This layer performs the checkpoint/restore swap on each accelerator node. The snapshot agent, a privileged DaemonSet, receives directives from the orchestrator and translates them into hardware-level operations — pausing accelerator processes, serializing device state to host DRAM, and restoring it when the job regains access. The agent is built around a pluggable backend interface, with cuda-checkpoint as the first implementation (more to come). Future backends will introduce faster snapshot mechanisms and more selective approaches, such as offloading specific memory addresses like LoRA adapters instead of full device state. The agent itself is designed to run standalone outside Kubernetes for bare metal and Slurm environments.

The flow: How it all comes together

When a workload finishes its current accelerator phase, its time-slice client library calls yield() to the time-slice orchestrator to release access. The orchestrator initiates the context switch by sending directives to the snapshot agent on each node in the group. The agent freezes the yielding workload's processes and moves its device state from accelerator memory into host DRAM.

With the accelerators vacated, the orchestrator grants the group lock to the next workload waiting in the queue. It directs the Snapshot Agents on those nodes to restore that workload's previously saved state from host DRAM back into accelerator memory, then unblocks the workload's pending acquire() call. The workload resumes execution exactly where it left off — no container restart, no framework reinitialization, no model reload from storage.

The yielding workload remains warm in host DRAM. When the orchestrator grants it the lock again, the Snapshot Agents perform the same swap in reverse.

Developer experience (client-side)
Researchers want to focus on core modeling logic rather than wrestling with low-level CUDA context switching or custom scheduling loops. If you use Ray or a similar platform to orchestrate your RL job, using time-slicing will have a minimal impact on the client side. In fact, there may not be any impact on the client side at all if you are queuing the training and sampling jobs separately at the platform level.

code_block: <ListValue: [StructValue([('code', 'from timeslice import TimeSliceOrchestratorClient\r\n\r\norchestrator = TimeSliceOrchestratorClient(target="orchestrator:50051")\r\n\r\n@orchestrator.on_accelerators(group_id="trainer-group")\r\ndef train_phase(model, trajectories):\r\n return model.update(trajectories)\r\n\r\n@orchestrator.on_accelerators(group_id="sampler-group")\r\ndef generate_phase(model, prompts):\r\n return model.generate(prompts)\r\n\r\n# Standard sequential loop — interleaved with other jobs under the hood\r\nfor epoch in range(EPOCHS):\r\n trajectories = generate_phase(policy, dataset)\r\n rewards = compute_rewards(trajectories)\r\n train_phase(policy, rewards)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f5164ff3d30>)])]>

Current release and future outlook

Today we are releasing the full time-slicing stack: the Snapshot Agent, the Accelerator Orchestrator, and the Python client libraries, each with a user guide for integrating time-slicing into your RL workloads.

Key roadmap highlights include:

Latency and state optimization: Expanding the Snapshot Agent with faster checkpoint/restore backends to minimize context-switch overhead, alongside application-aware backends for selective memory region snapshotting (e.g., swapping LoRA adapters instead of full model weights).
Automated scheduling and onboarding: Introducing an automated scheduler to profile running processes, identify time-sliceable structures, and handle job placement dynamically.
Cross-hardware compatibility: Extending data plane support beyond GPUs to TPUs and custom accelerator architectures.

Get started

Building robust, highly optimized RL infrastructure requires tight collaboration with the engineers and researchers running these workloads at scale.

If you are currently wrestling with low GPU utilization, synchronization stalls, or complex scheduling logic in your post-training pipelines, time-slicing can help. To get started, check out the following resources, and don’t forget to leave us your feedback!

Start using time-slicing during your RL run immediately with these user guides.
Try llm-d-router (kubernetes native) or the RL Scheduler (python library) user-guide for improved sampling throughput during the RL generation phase.
Explore the Weight Propagation Interface repo.
Join the discussion in the #sig-rl channel in the llm-d Slack.
Contribute by sharing your reference implementations, benchmarks, and edge cases to help us refine this path.

_{Thank you to Dolev Ish Am and Bogdan Berce for their contributions to this blog post.}

Your AI agents are ready. Is your data?

Thu, 23 Jul 2026 16:00:00 +0000

What’s one of the biggest bottlenecks stopping organizations from scaling their AI initiatives? It isn’t the capabilities of today’s models — it’s their access to business context and semantic meaning.

In the agentic era, enterprises need to go beyond simply storing data to activating it with trusted context, moving from passive systems of record to proactive systems of action.

But AI agents operate with nonlinear speed; for example, a single prompt can trigger the agent to independently browse, query, and execute across multiple systems, placing stress on the underlying infrastructure. If the compute, networking, and storage layers aren't optimized for agentic AI, the data platform sitting on top of them will buckle.

It’s no wonder that, according to our State of infrastructure report, 83% of organizations believe they require infrastructure upgrades to support production-grade agentic AI systems.

To solve this problem, we introduced the Agentic Data Cloud at Google Cloud Next 2026; unifying your data, AI models, and operational databases into a single System of Action. To make an Agentic Data Cloud work, it must be AI-native from the chip to the model. The underlying infrastructure must be able to accommodate agentic load.

Google’s Agentic Data Cloud

Let’s explore how the right infrastructure foundation empowers an Agentic Data Cloud to solve the biggest data challenges organizations face today.

Overcoming a lack of context

To be effective, agentic systems require access to context that is often found in fragmented data systems and legacy architectures. This can make it hard for agents to get this context, leading to incomplete, inaccurate results. In fact, our report found that 43% of IT leaders cite “difficulty integrating with legacy APIs and data sources” as their biggest agentic AI infrastructure gap.

But organizations cannot simply move massive datasets and connect them to AI without increasing complexity and cost.

Our Agentic Data Cloud solves this by leveraging a borderless Lakehouse running on open, flexible infrastructure. By accessing powerful native engines like BigQuery and Spanner over open standards (Apache Spark, Apache Iceberg), agents can read, reason over, and activate data across environments as if it were local, bypassing the latency and costs of traditional setups.

Escaping unnecessary manual work

Scaling agents on a patchwork of disconnected systems can create significant bottlenecks. In our research, 81% of leaders called out operational complexity and engineering overhead as top unforeseen expenses when scaling AI, citing the time engineers spend doing manual work to patch together AI agents across disparate systems.

To move from thinking to doing, agents must be able to connect real-time data across both analytical and operational sources. This requires vertical integration. When an Agentic Data Cloud is built on an AI-native infrastructure where the models, data systems, and underlying accelerators are co-designed, there are fewer network hops and tooling is better integrated. This unified system allows an agent to reach an insight and trigger secure transactions without the typical engineering overhead.

Bringing trust and knowledge to the data

It’s not enough for agents to just discover and query data. To take safe, accurate actions, agents also need rich context and business logic. Yet, 36% of leaders cite a lack of specialized, high-throughput vector databases used for AI model grounding, as a key infrastructure gap, hindering their ability to give agents context.

In order to work to their full potential, agents need a foundation which is built to read and write data systems in real-time, including legacy ERPs and third-party CRMs. It also gives them the long-term memory to recall a user’s preference from, say, three weeks ago, while executing a complex task today. And without this real-time automation, agents have to re-process data for every single query.

To provide context for AI, organizations are using Knowledge Catalog to aggregate and enrich data in their data lakes, and enable agentic searches. By extracting meaning from unstructured data and automatically generating semantics, the catalog acts as an active reasoning layer. That catalog in turn, must be backed by high-throughput infrastructure, so that agents can retrieve the right context.

The path forward

To turn AI into a true competitive advantage, it’s time to build a connected, active data ecosystem. Giving your agents seamless access to all of your data is a must to move from pilots to production, and this must be supported by an infrastructure that can handle the demands of the agentic era. The winners in 2026 and beyond won’t necessarily be the ones with the smartest agents. They’ll be the ones who can feed those agents the right knowledge — securely, cost-effectively, and at scale. Is your data ready for the agentic era?

See how leaders are taking an AI-optimized approach to architecture in the State of infrastructure in the agentic AI era report.

IDC: Why the right networking approach is foundational to agentic AI

Wed, 15 Jul 2026 16:00:00 +0000

Editor’s note: Today we hear from IDC on the results of its 2026 AI in Networking Special Report Survey exploring the enterprises' concerns about networking infrastructure to support the rise of agentic AI in their organizations. The survey was sponsored by Google Cloud.

Enterprises are moving quickly on AI pilots, but the move from pilot to production remains uneven. While AI models remain important, IDC research indicates that the pilot-to-production bottleneck is primarily infrastructure-centric, with core networking concerns emerging as one of the leading drivers of AI project delays and abandonment. In IDC's 2026 AI in Networking Special Report Survey:

32.6% of respondents cite security concerns: As AI workflows become more distributed and autonomous, enforcing consistent security and governance becomes more difficult.
26.8% of respondents cite challenges in automation: Manual operations and fragmented controls can slow deployment and make AI environments harder to scale.
24.7% of respondents cite staff time and talent restrictions: Limited skills and operational bandwidth can constrain an organization's ability to move AI initiatives into production.

Agentic AI specifically heightens these concerns by introducing more distributed and dynamic interactions across applications, services, APIs, tools, and data sources. In production environments, these interactions often span different agent frameworks, model providers, clouds, open-source tools, SaaS APIs, and internal applications, expanding both the operational scope and the security and governance surface area.

Networking for operational control, security, and governance at scale

Networking is the primary enabler of agentic interactions and plays a foundational role for intracloud and intercloud network- and services-layer connectivity, end-to-end security, and consistent governance. In agentic systems, networking increasingly extends into tighter service-centric controls that govern how distributed services identify one another, communicate, and exchange data securely. While AI workloads in general are increasing east-west traffic demands, agentic AI adds an additional layer of complexity by creating dynamic interactions that require tighter policy, visibility, and control closer to the application workflow.

From an infrastructure perspective, networking is much more than just a connectivity function. It is part of the infrastructure platform control plane that applies policy-based controls, supports observability, and helps maintain consistent security and governance across an AI agent's activity. This is significant because framework-level controls alone become insufficient in environments where agents and services span different runtimes, clouds, deployment models, and operating domains.

That is why an infrastructure-level approach becomes key. It does not replace application frameworks or orchestration environments, but it provides broader and more consistent policy implementation across a complex architectural landscape. As agentic AI becomes more autonomous and distributed, organizations need these controls built in as part of the infrastructure to reduce fragmented observability, inconsistent policy application, and unmanaged shadow agent activities. From a cloud infrastructure standpoint, this is where cloud network services become strategically important.

Balancing act: A platform vs. best-of-breed approach

Agentic AI systems are inherently fragmented because of underlying distributed workflows. Enterprises are already navigating a rapidly evolving landscape of business requirements, open-source components, emerging protocol standards, and new architecture patterns. In this context, choices between best-of-breed point solutions and platform-based approaches should be strategic rather than ideological.

Best-of-breed capabilities may be necessary to address specific technical requirements. But it is also true that point solutions introduced across a distributed agentic AI landscape can create inconsistent policies, operational complexity, and governance gaps. IDC research reflects this tension. In IDC’s 2026 AI in Networking Special Report Survey, organizations remained divided between platform and best-of-breed preferences for AI workloads; among respondents who favored platforms, the main reasons cited were stronger security (32.9%), reduced complexity (27.7%), and faster deployment (24.2%).

In IDC's view, a balance is important. Platforms can provide a consistent operational and policy foundation for AI deployments, but at the same time, they need to be modular and extensible to allow the inclusion of best-of-breed functionality as part of the platform toolset. The right platform for agentic AI should be open, flexible, and able to evolve. It should support integration with third-party and open-source tools, allow insertions of needed security and observability functions, and adapt without complete architectural rework.

This is a period of technology disruption. Businesses must meet their AI objectives while carefully managing dynamic agentic AI systems. In this environment, networking not only remains a connectivity piece of the AI infrastructure but becomes foundational to how organizations establish operational control, apply policy consistently, and maintain end-to-end trust across agentic workflows.

As agentic AI systems continue to evolve, the demands they place are unlikely to be addressed through best-of-breed point solutions alone. Operationalizing agentic AI at scale will require organizations to leverage the right networking approach, supported by infrastructure platforms that are open, flexible, and extensible, enabling a cohesive and adaptable security and governance framework.

Message from the sponsor
The autonomous and non-deterministic communications of agentic applications pose challenges for which the infrastructure and governance models of the cloud-native era are not prepared. In the agent-native era, an infrastructure-led approach is required to enable agentic applications at scale in production with effective governance and observability. An extensible platform based on open standards is critical in enabling the agentic journey today and through its maturity. Learn about the infrastructure imperatives and open standards that make a viable agentic infrastructure here.

Claude at scale on Google Cloud: Frontier AI, built for enterprise production

Tue, 14 Jul 2026 16:00:00 +0000

Running frontier AI in production is demanding — accelerators to manage, latency to hold steady across continents, regulated data to keep in-region, and long-context requests to serve reliably. Claude on Google Cloud is built for exactly this.

Like Monet and water lilies, frontier models and the enterprise platforms are often better together. In our case, Claude brings the reasoning, and Google Cloud brings the managed infrastructure, global reach, and compliance posture that enterprises already run on. Calling Claude becomes operationally identical to calling any other Google Cloud service — same Identity and Access Management (IAM), same VPC Service controls, same observability — so teams are able to spend their time building features instead of running inference infrastructure.

This post walks through what Claude on Google Cloud delivers in production across four areas:

Managed infrastructure that gives engineers their time back
Global endpoints that hold latency low, and uptime high for a worldwide user base
Security and data-sovereignty controls inherited straight from Google Cloud
Serving-layer features that keep cost and performance optimized at scale.

Managed infrastructure that frees engineering time

Claude on Google Cloud runs on fully managed infrastructure, so enterprise teams ship features instead of building clusters. Compute provisioning, auto-scaling logic, load balancing, and failover at frontier-model scale are handled by the platform — work that would otherwise occupy multiple teams full-time.

Claude is available through Agent Platform's Model Garden as a Model-as-a-Service offering, ready to use over standard REST / JSON over HTTP/1.1 or HTTP/2 endpoints. Invoking Claude is operationally identical to invoking any other Google Cloud service: the same IAM policies, the same VPC controls, and the same observability stack via Cloud Logging and Cloud Monitoring.

Serving Claude takes a few lines of Python using the AnthropicVertex client:

code_block: <ListValue: [StructValue([('code', 'from anthropic import AnthropicVertex\r\n\r\nclient = AnthropicVertex(\r\n project_id="your-project-id",\r\n region="us"\r\n)\r\n\r\nmessage = client.messages.create(\r\n model="claude-opus-4-8",\r\n max_tokens=1024,\r\n messages=[{"role": "user", "content": "Analyze this system architecture."}]\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f51662efcd0>)])]>

The same AnthropicVertex client handles prompt caching, tool use, structured outputs, streaming, and adaptive thinking; for batch inference, use Vertex AI Batch Prediction. Authentication uses Application Default Credentials; requests automatically inherit your project's IAM and VPC configuration.

Global reach with consistent latency and built-in failover

Serving a worldwide user base from a single endpoint produces high tail latency and a single point of failure. Most enterprises can't replicate inference infrastructure across continents while keeping performance consistent.

Agent Platform exposes three endpoint types for Claude, each solving a different production requirement:

Global endpoints route requests to a region with available AI compute capacity. For example, if us-central1 is capacity-constrained, traffic redirects to europe-west1 or another region with available capacity. That’s automatic failover and geographic load balancing without application-side routing logic. Global endpoints are ideal for maximum availability and lowest cost.
Regional endpoints like us-east5 or europe-west1 keep prompts, completions, and intermediate state inside a specific geographical boundary, making it ideal for low latency and data-residency requirements.
Multi-region endpoints give U.S. or EU data residency without single-region dependency. They dynamically route across regional endpoints providing built-in resilience against regional outages and capacity constraints.

The diagram below shows how applications reach Claude through these endpoint types, and how the Agent Platform serving layer routes traffic to the Compute AI clusters across regions:

Serving Claude Models From Regional & Global Endpoints

Serving Claude Models From Multi-Region Endpoints

Serving Claude Models From Regional Endpoints

Enterprise security and data sovereignty built in

Regulated workloads — financial services, healthcare, and government — get enterprise-grade security and data sovereignty without trading compliance for convenience, and without re-engineering the hardest layer to control: inference, where prompts, completions, and intermediate state all flow through the serving stack.

Claude on Agent Platform inherits Google Cloud's full security posture. FedRAMP High and HIPAA compliance enable deployment in government, healthcare, and financial services environments. VPC Service Controls let organizations define a perimeter around Agent Platform resources, preventing data exfiltration. IAM-native access control governs Claude endpoints with the same roles and policies that protect every other Google Cloud resource — no separate API keys to manage or rotate. Cloud Logging and Cloud Monitoring provide near real-time visibility into token usage, error rates, latency, and quota consumption.

Combined with the regional and multi-region endpoints above, this gives regulated customers a path to running frontier AI in production without re-auditing their compliance posture.

Optimized for cost and performance at scale

In production, cost and performance drive every architectural decision. Getting both right requires capabilities from two layers: Claude's native model features, and Google Cloud's serving infrastructure. Agent Platform supports both, so teams can optimize across the stack without managing them separately.

Claude-native capabilities, fully supported on Agent Platform

These features are built into Claude and available on Agent Platform without any additional configuration:

Prompt caching stores and reuses shared prefixes — long system prompts, legal documents, codebases — reducing request latency by up to 80% and cost by up to 90%.
Streaming responses over server-sent events deliver tokens as they are generated, critical for chat interfaces and coding assistants where perceived latency matters.
Extended and adaptive thinking lets Claude dynamically determine when and how much to reason through complex, multi-step problems — and allows users to dial the thinking effort directly, for example to control cost. Optimized for use cases like advanced code generation, mathematical reasoning, and multi-document analysis.
Extended context windows up to 1M tokens (for Claude Opus 4.6,Sonnet 4.6 and newer models) enable long-document analysis, large codebase reasoning, and multi-turn conversations at depth.

Google Cloud serving infrastructure

Agent Platform adds its own serving-layer capabilities on top of Claude's native features:

Batch prediction handles large-scale offline workloads — document classification, content moderation, bulk summarization — asynchronously at lower priority and reduced cost.
Provisioned throughput reserves dedicated inference capacity for mission-critical workloads, isolating them from public traffic and ensuring predictable performance during peak demand.
Memory management and scheduling for long-context requests is handled at the infrastructure layer,.

Together, these two layers give teams the full range of optimization levers — from model-level efficiency to infrastructure-level capacity control — on a single, unified platform.

From inference to agents

The same infrastructure that serves Claude inference powers the agent layer of Agent Platform on Google Cloud. The build-and-register flow has three steps:

Build with Claude. Claude is well-suited as an orchestration backbone — its extended context window, native tool use, and adaptive thinking make it effective at planning multi-step tasks and delegating to sub-agents. Pick Claude Opus, Sonnet, or Haiku from the Model Garden, then build with the Agent Development Kit (ADK) — code-first in Python, Go, Java, or TypeScript — deploy to Agent Runtime, Cloud Run or Google Kubernetes Engine.
Deploy the Agent to a Runtime. Depending on your use case, select Agent Runtime, Google Kubernetes Engine or GKE Agent Sandbox to run your deployed agents.
Interoperate over A2A. The Agent2Agent protocol runs at 150+ organizations, letting a registered Claude-powered agent delegate tasks to agents from SaaS and other service providers.

The result: a planning agent built on Claude can orchestrate sub-tasks across the broader agent ecosystem, under unified IAM, fully auditable, on the same infrastructure that serves the underlying inference.

Start building

Open the Agent Platform console, enable Claude in the Model Garden, and make your first API call with the AnthropicVertex SDK. Add prompt caching, provisioned throughput, and other features as your workload demands. When you're ready to go agentic, learn more about Claude on Agent Platform.

Reach out to your Google Cloud sales representative to discuss bringing Claude into your production environment at scale.

Google Cloud named Leader in the 2026 Gartner® Magic Quadrant™ for AI Infrastructure

Wed, 08 Jul 2026 16:00:00 +0000

In the agentic era, AI is evolving from answering questions to reasoning and taking action. Companies who want to lead in this next phase of AI need computing infrastructure that’s designed and optimized for these new requirements, helping them innovate faster, deliver compelling user and customer experiences, and optimize for cost and energy efficiency — all at massive scale.

Today, we are pleased to announce that Google has been named a Leader in the inaugural Gartner^Ⓡ Magic Quadrant™ for AI Infrastructure, positioned highest for ‘Ability to Execute’ and furthest for ‘Completeness of Vision’. We believe their findings validate our dedication to solving these challenges internally and for our customers.

Read the full report: 2026 Gartner Magic Quadrant™ for AI Infrastructure

Building on the infrastructure foundation powering Gemini

Today’s model and serving architectures require a fundamental rethinking of how silicon and software interact. We realized early on that the platform we envisioned couldn’t be bought off the shelf — we had to invent it. For over a decade, our infrastructure engineers and Google DeepMind researchers have worked shoulder to shoulder to co-design the entire stack for Gemini, YouTube, and Search. We make those innovations, together with popular third party and open source software, available to our customers through Google Cloud. Today our integrated stack serves 9 out of 10 frontier AI labs; capital markets firms like Citadel Securities; and enterprises like Mercedes Benz.

At the hardware layer, Gartner recognized our commitment to custom silicon as a core strength. Earlier this year we shared two new advancements in custom silicon, our 8th generation TPUs, engineered to solve enterprise scaling and memory bottlenecks at a systems level:

TPU 8t, the training powerhouse: Purpose-built to optimize training timelines, TPU 8t packs 9,600 chips into a single superpod, delivering the high-density compute required for frontier models with nearly 3x the compute performance per pod over the previous generation.
TPU 8i, the inference engine: Engineered to handle the collaborative, iterative work of specialized agents, TPU 8i breaks the memory wall for real-time agentic workflows, with 288 GB of high-bandwidth memory and 384 MB of on-chip SRAM — 3x more than the previous generation — keeping a model's active working set entirely on-chip.

While our TPU platforms push the boundaries of what is possible, we know that one size doesn't fit all. Different customers have different workloads, different requirements, and different use cases. So, we also partner deeply with NVIDIA to deliver the latest accelerated computing platforms as highly performant, reliable and scalable services in Google Cloud. We will be among the first to deliver A5X instances based on the next-generation Vera Rubin platform when it becomes available later this year, enabling customer choice. We also work closely with NVIDIA to integrate GPUs into many Google Cloud software services to give our customers easier access to accelerated computing.

To enable even more flexibility, we continue to contribute to open-source projects across the orchestration, inference engines, and framework layers through llm-d and vLLM. We also recently announced TorchTPU, which gives PyTorch developers portability without complex code rewrites while maximizing the performance of their deployment.

Get more performance per dollar on AI Hypercomputer

As your infrastructure investment grows, you need to balance raw performance and cost to make AI applications economically viable. Taking a ‘buy now, integrate later’ approach to AI is becoming unsustainable. By combining pre-integrated hardware and open software frameworks that feature flexible consumption models, we deliver a unified system engineered for better performance per dollar across training, reinforcement learning, and inference.

Gartner recognized our integrated AI Hypercomputer as a core strength. This AI-optimized infrastructure is engineered to drastically improve your performance per dollar:

A massive compute cluster is only as effective as the storage system feeding it data. Google Cloud Managed Lustre, powered by our new C4NX instances and Hyperdisk Exapools, now delivers 10 TB/s of bandwidth — up to 20x faster than other hyperscalers — while Rapid Buckets transforms object storage with up to 20 million operations per second, helping ensuring large-scale training checkpoints and recoveries happen near-instantly.
Our Virgo Network provides a high-bandwidth scale-out fabric capable of connecting more than one million TPUs across multiple data center sites into a training cluster, or up to 960,000 GPUs across multiple sites without performance degradation — transforming globally distributed infrastructure into a unified supercomputer.
GKE Inference Gateway enables scaling models in production with near-zero latency by combining LLM-aware routing, caching, and the disaggregated serving capabilities of llm-d, increasing throughput by up to 40% while reducing serving costs up to 30%.

Run AI on a fluid infrastructure at virtually any scale

In the agentic era, infrastructure cannot be a rigid, static constraint. It must be an intelligent resource that adapts to the shifting priorities of your business, scaling up with demand and down to zero when agents are idle, with consistent, reliable performance. On AI Hypercomputer, you can:

Train smarter and faster, using Cluster Director and Google Kubernetes Engine to scale up to 130,000 nodes. At the same time, squeeze up to 97% productivity (Goodput) out of every accelerator using TPU 8t together with software co-designed with Google DeepMind and integrated open-source frameworks — from JAX to Pathways and Pallas.
Enable secure, low latency agent execution with GKE Agent Sandbox. Because agents need to scale, GKE Agent Sandbox can sense agent bursts and respond rapidly — provisioning up to 300 sandboxes per second per cluster, then instantly scale back when agents sit idle, optimizing compute costs.
Run distributed enterprise and AI workloads consistently across multicloud, edge, and on premises environments with Cross-Cloud Network and Cloud WAN. This approach delivers low-latency, policy-driven connectivity across Google’s private global backbone spanning over 10+ million kilometers of fiber and over 200 countries and territories, with up to 40% higher performance than public internet routing.

Take the next steps on your journey with AI Hypercomputer

From frontier models, to billion user applications, AI Hypercomputer gives you the purpose-built hardware, open software, and flexible consumption models you need to improve AI performance, cost, and developer productivity. We are honored to see decades of experience building scalable, affordable and reliable AI systems rewarded with a leadership position in Gartner’s research.

You can download a complimentary copy of the 2026 Gartner Magic Quadrant™ for AI Infrastructure on our website.

Scaling Ray Serve LLM on GKE: Performance without losing the developer experience

Thu, 18 Jun 2026 16:00:00 +0000

Developers looking for LLM inference and model serving often turn to Ray Serve, a scalable model serving library with developer-friendly, Python-native APIs built by Anyscale. Combined with Google Kubernetes Engine (GKE), developers have a powerful, unified platform optimized for demanding LLM serving use cases, spanning from initial model development to online production serving.

However, that flexibility and feature set used to come at a cost to performance. But today, in partnership with Anyscale, we are delivering up to 5x higher throughput and 8x lower latency in Ray Serve, meeting the growing demands and rigorous performance requirements of state-of-the-art distributed inference, without having to sacrifice ease of use.

Scaling inference without the bottlenecks

Through our joint engineering partnership, we are introducing three major architectural optimizations that dramatically improve Ray Serve LLM's performance characteristics:

Ray Serve HAProxy integration: Ray Serve now builds in HAProxy to manage internal request routing and load balancing. This setup drastically reduces proxy overhead and prevents the Python runtime from saturating under high traffic.
Direct token streaming architecture: This architecture decouples the initial request path from the return stream. Tokens stream directly from individual model replicas back to the proxy, bypassing the ingress router completely for the streaming data path to cut latency.
v2 Ray executor backend for vLLM: The revamped Ray backend for vLLM moves Ray out of the data plane to enable asynchronous scheduling. This unifies the code path with native vLLM executors, closing the performance gap and helping to ensure Ray users benefit from the latest engine-level optimizations.

Benchmarking performance on GKE

We’ve also collaborated with Anyscale to benchmark the updated Ray Serve LLM on GKE clusters utilizing next-generation AI hardware, including Google Cloud A4 VMs powered by NVIDIA HGX B200 systems. We chose to run Gemma 4 E2B as a small, efficient model to isolate bottlenecks introduced from orchestration and routing. Our benchmarks compared the new Ray Serve LLM to its prior performance, as well as a plain vLLM setup using the Ray executor.

These technical enhancements deliver a transformative impact on performance, offering up to 5x higher throughput and 8x better latency compared to previous Ray Serve configurations.

The improved Ray Serve LLM demonstrated a remarkable improvement on a serving cluster with eight replicas, showing a scaling pattern that far exceeds previous performance, and showing comparable performance to running vLLM natively, but without the flexibility that Ray brings to the table.

We observe that with an increasing number of concurrent users, Ray is now able to scale up throughput while maintaining a low 99th percentile time-to-first-token, where previously it struggled. Now LLM practitioners don’t have to sacrifice Ray’s rich features and ecosystem to get production-grade performance on Kubernetes.

Why choose GKE for Ray Serve

GKE provides the foundational infrastructure that makes these software optimizations shine. When using the Ray Operator add-on for GKE, you get turnkey deployment across Google Cloud's AI accelerators, including automated horizontal scaling, monitoring, multi-cluster scaling, and built-in fault tolerance. GKE abstracts the complex parts of orchestrating distributed physical hardware, so your team can focus on refining your models and application logic with Ray.

Try Ray Serve LLM on GKE

We encourage developers to try out these enhancements in the latest Ray release (2.56 and later) and experience the future of high-performance LLM serving on GKE.

For more details, check out the following resources:

Report: GKE Inference Gateway delivers up to 92% faster AI responses

Tue, 09 Jun 2026 16:00:00 +0000

As generative AI moves from experimental pilots to massive production environments, the efficiency of your infrastructure becomes the ultimate differentiator. One way to get the most out of it and minimize costly accelerator idle time is to leverage the Google Kubernetes Engine (GKE) Inference Gateway, which intelligently routes generative AI workloads based on real-time model server metrics.

Instead of relying on traditional, naive round-robin load balancing — which frequently triggers expensive accelerator recomputation and spikes user latency — this native extension of the GKE Gateway utilizes advanced capabilities like prefix caching and model-aware routing. By ensuring requests land on the exact accelerator that is primed to process them right away, GKE transforms how you can serve your large language models (LLMs), with excellent hardware utilization and ultra-fast response times.

In fact, according to an independent benchmark report, GKE Inference Gateway outperforms the next leading managed Kubernetes service with 15.7% higher throughput, 92.8% shorter wait times, and 62.6% lower inter-token latency. This performance takes LLM-based applications from sluggish and expensive to fast and production-grade.

That performance tracks with Snap’s experience using GKE Inference Gateway.

“At Snap, we are integrating llm-d into our production AI infrastructure to facilitate high-performance inference at scale. By employing prefix-cache-aware routing, we have achieved prefix cache hit rates ranging up to 75-80%. We appreciate the open-source nature of llm-d, as it enables seamless integration with our Envoy-based Service Mesh.” - Vinay Kola, Senior Manager, Software Engineering, Snap Inc.

In this blog, we take a closer look at GKE Inference Gateway’s prefix caching, complete with examples. We also provide more details about its benchmark results. Let’s jump in.

The secret to low-latency AI: Prefix caching

Prefix caching optimizes LLM performance by storing the KV cache (activation states) of long, repetitive prompt prefixes. When consecutive user requests share the same system instructions, context, or documentation, the model entirely skips reprocessing those tokens. GKE Inference Gateway reads incoming request prefixes and matches them to the specific pods that already hold that data in memory. This eliminates the "thinking" tax on your GPUs and TPUs, turning heavy reasoning loops into near-instant answers.

Use case 1: Documentation and codebase Q&A with retrieval-augmented generation (RAG)

When querying massive enterprise repositories, you can ground your LLMs’ responses without any added latency by pinning entire documentation sets as static cached prefixes, using RAG.

Instead of forcing an LLM to re-read thousands of lines of API references or corporate wikis for every single user question, GKE Inference Gateway routes the query to a pod that already has that specific context warmed up in its KV cache. The LLM only has to compute the user's brief, dynamic question, completely bypassing expensive document re-evaluation.

code_block: <ListValue: [StructValue([('code', '[STATIC PREFIX - STAYS IN CACHE] You are an expert AI assistant specializing in technical documentation. Below is the complete API documentation for our software platform. Use this context to answer the user\'s questions accurately. If the answer cannot be found in the documentation, say "I cannot find that in the provided context." \r\n\r\n<documentation> [10,000+ words of API reference documentation, endpoints, error codes, etc.] </documentation> \r\n\r\n[DYNAMIC SUFFIX - CHANGES PER REQUEST] User Question: How do I handle a 429 rate limit error using the Python SDK?'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f515ff064f0>)])]>

Use case 2: Multi-turn chat

You can also use prefix caching to maintain customer service interactions across thousands of simultaneous sessions without compounding compute costs. You can do so by caching permanent system personas and core business rules directly on the LLM server.

In enterprise chat architectures, the base system prompt and reference tables remain completely identical across millions of customer interactions. GKE Inference Gateway handles these multi-turn conversations using context-aware routing to bypass repetitive token processing, so that your chatbot stays ultra-responsive even under peak traffic.

code_block: <ListValue: [StructValue([('code', '[STATIC PREFIX - STAYS IN CACHE] \r\n-System Persona: You are "FinBot", a helpful, empathetic, and compliant virtual assistant for ABC Banking Solutions. You must strictly adhere to the following rules: 1. Never provide concrete investment advice. 2. Always verify if the user is asking about checking or savings. 3. Keep your answers under 3 sentences. 4. If a user is angry, offer to connect them to a human manager. \r\n\r\nHere is the current interest rate table for May 2026: \r\n- Savings: 4.2% APR \r\n- Checking: 0.5% APR \r\n- CD (12-month): 5.1% APR \r\n\r\n[DYNAMIC SUFFIX - CHANGES PER REQUEST] User: Hi, I\'m trying to figure out how much I\'d make if I locked away $10,000 for a year?'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f515ff06fa0>)])]>

GKE outperforms alternative managed Kubernetes solutions

To validate these architectural advantages, Principled Technologies recently released an independent benchmark report comparing GKE (equipped with the GKE Inference Gateway) against a standard third-party managed Kubernetes service utilizing conventional round-robin HTTP load balancing.

Tested on a Llama 3.1 8B Instruct shared prefix workload using identical hardware (eight NVIDIA A100 40GB GPUs) the results reveal a massive performance gap between the two Kubernetes services. GKE didn't just win; it completely redefined inference efficiency across three critical metrics:

Higher throughput: 15.7% more tokens processed per second, enabling higher request capacity or reduced hardware needs for the same workload
Much faster time to first token (TTFT): 92.8% shorter wait times, producing dramatically quicker perceived response starts for interactive scenarios
Lower inter-token latency (ITL): 62.6% reduction, resulting in smoother and faster token streaming after the first token

Figure 3: Mean latency (normalized time per output token) of GKE with GKE Inference Gateway and third-party managed Kubernetes service on the Llama 3.1-8B Instruct LLM on the Shared prefix use case. Both solutions used the same hardware. Source: Principled Technologies

	GKE	3rd party ManagedKubernetes Service	GKE Advantage
Mean outputtoken throughput	7,169.21 outputtokens per second	6,042.05 outputtokens per second	15.7% more outputtoken throughput
Mean time tofirst token (TTFT)	188.36 ms	2624.73 ms	92.8% less TTFT
Mean inter-tokenlatency (ITL)	30.20 ms	81.03 ms	62.6% lower ITL

Figure 4: GKE with GKE Inference Gateway delivered superior AI inference compared to a third-party managed Kubernetes service using standard HTTP LB.

Ready to accelerate your gen AI inference workloads?

Whether you’re deploying inference workloads such as real-time customer support agents, dynamic coding assistants, or sub-second fraud detection models, infrastructure latency dictates your user experience. By ensuring shared prompt prefixes hit the active cache nearly 100% of the time, GKE Inference Gateway transforms your LLMs from sluggish, expensive reasoning engines into rapid, capital-efficient, production-grade powerhouses.

Ready to explore the performance advantage that GKE Inference Gateway can bring to your gen AI workloads? Access the full benchmark report here and watch this explainer video to learn more.

^{A special thanks to Dan Sullivan, Senior Performance Architect, Principled Technologies.}

How we evolved Google’s global and data center networks for the AI era

Tue, 26 May 2026 16:00:00 +0000

Over the last 25 years of building Google’s global network, we’ve navigated major architectural eras — from the Internet, to streaming, and the cloud. Today, we are squarely in the midst of a fourth: the AI era. The applications in the AI era are fundamentally different from the consumer and enterprise applications of the previous eras and impose a set of novel and demanding requirements — on compute resources, of course, but also on the network.

Consider the fundamental physical challenge, which is that it is far more difficult to move electrons (electrical power) than it is to move photons (data over fiber). Because the demand for AI compute frequently outpaces the space and power capacities of individual facilities, we strategically locate data centers near sustainable energy sources, or in locations with pathways to add clean energy sources to the local grid. Then, by utilizing the network to distribute AI workloads across campuses, we create a massive-scale, pooled hypercomputing resource that overcomes the power limitations of any single site.

To deliver this, we created an end-to-end, vertically integrated AI technology stack that comprises everything from chips to systems, to platforms and application and agentic ecosystems. This stack includes a portfolio of pre-built agents and applications; our Gemini Enterprise Agent Platform for you to build, scale, govern, and optimize your AI-enabled applications; world-class AI models; as well as our unified data platform. All this is anchored by our AI Hypercomputer, a unified infrastructure that combines purpose-built hardware and open software, and that comes with flexible consumption options. Our network, forged through decades of innovation, is the essential fabric of the AI Hypercomputer.

The network supporting this stack must meet the stringent bandwidth, scale, and performance needs of AI workloads. This applies not only within the campus, where the network must scale up and out, but also across the wide area network (WAN) along with high-bandwidth interconnects, to bring AI training data from its source to AI compute resources.

To address these challenges, we’ve reimagined three key pillars of our network infrastructure: the fabric inside the AI Hypercomputer, the fabric across the AI Hypercomputer, and our global network. Let’s take a closer look at each of these.

1. The fabric inside AI Hypercomputer

The massive scale of today’s AI models, fueled by the explosive growth of foundational AI model parameters, makes AI training very compute- and network-intensive.

This necessitates an exponential increase in required network bandwidth, with strict bounds on delay (e.g., tail latency) to accommodate AI workloads’ peculiar traffic patterns, which are characterized by sensitivity to performance variation and synchronized bursts, i.e., intense, coordinated, millisecond-level traffic spikes. Furthermore, since large-scale training jobs are uniquely vulnerable to failures and performance stragglers, maintaining high reliability and predictable performance is absolutely essential.

To address the scale, low latency, and high predictability that modern AI workloads require — as well as protection from extreme bursts — we’ve adopted a "campus as a computer" philosophy, decoupling our network into three distinct domains:

a scale-up domain for intra-pod connectivity
a dedicated east-west scale-out accelerator fabric
the Jupiter frontend network for north-south compute and storage access

This decoupled architecture provides three strategic advantages: it allows domains to evolve independently for faster innovation; provides a non-blocking scale-out network with massive training bandwidth; and helps ensure the network can be co-designed in lockstep with new ML accelerators, for superior hardware support.

Recently, we unveiled Virgo Network, our scale-out data center fabric specifically engineered for modern AI. Virgo utilizes high-radix switches and a flat, two-layer non-blocking topology to provide massive bisection bandwidth, while minimizing latency by reducing network tiers. Its multi-planar design, featuring independent control domains for each plane, provides hardware-level resilience and fault isolation. Furthermore, Virgo can expand across multiple data centers, removing physical building limitations and enabling flexible AI compute scaling.

The effectiveness of our network and accelerator codesign is perfectly illustrated by the recently debuted eighth generation TPUs. Within this architecture, Virgo Network can link 134,000 TPU 8t chips with up to 47 petabits/sec of non-blocking bi-sectional bandwidth in a single fabric. Virgo Network delivers up to 4x the bandwidth per TPU 8t accelerator over the previous generation, and 40% lower unloaded fabric latency for TPU 8t compared to the previous generation network for TPUs. In this setup, Virgo Network manages the raw accelerator traffic, while Jupiter provides reliable and rapid access to the global WAN and storage. When integrated with Pathways and JAX, this AI Hypercomputer networking engine facilitates near-linear scaling for up to a million TPU 8t chips in a single logical cluster.

Autonomous reliability: protecting workload goodput

Building a resilient megascale fabric represents only part of the challenge. In a cluster of hundreds of thousands of chips, hardware failures are a statistical certainty. A single stalled instance can stop an entire synchronous training job, wasting valuable compute cycles. As such, efficient fault localization is critical.

We engineered Virgo Network with autonomous reliability capabilities to maximize workload efficiency at scale, a.k.a., goodput. Expanding on our existing straggler detection, Virgo Network now also features automated hang detection. The moment a fail-stop event occurs, our specialized agents immediately localize the fault, isolate the faulty instance, and enable you to restore the training job from a checkpoint — getting your training timeline back on track, with minimal manual intervention. Learn more by watching this demo:

To complement these capabilities, we also use high-resolution, sub-millisecond telemetry to identify elusive network micro-bursts that are usually missed by conventional 30-second monitoring intervals. These high-resolution telemetry advancements enable more efficient network operations, better provisioning, and a lower mean time to recovery.

2. The fabric across AI Hypercomputer

The exponential growth of modern AI workloads requires us to scale and distribute AI workloads across multiple campuses over a WAN. At the same time, traditional networks weren’t built for the high bandwidth and extreme burstiness of AI traffic, and often fail to detect microbursts that can lead to severe performance degradation. We have developed a suite of innovations to optimize WAN performance for cross-site AI deployments, including:

A multi-shard global network that enables horizontal scaling. Our global network sustained a 10X WAN traffic growth from 2020 to 2025.
Tuning the fabric for essential availability, latency, and quality of service (QoS) attributes. Real-time microburst management helps ensure fair bandwidth allocation and infrastructure isolation across our multi-tenant infrastructure.
Multi-shard isolation to ensure each network shard operates with its own control, data, and management planes.

Combined with regional isolation and Protective Reroute, this architecture minimizes failure impact and shortens user-visible outages — delivering the beyond-nines reliability essential for AI workloads.

Providing high-speed, flexible, and cost-effective interconnectivity is also a priority. AI training relies on vast datasets that are often located on-premises or across various clouds. Given the high cost of AI compute, minimizing idle time is essential; for instance, upgrading from a 100 Gbps link to a 3.2 Tbps connection reduces the time to transfer a petabyte of data from 22.2 hours to just 0.7 hours — a 97% reduction in AI compute idle time spent waiting for data. Our AI-native Cloud Interconnect is purpose-built for the high-bandwidth and low-latency needs of AI workloads, featuring an optimized data path with 400 Gbps links that scale in 3.2 Tbps increments to reach petabit-per-second capacity. It also offers traffic differentiation and flexible connection options, including direct fiber peering and colocation facilities. AI-native Cloud Interconnect supports petabit-scale data transfer with reliable, private connectivity necessary for your cross-cloud AI training and serving.

3. A resilient global network for the age of inference

Applications serving AI inference to a global user population or supporting an agentic enterprise are far more demanding than conventional web apps. The need for opportunistic use of expensive AI compute available at distant locations, distributed service dependencies, and the burstiness of the traffic demand high bandwidth network with a global footprint, as well as deep peering to SaaS providers, ISPs, and hyperscalers. To maintain responsiveness and "always-on" availability, applications need low latency and a highly resilient network.

With its connectivity, scale, and resilience, Google’s global network is well-equipped to handle the demands of the age of AI inference. Our network spans more than 10 million kilometers of terrestrial and subsea fiber, connects our 43 cloud regions, and features 200+ edge locations, providing the essential footprint for serving AI inference. Our Premium Tier network delivers the low latency and reliability needed for consistent, high-quality global user experience. By optimizing traffic entry and exit points, the network significantly boosts application performance, with resilience at the core of this "always-on" infrastructure.

Building the future, together

As a Google Cloud customer, these network innovations are built directly into your environment. Google’s network delivers the massive scale, capacity, reliability and performance essential for your AI workloads.

The AI era demands more than just raw compute; it necessitates a robust network fabric to scale. Our vertically integrated AI technology stack — from silicon to software ecosystems — is powered by the AI Hypercomputer to accelerate your transformation and make AI helpful for everyone. Whether through our megascale fabric, resilient global network for inference, or AI-native Cloud Interconnect, we ensure your AI journey is efficient and reliable. We look forward to building this future with you.

Agent Sandbox on GKE is now available for everyone, and a first look at Agent Substrate

Wed, 20 May 2026 16:00:00 +0000

In just a short time, we’ve seen AI transition from simple chat interfaces to autonomous agents capable of function calling, code execution, and persistent terminal use. But to orchestrate these capabilities securely, agents need more than just intelligence — they need a robust, hyper-scalable, secure compute environment in which to execute code.

Since our preview announcement of GKE Agent Sandbox at KubeCon NA in November 2025, the community adoption has rapidly accelerated: we have seen more than 16x growth in sandboxes on Google Kubernetes Engine (GKE) in less than 5 months.

We’ve worked with key customers like Langchain and Lovable, and many others who are rapidly deploying millions of agents into production. Since its unveiling, Agent Sandbox has evolved rapidly, moving from a new project to a mature product with stable APIs. This stability is now fueling its integration into the broader agent ecosystem, where it serves as a critical infrastructure layer.

Today, we are excited to build on this momentum in two ways:

GKE Agent Sandbox is now generally available, giving you a secure, scalable foundation for your agent workloads
Introducing Agent Substrate, a new open source project aimed at continuing to push the limits of agentic infrastructure density

Secure, low-latency execution at scale

Agent Sandbox is an open-source, cloud-native execution environment built on Kubernetes, designed specifically for the unique demands of AI agents. It provides the foundational infrastructure to empower builders to safely and securely execute untrusted logic on top of their own infrastructure with industry-leading speed and efficiency.

With this release, we are delivering on the core requirements of modern agent workloads:

Reduce idle compute with pod snapshots: Agents often have short bursty cycles followed by longer idle periods. Instead of wasting valuable compute to keep the agent running, GKE Agent Sandbox integrates with Pod Snapshots to suspend your idle agent workloads and resume them in seconds upon request.
Low latency sandbox provisioning: Initializing a new sandbox instance for every request introduces unwanted seconds of cold start latency. GKE Agent Sandbox introduces a Sandbox API with an integrated warm pool. The Agent Sandbox API's integrated warm pool enables GKE to allocate 300 sandboxes per second, per cluster, at sub second latency, with 90% of allocations completing in 200 milliseconds.
Cost-effective warm pool: GKE Agent Sandbox warm pools keep pre-provisioned replicas ready to minimize sandbox startup latency. To minimize the cost of maintaining a sandbox warm pool, Agent Sandbox is integrated with standby capacity buffers (suspended VMs) to provide a cold pool of suspended sandboxes that can quickly replenish the warm pool for a fraction of the cost.
Ironclad security & isolation: Agent Sandbox natively supports gVisor and default-deny Kubernetes network policy. Agent Sandbox provides pluggable interfaces for open source sandboxes like Kata Containers, enabling users to customize their kernel isolation.

As the demand for compute continues to rise, this release ensures our customers have access to the broad range of Google Cloud compute options. GKE Agent Sandbox delivers up to 30% better price-performance when running on Axion processors than comparable hyperscaler cloud providers.

The next revolutionary step forward in agentic infrastructure Agentic workloads are simultaneously scaling up to the 10s to 100s of millions of instances while at the same time becoming increasingly idle, waiting for human interactions, events or triggers. These workloads continue to demand strong kernel and network isolation, making dense scheduling a challenge. Handling this level of scale and rapid suspend-and-resume is pushing the limits of the Kubernetes control plane.That’s why we are introducing Agent Substrate, a new open source project aimed at addressing the performance and density needs of ultra scale agents.

Agent Substrate introduces a new level of abstraction that moves agents onto and off of ready compute capacity (running in Kubernetes, of course) in real-time. Agent Substrate takes the core secure runtime and snapshotting capabilities of Agent Sandbox and pairs them with a minimal control plane designed to bypass some of the limitations of Kubernetes, without reinventing the rest of it.

This lets Agent Substrate optimize the critical paths to offer lower latency with higher scale and efficiency. While standard Kubernetes is optimized to handle thousands of long-running services, Agent Substrate is designed for the chatter of millions of sub-second tool calls that would otherwise overwhelm a standard control plane. It provides the perfect foundation for Agents, Agent Harnesses and Agent Runtimes, including the new Agent Executor project.

Agent Substrate’s goal is to explore every opportunity to make things move faster and scale bigger. Achieving this level of scale and efficiency is going to push the bounds of what current compute infrastructure can do, and no rock will be left unturned. One such exploration is to bring data locality into the core of the scheduler, ensuring that agent state and scheduling work together to shave off every possible millisecond of overhead.

Building the future in the open

In the early days of Kubernetes, the feedback and perspective from diverse contributors solving similar challenges was critical to setting the project up for success. We believe that agent infrastructure is at a similar inflection point. Today, we're hoping to recreate that magic of radically open and collaborative innovation to shape the future of agent infrastructure together. By kicking off the Agent Substrate project in the open, we are inviting the community to help design and build this critical next mode of infrastructure.

Get started today

As we look toward a future of autonomous agents, we are excited to continue to build the critical layers of the stack. We invite you to use Agent Sandbox to power your workloads today, and join us in the open-source community to collaborate on Agent Substrate – the next chapter in agent-native infrastructure.

Try Agent Sandbox on GKE
Contribute: Join the Agent Sandbox open-source community
Explore Agent Substrate

Benchmark and optimize LLMs on-device with AI Edge Portal

Wed, 20 May 2026 09:00:00 +0000

LLMs have become more powerful at smaller sizes, but deploying them to edge devices like smartphones remains a massive challenge. Today, developers have to optimize across a sprawling combination of accelerators, operating systems, and countless System-on-a-Chip (SoC) configurations, often relying on manual testing with just a handful of devices. Google AI Edge Portal helps solve these challenges.

By letting developers test ML workloads across a fleet of over 120 representative Android device types, Google AI Edge Portal provides deep insight into latency and performance across all CPU, GPU, and NPU backends.

Today, we are excited to announce two new capabilities that expand Google AI Edge Portal’s capabilities for the generative AI era: benchmarking and debugging on-device LLMs. These new services give developers what they need to optimize generative AI performance accurately and efficiently across the entire Android ecosystem.

Benchmark LLMs across over 120 different mobile devices

When a user interacts with an LLM-enabled experience in your app, they expect fast and consistent performance on their device. Common challenges like initialization time can result in your app appearing to freeze, or, in a worst case, crash completely if the model consumes all available memory.

With the latest release of Google AI Edge Portal, you can now run automated gen AI benchmarks directly on a physical lab of over 120 diverse Android devices and test for these scenarios specifically. Portal natively supports CPU and GPU benchmarking for LLMs in the LiteRT-LM format.

Customers can benchmark GenAI models on over 120 Android devices, viewing metrics including initialization time, prefill speed, decode speed, and peak memory usage.

When you trigger a gen AI benchmarking job with Portal, it profiles the critical metrics that dictate your end-users’ experience when interacting with your AI application on-device:

Metric	What it measures	Why it matters to you
Initialization time	Measures how long it takes to load your model into memory.	High initialization time can result in delays, or freeze the user interface when your application starts up.
Prefill speed	Captures how fast the device processes prompt tokens to generate the first output token.	Dictates the initial delay before the user sees the first response.
Decode speed	Captures how fast the model generates tokens during a response.	Dictates the speed at which output is generated.
Peak memory	Monitors maximum RAM usage.	Flags potential “out of memory” crash risk, especially prevalent on memory constrained devices.

With these insights, you can confidently decide which devices are ready to host your model and adjust or better optimize your LLMs for device targeting before shipping.

Debug performance easily with Model Explorer

Benchmarking is only useful if you can fix the discovered performance issues. When an LLM performs poorly, finding the root cause within the complex graph of multiple layers and thousands of nodes is a daunting task for developers, involving tedious and time-consuming searching that can take hours if not days.

To bridge this gap, we have added the ability to visualize and compare model graphs in Portal with ease. Through the natively integrated Model Explorer, our graph visualization tool, you can search and locate specific nodes, compare models side-by-side in the same tab, and view tensor shapes, trace inputs and outputs, and more. To further speed up debugging for teams, we also added the ability to take screenshots and share specific views directly with your collaborators in Google Cloud.

These visualizations are one of the most effective ways to identify targets for optimization, including:

Conversion: Model Explorer simplifies the identification of conversion anomalies through its dual-view comparison tool. This interface allows you to traverse complex model architectures by selectively expanding or collapsing specific layers, granting you the ability to analyze internal dependencies and structural nodes with precise granularity.
Quantization: Evaluate your quantization strategies by uploading custom error metrics as per-node data. This allows you to visually pinpoint precision loss, overlay granular per-layer data directly on the graph, and achieve an optimal balance between model footprint and output quality.
Optimization: Visualize hardware compatibility and conduct granular, per-op performance comparisons by overlaying your custom latency data directly onto the model graph to easily identify performance bottlenecks across different hardware accelerators.

With Model Explorer, you can view model graphs, search for specific layers, and compare models side-by-side to debug performance.

Start benchmarking LLMs on-device today

With the era of LLMs on-device here, we are excited to help close the critical gap in benchmarking to bring the power of AI to the thousands of types of smartphones on the market today. To utilize these latest features, please complete our sign-up form here to express interest.

Google AI Edge Portal is currently available in private preview for allowlisted Google Cloud customers. During this private preview period, access is provided at no charge, subject to the preview terms. All current allowlisted customers will receive access to these new features automatically.

We can’t wait to see what gen AI capabilities you are able to deploy across the full spectrum of devices with Google AI Edge Portal!

_{Thank you to the members of the team, and collaborators for their contributions in making the advancements in this release possible: Akshat Sharma, Ami Kubota, Charlie Xu, Chunlei Niu, Cormac Brick, Derek Bekebrede, Eric Yang, Jing Jin, Kathleen Low, Matthias Grundmann, Marissa Ikonomidis, Na Li, Ram Iyengar, Sachin Kotwani, Sommayah Soliman, Tenghui Zhu, Xiaoming Hu, Zi Yuan}

How Imgix processes 8 billion images daily with G4 VMs powered by NVIDIA Blackwell

Tue, 12 May 2026 16:00:00 +0000

The modern web is extremely visual. People are busy and easily-distracted, and smart companies know they have just seconds to attract would-be customers with compelling images, videos, animations, and other eye-catching elements. That’s why iconic brands like Bugatti, Yeti, Porsche, Spotify, and Sonos rely on Imgix to be the engine driving their online visual media.

Every day, Imgix serves more than 8 billion images and videos for brands like these and many others. With a platform designed to unify media optimization, AI transformation, and global delivery, Imgix ensures that its partners’ digital experiences are fast, personalized, and built for performance. Now more than ever, leading organizations are demanding real-time, high-fidelity media, and they need it to be fast.

To meet that demand, Imgix has evolved its infrastructure from private data centers to a full-stack, GPU-based environment on Google Cloud’s AI Hypercomputer. By transitioning to G4 VMs powered by NVIDIA RTX PRO 6000 Blackwell GPUs, Imgix ramped up its real-time processing capabilities, cutting median latency by 50% and increasing throughput per node by 6x. And it did all of that without changing its core application code.

The challenge: Instant visuals at scale

To capture people’s attention businesses need rich, fast-loading content that can reach millions of users simultaneously across a diverse array of devices.

A big part of that is real-time transformations — resizing, format negotiation, and applying artistic effects — and the computational power required for real-time transformations can be immense.

With inefficient technology, load times can be slow and brands risk giving their users poor experiences. Imgix’s solution to this challenge is a "just-in-time" philosophy. Achieving this requires high-performance instances. And with G4 VMs, they were able to process images instantly upon request rather than pre-rendering and storing millions of image variations.

Adopting the system that runs Google

When companies build on Google Cloud, they get more than just servers: they plug into the same intelligence engine powering Google's many billion-user products. Imgix is leveraging this structural advantage by using G4 VMs.

G4 VMs incorporate eight NVIDIA RTX PRO 6000 Blackwell GPUs, two AMD Turin CPUs, and Google Titanium offloads, which act as a dedicated administrative assistant for businesses’ servers. They handle the ”office chores” of security and data traffic in the background while the main processor does a company’s heavy lifting.

The G4 VM’s custom P2P interconnect yields up to 168% more throughput than standard configurations. With this architecture, Imgix can move all its image processing operations to NVIDIA GPUs and run multiple requests in parallel.

Inside the Imgix architecture

Imgix offers more than 150 different visual filters and its architecture is built to handle transformation requests dynamically based on which filters users choose. The pipeline has four primary stages:

Ingestion: The system retrieves assets directly from customers and routes them to a 2.5 petabyte storage cache on Google Cloud Storage (GCS). This high-speed layer replaces unreliable random web requests with a redundant, geographically distributed infrastructure.
Decoding: High-performance C libraries, supplemented by nvJPEG, decode assets into raw RGBA data. This leverages the G4 VM’s massive parallelism to handle multiple decoding stages, including Huffman decoding, Inverse DCT, and color space conversion.
Transformation: A custom Vulkan compute shader stack handles the core processing. Instead of fixed graphics pipelines, these shaders treat transformations (like resizing or masking) as parallel math problems rather than standard graphics tasks, enabling thousands of simultaneous pixel operations on the G4 VM clusters.
Encoding and Delivery: Once transformed, images are re-encoded using hardware-accelerated tools like NVENC and delivered via a global CDN. Because the G4 VM includes independent hardware engines for NVENC (encoding) and NVDEC (decoding), concurrent image manipulations on the CUDA cores aren’t slowed down.

Advanced video and image intelligence

Imgix is also using NVIDIA’s CUDA libraries for high-performance video analytics. By integrating NVIDIA DeepStream, it executes real-time object tracking within video streams for automated content analysis.

For static imagery, meanwhile, Imgix uses the nvJPEG library to offload computationally intensive JPEG decoding directly to the GPU. This prevents CPU bottlenecks during the ingestion of high-resolution assets while allowing the custom Vulkan compute shaders to begin immediate pixel-level transformations on the raw RGBA data residing in GPU memory.

The results: 50% faster and up to 6x more throughput

Thanks to its transition to G4 VMs, Imgix achieved the significant performance gains mentioned above without having to rewrite its core logic:

A 50% reduction in processing latency: It cut median latency from 100 milliseconds to 50 milliseconds.
A 5x to 6x increase in throughput: Its G4 VMs now handle up to six times the workload of its previous generation nodes.
Seamless migration: Imgix supported the G4 VMs by updating its Terraform scripts without needing to implement any application code changes.

"Building on Google Cloud's AI Hypercomputer isn't just about optimizing our current workloads; it's about future-proofing our platform. It gives us the foundational power to seamlessly weave advanced generative AI capabilities into real-time workflows, allowing our customers to push the boundaries of visual storytelling at global scale." - Alfonso Acosta, Head of Engineering, Imgix

Orchestrating at scale

To support the billions of image and video requests its customers process every day, Imgix built a sophisticated hybrid orchestration model:

Management: Google Cloud Run manages session and account layers.
Core Processing: Google Compute Engine-managed instance groups host the G4 VMs, which allows custom software to use the entire machine with no container "slicing."
Dynamic Scaling: Autoscaling relies on custom application metrics, such as machine queue length, rather than standard CPU use. This ensures that the stack’s most expensive elements are tuned for maximum efficiency.
Self-Healing: A custom mechanism monitors logs for driver faults, automatically "reaping" and restarting GPU instances without manual intervention.
Optimization: To maintain peak performance, Imgix uses NVIDIA Nsight Systems to identify and resolve code bottlenecks.

The future: From experimentation to execution

Even with the significant performance improvements it’s already achieved, Imgix is continuing to expand its AI infrastructure so its customers can access additional advanced capabilities like generative fill, background replacement, object removal, and image upscaling.

Features like these rely on high-performance machine learning systems that must process increasingly complex computations with no loss of speed or quality. By leveraging Google’s AI Hypercomputer, Imgix is now deploying and serving these models efficiently and offering its customers real-time, production-ready AI editing. And as demand grows for more dynamic and personalized visual experiences, this foundation is ensuring that Imgix can continue to deliver powerful capabilities reliably and at scale.

Get started

G4 VMs work natively with Google Compute Engine, Google Kubernetes Engine, Google Cloud Storage, and Vertex AI.

Dive deeper: Explore the Imgix architecture on GitHub.
Start building: Read the G4 VM documentation.

Cloud Storage Rapid: Turbocharged object storage for AI and analytics

Mon, 11 May 2026 17:00:00 +0000

At Google Cloud Next ’26 we announced Cloud Storage Rapid, a family of object storage capabilities for data-intensive workloads like AI and analytics. Out of the gate, Cloud Storage Rapid consists of Rapid Bucket (formerly Rapid Storage), a high-performance zonal object storage offering, and Rapid Cache (formerly Anywhere Cache), which accelerates reads on-demand and colocates compute and data for workloads in existing buckets.

Cloud Storage Rapid is our response to the generational shift in how organizations build with AI. Teams are training trillion-parameter models, deploying inference at global scale, and building autonomous agents that reason over vast amounts of enterprise data. While accelerators like GPUs and TPUs often get the spotlight, they have a critical dependency: storage.

Storage is the engine that feeds accelerators during training, and the fast-access layer that makes real-time inference responsive. But as models scale, storage performance can be a bottleneck. Every time an AI/ML cluster waits on a data read or a checkpoint write stalls, you are paying for expensive compute cycles that aren't doing useful work.

Historically, AI/ML practitioners have had to choose between the specialized performance of a niche, zonal storage system, and the reliability and scale of a global object store like Google Cloud Storage. Many developers value Cloud Storage for its simplicity, scalability, reliability, and cost-effectiveness, but as the AI era has progressed, they’ve been throwing hotter and hotter workloads at it, running training and inference workloads with thousands of GPUs and TPUs. We’ve reached a performance tipping point that traditional object storage was never meant to handle. The Rapid family provides multiple options for co-locating compute workloads directly with high-performance zonal storage. It minimizes I/O bottlenecks that can block accelerators, so that your GPUs and TPUs stay fully saturated and productive. In this blog, let’s take a closer look at Cloud Storage Rapid’s capabilities.

Rapid Bucket

Rapid Bucket (GA), helps Cloud Storage meet the evolving demands of massive-scale generative AI, analytics, and other high-performance workloads. It does so by leveraging Colossus, the Google distributed storage system that powers Gemini and YouTube, to provide massive read/write performance and ultra-low latency in a dedicated object storage zonal bucket.

Lightning-fast performance
By combining the sub-millisecond latency of block-like storage, the throughput of a parallel filesystem, and the scalability and ease of use of object storage, Rapid Bucket provides high performance from the same Cloud Storage that you know and love.

Highlights include:

Ultra-low latency: Achieve up to 20 million queries per second and sub-millisecond latency.
Massive scalability: Rapid Bucket delivers 15+ TB/s of aggregate read throughput from a single Rapid zonal bucket.
New semantics: Enable higher performance with new capabilities such as native appends, unlimited readers (while writing!), and vectored reads.

Optimized for AI and analytics
You can use Rapid Bucket for a variety of demanding scenarios, including AI/ML data preparation, training, checkpointing, batch and streaming analytics processing, and optimizing distributed database architectures.

Key benefits include:

Optimized accelerator utilization: With Rapid Bucket, we observed 50% reduced blocked GPU time and up to 2.5x faster data loading for multi-modal training runs.
Faster checkpointing: Rapid Bucket makes checkpoint restores up to 5x faster and writes 3.2x faster compared to traditional object storage. This ensures faster recovery from workload interruptions, minimizes wasted accelerator time, and increases overall efficiency.

>5x faster checkpoint restores with Rapid Bucket

>3.2x faster checkpoint writes with Rapid Bucket

You can get started with Rapid Bucket here.

Rapid Cache

Originally announced at Cloud Next ‘25, Rapid Cache accelerates bandwidth for AI/ML workloads like data prep, training, and bursty model loading for inference, delivering an aggregate read throughput of 2.5 TB/s for your existing buckets — with no code changes. For inference workloads, we’ve observed that Rapid Cache provides up to 2.1x (114%) accelerated model load, resulting in 47% TCO savings.

When combined with multi-region buckets, customers can flexibly access GPUs and TPUs distributed across regions in a geo, while maintaining a single bucket namespace. This eliminates the need for manually orchestrated data movements between buckets, while benefitting from zonally co-located high performance.

New: Rapid Cache ingest on write
Customers at some of the world’s largest frontier AI/ML labs told us that they were looking for ways to accelerate reads immediately after a write, such as checkpoint restore workloads or a data prep pipeline that then feeds training. Before, caching the data required an initial read to trigger ingestion, which was served directly from the bucket at standard performance.

Rapid Cache’s new ingest on write feature solves this by simultaneously writing data to the Rapid Cache as it is being written to a Cloud Storage bucket. This proactive approach eliminates the initial cache-miss penalty, and helps workloads benefit from an immediate cache hit on the very first read. This provides up to 2.2x faster checkpoint restore times, allowing training clusters to recover faster from interruption.

To enable ingest on write, simply modify the ingestion criteria of your existing Rapid Cache.

Rapid Cache’s simplicity and performance has resulted in explosive adoption. In just one year since General Availability, customers have deployed thousands of Rapid Caches with a 20x growth in caches deployed, In fact,Rapid Cache serves up to 20% of Cloud Storage’s global egress. Cutting-edge AI/ML customers deploy their workloads on Rapid Cache, including Anthropic who uses Rapid Cache to improve the resilience of their cloud workload by co-locating data with TPUs in a single zone and providing dynamically scalable read throughput up to 2.5TB/s.

Case study: Thinking Machines Lab
Thinking Machines Lab is an artificial intelligence research and product company. Its mission is to make AI systems that are adaptable and customizable, building a future where everyone has access to the knowledge and tools to make AI work for their unique needs and goals.

At Next ‘26, James Sun, Member of Technical Staff at Thinking Machines Lab, spoke at our session, Cloud Storage Rapid: Turbocharged object storage for AI & Analytics, where he presented about the needs of the data-hungry AI/ML workloads that Thinking Machines Lab runs for high-performance storage at scale.

Thinking Machines runs diverse workflows: data processing in Dataflow, Kafka, and Spark, multi-model training, and serving Tinker — a flexible API for fine-tuning open source models. Thinking Machines' workloads run on Google Cloud Storage, Sun explained. Running these data-intensive AI/ML workloads at such a large scale introduces significant infrastructure challenges.

The first is managing a hub and spoke data architecture, where data processing hubs are located in one primary region while training GPUs are spread across multiple regions. Historically, this has made manual data movement and lifecycle management a major operational pain point. Furthermore, Thinking Machines Lab's workloads such as data prep and pretraining workflows, which rely on massive-scale Spark workloads to prepare their multi-modal datasets, often spike from cold to hot instantly. Previously, these surges led to disruptive 429 errors, which stalled data processing and loading, and interrupted critical training cycles.

To minimize these bottlenecks, Thinking Machines Lab integrated Rapid Cache across their AI/ML pipeline, to positive results.

“Rapid Cache has become a core foundation of our AI/ML data infrastructure, supporting our critical workflows, from data prep and pretraining to training and model loading. By acting as a crucial bandwidth shield and booster, it enables us to scale our data-intensive workloads across our entire fleet without compromise, providing us with the on-demand high bandwidth and consistent stability that we need to innovate at speed.” - James Sun, Member of Technical Staff, Thinking Machines Lab

In short, Cloud Storage and Rapid Cache provides Thinking Machines Lab with:

Easy, instant, scalable, on demand bandwidth: The team now achieves stable read throughput peaks of over 1.8TB/s.
Enhanced stability: Rapid Cache has greatly reduced tail-end latencies and 429 errors, providing the consistent performance needed for multi-modal training.
Fleet-wide scalability: Combined with multi-region buckets, they can now scale data-intensive workloads across their entire fleet, meeting the demands of a rapidly growing compute scale without the hassle of manual data movement while benefiting from zonally colocated storage for high performance.
Operational efficiency: The use of Hierarchical Namespace (HNS) has optimized their massive Spark workloads for data preparation, by supporting fast directory renames, along with providing the ability to ramp QPS more quickly as they scale out clusters. Rapid Cache’s "ingest on write" capability helps ensure immediate cache hits for checkpoint restores.

Choose your rocket ship

Whether you are running data preparation, massive-scale training, or low-latency inference, Cloud Storage Rapid delivers high performance together with the reliability and scalability that Cloud Storage is known for.

Rapid Bucket delivers the highest Cloud Storage throughput and queries per second as well as the lowest latency for read/write use cases, such as analytics, AI training, checkpointing, and model serving. This helps to reduce storage bottlenecks and increase compute utilization.
Rapid Cache provides higher read bandwidth and tail latency stabilization in existing buckets, without code changes. Key use cases include AI training, checkpoint restores, and serving, as well as accelerator optionality via multi-region buckets.

Get started with the Cloud Storage Rapid family today!

Cluster-level reliability for trillion-parameter models on TPUs

Mon, 11 May 2026 16:30:00 +0000

Frontier AI models have redefined the unit of compute. At trillion-parameter scale, AI training requires thousands of interconnected components, orchestrated in industrial-scale deployments to operate as a single, massive entity.

Likewise, when it comes to reliability, aggregate infrastructure availability is what matters. Yet for almost two decades, instance-level reliability has been the cloud standard. Designed for microservices and horizontally scalable applications, instance-level reliability treats infrastructure as a collection of small independent units. This model is fundamentally inadequate for large-scale AI workloads.

We believe reliability must shift from an instance- to a cluster-level model.

For over a decade, Google has operated Tensor Processing Unit (TPU) clusters at scale, achieving reliability that meets the architectural requirements of modern AI workloads. In this blog, we’re presenting our cluster-level reliability framework for Google Cloud TPUs that focuses on collective performance at the superpod level, and one we use internally within Google to build the world’s most advanced AI models. This framework is the operational standard for TPUs in production today, and serves as the architectural blueprint for our recently announced eighth-generation TPUs.

Reliability for AI supercomputers

TPU superpods consist of thousands of chips arranged into cubes (64 TPUs), with high-speed Inter-Chip Interconnect (ICI) links connecting every chip within a cube and a dynamically configurable Optical Circuit Switch (OCS) network connecting all cubes to form a superpod.

For system-wide training progress, we must maximize the number of fully healthy cubes within a superpod. Because the performance of AI models relies on high-bandwidth, low-latency communication, every chip and ICI link within a cube must be operational for that unit to contribute to the training progress. Driven by these architectural realities, our cluster-level framework helps define how the industry can achieve reliability in the AI era, moving from instance-level reliability to availability of scale.

Deep dive: The mathematics of availability at scale

Instance-level reliability models are often deterministic, but industrial-scale AI deployments require a probabilistic approach over thousands of chips. In a traditional setup, you might track the Mean Time Between Failures (MTBF) of a single chip. However, at the scale of frontier AI, the cluster-level MTBF drops sharply as the number of components grows.

To visualize how quickly scaling can erode confidence, we can look at simple bounds like Markov’s inequality.

If we define X as the number of failed cubes, Markov’s inequality reminds us that as the expected number of failures E[X] increases with cluster size, the probability of staying below a strict failure threshold becomes increasingly difficult to guarantee without systemic architectural changes.

While Markov’s inequality provides a helpful rule of thumb for the risks at scale, we model the availability of scale using a binomial distribution of aggregate cluster health. For a superpod composed of n independent units (cubes), we define the probability of having at least k fully operational and interconnected cubes as the cumulative distribution of the success of n independent trials. To ensure a 95% confidence interval for training productivity, we solve for k where:

Where n represents the total cubes in a superpod and p represents the aggregate cube-level availability.

This model replaces the instance-level model with a topology-aware framework that mirrors actual performance requirements of large-scale training, ensuring that the larger block of compute is healthy and connected and can drive continuous training progress.

Scale of modern AI hardware

To demonstrate this new reliability model, we used Ironwood, Google’s generally available, seventh-generation TPU, and the custom silicon behind advanced models like Gemini and Nano Banana.

Pictured: Part of an Ironwood superpod, directly connecting 9,216 Ironwood TPUs in a single domain.

An Ironwood superpod is a dense, high-performance fabric consisting of 9,216 chips integrated into a single compute domain. It’s organized into 144 cubes, where each cube contains 64 chips. Within these cubes, ICI links create an extremely dense, all-to-all network fabric that provides massive bandwidth and low-latency connectivity for distributed operations within the cube. To form a superpod, 144 cubes are connected using OCS. For large jobs, capacity can be provisioned by interconnecting multiple cubes within a pod into one super-slice, or connecting multiple slices to form a multi-slice cluster. Cubes across multiple superpods can be connected over the datacenter network to run even larger workloads.

Using this model, we determine that the topological availability for an Ironwood superpod is 130 out of 144 cubes available for 95% of the month. This translates to a large compute block of 8,320 chips that are fully operational and interconnected via ICI and OCS, establishing a reliability model specifically optimized for hero jobs (the massive training runs of frontier AI).

The relationship between cluster size and its statistical availability is non-linear. By adjusting the required confidence level, we can identify the slice size that can be supported with statistical certainty. For researchers, this mapping provides a capacity availability curve. An organization with a workload that requires 99% availability for a mission-critical run can optimize their slice size to 125 cubes, while those pushing utmost scale can utilize 130 cubes at the 95% confidence interval.

Capacity availability curve for an Ironwood superpod (144 cubes)

This new reliability model maximizes the utility of the entire superpod through:

Full access: This model does not constrain capacity utilization; it focuses on the availability of fully healthy cubes. While a single chip or ICI failure results in the entire cube being classified as unhealthy, customers continue to have access to the remaining capacity within the cube. This makes most of the Ironwood superpod available for use while also optimizing the compute footprint for high-stakes, large-scale training.
Optimized resource usage: While the 130-cube model focuses primarily on large-scale training runs, the full superpod remains available for a heterogeneous mix of workloads. This allows researchers to utilize the remaining cubes for research experiments, inference, and dev/test workloads, maximizing the utility of the superpod without compromising the reliability of the main training run.

Our customers are using Ironwood at scale today and this model has empowered them to train their most demanding hero jobs.

Enhancing ML productivity

The goodput metric is the primary measure of ML productivity. Our new reliability standard provides the deterministic foundation for goodput and is engineered to maximize this metric for demanding hero jobs, so that the massive scale infrastructure required for frontier research is ready to perform as a single entity.

This model achieves high scheduling goodput, one of the three goodput metrics, by making the full set of resources available for massive-scale training runs. Combined with the software stack, this infrastructure-level availability helps deliver the high overall goodput. We achieve this through a three-layer reliability model:

Infrastructure: TPU superpods provide the capacity footprint to ensure the necessary scale is physically available and connected.
Frameworks: JAX and Pathways provide resilience, reconfiguring or hot-swapping around failed nodes to maintain forward progress without requiring a full restart.
Application: Fault-tolerance mechanisms like auto-checkpointing and multi-tier checkpointing preserve training state, so that lost progress is minimized in case of a failure.

Enabling the next generation of AI breakthroughs

The cluster-level reliability model marks the beginning of a new standard for the AI era, where an AI supercomputer is a dependable, industrial-scale engine for innovation. By aligning our reliability posture with the demands of frontier models, we’re enabling the next generation of AI breakthroughs to be faster, more reliable, and more predictable. Click here to learn more and get started with TPUs.

Introducing Virgo Network, Google’s scale-out AI data center fabric

Wed, 22 Apr 2026 12:00:00 +0000

The AI era requires a fundamental rethink of physical cloud architecture — networking, in particular. With foundational model parameters growing exponentially, traditional general-purpose networks are reaching their breaking points. To fuel the next decade of machine learning, Google designed Virgo Network, a new megascale AI data center fabric that embraces a "campus-as-a-computer" philosophy, and that underpins our AI Hypercomputer.

Legacy network designs simply cannot handle some of the constraints of modern AI:

Massive scale: Training demands now exceed the power and space of a single data center, requiring unified, multi-data-center domains.
Explosive bandwidth growth: Because foundational model training is heavily network-bound, the required bandwidth per accelerator has surged significantly over the last few years, creating throughput and congestion bottlenecks for older architectures.
Synchronized bursts: Intense, millisecond-level traffic spikes (figure 1) put immense pressure on network buffers. The outcome is that even a single "straggler" node can throttle the entire cluster’s performance.
Low latency: ML serving requires fast, consistent response times to deliver real-time inference, making strict latency control a critical architectural constraint.

Figure 1: Sub-millisecond line-rate bursts of an AI training workload

Reimagining the data center network

Meeting the demands of the AI era requires a fundamental shift away from general-purpose network design towards a specialized flat, low-latency network architecture. To address the unique scale and latency constraints, we leverage our proven Jupiter network for north-south traffic and are introducing a new fabric for east-west communication. The resulting architecture consists of three distinct and specialized layers that operate as one unified compute domain:

Scale-up domain: A high-bandwidth, low-latency interconnect fabric designed for tightly coupled communication between accelerators within a single pod.
Scale-out accelerator fabric (east-west): A dedicated accelerator-to-accelerator remote direct memory access (RDMA) fabric optimized for massive horizontal scale across pods. This layer is engineered for deterministic latency and maximum resilience, to provide high “goodput” for the ML workload.
Jupiter front-end network (north-south): A high-capacity fabric that provides fast, reliable access to distributed storage and general-purpose compute resources. It ensures that data access does not become a bottleneck for training and serving workloads, and is also used to scale-across multiple sites for very large training runs.

This architectural decoupling provides key strategic advantages:

Independent evolution: We can evolve and upgrade each network domain independently, preventing system-wide disruptions while accelerating the innovation cycle.
Dedicated scale-out bandwidth: A non-blocking network delivers massive bisectional bandwidth to accelerators for critical training tasks.
ML and network co-design: The network is built in lockstep with each new generation of ML accelerators, helping ensure the fabric is matched to the hardware it supports.

Figure 2: Data center network architecture

Introducing Virgo Network: Megascale data center fabric

Virgo Network is a scale-out fabric designed for the extreme requirements of modern AI workloads. Built on high-radix switches that reduce network layers by allowing more ports per switch, it employs a flat, two-layer non-blocking topology. Compared with traditional datacenter networks, this significantly reduces latency by minimizing network tiers. It features a multi-planar design with independent control domains to connect accelerators (figure 3). The accelerator racks also connect with the Jupiter north-south fabric to access compute and storage services. Together, this streamlined architecture delivers the massive bisection bandwidth and deterministic low latency necessary for both distributed training and serving workloads.

Figure 3: Megascale data center fabric (Virgo Network)

Virgo Network is the foundation of our next-generation accelerator designs and delivers the following advantages:

Massive fabric scale: Virgo Network can link 134,000 chips (TPU 8t) with up to 47 petabits/sec of non-blocking bi-sectional bandwidth in a single fabric.
Generational performance leap: With up to 4x the bandwidth per accelerator (TPU 8t) over the previous generation, Virgo Network delivers the bandwidth you need to get the full power of every chip.
Predictable low latency: Virgo Network delivers 40% lower unloaded fabric latency for TPUs compared to previous generation leading to more predictable performance for latency sensitive AI workloads.

Improving reliability at scale

In a system supporting hundreds of thousands of chips, hardware failures are inevitable. Because a single faulty component can disrupt a synchronized training job, reliability at scale is a primary focus. To maximize workload goodput, we designed the Virgo Network architecture around fault isolation, deep observability, and the rapid mitigation of hangs and stragglers.

At this scale, system-wide resilience requires a solid network foundation. Virgo Network integrates independent switching planes that provide robust fault isolation, protecting cluster-wide goodput from being degraded by localized hardware failures.

Figure 4: How fail-stop and fail-slow impact MTTR

Building on this foundation, we optimize the software and orchestration stack to maximize mean-time between interruptions (MTBI) and minimize mean-time to recovery (MTTR) through two primary areas:

Observability: Reliability at scale requires high-fidelity visibility. We use sub-millisecond telemetry to monitor network systems. This deep visibility allows us to detect transient congestion, optimize buffer management, and pinpoint the root causes of slowdowns across the hardware and software stack.
Identifying stragglers and hangs: Proactive monitoring is critical for identifying nodes that are experiencing performance degradation (stragglers) or that have stopped responding completely (hangs). By rapidly localizing these bottlenecks, with automated straggler and newly added hang detection, we accelerate the training job and protect it from localized slowdowns.

The foundation of the AI Hypercomputer

Virgo Network is a reimagined scale-out data center network custom-built for the stringent demands of modern AI workloads. This flat, multi-planar architecture unifies accelerators across pods into a single compute domain, addressing the bandwidth and scale limitations of traditional networks. By providing robust fault isolation directly at the hardware level, Virgo Network serves as the foundation for system-wide resilience, protecting synchronized workloads from localized hardware faults.

Ultimately, Virgo Network delivers the scale, predictable latency, and reliability necessary to accelerate the agentic AI era. To learn more about how we are building infrastructure for the future of AI, visit our AI infrastructure solutions page, explore the technical documentation, or attend the dedicated breakout session at Google Cloud Next.

Inside the eighth-generation TPU: An architecture deep dive

Wed, 22 Apr 2026 12:00:00 +0000

At Google, our TPU design philosophy has always been centered on three pillars: scalability, reliability, and efficiency. As AI models evolve from dense large language models (LLMs) to massive Mixture-of-Experts (MoEs) and reasoning-heavy architectures, the hardware must do more than just add floating point operations per second (FLOPS); it must evolve to meet the specific operational intensities of the latest workloads.

The rise of agentic AI requires infrastructure that can handle long context windows and complex sequential logic. At the same time, world models have emerged as a necessary evolution from current next-sequence-of-data architectures, which means newer agents are simulating future scenarios, anticipating consequences, and learning through "imagination" rather than risky trial-and-error. The eighth-generation TPUs (TPU 8t and TPU 8i) are our answer to these challenges, ensuring that every workload, from the first token of training to the final step of a multi-turn reasoning chain, is running on the most efficient path possible. They are built to efficiently train and serve world models like Google DeepMind’s Genie 3, enabling millions of agents to practice and refine their reasoning in diverse simulated environments.

TPU 8: Specialized by design

Recognizing that the infrastructure requirements for pre-training, post-training, and real-time serving have diverged, our eighth-generation TPUs introduce two distinct systems: TPU 8t and TPU 8i. These new systems are key components of Google Cloud's AI Hypercomputer, an integrated supercomputing architecture that combines hardware, software, and networking to power the full AI lifecycle. While both systems share the core DNA of Google’s AI stack and support the full AI lifecycle, each is built to address distinct bottlenecks and optimize efficiency for critical stages of development. Additionally, by integrating Arm-based Axion CPU headers across our eighth-generation TPU system, we’ve removed the host bottleneck caused by data preparation latency. Axion provides the compute headroom to handle complex data preprocessing and orchestration, so that TPUs stay fed and don’t stall.

TPU 8t: The pre-training powerhouse

Optimized for massive-scale pre-training and embedding-heavy workloads, TPU 8t utilizes our proven 3D torus network topology at an even larger scale of 9,600 chips in a single superpod. TPU 8t is designed for maximum throughput across hundreds of superpods, ensuring that training runs stay on schedule.

Here are some key advancements of TPU 8t over prior-generation TPUs:

The SparseCore advantage: Central to TPU 8t is the SparseCore, a specialized accelerator designed to handle the irregular memory access patterns of embedding lookups. While the Matrix Multiply Unit (MXU) handles matrix math, the SparseCore offloads data-dependent all-gather operations, amongst other collectives, preventing the zero-op bottlenecks that often plague general-purpose chips.
VPU/MXU overlap and balanced scaling: TPU 8t is designed to maximize provisioned FLOPs utilization. By implementing more balanced Vector Processing Unit (VPU) scaling, the architecture minimizes exposed vector operation time. This allows for better overlapping of quantization, softmax, and layernorms with the matrix multiplications in the MXU, helping the chip stay busy rather than waiting on sequential vector tasks.
Native FP4: TPU 8t introduces native 4-bit floating point (FP4) to overcome memory bandwidth bottlenecks, doubling MXU throughput while maintaining accuracy for large models even at lower-precision quantization. By reducing the bits per parameter, the platform minimizes energy-intensive data movement and allows larger model layers to fit within local hardware buffers for peak compute utilization.

Figure 1: TPU 8t ASIC block diagram

Virgo Network topology and up to 4x data center network increase: To support the massive data requirements of TPU 8t, we introduced Virgo Network. This new networking architecture enables up to 4x increased data center network (DCN) bandwidth on TPU 8t training over DCN. Virgo Network is a scale-out fabric designed for the extreme requirements of modern AI workloads. Built on high-radix switches that reduce network layers by allowing more ports per switch, it employs a flat, two-layer non-blocking topology. Compared with traditional datacenter networks, this significantly reduces latency by minimizing network tiers. It features a multi-planar design with independent control domains to connect TPU 8t chips. The TPU 8t racks also connect with the Jupiter north-south fabric to access compute and storage services. Together, this streamlined architecture delivers the massive bisection bandwidth and deterministic low latency necessary for enabling the world's largest training clusters with high availability.

With 2x scale-up bandwidth on the inter-chip interconnect (ICI) and up to 4x raw scale-out DCN bandwidth compared to the previous generation, TPU 8t drastically reduces data bottlenecks. Then, to further accelerate the development of frontier models, we scale distributed training beyond a single cluster. With JAX and Pathways, we can now scale to more than 1 million TPU chips in a single training cluster. Virgo Network can link over 134,000 TPU 8t chips with up to 47 petabits/sec of non-blocking bi-sectional bandwidth in a single fabric. This fabric delivers over 1.7K ExaFlops with near-linear scaling performance.

Figure 2: TPU 8t rack level connectivity to Virgo fabric

Faster storage access: We are introducing TPUDirect RDMA and TPU Direct Storage in TPU 8t. TPU Direct RDMA enables direct data transfers between the TPU's memory (HBM) and the Network Interface Cards (NICs), bypassing the host CPU and DRAM. This reduces latency and host system bottlenecks, increasing the effective bandwidth for TPU-to-TPU communication. Similarly, TPUDirect Storage bypasses CPU host bottlenecks by enabling direct memory access between the TPU and high-speed managed storage like 10T Lustre, effectively doubling the bandwidth for massive data transfers. This architecture allows the silicon to ingest training data at line rate, ensuring that the MXUs stay fully saturated even when processing large multimodal datasets.

By combining Managed Lustre 10T and TPUDirect Storage to route hundred-petabyte datasets directly to the silicon, TPU 8t prevents training delays caused by data ingestion bottlenecks. This delivers 10x faster storage access compared to training on seventh-generation Ironwood TPUs.

Figure 3: The top diagram shows the data transfer path without TPUDirect Storage. The bottom diagram shows TPU 8t data transfer with TPUDirect Storage between 2 TPU 8t chips and TPUDirect Storage with Managed 10T Lustre storage.

TPU 8i: The sampling and serving specialist

Optimized for post-training and high-concurrency reasoning, we designed TPU 8i with our highest on-chip SRAM, a new Collectives Acceleration Engine (CAE), and a new serving-optimized network topology called Boardfly.

Large on-chip SRAM: With 3x more on-chip SRAM over the previous generation, TPU 8i can host a larger KV Cache entirely on silicon, significantly reducing the idle time of the cores during long-context decoding.

Figure 4: TPU 8i ASIC block diagram

The Collectives Acceleration Engine (CAE): To solve the sampling bottleneck, TPU 8i uses the CAE, which aggregates results across cores with near-zero latency, specifically accelerating the reduction and synchronization steps required during auto-regressive decoding and "chain-of-thought" processing. For each TPU 8i chip, there are two Tensor Cores (TC) on-core dies and one CAE on the chiplet die, replacing four SparseCores (SCs) on core dies in previous-generation Ironwood TPU. By integrating a specialized CAE, TPU 8i further reduces the on-chip latency of collectives by 5x. Lower latency per collective operation means less time spent waiting, directly contributing to higher throughput required to run millions of agents concurrently.
Boardfly ICI topology: While the 3D torus allows connecting thousands of chips to be used in cohesion, a large mesh does have more hops between chips and higher all-to-all latencies. For 8i, we changed how the chips connect together in fully connected boards that are then aggregated into groups. Utilizing a high-radix design, we connect up-to 1,152 of these chips together, reducing the network diameter and the number of hops a data packet must take to cross the system. By slashing the hops required for all-to-all communication (the heart of MoE and reasoning models), Boardfly achieves up to a 50% improvement in latency for communication-intensive workloads.

Figure 5: TPU 8i hierarchical Boardfly topology building up from a building block of four fully connected chips into a fully connected group of eight boards, with 36 of such groups fully connected into a TPU 8i pod

Boardfly consists of the following elements, and its topology is hierarchical by nature:

Building Block (BB): Each tray forms a four-chip ring using internal ICI links, providing 16 external connections for broader networking.
Group (G): Eight boards are fully connected via copper cabling to create a localized group, utilizing 11 of the available external links for intra-group communication.
Pod structure: The final architecture scales to 36 groups (up to 1,024 active chips) linked through Optical Circuit Switches (OCS), ensuring a maximum latency of seven hops for any chip-to-chip communication.

Deep dive: The Boardfly vs. torus math

Why move away from the torus for TPU 8i? It comes down to network diameter.

In a 3D torus, nodes are arranged in a grid where each dimension wraps around like a ring. To reach the furthest possible chip in a 8 x 8 x 16 (1024-chip) configuration, a packet must traverse half the distance of each ring:

3D torus = 8/2(X) + 8/2(Y) + 16/2(Z) = 16 hops

While the torus is highly efficient for the neighbor-to-neighbor communication typical of dense training, it creates a latency tax for all-to-all communication patterns. In the era of reasoning models and MoE, where any chip may need to talk to any other chip to route a token, this hop count matters.

Boardfly’s high-radix topology is inspired by Dragonfly topology principles. By increasing the number of direct optical long-haul links between groups of boards, we flatten the network. For that same 1024-chip pod, Boardfly reduces the network diameter from 16 hops down to just seven.

This 56% reduction in network diameter translates directly to lower tail latency, so that the TPU 8i CAE isn't left waiting for data to arrive from across the pod.

Figure 6: A visual representation of the maximum seven-hop ICI network diameter via optical circuit switch on TPU 8i pod

TPU 8t and TPU 8i at a glance

Feature	TPU 8t	TPU 8i
Primary Workload	Large-scale pre-training	Sampling, serving, and reasoning
Network Topology	3D torus	Boardfly
Specialized Chip Features	SparseCore (Embeddings) & LLM Decoder Engine	CAE (Collectives Acceleration Engine)
HBM Capacity	216 GB	288 GB
On-Chip SRAM (Vmem)	128 MB	384 MB
Peak FP4 PFLOPs	12.6	10.1
HBM Bandwidth	6,528 GB/s	8,601 GB/s (~1.3x of TPU 8t)
CPU Header	Arm Axion	Arm Axion

Software enablement: A performance-first AI stack

Hardware is only as powerful as the software that drives it. The eighth generation of TPUs are built on the same performance-first stack we pioneered with the seventh-generation Ironwood TPUs, designed to make custom kernel development accessible without sacrificing the abstraction of high-level frameworks. This stack includes:

Pallas and Mosaic: We provide first-class support for Pallas, our custom kernel language that lets you write hardware-aware kernels in Python. This enables you to squeeze every drop of performance out of the TPU 8i CAE and the TPU 8t SparseCore.
Native PyTorch experience: We're thrilled to share that native PyTorch support for TPUs is now in preview. If you're currently building and serving models on PyTorch, we've made it easier than ever to start using TPUs. You can bring your existing models to our TPUs just as they are, complete with full support for the native features you rely on, such as Eager Mode.
Portability: The same JAX, PyTorch, or Keras code that runs on Ironwood scales to this generation. Accelerated Linear Algebra (XLA) handles the complex translation of the Boardfly topology and CAE synchronization behind the scenes, allowing you to focus on your model, not the interconnect.

Generation over generation: The performance leap

Our commitment to co-designing hardware and software continues to pay dividends. When compared to seventh-generation Ironwood TPU, the eighthgeneration TPUs deliver massive gains:

Training price-performance: TPU 8t delivers up to 2.7x performance-per-dollar improvement over Ironwood TPU for large-scale training.
Inference price-performance: TPU 8i delivers up to 80% performance-per-dollar improvement over Ironwood TPU, particularly at low-latency targets for large MoE models.
Energy efficiency: Both chips deliver up to 2x better performance-per-watt, critical for scaling the next generation of AI sustainably.

Looking ahead

To empower Google Cloud customers pioneering the next wave of innovation, we designed TPU 8t and TPU 8i as two distinct, specialized systems tailored to the multifaceted future demands of the AI lifecycle. TPU 8t and 8i are both purpose-built for the most demanding serving and training workloads, fully integrating with the AI Hypercomputer software stack: JAX, PyTorch, vLLM, XLA, and Pathways. This specialization and ground-up redesign, all in deep collaboration with Google Deepmind, delivers exceptional price-performance and power efficiency.

The modularity of our eighth-generation architecture provides a clear and unique roadmap for the future. Just as every major shift in the computing landscape has required infrastructure breakthroughs, so does the agentic era. Reasoning agents that plan, execute, and learn within continuous feedback loops cannot operate at peak efficiency on hardware that was originally optimized for traditional training or transactional inference; their operational intensity are fundamentally distinct. Our eighth-generation TPU infrastructure has evolved to meet these specific requirements head-on.

To learn more about the eighth-generation TPU family:

Storage innovations to accelerate your AI workloads at Next ‘26

Wed, 22 Apr 2026 12:00:00 +0000

At Google Cloud Next, we are announcing innovations across every layer of our storage stacks — performance, intelligence, and management — to ensure your data is as fast and as useful as the AI models, apps and agents you are building.

Why it matters: Storage is no longer just a place to keep data. When training AI models, storage is the engine that feeds data-hungry accelerators. During AI inference, it’s the access layer that makes it responsive, acting as the source for the context that AI agents need to be effective. When storage performance falls short, accelerators sit idle, agents respond slowly, and data remains invisible to AI models.

But storage performance is only half the battle; you also need storage that’s smart. With the help of Google’s AI models integrated directly into the storage layer, you’re no longer just storing bits, but data that has full context about its content. In this new era of smart storage, raw data becomes a valuable asset that’s ready to use by a variety of downstream AI and enterprise applications.

What’s new:

High-performance storage infrastructure: New Rapid family of features in Cloud Storage for high-performance object storage; delivering 10x performance enhancements plus a new cost-effective Dynamic tier for Google Cloud Managed Lustre.
Smart Storage: Unlocking unstructured data with automated metadata annotation, and AI agent connectivity via MCP.
Storage Intelligence: Streamlined data management through zero-configuration dashboards, aggregated activity views, and enhanced batch operations.
Enhanced ecosystem: Expanded capabilities across Google Cloud NetApp Volumes, Filestore for GKE, and our backup and data protection portfolio.

Let’s take a deeper look at the storage enhancements we are unveiling this week.

Storage infrastructure that keeps up with AI

As AI models scale, getting data from the storage to the compute layer fast enough can be a bottleneck. New storage capabilities move performance directly into the storage layer, reducing total cost of ownership (TCO) and keeping accelerators fully utilized.

Cloud Storage Rapid
Cloud-based object storage like our Cloud Storage is scalable and cost-effective, but bottlenecks can stall AI jobs and waste expensive compute cycles. Every time a training cluster waits on a read or a checkpoint write stalls, you're paying for accelerators that aren't doing useful work.

Cloud Storage Rapid marks a fundamental shift in designing AI infrastructure: you no longer have to choose between the reliability of object storage and the high performance of a specialized AI storage system. Cloud Storage Rapid lets you leverage the industry-leading durability, massive distributed scale, and cost-effective auto-tiering of object storage, while simultaneously achieving extreme throughput, frequent I/Os, and ultra-low latency. With native integrations into PyTorch and JAX, Cloud Storage Rapid is optimized out-of-box for the most popular AI/ML ecosystem frameworks, so that your data preparation, training, and inference workloads run on a high-performance and reliable foundation.

The Cloud Storage Rapid family consists of two offerings: Rapid Bucket and Rapid Cache.

Rapid Bucket, now generally available, leverages Colossus, the Google distributed storage system that powers Gemini and YouTube, to deliver more than 15 TB/s of bandwidth, 20 million requests per second, and sub-millisecond latency in a single zonal bucket. With access via high-performance gRPC and S3-compatible APIs, Rapid Bucket increases accelerator utilization for multi-modal training with 50% reduced GPU blocked time and 2.5x faster data loading. Checkpoint restores are 5x faster and checkpoint writes are 3.2x faster compared to traditional object storage, reducing workload interruptions and wasted GPU time.

Checkpoint writes are 3.2x faster and restores are 5x faster with Rapid Bucket

Rapid Cache, formerly Anywhere Cache, accelerates bandwidth for bursty workloads like model loading for inference, delivering an aggregate read throughput of 2.5 TB/s for existing buckets, with no code changes. The new ingest-on-write feature provides up to 2.2x faster checkpoint restores, allowing training clusters to recover faster from interruptions.

Rapid Cache’s combination of simplicity and performance has resulted in strong adoption, including from cutting-edge AI/ML customers like Thinking Machines Lab.

“Rapid Cache has become a core foundation of our AI/ML data infrastructure, supporting our critical workflows, from data prep and pretraining to training and model loading. By acting as a crucial bandwidth shield and booster, it enables us to scale our data-intensive workloads across our entire fleet without compromise, providing us with the on-demand high bandwidth and consistent stability that we need to innovate at speed.” - James Sun, Member of Technical Staff, Thinking Machines Lab

Google Cloud Managed Lustre
The Lustre parallel file system is the industry standard for organizations whose AI training and inference workloads require high throughput and sub-millisecond latency, and is trusted by AI labs and HPC centers worldwide to feed thousands of accelerators simultaneously and keep them saturated under pressure. Google Cloud Managed Lustre brings that capability as a fully managed service, and with today's announcements, it is the most performant managed Lustre offering available in any cloud.

Managed Lustre now delivers up to 10 TB/s of throughput — a 10x increase since last year and 4–20x higher than managed Lustre offerings from other hyperscalers for a single instance. Powered by C4NX VMs and Hyperdisk Exapools, Managed Lustre writes and restores checkpoints 2.6x faster when compared to other Google Cloud storage solutions.

The new Dynamic tier ($0.06/GB-month) delivers the low-latency performance required for intense AI workloads like training and checkpointing. By serving data from persistent disk rather than relying on object-based caching, we eliminate a performance cliff — helping ensure your data remains responsive and your accelerators stay productive. A single SKU provides simple, predictable billing without the hidden complexity of traditional data tiering.

“By integrating Managed Lustre we eliminated the typical onboarding bottlenecks, allowing us to hit the ground running with the inferencing workload. This high-throughput, low-latency storage keeps our B200 GPUs fully saturated, driving a substantial performance gain in LLM inference over the H200. For our customers, this translates directly into faster, more responsive AI agents that can handle complex reasoning at a fraction of the previous latency.” - Lavnaya Karanam, Software Engineering PMTS, Salesforce

Smart Storage: Context for the AI era

The beauty of an object storage system like Cloud Storage has long been its simplicity: the system knows the object’s name, its size, and when it was created. But if you want to understand the object’s content — what entities it references, whether it contains sensitive PII, or whether it’s relevant to a pending query — you need to use custom pipelines, separate databases, and bespoke enrichment systems.

AI has changed the equation. To fine-tune a model, you need to select the right objects from the get-go, from a corpus of millions. Building an agent requires retrieving the right context for each decision. To meet a compliance obligation, you need to know what every file contains up front, before it becomes a liability. In each case, the bottleneck isn’t compute or model quality — it’s the inability to describe, find, and act on objects at scale.

To bridge that gap between stored and usable data, last year we introduced Smart Storage, a set of capabilities built directly into Cloud Storage that makes every object self-describing. New Smart Storage capabilities include:

Automated annotations, which eliminates the need to build and maintain custom annotation pipelines. With Smart Storage enabled, Cloud Storage can now automatically generate context — including image annotations — so your data is discoverable and usable from the moment it lands. You pay to annotate the data once at write time, and every downstream system can use those annotations immediately for the life of the object.
Cloud Storage MCP server lets you read, write, and analyze Cloud Storage data using the standard MCP protocol.

Smart Storage enables these capabilities, and others, thanks to its object context, now generally available. This metadata substrate adds structured, mutable, IAM-governed context to every object. Customers write their own tags and classifications; Google's annotation pipelines automatically attach labels, extracted entities, and compliance signals.

With Smart Storage, ML teams can select training datasets from semantic criteria without building retrieval pipelines. AI agents can ground their reasoning in enterprise data without a separate retrieval layer.

Storage Intelligence: Data management at AI scale

As data estates grow to hundreds of petabytes, storage costs can spike without warning, and security blind spots can multiply across billions of objects. To manage this, teams have to stitch together multiple tools just to answer basic questions about their own data.

Last year we launched Storage Intelligence to give enterprises a unified management experience built directly into Cloud Storage. Today, 70% of our largest customers use Storage Intelligence, each of whom manage over 50 billion objects.

Storage Intelligence provides a single view across your entire project or organization, with unique capabilities like bucket relocations across regions. Today, we're making it significantly more powerful with:

New zero-configuration dashboards instantly surface cost anomalies and integrate Security Command Center’s Data Security Posture Management (DSPM) data governance feature, to detect critical security vulnerabilities across Cloud Storage — no setup required.
New object events and bucket activity tables in Insights Datasets now drive deeper cost analysis and accelerate operational tasks. You can use these insights to perform a wide range of analyses, from optimizing bucket placement based on egress patterns to quickly troubleshooting 429 errors by finding the impacted objects.
Enhanced batch operations make it even simpler to act on billions of objects with new change ACL and storage class operations, and support for multi-bucket operations.

Enhancing the storage ecosystem

Beyond our core storage offerings, we are streamlining how enterprises migrate to and protect data in the cloud.

Google Cloud NetApp Volumes: With the launch of Flex Unified, NetApp Volumes now provides a unified enterprise storage platform that bridges the data center and the cloud, provisioning both block (iSCSI, NVMe/TCP) and file (NFS/SMB) on the same storage pool. New ONTAP-mode lets you bring your existing automation (Terraform, Ansible) and ONTAP APIs directly to NetApp Volumes.
Filestore for GKE: Developers building AI workloads on Google Kubernetes Engine (GKE) can start small, with shares as small as 100 GiB, and scale capacity and IOPS independently. At the same time, tighter integration to the Colossus distributed file system provides more scale and enterprise capabilities.
Data protection: Google Cloud Backup and DR now features agentic AI capabilities that can autonomously audit your backup estate and remediate coverage gaps, with new GA integrations for AlloyDB and Filestore.

Where to start

As you navigate today’s generational AI shift, you need a storage foundation to support ever-larger, more intelligent, and autonomous models. With new high-performance and intelligent storage layers, plus enhanced storage management tools and a deeper data protection bench, Google Cloud’s storage platforms lets you understand and use your enterprise data in ways that weren’t previously possible, allowing you to:

Reduce the AI data bottleneck: Saturate compute and accelerate ROI. Keep your expensive GPUs and TPUs fully productive with high-throughput storage that delivers the extreme performance required for large-scale training and inference.
Build agent-ready data foundations: Shift from building custom pipelines to an active knowledge base where self-describing objects let AI agents instantly reason over data without manual prep.
Minimize blind spots across exabytes: Replace fragmented management tools with zero-configuration dashboards and datasets to instantly surface cost anomalies and security risks across billions of objects.
Embrace the storage ecosystem: Streamline migration and protection. Bridge your data center to the cloud, scale containerized apps, and automate data resilience with agentic AI.

Visit the Google Cloud Storage console to explore these new features, read more about Cloud Storage Rapid, or explore our Next '26 storage sessions.

What’s next in Google AI infrastructure: Scaling for the agentic era

Wed, 22 Apr 2026 12:00:00 +0000

AI is evolving from answering questions to reasoning and taking action. Companies who want to lead in today’s agentic era require computing infrastructure designed and optimized for these new requirements. Today at Google Cloud Next, we are introducing new AI infrastructure capabilities that help you innovate faster, deliver compelling user and customer experiences, and optimize for cost and energy efficiency — all at massive scale.

The shift to agentic intelligence

In the agentic era, a single intent triggers a chain reaction. Unlike chat, a primary AI agent decomposes goals into specific tasks for a fleet of specialized agents that then collaborate, preserve state, and use reinforcement learning to deliver outcomes in real-time.

This process scales intelligence per interaction, but also creates complexity that yesterday’s architectures cannot support without spiraling costs or performance bottlenecks. To scale efficiently and effectively, you must move beyond manually integrating fragmented components and technologies. To deliver agentic experiences that are smart, fast, scalable, and cost-effective, you need a unified infrastructure stack that spans purpose-built hardware, open software, and flexible consumption models.

Google’s AI Hypercomputer is AI-optimized infrastructure built for the agentic era, engineered to deliver on these new requirements. This is the same foundation that powers Google’s flagship Gemini models, consumer AI services, and enterprise AI offerings. Today, we are announcing a significant expansion of our AI infrastructure portfolio, including:

TPU 8t and TPU 8i, our eighth generation TPUs
A5X bare metal instances, powered by NVIDIA Vera Rubin NVL72
Axion N4A VMs, powered by our custom Axion Arm-based CPUs
Google Compute Engine 4th generation VMs, powered by Intel and AMD x86-based CPUs
Virgo Network, our breakthrough data center fabric for AI workloads
Google Cloud Managed Lustre, a high-performance parallel file system
Z4M VMs with high-capacity local SSD storage and RDMA for open parallel file systems
Dedicated KV Cache scalable storage subsystem
Native PyTorch support for TPUs
New Google Kubernetes Engine (GKE) capabilities for agent-native workload orchestration

Taken together, these capabilities will help you accelerate the development of models and complex agentic workflows to accelerate innovation, and deliver useful, responsive services to customers, all while reducing costs and using energy responsibly at scale.

Let’s take a closer look.

Introducing our eighth-generation TPU systems, purpose-built for agentic AI

Today, we’re pleased to announce the eighth generation of our Tensor Processing Units (TPUs), which for the first time includes two distinct chips and specialized systems, engineered specifically for the agentic era.

TPU 8t is our training powerhouse, specifically designed for high-throughput AI workloads. It redefines the scale of AI development, delivering nearly 3x higher compute performance than previous generations to shrink training timelines for massive models. It packs 9,600 chips in a single superpod to provide 121 exaflops of compute and two petabytes of shared memory connected through high-speed inter-chip interconnects (ICI). This massive pool of compute, unified memory, and doubled ICI bandwidth helps ensure that even the most complex models achieve near-linear scaling and maximum system utilization. We can now turn months of training into weeks with the power of 1 million+ TPU chips in a single cluster, orchestrated by Pathways and JAX.

TPU 8i is our breakthrough reasoning system for inference and reinforcement learning (RL), engineered to deliver the ultra-low latency required by agentic workflows and Mixture of Experts (MoE) models. By tripling on-chip SRAM to 384 MB and increasing high-bandwidth memory (HBM) to 288 GB, it breaks the memory wall, hosting massive KV Caches entirely on silicon. Additionally, it doubles ICI bandwidth to 19.2 Tb/s, reduces the ICI network diameter by over 50%, and introduces a dedicated Collectives Acceleration Engine (CAE), which reduces on-chip latency by up to 5x to minimize lag during high-concurrency requests. With this design, TPU 8i delivers 80% better performance per dollar for inference than the prior generation, enabling fast, interactive user experiences, cost-effectively.

TPU 8t and TPU 8i will be available to Cloud customers soon. To learn more, check out this deep dive on the architecture.

A5X with NVIDIA Vera Rubin platform

We know that one size doesn't fit all. Different customers have different workloads, different requirements, and different use cases. So, we also partner deeply with NVIDIA to deliver the latest GPU platforms as highly reliable and scalable services in Google Cloud. We will be among the first to deliver instances based on the next-generation Vera Rubin platform when it becomes available later this year.

We are also co-engineering the open-source Falcon networking protocol with NVIDIA via the Open Compute Project, pushing the frontiers of reliable transport protocols. A5X will implement a variety of innovative concepts from Falcon.

Thinking Machine Labs, for example, uses our NVIDIA-based infrastructure to power Tinker, an open platform for reinforcement learning and fine-tuning of frontier models for specialized use cases, achieving over 2x faster training and serving with Google’s AI Hypercomputer.

Fueling agentic logic and reinforcement learning with Axion, Intel, and AMD

While GPUs and TPUs are great for training and serving AI models, they need to be complemented with high-performance CPU-based services to handle the complex logic, tool-calls, and feedback loops that surround the core AI model. Our new Axion-powered N4A CPU instances deliver outstanding price-performance for these agent runtimes. In fact, GKE Agent Sandbox with Google Axion N4A offers up to 30% better price-performance than agent workloads on other hyperscalers. This efficiency extends across our entire portfolio, including our 4th generation Compute Engine VM families, powered by the latest x86 instances from Intel and AMD. These are specifically optimized for the broadest range of RL tasks, such as RL reward calculation, agent orchestration, and nested visualization, providing the optimal capabilities for every AI workload.

Virgo Network for data center scale-out fabric

As part of AI Hypercomputer, the Virgo Network is designed to meet the demanding requirements of modern large-scale AI workloads. Its collapsed fabric architecture with 4x the bandwidth of previous generations eliminates the "scaling tax" to deliver staggering peak computing power. This capacity helps the most ambitious AI workloads scale with near-linear efficiency.

With Virgo Network and TPU 8t, we can connect 134,000 TPUs into a single fabric in a single data center, and connect more than one million TPUs across multiple data center sites into a training cluster — essentially transforming globally distributed infrastructure into one seamless supercomputer.

We are also making Virgo Network available for A5X (powered by NVIDIA Vera Rubin NVL72), supporting up to 80,000 GPUs in a single data center, and up to 960,000 GPUs across multiple sites.

Storage: Minimizing data bottlenecks

A massive compute cluster is only as effective as the storage system feeding it data. To ensure storage is not a bottleneck while making compute faster, we are delivering four key storage advancements that let you:

Accelerate training and inference: Google Cloud Managed Lustre now delivers 10 TB/s of bandwidth — a 10x improvement over last year and up to 20x faster than other hyperscalers. We’ve also increased its capacity to 80 petabytes. These advancements are powered by our new C4NX instances and Hyperdisk Exapools.
Minimize latency: Managed Lustre can leverage new TPUDirect and RDMA to allow data to bypass the host, moving directly to the accelerators. By removing this processing overhead, your AI agents can respond with the near-instant speed users need.
Maintain peak utilization for training: Rapid Buckets on Google Cloud Storage transforms object storage with sub-millisecond latency and 20 million operations per second. This helps ensure large-scale training checkpoints and recoveries happen near-instantly, allowing your accelerators to maintain 95% utilization or higher, accelerating training cycles, while also providing cost-effective utilization of valuable TPUs and GPUs.
Build custom solutions: For ISVs and organizations that want to build storage solutions, we are launching the Z4M instance, specifically engineered for customers who want to integrate trusted parallel file systems like Vast Data or Sycomp. Each Z4M instance scales to a massive 168 TiB of local SSD capacity and can be deployed in RDMA clusters of thousands of machines.

These new storage options provide a comprehensive storage portfolio, giving you the raw power of the AI Hypercomputer stack with optimal storage services for each use-case.

GKE: Orchestration for agent-native workloads

In the agentic era, intelligence is only as effective as the speed at which it can be scaled. So, we’ve transformed GKE to serve as the premier orchestration engine for agent-native workloads.

Reducing latency across the stack
To support responsive agentic responses, we optimize every millisecond of the start-up and scale-out process. By streamlining how infrastructure responds to surges in demand, GKE ensures that your agents are ready the moment a user engages with the system. New in GKE are:

Accelerated node and pod startup: GKE nodes now start up to 4x faster, while pod startup times have been slashed by up to 80%.
Rapid model loading: Leveraging the run:AI Model Streamer and Rapid Cache in Google Cloud Storage, models now load 5x faster, removing a traditional storage bottleneck.

Intelligent routing with AI-powered Inference Gateway
Building on last year's introduction of GKE Inference Gateway, we are using "AI for AI" to solve the complexities of serving at scale.

Inference Gateway’s new predictive latency boost replaces heuristic guesswork with machine learning-driven, real-time capacity-aware routing. This intelligent orchestration cuts time-to-first-token (TTFT) latency by more than 70% without manual tuning. For businesses, this translates directly into more natural voice conversations and smooth, real-time interactions across a range of use cases.

Inference Gateway can be deployed alongside llm-d, a Kubernetes-native high-performance distributed LLM inference framework, which was recently accepted as a Cloud Native Computing Foundation (CNCF) Sandbox project. Google Cloud is proud to be a founding contributor to llm-d alongside Red Hat, IBM Research, CoreWeave, and NVIDIA, uniting around a clear, industry-defining vision: any model, any accelerator, any cloud.

Open software ecosystem for the full AI lifecycle

Hardware reaches its full potential through co-designed software. AI Hypercomputer enables engineers to move faster by providing native, optimized support for the industry’s most popular frameworks, including JAX, PyTorch, and vLLM. This open software layer reduces friction between development and deployment, translating to faster time-to-market and better resource efficiency.

We are now in preview with select customers with native PyTorch support for TPU, which we call TorchTPU. With TorchTPU, you can run models on TPUs as they are, with full support for native PyTorch features like Eager Mode. When you combine this with our robust support of vLLM on TPU, our message is clear: we always focus on building for openness and customer choice.

Your foundation for agentic growth

To innovate quickly and cost-effectively in the agentic era, you need a unified system that doesn’t compromise on performance or choice. That is exactly what AI Hypercomputer delivers. By co-designing every layer — from the silicon to the software — we remove the integration burden so your teams can focus on driving your business forward.

AI Hypercomputer also serves as the powerful foundation for Google’s entire ecosystem of high-level services. This integrated stack powers everything from Gemini Enterprise to the Gemini Enterprise Agent Platform, ensuring that all these infrastructure innovations translate directly into business value. By leveraging our fully managed services, such as our serverless training service and our new Managed RL API, you can apply AI Hypercomputer’s massive performance gains to customize Gemini with your own business logic, delivering sophisticated, agent-based solutions.

We’re looking forward to seeing what you build next with this updated and expanded AI platform.

A developer’s guide to architecting reliable GPU infrastructure at scale

Thu, 09 Apr 2026 22:00:00 +0000

May 28, 2026: This blog post outlining Google Cloud’s GPU AI/ML infrastructure reliability strategy was updated with a link to a new article on monitoring GPUs.

As we enter the era of multi-trillion parameter models, computational power has transitioned from a utility to a mission-critical strategic asset. To meet relentless training demand, organizations are no longer just building clusters — they are engineering massive, integrated compute ecosystems comprising hundreds of thousands of high-performance accelerators that are interconnected with an ultra-high-bandwidth networking backplane. At this unprecedented scale, raw performance thrives when it is built upon a foundation of systemic resilience.

In "always-on" mission-critical environments, the statistical probability of hardware variance becomes a primary constraint for reliability. When thousands of GPUs are operating at peak utilization for months at a time, a 0.01% performance fluctuation can trigger a systemic failure. The cost of training interruptions now measured in millions of dollars and weeks of lost progress, the industry's focus has shifted. The true frontier of training isn't just about the size of the cluster — it’s about the resilient system architecture that is able to power the next generation of AI workloads.

The core challenge for the industry goes beyond simple hardware fixes; it requires the creation of holistic software and infrastructure frameworks designed to withstand the inevitable disruptions of massive-scale computing. In an environment where AI/ML infrastructure represents a major capital expenditure on a company's balance sheet, partnering with a cloud provider that places a premium on infrastructure reliability is paramount.

Operational realities of AI at scale

The construction of a supercomputer utilizing hundreds of thousands of advanced GPUs involves significant operational complexity. Maintaining optimal utilization over several months to train a single large language Model (LLM) subjects the hardware to high levels of sustained performance that exceed the design parameters of conventional data center equipment. The advent of rackscale GPU architectures, such as the NVIDIA GB200 NVL72 and NVIDIA GB300 NVL72, has shifted the landscape. Considerations now extend beyond individual machines to encompass entire domains, impacting multiple interconnected trays with the potential to require coordinated management for AI/ML workloads to avoid disruptions.

The business implications of infrastructure instability

For organizations at the forefront of AI innovation, infrastructure reliability poses a significant commercial risk with substantial economic consequences.

High cost of failure: A single failure in a massive training job requires restarting from the last checkpoint, wiping out days or even weeks of progress. When infrastructure spend is a big capex, every failure counts.
Delayed time-to-market: In the fast-moving AI space, being first matters. Every day spent debugging hardware failures is a day delaying releasing new models while competitors are getting ahead. Reliability issues can directly slow down model iteration cycles, delaying product launches and feature updates.
Operational complexities: Manually managing a large GPU cluster is a resource-intensive task. Companies come to the cloud to reduce the cost of managing the infrastructure. Without systemic reliability investments, operations teams can get overwhelmed by a constant stream of alerts, forced to play "whack-a-mole" to identify, isolate, and replace faulty nodes thus affecting their time spent on planning for the future capacity and model demands.
Expensive workarounds to mitigate failure impact: To achieve a certain level of performance and Goodput, companies can end up needing to buy 10-20% more hardware than they actually need as a buffer.

Quantitative assessment: Key reliability metrics

Beyond traditional uptime measurements, the primary metrics Google Cloud uses to measure AI infrastructure health and stability are MTBI and Goodput.

Mean Time Between Interruption (MTBI): The average time a system runs before encountering an interruption. Includes instance terminations as well as every customer workload interruption that our systems can observe (example GPU XIDs).
Goodput: The amount of useful computational work completed per unit time.

Google Cloud’s methodology: Engineering systemic resilience

The objective has shifted from expecting total hardware perfection to engineering systems that demonstrate inherent resilience. We understand that trust in our infrastructure begins with reliability. Our approach is based on four principles:

Proactive prevention: We’ve integrated hardware validation, real-time telemetry, and automated remediation throughout the infrastructure lifecycle. This systemic approach to shift from reactive troubleshooting to proactive management optimizes the reliability of mission-critical GPUs systems at scale.
Continuous monitoring and intelligent detection: We have transformed raw data into actionable insights by synthesizing multi-layered telemetry through automated analysis, to proactively identify and resolve anomalies. This data-driven approach shifts our infrastructure from reactive maintenance to an intelligent, self-healing system that helps ensure continuous workload stability.
Transparency and control: We empower users with full visibility and control over GPU infrastructure health. We provide a comprehensive suite of observability metrics and direct tools, allowing customers to correlate hardware status with their workload Goodput and report faults.
Minimizing disruptions: Our control plane integrates smart scheduling with predictive health signals to enable improved workload migration via maintenance notifications. If unexpected issues arise, customers can enable automated remediations and fast recovery mechanisms to initiate rapid restoration of service.

We have covered an in-depth journey into these principles in our technical deep-dive post linked below. We are launching a comprehensive technical deep dive series to explore Google’s approach towards AI/ML infrastructure reliability for Google Cloud GPUs further. Check back here as we add links to learn about:

Proactive prevention: Inside Google Cloud's multi-layered GPU qualification process
GCE GPU monitoring guide: Visualizing infrastructure observability, operations and actionable insights
Continuous monitoring and intelligent detection: Using ML to predict and prevent GPU downtime (coming soon)
Minimizing disruptions: Smart scheduling and fast recovery systems for mission-critical GPU clusters (coming soon)

New GKE Cloud Storage FUSE Profiles take the guesswork out of configuring AI storage

Wed, 08 Apr 2026 16:30:00 +0000

In the world of AI/ML, data is the fuel that drives training and inference workloads. For Google Kubernetes Engine (GKE) users, Cloud Storage FUSE provides high-performance, scalable access to data stored in Google Cloud Storage. However, we learned from customers that getting the maximum performance out of Cloud Storage FUSE can be complex.

Today, we are excited to introduce GKE Cloud Storage FUSE Profiles, a new feature designed to automate performance tuning and accelerate data access for your AI/ML workloads (training, checkpointing, or inference) with minimal operational overhead. With these profiles, tuned for your specific workload needs, you can enjoy high performance of Cloud Storage FUSE out of the box.

Before (manual tuning)

code_block: <ListValue: [StructValue([('code', 'apiVersion: v1\r\nkind: PersistentVolume\r\nmetadata:\r\n name: serving-bucket-pv\r\nspec:\r\n accessModes:\r\n - ReadWriteMany\r\n capacity:\r\n storage: 64Gi\r\n persistentVolumeReclaimPolicy: Retain\r\n storageClassName: ""\r\n claimRef:\r\n name: serving-bucket-pvc\r\n mountOptions:\r\n - implicit-dirs\r\n - metadata-cache:ttl-secs:-1\r\n - metadata-cache:stat-cache-max-size-mb:-1\r\n - metadata-cache:type-cache-max-size-mb:-1\r\n - file-cache:max-size-mb:-1\r\n - file-cache:cache-file-for-range-read:true\r\n - file-system:kernel-list-cache-ttl-secs:-1\r\n - file-cache:enable-parallel-downloads:true\r\n - read_ahead_kb=1024\r\n csi:\r\n driver: gcsfuse.csi.storage.gke.io\r\n volumeHandle: BUCKET_NAME\r\n volumeAttributes:\r\n skipCSIBucketAccessCheck: "true"\r\n gcsfuseMetadataPrefetchOnMount: "true"\r\n---\r\napiVersion: v1\r\nkind: PersistentVolumeClaim\r\nmetadata:\r\n name: serving-bucket-pvc\r\nspec:\r\n accessModes:\r\n - ReadWriteMany\r\n resources:\r\n requests:\r\n storage: 64Gi\r\n volumeName: serving-bucket-pv\r\n storageClassName: ""\r\n–--\r\napiVersion: v1\r\nkind: Pod\r\nmetadata:\r\n name: gcs-fuse-csi-example-pod\r\n annotations:\r\n gke-gcsfuse/volumes: "true"\r\nspec:\r\n containers:\r\n # Your workload container spec\r\n ...\r\n volumeMounts:\r\n - name: serving-bucket-vol\r\n mountPath: /serving-data\r\n readOnly: true\r\n serviceAccountName: KSA_NAME \r\n volumes:\r\n - name: gke-gcsfuse-cache # gcsfuse file cache backed by RAM Disk\r\n emptyDir:\r\n medium: Memory \r\n - name: serving-bucket-vol\r\n persistentVolumeClaim:\r\n claimName: serving-bucket-pvc'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f51651099a0>)])]>

After (Cloud Storage FUSE mount options, CSI configs, and file cache medium automatically configured!)

code_block: <ListValue: [StructValue([('code', 'apiVersion: v1\r\nkind: PersistentVolume\r\nmetadata:\r\n name: serving-bucket-pv\r\nspec:\r\n accessModes:\r\n - ReadWriteMany\r\n capacity:\r\n storage: 64Gi\r\n persistentVolumeReclaimPolicy: Retain\r\n storageClassName: gcsfusecsi-serving\r\n claimRef:\r\n name: serving-bucket-pvc\r\n csi:\r\n driver: gcsfuse.csi.storage.gke.io\r\n volumeHandle: BUCKET_NAME\r\n---\r\napiVersion: v1\r\nkind: PersistentVolumeClaim\r\nmetadata:\r\n name: serving-bucket-pvc\r\nspec:\r\n accessModes:\r\n - ReadWriteMany\r\n resources:\r\n requests:\r\n storage: 64Gi\r\n volumeName: serving-bucket-pv\r\n storageClassName: gcsfusecsi-serving\r\n–--\r\napiVersion: v1\r\nkind: Pod\r\nmetadata:\r\n name: gcs-fuse-csi-example-pod\r\n annotations:\r\n gke-gcsfuse/volumes: "true"\r\nspec:\r\n containers:\r\n # Your workload container spec\r\n ...\r\n volumeMounts:\r\n - name: serving-bucket-vol\r\n mountPath: /serving-data\r\n readOnly: true\r\n serviceAccountName: KSA_NAME \r\n volumes: \r\n - name: serving-bucket-vol\r\n persistentVolumeClaim:\r\n claimName: serving-bucket-pvc'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f5165109c70>)])]>

The trouble with optimizing Cloud Storage FUSE

Optimizing Cloud Storage FUSE for high-performance workloads is a multi-dimensional problem. Historically, users had to navigate manual configuration guides that could span dozens of pages. And as AI/ML has evolved, Cloud Storage FUSE’s capabilities have also increased, with new mount options available to accelerate your workloads. The "right" settings were never static; they depended heavily on a variety of dynamic factors:

Bucket characteristics: The total size of your dataset and the number of objects significantly impact metadata and file cache requirements.
Infrastructure variability: Optimal configurations change based on whether you are using GPUs, TPUs, or general-purpose compute.
Node resources: Available RAM and Local SSD capacity determine how much data can be cached locally to minimize expensive round-trips to Cloud Storage.
Workload patterns: A training workload (high-throughput reads of large datasets) requires different tuning than a checkpointing workload (bursty, high-throughput writes) or a serving workload (latency-sensitive model loading).

In fact, many customers leave available performance on the table or face reliability issues (e.g., Pod Out-of-Memory kills) due to unoptimized or misconfigured Cloud Storage FUSE settings.

Introducing Cloud Storage FUSE Profiles for GKE

GKE Cloud Storage FUSE Profiles simplify this complexity with pre-defined, dynamically managed StorageClasses tailored for specific AI/ML patterns. Instead of manually adjusting dozens of mount options, you simply select a profile that matches your workload type.

These profiles operate on a layered model. They take the base best practices from Cloud Storage FUSE and add a GKE-specific intelligence layer. When you deploy a Pod using a profile, GKE automatically:

Scans your bucket (or a specific directory) to understand its size and object count.
Analyzes the target node to check for available RAM, Local SSD, and accelerator types.
Calculates optimal cache sizes and selects the best backing medium (RAM or Local SSD) automatically.

We are launching with three primary profiles:

gcsfusecsi-training: Optimized for high-throughput reads to keep GPUs and TPUs fed with data.
gcsfusecsi-serving: Optimized for model loading and inference, with automated Rapid Cache integration.
gcsfusecsi-checkpointing: Optimized for fast, reliable writes of large multi-gigabyte checkpoint files.

Using GKE Cloud Storage FUSE Profiles delivers several benefits:

Simplified tuning: Replace complex, error-prone manual configurations with three simple, purpose-built StorageClasses.
Dynamic, resource-aware optimization: The CSI driver automatically adjusts cache sizes based on real-time environment signals, so that you can maximize performance without risking node stability.
Accelerated read performance: The serving profile automatically triggers Rapid Cache, placing your data closer to your compute for faster cold-start model loading.
Granular performance insights: Gain visibility into automated tuning decisions through structured logs that detail exactly why specific cache sizes and mediums were selected for your Pod.

Using GKE Cloud Storage FUSE Profiles inference profile, we were able to reduce model loading time for a Qwen3-235B-A22B workload on TPUs (480GB) from 39 hours to just 14 minutes, helping customers achieve the maximum benefit of Cloud Storage FUSE GCSFuse out-of-the-box.

How to use Cloud Storage FUSE Profiles on GKE

To get started, ensure your cluster is running GKE version 1.35.1-gke.1616000 or later with the Cloud Storage FUSE CSI driver enabled.

1. Identify the StorageClass

GKE comes pre-installed with the profile-based StorageClasses. You can verify them with:

code_block: <ListValue: [StructValue([('code', 'kubectl get sc -l gke-gcsfuse/profile=true'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f5165109f40>)])]>

2. Create your PV and PVC

When creating your PersistentVolume, point it to your Cloud Storage bucket. GKE automatically initiates a bucket scan to determine the optimal configuration.

code_block: <ListValue: [StructValue([('code', 'apiVersion: v1\r\nkind: PersistentVolume\r\nmetadata:\r\n name: gcs-pv\r\nspec:\r\n accessModes:\r\n - ReadWriteMany\r\n capacity:\r\n storage: 5Gi\r\n persistentVolumeReclaimPolicy: Retain \r\n storageClassName: gcsfusecsi-training\r\n mountOptions:\r\n - only-dir=my-ml-dataset-subdirectory # Optional\r\n csi:\r\n driver: gcsfuse.csi.storage.gke.io\r\n volumeHandle: my-ml-dataset-bucket\r\n---\r\napiVersion: v1\r\nkind: PersistentVolumeClaim\r\nmetadata:\r\n name: gcs-pvc\r\nspec:\r\n accessModes:\r\n - ReadWriteMany\r\n resources:\r\n requests:\r\n storage: 5Gi\r\n storageClassName: gcsfusecsi-training\r\n volumeName: gcs-pv'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f5165109400>)])]>

3. Create your Deployment

Once your Persistent Volume Claim (PVC) is bound, simply consume it in your Deployment as you would any other volume. GKE mounts the volume with the precise settings your hardware and dataset require.

code_block: <ListValue: [StructValue([('code', 'apiVersion: apps/v1\r\nkind: Deployment\r\nmetadata:\r\n name: my-deployment\r\nspec:\r\n replicas: 3\r\n selector:\r\n matchLabels:\r\n app: my-app\r\n template:\r\n metadata:\r\n labels:\r\n app: my-app\r\n annotations:\r\n gke-gcsfuse/volumes: "true"\r\n spec:\r\n serviceAccountName: my-ksa\r\n containers:\r\n - name: my-container\r\n image: busybox\r\n volumeMounts:\r\n - name: my-gcs-volume\r\n mountPath: "/data"\r\n volumes:\r\n - name: my-gcs-volume\r\n persistentVolumeClaim:\r\n claimName: gcs-pvc'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f51651099d0>)])]>

After it's deployed, the CSI driver automatically calculates optimal cache sizes and mount options based on your node's resources, such as GPUs or TPUs, memory, Local SSD, the bucket or sub-directory size, and the sidecar resource limits.

Get started today

GKE Cloud Storage FUSE Profiles remove the guesswork from configuring your cloud storage for high performance. By moving from manual "knob-turning" to automated, workload-aware profiles, you can spend less time debugging storage throughput and more time building the next generation of AI.

Ready to get started? GKE Cloud Storage FUSE Profiles are generally available in version 1.35.1-gke.1616000. Explore the official documentation to configure Cloud Storage FUSE profiles in GKE for your AI/ML workloads!

AI infrastructure efficiency: Ironwood TPUs deliver 3.7x carbon efficiency gains

Mon, 06 Apr 2026 16:00:00 +0000

At Google, we are committed to being transparent about the environmental impact of our AI infrastructure, publishing metrics on the lifetime emissions of our chips — from manufacturing to powering these chips in the data center. Today, we are updating these metrics for our seventh-generation TPU, Ironwood, which demonstrates an approximately 3.7x improvement in Compute Carbon Intensity (CCI) compared to TPU v5p, the previous generation of performance-optimized TPUs.¹

In other words, despite the fact that AI is driving demand for additional compute resources, our ongoing work to optimize AI hardware is helping to improve the energy consumption and emissions of AI workloads.

Measuring AI accelerator efficiency: Compute Carbon Intensity (CCI)

To help manage the environmental impact of AI workloads, we monitor the Compute Carbon Intensity (CCI) of our AI accelerator hardware. CCI is defined in An Introduction to Life-Cycle Emissions of Artificial Intelligence Hardware²as the estimated amount of CO2 equivalent emitted for every utilized floating-point operation (CO2e/FLOP). This metric provides a holistic, chip-level view by including both the embodied emissions associated with manufacturing, transportation, and data center construction (Scope 3), as well as the operational emissions associated with running these chips in data centers (Scope 1 and 2).

The Ironwood advantage: high performance, low footprint

Google’s TPU CCI continues to improve with each chip generation. Drawing from empirical data measured in January 2026, Ironwood demonstrates a remarkable 3.7x improvement in CCI relative to TPU v5p. This accelerates efficiency gains from the 1.2x CCI improvement of TPU v5p relative to TPU v4, and demonstrates continued carbon efficiency optimization of Google’s performance-optimized TPU architecture.

These efficiency gains are driven by outsized compute performance increases between TPU generations relative to growth in machine energy consumption and manufacturing emissions. In fact, fleetwide measurements demonstrate a 5x improvement in utilized FLOPs across generations, from TPU v5p to Ironwood.³ Because the performance denominator in our CCI equation (CO2e/FLOP) is scaling faster than emissions, the net carbon cost per operation drops significantly with every new chip.

^{Figure 1: Ironwood’s accelerating CCI improvement measured on Google’s performance-optimized TPU cohort, considering January 2026 workloads.4}

Operating Google’s TPU fleet more efficiently

Updated TPU CCI metrics also offer a direct comparison to the measurement we published in 2025. Specifically, from October 2024 to January 2026, Google’s versatile TPU cohort ran more efficiently than what we reported previously:

TPU v5e achieved a 43% reduction in total CCI over 15 months, dropping to 228 gCO2e/EFLOP. This was driven by a 72% increase in average utilization.
Trillium, the sixth-generation TPU, saw a 20% reduction in total CCI over the same time period, bringing its emissions intensity down to 125 gCO2e/EFLOP.

^{Figure 2: Google’s versatile TPU cohort demonstrates deployment efficiency gains for the same TPU generations between October 2024 and January 2026.5}

These results demonstrate that Google continues to improve the carbon-efficiency of our AI infrastructure. While the massive scale of AI demand requires a significant and growing amount of power, our innovations allow us to deliver substantially more compute performance for every unit of energy consumed.

Decoupling energy and emissions from performance

To what can we attribute these improvements? Beyond Ironwood’s raw hardware capabilities, these CCI gains are further enabled by deep software and system-level optimizations across our infrastructure:

Software efficiency (MoE): The widespread adoption of sparse architectures, such as Mixture of Experts (MoE), routes computation only to necessary parameters. This drastically reduces the active FLOPs required per inference or training step without sacrificing model capacity or quality.
Lower precision math (FP8): By heavily leveraging 8-bit floating-point (FP8) formats, we effectively double compute throughput and halve memory bandwidth requirements compared to 16-bit formats. This shows that we can maintain output quality while exponentially decreasing the energy cost per mathematical operation.
Workload mix and intelligent scheduling: Advanced fleet orchestration continuously balances the workload mix across our infrastructure. By intelligently scheduling tasks, we ensure high continuous utilization rates, optimize duty cycles, and minimize the carbon penalty of idle power draw.

Scale sustainably with Google Cloud

AI’s trajectory requires infrastructure that can scale exponentially without an equivalent surge in carbon emissions. The 3.7x carbon efficiency improvement from TPU v5p to Ironwood demonstrates that we can achieve greater compute density while minimizing the growth of our energy and environmental footprint through deliberate hardware and software codesign. To learn more and get started with Ironwood, register your interest with this form.

_{1. Following the methodology published in an August 2025 technical report, we quantified the full lifecycle emissions of TPU hardware as a point-in-time snapshot across Google’s generations of TPUs as of January 2026. The functional unit for this study is one AI computer deployed in the data center, which includes one or more accelerator trays (containing TPUs) connected to one host tray (i.e., a computing server). Peripheral components beyond the tray (e.g., rack, shelf, and network equipment) and auxiliary computing and storage resources are excluded from the calculation of embodied and operational emissions. We include the electricity used in data center cooling in operational emissions. To estimate operational emissions from electricity consumption of running workloads, we used a one month sample of observed machine power data from our entire TPU fleet, applying Google’s 2024 average fleetwide carbon intensity. To estimate embodied emissions from manufacturing, transportation, and retirement, we performed a life-cycle assessment of the hardware. Data center construction emissions were estimated based on Google’s disclosed 2024 carbon footprint. These findings do not represent model-level emissions, nor are they a complete quantification of Google’s AI emissions. Based on the TPU location of a specific workload, CCI results of specific workloads may vary.
2. The authors would like to thank and acknowledge the co-authors of this paper for their important contributions to enable these results: Ian Schneider, Hui Xu, Stephan Benecke, Parthasarathy Ranganathan, and Cooper Elsworth.
3. This comparison considers the utilized FLOPS (BF16) between deployed TPU v5p and Ironwood chips in Google’s fleet in January 2026. This trend is consistent with the improvement in peak FLOPS (BF16) between v5p (459 FLOPS) and Ironwood (2,307 FLOPS).
4.The GHG protocol offers two accounting standards for operational emissions. Results presented here consider market-based emissions, which includes the impact of carbon-free energy purchases. Location-based accounting, which excludes carbon-free energy purchases, would raise operational CCI to 793, 712, and 195 gCO2e/EFLOP, respectively. The ratio of CCI improvements would be at a similar level, and Ironwood’s embodied CCI would drop from 23% to 8% of its total CCI.
5. To ensure a fair comparison across varying TPU utilizations, this analysis replicates the propensity score weighting methodology from the August 2025 technical report and compares January 2026 results to the results published in 2025. This statistical technique adjusts for duty cycle variations to balance the comparison of TPUs during a given time period. This empirical methodology results in small variations in calculated CCI between temporal periods, reflecting fluctuations in real-world energy consumption and hardware utilization across the global infrastructure.}

AI infrastructure

Minimize idle accelerators: Native RL job interleaving with co-operative time-slicing in llm-d

llm-d for RL infrastructure efficiency (the bigger picture)

The efficiency problem with RL loops

How co-operative time-slicing (RL job interleaving) helps

Time-slicing: High-level architecture

Current release and future outlook

Get started

Your AI agents are ready. Is your data?

Overcoming a lack of context

Escaping unnecessary manual work

Bringing trust and knowledge to the data

The path forward

Report: 83% of organizations need to upgrade their infrastructure to support agentic AI

IDC: Why the right networking approach is foundational to agentic AI

Networking for operational control, security, and governance at scale

Balancing act: A platform vs. best-of-breed approach

Claude at scale on Google Cloud: Frontier AI, built for enterprise production

Managed infrastructure that frees engineering time

Global reach with consistent latency and built-in failover

Enterprise security and data sovereignty built in

Optimized for cost and performance at scale

From inference to agents

Start building

Google Cloud named Leader in the 2026 Gartner® Magic Quadrant™ for AI Infrastructure

Building on the infrastructure foundation powering Gemini

Get more performance per dollar on AI Hypercomputer

Run AI on a fluid infrastructure at virtually any scale

Take the next steps on your journey with AI Hypercomputer

Scaling Ray Serve LLM on GKE: Performance without losing the developer experience

Scaling inference without the bottlenecks

Benchmarking performance on GKE

Why choose GKE for Ray Serve

Try Ray Serve LLM on GKE

Report: GKE Inference Gateway delivers up to 92% faster AI responses

The secret to low-latency AI: Prefix caching

GKE outperforms alternative managed Kubernetes solutions

Ready to accelerate your gen AI inference workloads?

How we evolved Google’s global and data center networks for the AI era

1. The fabric inside AI Hypercomputer

2. The fabric across AI Hypercomputer

3. A resilient global network for the age of inference

Building the future, together

Agent Sandbox on GKE is now available for everyone, and a first look at Agent Substrate

Secure, low-latency execution at scale

Building the future in the open

Get started today

Benchmark and optimize LLMs on-device with AI Edge Portal

Benchmark LLMs across over 120 different mobile devices

Debug performance easily with Model Explorer

Start benchmarking LLMs on-device today

How Imgix processes 8 billion images daily with G4 VMs powered by NVIDIA Blackwell

The challenge: Instant visuals at scale

Adopting the system that runs Google

Inside the Imgix architecture

Advanced video and image intelligence

The results: 50% faster and up to 6x more throughput

Orchestrating at scale

The future: From experimentation to execution

Get started

Google Cloud and NVIDIA expand AI innovation across industries at GTC 2026

Cloud Storage Rapid: Turbocharged object storage for AI and analytics

Rapid Bucket

Rapid Cache

Choose your rocket ship

Cluster-level reliability for trillion-parameter models on TPUs

Reliability for AI supercomputers

Deep dive: The mathematics of availability at scale

Scale of modern AI hardware

Enhancing ML productivity

Enabling the next generation of AI breakthroughs

Introducing Virgo Network, Google’s scale-out AI data center fabric

Reimagining the data center network

Introducing Virgo Network: Megascale data center fabric

Improving reliability at scale

The foundation of the AI Hypercomputer

Inside the eighth-generation TPU: An architecture deep dive

TPU 8: Specialized by design

TPU 8t: The pre-training powerhouse

TPU 8i: The sampling and serving specialist