AI infrastructure

ZenteiQ scales batch inference 4x with GKE Inference Gateway powered by llm-d

Wed, 21 Oct 2026 16:00:00 +0000

Today we hear from ZenteiQ, an artificial intelligence startup spun out of the Indian Institute of Science (IISc) and selected under the IndiaAI Mission to build sovereign scientific foundation models from the ground up. To pre-train models that understand physical laws, governing equations, and numerical solvers, ZenteiQ designed and built a large-scale batch synthetic data-generation pipeline that runs across multiple model families and feeds verified datasets into training runs. Google Kubernetes Engine (GKE) and GKE Inference Gateway gave ZenteiQ the inference-routing layer within its pipeline to scale aggregate inference throughput 4.07x and grow daily output from 1.47 billion to 5.97 billion generated tokens for every 1,000 accelerators. - Fisayo Feyisetan, Product Manager, Google Cloud

Unlike general-purpose language models trained primarily on web text and code, scientific foundation models must understand complex physical systems, mathematical governing equations, and numerical solvers across domains such as thermodynamics, computational fluid dynamics, and electromagnetics. At ZenteiQ, we are building sovereign scientific foundation models from scratch that interpret simulation outputs, evaluate equation formulations, and work through step-by-step mathematical derivations that obey physical laws.

Pre-training models on large datasets of governing equations takes hundreds of billions of reasoning tokens. Alongside curated scientific literature and simulation datasets, we also rely on a batch synthetic data-generation pipeline that we independently designed and implemented to generate, evaluate, and filter candidate derivations and solver validation problems before feeding them into our MaxText pre-training jobs.

Because we operate on a fixed compute allocation, every hour our accelerators spend generating and verifying training data is an hour taken away from pre-training. When data generation falls behind, multi-host accelerator clusters reserved for pre-training sit idle, waiting on datasets.

When data generation slows down pre-training

To maximize data generation speed, our pipeline separates prompt preparation on worker pods from model serving on vLLM across multi-node accelerator pools. Each batch moves through four stages: expanding scientific seed concepts into prompt batches, generating candidate reasoning traces with generator models, scoring responses using evaluator models, and running guardrail checks with a separate model family before packaging verified datasets for pre-training.

We originally ran these model pools on Kubernetes with standard ClusterIP round-robin load balancing across inference endpoints. As we scaled our pipeline to feed our 9B- and 35B-parameter model schedules, two routing bottlenecks began slowing down dataset creation.

Uneven engine queues: Our upstream worker pods expand seed concepts into prompts of widely varying complexity and response length. Because standard round-robin load balancing had no visibility into vLLM engine queues, it kept sending new requests to serving pods that were already busy processing long reasoning traces, while neighboring nodes had capacity to spare.
Redundant prefix computation: Prompts within a generation batch share roughly 70% to 80% of their instructional context and task specifications. Round-robin routing scattered requests with identical instructions across different serving instances, forcing each pod to recompute the same shared prompt prefix from scratch instead of reusing warm caches in accelerator memory.

Together, uneven queues and redundant prefill work caused our generation throughput to swing by as much as half within a single run, stretching out dataset creation and delaying the start of scheduled pre-training runs. We needed routing that understood live vLLM queue depths and prompt caches, without pulling our researchers off model development to build and maintain a custom inference scheduler.

Building on GKE Inference Gateway powered by llm-d

We already run our data preparation and MaxText pre-training jobs on GKE, which lets our team manage data generation and model training on the same Kubernetes platform. Following a recommendation from Google Cloud to address our routing bottlenecks without changing our underlying scientific methodology or pipeline design, we deployed GKE Inference Gateway, powered by the open-source llm-d distributed inference framework, in front of our vLLM accelerator pools.

In our pipeline, worker pods pull scientific seed documents and domain taxonomies from Cloud Storage and expand them into batches of multi-step prompts. Those batches flow through a single GKE Inference Gateway deployment that manages traffic across all of our model families and serving nodes.

Rather than maintaining isolated routing for each model tier, we route generation requests, evaluation queries, and guardrail checks across their respective model pools through the same gateway. Once candidate responses pass evaluation and guardrail checks, worker pods deduplicate and package the verified datasets for direct consumption by downstream MaxText pre-training jobs on GKE.

GKE Inference Gateway routes batch generation, evaluation, and guardrail requests across multi-node accelerator pools running vLLM before verified datasets are fed into MaxText pre-training.

From round-robin to load- and cache-aware routing

With GKE Inference Gateway and llm-d, routing decisions use live serving signals from our vLLM instances rather than static round-robin rotation. GKE Inference Gateway continuously monitors real-time queue depth and key-value (KV) cache utilization across our vLLM endpoints. When a serving node is busy decoding a long equation derivation, the gateway routes incoming requests to nodes with open capacity, so heavy prompts no longer pile up behind busy workers while neighboring nodes sit idle.

At the same time, the gateway routes requests with matching scientific instructions to the serving node where that prefix’s KV cache is already warm in accelerator memory. Because 70% to 80% of each prompt is shared instructional context while the generated reasoning trajectories vary from request to request, combining prefix-cache locality with live queue balancing lets our pods skip repeated prefill work without overloading any single replica.

Higher throughput, predictable training runs

Adopting GKE Inference Gateway powered by llm-d transformed the speed and predictability of our data generation pipeline. Across our production clusters, intelligent routing delivered:

4x higher fleet throughput: Aggregate throughput jumped from 16,964 to 69,114 tokens per second per 1,000 accelerators — growing daily output from 1.47 billion to nearly 6 billion generated tokens.
3.5x to 5.5x gains across individual model pools: Real-time queue balancing kept serving nodes evenly loaded throughout each run, preventing the mid-run slowdowns caused by long reasoning derivations.
14-percentage-point higher prefix-cache hit rate: Routing prompts with matching scientific instructions to warm caches avoided redundant prefill work before each request branched into its own derivation.

For our research and infrastructure teams, the most important outcome was schedule predictability. Quadrupling daily output meant verified datasets were ready on time, so our multi-host pre-training runs started on schedule instead of waiting on data generation.

What’s next for physical AI at ZenteiQ

As we scale our scientific foundation models from 9B to 35B parameters and beyond, we plan to use the same GKE Inference Gateway and llm-d routing setup for post-training reinforcement learning (RL) and automated numerical solver evaluation loops. Because each turn in those multi-turn rollouts builds on the context of the turns before it, they depend even more heavily on live queue balancing and warm prefix caches.

To try these capabilities on your own workloads, explore the GKE Inference Gateway documentation and check out the open-source llm-d project. You can also read how the Gemini platform team cut latency by 35% with GKE Inference Gateway.

Hardware-backed security at global scale: Confidential G4 with NVIDIA Blackwell is GA

Wed, 21 Oct 2026 11:30:00 +0000

Bringing autonomous agents into production means processing sensitive data and valuable intellectual property in real time. Protecting proprietary model weights, user prompts, and generated outputs requires security that safeguards data while it is actively being computed.

As part of our efforts to enable your enterprise securely, today we are announcing that Confidential Computing on Google Cloud’s G4 instances enabled by NVIDIA RTX PRO 6000 Blackwell Server edition GPUs is now generally available. This release integrates NVIDIA Confidential Computing directly into Google Cloud's AI Hypercomputer, delivering the dynamic performance needed to scale without compromising privacy.

When we announced Confidential G4 in preview earlier this year, we talked about its global footprint and scale. Today, these instances are generally available across 36 zones, using multiple consumption models, on Compute Engine VMs and GKE nodes.

In addition, support for Confidential Space enabling secure multi-party data sharing is rolling out to general availability in the coming weeks.

Protecting sensitive data and models at low latency

Protecting data in use shouldn't mean sacrificing GPU throughput or operational agility. By combining non-exportable hardware keys, verifiable remote attestation, and transparent memory encryption, Confidential G4 helps organizations maximize their investments in agentic and inference workloads. Enterprises can keep costs low while meeting strict regulatory and compliance requirements.

“As organizations deploy AI across increasingly complex cloud, on-premises, and agentic environments, protecting sensitive data is critical. Enterprises must also ensure that AI systems operate securely and consistently with their policies. Google Cloud Confidential G4 VMs powered by NVIDIA RTX PRO 6000 Blackwell GPUs provide a powerful foundation for privacy-preserving AI and AI operations. Confidential Core AI builds on Google Cloud to transform enterprise principles, policies, and operating requirements into real-time safeguards that monitor and govern AI agents, models, data, and tools. Together, these capabilities enable organizations to scale trusted AI on Google Cloud while maintaining control, confidentiality, and accountability.” - Taher Behbehani, Founder and CEO, Confidential Core AI

Additional benefits for enterprise AI workloads include:

End-to-end confidentiality and isolation: Encrypt data and model parameters in hardware-enforced execution environments to shield intellectual property and customer inputs from host system administrators, hypervisors, and access by cloud providers.
Verifiable cryptographic trust: Use hardware-rooted remote attestation to verify code, platform configuration, and environment integrity before releasing sensitive datasets and running workloads.
No code refactoring: Deploy models out of the box using standard Ubuntu OS images, NVIDIA drivers, and NVIDIA CUDA-X libraries without application changes or custom builds.

For Polaris Cloud, this ease of deployment was transformational.

“Polaris builds workspaces for AI agents, and our customers need to know where an agent's work actually ran and that it was cryptographically isolated along the way. Google Cloud Confidential G4 has been a game changer for that. We get hardware-backed provenance from the NVIDIA RTX PRO 6000 Blackwell GPU, which we bind into the receipt customers get back with their results. Virtually nothing else changed: same G4 image, same NVIDIA driver, same container runtime, one flag at launch, same performance. Overnight it upgraded our customer promise from 'we run your agent' to 'every workload you run with us is verifiable.” - Fred Esere, Founder and CEO, Polaris Cloud

Multi-party data collaboration: Run collaborative analytics and joint model training securely using Confidential Space. Multiple parties can combine sensitive datasets without revealing underlying private data to each other.

“Enterprises need to put their most valuable data to work with AI while protecting it during processing. NVIDIA Confidential Computing brings hardware-backed protection to GPU-accelerated workloads, helping safeguard sensitive data and proprietary models while in use. Together with Google Cloud, we’re bringing NVIDIA Blackwell performance and these security capabilities to Confidential G4 VMs so organizations can scale production AI with high performance and verifiable protection.” - Dave Salvator, Director of Accelerated Computing Products, NVIDIA

Testing performance

To understand the operational impact of memory encryption, we analyzed Confidential G4 instances against standard, non-confidential G4 instances on the g4-standard-48 machine shape. Our testing focused on widely adopted open-weight models that customers frequently deploy on single-GPU instances for their own inference pipelines.

Benchmarking across popular models including Llama 3, Llama 3.3, Resnet-50, and SDXL shows that Confidential G4 delivers performance on par with standard G4 instances for AI inference workloads, with less than 3.5% degradation in throughput (measured in tokens or images per second).

Note: Performance results are based on specific testing conditions and configurations. Individual workload performance may vary based on system setup, hardware choice, and software environment.

Our analysis can help you move fast while protecting sensitive data and meeting compliance requirements — a trade-off that real-world deployments are already validating.

”We benchmarked Google Cloud's Confidential G4 against a standard G4 running identical software on our workloads. We found that for a modest 5.2% increase in end-to-end latency, we get a hardware-level assurance that ensures unauthorized users outside our users' attested environment cannot eavesdrop on their conversations. We'd make that trade every time; our users get the convenience of the cloud with the privacy of on-premises.” - Andy Qin, CEO, Vertebrae

Getting started with Confidential G4s

With Confidential G4, improving enterprise security during model execution and agentic workloads doesn't require sacrificing performance, flexibility, or operational simplicity. By providing verifiable trust and robust hardware isolation with near-zero throughput degradation, organizations can securely deploy high-value models and protect critical data assets at cloud scale.

Get started today by following the Confidential VM with GPU documentation.

Google Cloud and NVIDIA expand co-engineering collaboration to scale infrastructure for the agentic era

Wed, 21 Oct 2026 11:30:00 +0000

As enterprises transition to continuous, multi-step agentic workflows, traditional infrastructure is hitting its limits. But raw performance is only part of the equation. To truly scale agentic AI, enterprises also need the flexibility to run the right models for the right tasks, alongside the data and operational sovereignty required to maintain verifiable control over both their sensitive intellectual property and the underlying infrastructure.

Today at NVIDIA GTC Berlin 2026, we are announcing new advancements across the stack to deliver this foundation: minimizing bottlenecks, expanding ecosystem choice, and introducing new sovereign AI and Confidential Computing capabilities to protect your most sensitive workloads with hardware-enforced security.

Architecting for the agentic era

To help you train, serve, and operate models at scale, Google Cloud’s AI Hypercomputer is built on compute, storage, networking, and software that we co-designed as a single system. Today, in collaboration with NVIDIA, we are announcing new updates to this agent-native infrastructure:

Accelerated computing with A5X instances based on NVIDIA Vera Rubin platform: These bare-metal GPU-accelerated instances feature the latest NVIDIA Vera Rubin NVL72 platform to deliver performance at scale. With general availability, coming next month, we are also expanding our infrastructure footprint to support EMEA data residency and sovereignty requirements with availability in new local regions.
Scale-out A5X networking with Virgo Network: Virgo Network supports a non-blocking intra- and inter-building scale-out fabric via mesh topology. This new capability will enable extreme horizontal scaling, allowing clusters to expand to 80,000 Rubin GPUs within a single site and up to 960,000 Rubin GPUs across multi-site deployments.
Storage with Managed Lustre and NVIDIA GPUDirect Storage: Google Cloud Managed Lustre will support NVIDIA GPUDirect Storage on A5X instances, giving training and inference data a direct RDMA path from the parallel file system into GPU memory, bypassing the CPU and system memory.
Long-context memory with KV Caching: A5X instances will also support advanced KV-caching capabilities designed to scale context memory beyond GPU high-bandwidth memory (HBM) for long-context and multi-turn agentic workloads. This high-speed caching layer will offload and reuse context across storage tiers, reducing redundant computation, lowering time-to-first-token, and improving inference goodput per dollar.
Workload orchestration with A5X instances on Google Kubernetes Engine (GKE): GKE brings a suite of features for distributed AI workloads, such as GKE Managed DRANET, which simplifies RDMA management by making network interface cards first-class native Kubernetes resources; and Kueue, which boosts performance of distributed workloads through topology-aware scheduling and resource management. This makes GKE and A5X instances a powerful combination for agentic workload orchestration.

“The true promise of agentic AI isn’t just speed, it’s freeing up human creativity from the friction of modern production pipelines. With Google Cloud’s A5X instances accelerated by NVIDIA Vera Rubin NVL72, our WPP Creative and Production teams can direct intelligent agents that perform frame-accurate quality checks and adapt brand assets seamlessly. Google Cloud gives us the reliability, scale and sovereign security required to meet our clients' most ambitious creative demands.” - Stephan Pretorius, Chief Technology Officer, WPP

Expanding open models and ecosystem innovation

Agentic workflows rarely rely on a single model. They route tasks to the most efficient model for the job. Your infrastructure should expand your options rather than restrict your roadmap. With Google Cloud, you can deploy infrastructure on your choice of software and NVIDIA AI infrastructure across multicloud, hybrid, and highly regulated environments. Today, we announced several updates to give you the freedom to build and adapt quickly:

Open models: We are adding support for models like Gemma, Mistral, NVIDIA Nemotron, and others to our Sovereign Cloud portfolio. This allows you to run and govern open-weight models, while retaining full custody of model weights and runtimes, avoiding lock-in while maintaining high-performance acceleration.
NVIDIA Cosmos 3 and NVIDIA Nemotron 3.5 SuperVL with NVIDIA NeMo now available on Gemini: You can now customize the new NVIDIA Nemotron 3.5 SuperVL model with Managed Training Clusters. Use an MTC-optimized training recipe to jumpstart your Nemotron post-training work with NVIDIA NeMo RL and NVIDIA NeMo Gym on Google Cloud. With NVIDIA Cosmos 3, you can generate virtual worlds where robots and autonomous vehicles can try, fail and improve before real-world deployment, supporting simulation-to-reality workflows.

Securing deployment with Sovereign AI and Confidential Computing

As agents take on more autonomous, mission-critical tasks, the data they process becomes increasingly sensitive. Organizations deploying these advanced workloads require hardware-enforced protection and operational control that don’t sacrifice innovation. Whether safeguarding intellectual property during model training in the cloud or executing mission-critical workloads in fully air-gapped environments, Google Cloud and NVIDIA help provide verifiable security and local custody.

Today, we are announcing new Sovereign AI capabilities for on-premises environments, alongside advancements in Confidential Computing for the public cloud:

Sovereign AI on-premises with Google Distributed Cloud

For mission-critical workloads that must remain or strictly isolated from the internet, Google Distributed Cloud (GDC) brings a managed cloud stack directly to your data center or edge facility. To advance sovereign AI initiatives, we are delivering new compute accelerators, multimodal models, and turnkey search capabilities across GDC air-gapped deployments:

NVIDIA Blackwell Ultra GPUs on GDC: Following our preview announced earlier this year, the latest NVIDIA HGX B300 systems are now generally available (GA) on GDC. This brings industry-leading compute density and energy efficiency on-premises, providing the high-throughput platform foundation that demanding local inference workloads require.
Gemini Flash models on GDC: We are expanding our sovereign model catalog with Gemini 3.5 Flash on GDC, accelerated by NVIDIA HGX B300. Organizations can now execute rapid, high-throughput multimodal reasoning, spanning text, audio, video, and imagery, entirely inside their air-gapped perimeter.
Enterprise search capabilities on GDC: Available in preview, our enterprise search capabilities with Gemini on GDC air-gapped delivers an out-of-the-box conversational interface powered by multimodal retrieval-augmented generation (RAG). These capabilities are powered by Gemini on GDC air-gapped with NVIDIA HGX B300. Now, analysts and knowledge workers can instantly query, summarize, and cross-examine proprietary documents, media, and unstructured archives without custom engineering, keeping all indexing, embeddings, and interactions strictly within your sovereign security boundary.

Beyond air-gapped on-premises deployments, you can also run sovereign AI workloads on NVIDIA AI infrastructure using Google Cloud Data Boundary to enforce controls such as regional data residency on our public cloud, or Google Cloud Dedicated for an isolated cloud run by a local independent partner such as S3NS in France or Thales in Germany.

Confidential Computing in the public cloud

As enterprise workloads scale, safeguarding highly sensitive intellectual property during active training, and user context and model IP during real-time inference is critical to maintaining a secure AI architecture. By combining Google Cloud’s advanced security runtimes with NVIDIA Confidential Computing, organizations can deploy high-performance generative AI models inside verified and trusted execution environments without sacrificing processing speed.

G4 instances with Confidential Compute are GA: We are expanding access to privacy-preserving AI with Confidential G4 instances enabled by NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs. G4 Confidential GPUs, with NVIDIA Confidential Computing provide upgraded security without compromising on performance.
Confidential Computing is available today on A4 instances enabled by NVIDIA HGX B200 and Intel TDX
We are introducing Confidential Bare Metal to Google Cloud, recognizing the value of single-tenant bare-metal performance and isolation for frontier model AI serving. Our innovation leads to A4X Max instances (enabled by NVIDIA GB300 NVL72) using NVIDIA Confidential Bare Metal. By extending hardware-rooted secure boot and attestation with encrypted DDR and NVLink interconnects, bare-metal AI workloads leverage this trusted execution boundary to protect user prompts and models during inference and training.

The Google Cloud advantage

Our co-engineering collaboration with NVIDIA integrates their full-stack accelerated computing platforms directly into AI Hypercomputer. This gives you the agent-native infrastructure you need to train, serve, and operate models at scale. Today's announcements deliver this unified foundation, combining the NVIDIA Vera Rubin platform and high-throughput networking with expanded support for open models.

Together, we also give you verifiable control over your data. With new sovereign AI deployments and confidential computing across NVIDIA infrastructure, you can protect your most sensitive workloads using hardware-enforced security and local data residency.

Because we design hardware and software together as a single system, you can run models across different architectures without rewriting code. Your compute cycles stay focused on the workload, so you ship AI products faster.

To get started with these new capabilities, please contact your Google Cloud sales representative.

TorchTPU is now open source: Run native PyTorch on TPUs

Tue, 20 Oct 2026 11:30:00 +0000

Today at PyTorch Conference 2026, we announced a range of new products and integrations designed to help you run PyTorch effectively on Google Cloud or in your own infrastructure. Notably, TorchTPU, our native PyTorch stack for Google TPUs, is now available as open source on GitHub. With TorchTPU, TPUs join GPUs and CPUs as a native option for PyTorch. You can now train, serve, and deploy models on TPUs using the tools you already rely on, including vLLM, SGLang, llm-d, TorchTitan, Ray, TorchRec, Hugging Face, LiteRT. Ultimately, this provides our customers choice across a broad range of workload-optimized computing options including TPUs, GPUs, and CPUs, with a consistent, open software and operational model across the full portfolio.

Run PyTorch code natively on TPUs

Co-developed with the PyTorch team at Meta, TorchTPU lets you run existing PyTorch workloads on TPUs with minimal code changes, using the familiar APIs. TorchTPU lets you debug your models with eager mode, then transition to production-level performance via torch.compile to optimize TPU utilization and reduce memory overhead. By supporting standard PyTorch APIs, advanced diagnostics, and native distributed primitives out of the box, TorchTPU unlocks high performance and operational efficiency from day one.

"At Meta, advancing the frontier of AI requires immense compute scale and deep hardware diversity. We cannot be constrained by a single hardware architecture. TorchTPU is a massive step forward — allowing us to co-develop hardware flexibly with the core machine learning systems team while supporting major LLM, Ads, and Recommendation systems workloads on key production use cases. It enables us to deploy our PyTorch workloads to Google TPUs, provide direct feedback to the engineering teams, and build the ecosystem flexibility everyone needs."- Jana van Greunen, Director of PyTorch Engineering, Meta

We also partnered with Hugging Face so that popular open-weight models across the Hugging Face Hub run natively on TorchTPU. In Transformers v5, deploying models to TPUs is now as simple as passing device_map="tpu". The Diffusers library runs natively on TPUs with support for eager mode, torch.compile, and tensor parallelism to scale large generative models across multiple accelerators. To maintain consistent reliability, we also built dedicated CI/CD pipelines for continuous nightly testing of the Hugging Face Transformers suite on TPU infrastructure.

“Hugging Face is committed to being an open, neutral platform where AI builders can choose the models, frameworks and hardware that work best for them. Hardware diversity is essential to making open-source AI accessible to everyone. Our collaboration with Google to bring native TorchTPU support to our core libraries, including Transformers, puts that commitment into practice. Together, we’re making it easier for the community to run popular models from the Hub efficiently on Google Cloud TPUs using familiar PyTorch workflows. Giving developers more choice in how and where they build helps the entire open-source ecosystem grow.” - Jeff Boudier, VP Product, Hugging Face

Optimizing inference

As reasoning and agentic models scale, inference cost and latency become the primary bottlenecks in production. We’ve integrated TorchTPU into widely used open-source serving engines and paired them with Kubernetes-native fleet routing across TPUs and GPUs to help you optimize throughput per dollar.

This foundation allows developers to scale workloads across diverse hardware while minimizing the friction of platform-specific tuning. To bring these framework-level enhancements to life in production, we have collaborated with leading open-source communities and industry partners. In the sections below, we explore how these partnerships are driving breakthrough efficiency across high-throughput serving with vLLM, structured agentic execution with SGLang, and large-scale distributed fleet orchestration with llm-d.

vLLM: High-throughput serving and memory efficiency

Developed in partnership with Inferact, the native PyTorch vLLM backend on TorchTPU enables developers to deploy open-weight models directly on TPUs with optimized, production-ready serving features and custom Pallas kernels. This integration supports continuous batching, quantization, and multi-chip parallelism on TPU hardware, while preserving upstream worker models, schedulers, and API servers to improve cost-performance ratios, without requiring changes to existing inference stacks.

"Our mission at Inferact has always been to make high-performance inference accessible, reliable, and cost-effective across the entire AI ecosystem. We are thrilled to partner on the launch of TorchTPU and bring first-class support for vLLM to TPUs. By bridging PyTorch’s optimized TPU capabilities directly into vLLM, developers can immediately run production workloads on TPUs with the high throughput, low latency, and memory efficiency they expect from our stack." - Simon Mo, CEO and co-founder, Inferact

SGLang: Accelerating agentic and structured workloads

Developed in partnership with RadixArk, our native PyTorch backend for SGLang accelerates highly structured and agentic workloads on TPUs. SGLang uses RadixAttention to reuse KV cache across requests, allowing repeated prompt prefixes to skip recomputation entirely. SGLang-TorchTPU preserves upstream worker models, schedulers, and API servers, using a disaggregation transfer backend that also acts as the I/O backend for HiCache to optimize latency and throughput for complex multi-step generation.

"At RadixArk, TorchTPU has supercharged our development velocity as we roll out SGLang support across diverse silicon. It’s given us a painless path to native TPU acceleration, passing massive efficiency gains and flexible compute choices directly down to our customers." - Banghua Zhu, CTO and co-founder, RadixArk

llm-d for distributed LLM inference

To run inference engines across large fleets, we are releasing llm-d 1.0 with Red Hat, IBM, and a community of more than 500 contributors and 25 organizations. As a Cloud Native Computing Foundation (CNCF) Sandbox project, llm-d connects two open-source communities: PyTorch, where vLLM and SGLang serve models, and Kubernetes, where distributed systems run and scale in production. The llm-d router sends each request to the replica that already holds its context, using prefix-cache-aware routing. It also splits compute-heavy prefill from memory-heavy decode. llm-d 1.0 makes that ready for production with multi-tenant flow control, router high availability, stable v1 APIs, and session-aware routing for agents. On Google Kubernetes Engine (GKE), you can have the choice of running on either GPUs or Google TPUs, with a common llm-d endpoint, and choose the hardware that fits each workload.

"At Zenteiq, we use GKE Inference Gateway powered by llm-d to accelerate synthetic data generation for our sovereign Scientific Foundation Models. By leveraging load-aware and prefix-cache-aware routing, we have achieved over 4× higher aggregate inference throughput, increasing daily generated token capacity from 1.47 billion to 5.97 billion tokens per 1,000 TPU v6e accelerators. The open-source foundation of llm-d, integrated with GKE, enables us to efficiently serve multiple model families at scale without developing and maintaining a custom inference scheduler." - Prof. Sashikumaar Ganesan, Founder, Zenteiq.ai

Training at scale

Scaling PyTorch training across thousands of accelerators requires fault-tolerant, compile-aware distributed frameworks. We have integrated TorchTPU into the PyTorch community’s core training, reinforcement learning (RL), and orchestration frameworks.

TorchTitan: Distributed high performance training at scale

TorchTitan is the PyTorch-native reference implementation for scalable distributed training and features a production-grade integration with TorchTPU. By connecting to PyTorch's torch.compile and TorchTitan’s compile friendly SimpleFSDP, specialized Pallas kernels, and graph-level optimizations, developers can build and train models on TPUs with high efficiency at scale.

TorchTitan provides recipes for pre-training large dense models (such as Llama 3 70B/405B, Qwen 3 8B/32B, and MuseGlimmer 30B), Mixture-of-Experts (MoE) such as Qwen 3 235B, and diffusion transformers (such as FLUX.1 12B), and has been validated running with up to 8000 Ironwood TPUs (TPU v7x). Beyond initial pre-training, TorchTitan serves as the scalable trainer inside post-training pipelines like VeRL for Group Relative Policy Optimization (GRPO) and RL alignment.To test large-scale efficiency on demanding open architectures, TorchTPU’s training paths via TorchTitan demonstrated up to 52% Model Flops Utilization (MFU) on Qwen 3 8B, showing near-linear scaling and stability across up to 8,192 Ironwood TPUs.

VeRL: Scalable post-training and reinforcement learning in preview

RL is a foundational technique for frontier labs that are advancing model capabilities, as well as for enterprises customizing models for domain-specific tasks. Yet, orchestrating distributed RL workloads remains one of the most complex challenges in modern AI infrastructure. Bridging that gap, VeRL has quickly become one of the most popular RL frameworks, supporting everything from verifiable math and coding rewards, to multi-turn agentic tool use at massive scale.

To help expand hardware choice for this thriving community, we upstreamed a TPU backend and an end-to-end reference recipe to VeRL. Built on a disaggregated trainer-sampler architecture designed to maximize training efficiency, our GRPO reference recipe on mathematical reasoning datasets demonstrates how developers can deploy VeRL workloads on TPUs while preserving their existing, hardware-agnostic model and algorithm code.

Ray: Orchestrating heterogenous compute

We’ve also added native TorchTPU support to Ray, enabling you to orchestrate PyTorch data processing, training, RL, and serving across CPUs, GPUs, and TPUs within a single workflow. With a few lines of resource configuration, you can schedule each stage of a heterogeneous AI pipeline onto the accelerator best suited for the job.

“At Ray Summit we talked about the PARK stack, PyTorch, AI models, Ray, and Kubernetes, because that's what teams are building production AI on today. TorchTPU embeds Google TPUs directly in that industry-standard stack. We've also worked closely with the Google team to support TPUs as a first-class accelerator in Ray. Coupled with native PyTorch support, teams can now seamlessly develop, scale, and monitor their workloads on TPUs." - Edward Oakes, Chief Architect, Anyscale

Deployment flexibility and operational efficiency

True portability requires standardizing deployment tools across different hardware profiles, including edge devices and cloud accelerators.

LiteRT joins the PyTorch Ecosystem

Many PyTorch models need to run on phones, laptops, browsers, and embedded devices. Today, LiteRT (formerly TensorFlow Lite), Google's high-performance on-device AI inference, joins the PyTorch ecosystem.

With LiteRT-Torch, you convert a PyTorch model with a single litert-torch.convert() call. It builds on torch.export() and Core ATen operators, so your model code stays in PyTorch. For LLMs, LiteRT Torch exports Hugging Face Transformer models to run with LiteRT-LM. The converted models run on Android, iOS, macOS, Windows, Linux, web, and IoT devices, with hardware acceleration across CPUs (via XNNPack), GPUs (via open source ML Drift), and NPUs from major chipsets provided through a single API.

TPU agent suite in early preview

To expand Pytorch adoption on TPUs, we are announcing that the TPU agent suite is in preview on GitHub. Integrating through plug-ins with several coding harness-model combinations, the TPU agent suite helps automate onboarding, diagnostics, user questions, and inference performance tuning for many PyTorch models.

Get started with PyTorch on Google Cloud

Together, these releases reinforce our commitment to making the Google TPU software ecosystem an open, developer-friendly environment for the PyTorch community. By bringing native TPU support upstream into the PyTorch stack across TorchTPU, Hugging Face, vLLM, SGLang, llm-d, TorchTitan, VeRL, and Ray — and contributing LiteRT to the PyTorch Ecosystem — we are giving you the flexibility to build, adapt, and scale AI on your terms.

Visit the Google Cloud documentation to get started running PyTorch on TPUs today.

Announcing llm-d 1.0: Open distributed inference for agents on Google Cloud TPUs and GPUs

Tue, 20 Oct 2026 11:30:00 +0000

Today at the PyTorch Conference, we released llm-d 1.0, an open-source, Kubernetes-native stack for distributed LLM inference. Co-founded by Google with Red Hat, IBM and others, we worked with a community of more than 500 contributors over 25 organizations on this Cloud Native Computing Foundation (CNCF) Sandbox project, bringing together two important communities: PyTorch, where vLLM and SGLang serve models, and Kubernetes, where distributed systems run and scale in production.

llm-d 1.0 arrives at a time when agents are changing the nature of inference. A year ago, most inference traffic was chat: one prompt in, one response out. Today, workloads are shifting to multi-step agents and reinforcement learning (RL) rollouts that fan out across tools, share long prefixes, and arrive in bursts. Serving them on a single machine — or behind a generic round-robin load balancer — breaks down fast. Replicas recompute the same prompts, accelerator memory fills up with duplicate state, and one bursty agent loop can stall interactive users.

llm-d solves this for teams serving their own models by turning a cluster of model servers into a coordinated serving pool. During the PyTorch Conference keynote, we showed this in action with 1,000 sandboxed agents on Agent Substrate on GKE hitting a single llm-d endpoint across vLLM, SGLang, and a mixed fleet of Google Cloud TPUs and GPUs. As traffic scaled across the three stages below, cache-aware routing and tiered KV cache kept each agent's steps on the replicas holding its context, while flow control queued bursty agent loops so interactive requests stayed fast.

Figure 1: Evolution of distributed inference from round-robin load balancing to cache-aware routing, tiered KV cache, and priority flow control in llm-d 1.0

What 1.0 provides

llm-d already delivered prefix- and load-aware routing, KV cache offloading, prefill/decode disaggregation, wide expert parallelism and batch inference across Google Cloud TPUs and GPUs. These capabilities provide:

Multi-tenant quality of service. Priority and fairness-aware routing via llm-d flow control, helping latency-sensitive requests and bursty agent workloads share a fleet to meet SLOs without starving one another.
A stable foundation. v1 APIs and a hardened request path, validated on multiple clouds and accelerator types.
Safer upgrades. Router high availability and zero-downtime updates, alongside DisaggregatedSet, which orchestrates prefill/decode-aware rolling updates without dropping context or disrupting active sessions.
Agent-aware serving. Session-aware routing, turn-priority scheduling, and native Responses API state, helping agents reuse context across turns.
Expanded TPU integration. Native PyTorch model serving with TorchTPU co-operative time-slicing to raise throughput and utilization across Google Cloud TPU fleets.

The components are designed so that you can compose them into your own bespoke production environment. Keep your engine, model and cluster, and adopt only the serving components and configurations you need.

llm-d under the hood

Across the areas below, Google engineers worked with the llm-d community to connect upstream Kubernetes primitives, PyTorch serving engines, and accelerator-aware scheduling into a composable stack.

Figure 2: Multi-node llm-d orchestration and single-node TorchTPU model engines on Google Kubernetes Engine and Google TPUs

One endpoint, many accelerators

An agent fleet is not one workload. Interactive turns need low latency, background loops and reinforcement learning (RL) rollouts need high throughput, and demand swings by the minute. llm-d puts your accelerator pools behind a single endpoint, with one router, one KV cache index, and one flow-control policy in front of the whole fleet. Agents call a single API; the router picks the replica for each step and keeps a session's context on the replicas that already hold it. Adding capacity, mixing accelerator generations, or moving a model to a different pool becomes a platform change, not an agent rewrite. TorchTPU brings Google Cloud TPUs into that fleet, running PyTorch models natively inside upstream vLLM and SGLang without model rewrites. For more, see the llm-d on Google Kubernetes Engine (GKE) TPU guide.

Routing that remembers context

The llm-d router builds on Gateway API Inference, the Kubernetes project for inference-optimized routing that we launched. On GKE, GKE Inference Gateway runs the llm-d router as a managed service, simplifying installation and management and extending routing across regions. The router tracks the replicas holding each cached prefix and weighs load and predicted latency when it picks one. When accelerator memory fills up, rather than dropping it, llm-d uses vLLM’s OffloadingConnector, Mooncake, or SGLang’s HiCache to move KV cache to host memory and then to SSD or Google Cloud Managed Lustre. This setup helps prevent prefix-eviction storms under heavy, multi-turn, and long-context conversational loads across mixed TPU and GPU fleets.

On an agentic coding workload on Ironwood TPU (TPU 7x), prefix-aware routing plus host memory offload cut median time-to-first-token (TTFT) by more than 85% and raised total throughput 2.4x versus a plain Kubernetes Service, as of October 2026 (source).

Disaggregation that upgrades safely

Prefill is compute-heavy and decode is memory-bound, so running both on the same replicas makes each wait on the other. llm-d runs prefill, decode, and multimodal encode as separate pools, with wide expert parallelism spreading large mixture-of-experts models across nodes. In 1.0, DisaggregatedSet — built on LeaderWorkerSet, the Kubernetes API for multi-host inference that we started — manages these pools as one unit. Rolling updates move prefill and decode together on matching versions while each phase scales independently. For more, see the disaggregated serving guide.

Async inference that runs alongside real-time inference

For latency-tolerant traffic, llm-d 1.0 introduces llm-d-async-processor, which pulls requests from queues like Google Cloud Pub/Sub or Redis and routes them alongside real-time traffic. Using queue and pool-dispatch gates, the processor checks available capacity and enforces strict priority lanes and quotas so background jobs run in slack capacity without starving interactive users.

Time-slicing to put idle RL capacity to work

During RL training, accelerators often sit idle 40% to 60% of the time during rollout generation or weight updates. llm-d now supports Google Cloud TPU and GPU time-slicing to interleave async inference into those quiet cycles, raising aggregate accelerator duty cycles from a ~40% baseline to >70% (source).

Benchmarks you can trust

Tuning a distributed inference stack across routers, KV cache tiers, and prefill/decode pools takes repeatable measurement under realistic multi-turn load. Prism is llm-d’s open-source analysis dashboard for exploring published benchmarks across llm-d’s well-lit paths, backed by standardized llm-d-benchmark recipes and inference-perf workloads. Inspect the exact configurations behind the numbers, or upload runs from your own cluster to compare throughput, latency, and cost curves side by side.

Running llm-d 1.0 on Google Cloud

llm-d builds on Kubernetes components you may already run: Gateway API, LeaderWorkerSet, Dynamic Resource Allocation, Prometheus, and OpenTelemetry. Running llm-d on Google Cloud brings these pieces together with managed infrastructure:

GKE Inference Gateway is the llm-d router operating in Gateway Mode, a managed GKE Gateway that handles the L7 proxy and load balancing and calls the llm-d Endpoint Picker on every request, simplifying installation and management so you don't run the proxy yourself. More importantly, GKE Inference Gateway is how you scale llm-d across clusters and regions with global failover and Model Armor safety checks.
Autoscaling on inference signals. KEDA and Horizontal Pod Autoscaler (HPA) scale pools based on queue depth, saturation, and predicted latency, while Kueue enforces per-model quota on shared accelerators.
Mixed accelerator node pools. Google Cloud TPU and GPU pools run behind the same llm-d endpoint, deployed with verified Helm charts.
KV cache offloading across layers. Google Cloud Managed Lustre provides a shared remote tier beneath local TPU and GPU HBM and host CPU RAM so long-context workloads run without exhausting accelerator memory.

In production

This GKE llm-d stack carries production traffic, and it has been tested at fleet scale:

Gemini platform (formerly Vertex AI) serves production traffic through GKE Inference Gateway, doubling its prefix cache hit rate from 35% to 70%, cutting TTFT by more than 35% for coding agents and cutting P95 TTFT by more than 50% under bursty chat traffic (source).
In a published benchmark across a 17,000-node multi-region GKE deployment serving a mixture-of-experts model in the US and Europe, multi-cluster GKE Inference Gateway and the llm-d router scaled throughput 2.9x across three clusters with a 99.9% success rate and less than 1% routing overhead (source).
In a published benchmark on A3 Mega cluster serving a 70B-parameter model, llm-d with multi-connector KV cache offloading to Google Cloud Managed Lustre in addition to Host CPU RAM, delivered an approximately 40% improvement in TTFT and a 30% reduction in end-to-end latency responding to 50k-token prompts compared to CPU RAM offload only, while reducing GPU-hour requirements by nearly 60% (source).

Organizations running llm-d on Google Cloud are putting these same capabilities to work in different ways, for instance prefix-aware routing to raise cache hit rates, open interfaces that fit an existing service mesh, and flow control to hold latency for priority tiers when traffic spikes.

“At Snap, we are integrating llm-d into our production AI infrastructure to facilitate high-performance inference at scale. By employing prefix-cache-aware routing, we have achieved prefix cache hit rates ranging up to 75-80%. We appreciate the open-source nature of llm-d, as it enables seamless integration with our Envoy-based Service Mesh." - Vinay Kola, Senior Manager, Software Engineering, Snap Inc.

"At ZenteiQ, we use GKE Inference Gateway powered by llm-d to accelerate synthetic data generation for our sovereign Scientific Foundation Models. By leveraging load-aware and prefix-cache-aware routing, we have achieved over 4× higher aggregate inference throughput, increasing daily generated token capacity from 1.47 billion to 5.97 billion tokens per 1,000 TPU v6e accelerators. The open-source foundation of llm-d, integrated with GKE, enables us to efficiently serve multiple model families at scale without developing and maintaining a custom inference scheduler.” - Prof. Sashikumaar Ganesan, Founder, Zenteiq.ai

“Serving frontier models across interactive applications and high-volume APIs means handling steep traffic spikes without letting bursty workloads degrade latency for priority customers. With llm-d's prefix-cache-aware routing, flow control and disaggregated serving support we can enforce strict priority bands across our commercial tiers, automatically holding or shedding lower-priority requests when pools saturate so paid traffic meets our latency SLOs. Mistral is a proud supporter of open-source software, and we will continue to contribute back to llm-d and DisaggregatedSet projects.” - Mathis Felardos, Lucas Fernandes-Martins and Nikhil Thomas, Mistral, Inference Engineers and llm-d contributors

Teams serving their own models have been asking for an open stack they can take apart and adapt to their own infrastructure, with the cache-aware routing, traffic control, and upgrade controls required in production. llm-d 1.0 gives them that and a community to build with.

Get started

Check out the full demo of llm-d with TPUs at the Google Cloud booth during the PyTorch Conference.
Deploy llm-d 1.0 on GKE with the verified Helm charts. See the llm-d on GKE documentation.
Start with prefix-aware routing in front of your existing vLLM deployment, then add flow control, KV cache offload (SGlang, vLLM) and disaggregation.
Compare results on Prism and contribute on GitHub.

Day 0 support for Reflection’s Beam on Google Cloud: 5x faster perf with llm-d and NVIDIA Dynamo

Fri, 16 Oct 2026 17:00:00 +0000

Today, Reflection released Beam, its first open-weight model built to give developers, researchers, and enterprises open control and customization for frontier software engineering and long-horizon agentic tasks. Released under a permissive Apache 2.0 license with BF16, FP8, and NVFP4 checkpoints available at launch, Beam sets a new standard for inference efficiency in the 500B-parameter class by solving complex reasoning and coding problems with 3x to 4x higher reasoning compute efficiency than comparable open models.

Alongside today’s release, Google Cloud is announcing day 0 support and ready-to-run deployment recipes for serving Beam on Google Cloud AI Hypercomputer and NVIDIA AI Infrastructure.

To help developers get the full benefit of Beam’s token efficiency in production, we worked with Reflection to optimize performance of the underlying serving stack using llm-d (with vLLM) and NVIDIA Dynamo (with SGLang) on Google Cloud:

"By deploying our Beam model on Google Cloud, we achieved up to 5x higher performance and 8x lower latency to optimize serving of agentic workloads." - Joe Spisak, VP of Product & Head of Open Source, Reflection

With today’s day 0 release on Google Cloud, developers get:

Compounding model and system efficiency: Combining Beam’s 3x to 4x reasoning compute efficiency with system-level optimizations on A4X Max instances (NVIDIA GB300 NVL72) for up to 5x higher throughput and 8x lower latency,* and G4 VMs (NVIDIA RTX PRO 6000) for up to 2x higher throughput and 8x lower latency.*
High-efficiency multi-turn serving: KV-cache-aware routing, prefill/decode disaggregation, speculative decoding and hierarchical KV cache offloading to Host DRAM and Local SSD reduce redundant prompt recomputation across long coding sessions.
Turnkey optimized deployment: Our recommended performance optimized recipes for agentic workloads with llm-d and NVIDIA Dynamo are available in AI-Hypercomputer/gpu-recipes, llm-d repo and Model Garden in Gemini platform with a few clicks.

*Based on customer benchmarking

Scaling inference requires system-level optimization

When evaluating models for agentic coding and tool use, inference efficiency happens at two distinct layers: how efficiently the model reasons and how efficiently the serving system executes multi-turn loops.

Model-level efficiency: Reflection’s Beam

Reflection designed Beam (~501B total parameters with ~23B active parameters per token, and an approximately 1M-token context window) specifically for text-native software engineering and multi-step tool use. It plugs directly into the coding harnesses developers already use, including Reflection’s Mirror CLI, Claude Code, Hermes, OpenCode, and , to resolve GitHub issues, run CLI and DevOps workflows, and complete long-horizon tasks.

Because Beam activates only 23B parameters per token and reaches solutions using 3x to 4x fewer reasoning tokens than other comparable open weight models, it dramatically reduces the inference-time compute required for each reasoning step.

System-level efficiency: AI Hypercomputer optimized serving

Once developers deploy an agent into production, a second set of bottlenecks could appear at the serving layer. In an agentic coding loop, the model is called dozens of times in a single session. With every turn, the coding harness appends new file reads, code diffs, compiler logs, and tool outputs to a shared conversation history that quickly grows to tens or hundreds of thousands of tokens.

Even when the model itself generates 3x to 4x fewer reasoning tokens on each turn, the accumulated context window across turns creates two major infrastructure challenges:

First, prefill compute becomes a bottleneck. Standard Kubernetes load balancers distribute requests using stateless round-robin routing. When a coding harness sends Turn 15 of a session, it often lands on a different replica than Turn 14. That replica is forced to recompute the entire historical prompt from scratch, wasting GPU compute and creating long prefill queues that delay other users.

Second, decode becomes bottlenecked by KV cache capacity. Holding long-context KV caches for dozens of concurrent agent sessions quickly fills up GPU High-Bandwidth Memory (HBM). Once HBM fills up, the serving engine has to evict older KV blocks. As soon as those blocks are evicted, cache hit rates drop and GPU throughput stalls.

To preserve Beam’s model-level efficiency at cluster scale, the serving stack has to be optimized as a distributed system. That is why our recipes pair Beam with llm-d (with vLLM) and NVIDIA Dynamo (with SGLang).

Inside the AI Hypercomputer inference architecture

To run Beam’s BF16, FP8, and NVFP4 checkpoints at peak efficiency on Day 0, we designed a vertically integrated architecture on AI Hypercomputer where each layer handles a specific part of the agentic serving lifecycle.

System optimized architecture for serving agentic traffic

Rather than treating inference pods as stateless endpoints, each layer of the stack is co-designed around the multi-turn agent lifecycle:

Prefix- and load-aware routing (GKE + llm-d EPP / Dynamo router): Tracks cached prefixes and queue depth across worker pods, steering each follow-up turn from a coding harness to the replica holding its warm KV prefix while preventing queue saturation via turn-priority flow control.
Disaggregated compute and speculative decoding (A4X Max & G4 instance): Runs vLLM and SGLang with native Blackwell NVFP4, Expert Parallelism (EP) for Beam’s 23B active experts, Data-Parallel Attention, and DFlash2 speculative decoding. Separating Prefill and Decode (P/D) pools ensures heavy prompt bursts never stall active token generation.
Tiered KV cache (GPU Memory -> DRAM & Local SSD) Active KV blocks stay in high-bandwidth GPU memory (Tier 1), while warm KV blocks evicted during long-horizon sessions overflow into host CPU DRAM and local NVMe SSDs (Tier 2) via vLLM’s OffloadingConnector or SGLang's HiCache, allowing historical context to be fetched in milliseconds instead of recomputed from scratch.

System optimizations for multi-turn agentic workloads

When serving multi-turn coding and tool-use workloads like the AgentX trajectory dataset across rising concurrency levels, each layer in the serving stack contributes to the performance and latency improvements for multi-turn, long context traffic patterns of agentic workloads:

A note on reading these charts. The two configurations are not comparable to each other. NVIDIA Dynamo ran on A4X Max instances with 32 x NVIDIA GB300 NVL72GPUs; llm-d ran on G4 VMs with 8 x NVIDIA RTX PRO 6000 GPUs. Each chart shows the gain over a baseline on the same hardware. More replicas give the router more warm KV caches to hit and more room to split prefill from decode, so the uplift grows with the deployment. Both llm-d and NVIDIA Dynamo scale up and down to match the underlying infrastructure.

NVIDIA Dynamo optimized perf vs baseline deployment of Reflection's Beam

llm-d optimized perf vs baseline deployment of Reflection's Beam

Across both platforms, layering these optimizations shifts the Pareto frontier to deliver higher output throughput and faster per-user interactivity simultaneously:

Baseline deployment (Stateless round-robin routing). In a standard Kubernetes deployment without prefix awareness, consecutive turns from the same coding session land on different replicas, forcing constant prefill recomputation on historical context and causing throughput to plateau as concurrency scales.
KV-cache-aware & flow control routing. Enabling prefix-aware routing in NVIDIA Dynamo or llm-d’s Sticky EPP with turn-priority flow control keeps multi-turn agent trajectories pinned to the replica holding their warm KV cache, immediately cutting Time-to-First-Token (TTFT) latency and freeing up GPU compute, resulting in a significant increase in token throughput.
Host DRAM and Local SSD (LSSD) KV offloading. Offloading evicted KV blocks from GPU memory into host DRAM and local NVMe SSDs prevents cache thrashing when dozens of long-context coding sessions exceed on-device GPU memory capacity.
Disaggregated serving, speculative decoding, and KV pool tuning. Tuning hybrid KV cache memory allocation, enabling speculative decoding, and disaggregating prefill and decode workers isolates token generation from heavy prompt bursts while the draft model multiplies output throughput to keep per-agent token interactivity fast.

Get started and meet us at PyTorchCon and GTC Berlin

This blog explored how you can leverage system level optimizations to scale performance to serve Reflection's Beam family of models for agentic workloads. All of the Kubernetes manifests, router configurations, and benchmarking scripts from this post are open-source and ready to run on your own GKE cluster.

Deploy the optimized stack: Explore the day 0 guides for running Reflection’s Beam with llm-d and NVIDIA Dynamo in the AI-Hypercomputer/gpu-recipes repository on A4X Max and G4, and Model Garden in Gemini platform.
Download the weights: Grab the Apache 2.0 BF16, FP8, and NVFP4 checkpoints on Reflection’s Hugging Face page.
Visit us at the llm-d and PyTorch Conference in San Jose: If you are attending PyTorch Conference North America in San Jose this October, stop by the llm-d community sessions and the Google Cloud booth #. We will be walking through live demos of KV-cache-aware routing, P/D disaggregation, and KV cache tiering for agentic workloads.
Come see us at NVIDIA GTC Berlin: If you are attending NVIDIA GTC Berlin, come connect with us at our booth #3026, or our Google Cloud Hub at w3.hub: just a short 5 minute walk from the venue. Learn more about our sessions.

Architecting autonomy: A design pattern for high-compute, governed agents in biotech

Fri, 16 Oct 2026 16:00:00 +0000

In production systems today, most AI agents operate as lightweight dispatchers. They parse a prompt, make an API call, retrieve a record, and hand back control. Their execution is fast, stateless and cheap. Biotechnology does not work this way.

Folding a protein target, screening a 50,000 compound ligand library, or evaluating structure-activity relationships (SAR) are not quick API lookups. They are sustained, stateful, GPU-bound simulations that run for minutes or even hours across specialized silicon. When you ask a general-purpose large language model (LLM) endpoint to coordinate a complex drug discovery workflow, the standard paradigm breaks: Timeouts occur, context windows overflow with atomic coordinates, and proprietary chemical Intellectual Property (IP) risks exposure to external networks.

Building enterprise grade research and development (R&D) agents requires moving past the monolithic assistant model. In this blog post, we will discuss how to use a three-layer architecture: separating cognitive orchestration (the brain) from computational execution (the muscle), and anchoring both inside an enterprise governance plane to build scalable agents for drug discovery use cases in biotech.

A three-layer pattern

The reference architecture below shows how to architect, optimize, and govern high-compute autonomous agents in biotechnology, specifically Drug Discovery.

Figure 1 — The three-layer pattern: a cognitive orchestrator (brain), a scalable GPU execution layer (muscle), and an enterprise governance plane.

The execution layer: scaling the muscle - high-compute execution

The execution layer is responsible for the heavy scientific computations. The first architectural decision is determining where workloads should run:

Strategic, proprietary pipelines require dedicated GPU infrastructure within your security perimeter
Routine or standardized workloads can safely be delegated to managed cloud endpoints

In this architecture, protein folding is driven by ESMFold and generative design by GenMol. Both run as NVIDIA Inference Microservices or NIMs deployed on Google Kubernetes Engine (GKE) using G4 VMs backed by NVIDIA RTX PRO 6000 Blackwell GPUs.

A critical hurdle in containerized GPU workloads is the cold-start penalty, the latency delay required to pull multi-gigabyte model weights from an image registry and stage them into GPU memory (VRAM).

To ensure low latency responsiveness, we build the following as a recommendation into the reference architecture:

Maintain warm pools: Pin the GKE node pool to a minimum of one active node.
Preload weights: Model weights are loaded directly into VRAM at pod initialization, ensuring containers are immediately ready for inference.
Resilient scheduling: GKE manages pod lifecycles and restarts failed instances without tearing down the underlying VM infrastructure.

This approach trades custom, manual CUDA kernel tuning for NVIDIA's pre-compiled, containerized runtimes, dramatically reducing infrastructure engineering overhead while ensuring deterministic throughput.

In biotech, chemical structures represent millions of dollars in proprietary IP. A docking request transmits SMILE strings or 3D coordinate matrices that must never leave company custody.

Containerized NIMs eliminate the friction between rapid prototyping and rigorous data isolation. During early exploration, researchers can query NVIDIA hosted endpoints. When transitioning to proprietary lead candidates, the exact same container is deployed to GKE inside a private VPC. The agent’s orchestration logic remains identical; only the target URI updates from an external proxy to an internal VPC endpoint. No structural data leaves the enterprise perimeter.

The cognitive layer: Multi-agent reasoning

With compute execution isolated, the cognitive layer manages agentic workflow sequencing. Built on Agent Runtime, within the Gemini platform, this reference architecture shows a system that deploys a hierarchical multi-agent structure, one lead orchestrator, and two specialized domain sub-agents (Lead Optimization and Safety Screening) . The multi-agent design can be modified to suit the use case at hand.

Based on this design, when an end-user submits a high-level task such as “ Analyze Candidate 4b against Target X and produce a SAR report”, the system applies four core orchestration patterns discussed below;

In-process native agent handoffs: The lead orchestrator delegates tasks to specialist sub-agents using Agent Runtime’s native SDK rather than network-serialized agent-to-agent HTTP protocols. This eliminates intermediate network hops and keeps internal handoffs instantaneous. Only the outermost boundary communicates outward to user-facing surfaces like the Gemini agent.
Passing bulky payloads by reference: LLMs degrade quickly when forced to ingest massive arrays of 3D spatial coordinates. In this architecture, raw structural data (PDB files, docking poses, trajectory logs) is written directly to Cloud Storage. Agents pass lightweight, signed object URIs and high-level structural summaries, keeping context windows compact and inference costs down.
Decoupling workflow logic from biochemical reasoning: General-purpose LLMs excel at task planning and synthesis, but are prone to hallucinating biochemical nuances. To control this behavior, this architecture maintains strong separation of concerns, namely, workflow and tasks decomposition are handled by Gemini. Domain chemistry and pharmacological reasoning are handled by specialized Nemotron models hosted on dedicated GPU nodes.
Context-driven evaluation over rigid cutoffs: Traditional computational screening enforces arbitrary numerical cutoffs (e.g., discard if binding affinity ,-7.5 kcal/mol) However, therapeutic context fundamentally alters risk tolerance. Preventive cardiovascular therapy taken daily by healthy patients requires extremely strict safety margins, or a novel oncology candidate targeting an intractable cancer may accept higher toxicity if efficacy is demonstrated.

By passing disease context directly into the specialized reasoning models, drug candidates are evaluated against dynamic, target-specific profiles rather than static thresholds, preventing the premature culling of viable drug leads.

Performance best practices for high-compute pipelines

Optimizing an agentic system that interfaces with heavy GPU compute requires challenging common engineering instincts:

Common Assumption	Architectural Reality	Recommended Best Practice
"Swap in a faster driver model to cut latency."	The primary orchestrator's token generation is rarely the bottleneck in multi-step R&D pipelines.	Optimize I/O, not token speed: Focus on reducing payload sizes and intermediate network hops rather than compromising orchestrator reasoning depth.
"Parallelize tool calls across workers."	External scientific services and containerized models often serialize requests internally on dedicated hardware.	Respect execution topology: Align client-side concurrency with backend queue mechanics to prevent pipeline lockouts and connection timeouts.
"Pass full simulation states between tools."	Large JSON bodies cause schema validation failures, prompt truncation, and token bloat.	Pass heavy payloads by reference: Store raw coordinates in object storage and pass lightweight URIs between agent boundaries.
"Return outputs only when the pipeline finishes."	Long multi-minute runs leave scientists blind to whether jobs are proceeding or stalled.	Stream intermediate stage completions: Emit progress updates at each completed milestone (e.g., folding complete, docking running) for real-time visibility.
"Chain multiple granular micro-tools."	Deep tool-chains increase the probability of hallucinated arguments and schema mismatches.	Collapse compute pipelines: Wrap end-to-end simulation pipelines behind a single, clean tool interface that takes clear parameters and returns a consolidated object.

The enterprise governance and asymmetric autonomy

Deploying high-compute agents in enterprise environments requires robust data sovereignty, audit trails, and process control. The Gemini platform establishes these safeguards natively with the following components:

Agent Identity: Every autonomous agent operates with a cryptographically verifiable identity. Thus, for a biotech use case, when a candidate molecule is advanced or rejected, the event is permanently logged to that specific agent principal.
Agent Gateway: This serves the single ingress/egress point for every interaction within the perimeter. Centralized IAM and regional residency rules help ensure sensitive molecular data remains within designated cloud regions.
Agent Registry: This feature maintains a curated catalog of approved agents, validated tool schemas, and authorized Model Context Protocol (MCP) servers, preventing rogue tools from executing in production.
Model Armor: This functionality inspects incoming prompts and outgoing payloads in real time to prevent prompt injection and data exfiltration.

The payoff is clear. Identity, RBAC, auditing, residency, and data protection are native properties of the Gemini platform; this allows enterprise customers to focus on science rather than having to build a compliance stack for every project.

In high-stakes research and development, full autonomy introduces unacceptable liability, while requiring approval for trivial steps can create frictions. The solution addressing this to use asymmetric gating in the following ways;

Automate reversible operations: For example, allow the agent to resolve chemical naming synonyms, pull structural PDBs, and run preliminary Lipinski screens autonomously without interrupting the researcher.

Gate irreversible operations: For instance, initiating multi-GPU docking run, advancing a lead compound to wet-lab synthesis, or archiving a therapeutic program must require explicit human sign-off.

The pattern, and the path forward

High-compute autonomous agents succeed when the boundaries between cognitive planning, numerical computation, and system governance are clean in the following ways:

Keep the orchestrator focused on decomposition and synthesis, leaving intensive biophysical simulation to specialized, containerized microservices and models.
Eliminate context bloat by passing complex chemical data by reference through cloud storage.
Anchor the workflow in enterprise governance, pairing cryptographic agent identity with asymmetric human oversight.

As generative and structural biology models expand toward whole-cell modeling and complex dynamic simulations, this decoupled architecture ensures research systems scale efficiently while keeping proprietary IP secure and auditable.

Reach out to your Google sales representative if you have any requirements to build a use case like this and we will be happy to talk and support you in your agent development journey on Google Cloud.

Temporal Serverless Workers on Cloud Run: Scale workers ahead of demand

Thu, 15 Oct 2026 16:00:00 +0000

The open-source Temporal execution platform is rapidly becoming a foundational layer of modern software infrastructure — especially for orchestration in distributed systems and AI agent workloads. However, deploying and scaling workers in Temporal comes with unique challenges. Workers poll for tasks instead of serving requests, so request-driven serverless platforms can be an awkward fit. This often results in running worker fleets on Kubernetes or VMs, which you have to manage and pay for continuously. You build a metrics pipeline and tune autoscaling triggers, and scaling still trails bursts because of metrics lag. Every deployment means juggling upgrades and versions, and you pay for that compute whether or not there's work in the queue.

We built Cloud Run worker pools for exactly this kind of pull-based work: Your instances connect to a queue or stream, take the next item, and process it. Starting today, Temporal Cloud, which is the fully managed version of Temporal’s open-source platform, can scale them for you. Currently in preview, Temporal Serverless Workers on Cloud Run let you deploy your workers as a worker pool, and Temporal sizes the pool to match the work on your task queues. When there's no work, it can scale the pool to zero. There's no cluster or autoscaler to run, and your worker code stays the same.

How Cloud Run worker pools fit with Temporal

A worker pool is Cloud Run's resource for continuous background work. It has no public endpoint, and each instance is allocated CPU for its whole lifetime, which matches how a Temporal worker runs. Temporal manages your Cloud Run worker pools as follows:

Your Temporal workers run as Cloud Run worker pool instances in your project and poll Temporal Cloud for tasks.
Temporal Cloud tracks your task queues and sets the worker pool's instance count through the Cloud Run Admin API, using the IAM permissions you grant. Temporal never opens a connection to your instances, so you don't need to open ports or add firewall rules for it.
You run each build version as its own Cloud Run worker pool. An older build's worker pool can sit at zero instances until a workflow that started on it needs it again.

Your Temporal workers run in your project, under your service account, and on your network. Temporal decides how many of them run.

What you can build with Temporal Serverless Workers

The combination of Cloud Run worker pools with Temporal Serverless Workers opens up a variety of interesting use cases:

Autoscaled agentic workers: AI agents run multi-step chains of LLM reasoning and tool calls. Temporal's durable execution and Cloud Run worker pools work together here. Temporal workers spin up instantly to handle bursts of LLM steps, embeddings or API calls, then scale to zero when the queue empties. Workers get keyless access to Google Cloud AI APIs. You can also attach a GPU for inference, and the workers attached to the GPUs will still scale to zero. Temporal runs each model and tool call as a durable activity; if a worker disappears mid-step, the activity retries on another worker and the workflow continues.

Long-running Workflows: Temporal Workflows can run indefinitely, allowing for building applications that track the state of real-world resources over long periods of time. This pattern is typically known as the “Entity Pattern,” because the Workflow represents the state of some entity. However, these long-running Workflows don’t usually have consistent compute requirements: Some individual steps are short, some are long, and other times, the Workflow is sleeping, waiting for external input, such as from a human or another system. Cloud Run worker pools provide flexible compute to support this pattern: Individual steps can run for as long as is required, and Temporal will automatically scale down the worker pool when there is no work to be done.

Bursty batch work: Worker pools bill only while instances run, so a worker pool that sits idle between batch runs costs nothing while it's at zero instances. For jobs like billing runs or media transcoding, Temporal scales the pool out as batches arrive and back in once the queue drains. Your maximum instance count caps how many instances you pay for. Tasks beyond that capacity wait in the queue rather than being dropped.

Why Cloud Run worker pools for Temporal workers

Before, running Temporal workers on Cloud Run services took some workarounds:

an HTTP listener, because services expect containers to listen on a port
instance-based billing, so workers kept their CPU between requests
minimum instances, so the service didn't scale to zero

By running Temporal workers on Cloud Run worker pools, you don’t need those workarounds anymore, and you have:

No endpoint to manage. There's no public URL or load balancer to lock down, and no health-check port to implement.
Steady CPU at a lower price. Instances don't idle or throttle. Cloud Run doesn't remove instances because they look idle, so Temporal alone decides when the pool shrinks. Worker pools list at almost 40% less per vCPU-hour and GB-hour than instance-billed Cloud Run services.
Scaling without revision churn. Changing the instance count doesn't create a new revision, so Temporal can resize the pool as often as the load changes.

How Temporal manages scaling and versioning of worker pools

Temporal decides when to add capacity using one main signal: the delay between a task landing in the task queue backlog and a worker picking it up. If that delay grows, the available workers are saturated. Scaling occurs as follows:

Server-side detection: The Temporal server tracks this backlog-to-worker latency for each task queue.
Push notification: When the latency crosses its trigger conditions, the server sends a push notification to the worker controller instance.
Provisioning: The worker controller then adds capacity to the worker pool. New instances take time to start and connect before they can poll for tasks, so capacity lags behind the signal.

Because Temporal keeps spare capacity, the pool grows ahead of demand instead of after tasks pile up. Scale-in is more conservative. Temporal waits for a stabilization period after the last time a task found no available worker (90 seconds by default), and it pauses between successive reductions. Cloud Run’s configurable graceful shutdown extends shutdown from 10 seconds to 10 minutes. That gives SIGTERM-aware workers time to finish or checkpoint. The instance count stays between a minimum and a maximum that you set. If the first task after an idle period can't wait for an instance to start, set the minimum above zero.

Deploying new workflow code without breaking running executions is hard. Temporal workflows are stateful and deterministic, so if you swap their code mid-run, you get non-determinism errors. Serverless Workers on Cloud Run solve this by requiring Temporal Worker Versioning. You don't patch code with conditional branches. Instead, you deploy each Worker Deployment Version as its own worker pool deployment, identified by a deploymentName and a buildId. Using distinct worker pools solves two problems:

Workflows that started on a particular Worker Deployment Version continue to run and complete on that version.
Each Worker Deployment Version is scaled and drained individually, matching compute spend and utilization to the requirements of your application. As existing workflows complete and drain on a previous Worker Deployment Version, the underlying Worker Pool is eventually scaled down to zero automatically.

You set a versioning behavior for each Workflow type, or set a Worker-level default:

PINNED: A workflow stays on the Worker version (Build ID) it started on and finishes there. Best for short-running workflows, order processing and multi-step APIs.
AUTO_UPGRADE: A workflow moves to the new "Current" or "Target" Worker Deployment Version as you roll it out. Best for long-running workflows, such as subscription billing or tasks that run for weeks.

Learn more about configuring worker versioning here.

Get started

Setup takes three steps:

Containerize your Temporal Worker and deploy the container as a Cloud Run worker pool, with the Build ID in the pool name. Keep credentials such as your Temporal API key in Secret Manager.
Connect Temporal to your worker pool by granting Temporal Cloud permission to manage the worker pool so it can scale it.
Create a Worker Deployment Version in Temporal that points at the pool's project, region, and name.

Temporal Serverless Workers on Cloud Run is in preview, so APIs and configuration may change before the stable release. To get started,

Learn how Serverless Workers on Cloud Run scale and handle versions.
Deploy your first worker with the setup guide.
See Cloud Run worker pools for CPU, memory, GPU and networking options.
Sign up for the configurable graceful shutdown preview.

AI21 achieves an 83% reduction in time-to-start for AI workloads with AI Hypercomputer

Fri, 02 Oct 2026 19:00:00 +0000

Editor’s note: AI21 Labs is a leading global AI lab with a long track record of building foundation models, most notably the Jamba family, and today focuses on specialized LLMs and agent optimization technology. By adopting Google Cloud AI Hypercomputer, AI21 cut high-priority job wait times from 72 hours to 12 and manual scheduling interventions from 20 per week to zero.

At AI21, we build foundation models and agent optimization products that help enterprises run agents at frontier quality, efficiently. Our language models, including the Jamba family, and our agent optimization product suite run demanding production workloads, including our own. We chose Google Cloud AI Hypercomputer to support them at scale.

To keep our model training runs highly utilized, we needed a performant, scalable environment codesigned across infrastructure, orchestration, and consumption models. Our model training runs on one of our shared Google Kubernetes Engine (GKE) clusters, pooling thousands of Google Cloud A3 (powered by NVIDIA H100 Tensor Core GPUs) and A3 Ultra (powered by NVIDIA H200 Tensor Core GPUs) instances, so any team can draw on the full capacity of the fleet rather than being boxed into its own slice. The cluster also trains models and agent-optimization workloads beyond the Jamba family. That approach keeps utilization high, and it makes scheduling hard.

The gridlock of high-utilization clusters

Prior to leveraging GKE for orchestration, we used to negotiate capacity by hand in Slack. If you needed capacity for a training run, you posted in #gpu-resources and hoped for the best.

That worked fine when the cluster had headroom. It stopped working once utilization pinned near 100%, which is where you want a reserved compute fleet to sit.

Over time, every request became a negotiation. Team leads spent their time refereeing compute disputes. Our high-priority jobs — the large, multi-node training runs that need half or more of the cluster at once and serve as the critical path for model projects — could sit blocked for up to 72 hours waiting for enough contiguous capacity to open up.

Figure 1: A typical day on #gpu-resources — manual requests, ad-hoc coordination, and researchers waiting on replies.

Scarcity created two distinct problems, and it took us a while to see them as separate. The first was contention: determining who gets compute access next, which we resolved through negotiation. The second was fragmentation: capacity that was technically free but scattered in pieces too small for a large job to use, a bin-packing problem no amount of negotiation could fix.

Figure 2: GPU fragmentation. 8 free GPUs scattered as 1+1+4+2 across four nodes can’t fit an 8-GPU workload.

Sometimes we had plenty of capacity free on paper, but it was scattered across different machines in chunks too small for a larger job to actually land. Without all-or-nothing admission, the cluster could reach a deadlock, with machines holding resources without doing useful work until someone stepped in manually.

It was clear the status quo wasn’t working and we needed something better.

Choosing the right scheduler

We looked at a few open-source batch schedulers, including Apache YuniKorn, Volcano, and Kueue. YuniKorn didn’t cover all our use cases. And while Volcano had more features, integrating it with our environment would have required replacing core Kubernetes scheduler components.

Kueue won on simplicity and integration. It worked with standard Kubernetes, didn’t require replacing core components, and didn’t force us to rewrite our job specs.

Pairing Kueue with AI Hypercomputer’s flexible, open operations through GKE also contributed to our success. We rely on GKE because it gives us the right level of control for compute-intensive AI work — close access to GPU hardware and drivers, without the overhead of managing raw instances ourselves. Kueue’s native integration with GKE, including with Google Cloud capacity types like Spot VMs and Dynamic Workload Scheduler, meant that once our reserved capacity filled up, the same scheduling logic could reach out to elastic capacity automatically instead of leaving jobs stuck.

Closing the loop with open source

Adopting Kueue turned into something bigger than a simple process change. Because we partner with Google Cloud, we have a direct line to a Technical Account Manager, who saw an opportunity to make AI21 a design partner for the Kueue team. This way, we wouldn’t just be a user, but a source of real production requirements that could help shape where the tool went next.

One of the first things to come out of that partnership was a requirements document we shared with the Kueue team, describing behavior we needed that didn’t exist yet: fair admission ordering across teams for multi-node jobs, without the preemption that usually comes bundled with fairness. The Kueue team built it. Admission Fair Sharing (AFS) reorders the admission queue to favor teams that have historically used less capacity without disrupting jobs already running. Around the same time, we also turned on Topology Aware Scheduling. This existing feature made Kueue aware of our physical cluster layout, so it can refuse to admit jobs that won’t fit on a single node rather than placing them with nowhere to run.

For us, that’s what the best-case open-source feedback loop looks like: Real requirements surfaced through production use, fed directly back into Kueue.

Same fleet, less friction

The Slack #gpu-resources channel is archived now. Every workload, whether a debug pod, a multi-node training run, or an inference deployment, gets queued, prioritized, and scheduled automatically. The results were immediate. Manual interventions dropped from about 20 a week to zero. High-priority jobs that used to wait up to 72 hours now wait 12. Fragmentation across the cluster fell from 15% to 8%, and the “zombie job” problem — workloads partially admitted with nowhere to run — is gone.

None of this changed our total cost. We run our reserved fleet at close to 100% utilization on purpose, so raw spend was never the variable we were optimizing. What changed is where our people’s time goes. Team leads aren’t refereeing compute disputes anymore, and researchers aren’t waiting on replies in Slack. That frees everyone to run more experiments and iterate faster.

What’s new in AI infrastructure and orchestration in September

Wed, 30 Sep 2026 16:00:00 +0000

We hereby declare September to be scalability month! As the world prepares for a surge of agentic fleets, we are shoring up our AI infrastructure and orchestration offerings to gracefully — and quickly — respond to that demand, all while maintaining workload isolation and security, and keeping costs in check. Read on to learn how these enhancements manifest across Google Cloud’s compute, network, storage, and orchestration offerings, plus new ways customers are using Google Cloud AI infrastructure, and third-party industry validation of our strategy.

Product, technology, and tools updates

Google Kubernetes Engine updates: The GKE team is all about improving the scalability of the platform, and in September, those improvements came in many shapes and sizes:

New feature: Need an execution runtime with higher density for your agentic workloads? We engineered the new open-source GKE Agent Substrate to run millions of sandboxes with 10x higher density than standard container runtimes. Agent Substrate also delivers sub-500ms resume operations at over 500 suspend/resume activations per second with a native zero-trust kernel and network isolation.
Product update: GKE now has scale-to-zero capabilities built-in. No need to configure complex components to scale your workloads down, thanks to the HPA with the Autoscaling Metric and support for KEP-2021, which do the job for you, out of the box. Read the blog to learn more.
Product update: Further, the GKE HPA (with the above-mentioned Autoscaling Metric) now lets you scale up and down based on custom PromQL metrics, in addition to standard metrics, allowing you to trigger workloads according to conditions that are meaningful and unique to your business. Read more here.
New feature: Yet another scalability feature is GKE Pod snapshots, which lets you save the running state of your workload, including CPU and GPU memory, and restore it on demand. According to internal tests, GKE Pod snapshots can reduce AI inference start-up by as much as 89%. Learn more here.
New migration tool: Finally, if you’ve always wanted to migrate your container workloads from AWS EKS to GKE but feared a daunting, high-friction engineering endeavor, we’ve just launched GKE agentic migration, a purpose-built agent plugin that replaces brittle, ad-hoc prompting with an AI-assisted migration pipeline protected by deterministic guardrails. Designed as a compilation of agent skills and a local Model Context Protocol (MCP) server, it uses AI to translate complex AWS EKS IaC and Kubernetes manifests directly into GKE landing zones. Get started with the onboarding guide.
Feature updates: Reinforcement learning (RL) and evaluation workloads are a beast: In a standard agentic RL loop, an LLM policy generates actions like code snippets on GPUs and executes them inside isolated CPU sandboxes to observe a reward signal. However, when scaling up this loop to support tens of thousands of parallel rollouts, infrastructure bottlenecks emerge, for instance idle accelerators, image cardinality, and a saturated control plane. To help, we developed GKE Agent Sandbox optimized for RL, plus an Agent Sandbox RL orchestration SDK and native integrations for popular RL gyms and harnesses. All are now generally available, and you can learn more here.

Storage updates: AI trains and creates lots of data, and that data has to live somewhere — in block storage systems, file systems, object stores and databases. We announced enhancements to our storage portfolio to help this critical layer roll with the agentic punches:

Product update: Filestore agent volumes offer high-performance, elastic, persistent file storage for agentic workloads. Thanks to its tight integration with GKE Agent Substrate and GKE Agent Sandbox, Filestore agent volumes automatically allocates and attaches a dedicated, isolated file workspace to GKE agent sandboxes in milliseconds. Request access to the preview here.
New product: If you run generative AI and RAG data layers — think Milvus, Pinecone, Qdrant, Vespa, Redis, and in-memory context caching — you may want to take a look at the M4N family of VMs, now GA, which offers the highest per-core IOPS and throughput of leading hyperscalers. Paired with Google Cloud's custom Titanium offload architecture and paired with Hyperdisk Extreme, M4N instances deliver up to 25,000 MiB/s (25 GiB/s) of aggregate host storage performance and up to 1 million IOPS.
New product: Another new Compute Engine product, Z4D, is GA, and a strong storage solution for AI/ML training and inference workloads. When configured as a bare metal instance, Z4D provides both the high local SSD (LSSD) capacity and low latency required by agentic microVMs, so you can run thousands of isolated sandboxes per host with native performance and efficiency. Learn about Z4D machines here.
Product update: Today’s AI training and inference pipelines create data faster than most storage management systems can keep up, creating challenges for teams trying to understand their storage estates. A new version of Storage Intelligence advisor makes it easier to answer the question: "What’s in my buckets?" and quickly identify unexpected changes. Then, enhanced batch operations let you automate bulk changes across your buckets — say, move storage classes, mass-delete stale or temporary data, or apply metadata, tagging, retention, or encryption changes. Learn about the latest in Storage Intelligence advisor here.
New product: Last but not least, a new version of AlloyDB for PostgreSQL brings together pioneering Google infrastructure — Colossus distributed file system, and Jupiter network — to power a new, no-compromises database architecture for the agentic era.

Practitioner guides and how-tos

How-to: Wish you could make GPUs and TPUs scattered around the globe behave as a single pool behind a single entry point? In this blog, we show you how to do just that. At the edge, the multi-cluster GKE Inference Gateway focuses on global, multi-region traffic distribution and high availability. Beneath that, the LLM-d router handles complex, memory-aware scheduling algorithms to keep utilization high. This architecture is deliberately runtime-, model-, and accelerator-agnostic, and in tests, routing traffic through the multi-cluster GKE Inference Gateway added less than 1% overhead.
Guide: Using or planning to use GKE on TPUs for AI model training or inference? Training massive Large Language Models (LLMs) or running high-throughput inference serving represents a significant investment in specialized AI hardware, such as Cloud TPUs and GPUs. To get the most out of every dollar spent, you need to understand workload lifecycle metrics in GKE. This detailed guide explains how to turn opaque cluster behaviors into actionable telemetry.

Customer and partner updates

GKE customer: Learn why gaming startup SeaVerse relies on GKE Agent Sandbox for its multi-tenant workloads, and how the platform helped it decrease its infrastructure costs by 60%.

Research, reports and deep-dives

In case you missed it, we’re also thrilled to share that Google has been named a leader, including achieving the highest score on either product or strategy, in three key analyst reports from Gartner and Forrester.

Google is a leader in The Forrester Wave™: Public Cloud Platforms, Q3 2026: Highest overall score of any cloud provider!
Google named a Leader in 2026 Gartner® Magic Quadrant™ for Strategic Cloud Platform Services: Positioned furthest for “Completeness of Vision” of all vendors evaluated.
Google is a Leader in the 2026 Gartner Magic Quadrant for Container Management: Positioned highest in “Ability to Execute” of all vendors evaluated.
Google Cloud achieved a Gold rating in the latest SemiAnalysis ClusterMax 3.0 report, which evaluates the reliability, performance, support, pricing, and security of GPU providers globally. See the full report for more.

August 2026

Product, technology, and tools updates

Product update: Filestore, Google Cloud’s first-party, secure, scalable NFS file service, has emerged as a popular storage platform for AI and agentic workflows, and now, it’s even better suited to the task, with a new backend storage layer built directly on Colossus, Google’s foundational distributed storage system. This new backend lets you provision IOPS independently from storage capacity, and is deeply integrated with GKE. In AI environments, this can help you service so-called agentic swarms — large groups of agents that need to read and write to a common dataset — without a drop off in performance. For more, check out the blog post.
New feature: gVisor sandboxes are now available in distributed Ray clusters on GKE. In partnership with Anyscale, we introduced an experimental library for Ray that brings gVisor, Google’s open-source application kernel, directly into distributed Ray clusters. gVisor provides lightweight environments with stronger isolation than ordinary containers, plus fast startup times and low memory overhead. To try out these sandboxing capabilities on GKE, head over to the Ray sandboxing User Guide.
Product update: Looking for high-performance, easy-to-use infrastructure on which to run a personal AI agent, but don’t want to spend a lot of money? New Cloud Run instances are dedicated, singleton compute runtimes on Cloud Run that won’t shut down when the agent is idle. Better yet, the cost to run a Cloud Run instance with 1 vCPU and 1 GiB of memory continuously for 30 days is just $5.70.

Practitioner guides, documentation and how-tos

How-to guide: Big news in Model Context Protocol (MCP) land: As of the 2026-07-28 specification, the protocol core is “completely stateless. The handshake is gone. The initialize / initialized handshake (SEP-2575) and the logical Mcp-Session-Id header (SEP-2567) have been removed entirely. Instead, every request is now self-describing and independent.” Whoa. Learn more about the changes that the latest MCP specification brings, and more importantly, how to implement them, in this Google Developers blog.
Guide: Real-time AI systems make a mess of traditional network load balancing techniques. “Instead of handling isolated requests, the backend has to manage a continuous, live bidirectional stream. You’re dealing with a constant stream of audio chunks, transcripts, model outputs, and synthesized speech flowing back and forth simultaneously.” Things only get worse when the user gets involved. “The server has to immediately halt its current speech generation, pivot to update the context, maybe trigger a new tool, and start drafting a different response; this must be done without dropping the connection.” For a new approach to managing load in the AI era, read Scaling real-time AI agents with session-aware load balancing.
How-to: Learn how to build an elastic, scalable LLM inference platform on GKE, even with a mix of different GPU accelerators. The proposed architecture combines Capacity Advisor and Compute Advisor, plus high-performance storage like RunAI:model streamer or GCPFuse with parallel downloads. Get all the details here.
Documentation: The thing about hosts with GPUs or TPUs is that you can’t use live migration to update them, setting up a maintenance challenge. In this new docs page, learn how to update accelerator-equipped hosts according to your tolerance for downtime for your training and inference workloads.
Documentation: Advanced Compute Images, or ACIs, are standardized image stacks for AI/ML and HPC infrastructure, so you don’t need to manually build your own custom images. In this new docs page, learn how to create an ACI image using the Google Cloud CLI, console, or SchedMD's Slurm workload manager.
Guide: AI workloads are notoriously difficult to architect, resource-intensive, and bursty, which can also lead to scaling bottlenecks and large pools of underutilized — or misutilized — compute resources. A new blog outlines the three main ways to achieve dynamic capacity management in Google Cloud: 1) scheduling capacity for planned downtime; 2) maintaining automated fallback capacity for unplanned downtime; and 3) relying on GKE’s core orchestration capabilities to automate resource allocation.

Customer and partner updates

Business orchestration software provider UiPath was dealing with spiky workloads, and wanted more predictable costs. To get there, it re-architected its infrastructure, moving from isolated clusters to a shared Google Cloud GPU fleet that included both A3 VM instances (NVIDIA H100 GPUs) for training with G4 VM instances (NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs) for inference. You can read more about their architecture here.
Mirendil, an frontier AI lab focused on accelerating AI development, announced that it is using AI Hypercomputer with both TPUs and NVIDIA GPUs to support its model pre-training and post-training applications.
Replenit, a retail CRM provider, built its AI decision engine in Google Cloud, using BigQuery, Gemini Enterprise Agent Platform, and open-source Gemma models that it runs on Cloud TPUs. This latter combination provided Replenit with 90% lower pipeline costs than their previous cloud provider, the company reports. Read the full case study for more.
Malachyte architected its AI-powered e-commerce recommendation platform on top of Bigtable, Managed Service for Apache Kafka, Pub/Sub, Compute Engine, and last but not least, GKE. See how it all comes together in this blog.

July 2026

Product, technology, and tools updates

Product update: Google Cloud Managed Lustre is now GA, and available in four distinct performance tiers that deliver throughput ranging from 125 MB/s, 250 MB/s, 500 MB/s, to 1000 MB/s per TiB of capacity — with the ability to scale up to 8 PB of storage capacity. The Managed Lustre solution is powered by DDN’s EXAScaler, combining DDN's decades of leadership in high-performance storage with Google Cloud's expertise in cloud infrastructure.
Product update: C4N network and storage optimized VMs are now GA. C4N is our first network- and block-storage-optimized VM series built to eliminate data-transfer bottlenecks. Powered by 5th Gen Intel Xeon Scalable processors and built on Google's Titanium offloading hardware, it achieves 400 Gbps network bandwidth, 95 million packets per second (MPPS), and up to 25 GiB/s of block storage throughput when paired with Hyperdisk Extreme.
New feature: GKE Dataplane V2 up to 15K Nodes with Network Policies (GA). This capability enables standard GKE clusters to scale up to 15,000 nodes while maintaining full active Network Policy enforcement, supporting the massive infrastructure needs of large enterprise and AI/ML customers.
New feature: Co-operative time-slicing in llm-d. If you’re running reinforcement learning (RL) workloads, you can now interleave independent RL jobs onto shared physical hardware, increasing aggregate accelerator duty cycles from a ~40% baseline up to 70% without impacting model convergence or accuracy.
New AI security tool: Looking to secure your AI supply chain on GKE, deploy AI workloads safely, and cut down on shadow AI? We open-sourced k8s-aibom, a lightweight, unprivileged Kubernetes controller that continuously monitors container clusters to automatically detect running AI runtimes (like vLLM and Triton) and generate standard CycloneDX Machine Learning Bill of Materials (ML-BOMs). Check out the k8s-aibom project and get involved.

Practitioner guides and how-tos

How-to guide: On July 27, Google announced Day 0 support for Moonshot AI’s Kimi K3 2.8-trillion-parameter open-weight model, the day weights were released. Whichever your preferred deployment path — via Model Garden, custom orchestration, or GKE with llm-d recipes — this guide offers detailed step-by-step instructions to help you evaluate and pilot Kimi K3 in Google Cloud.
How-to guide: Google Kubernetes Engine (GKE) managed DRANET supports both GPUs and TPUs. There are several configurations to use this implementation, including standard cluster (where you have full control) and autopilot cluster (where Google does the heavy configs for you). Take a deeper dive in the hands-on lab, GKE Autopilot clusters with TPUs, GKE managed DRANET and Gemma 4.
How-to guide: Learn to run Ray on TPUs, not GPUs. In Part 1 of this two-part series, we discuss TPU slices (hint: Ray thinks of them as just another accelerator on which to schedule), then walk through Ray’s various AI libraries (Part 2).
How-to guide: Evaluate TPUs for sample workloads using a new microbenchmark suite that helps you accurately assess whether a device is achieving its theoretical performance specifications, and to identify specific performance gaps or architecture-specific bottlenecks. Dive in here.
How-to guide: Scale your agents without killing your budget. Learn how GKE orchestration can help you safely pack more agents onto a fixed compute footprint with GKE Agent Sandbox and Pod snapshots. Whether your goal is performance or cost optimization, we teach you how to turn the right dials for optimal agent efficiency.
Technical blueprint: Inside the optimization of Mistral 3 large inference on Ironwood. This blog outlines how one Google team optimized Mistral 3 large MoE model inference on Google’s Ironwood (TPU v7x), achieving a 1.5x performance gain. They did so with hybrid sharding, replacing linear VPU summations with tree reductions, optimizing GMM/MLA kernels, and adopting asynchronous scheduling. As a result, they boosted throughput by up to 48% while maintaining benchmark accuracy neutrality. Read the full blog here.

Research, reports and deep-dives

Report: Google was named a Leader in the inaugural GartnerⓇ Magic Quadrant™ for AI Infrastructure, positioned highest for ‘Ability to Execute’ and furthest for ‘Completeness of Vision’. Gartner called out Google’s proprietary scalable compute, integrated AI Hypercomputer architecture, and the scale of our AI compute capacity as key strengths. Download a copy here.
Report: We recently surveyed more than 1,400 senior IT leaders for our State of AI Infrastructure report, and a resounding pattern emerged: The gap between AI ambition and infrastructure reality is widening. In fact, 83% of organizations say they require infrastructure upgrades to support production-grade agentic AI. Read the accompanying blog to understand how adapting your infrastructure to meet the demands that agentic applications place on your systems will help you move from pilot to production.

June 2026

Product, technology and tool updates

Product update: Protecting sensitive data used with AI is a critical part of advanced and secure cloud infrastructure. Confidential Computing cryptographically protects data in use in hardware-based Trusted Execution Environments (TEEs) with verifiable data integrity, and is now available on the accelerator-optimized G4 machine series, featuring NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs. Get started with Confidential G4 VMs and Confidential G4 GKE Nodes.
Developer resource: The new TPU Developer Hub is the place to go for model builders, optimizers, and developers to learn to unlock the full performance of Google Cloud TPUs. Read more in this blog.
New product: Scale your AI workloads with the new OpenTelemetry-Based TPU AI Telemetry Collector Agent. For the first time, you can route high-fidelity TPU hardware telemetry to Google Cloud Monitoring, Google Managed Prometheus, or your own self-hosted Grafana stack.

Practitioner guides and how-tos

How-to guide: Learn how to build high availability into an AI inference workload running on GKE Inference Gateway with TPUs, Cloud Storage FUSE and Dynamic Resource Allocation (DRA). This blog provides an overview, or you can get all the technical details in the hands-on codelab.
How-to guide: Did you know you can connect your AI agents to unstructured data in Cloud Storage via Model Context Protocol (MCP)? In this blog, learn about why would want to do that from three customer examples, then how to do it, choosing either a fully managed service, or a self-managed local server for more customization and control.

Research, reports and deep-dives

Report: According to an independent benchmark report, GKE Inference Gateway outperforms the next leading managed Kubernetes service with 15.7% higher throughput, 92.8% shorter wait times, and 62.6% lower inter-token latency. This performance can be attributed to its use of prefix caching, which optimizes LLM performance by storing the KV cache (activation states) of long, repetitive prompt prefixes. Learn more in the blog.
Architecture deep dive: A closer look at the cold start problem, this time for TPUs and GKE, and how the Run:ai Model Streamer can help change the dynamic.

Customer and partner updates

Customer win: Leveraging GKE, BigQuery, Cloud SQL, and Gemini Enterprise Agent Platform, Pager Health is eliminating operational fragmentation to deliver a simplified, personalized U.S. healthcare experience that transforms lives.
Customer win: Trustpilot, the customer review platform, built a high-volume streaming pipeline using fine-tuned Gemma models with Dataflow and Gemini Enterprise Agent Platform running on cost-optimized A2 VMs using A100 GPUs, as well as optimized version of vLLM maintained by Gemini Enterprise Agent Platform.

May 2026

Product, technology and tool updates

Product update: GKE Agent Sandbox is now generally available.
New open-source project: Agent Substrate is a new open-source project aimed at continuing to push the limits of agentic infrastructure density
New feature: Google AI Edge Portal, a solution for testing and benchmarking on-device machine learning (ML) at scale, now supports benchmarking and debugging on-device LLMs. Read more here.
Product deep dive: We went into depth about Cloud Storage Rapid, a new family of high-performance storage offerings for AI workloads. At launch, offerings include Rapid Bucket (formerly Rapid Storage), a high-performance zonal object storage offering, and Rapid Cache (formerly Anywhere Cache), which accelerates reads on-demand and colocates compute and data for workloads in existing buckets.

Research, reports and deep dives

Architecture deep dive: Google Global Infrastructure VP Bikash Koley and Engineering Fellow Arjun Singh provide a high-level overview of the challenges that AI workloads pose to network infrastructure, and discuss the deep enhancements we’ve made to our data center fabrics, WAN, and global networks to better support them.
Architecture deep dive: We unveiled a new cluster-level reliability model for developing frontier AI models on TPUs, ditching instance-level reliability

Customer and partner updates

Customer win: Visual media provider Imgix serves more than 8 billion images and videos from AI Hypercomputer equipped with G4 VMs powered by NVIDIA RTX PRO 6000 Blackwell GPUs.

Accelerating agentic RL and evaluation research velocity with 45x faster GKE Agent Sandbox

Tue, 29 Sep 2026 17:00:00 +0000

When scaling up agentic reinforcement learning (RL) and evaluation across massive parallel rollouts, frontier AI labs inevitably hit a bottleneck: Expensive GPU clusters sit idle, waiting minutes for CPU sandbox cold-starts, plus thousands of multi-gigabyte SWE-bench-style image pulls and scheduling backlogs. It’s a sandbox infrastructure problem that silently slows down your research and burns your training budget.

To solve this fundamental infrastructure bottleneck, today we are introducing GKE Agent Sandbox optimized for RL along with the Agent Sandbox RL orchestration SDK, plus native integrations for popular RL gyms and harnesses, now generally available.

As the operating system for modern AI, Kubernetes has evolved to power massive GPU/TPU training clusters and distributed inference. Now Kubernetes is expanding to drive the next AI compute frontier: agents. But unlike static workloads, agentic workloads evolve rapidly, so infrastructure must evolve just as fast. Rather than guessing at what RL researchers needed, we placed Kubernetes itself on an auto-research and verification loop driven by performance benchmarks and evaluations. We used heavy agentic benchmarks like SWE-bench to intentionally stress-test and break our own clusters. Every bottleneck that surfaced — from etcd timeouts to GPU idle spikes — was fed back into our development cycle to refine GKE’s core primitives.

This resulted in a purpose-built sandbox layer for agentic RL and eval workloads that features:

10x - 45x faster time-to-first-command: GKE can spin up a sandbox environment in 1–9 seconds instead of 45–85 seconds, keeping your expensive GPUs fully in use.
Reduced tail latency: We reduced the worst-case sandbox wait times from 7.5 minutes down to under 10 seconds.
3x less control-plane churn: Each RL training step triggers a rollout burst, where thousands of sandboxes are requested at once. The SDK has an in-place recycling strategy that reuses pods across rollouts instead of deleting and re-creating them. That means 3x fewer pods being created, which keeps the Kubernetes API server stable under the churn.

With this new primitive, AI labs and agent-native startups can now reliably run large scale agentic RL trajectories and evals simultaneously, minimizing accelerator idle time and drastically accelerating their research velocity.

The real-world challenges of agentic RL infrastructure

Before we talk about the solution, let’s be precise about what makes agentic RL so demanding for infrastructure in the first place. In a standard agentic RL loop, an LLM policy generates actions like code snippets on GPUs and executes them inside isolated CPU sandboxes to observe a reward signal. However, when scaling up this loop to support tens of thousands of parallel rollouts, three critical infrastructure bottlenecks emerge:

Accelerator idle costs, i.e., time-to-first-command (TTFC) and the tail latency trap: Agentic RL is a batch workload, and in a synchronous RL step, training cannot proceed until the slowest sandbox in the batch is ready. If standing up CPU sandboxes takes minutes to provision, pull images, and execute initial startup scripts, expensive accelerator capacity sits wasted.
Massive image cardinality: Standard container caching assumes a handful of static base images. However, in agentic RL, every single task (like thousands of GitHub repositories in SWE-bench or R2E-Gym) often requires a completely distinct OCI container image. Managing thousands of unique, large images per run can lead to severe image-pulling bottlenecks and storage friction.
Control-plane saturation: As a bursty batch workload, agentic RL typically executes batch-size image tasks and multiple rollouts per task (e.g. 4, 8, 16 rollouts per task), pushing an already large number of images to the extreme. For instance, one public dataset we tested against has 4,578 R2E images. Assuming four rollouts per image, that means 18,312 tasks, which require 18,312 sandboxes simultaneously. Standard Kubernetes control planes degrade significantly under these burst loads. Churning tens of thousands of ephemeral pods per minute creates API-server queue bottlenecks, pod state errors and triggers false node-health evictions.

These are not hypothetical problems. They are the daily reality for frontier AI labs training state-of-the-art agents.

GKE Agent Sandbox optimized for RL

To fan-out the code execution sandboxes, we set up a relatively modest cluster — a 10-node gVisor sandbox pool with GKE image streaming enabled, the Agent Sandbox controller, and an in-cluster SDK driver to claim warm pods. We tested various strategies and setups including a large number of images and high cardinality.

The infrastructure layer

GKE Agent Sandbox is the open Kubernetes primitive for secure agent execution, featuring built-in SandboxWarmPool capabilities that eliminate cold-start overhead by maintaining pre-initialized, healthy environments. By integrating SandboxWarmPool with GKE Image Streaming, we effectively support RL and eval workloads that demand high image cardinality — even with thousands of large images (>1.2GB), while delivering the exceptionally low TTFC required for responsive agentic training. Its snapshot, suspend and resume capabilities support checkpointing for error recovery, and sandbox forking for parallel agent branching logic to explore multiple trails.

This GKE primitive is already powering the agentic RL training infrastructure at Mistral AI, a frontier AI lab:

“To push the boundaries of reinforcement learning, you need infrastructure that can instantly scale to handle unpredictable demand. By leveraging GKE's high-performance Agent Sandbox for RL, we can seamlessly orchestrate hundreds of thousands of secure environments across clusters and handle spikes of over 30,000 sandboxes on a single cluster. It provides the reliable foundation we need to accelerate our model training and iteration cycles.” - Jean-Malo Delignon, Research Engineer, Mistral AI

Orchestration SDK and RL tool integrations

To help researchers harness this power without writing Kubernetes YAML or embedding custom daemons into container images, we built the Agent Sandbox RL orchestration SDK. It exposes a clean, async Python API with pluggable warm-pooling strategies tailored to your specific evaluation or RL training pattern. We also built native integrations for RL tools like Gymnasium, NVIDIA NeMo Gym, and OpenHands, with more to come.

Benchmarks: Reducing idle accelerator waste

We ran this on a 10-node gVisor sandbox pool against two workloads: a 500-image SWE-bench environment, and a harsher 4,578-image R2E corpus that does not fit in local disk.

1.TTFC and tail latency: 10x–45x faster

Core metric	Why it matters	Raw K8s pod baseline	GKE Agent Sandbox SDK	Benchmark gains
TTFC, average	Determines GPU idle time	44s – 85s (average)	1.1s – 8.8s (average)	10x faster
Tail latency — max TTFC (worst case)	The bottleneck for the batch	7.5 mins (450 seconds)	Strictly under 10 seconds	45x faster

Concurrency ranged from 500 simultaneous tasks and sandboxes up to 18,312 (4,578 R2E images × 4 rollouts). The gain held across every setup.

How we got here. We instrumented the controller, the SDK, and the RL fleet, then ran hundreds of comparison runs against that fixed 10-node budget. Two changes produced nearly all of the gain:

Image hydration moved off the critical path. A high-cardinality corpus far exceeds local disk, so pulling cold images mid-training creates I/O contention and multi-minute tails. The SDK plans image placement across nodes, and GKE Image Streaming plus upfront warm-pooling handle the rest — hydration finishes before the rollout ever asks for it.
The control plane is paced. High-concurrency rollouts trigger API-server thundering herds. Agent Sandbox Controller v1.0.0 adds rate controls that bound how fast warm pools refill, keeping etcd and the API server stable through the burst.

The tuning knobs, the benchmark harness, and the load tests behind these numbers all ship in the agent-sandbox-rl example, which emits human-readable and JSON reports so you can reproduce the comparison on your own cluster.

2. 3x less control-plane and pod lifecycle churn during multi-trajectory rollouts

Core metric	Why it matters	Raw K8s pod baseline	GKE Agent Sandbox SDK	Benchmark gains
Control-plane churn (4,578 images × 4 rollouts run)	Control-plane saturation	18,312	5,869	3.1x less churn

Raw Kubernetes recreates a pod for every trajectory step. At 18,312 tasks that inflates scheduling overhead, wastes disk I/O, and — in our large-scale tests — triggered unbounded garbage-collection loops. The SDK's in-place recycle strategy keeps the pod alive instead, running an in-pod git reset and repository checkout between rollout episodes. Pod creations drop 3.1x and the control plane stays flat through the burst.

The trade-off

Warming environments up front spends inexpensive CPU and background cluster time so that image streaming and readiness checks never land on the critical path. For an RL fleet that’s a straightforward trade: Accelerator idle time is the expensive resource, and this approach eliminates it.

One detail matters at training scale: The SDK counts and surfaces every failed sandbox as retriable rather than silently dropping it. An untracked drop is not just a lost rollout — it is reward bias.

Get started

Build today: Explore the Agent Sandbox repository or the Agent Sandbox RL repository for the Python SDK and native integrations for RL tools including Gymnasium, NVIDIA NeMo Gym, and OpenHands.
Push accelerator utilization further: Explore the llm-d co-operative time-slicing repository and its Snapshot Agent guide to dynamically interleave independent RL jobs onto shared physical hardware.
Learn more: Read the official documentation on deploying secure execution environments with GKE Agent Sandbox.

Google is a Leader in the 2026 Gartner Magic Quadrant for Container Management

Thu, 24 Sep 2026 19:00:00 +0000

We’re excited and proud to share that Gartner has recognized Google as a Leader for the fourth year in a row in the 2026 Gartner® Magic Quadrant™ for Container Management, based on its Completeness of Vision and Ability to Execute. Google was positioned highest in Ability to Execute of all vendors evaluated and we believe this validates the success of our mission to deliver a container platform that’s highly optimized for both performance and efficiency. We help global customers to build and run their most demanding and complex workloads at scale, including the next generation of AI and agentic applications.

In the accompanying 2026 Gartner Critical Capabilities for Container Management report, Google Cloud was ranked first in every use case: New Cloud Native Applications, Containerized Existing Applications, AI Training, AI Inference, Edge Applications, and Hybrid Applications.

Gartner predicts¹ that “By 2028, 95% of new AI deployments will use Kubernetes, up from less than 30% in 2025.” Containers power today’s most innovative apps and businesses — and deliver the infrastructure customers demand as they transform their businesses in the agentic era.

Google Cloud spearheaded the industry-wide cloud-native revolution when we introduced Kubernetes in 2014 and launched Google Kubernetes Engine (GKE), the world’s first managed Kubernetes service, in 2015. Our commitment to container platforms and the vibrant, innovative Kubernetes ecosystem has only grown stronger and deeper since. Alongside GKE, our serverless container platforms GKE Autopilot and Cloud Run dramatically lower operational costs and help developers deliver amazing containerized apps faster than ever before.

The massive acceleration in enterprise AI has inspired us to redefine infrastructure management for the AI era. In 2026 so far we’ve introduced a wide range of foundational improvements to shift GKE and Cloud Run into agent-native, high-performance platforms designed for autonomous AI systems, massive inference workloads, and secure runtime isolation. Whether you’re training AI at the frontier, launching an AI startup, or leading your enterprise AI transformation, we have the container platform you need. Important highlights include:

Delivering leading performance and efficiency for AI infrastructure

GKE predictive latency boost: Built into the GKE Inference Gateway, this ML-driven capability uses capacity-aware routing rather than static configurations to reduce Time-to-First-Token (TTFT) by up to 70%.
GKE automatic KV Cache storage tiering: Automatically shifts KV cache data across RAM, Local SSD, and Cloud Storage. This reduces memory bottlenecks, improving TTFT by 40% via RAM offloading and increasing throughput by 70% via Local SSDs for large prompt contexts. [1]
GKE accelerated container and model startups: GKE node spin-up times are up to 4x faster, and pod startup speeds have improved by up to 80%. Additionally, native run:AI Model Streamer integration pulls heavy models from Cloud Storage 5x faster.
Cloud Run on-demand serverless GPU scale-to-zero: Cloud Run supports NVIDIA RTX PRO 6000 Blackwell GPUs, allowing teams to serve 70B+ parameter models on-demand. Your services can go from zero to a fully provisioned GPU — with all drivers pre-installed — in under 5 seconds. Once active inference or fine-tuning runs complete, Cloud Run automatically scales instances back to zero, eliminating idle infrastructure costs.

Evolving Kubernetes for agentic infrastructure security and scale

GKE Agent Substrate: As an open-source, secure-by-default agent execution runtime, Agent Substrate is engineered to run millions of sandboxes with 10x higher density than standard container runtimes. Purpose-built for the era of autonomous agents, Substrate delivers sub-500ms resume operations at over 500 suspend/resume activations per second with a native zero-trust kernel and network isolation. Agent Substrate is available as an open-source solution that runs on any Kubernetes infrastructure and is optimized for GKE.
GKE Agent Sandbox: Built on gVisor kernel-isolation technology, Agent Sandbox isolates the host environment from untrusted, multi-agent AI code execution. It provides secure execution at scale, processing up to 300 sandboxes per second with sub-second latency and delivering up to 30% better price-performance when running on Axion processors than comparable hyperscaler cloud providers.
GKE Dataplane V2 scalability limits: Architectural capacity bounds for GKE clusters implementing active NetworkPolicies doubled from 7,500 nodes to 15,000 nodes per cluster, supporting the massive infrastructure needs of large enterprise and AI customers.
GKE intent-based autoscaling: GKE can now natively autoscale horizontally using application intent and custom metrics beyond basic hardware metrics. This reduces resource allocation reaction times from 25 seconds down to just 5 seconds.
Filestore agent volumes: a new offering that attaches and detaches NFS mounts in milliseconds, allowing agents to start/resume near-instantaneously, along with native Read-Write-Many (RWX) access and POSIX-compliant file locking to enable safe multi-agent collaboration without write collisions.

Next-gen developer experience with serverless containers

Whether you’re hosting a standard web API, running a heavy batch data job, processing an asynchronous message queue, or deploying a complex AI agent, Cloud Run handles it all under a single, unified serverless model that delivers an unmatched developer experience and maximum engineering velocity.

One-click prototyping in Google AI Studio: You can build and deploy full-stack applications directly within Google AI Studio, making it an exceptional environment for rapid prototyping and experimentation. With a single click, you can instantly package and publish your vibe-coded applications to Cloud Run.
Cloud Run instances: This new primitive manages individual, addressable, long-running singleton resources with integrated Cloud Storage volume mounts, allowing persistent background agents like OpenClaw to be deployed cost-effectively. With baseline shared-CPU configurations starting at a highly predictable flat rate of ~$5.70 per month (for 1 vCPU and 1 GiB of RAM), Cloud Run instances delivers an always-on, VM-like experience while bypassing the idle-cost penalties and operational overhead of traditional VMs.
Cloud Run sandboxes: Hard-isolated environments spin up in under 500 milliseconds to safely execute untrusted, model-generated code, protecting the host system from unauthorized access.

Take the next steps

As we reach for new heights of performance, security, and scale for our container platforms, we continue to build the future in the open. We invite you to explore Agent Sandbox and Agent Substrate today. We can’t wait to shape the future of agent infrastructure together with our customers and partners. Check out these resources to continue your learning journey:

Download your complimentary copy of the 2026 Gartner® Magic Quadrant™ for Container Management.
Try Agent Sandbox on GKE.
Contribute: Join the Agent Sandbox open-source community.
Explore Agent Substrate.
Join us at KubeCon North America 2026 in Salt Lake City, November 9-12. For even more fun, arrive a day early for GKE Day on November 9.
Discover why Google is also named a Leader in the 2026 Gartner® Magic Quadrant™ for Cloud-Native Application Platforms.
Start building the future of serverless applications today at cloud.run.

^{1. Gartner report: Critical Capabilities for Container Management, 8 September 2026}

^{Gartner, Magic Quadrant for Container Management, Dennis Smith, et al, 2 September 2026}^{Gartner, Critical Capabilities for Container Management, By Tony Iams, Wataru Katsurashima, Lucas Albuquerque, Dennis Smith, Bhuvie Chhabra, 8 September 2026.}^{Gartner and Magic Quadrant are trademarks of Gartner, Inc. and/or its affiliates.}^{Disclaimer: Gartner does not endorse any company, vendor, product or service depicted in its publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner publications consist of the opinions of Gartner’s business and technology insights organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this publication, including any warranties of merchantability or fitness for a particular purpose.}

Global AI routing with <1% overhead on multi-cluster GKE Inference Gateway

Mon, 21 Sep 2026 16:00:00 +0000

Demand for AI infrastructure is at an all-time high. Global accelerator shortages mean engineering teams can rarely get all the compute they need from just one data center — capacity comes a cluster here, a cluster there, often an ocean apart. At the same time, workloads are getting hungrier: Today’s long-running agentic workloads often have context windows of 100k to 800k+ tokens, which consume accelerator memory faster than any previous generation of AI traffic.

In this environment, the goal is to maximize "intelligence per dollar." Fragmented, poorly balanced infrastructure is rarely up to the task though, allowing expensive accelerators to sit idle, while requests queue up somewhere else.

To close that gap, we built a layered routing architecture that makes globally scattered capacity behave like a single pool behind a single entry point. At the edge, the multi-cluster GKE Inference Gateway focuses on global, multi-region traffic distribution and high availability. Beneath that, the LLM-d router handles the complex, memory-aware scheduling algorithms that keep utilization high.

This architecture is deliberately runtime-, model-, and accelerator-agnostic — it works across serving frameworks, model families, and GPU or TPU hardware. To make the results concrete rather than abstract, we recently benchmarked managing production-level global request routing at scale across a multi-region GKE deployment of 17,000 compute nodes spread across the US and Europe. The deployment served a leading Mixture of Experts (MoE) foundation model using SGLang.

The results: Scaling to three clusters achieved a near-linear throughput boost while maintaining a 99.9% success rate under heavy multi-client concurrency. Additionally, routing traffic through the multi-cluster GKE Inference Gateway added less than 1% overhead, delivering 99.5% of the throughput of a direct, local cluster call.

Read on to learn how it works, more on the benchmark results, and what it means for your own distributed inference deployment.

Three regions, one endpoint

The deployment spanned three GKE clusters in three geographic regions: us-east5 (the config cluster), us-west8, and europe-west4. However, from the client’s perspective, none of that geography exists. Requests hit a single global virtual IP, and the gateway decides — in real time — which cluster should serve each one.

What makes that decision smart rather than blind is telemetry. Instead of traditional round-robin routing at the network layer, the multi-cluster load balancer is configured to route traffic based on live application signals. Specifically, the Endpoint Picker Proxy (EPP) reads the KV-cache token utilization natively exposed by the underlying inference engines and emits it as a metric for the load balancer. When the load balancer sees a region running hot based on this emitted metric, it spills traffic to the next healthy region.

Multi-cluster GKE Inference Gateway topology. The config cluster holds routing configuration but sits outside the request path; each target cluster runs its own EPP and reports KV-cache utilization back to the load balancer.

Distributed LLM engines also operate differently than standard web apps. In a typical inference engine's distributed mode (such as tensor parallelism across multiple nodes), only the master (rank-0) pod serves the API. GKE already handles local routing using standard Service selectors and LeaderWorkerSet (LWS) to direct traffic exclusively to leader pods. The multi-cluster Inference Gateway also integrates with this foundation: It routes global traffic to the correct regional services, helping your cross-region load balancing respects your underlying multi-node topologies out of the box.

The net effect: Three isolated regional data centers start behaving like one cohesive global accelerator fleet, with failover and load balancing driven by what the models are actually doing.

Measuring the routing overhead

The first question every team asks about a global routing tier is almost always, ‘How much throughput am I giving up for cross-region capability?’

The benchmarks answer this directly: Deploying the multi-cluster GKE Inference Gateway to maximize your accelerator fleet doesn't have to come at the cost of throughput. Routing traffic through the Gateway added less than 1% overhead, delivering 99.5% of the throughput of a direct, local cluster call.

That’s the whole trade-off. All the benefits of global load balancing, essentially for free.

Linear scaling across regions

A bigger test is scale. In our test, growing the fleet from one cluster to three, spanning the US and Europe, while every client request originated from a single region (us-east5), put real pressure on the Gateway: If it couldn’t distribute load efficiently across those distances, throughput would flatten as hardware was added.

Instead, throughput multiplied almost exactly in line with capacity:

Fleet topology	Request throughput	Token throughput	Success rate
1 cluster (us-east5-a)	0.72 req/s	2,898 tok/s	99.87%
2 clusters (+ us-west8-a)	1.40 req/s	6,380 tok/s	99.95%
3 clusters (+ europe-west4- b)	2.10 req/s	8,457 tok/s	99.90%

Scaling to three clusters achieved a near-linear throughput boost while maintaining a 99.9% success rate under heavy multi-client concurrency.

Memory-aware routing in action

Round-robin load balancing is inadequate for serving LLMs because it treats every request as equal. They aren’t. Heavy prompts saturate GPU compute cores, long generations stress memory bandwidth, and long-context conversations quietly eat VRAM until the engine can’t schedule anything new.

Here, the pressure on memory bandwidth came from the routing signal chosen for this deployment. By mapping Inference Engine's native token-usage metric onto the Gateway’s KV-cache signal, the routing plane gained a real-time view of memory pressure across the entire 17,000 fleet. (Depending on the workload, the Gateway can route on other signals too, like queue depth or running concurrency.)

Under live production loads, as the primary region climbed toward its high-bandwidth memory (HBM) limits, the Gateway detected the saturation the moment the cluster crossed its 40% KV-cache utilization threshold; it then automatically began routing the overflow to the next healthy region. No operator intervention was needed. The complexity of running in multiple regions simply never reached the user.

The payoff

By routing traffic based on live KV-cache utilization, this GKE Inference Gateway setup effectively pools globally scattered compute capacity into one unified engine. For this deployment, the result was a near-linear throughput boost across three global regions, with virtually zero routing overhead.

This translates directly into maximizing 'intelligence per dollar,' extracting near-perfect proportional performance out of every accelerator you add to your fleet, rather than letting capital go to waste.

What this means for your team

If you’re planning your own distributed inference deployment, five lessons from this work stand out:

Smarter load balancing pays for itself. Round-robin routing wastes expensive GPU capacity because it can’t see memory or compute pressure. Routing on real-time application signals turns fragmented regional clusters into one efficient fleet — the difference between stranded hardware and 90%+ utilization of scarce compute.
Agentic workloads change the bottleneck. Long-running agents with extreme context windows exhaust memory long before there’s no more compute. If your routing layer can’t see memory pressure, your compute will strand compute behind full VRAM. Make KV-cache utilization a first-class routing signal.
AI traffic breaks web-era assumptions. Traditional load balancers are tuned for sub-second transactions; LLM requests can run for minutes. Plan connection limits and timeouts for AI-scale latency early, or expect aborted connections in production.
Your routing layer must integrate with native serving patterns. Distributed LLM engines have master-worker topologies where only certain pods can serve traffic. By pairing your Gateway with native Kubernetes constructs like LeaderWorkerSet (LWS), your global routing respects local pod topologies out of the box, saving your team from building custom proxy infrastructure.
For large foundation model builders, bet on an open, portable stack. Teams operating at frontier scale face the most acute capacity fragmentation, forcing them to hunt for compute resources across whichever regions have availability capacity. An open, portable inference stack such as LLM-d on GKE lets you absorb that capacity wherever it lands, rather than hard-wiring your serving architecture to any single cluster, region, or bespoke infrastructure.

Next steps

Ready to maximize your distributed accelerator efficiency and set up global cross-region load balancing with multi-cluster GKE Inference Gateway?

Deploy it yourself: Set up the multi-cluster GKE Inference Gateway.
Understand the architecture: About multi-cluster GKE Inference Gateway.
Learn about the cross-region spillover behavior featured in this post: About elastic cross-region high availability and Configure elastic cross-region high availability.

Changing the game: Using agentic AI to secure infrastructure code

Sat, 19 Sep 2026 01:00:00 +0000

Blog authors: Andrés Lagar-Cavilla, Distinguished Engineer; Stella Voutsina, Sr. Technical Program Manager; Parthasarathy Ranganathan, VP, Engineering Fellow

AI is accelerating software development at an unprecedented pace. But as code generation scales, so do the challenges of securing the code, especially emerging AI-based vulnerability exploitations. To meet these challenges, the Google AI and Infrastructure team is transforming how we approach security. In this article, we discuss new AI-native agentic methods that we’ve developed that systematically embed high-precision, pervasive vulnerability scanning and patching directly into Google’s software development lifecycle. By continuously scanning every code change across hundreds of millions of lines of code that we deploy onto our infrastructure, we are preventing hundreds of vulnerabilities per month from ever reaching our code base or production, defending our global network, AI infrastructure and our users.

Solution architecture and implementation

Pervasive pre-submit agentic scanning: security as part of ongoing software development

Traditionally, the technology industry relies on large one-off security scans that are slow and lack sufficient context. As a result, they often find vulnerabilities too late. Our approach instead focuses on pre-submit scanning, where we evaluate each code check-in (across every layer of the stack) in real-time using AI agents. By integrating the pre-submit scan into the tools developers already use, security becomes a continuous routine process, similar to rule checkers, readability reviews or other software development tools. Also, from an AI perspective, scanning each individual code change requires much less context than performing a large one-off scan, significantly improving the scan’s effectiveness.

The importance of localized threat models

For this initiative, we evolved Mantis, our open-source multi-agent review harness, to increase the precision of our security agents by matching them with a cohort of robust localized threat models. Rather than relying on static decoupled documents, the threat models use live codebase metadata. The scanning agent improves its accuracy further using a dependence call graph across packages and libraries to expand and refine its threat model context. Making threat models part of our ongoing vulnerability scanning encourages developers to continuously update threats and dependencies, keeping the models up-to-date. Using localized and precise threat model data translates to dramatic accuracy improvements, bringing our false-positive rates down to 3% in some cases.

Specialized triage agents speed up development

Vulnerability scanning as part of code check-in requires it to respond quickly to the developer or agents generating the code, so as not to impede engineering productivity. To get responses with low latency, we run a two-step validation process. First, we run a quick lightweight scan that validates its findings against a specialized triage agent. This agent programmatically checks the actual structure of the code (using abstract syntax tree parsing, call-graph traversal, and pre-indexed domain safety rules) to prove that the vulnerable path is actually reachable by an attacker. This agent gets over 92% precision and completes its work in less than a minute. Then, a post-submit scan as part of nightly integration testing serves as a second layer of defense, using off-peak cycles to test for vulnerabilities that may have been introduced across multiple changes.

Bug fix agents close the loop

Finding vulnerabilities is only half the battle. The last component of our solution is an automated bug-fix agent that uses the scan results and generated proofs (snippet of code that demonstrates how the vulnerability is exercised) to autonomously construct precise fixes that are consistent with our internal coding standards. The agent submits the fixes for human review as part of the original change request’s review, further reducing the time between detection and resolution.

Learnings and call to action

Embedding continuous scanning directly into the software development lifecycle has been a game changer at Google; its suggestions are widely adopted, and it’s prevented a multitude of vulnerabilities from being introduced into the codebase. But any organization wishing to improve security can adopt a similar AI-native approach, following these principles:

Keep systems separate: To prevent bias, keep the harnesses, rules, and context for each of your development, scanning, triage agents separate. Pair lightweight AI scans with deterministic, structural validation to drive down latency and improve accuracy.
Use context wisely: Feed your agents your existing threat models. Precise context is the answer to reducing false positives, and up-to-date threat models set a high floor on a team's security posture by improving the rate of true positives in presubmit scanning.
Build a good harness: While the choice of the underlying model is important, using a multi-agent harness can have substantial impact, by helping compensate for variability in model choice.
Automate the fix: Use agents to also propose human-in-the-loop fixes, to further reduce time-to-resolution.

If you want to get started on your own AI-native security transformation, Mantis is now available as open source for you to use and benefit from. You can also learn more about the fundamentals of cybersecurity and the other platforms that power this agentic pipeline: Google Cloud, Gemini Enterprise and Gemini models running on Trillium and Ironwood TPUs. And you can get inspiration from how agentic vulnerability scanning and remediation defends Google Cloud customers as an integral part of Google Cloud’s secure software development lifecycle (SDLC) effort.

^{With special recognition to critical team members who made this delivery possible: Stella Voutsina (Lead Program Manager), Yulong Zhang (Senior Staff Security Engineer, Mantis), and Nick Galloway (Staff Security Engineer, Mantis).}

For SeaVerse, GKE Agent Sandbox reduces infrastructure costs by 60%

Wed, 16 Sep 2026 16:00:00 +0000

Editor’s note: Today we hear from SeaVerse, a gaming startup from SeaArt that is building a platform for playable AI experiences, where users can open lightweight games, character chats, and interactive apps, or create their own experiences from a prompt. To support that creative loop, SeaVerse needed infrastructure that could run dynamic, multi-tenant sandbox workloads with strong isolation, low latency, better observability, and more flexible costs. Google Kubernetes Engine (GKE) and GKE Agent Sandbox gave SeaVerse the managed foundation from which to execute these AI workloads, helping the team reduce their infrastructure costs by up to 60%, while giving creators a faster path from idea to playable experiences. Read on to learn more.

What if AI were a playground? Welcome to SeaVerse, a creation-first platform for playable AI experiences. Here, an AI creation can be as peaceful as drawing a path for a snake to follow, or as chaotic as a music-backed stickman simulation. Some people come to play lightweight games. Others come to chat with AI characters, try interactive apps, create visual patterns, share what they made, or remix an idea into something new.

We built SeaVerse around a simple promise: Every experience should feel immediate and easy to share. A creator should be able to describe an idea in plain language, refine the result, and publish it in moments, without a traditional coding workflow.

Delivering that simplicity requires serious infrastructure. Every creation that users make moves through the same chain: generate, run, preview, debug, publish, remix. If any part of that chain is slow, unstable, or poorly isolated, users feel it immediately. That’s why we turned to GKE and GKE Agent Sandbox.

The infrastructure challenge of instant interaction

What looks effortless to a user is anything but on our end. Every creation on SeaVerse runs as a distinct workload and is expected to behave reliably from the first interaction.

Because each workload runs in its own environment, we needed clear security boundaries between users, creations, and sandboxes. But overly strict isolation could slow the very creative loop we were trying to protect, and when something went wrong, diagnosing it was costly. Our engineers had to trace problems across multiple parts of the execution chain with little visibility into what was happening inside the environment.

We explored existing sandbox approaches, but needed deeper kernel-level isolation and native observability at scale to support fast diagnosis across multi-tenant environments. Something had to change.

Building on GKE and GKE Agent Sandbox

We chose GKE because we needed a reliable, secure way to operate Kubernetes without turning our engineering team into a cluster maintenance team. GKE brought together the proven ecosystem and operational tooling we needed, freeing us to focus on building the platform rather than managing the infrastructure beneath it.

As a Kubernetes primitive designed for agent code execution and computer use, GKE Agent Sandbox addressed our requirement for strong isolation, enforcing strong security boundaries without slowing down the creation experience. By utilizing GKE Agent Sandbox with Kata Containers+Cloudhypervisor (microVM), we’ve achieved the perfect balance of multi-cloud flexibility and robust security, option to switch isolation runtime between microVM and gVisor, running our AI sandboxes safely. GKE empowers us to scale toward our long-term vision of supporting over a million sandboxes. Built on gVisor, it provides kernel-level isolation for dynamic sandbox workloads while preserving the Kubernetes orchestration model, so that they can be managed through the same scheduling, monitoring, and operations as the rest of the cluster.

With SeaVerse, users can generate interactive experiences from a single prompt. After an experience is generated, GKE Agent Sandbox supports the run, test, integration, and verification steps needed to make it ready to preview, refine, and publish. At general availability, it supports allocating up to 300 sandboxes per second, per cluster, with 90% of allocations completing in 200 milliseconds. Together, GKE and GKE Agent Sandbox gave us a reliable foundation for AI-generated interactive workloads that helped keep our team focused on the product experience.

From black box to glass box

Before GKE Agent Sandbox, a failed sandbox workload could feel like flying blind. We could often see that something had gone wrong, but didn’t have enough runtime status, metrics, or failure signals to understand why.

Now, Google Cloud’s native logging and monitoring reach directly into those sandboxed environments, giving us a clearer view of workload behavior, faster issue resolution, and a stronger foundation for managing multi-tenant workloads.

That visibility matters to developers, but it also matters to the platform’s users: A creator never sees the logs, the cluster, or the orchestration layer. They see whether an experience opens quickly, whether it responds when they draw, click, chat, or share, and whether they can keep building without friction.

Flexibility that translates to savings

GKE Agent Sandbox also changed how we think about cost. Previously, running secure sandboxed environments meant stronger dependencies on specific server types, which limited how precisely we could match resources to each workload. With GKE Agent Sandbox, we can run secure, isolated workloads on appropriately sized cloud VMs. This gives us greater flexibility in resource allocation and helped us cut our infrastructure costs by up to 60%.

That same flexibility extended to storage. Not all SeaVerse creations are built in a single session. Some evolve over time as creators return to refine them, build on earlier ideas, or invite others to remix what they’ve made. Our previous architecture didn’t support the persistent file-system capabilities those more complex use cases demanded, but that gap is gone now. We can attach persistent storage where workloads require it while maintaining the isolation boundaries that multi-tenant AI experiences need. For creators, that means experiences that are fast to open and easier to refine, revisit, and build on over time.

The next remix

Supporting creations that can evolve and deepen is central to what we’re building. It’s still early in what playable AI can become. As the platform grows, we need to keep strengthening what matters most: stability, observability, elastic scaling, and cost efficiency, all in service of a creator experience that stays fast, reliable, and expressive.

We’re also exploring additional Google Cloud tools to support smarter analytics and creation assistance. Gemini and agent models could help operators and creators better understand how experiences perform. BigQuery AI and ML capabilities can support use cases such as churn prediction, LTV and ROI prediction, and user segmentation. Multimodal tools such as Imagen and Veo on Gemini Enterprise Agent Platform open up new possibilities for material analysis, creative generation, and AI interactive content production.

Our goal is to make AI experiences feel immediate, expressive, and connected. With GKE and GKE Agent Sandbox, we have a stronger foundation for the next generation of playable AI.

Introducing Filestore agent volumes: fully managed storage for agent workspaces

Tue, 15 Sep 2026 16:00:00 +0000

From running build tools, to data analysis pipelines, to collaborative research, executing data-driven tasks is essential for any enterprise agent.

Today, platform teams often stitch together custom workarounds to address agent storage requirements, which could include shuttling state back and forth between agent sandboxes and centralized storage or manually managing local disks and/or self-hosted file systems. However, as agent fleets scale, these approaches force difficult trade-offs between cold-start latency, operational complexity, and the cost of idle, pre-allocated storage.

As organizations scale agent sandboxes to thousands or even millions of concurrent sessions, storage must evolve to overcome these trade-offs and meet the needs of these dynamic workloads, which require strict workspace isolation, instant session resumption, elastic pay-per-use economics, and fluid multi-agent collaboration.

To meet these emerging demands, we’re expanding our AI storage portfolio and announcing availability of Filestore agent volumes, a new, fully managed capability purpose-built to deliver high-performance, elastic file storage for scaling agentic workloads on Google Cloud.

Purpose-built storage for AI agent workspaces

Autonomous agents require isolated runtime environments to safely execute dynamic code, install third-party packages, and run tools without putting host infrastructure or tenant data at risk. While Agent Substrate on GKE and GKE Agent Sandbox provide the dedicated compute environments needed to run high-density agent fleets, those sandboxes also need dedicated persistent workspaces to operate on.

Filestore agent volumes within Google Cloud Filestore, give you purpose-built agentic storage to complement your agentic compute via a dynamic provisioning architecture designed specifically for the scale and elasticity of AI agent fleets. Co-designed with Agent Substrate to support agentic fleets at scale, Filestore agent volumes provide GKE sandboxes with instantaneous access to isolated, persistent file storage. When configured to leverage Filestore, GKE storage management happens behind the scenes: Every time GKE launches a sandbox for a new agent task, Filestore automatically allocates and attaches a dedicated, isolated file workspace to that environment in milliseconds. Platform teams don't need to manually create, attach, or tear down storage volumes for individual agent runs; instead, the system handles the entire volume lifecycle automatically as your agent fleet scales up and down. The result is an efficient, end-to-end infrastructure solution for cost-effective agent management that provides:

Granular isolation and enterprise guardrails: Agent platforms face security and data leakage risks when running untrusted, autonomous code. Filestore agent volumes enforce strict boundary controls and granular access permissions per workspace, ensuring agents operate exclusively within their designated directories and keeping dynamic toolchains strictly isolated across tenants.
Sub-second session resumption: Traditional storage provisioning approaches can introduce cold-start latency that stalls interactive agent sessions. Agent volumes attach and detach in milliseconds, making it possible for orchestrators to aggressively suspend idle sandboxes to save compute costs, and resume instantly when new tasks or user inputs arrive.
Smart lifecycle economics and pay-per-use pricing: Pre-allocating fixed-size, high-performance storage for thousands of short-lived or intermittent agent tasks can create massive storage waste. With agent volumes, platforms pay only for the storage capacity consumed and benefit from automatic lifecycle tiering. This means you get high performance without wasted spend: When your agents aren’t actively reading/modifying code or analyzing datasets, you can automatically shift idle workspace state to lower-cost storage.
Multi-agent collaboration: Coordinating multi-agent swarms can result in brittle data-passing pipelines and risk of file collisions. Built with native Read-Write-Many (RWX) support and POSIX file locking, agent volumes allow orchestrators to attach a single shared workspace across multiple agents. Collaborating agents can safely co-author, test, and review project files concurrently with file-level consistency and protection against write conflicts.

Powering next-generation agentic workloads

By providing an elastic, high-performance, and isolated file tier, Filestore agent volumes unlock a wide spectrum of agentic workloads and use cases in production:

Software engineering and coding sandboxes: Agentic coding platforms can spin up thousands of isolated workspaces where agents safely install libraries, write multi-file patches, run build tools, and execute unit tests, all leveraging standard POSIX file semantics with no need for storage-specific customization.
Collaborative multi-agent swarms: Complex workflows, such as a lead orchestrator delegating tasks to dedicated research, code generation, and validation sub-agents, can directly share a unified file tree. RWX support allows agents to co-author and review project files concurrently without write conflicts.
Interactive long-horizon workflows: For user-in-the-loop applications (such as agents that require asynchronous user approval or run multi-hour data analysis pipelines), platforms can suspend idle agent sandboxes to minimize compute waste, then resume execution on demand with sub-second responsiveness.

Get started today

If you are building an Agent-as-a-Service platform, scaling coding assistants, or deploying enterprise agent fleets, your storage tier should accelerate your innovation — not hinder it.

Filestore agent volumes are now available to all Google Cloud customers for non-production workloads. GA support for production workloads is available via allowlist. This new offering features out-of-the-box integrations with Agent Substrate on GKE and GKE Agent Sandbox to help you build responsive, scalable, and cost-efficient agent platforms today.

To request access to Filestore agent volumes, submit this form and visit the Filestore documentation and GKE documentation to learn more.

Google is a leader in The Forrester Wave™: Public Cloud Platforms, Q3 2026

Mon, 14 Sep 2026 16:00:00 +0000

We are excited to share that Google Cloud was named a Leader and received the highest score in the ‘current offering’ category in the Forrester Wave™: Public Cloud Platforms, Q3 2026 report, which examines the 10 most significant public cloud providers across 30 comprehensive criteria, Google also received the highest possible score in 23 out of 30 evaluation criteria, including, but not limited to vision, innovation, AI development services, database services, analytics services, containers and kubernetes services, modernization services, and security services. We believe Forrester’s recognition confirms our belief that to lead in the agentic era, you need a complete, integrated platform that’s engineered from the ground up, from silicon to systems to models.

Access the complimentary report: The Forrester Wave™: Public Cloud Platforms, Q3 2026.

Build on co-designed infrastructure proven in global enterprises

For over a decade, our infrastructure engineers, application developers, and AI researchers worked side by side to co-design infrastructure to power Gemini, Search, YouTube, Maps, and Gmail. We couldn't simply buy the platform and infrastructure we needed; we had to invent it. This led to the creation of everything from TPUs, the Transformer architecture, Kubernetes, Axion, and now Gemini.

In the agentic era, you need an integrated AI stack, where compute, orchestration software, modernization tools, and global networks operate together to give you more value from your investments — even if you’re not working at the frontiers of AI research. At Google Cloud, we’ve worked tirelessly to bring these breakthrough innovations to leading enterprises, startups, and frontier labs to help them achieve new levels of scale and efficiency, and we believe Forrester’s evaluation validates that strategy:

“Google Cloud’s vision is to enable the ‘agentic enterprise,’ and AI already permeates its platform, positioning the company to push further up the tech stack toward business users who increasingly shape AI adoption in the enterprise. Google Cloud is a good fit for enterprises seeking rapid technology innovation and a broad AI-enabled cloud platform.” - The Forrester Wave™: Public Cloud Platforms, Q3 2026 report

Run agents quickly on a secure, flexible platform

Most traditional infrastructure can’t keep pace with agents, and enterprises need a scalable alternative. But you don’t want a new, greenfield platform just for AI agents. Kubernetes is the proven industry standard for modern enterprise applications — from microservices and transactional databases to real-time LLM inference. We are evolving Google Kubernetes Engine (GKE) and our operations tooling so organizations can scale autonomous agents alongside traditional workloads on a single, proven platform.

Forrester gave Google Cloud the highest scores possible in Container and Kubernetes services, Serverless/FaaS services, and Operations management services, noting:

“Operators will find strong offerings in operations management as well as containers and Kubernetes services. Our evaluation did not identify significant capability gaps.”

Over the past three months, we’ve enhanced our infrastructure portfolio to help teams scale agentic workloads with enterprise predictability. Recent updates let you:

Safely execute untrusted agent code alongside traditional workloads with default-deny security using GKE Agent Sandbox (GA) and Cloud Run Sandboxes (preview), which provision lightweight, gVisor-isolated boundaries for your agent in under a second (and up to 300 sandboxes/sec per cluster).
Eliminate up to 90% of idle compute costs by serializing your container RAM state directly to Google Cloud Storage with GKE Pod Snapshots, allowing you to suspend idle agent sessions in ~100ms and resume them in ~280ms.
Cut time-to-first-token (TTFT) up to 70% and double cache-hit rates with predictive routing in GKE Inference Gateway, which uses a continuously trained ML model to make routing decisions based on real-time traffic data.

Ground your agents with real-time enterprise data

Agents are only as effective as the context that grounds them. Traditional distributed data topologies separate operational databases from analytical systems through fragmented, multi-hop pipelines. In the agentic era, this divide introduces multi-hop latency, stale context, and governance friction.

Our Agentic Data Cloud evolves the enterprise data platform from a static repository into a dynamic reasoning engine. It unifies transaction processing and analytical intelligence into an active system of action, providing the real-time context and deterministic responsiveness that autonomous workflows require. Google received 5/5 scores across the Database services, Analytics services, Data integration services, and Data Governance services criteria:

“Google Cloud’s traditional strength in database services and analytics drives strong performance, including multicloud and hybrid capabilities, along with an Agentic Data Cloud that bridges analytics and transactional systems.”

Over the past three months, we’ve introduced key capabilities to the Agentic Data Cloud to help customers unify their data estates:

Enable agents to query live financial and supply chain records without costly data movement using SAP BDC Connect for BigQuery (GA), which provides bi-directional, zero-copy data sharing between your SAP systems and BigQuery.
Map and infer business meaning across your entire data estate with Knowledge Catalog. You can now aggregate native context across your Google and partner data platforms, semantic models, and third-party catalogs, unifying them into a single, governed source of truth.
Access live data from Iceberg and BigQuery from the PostgreSQL data plane with Lakehouse federation. Perform live joins between AlloyDB's transactional data and historical insights in BigQuery or Iceberg without any data movement. You can also replicate data continuously to BigQuery and, importantly, to Iceberg tables directly from AlloyDB with Datastream.

The benchmark is set: Build what’s next on Google Cloud

We are honored that Forrester has named Google Cloud a Leader in The Forrester Wave™: Public Cloud Platforms, Q3 2026. We believe this recognition validates decades of foundational research, disciplined full-stack co-design, and our commitment to building an open, reliable cloud.

The era of fragmented infrastructure has come to an end. Whether your organization is an AI research lab scaling models across one million accelerator chips, a global financial exchange settling trillions in clearing systems, or an enterprise empowering millions of users with autonomous workflows, Google Cloud delivers the performance, scale, security, and data foundation to build what’s next.

Take the next step in your cloud journey:

Download the full report: Read the complete analysis in The Forrester Wave™: Public Cloud Platforms, Q3 2026.

Dynamic capacity management for AI infrastructure

Wed, 26 Aug 2026 13:30:00 +0000

The internet connected billions of people and mobile devices, putting computers in every hand. Now, we’re in the middle of the next big technology shift, deploying millions of autonomous AI agents to work alongside employees and end users. Today, we announced new FinOps controls for Gemini Enterprise to help organizations manage project-level AI spend and eliminate token shock. But the sheer scale of the agentic era is placing new constraints at every layer of the stack, including infrastructure. AI workloads are notoriously difficult to architect, resource-intensive, and bursty, which can also lead to scaling bottlenecks and large pools of underutilized — or misutilized — compute resources.

Organizations need insights to help them extract more value from their infrastructure investments. In this blog, we outline best practices for dynamic capacity management — scheduling and utilization strategies to help you run enterprise and AI applications on a single, flexible foundation with predictable cost and performance. These capabilities are designed to augment our on-demand, Spot and committed use discount (CUD) consumption models, which provide flexible pricing and discounting for your workloads. Let’s jump in.

Here's a quick summary

Three ways you can implement dynamic capacity management:

Schedule capacity for planned events. Schedule mission-critical resources (GPUs, TPUs and select VM families) ahead of planned events using calendar mode, or optimize costs for batch jobs with flexible start times using flex-start mode in Dynamic Workload Scheduler. Once you obtain the capacity, those resources are guaranteed for the specified duration.
Maintain service continuity by creating a fallback plan for every application. Define automated, prioritized hardware fallback lists using managed instance groups (MIGs) so your apps automatically pivot to the next approved compute option when your preferred option isn’t available.
Automate your entire capacity management lifecycle on a single, adaptive control plane. Google Kubernetes Engine (GKE) provides an agent-native environment to orchestrate the entire process — from fallback lists using Custom ComputeClasses, to granular hardware slicing with dynamic resource allocation, so agents can rapidly spin up in secure sandboxes and containers while it dynamically reallocating resources on the fly.

Why architectural flexibility matters

Ninety percent of enterprises want to deploy agents within the next three years, but only 17% of IT leaders feel confident their current IT setup can handle the load. Because these workloads have unique performance needs, organizations are racing to adopt specialized infrastructure, including accelerators (GPUs, TPUs) and CPUs with customized compute, memory, and storage ratios. However, agents also require access to enterprise applications and databases — often at a volume and scale that vastly exceeds typical human usage. Handling the intense demands of both agents and the applications they interact with requires a dynamic infrastructure. Infrastructure teams can leverage custom-designed processors like Google’s Axion to meet these needs, but hardware isn’t a complete solution. They also need ways to use that infrastructure wisely, solving execution inefficiencies to enable more flexibility across the stack.

How to overcome infrastructure constraints

Achieving this kind of flexibility requires a two-pronged approach: securing resources for the demand you can predict, and building automation to respond to the demand you can't. Combining the two, you can preschedule capacity for planned events and your infrastructure can adapt to unexpected changes without manual intervention.

1. Schedule capacity for planned events

You can secure mission-critical capacity ahead of scheduled milestones, offline training, or anticipated demand surges using Dynamic Workload Scheduler. By scheduling the resources you need up front, you optimize your spend and ensure you get access to the compute resources you need. Dynamic Workload Scheduler supports hardware accelerators (TPUs and GPUs) and select CPUs with two distinct modes:

Flex-start mode: Use this for latency-tolerant workloads like batch processing, model training, or offline fine-tuning. Instead of requiring resources immediately, you submit a defined duration request and the system intelligently queues your job, provisioning the resources as soon as capacity becomes available. This maximizes cost-efficiency and drastically improves your ability to obtain high-demand accelerators.
Calendar mode: Use this for mission-critical, time-bound events like a major product launch, a scheduled migration, or a seasonal traffic surge. By specifying the exact start and end dates of your event, you create a future reservation. This guarantees the requested capacity will be available when the event begins.

2. Maintain service continuity by creating a fallback plan for every application

Not every spike in traffic is predictable. You also need to plan for unexpected traffic from, say, a breaking news cycle or a sudden market shift that drives a surge in user activity. To help your services get the resources they need without interruption, you need a fallback plan — an automated, prioritized sequence of acceptable hardware configurations. This strategy:

Decouples your workloads from a single VM shape, size, or configuration. This allows them to run without manual intervention if your preferred option is unavailable
Allows you to execute a progressive tech refresh by adopting the newest VM generations as your primary choice while keeping older generations as an automatic fallback option.

If you run non-containerized workloads on Google Compute Engine, you can dynamically manage capacity with instance flexibility in managed instance groups (MIGs) and bulk VM creation. Instance flexibility lets you specify multiple machine types for your VM instances rather than being limited to a single machine type.

How it works: If your preferred machine type is temporarily unavailable, the MIG automatically provisions a compatible alternative from your list based on real-time capacity. When combined with location flexibility — by specifying multiple zones your MIGs can search within a region — you can drastically improve your provisioning success rate. If your MIGs use Spot VMs, Compute Engine automatically integrates with Spot capacity signals to prioritize machine types that offer longer estimated uptimes and lower risk of pre-emption.

You can also extend instance flexibility to your block storage layer by setting baseline disk defaults and configuring disk overrides so your storage adapts when a VM falls back to a different machine type.

How it works: Most of the time you can simply rely on our default options, omitting ‘disk type’ from the instance template entirely. However, for data disks that will outlive their associated VMs, it’s possible to enable a fast, durable Hyperdisk across multiple VM generations.

While Compute Engine provides instance flexibility for organizations working with virtual machines, GKE goes a step further and automates the entire capacity lifecycle from a single control plane. With GKE custom ComputeClasses, platform teams can design multi-dimensional fallback lists, automatically combine different VM machine families, sizes, and ratios, scale across multiple zones, and shift between on-demand and Spot VMs. By using Dynamic Workload Scheduler as a capacity target, and custom ComputeClasses to define the policy and priority, you can fully automate the capacity management lifecycle.

How it works: Once you’ve set up ComputeClasses, GKE automatically detects when a preferred node configuration is unavailable and falls back to your pre-approved alternative options in order of priority. When active migration is enabled, GKE gracefully migrates workloads back to higher-priority node configurations as capacity becomes available. For short-lived disks such as boot disks, GKE dynamically picks the right defaults based on the instance family. However, for long-term disks that will outlive the VM, you can use Hyperdisk.

Another GKE feature, dynamic resource allocation, helps eliminate wasteful, all-or-nothing hardware assignments by letting developers define advanced rules that dictate how resources are consumed.

How it works: Instead of claiming an entire GPU or TPU, your application specifies its exact parameters — such as total memory or number of cores — and the system allocates the perfect slice of hardware, helping to maximize utilization and reduce costs.

Take the next step toward dynamic infrastructure

Scaling AI shouldn’t mean linearly scaling your infrastructure budget or accumulating more tech debt. As these examples show, the right tools can help you overcome constraints and dramatically alter the value you get from your compute investments. Here are three steps to get started:

Audit your workloads for immediate cost-savings: Identify any applications currently tightly coupled to a single VM family, machine type, or availability zone, and map out viable alternative hardware shapes. Look beyond your existing configurations to evaluate new compute options that might better serve or act as alternatives based on your workload-level objectives. Then use Compute Engine MIGs, bulk VM creation or GKE Custom ComputeClasses to adopt them automatically, integrating them into your fallback lists.
Commit to a minimum spend for deeply discounted prices: Receive automatic discounts for sustained use, or up to 63% off when you sign up for Compute flexible committed use discounts, where your discount is tied to the resources you use regardless of the specific machine type or location.
Engage your account team: Reach out to your Google Cloud account team to craft a tailored capacity management strategy and configure your automated fallback lists.

Bringing gVisor sandboxes to distributed Ray clusters

Tue, 25 Aug 2026 16:00:00 +0000

The reinforcement learning (RL) ecosystem is rapidly adopting Ray as the unified compute runtime for complex post-training workflows. Across Google Cloud, we see customers using Ray for workloads ranging from multimodal data pipelines to frontier RL. But as agentic and reasoning models evolve, a critical bottleneck has emerged: orchestrating secure, isolated sandboxes at scale to safely execute dynamic rollouts, code generation, and multi-turn tool interactions. Today, in partnership with Anyscale, we are excited to introduce an experimental library for Ray that leverages agentic AI technologies being developed at Google to bring native, high-performance sandboxing directly into distributed Ray clusters.

Sandboxes as Ray Primitives

Ray has become a common runtime for orchestrating post-training workloads. Frameworks including veRL, NeMo-RL, SLIME, MILES, and SkyRL already use Ray to coordinate distributed trainers, inference engines, rollout workers, and other components.

When we designed Ray Sandboxing, an important goal was to make it fit naturally into the existing Ray programming model rather than introduce a separate abstraction for isolated execution. A sandbox has many of the same properties as other resources managed by Ray: it needs to be placed on a machine, assigned resources, created and destroyed, recovered from failures, and scaled with the surrounding workload. This led us to represent each high-level sandbox through a Ray Actor:

The Ray scheduler decides which node should run a sandbox and reserves the corresponding CPU and memory resources. The sandbox Actor manages its lifecycle, while gVisor provides the isolated execution environment on that node.

Starting in Ray 2.58, framework authors and researchers can manage sandboxed environments using the same Ray APIs and patterns they already use for the rest of their workload. For example:

code_block: <ListValue: [StructValue([('code', 'import ray\r\nfrom ray.experimental import sandbox\r\n\r\nray.init()\r\n# Create a gVisor sandbox environment and return an actor handle for a proxy actor\r\nsb = sandbox.create(\r\n cpu=1.0,\r\n memory="512Mi",\r\n image="python:3.12-slim"\r\n)\r\n# Execute code inside the sandbox\r\nresult = ray.get(sb.exec.remote("python -c \'import sys; print(sys.version)\'"))\r\nprint(result.stdout)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fdb5b3c7e10>)])]>

This creates a gVisor sandbox from an OCI-compatible image and returns a Ray Actor handle. Calls to exec are normal Ray Actor calls, so the sandbox can live anywhere in the cluster. The created actor is a proxy that will forward the operations to gVisor.

The sandbox API covers the basic lifecycle needed by agentic workloads:

Create environments from OCI container images
Set CPU and memory limits
Configure environment variables, working directories, and networking
Execute commands
Read, write, upload, and download files
Inspect sandbox state
Terminate or delete environments.

For lower-level use cases, SandboxRuntime provides direct access to local gVisor sandboxes and lets users modify the OCI specification before it is handed to gVisor. Here is an example how this API can be used to build a pool of local sandboxes inside of an actor:

code_block: <ListValue: [StructValue([('code', 'import ray\r\nfrom ray.experimental.sandbox.runtime import SandboxRuntime\r\n\r\n@ray.remote\r\nclass SandboxPool:\r\n def __init__(self, size: int = 3, image: str = "python:3.10-slim"):\r\n self.runtime = SandboxRuntime()\r\n self.sandboxes = [\r\n self.runtime.create(image=image, memory="512Mi")\r\n for _ in range(size)\r\n ]\r\n\r\n def run_command(self, index: int, command: str):\r\n return self.runtime.exec(self.sandboxes[index], command)\r\n\r\n def close(self):\r\n for sb_id in self.sandboxes:\r\n self.runtime.delete(sb_id)\r\n\r\n# Deploy an actor managing a pool of local sandboxes\r\npool = SandboxPool.remote(size=3)\r\nresult = ray.get(pool.run_command.remote(0, "python3 -c \'print(\\"Hello from pool!\\")\'"))\r\nprint(result.stdout)\r\nray.get(pool.close.remote())'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fdb5b3c5610>)])]>

Why gVisor?

Running model-generated code means treating the code inside the environment as untrusted. Ray Sandboxing uses gVisor, Google's open-source application kernel, as its initial sandbox runtime. gVisor implements a substantial portion of the Linux system-call interface in userspace, putting an additional isolation boundary between workloads and the host kernel. It is OCI-compatible, works with standard container images, and does not require exposing a Docker daemon or host Docker socket to the sandbox.

This combination is particularly useful for agentic workloads: environments remain lightweight enough to create dynamically while providing stronger isolation than executing generated code directly in ordinary containers. gVisor also provides sub-second sandbox startup and low per-sandbox memory overhead, making it possible to use sandboxes as relatively fine-grained distributed resources.

In future versions of Ray, we plan to extend support to other sandboxing runtimes such as Agent Substrate or Kata Containers.

Try Ray sandboxing on GKE

Check out the Ray documentation to learn more about Ray Sandboxes. To try out these sandboxing capabilities on GKE, head over to the Ray sandboxing User Guide. Have feedback or ideas? Join the discussion on the GitHub issue to collaborate on the future of Ray for reinforcement learning.

Empowering autonomous agents with advanced security governance

Mon, 24 Aug 2026 16:00:00 +0000

AI agents are the ultimate insiders. We grant them permission to read emails, query databases, and trigger API calls. They don’t just retrieve information, they take action.

Agents offer incredible potential for increased productivity and better customer experiences, but they also come with new security concerns. In our new State of AI infrastructure report, 79% of tech leaders cite security, governance, or operations as their most significant challenge to scaling inference.

While there’s still a crucial role for traditional security tools, the threat model has fundamentally changed. Autonomous workflows have redefined enterprise risk, so it's crucial that we give agents the access they need without compromising security.

The agentic paradox

The path to success starts with viewing governance as a driver for innovation. To be useful and secure, an agent needs access — and also guardrails. Yet 35% of senior IT decision makers cite insufficient security for multi-system access as a primary issue preventing agentic deployment.

Agents expand the surface area that defenders need to protect, and can introduce new threats, including tool poisoning and indirect prompt injection, where an attacker can hijack an agent’s logic through the data it processes. Managing the dynamic permissions that agents need to succeed at their tasks can also be a significant challenge, particularly as legacy security wasn’t designed for today’s automated threats.

Securing the chain of thought

Along with securing more identity and access issues, it’s important for defenders to secure both the network layer and the model.

Security leaders are increasingly shifting their focus from preventing breaches to verifying provenance to guard against misuse, including indirect prompt injection.

From an infrastructure perspective, what are your top security concerns related to AI?

From blocking to managing

We’ve looked at the new security challenges posed by agentic AI. You can’t solve them by simply locking down the system, as that defeats the purpose of autonomous agents.

Many organizations are turning to integrated, full-stack cloud platforms to give them greater oversight. 69% of surveyed executives now rate a full-stack platform as a critical requirement, and 80% say data compliance is the primary factor dictating that choice.

By adopting frameworks like the Secure AI Framework (SAIF) and moving to a central control plane, purpose-built platforms such as Gemini Enterprise Agent Platform, organizations can manage risk in three main areas:

Secure-by-default design: Embedding security directly into the AI development process to proactively guard against threats including prompt injection.
Agent governance and oversight: Adopting purpose-built permission and identity management for agents — giving greater control over agent interactions, exposing blind spots and limiting risks tools.
Human-in-the-loop control: Enforcing clear rules that automatically flag when an agent requires human approval before moving forward with a critical action.

Governance will guide you to success

The true value of a modern security foundation is its ability to encourage innovation. By embedding robust governance directly into a unified foundation, organizations can deploy agents with confidence across their most sensitive, business-critical workloads.

The leaders of the agentic era are re-architecting their stack to use security as a launchpad — empowering them to innovate securely and scale faster than their competition.

Find out more about how enterprise leaders are rethinking security for the agentic era in the State of AI infrastructure report.