Developers & Practitioners

GKE CPU startup boost: Accelerate app starts without over-provisioning

Fri, 02 Oct 2026 16:00:00 +0000

Whether you’re launching microservices in response to sudden traffic spikes, deploying new software releases, or scaling up application replicas, pod startup time is critical to maintaining a fast, responsive user experience for applications running on Google Kubernetes Engine (GKE).

Yet, platform engineers and developers face a persistent dilemma: Applications often demand significantly more CPU power during startup than they do during steady-state operations. Sizing CPU requests for normal, steady-state usage leads to CPU throttling during launch, which can result in sluggish cold starts and readiness probe timeouts. On the flip side, over-provisioning baseline CPU requests to satisfy short-lived startup bursts wastes valuable compute resources, inflating infrastructure bills.

Today, we are excited to announce CPU startup boost for GKE in preview. Integrated directly into GKE's Vertical Pod Autoscaler (VPA), CPU startup boost dynamically elevates a container's CPU allocation during initialization and seamlessly scales it back to baseline steady-state levels once the application is ready - all without restarting your containers.

Why modern applications need extra CPU at boot time

When a new container launches, it may perform intensive initialization tasks before it begins serving user requests. Depending on your tech stack, the following startup workloads require substantial CPU cycles:

Java JVM applications: Frameworks like Spring Boot require high CPU burst capacity for class loading, classpath scanning, instantiating dependency injection containers, and running Just-in-Time (JIT) compilation.
Node.js servers: Apps parse JavaScript files, build complex module dependency trees (require/import), and execute V8 engine optimization and JIT compilation passes during initial execution.
Python and AI/ML microservices: These services spend initial cycles importing heavy libraries (such as PyTorch, NumPy, or LangChain), compiling .pyc bytecode, establishing ORM database schemas, and pre-loading cache structures.

If you size CPU requests strictly for steady-state performance, these initialization workloads experience CPU throttling on launch, delaying readiness probes. To prevent slow cold starts, teams frequently overprovision CPU requests. However, once the application stabilizes, those extra CPU resources sit idle, increasing your cloud spend without adding value.

How CPU startup boost can help

CPU Startup Boost solves this by giving your workloads temporary vCPU "boosts" during launch, and automatically returning them to baseline once initialization completes.

Key benefits:

Faster cold starts: Reduce application initialization times by up to 2x, accelerating auto-scaling responsiveness during unexpected traffic surges.
Optimized cloud spend: Right-size steady-state CPU requests to fit actual runtime needs rather than paying for idle startup headroom.
Zero pod restarts: Dynamic resource resizing happens live inside the running container.
Flexible policy controls: Apply simple pod-level multiplier factors (e.g., 2x CPU during startup) or define granular, container-specific rules for complex multi-container pods.

Under the hood: Kubernetes In-place Pod Resize

Historically, changing a pod's resource requests or limits required deleting and recreating the pod. This disruptive process triggered container restarts, cache invalidation, and node rescheduling overhead.

To fix that, CPU startup boost builds on Kubernetes In-place Pod Resize (IPPR).

Tracked under KEP-1287, IPPR introduced dynamic, in-place resource mutation. Introduced as Alpha in Kubernetes 1.27, promoted to Beta in v1.33, and graduating to General Availability (GA) in v1.35, IPPR allows the Kubernetes control plane and kubelet to update container CPU and memory requests on running pods without restarting the container process.

GKE leverages IPPR within the VPA to apply startup CPU boosts at pod admission and smoothly step them down post-readiness.

How CPU startup boost works (pod lifecycle overview)

CPU startup boost operates across three distinct phases:

Admission phase: When you deploy a pod, the VPA admission webhook intercepts the creation request. It calculates the elevated CPU request based on your policy (e.g., 2x multiplier or +2 vCPUs) and injects the boosted CPU request along with tracking annotations into the pod spec.
Startup phase: The pod is scheduled and initialized with the boosted CPU allocation. Your application completes class loading, JIT compilation, or module parsing at top speed without experiencing CPU throttling.
Unboosting phase: As soon as the pod's readinessProbe passes (plus any configured durationSeconds cooldown delay), the VPA updater issues an in-place resize request. The CPU request steps back down to your baseline level while the container continues running uninterrupted.

Prerequisites and availability

CPU startup boost is available today in preview on GKE:

GKE version: Version 1.36.0-gke.4447000 or later on Standard and Autopilot clusters.
Cluster Modes: Enabled natively on GKE Autopilot (VPA is active by default). On GKE Standard, simply ensure Vertical Pod Autoscaling (VPA) is enabled.
Workload Support: Works with standard Kubernetes controllers, including Deployments and StatefulSets.

Getting started: Configuring CPU startup boost

Configuring CPU startup boost is as simple as adding a startupBoost section to your VerticalPodAutoscaler manifest.

Example 1: Pod-level boost with fixed steady-state (`updateMode: "Off"`)

If you want to use VPA purely for startup boost while keeping steady-state CPU requests locked to your manifest definitions, set updateMode to "Off":

code_block: <ListValue: [StructValue([('code', 'apiVersion: "autoscaling.k8s.io/v1"\r\nkind: VerticalPodAutoscaler\r\nmetadata:\r\n name: java-app-startup-boost\r\n namespace: default\r\nspec:\r\n targetRef:\r\n apiVersion: "apps/v1"\r\n kind: Deployment\r\n name: java-app\r\n updatePolicy:\r\n updateMode: "Off"\r\n startupBoost:\r\n cpu:\r\n type: "Factor"\r\n factor: 2\r\n durationSeconds: 10'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e67a9af10>)])]>

In this example, GKE doubles the container's CPU request during launch and holds the boosted allocation for 10 seconds after readiness probes pass before scaling back to baseline.

Example 2: Combining startup boost with continuous VPA auto-scaling

If you want GKE to boost CPU during launch and continuously optimize steady-state resources post-startup, set updateMode to "InPlaceOrRecreate":

code_block: <ListValue: [StructValue([('code', 'apiVersion: "autoscaling.k8s.io/v1"\r\nkind: VerticalPodAutoscaler\r\nmetadata:\r\n name: nodejs-app-vpa\r\n namespace: default\r\nspec:\r\n targetRef:\r\n apiVersion: "apps/v1"\r\n kind: Deployment\r\n name: nodejs-service\r\n updatePolicy:\r\n updateMode: "InPlaceOrRecreate"\r\n startupBoost:\r\n cpu:\r\n type: "Factor"\r\n factor: 3\r\n durationSeconds: 15'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e67a9a450>)])]>

Example 3: Granular container-level boost

For pods running sidecars (such as logging agents or service mesh proxies) that do not require extra CPU on boot, target specific app containers:

code_block: <ListValue: [StructValue([('code', 'apiVersion: "autoscaling.k8s.io/v1"\r\nkind: VerticalPodAutoscaler\r\nmetadata:\r\n name: app-container-boost\r\nspec:\r\n targetRef:\r\n apiVersion: "apps/v1"\r\n kind: Deployment\r\n name: API-gateway\r\n updatePolicy:\r\n updateMode: "Off"\r\n resourcePolicy:\r\n containerPolicies:\r\n - containerName: "web-app"\r\n mode: "Off"\r\n startupBoost:\r\n cpu:\r\n type: "Quantity"\r\n quantity: "2"\r\n durationSeconds: 5'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e67a995d0>)])]>

Verifying startup boost in your cluster

You can verify that GKE applied and downscaled the startup boost using kubectl:

1. Inspect pod annotations: Check for the vpaCpuStartupBoost tracking annotation:

code_block: <ListValue: [StructValue([('code', 'kubectl get pod <POD_NAME> -o yaml'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e67a99190>)])]>

Look for annotations indicating the original baseline and boosted CPU requests.

2. Monitor in-place resize events: Confirm that GKE downscaled the CPU request back to baseline after readiness:

code_block: <ListValue: [StructValue([('code', 'kubectl get events --field-selector reason=InPlaceResizedByVPA'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e67a9b390>)])]>

An event with reason=InPlaceResizedByVPA confirms successful in-place downscaling post-startup.

Best practices for production workloads

Pairing with Horizontal Pod Autoscaler (HPA): When using HPA alongside CPU startup boost, ensure a robust readinessProbe is defined and keep durationSeconds short (e.g., 0s–10s). This prevents HPA from falsely interpreting initialization CPU spikes as high steady-state load.
Handle traffic spikes with GKE capacity buffers API: By reducing the startup tax at scale, you can achieve higher workload density on fewer nodes to improve overall utilization. Consider adopting the GKE capacity buffers API to absorb sudden traffic surges with minimal operational overhead while maintaining strict SLOs.
GKE Autopilot resource ratios: On GKE Autopilot, remember that pods must maintain valid CPU-to-memory ratios. Ensure baseline memory allocations accommodate the boosted CPU ratio during startup.

Get started today

CPU startup boost gives GKE users the best of both worlds: lightning-fast cold starts for CPU-intensive workloads like Java, Node.js, and Python, paired with maximum resource efficiency and lower cloud costs.

Ready to accelerate your GKE workloads?

Explore the GKE CPU startup boost documentation.
Learn more about Vertical Pod Autoscaling on GKE.
Try CPU startup boost on your GKE Standard or Autopilot clusters running GKE 1.36.0-gke.4447000 or later!

Democratizing Managed Lustre with lower cost and frictionless development

Thu, 01 Oct 2026 13:00:00 +0000

This is the first of a two-part series exploring how Google Cloud is bringing the foundational values of a high-performance parallel filesystem–TB/s throughput, sub-ms latency at high client scale, and POSIX support–to a broader set of use cases and users.

Historically, due to the cost and special purpose nature of parallel filesystems, colder data had to be stored outside of the filesystem and AI developers have had to maintain separate, slower environments for writing code, compiling libraries, and managing repositories. This fragmentation increases the toil of manual data staging, dataset copying, and managing disjointed namespaces.

Google Cloud Managed Lustre is solving these problems through our 6 cents/GB*month Dynamic Tier and by optimizing Managed Lustre performance for a range of development tasks and workloads – making Managed Lustre a “One-Stop Shop” for high-performance AI and HPC workloads.

Lower Cost: More Lustre for Less with the Dynamic Tier

The Managed Lustre Dynamic Tier provides sub-ms latency for hot data, which allows you to store all of your data in a single namespace, and costs only 6 cents/GB*month.

Throughput, capacity scale and client scale: Throughput scales linearly with capacity up to 80 PB, while sub-ms latency for hot data remains stable as you scale to tens of thousands of clients.
Single-flat fee: Predictable pricing. No independent charges for disk media types, data movement within the namespace, or metadata IOPS.
Read Latencies: Sub-ms latencies for High-Performance Cache (SSD). The Capacity Pool (“HDD”) is built on Google Cloud Hyperdisk throughput, which has an average read latency of 10 to 30 ms.

Recommended workloads for Dynamic Tier

Multi-Epoch Training and/or Training with Optimized Fetch Sizes: Hot data is promoted to the High Performance Cache (SSD) after the first run. Larger data prefetch will allow you to take advantage of the Dynamic Tier cost structure and gain from low-latency SSD.
Write-Heavy Checkpointing: Bursty checkpoint writes land directly in the High Performance Cache. Older checkpoints are transparently demoted to the Capacity Pool (HDD).
Rapid Checkpoint Restore: New checkpoints are written to the High Performance Cache, enabling low-latency checkpoint restores.
Interactive Snappiness for Developers: Low-latency tasks like git cloning, compiling libraries, or running notebooks benefit from a local-disk feel (~300µs average read latencies) on the same shared workspace hosting large training sets.

Frictionless development: Lustre as a one-stop shop for developer’s workloads

In addition to Managed Lustre’s scalability for large AI and HPC workloads (checkpoint/restart/data-loading), it also meets the demands for interactive work, meaning developers can start on Managed Lustre and stay on Managed Lustre throughout the entire workload lifecycle:

Unified Foundation & Interactive Performance

Consolidates the AI and HPC lifecycle into a single namespace, providing a "local disk" feel for interactive work (Read more about the latency benefits of Managed Lustre experienced by Salesforce and others).

Latency: ~300µs average read latency—delivering up to 4x better responsiveness than alternative distributed file systems.
Accelerated Setup: Untar the Linux kernel in ~2 minutes (4.7x faster than alternative file solutions), run a 20-worker parallel git clone of Python in ~40 seconds, compile Python in ~200s.

High-Concurrency Broadcast & Cluster Startup

Managed Lustre maximizes GPU ROI by preventing storage bottlenecks during cluster initialization.

When thousands of worker nodes attempt to read the exact same file simultaneously (such as a shared model checkpoint, base weights, or container layer), traditional distributed file systems can choke on localized hotspotting, leaving high-cost GPU clusters idle for minutes.

Improves Aggregate Throughput for a large number of clients reading the same file: Demonstrates a 67% improvement over alternative file solutions.
Parallel Loading: Imports libraries like PyTorch across 4,000+ processes in under 60 seconds.

Run One-Stop Shop Workflows for Yourself

Here is the code for the tests we’ve run, so that you can perform your own testing.

Low latency for interactive access

We used fio to emulate small, low-concurrency reads and writes:

¹Storage system specs: 500 MBps per TiB tier of Managed Lustre, 108,000 GiB capacity. Zonal Filestore at 102,400 GiB capacity. Average throughput of 36.7 GB/s to 2,048 client VMs reading the same 40 GiB file.

code_block: <ListValue: [StructValue([('code', '# Read workload\r\nfio --ioengine=libaio --filesize=100M --ramp_time=2s \\\r\n --runtime=2m --time_based --numjobs=1 --direct=1 --verify=0 --randrepeat=0 \\\r\n --group_reporting --directory=~/LUSTRE_MOUNT \\\r\n --name=randread --blocksize=4k --iodepth=1 --readwrite=randread \\\r\n --buffer_compress_percentage=50\r\n\r\n# Write workload\r\nfio --ioengine=libaio --filesize=100M --ramp_time=2s \\\r\n --runtime=2m --time_based --numjobs=1 --direct=1 --verify=0 --randrepeat=0 \\\r\n --group_reporting --directory=~/LUSTRE_MOUNT \\\r\n --name=randwrite --blocksize=4k --iodepth=1 --readwrite=randwrite \\\r\n --buffer_compress_percentage=50'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e7414c5d0>)])]>

Accelerated setup

How to run Linux untar

code_block: <ListValue: [StructValue([('code', '# Download a kernel tarball\r\nwget -P /tmp https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.18.9.tar.xz\r\n\r\n# Extract the archive to the Lustre mount\r\nmkdir ~/LUSTRE_MOUNT/kernel\r\ntar -C ~/LUSTRE_MOUNT/kernel -xf /tmp/linux-5.18.9.tar.xz'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e7414ef50>)])]>

In the above use case, you will want to take care to avoid the metadata performance tax that can come from running as root (Namely, tar issues chown and chmod calls to make extracted files’ owner+permissions match the ones recorded in the archive.). If you still wish to run as root (and have verified that this approach is compatible with your setup), you may specify `--no-same-owner --no-same-permissions` in order to ensure that extracted files maintain root as owner and have root's default file permissions. In other words, it makes extraction as root behave like extraction as non-root (by ignoring the owner+permissions in the archive).

How to run Python gitclone

code_block: <ListValue: [StructValue([('code', 'git config --global checkout.workers 20\r\nmkdir ~/LUSTRE_MOUNT/python\r\ngit clone https://github.com/python/cpython.git ~/LUSTRE_MOUNT/python'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e6767ca90>)])]>

How to run Python compile

code_block: <ListValue: [StructValue([('code', 'pushd ~/LUSTRE_MOUNT/python\r\n./configure > /dev/null\r\nmake > /dev/null\r\npopd'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e6767f490>)])]>

High scale distribution

Aggregate throughput for distributing one large file to many nodes

Run the below on each client VM:

code_block: <ListValue: [StructValue([('code', '# Start fio in server mode\r\nfio --server'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e6767c510>)])]>

Run the below on a selected client VM:

code_block: <ListValue: [StructValue([('code', "# Create a 40 GiB file\r\nfio --name=job1 \\\r\n --ioengine=libaio \\\r\n --direct=1 \\\r\n --buffer_compress_percentage=50 \\\r\n --blocksize=4m \\\r\n --iodepth=32 \\\r\n --filesize=40g \\\r\n --readwrite=write \\\r\n --filename ~/LUSTRE_MOUNT/40gb_test\r\n\r\n# Create an fio job file for the read workload\r\ncat <<'EOF' > /tmp/read.fio\r\n[job1]\r\nfilename=${HOME}/LUSTRE_MOUNT/40gb_test\r\nrw=read\r\nbs=4m\r\nexitall_on_error=1\r\nEOF\r\n\r\n# Run the read workload using all client VMs in ~/hostfile\r\nfio --client ~/hostfile /tmp/read.fio"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e6767e210>)])]>

Parallel loading of libraries across many processes

Run the below on a selected client VM:

code_block: <ListValue: [StructValue([('code', '# Install PyTorch in a virtual env\r\npython3 -m venv ~/LUSTRE_MOUNT/env\r\nsource ~/LUSTRE_MOUNT/env/bin/activate\r\npip3 install --upgrade pip\r\npip3 install torch torchvision torchaudio \r\ndeactivate\r\n\r\n# Import PyTorch on all client VMs in ~/hostfile, 4 processes per host\r\nmpirun --allow-run-as-root --oversubscribe --hostfile ~/hostfile -N 4 \\\r\n bash -c \'source ~/LUSTRE_MOUNT/env/bin/activate && python3 -c "import torch"\''), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e6767d0d0>)])]>

Looking ahead and next steps

By eliminating the manual data staging tax and lowering entry costs with the Dynamic Tier, Google Cloud Managed Lustre is evolving from an elite, single-purpose engine into a highly versatile, unified storage fabric for the entire AI lifecycle.

In the second part of this series, we will focus on upcoming object integration features. Stay tuned!

Next steps

Run the benchmarks yourself (if you haven’t already): Deploy a Google Cloud Managed Lustre instance using the Google Cloud console and run tests provided above to benchmark your own workloads.
Explore the Dynamic Tier: Read the Google Cloud Managed Lustre Documentation to learn more.
Stay tuned for Part 2: In the next installment of this series, we will dive deep into upcoming object integration features and how they further simplify AI and HPC storage.
Get started with centralizing your development-to-training lifecycle on Google Cloud Managed Lustre!

Introducing the Server Side Cloud Swift SDK

Thu, 01 Oct 2026 04:00:00 +0000

For years, Swift was perceived mainly as a UI language tied to Apple client devices. With Swift 6 and strict concurrency checking, it has matured into a viable systems and cloud language, pairing Rust-like data-race safety with predictable, reference-counted performance.

To support this ecosystem, Google engineering has launched the official Google Cloud API Client Libraries for Swift. Built from the ground up for Swift 6.2+, this new SDK uses the latest non-blocking Swift NIO event loops, HTTP/2 multiplexing, gRPC transport, and zero-cost compile-time data race safety.

In this article, we'll walk you through all you need to know to get started, and to understand how the Server Side Cloud Swift SDK, or google-cloud-swift, works.

The rise of server-side Swift and cloud-native concurrency

Traditional backend development often forces a trade-off between developer ergonomics and resource utilization. While managed runtimes offer rapid development, lower-level systems languages provide finer control over memory and CPU footprint, often at the expense of feature velocity.

Server-side Swift aims to strike a practical balance. Swift pairs a lightweight runtime and Automatic Reference Counting (ARC) with expressive syntax. More importantly, Swift 6 introduces compile-time concurrency checking. When you share state between async tasks across a cloud microservice, the compiler enforces that types conform to Sendable. Data races are caught in your editor before a binary ever compiles or reaches production.

At the network layer, every request to Google Cloud APIs runs over event-driven, non-blocking sockets that scale across multicore Linux server environments without spawning system threads per connection.

Where to use the Swift SDK

The Server Side Cloud Swift SDK is engineered for server, container, and automated DevOps environments.

When you build high-throughput microservices with Swift web frameworks like Hummingbird or Vapor, google-cloud-swift provides native access to Cloud Storage, AI, Identity and Access Management (IAM), and over one hundred other Google Cloud services. You can containerize your executable on Linux and deploy directly to Cloud Run, Google Kubernetes Engine (GKE), or Compute Engine VMs.

Because the SDK compiles on macOS, and Linux, you can develop the backend in your preferred development environment, and then seamlessly deploy to production. And using Swift on both the frontend and backend allows you to share application-specific types across both.

The SDK also excels at platform engineering and DevOps automation. You can author cross-platform CLI utilities and data rotation scripts that run on your developer laptop or inside CI/CD pipelines. These tools authenticate automatically against Google Cloud using Application Default Credentials (ADC) or Workload Identity Federation.

If you're building an iOS, iPadOS, or visionOS app for the Apple App Store, you should not embed google-cloud-swift directly into your client bundle. Shipping Google Cloud service account keys or administrative credentials inside a client binary creates security risks. For direct client-side features, use the Firebase SDK for Apple Platforms to handle user authentication, real-time Firestore sync, and client-side security rules, or route requests through your own Cloud Run backend API.

Getting started with your IDE and packages

Because google-cloud-swift treats Linux and macOS as first-class citizens, you can develop on Apple hardware with Xcode or on Linux workstations with Visual Studio Code and swiftly.

To install the official Swift compiler on Linux workstations using the swiftly CLI installer, run:

code_block: <ListValue: [StructValue([('code', 'curl -O https://download.swift.org/swiftly/linux/swiftly-$(uname -m).tar.gz &&\r\ntar zxf swiftly-$(uname -m).tar.gz &&\r\n./swiftly init --quiet-shell-followup &&\r\n. "${SWIFTLY_HOME_DIR:-$HOME/.local/share/swiftly}/env.sh" &&\r\nhash -r'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e743ef3d0>)])]>

Alternatively, you can download prebuilt toolchain tarballs directly from official Swift Downloads for Ubuntu, Debian, Fedora, or Amazon Linux. Note that google-cloud-swift requires Swift 6.2 or later, so verify your compiler version with swift --version after installation.

To resolve Swift Package Manager bare repository trust warnings when cloning across Linux filesystems, configure Git before building with git config --global safe.bareRepository all.

Add the required packages to your Package.swift manifest:

code_block: <ListValue: [StructValue([('code', 'swift package add-dependency https://github.com/googleapis/swift-google-cloud-language-v2.git --from 0.4.0\r\nswift package add-target-dependency GoogleCloudLanguageV2 CloudBackendService --package swift-google-cloud-language-v2'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e743ef8d0>)])]>

On macOS you need to change the platforms directive:

code_block: <ListValue: [StructValue([('code', '// swift-tools-version: 6.2\r\nimport PackageDescription\r\n\r\nlet package = Package(\r\n name: "CloudBackendService",\r\n // Applied when compiling on Darwin/macOS; ignored by SPM on Linux targets\r\n platforms: [.macOS(.v15)],\r\n ... ...'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e743ed990>)])]>

In most environments a default-initialized client can make requests:

code_block: <ListValue: [StructValue([('code', 'import Foundation\r\nimport GoogleCloudLanguageV2\r\n\r\nfunc analyzeTextSentiment(text: String) async throws {\r\n // Initialize explicit API key credentials\r\n let client = try LanguageServiceClient()\r\n\r\n // Configure request using structured builder closure\r\n let document = Document().with {\r\n $0.type = .plainText\r\n $0.source = .content(text)\r\n }\r\n\r\n let response = try await client.analyzeSentiment(\r\n request: AnalyzeSentimentRequest().with { $0.document = document }\r\n )\r\n\r\n if let sentiment = response.documentSentiment {\r\n print("Document sentiment score: \\(sentiment.score)")\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e743efc10>)])]>

Notice how Document().with { ... } avoids verbose temporary variables or mutating setters by providing a clean, thread-safe configuration closure.

Networking, transport, and authentication

The repository splits infrastructure primitives into modular packages under packages/:

swift-google-cloud-auth: Implements Application Default Credentials discovery, service account JWT signing, external account exchange for Workload Identity Federation, and API keys.
swift-google-cloud-wkt: Provides idiomatic Swift types for Google Protocol Buffer well-known types, including nanosecond-precision Timestamp representations that bridge cleanly to Swift's Date.
swift-google-cloud-gax: Handles Google API Extensions such as automated retry loops, exponential backoff, and pagination state machines.

When you initialize any client library without arguments, Credentials.default() automatically scans your environment (GOOGLE_APPLICATION_CREDENTIALS, quota project variables, or the local Google Cloud CLI configuration) and authenticates connections over gRPC or HTTP/2.

If you need to programmatically override credentials with an API key or attach custom access headers, you can pass explicit configuration options. For example, you could modify the previous example to use the following:

code_block: <ListValue: [StructValue([('code', 'import Foundation\r\nimport GoogleCloudAuth\r\nimport GoogleCloudGax\r\nimport GoogleCloudLanguageV2\r\n\r\nfunc analyzeTextSentiment(apiKey: String, text: String) async throws {\r\n // Initialize explicit API key credentials\r\n let credentials = try Credentials(configuration: .apiKey(apiKey))\r\n let client = try LanguageServiceClient(\r\n ClientOptions().with { $0.credentials = credentials }\r\n )\r\n\r\n // Configure request using structured builder closure\r\n let document = Document().with {\r\n $0.type = .plainText\r\n $0.source = .content(text)\r\n }\r\n\r\n let response = try await client.analyzeSentiment(\r\n request: AnalyzeSentimentRequest().with { $0.document = document }\r\n )\r\n\r\n if let sentiment = response.documentSentiment {\r\n print("Document sentiment score: \\(sentiment.score)")\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e743eff10>)])]>

The autogenerated client ecosystem

Google Cloud operates a vast ecosystem of APIs whose schemas update regularly. The teams supporting google-cloud-swift use code generators to automatically update the client libraries with the latest features and with new APIs. Using code generators produces stable APIs, without disruptive breaking changes. While the releases are on a fixed cadence, please contact Cloud Customer Care if you need a particular feature or API urgently.

Whether you need to rotate keys in Secret Manager or invoke multimodal inference models via the Gemini API on Gemini Enterprise Agent Platform, the generated SDKs follow consistent naming and async/await signatures.

These generated clients offer more than plain unary RPC wrappers. They also offer wrappers that simplify application development. For example, iterating over long results involves fetching pages of results with one RPC, iterating over the page of results, and then preparing a new request to retrieve the following page. Using the generated clients this becomes an asynchronous iterator. This example shows how to query project secrets using GoogleCloudSecretManagerV1:

code_block: <ListValue: [StructValue([('code', 'import Foundation\r\nimport GoogleCloudSecretManagerV1\r\n\r\n@main\r\nstruct SecretManagerQuickstart {\r\n static func main() async throws {\r\n guard let projectId = CommandLine.arguments.dropFirst().first else {\r\n print("Usage: SecretManagerQuickstart <projectId>")\r\n exit(1)\r\n }\r\n\r\n // Connects using Application Default Credentials automatically\r\n let client = try SecretManagerServiceClient()\r\n\r\n let request = ListSecretsRequest().with {\r\n $0.parent = "projects/\\(projectId)"\r\n }\r\n\r\n // Async sequence streams pages of secrets automatically\r\n print("Secrets in project \\(projectId):")\r\n for try await item in try client.listSecretsByItem(request: request) {\r\n print(" - \\(item.name)")\r\n }\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e743ef410>)])]>

The pagination response returns an asynchronous sequence (AsyncSequence). You can iterate over items with for try await while the client library fetches subsequent pages in the background over non-blocking NIO channels.

Where to go next

With google-cloud-swift, server-side Swift developers can write end-to-end cloud infrastructure with compile-time race safety, native async/await ergonomic APIs, and zero OS thread congestion.

To inspect the source code, open issues, or contribute new veneers, visit the official repository at googleapis/google-cloud-swift.

Want to discuss server-side Swift architectures or Cloud Run containerization? Join the Google Developer Program to continue the conversation.

Empower your agents with the Google Cloud CLI remote MCP server

Wed, 30 Sep 2026 16:00:00 +0000

Today, we’re expanding our ecosystem of managed remote MCP servers by introducing the Google Cloud CLI remote MCP server in preview.

Powered by the popular gcloud and bq (BigQuery) command-line tools, this new server gives AI agents immediate, broad access to command-line operations for managing Google Cloud infrastructure and working with advanced BigQuery workflows securely and seamlessly.

Why CLI matters for AI agents

Agents are increasingly performing complex cloud operations, but standardizing how they interact with backend systems remains a challenge. The Google Cloud CLI remote MCP server bridges this gap by packaging the versatility of hundreds of gcloud and bq commands into one single MCP server. This results in two strong benefits for the agent:

Higher-level abstractions: CLI commands package complex multi-step workflows, validation checks, and high-level operations into unified commands rather than requiring multi-step API orchestration.
Leverages model training: LLMs are heavily pre-trained on public command-line documentation, syntaxes, and usage examples, making CLI invocation intuitive and highly accurate for models.

The benefits of putting CLI behind remote MCP

Managing cloud infrastructure with AI agents traditionally requires installing and maintaining Google Cloud CLI binaries inside agent execution environments. The Cloud CLI remote MCP server bridges CLI capabilities with MCP benefits by providing an isolated execution sandbox on Google Cloud infrastructure. This solves key infrastructure challenges:

Simplified dependency and runtime management: For teams building custom agents, maintaining local CLI versions and dependencies across dev, test, and production environments creates operational overhead. Remote MCP eliminates local installations and runtime maintenance.
Access for web-based agent endpoints: Web-hosted agent platforms and web interfaces (such as Gemini Enterprise and other hosted enterprise agent platforms) run in environments where users cannot control or install local packages. Remote MCP enables secure, managed access to Google Cloud CLI operations directly from these surfaces.

Enterprise-grade security and governance

Connecting an AI agent to your infrastructure requires strict, enterprise-ready safeguards. This remote server leverages Google Cloud's standard identity and governance frameworks to keep your environments secure:

Zero ambient credentials: The server isolates execution in a network-restricted proxy boundary with no ambient credentials. Authentication and authorization are handled through Agent Identity, OAuth 2.0, and Identity and Access Management (IAM).
Strict policy enforcement: Every command executed through the remote MCP server is run with the permissions of the authenticated caller identity. Both standard IAM permissions and organization policy service constraints are strictly enforced against downstream target resources.
Advanced protection with Model Armor: To minimize the risks associated with AI tool calling, the Cloud CLI remote MCP server integrates with Model Armor. You can proactively screen LLM prompts and responses to protect against risks like prompt injection and malicious inputs.
Cloud audit logging: The Cloud CLI remote MCP server can be configured to log every tool invocation to Audit Logs (Data Access logs under cloudcli.googleapis.com/mcp). Security teams can gain full visibility into caller identities, OAuth clients, and IAM authorization decisions (mcp.googleapis.com/tools.call) without exposing sensitive command payloads or personally identifiable information (PII).

Connecting to the Google Cloud CLI Remote MCP Server

Integrating cloud management into your agents no longer requires packaging Google Cloud CLI binaries, managing local execution runtimes, or maintaining dependencies inside agent container images. Because the Google Cloud CLI remote MCP server implements the standard Model Context Protocol, any MCP-compatible agent platform or orchestration runtime can connect immediately via standard configuration:

code_block: <ListValue: [StructValue([('code', '{\r\n "mcpServers": {\r\n "google-cloud-cli": {\r\n "uri": "https://cloudcli.googleapis.com/mcp",\r\n ...\r\n }\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e743ee850>)])]>

Your agent immediately gains access to execute gcloud and bq commands in a secure, network-isolated cloud sandbox.

Authentication is handled via keyless Agent Identity for hosted Google Cloud platforms, or standard OAuth 2.0 for external runtimes. For authentication options, see the MCP Authentication Guide.

Bringing infrastructure management to agents

The Cloud CLI remote MCP server exposes two powerful tools, run_gcloud_command and run_bq_command, giving your AI agents broad, immediate access to Google Cloud operations through natural language.

Managing cloud infrastructure with run_gcloud_command

With run_gcloud_command, agents can execute the full breadth of gcloud operations to manage, diagnose, and secure your Google Cloud environment. An example follows:

Observability and incident diagnostics: An agent streamlines incident diagnostics by automating command execution and reducing context-switching across tools.

Extending BigQuery operations with the run_bq_command tool

While the BigQuery MCP server already helps organizations analyze and explore data using AI agents, with the introduction of run_bq_command, agents can now tackle advanced BigQuery tasks such as resource allocation, job monitoring, and task scheduling by unlocking the full scope of bq CLI functionality. Key capabilities include:

Automating scheduled queries: An agent utilizes BigQuery Data Transfer Service configurations to schedule queries automatically.
Job and resource management: Gain deep insight into query execution details, including processed data volume, slot usage, and execution plans, as well as managing reservations.
Access and permissions control: Data administrators and owners can inspect and update table permissions directly through the agent.

Pricing and availability

The Google Cloud CLI MCP server is available today in public preview. There is no additional charge to use the MCP server itself. You pay only for the GCP resources you create and any applicable data transfer costs.

To get started, enable the Cloud CLI Execution API (`cloudcli.googleapis.com`) in your Google Cloud project, grant the required MCP Tool User (`roles/mcp.toolUser`) IAM role to your agent or user identity, and configure your MCP client to connect to `cloudcli.googleapis.com/mcp`.

Data Agent Kit is now GA: Bring Google Data Cloud to any coding agent

Wed, 30 Sep 2026 13:00:00 +0000

Today, Google Cloud Data Agent Kit is generally available. Data Agent Kit is a free set of Model Context Protocol (MCP) tools and agent skills that lets the coding agent you already use work directly with your Google Cloud data products, whether you're using Antigravity, Claude Code, Codex, or other popular tools.

With GA, we are adding support for BigQuery Graph, Bigtable, and Managed Service for Apache Spark access to your open Lakehouse, along with dozens of quality-of-life improvements that make everyday work faster and smoother.

What is Data Agent Kit?

Coding agents have become remarkably good at writing SQL, PySpark, and pipeline code. What they don't have by default is context about your environment: which tables exist, how they're partitioned, which ones your team trusts, or why last night's job failed. Without that, even a strong agent has to work from assumptions, and you end up pasting schemas and error logs into the chat to fill in the gaps.

Data Agent Kit fills that gap with two things:

MCP tools: Connections to more than 15 Google Data Cloud services, so your agent can inspect schemas, run queries, read job logs, and manage resources in your live environment.
Google-authored skills: Open-source instructions from Google Cloud engineers that teach your agent data best practices, like optimizing BigQuery SQL, designing Bigtable row keys, and building dbt (data build tool) pipelines.

You can use Data Agent Kit wherever you already work: as an IDE extension for VS Code, Antigravity IDE, Cursor, and other VS Code-compatible editors; as a plugin for Antigravity 2.0, Antigravity CLI, Claude Code, and Codex; or in Cloud Shell and Cloud Workstations, where it comes pre-installed. The IDE extension also brings a lightweight version of the Google Cloud console into your editor, so you can browse data, run queries, and review your agent's work without switching windows.

How it works

Say you ask your agent, "Forecast next month's demand for our top-selling products and check whether we have enough inventory to meet it." Data Agent Kit loads the relevant skills, so the agent follows Google's best practices for the task. It searches Knowledge Catalog to find the sales and inventory tables your team trusts. It then uses MCP tools to run a forecast in BigQuery, check current stock levels in AlloyDB for PostgreSQL, and bring the combined answer back to your editor or terminal. Every step runs with your own IAM permissions.

How Data Agent Kit connects to data.

What you can build

Data Agent Kit covers analytics, operational databases, the Lakehouse, and pipelines. Here's what that looks like in practice, starting with what's new at GA.

Analytics and graph: BigQuery and BigQuery Graph (New in GA)

Graphs are a natural way to explore relationships, like which products people buy together or how suppliers connect to your inventory. Building one usually means hand-writing CREATE PROPERTY GRAPH DDL, learning GQL, and working out which keys actually form edges.

Instead, you describe the graph you want and your agent builds it. The bigquery-graph-author skill maps your tables to nodes and edges, checks each proposed relationship against the actual data, and shows you a plan to approve before creating anything. It can even start from an ER diagram or data model you already have. The bigquery-graph-query skill then writes the GQL, and the graph visualizer in the IDE lets you click through the results.

Building and visualizing a BigQuery property graph.

Operational and real-time databases: Spanner, AlloyDB, Cloud SQL, and Bigtable (New in GA)

Some features have to load instantly, like a personalized feed, a live counter, or a "recently viewed" rail on your storefront. Bigtable is built for exactly that, and it rewards a well-designed row key.

With GA, Bigtable joins Spanner, AlloyDB, and Cloud SQL as a fully supported database in Data Agent Kit. The new bigtable-basics skill designs your schema around how the data will be read and flags hotspots and full table scans before you create anything. Your agent can then create the table and query it with GoogleSQL, with column families flattened into readable columns. In the IDE, you can browse Bigtable instances and tables in the catalog explorer and run queries from the SQL editor.

Agent-guided Bigtable schema design and querying.

Lakehouse and Spark: Managed Service for Apache Spark and Lakehouse for Apache Iceberg (New in GA)

Your agent could already query the Apache Iceberg tables in your Lakehouse through BigQuery. Now it can also work with those same tables using serverless Spark on Managed Service for Apache Spark, with no cluster to manage. Each session keeps its state, so temporary views carry across statements, and you get Iceberg's full feature set, including branching, time travel, and schema evolution.

Your tables don't all have to live on Google Cloud, either. The federate-lakehouse-catalog skill connects your Lakehouse to AWS Glue and Databricks Unity Catalog, so your agent can query that data in place without building an ingestion pipeline first.

Branching and querying Iceberg tables with Managed Service for Apache Spark.

Pipelines and orchestration: dbt, Dataform, and Managed Service for Apache Airflow

Once your logic works, your agent can turn it into a pipeline that runs on its own. It writes dbt or Dataform models, then the gcp-pipeline-orchestration skill schedules them together with your notebooks as an Orchestration Pipeline on Managed Service for Apache Airflow. Pipelines can also include Gemini Enterprise Agent Platform steps, like uploading a model or running batch inference.

In the IDE, you can follow each run on a visual pipeline canvas. If a task needs attention, click Diagnose to hand its logs to your agent. Troubleshooting skills for Airflow and Spark trace the root cause and propose a fix for you to approve.

Troubleshooting a failed Airflow DAG with an agent.

Improving the developer experience

GA also streamlines setup and day-to-day workflows. When you get started, you simply sign in once and select the Google Cloud services you use. Data Agent Kit automatically enables the required APIs, installs the matching skills, and configures your MCP servers with no manual setup files. Inside the IDE, the SQL editor and notebooks now support inline code generation, @ references to tables, and diff views for suggested changes. The extension also shares your active project, open file, and the error from the query you just ran with your agent, so asking it to "fix this query" just works.

We also made the core tools faster and more responsive. New Spark notebooks automatically create and select a Spark Connect runtime, and Spark SQL queries in the editor run in isolated sessions with built-in execution metrics. The catalog explorer now loads faster and includes BigQuery public datasets in the sidebar. Tuned notebook skills help your agent finish notebook tasks more quickly while using fewer tokens.

Ready for the enterprise

Data Agent Kit is included at no additional cost; you pay standard pricing only for the Google Cloud services your agent uses. Skills also steer the agent toward cost-aware query patterns, like checking partition keys and running a dry run before executing a BigQuery query to avoid accidental full-table scans. And because the skills are open source on GitHub, your team can audit them, fork them, or write custom skills for your own internal standards.

For access control, the agent connects as you or as a service account you impersonate, so row- and column-level security policies apply automatically. Admins can also govern MCP access with Identity and Access Management (IAM), screen MCP traffic with Model Armor, and scope agents with VPC Service Controls and Principal Access Boundary policies.

Install and get started

You can set up Data Agent Kit in under a minute in either your IDE or your terminal.

Option 1: Install the IDE Extension

IDE extension: Search for "Google Cloud Data Agent Kit" in the Extensions panel of VS Code, Antigravity IDE, Cursor, or any VS Code-compatible editor. You can also install it from the VS Code Marketplace or Open VSX.
Antigravity 2.0: Go to Settings > Customizations > Build with Google Plugins, then download Data Agent Kit.
Cloud Shell and Cloud Workstations: Already installed by default; just open the editor and sign in.

Option 2: Install the CLI Plugin

Run the command for your preferred coding agent using the official GoogleCloudPlatform/data-agent-kit-plugin repository:

code_block: <ListValue: [StructValue([('code', '# Antigravity CLI\r\nagy plugin install https://github.com/GoogleCloudPlatform/data-agent-kit-plugin\r\n\r\n# Claude Code\r\nclaude plugin install data-agent-kit-starter-pack@claude-plugins-official\r\n\r\n# Codex CLI\r\ncodex plugin marketplace add GoogleCloudPlatform/data-agent-kit-plugin\r\ncodex plugin add dak@dak-marketplace'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e6767e7d0>)])]>

Then try a first prompt:

code_block: <ListValue: [StructValue([('code', "What are this week's fastest rising search terms in the US that weren't in the last week's top 10? Use the BigQuery public Google Trends dataset."), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e67a98c90>)])]>

Data Agent Kit plugin running in Claude Code.

Next steps & hands-on resources

Read the docs: Explore the Data Agent Kit documentation and product overview page.
Explore the skills: Browse, star, and contribute on GitHub.
Build an analytics workflow: Try the Analytics with Data Agent Kit and Antigravity IDE codelab, and read the companion blog, Agentic analytics with the Data Agent Kit.
Build a data science pipeline: Try the Fraud detection pipeline with Data Agent Kit and Antigravity IDE codelab.

Graph Workflows in ADK: Everything You Need to Know

Tue, 29 Sep 2026 16:00:00 +0000

Graph engineering is the design work: breaking a task into nodes, connecting them with edges, and deciding where code, models, or people control the next step. The Agent Development Kit (ADK)'s Workflow turns that design into an executable process, with functions and agents doing the work. Through a refund example, this post shows how to run steps in parallel, route decisions, pause for human review, and process a list of cases. It also explains when to declare the paths in a static graph and when to let Python schedule further work as results arrive.

TL;DR: Using a refund workflow in ADK, we'll cover fan-out and fan-in, deterministic and agent routers, human-in-the-loop pauses, parallel workers, and dynamic orchestration— along with when to use a static graph or let Python decide what runs next.

Start with a single agent

We can give one agent the tools and instructions to handle the refund request from start to finish:

code_block: <ListValue: [StructValue([('code', 'refund_agent = Agent(\r\n name="refund_agent", model=MODEL,\r\n tools=[fetch_order, fetch_payment, fetch_history],\r\n instruction="""You handle refund requests.\r\n 1. Look up the order.\r\n 2. Check the payment record.\r\n 3. Check the customer\'s refund history.\r\n 4. Deny if there\'s an open chargeback or it\'s past 30 days.\r\n Approve if it\'s under $50 and they\'ve had fewer than three\r\n refunds this year.\r\n 5. Write the customer an email explaining the decision.""",\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f7e67cdf790>)])]>

This puts the model in charge of choosing the tools, applying the policy, and writing the reply. But the prompt already describes distinct pieces of work: three lookups, a decision, and a response. Making those pieces separate nodes lets us decide how each should run.

Assume the customer has selected an order and clicked “Request refund.” The app knows the order ID, so the workflow can begin with the lookups.

Let independent steps run together

The order, payment, and refund-history lookups all need the order ID, but none needs another lookup's result. Although the prompt lists them one after another, there is no reason for them to wait for each other. We can run all three in parallel.

The policy decision is different: it needs all three records. So the workflow splits into three paths, then brings their results together before continuing. These two moves are called fan-out and fan-in.

In ADK, the lookup functions can become nodes directly. A nested tuple starts them together, and a JoinNode waits for their results. First, the imports and a lookup signature:

code_block: <ListValue: [StructValue([('code', 'import asyncio\r\nfrom pydantic import BaseModel, Field\r\n\r\nfrom google.adk import Agent, Context, Event, Workflow\r\nfrom google.adk.events import RequestInput\r\nfrom google.adk.workflow import JoinNode, START, node\r\n\r\nasync def fetch_order(node_input: str) -> dict:\r\n ...'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f7e6710a310>)])]>

Each lookup receives the order ID through node_input and returns a dictionary. We can connect them like this:

code_block: <ListValue: [StructValue([('code', 'join_case = JoinNode(name="join_case")\r\n\r\nedges=[\r\n (START, (fetch_order, fetch_payment, fetch_history), join_case),\r\n]'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f7e6710b310>)])]>

Read this from left to right: start all three lookups, then continue through join_case once they finish.

The join returns a dictionary keyed by node name. The next node can read node_input["fetch_order"], node_input["fetch_payment"], and node_input["fetch_history"] without a model call to collect the results. The JoinNode guide explains the details.

If the payment lookup needed a transaction ID from the order lookup, those two would run in sequence. Dependencies determine the edges, even when the prompt lists every step in order.

You can learn more about fan out and fan in in this video:

Graph Engineering with ADK

One detail matters when turning tools into nodes: the parameter named node_input receives the previous node's output. Other names bind to ctx.state by default, so order_id would look for ctx.state["order_id"] and raise a ValueError if it is missing. Here, the str annotation also converts START's types.Content input to a string.

Route each request to the right workflow

So far, the customer has explicitly requested a refund. In a broader support conversation, we first need to identify what they want and send the request to the right process. “The shoes are the wrong size. Could you send me a different pair?” should go to an exchange workflow, while a request for money back should enter our refund workflow.

That introduces a router, a node that chooses which branch runs next.

A deterministic router follows explicit rules, the same inputs produce the same route.
A nondeterministic router can choose different branches for the same input.
An agent router uses a model to interpret the request, so its choice can vary.

Routers choose among the paths defined by the workflow. In our support example, we can use an agent to identify the customer's intent, then fixed rules to apply the refund policy.

Identify the intent with an agent router

An agent can interpret the customer's message and classify the request. In one ADK pattern, it returns a structured category, then a small function emits the corresponding Event(route=...) to send the request to the chosen workflow:

code_block: <ListValue: [StructValue([('code', 'Customer message → classification agent → route function\r\n ├─ REFUND → refund workflow\r\n ├─ EXCHANGE → exchange workflow\r\n └─ CLARIFY → ask a follow-up question'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e6710bd90>)])]>

The model identifies the intent; the graph defines the available destinations. If the request is unclear, the workflow can ask a follow-up question. ADK's routing sample shows this pattern.

This intent router would sit before our refund workflow. For the selected-order example, the “Request refund” button has already established the intent, so we can enter that workflow directly.

Apply the refund policy with a deterministic router

Inside the refund workflow, the three lookups give us the facts for another decision: approve, deny, or ask a person to review. This time, the policy gives us explicit thresholds, so a function can choose the path:

code_block: <ListValue: [StructValue([('code', 'def refund_policy(amount, days_ago, chargeback_open, priors):\r\n if chargeback_open or days_ago > 30:\r\n return "DENY"\r\n if amount < 50 and priors < 3:\r\n return "AUTO_APPROVE"\r\n return "MANUAL_REVIEW"\r\n\r\ndef route_refund(node_input):\r\n case = {**node_input["fetch_order"], **node_input["fetch_payment"],\r\n **node_input["fetch_history"]}\r\n route = refund_policy(case["amount_usd"], case["placed_days_ago"],\r\n case["chargeback_open"], case["prior_refunds_12mo"])\r\n return Event(output=case, route=route) # the function names the path'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f7e6710be90>)])]>

route_refund combines the records and returns the case with one of three route names: AUTO_APPROVE, DENY, or MANUAL_REVIEW. Manual review handles cases that meet neither automatic rule.

This is a deterministic router: the same case data produces the same decision, and we can test the policy without calling a model.

	Deterministic router	Agent router
Decisions come best from	Rules in code	A model interpreting the input
Best fit	Known facts and explicit policy	Meaning that is hard to capture in rules
Example	Deny an order older than 30 days	Recognize an exchange request
Model call for routing	None	Required

Both routers choose among defined paths. An agent router can sit inside a static graph: the model's choice varies, while the possible connections stay the same.

For an automatic approval or denial, the next step is to explain the decision to the customer. We give each route a notice agent that writes the reply from the case data. ADK passes the dictionary in Event(output=case, ...) to that agent as a JSON user message:

code_block: <ListValue: [StructValue([('code', 'approve_notice = Agent(\r\n name="approve_notice", model=MODEL,\r\n instruction="Tell the customer their refund is approved and when to expect the "\r\n "money, using the case JSON you receive. Short email, warm, no fluff.",\r\n)\r\ndenial_notice = Agent(\r\n name="denial_notice", model=MODEL,\r\n instruction="Tell the customer their refund was declined and exactly why, based "\r\n "on the case JSON you receive. If it carries a reviewer_note, that is "\r\n "the reason. Short email, direct and kind. Do not invent policy.",\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f7e67177cd0>)])]>

The refund workflow now has a path from the initial lookups to a decision and a reply, with a third branch for cases that need a person:

code_block: <ListValue: [StructValue([('code', 'workflow = Workflow(\r\n name="refund_decision",\r\n edges=[\r\n (START, (fetch_order, fetch_payment, fetch_history),\r\n join_case, route_refund),\r\n (route_refund, {"AUTO_APPROVE": approve_notice,\r\n "MANUAL_REVIEW": escalate_to_human,\r\n "DENY": denial_notice}),\r\n ],\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f7e671763d0>)])]>

The first chain fetches the records, joins them, and applies the policy. The second maps the router's decision to a destination. Automatic decisions go straight to a notice agent; MANUAL_REVIEW goes to the human-review node we'll define next.

That third branch needs more than another function call. A reviewer may take minutes or days to answer, so the workflow must pause with the case pending and continue once the person decides.

The review node yields RequestInput, which records the pending request and pauses the run. On resume, rerun_on_resume=True runs the node again, with the answer available in ctx.resume_inputs:

code_block: <ListValue: [StructValue([('code', 'REVIEW = "refund:review"\r\n\r\nclass ReviewDecision(BaseModel):\r\n approve: bool = Field(description="True to refund, False to decline.")\r\n note: str = Field("", description="Why, in the reviewer\'s words.")\r\n\r\n@node(rerun_on_resume=True)\r\nasync def escalate_to_human(ctx: Context, node_input: dict):\r\n answer = ctx.resume_inputs.get(REVIEW)\r\n if answer is None: # first pass: ask, then stop\r\n yield RequestInput(\r\n interrupt_id=REVIEW,\r\n message=f"Refund ${node_input[\'amount_usd\']} on order "\r\n f"{node_input[\'order_id\']}?",\r\n payload=node_input, # what the reviewer is shown\r\n response_schema=ReviewDecision,\r\n )\r\n return\r\n # second pass: the answer is here, so route on it\r\n yield Event(output={**node_input, "reviewer_note": answer.get("note", "")},\r\n route="AUTO_APPROVE" if answer["approve"] else "DENY")'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f7e67177090>)])]>

The response schema gives the node an approve value to route on and a note to carry forward. If the reviewer declines, the notice agent receives their reason with the case. One more edge connects the review decision to the reply:

code_block: <ListValue: [StructValue([('code', 'edges=[\r\n ...,\r\n (escalate_to_human, {"AUTO_APPROVE": approve_notice,\r\n "DENY": denial_notice}),\r\n]'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f7e67175b50>)])]>

ADK ships two variants of this. The one above is the single-node pattern: the node reruns and reads ctx.resume_inputs, as in the request_input_rerun sample. The request_input sample shows the two-node variant, where one node yields RequestInput and the reviewer's answer arrives as the next node's node_input — so there is no ctx.resume_inputs to find in that file. The companion scripts linked below include the code that sends the reviewer's answer back to the run.

A Workflow is also a node. This whole refund process can become one step in a larger customer-service workflow.

Apply the same step to a batch of cases

We now have a process for one refund. Suppose a batch of cases arrives with the records already collected. Each needs the same policy check, and the number of cases changes from batch to batch. We can apply one node to every item using a parallel worker.

In ADK, parallel_worker=True runs a node once per item in an input list and collects the results in the original order. We can reuse our policy function:

code_block: <ListValue: [StructValue([('code', '@node(parallel_worker=True)\r\ndef review_case(node_input):\r\n # Each worker receives one case from the input list.\r\n case = node_input\r\n decision = refund_policy(\r\n case["amount_usd"], case["placed_days_ago"],\r\n case["chargeback_open"], case["prior_refunds_12mo"],\r\n )\r\n return {"order_id": case["order_id"], "decision": decision}\r\n\r\ndef collect_decisions(node_input):\r\n # This node receives the list of worker results.\r\n return {"decisions": node_input}\r\n\r\nbatch_review = Workflow(\r\n name="batch_review",\r\n edges=[(START, review_case, collect_decisions)],\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f7e67109410>)])]>

Each worker receives one case, and collect_decisions receives the results as a list. No separate JoinNode is needed. The flag also works on agents—for example, to write an explanation for each case. The parallel_worker sample shows both forms.

The policy calculation here is small. Concurrency is more useful when each item waits on an API or model call, but the way inputs and results move stays the same.

Pattern	Work distributed	Collected output
Fan-out with `JoinNode`	Different notes doing independent jobs	Dictionary keyed by node name
Parallel worker	The same node for every item	List in input order

The batch size can change without changing the graph. A variable amount of work still fits inside a fixed process. See the parallel worker guide for execution details.

Let results shape the next step

Our refund process has known paths, even when a batch contains more cases or a reviewer takes longer to answer. But some work only becomes clear as we investigate.

Suppose a disputed refund reveals a second transaction. Checking it raises a delivery question that needs further investigation. Now each result can create follow-up work. A dynamic node can examine those results and schedule the next checks in Python.

ADK provides ctx.run_node to run another node and await its result. To see how that changes orchestration, let's first express our existing refund flow this way:

code_block: <ListValue: [StructValue([('code', 'HANDLERS = {\r\n "AUTO_APPROVE": approve_notice,\r\n "MANUAL_REVIEW": escalate_to_human,\r\n "DENY": denial_notice,\r\n}\r\n\r\n@node(rerun_on_resume=True)\r\nasync def refund_flow(ctx, node_input):\r\n # Step 1: fetch all three records at once.\r\n order, payment, history = await asyncio.gather(\r\n ctx.run_node(fetch_order, node_input, use_sub_branch=True),\r\n ctx.run_node(fetch_payment, node_input, use_sub_branch=True),\r\n ctx.run_node(fetch_history, node_input, use_sub_branch=True),\r\n )\r\n case = order | payment | history\r\n\r\n # Step 2: apply the refund policy — the same pure function, 0 LLM calls.\r\n decision = refund_policy(\r\n amount=case["amount_usd"],\r\n days_ago=case["placed_days_ago"],\r\n chargeback_open=case["chargeback_open"],\r\n priors=case["prior_refunds_12mo"],\r\n )\r\n\r\n # Step 3: run the chosen handler, and use its output as this node\'s output.\r\n await ctx.run_node(HANDLERS[decision], case, use_as_output=True)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f7e67176d90>)])]>

asyncio.gather runs the lookups together. Python combines their results, applies the policy, and runs the selected handler. use_as_output=True makes the handler's result the parent node's output without emitting it twice.

The dynamic version also adjusts the human-review node: after the answer arrives, it calls the chosen notice agent through ctx.run_node. The complete dynamic script (refund_dynamic.py) includes that variation.

The outer graph only needs an entry point:

code_block: <ListValue: [StructValue([('code', 'workflow = Workflow(\r\n name="refund_dynamic",\r\n edges=[(START, refund_flow)],\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f7e67176dd0>)])]>

In the static version, the edge list shows the branches and destinations. In this version, we read refund_flow to see them. The dynamic node guide covers this approach.

The human-review pause still works here. When the answer arrives, refund_flow runs again from the top, but completed ctx.run_node calls return their recorded outputs from session history. The lookups do not repeat, as direct function calls would. Keeping side effects inside child nodes lets completed calls replay their results when the parent resumes. Each child also gets its own trace span, and use_sub_branch=True keeps concurrent children's events on separate branches.

For this fixed refund process, the edge list remains easy to inspect. The Python version gives us a place to add the investigation logic described above: inspect a result, choose a follow-up node, and run independent checks together. A model might suggest what to investigate, while code limits the work—for example, three follow-ups per finding and two levels of investigation before human review.

Loops can still fit in a static graph. A fixed draft → check → revise process can use a conditional back-edge and a router that limits revisions. “Static” describes the possible connections; the actual path and iteration count can vary. The graph guide covers conditional cycles.

You can also place a dynamic node inside a static workflow, using Python for a stage that needs it while keeping the surrounding process visible.

Choose who decides what runs next

Start with a question: can you draw the possible workflow before the input arrives? Include branches, loops, and repeated stages. You do not need to predict the path each request will take.

What you need	Pattern
Independent jobs, then all their results	Fan-out and fan-in
A branch chosen by fixed rules	Deterministic router
A branch chosen by interpreting meaning	Agent router
A person's decision before continuing	`RequestInput`
The same step across a list	Parallel worker
Code that schedules work as results arrive	Dynamic node

Use an edge list when it makes those connections clear. Use dynamic orchestration when results create further work or Python expresses the control more naturally. A small, open-ended task may need only one agent and its tools.

In our refund workflow, the graph coordinates the lookups, code applies the policy, a person handles exceptions, and a model writes the reply. Graph engineering gives each a clear responsibility—and makes it easier to see how the process works.

Get started

Read: the workflow guide.
Explore: the workflow samples.
Build: the hands-on codelab, which applies these patterns to a marathon race-day coach.

Best practices guide for customizing Gemini models via Reinforcement Learning (RL)

Fri, 25 Sep 2026 16:00:00 +0000

Reinforcement learning (RL) has been a keystone of modern LLM post-training, but it demands large training clusters and access to model internals that external customers can't have with proprietary models like Gemini. So here at Google Cloud, we packaged it into a managed RL fine-tuning service (RLFT service) — you bring prompts and a reward function; we handle the infrastructure and the proprietary model internals.

Now, you can adapt Gemini with the service — teaching the model from a reward signal you define, rather than from a fixed set of labeled answers. This unlocks a class of problems that supervised fine-tuning (SFT) struggles with: tasks that are hard to demonstrate but easy to score.

In this guide, we will walk through practical best practices for using RL fine-tuning service. We'll start with a short tour of the RL training loop, how to decide if and when to use RL, and introduce how to get the most value from this approach.

What is RLFT?

RLFT adapts Gemini from a reward signal you define rather than labeled answers. Instead of authoring a large set of gold examples, you write one program that scores a response and the service improves the model against it — unlocking tasks that are hard to demonstrate but easy to verify: you can't hand-write the ideal SQL for every schema, but you can run the query and check the result.

At each training step the service generates multiple candidate responses to your prompts, scores them with your reward, and improves the model so that higher-scoring responses become more likely while it stays close to the original Gemini. The reinforcement learning that makes this work is fully managed — you never configure it. The one thing you own, and the thing that most determines your results, is the reward.

Three properties define what RLFT can and can't do:

It learns from the model's own outputs: It refines what the model already produces rather than copying an external target, so it tends to disturb unrelated capabilities less than SFT.
It rewards outcomes, not paths: Any response that reaches a good result earns reward, which fits open-ended tasks with many valid solutions.
It amplifies existing competence: It makes occasional success reliable, but it can't teach a skill the model never demonstrates.

When to use RLFT

Prompting and SFT handle most adaptation; exhaust them first. RLFT earns its keep when you can grade a response but can't cheaply author it, when SFT has plateaued on the metric that matters (faithfulness, schema validity, tone), or when the task has many equally valid answers a single reference target would wrongly penalize.

SFT and RLFT are complementary, not competing:

Direct RLFT when the base model already succeeds part of the time — enough for the reward to tell better answers from worse ones.
Two-stage SFT → RLFT when you have SFT data or the base success rate is too low for RL to gain traction. Use SFT as a short, cheap warm start — kept light, since over-fitting the demonstrations leaves less room for RL to improve — then continue into RL via Continuous Tuning, which initializes RL from the SFT checkpoint.

Across early adopters, these patterns show where RLFT delivers the most value — each scoring an outcome the business cares about but could never cheaply demonstrate.

Use cases for RLFT

AI-powered NPCs in games

What: In-character, on-brand dialogue held across long, multilingual, multi-turn conversations.
Problem: Off-the-shelf models break immersion — wrong language, hallucinated items, ignored players, repetitive loops.
Objective and reward: A Gemini autorater (LLM-as-a-judge) scores each turn on persona, flow, and game-state syntax, penalizing format and language errors.
Results: Loops and language drift disappeared and state syntax held, making shippable in-game characters viable at scale.

Structured entity extraction

What: Pulling a set of items from unstructured documents, such as supplier invoices and shipping manifests, into structured records automatically.
Problem: The long tail where SFT plateaus — missing required fields (recall) or inventing ones that aren't there (precision).
Objective and reward: A rule-based precision/recall reward forces every field to be grounded in the source text, not imitated from one gold answer.
Results: Field-level accuracy rose on noisy real-world documents where tuning had stalled, turning a manual review step into an automated one.

Content moderation

What: Applying intricate policies and decision trees at scale.
Problem: Models hallucinate false positives or reward-hack with invalid formats to dodge evaluation.
Objective and reward: A Cloud Run reward pairs format validation with a deterministic grader to enforce multi-step policy adherence.
Results: The model handled complex exemption carve-outs, sharply cut false positives, and stopped reward hacking — reducing the human-escalation volume that makes moderation expensive.

Code measured by execution

What: SQL or API calls graded on whether they actually run against customer data.
Problem: SFT mimics one reference query and breaks on unseen proprietary schemas.
Objective & Reward: A code-execution reward runs the code in a secure sandbox and pays out only if it compiles, executes, and returns the correct result.
Results: The model produced first-attempt executable queries at closed-frontier quality and lower inference cost, letting non-technical users query proprietary data in natural language.

Presentation slide generation via HTML

What: Multi-slide decks authored as HTML/CSS.
Problem: Training on text alone is blind to visual quality — overflows, clipped elements, and inconsistent styling slip through unnoticed.
Objective & Reward: A code-execution reward renders the slides and scores visual design, layout integrity, structural completeness, and rubric adherence.
Results: The model emitted modular, well-styled decks with cohesive themes and no layout overflow.

Where to start?

A dataset. A diverse set of prompts with a held-out validation split is enough for a first run — confirm the loop converges and reward moves the right way, then scale. Keep train and eval strictly separated; a contaminated eval hides overfitting.
A reward function. Your task specification as code or configs, and the dominant driver of quality. A good reward correlates with human preference, is robust to malformed output (catch the failed parse and return a clearly negative score rather than crashing), and resists reward hacking — ensemble judges, penalize length, floor degenerate outputs, and prefer a verifiable check over a model's opinion. Validate it offline before launch.

The service handles the rest; start from the defaults, watch reward and eval curves in the console, and take the checkpoint where validation reward saturates rather than the last step.

Get started today

What will you build? The tools are ready and waiting.

Documentation: Reinforcement Learning Fine-Tuning

Agent Factory recap: Agent harnesses, shifting left, and autonomous coding

Thu, 24 Sep 2026 23:00:00 +0000

In this episode of The Agent Factory, we explore the reality of building with autonomous agents alongside Ryan Lopopolo, a software engineer at Google Cloud and the person who coined the term agent harness. From throwing out manual code editors to treating team collaboration like leveling up RPG stats, Ryan breaks down how grounding models in rich context and shifting interventions left unlocks high levels of agent autonomy.

This post guides you through the key ideas from our conversation. Use it to quickly recap topics or dive deeper into specific segments with links and timestamps.

The Agent Harness - What is it?

Timestamp: [00:30]

An AI agent as we're defining it here is a large language model (LLM) plus an agent harness.

Think of the harness as everything wrapped around the LLM that isn't the model itself. For example, if you're working in Google Antigravity using Gemini 3.8 Flash, Gemini Flash is the LLM and Google Antigravity is the harness.

While an unassisted model can answer simple questions out of the box, it can't check live conditions or interact with your workspace on its own. When a user asks a question like "Why is the sky blue?", an unassisted LLM can respond without issue. However, when asked a question like "Should I wear a raincoat today?", the model can't answer on its own because it lacks the necessary data. The harness catches the intent, queries live weather tools, bundles that context back into the prompt, and hands it to the model to produce an informed answer.

Ryan Lopopolo on agent harnesses and autonomous coding

Tilde Thurium sat down with Ryan Lopopolo to discuss what it takes to run fully autonomous coding workflows in production. See the summary below!

Coining the harness and writing zero production code

Timestamp: [02:22]

The term agent harness grew out of Ryan's extensive work on autonomous coding agents, culminating in a February 2026 essay on leveraging coding models in an agent-first world. Ryan shared that he hasn't opened a traditional code editor since May of last year, maintaining that streak through his transition into Google Cloud. In this paradigm, engineers no longer author or review individual lines of syntax; instead, they operate at the level of natural language specifications and inspect the final artifacts, such as pull requests, documents, and spreadsheets. Then they determine whether the end result meets organizational standards.

Harness engineering is the study and the practice of putting a model into an environment where it can succeed. If you don't do that work, you end up doing what I call 'prompt and pray'. Ryan Lopopolo

Context curation and lazy prompting

Timestamp: [03:35]

Upfront harness investment pays off by allowing engineers to become lazy prompters. When the repository contains structured documentation, clear interfaces, and discoverable tools, you do not need to paste walls of text into a prompt box every morning

"I aspire to be an incredibly lazy prompter. If I have done the job to give the model the tools and context it needs to ground itself, I don't need to write a long prompt. It figures it out."

The model uses its harness to pull relevant context, allowing it to navigate large codebases and execute complex tasks without oversight.

Shifting left: engineering best practices as autonomous guardrails

Timestamp: [05:00]

When an agent fails, developers face a whole spectrum of interventions. The most common reflex is to fiddle with the prompt or retry, but that never scales across a team.

"The simplest, smooth-brain, stupidest intervention I can think of is literally just: try my prompt again without changing anything else. But shifting left means moving interventions earlier into the development lifecycle where they are cheapest and automated: from prompts, to repo docs, to linters, to tests, and all the way to upstream evals."

Instead of hoping the model guesses right on the next turn, shifting left embeds standards directly into the environment. Linters, tests, and AGENTS.md files act as durable memory and enforcement of what you think good looks like, making sure the agent stays on the rails without needing constant hand-holding.

Leveraging established tools and determinism

Timestamp: [06:50]

Agents shine when they're handed tools that already mirror patterns heavily represented in pre-training data. Pairing models with standard command-line interfaces moves reasoning into determinism, shifting the burden of context aggregation away from the model and onto reliable tools.

Ryan also shared an environmental design trick for context efficiency: structuring markdown files so link anchors sit directly beneath their corresponding prose blocks rather than inline. This prevents context clutter and mitigates "lost in the middle" retrieval issues. Because Ryan operates exclusively by reviewing end-state artifacts, keeping documentation readable allows him to easily inspect execution runs:

"I want to be able to look at the pull request and review it. If it made a bad decision, I need to know where it went off the rails so I can whack the agent on the head and make sure it does not make that same mistake again."

Long horizons and expanding the agentic loop

Timestamp: [09:45]

The central challenge of harness engineering is ensuring that agents cohere over long time horizons. Because human organizations produce software through iterative refinement rather than single-shot prompts, agent workflows must mirror that cadence. Harness engineering uses tightly scoped, reviewable pull requests to narrow the agent's state space. Stacking these high-confidence changes end-to-end allows supervisors to gradually expand the loop size, building trust until agents can autonomously execute large-scale initiatives, including entire language migrations

Curating agent teams like RPG stats

Timestamp: [11:58]

Rather than divvying up sprint tasks based on individual specialties, having a diverse team contribute to an agent turns it into a central producer of work that carries everyone's strengths. Ryan compared leveling up an agent's capabilities to building out a character sheet:

"[It's like] building out the stats of your RPG character. I get a new person on the team who is a React architect and boom! The attention that they pay is able to bump out the stats in front-end architecture and performance."

With that collective expertise baked into the environment, the agent can autonomously classify incoming work and activate the exact skills it needs on demand, operating as both a backend architect and a front-end specialist.

Accruing leverage in tools, not custom harnesses

Timestamp: [14:38]

For developers wondering whether to build their own custom agent harness, Ryan offered clear advice: don't build one from scratch. Standard harnesses already provide the foundational primitives: file reading, grep search, and command execution. Over-scaffolding an agent with rigid, bespoke frameworks creates technical debt and leads to sunk-cost traps when frontier models advance.

"If you focus all of your efforts on improving quality on tools and context, you can freely adopt the newest models as they come out and you'll be constantly accruing leverage into a bit of the system that will never become obsolete."

Operating Google Cloud and eliminating capability overhang

Timestamp: [16:47]

Discussing his work at Google Cloud, Ryan outlined his motivation to eliminate capability overhang: the delta between what frontier AI models are theoretically capable of and how much useful work is currently extracted in production. Because the cloud functions as a massive, programmable surface, equipping agents with direct interfaces to Google Cloud lets them manage and deploy infrastructure effectively, turning raw model capability into tangible enterprise utility.

Continually updating your priors on AI

Timestamp: [18:13]

The speed of AI development requires engineers and teams to actively unlearn old limitations and constantly reassess what these models can achieve. "It's very important to continually be updating what you think is possible with these lovely tools that we have," Ryan urged. What broke six months ago often runs effortlessly on today's frontier models. Rather than getting locked into rigid workflows, developers should build around the two highly extensible interfaces that will remain relevant across every model upgrade: tools and context.

"Agents will always need context in order to do that last mile adaptation into what you think good is. And as you can continue to... shift it to the left, in terms of increasingly capable tools which act as a form of memory and enforcement of what you think good looks like, you'll continually be amazed as the models are able to do more and more interesting things for you over time."

How To Build A Custom Harness

Next, Billy Jacobson started us off by showing how developers can customize their own agent harnesses for specific tasks.

Under the hood: Why build a custom harness?

Timestamp: [19:44]

Before jumping into code, Billy unpacked why developers should understand the mechanics of a harness rather than treating it like a black box. Recalling advice from an engineering mentor that "You can just use the framework, but a great engineer will really understand the framework", Billy explained that building a harness yourself is the best way to debug what happens when an agent breaks. You can evaluate three core design decisions for every workflow:

Looping: How many iterations should the agent run, and what conditions trigger an exit state?
Tools: What specific tools should the agent access, and when and how should it invoke them?
Memory: How important is conversational and operational memory, and when should it be retrieved or compacted?

Linear Agent Harness: Deterministic Single-Pass Execution

Timestamp: [21:34]

Billy demonstrated a minimalist linear harness designed for deterministic workflows where looping is unnecessary. This pattern is ideal for targeted inspections, file transformations, or single-turn data analyses where you want a high level of determinism and need the agent to perform the exact same execution flow every single time.

Closed-Loop Agent Harness: Iterative Test-Driven Repair

Timestamp: [22:45]

When tasks demand active bug fixing and refactoring, a closed-loop harness provides the iterative reasoning required to reach a verified resolution.

In this demo, Billy showcased an agent that applies an automated code edit to address a failing requirement, and the harness executes the unit test suite against the updated codebase. If the tests fail, the runtime captures standard failure logs and detailed stack traces, feeding those error diagnostics directly back into the agent's working memory. The process repeats continuously until all unit tests pass, backed by a five-iteration ceiling to prevent infinite loops and runaway execution costs.

Guardrail Harness with Google's Agent Development Kit (ADK)

Timestamp: [23:25]

For developers who require custom behavior without rewriting core orchestration plumbing from scratch, Google's Agent Development Kit (ADK) provides scaffolding with automated memory management and execution safeguards.

Billy walked through an example that leverages ADK's native context compaction to summarize older conversational turns, preventing context window bloat during extended debugging runs. Custom interception hooks inspect and filter shell actions before execution, automatically stopping high-risk operations such as recursive file deletions, database drops, or unauthorized remote git pushes. This architecture gives teams fine-grained control over tool execution boundaries while avoiding the maintenance burden of bespoke harness frameworks.

The 3-Layer Agent Dev Stack: Gemini 3.8 Flash, Google Antigravity, and Google Skills

Timestamp: [25:27]

Next up, Smitha Kolan broke down why coding agents do not always require heavier reasoning models, emphasizing that high performance stems from balancing the three layers of the agent stack: Model, Harness, and Knowledge.

"Your coding agent doesn't need a smarter model. It needs a better stack: model, harness, and knowledge. When all three click into place, everything changes."

She then walked through the three tools she's been loving recently, one for each layer of the stack.

Layer 1 | Model | Gemini 3.8 Flash: High-frequency agentic loops run between 20 and 60 sequential hops per task (inspecting files, updating functions, and executing unit tests). Because latency and API costs compound across iterations, a lightweight, responsive model like Gemini 3.8 Flash makes real-time agent loops practical without running up a massive bill.

Layer 2 | Harness | Google Antigravity with /boost: Default Antigravity handles standard navigation and component creation. On top of that, the /boost command spins up an orchestrator that coordinates specialized sub-agents in parallel and concludes with an independent audit pass before modifying files.

Layer 3 | Knowledge | Google Skills Repository: With over 19,000 GitHub stars and 100+ curated domain packages across Google Cloud, Firebase, Flutter, and Maps, this harness-agnostic repository injects precise domain context on demand, preventing agents from guessing cloud configurations

Your turn to build

Building effective coding agents requires moving past the reflex of simply swapping in larger models. As Ryan Lopopolo's philosophy of harness engineering illustrates, true developer leverage is achieved by shifting best practices to the left and investing in rich tools, deterministic verifiers, and well-curated context that survive model upgrades. When combined with fast inference models, structured orchestration harnesses, and modular domain knowledge, agents evolve from conversational novelties into dependable, autonomous engineering partners.

Ready to put it into practice? Explore the tools and resources covered in this episode:

Gemini 3.8 Flash: Fast, low-cost model for multi-hop agent loops.
Google Antigravity (/boost): Orchestrator harness for parallel coding agents and verification.
Google Skills: Modular domain knowledge for Google Cloud, Firebase, and Flutter.
Agent Development Kit (ADK): Custom harness middleware for safety guardrails and memory compaction.
Full episode video: The full interview and live Factory Floor code demos.

Connect with us

Smitha Kolan → LinkedIn | YouTube | X
Luke Schlangen → LinkedIn | website
Ryan Lopopolo → LinkedIn | Hyperbola | X
Tilde Thurium → LinkedIn
Billy Jacobson → LinkedIn | X
Mollie Pettit → LinkedIn | dev.to | X | Bluesky

How Google Cloud Networking Supports Your Fluid Compute Choices for AI Workloads

Thu, 24 Sep 2026 13:00:00 +0000

The availability of resources for AI workloads can be challenging across the industry, especially accelerators. This can slow your AI workload deployment if it’s built around a specific type of accelerator. The concept of fluid compute allows you to design your AI deployment with several options based on available resources that can fit your use case.

In this blog, we will explore how Google Cloud networking supports your AI workloads and considerations that are relevant to your choice of accelerator (GPU or TPU), as the backend networking component configuration is not exactly the same.

The resource options

After deciding the type of work you want to achieve with your AI deployment, another important component is the actual hardware to get this done. In this case, we want to run inference for a private LLM, and the target is the NVIDIA B200 GPU family which is available in the A4 VMs (a4-highgpu-8g).

Now we have identified what we want to get done and a possible compute option, but the challenge is: is this available?

To get access to resources, there are several options which include:

Dynamic Workload Scheduler (Flex-start VM): Queues workloads until all required accelerator nodes are available at the same time, provisioning them together and running non-preemptibly for up to seven days.
Dynamic Workload Scheduler (calendar mode): Enables reserving accelerator capacity 1 to 90 days in advance with guaranteed start and end times, ideal for scheduled pre-training runs and benchmarking.
Future reservations: Guarantees access to committed hardware in a specified zone beginning at a specific future date.
Flex reservations: Offers short-term commitment windows to secure scarce accelerator nodes without multi-year lock-in.
Dynamic node auto-provisioning and ComputeClasses: In Google Kubernetes Engine (GKE), defining multi-family fallback lists within ComputeClasses allows the cluster to automatically attempt provisioning alternative accelerator types if primary pools face regional constraints.
Spot VMs: Delivers surplus compute at substantial discounts for fault-tolerant, checkpointed batch jobs.

Networking your choices

The networking component of the accelerator varies based on your choice, so let's explore four configurations: standard networking, accelerated GPU networking (TCPX/TCPXO and RoCEv2), TPU networking, and Cloud Run.

Standard networking

Supported accelerators: NVIDIA T4 (N1 series), NVIDIA L4 (G2 series), NVIDIA A100 (A2 machine series single-node and multi-node), Cloud TPU v3, and Cloud TPU v5e (single-host/standalone slices).
Architecture: Nodes communicate over the primary Virtual Private Cloud (VPC) network using the Google Virtual NIC (gVNIC) over standard TCP/IP.
Workload fit: Provides straightforward portability across Google Cloud compute environments, supporting distributed data preprocessing, decoupled pipeline stages, independent inference replicas, and computer vision workloads using standard VPC routing and network policies.

Accelerated GPU Networking (TCPX/TCPXO and RoCEv2)

Distributed training and multi-node inference require specialized multi-rail network fabrics to handle massive parameter exchanges and collective communications.

GPUDirect-TCPX and TCPXO Fabrics

Supported accelerators: NVIDIA H100 (A3 High VMs with 4 rails) and NVIDIA H100 Mega (A3 Mega VMs with 8 rails).
Architecture: Uses custom GPUDirect-TCPX (4 dedicated VPCs) and GPUDirect-TCPXO (8 dedicated VPCs) offload engines to achieve high-throughput multi-rail GPU communication over standard Ethernet infrastructure without requiring native RDMA hardware.
Deployment blueprints: These multi-VPC topologies can be deployed in many ways including using pre-built blueprints from the Cluster Toolkit.

RoCEv2 Fabrics (VM and Bare Metal)

Supported accelerators: NVIDIA H200 (A3 Ultra VMs), NVIDIA B200 (A4 VMs), NVIDIA GB200 NVL72 (A4X VMs), and NVIDIA GB300 (A4X Max Bare Metal).
Zonal network profiles: RoCEv2 operates over a dedicated RDMA VPC attached to a specialized zonal network profile: VM instances (A3 Ultra, A4, A4X) use the ZONE-vpc-roce profile, while Bare Metal instances (such as A4X Max) utilize the dedicated ZONE-vpc-roce-metal bare-metal profile.
Rail-aligned fabrics: This dedicated VPC is isolated strictly for GPU communication and contains subnets mapped directly to the accelerator NICs. The backend is rail-aligned, with support for Jumbo Frames (MTU 8896), delivering non-blocking multi-terabit bandwidth with minimal cross-rail interference.
Automated plumbing with GKE Dynamic Resource Allocation Network (DRANET): When deploying these GPUs on GKE, the GKE managed DRANET can be used to automatically provision additional networks and assign drivers that map the RDMA network interfaces to the GPU. These can then be assigned and consumed directly in your workload pods using standard Kubernetes resource claims.
Turnkey deployment: You can deploy this entire end-to-end stack—including RDMA VPCs, MTU tuning, and DRA drivers—using automated blueprints from the Cluster Toolkit.

TPU Networking

Supported accelerators: Cloud TPU v4, Cloud TPU v5p, Cloud TPU v5e (multi-host Pod slices), Cloud TPU v6e (Trillium), and TPU7x (Ironwood).
Inter-chip interconnect (ICI): Inside a TPU Pod or slice, chips communicate directly over dedicated, ultra-low-latency optical links organized in 2D or 3D torus meshes, bypassing traditional network stacks entirely.
Optical circuit switches (OCS): In TPU v4 and TPU v5p SuperPods, software-reconfigurable OCS units dynamically change physical network topologies, route around faulty trays, and provision custom-sized accelerator slices without manual recabling.
Multi-NIC architecture (TPU v6e and Higher): While earlier TPU generations relied on ICI within a slice and single-NIC for host traffic, Cloud TPU v6e (Trillium) and TPU7x introduce a native multi-NIC architecture where worker nodes isolate standard Kubernetes management traffic onto a primary VPC while using secondary dedicated VPCs configured for high-throughput TPU data and cross-slice communication.
DRANET for TPU deployments: When deploying these TPUs on GKE, the GKE managed DRANET can be used to automatically provision additional networks and assign drivers for TPU communication. These can then be assigned and consumed directly in your workload pods using standard Kubernetes resource claims.
Data-center network (DCN) Multislice: For models scaling beyond an individual TPU slice, Cloud TPU Multislice connects multiple independent ICI meshes over Google's high-speed Jupiter Data Center Network utilizing these dedicated multi-NIC paths.

Cloud Run

Supported accelerators: NVIDIA L4 (G2 series) and NVIDIA RTX PRO 6000 (Blackwell) on Cloud Run GPU services.
Direct VPC egress: Binds serverless containers directly to your private VPC network using sub-minute IP allocation via Direct VPC Egress, enabling secure, low-latency access to internal data lakes, databases, and private APIs without traversing the public internet or requiring legacy connector VMs.

Summary

Google Cloud networking options support various accelerator types. When using fluid compute you can adjust your network setup to support the best design to optimise your workloads performance.

Next Steps

Take a deeper dive into Google Cloud AI infrastructure and networking architectures with these resources:

Want to ask a question, find out more, or share a thought? Please connect with me on LinkedIn.

A guide to speeding up your video processing with AlphaEvolve

Wed, 23 Sep 2026 16:00:00 +0000

In real-time streaming, every millisecond counts.

For example, at 30 frames per second (fps), developers have a strict frame budget of just 33.3 ms (and only 16.6 ms at 60 fps) to ingest camera frames, run neural segmentation, apply shaders, and composite output. Exceeding that budget by even a fraction of a millisecond leads to dropped frames and stuttering.

Manual optimization is notoriously tedious — requiring weeks of analyzing flame graphs and hand-tuning low-level code in Swift, C++, or Metal. While standard AI coding assistants can generate boilerplate, they can’t optimize against target hardware, benchmark real-world latency, or ensure optimizations preserve visual fidelity.

Autonomous, closed-loop evolutionary optimization changes this paradigm. Tools like AlphaEvolve pair cloud-scale model reasoning with local hardware execution, and we’re already seeing real-world impact. In partnership with Google, DoIt used AlphaEvolve to autonomously optimize production Swift code in a live macOS streaming app, uncovering performance headroom that manual profiling missed (read the full technical writeup).

While this post focuses on video pipelines, the split-loop pattern applies anywhere performance matters — from microservice throughput and database queries to ML tensor pipelines and embedded systems. In every case, the formula is the same: pair Gemini code generation in the cloud with your domain-specific benchmark harness and automated quality gates.

Today, we’ll show you how to use AlphaEvolve to speed up video processing—and apply these principles to your own performance bottlenecks:

Understanding the split-loop architecture: How AlphaEvolve decouples managed cloud generation (Gemini model ensemble on Google Cloud) from local evaluation (e.g. compiling and timing native Swift/Metal code).
Evaluator craft and quality gates: How to construct scoring functions using metrics like Structural Similarity Index (SSIM) to prevent evolutionary loops from gaming the benchmark (e.g., skipping rendering entirely to go fast).
Autonomous algorithmic discovery: How Gemini-driven evolutionary search can autonomously discover unprompted framework APIs and make intelligent engineering trade-offs (e.g., frame-caching limits).
Setting realistic performance boundaries: How to measure code optimization against physical hardware floors.

1. Understanding AlphaEvolve’s split-loop architecture

AlphaEvolve runs a closed-loop evolutionary process: given a seed program and a custom scoring function, a mixture of Gemini models proposes code variations, executes the scoring function against each candidate, keeps the highest-performing code, and iteratively climbs toward an optimal solution over multiple generations.

A core architectural advantage of AlphaEvolve is its clean separation into two halves:

The generation half (Google Cloud managed service): Contains the prompt sampler, Gemini model ensemble, and program database. Google Cloud handles the scale, prompt orchestration, and generation mechanics.
The evaluation half (customer managed compute): Scoring code quality is strictly domain-specific. You own the evaluator module entirely, running it on your own hardware or target architecture (in this case, macOS running native Swift code).

While AlphaEvolve is Python-first on the cloud generation side, evaluation can be written in any language. The custom evaluator compiles each Swift candidate using swift and executes it against a standard reference webcam clip.

2. Evaluator craft and quality gates

An automated optimization loop like AlphaEvolve never actually "sees" your video stream. It only sees the numeric fitness score your evaluator returns. If your evaluation metric has a blind spot, evolutionary code generation will aggressively exploit it.

In our early runs, a naive fitness score weighted toward raw latency produced an astonishing speedup: the model simply bypassed blur rendering entirely and returned unmodified frames in 0 ms.

Structural Similarity Index Measure (SSIM):

To prevent the model from gaming your benchmark, try building a two-tiered scoring function that pairs throughput with structural fidelity metrics like Structural Similarity Index (SSIM):

code_block: <ListValue: [StructValue([('code', 'speedup = baseline_ms_per_frame / candidate_ms_per_frame\r\nssim = mean_ssim_vs_golden\r\n\r\n#Disqualify any candidate falling below visual threshold\r\n\r\n\r\nif ssim < 0.98 or worst_frame_ssim < 0.95:\r\n return {"speedup": -1e12} # Disqualified\r\n\r\nreturn {"speedup": speedup, "ssim": ssim}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e74107410>)])]>

What does this give you?

The ability to test against worst-case clips: Never benchmark on static frames or blank cameras. Candidate code can easily pass an average SSIM gate on static backgrounds while failing completely during quick head turns.
You can track the minimum, not just the mean: Enforce both an average threshold and a per-frame floor to catch dropped frames or delayed mask updates.

Autonomous algorithmic discovery:

Most developers use generative AI for local micro-optimizations (e.g., inlining helper functions, unrolling loops, or tweaking memory pools). But when given architectural room, the evolutionary loop can discover systemic optimizations on its own.

Engineering lessons:

Provide framework context, not isolated loops: Include public SDK headers, interface definitions, or API reference symbols in the prompt or retrieval harness. An LLM cannot adopt a sequence-aware subsystem if its context window only contains an isolated frame-processing callback.
Expose multi-frame lifecycle hooks: Let your candidate code maintain a bounded state across executions (e.g., historical masks or cache timestamps) rather than enforcing pure, stateless functions.
Let quality gates police the trade-offs: When AlphaEvolve introduced temporal mask caching, it initially cached masks too aggressively, causing noticeable trailing artifacts. Because our SSIM gate penalized drift during motion, the search converged on a production-ready cache window without manual parameter tuning.

Setting realistic performance boundaries

A common pitfall in performance engineering is optimizing in the dark. If you achieve a 2x speedup, is that an incredible achievement, or did you leave another 3x on the table?

In real-time media, total frame time splits into two distinct categories:

Mutable software overhead: Memory allocations, buffer format conversions, thread context switches, and API dispatch friction.
Immutable hardware floors: Raw Neural Engine inference latency, GPU shader compute time, and hardware display synchronization.

To make the most of AlphaEvolve, developers should measure against theoretical maximum headroom

Before running optimization loops, here’s a few principles to keep in mind:

Build a "no-op" pipeline: Strip out Swift/C++ orchestration, data marshalling, and frame conversions. Dispatch only the pre-warmed ML model and bare GPU pass on a dummy buffer. The resulting time is your physical hardware lower bound.
Calculate your addressable ceiling: Your total possible optimization potential is:

3. Score against the hardware gap: Instead of arbitrary speedup multiples, measure optimization efficiency:

Get started

All benchmark code, test clips, evaluation scripts, and raw candidate logs are open source:

GitHub repository: AlphaEvolve Camera Background Blur Example
Detailed technical write-up of our case study with DoIt: Running AlphaEvolve on Your Own Code

The DevFest Community Workshop Experience: Building Real Agents Together

Fri, 18 Sep 2026 16:00:00 +0000

This week we kicked off the DevFest season in North America at Google Hudson Square in New York City with 80 engineers packed into the room. Typical technical workshops hand you a finished repo, tell you to blindly paste blocks of code into your terminal, and hope nothing crashes. You walk away with green checkmarks, but your brain stays on autopilot.

We've introduced a completely different experience called Workbench.

Workbench focuses on understanding core ideas and architectural models rather than obsessing over syntax and code snippets. Instead of getting bogged down in boilerplate, engineers spent the day grappling with the actual mental models behind graph engineering, self-evolving architectures, and automated self-patching harnesses.

A glimpse into the Workshop Experience

At the DevFest Community Workshop, we spent one intense day building long-running, self-evolving multi-agent systems powered by Google's agentic stack. Ricky Robinett, Senior Director of Developer Marketing, kicked off the day by diagnosing why so many engineering teams hit a wall with agents. Ricky broke down why prompt engineering fails as a safety mechanism: English is just a probabilistic suggestion, not an execution boundary.

Right after Ricky, Rachel Francois, Google Developer Groups (GDG) North America Program Lead, took the stage alongside GDG Brooklyn organizers to welcome the community and spotlight the power of local developer chapters. They set the tone for the entire day, reminding everyone that building durable software works best as a team sport where engineers share real-world patterns and build local networks that outlast any single framework.

Getting hands on with labs

Annie Wang & Christina Lin, Americas DevRel Team members, led the morning lab that put those runtime ideas to work. Attendees explored Google's Agent Development Kit (ADK), Veo 3.1, Memory Bank on Gemini Enterprise Agent Platform, and RAG Engine on Gemini Enterprise Agent Platform. Through Workbench, developers grasped the principle of separating state from active compute for long running tasks. Workflows paused cleanly mid-execution, waited out asynchronous human approvals, and resumed without running up idle compute costs.

After lunch, Logan Hennessy, Americas Developer Relations Engineer (DRE), and Kartik Derasari, Google Developer Expert (GDE), led a lab using auction history as insight for better bidding strategy. Attendees worked through the architecture by integrating BigQuery data into autonomous data engineering pipelines, reasoning about deterministic bidding logic and adding eval-gated, self-patching harnesses that catch spend anomalies and update runtime execution safely.

Between lab blocks, we ran fast-paced speed quizzes where developers raced to lock in their answers as quickly as possible. Screens flashed, fingers flew across keyboards, and seconds made the difference between topping the leaderboard or dropping five spots. Nothing beats watching a room full of serious engineers completely lose their cool over a live quiz leaderboard.

Join a DevFest Community Workshop this fall

New York was only round one. We are taking this exact experience on tour to five more cities this fall. Find your city and grab your seat before spots fill up:

Sunnyvale on September 30
Washington DC on October 6
Atlanta on October 30 (as a part of DevFest Atlanta)
Seattle on November 4
Boston on November 10

Best practices for handling cloud reliability incidents

Tue, 15 Sep 2026 16:00:00 +0000

Cloud outages can range from global service disruptions to issues isolated to a specific region, zone, or even just your project, workload or application. If you suspect a Google Cloud Platform outage is impacting your services, we recommend you follow a structured “Verify→ Investigate→Report→Resolve→Review" workflow to resolve it. And before that outage occurs, you should also have prepared your environment for an eventual disruption by designing for failure, and actively practicing the steps you need to take to restore service.

In this blog, we summarize the key reliability incident handling best practices to help you design and practice your reliability incident response capabilities and minimize impact. Rather than an exhaustive guide, this is meant as a primer on only the most important practices for advisory purposes. Please note that we do not cover additional practices specific to security incidents here.

Beyond the base steps covered here, you may want to also explore how AI agents and tools are starting to transform incident handling. Check out this episode of the Prodcast, where Googlers explore the latest trends of leveraging agentic AI in Site Reliability Engineering (SRE) to detect issues early and prevent disruptions. Try Cloud Assist investigations, or explore Agent Skills and remote managed MCP servers to give you another set of tools for quickly pinpointing an issue. Before getting into these advanced techniques, we focus below on the foundational steps to good incident handling.

1. Prepare

Long before things start to go sideways, you should have spent significant time preparing for an outage along at least four dimensions: design, data, playbooks and training.

Design: Think ahead and mitigate future incidents by designing automated response actions, like a load balancer shifting traffic away from slow or unresponsive instances, or by automating as much of your incident response playbook as possible. Review designs of all critical applications to automate as many actions as possible to accelerate response and recovery.
Data: When a disruption occurs, having meaningful data at your fingertips vastly improves response capabilities. Use Cloud Logging, Cloud Trace and Cloud Monitoring, or other third-party observability tools, and replicate that data to a redundant stack in a separate location from the systems being observed. Make sure, in advance of any incident, that time stamps are synced across your observability streams for easy correlation, or know how to do that on-demand during an outage, when time is of the essence.
Playbook: A well-thought-out playbook documenting your incident response processes, including crystal clear role and responsibility definitions for all personas, is paramount to efficient incident response. Who is responsible to do what? Who needs to be notified or mobilized for each type of disruption? How can they be reached? What tools and data are available? How are results communicated? How do teams hand over to the next shift during long running incidents? etc. Conduct a simulated incident response and critically review every step to find where your playbook needs clarification. Without clear responsibilities, mitigation inevitably takes longer.
Training: Hopefully, service disruptions are rare events. To ensure your staff knows and remembers how to react, they need to retrain on the process several times per year by running simulated cross-team incident response drills. A retrospective on the simulated exercise will help identify warranted improvements.

2. Verify

Despite your best efforts, sooner or later, a service disruption will occur, which you can detect via any number of mechanisms:

Observability tools (Google tools or third-party tools)
Unified Maintenance Management notifications for planned maintenance
Personalized Service Health notifications managed with alert policies
Proactive customer monitoring by Google

Now, you need to determine what broke and who should ultimately fix the problem:

Google, e.g., a bug, code roll-out, hardware failure, etc.
You, e.g., a configuration change, elevated load, quota ceiling, etc.
Third party, e.g., a directory hosted by a different cloud provider

If Google has declared an incident and started working to fix the problem, estimate whether you can possibly reestablish service sooner, for example by failing over to a secondary stack (see the ‘Typical Causes’ table below). You can determine whether Google has declared an incident and will provide a fix by consulting:

Personalized Service Health: Check this first. Personalized Service Health shows incidents specifically relevant to your projects and regions, distinguishing between incident types:.
- Emerging Incidents: Google has received an alert, on-callers are investigating, impact is yet unknown
- Confirmed Incidents: Google has investigated and found customers are impacted

Located within the Google Cloud console, Personalized Service Health often displays limited-scope incidents that don't appear on the public dashboard. Personalized Service Health also offers a mobile client for Android and iOS smartphones, assuming you can use your work ID and credentials on the phone.

Gemini Cloud Assist, which is integrated with Personalized Service Health, so you can use it to query that information in natural language.
Cloud Service Health dashboard: This is the public-facing non-authenticated web page for broad, severe incidents affecting many customers. Limited blast radius disruptions are not externalized to the public. All its content is available in Personalized Service Health as well. If ever Personalized Service Health goes down, Cloud Service Health serves as an alternative channel built on a separate infrastructure.
Known Issues: In the console, navigate to Support > Cases, view a case, and use the resource selector on the console toolbar to find the specific cloud resource you’re interested in. Then click Known issues. If your issue matches one listed here, you can link a support case to it, so you will receive automatic updates in your case record. If you don’t find a match, open a new support case. Google will automatically match the case to a related incident, as soon as one is declared.
Google declared incidents are updated as new information becomes available, so check back regularly, or set up a Personalized Service Health alert policy to be notified each time new information becomes available.

If you host cloud resources in multiple clouds, a good practice is to check early on whether the problem occurs for multiple cloud providers. If so, the problem is likely external to the providers and caused either by you or by a third-party service that your application interacts with.

3. Investigate

To determine the blast radius within your cloud footprint of Google-declared reliability incidents, first check Personalized Service Health updates for a description of the technical problem. Knowing what to look for will allow you to map your blast radius and decide on suitable contingency actions quicker.

If Google hasn’t declared an incident, try to rule out configuration errors or issues within your environment by checking:

Cloud Monitoring: Look for spikes in error rates (e.g. 5xx errors), increased latency, or drops in traffic in your dashboards.
Cloud Logs: Use Log Explorer to look for specific error messages like DEADLINE_EXCEEDED, SERVICE_UNAVAILABLE, or specific API errors.
Quotas: Ensure you haven't hit a project quota (e.g., CPU, API rate limits), which can often mimic the behavior of an outage.
Change history: Check your log of recently applied changes. Not all problems manifest immediately, but proximity on a timeline can be a powerful indicator of causality, even if it’s not proof. Also check whether Google rolled out any updates just before the symptoms started. See the Unified Maintenance Management interface in Cloud Hub.

Absent a clear culprit, such as a traffic spike or a DDOS attack, and if symptoms manifested immediately after rolling out a change, a good strategy is to back out that change and attempt to return to a last known good configuration.

4. Report

If the Cloud Service Health and Personalized Service Health dashboards are green but your metrics show a failure, you must report it to Google.

Determine priority:

P1 (Critical): Your production service is unusable or severely impacted with no workaround.
P2 (High): Significant impact or degradation, but a workaround may exist.
See guidance on setting priority and guidance on describing your issue.

File a case: Go to Support > Cases > Create Case in the console.

Explain quantifiable business impact to rationalize the submitted priority and prevent it from being reset when Cloud Support prioritizes cases. A clear and accurate rationale helps!

Essential information to include:

Project ID and affected region/zone
Timestamps (when it started and if it's ongoing) with a clearly labeled timezone
Specific error messages or log snippets
Scope: Is it affecting all users/systems, or a specific subset/location?

Escalation for Premium/Enhanced support

If you have a Premium or Enhanced support plan and a P1 case is not receiving the attention it requires, use the Escalate button within the support case in the console. This alerts a support manager to investigate and rectify the situation.

5. Resolve

By taking these steps, you are well on your way to resolving the outage. In the meantime, here are some ways to mitigate the impact of the outage and communicate with impacted stakeholders.

While waiting for a resolution:

Communicate: Notify your stakeholders and customers. Transparency helps manage expectations and reduces duplicate internal reports.
Fail over: If you have a multi-regional architecture, consider shifting traffic to a healthy region. As a best practice, first ensure that the disruption is at the infrastructure level and not at your workload level.
Check for workarounds: While working on a permanent fix, Google often posts temporary workarounds in the Service Health Dashboard updates, or in Personalized Service Health updates.
Consider your regulatory reporting requirements: Know whether your organization is subject to regulatory reporting requirements, and what the required deadlines are for both initial and follow-up reporting. Google Cloud prepares Incident Reports for incidents that meet certain criteria — see details here for how to get those reports. Premium Support customers can also request an Incident Summary, which is an Incident Report customized to your account’s specific hosting location, time stamps, etc.

De-escalation and closure

Once systems are stable, Google downgrades the severity levels and deactivates the active on-call escalation chain. Google only closes an incident in Personalized Service Health when it has taken all the mitigation steps covering all impacted customers. Your specific services might be restored sooner than the incident closure time, if other customers are restored later than you. The incident is officially closed on the Google Cloud Status Dashboard when systems have run stably for a designated auto-close duration. Verify that your services are operating normally at this point. And if your incident responders aren’t compensated for extra time spent on the incident, find a way to thank them.

6. Review

After the problem has been fixed and operations have returned to a normal, steady state, it’s time to conduct a post-mortem analysis to identify how your team can respond better in future service disruptions. A “blameless” approach is essential to surfacing meaningful and impactful improvements that can be made to your incident response process. Ask questions like:

What went well?
What could we have done better?
Where did we get lucky?
Where did we get unlucky?

Then decide what changes can be made to improve your playbook, tools and training.

At Google, we often publish a post-mortem or Incident Report for major outages, available via Personalized Service Health. Review this to understand the root cause and adjust your own disaster recovery plans to prevent or reduce future impact. Customers with a Premium Support plan can request an Incident Summary for a Google-caused incident they were impacted by and for which they opened a P1 case. An Incident Summary is an Incident Report customized for your environment (e.g., start and end times of impact).

Typical causes, comms and prevention strategies

To help you prepare and plan ahead, here’s an overview of some typical incidents based on the symptoms reported in Cloud Service Health and Personalized Service Health along with guidance on what Google communications to expect, and some generic mitigation or prevention strategies you can build into your playbooks.

Blast radius	Typical cause	Comms	Strategy
Single zone or region.Subset of products.	Typical of a software problem triggered by a rollout. Learning points: - Understand the location scope (zones and regions) of your workload - Products can depend on other products	Major incidents are communicated via Cloud Service Health.Major and Minor (by number of customers, not severity) incidents are communicated via Personalized Service Health. Highly localized incidents are not communicated via Cloud Service Health or Personalized Service Health.	Fail over, if so configured, but verify the health of the secondary stack first.
Single zone.Most or all products.	Typical of a power or cooling issue.	Check Cloud Service Health and Personalized Service Health.	Fail over to a different zone, if so configured.
Single region. Most or all products.	Typical of a backbone networking infrastructure issue	Check Cloud Service Health and Personalized Service Health.	Fail over to a different region, if so configured.
Control plane issue for a product	Typical of a late detected issue	Communicated via Personalized Service Health if significant customer impact is verified.	Look for workarounds. Wait for Google to fix. Fail over, if so configured.
Multi-regional issue with a global product	Rare but possible, typically detected quickly. Learnings: Mitigation options can be limited. Try regional variants, alternative products with similar functionality	Check Cloud Service Health and Personalized Service Health.	Wait for Google to fix. In the meantime, verify via Google Comms and your own investigation that this is truly Google’s problem to fix.
Capacity / Stockout issue	System-level demand exceeding capacity in the product/location/model. (Cloud is designed to scale, but limits always exist, so proper planning is advised)	Error message. No incident will be declared.	Place reservations for predicted capacity needs (if cost is acceptable). Flexibility in zone placement can also help.
Quota exhaustion	Difficult / inaccurate prediction of traffic	Error message. No incident will be declared.	Review consumption trends against ceiling regularly.

Go deeper

This document offers only a condensed summary of key points. If you have an active Premium Support contract with Google Cloud, reach out to your account team for a deeper review of your response plans. For a comprehensive treatise on how to build reliable services and how to respond to incidents, we strongly recommend Google’s SRE Book, which is available as a free download. A new version of the SRE book is releasing ~Oct 2026 and will be available for purchase on O’Reilly Media. We’re also working on a future primer that explores AI-supported incident handling in-depth — stay tuned!

Introducing the Google Cloud Developer Plugin for AI Coding Agents

Thu, 10 Sep 2026 10:53:00 +0000

Agent skills fit well alongside documentation and remote MCP servers as ways of enabling the success of your AI workflows. They reduce context window usage for certain use cases, and they're straightforward to install. However, you might have noticed that managing individual skills can be unwieldy, or that some skills are most useful when they act alongside other skills or MCP servers toward the same goal. That's where plugins come in to help.

Today, we're thrilled to announce a new Google Cloud plugin for AI coding agents! Designed as installable bundles, agent plugins equip the AI agent of your choice with skills and tools to be more effective on Google Cloud.

Solving the tool coupling problem

As you expand your usage of coding agents, you might find that they become significantly more capable when they use related skills in tandem or with complementary context and tooling. For example, an agent analyzing infrastructure is more effective when combining domain knowledge, workflow recommendations, and the ability to interact with a live environment together.

Plugins solve this coupling challenge by packaging related capabilities into cohesive, installable bundles. This allows you to take advantage of both broad foundational capabilities and deep, product-specific tools without managing complex dependencies. In the Google Agent Skills repository, you'll start to see the following for Google Cloud popping up over time:

Foundational plugins: Essential platform-wide guidance for things like documentation discovery, project configuration, and architectural design.
Domain-focused plugins: Specialized knowledge and best practices for technical areas in the context of Google Cloud.

For this release, we've started with a foundational plugin that supports agent functionality for all Google Cloud users, focusing on making it easier for agents to retrieve Google Cloud-related skills, make use of official documentation, and handle programmatic interactions with Google Cloud.

Built on an open standard

We've also built our plugin in compliance with the Agent Plugins specification, an open, vendor-neutral standard for packaging Agent Skills and Model Context Protocol (MCP) servers into portable, interoperable units. Rather than requiring developers to maintain different configurations and wrappers for every AI assistant, the Agent Plugins standard provides a unified manifest and directory structure.

Our Google Cloud plugin adopts this standard to ensure that developers across a variety of AI coding environments get consistent, high-quality access to tools that help them succeed with Google Cloud. That includes not only the plugins we talk about today, but all other plugins published to the Google Agent Skills repository as well.

Let's take a look at the flagship plugin that we've just published in the Google Agent Skills repository: google-cloud-developer. This plugin exists to help agents successfully navigate the fundamentals of interacting with Google Cloud: things like authentication, authorization, managing projects, and guardrails for gcloud CLI operations. This plugin also bundles configuration for the Developer Knowledge MCP server, which gives agents up-to-date grounding in Google's official developer documentation.

Plugin in action: Project onboarding and identity authentication

To see how this plugin works, consider a situation where you're bootstrapping a new project as part of working on a script. With the google-cloud-developer plugin installed, you can prompt your agent:

I'm brand new to this platform, and I need to get an account and a first project with billing set up. Then, I need my local machine authenticated so a script that I'm writing can call the APIs as a service identity instead of as me.

Environment awareness: The agent silently runs background checks against your live environment for prerequisites like CLI availability and potential existing projects or organizations.
Review: The agent considers IAM best practices to avoid risks that might be assumed as part of the prompt, like accidental key leaks or git commits.
Interaction with guardrails: The agent outlines a workflow roadmap and offers to act on those steps before modifying any resources.

Installing Google Cloud plugins

Because Google Cloud plugins are available from the open Google Agent Skills repository and adhere to the standard Agent Plugins layout, adding them to your environment is straightforward. For example, here's how you'd install the google-cloud-developer plugin:

Antigravity CLI

Install the plugin directly via the CLI using its path in the Google Agent Skills repository:

code_block: <ListValue: [StructValue([('code', 'agy plugin install https://github.com/google/skills/plugins/cloud/google-cloud-developer'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e679959d0>)])]>

Enable the Developer Knowledge API in your Google Cloud project by using the gcloud CLI:

code_block: <ListValue: [StructValue([('code', 'gcloud services enable developerknowledge.googleapis.com --project=<YOUR_PROJECT_ID>'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e75118e10>)])]>

Claude Code

Add the Google plugins marketplace, then install the plugin:

code_block: <ListValue: [StructValue([('code', 'claude plugin marketplace add google/skills\r\nclaude plugin install google-cloud-developer@google-plugins'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e67c35510>)])]>

Enable the Developer Knowledge API in your Google Cloud project by using the gcloud CLI:

code_block: <ListValue: [StructValue([('code', 'gcloud services enable developerknowledge.googleapis.com --project=<YOUR_PROJECT_ID>'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e673847d0>)])]>

Create an API key for the Developer Knowledge API by following the instructions Create and secure the API key. Save the key you create in a secure location.

Export the API key to your environment before starting Claude Code:

code_block: <ListValue: [StructValue([('code', 'export DEVELOPERKNOWLEDGE_API_KEY=<YOUR_API_KEY>'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e67385e50>)])]>

Note: The DEVELOPERKNOWLEDGE_API_KEY environment variable needs to be set in the environment before you start Claude Code. Consider adding this export to your shell's startup script (e.g. .bashrc, .zshrc) for convenience.

Codex CLI

Add the Google plugins marketplace, then install the plugin:

code_block: <ListValue: [StructValue([('code', 'codex plugin marketplace add google/skills\r\ncodex plugin add google-cloud-developer@google-plugins'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e67317ad0>)])]>

Enable the Developer Knowledge API in your Google Cloud project by using the gcloud CLI:

code_block: <ListValue: [StructValue([('code', 'gcloud services enable developerknowledge.googleapis.com --project=<YOUR_PROJECT_ID>'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e66fdfa90>)])]>

Create an API key for the Developer Knowledge API by following the instructions Create and secure the API key. Save the key you create in a secure location.

Enable authenticated access to the Developer Knowledge MCP server by updating ~/.codex/config.toml (or your project's .codex/config.toml) to include the following lines:

code_block: <ListValue: [StructValue([('code', '[mcp_servers.developer-knowledge]\r\n url = "https://developerknowledge.googleapis.com/mcp"\r\n env_http_headers = { "X-Goog-Api-Key" = "DEVELOPERKNOWLEDGE_API_KEY" }'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e66fdc610>)])]>

Export the API key to your environment before starting Codex:

code_block: <ListValue: [StructValue([('code', 'export DEVELOPERKNOWLEDGE_API_KEY=<YOUR_API_KEY>'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e66fdda90>)])]>

Note: The DEVELOPERKNOWLEDGE_API_KEY environment variable needs to be set in the environment before you start Codex. Consider adding this export to your shell's startup script (e.g. .bashrc, .zshrc) for convenience.

Note: After performing this configuration update, a status of not logged in for the Developer Knowledge MCP server is expected and doesn't block access to the server.

Next Steps

If you're already a Google Cloud user, try the above installation steps to set up your agent for success. We think you'll like what you see! For those who want a more guided approach, our new codelab will walk you through the installation and initial exploration of the plugin in Antigravity.

If you're new to Google Cloud, you can also get started with instructions in our documentation to set yourself up for local development.

The most curious readers can also take a deeper look at the plugins and agent skills available to use today in the Google Agent Skills repository.

Power agent hubs or custom harnesses with the Antigravity SDK in one toolkit

Tue, 08 Sep 2026 16:00:00 +0000

Enterprise agent adoption isn’t one-size-fits-all. While many teams will opt for managed commercial platforms, such as Gemini Enterprise Agent Platform for turnkey agent deployment and governance, developers with bespoke workflows or custom execution engines often choose to build their own lightweight agent hubs.

If you are building a centralized agent hub from the ground up, you need tools that run predictably, log everything, and stay in their sandbox. The Antigravity SDK gives you the exact runtime engine used in Antigravity 2.0 and the Antigravity CLI, adding declarative safety policies, real-time telemetry, and stateful multi-turn persistence straight into your application. When the core runtime updates, your SDK agents get those optimizations automatically. That's why today, we're breaking down how the Antigravity SDK powers a complete multi-agent control plane.

How Antigravity comes together

A multi-agent control plane monitors and manages LLM workloads. It shows you exactly what the agent is thinking, which tools it calls, and how it stores state.

It consists of two critical components:

1. Antigravity SDK agent core: The runtime that manages model interactions (like Gemini 3.1 Pro and Gemini 3.8 Flash), runs tools, generates thinking traces, and executes skills.

2. Observability and telemetry middleware: An event-driven layer powered by Antigravity SDK Lifecycle Hooks. It intercepts agent actions like step starts, thinking updates, and tool calls, and streams telemetry over WebSockets to your dashboard.

Use case: Multi-agent monitoring and interactive control

Let's explore a scenario where an organization is building or maintains a custom agent hub and wants to integrate Antigravity SDK-powered agents.

The problem: An operations engineer needs to monitor multiple active agents (e.g., gemini-pro-agent, github-agent, email-agent) performing background research, document summarization, and task scheduling. Traditionally, observing agent progress requires:

Tailing fragmented console logs across multiple terminal windows
Manually inspecting JSON transcripts to diagnose stuck or failing tool calls
Lack of visibility into which Skills or MCP connectors are loaded for a given agent session
Difficulty tracking cumulative token usage and execution latency

The solution: This post walks through each one: the streaming API for real-time observation, lifecycle hooks for telemetry and interception, the policy engine for steering, skills for capability management, and session state for persistence. With an SDK-powered dashboard, operators get a single view into what every agent is doing.

What happens behind the scenes?
When an operator or dashboard interacts with an Antigravity agent, the runtime coordinates execution through five core mechanisms:

Session initialization and state attachment (save_dir & conversation_id):The runtime initializes or reattaches to a session, binding execution to a root save_dir. Multi-turn trajectory logs, tool receipts, and artifacts are preserved under traj-<conversation_id> for persistent auditability.
Skill resolution (skills_paths):Domain-specific capabilities and instructions are resolved directly from filesystem paths pointing to SKILL.md bundles, dynamically augmenting the agent's system prompt without an external registry.
Concurrent stream generation (ChatResponse):The runtime exposes three concurrent async iterators over the single model response:

response (yields visible text tokens)
response.thoughts (yields internal chain-of-thought reasoning deltas)
response.tool_calls (yields typed ToolCall events containing .name and .args)

Declarative sandboxing and built-in tool execution:When the agent performs workspace operations, built-in tools (list_directory, find_file, search_directory, view_file, create_file, edit_file) execute strictly within configured workspaces directories governed by safety policies (such as policy.workspace_only()).
Telemetry interception via lifecycle hooks:Decorated async hook functions (@hooks.on_session_start, @hooks.pre_tool_call_decide, @hooks.post_tool_call, @hooks.on_session_end) intercept agent transitions in real time, validating or modifying tool calls and broadcasting telemetry payloads over WebSockets to the live dashboard.

The Antigravity SDK organizes these responsibilities into four core building blocks:

1. Modular capabilities with Skills

Skills provide reusable, domain-specific instruction bundles and reference assets that agents load dynamically. Rather than managing an in-memory registry, skills are resolved directly from filesystem directories containing a SKILL.md file:

code_block: <ListValue: [StructValue([('code', 'from google.antigravity import Agent, LocalAgentConfig\r\n\r\n# Pass directory paths containing SKILL.md bundles directly to config.\r\n# The runtime dynamically resolves and injects them into the prompt.\r\nconfig = LocalAgentConfig(\r\n model="gemini-3.8-flash",\r\n system_instructions=(\r\n "You are an enterprise operations assistant equipped with "\r\n "specialized operational skills."\r\n ),\r\n skills_paths=["./skills/research", "./skills/code_review"],\r\n)\r\n\r\nasync with Agent(config) as agent:\r\n response = await agent.chat("Analyze the deployment logs.")'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e743be310>)])]>

2. Sandboxed built-in tools and workspace scoping

The SDK provides production-ready file and workspace tools out of the box, which removes the need to write custom filesystem wrappers. When paired with workspaces and declarative safety policies, tools are strictly confined to authorized directories:

code_block: <ListValue: [StructValue([('code', 'from google.antigravity import Agent, LocalAgentConfig, types\r\nfrom google.antigravity.policies import policy\r\n\r\nconfig = LocalAgentConfig(\r\n model="gemini-3.8-flash",\r\n # Selectively enable built-in tools via CapabilitiesConfig\r\n capabilities=types.CapabilitiesConfig(\r\n enabled_tools=[\r\n types.BuiltinTools.LIST_DIR, # "list_directory"\r\n types.BuiltinTools.FIND_FILE, # "find_file"\r\n types.BuiltinTools.SEARCH_DIR, # "search_directory"\r\n types.BuiltinTools.VIEW_FILE, # "view_file"\r\n types.BuiltinTools.CREATE_FILE, # "create_file"\r\n types.BuiltinTools.EDIT_FILE, # "edit_file"\r\n ]\r\n ),\r\n # Enforce filesystem isolation: operations outside these paths are blocked\r\n workspaces=["./workspace"],\r\n policies=[policy.workspace_only()],\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e75212250>)])]>

3. Session isolation and trajectory persistence (`save_dir` & `conversation_id`)

State persistence in the Antigravity SDK is managed through declarative configuration rather than an external database. Specifying a save_dir establishes a root directory where full turn trajectories, tool receipts, and artifacts are preserved under traj-<conversation_id>:

code_block: <ListValue: [StructValue([('code', 'from google.antigravity import Agent, LocalAgentConfig\r\n\r\nconfig = LocalAgentConfig(\r\n model="gemini-3.8-flash",\r\n # Root directory storing all conversation trajectories\r\n save_dir="./storage/sessions",\r\n # Supply conversation_id to reattach to an existing trajectory;\r\n # omit it to let the SDK mint a new ID on the first turn.\r\n conversation_id="ops-session-20260820-001",\r\n)\r\n\r\nasync with Agent(config) as agent:\r\n # Resumes prior context and continues the multi-turn session seamlessly\r\n response = await agent.chat("Summarize the issues identified in the last turn.")'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e75213e50>)])]>

4. Real-time telemetry and interception with lifecycle hooks

Lifecycle hooks allow dashboards and monitoring engines to observe and steer every stage of execution. Using decorated async functions, you can stream status updates over WebSockets, inspect tool parameters, and enforce human-in-the-loop approvals before tools run:

code_block: <ListValue: [StructValue([('code', 'from google.antigravity import Agent, LocalAgentConfig, types\r\nfrom google.antigravity.hooks import hooks\r\n\r\n# 1. Session start & end telemetry\r\n@hooks.on_session_start\r\nasync def on_session_start():\r\n broadcast_to_dashboard({"type": "STATUS", "status": "RUNNING"})\r\n\r\n@hooks.on_session_end\r\nasync def on_session_end():\r\n broadcast_to_dashboard({"type": "STATUS", "status": "IDLE"})\r\n\r\n# 2. Intercept tool calls before execution (human-in-the-loop / audit gate)\r\n@hooks.pre_tool_call_decide\r\nasync def intercept_tool(tool_call: types.ToolCall) -> types.HookResult:\r\n broadcast_to_dashboard({\r\n "type": "TOOL_CALL",\r\n "tool": tool_call.name,\r\n "args": tool_call.args,\r\n })\r\n # Return HookResult to approve or block execution\r\n return types.HookResult(allow=True)\r\n\r\n# 3. Post-execution tool receipts\r\n@hooks.post_tool_call\r\nasync def record_tool_result(result):\r\n broadcast_to_dashboard({\r\n "type": "TOOL_RESULT",\r\n "tool": result.name,\r\n "error": getattr(result, "error", None),\r\n })\r\n\r\nconfig = LocalAgentConfig(\r\n model="gemini-3.8-flash",\r\n hooks=[on_session_start, on_session_end, intercept_tool, record_tool_result],\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e752110d0>)])]>

Get started

Get started with your own enterprise agent control plane using the following resources:

Spanner migrations: Automating dual-write with Antigravity CLI for minimal disruption

Fri, 04 Sep 2026 16:00:00 +0000

When Google's Finance Engineering team needed to modernize their legacy data layer, they chose Spanner, a globally distributed, strongly consistent, multi-model database with high availability capabilities. But migrating to Spanner without taking production services offline was a daunting engineering challenge: As the internal team responsible for the application, we needed to manually rewrite dual-write logic across dozens of Data Access Objects (DAOs), a process that is slow and prone to human error. Further, doing so without disruption would have required implementing multi-phase dual-write architectures across every DAO in our codebase.

To solve this, we took an alternative approach: We built an automated refactoring pipeline powered by Antigravity CLI in headless mode. This helped us accelerate our migration velocity significantly while maintaining strict data parity in our staging environments as we prepare for production.

The challenge: Anatomy of a dual-write migration

When migrating high-throughput production services where financial accuracy is essential, simple cutover scripts do not work. You must verify that both the legacy datastore and Spanner receive identical writes simultaneously until all the historical data backfills and verifications are complete.

We structured our migration across three distinct phases:

Historical backfill: Copying existing historical records to Spanner while maintaining referential integrity.
Dual-write / dual-read implementation: Modifying every DAO to write mutations to both the primary store and Cloud Spanner in parallel during the migration window.
Automated API verification and parity checking: Intercepting RPC traffic and verifying end-to-end that every write lands with byte-for-byte equivalence across both stores.

The architectural pattern is clean, but at our scale, we began to encounter friction. That’s because each DAO requires:

A dedicated MutationConverter class mapping complex domain models to Spanner schema columns
Dual-write branch handling and rollback or error-reporting logic
A suite of unit tests verifying both primary and Spanner writes using fake time sources and test doubles (FakeTimeSource)

Performing these identical, high-precision code changes across 30+ DAOs by hand would have taken months of engineering time.

The solution: Standardized mutation converter patterns

To verify that our automation pipeline could reliably generate clean code, we first standardized our DAO refactoring pattern around a decoupled MutationConverter interface.

Instead of embedding raw Spanner table names and column assignments directly inside core DAO business logic, we isolate Spanner schema translation into dedicated converter units:

code_block: <ListValue: [StructValue([('code', '// Example of the standardized pattern generated by our pipeline\r\n\r\ntype BpcTransferAmountsMutationConverter interface {\r\n ToInsertMutation(entity *model.BpcTransferAmount) (*spanner.Mutation, error)\r\n ToUpdateMutation(entity *model.BpcTransferAmount) (*spanner.Mutation, error)\r\n}\r\n\r\ntype bpcTransferAmountsMutationConverterImpl struct {\r\n tableName string\r\n}\r\n\r\nfunc (c *bpcTransferAmountsMutationConverterImpl) ToInsertMutation(entity *model.BpcTransferAmount) (*spanner.Mutation, error) {\r\n if entity == nil {\r\n return nil, errors.New("entity cannot be nil")\r\n }\r\n \r\n // Map domain fields to Cloud Spanner table schema\r\n cols := []string{"TransferId", "AmountCents", "CurrencyCode", "LastModifiedTimestamp"}\r\n vals := []interface{}{\r\n entity.TransferId,\r\n entity.AmountCents,\r\n entity.CurrencyCode,\r\n spanner.CommitTimestamp, // Use Spanner commit timestamps\r\n }\r\n \r\n return spanner.Insert(c.tableName, cols, vals), nil\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e753f6910>)])]>

By establishing a rigid, deterministic contract between the DAO and the Spanner SDK (spanner.Mutation), we created an exact target specification that an AI coding agent could reason about and generate reliably.

Why use Antigravity CLI in headless mode?

Interactive AI chat interfaces in IDEs work well for exploratory coding, but they are poorly suited for systematic, multi-file code updates across an entire codebase. When you need to apply repeatable refactoring to dozens of targets without missing edge cases, you need automated workflows.

We addressed this by building an orchestration script (migration_ui.py) that runs Antigravity CLI in headless mode (-p).

Headless mode lets Antigravity run directly inside shell scripts, continuous integration pipelines, and background automation jobs without requiring manual terminal prompts. This approach helped us scale our work in three key ways:

Deterministic prompt architectures: We treated our prompts as version-controlled engineering artifacts. We codified precise rules handling common Spanner edge cases — such as timestamp serialization, nullability conversions, mutation ambiguity, and FakeTimeSource test injection — directly into reusable prompt templates.
Batch execution and automated verification: Our orchestration script takes a target DAO name as input, retrieves the existing single-write source code and schema, and feeds it to headless Antigravity alongside our structural conventions. Antigravity generates the new converter, the refactored dual-write DAO, and corresponding unit tests. The script then runs blaze test. If a linter error or test assertion fails, the error log feeds directly back into Antigravity for self-correction.
Overnight execution at scale: Because the loop runs unattended, engineers can queue up 10 DAOs at the end of the day. By morning, the pipeline generates, tests, and validates 10 clean changelists ready for human code review.

Results and key takeaways for cloud engineers

Combining Spanner's distributed database primitives with Antigravity CLI's headless automation produced clear benefits across our engineering organization:

Significant reduction in migration effort: DAO dual-write migrations that previously required extensive manual coding and testing were completed and reviewed in a fraction of the time
Highly reliable data migration: Because every generated DAO adhered to the exact same tested MutationConverter pattern and underwent automated unit testing against Spanner test doubles, we sustained high data fidelity during our extensive migration testing.
Focus on higher-value engineering: Engineers avoided repetitive boilerplate refactoring, giving them time to focus on data modeling, architectural resilience, and performance optimization.

Three tips for your next database migration

Decouple schema translation first: Before writing migration scripts, define a strict interface (like our MutationConverter) that isolates your new cloud database SDK requirements from your existing business logic. AI agents work best when given clear, bounded design patterns.
Move from interactive chat to headless automation: When executing repetitive refactoring across more than three or four files, invest in scripted, headless workflows. Treating prompt inputs and test verifications as automated build steps help maintain quality and consistency.
Let the build system act as your guardrail: Connect your AI generation loop directly to your build and test harness (bazel test or go test). This lets the model fix compile and assertion errors before a developer reviews the code.

Get started

Whether you’re migrating financial systems or building cloud-native applications from scratch, Spanner and Antigravity provide a foundation for scalable software development.

Explore Cloud Spanner: Learn more about Spanner's distributed architecture Google Cloud Spanner documentation.
Discover Gemini for Developers: See how AI-assisted coding and headless CLI automation can assist your engineering workflows at Google Cloud AI for Developers.

Not All LLM Workloads Are Equal: Benchmarking TPU Performance on Classification vs. Generation

Fri, 04 Sep 2026 15:36:00 +0000

Moving Large Language Models (LLMs) from experimental prototypes into enterprise production exposes a critical truth: your infrastructure dictates both your performance ceilings and your unit economics. Standard hardware benchmarks often ignore a fundamental reality—not all LLM requests stress the silicon in the same way.

In this post, we dive into a comprehensive benchmarking exercise comparing Gemma 3 12B and Gemma 3 27B on Google Cloud TPU v6e to answer a crucial architectural question: How does TPU infrastructure actually perform when tasked with structurally distinct workloads at scale?

Key Findings and Suggestions

Before diving into the methodology, here are the critical takeaways for architects deploying Gemma 3 on TPU v6e:

The Generation Performance Wall

For decode-heavy generation tasks, the Gemma 3 27B model hits a strict performance wall past 64 concurrent users, plateauing at a 4.12x normalized throughput multiplier at 128 users. In contrast, the 12B model scales up to an 8.19x multiplier.

Suggestion: If your workload requires high-concurrency generation, downsize to the 12B model, or set strict pod-autoscaling limits capping concurrent requests at 64 per replica for the 27B model.

The Classification Parity

For prefill-heavy classification tasks, model parameter size matters significantly less. Both the 12B and 27B models achieve similar peak scaling (around 6.0x to 6.4x normalized throughput at 128 users) without saturating the TPUs.

Suggestion: You can safely deploy larger, more capable models for summarization or classification workflows without paying a throughput penalty. The average --max-num-seqs or --max-model-len should be kept judiciously based on the average user load and average tokens per request, without which there might be request drops.

Designing Around the Wall

Hardware saturation manifests as severe latency spikes and silent request dropouts. To mitigate this, do not rely on standard CPU/Memory scaling triggers. Instead, scale based on End-to-End (E2E) latency metrics, and implement aggressive vLLM bucket padding optimizations (VLLM_TPU_BUCKET_PADDING_GAP) to conserve memory.

The Architecture Setup

The inference stack can be divided into three core pillars:

1. Infrastructure: GKE & TPU

The foundation of our deployment is a Google Kubernetes Engine (GKE) Autopilot cluster. Connected to this is a single-host TPU v6e node pool configured with a 2x2 chip topology.

2. Software & Tools: vllm

For the serving framework, we leveraged vllm via vllm-project/tpu-inference.

3. Models: Gemma 3 12B and 27B

We evaluated two highly capable open-weights models: Gemma 3 12B and Gemma 3 27B. These models were accessed via HuggingFace.

The Workloads: Classification vs. Generation

Not all LLM requests stress the system equally. We benchmarked two distinct scenarios: Classification and Generation, across 16, 32, 64, and 128 concurrent users:

Classification (High Input, Low Output): This use case mimics an e-commerce compliance task. The prompt includes large blocks of product rules, item descriptions, and OCR-extracted text. The output is exceptionally small—typically just classifying an item as "Allow" or "Prohibit". Input Sequence Length (ISL) is ~4,000 tokens and Output Sequence Length (OSL) is ~10 tokens.
Generation (Low/Medium Input, High Output): This use case mimics long-form text generation. The prompt requests a detailed, analytical policy brief on the future of AI in the labor market. The model spends the majority of its time decoding and streaming out hundreds of tokens. Input Sequence Length (ISL) is 500 tokens and Output Sequence Length (OSL) is ~1,000 tokens.

Results and Observations

We measured metrics like Throughput (requests/sec), End-to-End Latency and the results provided some fascinating insights into how parameter size and hardware bandwidth interact. To ensure architectural consistency, every benchmark was executed using the vllm-project/tpu-inference hardware plugin, leveraging a standardized global serving configuration of max-model-len=128000, max-num-batched-tokens=8192, and max-num-seqs=512.

Generation Scaling Divergence

In Generation tasks, both models perform similarly up to 64 concurrent users. However, at 128 concurrent users, the Gemma 3 12B model shows significantly better scaling, achieving an 8.19x normalized throughput multiplier compared to a 4.12x plateau for the Gemma 3 27B model (normalized against the Gemma 3 12B baseline at 16 users). This suggests that the larger 27B model hits memory or compute limits much earlier under high generation loads.

Concurrent Users	Gemma 3 12B Throughput (req/s)	Gemma 3 27B Throughput (req/s)
16 users	1.00 x	1.05 x
32 users	1.98 x	1.97 x
64 users	2.96 x	4.00 x
128 users	8.19 x	4.12 x

aside_block: <ListValue: [StructValue([('title', 'Pro Tip → Metrics Inflation at High Concurrency'), ('body', <wagtail.rich_text.RichText object at 0x7f7e7414e5d0>), ('btn_text', ''), ('href', ''), ('image', None)])]>

Classification Performance Parity

In Classification tasks, there is negligible difference in scaling behavior between the Gemma 3 12B and Gemma 3 27B models. Both models operate efficiently within the hardware's capacity and scale well, reaching peak normalized throughputs of approximately 6.04x to 6.37x at 128 concurrent users (normalized against the Gemma 3 12B baseline at 16 users).

Concurrent Users	Gemma 3 12B Throughput (req/s)	Gemma 3 27B Throughput (req/s)
16 users	1.00 x	0.76x
32 users	1.18x	1.53x
64 users	2.04x	3.15x
128 users	6.37x	6.04x

Latency Threshold Analysis

End-to-End (E2E) latency exhibits different scaling behaviors depending on the model size and task. When using identical serving hyperparameters (--max-num-seqs=512), the Gemma 3 12B model's Classification latency roughly doubles when moving from 32 users to 64 users, indicating resource contention. However, for the larger Gemma 3 27B model, Classification latency remains relatively flat between 32 and 64 users before doubling at the 128-user mark.

Model	Task	16 Users	32 Users	64 Users	128 Users
Gemma 3 12B	Generation	1.00x	1.13x	1.40x	1.70x
Gemma 3 12B	Classification	1.00x	0.99x	1.79x	2.90x
Gemma 3 27B	Generation	1.20x	1.68x	2.93x	3.33x
Gemma 3 27B	Classification	1.20x	1.95x	1.95x	3.88x

aside_block: <ListValue: [StructValue([('title', 'A Crucial TPU Optimization Technique'), ('body', <wagtail.rich_text.RichText object at 0x7f7e677a5ed0>), ('btn_text', ''), ('href', ''), ('image', None)])]>

Conclusion

Benchmarking Gemma 3 12B and 27B models on Google Cloud TPU v6e architecture reveals that raw parameter count is not the sole predictor of inference performance; rather, the interaction between the serving framework, hardware topology, and workload token ratios dictates efficiency. For generation tasks (low input, high output), the 12B model proves superior at high concurrency, sustaining an 8.19x relative throughput multiplier where the 27B model saturates at 4.12x. Conversely, for prefill-heavy classification tasks, both models perform similarly, allowing organizations to deploy larger models without a severe scaling penalty. Our evaluation also mapped exact hardware saturation thresholds—such as End-to-End latency doubling at 64 users for classification and hitting a cliff at 128 users for generation—enabling precise, data-driven auto-scaling triggers rather than costly over-provisioning. Ultimately, achieving these peak metrics requires aggressive tuning of vllm parameters, such as adjusting batched tokens and configuring TPU-specific bucket padding to prevent compute waste, proving that cost-effective AI infrastructure must strictly align model selection and serving configurations to the unique input/output profiles of production workloads.

Ready to scale your LLM workloads?

Don't let unoptimized infrastructure bottleneck your enterprise AI rollouts. Now that you know how different workload shapes impact hardware saturation, it's time to put these insights into practice:

Use these benchmarks to right-size your production architecture. Safely leverage the larger Gemma 3 27B for prefill-heavy classification tasks without a throughput penalty, but consider switching to the 12B model to maintain linear scaling for decode-heavy generation at high concurrency.
Deploy using Google Kubernetes Engine (GKE) with TPU v6e node pools to build a highly scalable, managed AI foundation and dedicated vllm-project/tpu-inference hardware plugin. Alternatively, you can also deploy via Model Garden on Gemini Enterprise Agent Platform or you can spin up TPU VMs for serving Gemma 3 models.

Have you encountered similar performance walls in your own production deployments? Share your scaling strategies, ask questions, and join the discussion in the Google Cloud Community forums.

Announcing the Google Gen AI SDK for Kotlin 1.0: Idiomatic multiplatform access to Gemini

Thu, 03 Sep 2026 09:00:00 +0000

Integrating modern generative AI capabilities into Kotlin applications shouldn't require juggling raw HTTP clients or bridging disparate Java libraries. Today, we're excited to announce the 1.0 release of the Google Gen AI SDK for Kotlin (google-genai-kotlin). You can dive right into the code, explore runnable samples, and star the project today on GitHub at googleapis/kotlin-genai.

Built from the ground up as a Kotlin Multiplatform (KMP) library, the SDK brings idiomatic Kotlin paradigms (including first-class Coroutines, asynchronous Flow streaming, and immutable data classes with named and default parameters) to developers specifically targeting the JVM (backend services, serverless functions, desktop).

The SDK provides a unified surface to interact with both the Gemini Developer API (Google AI Studio) and the Gemini Enterprise Agent Platform (on Google Cloud) with minimal configuration tweaks.

Although this SDK is a Kotlin Multiplatform library, using it directly from a mobile app is blocked for security reasons. Instead, use Firebase AI Logic for direct access to Gemini models from mobile apps.

1. Getting started: Adding the dependency

The SDK is published to Maven Central under com.google.genai:google-genai-kotlin.

Kotlin Multiplatform (KMP)

For multiplatform applications, add the dependency to your commonMain source set:

code_block: <ListValue: [StructValue([('code', '// build.gradle.kts\r\nkotlin {\r\n sourceSets {\r\n commonMain.dependencies {\r\n implementation("com.google.genai:google-genai-kotlin:1.0.0")\r\n }\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e670d2590>)])]>

Standard JVM projects

For single-platform Kotlin projects, Gradle automatically selects the optimal variant via Gradle Module Metadata:

code_block: <ListValue: [StructValue([('code', '// build.gradle.kts\r\ndependencies {\r\n implementation("com.google.genai:google-genai-kotlin:1.0.0")\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e6705add0>)])]>

2. Unary and streaming text generation and chat

The primary entry point is the Client class. It manages HTTP connections and authentication automatically based on your environment variables (GEMINI_API_KEY or GOOGLE_API_KEY for Google AI Studio, and GOOGLE_GENAI_USE_ENTERPRISE=true with standard Google Cloud Application Default Credentials).

Single prompt request with Gemini Flash

Using Kotlin's use extension ensures the client's underlying network engine and HTTP connections are released cleanly:

code_block: <ListValue: [StructValue([('code', 'import com.google.genai.kotlin.Client\r\nimport kotlinx.coroutines.runBlocking\r\n\r\nfun main() = runBlocking {\r\n Client().use { client ->\r\n val response = client.models.generateContent(\r\n model = "gemini-flash-latest",\r\n text = "Explain quantum entanglement in two sentences."\r\n )\r\n\r\n println(response.text)\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e67059bd0>)])]>

Low-latency streaming with Coroutines `Flow`

For interactive UIs and responsive CLI tools, generateContentStream returns a cold Kotlin Coroutine Flow<GenerateContentResponse>, delivering token chunks in real time:

code_block: <ListValue: [StructValue([('code', 'import com.google.genai.kotlin.Client\r\nimport kotlinx.coroutines.runBlocking\r\n\r\nfun main() = runBlocking {\r\n Client().use { client ->\r\n val responseFlow = client.models.generateContentStream(\r\n model = "gemini-flash-latest",\r\n text = "Outline the key architectural patterns for microservices on Google Cloud."\r\n )\r\n\r\n responseFlow.collect { chunk ->\r\n chunk.text?.let { print(it) }\r\n }\r\n println()\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e6705ad10>)])]>

Multi-turn conversations (chat)

Managing conversation history manually across request turns can become tedious. The SDK includes a dedicated chats service that automatically maintains context, appends turns, formats conversation history, and handles function calling:

code_block: <ListValue: [StructValue([('code', 'import com.google.genai.kotlin.Client\r\nimport com.google.genai.kotlin.types.Content\r\nimport com.google.genai.kotlin.types.GenerateContentConfig\r\nimport kotlinx.coroutines.runBlocking\r\n\r\nfun main() = runBlocking {\r\n Client().use { client ->\r\n val config = GenerateContentConfig(\r\n systemInstruction = Content.fromText("You are an expert Google Cloud Solutions Architect.")\r\n )\r\n\r\n // Create a multi-turn chat session\r\n val chat = client.chats.create(\r\n model = "gemini-flash-latest",\r\n config = config\r\n )\r\n\r\n // Turn 1\r\n val firstResponse = chat.sendMessage("We are designing an event-driven ingestion pipeline on Google Cloud.")\r\n println("Gemini: ${firstResponse.text}\\n")\r\n\r\n // Turn 2: context from the first turn is included automatically\r\n val secondResponse = chat.sendMessage("Which managed messaging service should we choose: Pub/Sub or Kafka?")\r\n println("Gemini: ${secondResponse.text}\\n")\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e6705b710>)])]>

You can also use chat.sendMessageStream(...) for streaming multi-turn chat responses.

3. Multimodal analysis grounded with Google Search

Gemini's multimodal reasoning is especially effective when combined with external verification. For instance, when analyzing technical, medical, or scientific diagrams, you can attach Google Search Grounding to cross-check factual claims against live web sources.

code_block: <ListValue: [StructValue([('code', 'import com.google.genai.kotlin.Client\r\nimport com.google.genai.kotlin.types.*\r\nimport java.io.File\r\nimport kotlinx.coroutines.runBlocking\r\n\r\nfun main() = runBlocking {\r\n Client().use { client ->\r\n val imageBytes = File("src/main/resources/medical_diagram.png").readBytes()\r\n\r\n val content = Content(\r\n parts = listOf(\r\n Part(inlineData = Blob(mimeType = "image/png", data = imageBytes)),\r\n Part(text = "Is this anatomical diagram accurate? Verify labels against authoritative medical sources.")\r\n )\r\n )\r\n\r\n // Enable Google Search as a grounding tool\r\n val config = GenerateContentConfig(\r\n tools = listOf(Tool(googleSearch = GoogleSearch()))\r\n )\r\n\r\n val response = client.models.generateContent(\r\n model = "gemini-flash-latest",\r\n content = content,\r\n config = config\r\n )\r\n\r\n println("=== Analysis ===")\r\n println(response.text)\r\n\r\n // Inspect citations and search queries\r\n val grounding = response.groundingMetadata\r\n println("\\n=== Search Queries Executed ===")\r\n grounding?.webSearchQueries?.forEach { println("- $it") }\r\n\r\n println("\\n=== Grounding Sources ===")\r\n grounding?.groundingChunks?.mapNotNull { it.web }?.forEach { source ->\r\n println("- ${source.title}: ${source.uri}")\r\n }\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e67059490>)])]>

4. Visual generation and conversational editing: The Gemini 3 image family

The SDK provides full support for Google's latest image generation models (popularly known as the Nano Banana series of models on leaderboards).

Generating and Saving an Image

Generated image bytes are delivered directly in the response parts as a Blob:

code_block: <ListValue: [StructValue([('code', 'import com.google.genai.kotlin.Client\r\nimport java.io.File\r\nimport kotlinx.coroutines.runBlocking\r\n\r\nfun main() = runBlocking {\r\n Client().use { client ->\r\n val response = client.models.generateContent(\r\n model = "gemini-3.1-flash-image", // Nano Banana 2\r\n text = "A photorealistic blueprint of an eco-friendly modern datacenter, isometric view, 4k"\r\n )\r\n\r\n val imagePart = response.parts?.firstOrNull { it.inlineData != null }\r\n imagePart?.inlineData?.data?.let { bytes ->\r\n File("datacenter_blueprint.png").writeBytes(bytes)\r\n println("Image generated and saved successfully.")\r\n }\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e6705b610>)])]>

Conversational image-to-image editing

You can pass existing images and conversational edit instructions in the same request:

code_block: <ListValue: [StructValue([('code', 'val originalImage = File("input.png").readBytes()\r\n\r\nval editPrompt = Content(\r\n parts = listOf(\r\n Part(inlineData = Blob(mimeType = "image/png", data = originalImage)),\r\n Part(text = "Change the daylight illumination to a dramatic twilight skyline with illuminated windows.")\r\n )\r\n)\r\n\r\nval editResponse = client.models.generateContent(\r\n model = "gemini-3-pro-image", // Nano Banana Pro\r\n content = editPrompt\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e67058810>)])]>

5. Real-time bidirectional interaction with Gemini Live

For low-latency voice, audio, and live multimodal interactions, the SDK supports the Gemini Live API via persistent WebSocket connections using client.live.connect(...):

code_block: <ListValue: [StructValue([('code', 'import com.google.genai.kotlin.Client\r\nimport com.google.genai.kotlin.types.AudioTranscriptionConfig\r\nimport com.google.genai.kotlin.types.LiveConnectConfig\r\nimport kotlinx.coroutines.launch\r\nimport kotlinx.coroutines.runBlocking\r\n\r\nfun main() = runBlocking {\r\n Client().use { client ->\r\n val model = if (client.enterprise) "gemini-live-2.5-flash-native-audio"\r\n else "gemini-3.1-flash-live-preview"\r\n\r\n val config = LiveConnectConfig(\r\n outputAudioTranscription = AudioTranscriptionConfig()\r\n )\r\n\r\n // Establish real-time bidirectional WebSocket session\r\n client.live.connect(model, config).use { session ->\r\n println("Connected to Gemini Live session!")\r\n\r\n // Launch collector for server messages (audio and text transcriptions)\r\n val receiveJob = launch {\r\n session.receive().collect { serverMessage ->\r\n serverMessage.serverContent?.outputTranscription?.text?.let { text ->\r\n print(text)\r\n }\r\n }\r\n }\r\n\r\n // Stream real-time text (or raw PCM audio blobs via session.sendRealtimeInput(audio = ...))\r\n session.sendRealtimeInput(text = "Hello Gemini! Give me a 5-second motivational quote.")\r\n\r\n // When finished, clean up\r\n receiveJob.cancel()\r\n session.closeSession()\r\n }\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e67059950>)])]>

6. Structured tool and function calling

When building agentic workflows or bridging LLMs with backend microservices, developers can pass structured JSON schemas via FunctionDeclaration. The model will intelligently select when to invoke the tool:

code_block: <ListValue: [StructValue([('code', 'val telemetryTool = FunctionDeclaration(\r\n name = "getDatacenterMetrics",\r\n description = "Fetch real-time CPU and thermal telemetry for a Google Cloud region",\r\n parameters = Schema(\r\n type = Type.OBJECT,\r\n properties = mapOf("region" to Schema(type = Type.STRING)),\r\n required = listOf("region")\r\n )\r\n)\r\n\r\nval response = client.models.generateContent(\r\n model = "gemini-flash-latest",\r\n text = "Check telemetry for europe-west1",\r\n config = GenerateContentConfig(\r\n tools = listOf(Tool(functionDeclarations = listOf(telemetryTool)))\r\n )\r\n)\r\n\r\nresponse.functionCalls?.firstOrNull()?.let { call ->\r\n println("Model triggered tool: ${call.name} with arguments: ${call.args}")\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e67059410>)])]>

Additionally, when using the chats service, you can take advantage of Automatic Function Calling (AFC), which means that functions declared in the chat conversation can be invoked automatically and transparently by the SDK on your behalf, as you can see in the following example:

code_block: <ListValue: [StructValue([('code', 'fun main() = runBlocking {\r\n // A mocked function\r\n val getWeather = callableFunction("get_weather", paramName = "city") { city: String ->\r\n "18 degrees and sunny in $city"\r\n }\r\n\r\n Client().use { client ->\r\n val chat = client.chats.create(\r\n model = "gemini-flash-latest",\r\n automaticFunctionCalling = AutomaticFunctionCalling(getWeather),\r\n )\r\n\r\n // SDK calls get_weather if needed in this conversation\r\n println(chat.sendMessage("What is the weather in Zurich?").text)\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e6705b350>)])]>

What's next?

With the 1.0 release of the Google Gen AI SDK for Kotlin, Kotlin developers across backend server ecosystems (Ktor, Spring Boot, Quarkus, Micronaut) now have a clean, multiplatform foundation for building generative AI applications.

To learn more and get started, check out the following resources:

GitHub Repository: Check out the source, stars, and discussions at github.com/googleapis/kotlin-genai.
Documentation and Samples: Explore the Kotlin Gen AI sample suite.
Feedback: File issues, suggest features, or submit pull requests directly on GitHub.

We look forward to seeing what you build with Kotlin and Gemini!

Simplify your resilience testing strategy with Fault Injection Testing

Wed, 26 Aug 2026 16:00:00 +0000

When databases fail and network paths falter, you still need your mission-critical cloud services to stay online. Yet guaranteeing high availability has become increasingly difficult because of the complexity of modern distributed systems.

To help you maintain availability and reliability during adverse events, we’re announcing Fault Injection Testing in preview. Fault Injection Testing is designed to help developers and architects automate failure testing to ensure predictable behavior during disruptions.

By deliberately introducing faults into your environment, you can verify your safety mechanisms before an actual outage impacts your customers.

Why native resilience testing matters

Unlike in self-hosted data centers, cloud applications offer less direct access to underlying infrastructure to facilitate failover testing.

Without native tools to prove your application can survive a failure, you risk a critical gap in your reliability strategy that exposes you to several risks:

Damaged trust and reputation: Frequent failures or poor performance lead to customer dissatisfaction and long-term damage to your brand's image.
Compliance and regulatory penalties: For many industries, particularly financial institutions, failing to prove disaster recovery capabilities can lead to non-compliance, audits, and fines.
Migration delays: Large-scale migrations often stop when teams cannot verify that critical applications will remain stable during a zone failure.

How Fault Injection Testing works

Fault Injection Testing allows you to run experiments by creating experiment templates. These templates act as blueprints, defining the specific fault to be injected and the resources that will be targeted for the experiment.

In this public preview, you can test two primary failure scenarios:

Failover Cloud SQL: This fault triggers a failover of a high availability Cloud SQL instance from the primary zone to a standby zone.
Degrade application traffic: This allows you to selectively add latency and HTTP error codes through an Application Load Balancer.

Before any fault is injected, Fault Injection Testing performs an automated dry run. This read-only simulation checks your permissions and provides an up-to-date list of every resource that will be affected.

Once you verify the scope, you can manually start the injection. The duration you defined in the template will run its course, and the faults will be reverted at the expiration of the timer.

During the experiment, you can verify that your application is behaving as you planned. If things do not go as planned, you can use the stop and revert capability to immediately halt the experiment and begin restoring resources to their normal state.

During preview, we recommend as a best practice to use Fault Injection Testing (FIT) in a non-production environment. Preview is an opportunity to get early access to learn how the service fits and complements your existing testing practices, and to provide us with your feedback to improve the product as well!

Built for the enterprise

Partners like KeyBank and Servier are already using Fault Injection Testing to validate their deployments. By using native fault injection, these organizations can approximate demanding failure scenarios — such as zonal outages — to help ensure their services remain stable.

Get started with Fault Injection Testing

Fault Injection Testing is available through the Google Cloud console, the gcloud CLI, and REST APIs.

Request preview access: Talk to your Google Cloud Account Team to add your project to the preview.
Enable the API: Search for "Fault Testing API" in your Google Cloud console and select enable.
Assign roles: Ensure your team has the roles/faulttesting.operator role to configure and run experiments.
Run your first dry run: Create a template for a Cloud SQL or load balancer resource in a non-production environment and execute a dry run to see the potential impact.

For more details on implementation, talk to your account team, or view the User Guide for Fault Injection Testing.

How Uber improves network reliability while unblocking cloud migration

Wed, 26 Aug 2026 16:00:00 +0000

Uber has a lot in common with the cities it serves. Both are always changing and growing, both must carefully manage the resulting traffic to prevent congestion and sprawl.

Uber has continuously evolved its technical strategies to manage its expanding network, and this careful planning and constant evolution helps ensure that application traffic across its entire platform runs smoothly. Ultimately, maintaining a reliable, high-scale platform that operates seamlessly at any given time is key to preserving user trust.

One important solution in this effort has been application awareness on Cloud Interconnect. An industry-first tool for application prioritization across hybrid networks, application awareness on Cloud Interconnect has helped Uber prioritize critical traffic to ensure business continuity during potential network congestion events.

Uber acted as an early design partner for application awareness on Cloud Interconnect, helping ensure that this capability met the demands of Uber’s global-scale operations. It not only improved Uber’s daily operations, it also gave Uber the confidence to move forward with a Google Cloud migration, with confidence that there would be less risk of service interruptions during switchovers.

In this post, we’ll explain the features Uber most sought and why, the inner workings of application awareness on Cloud Interconnect, and how it can help other organizations as well.

Prioritizing critical traffic

When migrating distributed, hybrid, or multicloud applications at a global scale, network reliability becomes a primary concern. Even the most worthwhile migrations may not seem worth it if such migrations interrupt ongoing service. For organizations like Uber, moving vast amounts of data to support large data analytics workload — including emerging AI use cases — can saturate network links, resulting in increased reliability risk for their critical application traffic.

With standard cloud interconnect approaches, enterprises typically apply simple bandwidth overprovisioning to meet extreme infrastructure needs. But with today's hybrid cloud demands, and given the size of an organization like Uber, overprovisioning network capacity for peak usage is often too costly and unreliable.

The shortcomings of overprovisioning only become magnified with the integration of cutting-edge AI innovations. Uber needs systems in place that can take on massive data transfers without congesting its network and protecting the performance of business-critical applications.

With the benefit of application awareness on Cloud Interconnect, including the four major features of application awareness — traffic handling, congestion response, latency management, and cost efficiency — Uber was able to achieve the networking optimization its modern tech stack requires.

Starting with a private preview, Uber deployed this feature across its infrastructure, beginning with Google Cloud Interconnect deployments in Phoenix, Arizona, and Ashburn, Virginia. Application awareness on Cloud Interconnect allows Uber to classify and prioritize end-user application traffic over less time-sensitive data using DSCP marking and configured queuing profiles.

In the following chart, we look at the four key features of application awareness on Cloud Interconnect, how they differ from legacy approaches, and how they help provide better operational continuity for organizations like Uber.

Feature	Standard interconnect solutions	Application awareness on Cloud Interconnect
Traffic handling	All traffic treated equally (first-in, first-out)	Traffic classified into six distinct traffic classes
Congestion response	High-priority application traffic may be dropped during bursts	Business-critical traffic is protected via strict priority or bandwidth sharing policies
Latency management	Unpredictable latency for high priority applications	Predictable and consistent low-latency for time-sensitive workloads
Cost efficiency	Requires expensive overprovisioning to absorb peaks	Efficient bandwidth utilization and lower TCO

Uber's key takeaways

For Uber, the business value of being able to prioritize business-critical traffic on its networks by deploying application awareness on Cloud Interconnect was immediate. And in doing so, Uber has also created a blueprint that other enterprises with similar hybrid cloud challenges can replicate. The core elements of that blueprint include:

Ensuring business continuity: Uber can decide in real time which application traffic to prioritize during major, high-traffic events. This means that mission critical applications stay up and running during even extreme events (both planned and unplanned). Uber leadership has called application awareness on Cloud Interconnect important for its global operations.
Efficient bandwidth utilization: Instead of blindly overprovisioning bandwidth to prevent congestion, application awareness allows Uber to better utilize their existing Cloud Interconnect capacity aligned with their expected network bandwidth needs. The result is lower total cost of ownership for network infrastructure.
Unblocked workload migration: By protecting critical applications from network congestion, Uber was able to migrate significant workloads to Google Cloud and, in the process, dramatically reduce operational overhead.

"Application awareness on Cloud Interconnect was the key that unlocked our ability to migrate more strategic workloads to Google Cloud and is critical for maintaining service reliability during peak global demand. By allowing us to intelligently prioritize traffic, it helps us ensure that we can protect our higher priority services and make our infrastructure more efficient, lowering our total cost of ownership. This wasn't just a feature deployment; it was a deep engineering partnership that delivered a solution critical to our business." – Harry Liu, Director of Engineering, Uber

Securing network reliability for AI and beyond

As more enterprises integrate cloud-based AI models, distributed applications, and data analytics, it's becoming a business imperative to be ready to handle the massive data transfers that follow. But in doing so, they also have to ensure they never compromise the reliability of their critical applications.

With application awareness on Cloud Interconnect, Uber demonstrated that moving beyond simple bandwidth overprovisioning to protect business-critical traffic was an essential step to building the stability required to embrace modern hybrid and multicloud strategies.

You can read our blog about the potential of Cloud Interconnect across industries to learn more about what the service can bring to your organization, and if you’re ready to explore more, our team of networking and industry experts are ready to help.

Your chance to start building AI agents from the absolute basics

Wed, 26 Aug 2026 09:07:00 +0000

Have you been hearing a lot about "AI agents" lately but aren't sure how to actually start building them? You don't need a background in machine learning or years of software experience to get started. The best way to learn is by doing, which is why we built Agent Valley.

Agent Valley is a free, 5-week live learning series designed to take you from scratch to building your very own hands-on agent systems. And instead of staring at boring terminal lines, you’ll be building and playing inside a tiny, low-poly virtual world!

Meet your instructor

You’ll be learning directly from Annie Wang, one of our top Google DevRel Engineers. She designed this course from the ground up to be fully hands-on, interactive, and beginner-friendly. If you want to learn how AI systems are built by the people actually designing them at Google, this is your chance.

How we'll learn together

You’ll learn by building in a split-screen workspace on your laptop. On Day 1, you'll describe and summon a custom low-poly companion that serves as your play character and save file. As you guide your companion through the valley's five districts, a live Runtime Inspector sits right beside the game, showing you exactly what the AI is thinking, deciding, and costing in real-time. Setup is completely zero-stress. Google will provide the environment for running these exercises, so you can dive straight into building.

Agent 101 Live with 5 modular sessions (Jump in anytime!)

Week 1: The Summoning Grove (CONTROL) · Get started by summoning your companion and learning how to keep its memory and traits consistent across a conversation.
Week 2: The Buildyard (DECOMPOSE) · Learn how to break a big project down so multiple AI assistants can work together in parallel without stepping on each other's toes.
Week 3: Market Street (COORDINATE) · Open up a virtual shop! You'll learn how to write reliable code so transactions and returns go smoothly without crashing.
Week 4: The Archive (REMEMBER) · Give your companion a memory. Learn how to help your agent remember past details without getting confused or making things up.
Week 5: The Night Market (LIVE) · The grand finale. Learn how to make your agent react live to events in the world (like fireworks or stage lights) while keeping the system fast and affordable.

Join the livestream

5 Tue starting Sep 1 · 10:00 AM (Pacific Time)
Anyone new to AI agents who wants to learn by coding and playing.
RSVP Here: goo.gle/agent101

Developers & Practitioners

GKE CPU startup boost: Accelerate app starts without over-provisioning

Why modern applications need extra CPU at boot time

How CPU startup boost can help

Under the hood: Kubernetes In-place Pod Resize

How CPU startup boost works (pod lifecycle overview)

Prerequisites and availability

Getting started: Configuring CPU startup boost

Example 1: Pod-level boost with fixed steady-state (updateMode: "Off")

Example 2: Combining startup boost with continuous VPA auto-scaling

Example 3: Granular container-level boost

Verifying startup boost in your cluster

Best practices for production workloads

Get started today

Democratizing Managed Lustre with lower cost and frictionless development

Lower Cost: More Lustre for Less with the Dynamic Tier

Recommended workloads for Dynamic Tier

Frictionless development: Lustre as a one-stop shop for developer’s workloads

Unified Foundation & Interactive Performance

High-Concurrency Broadcast & Cluster Startup

Run One-Stop Shop Workflows for Yourself

Low latency for interactive access

Accelerated setup

How to run Linux untar

How to run Python gitclone

High scale distribution

Aggregate throughput for distributing one large file to many nodes

Parallel loading of libraries across many processes

Looking ahead and next steps

Next steps

Introducing the Server Side Cloud Swift SDK

The rise of server-side Swift and cloud-native concurrency

Where to use the Swift SDK

Getting started with your IDE and packages

Networking, transport, and authentication

The autogenerated client ecosystem

Where to go next

Empower your agents with the Google Cloud CLI remote MCP server

Why CLI matters for AI agents

The benefits of putting CLI behind remote MCP

Enterprise-grade security and governance

Connecting to the Google Cloud CLI Remote MCP Server

Bringing infrastructure management to agents

Extending BigQuery operations with the run_bq_command tool

Pricing and availability

Data Agent Kit is now GA: Bring Google Data Cloud to any coding agent

What is Data Agent Kit?

How it works

What you can build

Analytics and graph: BigQuery and BigQuery Graph (New in GA)

Operational and real-time databases: Spanner, AlloyDB, Cloud SQL, and Bigtable (New in GA)

Lakehouse and Spark: Managed Service for Apache Spark and Lakehouse for Apache Iceberg (New in GA)

Pipelines and orchestration: dbt, Dataform, and Managed Service for Apache Airflow

Improving the developer experience

Ready for the enterprise

Install and get started

Option 1: Install the IDE Extension

Option 2: Install the CLI Plugin

Next steps & hands-on resources

Graph Workflows in ADK: Everything You Need to Know

Start with a single agent

Let independent steps run together

Graph Engineering with ADK

Route each request to the right workflow

Identify the intent with an agent router

Apply the refund policy with a deterministic router

Apply the same step to a batch of cases

Let results shape the next step

Choose who decides what runs next

Get started

Best practices guide for customizing Gemini models via Reinforcement Learning (RL)

What is RLFT?

When to use RLFT

Use cases for RLFT

Where to start?

Get started today

Agent Factory recap: Agent harnesses, shifting left, and autonomous coding

The Agent Harness - What is it?

Ryan Lopopolo on agent harnesses and autonomous coding

Coining the harness and writing zero production code

Example 1: Pod-level boost with fixed steady-state (`updateMode: "Off"`)

3. Session isolation and trajectory persistence (`save_dir` & `conversation_id`)