Developers & Practitioners

Claude Opus 5.5 and Claude Sonnet 5.5 are now available in Google Antigravity

Tue, 20 Oct 2026 16:00:00 +0000

Whether you are iterating on a quick bug fix or planning a complex migration, different tasks call for different models.

In keeping with Google Cloud's commitment to an open ecosystem and to give developers the right model for every job, Claude Opus 5.5 and Claude Sonnet 5.5 are now available in Google Antigravity for Google Cloud customers. Claude Opus 5.5 is built for complex work while Claude Sonnet 5.5 is a faster, lower-cost complement.

Once enabled by an administrator, developers can select Claude Opus 5.5 and Sonnet 5.5 directly from the model picker. Administrators can also select Google’s Gemini models, including Gemini 3.8 Flash and Gemini 4 Argon. The models apply across Antigravity 2.0, Antigravity CLI and our recently released Antigravity IDE extensions for VS Code, JetBrains IntelliJ, and Visual Studio. This lets you choose the right model for the job, under a unified enterprise billing and security boundary.

In this post, we’ll share two engineering patterns for building in Antigravity using these models. To view the entire list of models supported in Antigravity across subscriptions, see our documentation here.

Multi-model engineering patterns in Antigravity

Having multiple model families available inside the same agent harness unlocks practical multi-model workflows that improve code quality while controlling latency and cost.

Pattern 1: Cross-model verification ("LLM-as-a-Judge") in loop and graph workflows

When building agentic loops or multi-step graph workflows, using the same model to both write code and review its own output can reinforce model-specific blind spots. Pairing two distinct model families, one as the implementer and another as the validator (judge), provides independent verification before moving to the next stage of a workflow.

In Antigravity, you can run this pattern either interactively or via automated subagents:

Plan and implement with Model A: Start in planning mode using Claude Opus 5.5 or Gemini 4 Argon to design and implement a multi-file change.
Validate with Model B: Before committing or advancing the loop, switch models, or invoke a one-shot /model check in the CLI, to have a different model family critique the implementation against your test suite and requirements.

Pattern 2: Pairing reasoning models with Flash subagents

Complex engineering tasks often combine a few high-reasoning decisions with dozens of high-volume, mechanical lookups (searching files, grepping logs, checking lint output, or classifying test failures). Running every lookup on a frontier model can increase both latency and spend.

Antigravity's Subagent customization system (.agents/agents/*.md) lets you assign specific model tiers (inherit, flash, or pro) to specialized subagents. When your primary session is running a more powerful model.

Subagents configured with model: inherit use your active session model for deep reasoning subtasks.

Subagents configured with model: flash offload high-speed search, classification, and triage work to Gemini Flash models.

For example, you can define a fast log-triage subagent in .agents/agents/test-triager.md:

code_block: <ListValue: [StructValue([('code', '---\r\nname: test-triager\r\ndescription: Runs unit tests, classifies failures, and returns a concise summary of failing assertions.\r\nmodel: flash\r\ntools:\r\n - read\r\n - edit\r\n---\r\n\r\nRun the requested test targets, classify any failures by root cause (compilation error, assertion failure, or timeout), and return a structured summary to the parent agent.'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe68bc2e510>)])]>

And a cross-model code reviewer subagent in .agents/agents/gemini-judge.md:

code_block: <ListValue: [StructValue([('code', "---\r\nname:cross-model-judge\r\ndescription: Independent LLM-as-a-judge verifier that audits diffs and edge cases using Gemini Pro.\r\nmodel: pro\r\ntools:\r\n - read\r\n---\r\n\r\nReview the current workspace diff against the user's requirements. Check for regressions, missing edge-case tests, and security issues. Output a pass/fail verdict with specific line references."), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe68bb2efd0>)])]>

With this setup, you can delegate rapid test-and-classify loops to test-triager and trigger cross-model-judge to validate the final diff, combining the strengths of each model in a single workflow.

Getting started

Claude Opus 5.5 and Claude Sonnet 5.5 are available now for Google Cloud customers using Antigravity:

Administrators (Gemini for business): Third-party partner models do not draw from the pooled Gemini Enterprise Standard and Plus edition monthly token quota. Instead, usage is billed on a pay-as-you-go basis through your existing Google Cloud Billing account, so overages must be enabled. Visit “Gemini Enterprise” > “Settings” > “AI developer tools” in the Google Cloud Console to configure overage billing, spend limits and authorize partner models for your organization. See Configure AI developer tools settings for step-by-step instructions.
Administrators (Gemini platform): You can also enable partner models if you are using Antigravity through the API in Gemini platform. Enable Claude models in Model Garden and complete the standard Anthropic terms.
Developers: Update to the latest version of Antigravity 2.0, the Antigravity CLI, or the Antigravity IDE extensions, and open the model picker (or run /model) to select the right model for your task. To learn more about enterprise governance and multi-agent configuration, visit the Google Antigravity Enterprise documentation. You can switch the active model between turns in the same conversation thread while preserving workspace state, open artifacts, and terminal context.

Note: Lightweight first-party Gemini models continue to run in the background to handle high-frequency client-side coordination, such as context management and summaries. They consume first-party tokens and can incur charges.

Developer’s guide to implementing MCP apps in Gemini

Mon, 19 Oct 2026 16:00:00 +0000

Enterprise assistants still respond mostly in text. However, that can break down for visual, multi-step work, where every adjustment costs another slow chat turn.

MCP Apps (SEP-1865) solves this. This open Model Context Protocol (MCP) extension pairs a tool with a live UI. In Gemini Enterprise, that UI renders directly in the chat flow inside a sandboxed iframe, and can expand to fullscreen or a side panel. Edits run locally while the workspace and assistant stay synchronized.

You also avoid rebuilding core foundations: Gemini provides workforce sign-in, governed data access, and the Default Assistant, while you host your MCP server on a private Cloud Run service.

Why we need MCP Apps

Enterprise assistants have evolved through three generations:

Generation 1 (Text chatbots): Returns static markdown with no interactive controls.
Generation 2 (Standard MCP tools): Invokes live APIs, but outputs static text. Every change requires another model turn.
Generation 3 (Interactive MCP Apps): Pairs each tool with a live workspace. Adjustments happen in the widget without a model turn, while the assistant stays in sync.

Building MCP Apps in Gemini delivers six benefits:

Responsive user experience: Single-click actions execute directly in the panel without round-tripping through the chat.
Purpose-built workflows: Deliver visual redaction views for legal review, 3D viewers for CAD, or sensitivity models for finance.
Enterprise security: Workforce OAuth sign-in, private Cloud Run ingress, and a sandboxed iframe protect every layer.
Human-in-the-loop governance: Experts inspect and approve high-stakes actions before execution.
Open standard: Portable across compatible hosts, with automatic text fallback.
Broad connectivity: Connect to BigQuery, Agent Platform, Cloud Storage, or internal ERP and CRM systems.

High-level architecture

Three decoupled tiers work together:

Client browser: Shows the chat alongside the MCP App widget in the side panel.
Gemini Enterprise: The default assistant calls tools, the AppBridge proxy relays widget messages, and the connector manages OAuth.
Google Cloud (your project): You deploy and host the MCP server on Cloud Run, which exposes one JSON-RPC endpoint (/mcp), runs background jobs, and serves the single-file HTML widget.

Each tier talks only to its neighbor, letting Gemini Enterprise enforce security and routing.

Authentication with two independent layers

Each user signs in through OAuth 2.0. On every call, Gemini Enterprise sends its service agent token alongside the user's OAuth token:

Layer 1 (Cloud Run IAM): Enforced at Google's edge, this answers "which service may call?" Public access is disabled; unauthorized callers receive HTTP 403.
Layer 2 (In-App OAuth): Enforced inside the application, this answers "which user is calling?" Failed verification returns HTTP 401, prompting the user to sign in again.

Cloud Run IAM cannot distinguish end users, so both layers are required. Outbound calls use a least-privilege service account.

Build, deploy on Google Cloud, and register in Gemini Enterprise

First, build your custom MCP server (exposing POST /mcp and a bundled ui:// HTML widget) using the MCP Apps SDK, or use Google Antigravity to scaffold and deploy your MCP App with this starter prompt:

Build an MCP App for Gemini Enterprise using the create-mcp-app skill (https://github.com/modelcontextprotocol/ext-apps/tree/main/plugins/mcp-apps/skills/create-mcp-app).
Workflow: [describe your input data, side-panel UI controls, and approval or export actions].
Generate a single-file HTML widget, a Streamable HTTP backend (POST /mcp) with <2s async tool responses and get_app_state polling, and Cloud Run deployment commands (--no-allow-unauthenticated, --no-cpu-throttling, --min-instances=1).

Once your MCP server code is ready, deploy it to Cloud Run and connect it to Gemini Enterprise:

Step 1: Enable APIs for Cloud Run, Discovery Engine, Agent Platform, and Organization Policy.
Step 2: Deploy your MCP server container to Cloud Run from source (gcloud run deploy <service> --source .) with --no-allow-unauthenticated, --no-cpu-throttling so background work does not freeze between polls, and --min-instances=1.
Step 3: Grant the invoker role (roles/run.invoker) to your Discovery Engine service agent (Layer 1).
Step 4: Create an OAuth 2.0 Web Client with redirect URI https://vertexaisearch.cloud.google.com/oauth-redirect(Layer 2).
Step 5: Override the discoveryengine.managed.disableCustomMcpServerConnector constraint and complete the Before you begin prerequisites (roles/discoveryengine.editor for the admin, allowedEgressFqdns for your server and OAuth hosts, and custom_mcpin allowedDataSources if enforced). Then, in the Google Cloud Console under Gemini Enterprise > Data stores > Create data store, select Custom MCP Server and register the connector:

Configuration field	Value
MCP Server URL	`https://<your-cloud-run-service>.run.app/mcp`
Authorization URL	`https://accounts.google.com/o/oauth2/v2/auth`
Token URL	`https://oauth2.googleapis.com/token`
Client ID and Secret	Your Web OAuth 2.0 Client ID and Secret
Scopes	`openid email profile`
Enable PKCE Support	Checked (recommended)
Use HTTP Basic Auth	Unchecked (avoids `invalid_client` errors)

Step 6: Enable actions on your data store. Gemini Enterprise imports your MCP server's tools as actions and disables them by default. In the Cloud Console, open Gemini Enterprise > Data stores > [Your MCP Data Store] > Actions, click Reload custom actions (which calls tools/list on your server), select your model-visible tools, and click Enable actions.

Step 7: Connect the data store to your app. Open Gemini Enterprise > Apps > [Your App] > Connected data stores > Add existing data stores, and select your MCP data store. Connecting it as a data store gives the built-in Default Assistant access to your MCP tools and UI widget directly in main chat ( do not use "Add Agent," which is for external sub-agents rather than MCP data stores).

Build your own

An MCP App pairs backend tools with a live workspace in Gemini Enterprise. You do not build a chat frontend or identity pipeline; you deploy one Cloud Run container and attach it as a Data Store, making Gemini Enterprise where enterprise work gets done.

Try Gemini here
Co-build with Google Antigravity here
Build on Google Cloud: MCP Apps Specification (SEP-1865), Cloud Run, and Google Cloud AI.

Four evaluation engineering lessons learned from building agent plugins

Fri, 16 Oct 2026 16:00:00 +0000

The real challenge of building an agentic system isn't the construction, it's the measurement.

Evaluation engineering is a critical foundation for building reliable AI systems. Setting up systematic, repeatable tests to measure your agent’s true behavior can be the difference between shipping a tool you can trust and one that silently breaks, fails to call APIs, or makes mistakes.

I’ve run evaluation engineering initiatives for years at Google and developing the newly released Google Cloud Developer plugin served as a live testbed to stress-test these key principles across real-world developer workflows.

Here are four eval lessons we learned from building the new plugin that you can apply to your own builds.

1. Grade the execution steps instead of the summary

When we first evaluated agents, we set up an LLM-as-judge to inspect the agent’s final text reply and scored it against a checklist. However, we quickly realized it’s fundamentally flawed because the grader only sees what the agent wrote, not what it actually did. When we ran our initial 11-prompt evaluation suite for the Google Cloud Developer plugin, the automated grader marked it as failing three prompts. But when we opened the raw tool-call logs and read the transcripts, the plugin had actually passed all three. On a Cloud Run cleanup prompt, for instance, our rubric required checking command syntax before running anything. The agent ran gcloud help run services list as its very first step. It did the work, but its final message was focused on warning the user about permanent data loss rather than reciting that it had checked the help docs.

The grader penalized the agent for doing the right thing simply because it didn’t brag about it in the summary. If your rubric line says "did it run X?", a LLM judge is guessing unless you evaluate the actual tool-call trajectory.

Takeaway: Don't grade an agent on its closing pitch. Score the raw execution trajectory to verify what commands were actually executed, validated, and scoped.

2. Design evals for asynchronous operations

When you start reading tool-call trajectories, how you extract that telemetry matters just as much as what you're reading. Extracting agent telemetry into a flattened, sequential list of events obscures concurrent tool dispatch, which is one of the critical causes of behavioral failure. Modern agent harnesses often batch multiple tool invocations into a single assistant turn using an array of calls.

In our evaluations, when an agent was asked to inspect an environment and look up documentation, the harness dispatched the skill-loading call and two documentation searches simultaneously within a single tool_calls payload. Because activating a skill is a multi-stage process, where the initial call returns a short launch stub and the subsequent turn injects the full instruction body, the instructions meant to govern the search arrived in the prompt context after the system had already executed the searches.

The system never gave the skill a dedicated turn to let its guidance influence the agent's actions. If telemetry pipelines strip out turn structures and merge calls into a flat timeline, engineers will waste days wondering why an agent disregarded unambiguous instructions that a developer embedded in a skill.

Takeaway: Preserve turn structure in telemetry extractors so you can identify when client-side parallel dispatch issues tool calls before injected guidance enters the context window.

3. Test live runs and "dry runs" side-by-side

You need to inspect real API calls to catch harness quirks, like parallel dispatch. But if you only test your agent in one environment condition, you're flying half-blind. From day one, you need to run two conditions side-by-side: a live run and a plan-only run. In the live condition, the agent has real, sandboxed credentials and actually executes commands. This is how you see if syntax works, parameters serialize cleanly, and the agent exercises restraint before doing something destructive.

Live runs frequently hit an unexpected permission error on step two and abort before you ever reach the behavior you’re trying to test. That's why you need the plan-only condition, where you strip credentials and tell the agent to explain what it would do without running anything. This strips out permission roadblocks and shows you the model’s pure reasoning and routing intent. Roughly half our most important findings came from the delta between the two.

For example, on one of our onboarding prompts, a live run was blocked immediately on a project list call. The plan-only run sailed right through and produced an 8,000-character plan that completely ignored organization policies, admin approvals, and governance. The live run masked a major cognitive flaw that the plan-only run exposed in seconds.

Takeaway: Run live and plan-only evaluations side by side. Live runs test actual execution and safety restraint; plan-only runs expose the model's raw reasoning and routing without getting tripped up by permission errors.

4. Use “golden sets” as durable baselines

Frontier model architectures change every few months, agent harnesses improve, and plugin manifest standards will inevitably evolve. When building the Google Cloud Developer plugin, we treated evaluations as continuous integration testing for these non-deterministic environments. That means we stress-tested the composite stack, including the harness, the live model, and the tools running together on realistic tasks.

Instead of guessing from the final text output, this approach captures the whole trajectory: what the agent thought, what commands it actually ran, and whether it solved the problem without breaking anything. Then you score that run against golden sets curated by real human experts who know what the ideal path looks like.

Takeaway: Build golden sets based on what the entire system should actually do in the wild, including your harness, model, and plugins. Models change, but those baseline runs are your safety net.

Reminder: Use evaluation suites over “one-shot” demos

A common mistake I see among engineers is what I like to call the one-shot demo. This is when someone feeds a prompt into a LLM, watches it build a complete 3D game or solves a Rubik’s cube in a single pass and calls it an eval. But in reality, this is just a vanity metric.

To do quality work with AI, I truly believe that you need evaluation suites. Without evaluations, teams are like literally running into a dark cave blind. Engineering an enterprise-grade agent means testing the system's ability to handle multi-step interactions, manage permissions, and avoid destructive outcomes like accidentally deleting a project because you asked it to "find opportunities to save costs."

Where to go from here

Building an agent plugin or MCP server is the beginning. Shipping something you can actually trust with real cloud infrastructure requires significantly more work in evaluations to ensure it’s doing what you expect it to do. At the end of the day, the only thing that outlasts model churn is a stress-tested eval suite. Check out the Google Cloud Developer plugin documentation or explore this codelab to get started with the plugin in Antigravity.

One Agent Development Kit, five languages

Tue, 13 Oct 2026 12:22:00 +0000

When we talk about agents, that doesn't necessarily mean we're talking about Python. In today's modern ecosystem, an agent might run as a high-concurrency Go microservice on Cloud Run, or live inside a browser tab calling Chrome's on-device model. Yet many agent frameworks assume a Python-only stack. That assumption forces teams to either spin up a Python service just for their agent workloads, or to build agent orchestration logic from scratch in their own language.

Google's Agent Development Kit (ADK) removes that trade-off with coverage for Python, Go, TypeScript, Java, and Kotlin. Each version of the ADK is built from the ground up for its own runtime and type system, while sharing a unified architecture with interoperability through open protocols.

A shared foundation

Picking a language doesn't lock in your architecture: all five support the same open protocols. ADK includes a Model Context Protocol client in every language, so a custom tool exposed as an MCP server can be called from any of them immediately.

For multi-agent systems, ADK supports the Agent2Agent (A2A) protocol, allowing an agent to communicate with a remote agent over HTTP. Through A2A, agents written in different languages or frameworks are interoperable.

Starting agent-first

To get started, run the one-time setup command to check your Google Cloud authentication, install the CLI, and then set up first-party ADK skills in coding agents like Antigravity, Claude Code, or Codex:

code_block: <ListValue: [StructValue([('code', 'uvx google-agents-cli setup'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690641690>)])]>

Then, run the scaffold command to generate a working project with A2A endpoints and deployment config for Cloud Run, Gemini Enterprise Agent Runtime, or Google Kubernetes Engine (GKE):

code_block: <ListValue: [StructValue([('code', '# Scaffold an ADK project (use --agent adk for Python or --agent adk_go for Go)\r\nagents-cli scaffold create my-go-agent \\\r\n --agent adk_go \\\r\n --deployment-target cloud_run \\\r\n --region us-central1'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe6906401d0>)])]>

Both the ADK for Python and ADK for Kotlin repositories also ship agent builder skills in their source trees, and the adk.dev/llms.txt resource gives any coding agent on-demand API documentation across all five languages.

Coding-agent skills scaffold your project while MCP and A2A connect all five language runtimes.

Hands-on view of each ADK

The clearest way to feel the difference between the five languages is to define the same weather tool in each one and see how its design fits the runtime.

Python

In ADK for Python, you don't need a decorator or a schema builder class. You pass a plain Python function into the tools parameter, and ADK inspects its signature, type hints, and docstring at startup to automatically construct the tool declaration and Pydantic parameter schema:

code_block: <ListValue: [StructValue([('code', 'from google.adk.agents.llm_agent import Agent\r\n\r\n\r\ndef get_weather(city: str) -> dict:\r\n """Retrieves the current weather report for a specified city.\r\n\r\n Args:\r\n city (str): The name of the city for which to retrieve the weather report.\r\n """\r\n return {"status": "success", "report": f"Sunny and 72F in {city}."}\r\n\r\n\r\nroot_agent = Agent(\r\n name="weather_agent",\r\n model="gemini-3.8-flash",\r\n instruction="Answer questions about the weather in a city.",\r\n tools=[get_weather],\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7fe6906411d0>)])]>

Beyond fast prototyping, Python is ADK's reference implementation, shipping the broadest set of Google Cloud integrations (BigQuery, Spanner, Bigtable) and model adapters (Gemini, Anthropic, OpenAI, LiteLLM). Paired with the adk eval or agents-cli eval run commands, its evaluation tooling scores agent trajectories against golden datasets in continuous integration (CI) to test the whole system end to end, even when your production service runs in another language.

Go

While Python leads in experimentation and evaluation, ADK for Go hits the sweet spot for production backend engineering.

A Go tool function takes the agent.Context configuration and a strongly typed input struct. The functiontool.New function uses Go generics to infer both the input and output schemas from your struct tags:

code_block: <ListValue: [StructValue([('code', 'type GetWeatherArgs struct {\r\n\tCity string `json:"city" jsonschema:"The city to look up weather for."`\r\n}\r\n\r\ntype GetWeatherResult struct {\r\n\tReport string `json:"report"`\r\n}\r\n\r\nfunc getWeather(_ agent.Context, args GetWeatherArgs) (GetWeatherResult, error) {\r\n\treturn GetWeatherResult{Report: "Sunny and 72F in " + args.City}, nil\r\n}\r\n\r\nweatherTool, err := functiontool.New(functiontool.Config{\r\n\tName: "get_weather",\r\n\tDescription: "Retrieves the current weather report for a city.",\r\n}, getWeather)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690640bd0>)])]>

Because the functiontool.New function is generic over your handler's signature, the Go compiler verifies that your function matches the expected contract before your binary ever runs.

Go was also the first language after Python to ship the ADK 2.0 Graph Workflow runtime. When you need deterministic control flow alongside LLM reasoning, the workflow.New function wires agents and plain Go functions into a directed graph with conditional routing and parallel steps:

code_block: <ListValue: [StructValue([('code', 'ag, err := workflowagent.New("order_pipeline", []workflow.Edge{\r\n\t{From: workflow.Start, To: triageNode},\r\n\t{From: triageNode, To: fulfillNode, Route: workflow.StringRoute("fulfill")},\r\n\t{From: fulfillNode, To: notifyNode},\r\n})'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690643150>)])]>

Where Go really stands out is in production on Cloud Run and GKE. A compiled Go agent packages onto a roughly 2 MiB Google distroless static base image, with no language VM or heavy dependency tree. A static Go binary binds to its port as soon as the container starts, skipping the module imports of dynamic languages and the JAR unpacking and JIT warmup of JVM services.

After it's running, Go's concurrency model fits agent workloads naturally. Each goroutine starts with a 2 KB stack and minimal scheduling overhead, so a single instance can multiplex thousands of concurrent streaming requests and tool calls. ADK for Go also includes built-in A2A server handlers and OpenTelemetry tracing out of the box.

TypeScript

TypeScript erases type annotations when compiling to JavaScript, so runtime reflection isn't an option. Instead, ADK for TypeScript pairs with Zod. You pass a z.object(...) schema to the FunctionTool function, and TypeScript statically infers the parameter types of your execute callback from that single declaration:

code_block: <ListValue: [StructValue([('code', 'import { FunctionTool, LlmAgent } from "@google/adk";\r\nimport { z } from "zod";\r\n\r\nconst getWeatherTool = new FunctionTool({\r\n name: "get_weather",\r\n description: "Retrieves the current weather report for a city.",\r\n parameters: z.object({\r\n city: z.string().describe("The name of the city."),\r\n }),\r\n execute: async ({ city }) => {\r\n return { status: "success", report: `Sunny and 72F in ${city}.` };\r\n },\r\n});\r\n\r\nexport const rootAgent = new LlmAgent({\r\n name: "weather_agent",\r\n model: "gemini-3.8-flash",\r\n instruction: "Answer questions about the weather in a city.",\r\n tools: [getWeatherTool],\r\n});'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690641350>)])]>

Your runtime input validation, the JSON Schema sent to the model, and your IDE autocomplete inside the execute callback all flow from one source of truth.

TypeScript also runs on the ADK 2.0 Graph Workflow engine and ships a standalone browser bundle. Beyond Node.js services, you can run an ADK agent directly in a browser tab using WebMCP for page-level tools and Chrome's on-device Prompt API.

Java

ADK for Java snaps right in for teams who have standardized on Maven, Gradle, and Spring Boot. You add @Schema annotations to method parameters and assemble agents by using a fluent LlmAgent.builder() API:

code_block: <ListValue: [StructValue([('code', 'public class WeatherTools {\r\n public static Map<String, String> getWeather(\r\n @Schema(name = "city", description = "The name of the city")\r\n String city) {\r\n return Map.of("status", "success", "report", "Sunny and 72F in " + city);\r\n }\r\n}\r\n\r\nLlmAgent agent = LlmAgent.builder()\r\n .name("weather_agent")\r\n .model("gemini-3.8-flash")\r\n .instruction("Answer questions about the weather in a city.")\r\n .tools(FunctionTool.create(WeatherTools.class, "getWeather"))\r\n .build();'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690641f90>)])]>

At runtime, the runner streams agent events reactively through RxJava. For local testing and existing JVM apps, ADK for Java ships a Maven Dev UI plugin alongside first-party bridges for Spring AI and LangChain4j.

Kotlin

ADK for Kotlin takes a strong stance on build-time safety. Its Kotlin Symbol Processing (KSP) compiler plugin reads @Tool and @Param annotations at build time and emits the tool schema directly into bytecode:

code_block: <ListValue: [StructValue([('code', 'class WeatherTools {\r\n @Tool(\r\n name = "get_weather",\r\n description = "Retrieves the current weather report for a city.",\r\n )\r\n suspend fun getWeather(\r\n @Param("The name of the city.") city: String\r\n ): Map<String, String> = mapOf("status" to "success", "report" to "Sunny and 72F in $city")\r\n}\r\n\r\nval agent = LlmAgent(\r\n model = Gemini("gemini-3.8-flash"),\r\n name = "weather_agent",\r\n instruction = Instruction("Answer questions about the weather in a city."),\r\n tools = WeatherTools().generatedTools(),\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe6906417d0>)])]>

If a tool signature uses an unsupported type or declares a default value on a non-nullable parameter, your Gradle build fails immediately instead of breaking in production. Because the .generatedTools() function uses zero runtime reflection, it survives Android R8 shrinking.

That zero-reflection design makes it possible to run the entire agent loop locally on Android. It offers JVM server support, including a Spring AI bridge and experimental ADK 2.0 graph workflows. On-device, ADK for Kotlin integrates with LiteRT-LM for tool calling and ML Kit (Gemini Nano) for text generation. It also works with Room and AppSearch for local state, and Android AppFunctions for OS-level actions, all without requiring a network connection.

Choice without compromise

Having five native languages in ADK means you don't have to pick a single runtime for your entire organization. Because every version of ADK speaks MCP and A2A, each component of your system can use the language that fits it best.

What languages have you used to build agents with Google's ADK? Let me know what stack you're running on X, LinkedIn, or Bluesky.

10 Years of the SRE book: Announcing the second edition of "Site Reliability Engineering”

Mon, 12 Oct 2026 16:00:00 +0000

A decade ago, Google released Site Reliability Engineering: How Google Runs Production Systems with O'Reilly Media. What began as a collection of our practices for managing large-scale infrastructure became a bestselling book, and inspired many organizations to adopt the reliability practices it outlined: applying software engineering to operations to build systems that are scalable, reliable, and sustainable.

Over the past ten years, computing landscapes have undergone profound shifts. AI, microservices, multi-cloud deployments, serverless paradigms, and platform engineering have all reshaped how we build and operate software. To mark the 10th anniversary of the book's release, we are excited to announce the completely rewritten second edition of the Google SRE book, reflecting a decade of lessons learned both inside Google and across the broader industry, to offer fresh guidance tailored for today's operational challenges.

Why update the book?

When we published the first edition, many organizations were focused on early cloud migrations and just starting to transition from monolithic architectures. Today, with organizations delivering software at a high velocity and at global scale, the challenge is no longer adopting cloud infrastructure, but maintaining high reliability in the face of the immense complexity.

Modern production environments operate under constant change. System boundaries are more complex, dependencies are deep, and failure modes are more complex and damaging than ever before. The explosion of resource-intensive workloads like generative AI has changed the economics and operational requirements of infrastructure management.

Writing this second edition required us to take a step back and evaluate what it takes to successfully implement SRE today.

A decade of evolution: How SRE practices have progressed

The second edition takes a deep dive into how foundational site reliability engineering practices have matured over the last ten years, shifting from manual, task-specific work toward automated, platform-wide capabilities. Consider, for example:

SLOs and error budgets: In 2016, teams relied on basic availability metrics and manual error budget policies. Today, practices have matured into user-centric, multi-tiered SLOs with dynamic burn-rate alerting and automated policy enforcement.

Monitoring and observability: Early setups centered on static dashboards, basic logs, and manual alert triage. Modern observability leverages OpenTelemetry, distributed tracing, and automated anomaly correlation to find root causes quickly.

Automation and platforms: Operations have shifted from writing custom scripts for isolated tasks to using declarative infrastructure as code, internal developer platforms, and resilient, self-healing systems.

Reliability testing: Rather than relying solely on scheduled game days and synthetic load tests, modern engineering embeds continuous resilience checks, automated pipeline testing, and live production canary verification.

Capacity and resource management: Manual provisioning and static forecasting have been replaced by dynamic auto-scaling, cloud FinOps alignment, and dedicated cost and cluster optimization for GPU and TPU workloads.

Organizational models: SRE has expanded beyond dedicated teams assigned to specific applications. Today's organizations use hybrid models, platform engineering integrations, and reliability champions embedded across product teams.

SRE in the age of AI

Then there’s the dual relationship between SRE and artificial Intelligence: how SRE supports AI infrastructure, and how SRE teams use AI to enhance operational reliability.

Running large language models (LLMs) and foundation models introduces novel reliability challenges. The book outlines methodologies for operating AI infrastructure at scale:

Managing massive GPU and TPU hardware cluster reliability and network topologies
Monitoring non-deterministic outputs and latency characteristics unique to AI
Ensuring pipeline reliability for continuous model training, fine-tuning, and data processing

AI is also rapidly becoming a force multiplier for reliability engineers. The second edition shares practical frameworks for enhancing human operational efficiency:

Using generative AI tools to assist in real-time incident triage and postmortem generation
Interrogating complex system telemetry using natural language queries
Using predictive machine learning models to identify resource bottlenecks before outages occur

What hasn’t changed? The importance of culture

While technologies and architectures continuously evolve, the core principles of SRE remain intact, and are key to establishing a reliability-focused culture. Accordingly, the second edition re-emphasizes and expands upon the human elements of system engineering, and how to get there.

Whether you’re new to the field, a practitioner, an engineering manager, or a technology executive, the rewritten second edition of Site Reliability Engineering serves as your definitive roadmap for the next era of production systems.

You can order your copy today in physical format (see Google Books) or digital format from O'Reilly.

Announcing MCP Toolbox Java SDK v1.0: Agentic data access for the enterprise

Tue, 06 Oct 2026 19:00:00 +0000

Co-authors:
Stenal Jolly, Strategic Cloud Engineer, Google
Anubhav Dhawan, Software Engineer, Google

Following the landmark announcement of MCP Toolbox v1.0, we're thrilled to announce that the MCP Toolbox Java SDK has officially reached version 1.0.

This release brings first-class, type-safe agent orchestration to one of the world's most widely adopted enterprise ecosystems. Java's mature architecture is purpose-built for rigorous demands, providing the high concurrency, strict transactional integrity, and robust state management required to safely scale mission-critical AI agents in production.

In this post, we'll tell you about what's new in Java SDK v1.0, show you a real-world example, and help you get started with your own implementation.

MCP: The universal interface

Today, developers face a compounding integration bottleneck: if you have N different AI models and M enterprise data sources, you must build, secure, and maintain N × M bespoke, custom connections. This lack of a unified integration layer forces engineering teams to rely on a fragmented web of ad-hoc pipelines. As a result, scaling an agentic architecture quickly becomes unsustainable, exposing sensitive enterprise databases to severe security vulnerabilities, fragmented access controls, and massive maintenance overhead.

Eliminating the fragmented web of custom integrations is the core problem solved by the Model Context Protocol (MCP). Acting as a universal interface—the "USB Type-C" for AI orchestration—MCP decouples models from data sources. Instead of writing custom or managed API integration code for every new model or database, developers write to a single, standardized protocol. This approach allows any MCP-compliant agent to securely and immediately interact with any MCP-enabled system. The MCP connection lets developers connect agents to real-world systems without building bespoke integrations for every new model.

What's new in Java SDK v1.0: Built for production workloads

When we announced the public Beta for the MCP Toolbox Java SDK, our goal was to bring first-class, type-safe agent orchestration to enterprise Java environments. Since then, we've collaborated with developers and open-source contributors to harden our APIs.

The v1.0 release marks a stable, backwards-compatible foundation suitable for enterprise workloads. Here's what's new and hardened since our v0.2 release:

Transport layer abstraction & HttpMcpTransport: We introduced a clean transport layer abstraction alongside HttpMcpTransport. This feature decouples the core protocol logic from underlying HTTP clients, making it easy to swap network implementations or customize connection pooling.
Decoupled client authentication: To simplify enterprise security compliance, client authentication is now decoupled using CredentialsProvider and AuthMethods classes. Credentials are resolved asynchronously on every request, so teams can refresh tokens dynamically or plug in their own token source (Google OIDC via ADC ships in the box, anything else is a one-method interface).
Default parameter support: Native support for default values in tool parameters, reducing prompt payload sizes and enhancing agent reliability.
Pruning bound parameters: Sensitive parameters that are bound server-side (like tenant_id) are now automatically stripped from exposed tool definitions so the LLM can't manipulate them.
Version selection & session tracking: Standardized MCP version selection and robust session tracking ensure consistent protocol negotiation and conversation-state lifecycles.
HTTP credential exposure warnings: Added built-in detection that warns you at runtime when credentials are about to travel over a plaintext HTTP connection.
Generic client headers map: Easily attach custom corporate proxy headers, transaction tracing IDs, or correlation metadata to all outgoing requests.

Get started with the Java SDK v1.0

We designed the MCP Toolbox Java SDK to be frictionless for enterprise teams. Just add the following dependency to your pom.xml:

code_block: <ListValue: [StructValue([('code', '<dependency>\r\n <groupId>com.google.cloud.mcp</groupId>\r\n <artifactId>mcp-toolbox-sdk-java</artifactId>\r\n <version>1.0.0</version>\r\n</dependency>'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690d80ad0>)])]>

Real-world example: The autonomous transit concierge

To demonstrate the power of the Java SDK combined with AlloyDB, let's look at an enterprise use case.

Meet Cymbal Transit, a fictitious intercity bus network. Customers don't want to click through nested dropdown menus to plan a trip. They want to ask:

"I need to get from New York to Boston tomorrow morning. Can I bring my Golden Retriever? If so, book me the fastest trip."

To answer this question, an AI agent must cross-reference unstructured data (pet policies) with structured data (schedules and seat availability) and execute a transaction (booking)—all while maintaining the context of the conversation.

The foundation: AlloyDB schema with native embeddings

We used AlloyDB for this implementation because it handles relational data and high-dimensional vectors natively. Set up your database tables with these statements:

code_block: <ListValue: [StructValue([('code', "-- Enable necessary extensions for semantic search and embeddings\r\nCREATE EXTENSION IF NOT EXISTS vector;\r\nCREATE EXTENSION IF NOT EXISTS google_ml_integration;\r\n\r\n-- Table 1: Transit Policies (Unstructured Data for RAG)\r\nCREATE TABLE transit_policies (\r\n policy_id SERIAL PRIMARY KEY,\r\n category VARCHAR(50),\r\n policy_text TEXT,\r\n policy_embedding vector(768)\r\n);\r\n\r\n-- Table 2: Intercity Bus Schedules (Structured Data)\r\nCREATE TABLE bus_schedules (\r\n trip_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),\r\n origin_city VARCHAR(100),\r\n destination_city VARCHAR(100),\r\n departure_time TIMESTAMP,\r\n arrival_time TIMESTAMP,\r\n available_seats INT DEFAULT 50,\r\n ticket_price DECIMAL(6,2)\r\n);\r\n\r\n-- Table 3: Booking Ledger (Transactional Action Data)\r\nCREATE TABLE bookings (\r\n booking_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),\r\n trip_id UUID REFERENCES bus_schedules(trip_id),\r\n passenger_id VARCHAR(100),\r\n status VARCHAR(20) DEFAULT 'CONFIRMED',\r\n booking_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP\r\n);"), ('language', 'lang-sql'), ('caption', <wagtail.rich_text.RichText object at 0x7fe690d83fd0>)])]>

Mapping intents to SQL: The `tools.yaml`

The MCP Toolbox lets you define custom tools securely. Rather than granting the LLM direct database access, a tools.yaml configuration maps natural language intents directly to parameterized, safe queries:

code_block: <ListValue: [StructValue([('code', 'kind: source\r\nname: alloydb\r\ntype: alloydb-postgres\r\nproject: my-project\r\nregion: us-central1\r\ncluster: my-cluster\r\ninstance: my-instance\r\ndatabase: postgres\r\n---\r\nkind: tool\r\nname: query-schedules\r\ntype: postgres-sql\r\nsource: alloydb\r\ndescription: Find available bus schedules between cities.\r\nparameters:\r\n - name: origin\r\n type: string\r\n description: The departure city name.\r\n - name: destination\r\n type: string\r\n description: The arrival city name.\r\n - name: limit\r\n type: integer\r\n description: Maximum number of schedules to return.\r\n default: 5\r\nstatement: |\r\n SELECT CAST(trip_id AS TEXT) AS trip_id, departure_time, ticket_price\r\n FROM bus_schedules\r\n WHERE lower(origin_city) = lower($1) AND lower(destination_city) = lower($2)\r\n ORDER BY departure_time ASC\r\n LIMIT $3'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690d82a50>)])]>

For the complete YAML file, see the tools.yaml file in the mcp-toolbox-sdk-java repository.

Stateful agent architecture in Spring Boot

The hardest part of building conversational AI in enterprise applications is managing state: when a user asks, "What times are available?" and follows up with, "Book the 8 AM one," the agent must remember prior context across turns.

Using the Java MCP Toolbox SDK with Spring Boot and LangChain4j, we can cleanly maintain conversational memory in the HTTP Session and we can cleanly separate the agent into two declarative components:

A declarative agent interface that manages the prompt, tools, and conversational memory via an HTTP session.
A tool execution service that routes agent requests directly to the MCP Toolbox server.

code_block: <ListValue: [StructValue([('code', 'interface TransitAgent {\r\n @SystemMessage({\r\n "You are the Cymbal Transit Concierge.",\r\n "Use the \'querySchedules\' tool for finding schedules.",\r\n "Use \'bookTicket\' to execute transactions.",\r\n "Use \'searchPolicies\' to look up luggage and pet rules."\r\n })\r\n String chat(@MemoryId String sessionId, @UserMessage String userMessage);\r\n}\r\n\r\n@Service\r\nclass TransitAgentTools {\r\n // These methods automatically invoke our MCP Toolbox server!\r\n @Tool("Query specific schedules between an origin and destination city.")\r\n public String querySchedules(String origin, String destination) { ... }\r\n\r\n @Tool("Book a ticket for a passenger.")\r\n public String bookTicket(String tripId, String passengerName) { ... }\r\n\r\n @Tool("Search transit policies for luggage and pet rules.")\r\n public String searchPolicies(String query) { ... }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690d801d0>)])]>

Notice how the @MemoryId annotation abstracts session tracking: Spring Boot automatically correlates conversational context to the user's HTTP session. Meanwhile, LangChain4j and the MCP Toolbox handle schema translation and tool routing behind the scenes—no handwritten if/else intent parsing required.

By pairing the MCP Toolbox Java SDK with LangChain4j, we achieve clean separation of concerns and effortless state management:

Zero-boilerplate session management: The @MemoryId String sessionId parameter binds conversation history directly to the user's HTTP session.
Declarative agent contract: The TransitAgent interface defines the model's persona and system instructions without complex prompt templating.
Type-safe tool execution: The TransitAgentTools Spring service wraps remote MCP database tools as native Java methods.

This architecture ensures your agent remains modular: you can refine prompt guidance in the interface, manage user sessions automatically, and execute secure database queries through MCP Toolbox without tight coupling.

Connecting the dots: Listing, invoking, and executing tools in Java v1.0

Now let's look under the hood of the TransitAgentTools interface. Inside those LangChain4j @Tool methods on our Spring @Service, the MCP Toolbox Java SDK handles the heavy lifting—bridging your Java service methods to the MCP tools defined in the tools.yaml file. In just a few lines of type-safe code, we can initialize our client using the new v1.0 decoupled authentication and headers abstractions:

code_block: <ListValue: [StructValue([('code', '// 1. Initialize the Client with Decoupled Auth and Custom Headers (v1.0)\r\nString serviceUrl = "https://toolbox-my-project-uc.a.run.app/mcp";\r\nMcpToolboxClient mcpClient = McpToolboxClient.builder()\r\n .baseUrl(serviceUrl)\r\n .credentialsProvider(new GoogleCredentialsProvider(serviceUrl)) // Decoupled OIDC credentials\r\n .headers(Map.of( // Generic client headers\r\n "X-Correlation-ID", "enterprise-session-abc123",\r\n "X-Client-Platform", "Spring-Boot"\r\n ))\r\n .build();\r\n\r\n// 2. Listing Discoverable Tools\r\nmcpClient.listTools().thenAccept(tools -> {\r\n System.out.println("Successfully discovered " + tools.size() + " tools.");\r\n});\r\n\r\n// 3. Invoking a Tool (Read-Only Data with Default Parameter Support)\r\n// "limit" is omitted: the SDK fills it from the default in the tool definition\r\nString schedules = mcpClient.loadTool("query-schedules")\r\n .thenCompose(tool -> tool.execute(Map.of(\r\n "origin", "New York",\r\n "destination", "Boston")))\r\n .join().text();\r\n\r\n// 4. Executing a Transactional Tool (Using Bound Parameters)\r\nAuthTokenGetter toolAuthGetter = () -> CompletableFuture.completedFuture(myIdToken);\r\n\r\nString bookingConfirmation = mcpClient.loadTool("book-ticket", Map.of("google_auth", toolAuthGetter))\r\n // Bind the authenticated user context securely. bindParam returns a new immutable\r\n // Tool, and the bound parameter is pruned from the definition exposed to the LLM!\r\n .thenCompose(tool -> tool.bindParam("passenger_name", "Jane Doe")\r\n // Execute the mutable transaction\r\n .execute(Map.of("trip_id", "123e4567-e89b-12d3-a456-426614174000")))\r\n .join().text();'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690d80490>)])]>

Secure by default: authentication and deployment

Moving an AI agent to production requires rock-solid credential handling and an infrastructure that scales with demand. Let's look at how you can enforce credential safety across environments and deploy independently on Cloud Run.

Application Default Credentials (ADC) & safety

By using the GoogleCredentialsProvider service, your Java app inherits its secure identity from its execution environment (whether local or in Google Cloud) through Application Default Credentials (ADC)—no hard-coded keys, with OIDC tokens minted and cached per audience under the hood. Furthermore, v1.0 offers HTTP Credential Exposure Warnings that automatically detect when credentials are about to travel over a plaintext HTTP connection and emit a runtime warning telling you to switch to HTTPS.

Deploying the fleet to Cloud Run

Because MCP Toolbox and the Spring Boot Agent are fully decoupled, they scale independently on Google Cloud Run to meet high concurrency and stateful conversation requirements.

To set up and configure Toolbox on Cloud Run, download the open-source MCP Toolbox for Databases and then follow the deployment guide.

Get started today

With MCP Toolbox Java SDK v1.0, enterprise Java teams can wire Spring Boot and LangChain4j agents to the Toolbox server, and through it, to AlloyDB and every other supported data source. When you use the toolbox, arguments are validated against the tool definition before they leave the JVM, authentication is decoupled, and custom headers are attached to every outgoing request. The following implementation steps will help you get started.

Step 1: Add the Dependency

To start building with the SDK, add the following dependency to your Maven project's pom.xml file:

code_block: <ListValue: [StructValue([('code', '<dependency>\r\n <groupId>com.google.cloud.mcp</groupId>\r\n <artifactId>mcp-toolbox-sdk-java</artifactId>\r\n <version>1.0.0</version>\r\n \r\n </dependency>'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690d826d0>)])]>

Step 2: Explore Resources & Demos

GitHub Repository: Java SDK for interacting with the MCP Toolbox for Databases
Official Documentation: MCP Toolbox for Databases

Demo Application: To experience the Java SDK V1.0 for MCP Toolbox latest, try the sample application Cymbal transit project.

Gradle implementation

If your team uses Gradle instead of Maven, remember they will need to translate this dependency:

code_block: <ListValue: [StructValue([('code', "implementation 'com.google.cloud.mcp:mcp-toolbox-sdk-java:1.0.0'"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690d81a90>)])]>

Automatic version tracking

If you copy this setup into automated internal repositories, keep the XML comment  intact. It's required by the release manager's deployment scripts to automatically bump versions.

Now that you can integrate your modern agentic tools and servers to your enterprise Java applications with a stale MCP Toolbox Java SDK, get started today!

Networking for AI inference model serving - GKE only and for all other backends

Tue, 06 Oct 2026 12:00:00 +0000

Enterprises and individual developers frequently run multiple AI inference models. The right architecture can simplify how the models are called while also providing centralized governance. In this post, we'll look at two reference architectures focused on networking AI inference model serving: one for Google Kubernetes Engine (GKE) and one all other backend types. First, we'll explore the commonalities between the reference architectures that you'll see later. Then we'll explore unique components of the architecture for GKE backends and finally, we'll go over the elements of the architecture for all backend types.

The entry point

You can expose your model deployment behind a stable, secure, and reliable entry point that acts as the front end for inference calls. This entry point also acts as a control zone where policy, security, and logic can be enforced. Both the Cloud Load balancer and the Inference Gateway provide entry point capability. These types of endpoints can terminate secure connections with TLS, integrate with API management components, extend functionally with service extensions, and capitalize on capabilities of Model Armor for added security.

Common services in the designs

Both reference designs use these services:

Private Service Connect inference endpoint: Anchors the entry point inside your consumer Virtual Private Cloud (VPC) network. Traffic hits a private internal IP address, keeping inference calls in your private network.
Apigee API Management (Optional): Integrates via an Apigee Extension Processor callout to handle client identity verification, rate limits, and quota enforcement before requests ever reach compute resources.
Model Armor: Serves as an inline AI safety checkpoint, screening prompts and output completions against prompt injection and sensitive data leakage.

Design pattern serving on GKE only

This section focuses on a GKE-only backend design. To understand the full end-to-end concept, please read the entire architecture document Networking for AI inference model serving on GKE. The design pattern is based on this diagram:

In addition to the common services identified in the previous section, the design for GKE uses these components:

GKE Inference Gateway: Deployed as an internal Application Load Balancer (gke-l7-rilb). It acts as a specialized ingress engine that parses incoming request payloads, evaluates HTTPRoute rules, and steers queries to appropriate model-serving targets.
Inference pools: A logical group containing replicas of the same model. When the Gateway receives a prompt, it evaluates HTTPRoute rules to select the appropriate inference pool based on the model identifier. Pools have an initial size and can be configured to autoscale dynamically.
Model replica sets: Individual model replicas (inference server instances) deployed across single-node or multi-node GPU or TPU node pools. A replica set represents a uniform group of these model replicas.

The traffic flow GKE example

A client application that uses this GKE-based architecture to call a backend model would go through a flow like this:

Ingress: A client application in the consumer VPC issues an OpenAI-compatible API call to the local Private Service Connect endpoint, routing directly to the GKE Inference gateway using a regional internal Application Load Balancer.
Payload inspection: The Gateway reads the target model parameter specified in the request body and adds it to the HTTP headers.
Control plane validation: If Apigee is used, it checks client credentials and quotas. Model Armor screens the prompt for policy violations or data leakage.
Backend selection: The Gateway evaluates HTTPRoute mappings to identify the target pool, matches shared prefix cache context, and routes to the lowest-load GPU or TPU replica based on real-time Prometheus data.
Egress: The replica runs the inference workload. Output tokens pass through Model Armor for final response verification before streaming back over the private connection.

Design pattern serving on all backends

This section focuses on multiple backend types which can be used for inference, and it provides an overview of the architecture in the following diagram. To understand the full end-to-end concept, please read the entire architecture document Networking for AI inference model serving on all backends.

For architectures spanning mixed environments such as GKE, Cloud Run, Agent Platform, on-premises data centers, or external clouds, these additional components are used:

Regional internal Application Load Balancer: Serves as the central Layer 7 routing proxy that manages routing logic, SSL termination, and Service Extensions callouts.
Inference Payload Processor (Service Extensions): This is similar to body-based routing as used in the GKE Inference Gateway, but to enable the functionality on an Application Load Balancer a service extension is needed. A lightweight Cloud Run callout inspects the JSON body of incoming OpenAI API requests, extracts the target model identifier, and writes an X-Gateway-Model-Name header to drive URL map routing.
Network Endpoint Group (NEG): Delivers flexible routing to heterogeneous backends based on the injected model header.

All-backends traffic flow example

A client application that uses this architecture to call a backend model would go through a flow like this:

Private ingress: The client application targets the Private Service Connect endpoint over private IP address space. The regional internal Application Load Balancer receives the request.
Model name extraction: The load balancer sends the payload to the Cloud Run body-based router callout, which inspects the JSON payload and injects the X-Gateway-Model-Name header.
Policy and safety enforcement: The request passes to Apigee for identity and quota validation, then to Model Armor to scrub sensitive data and block malicious prompts.
NEG routing: The load balancer URL map inspects the model header and forwards the request to the matching backend NEG (Agent Platform, GKE, Cloud Run, Hybrid, or Internet).
Private delivery: The target backend executes the model prompt, Model Armor screens the completion, and the result returns privately along the ingress path.

What's next

Take a deeper dive into building AI workloads on Google Cloud:

Document set: Agentic AI architecture guides
Architecture Center: Multi-agent private networking patterns in Google Cloud
Document: Gemini Enterprise Agent Platform networking access overview

Want to ask a question, find out more, or share a thought? Please connect with me on Linkedin.

GKE CPU startup boost: Accelerate app starts without over-provisioning

Fri, 02 Oct 2026 16:00:00 +0000

Whether you’re launching microservices in response to sudden traffic spikes, deploying new software releases, or scaling up application replicas, pod startup time is critical to maintaining a fast, responsive user experience for applications running on Google Kubernetes Engine (GKE).

Yet, platform engineers and developers face a persistent dilemma: Applications often demand significantly more CPU power during startup than they do during steady-state operations. Sizing CPU requests for normal, steady-state usage leads to CPU throttling during launch, which can result in sluggish cold starts and readiness probe timeouts. On the flip side, over-provisioning baseline CPU requests to satisfy short-lived startup bursts wastes valuable compute resources, inflating infrastructure bills.

Today, we are excited to announce CPU startup boost for GKE in preview. Integrated directly into GKE's Vertical Pod Autoscaler (VPA), CPU startup boost dynamically elevates a container's CPU allocation during initialization and seamlessly scales it back to baseline steady-state levels once the application is ready - all without restarting your containers.

Why modern applications need extra CPU at boot time

When a new container launches, it may perform intensive initialization tasks before it begins serving user requests. Depending on your tech stack, the following startup workloads require substantial CPU cycles:

Java JVM applications: Frameworks like Spring Boot require high CPU burst capacity for class loading, classpath scanning, instantiating dependency injection containers, and running Just-in-Time (JIT) compilation.
Node.js servers: Apps parse JavaScript files, build complex module dependency trees (require/import), and execute V8 engine optimization and JIT compilation passes during initial execution.
Python and AI/ML microservices: These services spend initial cycles importing heavy libraries (such as PyTorch, NumPy, or LangChain), compiling .pyc bytecode, establishing ORM database schemas, and pre-loading cache structures.

If you size CPU requests strictly for steady-state performance, these initialization workloads experience CPU throttling on launch, delaying readiness probes. To prevent slow cold starts, teams frequently overprovision CPU requests. However, once the application stabilizes, those extra CPU resources sit idle, increasing your cloud spend without adding value.

How CPU startup boost can help

CPU Startup Boost solves this by giving your workloads temporary vCPU "boosts" during launch, and automatically returning them to baseline once initialization completes.

Key benefits:

Faster cold starts: Reduce application initialization times by up to 2x, accelerating auto-scaling responsiveness during unexpected traffic surges.
Optimized cloud spend: Right-size steady-state CPU requests to fit actual runtime needs rather than paying for idle startup headroom.
Zero pod restarts: Dynamic resource resizing happens live inside the running container.
Flexible policy controls: Apply simple pod-level multiplier factors (e.g., 2x CPU during startup) or define granular, container-specific rules for complex multi-container pods.

Under the hood: Kubernetes In-place Pod Resize

Historically, changing a pod's resource requests or limits required deleting and recreating the pod. This disruptive process triggered container restarts, cache invalidation, and node rescheduling overhead.

To fix that, CPU startup boost builds on Kubernetes In-place Pod Resize (IPPR).

Tracked under KEP-1287, IPPR introduced dynamic, in-place resource mutation. Introduced as Alpha in Kubernetes 1.27, promoted to Beta in v1.33, and graduating to General Availability (GA) in v1.35, IPPR allows the Kubernetes control plane and kubelet to update container CPU and memory requests on running pods without restarting the container process.

GKE leverages IPPR within the VPA to apply startup CPU boosts at pod admission and smoothly step them down post-readiness.

How CPU startup boost works (pod lifecycle overview)

CPU startup boost operates across three distinct phases:

Admission phase: When you deploy a pod, the VPA admission webhook intercepts the creation request. It calculates the elevated CPU request based on your policy (e.g., 2x multiplier or +2 vCPUs) and injects the boosted CPU request along with tracking annotations into the pod spec.
Startup phase: The pod is scheduled and initialized with the boosted CPU allocation. Your application completes class loading, JIT compilation, or module parsing at top speed without experiencing CPU throttling.
Unboosting phase: As soon as the pod's readinessProbe passes (plus any configured durationSeconds cooldown delay), the VPA updater issues an in-place resize request. The CPU request steps back down to your baseline level while the container continues running uninterrupted.

Prerequisites and availability

CPU startup boost is available today in preview on GKE:

GKE version: Version 1.36.0-gke.4447000 or later on Standard and Autopilot clusters.
Cluster Modes: Enabled natively on GKE Autopilot (VPA is active by default). On GKE Standard, simply ensure Vertical Pod Autoscaling (VPA) is enabled.
Workload Support: Works with standard Kubernetes controllers, including Deployments and StatefulSets.

Getting started: Configuring CPU startup boost

Configuring CPU startup boost is as simple as adding a startupBoost section to your VerticalPodAutoscaler manifest.

Example 1: Pod-level boost with fixed steady-state (`updateMode: "Off"`)

If you want to use VPA purely for startup boost while keeping steady-state CPU requests locked to your manifest definitions, set updateMode to "Off":

code_block: <ListValue: [StructValue([('code', 'apiVersion: "autoscaling.k8s.io/v1"\r\nkind: VerticalPodAutoscaler\r\nmetadata:\r\n name: java-app-startup-boost\r\n namespace: default\r\nspec:\r\n targetRef:\r\n apiVersion: "apps/v1"\r\n kind: Deployment\r\n name: java-app\r\n updatePolicy:\r\n updateMode: "Off"\r\n startupBoost:\r\n cpu:\r\n type: "Factor"\r\n factor: 2\r\n durationSeconds: 10'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe6911db910>)])]>

In this example, GKE doubles the container's CPU request during launch and holds the boosted allocation for 10 seconds after readiness probes pass before scaling back to baseline.

Example 2: Combining startup boost with continuous VPA auto-scaling

If you want GKE to boost CPU during launch and continuously optimize steady-state resources post-startup, set updateMode to "InPlaceOrRecreate":

code_block: <ListValue: [StructValue([('code', 'apiVersion: "autoscaling.k8s.io/v1"\r\nkind: VerticalPodAutoscaler\r\nmetadata:\r\n name: nodejs-app-vpa\r\n namespace: default\r\nspec:\r\n targetRef:\r\n apiVersion: "apps/v1"\r\n kind: Deployment\r\n name: nodejs-service\r\n updatePolicy:\r\n updateMode: "InPlaceOrRecreate"\r\n startupBoost:\r\n cpu:\r\n type: "Factor"\r\n factor: 3\r\n durationSeconds: 15'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe6911d8ed0>)])]>

Example 3: Granular container-level boost

For pods running sidecars (such as logging agents or service mesh proxies) that do not require extra CPU on boot, target specific app containers:

code_block: <ListValue: [StructValue([('code', 'apiVersion: "autoscaling.k8s.io/v1"\r\nkind: VerticalPodAutoscaler\r\nmetadata:\r\n name: app-container-boost\r\nspec:\r\n targetRef:\r\n apiVersion: "apps/v1"\r\n kind: Deployment\r\n name: API-gateway\r\n updatePolicy:\r\n updateMode: "Off"\r\n resourcePolicy:\r\n containerPolicies:\r\n - containerName: "web-app"\r\n mode: "Off"\r\n startupBoost:\r\n cpu:\r\n type: "Quantity"\r\n quantity: "2"\r\n durationSeconds: 5'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690d80250>)])]>

Verifying startup boost in your cluster

You can verify that GKE applied and downscaled the startup boost using kubectl:

1. Inspect pod annotations: Check for the vpaCpuStartupBoost tracking annotation:

code_block: <ListValue: [StructValue([('code', 'kubectl get pod <POD_NAME> -o yaml'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe6914e6810>)])]>

Look for annotations indicating the original baseline and boosted CPU requests.

2. Monitor in-place resize events: Confirm that GKE downscaled the CPU request back to baseline after readiness:

code_block: <ListValue: [StructValue([('code', 'kubectl get events --field-selector reason=InPlaceResizedByVPA'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe6914e7290>)])]>

An event with reason=InPlaceResizedByVPA confirms successful in-place downscaling post-startup.

Best practices for production workloads

Pairing with Horizontal Pod Autoscaler (HPA): When using HPA alongside CPU startup boost, ensure a robust readinessProbe is defined and keep durationSeconds short (e.g., 0s–10s). This prevents HPA from falsely interpreting initialization CPU spikes as high steady-state load.
Handle traffic spikes with GKE capacity buffers API: By reducing the startup tax at scale, you can achieve higher workload density on fewer nodes to improve overall utilization. Consider adopting the GKE capacity buffers API to absorb sudden traffic surges with minimal operational overhead while maintaining strict SLOs.
GKE Autopilot resource ratios: On GKE Autopilot, remember that pods must maintain valid CPU-to-memory ratios. Ensure baseline memory allocations accommodate the boosted CPU ratio during startup.

Get started today

CPU startup boost gives GKE users the best of both worlds: lightning-fast cold starts for CPU-intensive workloads like Java, Node.js, and Python, paired with maximum resource efficiency and lower cloud costs.

Ready to accelerate your GKE workloads?

Explore the GKE CPU startup boost documentation.
Learn more about Vertical Pod Autoscaling on GKE.
Try CPU startup boost on your GKE Standard or Autopilot clusters running GKE 1.36.0-gke.4447000 or later!

Introducing the Server Side Cloud Swift SDK

Thu, 01 Oct 2026 04:00:00 +0000

For years, Swift was perceived mainly as a UI language tied to Apple client devices. With Swift 6 and strict concurrency checking, it has matured into a viable systems and cloud language, pairing Rust-like data-race safety with predictable, reference-counted performance.

To support this ecosystem, Google engineering has launched the official Google Cloud API Client Libraries for Swift. Built from the ground up for Swift 6.2+, this new SDK uses the latest non-blocking Swift NIO event loops, HTTP/2 multiplexing, gRPC transport, and zero-cost compile-time data race safety.

In this article, we'll walk you through all you need to know to get started, and to understand how the Server Side Cloud Swift SDK, or google-cloud-swift, works.

The rise of server-side Swift and cloud-native concurrency

Traditional backend development often forces a trade-off between developer ergonomics and resource utilization. While managed runtimes offer rapid development, lower-level systems languages provide finer control over memory and CPU footprint, often at the expense of feature velocity.

Server-side Swift aims to strike a practical balance. Swift pairs a lightweight runtime and Automatic Reference Counting (ARC) with expressive syntax. More importantly, Swift 6 introduces compile-time concurrency checking. When you share state between async tasks across a cloud microservice, the compiler enforces that types conform to Sendable. Data races are caught in your editor before a binary ever compiles or reaches production.

At the network layer, every request to Google Cloud APIs runs over event-driven, non-blocking sockets that scale across multicore Linux server environments without spawning system threads per connection.

Where to use the Swift SDK

The Server Side Cloud Swift SDK is engineered for server, container, and automated DevOps environments.

When you build high-throughput microservices with Swift web frameworks like Hummingbird or Vapor, google-cloud-swift provides native access to Cloud Storage, AI, Identity and Access Management (IAM), and over one hundred other Google Cloud services. You can containerize your executable on Linux and deploy directly to Cloud Run, Google Kubernetes Engine (GKE), or Compute Engine VMs.

Because the SDK compiles on macOS, and Linux, you can develop the backend in your preferred development environment, and then seamlessly deploy to production. And using Swift on both the frontend and backend allows you to share application-specific types across both.

The SDK also excels at platform engineering and DevOps automation. You can author cross-platform CLI utilities and data rotation scripts that run on your developer laptop or inside CI/CD pipelines. These tools authenticate automatically against Google Cloud using Application Default Credentials (ADC) or Workload Identity Federation.

If you're building an iOS, iPadOS, or visionOS app for the Apple App Store, you should not embed google-cloud-swift directly into your client bundle. Shipping Google Cloud service account keys or administrative credentials inside a client binary creates security risks. For direct client-side features, use the Firebase SDK for Apple Platforms to handle user authentication, real-time Firestore sync, and client-side security rules, or route requests through your own Cloud Run backend API.

Getting started with your IDE and packages

Because google-cloud-swift treats Linux and macOS as first-class citizens, you can develop on Apple hardware with Xcode or on Linux workstations with Visual Studio Code and swiftly.

To install the official Swift compiler on Linux workstations using the swiftly CLI installer, run:

code_block: <ListValue: [StructValue([('code', 'curl -O https://download.swift.org/swiftly/linux/swiftly-$(uname -m).tar.gz &&\r\ntar zxf swiftly-$(uname -m).tar.gz &&\r\n./swiftly init --quiet-shell-followup &&\r\n. "${SWIFTLY_HOME_DIR:-$HOME/.local/share/swiftly}/env.sh" &&\r\nhash -r'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690aafcd0>)])]>

Alternatively, you can download prebuilt toolchain tarballs directly from official Swift Downloads for Ubuntu, Debian, Fedora, or Amazon Linux. Note that google-cloud-swift requires Swift 6.2 or later, so verify your compiler version with swift --version after installation.

To resolve Swift Package Manager bare repository trust warnings when cloning across Linux filesystems, configure Git before building with git config --global safe.bareRepository all.

Add the required packages to your Package.swift manifest:

code_block: <ListValue: [StructValue([('code', 'swift package add-dependency https://github.com/googleapis/swift-google-cloud-language-v2.git --from 0.4.0\r\nswift package add-target-dependency GoogleCloudLanguageV2 CloudBackendService --package swift-google-cloud-language-v2'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690aad050>)])]>

On macOS you need to change the platforms directive:

code_block: <ListValue: [StructValue([('code', '// swift-tools-version: 6.2\r\nimport PackageDescription\r\n\r\nlet package = Package(\r\n name: "CloudBackendService",\r\n // Applied when compiling on Darwin/macOS; ignored by SPM on Linux targets\r\n platforms: [.macOS(.v15)],\r\n ... ...'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690aac690>)])]>

In most environments a default-initialized client can make requests:

code_block: <ListValue: [StructValue([('code', 'import Foundation\r\nimport GoogleCloudLanguageV2\r\n\r\nfunc analyzeTextSentiment(text: String) async throws {\r\n // Initialize explicit API key credentials\r\n let client = try LanguageServiceClient()\r\n\r\n // Configure request using structured builder closure\r\n let document = Document().with {\r\n $0.type = .plainText\r\n $0.source = .content(text)\r\n }\r\n\r\n let response = try await client.analyzeSentiment(\r\n request: AnalyzeSentimentRequest().with { $0.document = document }\r\n )\r\n\r\n if let sentiment = response.documentSentiment {\r\n print("Document sentiment score: \\(sentiment.score)")\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690aac510>)])]>

Notice how Document().with { ... } avoids verbose temporary variables or mutating setters by providing a clean, thread-safe configuration closure.

Networking, transport, and authentication

The repository splits infrastructure primitives into modular packages under packages/:

swift-google-cloud-auth: Implements Application Default Credentials discovery, service account JWT signing, external account exchange for Workload Identity Federation, and API keys.
swift-google-cloud-wkt: Provides idiomatic Swift types for Google Protocol Buffer well-known types, including nanosecond-precision Timestamp representations that bridge cleanly to Swift's Date.
swift-google-cloud-gax: Handles Google API Extensions such as automated retry loops, exponential backoff, and pagination state machines.

When you initialize any client library without arguments, Credentials.default() automatically scans your environment (GOOGLE_APPLICATION_CREDENTIALS, quota project variables, or the local Google Cloud CLI configuration) and authenticates connections over gRPC or HTTP/2.

If you need to programmatically override credentials with an API key or attach custom access headers, you can pass explicit configuration options. For example, you could modify the previous example to use the following:

code_block: <ListValue: [StructValue([('code', 'import Foundation\r\nimport GoogleCloudAuth\r\nimport GoogleCloudGax\r\nimport GoogleCloudLanguageV2\r\n\r\nfunc analyzeTextSentiment(apiKey: String, text: String) async throws {\r\n // Initialize explicit API key credentials\r\n let credentials = try Credentials(configuration: .apiKey(apiKey))\r\n let client = try LanguageServiceClient(\r\n ClientOptions().with { $0.credentials = credentials }\r\n )\r\n\r\n // Configure request using structured builder closure\r\n let document = Document().with {\r\n $0.type = .plainText\r\n $0.source = .content(text)\r\n }\r\n\r\n let response = try await client.analyzeSentiment(\r\n request: AnalyzeSentimentRequest().with { $0.document = document }\r\n )\r\n\r\n if let sentiment = response.documentSentiment {\r\n print("Document sentiment score: \\(sentiment.score)")\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690aaf0d0>)])]>

The autogenerated client ecosystem

Google Cloud operates a vast ecosystem of APIs whose schemas update regularly. The teams supporting google-cloud-swift use code generators to automatically update the client libraries with the latest features and with new APIs. Using code generators produces stable APIs, without disruptive breaking changes. While the releases are on a fixed cadence, please contact Cloud Customer Care if you need a particular feature or API urgently.

Whether you need to rotate keys in Secret Manager or invoke multimodal inference models via the Gemini API on Gemini Enterprise Agent Platform, the generated SDKs follow consistent naming and async/await signatures.

These generated clients offer more than plain unary RPC wrappers. They also offer wrappers that simplify application development. For example, iterating over long results involves fetching pages of results with one RPC, iterating over the page of results, and then preparing a new request to retrieve the following page. Using the generated clients this becomes an asynchronous iterator. This example shows how to query project secrets using GoogleCloudSecretManagerV1:

code_block: <ListValue: [StructValue([('code', 'import Foundation\r\nimport GoogleCloudSecretManagerV1\r\n\r\n@main\r\nstruct SecretManagerQuickstart {\r\n static func main() async throws {\r\n guard let projectId = CommandLine.arguments.dropFirst().first else {\r\n print("Usage: SecretManagerQuickstart <projectId>")\r\n exit(1)\r\n }\r\n\r\n // Connects using Application Default Credentials automatically\r\n let client = try SecretManagerServiceClient()\r\n\r\n let request = ListSecretsRequest().with {\r\n $0.parent = "projects/\\(projectId)"\r\n }\r\n\r\n // Async sequence streams pages of secrets automatically\r\n print("Secrets in project \\(projectId):")\r\n for try await item in try client.listSecretsByItem(request: request) {\r\n print(" - \\(item.name)")\r\n }\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690aafdd0>)])]>

The pagination response returns an asynchronous sequence (AsyncSequence). You can iterate over items with for try await while the client library fetches subsequent pages in the background over non-blocking NIO channels.

Where to go next

With google-cloud-swift, server-side Swift developers can write end-to-end cloud infrastructure with compile-time race safety, native async/await ergonomic APIs, and zero OS thread congestion.

To inspect the source code, open issues, or contribute new veneers, visit the official repository at googleapis/google-cloud-swift.

Want to discuss server-side Swift architectures or Cloud Run containerization? Join the Google Developer Program to continue the conversation.

Democratizing Managed Lustre with lower cost and frictionless development

Thu, 01 Oct 2026 04:00:00 +0000

This is the first of a two-part series exploring how Google Cloud is bringing the foundational values of a high-performance parallel filesystem–TB/s throughput, sub-ms latency at high client scale, and POSIX support–to a broader set of use cases and users.

Historically, due to the cost and special purpose nature of parallel filesystems, colder data had to be stored outside of the filesystem and AI developers have had to maintain separate, slower environments for writing code, compiling libraries, and managing repositories. This fragmentation increases the toil of manual data staging, dataset copying, and managing disjointed namespaces.

Google Cloud Managed Lustre is solving these problems through our 6 cents/GB*month Dynamic Tier and by optimizing Managed Lustre performance for a range of development tasks and workloads – making Managed Lustre a “One-Stop Shop” for high-performance AI and HPC workloads.

Lower Cost: More Lustre for Less with the Dynamic Tier

The Managed Lustre Dynamic Tier provides sub-ms latency for hot data, which allows you to store all of your data in a single namespace, and costs only 6 cents/GB*month.

Throughput, capacity scale and client scale: Throughput scales linearly with capacity up to 80 PB, while sub-ms latency for hot data remains stable as you scale to tens of thousands of clients.
Single-flat fee: Predictable pricing. No independent charges for disk media types, data movement within the namespace, or metadata IOPS.
Read Latencies: Sub-ms latencies for High-Performance Cache (SSD). The Capacity Pool (“HDD”) is built on Google Cloud Hyperdisk throughput, which has an average read latency of 10 to 30 ms.

Recommended workloads for Dynamic Tier

Multi-Epoch Training and/or Training with Optimized Fetch Sizes: Hot data is promoted to the High Performance Cache (SSD) after the first run. Larger data prefetch will allow you to take advantage of the Dynamic Tier cost structure and gain from low-latency SSD.
Write-Heavy Checkpointing: Bursty checkpoint writes land directly in the High Performance Cache. Older checkpoints are transparently demoted to the Capacity Pool (HDD).
Rapid Checkpoint Restore: New checkpoints are written to the High Performance Cache, enabling low-latency checkpoint restores.
Interactive Snappiness for Developers: Low-latency tasks like git cloning, compiling libraries, or running notebooks benefit from a local-disk feel (~300µs average read latencies) on the same shared workspace hosting large training sets.

Frictionless development: Lustre as a one-stop shop for developer’s workloads

In addition to Managed Lustre’s scalability for large AI and HPC workloads (checkpoint/restart/data-loading), it also meets the demands for interactive work, meaning developers can start on Managed Lustre and stay on Managed Lustre throughout the entire workload lifecycle:

Unified Foundation & Interactive Performance

Consolidates the AI and HPC lifecycle into a single namespace, providing a "local disk" feel for interactive work (Read more about the latency benefits of Managed Lustre experienced by Salesforce and others).

Latency: ~300µs average read latency—delivering up to 4x better responsiveness than alternative distributed file systems.
Accelerated Setup: Untar the Linux kernel in ~2 minutes (4.7x faster than alternative file solutions), run a 20-worker parallel git clone of Python in ~40 seconds, compile Python in ~200s.

High-Concurrency Broadcast & Cluster Startup

Managed Lustre maximizes GPU ROI by preventing storage bottlenecks during cluster initialization.

When thousands of worker nodes attempt to read the exact same file simultaneously (such as a shared model checkpoint, base weights, or container layer), traditional distributed file systems can choke on localized hotspotting, leaving high-cost GPU clusters idle for minutes.

Improves Aggregate Throughput for a large number of clients reading the same file: Demonstrates a 67% improvement over alternative file solutions.¹
Parallel Loading: Imports libraries like PyTorch across 4,000+ processes in under 60 seconds.

¹Storage system specs: 500 MBps per TiB tier of Managed Lustre, 108,000 GiB capacity. Zonal Filestore at 102,400 GiB capacity. Average throughput of 36.7 GB/s to 2,048 client VMs reading the same 40 GiB file.

Run One-Stop Shop Workflows for Yourself

Here is the code for the tests we’ve run, so that you can perform your own testing.

Low latency for interactive access

We used fio to emulate small, low-concurrency reads and writes:

code_block: <ListValue: [StructValue([('code', '# Read workload\r\nfio --ioengine=libaio --filesize=100M --ramp_time=2s \\\r\n --runtime=2m --time_based --numjobs=1 --direct=1 --verify=0 --randrepeat=0 \\\r\n --group_reporting --directory=~/LUSTRE_MOUNT \\\r\n --name=randread --blocksize=4k --iodepth=1 --readwrite=randread \\\r\n --buffer_compress_percentage=50\r\n\r\n# Write workload\r\nfio --ioengine=libaio --filesize=100M --ramp_time=2s \\\r\n --runtime=2m --time_based --numjobs=1 --direct=1 --verify=0 --randrepeat=0 \\\r\n --group_reporting --directory=~/LUSTRE_MOUNT \\\r\n --name=randwrite --blocksize=4k --iodepth=1 --readwrite=randwrite \\\r\n --buffer_compress_percentage=50'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690312b10>)])]>

Accelerated setup

How to run Linux untar

code_block: <ListValue: [StructValue([('code', '# Download a kernel tarball\r\nwget -P /tmp https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.18.9.tar.xz\r\n\r\n# Extract the archive to the Lustre mount\r\nmkdir ~/LUSTRE_MOUNT/kernel\r\ntar -C ~/LUSTRE_MOUNT/kernel -xf /tmp/linux-5.18.9.tar.xz'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe6914e7050>)])]>

In the above use case, you will want to take care to avoid the metadata performance tax that can come from running as root (Namely, tar issues chown and chmod calls to make extracted files’ owner+permissions match the ones recorded in the archive.). If you still wish to run as root (and have verified that this approach is compatible with your setup), you may specify `--no-same-owner --no-same-permissions` in order to ensure that extracted files maintain root as owner and have root's default file permissions. In other words, it makes extraction as root behave like extraction as non-root (by ignoring the owner+permissions in the archive).

How to run Python gitclone

code_block: <ListValue: [StructValue([('code', 'git config --global checkout.workers 20\r\nmkdir ~/LUSTRE_MOUNT/python\r\ngit clone https://github.com/python/cpython.git ~/LUSTRE_MOUNT/python'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe6914e5350>)])]>

How to run Python compile

code_block: <ListValue: [StructValue([('code', 'pushd ~/LUSTRE_MOUNT/python\r\n./configure > /dev/null\r\nmake > /dev/null\r\npopd'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe6914e6dd0>)])]>

High scale distribution

Aggregate throughput for distributing one large file to many nodes

Run the below on each client VM:

code_block: <ListValue: [StructValue([('code', '# Start fio in server mode\r\nfio --server'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe68bc2d310>)])]>

Run the below on a selected client VM:

code_block: <ListValue: [StructValue([('code', "# Create a 40 GiB file\r\nfio --name=job1 \\\r\n --ioengine=libaio \\\r\n --direct=1 \\\r\n --buffer_compress_percentage=50 \\\r\n --blocksize=4m \\\r\n --iodepth=32 \\\r\n --filesize=40g \\\r\n --readwrite=write \\\r\n --filename ~/LUSTRE_MOUNT/40gb_test\r\n\r\n# Create an fio job file for the read workload\r\ncat <<'EOF' > /tmp/read.fio\r\n[job1]\r\nfilename=${HOME}/LUSTRE_MOUNT/40gb_test\r\nrw=read\r\nbs=4m\r\nexitall_on_error=1\r\nEOF\r\n\r\n# Run the read workload using all client VMs in ~/hostfile\r\nfio --client ~/hostfile /tmp/read.fio"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe69246d0d0>)])]>

Parallel loading of libraries across many processes

Run the below on a selected client VM:

code_block: <ListValue: [StructValue([('code', '# Install PyTorch in a virtual env\r\npython3 -m venv ~/LUSTRE_MOUNT/env\r\nsource ~/LUSTRE_MOUNT/env/bin/activate\r\npip3 install --upgrade pip\r\npip3 install torch torchvision torchaudio \r\ndeactivate\r\n\r\n# Import PyTorch on all client VMs in ~/hostfile, 4 processes per host\r\nmpirun --allow-run-as-root --oversubscribe --hostfile ~/hostfile -N 4 \\\r\n bash -c \'source ~/LUSTRE_MOUNT/env/bin/activate && python3 -c "import torch"\''), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe69246c1d0>)])]>

Looking ahead and next steps

By eliminating the manual data staging tax and lowering entry costs with the Dynamic Tier, Google Cloud Managed Lustre is evolving from an elite, single-purpose engine into a highly versatile, unified storage fabric for the entire AI lifecycle.

In the second part of this series, we will focus on upcoming object integration features. Stay tuned!

Next steps

Run the benchmarks yourself (if you haven’t already): Deploy a Google Cloud Managed Lustre instance using the Google Cloud console and run tests provided above to benchmark your own workloads.
Explore the Dynamic Tier: Read the Google Cloud Managed Lustre Documentation to learn more about the Dynamic Tier.
Stay tuned for Part 2: In the next installment of this series, we will dive deep into upcoming object integration features and how they further simplify AI and HPC storage.
Get started with centralizing your development-to-training lifecycle on Google Cloud Managed Lustre!

Empower your agents with the Google Cloud CLI remote MCP server

Wed, 30 Sep 2026 16:00:00 +0000

Today, we’re expanding our ecosystem of managed remote MCP servers by introducing the Google Cloud CLI remote MCP server in preview.

Powered by the popular gcloud and bq (BigQuery) command-line tools, this new server gives AI agents immediate, broad access to command-line operations for managing Google Cloud infrastructure and working with advanced BigQuery workflows securely and seamlessly.

Why CLI matters for AI agents

Agents are increasingly performing complex cloud operations, but standardizing how they interact with backend systems remains a challenge. The Google Cloud CLI remote MCP server bridges this gap by packaging the versatility of hundreds of gcloud and bq commands into one single MCP server. This results in two strong benefits for the agent:

Higher-level abstractions: CLI commands package complex multi-step workflows, validation checks, and high-level operations into unified commands rather than requiring multi-step API orchestration.
Leverages model training: LLMs are heavily pre-trained on public command-line documentation, syntaxes, and usage examples, making CLI invocation intuitive and highly accurate for models.

The benefits of putting CLI behind remote MCP

Managing cloud infrastructure with AI agents traditionally requires installing and maintaining Google Cloud CLI binaries inside agent execution environments. The Cloud CLI remote MCP server bridges CLI capabilities with MCP benefits by providing an isolated execution sandbox on Google Cloud infrastructure. This solves key infrastructure challenges:

Simplified dependency and runtime management: For teams building custom agents, maintaining local CLI versions and dependencies across dev, test, and production environments creates operational overhead. Remote MCP eliminates local installations and runtime maintenance.
Access for web-based agent endpoints: Web-hosted agent platforms and web interfaces (such as Gemini Enterprise and other hosted enterprise agent platforms) run in environments where users cannot control or install local packages. Remote MCP enables secure, managed access to Google Cloud CLI operations directly from these surfaces.

Enterprise-grade security and governance

Connecting an AI agent to your infrastructure requires strict, enterprise-ready safeguards. This remote server leverages Google Cloud's standard identity and governance frameworks to keep your environments secure:

Zero ambient credentials: The server isolates execution in a network-restricted proxy boundary with no ambient credentials. Authentication and authorization are handled through Agent Identity, OAuth 2.0, and Identity and Access Management (IAM).
Strict policy enforcement: Every command executed through the remote MCP server is run with the permissions of the authenticated caller identity. Both standard IAM permissions and organization policy service constraints are strictly enforced against downstream target resources.
Advanced protection with Model Armor: To minimize the risks associated with AI tool calling, the Cloud CLI remote MCP server integrates with Model Armor. You can proactively screen LLM prompts and responses to protect against risks like prompt injection and malicious inputs.
Cloud audit logging: The Cloud CLI remote MCP server can be configured to log every tool invocation to Audit Logs (Data Access logs under cloudcli.googleapis.com/mcp). Security teams can gain full visibility into caller identities, OAuth clients, and IAM authorization decisions (mcp.googleapis.com/tools.call) without exposing sensitive command payloads or personally identifiable information (PII).

Connecting to the Google Cloud CLI Remote MCP Server

Integrating cloud management into your agents no longer requires packaging Google Cloud CLI binaries, managing local execution runtimes, or maintaining dependencies inside agent container images. Because the Google Cloud CLI remote MCP server implements the standard Model Context Protocol, any MCP-compatible agent platform or orchestration runtime can connect immediately via standard configuration:

code_block: <ListValue: [StructValue([('code', '{\r\n "mcpServers": {\r\n "google-cloud-cli": {\r\n "uri": "https://cloudcli.googleapis.com/mcp",\r\n ...\r\n }\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe6904bf610>)])]>

Your agent immediately gains access to execute gcloud and bq commands in a secure, network-isolated cloud sandbox.

Authentication is handled via keyless Agent Identity for hosted Google Cloud platforms, or standard OAuth 2.0 for external runtimes. For authentication options, see the MCP Authentication Guide.

Bringing infrastructure management to agents

The Cloud CLI remote MCP server exposes two powerful tools, run_gcloud_command and run_bq_command, giving your AI agents broad, immediate access to Google Cloud operations through natural language.

Managing cloud infrastructure with run_gcloud_command

With run_gcloud_command, agents can execute the full breadth of gcloud operations to manage, diagnose, and secure your Google Cloud environment. An example follows:

Observability and incident diagnostics: An agent streamlines incident diagnostics by automating command execution and reducing context-switching across tools.

Extending BigQuery operations with the run_bq_command tool

While the BigQuery MCP server already helps organizations analyze and explore data using AI agents, with the introduction of run_bq_command, agents can now tackle advanced BigQuery tasks such as resource allocation, job monitoring, and task scheduling by unlocking the full scope of bq CLI functionality. Key capabilities include:

Automating scheduled queries: An agent utilizes BigQuery Data Transfer Service configurations to schedule queries automatically.
Job and resource management: Gain deep insight into query execution details, including processed data volume, slot usage, and execution plans, as well as managing reservations.
Access and permissions control: Data administrators and owners can inspect and update table permissions directly through the agent.

Pricing and availability

The Google Cloud CLI MCP server is available today in public preview. There is no additional charge to use the MCP server itself. You pay only for the GCP resources you create and any applicable data transfer costs.

To get started, enable the Cloud CLI Execution API (`cloudcli.googleapis.com`) in your Google Cloud project, grant the required MCP Tool User (`roles/mcp.toolUser`) IAM role to your agent or user identity, and configure your MCP client to connect to `cloudcli.googleapis.com/mcp`.

Data Agent Kit is now GA: Bring Google Data Cloud to any coding agent

Wed, 30 Sep 2026 13:00:00 +0000

Today, Google Cloud Data Agent Kit is generally available. Data Agent Kit is a free set of Model Context Protocol (MCP) tools and agent skills that lets the coding agent you already use work directly with your Google Cloud data products, whether you're using Antigravity, Claude Code, Codex, or other popular tools.

With GA, we are adding support for BigQuery Graph, Bigtable, and Managed Service for Apache Spark access to your open Lakehouse, along with dozens of quality-of-life improvements that make everyday work faster and smoother.

What is Data Agent Kit?

Coding agents have become remarkably good at writing SQL, PySpark, and pipeline code. What they don't have by default is context about your environment: which tables exist, how they're partitioned, which ones your team trusts, or why last night's job failed. Without that, even a strong agent has to work from assumptions, and you end up pasting schemas and error logs into the chat to fill in the gaps.

Data Agent Kit fills that gap with two things:

MCP tools: Connections to more than 15 Google Data Cloud services, so your agent can inspect schemas, run queries, read job logs, and manage resources in your live environment.
Google-authored skills: Open-source instructions from Google Cloud engineers that teach your agent data best practices, like optimizing BigQuery SQL, designing Bigtable row keys, and building dbt (data build tool) pipelines.

You can use Data Agent Kit wherever you already work: as an IDE extension for VS Code, Antigravity IDE, Cursor, and other VS Code-compatible editors; as a plugin for Antigravity 2.0, Antigravity CLI, Claude Code, and Codex; or in Cloud Shell and Cloud Workstations, where it comes pre-installed. The IDE extension also brings a lightweight version of the Google Cloud console into your editor, so you can browse data, run queries, and review your agent's work without switching windows.

How it works

Say you ask your agent, "Forecast next month's demand for our top-selling products and check whether we have enough inventory to meet it." Data Agent Kit loads the relevant skills, so the agent follows Google's best practices for the task. It searches Knowledge Catalog to find the sales and inventory tables your team trusts. It then uses MCP tools to run a forecast in BigQuery, check current stock levels in AlloyDB for PostgreSQL, and bring the combined answer back to your editor or terminal. Every step runs with your own IAM permissions.

How Data Agent Kit connects to data.

What you can build

Data Agent Kit covers analytics, operational databases, the Lakehouse, and pipelines. Here's what that looks like in practice, starting with what's new at GA.

Analytics and graph: BigQuery and BigQuery Graph (New in GA)

Graphs are a natural way to explore relationships, like which products people buy together or how suppliers connect to your inventory. Building one usually means hand-writing CREATE PROPERTY GRAPH DDL, learning GQL, and working out which keys actually form edges.

Instead, you describe the graph you want and your agent builds it. The bigquery-graph-author skill maps your tables to nodes and edges, checks each proposed relationship against the actual data, and shows you a plan to approve before creating anything. It can even start from an ER diagram or data model you already have. The bigquery-graph-query skill then writes the GQL, and the graph visualizer in the IDE lets you click through the results.

Building and visualizing a BigQuery property graph.

Operational and real-time databases: Spanner, AlloyDB, Cloud SQL, and Bigtable (New in GA)

Some features have to load instantly, like a personalized feed, a live counter, or a "recently viewed" rail on your storefront. Bigtable is built for exactly that, and it rewards a well-designed row key.

With GA, Bigtable joins Spanner, AlloyDB, and Cloud SQL as a fully supported database in Data Agent Kit. The new bigtable-basics skill designs your schema around how the data will be read and flags hotspots and full table scans before you create anything. Your agent can then create the table and query it with GoogleSQL, with column families flattened into readable columns. In the IDE, you can browse Bigtable instances and tables in the catalog explorer and run queries from the SQL editor.

Agent-guided Bigtable schema design and querying.

Lakehouse and Spark: Managed Service for Apache Spark and Lakehouse for Apache Iceberg (New in GA)

Your agent could already query the Apache Iceberg tables in your Lakehouse through BigQuery. Now it can also work with those same tables using serverless Spark on Managed Service for Apache Spark, with no cluster to manage. Each session keeps its state, so temporary views carry across statements, and you get Iceberg's full feature set, including branching, time travel, and schema evolution.

Your tables don't all have to live on Google Cloud, either. The federate-lakehouse-catalog skill connects your Lakehouse to AWS Glue and Databricks Unity Catalog, so your agent can query that data in place without building an ingestion pipeline first.

Branching and querying Iceberg tables with Managed Service for Apache Spark.

Pipelines and orchestration: dbt, Dataform, and Managed Service for Apache Airflow

Once your logic works, your agent can turn it into a pipeline that runs on its own. It writes dbt or Dataform models, then the gcp-pipeline-orchestration skill schedules them together with your notebooks as an Orchestration Pipeline on Managed Service for Apache Airflow. Pipelines can also include Gemini Enterprise Agent Platform steps, like uploading a model or running batch inference.

In the IDE, you can follow each run on a visual pipeline canvas. If a task needs attention, click Diagnose to hand its logs to your agent. Troubleshooting skills for Airflow and Spark trace the root cause and propose a fix for you to approve.

Troubleshooting a failed Airflow DAG with an agent.

Improving the developer experience

GA also streamlines setup and day-to-day workflows. When you get started, you simply sign in once and select the Google Cloud services you use. Data Agent Kit automatically enables the required APIs, installs the matching skills, and configures your MCP servers with no manual setup files. Inside the IDE, the SQL editor and notebooks now support inline code generation, @ references to tables, and diff views for suggested changes. The extension also shares your active project, open file, and the error from the query you just ran with your agent, so asking it to "fix this query" just works.

We also made the core tools faster and more responsive. New Spark notebooks automatically create and select a Spark Connect runtime, and Spark SQL queries in the editor run in isolated sessions with built-in execution metrics. The catalog explorer now loads faster and includes BigQuery public datasets in the sidebar. Tuned notebook skills help your agent finish notebook tasks more quickly while using fewer tokens.

Ready for the enterprise

Data Agent Kit is included at no additional cost; you pay standard pricing only for the Google Cloud services your agent uses. Skills also steer the agent toward cost-aware query patterns, like checking partition keys and running a dry run before executing a BigQuery query to avoid accidental full-table scans. And because the skills are open source on GitHub, your team can audit them, fork them, or write custom skills for your own internal standards.

For access control, the agent connects as you or as a service account you impersonate, so row- and column-level security policies apply automatically. Admins can also govern MCP access with Identity and Access Management (IAM), screen MCP traffic with Model Armor, and scope agents with VPC Service Controls and Principal Access Boundary policies.

Install and get started

You can set up Data Agent Kit in under a minute in either your IDE or your terminal.

Option 1: Install the IDE Extension

IDE extension: Search for "Google Cloud Data Agent Kit" in the Extensions panel of VS Code, Antigravity IDE, Cursor, or any VS Code-compatible editor. You can also install it from the VS Code Marketplace or Open VSX.
Antigravity 2.0: Go to Settings > Customizations > Build with Google Plugins, then download Data Agent Kit.
Cloud Shell and Cloud Workstations: Already installed by default; just open the editor and sign in.

Option 2: Install the CLI Plugin

Run the command for your preferred coding agent using the official GoogleCloudPlatform/data-agent-kit-plugin repository:

code_block: <ListValue: [StructValue([('code', '# Antigravity CLI\r\nagy plugin install https://github.com/GoogleCloudPlatform/data-agent-kit-plugin\r\n\r\n# Claude Code\r\nclaude plugin install data-agent-kit-starter-pack@claude-plugins-official\r\n\r\n# Codex CLI\r\ncodex plugin marketplace add GoogleCloudPlatform/data-agent-kit-plugin\r\ncodex plugin add dak@dak-marketplace'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe6912852d0>)])]>

Then try a first prompt:

code_block: <ListValue: [StructValue([('code', "What are this week's fastest rising search terms in the US that weren't in the last week's top 10? Use the BigQuery public Google Trends dataset."), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690765150>)])]>

Data Agent Kit plugin running in Claude Code.

Next steps & hands-on resources

Read the docs: Explore the Data Agent Kit documentation and product overview page.
Explore the skills: Browse, star, and contribute on GitHub.
Build an analytics workflow: Try the Analytics with Data Agent Kit and Antigravity IDE codelab, and read the companion blog, Agentic analytics with the Data Agent Kit.
Build a data science pipeline: Try the Fraud detection pipeline with Data Agent Kit and Antigravity IDE codelab.

Graph Workflows in ADK: Everything You Need to Know

Tue, 29 Sep 2026 16:00:00 +0000

Graph engineering is the design work: breaking a task into nodes, connecting them with edges, and deciding where code, models, or people control the next step. The Agent Development Kit (ADK)'s Workflow turns that design into an executable process, with functions and agents doing the work. Through a refund example, this post shows how to run steps in parallel, route decisions, pause for human review, and process a list of cases. It also explains when to declare the paths in a static graph and when to let Python schedule further work as results arrive.

TL;DR: Using a refund workflow in ADK, we'll cover fan-out and fan-in, deterministic and agent routers, human-in-the-loop pauses, parallel workers, and dynamic orchestration— along with when to use a static graph or let Python decide what runs next.

Start with a single agent

We can give one agent the tools and instructions to handle the refund request from start to finish:

code_block: <ListValue: [StructValue([('code', 'refund_agent = Agent(\r\n name="refund_agent", model=MODEL,\r\n tools=[fetch_order, fetch_payment, fetch_history],\r\n instruction="""You handle refund requests.\r\n 1. Look up the order.\r\n 2. Check the payment record.\r\n 3. Check the customer\'s refund history.\r\n 4. Deny if there\'s an open chargeback or it\'s past 30 days.\r\n Approve if it\'s under $50 and they\'ve had fewer than three\r\n refunds this year.\r\n 5. Write the customer an email explaining the decision.""",\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7fe6905e18d0>)])]>

This puts the model in charge of choosing the tools, applying the policy, and writing the reply. But the prompt already describes distinct pieces of work: three lookups, a decision, and a response. Making those pieces separate nodes lets us decide how each should run.

Assume the customer has selected an order and clicked “Request refund.” The app knows the order ID, so the workflow can begin with the lookups.

Let independent steps run together

The order, payment, and refund-history lookups all need the order ID, but none needs another lookup's result. Although the prompt lists them one after another, there is no reason for them to wait for each other. We can run all three in parallel.

The policy decision is different: it needs all three records. So the workflow splits into three paths, then brings their results together before continuing. These two moves are called fan-out and fan-in.

In ADK, the lookup functions can become nodes directly. A nested tuple starts them together, and a JoinNode waits for their results. First, the imports and a lookup signature:

code_block: <ListValue: [StructValue([('code', 'import asyncio\r\nfrom pydantic import BaseModel, Field\r\n\r\nfrom google.adk import Agent, Context, Event, Workflow\r\nfrom google.adk.events import RequestInput\r\nfrom google.adk.workflow import JoinNode, START, node\r\n\r\nasync def fetch_order(node_input: str) -> dict:\r\n ...'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7fe69010add0>)])]>

Each lookup receives the order ID through node_input and returns a dictionary. We can connect them like this:

code_block: <ListValue: [StructValue([('code', 'join_case = JoinNode(name="join_case")\r\n\r\nedges=[\r\n (START, (fetch_order, fetch_payment, fetch_history), join_case),\r\n]'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7fe690108ed0>)])]>

Read this from left to right: start all three lookups, then continue through join_case once they finish.

The join returns a dictionary keyed by node name. The next node can read node_input["fetch_order"], node_input["fetch_payment"], and node_input["fetch_history"] without a model call to collect the results. The JoinNode guide explains the details.

If the payment lookup needed a transaction ID from the order lookup, those two would run in sequence. Dependencies determine the edges, even when the prompt lists every step in order.

You can learn more about fan out and fan in in this video:

Graph Engineering with ADK

One detail matters when turning tools into nodes: the parameter named node_input receives the previous node's output. Other names bind to ctx.state by default, so order_id would look for ctx.state["order_id"] and raise a ValueError if it is missing. Here, the str annotation also converts START's types.Content input to a string.

Route each request to the right workflow

So far, the customer has explicitly requested a refund. In a broader support conversation, we first need to identify what they want and send the request to the right process. “The shoes are the wrong size. Could you send me a different pair?” should go to an exchange workflow, while a request for money back should enter our refund workflow.

That introduces a router, a node that chooses which branch runs next.

A deterministic router follows explicit rules, the same inputs produce the same route.
A nondeterministic router can choose different branches for the same input.
An agent router uses a model to interpret the request, so its choice can vary.

Routers choose among the paths defined by the workflow. In our support example, we can use an agent to identify the customer's intent, then fixed rules to apply the refund policy.

Identify the intent with an agent router

An agent can interpret the customer's message and classify the request. In one ADK pattern, it returns a structured category, then a small function emits the corresponding Event(route=...) to send the request to the chosen workflow:

code_block: <ListValue: [StructValue([('code', 'Customer message → classification agent → route function\r\n ├─ REFUND → refund workflow\r\n ├─ EXCHANGE → exchange workflow\r\n └─ CLARIFY → ask a follow-up question'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690108290>)])]>

The model identifies the intent; the graph defines the available destinations. If the request is unclear, the workflow can ask a follow-up question. ADK's routing sample shows this pattern.

This intent router would sit before our refund workflow. For the selected-order example, the “Request refund” button has already established the intent, so we can enter that workflow directly.

Apply the refund policy with a deterministic router

Inside the refund workflow, the three lookups give us the facts for another decision: approve, deny, or ask a person to review. This time, the policy gives us explicit thresholds, so a function can choose the path:

code_block: <ListValue: [StructValue([('code', 'def refund_policy(amount, days_ago, chargeback_open, priors):\r\n if chargeback_open or days_ago > 30:\r\n return "DENY"\r\n if amount < 50 and priors < 3:\r\n return "AUTO_APPROVE"\r\n return "MANUAL_REVIEW"\r\n\r\ndef route_refund(node_input):\r\n case = {**node_input["fetch_order"], **node_input["fetch_payment"],\r\n **node_input["fetch_history"]}\r\n route = refund_policy(case["amount_usd"], case["placed_days_ago"],\r\n case["chargeback_open"], case["prior_refunds_12mo"])\r\n return Event(output=case, route=route) # the function names the path'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7fe6901096d0>)])]>

route_refund combines the records and returns the case with one of three route names: AUTO_APPROVE, DENY, or MANUAL_REVIEW. Manual review handles cases that meet neither automatic rule.

This is a deterministic router: the same case data produces the same decision, and we can test the policy without calling a model.

	Deterministic router	Agent router
Decisions come best from	Rules in code	A model interpreting the input
Best fit	Known facts and explicit policy	Meaning that is hard to capture in rules
Example	Deny an order older than 30 days	Recognize an exchange request
Model call for routing	None	Required

Both routers choose among defined paths. An agent router can sit inside a static graph: the model's choice varies, while the possible connections stay the same.

For an automatic approval or denial, the next step is to explain the decision to the customer. We give each route a notice agent that writes the reply from the case data. ADK passes the dictionary in Event(output=case, ...) to that agent as a JSON user message:

code_block: <ListValue: [StructValue([('code', 'approve_notice = Agent(\r\n name="approve_notice", model=MODEL,\r\n instruction="Tell the customer their refund is approved and when to expect the "\r\n "money, using the case JSON you receive. Short email, warm, no fluff.",\r\n)\r\ndenial_notice = Agent(\r\n name="denial_notice", model=MODEL,\r\n instruction="Tell the customer their refund was declined and exactly why, based "\r\n "on the case JSON you receive. If it carries a reviewer_note, that is "\r\n "the reason. Short email, direct and kind. Do not invent policy.",\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7fe690109150>)])]>

The refund workflow now has a path from the initial lookups to a decision and a reply, with a third branch for cases that need a person:

code_block: <ListValue: [StructValue([('code', 'workflow = Workflow(\r\n name="refund_decision",\r\n edges=[\r\n (START, (fetch_order, fetch_payment, fetch_history),\r\n join_case, route_refund),\r\n (route_refund, {"AUTO_APPROVE": approve_notice,\r\n "MANUAL_REVIEW": escalate_to_human,\r\n "DENY": denial_notice}),\r\n ],\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7fe69010be50>)])]>

The first chain fetches the records, joins them, and applies the policy. The second maps the router's decision to a destination. Automatic decisions go straight to a notice agent; MANUAL_REVIEW goes to the human-review node we'll define next.

That third branch needs more than another function call. A reviewer may take minutes or days to answer, so the workflow must pause with the case pending and continue once the person decides.

The review node yields RequestInput, which records the pending request and pauses the run. On resume, rerun_on_resume=True runs the node again, with the answer available in ctx.resume_inputs:

code_block: <ListValue: [StructValue([('code', 'REVIEW = "refund:review"\r\n\r\nclass ReviewDecision(BaseModel):\r\n approve: bool = Field(description="True to refund, False to decline.")\r\n note: str = Field("", description="Why, in the reviewer\'s words.")\r\n\r\n@node(rerun_on_resume=True)\r\nasync def escalate_to_human(ctx: Context, node_input: dict):\r\n answer = ctx.resume_inputs.get(REVIEW)\r\n if answer is None: # first pass: ask, then stop\r\n yield RequestInput(\r\n interrupt_id=REVIEW,\r\n message=f"Refund ${node_input[\'amount_usd\']} on order "\r\n f"{node_input[\'order_id\']}?",\r\n payload=node_input, # what the reviewer is shown\r\n response_schema=ReviewDecision,\r\n )\r\n return\r\n # second pass: the answer is here, so route on it\r\n yield Event(output={**node_input, "reviewer_note": answer.get("note", "")},\r\n route="AUTO_APPROVE" if answer["approve"] else "DENY")'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7fe69010bb50>)])]>

The response schema gives the node an approve value to route on and a note to carry forward. If the reviewer declines, the notice agent receives their reason with the case. One more edge connects the review decision to the reply:

code_block: <ListValue: [StructValue([('code', 'edges=[\r\n ...,\r\n (escalate_to_human, {"AUTO_APPROVE": approve_notice,\r\n "DENY": denial_notice}),\r\n]'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7fe69010bdd0>)])]>

ADK ships two variants of this. The one above is the single-node pattern: the node reruns and reads ctx.resume_inputs, as in the request_input_rerun sample. The request_input sample shows the two-node variant, where one node yields RequestInput and the reviewer's answer arrives as the next node's node_input — so there is no ctx.resume_inputs to find in that file. The companion scripts linked below include the code that sends the reviewer's answer back to the run.

A Workflow is also a node. This whole refund process can become one step in a larger customer-service workflow.

Apply the same step to a batch of cases

We now have a process for one refund. Suppose a batch of cases arrives with the records already collected. Each needs the same policy check, and the number of cases changes from batch to batch. We can apply one node to every item using a parallel worker.

In ADK, parallel_worker=True runs a node once per item in an input list and collects the results in the original order. We can reuse our policy function:

code_block: <ListValue: [StructValue([('code', '@node(parallel_worker=True)\r\ndef review_case(node_input):\r\n # Each worker receives one case from the input list.\r\n case = node_input\r\n decision = refund_policy(\r\n case["amount_usd"], case["placed_days_ago"],\r\n case["chargeback_open"], case["prior_refunds_12mo"],\r\n )\r\n return {"order_id": case["order_id"], "decision": decision}\r\n\r\ndef collect_decisions(node_input):\r\n # This node receives the list of worker results.\r\n return {"decisions": node_input}\r\n\r\nbatch_review = Workflow(\r\n name="batch_review",\r\n edges=[(START, review_case, collect_decisions)],\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7fe6901094d0>)])]>

Each worker receives one case, and collect_decisions receives the results as a list. No separate JoinNode is needed. The flag also works on agents—for example, to write an explanation for each case. The parallel_worker sample shows both forms.

The policy calculation here is small. Concurrency is more useful when each item waits on an API or model call, but the way inputs and results move stays the same.

Pattern	Work distributed	Collected output
Fan-out with `JoinNode`	Different notes doing independent jobs	Dictionary keyed by node name
Parallel worker	The same node for every item	List in input order

The batch size can change without changing the graph. A variable amount of work still fits inside a fixed process. See the parallel worker guide for execution details.

Let results shape the next step

Our refund process has known paths, even when a batch contains more cases or a reviewer takes longer to answer. But some work only becomes clear as we investigate.

Suppose a disputed refund reveals a second transaction. Checking it raises a delivery question that needs further investigation. Now each result can create follow-up work. A dynamic node can examine those results and schedule the next checks in Python.

ADK provides ctx.run_node to run another node and await its result. To see how that changes orchestration, let's first express our existing refund flow this way:

code_block: <ListValue: [StructValue([('code', 'HANDLERS = {\r\n "AUTO_APPROVE": approve_notice,\r\n "MANUAL_REVIEW": escalate_to_human,\r\n "DENY": denial_notice,\r\n}\r\n\r\n@node(rerun_on_resume=True)\r\nasync def refund_flow(ctx, node_input):\r\n # Step 1: fetch all three records at once.\r\n order, payment, history = await asyncio.gather(\r\n ctx.run_node(fetch_order, node_input, use_sub_branch=True),\r\n ctx.run_node(fetch_payment, node_input, use_sub_branch=True),\r\n ctx.run_node(fetch_history, node_input, use_sub_branch=True),\r\n )\r\n case = order | payment | history\r\n\r\n # Step 2: apply the refund policy — the same pure function, 0 LLM calls.\r\n decision = refund_policy(\r\n amount=case["amount_usd"],\r\n days_ago=case["placed_days_ago"],\r\n chargeback_open=case["chargeback_open"],\r\n priors=case["prior_refunds_12mo"],\r\n )\r\n\r\n # Step 3: run the chosen handler, and use its output as this node\'s output.\r\n await ctx.run_node(HANDLERS[decision], case, use_as_output=True)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7fe69010a3d0>)])]>

asyncio.gather runs the lookups together. Python combines their results, applies the policy, and runs the selected handler. use_as_output=True makes the handler's result the parent node's output without emitting it twice.

The dynamic version also adjusts the human-review node: after the answer arrives, it calls the chosen notice agent through ctx.run_node. The complete dynamic script (refund_dynamic.py) includes that variation.

The outer graph only needs an entry point:

code_block: <ListValue: [StructValue([('code', 'workflow = Workflow(\r\n name="refund_dynamic",\r\n edges=[(START, refund_flow)],\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7fe69010ab50>)])]>

In the static version, the edge list shows the branches and destinations. In this version, we read refund_flow to see them. The dynamic node guide covers this approach.

The human-review pause still works here. When the answer arrives, refund_flow runs again from the top, but completed ctx.run_node calls return their recorded outputs from session history. The lookups do not repeat, as direct function calls would. Keeping side effects inside child nodes lets completed calls replay their results when the parent resumes. Each child also gets its own trace span, and use_sub_branch=True keeps concurrent children's events on separate branches.

For this fixed refund process, the edge list remains easy to inspect. The Python version gives us a place to add the investigation logic described above: inspect a result, choose a follow-up node, and run independent checks together. A model might suggest what to investigate, while code limits the work—for example, three follow-ups per finding and two levels of investigation before human review.

Loops can still fit in a static graph. A fixed draft → check → revise process can use a conditional back-edge and a router that limits revisions. “Static” describes the possible connections; the actual path and iteration count can vary. The graph guide covers conditional cycles.

You can also place a dynamic node inside a static workflow, using Python for a stage that needs it while keeping the surrounding process visible.

Choose who decides what runs next

Start with a question: can you draw the possible workflow before the input arrives? Include branches, loops, and repeated stages. You do not need to predict the path each request will take.

What you need	Pattern
Independent jobs, then all their results	Fan-out and fan-in
A branch chosen by fixed rules	Deterministic router
A branch chosen by interpreting meaning	Agent router
A person's decision before continuing	`RequestInput`
The same step across a list	Parallel worker
Code that schedules work as results arrive	Dynamic node

Use an edge list when it makes those connections clear. Use dynamic orchestration when results create further work or Python expresses the control more naturally. A small, open-ended task may need only one agent and its tools.

In our refund workflow, the graph coordinates the lookups, code applies the policy, a person handles exceptions, and a model writes the reply. Graph engineering gives each a clear responsibility—and makes it easier to see how the process works.

Get started

Read: the workflow guide.
Explore: the workflow samples.
Build: the hands-on codelab, which applies these patterns to a marathon race-day coach.

Best practices guide for customizing Gemini models via Reinforcement Learning (RL)

Fri, 25 Sep 2026 16:00:00 +0000

Reinforcement learning (RL) has been a keystone of modern LLM post-training, but it demands large training clusters and access to model internals that external customers can't have with proprietary models like Gemini. So here at Google Cloud, we packaged it into a managed RL fine-tuning service (RLFT service) — you bring prompts and a reward function; we handle the infrastructure and the proprietary model internals.

Now, you can adapt Gemini with the service — teaching the model from a reward signal you define, rather than from a fixed set of labeled answers. This unlocks a class of problems that supervised fine-tuning (SFT) struggles with: tasks that are hard to demonstrate but easy to score.

In this guide, we will walk through practical best practices for using RL fine-tuning service. We'll start with a short tour of the RL training loop, how to decide if and when to use RL, and introduce how to get the most value from this approach.

What is RLFT?

RLFT adapts Gemini from a reward signal you define rather than labeled answers. Instead of authoring a large set of gold examples, you write one program that scores a response and the service improves the model against it — unlocking tasks that are hard to demonstrate but easy to verify: you can't hand-write the ideal SQL for every schema, but you can run the query and check the result.

At each training step the service generates multiple candidate responses to your prompts, scores them with your reward, and improves the model so that higher-scoring responses become more likely while it stays close to the original Gemini. The reinforcement learning that makes this work is fully managed — you never configure it. The one thing you own, and the thing that most determines your results, is the reward.

Three properties define what RLFT can and can't do:

It learns from the model's own outputs: It refines what the model already produces rather than copying an external target, so it tends to disturb unrelated capabilities less than SFT.
It rewards outcomes, not paths: Any response that reaches a good result earns reward, which fits open-ended tasks with many valid solutions.
It amplifies existing competence: It makes occasional success reliable, but it can't teach a skill the model never demonstrates.

When to use RLFT

Prompting and SFT handle most adaptation; exhaust them first. RLFT earns its keep when you can grade a response but can't cheaply author it, when SFT has plateaued on the metric that matters (faithfulness, schema validity, tone), or when the task has many equally valid answers a single reference target would wrongly penalize.

SFT and RLFT are complementary, not competing:

Direct RLFT when the base model already succeeds part of the time — enough for the reward to tell better answers from worse ones.
Two-stage SFT → RLFT when you have SFT data or the base success rate is too low for RL to gain traction. Use SFT as a short, cheap warm start — kept light, since over-fitting the demonstrations leaves less room for RL to improve — then continue into RL via Continuous Tuning, which initializes RL from the SFT checkpoint.

Across early adopters, these patterns show where RLFT delivers the most value — each scoring an outcome the business cares about but could never cheaply demonstrate.

Use cases for RLFT

AI-powered NPCs in games

What: In-character, on-brand dialogue held across long, multilingual, multi-turn conversations.
Problem: Off-the-shelf models break immersion — wrong language, hallucinated items, ignored players, repetitive loops.
Objective and reward: A Gemini autorater (LLM-as-a-judge) scores each turn on persona, flow, and game-state syntax, penalizing format and language errors.
Results: Loops and language drift disappeared and state syntax held, making shippable in-game characters viable at scale.

Structured entity extraction

What: Pulling a set of items from unstructured documents, such as supplier invoices and shipping manifests, into structured records automatically.
Problem: The long tail where SFT plateaus — missing required fields (recall) or inventing ones that aren't there (precision).
Objective and reward: A rule-based precision/recall reward forces every field to be grounded in the source text, not imitated from one gold answer.
Results: Field-level accuracy rose on noisy real-world documents where tuning had stalled, turning a manual review step into an automated one.

Content moderation

What: Applying intricate policies and decision trees at scale.
Problem: Models hallucinate false positives or reward-hack with invalid formats to dodge evaluation.
Objective and reward: A Cloud Run reward pairs format validation with a deterministic grader to enforce multi-step policy adherence.
Results: The model handled complex exemption carve-outs, sharply cut false positives, and stopped reward hacking — reducing the human-escalation volume that makes moderation expensive.

Code measured by execution

What: SQL or API calls graded on whether they actually run against customer data.
Problem: SFT mimics one reference query and breaks on unseen proprietary schemas.
Objective & Reward: A code-execution reward runs the code in a secure sandbox and pays out only if it compiles, executes, and returns the correct result.
Results: The model produced first-attempt executable queries at closed-frontier quality and lower inference cost, letting non-technical users query proprietary data in natural language.

Presentation slide generation via HTML

What: Multi-slide decks authored as HTML/CSS.
Problem: Training on text alone is blind to visual quality — overflows, clipped elements, and inconsistent styling slip through unnoticed.
Objective & Reward: A code-execution reward renders the slides and scores visual design, layout integrity, structural completeness, and rubric adherence.
Results: The model emitted modular, well-styled decks with cohesive themes and no layout overflow.

Where to start?

A dataset. A diverse set of prompts with a held-out validation split is enough for a first run — confirm the loop converges and reward moves the right way, then scale. Keep train and eval strictly separated; a contaminated eval hides overfitting.
A reward function. Your task specification as code or configs, and the dominant driver of quality. A good reward correlates with human preference, is robust to malformed output (catch the failed parse and return a clearly negative score rather than crashing), and resists reward hacking — ensemble judges, penalize length, floor degenerate outputs, and prefer a verifiable check over a model's opinion. Validate it offline before launch.

The service handles the rest; start from the defaults, watch reward and eval curves in the console, and take the checkpoint where validation reward saturates rather than the last step.

Get started today

What will you build? The tools are ready and waiting.

Documentation: Reinforcement Learning Fine-Tuning

Agent Factory recap: Agent harnesses, shifting left, and autonomous coding

Thu, 24 Sep 2026 23:00:00 +0000

In this episode of The Agent Factory, we explore the reality of building with autonomous agents alongside Ryan Lopopolo, a software engineer at Google Cloud and the person who coined the term agent harness. From throwing out manual code editors to treating team collaboration like leveling up RPG stats, Ryan breaks down how grounding models in rich context and shifting interventions left unlocks high levels of agent autonomy.

This post guides you through the key ideas from our conversation. Use it to quickly recap topics or dive deeper into specific segments with links and timestamps.

The Agent Harness - What is it?

Timestamp: [00:30]

An AI agent as we're defining it here is a large language model (LLM) plus an agent harness.

Think of the harness as everything wrapped around the LLM that isn't the model itself. For example, if you're working in Google Antigravity using Gemini 3.8 Flash, Gemini Flash is the LLM and Google Antigravity is the harness.

While an unassisted model can answer simple questions out of the box, it can't check live conditions or interact with your workspace on its own. When a user asks a question like "Why is the sky blue?", an unassisted LLM can respond without issue. However, when asked a question like "Should I wear a raincoat today?", the model can't answer on its own because it lacks the necessary data. The harness catches the intent, queries live weather tools, bundles that context back into the prompt, and hands it to the model to produce an informed answer.

Ryan Lopopolo on agent harnesses and autonomous coding

Tilde Thurium sat down with Ryan Lopopolo to discuss what it takes to run fully autonomous coding workflows in production. See the summary below!

Coining the harness and writing zero production code

Timestamp: [02:22]

The term agent harness grew out of Ryan's extensive work on autonomous coding agents, culminating in a February 2026 essay on leveraging coding models in an agent-first world. Ryan shared that he hasn't opened a traditional code editor since May of last year, maintaining that streak through his transition into Google Cloud. In this paradigm, engineers no longer author or review individual lines of syntax; instead, they operate at the level of natural language specifications and inspect the final artifacts, such as pull requests, documents, and spreadsheets. Then they determine whether the end result meets organizational standards.

Harness engineering is the study and the practice of putting a model into an environment where it can succeed. If you don't do that work, you end up doing what I call 'prompt and pray'. Ryan Lopopolo

Context curation and lazy prompting

Timestamp: [03:35]

Upfront harness investment pays off by allowing engineers to become lazy prompters. When the repository contains structured documentation, clear interfaces, and discoverable tools, you do not need to paste walls of text into a prompt box every morning

"I aspire to be an incredibly lazy prompter. If I have done the job to give the model the tools and context it needs to ground itself, I don't need to write a long prompt. It figures it out."

The model uses its harness to pull relevant context, allowing it to navigate large codebases and execute complex tasks without oversight.

Shifting left: engineering best practices as autonomous guardrails

Timestamp: [05:00]

When an agent fails, developers face a whole spectrum of interventions. The most common reflex is to fiddle with the prompt or retry, but that never scales across a team.

"The simplest, smooth-brain, stupidest intervention I can think of is literally just: try my prompt again without changing anything else. But shifting left means moving interventions earlier into the development lifecycle where they are cheapest and automated: from prompts, to repo docs, to linters, to tests, and all the way to upstream evals."

Instead of hoping the model guesses right on the next turn, shifting left embeds standards directly into the environment. Linters, tests, and AGENTS.md files act as durable memory and enforcement of what you think good looks like, making sure the agent stays on the rails without needing constant hand-holding.

Leveraging established tools and determinism

Timestamp: [06:50]

Agents shine when they're handed tools that already mirror patterns heavily represented in pre-training data. Pairing models with standard command-line interfaces moves reasoning into determinism, shifting the burden of context aggregation away from the model and onto reliable tools.

Ryan also shared an environmental design trick for context efficiency: structuring markdown files so link anchors sit directly beneath their corresponding prose blocks rather than inline. This prevents context clutter and mitigates "lost in the middle" retrieval issues. Because Ryan operates exclusively by reviewing end-state artifacts, keeping documentation readable allows him to easily inspect execution runs:

"I want to be able to look at the pull request and review it. If it made a bad decision, I need to know where it went off the rails so I can whack the agent on the head and make sure it does not make that same mistake again."

Long horizons and expanding the agentic loop

Timestamp: [09:45]

The central challenge of harness engineering is ensuring that agents cohere over long time horizons. Because human organizations produce software through iterative refinement rather than single-shot prompts, agent workflows must mirror that cadence. Harness engineering uses tightly scoped, reviewable pull requests to narrow the agent's state space. Stacking these high-confidence changes end-to-end allows supervisors to gradually expand the loop size, building trust until agents can autonomously execute large-scale initiatives, including entire language migrations

Curating agent teams like RPG stats

Timestamp: [11:58]

Rather than divvying up sprint tasks based on individual specialties, having a diverse team contribute to an agent turns it into a central producer of work that carries everyone's strengths. Ryan compared leveling up an agent's capabilities to building out a character sheet:

"[It's like] building out the stats of your RPG character. I get a new person on the team who is a React architect and boom! The attention that they pay is able to bump out the stats in front-end architecture and performance."

With that collective expertise baked into the environment, the agent can autonomously classify incoming work and activate the exact skills it needs on demand, operating as both a backend architect and a front-end specialist.

Accruing leverage in tools, not custom harnesses

Timestamp: [14:38]

For developers wondering whether to build their own custom agent harness, Ryan offered clear advice: don't build one from scratch. Standard harnesses already provide the foundational primitives: file reading, grep search, and command execution. Over-scaffolding an agent with rigid, bespoke frameworks creates technical debt and leads to sunk-cost traps when frontier models advance.

"If you focus all of your efforts on improving quality on tools and context, you can freely adopt the newest models as they come out and you'll be constantly accruing leverage into a bit of the system that will never become obsolete."

Operating Google Cloud and eliminating capability overhang

Timestamp: [16:47]

Discussing his work at Google Cloud, Ryan outlined his motivation to eliminate capability overhang: the delta between what frontier AI models are theoretically capable of and how much useful work is currently extracted in production. Because the cloud functions as a massive, programmable surface, equipping agents with direct interfaces to Google Cloud lets them manage and deploy infrastructure effectively, turning raw model capability into tangible enterprise utility.

Continually updating your priors on AI

Timestamp: [18:13]

The speed of AI development requires engineers and teams to actively unlearn old limitations and constantly reassess what these models can achieve. "It's very important to continually be updating what you think is possible with these lovely tools that we have," Ryan urged. What broke six months ago often runs effortlessly on today's frontier models. Rather than getting locked into rigid workflows, developers should build around the two highly extensible interfaces that will remain relevant across every model upgrade: tools and context.

"Agents will always need context in order to do that last mile adaptation into what you think good is. And as you can continue to... shift it to the left, in terms of increasingly capable tools which act as a form of memory and enforcement of what you think good looks like, you'll continually be amazed as the models are able to do more and more interesting things for you over time."

How To Build A Custom Harness

Next, Billy Jacobson started us off by showing how developers can customize their own agent harnesses for specific tasks.

Under the hood: Why build a custom harness?

Timestamp: [19:44]

Before jumping into code, Billy unpacked why developers should understand the mechanics of a harness rather than treating it like a black box. Recalling advice from an engineering mentor that "You can just use the framework, but a great engineer will really understand the framework", Billy explained that building a harness yourself is the best way to debug what happens when an agent breaks. You can evaluate three core design decisions for every workflow:

Looping: How many iterations should the agent run, and what conditions trigger an exit state?
Tools: What specific tools should the agent access, and when and how should it invoke them?
Memory: How important is conversational and operational memory, and when should it be retrieved or compacted?

Linear Agent Harness: Deterministic Single-Pass Execution

Timestamp: [21:34]

Billy demonstrated a minimalist linear harness designed for deterministic workflows where looping is unnecessary. This pattern is ideal for targeted inspections, file transformations, or single-turn data analyses where you want a high level of determinism and need the agent to perform the exact same execution flow every single time.

Closed-Loop Agent Harness: Iterative Test-Driven Repair

Timestamp: [22:45]

When tasks demand active bug fixing and refactoring, a closed-loop harness provides the iterative reasoning required to reach a verified resolution.

In this demo, Billy showcased an agent that applies an automated code edit to address a failing requirement, and the harness executes the unit test suite against the updated codebase. If the tests fail, the runtime captures standard failure logs and detailed stack traces, feeding those error diagnostics directly back into the agent's working memory. The process repeats continuously until all unit tests pass, backed by a five-iteration ceiling to prevent infinite loops and runaway execution costs.

Guardrail Harness with Google's Agent Development Kit (ADK)

Timestamp: [23:25]

For developers who require custom behavior without rewriting core orchestration plumbing from scratch, Google's Agent Development Kit (ADK) provides scaffolding with automated memory management and execution safeguards.

Billy walked through an example that leverages ADK's native context compaction to summarize older conversational turns, preventing context window bloat during extended debugging runs. Custom interception hooks inspect and filter shell actions before execution, automatically stopping high-risk operations such as recursive file deletions, database drops, or unauthorized remote git pushes. This architecture gives teams fine-grained control over tool execution boundaries while avoiding the maintenance burden of bespoke harness frameworks.

The 3-Layer Agent Dev Stack: Gemini 3.8 Flash, Google Antigravity, and Google Skills

Timestamp: [25:27]

Next up, Smitha Kolan broke down why coding agents do not always require heavier reasoning models, emphasizing that high performance stems from balancing the three layers of the agent stack: Model, Harness, and Knowledge.

"Your coding agent doesn't need a smarter model. It needs a better stack: model, harness, and knowledge. When all three click into place, everything changes."

She then walked through the three tools she's been loving recently, one for each layer of the stack.

Layer 1 | Model | Gemini 3.8 Flash: High-frequency agentic loops run between 20 and 60 sequential hops per task (inspecting files, updating functions, and executing unit tests). Because latency and API costs compound across iterations, a lightweight, responsive model like Gemini 3.8 Flash makes real-time agent loops practical without running up a massive bill.

Layer 2 | Harness | Google Antigravity with /boost: Default Antigravity handles standard navigation and component creation. On top of that, the /boost command spins up an orchestrator that coordinates specialized sub-agents in parallel and concludes with an independent audit pass before modifying files.

Layer 3 | Knowledge | Google Skills Repository: With over 19,000 GitHub stars and 100+ curated domain packages across Google Cloud, Firebase, Flutter, and Maps, this harness-agnostic repository injects precise domain context on demand, preventing agents from guessing cloud configurations

Your turn to build

Building effective coding agents requires moving past the reflex of simply swapping in larger models. As Ryan Lopopolo's philosophy of harness engineering illustrates, true developer leverage is achieved by shifting best practices to the left and investing in rich tools, deterministic verifiers, and well-curated context that survive model upgrades. When combined with fast inference models, structured orchestration harnesses, and modular domain knowledge, agents evolve from conversational novelties into dependable, autonomous engineering partners.

Ready to put it into practice? Explore the tools and resources covered in this episode:

Gemini 3.8 Flash: Fast, low-cost model for multi-hop agent loops.
Google Antigravity (/boost): Orchestrator harness for parallel coding agents and verification.
Google Skills: Modular domain knowledge for Google Cloud, Firebase, and Flutter.
Agent Development Kit (ADK): Custom harness middleware for safety guardrails and memory compaction.
Full episode video: The full interview and live Factory Floor code demos.

Connect with us

Smitha Kolan → LinkedIn | YouTube | X
Luke Schlangen → LinkedIn | website
Ryan Lopopolo → LinkedIn | Hyperbola | X
Tilde Thurium → LinkedIn
Billy Jacobson → LinkedIn | X
Mollie Pettit → LinkedIn | dev.to | X | Bluesky

How Google Cloud Networking Supports Your Fluid Compute Choices for AI Workloads

Thu, 24 Sep 2026 13:00:00 +0000

The availability of resources for AI workloads can be challenging across the industry, especially accelerators. This can slow your AI workload deployment if it’s built around a specific type of accelerator. The concept of fluid compute allows you to design your AI deployment with several options based on available resources that can fit your use case.

In this blog, we will explore how Google Cloud networking supports your AI workloads and considerations that are relevant to your choice of accelerator (GPU or TPU), as the backend networking component configuration is not exactly the same.

The resource options

After deciding the type of work you want to achieve with your AI deployment, another important component is the actual hardware to get this done. In this case, we want to run inference for a private LLM, and the target is the NVIDIA B200 GPU family which is available in the A4 VMs (a4-highgpu-8g).

Now we have identified what we want to get done and a possible compute option, but the challenge is: is this available?

To get access to resources, there are several options which include:

Dynamic Workload Scheduler (Flex-start VM): Queues workloads until all required accelerator nodes are available at the same time, provisioning them together and running non-preemptibly for up to seven days.
Dynamic Workload Scheduler (calendar mode): Enables reserving accelerator capacity 1 to 90 days in advance with guaranteed start and end times, ideal for scheduled pre-training runs and benchmarking.
Future reservations: Guarantees access to committed hardware in a specified zone beginning at a specific future date.
Flex reservations: Offers short-term commitment windows to secure scarce accelerator nodes without multi-year lock-in.
Dynamic node auto-provisioning and ComputeClasses: In Google Kubernetes Engine (GKE), defining multi-family fallback lists within ComputeClasses allows the cluster to automatically attempt provisioning alternative accelerator types if primary pools face regional constraints.
Spot VMs: Delivers surplus compute at substantial discounts for fault-tolerant, checkpointed batch jobs.

Networking your choices

The networking component of the accelerator varies based on your choice, so let's explore four configurations: standard networking, accelerated GPU networking (TCPX/TCPXO and RoCEv2), TPU networking, and Cloud Run.

Standard networking

Supported accelerators: NVIDIA T4 (N1 series), NVIDIA L4 (G2 series), NVIDIA A100 (A2 machine series single-node and multi-node), Cloud TPU v3, and Cloud TPU v5e (single-host/standalone slices).
Architecture: Nodes communicate over the primary Virtual Private Cloud (VPC) network using the Google Virtual NIC (gVNIC) over standard TCP/IP.
Workload fit: Provides straightforward portability across Google Cloud compute environments, supporting distributed data preprocessing, decoupled pipeline stages, independent inference replicas, and computer vision workloads using standard VPC routing and network policies.

Accelerated GPU Networking (TCPX/TCPXO and RoCEv2)

Distributed training and multi-node inference require specialized multi-rail network fabrics to handle massive parameter exchanges and collective communications.

GPUDirect-TCPX and TCPXO Fabrics

Supported accelerators: NVIDIA H100 (A3 High VMs with 4 rails) and NVIDIA H100 Mega (A3 Mega VMs with 8 rails).
Architecture: Uses custom GPUDirect-TCPX (4 dedicated VPCs) and GPUDirect-TCPXO (8 dedicated VPCs) offload engines to achieve high-throughput multi-rail GPU communication over standard Ethernet infrastructure without requiring native RDMA hardware.
Deployment blueprints: These multi-VPC topologies can be deployed in many ways including using pre-built blueprints from the Cluster Toolkit.

RoCEv2 Fabrics (VM and Bare Metal)

Supported accelerators: NVIDIA H200 (A3 Ultra VMs), NVIDIA B200 (A4 VMs), NVIDIA GB200 NVL72 (A4X VMs), and NVIDIA GB300 (A4X Max Bare Metal).
Zonal network profiles: RoCEv2 operates over a dedicated RDMA VPC attached to a specialized zonal network profile: VM instances (A3 Ultra, A4, A4X) use the ZONE-vpc-roce profile, while Bare Metal instances (such as A4X Max) utilize the dedicated ZONE-vpc-roce-metal bare-metal profile.
Rail-aligned fabrics: This dedicated VPC is isolated strictly for GPU communication and contains subnets mapped directly to the accelerator NICs. The backend is rail-aligned, with support for Jumbo Frames (MTU 8896), delivering non-blocking multi-terabit bandwidth with minimal cross-rail interference.
Automated plumbing with GKE Dynamic Resource Allocation Network (DRANET): When deploying these GPUs on GKE, the GKE managed DRANET can be used to automatically provision additional networks and assign drivers that map the RDMA network interfaces to the GPU. These can then be assigned and consumed directly in your workload pods using standard Kubernetes resource claims.
Turnkey deployment: You can deploy this entire end-to-end stack—including RDMA VPCs, MTU tuning, and DRA drivers—using automated blueprints from the Cluster Toolkit.

TPU Networking

Supported accelerators: Cloud TPU v4, Cloud TPU v5p, Cloud TPU v5e (multi-host Pod slices), Cloud TPU v6e (Trillium), and TPU7x (Ironwood).
Inter-chip interconnect (ICI): Inside a TPU Pod or slice, chips communicate directly over dedicated, ultra-low-latency optical links organized in 2D or 3D torus meshes, bypassing traditional network stacks entirely.
Optical circuit switches (OCS): In TPU v4 and TPU v5p SuperPods, software-reconfigurable OCS units dynamically change physical network topologies, route around faulty trays, and provision custom-sized accelerator slices without manual recabling.
Multi-NIC architecture (TPU v6e and Higher): While earlier TPU generations relied on ICI within a slice and single-NIC for host traffic, Cloud TPU v6e (Trillium) and TPU7x introduce a native multi-NIC architecture where worker nodes isolate standard Kubernetes management traffic onto a primary VPC while using secondary dedicated VPCs configured for high-throughput TPU data and cross-slice communication.
DRANET for TPU deployments: When deploying these TPUs on GKE, the GKE managed DRANET can be used to automatically provision additional networks and assign drivers for TPU communication. These can then be assigned and consumed directly in your workload pods using standard Kubernetes resource claims.
Data-center network (DCN) Multislice: For models scaling beyond an individual TPU slice, Cloud TPU Multislice connects multiple independent ICI meshes over Google's high-speed Jupiter Data Center Network utilizing these dedicated multi-NIC paths.

Cloud Run

Supported accelerators: NVIDIA L4 (G2 series) and NVIDIA RTX PRO 6000 (Blackwell) on Cloud Run GPU services.
Direct VPC egress: Binds serverless containers directly to your private VPC network using sub-minute IP allocation via Direct VPC Egress, enabling secure, low-latency access to internal data lakes, databases, and private APIs without traversing the public internet or requiring legacy connector VMs.

Summary

Google Cloud networking options support various accelerator types. When using fluid compute you can adjust your network setup to support the best design to optimise your workloads performance.

Next Steps

Take a deeper dive into Google Cloud AI infrastructure and networking architectures with these resources:

Want to ask a question, find out more, or share a thought? Please connect with me on LinkedIn.

A guide to speeding up your video processing with AlphaEvolve

Wed, 23 Sep 2026 16:00:00 +0000

In real-time streaming, every millisecond counts.

For example, at 30 frames per second (fps), developers have a strict frame budget of just 33.3 ms (and only 16.6 ms at 60 fps) to ingest camera frames, run neural segmentation, apply shaders, and composite output. Exceeding that budget by even a fraction of a millisecond leads to dropped frames and stuttering.

Manual optimization is notoriously tedious — requiring weeks of analyzing flame graphs and hand-tuning low-level code in Swift, C++, or Metal. While standard AI coding assistants can generate boilerplate, they can’t optimize against target hardware, benchmark real-world latency, or ensure optimizations preserve visual fidelity.

Autonomous, closed-loop evolutionary optimization changes this paradigm. Tools like AlphaEvolve pair cloud-scale model reasoning with local hardware execution, and we’re already seeing real-world impact. In partnership with Google, DoIt used AlphaEvolve to autonomously optimize production Swift code in a live macOS streaming app, uncovering performance headroom that manual profiling missed (read the full technical writeup).

While this post focuses on video pipelines, the split-loop pattern applies anywhere performance matters — from microservice throughput and database queries to ML tensor pipelines and embedded systems. In every case, the formula is the same: pair Gemini code generation in the cloud with your domain-specific benchmark harness and automated quality gates.

Today, we’ll show you how to use AlphaEvolve to speed up video processing—and apply these principles to your own performance bottlenecks:

Understanding the split-loop architecture: How AlphaEvolve decouples managed cloud generation (Gemini model ensemble on Google Cloud) from local evaluation (e.g. compiling and timing native Swift/Metal code).
Evaluator craft and quality gates: How to construct scoring functions using metrics like Structural Similarity Index (SSIM) to prevent evolutionary loops from gaming the benchmark (e.g., skipping rendering entirely to go fast).
Autonomous algorithmic discovery: How Gemini-driven evolutionary search can autonomously discover unprompted framework APIs and make intelligent engineering trade-offs (e.g., frame-caching limits).
Setting realistic performance boundaries: How to measure code optimization against physical hardware floors.

1. Understanding AlphaEvolve’s split-loop architecture

AlphaEvolve runs a closed-loop evolutionary process: given a seed program and a custom scoring function, a mixture of Gemini models proposes code variations, executes the scoring function against each candidate, keeps the highest-performing code, and iteratively climbs toward an optimal solution over multiple generations.

A core architectural advantage of AlphaEvolve is its clean separation into two halves:

The generation half (Google Cloud managed service): Contains the prompt sampler, Gemini model ensemble, and program database. Google Cloud handles the scale, prompt orchestration, and generation mechanics.
The evaluation half (customer managed compute): Scoring code quality is strictly domain-specific. You own the evaluator module entirely, running it on your own hardware or target architecture (in this case, macOS running native Swift code).

While AlphaEvolve is Python-first on the cloud generation side, evaluation can be written in any language. The custom evaluator compiles each Swift candidate using swift and executes it against a standard reference webcam clip.

2. Evaluator craft and quality gates

An automated optimization loop like AlphaEvolve never actually "sees" your video stream. It only sees the numeric fitness score your evaluator returns. If your evaluation metric has a blind spot, evolutionary code generation will aggressively exploit it.

In our early runs, a naive fitness score weighted toward raw latency produced an astonishing speedup: the model simply bypassed blur rendering entirely and returned unmodified frames in 0 ms.

Structural Similarity Index Measure (SSIM):

To prevent the model from gaming your benchmark, try building a two-tiered scoring function that pairs throughput with structural fidelity metrics like Structural Similarity Index (SSIM):

code_block: <ListValue: [StructValue([('code', 'speedup = baseline_ms_per_frame / candidate_ms_per_frame\r\nssim = mean_ssim_vs_golden\r\n\r\n#Disqualify any candidate falling below visual threshold\r\n\r\n\r\nif ssim < 0.98 or worst_frame_ssim < 0.95:\r\n return {"speedup": -1e12} # Disqualified\r\n\r\nreturn {"speedup": speedup, "ssim": ssim}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690fa4dd0>)])]>

What does this give you?

The ability to test against worst-case clips: Never benchmark on static frames or blank cameras. Candidate code can easily pass an average SSIM gate on static backgrounds while failing completely during quick head turns.
You can track the minimum, not just the mean: Enforce both an average threshold and a per-frame floor to catch dropped frames or delayed mask updates.

Autonomous algorithmic discovery:

Most developers use generative AI for local micro-optimizations (e.g., inlining helper functions, unrolling loops, or tweaking memory pools). But when given architectural room, the evolutionary loop can discover systemic optimizations on its own.

Engineering lessons:

Provide framework context, not isolated loops: Include public SDK headers, interface definitions, or API reference symbols in the prompt or retrieval harness. An LLM cannot adopt a sequence-aware subsystem if its context window only contains an isolated frame-processing callback.
Expose multi-frame lifecycle hooks: Let your candidate code maintain a bounded state across executions (e.g., historical masks or cache timestamps) rather than enforcing pure, stateless functions.
Let quality gates police the trade-offs: When AlphaEvolve introduced temporal mask caching, it initially cached masks too aggressively, causing noticeable trailing artifacts. Because our SSIM gate penalized drift during motion, the search converged on a production-ready cache window without manual parameter tuning.

Setting realistic performance boundaries

A common pitfall in performance engineering is optimizing in the dark. If you achieve a 2x speedup, is that an incredible achievement, or did you leave another 3x on the table?

In real-time media, total frame time splits into two distinct categories:

Mutable software overhead: Memory allocations, buffer format conversions, thread context switches, and API dispatch friction.
Immutable hardware floors: Raw Neural Engine inference latency, GPU shader compute time, and hardware display synchronization.

To make the most of AlphaEvolve, developers should measure against theoretical maximum headroom

Before running optimization loops, here’s a few principles to keep in mind:

Build a "no-op" pipeline: Strip out Swift/C++ orchestration, data marshalling, and frame conversions. Dispatch only the pre-warmed ML model and bare GPU pass on a dummy buffer. The resulting time is your physical hardware lower bound.
Calculate your addressable ceiling: Your total possible optimization potential is:

3. Score against the hardware gap: Instead of arbitrary speedup multiples, measure optimization efficiency:

Get started

All benchmark code, test clips, evaluation scripts, and raw candidate logs are open source:

GitHub repository: AlphaEvolve Camera Background Blur Example
Detailed technical write-up of our case study with DoIt: Running AlphaEvolve on Your Own Code

The DevFest Community Workshop Experience: Building Real Agents Together

Fri, 18 Sep 2026 16:00:00 +0000

This week we kicked off the DevFest season in North America at Google Hudson Square in New York City with 80 engineers packed into the room. Typical technical workshops hand you a finished repo, tell you to blindly paste blocks of code into your terminal, and hope nothing crashes. You walk away with green checkmarks, but your brain stays on autopilot.

We've introduced a completely different experience called Workbench.

Workbench focuses on understanding core ideas and architectural models rather than obsessing over syntax and code snippets. Instead of getting bogged down in boilerplate, engineers spent the day grappling with the actual mental models behind graph engineering, self-evolving architectures, and automated self-patching harnesses.

A glimpse into the Workshop Experience

At the DevFest Community Workshop, we spent one intense day building long-running, self-evolving multi-agent systems powered by Google's agentic stack. Ricky Robinett, Senior Director of Developer Marketing, kicked off the day by diagnosing why so many engineering teams hit a wall with agents. Ricky broke down why prompt engineering fails as a safety mechanism: English is just a probabilistic suggestion, not an execution boundary.

Right after Ricky, Rachel Francois, Google Developer Groups (GDG) North America Program Lead, took the stage alongside GDG Brooklyn organizers to welcome the community and spotlight the power of local developer chapters. They set the tone for the entire day, reminding everyone that building durable software works best as a team sport where engineers share real-world patterns and build local networks that outlast any single framework.

Getting hands on with labs

Annie Wang & Christina Lin, Americas DevRel Team members, led the morning lab that put those runtime ideas to work. Attendees explored Google's Agent Development Kit (ADK), Veo 3.1, Memory Bank on Gemini Enterprise Agent Platform, and RAG Engine on Gemini Enterprise Agent Platform. Through Workbench, developers grasped the principle of separating state from active compute for long running tasks. Workflows paused cleanly mid-execution, waited out asynchronous human approvals, and resumed without running up idle compute costs.

After lunch, Logan Hennessy, Americas Developer Relations Engineer (DRE), and Kartik Derasari, Google Developer Expert (GDE), led a lab using auction history as insight for better bidding strategy. Attendees worked through the architecture by integrating BigQuery data into autonomous data engineering pipelines, reasoning about deterministic bidding logic and adding eval-gated, self-patching harnesses that catch spend anomalies and update runtime execution safely.

Between lab blocks, we ran fast-paced speed quizzes where developers raced to lock in their answers as quickly as possible. Screens flashed, fingers flew across keyboards, and seconds made the difference between topping the leaderboard or dropping five spots. Nothing beats watching a room full of serious engineers completely lose their cool over a live quiz leaderboard.

Join a DevFest Community Workshop this fall

New York was only round one. We are taking this exact experience on tour to five more cities this fall. Find your city and grab your seat before spots fill up:

Sunnyvale on September 30
Washington DC on October 6
Atlanta on October 30 (as a part of DevFest Atlanta)
Seattle on November 4
Boston on November 10

Best practices for handling cloud reliability incidents

Tue, 15 Sep 2026 16:00:00 +0000

Cloud outages can range from global service disruptions to issues isolated to a specific region, zone, or even just your project, workload or application. If you suspect a Google Cloud Platform outage is impacting your services, we recommend you follow a structured “Verify→ Investigate→Report→Resolve→Review" workflow to resolve it. And before that outage occurs, you should also have prepared your environment for an eventual disruption by designing for failure, and actively practicing the steps you need to take to restore service.

In this blog, we summarize the key reliability incident handling best practices to help you design and practice your reliability incident response capabilities and minimize impact. Rather than an exhaustive guide, this is meant as a primer on only the most important practices for advisory purposes. Please note that we do not cover additional practices specific to security incidents here.

Beyond the base steps covered here, you may want to also explore how AI agents and tools are starting to transform incident handling. Check out this episode of the Prodcast, where Googlers explore the latest trends of leveraging agentic AI in Site Reliability Engineering (SRE) to detect issues early and prevent disruptions. Try Cloud Assist investigations, or explore Agent Skills and remote managed MCP servers to give you another set of tools for quickly pinpointing an issue. Before getting into these advanced techniques, we focus below on the foundational steps to good incident handling.

1. Prepare

Long before things start to go sideways, you should have spent significant time preparing for an outage along at least four dimensions: design, data, playbooks and training.

Design: Think ahead and mitigate future incidents by designing automated response actions, like a load balancer shifting traffic away from slow or unresponsive instances, or by automating as much of your incident response playbook as possible. Review designs of all critical applications to automate as many actions as possible to accelerate response and recovery.
Data: When a disruption occurs, having meaningful data at your fingertips vastly improves response capabilities. Use Cloud Logging, Cloud Trace and Cloud Monitoring, or other third-party observability tools, and replicate that data to a redundant stack in a separate location from the systems being observed. Make sure, in advance of any incident, that time stamps are synced across your observability streams for easy correlation, or know how to do that on-demand during an outage, when time is of the essence.
Playbook: A well-thought-out playbook documenting your incident response processes, including crystal clear role and responsibility definitions for all personas, is paramount to efficient incident response. Who is responsible to do what? Who needs to be notified or mobilized for each type of disruption? How can they be reached? What tools and data are available? How are results communicated? How do teams hand over to the next shift during long running incidents? etc. Conduct a simulated incident response and critically review every step to find where your playbook needs clarification. Without clear responsibilities, mitigation inevitably takes longer.
Training: Hopefully, service disruptions are rare events. To ensure your staff knows and remembers how to react, they need to retrain on the process several times per year by running simulated cross-team incident response drills. A retrospective on the simulated exercise will help identify warranted improvements.

2. Verify

Despite your best efforts, sooner or later, a service disruption will occur, which you can detect via any number of mechanisms:

Observability tools (Google tools or third-party tools)
Unified Maintenance Management notifications for planned maintenance
Personalized Service Health notifications managed with alert policies
Proactive customer monitoring by Google

Now, you need to determine what broke and who should ultimately fix the problem:

Google, e.g., a bug, code roll-out, hardware failure, etc.
You, e.g., a configuration change, elevated load, quota ceiling, etc.
Third party, e.g., a directory hosted by a different cloud provider

If Google has declared an incident and started working to fix the problem, estimate whether you can possibly reestablish service sooner, for example by failing over to a secondary stack (see the ‘Typical Causes’ table below). You can determine whether Google has declared an incident and will provide a fix by consulting:

Personalized Service Health: Check this first. Personalized Service Health shows incidents specifically relevant to your projects and regions, distinguishing between incident types:.
- Emerging Incidents: Google has received an alert, on-callers are investigating, impact is yet unknown
- Confirmed Incidents: Google has investigated and found customers are impacted

Located within the Google Cloud console, Personalized Service Health often displays limited-scope incidents that don't appear on the public dashboard. Personalized Service Health also offers a mobile client for Android and iOS smartphones, assuming you can use your work ID and credentials on the phone.

Gemini Cloud Assist, which is integrated with Personalized Service Health, so you can use it to query that information in natural language.
Cloud Service Health dashboard: This is the public-facing non-authenticated web page for broad, severe incidents affecting many customers. Limited blast radius disruptions are not externalized to the public. All its content is available in Personalized Service Health as well. If ever Personalized Service Health goes down, Cloud Service Health serves as an alternative channel built on a separate infrastructure.
Known Issues: In the console, navigate to Support > Cases, view a case, and use the resource selector on the console toolbar to find the specific cloud resource you’re interested in. Then click Known issues. If your issue matches one listed here, you can link a support case to it, so you will receive automatic updates in your case record. If you don’t find a match, open a new support case. Google will automatically match the case to a related incident, as soon as one is declared.
Google declared incidents are updated as new information becomes available, so check back regularly, or set up a Personalized Service Health alert policy to be notified each time new information becomes available.

If you host cloud resources in multiple clouds, a good practice is to check early on whether the problem occurs for multiple cloud providers. If so, the problem is likely external to the providers and caused either by you or by a third-party service that your application interacts with.

3. Investigate

To determine the blast radius within your cloud footprint of Google-declared reliability incidents, first check Personalized Service Health updates for a description of the technical problem. Knowing what to look for will allow you to map your blast radius and decide on suitable contingency actions quicker.

If Google hasn’t declared an incident, try to rule out configuration errors or issues within your environment by checking:

Cloud Monitoring: Look for spikes in error rates (e.g. 5xx errors), increased latency, or drops in traffic in your dashboards.
Cloud Logs: Use Log Explorer to look for specific error messages like DEADLINE_EXCEEDED, SERVICE_UNAVAILABLE, or specific API errors.
Quotas: Ensure you haven't hit a project quota (e.g., CPU, API rate limits), which can often mimic the behavior of an outage.
Change history: Check your log of recently applied changes. Not all problems manifest immediately, but proximity on a timeline can be a powerful indicator of causality, even if it’s not proof. Also check whether Google rolled out any updates just before the symptoms started. See the Unified Maintenance Management interface in Cloud Hub.

Absent a clear culprit, such as a traffic spike or a DDOS attack, and if symptoms manifested immediately after rolling out a change, a good strategy is to back out that change and attempt to return to a last known good configuration.

4. Report

If the Cloud Service Health and Personalized Service Health dashboards are green but your metrics show a failure, you must report it to Google.

Determine priority:

P1 (Critical): Your production service is unusable or severely impacted with no workaround.
P2 (High): Significant impact or degradation, but a workaround may exist.
See guidance on setting priority and guidance on describing your issue.

File a case: Go to Support > Cases > Create Case in the console.

Explain quantifiable business impact to rationalize the submitted priority and prevent it from being reset when Cloud Support prioritizes cases. A clear and accurate rationale helps!

Essential information to include:

Project ID and affected region/zone
Timestamps (when it started and if it's ongoing) with a clearly labeled timezone
Specific error messages or log snippets
Scope: Is it affecting all users/systems, or a specific subset/location?

Escalation for Premium/Enhanced support

If you have a Premium or Enhanced support plan and a P1 case is not receiving the attention it requires, use the Escalate button within the support case in the console. This alerts a support manager to investigate and rectify the situation.

5. Resolve

By taking these steps, you are well on your way to resolving the outage. In the meantime, here are some ways to mitigate the impact of the outage and communicate with impacted stakeholders.

While waiting for a resolution:

Communicate: Notify your stakeholders and customers. Transparency helps manage expectations and reduces duplicate internal reports.
Fail over: If you have a multi-regional architecture, consider shifting traffic to a healthy region. As a best practice, first ensure that the disruption is at the infrastructure level and not at your workload level.
Check for workarounds: While working on a permanent fix, Google often posts temporary workarounds in the Service Health Dashboard updates, or in Personalized Service Health updates.
Consider your regulatory reporting requirements: Know whether your organization is subject to regulatory reporting requirements, and what the required deadlines are for both initial and follow-up reporting. Google Cloud prepares Incident Reports for incidents that meet certain criteria — see details here for how to get those reports. Premium Support customers can also request an Incident Summary, which is an Incident Report customized to your account’s specific hosting location, time stamps, etc.

De-escalation and closure

Once systems are stable, Google downgrades the severity levels and deactivates the active on-call escalation chain. Google only closes an incident in Personalized Service Health when it has taken all the mitigation steps covering all impacted customers. Your specific services might be restored sooner than the incident closure time, if other customers are restored later than you. The incident is officially closed on the Google Cloud Status Dashboard when systems have run stably for a designated auto-close duration. Verify that your services are operating normally at this point. And if your incident responders aren’t compensated for extra time spent on the incident, find a way to thank them.

6. Review

After the problem has been fixed and operations have returned to a normal, steady state, it’s time to conduct a post-mortem analysis to identify how your team can respond better in future service disruptions. A “blameless” approach is essential to surfacing meaningful and impactful improvements that can be made to your incident response process. Ask questions like:

What went well?
What could we have done better?
Where did we get lucky?
Where did we get unlucky?

Then decide what changes can be made to improve your playbook, tools and training.

At Google, we often publish a post-mortem or Incident Report for major outages, available via Personalized Service Health. Review this to understand the root cause and adjust your own disaster recovery plans to prevent or reduce future impact. Customers with a Premium Support plan can request an Incident Summary for a Google-caused incident they were impacted by and for which they opened a P1 case. An Incident Summary is an Incident Report customized for your environment (e.g., start and end times of impact).

Typical causes, comms and prevention strategies

To help you prepare and plan ahead, here’s an overview of some typical incidents based on the symptoms reported in Cloud Service Health and Personalized Service Health along with guidance on what Google communications to expect, and some generic mitigation or prevention strategies you can build into your playbooks.

Blast radius	Typical cause	Comms	Strategy
Single zone or region.Subset of products.	Typical of a software problem triggered by a rollout. Learning points: - Understand the location scope (zones and regions) of your workload - Products can depend on other products	Major incidents are communicated via Cloud Service Health.Major and Minor (by number of customers, not severity) incidents are communicated via Personalized Service Health. Highly localized incidents are not communicated via Cloud Service Health or Personalized Service Health.	Fail over, if so configured, but verify the health of the secondary stack first.
Single zone.Most or all products.	Typical of a power or cooling issue.	Check Cloud Service Health and Personalized Service Health.	Fail over to a different zone, if so configured.
Single region. Most or all products.	Typical of a backbone networking infrastructure issue	Check Cloud Service Health and Personalized Service Health.	Fail over to a different region, if so configured.
Control plane issue for a product	Typical of a late detected issue	Communicated via Personalized Service Health if significant customer impact is verified.	Look for workarounds. Wait for Google to fix. Fail over, if so configured.
Multi-regional issue with a global product	Rare but possible, typically detected quickly. Learnings: Mitigation options can be limited. Try regional variants, alternative products with similar functionality	Check Cloud Service Health and Personalized Service Health.	Wait for Google to fix. In the meantime, verify via Google Comms and your own investigation that this is truly Google’s problem to fix.
Capacity / Stockout issue	System-level demand exceeding capacity in the product/location/model. (Cloud is designed to scale, but limits always exist, so proper planning is advised)	Error message. No incident will be declared.	Place reservations for predicted capacity needs (if cost is acceptable). Flexibility in zone placement can also help.
Quota exhaustion	Difficult / inaccurate prediction of traffic	Error message. No incident will be declared.	Review consumption trends against ceiling regularly.

Go deeper

This document offers only a condensed summary of key points. If you have an active Premium Support contract with Google Cloud, reach out to your account team for a deeper review of your response plans. For a comprehensive treatise on how to build reliable services and how to respond to incidents, we strongly recommend Google’s SRE Book, which is available as a free download. A new version of the SRE book is releasing ~Oct 2026 and will be available for purchase on O’Reilly Media. We’re also working on a future primer that explores AI-supported incident handling in-depth — stay tuned!

Introducing the Google Cloud Developer Plugin for AI Coding Agents

Thu, 10 Sep 2026 10:53:00 +0000

Agent skills fit well alongside documentation and remote MCP servers as ways of enabling the success of your AI workflows. They reduce context window usage for certain use cases, and they're straightforward to install. However, you might have noticed that managing individual skills can be unwieldy, or that some skills are most useful when they act alongside other skills or MCP servers toward the same goal. That's where plugins come in to help.

Today, we're thrilled to announce a new Google Cloud plugin for AI coding agents! Designed as installable bundles, agent plugins equip the AI agent of your choice with skills and tools to be more effective on Google Cloud.

Solving the tool coupling problem

As you expand your usage of coding agents, you might find that they become significantly more capable when they use related skills in tandem or with complementary context and tooling. For example, an agent analyzing infrastructure is more effective when combining domain knowledge, workflow recommendations, and the ability to interact with a live environment together.

Plugins solve this coupling challenge by packaging related capabilities into cohesive, installable bundles. This allows you to take advantage of both broad foundational capabilities and deep, product-specific tools without managing complex dependencies. In the Google Agent Skills repository, you'll start to see the following for Google Cloud popping up over time:

Foundational plugins: Essential platform-wide guidance for things like documentation discovery, project configuration, and architectural design.
Domain-focused plugins: Specialized knowledge and best practices for technical areas in the context of Google Cloud.

For this release, we've started with a foundational plugin that supports agent functionality for all Google Cloud users, focusing on making it easier for agents to retrieve Google Cloud-related skills, make use of official documentation, and handle programmatic interactions with Google Cloud.

Built on an open standard

We've also built our plugin in compliance with the Agent Plugins specification, an open, vendor-neutral standard for packaging Agent Skills and Model Context Protocol (MCP) servers into portable, interoperable units. Rather than requiring developers to maintain different configurations and wrappers for every AI assistant, the Agent Plugins standard provides a unified manifest and directory structure.

Our Google Cloud plugin adopts this standard to ensure that developers across a variety of AI coding environments get consistent, high-quality access to tools that help them succeed with Google Cloud. That includes not only the plugins we talk about today, but all other plugins published to the Google Agent Skills repository as well.

Let's take a look at the flagship plugin that we've just published in the Google Agent Skills repository: google-cloud-developer. This plugin exists to help agents successfully navigate the fundamentals of interacting with Google Cloud: things like authentication, authorization, managing projects, and guardrails for gcloud CLI operations. This plugin also bundles configuration for the Developer Knowledge MCP server, which gives agents up-to-date grounding in Google's official developer documentation.

Plugin in action: Project onboarding and identity authentication

To see how this plugin works, consider a situation where you're bootstrapping a new project as part of working on a script. With the google-cloud-developer plugin installed, you can prompt your agent:

I'm brand new to this platform, and I need to get an account and a first project with billing set up. Then, I need my local machine authenticated so a script that I'm writing can call the APIs as a service identity instead of as me.

Environment awareness: The agent silently runs background checks against your live environment for prerequisites like CLI availability and potential existing projects or organizations.
Review: The agent considers IAM best practices to avoid risks that might be assumed as part of the prompt, like accidental key leaks or git commits.
Interaction with guardrails: The agent outlines a workflow roadmap and offers to act on those steps before modifying any resources.

Installing Google Cloud plugins

Because Google Cloud plugins are available from the open Google Agent Skills repository and adhere to the standard Agent Plugins layout, adding them to your environment is straightforward. For example, here's how you'd install the google-cloud-developer plugin:

Antigravity

Install the Google Cloud Developer plugin directly from the Marketplace in Antigravity.

Alternatively, you can install the plugin directly using the Antigravity CLI:

code_block: <ListValue: [StructValue([('code', 'agy plugin install google-cloud-developer'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690322890>)])]>

Claude Code

Add the Google plugins marketplace, then install the plugin:

code_block: <ListValue: [StructValue([('code', 'claude plugin marketplace add google/skills\r\nclaude plugin install google-cloud-developer@google-plugins'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe6903205d0>)])]>

Enable the Developer Knowledge API in your Google Cloud project by using the gcloud CLI:

code_block: <ListValue: [StructValue([('code', 'gcloud services enable developerknowledge.googleapis.com --project=<YOUR_PROJECT_ID>'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690322a10>)])]>

Create an API key for the Developer Knowledge API by following the instructions Create and secure the API key. Save the key you create in a secure location.

Export the API key to your environment before starting Claude Code:

code_block: <ListValue: [StructValue([('code', 'export DEVELOPERKNOWLEDGE_API_KEY=<YOUR_API_KEY>'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690320310>)])]>

Note: The DEVELOPERKNOWLEDGE_API_KEY environment variable needs to be set in the environment before you start Claude Code. Consider adding this export to your shell's startup script (e.g. .bashrc, .zshrc) for convenience.

Codex CLI

Add the Google plugins marketplace, then install the plugin:

code_block: <ListValue: [StructValue([('code', 'codex plugin marketplace add google/skills\r\ncodex plugin add google-cloud-developer@google-plugins'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690323f10>)])]>

Enable the Developer Knowledge API in your Google Cloud project by using the gcloud CLI:

code_block: <ListValue: [StructValue([('code', 'gcloud services enable developerknowledge.googleapis.com --project=<YOUR_PROJECT_ID>'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690323490>)])]>

Create an API key for the Developer Knowledge API by following the instructions Create and secure the API key. Save the key you create in a secure location.

Enable authenticated access to the Developer Knowledge MCP server by updating ~/.codex/config.toml (or your project's .codex/config.toml) to include the following lines:

code_block: <ListValue: [StructValue([('code', '[mcp_servers.developer-knowledge]\r\n url = "https://developerknowledge.googleapis.com/mcp"\r\n env_http_headers = { "X-Goog-Api-Key" = "DEVELOPERKNOWLEDGE_API_KEY" }'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe6903219d0>)])]>

Export the API key to your environment before starting Codex:

code_block: <ListValue: [StructValue([('code', 'export DEVELOPERKNOWLEDGE_API_KEY=<YOUR_API_KEY>'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe690323bd0>)])]>

Note: The DEVELOPERKNOWLEDGE_API_KEY environment variable needs to be set in the environment before you start Codex. Consider adding this export to your shell's startup script (e.g. .bashrc, .zshrc) for convenience.

Note: After performing this configuration update, a status of not logged in for the Developer Knowledge MCP server is expected and doesn't block access to the server.

Next Steps

If you're already a Google Cloud user, try the above installation steps to set up your agent for success. We think you'll like what you see! For those who want a more guided approach, our new codelab will walk you through the installation and initial exploration of the plugin in Antigravity.

If you're new to Google Cloud, you can also get started with instructions in our documentation to set yourself up for local development.

The most curious readers can also take a deeper look at the plugins and agent skills available to use today in the Google Agent Skills repository.

Developers & Practitioners

Claude Opus 5.5 and Claude Sonnet 5.5 are now available in Google Antigravity

Multi-model engineering patterns in Antigravity

Pattern 1: Cross-model verification ("LLM-as-a-Judge") in loop and graph workflows

Pattern 2: Pairing reasoning models with Flash subagents

Getting started

Developer’s guide to implementing MCP apps in Gemini

Why we need MCP Apps

Building MCP Apps in Gemini delivers six benefits:

High-level architecture

Authentication with two independent layers

Build, deploy on Google Cloud, and register in Gemini Enterprise

Build your own

Four evaluation engineering lessons learned from building agent plugins

1. Grade the execution steps instead of the summary

2. Design evals for asynchronous operations

3. Test live runs and "dry runs" side-by-side

4. Use “golden sets” as durable baselines

Where to go from here

One Agent Development Kit, five languages

A shared foundation

Starting agent-first

Hands-on view of each ADK

Python

Go

TypeScript

Java

Kotlin

Choice without compromise

10 Years of the SRE book: Announcing the second edition of "Site Reliability Engineering”

Why update the book?

A decade of evolution: How SRE practices have progressed

SRE in the age of AI

What hasn’t changed? The importance of culture

Announcing MCP Toolbox Java SDK v1.0: Agentic data access for the enterprise

MCP: The universal interface

What's new in Java SDK v1.0: Built for production workloads

Get started with the Java SDK v1.0

Real-world example: The autonomous transit concierge

The foundation: AlloyDB schema with native embeddings

Mapping intents to SQL: The tools.yaml

Stateful agent architecture in Spring Boot

Connecting the dots: Listing, invoking, and executing tools in Java v1.0

Secure by default: authentication and deployment

Application Default Credentials (ADC) & safety

Deploying the fleet to Cloud Run

Get started today

Step 1: Add the Dependency

Step 2: Explore Resources & Demos

Gradle implementation

Automatic version tracking

Networking for AI inference model serving - GKE only and for all other backends

The entry point

Common services in the designs

Design pattern serving on GKE only

The traffic flow GKE example

Design pattern serving on all backends

All-backends traffic flow example

What's next

GKE CPU startup boost: Accelerate app starts without over-provisioning

Why modern applications need extra CPU at boot time

How CPU startup boost can help

Under the hood: Kubernetes In-place Pod Resize

How CPU startup boost works (pod lifecycle overview)

Prerequisites and availability

Getting started: Configuring CPU startup boost

Example 1: Pod-level boost with fixed steady-state (updateMode: "Off")

Example 2: Combining startup boost with continuous VPA auto-scaling

Example 3: Granular container-level boost

Verifying startup boost in your cluster

Best practices for production workloads

Get started today

Introducing the Server Side Cloud Swift SDK

The rise of server-side Swift and cloud-native concurrency

Where to use the Swift SDK

Getting started with your IDE and packages

Networking, transport, and authentication

The autogenerated client ecosystem

Where to go next

Democratizing Managed Lustre with lower cost and frictionless development

Mapping intents to SQL: The `tools.yaml`

Example 1: Pod-level boost with fixed steady-state (`updateMode: "Off"`)