Application Development

How to build production-ready AI agents with Google-managed MCP servers

Fri, 27 Mar 2026 16:00:00 +0000

As developers build AI agents with more sophisticated reasoning systems, they require higher-quality fuel–in the form of enterprise data and specialized tools–to drive real business value. To get the most out of that octane-rich mix, we offer Google-managed model context protocol (MCP) servers: an engine purpose-built for AI agents to interact securely with Google and Google Cloud services.

These Google-hosted, fully-managed endpoints allow AI agents to communicate with Google Maps, BigQuery, Google Kubernetes Engine, Cloud Run, and many other Google services. As we boldly build AI agents, ensuring that we’re also building responsibly is critical.

In this guide, we demonstrate how to build agents securely on our managed MCP servers.

Why you should use Google-managed MCP servers

Transitioning from local experimentation to enterprise-grade AI requires adopting a robust, managed infrastructure that prioritizes scale and oversight. These are the key benefits that we offer:

Production readiness: While open-source MCP servers are great for local development, they struggle in production with scalability, single points of failure, and management overhead. Google’s managed MCP servers require no infrastructure provisioning because we handle the hosting, scaling, and security.
Unified discoverability: You can publicly query and easily discover all available MCP endpoints for Google services (such as maps.googleapis.com/mcp) using a simple directory service.
Enterprise security: Google MCP servers offer native integrations with the Google Cloud security stack, including Cloud IAM, VPC-SC and Model Armor.
Integrated observability and auditability: Google MCP servers are integrated with Cloud Audit Logs, offering a centralized view of all tool-calling activity. This allows platform teams to monitor agent performance, ensure compliance, and troubleshoot interactions through a single enterprise-grade logging pane.

Figure 1: Google MCP Servers high-level architecture diagram

An AI agent example using Google MCP server with ADK

Cityscape is a demo agent built with Google's Application Development Kit (ADK) that turns a simple text prompt — like "Generate a cityscape for Kyoto" — into a unique, AI-generated city image. It uses the Google Maps Grounding Lite-managed MCP server for trusted location information and the Nano Banana model (via a local MCP server) for image generation.

The lightweight app is then easily deployed to Google Cloud Run, a serverless runtime, to interact with users. Below are two examples of the images generated by the agent based on the local real-time weather conditions.

Figure 2: Example images generated by the Cityscape agent with real time weather info

1. Calling a Google MCP server from the ADK agent:

As demonstrated in the get_weather code snippet below, the Cityscape agent utilizes a Streamable HTTP endpoint to interface with the Google Maps MCP server. It provides the agent with real-time weather conditions for a given city, which are then used to set the atmospheric mood in the generated cityscape image.

Because it's a Google-managed remote MCP server, Google handles the hosting, scaling, and security — so your agent benefits from automatic scaling to handle any traffic level, built-in reliability with Google's production infrastructure, and enterprise-grade security out of the box. There's no infrastructure to manage — you just point to the Maps URL like below and authenticate with an API key, making it ideal for production deployments.

code_block: <ListValue: [StructValue([('code', '# Remote Google MCP server: connects to Google Maps Grounding Lite \r\n# to fetch real-time weather conditions for a given city.\r\nget_weather = McpToolset(\r\n connection_params=StreamableHTTPConnectionParams(\r\n url="https://mapstools.googleapis.com/mcp",\r\n headers={"X-Goog-Api-Key": os.environ["MAPS_API_KEY"] }\r\n ),\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f486047cd90>)])]>

While the Google Maps Grounding Lite is a Google-managed remote endpoint, the Cityscape agent also demonstrates the other end of the spectrum — a locally hosted MCP server for image generation. The nano_banana toolset connects to the GenMedia MCP server using StdioConnectionParams.

With this setup, the agent generates a stylized isometric cityscape image, incorporating the landmarks and weather data gathered earlier. Running a self-hosted MCP server gives you full control over the process lifecycle and environment configuration, but requires a local binary on the host machine or a sidecar container, which adds setup complexity compared to the hosted approach.

code_block: <ListValue: [StructValue([('code', '# Self-hosted MCP server: launches the GenMedia MCP server (mcp-gemini-go)\r\n# as a subprocess to generate cityscape images via the Gemini image model.\r\nnano_banana = McpToolset(\r\n connection_params=StdioConnectionParams(\r\n server_params=StdioServerParameters(\r\n command="mcp-gemini-go",\r\n env=dict(os.environ, PROJECT_ID=os.environ["GOOGLE_CLOUD_PROJECT"]),\r\n ),\r\n timeout=60,\r\n ),\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f486047c550>)])]>

ADK supports Google-managed, remote, and self-hosted MCP servers. The former gives you production-ready infrastructure with zero operations overhead, while the latter two offer flexibility for custom or experimental tools.

2. Enterprise-grade security and content guardrails

Security in the agentic era can not be an afterthought. Here’s how two key security features can be applied to our Cityscape agent.

Granular control of MCP tools via IAM Deny policies

Google Cloud lets you control MCP tool access using IAM deny policies — the same governance framework you already use for other Google Cloud resources.

Now imagine we extend the Cityscape agent by adding a BigQuery MCP server — perhaps to query a dataset of historical cityscape metadata or population statistics. The BigQuery MCP server exposes both read-only tools like get_dataset_info and list_datasets, as well as write tools like execute_sql that can modify data.

In our use case, the agent should only query BigQuery for information — it should never execute SQL that inserts, updates, or deletes data. With Google-managed MCP servers, you don't have to rely on prompt engineering alone to enforce this.

Instead, you apply an IAM Deny policy that blocks any tool not annotated as read-only:

code_block: <ListValue: [StructValue([('code', '// IAM deny policy: blocks all MCP tool calls that are not read-only.\r\n{\r\n "rules": [\r\n {\r\n "denyRule": {\r\n "deniedPrincipals": ["principalSet://goog/public:all"],\r\n "deniedPermissions": ["mcp.googleapis.com/tools.call"],\r\n "denialCondition": {\r\n "title": "Deny read-write tools",\r\n "expression": "api.getAttribute(\'mcp.googleapis.com/tool.isReadOnly\', false) == false"\r\n }\r\n }\r\n }\r\n ]\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f486047ca30>)])]>

Apply it with:

code_block: <ListValue: [StructValue([('code', 'gcloud iam policies create mcp-deny-policy \\\r\n --attachment-point=cloudresourcemanager.googleapis.com/projects/$PROJECT_ID \\\r\n --kind=denypolicies \\\r\n --policy-file=policy.json'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f486047c070>)])]>

With this policy applied, the agent can freely look up dataset schemas, but any attempt to call execute_sql — whether intentional or triggered by a prompt injection — is blocked at the platform level before it ever reaches BigQuery. This is defense-in-depth: Your agent's instructions say "only read data," but IAM enforces it — regardless of what the LLM decides to do.

Content security with Model Armor

Model Armor integrates directly with Google Cloud MCP servers to sanitize all MCP tool calls and responses at the project level. Once enabled, it acts as an inline security layer that scans for:

Prompt injection attacks
Malicious URIs (such as phishing links)
Dangerous content that violates responsible AI filters

Returning to our Cityscape agent, imagine a user submitting: "Generate a cityscape for http://malicious-site.com".

With Model Armor enabled, the MCP tool call is scanned before it reaches the Maps server. Malicious URIs, prompt injection attempts, and dangerous content are blocked automatically — no custom validation code needed in your agent.

Enabling it is a two-step process. First, configure a floor setting that defines your minimum security filters:

code_block: <ListValue: [StructValue([('code', 'gcloud model-armor floorsettings update \\\r\n --full-uri=\'projects/$PROJECT_ID/locations/global/floorSetting\' \\\r\n --enable-floor-setting-enforcement=TRUE \\\r\n --add-integrated-services=GOOGLE_MCP_SERVER \\\r\n --google-mcp-server-enforcement-type=INSPECT_AND_BLOCK \\\r\n --enable-google-mcp-server-cloud-logging \\\r\n --malicious-uri-filter-settings-enforcement=ENABLED \\\r\n --add-rai-settings-filters=\'[{"confidenceLevel": "MEDIUM_AND_ABOVE", "filterType": "DANGEROUS"}]\''), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f486047c220>)])]>

Then enable content security for your all Google MCP servers in your project:

code_block: <ListValue: [StructValue([('code', 'gcloud beta services mcp content-security add modelarmor.googleapis.com \\\r\n --project=$PROJECT_ID'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f4862b26af0>)])]>

Once enabled, all MCP traffic in the project is automatically scanned — regardless of which agent or client originates the call. Blocked requests are logged to Cloud Logging, giving you full observability into potential threats.

Getting started

Google MCP servers remove the infrastructure hurdles that keep AI agents stuck in prototyping. By combining managed endpoints with platform-level security — IAM deny policies, Model Armor, and Cloud Audit Logs — you get a production-ready foundation with minimum ops overhead. The era of the autonomous agent is here: Make sure your stack is ready.

ADK Cityscape agent code repo here
Read more about Google MCP servers and supported services here
Hands-on codelab: Local to Cloud — Full-stack app migration with Gemini CLI, Cloud Run, and Cloud SQL MCP servers
Build AI agents with Google Cloud Run: a serverless runtime for your agentic AI apps

Centralized policy meets distributed logic: Getting to know Eventarc Advanced

Fri, 27 Feb 2026 17:00:00 +0000

Enterprise architects often face a fundamental dilemma: choosing between developer agility and organizational control. Development teams need to move fast and deploy independent microservices without waiting for permission. Security and compliance teams need to be safe, and ensure that data flow is observable and governed by policies.

That’s why we built Eventarc Advanced, a serverless eventing platform and the evolution of Eventarc Standard. Eventarc Advanced provides an improved architectural pattern for the modern cloud, where centralized policy meets distributed logic. By clearly separating the governance layer (the "bus") from the processing layer (the "pipeline"), Eventarc Advanced gives SecOps teams the visibility and control they demand, while freeing developers to choreograph AI agents and build event-driven applications with the autonomy they want. Eventarc Advanced became generally available in August 2025.

In this blog, we take a deeper look at the evolution of integration architectures — from service buses, to microservices, to where we are today — and go into depth with a real-world example. Let’s jump in.

The evolution of integration architectures

To understand the value of this new pattern, it helps to look at where we came from and why previous architecture patterns forced a compromise.

The centralized bottleneck of the Enterprise Service Bus

One early integration architecture approach was the Enterprise Service Bus (ESB), which prioritized centralized control. The ESB emerged to solve the "spaghetti architecture" of point-to-point integrations by providing a centralized communication layer that standardized how disparate systems interact. However, it often introduced serious pitfalls.

The primary issue was what’s referred to as a centralized logic trap. Organizations frequently embedded complex business logic — transformations and orchestration — directly into the governance layer. The resulting middleware layer was opaque, with critical business rules hidden from the developers who owned the services.

Consequently, integration changes typically required the intervention of a central middleware team. Development teams lost autonomy, forced to queue behind integration specialists to ship even minor features, often waiting weeks for updates.

Microservices’ governance gap

To address this, the industry shifted toward microservices (often described as "smart endpoints and dumb pipes"), distributing logic to give teams the autonomy they were looking for. For synchronous traffic (REST, gRPC), tools like API gateways and service meshes restored a layer of governance by enforcing policies like authentication and rate limiting at the infrastructure level.

However, as architectures shifted to Event-Driven Architecture (EDA) for greater resilience and decoupling, a new gap emerged. In a distributed, asynchronous world, centralized control often vanished. This created a governance gap where SecOps teams struggled to maintain order. Three issues emerged to the forefront:

The visibility void: Without a central policy, shadow IT services could silently subscribe to sensitive events without detection.
The policy problem: Enforcing data residency or PII masking is nearly impossible when the broker treats every message as an opaque blob.
The dependency risk: Without clear contracts, changing an event schema risks silently breaking unknown downstream consumers.

A new pattern: Centralized policy, distributed logic

Eventarc Advanced addresses the trade-off between control and speed with a novel architectural pattern: centralized policy meets distributed logic.

Eventarc Advanced maps these distinct responsibilities to two specific architectural resources that each correspond to a distinct role:

The bus: This governance layer is a managed, centralized hub where platform administrators enforce global constraints before events are routed. It synthesizes the centralized routing of the legacy ESB with the modern security architecture of a service mesh. It handles Identity and Access Management (IAM), including content-based access control, to strictly define who can publish, and integrates with VPC Service Controls to prevent data exfiltration.
The pipeline: Think of this distributed, team-owned resource as developers’ integration logic layer. This is where eventing patterns for AI agents and microservices are unlocked, allowing developers to configure event flow and delivery according to their specific business logic. Unlike many service meshes that treat data as opaque bits, the pipeline understands content. Developers can transform events, convert payloads between formats (like JSON to Avro), and configure retry policies and authentication independently.

In other words, by decoupling these duties, Eventarc Advanced provides the control of an ESB with the agility of microservices and the resilience of modern event-driven architectures.

How it works: A retail event mesh example

A typical Eventarc Advanced solution can be implemented with minimal configuration, providing a streamlined experience for both administrative governance and distributed integration logic. To see this model in practice, let's look at a real-world implementation of a retail event mesh.

Imagine an ecosystem at a global retailer with four autonomous teams in charge of the following services:

Commerce
Finance
Logistics
Intelligence (AI Insights Agent)

In a traditional setup, aligning these teams is difficult. The Intelligence team wants access to everything for their models, Finance wants to lock everything down for compliance, Logistics just needs a stable schema to ship boxes, and Commerce needs to roll out new features at a moment’s notice.

The foundation: Built on CloudEvents

Eventarc Advanced uses a data model based on the open CloudEvents standard, which can carry any type of payload. This helps ensure governance and discoverability while retaining flexibility. In our example, before a single event is published, the platform administrator mandates that every message must contain standard attributes and a specific custom extension for governance.

In this example, every event on the bus must carry the following attributes:

type: Standard identifiers for the event instance (e.g., com.retail.order.created)
source: A standard attribute identifying the producer (e.g., //commerce/frontend)
data_sensitivity: A custom extension attribute to categorize risk

In addition, the organization defines three data sensitivity levels:

restricted (High): Severe risk data like Credit Card Tokens or Tax IDs
confidential (Medium): PII like home addresses
general (Low): Safe operational data like Order IDs

This standardized metadata layer allows the bus to enforce policies based on specific attribute names — checking who sent the data (source) and what kind of data it is (data_sensitivity).

The workflow

With this model, the lifecycle of a single order becomes a secure flow where sensitivity changes at every step.

Order placement: The Commerce service publishes order.created to the Bus. The event’s data sensitivity is tagged as general. The AI Insights Agent service subscribes to analyze market trends.
Payment authorization: The Commerce service publishes payment.authorized tagged as restricted (containing a secure token). The Finance service subscribes to capture the token and executes the charge.
Settlement: The Finance service publishes payment.success tagged as general, signaling the transaction is safe to fulfill without exposing financial secrets. Logistics subscribes to ship the box, and Intelligence AI Insights Agent is triggered to evaluate market trends for the next supply chain cycle.
Fulfillment: The Logistics service publishes shipment.ready tagged as confidential (containing the customer phone number). The Logistics own notification pipeline subscribes to it to trigger an SMS notification.

In a legacy architecture, mixing PCI, PII, and operational data on a single bus would be a compliance nightmare. With Eventarc Advanced, it’s a solved problem.

The bus: the governance layer

The platform administrator implements a secure strategy on the bus. Rather than blindly trusting internal services, they enforce global policies that inspect these CloudEvents attributes using fine-grained access control (FGAC).

Enforcing source integrity

To ensure a compromised service cannot spoof events, the bus administrator enforces the producer's identity to match the source attribute.

For example, a bus policy can state that only the principal sa-commerce@retail.com can publish events that match the expression message.source.startsWith("//commerce/"). If the Intelligence AI Insights Agent service tries to publish an event claiming to be from //commerce/payments, the bus rejects the request.

Enforcing a data classification

To ensure every event is categorized, the bus administrator requires that every payload received by the bus includes a valid sensitivity attribute. A bus policy can check that message.data_sensitivity is one of ['general', 'confidential', 'restricted']. This guarantees that the event mesh contains only classified, governance-ready data.

The Pipeline: the logic layer - autonomous team innovation

With the security posture established on the bus, development teams can then use pipelines to solve complex integration challenges entirely within their own domains. Let’s take a look at a few of these challenges.

Schema-aware formats conversion and payload transformation

The Logistics team decides to upgrade their warehouse robots to use high-efficiency protocol buffers. Instead of forcing the Finance team to change their JSON output (which would break other consumers), Logistics configures a transformation step in their own pipeline.

A typical com.retail.payment.success event from Finance arrives as JSON:

code_block: <ListValue: [StructValue([('code', '{\r\n "id": "89d5663e-789e-4d9f-a65f-f7d83742d987",\r\n "source": "//finance/ledger",\r\n "type": "com.retail.payment.success",\r\n "data_sensitivity": "general",\r\n "datacontenttype": "application/json",\r\n "data": {\r\n "order_number": "ORD-2023-8841",\r\n "total_amount": 249.99,\r\n "currency": "USD",\r\n "transaction_id": "tx_77382910",\r\n "status": "SETTLED"\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f48621b6f70>)])]>

The warehouse robots service expects a binary Protobuf message:

code_block: <ListValue: [StructValue([('code', 'message PaymentConfirmed {\r\n string order_id = 1;\r\n double insured_value = 2;\r\n string currency_code = 3;\r\n string ledger_reference = 4;\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f486566dbe0>)])]>

The Logistics team configures their pipeline to accept json as input and output to protobuf. To map the data, they use Common Expression Language (CEL) to configure a transformation:

code_block: <ListValue: [StructValue([('code', '// CEL Transformation to Construct the target Protobuf message\r\n{\r\n "order_id": message.data.order_number,\r\n // 110% of total to cover replacement cost\r\n "insured_value": message.data.total_amount * 1.1,\r\n // Standardize currency to uppercase\r\n "currency_code": message.data.currency.upperAscii(),\r\n "ledger_reference": message.data.transaction_id,\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f486575fa60>)])]>

This transformation not only maps the input but also applies business logic — calculating the insured value and normalizing the currency code. The Logistics team implements this modernization without a single meeting with the Finance team.

Agentic workflows: Filtering and triggering AI agents

Eventarc Advanced enables agentic workflows by allowing pipelines to communicate directly with AI agents using open standard protocols like Agent2Agent (A2A) and Model Context Protocol (MCP), while also offering rich capabilities like filtering to optimize when those agents are invoked.

The Intelligence team uses a pipeline they name ai-insights and the A2A protocol to connect with an AI Insights Agent that proactively analyzes market trends based on placed orders. Because the agent’s processing is resource-intensive, the team uses a filter to only invoke the agent for high-value orders that warrant deeper analysis.

The pipeline filter is configured with the following expression:

code_block: <ListValue: [StructValue([('code', 'message.type == "order.created" && \r\ndouble(message.amount) > 5000.0'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f4862af8700>)])]>

When the filter is passed, the pipeline uses a HTTP Message Destination Binding (MDB) expression to directly trigger the agent. By defining a CEL template, the pipeline handles the complexity of constructing a native A2A SendMessage request to the AI strategic insights agent. This allows the agent to receive a conversational prompt derived from technical event data without any manual "glue code":

code_block: <ListValue: [StructValue([('code', '{\r\n "headers": headers.merge({ "Content-Type": "application/json", "A2A-Version": "1.0" }),\r\n "body": {\r\n "jsonrpc": "2.0",\r\n "id": message.id,\r\n "method": "message/send",\r\n "params": {\r\n "message": {\r\n "messageId": message.id,\r\n "role": "ROLE_USER",\r\n "parts": [\r\n { \r\n "text": "Analyze Order " + message.data.order_number + " for market trends." \r\n }\r\n ]\r\n }\r\n }\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f4864337370>)])]>

A similar prompt message can be crafted for other popular agentic communication protocols like MCP.

This combination of filtering and agentic protocol translation ensures that AI resources are used precisely where they add value. The Intelligence team implements this independently – without writing ingestion code or coordinating with the Commerce or Admin team. The agent can then publish its own strategic recommendation back to the bus, enabling a choreography of AI experts that turns standard cloud events into competitive intelligence.

Advanced API request modeling

When a shipment is ready, the Logistics team uses a pipeline to send an SMS using a legacy gateway API. Integrating with legacy third-party APIs often requires writing "glue code" services just to format requests.

The Logistics team eliminates this maintenance burden by configuring a dedicated pipeline to fully construct the exact request expected by the legacy service.

They use a HTTP Message Destination Binding CEL expression, which standardizes the phone number and maps it to the X-SMS-To HTTP header required by the API. It also construct the SMS text:

code_block: <ListValue: [StructValue([('code', '{\r\n "headers": { "X-SMS-To", \r\n message.data.phone.matches(\'^\\\\+1\') ?\r\n message.data.phone : \r\n \'+1\' + message.data.phone \r\n },\r\n\r\n "body": {\r\n "sms_text": "Order " + message.data.order_id + " shipped!"\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f4862ebb580>)])]>

Finally, they configure a robust retry policy (linear backoff, max five attempts) directly on the pipeline, so that temporary network interruptions don't result in lost notifications. In addition to HTTP endpoints, the pipeline supports guaranteed delivery and out-of-the-box authentication for destinations like Cloud Run, Pub/Sub, Bus, Workflows, and over 200 Google services.

The future of agile integration

Eventarc Advanced closes an important gap in event-driven architectures: It brings the same level of maturity to asynchronous communication by introducing the pattern of centralized policy, distributed logic.

For the Platform team, Eventarc Advanced provides assurance that a bus can strictly enforce integrity and confidentiality on every message, bringing "service-mesh-like" security to the event layer.
For the developer, it restores autonomy. The pipeline allows teams to filter, transform, convert, and route events to fit their specific needs, enabling them to treat events as first-class products rather than opaque artifacts.

This architecture lays the foundation for the next generation of intelligent applications. A secure, typed, and trustworthy event mesh can serve as the backbone for generative AI agents and real-time analytics, allowing you to safely expose business context to the systems that need it most.

Get started

Don't let governance slow down your innovation. Here are some Eventarc Advanced resources to get you on your way:

Learn more: Dive into the full capabilities of the Bus and Pipeline in the Eventarc Advanced documentation.
Get hands-on: Deploy the "Retail Event Mesh" scenario yourself and explore enterprise patterns with our Quickstarts and Tutorials.
Start building: Go to the Google Cloud console to configure your first bus and pipeline today.
Let's talk: Have a complex enterprise use case? Contact Google Cloud Sales to discuss how Eventarc Advanced fits into your broader integration strategy.

Powering the next generation of agents with Google Cloud databases

Wed, 18 Feb 2026 18:00:00 +0000

For developers building AI applications, including custom agents and chatbots, the open-source Model Context Protocol (MCP) standard enables your innovations to access data and tools consistently and securely. At the end of 2025, we introduced managed and remote MCP support for services like Google Maps and BigQuery, establishing a standard method for AI to connect with tools, and effectively creating a universal interface for applications. Today, we are expanding this offering to include PostgreSQL with AlloyDB, Spanner and Cloud SQL, as well as Firestore and Bigtable for high-performance NoSQL workloads, and introducing a new Developer Knowledge MCP server, which presents an API to connect IDEs to Google’s documentation. These servers run in Google Cloud, providing a secure interface for Gemini and other MCP-compliant clients to easily interact with data and infrastructure.

With the launch of Gemini 3, developers gained advanced reasoning capabilities to plan, build, and solve complex problems. But for an AI model to function as a useful "agent," it must reliably interact with its environment. Today’s announcement extends these capabilities more broadly to the database tools our customers leverage daily as the backbone of their work environment.

To connect your agents to these servers, you don’t need to deploy infrastructure. Just configure the MCP server endpoint in the agent configuration and immediately gain access to your operational data, backed by enterprise-grade auditing, observability and governance. With no infrastructure management, you can scale your agentic workloads without incurring operational overhead.

Bringing operational data to agents

These new managed servers enable agents to access specific capabilities across our portfolio:

AlloyDB for PostgreSQL: Agents can interact with PostgreSQL workloads, enabling tasks such as schema creation, diagnosing complex queries for slowness and performing vector similarity search.
Spanner: With unified multi-model capabilities in Spanner such as Spanner Graph, agents can model and query complex relationships directly alongside relational and semantic data using standard (SQL and GQL) queries. This allows agents to quickly uncover deep insights (like identifying fraud rings or generating product recommendations) using the MCP tools at its disposal.
Cloud SQL for PostgreSQL, MySQL and SQL Server: Developers and database administrators can use the Cloud SQL MCP Server across MySQL, PostgreSQL, and SQL Server fleets for natural language interactions with the database, AI-assisted app development, query performance optimization and database troubleshooting via agents.
Bigtable: Bigtable’s flexible schema and high-throughput ingestion capabilities are commonly used for building digital integration hubs and managing time series data. MCP simplifies automating operational workflows and developing agentic customer support, CRM, human resources, IT operations, supply chain and logistics applications with this data.
Firestore: Focused on mobile and web development, the Firestore MCP server enables agents to sync with live document collections. This supports dynamic interactions such as checking user session states or verifying order statuses via natural language prompts.

Managing applications and infrastructure

Beyond data retrieval, we are enabling agents to help build and manage applications. The Developer Knowledge MCP server connects IDEs to Google’s documentation, allowing agents to answer technical questions and troubleshoot code with relevant context.

Security and governance

Connecting an agent to a database requires robust security and governance. These servers are built on Google Cloud's standard identity and observability frameworks:

Identity-first security: Authentication is handled entirely through Identity and Access Management (IAM) rather than shared keys. This ensures agents can only access the specific tables or views explicitly authorized by the user.
Full observability: To track agent activity, every query and action taken via these MCP servers is logged in Cloud Audit Logs. This provides security teams with a record of every database interaction, maintaining visibility alongside ease of access.

Demo: From local code to managed data

Let’s see these new MCP servers in action.

Imagine an agent designed to automate the migration of a full-stack event management platform for fitness communities. Through a series of natural language instructions in the Gemini CLI, the agent utilizes the Cloud SQL remote MCP server to provision a managed PostgreSQL instance, apply the correct schema, and securely migrate your local data. You don't need to master complex gcloud commands or become a Cloud SQL expert; the agent handles the heavy lifting. This transition is architected in real-time by the Developer Knowledge MCP server, which references official documentation to guide the agent through best practices — easily upgrading your application's backbone from local storage to a fully managed enterprise database.

Support for third-party agents

Because these servers follow the open MCP standard, they also work with your favorite AI agents. You can easily connect clients like Anthropic’s Claude by adding a Custom Connector in the settings. Simply point it to your Google Cloud database MCP endpoint, and you are ready to start building — no complex configuration files required.

What’s next

We’ll continue to expand this ecosystem in the coming months with managed MCP support for Looker, Database Migration Service (DMS), BigQuery Migration Service, Memorystore, Database Center, Pub/Sub, Kafka and more.

To start building secure, data-driven agents, explore our guides for AlloyDB, Spanner, Cloud SQL, Bigtable, and Firestore. You can also check out these codelabs for Cloud SQL and Spanner, along with this demo video walking through the app migration to Google Cloud.

The platform usage trap part 1: Why high activity doesn’t necessarily mean high value

Wed, 04 Feb 2026 18:00:00 +0000

For any organization that has invested in an internal developer platform, a question inevitably arises: Is it actually working?

Simply tracking adoption rates won't tell you if your platform is truly delivering value to your developers. This was the challenge faced by John Lewis, a major UK retailer. In our previous articles (parts 1 and 2) we introduced the John Lewis Digital Platform (JLDP) and how it enabled dozens of product teams to build high-quality software rapidly to power www.johnlewis.com and other critical applications. But how did they know that the platform was actually successful? Traditional product metrics like revenue and sales don’t translate easily to this world. When you focus only on whether your tenants use the platform, you don’t understand whether it’s bringing them value.

In this article, Alex Moss from the John Lewis platform team discusses how they moved beyond simple usage metrics to develop a sophisticated, multi-stage approach to measuring the real value of their platform — a journey that took them from lead-time metrics, to DORA, and finally to a "Technical Health" score. Along the way, they explore how the JLDP’s purpose evolved — and its value along with it. - Darren Evans

Initial measurement: A focus on platform value

In the early days of the platform, understanding its value was actually much easier. This was because the platform was created with a very clear purpose: to enable speed of change. The John Lewis business wanted to create multiple product teams working on several features of johnlewis.com in parallel, and to put those features in front of customers quickly for feedback.

Its origins in the world of the company’s John Lewis Digital online business resulted in it being treated as a product from a very early stage, and therefore integrated with that area’s reporting mechanisms too. Thus, it became normal to link the platform objectives to the online business’s broader goals each quarter and report on measurable key results. This kept the focus on the reasons the platform is important: do improvements to the platform continue to justify using it over seeking out a different one? We cannot afford to rest on our laurels!

The six annual measures reported against every quarter. The specific measures have varied over the years.

In addition to this, in the first few years of the platform’s existence, there were three simple metrics that best indicated how the platform was living up to the rationale for creating it:

Service Creation Lead Time: How long it took to create a tenancy (the space in which a product team was creating their software)
Onboarding Lead Time: How long it took that product team to deploy something into production
First Customer Lead Time: How long it took that product team to designate their service as “live to customers”

Some screenshots from the early version of the platform's self-written service catalogue, tracking the three metrics mentioned

This was then combined with the number of tenants present on the platform into a report, which was displayed as part of an initial home-grown Service Catalogue shown above (which was later replaced with Backstage). This report served two purposes:

A very clear visualization for stakeholders of how much their platform was being adopted, and how fast they were able to get up and running (in particular, “Service Creation” being measured in single-digit hours, in comparison to the weeks teams would traditionally have had to wait). This is important, because in the early days of your product, you need to justify its continued growth and investment.
A useful way for the platform team themselves (and stakeholders) to see which teams were taking their time about getting something into production. Is my product actually helping you? And if not, what more could we be doing?

Using this as a conversation-starter with our tenants opened doors to rich sources of feedback that could be turned into platform features: When we asked tenants “What’s stopping you from going live?”, they often answered that the product they were building was simply complex. But we also often saw that our own processes were getting in the way. This was important, as we could then do something about it.

The easiest of these barriers for us to overcome were typically technology-related. In previous articles, we covered two examples, “My team is spending a lot of time writing Terraform to provision PubSub,” and “we’re having trouble learning how to use Kubernetes.” To help, the platform team created “paved roads” to enable self-service provisioning or simplification of Kubernetes, significantly reducing these burdens for teams.

The more significant opportunities to streamline getting new services live were a result of our processes (e.g., security approvals) — and if your platform is empowered to simplify these sorts of organizational functions, then the gains can be extremely beneficial. One such example was the Information Security risk assurance process. Gaining the necessary security sign-offs and producing the required documentation was a necessary but time-consuming task, and - with the rate of change in the business - this was often something that many teams were going through in parallel. Our platform team successfully negotiated a simplified process for its tenants. It was able to do this because, by being resident on the platform, they could guarantee that security controls were in place and that policies were being followed. This was a direct result of the platform building features to meet those needs, and being able to provide evidence that they were being used — removing the need for the tenant team to either document or invent this themselves. This is still simplifying the developer experience through platform engineering, even though the solution is a less technically-based one.

Sometimes the conversation resulted in feedback that wasn’t even platform-shaped — for example, helping teams understand concepts like feature flagging and dark launching, or software design options to help break dependencies with legacy systems. John Lewis’ platform teams are staffed with experienced engineers, ideally ones with software development experience, which helps a lot with these sorts of interactions.

A key point here is that by measuring how effectively teams were making it into production, we could identify who to talk to and elucidate the feedback we needed on what problems needed to be addressed. Simply relying on your tenants thinking of this themselves when they don’t see the bigger picture (or have other priorities) is not nearly as effective.

We then combined the process with more traditional approaches such as sending out a survey or use of Net Promoter Scoring to help build popularity in the product. The results of these were usually very positive, and could be used to generate mindshare — especially where a product team was comfortable talking about their positive experiences in internal tech conferences and the like.

Helping understand team performance

A few years into the life of the platform, our emphasis started to shift. There was less of a need to prove the value of the platform — the business and our engineers were happy — so we shifted from “how can we get you into production as quickly as possible” towards “how can we enable you to continue to be as fast, but also reduce friction, in your day-to-day activities.” This led us towards DORA metrics.

Our initial DORA implementations involved mining information from our systems of record for change and incident, complimented by our already-mature observability stack for availability data, as well as pulling events from things like cloud audit logs. We built software to do this and stored it in BigQuery, which enabled us to visualize the data in our home-grown Service Catalogue tool. Later, we moved this into Grafana dashboards instead, which are still in use today:

Looking for patterns in this data led to us discovering additional features that would be useful for us to build. Two major examples of this were handling change, and operational readiness.

JLP’s service management processes were geared towards handling complex release processes across multiple large systems and/or teams - but we had fundamentally changed our architecture by adopting microservices. This empowered teams to release independently at will, and therefore manage the consequences of failed changes themselves. We used the data we’d collected about change failure rates and frequency of small releases to justify a different approach: allowing tenants to automatically raise and close changes as part of their CI/CD pipelines. After clearing this approach with our Service Management team, we developed a CLI tool that teams could use within their pipelines. This had the additional benefit of allowing us to capture useful data at point of release, rather than scraping more awkward data sources. The automated change “carrot” was very popular and was widely adopted, shifting the approval point left to the pull request rather than later in the release process. This reduced time wastage, change-set size and risk of collisions.

In a similar vein, with more teams operating their own services, the need for a central site-wide operations team was reduced. We could see from our metrics that teams practicing “You Build It, You Run It” had fewer incidents and were resolving them much more quickly. We used this as evidence to bring in tooling to help them respond to incidents faster, and decouple the centralized ops teams from those processes — in some cases allowing them to focus on legacy systems, and in others, removing the need for the service entirely (which resulted in significant cost savings, despite the fact that we had more individual product teams on-call). This, and supporting observability and alerting tooling, was all configured through the platform’s paved-road pipeline described in our previous article.

The DORA metrics helped us architecturally as well. Operational data shined a light on the brittleness of third-party and legacy services, thereby driving greater investment into resilience engineering, alternative solutions, and in some cases, causing us to re-evaluate our build vs. buy decisions.

Choosing what to measure

It’s very important to choose wisely about what to measure. Experts in the field (such as Laura Tacho) influenced us to avoid vanity metrics and to be cautious about interpreting the ones we do collect. It’s also important for metrics to be meaningful to the target audience, and presented accordingly.

As an example, we communicate about cost and vulnerability with our teams, but the form this takes depends on the intended audience’s role. For example, we send new vulnerabilities or spikes in cost directly to product teams’ collaboration channels, because experience has taught us that having our engineers see these vulnerabilities results in a faster response. On the other hand, for compliance reporting or review by team leads, reports are more effective at summarising the areas that need action. Because if we know one thing, it’s that nobody wants to be a leader of the “vulnerabilities outside of policy” dashboard!

It was not unusual for us to historically look at measures such as the number or frequency of incidents. But in a world of highly automated response systems, this is a trap, as alerts can be easily duplicated. Focusing too much on a number can drive the wrong behavior — at worst, deliberately avoiding creating an incident at all! Instead, it’s much better to focus on the impact of the parent incident and how long it took to recover. Another example is reporting on the number of vulnerabilities. Imagine you have a package that is used extensively across many components in a distributed system. Disclosing that the package has a vulnerability can create a false sense of scale, when in fact patching the base image deals with the problem swiftly. Instead, it’s better to look at the speed of response than a pre-agreed policy based on severity. This is both a much more effective and reasonable metric for teams to act on, so we see better engagement.

It’s very important that you put across as much context as possible when presenting the data so that the right conclusions can be drawn — especially where those reports are seen by decision-makers. With that in mind, we combined raw metrics we could visualize with user opinion about them. This helped to bring that missing context: Is the team that’s suffering from a high change failure rate also struggling with its release processes and batch size? Is the team that’s not addressing vulnerabilities quickly also reporting that they’re spending too much time on feature development and not enough on operational matters? We reached for a different tool — DX — to help us bring this sort of information to bear. In our follow-up article, we’ll elaborate on how we did this and how it prompted us to expand the data we collected about our tenants. Stay tuned!

To learn more about shifting down with platform engineering on Google Cloud, start here.

The platform usage trap part 2: Choosing meaningful monitoring metrics

Wed, 04 Feb 2026 18:00:00 +0000

In part one of this article, Alex Moss from the John Lewis Partnership covered the metrics that they use to measure the value of their developer platform. Now, let's talk about a crucial aspect of any measurement strategy: choosing the right things to measure. It's easy to get lost in a sea of data or to focus on metrics that look impressive, but don't actually reflect the health of your platform or the experience of your developers. Here, Alex shares the John Lewis philosophy on how to choose meaningful metrics and present them in a way that drives the right conversations and actions, ensuring that the data is always presented with as much context as possible. - Darren Evans

While the solution we detailed in the first half of this article worked very well, relying solely on objective measures comes with a number of traps. They are very easy to misinterpret: either wasting time (“the team is working on another product at the moment”) or not telling the right story (“the incident wasn’t closed properly”). This leads to a scaling challenge: Chatting with a small number of teams to understand a situation is one thing. But when you are only one small team trying to build a product, and you need to talk across several dozen teams, it’s not so easy.

Collecting engineers’ subjective feedback

We needed a way to collate more subjective feedback, ideally in a form that we could visualize and contrast to the objective DORA and other service metrics we held.

Our initial attempt at this involved creating Service Operability Assessments — questionnaires that tenants fill in every quarter. Service Operability Assessments are intended to hold a series of thought-provoking questions aimed at whether the team is following good practices for running their service. This worked well with an experienced facilitator (usually a senior platform engineer) who could ask further probing questions and pull out the key feedback and actions. But as you might imagine, this suffered from scaling challenges. We eventually let this be handled entirely self-service — an imperfect system, since many teams are quite happy to just copy/paste their answers from the previous quarter, which may or may not reflect reality!

We then learned about a tool called the DX platform, which significantly changed how we approached this, and which is now used across our entire Engineering community. It works by surveying individual engineers (rather than teams) for a few minutes every three months. The questions are curated based on DX’s research, backed by the founders of DORA and other similar frameworks. We’ve found it very helpful to be able to slice the results in different ways, including looking at areas across whole platforms or deep-diving on particular teams. The latter, in combination with our DORA data, makes for rich conversations. For example, in the DX tool, a team which recently suffered through some highly impactful incidents might also have registered concerns on “Production Debugging,” while another team that saw a marked drop in release frequency flagged worries around “Change Confidence” or “Ease of Release.” The platforms team can at this point step in to offer advice or potentially implement new features to help with the issues the teams are seeing.

The pre-built drivers and reports in DX are tremendously useful, but we also augment it with our own custom queries to help us understand areas of current focus. For example, we measure Customer Satisfaction (CSAT) for the platform and its portal (Backstage), and collect data on how long it takes for a newcomer to begin submitting pull requests and ask them about how they found the onboarding process. We also recently started assessing engineers’ opinions on the effectiveness of AI coding assistants to help justify further investment in them (instead of just relying on market insight).

An example of where this helped focus our efforts was with documentation, namely, building capabilities into our Backstage developer portal to make it easier for teams to view each others’ docs through pipelines that automatically publish content and make it discoverable.

Service health - Feature adoption & beyond

Outside of the insights we generate from the likes of DORA and DX, we’ve recently begun questioning not only whether the platform itself is valuable, but whether tenants are getting the value they should from it. In other words, we’ve effectively started to measure platform feature adoption.

To do this, we built out what we refer to internally as our Technical Health feature. It takes the form of a custom plugin that integrates with our Backstage Developer Portal, which then queries an in-house API that surfaces data fed from a large number of small jobs that collect information on the things we want to measure. These jobs are independently releasable themselves, which allowed us to scale this up pretty quickly.

We currently capture four categories of health measures:

Technical health: We currently have 17 “technical” measures. Examples here include measuring whether teams are using our paved road pipeline and custom Microservice CRD (see previous articles 1 and 2) rather than “terraforming” their own resources, following our recommended Kubernetes practices (such as resource sizing, disruption budgets and lifecycle probes), keeping base images up to date, and the like. We also include some “softer” technical measures such as whether they are running pipelines frequently enough to pick up changes (we don’t run this for teams), reviewing their operability assessments, staying on top of git branches, and so on.
Operational readiness: Then, there are 18 measures relating to operational health — things like whether a pre-flight configuration is in place, whether runbooks are written, docs have been published, and so on. This is an evolution of an Operational Readiness checklist from several years ago (back when we used to have separate Delivery and Operations teams, and therefore these sorts of checks were mandatory for “handover”). We tailored this checklist to the specific features of the platform that help teams achieve good operability, rather than being a generic list. This also serves to help our Service Management team feel confident that the right practices are being followed, thereby eliminating a point of friction when carrying out manual reviews.
Migrations: From time to time, the Platform requires tenants to carry out work to keep up with changes to the platform itself. A classic example of this is getting teams to deal with deprecated Kubernetes API versions. This also includes adoption of different features that we want to drive more forcefully in order to remove the older way of doing things (say for example, in favour of something more secure). We found that as the Platform grew, we had a long tail of migration work that we needed teams to perform, providing an easy way for Product Managers and Delivery Leads to prioritize their teams’ workloads.
Broader engineering practices: We recently opened up the feature to allow other teams to contribute — in this case, our Engineering leadership — to build in their own measures, such as whether teams are keeping up to date with versions of our design system or whether they’re following broader engineering practices that extend beyond just the JL Digital Platform.

We present this data through aggregated views (like the example shown below), as well as individual tasks and broader leaderboards — all designed to catch the eye of those with influence over a team’s priorities. We’ve found that the desire for an engineer to turn a traffic-light green can be a powerful motivator — far more effective than relying on documentation or announcements.

This technology works through custom plugins that we’ve built for the Backstage Portal. Each “health check” is itself its own microservice (often running as a job) which interrogates the appropriate system to determine whether the measure is met. For example, one microservice checks that a PodDisruptionBudget has been created by querying Kubernetes directly, while another that looks at whether distroless base images are in use, does so by inspecting container image layers. There’s a template for creating new metrics, which makes it easy for engineers to create new ones — including those outside the platform team themselves. The results are stored in BigQuery, with an API to make Backstage plugin development simpler.

A reality of introducing measures like this is that it drives more work into the product teams. It is important that your culture be ready for this. If we had implemented these measures very early in the platform’s life, this would likely have affected how the product was perceived — perhaps as very strict or inhibiting the pace of change with guardrails. This can negatively impact overall adoption. By introducing these later on, we benefited from many tenants who already saw the platform as very valuable, as well as the confidence that we had selected the right measures and could apply them consistently. That said, we did still see a small drop in CSAT for the platform after we started doing this. We try to be considerate about the pace that we launch each measure to give product teams the time to absorb the work, as well as provide a means for teams to suppress the indicators that aren’t relevant to them. For example, a tenant might deliberately choose not to use pod autoscaling for performance reasons, or have a functional reason why they can’t use our Microservice CRD.

The introduction of these sorts of assurance measures on tenant behaviour is a reflection of the maturity of the platform. In the early days, we relied on highly skilled teams to do the right thing whilst going fast. But as time has passed, we’ve witnessed a variety of skills and capabilities, combined with shifts in ownership of services, that pushed us to introduce techniques to drive the right outcomes. This is also due to the platform itself becoming complex — the cognitive load for a new team is much higher than it was, due to all its new features. We needed to put some lights along the edges of our paved road to help teams stay on it!

Throughout this evolution, we’ve continued to report on our key results for the business themselves: Are we still doing what they want of us? This has naturally shifted from “go fast, enable teams” (which we largely see as a solved problem, to be honest) towards “do it safely, and manage your technical debt.”

Are you being served? Key takeaways

Long story short, the question of whether a developer platform has value is complex, and can be answered in many ways. As you embark on building out — and quantifying — your own developer platform, here are a few concluding thoughts to keep in mind:

Measurement is a journey, not a destination: Start by measuring something meaningful to your stakeholders, but be prepared to adapt as your platform evolves. In the beginning, it’s okay to prioritize further investment in your product, but it’s better to actually measure how the platform is enabling your teams. The things that mattered when you were initially proving out the platform’s viability are unlikely to be what are important several years later when your features are more mature and your priorities have shifted.
Listen to the humans: Don’t assume that just because your platform is being used, that it is providing value. The most powerful metrics are often qualitative; engineers wanting to use your tool and CSAT are strong signals, but asking them questions about how they are using it is a better way to gain insight into how you can improve it. It is hard to figure out what’s working (and what isn’t) through measurement alone.
Data is for enabling, not just reporting: Use your insights to help teams improve, not just to show graphs to leadership. Further, be transparent about what specific data led you to act. For example, when you see a dip in release frequency for a specific team, use that data to start a conversation about potential roadblocks rather than simply flagging it as a problem. By doing this, you build the trust and goodwill with both leadership and your tenants to keep moving the platform forward.

_{The evolution of the John Lewis Partnership’s measurement strategy serves as a compelling case study. By transitioning from basic lead-time tracking to a holistic model — blending DORA metrics with qualitative developer feedback — they demonstrated that true platform success is defined by the genuine value it delivers, not merely by adoption rates.}

_{To learn more about platform engineering on Google Cloud, check out some of our other articles: Using Platform Engineering to simplify the developer experience - part one, part two, 5 myths about platform engineering: what it is and what it isn’t and Another five myths about platform engineering. We also recommend reading about App Hub, our foundational tool for managing application-centric governance across your organization.}

Monitoring Google ADK agentic applications with Datadog LLM Observability

Fri, 23 Jan 2026 17:00:00 +0000

Google’s Agent Development Kit (ADK) gives you the building blocks to create powerful agentic systems. These multi-step agents can plan, loop, collaborate, and call tools dynamically to solve problems on their own. However, this flexibility also makes them unpredictable, leading to potential issues like incomplete outputs, unexpected costs, and security risks. To help you manage this complexity, Datadog LLM Observability now provides automatic instrumentation for systems built with ADK. This integration gives you the visibility to monitor agent behavior, track costs and errors, and optimize agents for response quality and safety through offline experimentation and online evaluation without extensive manual setup.

This is significant as agentic systems are complex, and interagent interactions and the non-deterministic nature of LLMs makes it difficult to predict responses.

Common risks when running these agents include:

Pace of change: New foundation models drop weekly and “best-practice” prompting patterns change just as fast. Teams must constantly evaluate new combinations.
Multi-agent handoffs: If one agent produces low-quality output, it can cascade downstream and cause other agents to make poor decisions.
Loops and retries: Planners can get stuck calling the same tool repeatedly, such as retrying a search query indefinitely, which causes latency spikes.
Hidden costs: A single misrouted planner step can multiply token usage or API calls, pushing costs over budget.
Safety and accuracy: LLM responses may contain hallucinations, sensitive data, or prompt injection attempts, risking security incidents and reduced customer trust.

Finally, ADK is just one of many agentic frameworks available on the market. Having to manually instrument it only adds another learning curve to an already tedious and error-prone process.

Trace agent decisions and unexpected behaviors

Datadog LLM Observability addresses these pains by automatically instrumenting and tracing your ADK agents, so you can start evaluating your agents offline and monitoring them in production in minutes — without code changes. This allows you to visualize every step and planner choice — from agent orchestration to tool calls — on a single trace timeline.

For example, if an agent selects an incorrect tool to respond to a user query, it can yield unexpected errors or inaccurate responses. You can use Datadog’s visualizations to pinpoint the exact step where the incorrect tool was selected, making troubleshooting easier and helping you reproduce the issue.

Monitor token usage and latency

Sudden increases in latency or cost are often a sign of trouble in agentic applications. Datadog lets you view token usage and latency per tool, branch, and workflow to identify where errors happened and how they affected downstream steps.

For example, if a planner agent retries a summarization tool five times, it can significantly increase latency. Datadog highlights these loops, showing you exactly how long they took and the associated cost impact.

Evaluate agent response quality and security

Operational performance metrics like latency are critical monitoring signals, but for a holistic view of how agentic applications are performing, teams also need to evaluate the semantic quality of the LLM and agentic responses. Datadog provides built-in evaluations to detect hallucinations, PII leaks, prompt injections, and unsafe responses.

You can also add custom evaluators, including LLM-as-a-judge evaluators, for domain-specific checks. For instance, if a retrieval agent fetches irrelevant documents that lead to off-topic answers, a custom evaluator can flag that trace as having low retrieval relevance.

Iterate quickly and confidently with experiments

When you roll out a new system prompt, you might notice spikes in latency or drifts in output consistency. Datadog allows you to replay production LLM calls in its Playground to test different models, prompts, or parameters to find the configurations that move you closer to your ideal behavior.

From there, you can run structured experiments to compare versions side-by-side using datasets built from real traffic to optimize operational and functional performance. Because every agent step is logged through ADK instrumentation, you have the full context you need to reproduce regressions and validate fixes before you deploy.

Get started with Datadog LLM Observability

Datadog LLM Observability simplifies monitoring and debugging for Google ADK systems, helping users debug agent operations, evaluate responses, iterate quickly, and validate changes before deploying them to production.

You can get started today with the latest version of the LLM Observability SDK, or start a free trial if you are new to Datadog.

For more information on how to debug agent operations and evaluate responses, view Datadog’s LLM Observability documentation.

A gRPC transport for the Model Context Protocol

Tue, 13 Jan 2026 17:30:00 +0000

AI agents are moving from test environments to the core of enterprise operations, where they must interact reliably with external tools and systems to execute complex, multi-step goals. The Model Context Protocol (MCP) is the standard that makes this agent to tool communication possible. In fact, just last month we announced the release of fully-managed, remote MCP servers. Developers can now simply point their AI agents or standard MCP clients like Gemini CLI to a globally-consistent and enterprise-ready endpoint for Google and Google Cloud services.

MCP uses JSON-RPC as its standard transport. This brings many benefits as it combines an action-oriented approach with natural language payloads that can be directly relayed by agents in their communication with foundational models. Yet many organizations rely on gRPC, a high-performance, open source implementation of the remote procedure call (RPC) model. Enterprises that have adopted the gRPC framework must adapt their tooling to be compatible with the JSON-RPC transport used by MCP. Today, these enterprises need to deploy transcoding gateways to translate between JSON-RPC MCP requests and their existing gRPC-based services.

An interesting alternative to MCP transcoding is to use gRPC as the custom transport for MCP. Many gRPC users are actively experimenting with this option by implementing their own custom MCP servers. At Google Cloud, we use gRPC extensively to enable services and offer APIs at a global scale, and we’re committed to sharing the technology and expertise that has resulted from this pervasive use of gRPC. Specifically, we’re committed to supporting gRPC practitioners in their journey to adopt MCP in production, and we’re actively working with the MCP community to explore mechanisms to support gRPC as a transport for MCP. The MCP core maintainers have arrived at an agreement to support pluggable transports in the MCP SDK, and in the near future, Google Cloud will contribute and distribute a gRPC transport package to be plugged into the MCP SDKs. A community-backed transport package will enable gRPC practitioners to deploy MCP with gRPC in a consistent and interoperable manner.

The use of gRPC as a transport avoids the need for transcoding and helps maintain operational consistency for environments that are actively using gRPC. In the rest of this post, we explore the benefits of using gRPC as a transport for MCP and how Google Cloud is supporting this journey.

The choice of RPC transport

For organizations already using gRPC for their services, gRPC support allows them to continue to use their existing tooling to access services via MCP without altering the services or implementing transcoding proxies. These organizations are on a journey to keep the benefits of gRPC as MCP becomes the mechanism for agents to access services.

“Because gRPC is our standard protocol in the backend, we have invested in experimental support for MCP over gRPC internally. And we already see the benefits: ease of use and familiarity for our developers, and reducing the work needed to build MCP servers by using the structure and statically typed APIs.” - Stefan Särne, Senior Staff Engineer and Tech Lead for Developer Experience, Spotify

Benefits of gRPC

Using gRPC as a transport aligns MCP with the best practices of modern gRPC-based distributed systems, improving performance, security, operations, and developer productivity.

Performance and efficiency

The performance advantages of gRPC provide a big boost in efficiency, thanks to the following attributes:

Binary encoding (protocol buffers): gRPC uses protocol buffers (Protobufs) for binary encoding, shrinking message sizes by up to 10x compared to JSON. This means less bandwidth consumption and faster serialization/deserialization, which translates to lower latency for tool calls, reduced network costs, and a much smaller resource footprint.
Full duplex bidirectional streaming: gRPC natively supports the client (the agent) and the server (the tool), sending continuous data streams to each other simultaneously over a single, persistent connection. This feature is a game-changer for agent-tool interaction, opening the door to truly interactive, real-time agentic workflows without requiring application-level connection synchronization.
Built-in flow control (backpressure): gRPC includes native flow control to prevent a fast-sending tool from overwhelming the agent.

Enterprise-grade security and authorization

gRPC treats security as a first-class citizen, with enterprise-grade features built directly into its core, including:

Mutual TLS (mTLS): Critical for Zero Trust architectures, mTLS authenticates both the client and the gRPC-powered server, preventing spoofing and helping to ensure only trusted services communicate.
Strong authentication: gRPC offers native hooks for integrating with industry-standard token-based authentication (JWT/OAuth), providing verifiable identity for every AI agent.
Method-level authorization: You can enforce authorization policies directly on specific RPC methods or MCP tools (e.g., an agent is authorized to ReadFile but not DeleteFile), helping to ensure strict adherence to the principle of least privilege and combating "excessive agency."

Operational maturity and developer productivity

gRPC provides a powerful, integrated solution that helps offload resiliency measures and improves developer productivity through extensibility and reusability. Some of its capabilities include:

Unified observability: Native integration with distributed tracing (OpenTelemetry) and structured error codes provides a complete, auditable trail of every tool call. Developers can trace a single user prompt through every subsequent microservice interaction.
Robust resiliency: Features like deadlines, timeouts, and automatic flow control prevent a single unresponsive tool from causing system-wide failures. These features allow a client to specify a policy for a tool call that the framework automatically cancels if exceeded, preventing a cascading failure.
Polyglot development: gRPC generates code for 11+ languages, allowing developers to implement MCP Servers in the best language for the job while maintaining a consistent, strongly-typed contract.
Schema-based input validation: Protobuf's strict typing mitigates injection attacks and simplifies the development task by rejecting malformed inputs at the serialization layer.
Error handling and metadata: The framework provides a standardized set of error codes (e.g., UNAVAILABLE, PERMISSION_DENIED) for reliable client handling, and clients can send and receive out-of-band information as key-value pairs in metadata (e.g., for tracing IDs) without cluttering the main request.

Get started

As a founding member of the Agentic AI Foundation and a core contributor to the MCP specification, Google Cloud, along with other members of the community, has championed the inclusion of pluggable transport interfaces in the MCP SDK. Participate and communicate your interest in having gRPC as a transport for MCP:

Express your interest in enabling gRPC as an MCP transport. Contribute to the active pull request for pluggable transport interfaces for the Python MCP SDK.
Join the community that is shaping the future of communications for AI and help advance the Model Context Protocol. Contributor Communication - Model Context Protocol.
Contact us. We want to learn from your experience and support your journey.

Connect your enterprise data to Google’s new Antigravity IDE

Mon, 15 Dec 2025 17:00:00 +0000

The AI state of the art is shifting rapidly from simple chat interfaces to autonomous agents capable of planning, executing, and refining complex workflows. In this new landscape, the ability to ground these intelligent agents in your enterprise data is key to unlocking true business value. Google Cloud is at the forefront of this shift, empowering you to build robust, data-driven applications quickly and accurately.

Last month, Google announced Antigravity, an AI-first integrated development environment (IDE). And now, you can now give the AI agents you build in Antigravity direct, secure access to the trusted data infrastructure that powers your organization, turning abstract reasoning into concrete, data-aware action. With Model Context Protocol (MCP) servers powered by MCP Toolbox for Databases now available within Antigravity, you can securely connect your AI agents to services like AlloyDB for PostgreSQL, BigQuery, Spanner, Cloud SQL, Looker and others within Google’s Data Cloud, all within your development workflow.

Why use MCP in Antigravity?

We designed Antigravity to keep you in the flow, but the power of an AI agent is limited by what it "knows." To build truly useful applications, your agent needs to understand your data. MCP acts as the universal translator. You can think of it like a USB-C port for AI. It allows the LLMs in your IDE to plug into your data sources in a standardized way. By integrating pre-built MCP servers directly into Antigravity, you don’t need to perform any manual configuration. Your agents can now converse directly with your databases, helping you build and iterate faster without ever leaving the IDE.

Getting started with MCP servers

In Antigravity, connecting an agent to your data is a UI-driven experience, eliminating the challenges we’ve all faced when wrestling with complex configuration files just to get a database connection running. Here’s how to get up and running.

1. Discover and launch

You can find MCP servers for Google Cloud in the Antigravity MCP Store. Search for the service you need, such as "AlloyDB for PostgreSQL" or "BigQuery," and click on Install to start the setup process.

Launching the Antigravity MCP store

2. Configure your connection

Antigravity presents a form where you can add your service details such as Project ID and region. You can also enter your password or have Antigravity use your Identity and Access Management (IAM) credentials for additional security. These are stored securely, so your agent can access the tools it needs without exposing raw secrets in your chat window.

Installing the AlloyDB for PostgreSQL MCP Server

See your agents in action

Once connected to Antigravity, your agent gains a suite of "tools" (executable functions) that it can use to assist you, and help transform your development and observability experience across different services. Let’s take a look at a couple of common scenarios.

Streamlining database tasks with AlloyDB for PostgreSQL

When building against a relational database like PostgreSQL, you may spend time switching between your IDE and a SQL client to check schema names or test queries. With the AlloyDB MCP server, your agent handles that context and gains the ability to perform database administration and generate high-quality SQL code you can include in your apps — all within the Antigravity interface.

For example:

Schema exploration: The agent can use list_tables and get_table_schema to read your database structure and explain relationships to you instantly.
Query development: Ask the agent to "Write a query to find the top 10 users," and it can use execute_sql to run it and verify the results immediately.
Optimization: Before you commit code, use the agent to run get_query_plan to ensure your logic is performant.

Antigravity agent using the MCP tools

Unlocking analytics with BigQuery

For data-heavy applications, your agent can act as a helpful data analyst. Leveraging the BigQuery MCP server, it can, for example:

Forecast: Use forecast to predict future trends based on historical data.
Search the catalog: Use search_catalog to discover and manage data assets.
Augmented analytics: Use analyze_contribution to understand the impact of different factors on data metrics.

Building on truth with Looker

Looker acts as your single source of truth for business metrics. Looker’s MCP server allows your agent to bridge the gap between code and business logic, for example:

Ensuring metric consistency: No more guessing whether a field is named total_revenue or revenue_total. Use get_explores and get_dimensions to ask your agent, "What is the correct measure for Net Retention?" and receive the precise field reference from the semantic model.
Instantly validating logic: Don't wait to deploy a dashboard to test a theory. Use run_query to execute ad-hoc tests against the Looker model directly in your IDE, so that your application logic matches the live data.
Auditing reports: Use run_look to pull results from existing saved reports, allowing you to verify that your application's output aligns with official business reporting.

Build with data in Antigravity

By integrating Google’s Data Cloud MCP servers into Antigravity, it’s easier than ever to use AI to discover insights and develop new applications. Now, with access to a wide variety of data sources that run your business, get ready to take the leap from simply talking to your code, to creating new experiences for your users.

To get started, check out the following resources:

Documentation: Connecting to AlloyDB using MCP
GitHub: MCP Toolbox for Databases

Streamline the design and deployment of application infrastructure with Application Design Center, now GA

Mon, 08 Dec 2025 17:00:00 +0000

Earlier this year, we unveiled a big investment in platform and developer team productivity, with the launch of Application Design Center, helping them streamline the design and deployment of cloud application infrastructure, while ensuring applications are secure, reliable, and aligned with best practices. And today, Application Design Center is generally available.

We built Application Design Center to put applications at the center of your cloud experience, with a visual, canvas-style and AI-powered approach to design and modify Terraform-backed application templates. It also offers full lifecycle management that’s aligned with DevOps best practices across application design and deployment.

Application Design Center is a core component of our application-centric cloud experience. When you use Application Design Center to design and deploy your application infrastructure, your applications are easily discoverable, observable, and manageable. Application Design Center works in concert with App Hub to automatically register application deployments, enabling a unified view and control plane for your application portfolio, and Cloud Hub, to provide operational insights for your applications.

“Google Application Design Center is a valuable enabler for Platform Engineering, providing a structured approach to harmonizing resource creation in Google Cloud Platform. By aligning tools, processes, and technologies, it streamlines workflows, reducing friction between development, operations, and other teams. This harmonization enhances collaboration, accelerates delivery, and ensures consistency across Google Cloud environments.” - Ervis Duraj, Principal Engineer, MediaMarktSaturn Technology

The gateway to an app-centric cloud

Our goal with Application Design Center is for you to innovate more, and administer less. It consists of four key elements to help you minimize administrative overhead and maximize efficiency, so you can design and deploy applications with integrated best practices and essential guardrails. Let’s take a closer look.

1. Terraform components and application templates
Develop applications faster with our growing library of opinionated application templates. These provide well-architected patterns and pre-built components, including innovative "AI inference templates" to help you leverage AI to create dynamic and intelligent application foundations. As an example, at launch, Application Design Center provides opinionated templates for Google Kubernetes Engine (GKE) clusters (Standard, Autopilot and NodePool) to run AI inference workloads using a variety of LLM models, as well as for enterprise-grade production clusters or single-region web app clusters.

You can also ingest and manage your existing Terraform configurations (“Bring your own Terraform”) directly from Git repositories. Once imported, you can use Application Design Center to design with your own Terraform, or in combination with Google-provided Terraform, to create standardized, opinionated infrastructure patterns for sharing and reuse across your application teams.

2. AI-powered design for rapid application designing and prototyping
Application Design Center integrates with Google's Gemini Cloud Assist Design Agent, empowering you to design actual, deployable application infrastructure application templates on Google Cloud that you can export as Terraform infrastructure-as-code.

With Gemini Cloud Assist, you can describe your application design intents using natural language. In return, Gemini interactively generates multi-product application template suggestions, complete with visual architecture diagrams and summarized benefits. You can then refine these proposals through multi-turn reasoning or by directly manipulating the architecture within the Application Design Center canvas.

Additionally, all designs that you create with Gemini are automatically observable, optimizable, and enabled for troubleshooting assistance during runtime, thanks to their tight integration with Gemini Cloud Assist.

3. A secure, sharable catalog of application templates with full lifecycle management
Platform admins can curate a collection of application templates built from Google's best-practice components. This provides developers a trusted, self-service experience from which they can quickly discover and deploy compliant applications. Tight integration with Cloud Hub transforms these governed templates into a live operational command center, complete with unified visibility into the health and deployment status of the resulting applications. This closes the critical loop between design and runtime, so that your production environments reflect your organization’s approved architectural standards.

Also, Application Design Center’s robust application template revisions serve as an immutable audit trail. It automatically detects and flags configuration drift between your intended designs and deployed applications, so that developers can remediate unauthorized changes or safely push approved configuration updates. This helps ensure continuous state consistency and compliance from Day 1 and through the subsequent evolution of your application.

4. GitOps integration automating developers’ day-to-day software design lifecycle tasks
By integrating Application Design Center into existing CI/CD workflows, platform teams empower developers to own the complete software delivery lifecycle right from their IDE. Developers can leverage compliant application and infrastructure (IaC) code using Application Design Center application templates.

Further, every infrastructure decision made through Application Design Center is committed to code, versioned, and auditable. Specifically, developers can download the application IaC template from Application Design Center and import it into their app repos (the single source of truth), clone their repo, and edit the Terraform directly in their local IDEs. Any modifications go through a Git pull request for review. Once approved, this automatically triggers the existing CI/CD setup to build, test, and deploy both app and infra changes in lockstep. This unified approach minimizes friction, enforcing "golden paths" and providing an end-to-end automated pathway from a line of code in the IDE to a fully deployed change in production.

What's new since preview

This GA launch is packed with features that users have been asking for. We’re excited to share powerful new capabilities: enterprise-grade governance and security with public APIs and gcloud CLI support; full compatibility with VPC service controls; bring your own Terraform and GitOps support for integration with your existing application patterns and automation pipelines; agentic application patterns using GKE templates (Standard, Autopilot and NodePool); and finally, a simplified onboarding experience with app-managed project support, making Application Design Center an AI-powered engine for your applications on Google Cloud.

Get started today

To help you get started, Google provides a growing library of curated Google application templates built by experts. These templates combine multiple Google Cloud products and best practices to serve common use cases, which you can configure for deployment, and view as infrastructure as code in-line. Platform teams can then create and securely share the catalogs and collaborate with teammates on designs and self-service deployment for developers. For enterprises with existing Terraform patterns and assets, Application Design Center interoperates by enabling their import and reuse within its native design and configuration experience.

Ready to experience the power of Application Design Center? You can learn more about ADC and get started building in minutes using the quickstart. You can start building your first AI-powered application template in minutes, free of cost, and quickly deploy applications with working code. For deeper insights, explore the comprehensive public documentation here. We can't wait to see how you innovate with the Application Design Center!

Responding to CVE-2025-55182: Secure your React and Next.js workloads

Wed, 03 Dec 2025 23:00:00 +0000

Editor's note: This blog was updated on Dec. 4, 5, 7, and 12, 2025, with additional guidance on Cloud Armor WAF rule syntax, and WAF enforcement across App Engine Standard, Cloud Functions, and Cloud Run.

Earlier today, Meta and Vercel publicly disclosed two vulnerabilities that expose services built using the popular open-source frameworks React Server Components (CVE-2025-55182) and Next.js to remote code execution risks when used for some server-side use cases. At Google Cloud, we understand the severity of these vulnerabilities, also known as React2Shell, and our security teams have shared their recommendations to help our customers take immediate, decisive action to secure their applications.

Vulnerability background

The React Server Components framework is commonly used for building user interfaces. On Dec. 3, 2025, CVE.org assigned this vulnerability as CVE-2025-55182. The official Common Vulnerability Scoring System (CVSS) base severity score has been determined as Critical, a severity of 10.0.

Vulnerable versions: React 19.0, 19.1.0, 19.1.1, and 19.2.0
Patched in React 19.2.1
Fix: https://github.com/facebook/react/commit/7dc903cd29dac55efb4424853fd0442fef3a8700
Announcement: https://react.dev/blog/2025/12/03/critical-security-vulnerability-in-react-server-components

Next.js is a web development framework that depends on React, and is also commonly used for building user interfaces. (The Next.js vulnerability was referenced as CVE-2025-66478 before being marked as a duplicate.)

Vulnerable versions: Next.js 15.x, Next.js 16.x, Next.js 14.3.0-canary.77 and later canary releases
Patched versions are listed here.
Fix: https://github.com/vercel/next.js/commit/6ef90ef49fd32171150b6f81d14708aa54cd07b2
Announcement: https://nextjs.org/blog/CVE-2025-66478

Google Threat Intelligence Group (GTIG) has also published a new report to help understand the specific threats exploiting React2Shell.

We strongly encourage organizations who manage environments relying on the React and Next.js frameworks to update to the latest version, and take the mitigation actions outlined below.

Mitigating CVE-2025-55182

We have created and rolled out a new Cloud Armor web application firewall (WAF) rule designed to detect and block exploitation attempts related to CVE-2025-55182. This new rule is available now and is intended to help protect your internet-facing applications and services that use global or regional Application Load Balancers. We recommend deploying this rule as a temporary mitigation while your vulnerability management program patches and verifies all vulnerable instances in your environment.

For customers using App Engine Standard, Cloud Functions, Cloud Run, Firebase Hosting or Firebase App Hosting, we provide an additional layer of defense for serverless workloads by automatically enforcing platform-level WAF rules that can detect and block the most common exploitation attempts related to CVE-2025-55182.

For Project Shield users, we have deployed WAF protections for all sites and no action is necessary to enable these WAF rules. For long-term mitigation, you will need to patch your origin servers as an essential step to eliminate the vulnerability (see additional guidance below).

Cloud Armor and the Application Load Balancer can be used to deliver and protect your applications and services regardless of whether they are deployed on Google Cloud, on-premises, or on another infrastructure provider. If you are not yet using Cloud Armor and the Application Load Balancer, please follow the guidance further down to get started.

While these platform-level rules and the optional Cloud Armor WAF rules (for services behind an Application Load Balancer) help mitigate the risk from exploits of the CVE, we continue to strongly recommend updating your application dependencies as the primary long-term mitigation.

Deploying the cve-canary WAF rule for Cloud Armor

To configure Cloud Armor to detect and protect from CVE-2025-55182, you can use the cve-canary preconfigured WAF rule leveraging the new ruleID that we have added for this vulnerability. This rule is opt-in only, and must be added to your policy even if you are already using the cve-canary rules.

In your Cloud Armor backend security policy, create a new rule and configure the following match condition:

code_block: <ListValue: [StructValue([('code', "(has(request.headers['next-action']) || has(request.headers['rsc-action-id']) || request.headers['content-type'].contains('multipart/form-data') || request.headers['content-type'].contains('application/x-www-form-urlencoded')) && evaluatePreconfiguredWaf('cve-canary',{'sensitivity': 0, 'opt_in_rule_ids': ['google-mrs-v202512-id000001-rce','google-mrs-v202512-id000002-rce']})"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f4863aaa040>)])]>

This can be accomplished from the Google Cloud console by navigating to Cloud Armor and modifying an existing or creating a new policy.

Cloud Armor rule creation in the Google Cloud console.

Alternatively, the gcloud CLI can be used to create or modify a policy with the requisite rule:

code_block: <ListValue: [StructValue([('code', 'gcloud compute security-policies rules create PRIORITY_NUMBER \\\r\n --security-policy SECURITY_POLICY_NAME \\\r\n --expression "(has(request.headers[\'next-action\']) || has(request.headers[\'rsc-action-id\']) || request.headers[\'content-type\'].contains(\'multipart/form-data\') || request.headers[\'content-type\'].contains(\'application/x-www-form-urlencoded\')) && evaluatePreconfiguredWaf(\'cve-canary\',{\'sensitivity\': 0, \'opt_in_rule_ids\': [\'google-mrs-v202512-id000001-rce\',\'google-mrs-v202512-id000002-rce\']})" \\\r\n --action=deny-403'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f4863aaa250>)])]>

Additionally, if you are managing your rules with Terraform, you may implement the rule via the following syntax:

code_block: <ListValue: [StructValue([('code', 'rule {\r\n action = "deny(403)"\r\n priority = "PRIORITY_NUMBER"\r\n match {\r\n expr {\r\n expression = "(has(request.headers[\'next-action\']) || has(request.headers[\'rsc-action-id\']) || request.headers[\'content-type\'].contains(\'multipart/form-data\') || request.headers[\'content-type\'].contains(\'application/x-www-form-urlencoded\')) && evaluatePreconfiguredWaf(\'cve-canary\',{\'sensitivity\': 0, \'opt_in_rule_ids\': [\'google-mrs-v202512-id000001-rce\',\'google-mrs-v202512-id000002-rce\']})"\r\n }\r\n }\r\n description = "Applies protection for CVE-2025-55182 (React/Next.JS)"\r\n }'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f4863aaa2b0>)])]>

Verifying WAF rule safety for your application and consuming telemetry

Cloud Armor rules can be configured in preview mode, a logging-only mode to test or monitor the expected impact of the rule without Cloud Armor enforcing the configured action. We recommend that the new rule described above first be deployed in preview mode in your production environments so that you can see what traffic it would block.

Once you verify that the new rule is behaving as desired in your environment, then you can disable preview mode to allow Cloud Armor to actively enforce it.

Cloud Armor per-request WAF logs are emitted as part of the Application Load Balancer logs to Cloud Logging. To see what Cloud Armor’s decision was on every request, load balancer logging first needs to be enabled on a per backend service basis. Once it is enabled, all subsequent Cloud Armor decisions will be logged and can be found in Cloud Logging by following these instructions.

Interaction of Cloud Armor rules with vulnerability scanning tools

There has been a proliferation of scanning tools designed to help identify vulnerable instances of React and Next.js in your environments. Many of those scanners are designed to identify the version number of relevant frameworks in your servers and do so by crafting a legitimate query and inspecting the response from the server to detect the version of React and Next.js that is running.

Our WAF rule is designed to detect and prevent exploit attempts of CVE-2025-55182. As the scanners discussed above are not attempting an exploit, but sending a safe query to elicit a response revealing indications of the version of the software, the above Cloud Armor rule will not detect or block such scanners.

If the findings of these scanners indicate a vulnerable instance of software protected by Cloud Armor, that does not mean that an actual exploit attempt of the vulnerability will successfully get through your Cloud Armor security policy. Instead, such findings mean that the version React or Next.js detected is known to be vulnerable and should be patched.

How to get started with Cloud Armor for new users

If your workload is already using an Application Load Balancer to receive traffic from the internet, you can configure Cloud Armor to protect your workload from this and other application-level vulnerabilities (as well as DDoS attacks) by following these instructions.

If you are not yet using an Application Load Balancer and Cloud Armor, you can get started with the external Application Load Balancer overview, the Cloud Armor overview, and the Cloud Armor best practices.

If your workload is using Cloud Run, Cloud Run functions, or App Engine and receives traffic from the internet, you must first set up an Application Load Balancer in front of your endpoint to leverage Cloud Armor security policies to protect your workload. You will then need to configure the appropriate controls to ensure that Cloud Armor and the Application Load Balancer can’t be bypassed.

Best practices and additional risk mitigations

Once you configure Cloud Armor, we recommend consulting our best practices guide. Be sure to account for limitations discussed in the documentation to minimize risk and optimize performance while ensuring the safety and availability of your workloads.

Serverless platform protections

Google Cloud is enforcing platform-level protections across App Engine Standard, Cloud Functions, and Cloud Run to automatically help protect against common exploit attempts of CVE-2025-55182. This protection supplements the protections already in place for Firebase Hosting and Firebase App Hosting.

What this means for you:

Applications deployed to those serverless services benefit from these WAF rules that are enabled by default to help provide a base level of protection without requiring manual configuration.
These rules are designed to block known malicious payloads targeting this vulnerability.

Important considerations:

Patching is still critical: These platform-level defenses are intended to be a temporary mitigation. The most effective long-term solution is to update your application's dependencies to non-vulnerable versions of React and Next.js, and redeploy them.
Potential impacts: While unlikely, if you believe this platform-level filtering is incorrectly impacting your application's traffic, please contact Google Cloud Support and reference issue number 465748820.

Long-term mitigation: Mandatory framework update and redeployment

While WAF rules provide critical frontline defense, the most comprehensive long-term solution is to patch the underlying frameworks.

While Google Cloud is providing platform-level protections and Cloud Armor options, we urge all customers running React and Next.js applications on Google Cloud to immediately update their dependencies to the latest stable versions (React 19.2.1 or the relevant version of Next.js listed here), and redeploy their services.

This applies specifically to applications deployed on:

Cloud Run, Cloud Run functions, or App Engine: Update your application dependencies with the updated framework versions and redeploy.
Google Kubernetes Engine (GKE): Update your container images with the latest framework versions and redeploy your pods.
Compute Engine: The public OS images provided by Google Cloud do not have React or Next.js packages installed by default. If you have installed a custom OS with the affected packages, update your workloads to include the latest framework versions and enable WAF rules in front of all workloads.
Firebase: If you’re using Cloud Functions for Firebase, Firebase Hosting, or Firebase App Hosting, update your application dependencies with the updated framework versions and redeploy. Firebase Hosting and App Hosting are also automatically enforcing a rule to limit exploitation of CVE-2025-55182 through requests to custom and default domains.

Patching your applications is an essential step to eliminate the vulnerability at its source and ensure the continued integrity and security of your services.

We will continue to monitor the situation closely and provide further updates and guidance as necessary. Please refer to our official Google Cloud Security advisories for the most current information and detailed steps.

If you have any questions or require assistance, please contact Google Cloud Support and reference issue number 465748820.

From interaction to insight: Announcing BigQuery Agent Analytics for the Google ADK

Thu, 20 Nov 2025 17:00:00 +0000

In a world of agentic AI, building an agent is only half the battle. The other half is understanding how users are interacting with it. What are their most common requests? Where do they get stuck? What paths lead to successful outcomes? Answering these questions is the key to refining your agent and delivering a better user experience. These insights are also super critical for optimizing agent performance.

Today, we're making it easier for agent developers in Google’s Agent Development Kit (ADK) to answer these questions. With a single line of code, ADK developers can stream agent interaction data directly to BigQuery and get insights into their agent activity in a scalable manner. To do so, we are introducing BigQuery Agent Analytics, a new plugin for ADK that exports your agent's interaction data directly into BigQuery to capture, analyze, and visualize agent performance, user interaction, and cost.

With your agent interaction data centralized in BigQuery, analyzing critical metrics such as latency, token consumption, and tool usage is straightforward. Creating custom dashboards in tools like Looker Studio or Grafana is easy. Furthermore, you can leverage cutting-edge BigQuery capabilities including generative AI functions, vector search, and embedding generation, to perform sophisticated analysis. This enables you to cluster agent interactions, precisely gauge agent performance, and rapidly pinpoint common user queries or systemic failure patterns — all of which are essential for refining the agent experience. You can also join interaction data with relevant business datasets — for instance, linking support agent interactions with CSAT scores — to accurately measure the agent's real-world impact. This entire capability is unlocked with a minimal code change.

This plugin is available in preview for ADK users today, with support for other agent frameworks soon to follow.

See the plugin in action in the following video.

Understanding BigQuery Agent Analytics

The BigQuery Agent Analytics plugin is a very lightweight way of streaming various agent activity data directly to your BigQuery table. It consists of three main components:

ADK Plugin: With a single line of code, the new ADK plugin can stream agent activity like requests, responses, LLM tool calls, etc. to a BigQuery table.
Predefined BigQuery schema: We provide an optimized table schema out-of-the-box that stores rich details about user interactions, agent responses, and tool usage.
Low-cost, high-performance streaming: The plugin uses the BigQuery Storage Write API to stream events directly to BigQuery in real-time.

Why it matters: Data-driven agent development

By integrating your agent's analytic data in BigQuery, you can go from viewing basic metrics to generating deep, actionable insights. Specifically, this integration lets you:

Visualize agent usage and interactions: Gain a clear understanding of your agent's performance. Easily track key operational metrics like token consumption and tool usage to monitor costs and resource allocation.
Evaluate agent quality with advanced AI: Go beyond simple metrics by using BigQuery's advanced AI capabilities. Leverage AI functions and vector search to perform quality analysis on conversation data, identifying areas for improvement with greater precision.
Learn by conversing with your agent data: Create a conversational data agent that works directly with your new observability data. This allows you and your team to ask questions about your agent activity in natural language and get immediate insights, without writing complex queries.

How It works

We've designed the process of setting up robust analytics pipeline to be as simple as possible:

1. Add the required code: This plugin requires use of ADK’s application(apps) component when building the agent. The following code demonstrates how to initialize the new plugin and make it part of your app.

code_block: <ListValue: [StructValue([('code', '# --- Initialize the Plugin ---\r\nbq_logging_plugin = BigQueryAgentAnalyticsPlugin(\r\n project_id=PROJECT_ID, \r\n dataset_id=DATASET_ID, \r\n table_id="agent_events" # Optional \r\n)\r\n\r\n# --- Initialize Model and the root agent ---\r\nllm = Gemini(\r\n model="gemini-2.5-flash",\r\n)\r\n\r\nroot_agent = Agent(\r\n model=llm,\r\n name=\'my_adk_agent\',\r\n instruction="You are a helpful assistant"\r\n\r\n)\r\n\r\n# --- Create the App ---\r\napp = App(\r\n name="my_adk_agent",\r\n root_agent=root_agent,\r\n plugins=[bq_logging_plugin], # Register the plugin here\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f4865ad73a0>)])]>

2. Choose what to stream and customize pre-processing: You have full control over what data you send to BigQuery. Choose the specific events you want to stream, so that you only capture the data that is most relevant to your needs. The following code example redacts dollar amounts before logging.

code_block: <ListValue: [StructValue([('code', 'import json\r\nimport re\r\n\r\nfrom google.adk.plugins.bigquery_agent_analytics_plugin import BigQueryLoggerConfig\r\n\r\n\r\ndef redact_dollar_amounts(event_content: Any) -> str:\r\n """\r\n Custom formatter to redact dollar amounts (e.g., $600, $12.50)\r\n and ensure JSON output if the input is a dict.\r\n """\r\n text_content = ""\r\n if isinstance(event_content, dict):\r\n text_content = json.dumps(event_content)\r\n else:\r\n text_content = str(event_content)\r\n\r\n # Regex to find dollar amounts: $ followed by digits, optionally with commas or decimals.\r\n # Examples: $600, $1,200.50, $0.99\r\n redacted_content = re.sub(r\'\\$\\d+(?:,\\d{3})*(?:\\.\\d+)?\', \'xxx\', text_content)\r\n return redacted_content\r\n\r\nconfig = BigQueryLoggerConfig(\r\n enabled=True,\r\n event_allowlist=["LLM_REQUEST", "LLM_RESPONSE"], # Only log these events\r\n shutdown_timeout=10.0, # Wait up to 10s for logs to flush on exit\r\n client_close_timeout=2.0, # Wait up to 2s for BQ client to close\r\n max_content_length=500, # Truncate content to 500 chars (default)\r\n content_formatter=redact_dollar_amounts, # Redact the dollar amounts in the logging content\r\n)\r\n\r\nplugin = BigQueryAgentAnalyticsPlugin(..., config=config)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f4865ad7460>)])]>

And that’s it — the plugin handles the rest, including auto-creating the necessary BigQuery table with the correct schema, and streaming the agent data in real-time.

Now you are ready to analyze your agent metrics, using familiar BigQuery semantics. Here is an illustration of your logs as they appear in the BigQuery table using a “select * limit 10” on non-empty columns.

Get started today

It's time to unlock the full potential of your agents. With the new BigQuery Agent Analytics you can answer critical usage questions to refine your agent, optimize performance, and deliver a superior user experience.There is more to come in the near future, including integration with LangGraph to advanced analysis for multimodal agent interactions.

To get started, check out the Google Cloud BigQuery Agent Analytics documentation on the Google ADK site. For a guided walkthrough on using this plugin, we invite you to explore our comprehensive new codelab.

We’re excited to see the amazing, data-driven conversational experiences you build.

Introducing Agent Sandbox: Strong guardrails for agentic AI on Kubernetes and GKE

Tue, 11 Nov 2025 12:00:00 +0000

Google and the cloud-native community have consistently strengthened Kubernetes to support modern applications. At KubeCon EU 2025 earlier this year, we announced a series of enhancements to Kubernetes to better support AI inference. Today, at KubeCon NA 2025, we’re focused on making Kubernetes the most open and scalable platform for AI agents, with the introduction of Agent Sandbox.

Consider the challenge that AI agents represent. AI agents help applications go from answering simple queries to performing complex, multi-step tasks to achieve the users objective. Provided a request like “visualize last quarters sales data”, the agent has to use one tool to query the data and another to process that data into a graph and return to the user. Where traditional software is predictable, AI agents can make their own decisions about when and how to use tools at their disposal to achieve a user's objective, including generating code, using computer terminals and even browsers.

Without strong security and operational guardrails, orchestrating powerful, non-deterministic agents can introduce significant risks. Providing kernel-level isolation for agents that execute code and commands is non-negotiable. AI and agent-based workloads also have additional infrastructure needs compared to traditional applications. Most notably, they need to orchestrate thousands of sandboxes as ephemeral environments, rapidly creating and deleting them as needed while ensuring they have limited network access.

With its maturity, security, and scalability, we believe Kubernetes provides the most suitable foundation for running AI agents. Yet it still needs to evolve to meet the needs of agent code execution and computer use scenarios. Agent Sandbox is a powerful first step in that direction.

Strong isolation at scale

Agentic code execution and computer use require an isolated sandbox to be provisioned for each task. Further, users expect infrastructure to keep pace even as thousands of sandboxes are scheduled in parallel.

At its core, Agent Sandbox is a new Kubernetes primitive built with the Kubernetes community that’s designed specifically for agent code execution and computer use, delivering the performance and scale needed for the next generation of agentic AI workloads. Foundationally built on gVisor with additional support for Kata Containers for runtime isolation, Agent Sandbox provides a secure boundary to reduce the risk of vulnerabilities that could lead to data loss, exfiltration or damage to production systems. We’re continuing our commitment to open source, building Agent Sandbox as a Cloud Native Computing Foundation (CNCF) project in the Kubernetes community.

Enhanced performance on GKE

At the same time, you need to optimize performance as you scale your agents to deliver the best agent user-experience at the lowest cost. When you use Agent Sandbox on Google Kubernetes Engine (GKE), you can leverage managed gVisor in GKE Sandbox and the container-optimized compute platform to horizontally scale your sandboxes faster. Agent Sandbox also enables low-latency sandbox execution by enabling administrators to configure pre-warmed pools of sandboxes. With this feature, Agent Sandbox delivers sub-second latency for fully isolated agent workloads, up to a 90% improvement over cold starts.

The same isolation property that makes a sandbox safe, makes it more susceptible to compute underutilization. Reinitializing each sandbox environment with a script can be brittle and slow, and idle sandboxes often waste valuable compute cycles. In a perfect world, you could take a snapshot of running sandbox environments to start them from a specific state.

Pod Snapshots is a new, GKE-exclusive feature that enables full checkpoint and restore of running pods. Pod Snapshots drastically reduces startup latency of agent and AI workloads. When combined with Agent Sandbox, Pod Snapshots lets teams provision sandbox environments from snapshots, so they can start up in seconds. GKE Pod Snapshots supports snapshot and restore of both CPU- and GPU-based workloads, bringing pod start times from minutes down to seconds. With Pod Snapshots, any idle sandbox can be snapshotted and suspended, saving significant compute cycles with little to no disruption for end-users.

Built for AI engineers

Teams building today’s agentic AI or reinforcement learning (RL) systems should not have to be infrastructure experts. We built Agent Sandbox with AI engineers in mind, designing an API and Python SDK that lets them manage the lifecycle of their sandboxes, without worrying about the underlying infrastructure.

code_block: <ListValue: [StructValue([('code', 'from agentic_sandbox import Sandbox\r\n\r\n# The SDK abstracts all YAML into a simple context manager \r\nwith Sandbox(template_name="python3-template",namespace="ai-agents") as sandbox:\r\n\r\n # Execute a command inside the sandbox\r\n result = sandbox.run("print(\'Hello from inside the sandbox!\')")'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f4865aaee50>)])]>

This separation of concern enables both an AI developer-friendly experience and the operational control and extensibility that Kubernetes administrators and operators expect.

Get started today

Agentic AI represents a profound shift for software development and infrastructure teams. Agent Sandbox and GKE can help deliver the isolation and performance your agents need. Agent Sandbox is available in open source and can be deployed on GKE today. GKE Pod Snapshots is available in limited preview and will be available to all GKE customers later this year. To get started, check out the Agent Sandbox documentation and quick start. We are excited to see what you build!

Chaos engineering on Google Cloud: Principles, practices, and getting started

Mon, 13 Oct 2025 16:00:00 +0000

As engineers, we all dream of perfectly resilient systems — ones that scale perfectly, provide a great user experience, and never ever go down. What if we told you the key to building these kinds of resilient systems isn't avoiding failures, but deliberately causing them? Welcome to the world of chaos engineering, where you stress test your systems by introducing chaos, i.e., failures, into a system under a controlled environment. In an era where downtime can cost millions and destroy reputations in minutes, the most innovative companies aren't just waiting for disasters to happen — they're causing them and learning from the resulting failures, so they can build immunity to chaos before it strikes in production.

Chaos engineering is useful for all kinds of systems, but particularly for cloud-based distributed ones. Modern architectures have evolved from monolithic to microservices-based systems, often comprising hundreds or thousands of services. These complex service dependencies introduce multiple points of failure, and it’s difficult if not impossible to predict all the possible failure modes through traditional testing methods. When these applications are deployed on the cloud, they are deployed across multiple availability zones and regions. This increases the likelihood of failure due to the highly distributed nature of cloud environments and the large number of services that coexist within them.

A common misconception is that cloud environments automatically provide application resiliency, eliminating the need for testing. Although cloud providers do offer various levels of resiliency and SLAs for their cloud products, these alone do not guarantee that your business applications are protected. If applications are not designed to be fault-tolerant or if they assume constant availability of cloud services, they will fail when a particular cloud service they depend on is not available.

In short, chaos engineering can take a team's worst "what if?" scenarios and transform them into well-rehearsed responses. Chaos engineering isn’t about breaking systems — engineering chaotically, as it were — it's about building teams that face production incidents with the calm confidence that only comes from having weathered that chaos before, albeit in controlled conditions.

Google Cloud’s Professional Service Organization (PSO) Enterprise Architecture team consults on and provides hands-on expertise on customers’ cloud transformation journeys, including application development, cloud migrations, and enterprise architecture. And when advising on designing resilient architecture for cloud environments, we routinely introduce the principles and practices of chaos engineering and Site Reliability Engineering (SRE) practices.

In this first blog post in a series, we explain the basics of chaos engineering — what it is and its core principles and elements. We then explore how chaos engineering is particularly helpful and important for teams running distributed applications in the cloud. Finally, we’ll talk about how to get started, and point you to further resources.

Understanding chaos engineering

Chaos engineering is a methodology invented by Netflix in 2010 when it created and popularized ‘Chaos Monkey’ to address the need to build more resilient and reliable systems in the face of increasing complexity in their AWS environment. Around the same time, Google introduced Disaster Resilience Testing, or DiRT, which enabled continuous and automated disaster readiness, response, and recovery of Google’s business, systems, and data. Here on Google Cloud’s PSO team, we offer various services to help customers implement DiRT as part of SRE practices. These offerings also include training on how to perform DiRT on applications and systems operating on Google Cloud. The central concept is straightforward: deliberately introduce controlled disruptions into a system to identify vulnerabilities, evaluate its resilience, and enhance its overall reliability.

As a proactive discipline, chaos engineering enables organizations to identify weaknesses in their systems before they lead to significant outages or failures, where a system includes not only the technology components but also the people and processes of an organization. By introducing controlled, real-world disruptions, chaos engineering helps test a system's robustness, recoverability, and fault tolerance. This approach allows teams to uncover potential vulnerabilities, so that systems are better equipped to handle unexpected events and continue functioning smoothly under stress.

Principles and practices of chaos engineering

Chaos engineering is guided by a set of core principles about why it should be done, while practices define what needs to be done.

Below are the principles of chaos engineering:

Build a hypothesis around steady state: Prior to initiating any disruptive actions, you need to define what "normal" looks like for your system, commonly referred to as the "steady state hypothesis."
Replicate real-world conditions: Chaos experiments should emulate realistic failure scenarios that the system might encounter in a production environment.
Run experiments in production: Chaos engineering is firmly rooted in the belief that only a production environment with real traffic and dependencies can provide an accurate picture of resiliency. This is what separates chaos engineering from traditional testing.
Automate experiments: Make resiliency testing part of a continuous ongoing process rather than a one-off test.
Determine the blast radius: Experiments should be meticulously designed to minimize adverse impacts on production systems. This requires categorizing applications and services in different tiers based on the impact the experiments can have on customers and other applications and services.

With these principles established, follow these practices when conducting a chaos engineering experiment:

Define steady state: Identifies the specific metrics (e.g., latency, throughput) that you will look at and establish a baseline for them.
Formulate a hypothesis: This is the practice of creating a single testable statement, for example, ‘By deleting this container pod, user login will not be affected’. Hypotheses are generally created by identifying customer user journeys and deriving test scenarios from them.
Use a controlled environment: While one chaos engineering principle states that experiments need to run in production, you should still start small and run your experiment in a non-production environment first, learn and adjust, and then gradually expand the scope to production environment.
Inject failures: This is the practice of causing disruption by injecting failures either directly into the system (e.g., deleting a VM, stopping a database instance) or indirectly by injecting failures in the environment (e.g. deleting a network route, adding a firewall rule).
Automate experimental execution: Automation is crucial for establishing chaos engineering as a repeatable and scalable practice. This includes using automated tools for fault injection (e.g., making it part of a CI/CD pipeline) and automated rollback mechanisms.
Derive actionable insights: The primary objective of using chaos engineering is to gain insights into system vulnerabilities, thereby enhancing resilience. This involves rigorous analysis of experimental results; identifying weaknesses and areas for improvement; and disseminating findings to relevant teams to inform subsequent experimental design and system enhancements.

In other words, chaos engineering isn't about breaking things for the sake of it, but about building more resilient systems by understanding their limitations and addressing them proactively.

Elements of chaos engineering

Here are the core elements you'll use in a chaos engineering experiment, derived from these five principles:

Experiments: A chaos experiment constitutes a deliberate, pre-planned procedure wherein faults are introduced into a system to ascertain its response.
Steady-state hypotheses: A steady-state hypothesis defines the baseline operational state, or "normal" behavior, of the system under evaluation.
Actions: An action represents a specific operation executed upon the system being experimented on.
Probes: A probe provides a mechanism for observing defined conditions within the system during experimentation.
Rollbacks: An experiment may incorporate a sequence of actions designed to reverse any modifications implemented during the experiment.

Getting started with chaos engineering

Now that you have a good understanding of chaos engineering and why to use it in your cloud environment, the next step is to try it out for yourself in your own development environment.

There are multiple chaos engineering solutions in the market; some are paid products and some are open-source frameworks. To get started quickly, we recommend that you use Chaos Toolkit as your chaos engineering framework.

Chaos Toolkit is an open-source framework written in Python that provides a modular architecture where you can plug in other libraries (also known as ‘drivers’) to extend your chaos engineering experiments. For example, there are extension libraries for Google Cloud, Kubernetes, and many other technologies. Since Chaos Toolkit is a Python-based developer tool, you can begin by configuring your Python environment. You can find a good example of a Chaos Toolkit experiment and step-by-step explanation here.

Finally, to enable Google Cloud customers and engineers to introduce chaos testing in their applications, we’ve created a series of Google Cloud-specific chaos engineering recipes. Each recipe covers a specific scenario to introduce chaos in a particular Google Cloud service. For example, one recipe covers introducing chaos in an application/service running behind a Google Cloud internal or external application load balancer; another recipe covers simulating a network outage between an application running on Cloud Run and connecting to a Cloud SQL database by leveraging another Chaos Toolkit extension named ToxiProxy.

You can find a complete collection of recipes, including step-by-step instructions, scripts, and sample code, to learn how to introduce chaos engineering in your Google Cloud environment on GitHub. Then, stay tuned for subsequent posts, where we’ll talk about chaos engineering techniques, such as how to introduce faults into your Google Cloud environment.

Automate app deployment and security analysis with new Gemini CLI extensions

Wed, 10 Sep 2025 14:00:00 +0000

Find and fix security vulnerabilities. Deploy your app to the cloud. All without leaving your command-line.

Today, we’re closing the gap between your terminal and the cloud with a first look at the future of Gemini CLI, delivered through two new extensions: security extension and Cloud Run extension. These extensions are designed to handle critical parts of your workflows with simple, intuitive commands:

1) /security:analyze performs a comprehensive scan right in your local repository, with support for GitHub pull requests coming soon. This makes security a natural part of your development cycle.

2) /deploy deploys your application to Cloud Run, our fully managed serverless platform, in just a few minutes.

These commands are the first expression of a new extensibility framework for Gemini CLI. While we'll be sharing more about the full Gemini CLI extension world soon, we couldn't wait to get these capabilities into your hands. Consider this a sneak peak of what’s coming next!

Security extension: automate security analysis with /security:analyze

To help teams address software vulnerabilities early in the development lifecycle, we are launching the Gemini CLI Security extension. This new open-source tool automates security analysis, enabling you to proactively catch and fix issues using the /security:analyze command at the terminal or through a soon-coming GitHub Actions integration.

Integrated directly into your local development workflow and CI/CD pipeline, this extension:

Analyzes code changes: When triggered, the extension automatically takes the git diff of your local changes or pull request.
Identifies vulnerabilities: Using a specialized prompt and tools, Gemini CLI analyzes the changes for a wide range of potential vulnerabilities, such as hardcoded-secrets, injection vulnerabilities, broken access control, and insecure data handling.
Provides actionable feedback: Gemini returns a detailed, easy-to-understand report directly in your terminal or as a comment on your pull request. This report doesn't just flag issues; it explains the potential risks and provides concrete suggestions for remediation, helping you fix issues quickly and learn as you go.

And after the report is generated, you can also ask Gemini CLI to save it to disk or even implement fixes for each issue.

Getting started with /security:analyze

Integrating security analysis into your workflow is simple. First, download the Gemini CLI and install the extension (requires Gemini CLI v0.4.0+):

code_block: <ListValue: [StructValue([('code', 'gemini extensions install https://github.com/google-gemini/gemini-cli-security'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f4863b28610>)])]>

Then you can start run your first scan:

Locally: After making local changes, simply run /security:analyze in the Gemini CLI.
In CI/CD (Coming Soon): We're bringing security analysis directly into your CI/CD workflow. Soon, you’ll be able to configure the GitHub Action to automatically review pull requests as they are opened.

This is just the beginning. The team is actively working on further enhancing the extension's capabilities, and we are also inviting the community to contribute to this open source project by reporting bugs, suggesting features, continuously improving security practices and submitting code improvements.

For complete documentation and to contribute, visit the official GitHub repository.

Cloud Run extension: automate deployment with /deploy

The /deploy command in Gemini CLI automates the entire deployment pipeline for your web applications. You can now deploy a project directly from your local workspace. Once you issue the command, Gemini returns a public URL for your live application.

The /deploy command automates a full CI/CD pipeline to deploy web applications and cloud services from the command line using the Cloud Run MCP server. What used to be a multi-step process of building, containerizing, pushing, and configuring is now a single, intuitive command from within the Gemini CLI.

You can access this feature across three different surfaces – in Gemini CLI in the terminal, in VS Code via Gemini Code Assist agent mode, and in Gemini CLI in Cloud Shell.

Use /deploy command in Gemini CLI at the terminal to deploy application to Cloud Run

Get started with /deploy:

For existing Google Cloud users, getting started with /deploy is straightforward in Gemini CLI at the terminal:

Prerequisites: You'll need the gcloud CLI installed and configured on your machine and have an existing app or use Gemini CLI to create one.

Step 1: Install the Cloud Run extension
The /deploy command is enabled through a Model Context Protocol (MCP) server, which is included in the Cloud Run extension. To install the Cloud Run extension (Requires Gemini CLI v0.4.0+), run this command:

code_block: <ListValue: [StructValue([('code', 'gemini extensions install https://github.com/GoogleCloudPlatform/cloud-run-mcp'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f4863b28b50>)])]>

Step 2: Authenticate with Google Cloud
Ensure your local environment is authenticated to your Google Cloud account by running:

code_block: <ListValue: [StructValue([('code', 'gcloud auth login\r\ngcloud auth application-default login'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f4863b28c70>)])]>

Step 3: Deploy your app
Navigate to your application's root directory in your terminal and type gemini to launch Gemini CLI. Once inside, type /deploy to deploy your app to Cloud Run.

That's it! In a few moments, Gemini CLI will return a public URL where you can access your newly deployed application. You can also visit the Google Cloud Console to see your new service running in Cloud Run.

Besides Gemini CLI at the terminal, this feature can also be accessed in VS Code via Gemini Code Assist agent mode, powered by Gemini CLI, and in Gemini CLI in Cloud Shell, where the authentication step will be automatically handled out of the box.

Use /deploy command to deploy application to Cloud Run in VS Code via Gemini Code Assist agent mode.

Building a robust extension ecosystem

The Security and Cloud Run extensions are two of the first extensions from Google built on our new framework, which is designed to create a rich and open ecosystem for the Gemini CLI. We are building a platform that will allow any developer to extend and customize the CLI's capabilities, and this is just an early preview of the full platform's potential. We will be sharing a more comprehensive look at our extensions platform soon, including how you can start building and sharing your own.

Try Gemini CLI today, visit the GitHub here.

Simplify complex eventing at Scale with Eventarc Advanced

Fri, 29 Aug 2025 16:00:00 +0000

Modern application development requires organizations to invest not only in scale but also in simplification and central governance. This means more than message routing; it requires a simple, unified messaging platform that can intelligently filter, transform, and govern the flow of information in real-time, taming complexity all in one place.

Today, we are excited to announce the general availability of Eventarc Advanced, a unified, serverless eventing platform that goes beyond simple routing by combining real-time filtering, transformation, management, and delivery in one place — for a complex, multi-source event-driven architecture.

Evolving Eventarc to handle complexity

Eventarc Advanced is an evolution of Eventarc Standard and offers out-of-the-box integration patterns to simplify your eventing needs.

With Eventarc Advanced, organizations can

Integrate existing services using Publish API and leverage Google Cloud events to build sophisticated event-driven applications.
Centrally manage, secure, and observe the flow of messages across services with support for per-message fine-grained access control.
Intelligently route messages to appropriate destinations based on flexible message criteria.
Transform and convert events in real-time, with support for multiple payload formats and built-in capability to transform event attributes.
Publish to Google Cloud services using HTTP binding.

With Eventarc Advanced, you can build sophisticated eventing systems. In contrast, Eventarc Standard is best for simple one-to-one eventing needs involving Google Cloud events (comparison).

Eventarc Advanced’s key technical features include:

Publish API to ingest custom and third-party messages using CloudEvents format (details).
Message bus that acts as the central nervous system of your event-driven architecture, providing centralized observability, security and management. Message bus is based on Envoy and uses the policy engine of Cloud Load Balancers and Cloud Service Mesh.

Your existing systems can publish messages to a central message bus that can be intelligently routed to appropriate consumers based on flexible criteria. The message bus simplifies event management and reduces operational overhead.
You can gain insights into your message flows with centralized monitoring, logging, and tracing capabilities. Logs are captured in Cloud Logging, providing detailed information about event processing and errors.

Out-of-the-box event mediation capabilities to adapt messages on the fly without modifying your source or destination services, and to handle different events through support for multiple payload formats (Avro, JSON, Protobuf) and built-in capability to transform event attributes.

Eventarc Advanced incorporates error-handling by offering reliable event delivery and graceful recovery from transient failures.

Empowering developers and operators

We designed Eventarc Advanced to cater to the needs of both developers and operators:

“Simplicity” for developers: Focus on building your core application features, not on complex event routing logic. Eventarc Advanced provides a unified API and a consistent experience, letting you build decoupled, reliable, and scalable services including real-time transformations.
“Centralized governance” for platform operators: Simplify the setup and management of your eventing infrastructure. Centralized governance across projects / teams, plus monitoring and logging make it easier to identify and resolve issues, reducing operational overhead.

How Eventarc Advanced works

Imagine an order processing system where orders are created, payments are processed, and items are shipped. Each action is an "event," and in a complex system, managing this flow can be challenging. This is where Eventarc Advanced comes in. It provides a centralized way to manage, observe, and route all your application's events. Let's explore how it works.

Set up your message bus
At the heart of Eventarc Advanced is a message bus that acts as the central nervous system for your event-driven application. Every event, regardless of its origin, is sent to the message bus to be analyzed and routed. This central hub is where you can define security policies, controlling exactly who can send events and what kind are allowed.

In our example, you would create a message bus to receive all order-related events. Whether an order is newly created, its payment is confirmed, or its status changes to "shipped," the events land here.

Connect your event sources
Next, connect your sources that generate order events. Event sources are the services and applications that generate events and feed them into your message bus. Eventarc Advanced makes this easy, supporting a wide range of sources, including:

Google API events
External apps or custom systems via Publish API

In our example, the event source could be a custom service using the Publish API. Every time a new order is saved or an existing one is updated, it automatically sends an event to your message bus.

Configure pipelines and destinations
This is another area where Eventarc Advanced shines. With events flowing into your message bus, you can configure pipelines to intelligently route them to the correct destinations, allowing you to filter, transform, and direct events with precision.

In the above example,

New order notification: You can set up a filter that looks for events with status: "new". This pipeline routes these events to a notification service that sends an order confirmation email to the customer.
Fraud detection: For high-value orders (e.g., amount > $1000), you can apply a transformation and route it to a specialized fraud detection service for analysis.

Unlocking new possibilities

Eventarc Advanced opens up new possibilities for your applications and workflows:

Large-scale application integration: Connect numerous services and agents, enabling them to communicate asynchronously and reliably, even across different event formats and schemas.
Event streaming for AI and analytics: Handle the influx of data from IoT devices and AI workloads by filtering and transforming them before feeding them into your analytics pipelines.
Hybrid and multi-cloud deployments: Extend your event-driven architectures beyond Google Cloud, integrating with on-premises systems and other cloud providers.

What's next

As today’s applications become increasingly agentic, distributed and data-driven, the need for efficient and secure event orchestration is more critical than ever. With upcoming native support for Service Extensions to insert custom code into the data path and services like Model Armor, Eventarc Advanced’s message bus provides security and networking controls for agent communications.

Eventarc Advanced is available today. To learn more about Eventarc Advanced, see the documentation. To learn more about event-driven architectures, visit our Architecture Center based on Google Cloud best practices. Get ready to take your event-driven architectures to the next level!

Don’t just speculate, investigate! Gemini Cloud Assist now offers root-cause analysis

Fri, 22 Aug 2025 16:00:00 +0000

Debugging in a complex, distributed cloud environment can feel like searching for a needle in a haystack. The sheer volume of data, intertwined dependencies, and ephemeral issues make traditional troubleshooting methods time-consuming and often reactive. Just as modern software development demands more context for effective debugging, so too does cloud operations.

Gemini Cloud Assist, a key product in the Google Cloud with Gemini portfolio, simplifies the way you manage your applications with AI-powered assistance to help you design, deploy, and optimize your apps, so you can reach your efficiency, cost, reliability, and security goals.

Then there’s Gemini Cloud Assist investigations, a root-cause analysis (RCA) AI agent for troubleshooting infrastructure and applications that is now available in preview.

When you encounter an issue, you can initiate an investigation from various places like the Logs Explorer, Cloud Monitoring alerts, or directly from the Gemini chat panel. Cloud Assist then analyzes data from multiple sources, including logs, configurations, and metrics, to produce ranked and filtered "Observations" that provide insights into your environment's state. It synthesizes these observations to diagnose probable root causes, explains the context, and recommends the next steps or fixes to resolve the problem. If you need more help, your investigation, along with all its context, can be seamlessly transferred into a Google Cloud support case to expedite resolution with a support engineer.

How Gemini Cloud Assist investigations works

Gemini Cloud Assist investigations helps to find the root cause of an issue using a combination of capabilities:

Programmatic, proactive, and interactive access: Trigger or consume your investigation through API calls, chat menu, or UI for proactive or interactive troubleshooting.
Contextualization: Investigations discover the most relevant resources, data sources, and APIs in your environment to provide focused troubleshooting.
Comprehensive signal analysis: Investigations perform deep analysis in parallel across Cloud Logs, Cloud Asset Inventory, App Hub, Metrics, Errors, and Log Themes to uncover anomalies, configuration changes, performance bottlenecks, and recurring issues.
AI-powered insights and recommendations: Utilizing specialized knowledge sources, like Google Cloud support knowledgebases and internal runbooks, investigations generate probable root cause and actionable recommendations.
Interactive collaboration - Chat with and share investigations across your team for collaborative troubleshooting between you, your team, and Gemini Cloud Assist.
Handoff to Google Cloud Support: Convert your investigation directly to a support case without losing any time or context.

Programmatic, proactive, and interactive investigations

Early users are thrilled with the speed and effectiveness with which Cloud Assist investigations helps them troubleshoot and resolve tough problems.

"At ZoomInfo, maintaining uptime is critical, but equally important is ensuring our engineers can swiftly and effectively troubleshoot complex issues. By integrating Gemini Cloud Assist investigations early into our development process, we've accelerated troubleshooting across all levels of our engineering team. Engineers at every experience level can now rapidly diagnose and resolve problems, reducing some resolution times from hours to minutes. This efficiency enables our teams to spend more energy innovating and less time on reactive problem-solving. Gemini Cloud Assist investigations isn't just a troubleshooting aid; it's a key driver of productivity and innovation." - Yasin Senturk, DevOps Engineer at ZoomInfo

“I'm really impressed by how Gemini Cloud Assist Investigations feature in 2 minutes turned over with some valid suggestions on the potential root causes, and the first one being the actual culprit! I was able to mitigate the whole issue within an hour. Gemini Cloud Assist really saved my weekend!” - Chuanzhen Wu, SRE, Google Waze

Let’s walk through Gemini Cloud Assist investigations’ capabilities in a bit more detail.

Programmatic, proactive, and interactive access
You can start an investigation directly from various points within Google Cloud, such as error messages in Logs Explorer or specific product pages (like Google Kubernetes Engine or Cloud Run), or from the central Investigations page, where you can provide context like error messages, affected resources, and observation time. Gemini Cloud Assist investigations also provides an API, allowing you to integrate it into existing workflows such as Slack or other incident management tools. If the root cause of an issue requires further assistance, you can trigger a Google Cloud support case with the Investigation response so support engineers can proceed from that point.

Contextualization
Investigations can start with a natural language description, error message, log snippets, or any combination of information that you have about your issue. It starts by gathering the initial context related to your issue, then builds a topology of relevant resources and all the associated data sources that might provide insights to the root cause.

Investigations uses both public and private knowledge, playbooks informed by Google SRE and Google Cloud Support issues, and your topology, grounding itself in similar issues before generating a troubleshooting plan for your issue. This context becomes key in providing focused and comprehensive signal analysis.

Comprehensive signal analysis
Once the investigation runs, you’ll see the observations that it starts to collect from your project. The investigation goes beyond surface-level observations; it automatically analyzes critical data sources across your Google Cloud environment, including:

Google Cloud logs: Sifting through vast log data to identify anomalies and critical events
Cloud Asset Inventory: Understanding changes in your resource configurations and their potential impact
Metrics (coming soon): Correlating performance data to pinpoint resource exhaustion or unexpected behavior
Errors: Aggregating and categorizing errors to highlight patterns and recurring problems
Log themes: Identifying common patterns and themes within log data to provide a higher-level view of issues

AI-powered insights and recommendations
Observations are the basis of Gemini Cloud Assist investigations’ root-cause insights and recommendations. Leveraging Gemini’s analytical capabilities, Cloud Assist synthesizes observations from disparate data sources, ranking and filtering information to focus on the most relevant details. Crucially, investigations draw upon differentiated knowledge sources and publicly available documentation, such as extensive Google Cloud support troubleshooting knowledge and internal runbooks, to generate highly accurate and relevant insights and observations. It then generates:

Probable root cause: Provides clear hypotheses about the underlying cause of the issue, complete with contextual explanations
Actionable recommendations: Offers concrete next steps for troubleshooting or even direct fixes, helping you resolve incidents faster

Handoff to Google Support teams
If an issue proves particularly elusive, with the click of a button, investigations packages context, observations, and hypotheses into a support case, for faster issue resolution. This is why you might want to run an investigation before contacting Google support teams about an issue.

Get started with Gemini Cloud Assist investigations today

Ready to get to the root of your troubles faster? Try investigations now by investigating any error logs from the Log Explorer console. Or create an investigation directly and describe any issues you might be having.

How Renault Group is using Google’s software-defined vehicle industry solution

Wed, 16 Jul 2025 16:00:00 +0000

It’s funny to think of Renault Group, the massive European car manufacturer, as a software company, but in many ways, it is. Renault Group subsidiary Ampere Software Technology is dedicated to developing and integrating advanced software solutions for intelligent electric vehicles, aiming to create software-defined vehicles (SDVs) with enhanced customer experiences and new services.

Ampere develops Renault Group’s software-defined vehicle based on Google’s AAOS SDV solution. But like all software companies, it struggled to contain costs, sync code bases, maintain adequate testing regimens, and onboard new talent.

Building on the existing partnership between Google Cloud and Renault Group, Ampere chose a Google Cloud solution for its software-defined vehicle development. This solution, leveraging Google Cloud Workstations and Gemini Code Assist, effectively streamlined the process, making it more secure and productive by eliminating many common development hurdles.

For security-conscious enterprises, Cloud Workstations offer fully managed development environments. Concurrently, Gemini Code Assist, driven by Gemini 2.5, provides secure generative AI coding assistance and agents across the entire software development lifecycle. And by utilizing Google's virtual twin technology, specifically developed for Google’s AAOS SDV and AAOS IVI (Android Automotive OS for IVI), Ampere constructed full digital counterparts to their automobiles.

Let’s take a closer look at the components in this solution:

Google Cloud Workstations

Google Cloud Workstations significantly boosts Android Open Source Project (AOSP) developer productivity in general. In the context of Ampere, it offers on-demand development environments with persistent disks, pre-synced with the customer’s AAOS SDV and AAOS IVI repositories. This eliminates lengthy sync and build times, allowing developers to access their work from anywhere. Ampere's Platform Admins provision these workstations, drastically cutting down the time it takes for developers to become productive. Developers have instant access to powerful virtual machines with ample vCPUs, RAM, and fast SSD storage — important for the demanding emulators that they use. This resource elasticity prevents bottlenecks and accelerates development. Then, secure, authenticated cloud access and Google Cloud security tools helps to significantly reduce IP leaks and unauthorized access. Finally, having a consistent development environment prevents "works on my machine" problems and reduces debugging time, while flexible access and disk configurations enhance AAOS SDV and AAOS IVI developers’ productivity by enabling workstation access from anywhere, and preserving codebases, configurations, and build artifacts across sessions via persistent disks, eliminating repeated repo syncs.

aside_block: <ListValue: [StructValue([('title', 'Try Google Cloud for free'), ('body', <wagtail.rich_text.RichText object at 0x7f4862bdd850>), ('btn_text', 'Get started for free'), ('href', 'https://console.cloud.google.com/freetrial?redirectPath=/welcome'), ('image', None)])]>

Gemini Code Assist for AI-powered Android development

To help, Ampere offered their developers Android Studio and Code OSS IDEs integrated with Gemini Code Assist, helping to address code management complexity, reduce steep learning curves, and prevent errors. Gemini Code Assist uses retrieval-augmented generation (RAG) to access Ampere’s private codebases and documentation, providing relevant and accurate code suggestions tailored to their Android development standards and conventions. It sped up understanding of their vast codebases by explaining functions, summarizing modules, and suggesting next steps, benefiting new developers and those working on different parts of the SDV software. It also helped boost their Android development productivity by automating boilerplate code, suggesting APIs, and finding potential problems, letting developers concentrate on core SDV logic instead of repetitive tasks.

The virtual twin

Google Cloud enabled Ampere’s AAOS SDV developers and testers to use a "virtual twin" of a car, resolving resource and complexity issues associated with physical or poorly managed virtual testing. Developers can use powerful Compute Engine instances and specialized Android emulators like Cuttlefish to create accurate virtual vehicle embedded systems. This enables rigorous software testing with virtual hardware, helping to ensure robust performance before building physical prototypes. AAOS SDV developers can also use scalable virtual devices for parallel testing, comprehensive regression suites, and simulations, accelerating the "test" phase of the CI/CD pipeline and improving the SDV lifecycle. The virtual twin is integrated with the Cloud Workstations development environment and the customer’s CI/CD pipelines (e.g., powered by GKE and GitLab), allowing developers to build their AAOS SDV changes on their workstation, trigger automated tests on a fleet of virtual twins, and get immediate feedback on their code.

Tangible returns of modernized SDV development

By combining the robust, managed infrastructure of Cloud Workstations, Gemini Code Assist’s intelligent assistance, and virtual twins, Google Cloud is helping Renault modernize automotive software development, accelerate innovation, and bring new features to market at unprecedented speeds.

“...to invest and build the software platform for the software defined vehicle in Europe ..you need the tools and this is where Google shines ... At the heart of it is AI and when we talk about code generation, instantiation of things that you need to be running immediately versus waiting for the thing to compile and having be available to the developer … to make our engineers more efficient so we can do more with less time because we are challenged with time.” - Henry Bzeih, Ampere Chief Software Officer (Renault Group)

Google Cloud and Gemini Code Assist offer automotive OEMs a transformational shift, extending beyond mere tool adoption to significantly impacting business results. This transition enhances competitiveness, profitability, and innovation speed.

Traditionally, onboarding new developers takes days and is costly. AAOS SDV development often involves time-consuming setup, dependency management, and build system troubleshooting. By reducing environment setup time from days to minutes — including repository syncing and toolchain configuration — and utilizing AI assistance, the development process is vastly accelerated.

OEMs worry about intellectual property leaks from local devices. Cloud Workstations addresses this concern by operating within the customer's Virtual Private Cloud. This approach prevents source code from being synced locally and exposed on endpoints, reducing security risks.

While cloud infrastructure has associated costs, it yields substantial cost optimization. Eliminating high-end local machines, minimizing wasted developer time on environment management, and speeding up timelines all lower total development costs. The ability to quickly adjust cloud resources ensures payment only for active usage, avoiding idle hardware expenses, and a better return on engineering investment.

Finally, having a virtual twin of the car improves quality assurance and validation. Instead of relying on limited prototypes or unreliable local emulators, developers can use detailed virtual car models, facilitating faster iteration, scalable testing, early bug detection, and advanced scenario simulation.

Automotive companies are not only adopting new technologies but are also reshaping their development capabilities by utilizing Cloud Workstations and Gemini Code Assist. For more, watch the fireside chat with Henry Bzeih, Ampere Chief Software Officer (Renault Group) on their success with this SDV Industry solution.

And if you’re in the automotive industry, you can get started on setting up custom AAOS SDV or AAOS IVI development environments with Gemini Code Assist by referring to our GitHub repository. And of course, your Google Partner Engineering or Customer Engineering contacts are ready to help!

How Jina AI built its 100-billion-token web grounding system with Cloud Run GPUs

Fri, 11 Jul 2025 16:00:00 +0000

Editor’s note: The Jina AI Reader is a specialized tool that transforms raw web content from URLs or local files into a clean, structured, and LLM-friendly format. In this post, Han Xiao details how Cloud Run empowers Jina AI to build a secure, reliable, and massively scalable web scraping system that remains economically viable. This post explores the collaborative innovation, technical hurdles, and breakthrough achievements behind Jina Reader, a web grounding system now processing 100 billion tokens daily.

When Jina Reader launched in April 2024, its explosive growth — serving more than 10 million requests and 100 billion tokens daily — confirmed huge demand for reliable, LLM-friendly web content. Jina Reader isn't just another scraper; it takes a different approach to how AI systems consume web content by transforming raw, noisy web pages into clean, structured markdown.

The core challenge for any AI system processing web data is the "web grounding problem." Modern websites are a chaotic mix of content, ads, tracking scripts, and dynamic JavaScript, creating an overwhelming noise-to-signal ratio. Traditional scrapers struggle with this complexity, often failing on dynamic single-page applications or generating unusable, ungrounded data for LLMs. Jina Reader’s breakthrough, ReaderLM-v2, is a purpose-built 1.5-billion-parameter language model that intelligently extracts content, trained on millions of documents to understand web structure beyond simple rules.

FIgure 1: Jina Reader: a sophisticated browser automation system

Cloud Run: The engine behind Jina Reader's scale

Jina Reader faced inherent burstiness and unpredictability of web scraping workloads. Traditional virtual machine setups meant either costly over-provisioning or critical failures under load. Google Cloud Run became the essential solution, enabling Jina Reader to build a web scraping system that is secure, reliable, massively scalable, and economically viable.

The web grounding app (the browser automation system that scrapes and cleans web content) is hosted on Cloud Run (CPU). It runs full Chrome browser instances.
ReaderLM-v2 is a purpose-built 1.5-billion-parameter language model for HTML-to-markdown conversion that runs on Cloud Run with serverless GPUs.

Cloud Run directly addressed several critical issues:

Optimized Performance: The deep collaboration between Jina Reader and Google Cloud engineering was essential. We jointly optimized container lifecycle management for browser automation, reducing startup times from over 10 seconds to under two seconds through prewarming, optimized images, and intelligent resource allocation. For ReaderLM-v2, Google's team helped create custom container configurations to efficiently run a 1.5-billion-parameter model on Cloud Run GPUs. The on-demand scaling and fast start capabilities of Cloud Run GPUs were critical in helping optimize model performance, directly impacting our ability to process 100 billion tokens daily.

Figure 2: On-demand AI inference with Cloud Run GPUs (hosting ReaderLM-v2 model)

True Scale-to-Zero Serverless: Cloud Run's ability to run full Chrome browser instances allowed cost-effective operations. Each request spawns an isolated container with its own headless Chrome, and crucially, these containers disappear when the request is done. This ephemeral nature is vital for processing untrusted web content, mitigating security risks and memory leaks.
Global Multi-Regional Deployment: Cloud Run's global presence ensures requests are processed close to both the users and target websites. This significantly minimizes latency and boosts success rates, even against geo-restricted content.
Massive & Automatic Scaling: The platform seamlessly scales from a handful to over 1,000 container instances during peak traffic, handling the unpredictable nature of web scraping without manual intervention.
Economic Viability: With Cloud Run's pay-per-use model, Jina Reader can offer a generous free tier to end users while maintaining profitability even with substantial monthly usage. This pricing flexibility was fundamental to our widespread adoption.
Resilience and Operational Excellence: During a recent sustained DDoS attack, Cloud Run's serverless architecture proved invaluable. It scaled up to absorb massive loads (over 100,000 requests per minute), while intelligent rate limiting filtered malicious traffic. Critically, costs returned to normal immediately after the attack subsided due to its scale-to-zero capability. The system has maintained over 99.9% uptime.

Conclusion

Building Jina Reader on Google Cloud Run proved that AI capabilities and cloud-native architecture are complementary. Cloud Run's unique capabilities — serverless GPUs, container isolation, global deployment and scale-to-zero economics — made the architecture possible. Our close partnership demonstrates that deep integration between AI-first systems and modern cloud infrastructure can create capabilities previously thought impossible, enabling us to process 100 billion tokens every day.

You can discover more about Cloud Run GPUs on our product page, and if you want to learn how to host a large language model on Cloud Run, watch this video.

aside_block: <ListValue: [StructValue([('title', 'Try Google Cloud for free'), ('body', <wagtail.rich_text.RichText object at 0x7f4862e89700>), ('btn_text', 'Get started for free'), ('href', 'https://console.cloud.google.com/freetrial?redirectPath=/welcome'), ('image', None)])]>

Using Platform Engineering to simplify the developer experience - part one

Thu, 26 Jun 2025 16:00:00 +0000

Editor's note: This is part one of the story. After you’re finished reading, head over to part two.

In 2017, John Lewis, a major UK retailer with a £2.5bn annual online turnover, was hampered by its monolithic e-commerce platform. This outdated approach led to significant cross-team dependencies, cumbersome and infrequent releases (monthly at best), and excessive manual testing, all further hindered by complex on-premises infrastructure. What was needed were some bold decisions to drive a quick and significant transformation.

The John Lewis engineers knew there was a better way. Working with Google Cloud, they modernized their e-commerce operations with Google Kubernetes Engine. They started with the frontend, and started to see results fast: the frontend was moved onto Google Cloud in mere months, releases to the frontend browser journey started to happen weekly, and the business gladly backed expansion into other areas.

At the same time, the team had a broader strategy in mind: to take a platform engineering approach, creating many product teams who built their own microservices to replace the functionality of the legacy commerce engine, as well as creating brand new experiences for customers.

And so The John Lewis Digital Platform was born. The vision was to empower development teams and arm them with the tools and processes they needed to go to market fast, with full ownership of their own business services. The team’s motto? "You Build It. You Run It. You Own It." This decentralization of development and operational responsibilities would also enable the team to scale.

This article features insights from Principal Platform Engineer Alex Moss, who delves into their strategy, platform build, and key learnings of John Lewis’ journey to modernize and streamline its operations with platform engineering — so you can begin to think about how you might apply platform engineering to your own organization.

aside_block: <ListValue: [StructValue([('title', 'Try Google Cloud for free'), ('body', <wagtail.rich_text.RichText object at 0x7f48653890d0>), ('btn_text', 'Get started for free'), ('href', 'https://console.cloud.google.com/freetrial?redirectPath=/welcome'), ('image', None)])]>

Step 1: From monolithic to multi-tenant

In order to make this happen, John Lewis needed to adopt a multi-tenant architecture — one tenant for each business service, allowing each owning team to work independently without risk to others -- and thereby permitting the Platform team to give the team a greater degree of freedom.

Knowing that the business' primary objective was to greatly increase the number of product teams helped inform our initial design thinking, positioning ourselves to enable many independent teams even though we only had a handful of tenants.

This foundational design has served us very well and is largely unchanged now, seven years later. Central to the multi-tenant concept is what we chose to term a "Service" — a logical business application, usually composed of several microservices plus components for storing data.

We largely position our platform as a “bring your own container” experience, but encourage teams to make use of other Google Cloud services — particularly for handling state. Adopting services like Firestore and Pub/Sub reduces the complexity that our platform team has to work with, particularly for areas like resilience and disaster recovery. We also favor Kubernetes over compute products like Cloud Run because it strikes the right balance for us between enabling development teams to have freedom whilst allowing our platform to drive certain certain behaviours, e.g., the right level of guardrails, without introducing too much friction.

On our platform, Product Teams (i.e., tenants) have a large amount of control over their own Namespaces and Projects. This allows them to prototype, build, and ultimately operate, their workloads without dependency on others — a crucial element of enabling scale.

Our early-adopter teams were extremely helpful in helping evolve the platform; they were accepting of the lack of features and willing to develop their own solutions, and provided very rich feedback on whether we were building something that met their needs.

The first tenant to adopt the platform was rebuilding the johnlewis.com, search capability, replacing a commercial-off-the-shelf solution. This team was staffed with experienced engineers familiar with modern software development and the advantages of a microservice-based architecture. They quickly identified the need for supporting services for their application to store data and asynchronously communicate between their components. They worked with the Platform Team to identify options, and were onboard with our desire to lean into Google Cloud native services to avoid running our own databases or messaging. This led to us adopting Cloud Datastore and Pub/Sub for our first features that extended beyond Google Kubernetes Engine.

All roads lead to success

A risk with a platform that allows very high team autonomy is that it can turn into a bit of a wild-west of technology choices and implementation patterns. To handle this, but to do so in a way that remained developer-centric, we adopted the concept of a paved road, analogous to a “golden path.”

We found that the paved road approach made it easier to:

build useful platform features to help developers do things rapidly and safely
share approaches and techniques, and engineers to move between teams
demonstrate to the wider organisation that teams are following required practices (which we do by building assurance capabilities, not by gating release)

The concept of the paved road permeates most of what the platform builds, and has inspired other areas of the John Lewis Partnership beyond the John Lewis Digital space.

Our paved road is powered by two key features to enable simplification for teams:

The Paved Road Pipeline. This operates on the whole Service and drives capabilities such as Google Cloud resource provisioning and observability tools.
The Microservice CRD. As the name implies, this is an abstraction at the microservice level. The majority of the benefit here is in making it easier for teams to work with Kubernetes.

Whilst both features were created with the developer experience in mind, we discovered that they also hold a number of benefits for the platform team too.

The Paved Road Pipeline is driven by a configuration file — in yaml (of course!) — which we call the Service Definition. This allows the team that owns the tenancy to describe, through easy-to-reason-about configuration, what they would like the platform to provide for them. Supporting documentation and examples help them understand what can be achieved. Pushes to this file then drive a CI/CD pipeline for a number of platform-owned jobs, which we refer to as provisioners. These provisioners are microservices-like themselves in that they are independently releasable and generally focus on performing one task well. Here are some examples of our provisioners and what they can do:

Create Google Cloud resources in a tenant’s Project. For example, Buckets, PubSub, and Firestore — amongst many others
Configure platform-provided dashboards and custom dashboards based on golden-signal and self-instrumented metrics
Tune alert configurations for a given microservice’s SLOs, and the incident response behaviour for those alerts

Our product teams are therefore freed from the need to familiarize themselves deeply with how Google Cloud resource provisioning works, or Infrastructure-as-Code (IaC) tooling for that matter. Our preferred technologies and good practices can be curated by our experts, and developers can focus on building differentiating software for the business, while remaining fully in control of what is provisioned and when.

Earlier, we mentioned that this approach has the added benefit of being something that the platform team can rely upon to build their own features. The configuration updated by teams for their Service can be combined with metadata about their team and surfaced via an API and events published to Pub/Sub. This can then drive updates to other features like incident response and security tooling, pre-provision documentation repositories, and more. This is an example of how something that was originally intended as a means to help teams avoid writing their own IaC can also be used to make it easier for us to build platform features, further improving the value-add — without the developer even needing to be aware of it!

We think this approach is also more scalable than providing pre-built Terraform modules for teams to use. That approach still burdens teams with being familiar with Terraform, and versioning and dependency complexities can create maintenance headaches for platform engineers. Instead, we provide an easy-to-reason-about API and deliberately burden the platform team, ensuring that the Service provides all the functionality our tenants require. This abstraction also means we can make significant refactoring choices if we need to.

Adopting this approach also results in a broad consistency in technologies across our platform. For example, why would a team implement Kafka when the platform makes creating resources in Pub/Sub so easy? When you consider that this spans not just the runtime components that assemble into a working business service, but also all the ancillary needs for operating that software — resilience engineering, monitoring & alerting, incident response, security tooling, service management, and so on— this has a massive amplifying effect on our engineers’ productivity. All of these areas have full paved road capabilities on the John Lewis Digital Platform, reducing the cognitive load for teams in recognizing the need for, identifying appropriate options, and then implementing technology or processes to use them.

That being said, one of the reasons we particularly like the paved road concept is because it doesn't preclude teams choosing to "go off-road." A paved road shouldn’t be mandatory, but it should be compelling to use, so that engineers aren’t tempted to do something else. Preventing use of other approaches risks stifling innovation and the temptation to think the features you've built are "good enough." The paved road challenges our Platform Engineers to keep improving their product so that it continues to meet our Developers' changing needs. Likewise, development teams tempted to go off-road are put off by the increasing burden of replicating powerful platform features.

The needs of our Engineers don’t remain fixed, and Google Cloud are of course releasing new capabilities all the time, so we have extended the analogy to include a “dusty path” representing brand new platform features that aren’t as feature-rich as we’d like (perhaps they lack self-service provisioning or out-the-box observability). Teams are trusted to try different options and make use of Google Cloud products that we haven't yet paved. The Paved Road Pipeline allows for this experimentation - what we term "snowflaking". We then have an unofficial "rule of three", whereby if we notice at least 3 teams requesting the same feature, we move to make the use of it self-service.

At the other end of the scale, teams can go completely solo — which we refer to as “crazy paving” — and might be needed to support wild experimentation or to accommodate a workload which cannot comply with the platform’s expectations for safe operation. Solutions in this space are generally not long-lived.

In this article, we've covered how John Lewis revolutionized its e-commerce operations by adopting a multi-tenant, "paved road" approach to platform engineering. We explored how this strategy empowered development teams and streamlined their ability to provision Google Cloud resources and deploy operational and security features.

In part 2 of this series, we'll dive deeper into how John Lewis further simplified the developer experience by introducing the Microservice CRD. You'll discover how this custom Kubernetes abstraction significantly reduced the complexity of working with Kubernetes at the component level, leading to faster development cycles and enhanced operational efficiency.

To learn more about shifting down with platform engineering on Google Cloud, you can find more information available here. To learn more about how Google Kubernetes Engine (GKE) empowers developers to effortlessly deploy, scale, and manage containerized applications with its fully managed, robust, and intelligent Kubernetes service, you can find more information here.

Using Platform Engineering to simplify the developer experience - part two

Thu, 26 Jun 2025 16:00:00 +0000

In our previous article we introduced the John Lewis Digital Platform and its approach to simplifying the developer experience through platform engineering and so-called paved road features. We focused on the ways that platform engineering enables teams to create resources in Google Cloud and deploy the platform's operational and security features within dedicated tenant environments. In this article, we will build upon that concept for the next level of detail — how the platform simplifies build and run at a component (typically for us, a microservice) level too.

Within just over a year, the John Lewis Digital Platform had fully evolved into a product. We had approximately 25 teams using our platform, with several key parts of the johnlewis.com retail website running in production. We had built a self-service capability to help teams provision resources in Google Cloud, and firmly established that the foundation of our platform was on Google Kubernetes Engine (GKE). But we were hearing signals from some of the recent teams that there was a learning curve to Kubernetes. This was expected — we were driving a cultural change for teams to build and run their own services, and so we anticipated that our application developers would need some Kubernetes skills to support their own software. But our vision was that we wanted to make developers' lives easier — and their feedback was clear. In some cases, we observed that teams weren't following "good practice" (despite the existence of good documentation!) such as not using anti-affinity rules or PodDisruptionBudgets to help their workloads tolerate failure.

aside_block: <ListValue: [StructValue([('title', 'Try Google Cloud for free'), ('body', <wagtail.rich_text.RichText object at 0x7f4863aaaa30>), ('btn_text', 'Get started for free'), ('href', 'https://console.cloud.google.com/freetrial?redirectPath=/welcome'), ('image', None)])]>

All the way back in 2017, Kelsey Hightower wrote: “Kubernetes is a platform for building platforms. It's a better place to start, not the endgame.”

Kelsey's quote inspired us to act. We had the idea to write our own custom controller to simplify the point of interaction for a developer with Kubernetes — a John Lewis-specific abstraction that aligned to our preferred approaches. And thus the JL Microservice was born.

To do this, we declared a Kubernetes CustomResourceDefinition with a simplified specification containing just the fields we felt our developers needed to set. For example, as we expect our tenants to build and operate their applications themselves, attributes such as the number of replicas and the amount of resources needed are best left up to the developers themselves. But do they really need to be able to customize the rules defining how to distribute pods across nodes? How often do they need to change the Service pointing towards their Deployment? When we looked closer, we realized just how much duplication there was — our analysis at the time suggested that only around 33% of the lines in the yaml files developers were producing were relevant to their application. This was a target-rich scenario for simplification.

To help us build this feature, we selected Kubebuilder, using it to declare our CustomResourceDefinition and then build the Controller (what we call MicroserviceManager). This turned out to be a beneficial decision — initial prototyping was quick, and the feature was launched a few months later, and very well-received. Our team had to skill up in the Go programming language, but this trade-off felt worthwhile due to the advantages Kubebuilder was bringing to the table, and it has continued to be helpful for other software engineering since.

The initial implementation replaced an engineer's need to understand and fully configure a Deployment and Service, instead applying a much briefer yaml file containing only the fields they need to change. As well as direct translation of identical fields (image and replicas are equivalent to what you would see in a Deployment, for example), it also allowed us to simplify the choices made by the Kubernetes APIs, because in John Lewis we didn't need some of that functionality. For example, writablePaths: [] is an easy concept for our engineers to understand, and behind the scenes, our controller is converting those into the more complex combination of Volumes and VolumeMounts. Likewise, visibleToOtherServices: true is an example of us simplifying the interaction with Kubernetes NetworkPolicy — rather than requiring teams to read our documentation to understand the necessary incantations to label their resources correctly, the controller understands those conventions and handles it for them.

With the core concept of the Microservice resource established, we were able to improve the value-add by augmenting it with further features. We rapidly extended it out to define our Prometheus scrape configuration, then more complex features such as allowing teams to declare that they use Google Cloud Endpoints, and have the controller inject the necessary sidecar container into their Deployment and wiring it up to the Service. As we added more features, existing tenants converted to use this specification, and it now makes up the majority of workloads declared on the platform.

Moving the platform boundary

Our motivation to build MicroserviceManager was focused on making developers' lives easier. But we discovered an additional benefit that we had not initially expected - it was something we could greatly benefit from within the platform as well. It enabled us to make changes behind the scenes without needing to involve our tenants — reducing toil for them and making it easier for us to improve our product. This was a slightly unexpected but an exceptionally powerful benefit. It is generally difficult to change the agreement that you’ve established between your tenants and the platform, and creating an abstraction like this has allowed us to bring more under our control, for everyone’s benefit.

An example of this was something we observed through our live load testing of johnlewis.com when certain workloads burst up to several hundred Pods — numbers that exceeded the typical number of Nodes we had running in the cluster. This led to new Node creation — therefore slower Pod autoscaling and poor bin-packing. Experienced Kubernetes operators can probably guess what was happening here: our default antiAffinity rules were set to optimize for resilience such that no more than one replica was allowed on any given Node. The good news though was that because the workloads were under the control of our Microservice Manager, rather than us having to instruct our tenants to copy the relevant yaml into their Deployments, it was a straightforward change for us to replace the antiAffinity rules with the more modern podTopologyConstraints, allowing us to customize the number of replicas that could be stacked on a Node for workloads exceeding a certain replica count. And this happened with no intervention from our tenants.

A more complex example of this was when we rolled out our service mesh. In keeping with our general desire to let Google Cloud handle the complexity of running control planes components, we opted to use Google's Cloud Service Mesh product. But even then, rolling out a mesh to a business-critical platform in constant use is not without its risks. Microservice Manager allowed us to control the rate at which we enrolled workloads into the mesh through the use of a feature flag on the Microservice resource. We could start rollout with platform-owned workloads first to test our approach, then make tenants aware of the flag for early adopters to validate and take advantage of some of Cloud Service Mesh’s features. To scale the rollout, we could then manipulate the flag to release in waves based on business importance, providing an opt-out mechanism if needed to. This again greatly simplified the implementation — product teams had very little to do, and we avoided having to chase approximately 40 teams running hundreds of Microservices to make the appropriate changes in their configuration. This feature flagging technique is something we make extensive use of to support our own experimentation.

Beyond the microservice

Building the Microservice Manager has led to further thinking in Kubernetes-native ways: the Custom Resource + Controller concept is a powerful technique, and we have built other features since using it. One example is a controller that converts the need for external connectivity into Istio resources to route via our egress gateway. Istio in particular is an example of a very powerful platform capability that comes with a high cognitive load for its users, and so is a perfect example of where platform engineering can help manage that for teams whilst still allowing them to take advantage of it. We have a number of ideas in this area now that our confidence in the technology has grown.

In summary, the John Lewis Partnership leveraged Google Cloud and platform engineering to modernize their e-commerce operations and developer experience. By implementing a "paved road" approach with a multi-tenant architecture, they empowered development teams, accelerated deployment cycles, and simplified Kubernetes interactions using a custom Microservice CRD. This strategy allowed them to scale effectively and enhance the developer experience by reducing complexity while maintaining operational efficiency and scaling engineering teams effectively.

To learn more about platform engineering on Google Cloud, check out some of our other articles: 5 myths about platform engineering: what it is and what it isn’t, Another five myths about platform engineering, and Light the way ahead: Platform Engineering, Golden Paths, and the power of self-service.