Developers & Practitioners

Migrating to Google Cloud’s Application Load Balancer: A practical guide

Fri, 10 Apr 2026 16:00:00 +0000

Migrating your existing application load balancer infrastructure from an on-premises hardware solution to Cloud Load Balancing offers substantial advantages in scalability, cost-efficiency, and tight integration within the Google Cloud ecosystem. Yet, a fundamental question often arises: "What about our current load balancer configurations?"

Existing on-premises load balancer configurations often contain years of business-critical logic for traffic manipulation. The good news is that not only can you fully migrate existing functionalities, but this migration also presents a significant opportunity to modernize and simplify your traffic management.

This guide outlines a practical approach for migrating your existing load balancer to Google Cloud’s Application Load Balancer. It addresses common functionalities, leveraging both its declarative configurations and the innovative, event-driven Service Extensions edge compute capability.

A simple, phased approach to migration

Transitioning from an imperative, script-based system to a cloud-native, declarative-first model requires a structured plan. We recommend a straightforward, four-phase approach.

Phase 1: Discovery and mapping

Before commencing any migration, you must understand what you have. Analyze and categorize your current load balancer configurations. What is each rule's intent? Is it performing a simple HTTP-to-HTTPS redirect? Is it engaged in HTTP header manipulation (addition or removal)? Or is it handling complex, custom authentication logic?

Most configurations typically fall into two primary categories:

Common patterns: Logic that is common to most web applications, such as redirects, URL rewrites, basic header manipulation, and IP-based access control lists (ACLs).
Bespoke business logic: Complex logic unique to your application, like custom proprietary token authentication, advanced header extraction / replacement, dynamic backend selection based on HTTP attributes, or HTTP response body manipulation.

Phase 2: Choose your Google Cloud equivalent

Once your rules are categorized, the next step involves mapping them to the appropriate Google Cloud feature. This is not a one-to-one replacement; it's a strategic choice.

Option 1: the declarative path (for ~80% of rules)
For the majority of common patterns, leveraging the Application Load Balancer's built-in declarative features is usually the best approach. Instead of a script, you define the desired state in a configuration file. This is simpler to manage, version-control, and scale.

Common patterns to declarative feature mapping:

Redirects/rewrites -> Application Load Balancer URL maps
ACLs/throttling -> Google Cloud Armor security policies
Session persistence -> backend service configuration

Option 2: The programmatic path (for complex, bespoke rules)
When dealing with complex, bespoke business logic, you have a programmatic equivalent: Service Extensions, a powerful edge compute capability that allows you to inject custom code (written in Rust, C++ or Go) directly into the load balancer's data path. This approach gives you flexibility in a modern, managed, and high-performance framework.

This flowchart helps you decide the appropriate Google Cloud feature for each configuration

Phase 3: Test and validate

Once you’ve chosen the appropriate path for your configurations, you are ready to deploy your new Application Load Balancer configuration in a staging environment that mirrors your production setup. Thoroughly test all application functionality, paying close attention to the migrated logic. Use a combination of automated testing and manual QA to validate the redirects, security policies, and that the custom Service Extensions logic are behaving as expected.

Phase 4: Phased cutover (canary deployment)

Don't flip a single switch for all your traffic; instead, implement a phased migration strategy. Start the transitioning process by routing a small percentage of production traffic (e.g., 5-10%) to your new Google Cloud load balancer. During this initial period, be sure to monitor key metrics like latency, error rates, and application performance. As you gain confidence, you can progressively increase the percentage of traffic routed to the Application Load Balancer. Always have a clear rollback plan to revert back to the legacy infrastructure in the event you encounter critical issues.

Best practices for a smooth migration

Drawing from our practical experience, we have compiled the following recommendations to assist you in planning your load balancer migrations.

Analyze first, migrate second: A thorough analysis of your existing configurations is the most critical step. Don't "lift and shift" logic that is no longer needed.
Prefer declarative: Always default to Google Cloud's managed, declarative features (URL Maps, Cloud Armor) first. They are simpler, more scalable, and require less maintenance.
Use Service Extensions strategically: Reserve Service Extensions for the complex, bespoke business logic that declarative features cannot handle.
Monitor everything: Continuously monitor both your existing load balancers and Google Cloud load balancers during the migration. Watch key metrics like traffic volume, latency, and error rates to detect and address issues instantly.
Train your team: Ensure your team is trained on Cloud Load Balancing concepts. This will empower them to effectively operate and maintain the new infrastructure.

Migrating from the existing on-premises load balancer infrastructure is more than just a technical task, it's an opportunity to modernize your application delivery. By thoughtfully mapping your current load balancing configurations and capabilities to either declarative Application Load Balancer features or programmatic Service Extensions, you can build a more scalable, resilient, and cost-effective infrastructure destined for future demands.

To get started, review the Application Load Balancer and Service Extensions features and advanced capabilities to come up with the right design for your application. For more guidance and complex use cases, contact your Google Cloud team.

Create Expert Content: Local Testing of a Multi-Agent System with Memory

Fri, 10 Apr 2026 08:11:00 +0000

In support of our mission to accelerate the developer journey on Google Cloud, we built Dev Signal: a multi-agent system designed to transform raw community signals into reliable technical guidance by automating the path from discovery to expert creation.

In part 1 and part 2 of this series, we established the essential groundwork by standardizing the core capabilities through the Model Context Protocol (MCP) and constructing a multi-agent architecture integrated with the Vertex AI memory bank to provide long-term intelligence and persistence. Now, we'll explore how to test your multi-agent system locally!

If you’d like to dive straight into the code and explore it at your own pace, you can clone the repository here.

Testing the agent Locally

Before transitioning your agentic system to Google Cloud Run, it is essential to ensure that its specialized components work seamlessly together on your workstation. This testing phase allows you to validate trend discovery, technical grounding, and creative drafting within a local feedback loop, saving time and resources during the development process.

In this section, you will configure your local secrets, implement environment-aware utilities, and use a dedicated test runner to verify that Dev Signal can correctly retrieve user preferences from the Vertex AI memory bank on the cloud. This local verification ensures that your agent's "brain" and "hands" are properly synchronized before moving to deployment.

Environment Setup

Create a .env file in your project root. These variables are used for local development and will be replaced by Terraform/Secret Manager in production.

Paste this code in dev-signal/.env and update with your own details.

Note: GOOGLE_CLOUD_LOCATION is set as global because that is where Gemini-3-flash-preview is supported. We will use GOOGLE_CLOUD_LOCATION for the model location.

code_block: <ListValue: [StructValue([('code', '# Google Cloud Configuration\r\nGOOGLE_CLOUD_PROJECT=your-project-id\r\nGOOGLE_CLOUD_LOCATION=global\r\nGOOGLE_CLOUD_REGION=us-central1\r\nGOOGLE_GENAI_USE_VERTEXAI=True\r\nAI_ASSETS_BUCKET=your_bucket_name\r\n\r\n# Reddit API Credentials\r\nREDDIT_CLIENT_ID=your_client_id\r\nREDDIT_CLIENT_SECRET=your_client_secret\r\nREDDIT_USER_AGENT=my-agent/0.1\r\n\r\n# Developer Knowledge API Key\r\nDK_API_KEY=your_api_key'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3602243e80>)])]>

Helper Utilities

Create a new directory for your application utils.

code_block: <ListValue: [StructValue([('code', 'cd dev_signal_agent\r\nmkdir app_utils\r\ncd app_utils'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3602243f10>)])]>

Environment configuration

This module standardizes how the agent discovers the active Google Cloud Project and Region, ensuring a seamless transition between development environments. Using load_dotenv(), the script first checks for local configurations before falling back to google.auth.default() or environment variables to retrieve the Project ID. This automated approach ensures your agent is properly authenticated and grounded in the correct cloud context without requiring manual configuration changes.

Beyond basic project discovery, the script provides a robust Secret Management layer. It attempts to resolve sensitive credentials, such as Reddit API keys, first from the local environment (for rapid development) and then dynamically from the Google Cloud Secret Manager API for production security. By returning these as a dictionary rather than injecting them into environment variables, the module maintains a clean security posture.

The script further calibrates the environment by distinguishing between global and regional requirements for different AI services. It specifically assigns the "global" location for models to access cutting-edge preview features while designating a regional location, such as us-central1, for infrastructure like the Vertex AI Agent Engine. By finalizing this setup with a global SDK initialization, the module integrates these settings into the session, allowing the rest of your application to interact with models and memory banks without having to repeatedly pass project or location parameters.

Paste this code in dev_signal_agent/app_utils/env.py

code_block: <ListValue: [StructValue([('code', 'import os\r\nimport google.auth\r\nimport vertexai\r\nfrom google.cloud import secretmanager\r\nfrom dotenv import load_dotenv\r\n\r\ndef _fetch_secrets(project_id: str):\r\n """Fetch secrets from Secret Manager and return them as a dictionary."""\r\n secrets_to_fetch = ["REDDIT_CLIENT_ID", "REDDIT_CLIENT_SECRET", "REDDIT_USER_AGENT", "DK_API_KEY"]\r\n fetched_secrets = {}\r\n\r\n # First, check local environment (for local development via .env)\r\n for s in secrets_to_fetch:\r\n val = os.getenv(s)\r\n if val:\r\n fetched_secrets[s] = val\r\n\r\n # If keys are missing (common in production), fetch from Secret Manager API\r\n if len(fetched_secrets) < len(secrets_to_fetch):\r\n client = secretmanager.SecretManagerServiceClient()\r\n for secret_id in secrets_to_fetch:\r\n if secret_id not in fetched_secrets:\r\n name = f"projects/{project_id}/secrets/{secret_id}/versions/latest"\r\n try:\r\n response = client.access_secret_version(request={"name": name})\r\n # DO NOT set os.environ[secret_id] here. \r\n # Keep it in this dictionary only.\r\n fetched_secrets[secret_id] = response.payload.data.decode("UTF-8")\r\n except Exception as e:\r\n print(f"Warning: Could not fetch {secret_id} from Secret Manager: {e}")\r\n\r\n return fetched_secrets\r\n\r\ndef init_environment():\r\n """Consolidated environment discovery."""\r\n load_dotenv()\r\n try:\r\n _, project_id = google.auth.default()\r\n except Exception:\r\n project_id = os.getenv("GOOGLE_CLOUD_PROJECT")\r\n \r\n model_location = os.getenv("GOOGLE_CLOUD_LOCATION", "global")\r\n service_location = os.getenv("GOOGLE_CLOUD_REGION", "us-central1")\r\n \r\n secrets = {}\r\n if project_id:\r\n vertexai.init(project=project_id, location=service_location)\r\n # Fetch secrets into a local variable\r\n secrets = _fetch_secrets(project_id)\r\n \r\n return project_id, model_location, service_location, secrets'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f35fffceeb0>)])]>

Local testing script

The Google ADK comes with a built-in Web UI, This UI is excellent for visualizing agent logic and tool composition.

You can launch it by running in the project root:

code_block: <ListValue: [StructValue([('code', 'uv run adk web'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f35ec3398e0>)])]>

However, the default Web UI will not test the long-term memory integration described in this tutorial because it is not pre-connected to a Vertex AI memory session. By default, the generic UI often relies on in-memory services that do not persist data across sessions. Therefore, we use the dedicated test_local.py script to explicitly initialize the VertexAiMemoryBankService. This ensures that even in a local environment, your agent is communicating with the real cloud-based memory bank to validate preference persistence.

The test_local.py script:

Connects to the real Vertex AI Agent Engine in the cloud for memory storage.
Uses an in-memory session service for local chat history (so you can wipe it easily).
Run a chat loop where you can talk to your agent.

Go back to the root folder dev-signal:

code_block: <ListValue: [StructValue([('code', 'cd ../..'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f35ec339640>)])]>

Paste this code in dev-signal/test_local.py

code_block: <ListValue: [StructValue([('code', 'import asyncio\r\nimport os\r\nimport google.auth\r\nimport vertexai\r\nimport uuid\r\nfrom dotenv import load_dotenv\r\nfrom google.adk.runners import Runner\r\nfrom google.adk.memory.vertex_ai_memory_bank_service import VertexAiMemoryBankService\r\nfrom google.adk.sessions import InMemorySessionService\r\nfrom vertexai import agent_engines\r\nfrom google.genai import types\r\nfrom dev_signal_agent.agent import root_agent\r\n\r\n# Load environment variables\r\nload_dotenv()\r\n\r\nasync def main():\r\n # 1. Setup Configuration\r\n project_id = os.getenv("GOOGLE_CLOUD_PROJECT")\r\n # Agent Engine (Memory) MUST use a regional endpoint\r\n resource_location = "us-central1"\r\n agent_name = "dev-signal"\r\n \r\n print(f"--- Initializing Vertex AI in {resource_location} ---")\r\n vertexai.init(project=project_id, location=resource_location)\r\n\r\n # 2. Find the Agent Engine Resource for Memory\r\n existing_agents = list(agent_engines.list(filter=f"display_name={agent_name}"))\r\n if existing_agents:\r\n agent_engine = existing_agents[0]\r\n agent_engine_id = agent_engine.resource_name.split("/")[-1]\r\n print(f"✅ Using persistent Memory Bank from Agent: {agent_engine_id}")\r\n else:\r\n print(f"❌ Error: Agent Engine \'{agent_name}\' not found. Please deploy with Terraform first.")\r\n return\r\n\r\n # 3. Initialize Services\r\n # We use InMemorySessionService for easier local testing (IDs are flexible)\r\n # BUT we use VertexAiMemoryBankService for REAL cloud persistence\r\n session_service = InMemorySessionService()\r\n \r\n memory_service = VertexAiMemoryBankService(\r\n project=project_id,\r\n location=resource_location,\r\n agent_engine_id=agent_engine_id\r\n )\r\n\r\n # 4. Create a Runner\r\n runner = Runner(\r\n agent=root_agent,\r\n app_name="dev-signal",\r\n session_service=session_service,\r\n memory_service=memory_service \r\n )\r\n\r\n # 5. Run a Test Loop\r\n user_id = "local-tester"\r\n \r\n print("\\n--- TEST SCENARIO ---")\r\n print("1. Start a session, tell the agent your preference (e.g., \'write in rhymes\').")\r\n print("2. Type \'new\' to start a FRESH session (local state wiped).")\r\n print("3. Ask for a blog post. The agent should retrieve your preference from the CLOUD memory.")\r\n \r\n current_session_id = f"session-{str(uuid.uuid4())[:8]}"\r\n await session_service.create_session(\r\n app_name="dev-signal",\r\n user_id=user_id,\r\n session_id=current_session_id\r\n )\r\n print(f"\\n--- Chat Session (ID: {current_session_id}) ---")\r\n\r\n while True:\r\n user_input = input("\\nYou: ")\r\n \r\n if user_input.lower() in ["exit", "quit"]:\r\n break\r\n \r\n if user_input.lower() == "new":\r\n # Simulate starting a completely fresh session\r\n current_session_id = f"session-{str(uuid.uuid4())[:8]}"\r\n await session_service.create_session(\r\n app_name="dev-signal",\r\n user_id=user_id,\r\n session_id=current_session_id\r\n )\r\n print(f"\\n--- Fresh Session Started (ID: {current_session_id}) ---")\r\n print("(Local history is empty, retrieval must come from Memory Bank)")\r\n continue\r\n\r\n print("Agent is thinking...")\r\n async for event in runner.run_async(\r\n user_id=user_id,\r\n session_id=current_session_id,\r\n new_message=types.Content(parts=[types.Part(text=user_input)])\r\n ):\r\n if event.content and event.content.parts:\r\n for part in event.content.parts:\r\n if part.text:\r\n print(f"Agent: {part.text}")\r\n \r\n if event.get_function_calls():\r\n for fc in event.get_function_calls():\r\n print(f"?️ Tool Call: {fc.name}")\r\n\r\nif __name__ == "__main__":\r\n asyncio.run(main())'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f35ec339820>)])]>

Running the Test

First, ensure you have your Application Default Credentials set up:

code_block: <ListValue: [StructValue([('code', 'gcloud auth application-default login'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f35ec339760>)])]>

Then run the script:

code_block: <ListValue: [StructValue([('code', 'uv run test_local.py'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f35ec339790>)])]>

Test Scenario

This scenario validates the full end-to-end lifecycle of the agent: from discovery and research to multimodal content creation and long-term memory retrieval.

Phase 1: Teaching & Multimodal Creation (Session 1)

Goal: Establish technical context and set a specific stylistic preference.

Discovery

Ask the agent to find trending Cloud Run topics.

Input: "Find high-engagement questions about AI agents on Cloud Run from the last 21 days."

Research

Instruct the agent to perform a deep dive on a specific result.

Input: "Use the GCP Expert to research topic #1."

Personalization

Request a blog post and explicitly set your style preference.

Input: "Draft a blog post based on this research. From now on, I want all my technical blogs written in the style of a 90s Rap Song."

Image generation

Ask the agent to generate an image that demonstrates the main ideas in the blog using the Nano Banana Pro tool. The image would be saved to your bucket in Google Cloud and you should get the path to see it which will look like this: https://storage.mtls.cloud.google.com/...

Phase 2: Long-Term Memory Recall (Session 2)

Goal: Verify the agent recalls preferences across a completely fresh session.

Type new in the console to wipe local session history and start a fresh state.
Retrieval) Inquire about your stored preferences to test the Vertex AI memory bank.

Input: "What are my current topics of interest and what is my preferred blogging style?"

Verification: Confirm the agent successfully retrieves your "AI Agents on Cloud Run" interest and "Rap" style from the cloud.

Final Test: Ask for a new blog on a different topic (e.g., "GKE Autopilot") and ensure it is automatically written as a rap song without being prompted.

Summary

In this part of our series we focused on verifying the agent's functionality in a local environment before proceeding to cloud deployment. By configuring local secrets and utilizing environment-aware utilities, we used a dedicated test runner to confirm that the core reasoning and tool logic are properly integrated. We successfully validated the full lifecycle: from Reddit discovery to expert content creation, confirming that the agent correctly retrieves preferences from the cloud-based Vertex AI memory bank even in completely fresh sessions.

Ready to run the test scenario yourself? Clone the repository and try the test_local.py script to see 'Dev Signal' retrieve your preferences from the Vertex AI memory bank in real-time. For a deeper dive into the underlying mechanics of memory orchestration, check out this quickstart guide.

In the final part of this series, we will transition our prototype into production service on Google Cloud Run using Terraform for secure infrastructure and explore the roadmap to production excellence through continuous evaluation and security

Special thanks to Remigiusz Samborski for the helpful review and feedback on this article.

For more content like this, Follow me on Linkedin and X.

Experimenting with GPUs: GKE managed DRANET and Inference Gateway AI Deployment

Wed, 08 Apr 2026 10:05:00 +0000

Building and serving models on infrastructure is a strong use case for businesses. In Google Cloud, you have the ability to design your AI infrastructure to suit your workloads. Recently, I experimented with Google Kubernetes Engine (GKE) managed DRANET while deploying a model for inference with NVIDIA B200 GPUs on GKE. In this blog, we will explore this setup in easy to follow steps.

What is DRANET

Dynamic Resource Allocation (DRA) is a feature that lets you request and share resources among Pods. DRANET allows you to request and allocate networking resources for your Pods, including network interfaces that support TPUs & Remote Direct Memory Access (RDMA). In my case, the use of high-end GPUs.

How GPU RDMA VPC works

The RDMA network is set up as an isolated VPC, which is regional and assigned a network profile type. In this case, the network profile type is RoCEv2. This VPC is dedicated for GPU-to-GPU communication. The GPU VM families have RDMA capable NICs that connect to the RDMA VPC. The GPUs communicate between multiple nodes via this low latency, high speed rail aligned setup.

Design pattern example

Our aim was to deploy a LLM model (Deepseek) onto a GKE cluster with A4 nodes that support 8 B200 GPUs and serve it via GKE Inference gateway privately. To set up an AI Hypercomputer GKE cluster, you can use the Cluster Toolkit, but in my case, I wanted to test the GKE managed DRANET dynamic setup of the networking that supports RDMA for the GPU communication.

This design utilizes the following services to provide an end-to-end solution:

VPC: Total of 3 VPC. One VPC manually created, two created automatically by GKE managed DRANET, one standard and one for RDMA.
GKE: To deploy the workload.
GKE Inference gateway: To expose the workload internally using a regional internal Application Load Balancers type gke-l7-rilb.
A4 VM’s: These support RoCEv2 with NVIDIA B200 GPU.

Putting it together

To get access to the A4 VM a future reservation was used. This is linked to a specific zone.

Begin: Set up the environment

Create a standard VPC, with firewall rules and subnet in the same zone as the reservation.
Create a proxy-only subnet this will be used with the Internal regional application load balancer attached to the GKE inference gateway

Next: Create a standard GKE cluster node and default node pool.

code_block: <ListValue: [StructValue([('code', 'gcloud container clusters create $CLUSTER_NAME \\\r\n --location=$ZONE \\\r\n --num-nodes=1 \\\r\n --machine-type=e2-standard-16 \\\r\n --network=${GVNIC_NETWORK_PREFIX}-main \\\r\n --subnetwork=${GVNIC_NETWORK_PREFIX}-sub \\\r\n --release-channel rapid \\\r\n --enable-dataplane-v2 \\\r\n --enable-ip-alias \\\r\n --addons=HttpLoadBalancing,RayOperator \\\r\n --gateway-api=standard \\\r\n --enable-ray-cluster-logging \\\r\n --enable-ray-cluster-monitoring \\\r\n --enable-managed-prometheus \\\r\n --enable-dataplane-v2-metrics \\\r\n --monitoring=SYSTEM'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f35fffaaac0>)])]>

Once that is complete you can connect to your cluster:

code_block: <ListValue: [StructValue([('code', 'gcloud container clusters get-credentials $CLUSTER_NAME --zone $ZONE --project $PROJECT'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f35fffaa670>)])]>

Create a GPU node pool (this example uses, A4 VM with reservation) and additionals flags:

---accelerator-network-profile=auto (GKE automatically adds the gke.networks.io/accelerator-network-profile: auto label to the nodes)

--node-labels=cloud.google.com/gke-networking-dra-driver=true (Enables DRA for high-performance networking)

code_block: <ListValue: [StructValue([('code', 'gcloud beta container node-pools create $NODE_POOL_NAME \\\r\n --cluster $CLUSTER_NAME \\\r\n --location $ZONE \\\r\n --node-locations $ZONE \\\r\n --machine-type a4-highgpu-8g \\\r\n --accelerator type=nvidia-b200,count=8,gpu-driver-version=latest \\\r\n --enable-autoscaling --num-nodes=1 --total-min-nodes=1 --total-max-nodes=3 \\\r\n --reservation-affinity=specific \\\r\n--reservation=projects/$PROJECT/reservations/$RESERVATION_NAME/reservationBlocks/$BLOCK_NAME \\\r\n --accelerator-network-profile=auto \\\r\n--node-labels=cloud.google.com/gke-networking-dra-driver=true'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f35fffaa8b0>)])]>

Next: Create a ResourceClaimTemplate, which will be used to attach the networking resources to your deployments. The deviceClassName: mrdma.google.com is used for GPU workloads:

code_block: <ListValue: [StructValue([('code', 'apiVersion: resource.k8s.io/v1\r\nkind: ResourceClaimTemplate\r\nmetadata:\r\n name: all-mrdma\r\nspec:\r\n spec:\r\n devices:\r\n requests:\r\n - name: req-mrdma\r\n exactly:\r\n deviceClassName: mrdma.google.com\r\n allocationMode: All'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f35fffaacd0>)])]>

Deploy model and inference

Now that a cluster and node pool is setup, we can deploy a model and serve it via Inference gateway. In my experiment I used DeepSeek but this could be any model.

Deploy model and services

The nodeSelector: gke.networks.io/accelerator-network-profile: auto is used to assign to the GPU node
The resourceClaims: attaches the resource we defined for networking

Create a secret (I used Hugging Face token):

code_block: <ListValue: [StructValue([('code', 'kubectl create secret generic hf-secret \\\r\n --from-literal=hf_token=${HF_TOKEN}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f35fffaaa60>)])]>

Deployment

code_block: <ListValue: [StructValue([('code', 'apiVersion: apps/v1\r\nkind: Deployment\r\nmetadata:\r\n name: deepseek-v3-1-deploy\r\nspec:\r\n replicas: 1\r\n selector:\r\n matchLabels:\r\n app: deepseek-v3-1\r\n template:\r\n metadata:\r\n labels:\r\n app: deepseek-v3-1\r\n ai.gke.io/model: deepseek-v3-1\r\n ai.gke.io/inference-server: vllm\r\n examples.ai.gke.io/source: user-guide\r\n spec:\r\n containers:\r\n - name: vllm-inference\r\n image: us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:20250819_0916_RC01\r\n resources:\r\n requests:\r\n cpu: "190"\r\n memory: "1800Gi"\r\n ephemeral-storage: "1Ti"\r\n nvidia.com/gpu: "8"\r\n limits:\r\n cpu: "190"\r\n memory: "1800Gi"\r\n ephemeral-storage: "1Ti"\r\n nvidia.com/gpu: "8"\r\n claims:\r\n - name: rdma-claim\r\n command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]\r\n args:\r\n - --model=$(MODEL_ID)\r\n - --tensor-parallel-size=8\r\n - --host=0.0.0.0\r\n - --port=8000\r\n - --max-model-len=32768\r\n - --max-num-seqs=32\r\n - --gpu-memory-utilization=0.90\r\n - --enable-chunked-prefill\r\n - --enforce-eager\r\n - --trust-remote-code\r\n env:\r\n - name: MODEL_ID\r\n value: deepseek-ai/DeepSeek-V3.1\r\n - name: HUGGING_FACE_HUB_TOKEN\r\n valueFrom:\r\n secretKeyRef:\r\n name: hf-secret\r\n key: hf_token\r\n volumeMounts:\r\n - mountPath: /dev/shm\r\n name: dshm\r\n livenessProbe:\r\n httpGet:\r\n path: /health\r\n port: 8000\r\n initialDelaySeconds: 1800\r\n periodSeconds: 10\r\n readinessProbe:\r\n httpGet:\r\n path: /health\r\n port: 8000\r\n initialDelaySeconds: 1800\r\n periodSeconds: 5\r\n volumes:\r\n - name: dshm\r\n emptyDir:\r\n medium: Memory\r\n nodeSelector:\r\n gke.networks.io/accelerator-network-profile: auto\r\n resourceClaims:\r\n - name: rdma-claim\r\n resourceClaimTemplateName: all-mrdma\r\n---\r\napiVersion: v1\r\nkind: Service\r\nmetadata:\r\n name: deepseek-v3-1-service\r\nspec:\r\n selector:\r\n app: deepseek-v3-1\r\n type: ClusterIP\r\n ports:\r\n - protocol: TCP\r\n port: 8000\r\n targetPort: 8000\r\n---\r\napiVersion: monitoring.googleapis.com/v1\r\nkind: PodMonitoring\r\nmetadata:\r\n name: deepseek-v3-1-monitoring\r\nspec:\r\n selector:\r\n matchLabels:\r\n app: deepseek-v3-1\r\n endpoints:\r\n - port: 8000\r\n path: /metrics\r\n interval: 30s'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f35ecb46b80>)])]>

Deploy GKE Inference Gateway

This install needed Custom Resource Definitions (CRDs) in your GKE cluster:

For GKE versions 1.34.0-gke.1626000 or later, install only the alpha InferenceObjective CRD:

code_block: <ListValue: [StructValue([('code', 'kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/v1.0.0/config/crd/bases/inference.networking.x-k8s.io_inferenceobjectives.yaml'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f35ecb466a0>)])]>

Create Inference pool

code_block: <ListValue: [StructValue([('code', 'helm install deepseek-v3-pool \\\r\n oci://registry.k8s.io/gateway-api-inference-extension/charts/inferencepool \\\r\n --version v1.0.1 \\\r\n --set inferencePool.modelServers.matchLabels.app=deepseek-v3-1 \\\r\n --set provider.name=gke \\\r\n --set inferenceExtension.monitoring.gke.enabled=true'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f35ecb46970>)])]>

Create the Gateway, HTTPRoute and InferenceObjective

code_block: <ListValue: [StructValue([('code', '# 1. The Regional Internal Gateway (ILB)\r\napiVersion: gateway.networking.k8s.io/v1\r\nkind: Gateway\r\nmetadata:\r\n name: deepseek-v3-gateway\r\n namespace: default\r\nspec:\r\n gatewayClassName: gke-l7-rilb\r\n listeners:\r\n - name: http\r\n protocol: HTTP\r\n port: 80\r\n allowedRoutes:\r\n namespaces:\r\n from: Same\r\n---\r\n# 2. The HTTPRoute (Routing to the Pool)\r\napiVersion: gateway.networking.k8s.io/v1\r\nkind: HTTPRoute\r\nmetadata:\r\n name: deepseek-v3-route\r\n namespace: default\r\nspec:\r\n parentRefs:\r\n - name: deepseek-v3-gateway\r\n rules:\r\n - matches:\r\n - path:\r\n type: PathPrefix\r\n value: /\r\n backendRefs:\r\n - group: inference.networking.k8s.io\r\n kind: InferencePool\r\n name: deepseek-v3-pool\r\n---\r\n# 3. The Inference Objective (Performance Logic)\r\napiVersion: inference.networking.x-k8s.io/v1alpha2\r\nkind: InferenceObjective\r\nmetadata:\r\n name: deepseek-v3-objective\r\n namespace: default\r\nspec:\r\n poolRef:\r\n name: deepseek-v3-pool'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f35ecb46490>)])]>

Once complete, you can create a test VM in your main VPC and make a call to the IP address of the GKE Inference Gateway:

code_block: <ListValue: [StructValue([('code', 'curl -N -s -X POST "http://$GATEWAY_IP/v1/chat/completions" \\\r\n -H "Content-Type: application/json" \\\r\n -d \'{\r\n "model": "deepseek-ai/DeepSeek-V3.1",\r\n "messages": [{"role": "user", "content": "Box A: red. Box B: blue. Box C: empty. Move A to C, Move B to A, Swap B and C. Where is red?"}],\r\n "stream": true\r\n }\' | stdbuf -oL grep "data: " | sed -u \'s/^data: //\' | grep -v "\\[DONE\\]" | \\\r\n jq --unbuffered -rj \'.choices[0].delta | (.reasoning_content // .reasoning // .content // empty)\''), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f35ecb464c0>)])]>

Next Steps

Take a deeper dive into GKE managed DRANET and GKE Inference Gateway, review the following.

Blog: DRA: A new era of Kubernetes device management with Dynamic Resource Allocation
Document set: DRANET
Documentation: AI Hypercomputer

Want to ask a question, find out more or share a thought? Please connect with me on Linkedin.

See beyond the IP and secure URLs with Google Cloud NGFW

Tue, 07 Apr 2026 17:30:00 +0000

In a cloud-first world, traditional IP-based defenses are no longer enough to protect your perimeter. As services migrate to shared infrastructure and content delivery networks, relying on static IP addresses and FQDNs can create security gaps.

Because single IP addresses can host multiple services, and IPs addresses can change frequently, we are introducing domain filtering with a wildcard capability in Cloud Next Generation Firewall (NGFW) Enterprise. This new capability provides increased security and granular policy controls.

Why domain and SNI filtering matters

The Cloud NGFW URL filtering service performs deep inspections of HTTP payloads to secure workloads against threats from both public and internal networks. This service elevates security controls to the application layer and helps restrict access to malicious domains.

Key use cases include:

Granular egress control: This capability enables the precise allowing and blocking of connections based on domain names and SNI information found in egress HTTP(S) messages. By inspecting Layer 7 (L7) headers, it offers significantly finer control than traditional filtering based solely on IP addresses and FQDNs, which can be inefficient when a single IP hosts multiple services.
Control access without decrypting: For organizations that prefer not to perform full TLS decryption on their traffic, Cloud NGFW can still enforce security policies by controlling traffic based on SNI headers provided during the TLS handshake. This allows for effective domain-level filtering while maintaining end-to-end encryption for privacy or compliance reasons.
Reduced operational overhead: Implementing domain-based filtering helps reduce the constant maintenance typically required to track frequently changing IP addresses and DNS records. By focusing on stable domain identities rather than dynamic network attributes, security teams can minimize the manual effort involved in updating firewall rulebases.
Flexible matching: The service utilizes matcher strings within URL lists, supporting limited wildcard domains to define criteria for both domains and subdomains. For example, using a wildcard like *.example.com allows a single filter to cover all associated subdomains, providing a more scalable solution than defining thousands of individual FQDN entries.
Improved security: URL filtering significantly enhances the security posture by protecting against sophisticated flaws like SNI header spoofing. By evaluating L7 headers before allowing access to an application, Cloud NGFW ensures that attackers cannot bypass security controls by simply spoofing lower-layer identifiers.

How Cloud NGFW URL filtering works

The URL filtering service functions by inspecting traffic at L7 using a distributed architecture.

Cloud NGFW URL filtering service

You can get started with URL filtering in three simple steps.

Deploy Cloud NGFW endpoints:

The first step is to create and deploy a Cloud NGFW endpoint in a zone. The NGFW endpoint is an organization level resource. Please ensure you have the right permission before deploying the endpoint.
Once the endpoint is deployed you can associate it to one or more VPCs of your choice.

Create security profiles and security profile groups:

The URL filtering security profile holds the URL filters with matcher strings and an action (allow or deny).
The security profile group acts as a container for these security profiles, which is then referenced by a firewall policy rule. Create URL filtering security profiles with desired URLs, wildcard FQDNs and add them to a security profile group.
Once the security profile group is created, you will need to reference the security profile group in firewall policies.

Policy enforcement:

You enable the service by configuring a hierarchical or global network firewall policy rule using the apply_security_profile_group action, specifying the name of your security profile group.

For more information about configuring a firewall policy rule, see the following:

Getting started

Get started with Cloud NGFW URL filtering by visiting our documentation and codelab.

Envoy: A future-ready foundation for agentic AI networking

Fri, 03 Apr 2026 16:00:00 +0000

In today's agentic AI environments, the network has a new set of responsibilities.

In a traditional application stack, the network mainly moves requests between services. But as discussed in a recent white paper, Cloud Infrastructure in the Agent-Native Era, in an agentic system the network sits in the middle of model calls, tool invocations, agent-to-agent interactions, and policy decisions that can shape what an agent is allowed to do. The rapid proliferation of agents, often built on diverse frameworks, necessitates a consistent enforcement of governance and security across all agentic paths at scale. To achieve this, the enforcement layer must shift from the application level to the underlying infrastructure. That means the network can no longer operate as a blind transport layer. It has to understand more, enforce better, and adapt faster. This shift is precisely where Envoy comes in.

As a high-performance distributed proxy and universal data plane, Envoy is built for massive scale. Trusted by demanding enterprise environments, including Google Cloud, it supports everything from single-service deployments to complex service meshes using Ingress, Egress, and Sidecar patterns. Because of its deep extensibility, robust policy integration, and operational maturity, Envoy is uniquely suited for an era where protocols change quickly and the cost of weak control is steep. For teams building agentic AI, Envoy is more than a concept: it's a practical, production-ready foundation.

Agentic AI changes the networking problem

Agentic workloads still often use HTTP as a transport, but they break some of the assumptions that traditional HTTP intermediaries rely on. Protocols such as Model Context Protocol (MCP) and Agent2agent (A2A) use JSON-RPC or gRPC over HTTP, adding protocol-level phases such as MCP initialization, where client and server exchange their capabilities, on top of standard HTTP request/response semantics. The key aspects of agentic systems that require intermediaries to adapt include:

Diverse enterprise governance imperatives. The primary challenge is satisfying the wide spectrum of non-negotiable enterprise requirements for safety, security, data privacy, and regulatory compliance. These needs often go beyond standard network policies and require deep integration with internal systems, custom logic, and the ability to rapidly adapt to new organizational rules or external regulations. This demands a highly extensible framework where enterprises can plug in their specific governance models.
Policy attributes live inside message bodies, not headers. Unlike traditional web traffic where policy inputs like paths and headers are readily accessible, agentic protocols frequently bury critical attributes (e.g., model names, tool calls, resource IDs) deep within JSON-RPC or gRPC payloads. This shift requires intermediaries to possess the ability to parse and understand message contents to apply context-aware policies.
Handling diverse and evolving protocol characteristics. Agentic protocols are not uniform. Some, like MCP with Streamable HTTP, can introduce stateful interactions requiring session management across distributed proxies (e.g., using Mcp-Session-Id). The need to support such varied behaviors, along with future protocol innovations, reinforces the necessity of an inherently adaptable and extensible networking foundation.

These factors mean enterprises need more than just connectivity. The network must now serve as a central point for enforcing the crucial governance needs mentioned earlier. This includes providing capabilities like centralized security, comprehensive auditability, fine-grained policy enforcement, and dynamic guardrails, all while keeping pace with the rapid evolution of protocols and agent behaviors. Put simply, agentic AI transforms the network from a mere transit path into a critical control point.

Why Envoy fits this shift

Envoy is a strong fit for agentic AI networking for three reasons. Envoy is:

Battle-tested. Enterprises already rely on Envoy in high-scale, security-sensitive environments, making it a credible platform to anchor a new generation of traffic management and policy enforcement.
Extensible. Envoy can be extended through native filters, Rust modules, WebAssembly (Wasm) modules, and external processing patterns. That gives platform teams room to adopt new protocols without having to rebuild their networking layer every time the ecosystem changes.
Operationally useful today. Envoy already acts as a gateway, enforcement point, observability layer, and integration surface for control planes. That makes it a practical choice for organizations that need to move now, not after the standards settle.

Building on these core strengths, Envoy has introduced specific architectural advancements to meet the unique demands of agentic networking:

1. Envoy understands agent traffic

The first requirement for agentic networking is simple: The gateway needs to understand what the agent is actually trying to do.

That’s harder than it sounds. In protocols such as MCP, A2A, and OpenAI-style APIs, important policy signals may live inside the request body. Traditional HTTP proxies are optimized to treat bodies as opaque byte streams. That design is efficient, but it limits what the proxy can enforce. For protocols that use JSON messages, a proxy may need to buffer the entire request body to locate attribute values needed for policy application — especially when those attributes appear at the end of the JSON message. Business logic specific to gen AI protocols, such as rate limiting based on consumed tokens, may also require parsing server responses.

Envoy addresses this by deframing protocol messages carried over HTTP and exposing useful attributes to the rest of the filter chain. The extensibility model for gen AI protocols was guided by two goals:

Easy reuse of existing HTTP extensions that work with gen AI protocols out of the box, such as RBAC or tracers.
Easy access to deframed messages for gen-AI-specific extensions, so that developers can focus on gen AI business logic without needing to deal with HTTP or JSON envelopes.

Based on these goals, new extensions for gen AI protocols are still built as HTTP extensions and configured in the HTTP filter chain. This provides flexibility to mix HTTP-native business logic, such as OAuth or mTLS authorization, with gen AI protocol logic in a single chain. A deframing extension parses the protocol messages carried by HTTP and provides an ambient context with extracted attributes, or even the entirety of parsed messages, to downstream extensions via well-known filter state and metadata values.

Instead of forcing every policy component to parse JSON envelopes or protocol-specific message formats on its own, Envoy makes those attributes available as structured metadata. Once the gateway has deframed protocol messages, existing Envoy extensions such as ext_authz or RBAC can read protocol properties to evaluate policies using protocol-specific attributes such as tool names for MCP, message attributes for A2A, or model names for OpenAI.

Access logs can include message attributes for enhanced monitoring and auditing. The protocol attributes are also available to the Common Expression Language (CEL) runtime, simplifying creation of complex policy expressions in RBAC or composite extensions.

Buffering and memory management
Envoy is designed to use as little memory as possible when proxying HTTP requests. However, parsing agentic protocols may require an arbitrary amount of buffer space, especially when extensions require the entire message to be in memory. The flexibility of allowing extensions to use larger buffers needs to be balanced with adequate protection from memory exhaustion, especially in the presence of untrusted traffic.

To achieve this, Envoy now provides a per-request buffer size limit. Buffers that hold request data are also integrated with the overload manager, enabling a full range of protective actions under memory pressure, such as reducing idle timeouts or resetting requests that consume the most memory for an extended duration. These changes pave the way for Envoy to serve as a gateway and policy-enforcement point for gen AI protocols without compromising its resource efficiency.

2. Envoy enforces policy on things that matter

Understanding traffic is only useful if the gateway can act on it.

In agentic systems, policy is not just about which service an agent can reach. It’s about which tools an agent can call, which models it can use, what identity it presents, how much it can consume, and what kinds of outputs require additional controls. Those are higher-value decisions than simple layer-4 or path-based controls, and they are exactly the kinds of controls enterprises care about when agents are allowed to take action on their behalf.

Envoy is well-positioned here because it can combine transport-level security with application-aware policy enforcement. Teams can authenticate workloads with mTLS and SPIFFE identities, then enforce protocol-specific rules with RBAC, external authorization, external processing, access logging, and CEL-based policy expressions.

This capability is crucial because it lets platform teams decouple agent development from enforcement. Developers can focus on building useful agents, while operators enforce a consistent zero-trust posture at the network layer, even as tools, models, and protocols continue to change.A prime example of this zero-trust decoupling is the critical "user-behind-agent" scenario, where an AI agent must execute tasks on a human user's behalf. Traditionally, handing user credentials directly to an application introduces severe security risks — if the agent is compromised or manipulated via prompt injection, an attacker could exfiltrate or misuse those credentials. By offloading identity management to Envoy, the proxy can automatically insert user delegation tokens into outbound requests at the infrastructure layer. Because the agent never directly holds the sensitive credential, the risk of a compromised agent misusing or leaking the token is completely neutralized, ensuring actions remain strictly bound to the user's actual permissions.

Case study: Restricting an agent to specific GitHub MCP tools
Consider an agent that triages GitHub issues.

The GitHub MCP server may expose dozens of tools, but the agent may only need a small read-only subset, such as list_issues, get_issue, and get_issue_comments. In most enterprises, that difference matters. A useful agent should not automatically become an unrestricted one.

With Envoy in front of the MCP server, the gateway can verify the agent identity using SPIFFE during the mTLS handshake, parse the MCP message via the deframing filter, extract the requested method and tool name, and enforce a policy that allows only the approved tool calls for that specific agent identity. RBAC uses metadata created by the MCP deframing filter to check the method and tool name in the MCP message:

code_block: <ListValue: [StructValue([('code', 'envoy.filters.http.rbac:\r\n "@type": type.googleapis.com/envoy.extensions.filters.http.rbac.v3.RBACPerRoute\r\n rbac:\r\n rules:\r\n policies:\r\n github-issue-reader-policy:\r\n permissions:\r\n - and_rules:\r\n rules:\r\n - sourced_metadata:\r\n metadata_matcher:\r\n filter: envoy.http.filters.mcp\r\n path: [{ key: "method" }]\r\n value: { string_match: { exact: "tools/call" } }\r\n - sourced_metadata:\r\n metadata_matcher:\r\n filter: envoy.http.filters.mcp\r\n path: [{ key: "params" }, { key: "name" }]\r\n value:\r\n or_match:\r\n value_matchers:\r\n - string_match: { exact: "list_issues" }\r\n - string_match: { exact: "get_issue" }\r\n - string_match: { exact: "get_issue_comments" }\r\n principals:\r\n - authenticated:\r\n principal_name:\r\n exact: "spiffe://cluster.local/ns/github-agents/sa/issue-triage-agent"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3600e814c0>)])]>

That’s the real value: Policy is enforced centrally, close to the traffic, and in terms that match the agent's actual behavior.

Beyond static rules: External authorization
A complex compliance policy that can’t be expressed using RBAC rules can be implemented in an external authorization service using the ext_authz protocol. Envoy provides MCP message attributes along with HTTP headers in the context of the ext_authz RPC. It can also forward the agent's SPIFFE identity from the peer certificate:

code_block: <ListValue: [StructValue([('code', 'http_filters:\r\n - name: envoy.filters.http.ext_authz\r\n typed_config:\r\n "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz\r\n grpc_service:\r\n envoy_grpc:\r\n cluster_name: auth_service_cluster\r\n include_peer_certificate: true\r\n metadata_context_namespaces:\r\n - envoy.http.filters.mcp'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3600675cd0>)])]>

This allows external services to make authorization decisions based on the full combination of agent identity, MCP method, tool name, and any other protocol attributes, without the agent or the MCP server needing to be aware of the policy layer.

Protocol-native error responses
When Envoy denies a request, the error should be meaningful to the calling agent. For MCP traffic, Envoy can use local_reply_config to map HTTP error codes to appropriate JSON-RPC error responses. For example, a 403 Forbidden can be mapped to a JSON-RPC response with isError: true and a human-readable message, ensuring the agent receives a protocol-appropriate denial rather than an opaque HTTP status code.

3. Envoy supports stateful agent interactions at scale

Not all agent traffic is stateless. Some protocols, including Streamable HTTP for MCP, can rely on session-oriented behavior. That creates a new challenge for intermediaries, especially when traffic flows through multiple gateway instances to achieve scale and resilience. An MCP session effectively binds the agent to the server that established it, and all intermediaries need to know this to direct incoming MCP connections to the correct server.

If a session is established on one backend, later requests in that conversation need to reach the right destination. That sounds straightforward for a single-proxy deployment, but it becomes more complicated in horizontally scaled systems, where multiple Envoy instances may handle different requests from the same agent.

Passthrough gateway
In the simpler passthrough mode, Envoy establishes one upstream connection for each downstream connection. Its primary use is enforcing centralized policies, such as client authorization, RBAC, rate limiting, and authentication, for external MCP servers. The session state transferred between intermediaries needs to include only the address of the server that established the session over the initial HTTP connection, so that all session-related requests are directed to that server.

Session state transfer between different Envoy instances is achieved by appending encoded session state to the MCP session ID provided by the MCP server. Envoy removes the session-state suffix from the session ID before forwarding the request to the destination MCP server. This session stickiness is enabled by configuring Envoy's envoy.http.stateful_session.envelope extension.

Aggregating gateway
In aggregating mode, Envoy acts as a single MCP server by aggregating the capabilities, tools, and resources of multiple backend MCP servers. In addition to enforcing policies, this simplifies agent configuration and unifies policy application for multiple MCP servers.

Session management in this mode is more complicated because the session state also needs to include mapping from tools and resources to the server addresses and session IDs that advertised them. The session ID that Envoy provides to the agent is created before tools or resources are known, and the mapping has to be established later, after the MCP initialization phases between Envoy and the backend MCP servers are complete.

One approach, currently implemented in Envoy, is to combine the name of a tool or resource with the identifier and session ID of its origin server. The exact tool or resource names are typically not meaningful to the agent and can carry this additional provenance information. If unmodified tool or resource names are desirable, another approach is to use an Envoy instance that does not have the mapping, and then recreate it by issuing a tools/list command before calling a specific tool. This trades latency for the complexity of deploying an external global store of MCP sessions, and is currently in planning based on user feedback.

This matters because it moves Envoy beyond simple traffic forwarding. It allows Envoy to serve as a reliable intermediary for real agent workflows, including those spanning multiple requests, tools, and backends.

4. Envoy supports agent discovery

Envoy is adding support for the A2A protocol and agent discovery via a well-known AgentCard endpoint. AgentCard, a JSON document with agent capabilities, enables discovery and multi-agent coordination by advertising skills, authentication requirements, and service endpoints. The AgentCard can be provisioned statically via direct response configuration or obtained from a centralized agent registry server via xDS or ext_proc APIs. A more detailed description of A2A implementation and agent discovery will be published in a forthcoming blog post.

5. Envoy is a complete solution for agentic networking challenges

Building on the same foundation that enabled policy application for MCP protocol in demanding deployments, Envoy is adding support for OpenAI and transcoding of agentic protocols into RESTful HTTP APIs. This transcoding capability simplifies the integration of gen AI agents with existing RESTful applications, with out-of-the-box support for OpenAPI-based applications and custom options via dynamic modules or Wasm extensions. In addition to transcoding, Envoy is being strengthened in critical areas for production readiness, such as advanced policy applications like quota management, comprehensive telemetry adhering to OpenTelemetry semantic conventions for generative AI systems, and integrated guardrails for secure agent operation.

Guardrails for safe agents
The next significant area of investment is centralized management and application of guardrails for all agentic traffic. Integrating policy enforcement points with external guardrails presently requires bespoke implementation and this problem area is ripe for standardization.

Control planes make this operational

The gateway is only part of the story. To achieve this policy management and rollout at scale, a separate control plane is required to dynamically configure the data plane using the xDS protocol, also known as the universal data plane API.

That is where control planes become important. Cloud Service Mesh, alongside open-source projects such as Envoy AI Gateway and kube-agentic-networking, uses Envoy as the data plane while giving operators higher-level ways to define and manage policy for agentic workloads.

This combination is powerful: Envoy provides the enforcement and extensibility in the traffic path, while control planes provide the operating model teams need to deploy that capability consistently.

Why this matters now

The shift towards agentic systems and gen AI protocols such as MCP, A2A, and OpenAI necessitates an evolution in network intermediaries. The primary complexities Envoy addresses include:

Deep protocol inspection. Protocol deframing extensions extract policy-relevant attributes (tool names, model names, resource paths) from the body of HTTP requests, enabling precise policy enforcement where traditional proxies would only see an opaque byte stream.
Fine-grained policy enforcement. By exposing these internal attributes, existing Envoy extensions like RBAC and ext_authz can evaluate policies based on protocol-specific criteria. This allows network operators to enforce a unified, zero-trust security posture, ensuring agents comply with access policies for specific tools or resources.
Stateful transport management. Envoy supports managing session state for the Streamable HTTP transport used by MCP, enabling robust deployments in both passthrough and aggregating gateway modes, even across a fleet of intermediaries.

Agentic AI protocols are still in their early stages, and the protocol landscape will continue to evolve. That’s exactly why the networking layer needs to be adaptable. Enterprises should not have to rebuild their security and traffic infrastructure every time a new agent framework, transport pattern, or tool protocol gains traction. They need a foundation that can absorb change without sacrificing control.

Envoy brings together three qualities that are hard to get in one place: proven production maturity, deep extensibility, and growing protocol awareness for agentic workloads. By leveraging Envoy as an agent gateway, organizations can decouple security and policy enforcement from agent development code.

That makes Envoy more than just a proxy that happens to handle AI traffic. It makes Envoy a future-ready foundation for agentic AI networking.

^{Special thanks to the additional co-authors of this blog: Boteng Yao, Software Engineer, Google and Tianyu Xia, Software Engineer, Google and Sisira Narayana, Sr Product Manager, Google.}

Activating Your Data Layer for Production-Ready AI

Thu, 02 Apr 2026 13:18:00 +0000

When discussing applications and systems using generative AI and the new opportunities they present, one component of the ecosystem is irreplaceable - data. Specifically, the data that companies gather, hold, and use daily. This data serves as the backbone for applications, analytics, knowledge bases, and much more. We use databases to store and work with this data, and most, if not all, AI-driven initiatives and new applications are going to use that data layer.

But how can we start to use the data in our AI systems? Let me introduce you to some of the labs showing how to prepare and use the data with AI models in Google databases.

Semantic Search: Text Embeddings in Database

Our journey starts by preparing our data for semantic search and running first tests to augment the Gen AI model's response by grounding it with your semantic search results. The grounding data is the basis for RAG (Retrieval Augmented Generation). Then, you can improve the performance of your search by indexing your embeddings using the latest indexing techniques.

One of the options is the Google AlloyDB database, which has direct integration with AI models and supports the most demanding workloads. The following lab guides us through all the steps, starting from creating an AlloyDB cluster, loading sample data, and generating embeddings, to using those embeddings to generate an augmented response from the Gen AI model.

aside_block: <ListValue: [StructValue([('title', 'Go to the lab!'), ('body', <wagtail.rich_text.RichText object at 0x7f36023f6550>), ('btn_text', ''), ('href', ''), ('image', None)])]>

AI integration is not limited to AlloyDB. All Google Cloud databases have AI integration and are capable of generating and using embeddings for semantic search. For example, if you are using Cloud SQL, you can also generate and use embeddings for semantic search directly within your existing PostgreSQL or MySQL instances.

The next two labs are very similar to the previous one, but instead of Google AlloyDB for PostgreSQL, we are using Cloud SQL for PostgreSQL and Cloud SQL for MySQL to use semantic search as the grounding engine for the model's response. Some steps are of course different due to variations in SQL language and different database engines, but the main idea stays the same: use our data to ground the model response and improve output.

aside_block: <ListValue: [StructValue([('title', 'Go to the labs!'), ('body', <wagtail.rich_text.RichText object at 0x7f36023f6d90>), ('btn_text', ''), ('href', ''), ('image', None)])]>

Semantic search using text data is one of the cornerstones and important features making responses much more reliable and useful, but Google Gen AI models can offer much more. Let's talk about multimodal search.

Multimodal Embeddings: Bring Images to the Search

In real life, of course, we use all our senses, including vision, to evaluate the world around us. The Google multimodal embedding models bring an additional layer of understanding, improving search by using embeddings not only for text but also for images.

In the following lab, we use a catalog of products placed in AlloyDB and supplemented by images in Google Cloud Storage. In the lab, we show how we can use both text descriptions and images for semantic search, supplementing and replacing each other, naturally incorporating search based on image input into our response.

aside_block: <ListValue: [StructValue([('title', 'Go to the lab!'), ('body', <wagtail.rich_text.RichText object at 0x7f36023f6790>), ('btn_text', ''), ('href', ''), ('image', None)])]>

Preparing the data and making first steps are important for a general understanding of RAG and tools available for the search, but Google has other cases when direct AI integration can help with your data analysis without any data preparations.

AlloyDB AI Functions and Reranking

Google AlloyDB database comes with additional AI integrations that help you use some AI capabilities without data preparation. For example, the AI.IF function can perform semantic search on the fly, evaluating sentiment or comparing data in columns with a natural language query, returning results filtered by the query condition. Also, you can apply a ranking function to the search output, improving the final result. You can try some of the new functionality using the following lab and let us know if it can help in your use case.

aside_block: <ListValue: [StructValue([('title', 'Go to the lab!'), ('body', <wagtail.rich_text.RichText object at 0x7f35ec6fa4c0>), ('btn_text', ''), ('href', ''), ('image', None)])]>

But what if somebody is not particularly savvy with SQL or not familiar with the data structure in your database? The AlloyDB NL2SQL can help you with that.

Generate SQL using AlloyDB AI Natural Language

The "alloydb_ai_nl" AlloyDB extension allows you not only to generate SQL queries based on default metadata available out-of-the-box but to build either automatic or custom context, helping to make the best of the query generation.

The NL2SQL functions can add a layer describing your data structure, relations between tables, and metadata based on real data samples from your tables without compromising the data itself, providing necessary information helping the AI model to understand how to build the best query. The following lab helps you to start with the new features and generate your first queries based on your data schema.

aside_block: <ListValue: [StructValue([('title', 'Go to the lab!'), ('body', <wagtail.rich_text.RichText object at 0x7f35ec6fa760>), ('btn_text', ''), ('href', ''), ('image', None)])]>

From Tests to Production

Those labs are part of the From Data Foundations to Advanced RAG module of our Production-Ready AI with Google Cloud program. Check the other modules and see if they can help you to adopt the AI capabilities provided by our Google Cloud services and tools. The end game goal is a high quality application using the full potential of modern technologies.

And stay tuned on release notes for ALloyDB and Cloud SQL - the engineering team is busy working on new features and improvements. Happy testing.

Create Expert Content: Architect A Personalized Multi-Agent System with Long-Term Memory

Tue, 31 Mar 2026 09:31:00 +0000

In support of our mission to accelerate the developer journey on Google Cloud, we built Dev Signal—a multi-agent system designed to transform raw community signals into reliable technical guidance by automating the path from discovery to expert creation.

In the first part of this series for the Dev Signal, we laid the essential groundwork for this system by establishing a project environment and equipping core capabilities through the Model Context Protocol (MCP). We standardized our external integrations, connecting to Reddit for trend discovery, Google Cloud Docs for technical grounding, and building a custom Nano Banana Pro MCP server for multimodal image generation. If you missed Part 1 or want to explore the code directly, you can find the complete project implementation in our GitHub repository.

Now, in Part 2, we focus on building the multi-agent architecture and integrating the Vertex AI memory bank to personalize these capabilities. We will implement a Root Orchestrator that manages three specialist agents: the Reddit Scanner, GCP Expert, and Blog Drafter, to provide a seamless flow from trend discovery to expert content creation. We will also integrate a long-term memory layer that enables the agent to learn from your feedback and persist your stylistic preferences across different conversations. This ensures that Dev Signal doesn't just process data, but actually learns to match your professional voice over time.

Infrastructure and Model Setup

First, we initialize the environment and the shared Gemini model.

Paste this code in dev_signal_agent/agent.py

code_block: <ListValue: [StructValue([('code', 'from google.adk.agents import Agent\r\nfrom google.adk.apps import App\r\nfrom google.adk.models import Gemini\r\nfrom google.adk.tools import google_search, AgentTool, load_memory_tool, preload_memory_tool\r\nfrom google.adk.tools.tool_context import ToolContext\r\nfrom google.genai import types\r\nfrom dev_signal_agent.app_utils.env import init_environment\r\nfrom dev_signal_agent.tools.mcp_config import (\r\n get_reddit_mcp_toolset, \r\n get_dk_mcp_toolset, \r\n get_nano_banana_mcp_toolset\r\n)\r\n\r\nPROJECT_ID, MODEL_LOC, SERVICE_LOC, SECRETS = init_environment()\r\n\r\n\r\nshared_model = Gemini(\r\n model="gemini-3-flash-preview", \r\n vertexai=True, \r\n project=PROJECT_ID, \r\n location=MODEL_LOC,\r\n retry_options=types.HttpRetryOptions(attempts=3),\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f35fffc7f70>)])]>

Memory Ingestion Logic

We want Dev Signal to do more than just follow instructions - we want it to learn from you. By capturing your preferences, such as specific technical interests on Reddit or a preferred blogging style, the agent can personalize its output for future use. To achieve this, we use the Vertex AI memory bank to persist session history across different conversations.

Long-term Memory

We automate this through the save_session_to_memory_callback function. This callback is configured to run automatically after every turn, ensuring that session details are captured and stored in the memory bank without manual intervention.

How Managed Memory Works:

Ingestion: The save_session_to_memory_callback sends the conversation data to Vertex AI.
Embedding: Vertex AI converts the text into numerical vectors (embeddings) that capture the semantic meaning of your preferences.
Storage: These vectors are stored in a managed index, enabling the agent to perform semantic searches and retrieve relevant history in future sessions.
Retrieval: The agent recalls this history using built-in ADK tools. The PreloadMemoryTool proactively brings in context at the start of an interaction, while the LoadMemoryTool allows the agent to fetch specific memories on an as-needed basis.

Paste this code in dev_signal_agent/agent.py

code_block: <ListValue: [StructValue([('code', 'async def save_session_to_memory_callback(*args, **kwargs) -> None:\r\n """\r\n Defensive callback to persist session history to the Vertex AI memory bank.\r\n """\r\n ctx = kwargs.get("callback_context") or (args[0] if args else None)\r\n \r\n # Check connection to Memory Service\r\n if ctx and hasattr(ctx, "_invocation_context") and ctx._invocation_context.memory_service:\r\n # Save the session!\r\n await ctx._invocation_context.memory_service.add_session_to_memory(\r\n ctx._invocation_context.session\r\n )'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f35fffc7d00>)])]>

Short-term Memory

The add_info_to_state function serves as the agent's short-term working memory, allowing the gcp_expert to reliably hand off its detailed findings to the blog_drafter within the same session. This working memory and the conversation transcript are managed by the Vertex AI Session Service to ensure that active context survives server restarts or transient failures.

The boundary between session-based state and long-term persistence - It is important to note that while this service provides stability during an active interaction, this short-term memory does not persist between different sessions. Starting a fresh session ID effectively resets this working state, ensuring a clean slate for new tasks. Cross-session continuity, where the agent remembers your stylistic preferences or past feedback, is handled by the Vertex AI Memory Bank.

Paste this code in dev_signal_agent/agent.py

code_block: <ListValue: [StructValue([('code', 'def add_info_to_state(tool_context: ToolContext, key: str, data: str) -> dict:\r\n tool_context.state[key] = data\r\n return {"status": "success", "message": f"Saved \'{key}\' to state."}'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f35fffc7df0>)])]>

Specialist 1: Reddit Scanner (Discovery)

The Reddit scanner is our “Trend Spotter," it identifies high-engagement questions from the last 21 days (3 weeks) to ensure that all research findings remain both timely and relevant.

Memory Usage: It leverages load_memory to retrieve your past areas of interest and preferred topics from the Vertex AI memory bank If relevant history exists, the agent prioritizes those specific topics in its search to provide a personalized discovery experience.

Beyond simple retrieval, each sub-agent actively updates its memories by listening for new preferences and explicitly acknowledging them during the chat. This process captures relevant information in the session history, where an automated callback then persists it to the long-term Vertex AI memory bank for future use.

This memory management is supported by two distinct retrieval patterns within the Google Agent Development Kit (ADK). The first is the PreloadMemoryTool, which proactively brings in historical context at the beginning of every interaction to ensure the agent is fully briefed before addressing the current request. The second is the LoadMemoryTool, which the agent uses on an as-needed basis, calling upon it only when it decides that deeper past knowledge would be beneficial for the current step in the workflow.

Paste this code in dev_signal_agent/agent.py

code_block: <ListValue: [StructValue([('code', '# Singleton toolsets\r\nreddit_mcp = get_reddit_mcp_toolset(\r\n client_id=SECRETS.get("REDDIT_CLIENT_ID", ""),\r\n client_secret=SECRETS.get("REDDIT_CLIENT_SECRET", ""),\r\n user_agent=SECRETS.get("REDDIT_USER_AGENT", "")\r\n)\r\nreddit_scanner = Agent(\r\n name="reddit_scanner",\r\n model=shared_model,\r\n instruction="""\r\n You are a Reddit research specialist. Your goal is to identify high-engagement questions \r\n from the last 3 weeks on specific topics of interest, such as AI/agents on Cloud Run.\r\n \r\n Follow these steps:\r\n 1. **MEMORY CHECK**: Use `load_memory` to retrieve the user\'s **past areas of interest** and **preferred topics**. Calibrate your search to align with these interests.\r\n 2. Use the Reddit MCP tools to search for relevant subreddits and posts.\r\n 3. Filter results for posts created within the last 21 days (3 weeks).\r\n 4. Analyze "high-engagement" based on upvote counts and the number of comments.\r\n 5. Recommend the most important and relevant questions for a technical audience.\r\n 6. **CRITICAL**: For each recommended question, provide a direct link to the original thread and a concise summary of the discussion.\r\n 7. **CAPTURE PREFERENCES**: Actively listen for user preferences, interests, or project details. Explicitly acknowledge them to ensure they are captured in the session history for future personalization.\r\n """,\r\n tools=[reddit_mcp, load_memory_tool.LoadMemoryTool()],\r\n after_agent_callback=save_session_to_memory_callback,\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f35fffc7b80>)])]>

Specialist 2: GCP Expert (Grounding)

The GCP expert is our "The Technical Authority". It triangulates facts by synthesizing official documentation from the Google Cloud Developer Knowledge MCP Server, community sentiment from Reddit, and broader context from Google Search.

Paste this code in dev_signal_agent/agent.py

code_block: <ListValue: [StructValue([('code', 'dk_mcp = get_dk_mcp_toolset(api_key=SECRETS.get("DK_API_KEY", ""))\r\n\r\n\r\nsearch_agent = Agent(\r\n name="search_agent",\r\n model=shared_model,\r\n instruction="Execute Google Searches and return raw, structured results (Title, Link, Snippet).",\r\n tools=[google_search],\r\n)\r\ngcp_expert = Agent(\r\n name="gcp_expert",\r\n model=shared_model,\r\n instruction="""\r\n You are a Google Cloud Platform (GCP) documentation expert. \r\n Your goal is to provide accurate, detailed, and cited answers to technical questions by synthesizing official documentation with community insights.\r\n \r\n For EVERY technical question, you MUST perform a comprehensive research sweep using ALL available tools:\r\n \r\n 1. **Official Docs (Grounding)**: Use DeveloperKnowledge MCP (`search_documents`) to find the definitive technical facts.\r\n 2. **Social Media Research (Reddit)**: Use the Reddit MCP to research the question on social media. This allows you to find real-world user discussions, common pain points, or alternative solutions that might not be in official documentation.\r\n 3. **Broader Context (Web/Social)**: Use the `search_agent` tool to find recent technical blogs, social media discussions, or tutorials.\r\n \r\n Synthesize your answer:\r\n - Start with the official answer based on GCP docs.\r\n - Add "Social Media Insights" or "Common Issues" sections derived from Reddit and Web Search findings.\r\n - **CRITICAL**: After providing your answer, you MUST use the `add_info_to_state` tool to save your full technical response under the key: `technical_research_findings`.\r\n - Cite your sources specifically at the end of your response, providing **direct links** (URLs) to the official documentation, blog posts, and Reddit threads used.\r\n - **CAPTURE PREFERENCES**: Actively listen for user preferences, interests, or project details. Explicitly acknowledge them to ensure they are captured in the session history for future personalization.\r\n """,\r\n tools=[dk_mcp, AgentTool(search_agent), reddit_mcp, add_info_to_state],\r\n after_agent_callback=save_session_to_memory_callback,\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f35fffc7c10>)])]>

Specialist 3: Blog Drafter (Creativity)

The blog drafter is our Content Creator. It drafts the blog based on the expert's findings and offers to generate visuals.

Memory Usage: It checks load_memory for the user's preferred writing style (e.g. "Witty", "Rap") stored in the Vertex AI memory bank.

Paste this code in dev_signal_agent/agent.py

code_block: <ListValue: [StructValue([('code', 'nano_mcp = get_nano_banana_mcp_toolset()\r\n\r\n\r\nblog_drafter = Agent(\r\n name="blog_drafter",\r\n model=shared_model,\r\n instruction="""\r\n You are a professional technical blogger specializing in Google Cloud Platform. \r\n Your goal is to draft high-quality blog posts based on technical research provided by the GDE expert and reliable documentation.\r\n \r\n You have access to the research findings from the gcp_expert_agent here:\r\n {{ technical_research_findings }}\r\n \r\n Follow these steps:\r\n 1. **MEMORY CHECK**: Use `load_memory` to retrieve past blog posts, **areas of interest**, and user feedback on writing style. Adopt the user\'s preferred style and depth.\r\n 2. **REVIEW & GROUND**: Review the technical research findings provided above. **CRITICAL**: Use the `dk_mcp` (Developer Knowledge) tool to verify key facts, technical limitations, and API details. Ensure every claim in your blog is grounded in official documentation.\r\n 3. Draft a blog post that is engaging, accurate, and helpful for a technical audience.\r\n 4. Include code snippets or architectural diagrams if relevant.\r\n 5. Provide a "Resources" section with links to the official documentation used.\r\n 6. Ensure the tone is professional yet accessible, while adhering to any style preferences found in memory.\r\n 7. **VISUALS**: After presenting the drafted blog post, explicitly ask the user: "Would you like me to generate an infographic-style header image to illustrate these key points?" If they agree, use the `generate_image` tool (Nano Banana).\r\n 8. **CAPTURE PREFERENCES**: Actively listen for user preferences, interests, or project details. Explicitly acknowledge them to ensure they are captured in the session history for future personalization.\r\n """,\r\n tools=[dk_mcp, load_memory_tool.LoadMemoryTool(), nano_mcp],\r\n after_agent_callback=save_session_to_memory_callback,\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f3600e56190>)])]>

The Root Orchestrator

The root agent serves as the system's strategist, managing a team of specialist agents and orchestrating their actions based on the specific goals provided by the user. At the start of a conversation, the orchestrator retrieves memory to establish context by checking for the user's past areas of interest, preferred topics, or previous projects.

Paste this code in dev_signal_agent/agent.py

code_block: <ListValue: [StructValue([('code', 'root_agent = Agent(\r\n name="root_orchestrator",\r\n model=shared_model,\r\n instruction="""\r\n You are a technical content strategist. You manage three specialists:\r\n 1. reddit_scanner: Finds trending questions and high-engagement topics on Reddit.\r\n 2. gcp_expert: Provides technical answers based on official GCP documentation.\r\n 3. blog_drafter: Writes professional blog posts based on technical research.\r\n \r\n Your responsibilities:\r\n - **MEMORY CHECK**: At the start of a conversation, use `load_memory` to check if the user has specific **areas of interest**, preferred topics, or past projects. Tailor your suggestions accordingly.\r\n - **CAPTURE PREFERENCES**: Actively listen for user preferences, interests, or project details. Explicitly acknowledge them to ensure they are captured in the session history for future personalization.\r\n - If the user wants to find trending topics or questions from Reddit, delegate to reddit_scanner.\r\n - If the user has a technical question or wants to research a specific theme, delegate to gcp_expert.\r\n - **CRITICAL**: After the gcp_expert provides an answer, you MUST ask the user: \r\n "Would you like me to draft a technical blog post based on this answer?"\r\n - If the user agrees or asks to write a blog, delegate to blog_drafter.\r\n - Be proactive in helping the user navigate from discovery (Reddit) to research (Docs) to content creation (Blog).\r\n """,\r\n tools=[load_memory_tool.LoadMemoryTool(), preload_memory_tool.PreloadMemoryTool()],\r\n after_agent_callback=save_session_to_memory_callback,\r\n sub_agents=[reddit_scanner, gcp_expert, blog_drafter]\r\n)\r\n\r\napp = App(root_agent=root_agent, name="dev_signal_agent")'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f3600e56a00>)])]>

Summary

In this part of our series, we built multi-agent architecture and implemented a robust, dual-layered memory system. We established a Root Orchestrator, managing three specialist agents: a Reddit Scanner for trend discovery, a GCP Expert for technical grounding, and a Blog Drafter for creative content creation.

By utilizing short-term state to pass information reliably between specialists and integrating the Vertex AI memory bank for long-term persistence, we’ve enabled the agent to learn from your feedback and remember specific writing styles across different conversations.

In part 3, we will show you how to test the agent locally to verify these components on your workstation, before transitioning to a full production deployment on Google Cloud Run in part 4. Can't wait for Part 3? The full implementation is already available for you to explore on GitHub.

To learn more about the underlying technology, explore the Vertex AI Memory Bank overview or dive into the official ADK Documentation to see how to orchestrate complex multi-agent workflows.

Special thanks to Remigiusz Samborski for the helpful review and feedback on this article.

For more content like this, Follow me on Linkedin and X.

Five techniques to reach the efficient frontier of LLM inference

Fri, 27 Mar 2026 10:02:00 +0000

Every dollar that you spend on model inference buys you a position on a graph of latency and throughput. On this plot is a curve of optimal configurations, where you've squeezed the maximum possible performance from your hardware. That curve, borrowed from portfolio theory in finance, is the efficient frontier.

With the assumption that you have a fixed budget for hardware, you can trade latency for throughput. But, you can't improve one aspect without sacrificing the other, unless the frontier curve itself moves. There are two fundamentally different dynamics at play, and this is the central insight for anyone running LLMs in production.

The first dynamic is getting to the frontier, which involves applying the full stack of techniques available to you today. This part is within your control. Continuous batching, paged attention, intelligent routing, speculative decoding, and quantization all exist right now. If you're not using these techniques, you're operating below the frontier and leaving performance on the table.

The second dynamic is that the frontier itself is constantly moving outward. This part is largely outside of your control. Researchers publish new algorithms. Hardware vendors ship new architectures. Open-source projects mature. Each breakthrough redefines what's physically achievable and expands the curve so that yesterday's optimal configuration is today's inefficiency.

Your job as a platform engineer is to stay as close to the frontier as possible as you build infrastructure that's flexible enough to absorb each new advance as it arrives. This article gives you the tools to do just that.

Why inference has an efficient frontier

Every LLM request has two computational phases, and they can have bottlenecks for different hardware resources.

1. Prefill (Compute-Bound): In this phase, the GPU processes your entire input prompt at one time to build the key-value (KV) cache for the attention mechanism. Because the instructions are batch-processed in parallel, the GPU's compute cores (tensor cores) are highly utilized. This phase is fast and efficient: the processors have all of the data that they need, immediately available, to perform massive matrix multiplications. Longer prompts just mean more computations.

2. Decode (Memory-Bandwidth-Bound): This phase generates new tokens, one at a time, autoregressively. To generate only one single token, the GPU can't batch the work. It must fetch the entire model's weights and the growing KV cache from High-Bandwidth Memory (HBM) into the compute cores. Then, the GPU needs to calculate that one token, and then waits to do it all over again for the next one.

This mismatch is the fundamental reason that the frontier exists. You can't optimize a single system for both phases simultaneously without making some tradeoffs.

The two axes of inference

Instead of risk and return, the efficient frontier of LLM inference measures a different fundamental tradeoff, with the assumption that the hardware budget is fixed:

Axis	Key metrics measured	Hardware constraint
Latency (the X-Axis)	Time to First Token (TTFT) + Time Between Tokens (TBT)	Compute (prefill) and memory bandwidth (decode)
Throughput (the Y-Axis)	Total tokens per second across all concurrent users	Batch size × memory capacity

Cost is the constraint that buys the graph of latency and throughput itself. If you increase your hardware budget, or the industry invents a new algorithmic breakthrough, the entire frontier curve shifts outward. For a given budget and software stack, you can apply today's best practices to move from a sub-optimal point towards that frontier.

Getting to the frontier: Five techniques within your control

Most production inference systems today operate below the frontier. They're leaving performance on the table, not because better techniques don't exist, but because they haven't adopted them yet. Everything described in this section is available today. If you're not applying these techniques, you're choosing to operate below the curve.

1. Semantic routing across model tiers

Not every query needs a 400B parameter model. Simple classification, summarization, or formatting tasks can be routed to smaller, quantized models that are orders of magnitude cheaper per token. A lightweight classifier at the gateway edge analyzes query complexity and routes accordingly: frontier-class models for hard reasoning, and small models for everything else.

Semantic routing pushes your system dramatically closer to its theoretical maximum throughput, and avoids wasted cycles on easy tasks, without sacrificing aggregate output quality.

2. Prefill and decode disaggregation

Physically separating prefill and decode phases onto different hardware is one of the most architecturally significant optimizations available today.

The prefill phase needs compute-dense GPUs. The decode phase needs high-bandwidth memory. If you force both phases onto the same GPU, then one resource is always underutilized.

To push both phases toward their theoretical hardware limits independently, run dedicated prefill clusters and decode clusters. Connect these clusters with high-speed networks that transfer only the compressed KV cache state to the same GPU, then one resource is always underutilized.

3. Quantization: Trading precision for speed

When you reduce model weights from FP16 to the INT8 or INT4 formats, you can reduce the memory footprint to half or a quarter. Because the decode phase is memory-bandwidth-bound, 4-bit weights can be read up to 4× faster than 16-bit weights. This approach provides a direct TBT improvement.

The tradeoff is quality because naive quantization degrades model outputs. Modern techniques like Activation-aware Weight Quantization (AWQ) and GPTQ preserve the quality of sensitive weights, but aggressively compress others, to achieve near-FP16 quality at INT4 speeds.

4. Context routing: The biggest lever that most teams miss

In a production deployment with dozens of model replicas, the routing layer is where the biggest competitive advantages are won or lost today.

In 2026, prefix caching is foundational. If ten users ask questions about the exact same 100-page RAG document, or use the identical massive system prompt, you shouldn't run the compute-heavy prefill phase ten times. You should compute the KV cache once, store it, and then let the other nine users reuse it. This approach slashes TTFT by up to 85% and drastically reduces compute costs.

But, there's a catch: a standard L4 load balancer scatters requests randomly. If user 2's request lands on a different GPU than user 1's request, the prefix cache is useless. The system has to recompute the cache from scratch.

This is why context-aware L7 routing is the differentiator. An intelligent router inspects the incoming prompt's prefix and intentionally routes the request to the specific pod that already holds that context in its cache. You stop wasting compute power on redundant work and instantly push your latency and throughput closer to the physical limits of your hardware.

5. Speculative decoding

Remember: during the decode phase, tensor cores are mostly idle because there's a bottleneck on memory bandwidth. Speculative decoding exploits this wasted computation power.

A small, fast "draft" model generates several candidate tokens cheaply. The large target model then verifies all of the candidates in a single forward pass, which is a parallel compute-bound operation, rather than a sequential memory-bound one. If the draft model predicted the candidates correctly, you've generated 4-5 tokens for the memory cost of one.

This approach directly breaks the TBT floor set by memory bandwidth. If you're not using speculative decoding for latency-sensitive workloads, you're not leveraging one of the most impactful optimizations available.

Although the addition of a draft model can introduce some operational complexity and slightly increase compute costs, the draft model is relatively tiny compared to the main model. This tradeoff for latency is worthwhile.

Note that some newer models have introduced self-speculative decoding, which eliminates the overhead of managing a second model. These models use specialized internal layers (often called prediction heads) that are trained to predict extra future tokens simultaneously. These models generally achieve a highly meaningful token hit rate.

Case study: How Vertex AI moved closer to the frontier

The Vertex AI engineering team moved closer to the frontier when they adopted GKE Inference Gateway, which is built on the standard Kubernetes Gateway API. Inference Gateway intercepted requests at Layer 7 and added two critical layers of intelligence:

Load-aware routing: It scraped real-time metrics (like KV cache utilization and queue depth) directly from the model server's Prometheus endpoints. This process routes requests to the pod that can serve them the fastest.
Content-aware routing: Crucially, it inspected request prefixes and routed traffic to the pod that already held that specific context in its KV cache. This process avoids expensive re-computation.

When the production workloads were migrated to this intelligent routing architecture, the Vertex AI team proved that optimizing the network layer is key to unlocking performance at scale. Validated on production traffic, the results were stark:

35% faster TTFT for Qwen3-Coder (context-heavy coding agent workloads)
2x better P95 tail latency (52% improvement) for DeepSeek V3.1 (bursty chat workloads)
Doubled prefix cache hit rate (optimized from 35% to 70%)

The bottom line

LLM inference has an efficient frontier, which represents a hard boundary where latency and throughput are optimally balanced for a given compute budget.

Getting to that frontier is within your control. The techniques exist today: continuous batching, paged attention, intelligent L7 routing, speculative decoding, quantization, and prefill and decode disaggregation. The GKE Inference Gateway case study shows that routing alone, without changing hardware, models, or cluster size, cut TTFT by 35% and doubled cache efficiency. If you're not applying the full stack, you're operating below the curve and overpaying for every token.

The frontier itself keeps moving outward. This part is outside of your control. Researchers publish new algorithms, hardware vendors ship new architectures, and open-source serving frameworks integrate these algorithms and architectures. Something that was cutting-edge optimization 18 months ago became a baseline table stake. Your job isn't to predict which breakthrough comes next; it's to build infrastructure flexible enough to absorb it when it arrives.

The organizations that will win on inference economics aren't the ones with the most GPUs. They're the ones that systematically close the gap to today's frontier while they stay ready for tomorrow's.

Have you applied any of these optimization techniques to your own LLM inference workloads? I'd love to hear about your experience! Share what you've built with me on LinkedIn, X, or Bluesky!

The new AI literacy: Insights from student developers

Thu, 26 Mar 2026 17:00:00 +0000

AI has made it easier than ever for student developers to work efficiently, tackle harder problems, and pursue ambitious projects. But for students earning technical degrees, these new capabilities also create genuine tensions around learning.

How much should I use AI? What should I use it for?

As 90% of technology professionals now use AI in their daily work according to Google's DORA 2025 report, understanding how the next generation navigates these tools matters more than ever. Contrary to fears that students use AI to cheat or are becoming intellectually lazy, our research with UC Berkeley students reveals something different. Students treated AI as a learning partner rather than a shortcut, using it strategically for some tasks while deliberately turning it off for others.

As AI becomes foundational to software development, the question isn't whether to adopt these tools but how to work with them thoughtfully. The students at UC Berkeley are showing us one answer: with curiosity, caution, and a commitment to genuine learning that technology can support but never replace.

The research

Our team of four student researchers (Andrew Harlan, Mindy Tsai, Kenny Ly, and Karissa Wong) conducted a mixed methods research project with UC Berkeley students in Computer Science, Electrical Engineering, Design, and Data Science to understand how they're integrating AI into their academic work.

A separate UC Berkeley study (conducted by Edward Fraser, Jessie Deng, and Eileen Thai) used eye-tracking technology to observe how developers with one to five years of experience actually interact with AI coding assistants. Both student teams were supported by dedicated mentors, with Googlers Harini Sampath, Becky Sohn, and Derek DeBellis advising the mixed methods research, and UC Berkeley Professor John Chuang, PhD, advising the eye-tracking study.

Together, these studies reveal three key insights about how students balance AI's capabilities with their need to develop genuine expertise. The patterns emerging among students closely mirror what DORA research has found in professional developers.

Finding #1: The 24/7 office hour

AI as a tutor, not a shortcut

When asked to describe their relationship with AI, every student in our study used educational terms. They referred to AI as a "tutor" or "teacher," not an assistant or productivity tool.

"AI is a teacher...in the sense that it is most helpful for understanding dense content and potentially parts of code that are prewritten in the database to allow for fundamental understanding of the project."

"I use [AI] as my own private tutor...to [cover] any specific topics in the classes or lectures...not just in CS classes but in all classes."

This framing matters because it reveals strategic use rather than dependency. Rather than asking AI to complete assignments, students described using AI metacognitively to identify gaps in their knowledge, clarify confusing concepts, and guide their learning process. They used AI to summarize academic papers mentioned in lectures so they could decide which ones warranted deeper reading. They asked AI to explain why their code produced specific errors.

One student explained their workflow:

"When I don't understand what my professor is explaining, I ask AI to help me understand the concept or what a piece of code is doing. If I don't know how to begin a lab, I give the prompt to AI to figure out where to start, then write the code myself and ask AI to correct my work."

For students with learning disabilities, this constant availability addresses a real access gap:

"As a student with a learning disability, I need more time to understand a problem. AI has helped me a lot—it's like having a 24/7 TA."

By extending access beyond limited office hours, AI allows students to iterate on their understanding without waiting for help. This frees up cognitive space for higher-level thinking:

"I spend less time actually coding and more time on big picture ideation. Now, my time is spent thinking through logic, concepts, and coming up with ideas creatively, rather than producing code manually."

These accounts portray AI as a scaffold for exploration rather than a producer of finished work. This mirrors what DORA research found: when AI handles routine toil, developers can focus more energy on delivering user value.

Finding #2: Active resistance to overdependence

Building guardrails to protect learning

Despite embracing AI as a learning tool, students expressed genuine anxiety about becoming too dependent on it.

"If AI disappeared, I'd struggle more with figuring out how to solve things on my own."

In a recent study using EEG to measure brain activity during essay writing, researchers found that AI users showed weaker cognitive engagement patterns compared to those using search engines or no tools, and frequent AI users who later wrote without assistance remembered less of their content and felt less ownership over it, what the authors termed "cognitive debt”.¹

Our research revealed a positive signal: rather than passively accepting this risk, students responded by establishing deliberate boundaries.

One mechanical engineering student described how she's developed a competency-based system over years of working with electronics:

"When I use basic sensors like a servo or ultrasonic, I can still code that myself. But when I have more complex sensors where I don't necessarily know the exact functions, that's when I'll use AI." She explained her reasoning: "I have the background to understand why things aren't working, but I don't always know the direct language to fix it, so AI is good for helping overcome that."

For a recent project building a tactile storytelling tool, she knew the basic concept but needed help structuring the counting and comparison system. "AI was really useful in setting up that structure, but I still had to code after to fine-tune it." She's clear about the division of labor: "I'm still working with doing the code myself. I wouldn't say that I'm just handing it off like a technical expert. I'm working in tandem with it. I have to be the initiator of what I want it to actually do. If I just give it a blind request, it's not useful at all."

Even when students do engage AI, they often set explicit rules:

"Sometimes I tell AI not to give me the full answer, just to guide me in the right direction."

Students have developed several specific strategies to prevent overreliance:

Limiting access to powerful models:

"I don't want to pay for AI tools because it could lead me to overuse the models."

Alternating between assisted and unassisted work:

"I have actually gone back to hand-coding for certain things, like a for-loop for example."

Warning against "vibe coding":

"AI tools can definitely be a good companion to boost developer productivity. However, one needs to be very mindful and not get used to vibe coding. It's very important to understand and validate the code AI is generating and use it appropriately."

This anxiety is itself metacognitive awareness. Students recognize that the path of least resistance may not be the path of greatest learning. This mirrors DORA's findings: despite 90% adoption, about 30% of practitioners report little to no trust in AI-generated code. Effective AI use requires mastering critical evaluation and verification, not just adoption.

Finding #3: Knowing when to use AI and when to turn it off

What the eye-tracking data reveals

A separate study using eye-tracking technology provides behavioral validation. When researchers observed developers with one to five years of experience interacting with AI coding assistants, they found stark differences in AI engagement depending on task type:

During interpretive tasks requiring deep understanding: <1% visual attention on AI
During mechanical tasks like boilerplate code: 19% visual attention on AI

Developers actively ignored AI suggestions during complex work, even when those suggestions were accurate and could save time. AI creates cognitive load during deep understanding work, and experienced developers know when to turn it off.

Strategic selectivity, not blanket adoption

Students in our interviews echoed this context-dependent approach:

"I typically use AI to generate ideas for a starting point."

"Despite knowing AI was allowed, I wanted to go through the friction of learning and failing and having space for creativity."

Customization matters

Most AI coding assistants now let developers toggle inline suggestions, enable on-demand only modes, or adjust suggestion frequency. By experimenting with these settings, developers can align AI behavior with the cognitive demands of different tasks, reducing disruption during deep work while maintaining assistance for routine tasks.

What this means for the industry

Students are modeling the future of AI-augmented development

The students in these studies are ahead of the curve. They've developed a literacy that knows when to engage AI, how to verify its output, and when to work manually to preserve understanding. For teams navigating AI adoption, the student experience offers direction:

Experiment with customization to find configurations that support rather than disrupt work
Build verification practices into workflows rather than accepting suggestions uncritically
Create space for unassisted work on complex problems where understanding matters more than speed

To learn more about how professionals across the industry are navigating AI adoption, download the DORA 2025 State of AI-assisted Software Development Report. You can also read the full research articles from our collaboration with researchers at UC Berkeley.

^{1. Kosmyna, Nataliya, et al. "Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task." arXiv, 10 June 2025, doi:10.48550/arXiv.2506.08872. Accessed 28 Jan. 2026.}

Building Distributed AI Agents

Wed, 18 Mar 2026 19:00:00 +0000

Let's be honest: building an AI agent that works once is easy. Building an AI agent that works reliably in production, integrated with your existing React or Node.js application? That's a whole different ball game.

(TL;DR: Want to jump straight to the code? Check out the Course Creator Agent Architecture on GitHub.)

We've all been there. You have a complex workflow—maybe it's researching a topic, generating content, and then grading it. You shove it all into one massive Python script or a giant prompt. It works on your machine, but the moment you try to hook it up to your sleek frontend, things get messy. Latency spikes, debugging becomes a nightmare, and scaling is impossible without duplicating the entire monolith.

But what if you didn't have to rewrite your entire application to accommodate AI? What if you could just... plug it in?

In this post, we're going to explore a better way: the orchestrator pattern. Instead of just one powerful agent that does everything, we'll build a team of specialized, distributed microservices. This approach lets you integrate powerful AI capabilities directly into your existing frontend applications without the headache of a monolithic rewrite.

We'll use Google's Agent Development Kit (ADK) to build the agents, the Agent-to-Agent (A2A) protocol to connect them and let them communicate with each other, and deploy them as scalable microservices on Cloud Run.

Why Distributed Agents? (And Why Your Frontend Team Will Love You)

Imagine you have a polished Next.js application. You want to add a "Course Creator" feature.

If you build a monolithic agent, your frontend has to wait for a single, long-running process to finish everything. If the research part hangs, the whole request times out. Additionally, you won’t have the opportunity to scale separate agents as needed. For example, if your judge agent requires more processing, you’ll have to scale all your agents up, instead of just the judge agent.

By adopting a distributed orchestrator pattern, you gain scalability and flexibility:

Seamless integration: Your frontend talks to one endpoint (the orchestrator), which manages the chaos behind the scenes.
Independent scaling: Is the judge step slow? Scale just that service to 100 instances. Your research service can stay small.
Modularity: You can write the high-performance networking parts in Go and the data science parts in Python. They just speak HTTP.

The Blueprint: Course Creator App

Let's build that course creator system. We'll break it down into three distinct specialists:

The researcher: A specialist that digs up information.
The judge: A QA specialist that ensures quality.
The orchestrator: The manager that coordinates the work and talks to your frontend.

Step 1: Hiring the Specialist (The Researcher)

First, we need someone to do the legwork. We'll build a focused agent using ADK whose only job is to use Google Search.

code_block: <ListValue: [StructValue([('code', '# researcher/app/agent.py\r\nfrom google.adk.agents import Agent\r\nfrom google.adk.tools import google_search\r\n\r\nresearcher = Agent(\r\n name="researcher",\r\n model="gemini-2.5-flash",\r\n description="Gathers information on a topic using Google Search.",\r\n instruction="""\r\n You are an expert researcher. Your goal is to find comprehensive information.\r\n Use the `google_search` tool to find relevant information.\r\n Summarize your findings clearly.\r\n """,\r\n tools=[google_search],\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f360075e8e0>)])]>

See? Simple. It doesn't know about courses or frontends. It just researches.

Step 2: The Judge (Structured Output)

We can't have our agents rambling. We need strict pass or fail grades so our code can make decisions. We use Pydantic to enforce this contract.

code_block: <ListValue: [StructValue([('code', '# judge/app/agent.py\r\nfrom pydantic import BaseModel, Field\r\nfrom typing import Literal\r\n\r\nclass JudgeFeedback(BaseModel):\r\n status: Literal["pass", "fail"] = Field(\r\n description="Whether the research is sufficient (\'pass\') or needs more work (\'fail\')."\r\n )\r\n feedback: str = Field(\r\n description="Detailed feedback on what is missing."\r\n )\r\n\r\njudge = Agent(\r\n name="judge",\r\n model="gemini-2.5-flash",\r\n description="Evaluates research findings.",\r\n instruction="""\r\n You are a strict editor. Evaluate the findings.\r\n If they are missing key info, output status=\'fail\' and provide feedback.\r\n """,\r\n output_schema=JudgeFeedback, # Enforce the contract!\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f360075e1c0>)])]>

Now, when the judge speaks, it speaks JSON. Your application logic can trust it.

Step 3: The Universal Language (A2A Protocol)

Here's the magic. We wrap these agents as web services using the A2A Protocol. Think of it as a universal language for agents. It lets them describe what they do (agent.json) and talk over standard HTTP.

code_block: <ListValue: [StructValue([('code', '# researcher/app/server.py\r\nfrom fastapi import FastAPI\r\nfrom a2a.server.apps import A2AFastAPIApplication\r\nfrom app.agent import app as adk_app\r\n\r\n# ... setup runner ...\r\n\r\n# Create the A2A App wrapper\r\na2a_app = A2AFastAPIApplication(agent_card=agent_card, http_handler=request_handler)\r\n\r\napp = FastAPI(lifespan=lifespan)\r\n\r\n# Register routes: /.well-known/agent.json and /rpc\r\na2a_app.add_routes_to_app(app)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f360075ef70>)])]>

Now, your researcher is a microservice running on port 8000. It's ready to be called by anyone—including your orchestrator.

Step 4: The Orchestrator Pattern

This is where it all comes together. The orchestrator is the general contractor. It doesn't do the research; it hires the researcher. It doesn't make judgments; it asks the judge.

Crucially, this is the only agent your frontend needs to know about.

code_block: <ListValue: [StructValue([('code', '# orchestrator/app/agent.py\r\nfrom google.adk.agents import LoopAgent, SequentialAgent\r\nfrom google.adk.agents.remote_a2a_agent import RemoteA2aAgent\r\n\r\n# Connect to the remote Researcher service\r\nresearcher = RemoteA2aAgent(\r\n name="researcher",\r\n agent_card="http://researcher-service:8000/.well-known/agent.json",\r\n description="Gathers information on a topic."\r\n)\r\n\r\n# Connect to the remote Judge service\r\njudge = RemoteA2aAgent(\r\n name="judge",\r\n agent_card="http://judge-service:8000/.well-known/agent.json",\r\n description="Evaluates research findings."\r\n)\r\n\r\n# The Orchestrator manages the loop\r\nresearch_loop = LoopAgent(\r\n name="research_loop",\r\n sub_agents=[researcher, judge, escalation_checker],\r\n max_iterations=3,\r\n)\r\n\r\n# The full pipeline\r\nroot_agent = SequentialAgent(\r\n name="course_creation_pipeline",\r\n sub_agents=[research_loop, content_builder],\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f360075ea30>)])]>

The orchestrator handles the complexity—retries, loops, state management—so your frontend stays clean and simple.

Deployment: The "Grocery Store" Model

Deploying this system on Cloud Run gives you what I call the "grocery store" model. If the checkout lines (researcher tasks) get long, you don't build a new store. You just open more registers. Cloud Run scales your researcher service independently to handle the load, while your judge service stays lean.

Caveats & Security Considerations

Of course, with great power comes great responsibility (and security reviews).

Authentication: In this demo, agents talk over open HTTP. In production, you must lock this down. Use mTLS, OIDC, or API keys to ensure that only your orchestrator can talk to your researcher.
Latency: Every hop adds time. Use this pattern for coarse-grained tasks (like "research this topic") rather than chatty, low-level interactions.
Error handling: Networks fail. Your orchestrator needs to be robust enough to handle timeouts and retries gracefully.

Ready to Build?

Stop trying to build one giant agent that does it all. By using the orchestrator pattern and distributed microservices, you can build AI systems that are scalable, maintainable, and—best of all—play nicely with the apps that you already have.

Want to see the code? Check out the full Course Creator Agent Architecture on GitHub.

And if you're ready to deploy, get started with Cloud Run, ADK, and A2A to bring your agent team to life.

Create Expert Content: Building Capabilities for a Multi-Agent System with Google ADK, MCP, and Cloud Run

Wed, 18 Mar 2026 09:18:00 +0000

My team’s mission is to accelerate the developer journey from writing code to running secure AI workloads on Google Cloud. To help developers succeed, we focus on identifying their most pressing questions and building demos that provide straightforward, easy-to-implement solutions.

Recently, I was struck with inspiration when the new Developer Knowledge MCP server was released. It led me to build Dev Signal—a multi-agent system designed with Google Agent Development Kit (ADK)—to identify technical questions from Reddit, research them using official documentation, and draft detailed technical blogs. Dev Signal also provides custom visuals using Nano Banana Pro. I even integrated a long-term memory layer so the agent remembers my specific preferences and blogging style.

By connecting my coding assistant, Gemini CLI, to the developer knowledge MCP server, I built and deployed this entire system to Google Cloud Run in just two days.

Whether you want to learn how to architect a complex multi-agent system with long term memory, leverage local and remote MCP servers for tool standardization, or write detailed Terraform scripts for secure Cloud Run deployment, I'll show you how!

If you’d rather dive straight into the code and explore it at your own pace, you can clone the repository here.

What you'll learn

In this four-part blog series, I’ll walk you through the step-by-step process of how I brought this project to life. Each blog post captures the journey of building and deploying Dev Signal:

Part 1: Tools for building agent capabilities – You’ll begin by setting up your project environment and equipping your agent with tools using the Model Context Protocol (MCP). You’ll learn how to connect to Reddit for trend discovery, Google Cloud docs for technical grounding, and a custom Nano Banana Pro tool for image generation.
Part 2: The Multi-Agent Architecture with long term memory – You’ll build the "brain" of the system by implementing a root orchestrator and a team of specialized agents. You’ll also integrate the Vertex AI memory bank, enabling the agent to learn and persist your preferences across sessions.
Part 3: Testing the agent Locally – Before moving to the cloud, you’ll synchronize the agent's components and verify its performance on your workstation. You’ll use a dedicated test runner to simulate the full lifecycle of discovery, research, and multimodal creation, with a special focus on validating long-term memory persistence by connecting your local agent directly to the cloud-based Vertex AI memory bank.
Part 4: Deployment to Cloud Run and the Path to Production – Finally, you’ll deploy your service on Google Cloud Run using Terraform for reproducible infrastructure. You’ll also discuss the next steps required for a high quality secure production system.

Getting started with Dev Signal

Dev Signal is an intelligent monitoring agent designed to filter noise and create value. Dev Signal operates in the following ways:

Discovery: Scouts Reddit for high-engagement technical questions.
Grounding: Researches answers using official Google Cloud documentation to ensure accuracy.
Creation: Drafts professional technical blog posts based on its findings.
Multimodal Generation: Generates custom infographic headers for those posts.
Long-Term Memory: Uses Vertex AI memory bank to remember your feedback across different sessions.

Prerequisites

Before you begin, verify the following is installed:

Python 3.12+
uv (Python package manager): curl -LsSf https://astral.sh/uv/install.sh | sh
Google Cloud SDK (gcloud CLI) installed and authenticated.
Terraform (for infrastructure as code).
Node.js & npm (required for the Reddit MCP tool).

You will also need:

A Google Cloud Project with billing enabled.
APIs Enabled: Vertex AI, Cloud Run, Secret Manager, Artifact Registry.
Reddit API Credentials (Client ID, Secret) - You can get these from the Reddit Developer Portal.
Developer Knowledge API Key (for Google Cloud docs search) - Instructions on how to get it are here.

Project Setup

The Dev Signal system was built by first running the Agent Starter Pack, following the automated architect workflow described in the Agent Factory episode by Remigiusz Samborski and Vlad Kolesnikov. This foundation provided the project’s modular directory structure, which is used to separate concerns between Agent Logic, Server Code, Utilities, and Tools.

The starter pack acts as a powerful starting point because it automates the creation of professional infrastructure, CI/CD pipelines, and observability tools in seconds. This allows you to focus entirely on the agent’s unique intelligence while ensuring the underlying platform remains secure and scalable. By building on top of this generated boilerplate with AI assistance from Gemini CLI and Antigravity, the development process is highly accelerated.

The agent starter pack high level architecture:

1. Initialize the Project

Create a new directory for your project and initialize it. We'll use uv, which is an extremely fast Python package manager.

code_block: <ListValue: [StructValue([('code', 'uv init dev-signal'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f35eca546a0>)])]>

2. Folder Structure

Our project will follow this structure. We will populate these files step-by-step.

code_block: <ListValue: [StructValue([('code', 'dev-signal/\r\n├── dev_signal_agent/\r\n│ ├── __init__.py\r\n│ ├── agent.py # Agent logic & orchestration\r\n│ ├── fast_api_app.py # Application server & memory connection\r\n│ ├── app_utils/ # Env Config\r\n│ │ └── env.py\r\n│ └── tools/ # External capabilities\r\n│ ├── __init__.py\r\n│ ├── mcp_config.py # Tool configuration (Reddit, Docs)\r\n│ └── nano_banana_mcp/# Custom local image generation tool\r\n│ ├── __init__.py\r\n│ ├── main.py\r\n│ ├── nano_banana_pro.py\r\n│ ├── media_models.py\r\n│ ├── storage_utils.py\r\n│ └── requirements.txt\r\n├── deployment/\r\n│ └── terraform/ # Infrastructure as Code\r\n├── .env # Local secrets (API keys)\r\n├── Makefile # Shortcuts for building/deploying\r\n├── Dockerfile # Container definition\r\n└── pyproject.toml # Dependencies'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f35eca543a0>)])]>

3. Define Dependencies

Update your pyproject.toml with the necessary dependencies. We use google-adk for the agent framework and google-genai for the model interaction.

code_block: <ListValue: [StructValue([('code', '[project]\r\nname = "dev-signal"\r\nversion = "0.1.0"\r\ndescription = "A multi-agent system for monitoring and content creation."\r\nreadme = "README.md"\r\nrequires-python = ">=3.12, <3.14"\r\ndependencies = [\r\n "google-adk>=0.1.0",\r\n \xa0"google-genai>=1.0.0",\r\n "mcp>=1.0.0",\r\n \xa0"python-dotenv>=1.0.0",\r\n "fastapi>=0.110.0",\r\n "uvicorn>=0.29.0",\r\n "google-cloud-logging>=3.0.0",\r\n "google-cloud-aiplatform>=1.38.0",\r\n \xa0"fastmcp>=2.13.0",\r\n "google-cloud-storage>=3.6.0",\r\n "google-auth>=2.0.0",\r\n "google-cloud-secret-manager>=2.26.0",\r\n]'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f35eca54fa0>)])]>

Run uv sync to install everything.

Create a new directory for the agent code.

code_block: <ListValue: [StructValue([('code', 'mkdir dev_signal_agent\r\ncd dev_signal_agent'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f35eca54eb0>)])]>

Building the agent capabilities: MCP tools

Our agent needs to interact with the outside world. We use the Model Context Protocol (MCP) to standardize this. The Model Context Protocol (MCP) is a universal standard for connecting AI agents to external data and tools. Instead of writing custom API wrappers, we use standard MCP servers. This allows us to connect to APIs (Reddit), Knowledge Bases (Google Cloud Docs), and even local scripts (Image Generation using Nano Banana Pro) using a common interface. Create a new directory for the agent tools.

code_block: <ListValue: [StructValue([('code', 'mkdir tools\r\ncd tools'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f35eca540d0>)])]>

Tools Configuration

We'll define our toolsets in dev_signal_agent/tools/mcp_config.py.

This file defines the connection parameters for our three main tools.

Reddit: Connected via a local stdio subprocess.
Developer Knowledge: Connected via a remote HTTP endpoint.
Nano Banana: Connected via a local stdio subprocess (our custom Python script).

Reddit Search (Discovery Tool)

The Reddit MCP server acts as a bridge to the Reddit API, allowing your agent to discover trending posts and analyze engagement without you having to write complex API wrappers. To ensure portability, the code uses a "find or fetch" strategy: it first checks for a local installation and, if missing, automatically uses npx to download and run the server on demand.

Instead of a network connection, the agent launches the server as a local subprocess and communicates via standard input and output (stdio). Within the Google ADK, the McpToolset class acts as a universal wrapper that standardizes these connections, enabling your agent to interact with various tools, from community resources to custom scripts like the Nano Banana image generator, using a common interface. By securely passing API credentials through environment variables, the system ensures these "plug-and-play" modules function as a seamless bridge between the AI and external platforms.

Paste this code in dev_signal_agent/tools/mcp_config.py:

code_block: <ListValue: [StructValue([('code', 'import os\r\nimport shutil\r\nfrom mcp import StdioServerParameters\r\nfrom google.adk.tools import McpToolset\r\nfrom google.adk.tools.mcp_tool import StreamableHTTPConnectionParams, StdioConnectionParams\r\n\r\ndef get_reddit_mcp_toolset(client_id: str = "", client_secret: str = "", user_agent: str = ""):\r\n """\r\n Connects to the Reddit MCP server.\r\n This server runs as a local subprocess (stdio) and proxies requests to the Reddit API.\r\n """\r\n # Check if \'reddit-mcp\' is installed globally, otherwise use npx to run it\r\n cmd = "reddit-mcp" if shutil.which("reddit-mcp") else "npx"\r\n args = [] if shutil.which("reddit-mcp") else ["-y", "--quiet", "reddit-mcp"]\r\n \r\n # Inject secrets into the environment of the subprocess only\r\n env = {\r\n **os.environ, \r\n "DOTENV_CONFIG_SILENT": "true", \r\n "LANG": "en_US.UTF-8"\r\n }\r\n\r\n if client_id: env["REDDIT_CLIENT_ID"] = client_id\r\n if client_secret: env["REDDIT_CLIENT_SECRET"] = client_secret\r\n if user_agent: env["REDDIT_USER_AGENT"] = user_agent\r\n\r\n return McpToolset(\r\n connection_params=StdioConnectionParams(\r\n server_params=StdioServerParameters(\r\n command=cmd, \r\n args=args, \r\n env=env # Pass injected secrets directly to the subprocess\r\n ),\r\n timeout=120.0\r\n )\r\n )'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f35eca54400>)])]>

Google Cloud Docs (Knowledge Tool)

The Developer Knowledge MCP server provides grounding for your agent by allowing it to search the entire corpus of official Google Cloud documentation. Unlike the local Reddit server, this is a managed service hosted by Google and accessed as a remote endpoint over the internet. It exposes specialized tools like google_developer_documentation_search for semantic queries and google_developer_documentation_fetch to retrieve full markdown content, ensuring that every technical claim the agent makes is supported by definitive, up-to-date facts.

Note: you can also connect your coding assistant tools such as Gemini CLI or Antigravity to the developer knowledge MCP server to empower them with handy up to date Google Cloud documentation. I used it when writing this blog!

To connect, the agent uses the McpToolset class with StreamableHTTPConnectionParams, pointing to a web URL instead of launching a local process. It securely authenticates using a DK_API_KEY (create your api key) passed in the request headers, allowing the agent to perform a "comprehensive research sweep" across official docs, community sentiment, and broader web context through a single standardized interface.

Paste this code in dev_signal_agent/tools/mcp_config.py:

code_block: <ListValue: [StructValue([('code', 'def get_dk_mcp_toolset(api_key: str = ""):\r\n """\r\n Connects to Developer Knowledge (Google Cloud Docs).\r\n This is a remote MCP server accessed via HTTP.\r\n """\r\n headers = {}\r\n if api_key:\r\n headers["X-Goog-Api-Key"] = api_key\r\n else:\r\n # Fallback to os.environ for local testing if not passed via API\r\n headers["X-Goog-Api-Key"] = os.getenv("DK_API_KEY", "")\r\n\r\n return McpToolset(\r\n connection_params=StreamableHTTPConnectionParams(\r\n url="https://developerknowledge.googleapis.com/mcp",\r\n headers=headers\r\n )\r\n )'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f35eca54190>)])]>

The Image Generator (Nano Banana MCP)

While we've used external MCP servers for Reddit and documentation, we can also build our own custom MCP server to wrap specific Python logic. In this case, we are creating an image generation tool powered by Gemini 3 Pro Image (also known as Nano Banana Pro). This demonstrates that any Python function can be standardized into a tool that any agent can understand.

How the image generation works:

FastMCP: We use the fastmcp library to drastically simplify server creation, allowing us to register Python functions as tools with just a few lines of code.
Gemini Integration: The server uses the Google GenAI SDK to call the gemini-3-pro-image-preview model, which converts the agent's descriptive prompts into raw image bytes.
GCS Upload & Hosting: Because agent interfaces typically require a URL to display images, the server automatically uploads the generated bytes to Google Cloud Storage (GCS) and returns a public link.

To connect this local tool, we use StdioConnectionParams because the server runs as a local subprocess communicating via standard input and output. This transport method directly matches the transport="stdio" configuration we will define in our server entrypoint, ensuring a seamless connection for your custom local scripts.

The following code defines the MCP connection in dev_signal_agent/tools/mcp_config.py. We use uv run to ensure the server starts in an isolated environment with all its dependencies correctly installed.

Paste this code in dev_signal_agent/tools/mcp_config.py:

code_block: <ListValue: [StructValue([('code', 'def get_nano_banana_mcp_toolset():\r\n """\r\n Connects to our local \'Nano Banana\' image generator.\r\n This demonstrates how to wrap a local Python script as an MCP tool.\r\n """\r\n path = os.path.join("dev_signal_agent", "tools", "nano_banana_mcp", "main.py")\r\n bucket = os.getenv("AI_ASSETS_BUCKET") \r\n return McpToolset(\r\n connection_params=StdioConnectionParams(\r\n server_params=StdioServerParameters(\r\n command="uv", \r\n args=["run", path], \r\n env={**os.environ, "AI_ASSETS_BUCKET": bucket}\r\n ),\r\n timeout=600.0 # Image generation can take time\r\n )\r\n )'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f35eca54e20>)])]>

Implementing the Nano Banana Pro Server Logic

Now, we will implement the actual logic for this server. This implementation is based on the Agent Factory demo code by Remigiusz Samborski. While Remi's original code provides instructions for deploying the MCP server to Cloud Run, we will run it here as a local subprocess for faster development and testing.

To get started, create the directory for our new server:

code_block: <ListValue: [StructValue([('code', 'mkdir -p dev_signal_agent/tools/nano_banana_mcp\r\ncd dev_signal_agent/tools/nano_banana_mcp'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f36026c5a60>)])]>

The Server Entrypoint (`main.py` )

This file acts as the "brain" that initializes and starts the MCP server.

FastMCP Initialization: We use the FastMCP library to create a server named "MediaGenerators" and register our generate_image function as a tool
Safe Logging: The _initialize_console_logging function is critical. It forces all logs to sys.stderr. This is because the MCP "stdio" transport uses sys.stdout for communication between the agent and the tool; standard logs sent to stdout would corrupt that protocol.
Execution: The mcp.run(transport="stdio") line starts the server as a local subprocess, allowing it to listen for requests from your agent via standard input.

Paste this code in dev_signal_agent/tools/nano_banana_mcp/main.py

code_block: <ListValue: [StructValue([('code', 'import logging\r\nimport os\r\nimport sys\r\nfrom fastmcp import FastMCP\r\nfrom dotenv import load_dotenv\r\nfrom nano_banana_pro import generate_image\r\n\r\ndef _initialize_console_logging(min_level: int = logging.INFO):\r\n # Ensure logs go to STDERR so they don\'t break the MCP stdio protocol\r\n handler = logging.StreamHandler(sys.stderr)\r\n logging.basicConfig(level=min_level, handlers=[handler], force=True)\r\n\r\ntools = [generate_image]\r\nmcp = FastMCP(name="MediaGenerators", tools=tools)\r\n\r\nif __name__ == "__main__":\r\n load_dotenv()\r\n _initialize_console_logging()\r\n mcp.run(transport="stdio")'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f36026c5580>)])]>

The Generation Logic (`nano_banana_pro.py)`

This is where the actual image generation happens using Gemini.

GenAI Client: We initialize the genai.Client() to interact with Google's generative models.
Model Selection: It specifically targets the gemini-3-pro-image-preview model. We set the response_modalities to "IMAGE" to tell the model we want pixels, not just text.
Robustness: The code includes a MAX_RETRIES loop (set to 5) to handle any transient generation errors, ensuring the agent has multiple attempts to get a valid image.
Byte Processing: Once the model generates the image, it arrives as raw inline data. We extract these bytes and call our helper to move them to the cloud.
URI Conversion: Finally, it replaces the internal gs:// path with a browser-accessible https:// URL so the user can actually see the image.

Paste this code in dev_signal_agent/tools/nano_banana_mcp/nano_banana_pro.py

code_block: <ListValue: [StructValue([('code', 'import logging\r\nfrom typing import Literal, Optional\r\nfrom google import genai\r\nfrom google.genai import types\r\nfrom media_models import MediaAsset\r\nfrom storage_utils import upload_data_to_gcs\r\n\r\nAUTHORIZED_URI = "https://storage.mtls.cloud.google.com/"\r\nMAX_RETRIES = 5\r\n\r\nasync def generate_image(\r\n prompt: str,\r\n aspect_ratio: Literal["16:9", "9:16"] = "16:9",\r\n) -> MediaAsset:\r\n """Generates an image using Gemini 3 Image model."""\r\n genai_client = genai.Client()\r\n content = types.Content(parts=[types.Part.from_text(text=prompt)], role="user")\r\n \r\n logging.info(f"Starting image generation for prompt: {prompt[:50]}...")\r\n asset = MediaAsset(uri="")\r\n \r\n for _ in range(MAX_RETRIES):\r\n response = genai_client.models.generate_content(\r\n model="gemini-3-pro-image-preview",\r\n contents=[content],\r\n config=types.GenerateContentConfig(\r\n response_modalities=["IMAGE"],\r\n image_config=types.ImageConfig(aspect_ratio=aspect_ratio)\r\n )\r\n )\r\n if response and response.parts:\r\n for part in response.parts:\r\n if part.inline_data and part.inline_data.data:\r\n # Upload the raw bytes to GCS\r\n gcs_uri = await upload_data_to_gcs(\r\n "mcp-tools",\r\n part.inline_data.data,\r\n part.inline_data.mime_type\r\n )\r\n asset = MediaAsset(uri=gcs_uri)\r\n break\r\n if asset.uri: break\r\n\r\n if not asset.uri:\r\n asset.error = "No image was generated."\r\n else:\r\n # Convert gs:// URI to an HTTP accessible URL if needed\r\n asset.uri = asset.uri.replace(\'gs://\', AUTHORIZED_URI)\r\n logging.info(f"Image URL: {asset.uri}")\r\n \r\n return asset'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f36026c54c0>)])]>

GCS Upload Helper (`storage_utils.py)`

Since agents need a web link to display images, this utility handles the hosting on Google Cloud Storage (GCS).

Dynamic Bucket Selection: It looks for a bucket name in your environment variables, falling back from AI_ASSETS_BUCKET to LOGS_BUCKET_NAME to ensure it always has a place to save data.
Unique Filenames: We use an MD5 hash of the raw image data to create a unique filename. This prevents filename collisions and acts as a simple way to avoid duplicate uploads of the same image.
Cloud Upload: The blob.upload_from_string method pushes the raw image bytes directly to your GCS bucket.

Paste this code in dev_signal_agent/tools/nano_banana_mcp/storage_utils.py

code_block: <ListValue: [StructValue([('code', 'import hashlib\r\nimport mimetypes\r\nimport os\r\nfrom google.cloud.storage import Client, Blob\r\nfrom dotenv import load_dotenv\r\n\r\nload_dotenv()\r\nstorage_client = Client()\r\nai_bucket_name = os.environ.get("AI_ASSETS_BUCKET") or os.environ.get("LOGS_BUCKET_NAME")\r\nai_bucket = storage_client.bucket(ai_bucket_name)\r\n\r\nasync def upload_data_to_gcs(agent_id: str, data: bytes, mime_type: str) -> str:\r\n file_name = hashlib.md5(data).hexdigest()\r\n ext = mimetypes.guess_extension(mime_type) or ""\r\n blob_name = f"assets/{agent_id}/{file_name}{ext}"\r\n blob = Blob(bucket=ai_bucket, name=blob_name)\r\n blob.upload_from_string(data, content_type=mime_type, client=storage_client)\r\n return f"gs://{ai_bucket_name}/{blob_name}"'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f36026c5490>)])]>

Data Model (`media_models.py`)

This file ensures that our data follows a strict structure (Schema).

Structured Output: By using a Pydantic BaseModel, we guarantee that the tool always returns a consistent JSON object containing a uri (the link) and an optional error message. This makes it much easier for the AI agent to understand and process the tool's result.

Paste this code in dev_signal_agent/tools/nano_banana_mcp/media_models.py

code_block: <ListValue: [StructValue([('code', 'from typing import Optional\r\nfrom pydantic import BaseModel\r\n\r\nclass MediaAsset(BaseModel):\r\n uri: str\r\n error: Optional[str] = None'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f36026c58b0>)])]>

Tool Dependencies (`requirements.txt)`

While we use uv to run our code, a requirements.txt file remains essential because it defines the specific dependencies uv needs to install for the Nano Banana server to function. This provides the necessary "ingredients" to set up the isolated environment before the server starts.

This file lists the three core libraries required for this tool:

google-cloud-storage: Used for hosting the generated images on the cloud.
google-genai: Provides the logic for the Gemini 3 Pro image generation.
fastmcp: The framework that turns our Python script into a standardized MCP tool.

Paste this code in dev_signal_agent/tools/nano_banana_mcp/requirements.txt

code_block: <ListValue: [StructValue([('code', 'google-cloud-storage==3.6.*\r\ngoogle-genai==1.52.*\r\nfastmcp==2.13.*'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f36026c5c40>)])]>

Summary

In this first part of our series, we focused on establishing the agent's core capabilities by standardizing its external integrations through the Model Context Protocol (MCP). We initialized the project using uv for high-speed dependency management and successfully configured three critical toolsets: Reddit for trend discovery, Google Cloud Docs for technical grounding, and a custom Nano Banana MCP server for multimodal image generation. By utilizing the Google ADK’s McpToolset, we’ve abstracted away complex API logic into simple, plug-and-play modules, ensuring that our tools share a common interface that decouples integration from intelligence.

For a deeper look into our technical foundation, you can explore the Developer Knowledge MCP server to learn more about knowledge grounding or visit the Google ADK GitHub repository to explore the framework's core capabilities

With our toolset fully configured and ready for action, we can now move to Part 2, where we will build the multi-agent architecture and integrate the Vertex AI memory bank to orchestrate these capabilities. You can also jump ahead to Part 3, where we will show you how to test the agent locally to verify these components on your workstation. If you’d like to dive ahead, you can explore the complete code for the entire series in our GitHub repository.

Special thanks to Remigiusz Samborski for the helpful review and feedback on this article.

For more content like this, follow me on Linkedin and X.

Introducing multi-cluster GKE Inference Gateway: Scale AI workloads around the world

Tue, 17 Mar 2026 16:00:00 +0000

The world of artificial intelligence is moving fast, and so is the need to serve models reliably and at scale. Today, we're thrilled to announce the preview of multi-cluster GKE Inference Gateway to enhance the scalability, resilience, and efficiency of your AI/ML inference workloads across multiple Google Kubernetes Engine (GKE) clusters — even those spanning different Google Cloud regions.

Built as an extension of the GKE Gateway API, the multi-cluster Inference Gateway leverages the power of multi-cluster Gateways to provide intelligent, model-aware load balancing for your most demanding AI applications.

Why multi-cluster for AI inference?

As AI models grow in complexity and users become more global, single-cluster deployments can face limitations:

Availability risks: Regional outages or cluster maintenance can impact service.
Scalability caps: Hitting hardware limits (GPUs/TPUs) within a single cluster or region.
Resource silos: Underutilized accelerator capacity in one cluster can’t be used by another
Latency: Users far from your serving cluster may experience higher latency

The multi-cluster GKE Inference Gateway addresses these challenges head-on, providing a variety of features and benefits:

Enhanced high reliability and fault tolerance: Intelligently route traffic across multiple GKE clusters, including across different regions. If one cluster or region experiences issues, traffic is automatically re-routed, minimizing downtime.
Improved scalability and optimized resource usage: Pool and leverage GPU/TPU resources from various clusters. Handle demand spikes by bursting beyond the capacity of a single cluster and efficiently utilize available accelerators across your entire fleet.
Globally optimized, model-aware routing: The Inference Gateway can make smart routing decisions using advanced signals. With GCPBackendPolicy, you can configure load balancing based on real-time custom metrics, such as the model server's KV cache utilization metric, so that requests are sent to the best-equipped backend instance. Other modes like in-flight request limits are also supported.
Simplified operations: Manage traffic to a globally distributed AI service through a single Inference Gateway configuration in a dedicated GKE "config cluster," while your models run in multiple "target clusters."

How it works

In GKE Inference Gateway there are two foundational resources, InferencePool and InferenceObjective. An InferencePool acts as a resource group for pods that share the same compute hardware (like GPUs or TPUs) and model configuration, helping to ensure scalable and high-availability serving. An InferenceObjective defines the specific model names and assigns serving priorities, allowing Inference Gateway to intelligently route traffic and multiplex latency-sensitive tasks alongside less urgent workloads.

With this release, the system uses Kubernetes Custom Resources to manage your distributed inference service. InferencePool resources in each "target cluster" group model-server backends. These backends are exported and become visible as GCPInferencePoolImport resources in the "config cluster." Standard Gateway and HTTPRoute resources in the config cluster define the entry point and routing rules, directing traffic to these imported pools. Fine-grained load-balancing behaviors, such as using CUSTOM_METRICS or IN_FLIGHT requests, are configured using the GCPBackendPolicy resource attached to GCPInferencePoolImport.

This architecture enables use cases like global low-latency serving, disaster recovery, capacity bursting, and efficient use of heterogeneous hardware.

For more information about GKE Inference Gateway core concepts check out our guide.

Get started today

As you scale your AI inference serving workloads to more users in more places, we're excited for you to try multi-cluster GKE Inference Gateway. To learn more and get started, check out the documentation:

Build Resilient LLM Applications on Vertex AI and Reduce 429 Errors

Thu, 12 Mar 2026 16:00:00 +0000

Building applications powered by Large Language Models (LLMs) on Vertex AI is exciting, but hitting a 429 error can be a frustrating roadblock. These errors signal that your requests are coming in faster than the service can handle them at that moment.

Last year, we published a guide to handling these 429 errors. In this article, we’ll dig deeper into Vertex AI’s consumption models and dives into architectural best practices for managing request flows. This way, you can build smooth, resilient, and truly scalable AI applications.

Choosing the right consumption option

Vertex AI provides a range of consumption models designed to accommodate various API traffic types and volumes. Your primary strategy for minimizing 429 errors is selecting the consumption model that best aligns with your application’s unique traffic patterns.

Default options: The default option with Gemini on Vertex AI is Standard Pay-as-you-go (Paygo). For Standard Pay-as-you-go (Paygo) traffic, Vertex AI uses a system with Usage Tiers. This dynamic approach allocates resources from a shared pool, where your organization’s historical spend determines your Usage Tier and baseline throughput (TPM). This baseline provides a predictable performance floor for typical workloads, while still allowing your application to burst beyond it on a best-effort basis.

If your application generates critical, user-facing traffic that can be unpredictable and require higher reliability than Standard Paygo, Priority Paygo is designed for you. By adding the priority header to your requests, you signal that this traffic should be prioritized, reducing the likelihood of being throttled.

For applications with consistently high volumes of real-time traffic, Provisioned Throughput (PT) is the only consumption option that provides isolation from the shared PayGo pool, offering a stable experience even during heavy contention on PayGo. With PT, you reserve and pay for a guaranteed throughput, ensuring your important traffic flows smoothly. To learn more about PT on Vertex AI, visit our guide here.

Cost-effective options: For traffic that isn't latency sensitive, Vertex AI offers more cost-effective options. The Flex PayGo is suited for latency-tolerant traffic, processing requests at a lower price. Large-scale, asynchronous jobs, such as offline analysis or bulk data enrichment, are best handled by Batch. This service manages the entire workflow, including scaling and retries, over a longer period (around 24 hours), insulating your main application from this heavy load.

Complex applications and hybrid approaches: Complex applications often leverage a hybrid approach: PT for essential real-time traffic, Priority Paygo for fluctuating traffic, Standard Paygo for general requests, and Batch/Flex for latency-tolerant and offline request flows.

Five ways to reduce 429 errors on Vertex AI

1. Implement smart retries

When your application encounters a temporary overload error like a 429 (Resource Exhausted) or 503 (Service Unavailable), an immediate retry is not recommended. The best practice is to implement a retry strategy called Exponential Backoff with Jitter. Exponential backoff means that the delay between retry attempts increases exponentially usually up to a predefined maximum delay. This gives the service time to recover from the overload condition.

SDK & libraries: The Google Gen AI SDK includes native retry behavior that can be configured via HttpRetryOptions in client parameters. However, you can also leverage specialized libraries like Tenacity (for Python) or build a custom solution. For a deeper dive, refer to this blog post.
Agentic workflows: For developing agents, the Agent Development Kit (ADK) offers a Reflect and Retry plugin that builds resilience into AI workflows by automatically intercepting 429 errors.
Infrastructure & Gateway: Another robust option for building resilience is circuit breaking with Apigee, which enables you to manage traffic distribution and implement graceful failure handling.

2. Leverage global model routing

Vertex AI's infrastructure is distributed across multiple regions. By default, if you target a specific regional endpoint, your request is served from that region. This means your application's availability is tied to the capacity of that single region. This is where the global endpoint becomes an effective tool for enhancing availability and resilience. Instead of being locked into one region, the global endpoint routes your traffic across a fleet of regions where there may be more availability, reducing the potential error rate.

3. Reduce payload via context caching

An effective way to reduce the load on Vertex AI is to avoid making calls for repetitive queries. Many production applications, especially chatbots and support systems, see similar questions asked frequently. Instead of re-processing these, you can implement context caching. With Context Caching, Gemini reuses precomputed cached tokens, allowing you to reduce your API traffic and throughput. This not only saves costs but also reduces latency for repeated content within your prompts.

4. Optimize prompts

Reducing the token count in each request directly lowers your TPM consumption and costs.

Summarization with Flash-Lite: Before sending a long conversation history to a model like Gemini Pro, use a lightweight model like Gemini 2.5 Flash-Lite to summarize the context.
Agent memory optimization: For Agentic workloads you can leverage Vertex AI Agent Engine Memory Bank. Features like Memory Extraction and Consolidation allow you to distill meaningful facts from a conversation, ensuring your agent remains context-aware without raw chat history.
Prompt hygiene: Review your prompts and reduce overly verbose JSON schema descriptions (if the model is already familiar) and stripping excessive whitespace or redundant formatting.

5. Shape traffic

Sudden bursts of requests are a primary cause of 429 errors. Even if your average traffic rate is low, sharp spikes can strain resources. The goal is to smoothen traffic, spreading requests out over time.

Get started

Ready to put these patterns into practice? Explore the Vertex AI samples on GitHub, or jumpstart your next project with the Google Cloud Beginner’s Guide, Vertex AI quickstart or start building your next AI agent with the Agent Development Kit (ADK)

Calling all devs: Build the future of Multimodal AI in the Gemini Live Agent Challenge

Fri, 06 Mar 2026 17:00:00 +0000

Hey builders! Stop typing, and start interacting! We are moving beyond the text box. The future of AI is all about immersive, real-time experiences. To celebrate multimodal AI, we’re challenging you to build the next generation of agents that can help you see, hear, speak, and create in the Gemini Live Agent Challenge.

Build Multimodal AI agents in the Gemini Live Agent Challenge

Why join?

Hands-on learning with Gemini Live API: This is your shot to build the future of immersive AI agents on Google Cloud. We have everything you need to get started: Quickstarts, tutorials, access to the Agent Development Kit (ADK), and webinars hosted by our experts.
Showcase your skills: You’ll have the opportunity to break out of the traditional "text box" paradigm. Choose from three exciting categories—The Live Agent, The Creative Storyteller, or The UI Navigator—to demonstrate the power of your solution .

Think you have what it takes to win?

Build a solution to showcase your multimodal agent and you could potentially win a share of $80,000 in prizes:

Overall grand prize: A trip to Google Cloud Next ’26 in Las Vegas (includes tickets, a travel stipend, and a chance to present on stage), $25,000 in USD, $3,000 in Google Cloud Credits for use with a Cloud Billing Account, virtual coffee with a Google Cloud team member, and the potential opportunity to be featured on our social channels.
Category winners: A trip to Google Cloud Next ’26 in Las Vegas (includes tickets), $10,000 in USD, $1,000 in Google Cloud Credits for use with a Cloud Billing Account, virtual coffee with a Google Cloud team member, and the potential opportunity to be featured on our social channels.
Subcategory winners: $5,000 in USD and $500 in Google Cloud Credits for use with a Cloud Billing Account
Honorable mentions: $2,000 in USD and $500 in Google Cloud Credits for use with a Cloud Billing Account

Dig into Multimodal AI

Your mission is to build and deploy an AI agent on Google Cloud that utilizes multimodal inputs and outputs. We want you to go beyond the traditional text-in/text-out approach.

Whether you are building a real-time translator or a visual web navigator, your agent should interpret the world around it. Here is some inspiration:

The live agent: Build an agent we can talk to naturally that handles interruptions gracefully. Think real-time translators, vision-enabled tutors, and more.
The creative storyteller: Blend text, images, audio, and video into one seamless experience. Imagine building an interactive storybook or a full marketing asset generator in a single workflow.
The UI navigator: Create a helping hand that interprets visual screens. Maybe you want to create a universal web navigator or a visual QA tester that performs actions based on user intent.

Crucial note: Your project must use a Gemini model (like Gemini 3 or Nano Banana) and the Gen AI SDK or Agent Development Kit (ADK). Lastly, you must use at least one Google Cloud service, such as Firestore, CloudSQL, Cloud Run, or Vertex AI.

Ready to start building?

Head over to our hackathon website to register, watch the kickoff video, and review the official rules. Submissions are open until March 16, 2026.

Register for the Gemini Live Agent Challenge

Cost-Effective AI with Ollama, GKE GPU Sharing, and vCluster

Fri, 06 Mar 2026 16:00:00 +0000

As organizations scale their AI workloads, two major challenges often emerge: the high cost of underutilized GPUs and the operational complexity of managing isolated environments for multiple teams. Traditionally, assigning a whole GPU to a single pod is inefficient, but managing separate clusters for every team is operationally heavy.

In this post, we'll demonstrate how to solve both problems by combining Google Kubernetes Engine (GKE) GPU time-sharing with vCluster for multi-tenancy. We'll deploy Ollama to serve open models (like Mistral) in isolated virtual environments that share the same physical GPU infrastructure.

The Architecture: Virtual Clusters on Shared Hardware

The architecture leverages GKE Autopilot to abstract away the physical infrastructure. Instead of managing nodes, you simply deploy workloads, and Autopilot provisions the necessary hardware on demand, including GPUs, drivers, etc.

This setup lets teams have their own isolated environments, APIs, and Ollama instances, and potentially different models, while running on the same cost-effective, shared GPU nodes. For example, Team A (e.g., Legal Research) and Team B (e.g., Customer Support) can work in separate environments while they share GPU resources.

vCluster lets you create virtual Kubernetes clusters on top of an existing Kubernetes cluster. It supports various tenancy modes, including the shared nodes model that's shown in the diagram, where each virtual cluster gets its own isolated control plane while sharing the underlying worker nodes. Each virtual cluster can be accessed independently by teams who get full admin access to their cluster without interfering with others. This model also lets you leverage host cluster features when needed, and you have the ability to deploy your own controllers and CRDs inside each virtual cluster.

When you use vCluster, you can use any of these tenancy modes:

Shared nodes: The shared nodes mode allows multiple virtual clusters to run workloads on the same physical Kubernetes nodes. This configuration is ideal for scenarios where maximizing resource utilization is a top priority—especially for internal developer environments, CI/CD pipelines, and cost-sensitive use cases.
Private nodes: Using private nodes is a mode for vCluster where, instead of sharing the host cluster's worker nodes, individual worker nodes are joined to a vCluster. These private nodes act as the vCluster's worker nodes and they aren't shared with other vClusters on the same host cluster.
Auto nodes: You can configure vCluster to automatically provision and join worker nodes based on the node and resource requirements. To use auto nodes, you need vCluster Platform installed and vCluster needs to be connected to it.
Standalone: vCluster Standalone is a different architecture mode for vCluster for the control plane and node. The standalone mode doesn't require a host cluster. vCluster is deployed directly onto nodes like other Kubernetes distributions. vCluster Standalone can run on any type of node, whether it's a bare-metal node or a VM. It provides the strictest isolation for workloads because there's no shared host cluster for the control plane or worker nodes.

Deployment

To follow along on the deployment steps, make sure that you have the following installed:

Step 1: Set up and Create the GKE Autopilot Cluster

Unlike GKE Standard, we don't need to calculate node counts or configure node pools manually. Instead, we'll automatically create the cluster and then get credentials.

Set environment variables and create a GKE Autopilot cluster:

export PROJECT_ID=YOUR_PROJECT_ID
export REGION=YOUR_REGION_ID
# Create GKE Autopilot cluster
gcloud container clusters create-auto vcluster-gpu-sharing \
  --region=$REGION --project $PROJECT_ID

Replace YOUR_PROJECT_ID and YOUR_REGION_ID with the Google Cloud project and region that you want to use.

Get the credentials to configure your local kubectl:

gcloud container clusters get-credentials vcluster-gpu-sharing \
  --region $REGION --project $PROJECT_ID

Step 2: Create Virtual Clusters (vClusters)

With the Autopilot cluster running, we can now create isolated environments for our tenants. We'll create two vClusters, demo1 and demo2. You'll need a vcluster.yaml manifest file for configuration.

When you use GKE Autopilot, it might take a few minutes to create the first vCluster. This is because vCluster waits for its own control plane pods to be up and running. Because Autopilot provisions the underlying nodes dynamically in response to this new workload, there's a brief delay while the infrastructure is initialized.

code_block: <ListValue: [StructValue([('code', '# Create the vcluster configuration file\r\ncat <<EOF > vcluster.yaml\r\n# Place your vCluster configuration here. \r\n# For GPU workloads on GKE Autopilot, this typically involves \r\n# enabling node synchronization so the vCluster can see the \r\n# underlying GPU nodes provided by Autopilot.\r\nsync:\r\n fromHost:\r\n ingressClasses:\r\n enabled: true\r\n nodes:\r\n enabled: true\r\n toHost:\r\n ingresses:\r\n enabled: true\r\nEOF\r\n\r\n# Create the first virtual cluster\r\nvcluster create demo1 -n demo1 -f vcluster.yaml\r\n\r\n# Create the second virtual cluster\r\nvcluster create demo2 -n demo2 -f vcluster.yaml'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3601e8a370>)])]>

Note: If you receive an error warning that you're trying to create a vCluster inside another, select no and then switch back to the correct host context.

Step 3: Deploy Ollama to the Virtual Cluster

We start by creating the deployment manifest for Ollama. This manifest deploys Ollama and uses a Kubernetes Service to expose it on port 11434.

Create the deployment manifest for Ollama. This manifest deploys Ollama and it uses a Kubernetes Service to expose it on port 11434. Nodes are selected that use GPU time-sharing.

# Create Ollama deployment manifest
cat <<EOF > ollama.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
 name: ollama
 namespace: default
spec:
 replicas: 1
 selector:
   matchLabels:
     app: ollama
 template:
   metadata:
     labels:
       app: ollama
   spec:
     nodeSelector:
    # Selects nodes that use GPU time-sharing.
    # Selects nodes that allow a specific number of containers
    #  to share the underlying GPU.
    # Select nodes with Nvidia L4 GPUs
       cloud.google.com/gke-gpu-sharing-strategy: "time-sharing"
       cloud.google.com/gke-max-shared-clients-per-gpu: "5"
       cloud.google.com/gke-accelerator: nvidia-l4
     containers:
     - name: ollama
       image: ollama/ollama:latest
       ports:
       - containerPort: 11434
       resources:
         limits:
           nvidia.com/gpu: 1
---
apiVersion: v1
kind: Service
metadata:
 name: ollama
 namespace: default
spec:
 selector:
   app: ollama
 ports:
 - port: 11434
   targetPort: 11434
 type: ClusterIP
EOF

When the vCluster is active, switch contexts to work inside demo1:

# Connect to the virtual cluster demo1
vcluster connect demo1 -n demo1

Deploy Ollama in the virtual environment:
```
# Apply your deployment manifest
kubectl apply -f ollama.yaml
```
Even though we're in a virtual cluster, when we create pods that request GPUs, the request is synced to the host. GKE Autopilot detects this requirement and automatically attaches the necessary GPU hardware to the nodes that are running your workloads.

Step 4: Pulling and Testing the Model

With the server running, perform the model pull and test entirely within the virtual cluster context:
```
# Execute the pull command inside the pod
kubectl exec -it <pod-name> -- ollama pull mistral
```

Verify the API:

# Port forward the Ollama service
kubectl port-forward svc/ollama 8080:11434
# Send a chat request in a new window
curl -s http://localhost:8080/api/chat \
 -H "Content-Type: application/json" \
 -d '{ "model": "mistral", "stream": false, "messages": [ {"role": "user", "content": "Explain GKE Autopilot"} ] }' | jq -r '.message.content'

Step 5: Deploy Ollama to vCluster demo2

Repeat the steps to deploy Ollama and pull the model to the second virtual cluster:

code_block: <ListValue: [StructValue([('code', '# Connect to the virtual cluster\r\nvcluster connect demo2 -n demo2\r\n\r\n# Apply your deployment manifest\r\nkubectl apply -f ollama.yaml\r\n\r\n# Execute the pull command inside the pod\r\nkubectl exec -it <pod-name> -- ollama pull mistral\r\n\r\n# Port forward the Ollama service\r\nkubectl port-forward svc/ollama 8080:11434\r\n\r\n# Send a chat request in a new window\r\ncurl -s http://localhost:8080/api/chat \\\r\n -H "Content-Type: application/json" \\\r\n -d \'{ "model": "mistral", "stream": false, "messages": [ {"role": "user", "content": "Explain GKE Autopilot"} ] }\' | jq -r \'.message.content\''), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3601e8a250>)])]>

Verify the Underlying Infrastructure

Now let's switch back to the host cluster context and see what's going on.

Check how many nodes have been provisioned and where are the Ollama pods running:

# List the available contexts
kubectx
# Switch to the host cluster context
kubectx gke_$PROJECT_ID_$REGION_vcluster-gpu-sharing
# List nodes
Kubectl nodes

You should see two nodes. One is running the vCluster components. The other runs the Ollama instances with L4 GPUs. Your output should look like this (node names will be different):

# Output of kubectl get nodes
$ kubectl get nodes
NAME                                                  STATUS   ROLES    AGE    VERSION
gk3-vcluster-gpu-sharing-nap-1w88cyly-895203e4-xbqk   Ready    <none>   7h8m   v1.33.5-gke.2072000
gk3-vcluster-gpu-sharing-pool-2-0a984fed-7mff         Ready    <none>   4d     v1.33.5-gke.2072000

Check where the Ollama pods are running:
```
# Check the Nodes running the Ollama pods
kubectl get pods -n demo1 -o wide
kubectl get pods -n demo2 -o wide
```
Notice that both Ollama pods are running on the same node. This node has been provisioned by GKE Autopilot with L4 GPUs and GPU Sharing configured.

Conclusion

By using GKE Autopilot, we've removed the need to manually configure GPU node pools or time-sharing strategies. Autopilot provides resources dynamically, while vCluster ensures that Team A's Legal Research data and Team B's Customer Support bots remain completely isolated. This implementation provides a robust, low-maintenance platform for scaling AI workloads.

Data Strategy = AI Strategy Series: Transforming Developers into AI Architects with Google Cloud

Tue, 03 Mar 2026 19:00:00 +0000

Your agent is only as good as your data grounding. If your data is messy, your agent will be highly confident but possibly still hallucinating. In 2026, your Data Strategy and your AI Strategy are now the same thing. You cannot have one without the other.

This series, "Data Strategy = AI Strategy," focuses on the various aspects of this strategy and it can help you architect workflows that are more deterministic while still building autonomous agents. This post is just the first episode in which we focus on data and AI architecture convergence.

The industry is reaching a critical inflection point. Although 2024 and 2025 were defined by the "API era"—where developers learned to integrate LLMs through apps & endpoints—2026 demands a shift towards enterprise architecture. To build production-ready applications, developers must transition from writing prompts to designing intelligent end-to-end stacks.

The challenge isn't just about the AI model that you use; it's about the infrastructure around it. For an application to be enterprise-grade, it must meet the requirements of three critical pillars: speed, scale, and security. We focus on these pillars in this article: moving away from building agents that are focused on AI adoption to architecting agents that are grounded in well-strategized context. More specifically, this blog and the linked codelabs provide a hands-on learning path that shows you how to build an architecture by using Google's data cloud. This approach uses relational databases that are easily 100% PostgreSQL compatible.

The Strategic Pivot: The Database as the Context Engine

In a modern AI stack, the database is no longer just a storage layer; it has become the context engine. Our strategy centers on using fully PostgreSQL-compatible services like AlloyDB for PostgreSQL and Cloud SQL to eliminate the primary bottlenecks of AI in production: latency, AI capabilities, and retrieval accuracy.

To enable this transition from developer to architect, the learning path focuses on eliminating infrastructure friction and prioritizing high-level architectural design.

1. Eliminating the Infrastructure Tax

Historically, the transition from local prototyping to cloud-scale deployment was hindered by the infrastructure tax—the hours spent on configuring clusters, instances, and VPC network peering. By introducing automated setup utilities, we let developers bypass these configuration hurdles.

The result is a shift in focus: instead of managing infrastructure, developers can spend their time on designing secure data flows and high-throughput vector pipelines. In our recent instructor-led Code Vipassana sessions, each participating developer saved over an hour of time in each lab because of this shift. This approach effectively accelerates the path to production.

2. Building for Scale: One Million Vectors, Zero Loops

To build enterprise architecture, you need to move beyond small-scale demos. We focus on batch processing for embeddings to expedite vector search processes. AlloyDB can generate embeddings at scale directly within the database layer. By using this capability, we can eliminate the latency of traditional loops, which allows us to do real-time analytics on massive datasets.

3. Sovereign Intelligence and Row-Level Security (RLS)

Security in AI is more than only a firewall; it's about data governance. We emphasize the use of row-level security (RLS) to help ensure that AI agents can access only the specific data they're authorized to see. This private vault architecture is essential for regulated industries where data isolation is a non-negotiable requirement. Imagine your user talking to your agent and learning about another user or a benchmark. Baking the data level security into the database is not an option anymore. We cannot rely on agents to make the call on who should be informed of what.

The Architectural Learning Path

We have curated a series of hands-on technical labs that form a complete narrative of enterprise AI development. Each lab represents a specific layer of the intelligent stack

Most of the labs use AlloyDB. However, the momentum of this architectural strategy has also extended to the Cloud SQL ecosystem. Our learning path includes a couple of alternative labs for Cloud SQL for PostgreSQL users.

Our recommended learning path includes the following core architectural labs:

AlloyDB Quick Setup Lab

This lab serves as the entry point for architects, by demonstrating how to provision a high-performance AlloyDB cluster with the required VPC and network settings in minutes. It focuses on the day 0 operations that help to ensure a secure and scalable foundation for all subsequent AI logic.
Connect your app to AlloyDB data and deploy on Cloud Run (or Connect to Cloud SQL and deploy on Cloud Run)

Moving into deployment, this lab explores the architecture of serverless applications. Developers learn how to connect Cloud Run services to AlloyDB (or Cloud SQL), with a focus on using managed identities and connection strings to improve security. This approach helps ensure that the application layer is as robust as the data layer.
Build a Real-Time Surplus Engine: Gemini 3 Flash & AlloyDB (or Gemini 3 Flash & Cloud SQL)

This lab addresses the pillar for speed by building an end-to-end, data-driven AI app. It demonstrates how to use the high-efficiency Gemini 3 Flash model to process streaming data and generate real-time insights. This approach creates a responsive feedback loop between the database and the end user.
One Million Vectors, Zero Loops: Scale with AlloyDB

Focused on scale, this lab dives deeply into the vector search process. Architects learn how to implement batch processing for embeddings directly within the database. This approach bypasses application-layer bottlenecks, which enables the ingestion and search of millions of vectors with enterprise-grade performance.
The Private Vault: Zero Trust Intelligence with RLS

The final piece of the architectural puzzle focuses on improved security. This lab guides developers through building zero trust agents. By implementing RLS in PostgreSQL, developers can help ensure that their AI agents respect user-specific data boundaries. This approach provides a blueprint for compliant AI systems with enhanced security.

Designing the Future

By removing the friction from infrastructure and focusing on the core principles of speed, scale, and security, we can empower a new generation of AI architects. This strategic shift can help ensure that the applications built today are ready for the production demands of tomorrow.

To join our upcoming instructor-led, hands-on sessions and begin your transformation from developer to architect, sign up for Code Vipassana.

Announcing the MCP Toolbox Java SDK

Tue, 03 Mar 2026 09:29:00 +0000

Engineering teams are moving beyond simple chatbots to build agentic systems that interact directly with mission critical databases. However, building these enterprise agents often means hitting an integration wall of custom glue code, brittle APIs, and complex database logic.

To replace these hardcoded bottlenecks with a secure, unified control plane, we are thrilled to announce the Java SDK for the Model Context Protocol (MCP) Toolbox for Databases. This release brings first-class, typesafe agent orchestration to the world’s most widely adopted enterprise ecosystem. Java's mature architecture is purpose-built for these rigorous demands, providing the high concurrency, strict transactional integrity, and robust state management required to safely scale mission-critical AI agents in production.

MCP: The USB Type-C for AI Agents

Think of the Model Context Protocol (MCP) as a universal translator for AI.

Created to standardize how AI models connect to external tools and datasets, MCP replaces custom, fragmented integration scripts with a secure, universal protocol. Whether your agent needs to execute a transactional SQL query, search through thousands of policy documents, or trigger a REST API, MCP provides a single, unified interface.

With MCP Toolbox for databases, we’ve made implementing this protocol effortless.

MCP Toolbox for Databases

MCP Toolbox for Databases is an open source MCP server for databases. It natively supports 42 different data sources spanning AlloyDB, Cloud SQL, Cloud Spanner, and many more including third party data sources as well. Crucially, it gives you the ability to define custom tools that safely map an AI agent's natural language intents directly to specific database operations. It enables you to develop tools easier, faster, and more securely by handling the complexities such as connection pooling, authentication, and more.

We already provide robust, production-ready SDKs for Python, JavaScript, TypeScript, and Go. But when it comes to "Day 2" production workloads—where high concurrency, transactional integrity, and conversational state management are non-negotiable—Java and Spring Boot remain the undisputed heavyweights.

With the new Java SDK, you can natively build stateful, highly concurrent multi-agent systems without ever leaving your preferred tech stack. This SDK brings first-class, type-safe orchestration to Java, which is potentially a major priority for enterprise architects.

Get Started with the Java SDK

We’ve designed the MCP Toolbox Java SDK to be frictionless for enterprise teams. You can start building your agents today.

Add the Dependency

To bring the MCP Toolbox into your Java or Spring Boot project, simply add the following dependency to your pom.xml:

code_block: <ListValue: [StructValue([('code', '<dependency>\r\n <groupId>com.google.cloud.mcp</groupId>\r\n <artifactId>mcp-toolbox-sdk-java</artifactId>\r\n <version>0.2.0</version>\r\n</dependency>'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3601ee0310>)])]>

That’s it!!! You’d be good to go, just like we did in this enterprise grade use case sample below!!

Real World Example: The Autonomous Transit Concierge

To demonstrate the power of the Java SDK for MCP Toolbox combined with AlloyDB, let’s look at a use case from the transportation sector.

Meet Cymbal Transit, a fictitious intercity bus network. Customers don't want to click through 15 dropdown menus to plan a trip. They want to ask:

"I need to get from New York to Boston tomorrow morning. Can I bring my Golden Retriever? If so, book me the fastest trip."

To answer this, an AI agent must seamlessly cross-reference unstructured data (pet policies) with structured data (schedules, seat availability) and execute a transaction (booking)—all while remembering the context of the conversation.

Here is how we build this using the Java SDK for MCP Toolbox.

The Foundation: Database! (AlloyDB Schema with Native Embeddings)

AlloyDB is the perfect engine for this because it handles relational data and high-dimensional AI vectors in a single query engine. Even better, AlloyDB can generate embeddings natively using the google_ml_integration extension, meaning your Java app doesn't have to shuffle text back and forth to an embedding API.

First, set up AlloyDB cluster and instance by following this quick one-click deploy lab.

Then, set up database objects using the SQL statements below:

code_block: <ListValue: [StructValue([('code', "-- Enable necessary extensions for AI semantic search and embedding generation\r\nCREATE EXTENSION IF NOT EXISTS vector;\r\nCREATE EXTENSION IF NOT EXISTS google_ml_integration;\r\n\r\n-- Table 1: Transit Policies (Unstructured Data for RAG)\r\nCREATE TABLE transit_policies (\r\n policy_id SERIAL PRIMARY KEY,\r\n category VARCHAR(50),\r\n policy_text TEXT,\r\n policy_embedding vector(768) \r\n);\r\n\r\n-- Table 2: Intercity Bus Schedules (Structured Data)\r\nCREATE TABLE bus_schedules (\r\n trip_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),\r\n origin_city VARCHAR(100),\r\n destination_city VARCHAR(100),\r\n departure_time TIMESTAMP,\r\n arrival_time TIMESTAMP,\r\n available_seats INT DEFAULT 50,\r\n ticket_price DECIMAL(6,2)\r\n);\r\n\r\n-- Table 3: Booking Ledger (Transactional Action Data)\r\nCREATE TABLE bookings (\r\n booking_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),\r\n trip_id UUID REFERENCES bus_schedules(trip_id),\r\n passenger_id VARCHAR(100),\r\n status VARCHAR(20) DEFAULT 'CONFIRMED',\r\n booking_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP\r\n);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3601ee0970>)])]>

With our tables defined and vector support enabled, AlloyDB is now ready to serve as the unified brain for both our structured transactional data and semantic knowledge base.

Ingesting Records and Generating Real Embeddings

We need a robust dataset to ensure our agent's context window has real options to reason over. Using PostgreSQL's powerful generate_series in AlloyDB, we can instantly seed our database with over 200 realistic bus trips for tomorrow. We have taken this approach to ingesting mock data for this demo application.

code_block: <ListValue: [StructValue([('code', '-- 1. Insert Unstructured Policies and GENERATE REAL EMBEDDINGS natively in AlloyDB\r\nINSERT INTO transit_policies (category, policy_text, policy_embedding) \r\n... (refer repo for full statement)\r\n\r\n-- 2. Generate 200+ Realistic Schedules for the Next 7 Days using generate_series\r\nINSERT INTO bus_schedules (origin_city, destination_city, departure_time, arrival_time, ticket_price, available_seats)\r\n... (refer repo for full statement)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3601ee0a00>)])]>

Refer to the repo for the full code.

Just like that, our database is dynamically populated with realistic schedules and natively generated embeddings, giving our AI agent immediate access to a rich, queryable environment.

Stateful Agent Architecture in Spring Boot

The hardest part of building a conversational UI is session management. If a user asks, "What times are available?" and then says, "Book the 8 AM one," the agent needs to remember the context.

Using the Java MCP Toolbox SDK with Spring Boot and LangChain4j, we can seamlessly maintain conversational memory in the HTTP Session and inject it into the agent's thought process. By pairing a modern frontend with this stateful backend, your enterprise application becomes a continuous, intelligent workspace.

Instead of writing massive if/else blocks to parse user intent, we simply define a declarative AI interface and bind our MCP tools to it. How simple it is to bring orchestration logic to the code without having to hard-code detailed queries or add blocks of static conditional code:

code_block: <ListValue: [StructValue([('code', 'interface TransitAgent {\r\n @SystemMessage({\r\n "You are the Cymbal Transit Concierge.",\r\n "Use the \'querySchedules\' tool for finding schedules.",\r\n "Use \'bookTicket\' to execute transactions.",\r\n "Use \'searchPolicies\' to look up luggage and pet rules."\r\n })\r\n String chat(@MemoryId String sessionId, @UserMessage String userMessage);\r\n}\r\n\r\n@Service\r\nclass TransitAgentTools {\r\n // These methods automatically call our MCP Toolbox for Databases server!\r\n @Tool("Query specific schedules between an origin and destination city.")\r\n public String querySchedules(String origin, String destination) { ... }\r\n\r\n @Tool("Book a ticket for a passenger.")\r\n public String bookTicket(String tripId, String passengerName) { ... }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3601ee09a0>)])]>

By cleanly separating the LLM prompt logic from the actual tool execution, LangChain4j ensures our agent remains focused, predictable, and remarkably easy to maintain over time. By pairing a modern frontend with this stateful Spring Boot backend, your enterprise application becomes a continuous, intelligent workspace rather than a series of disconnected prompts.

Mapping Intents to SQL: The tools.yaml

The true magic of MCP Toolbox for Databases is how you define these custom tools. Your Java application doesn't need direct SQL access, nor does the LLM need to hallucinate table schemas. Instead, you provide the MCP server with a clean tools.yaml configuration. This file securely maps the agent’s tool calls directly to parameterized SQL statements in AlloyDB.

code_block: <ListValue: [StructValue([('code', 'tools:\r\n query-schedules:\r\n kind: postgres-sql\r\n source: alloydb\r\n description: Find available bus schedules between cities.\r\n parameters:\r\n - name: origin\r\n type: string\r\n - name: destination\r\n type: string\r\n statement: |\r\n SELECT CAST(trip_id AS TEXT) AS trip_id, departure_time, ticket_price \r\n FROM bus_schedules \r\n WHERE lower(origin_city) = lower($1) AND lower(destination_city) = lower($2)\r\n\r\n search-policies:\r\n ... refer to the repo for the full tools.yaml'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3601ee0eb0>)])]>

With this simple declarative configuration, you've bridged the gap between natural language intents and complex SQL queries—without ever exposing your database schema directly to the LLM.

Connecting the Dots: Listing, Invoking, and Executing Tools in Java

Now that our database and tools are configured, the MCP Toolbox Java SDK handles the heavy lifting of interacting with them. The SDK provides an intuitive, type-safe API to securely discover, query, and execute transactions from your Spring Boot service.

code_block: <ListValue: [StructValue([('code', '// 1. Initialize the Client\r\nMcpToolboxClient mcpClient = McpToolboxClient.builder()\r\n .baseUrl("https://toolbox-my-project-uc.a.run.app")\r\n .apiKey(myIdToken) \r\n .build();\r\n\r\n// 2. Listing Discoverable Tools\r\nmcpClient.listTools().thenAccept(tools -> {\r\n System.out.println("Successfully discovered " + tools.size() + " tools.");\r\n});\r\n\r\n// 3. Invoking a Tool (Read-Only Data)\r\nString schedules = mcpClient.invokeTool("query-schedules", Map.of(\r\n "origin", "New York",\r\n "destination", "Boston"\r\n)).join().content().get(0).text();\r\n\r\n// 4. Executing a Transactional Tool (Requires Bound Authentication)\r\nAuthTokenGetter toolAuthGetter = () -> CompletableFuture.completedFuture(myIdToken);\r\n\r\nString bookingConfirmation = mcpClient.loadTool("book-ticket", Map.of("google_auth", toolAuthGetter))\r\n .thenCompose(tool -> {\r\n // Bind the authenticated user context securely\r\n tool.bindParam("passenger_name", "Jane Doe");\r\n // Execute the mutable transaction\r\n return tool.execute(Map.of("trip_id", "123e4567-e89b-12d3-a456-426614174000"));\r\n }).join().content().get(0).text()'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3601ee0ac0>)])]>

And there you have it—in just a few lines of type-safe Java, your Spring Boot application is securely discovering and executing remote tools as if they were local methods. Notice the bindParam method above? This powerful feature allows you to securely inject application-level context (like the authenticated user's identity) directly into the database transaction, bypassing the LLM entirely. You can learn more about this in the MCP Toolbox Java SDK documentation.

Zero-Config Security with Application Default Credentials (ADC)

Notice there are no hardcoded secrets or JSON keys to manage! By fetching myIdToken using Google's Application Default Credentials (ADC) under the hood, your Java app automatically inherits its secure identity directly from the environment. Whether you are developing locally via the gcloud CLI or running in production, your application stays secure by default with zero manual credential configuration.

Deploying the Fleet to Cloud Run

Remember those "Day 2" enterprise requirements we mentioned earlier? To successfully handle high concurrency, transactional integrity, and maintain stateful conversations at scale, your architecture needs to be robust. Because the MCP Toolbox for Databases and our Spring Boot Agent are fully decoupled, they can scale independently on Google Cloud Run to meet those exact demands.

First, you deploy the open-source MCP Toolbox for Databases as its own secure service:

code_block: <ListValue: [StructValue([('code', '# Download the toolbox CLI\r\nexport VERSION=0.27.0\r\ncurl -L -o toolbox https://storage.googleapis.com/genai-toolbox/v$VERSION/linux/amd64/toolbox\r\nchmod +x toolbox'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3601ee0190>)])]>

Next, deploy the local toolbox server to Cloud Run:Follow instructions here in the official documentation.

Once the Toolbox is running, it acts as the secure, highly scalable bridge between your database and the outside world.

Next, deploy your Java Spring Boot Agent, injecting the dynamically generated MCP Toolbox URL and your Vertex AI settings as environment variables:

Set up the Cymbal Transit Agent App:

Clone the repo.
Then, you deploy your Java Spring Boot Agent, injecting the dynamically generated MCP Toolbox URL and your Vertex AI settings as environment variables:

code_block: <ListValue: [StructValue([('code', 'gcloud run deploy cymbal-transit \\\r\n --source . \\\r\n --allow-unauthenticated \\\r\n --set-env-vars GCP_PROJECT_ID=my-project,GCP_REGION=us-central1,GEMINI_MODEL_NAME=gemini-2.5-flash,MCP_TOOLBOX_URL=https://toolbox-...'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3601ee0cd0>)])]>

With a single command, your stateful, multi-agent enterprise application is live on Cloud Run, ready to securely orchestrate workflows on behalf of your users!

The Era of Hardcoded Integrations is Over

The transition from stateless chatbots to autonomous, transactional agents is the defining technological shift of this decade. But agents are only as powerful as the systems they can securely interact with.

With the MCP Toolbox for Databases Java SDK, enterprise developers finally have a native, elegant, and highly scalable way to give their AI agents read-and-write access to the mission-critical systems of record that run their business.

Ready to build your own stateful enterprise agents? Explore the official MCP Toolbox Java SDK GitHub Repository to get started, and check out the demo application Cymbal Bus Agent GitHub Repository to explore the complete source code and try it out today!

Designing private network connectivity for RAG-capable gen AI apps

Mon, 02 Mar 2026 17:00:00 +0000

The flexibility of Google Cloud allows enterprises to build secure and reliable architecture for their AI workloads. In this blog we will look at a reference architecture for private connectivity for retrieval-augmented generation (RAG)-capable generative AI applications. This architecture is for scenarios where communications of the overall system must use private IP addresses and must not traverse the internet.

The power of RAG

RAG is a powerful technique used to optimize the output of large language models (LLMs) by grounding them in specific, authoritative knowledge bases outside of their original training data. RAG allows an application to retrieve relevant information from your documents, datasources, or databases in real time. This retrieved context is then provided to the model alongside the user’s query, helping to ensure that the AI’s responses are accurate, verifiable, and highly relevant to your business. This improves the quality of responses and reduces hallucinations.

This approach is helpful because it allows you to direct generative AI to use a designated source of truth, rather than relying solely on the model's pre-existing knowledge, and without needing to retrain or fine-tune the model itself.

Design pattern example

To understand how to think about setting up your network for private connectivity for a RAG application in a regional design, let's look at the design pattern.

The setup comprises an external network (on-prem and other clouds) and Google Cloud environments consisting of a routing project, a Shared VPC host project for RAG, and three specialized service projects: data ingestion, serving, and frontend.

This design utilizes the following services to provide an end-to-end solution:

Cloud Interconnect or Cloud VPN: To securely connect from your on-premises or other clouds to the routing VPC network
Network Connectivity Center: Used as an orchestration framework to manage connectivity between the routing VPC network and the RAG VPC network via VPC spokes and hybrid spokes
Cloud Router: In the routing project, facilitates dynamic BGP route exchange between the external network and Google Cloud
Private Service Connect: Provides a private endpoint in the routing VPC network to reach the Cloud Storage bucket for data ingestion without traversing the public internet
Shared VPC: Host project architecture that allows multiple service projects to use a common, centralized VPC network
Google Cloud Armor and Application Load Balancer: Placed in the frontend service project to provide security and traffic management for user interaction
VPC Service Controls: Creates a managed security perimeter around all resources to mitigate data exfiltration risks

The traffic flow

RAG population flow

In the diagram, the green dashed line shows the RAG population flow, which describes how data travels from data engineers to vector storage.

From the external network, data travels over Cloud Interconnect or Cloud VPN.
In the routing projects it uses the Private Service Connect endpoint to get to the Cloud Storage bucket.
From the Cloud Storage bucket in the Data Ingestion service project, the data ingestion subsystem processes the raw data.
The AI model creates vectors from the chunks, returns them to the data ingestion subsystem, which writes them to the RAG datastore in the serving service project.

Inference flow

In the diagram, the orange dashed line shows the inference flow, which describes customer or user requests.

The request travels over Cloud Interconnect or Cloud VPN to the routing VPC network and then over the VPC spoke to the RAG VPC network.
The request reaches the Application Load Balancer protected by Cloud Armor; once allowed, it passes it to the frontend subsystem.
The frontend subsystem forwards the request to the serving subsystem, which augments the prompt with data from the RAG datastore and generates a response via the AI model.
The system generates a response via the AI model, and the grounded response is returned along the same path to the requestor.

Management and routing

In the diagram, the blue dotted lines represent the Network Connectivity Center hybrid and VPC spokes that manage the control plane and route orchestration between the routing network and the RAG VPC network. This ensures that routes learned from the external network are appropriately propagated across the environment.

Please read the entire architecture document Private connectivity for RAG-capable generative AI applications to understand the specific including IAM permissions, VPC Service Controls, and deployment considerations.

Next steps

Take a deeper dive into the Cross-Cloud Network, and other guides about generative AI with RAG:

Document set: Generative AI with RAG
Document: Cross-Cloud Network for distributed applications
Blog: Build Your First ADK Agent Workforce

Want to ask a question, find out more or share a thought? Please connect with me on Linkedin.

From "Vibe Checks" to Continuous Evaluation: Engineering Reliable AI Agents

Fri, 27 Feb 2026 17:00:00 +0000

I live through the same story with every single AI agent. After weeks of experiments and tests, it works like a charm. Suddenly, someone comes with a question that the agent fails to answer properly. I rush to make a change by tweaking one of the prompts. After a handful of tweaks, the failed prompt produces good results. I try a few of my favorite prompts and it works like a charm. Another new question, another perfect hit. I push it to production.

Less than 24 hours later, user reports start trickling in. The agent is hallucinating dates. It fails to cite sources for obscure topics. A little change that felt so solid ended up sabotaging dozens of other use cases that I haven't bothered to verify.

This is the vibe check trap.

The Vibe Check Trap

In the classical software world, if you change a line of code, you run unit tests. The predicate assert 2 + 2 == 4 will never statistically drift. Integration tests are more complex and flaky, but they're still largely stable in well-maintained projects. But in the world of Generative AI, we're building software on top of probabilistic foundations. A prompt that works 99% of the time today might work 92% of the time tomorrow just because the underlying model's weight distribution shifted slightly, or because the temperature parameter introduced a new token sequence. A minor change in the prompt or grounding data format might trigger a significant regression in the model's answers.

Relying on vibe checks—manually chatting with the agent to see if it feels right—is a recipe for disaster in production. It's subjective, unscalable, and susceptible to confirmation bias.

This guide is for software engineers who are ready to graduate from building demos to building production-grade AI systems.

In this post, we'll explore how to apply the engineering discipline of continuous evaluation (CE) for AI agents. With CE, you refine your agent's prompts, tools, and logic by using a combination of production monitoring, automated LLM-as-a-judge scoring, and human feedback. We'll show you how to apply CE using specific tools from the Google Cloud ecosystem: Agent Development Kit (ADK), Vertex AI Gen AI evaluation service, and Cloud Run.

1. The Engineering Mindset: Discovery vs. Defense

To organize our work effectively, we must distinguish between two fundamental modes of AI engineering. In traditional DevOps, these map roughly to Development and QA/Ops, but the distinction is sharper here due to the stochastic nature of large language models (LLMs). In AI engineering, these modes translate to discovery mode and defense mode.

Discovery Mode (The Lab)

This is the creative phase. You're an explorer.

Activities: Prompt engineering, tool selection, model selection.
Goal: Raise the ceiling. You want to see if the model is capable of solving a complex reasoning task at least once.
Methodology:

Few-shot iteration: Providing examples in the prompt to guide the model's behavior.
Red teaming: Actively trying to break the model with adversarial inputs to find edge cases.
Vibe checks: Yes, here they're useful! They help you build intuition about the model's personality and latency.

Outcome: A golden prompt that works perfectly for your specific reference examples.

Defense Mode (The Factory)

This is the industrialization phase. You're a reliability engineer.

Activities: Regression testing, shadow traffic, monitoring.
Goal: Protect the floor. You want to ensure that the average performance across 10,000 requests meets your Service Level Objectives (SLOs).
Methodology:

Dataset-driven evaluation: Running the prompt against hundreds of diverse examples, not just the three you memorize.
Strict gating: Automatically failing a build if the grounding score drops below 0.85.
Automated metrics: No humans involved in the loop.

Outcome: A deployed system that you can sleep through the night with.

Comparison Table

Feature	Discovery mode	Defense mode
Primary goal	Innovation (new capabilities)	Stability (reliability)
Sample size	1 to 10 inputs	50 to 10,000 inputs
Evaluation method	Human eye (vibe check)	Automated evaluators (LLMs/code)
Latency tolerance	High (waiting for reasoning)	Low (SLO enforcement)
Cost sensitivity	Low (development environments)	High (production scale)

The failure mode: Most teams stay in discovery mode forever. They treat every bug report as a reason to tweak the prompt, push it live, and pray. This creates a game of whack-a-mole where fixing one hallucination causes two more. To exit this trap, we need regression testing, which we'll show you how to implement next.

2. The Reference System: Architecture of a Course Creator

To demonstrate the defense mode principles concretely, we'll analyze a Course Creator System. This isn't a single all-in-one agent, or a monolithic prompt trying to do everything: it's a distributed multi-agent system that's composed of multiple specialized agents. This architecture follows the principle of separation of concerns.

aside_block: <ListValue: [StructValue([('title', 'Check out the Agent Evaluation Lab repository'), ('body', <wagtail.rich_text.RichText object at 0x7f36026429a0>), ('btn_text', ''), ('href', 'https://github.com/vladkol/agent-evaluation-lab'), ('image', None)])]>

The system is built on Cloud Run for serverless scalability and it uses the Agent2Agent (A2A) Protocol for standardized inter-agent communication.

The Agent Roster

Each agent does its specific piece of work. The researcher collects information, the judge evaluates the collected data, the content builder composes it into a well-structured course, and the orchestrator controls this mighty team!

1. The Researcher (The Hunter)

Role: Information retrieval.
Tools: Custom wikipedia_search.
Personality: Objective, fact-focused.
Input: A query string (e.g., "history of neural networks").
Output: Text of the most relevant Wikipedia page.

Wikipedia Search Tool:

code_block: <ListValue: [StructValue([('code', '# agents/researcher/agent.py\r\ndef wikipedia_search(query: str) -> str:\r\n """Searches Wikipedia for a given query."""\r\n pages = search(query, results=1)\r\n if pages:\r\n return page(pages[0], auto_suggest=False).content\r\n return ""'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f36026426a0>)])]>

2. The Judge (The Critic)

Role: Quality assurance.
Tools: None.
Personality: Strict, pedantic.
Mechanism: It uses structured output (Pydantic objects) to return a formal verification result.

Why is the judge a separate agent? An agent detecting its own hallucinations is notoriously unreliable. A separate judge agent provides a necessary adversarial check.

3. The Content Builder (The Writer)

Role: Synthesis and formatting.
Tools: None.
Personality: Creative, educational.
Responsibility: It takes the raw, verified facts from the researcher and it structures them into a cohesive course module, such as "Introduction", "Chapter 1", etc.

4. The Orchestrator (The Manager)

Role: Workflow management.
Mechanism: It implements a SequentialAgent.
Logic:

Call the research loop:

Ask the researcher to gather data.
Ask the judge to evaluate data.
If judge says "Fail": Send feedback to the researcher (restart the loop).
If the judge says "Pass": Break the loop and continue to the next step.

Call the content builder to build the comprehensive course content.

Why is the orchestrator a separate agent? Using a separate orchestrator agent isolates the control flow logic from the generation logic.

The Course-Building Multi-Agent System

The architecture of the multi-agent system is set up like this:

This multi-agent system has a nice web app (in the app folder of the repository). A little service exposes a frontend and calls the orchestrator agent service by using our ADK FastAPI integration. The user request flow looks like this:

The A2A Protocol Benefits

The Agent2Agent (A2A) Protocol standardizes how these agents communicate with each other. Instead of wrapping the researcher as a function call or a generic tool within the orchestrator's prompt (which limits its capabilities), A2A lets the orchestrator interact with the researcher as a full peer service.

This protocol solves the "N × N" integration problem. All agents speak the same language (HTTP + JSON schemas), making the system modular and easy to extend. If we want to replace the researcher with a different implementation, the orchestrator doesn't need to change a single line of code.

This modularity is also the key to our evaluation strategy. Because the agents are loosely coupled services, we don't have to evaluate the entire system at once. Instead, we can target individual components.

Shared Architecture Components

To make this distributed system reliable and observable, we use a set of shared utility components across all of our agents:

shared/adk_app.py: This is the backbone of every agent service. It builds on top of ADK and FastAPI. It automatically configures these components:

A2A middleware: Handles the exchange of agent cards or self-description, dynamically rewriting URLs to match the current deployment. A2A middleware is useful for shadow revisions, where you deploy a new version of your agent to handle simulated traffic from an evaluation pipeline.
OpenTelemetry middleware: Automatically captures every incoming request as a Trace Span.

shared/traced_authenticated_httpx.py: A hardened HTTP client for inter-agent communication.

Authentication: Handles the complexities of Google Cloud service-to-service authentication (OIDC tokens), ensuring zero-trust security between agents.
Trace propagation: Injects the traceparent header into every outgoing request. The header context lets Cloud Trace stitch together the graph that shows how the orchestrator called the researcher. We'll discuss that more later, when we take a look at distributed tracing.

shared/a2a_utils.py: Provides the logic for dynamic agent cards. In Cloud Run, a service might be accessed through a public URL or through a revision-specific URL. This utility ensures that the agent always tells its peers the correct address to call back.

3. The Evaluation Taxonomy: A Deep Dive

Before we write code, we must define our units of measurement. "Is this agent good?" isn't a valid engineering question. We need to define "good" as testable dimensions. To do that, we can categorize evaluation metrics into a hierarchy of sophistication.

Level 1: Computation-Based Metrics

These are deterministic checks against a ground truth or rigid rules. They're the closest to traditional software unit and integration tests.

Classic NLP metrics like ROUGE and BLEU: Is the result sufficiently similar to the reference answer?
JSON validity: If your agent must output JSON, does it parse?
Prohibited phrases: Does the output contain "I am an AI language model"?
Exact match or Regex: For extraction tasks (e.g., getting a date YYYY-MM-DD), does the output match the pattern?
Execution trajectory (including agent tool trajectory): Did the agent call certain tools in a particular order with specific parameters?

Reference-based or Reference-free metrics

Some metrics require a ground truth reference answer to compare the result to. ROUGE and BLEU are perfect examples of that. Other metrics might evaluate the result on different criteria, such as output format ("Did the agent produce correct JSON?") or prohibited words—they don't need a ground truth answer for comparison.

Level 2: Rubric-Based Metrics

This is the standard for semantic evaluation. We use a powerful LLM-as-a-judge model (like Gemini Pro) to grade the agent's output. To evaluate agent answers, these metrics use sophisticated battle-tested and rigorously maintained prompts.

These are the core concepts that are related to LLM-based evaluation metrics:

Rubrics: The criteria for how to rate the response of an LLM model or application. Basically, it's a composite prompt that can be pre-defined or dynamically generated.
Metrics: A score that measures the model output against the rating rubrics.

Rubric-based metrics incorporate LLMs into these kinds of evaluation workflows:

Adaptive rubrics: Rubrics are dynamically generated for each prompt. Responses are evaluated with granular, explainable pass or fail feedback that's specific to the prompt.
Static rubrics: Rubrics are defined explicitly and the same rubric applies to all prompts. Responses are evaluated with the same set of numerical scoring-based evaluators, with a single numerical score (such as 1 to 5) per prompt. Static rubrics are used when the exact same criteria is required across all prompts (e.g., "Check whether the agent answered the user's question").

Just like with computation-based metrics, rubric-based metrics might or might not require a ground truth reference. It's critical to choose the right one for your task.

A. Reference-Free Metrics

You don't have a specific correct answer, but you rely on general principles of quality. Use this approach for open-ended generation like emails, poems, or generic advice. Examples of reference-free metrics include these:

Response quality: A comprehensive and adaptive rubrics metric that evaluates the overall quality of an agent's response as follows:

It automatically generates a broad range of criteria based on the agent configuration (developer instruction and declarations for tools that are available to the agent) and the user's prompt.
Then it assesses the generated criteria based on tool usage in intermediate events and the final answer by the agent.

Coherence: Evaluates whether the text is logical and grammatically correct.
Safety: Evaluates whether the text violates safety policies, such as by inclusion of hate speech or personally identifiable information (PII).
Instruction-following: A targeted and adaptive rubrics metric that measures how well the response adheres to the specific constraints and instructions that are given in the prompt.

B. Reference-Based Metrics

You have a golden answer and you want to ensure that the agent's response creates the same meaning, even if it's phrased differently. If the reference is "Paris" and the agent says "Capital of France, which is Paris", a regex might fail, but an LLM judge will pass it. A reference-based metric checks for a response match to determine whether the answer matches the reference response or ground truth.

Level 3: Vertex AI Managed Metrics

Google's Vertex AI Gen AI evaluation service provides pre-built, calibrated models called autoraters for these dimensions. Autoraters are superior to creating your own judge prompt because they're benchmarked against human raters and they're maintained by Google. These are a few examples of autoraters:

GROUNDING: The most critical metric for RAG. It takes context + response and checks whether the response is fully supported by the context. It assigns a score from 0 to 1.
SAFETY: Automatically flags hate speech, harassment, and dangerous content.
TOOL_USE_QUALITY: Specifically tailored for agents to evaluate whether the agent made an appropriate tool call and whether the argument was correct. This metric doesn't require comparison to a tool call reference that's considered correct. Instead, the evaluator makes a judgment based on the tool description, the agent description, and the context of the conversation.

For more information, see the complete list of managed rubric-based metrics in Vertex AI.

Static and Adaptive Rubrics

Static rubrics like "Rate helpfulness 1-5" suffer from high variance. Adaptive rubrics solve that issue by dynamically generating a test case for each prompt.

Rubric generation: The system analyzes the user prompt and reference.

Prompt: "Compare the battery life of Pixel 9 and iPhone 16."
System generates criteria:

Criteria 1: Mentions Pixel 9 mAh?
Criteria 2: Mentions iPhone 16 video playback hours?
Criteria 3: Is neutral?

Rubric grading: The judge checks these boolean conditions.

This turns a subjective vibe check into an objective report card that explains exactly what was missing.

Gen AI evaluation service provides a comprehensive set of metrics that are based on static and adaptive rubrics. You can also create your own adaptive rubrics by using the GenAI Client in Vertex AI SDK.

4. The Fuel: Designing Your Evaluation Dataset

Garbage in, garbage out. Your evaluation is only as good as your dataset. A proper evaluation dataset is a collection of examples (rows). In our system, we use a JSON-based format where columns represent different inputs and expected outputs.

The following is an actual example from our evaluator/eval_data_researcher.json dataset. It's structured as columns for efficient pandas loading:

code_block: <ListValue: [StructValue([('code', '{\r\n "prompt": {\r\n "0": "History of Rome",\r\n "1": "Pythagorean theorem"\r\n },\r\n "reference": {\r\n "0": "# The History of Rome\\n\\nThe history of Rome spans over 2,500 years...",\r\n "1": "## The Pythagorean Theorem: A Cornerstone of Geometry..."\r\n },\r\n "reference_trajectory": {\r\n "0": [\r\n {\r\n "tool_name": "wikipedia_search",\r\n "tool_input": { "query": "History of Rome" }\r\n }\r\n ],\r\n "1": [\r\n {\r\n "tool_name": "wikipedia_search",\r\n "tool_input": { "query": "Pythagorean theorem" }\r\n }\r\n ]\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3602642a90>)])]>

Components of the Dataset

prompt: The input to the agent.

Example: "What is the return policy for item #123?"

reference (optional): The ideal answer (for reference-based metrics).

Example: "Item #123 can be returned within 30 days."

reference_trajectory (optional): The simple gold standard for tool usage.

This option lets us verify whether our agent is thinking correctly. If the prompt asks for "Population of Tokyo" and the trajectory shows a call to get_weather("Tokyo"), the agent has failed fundamentally, even if it hallucinates the correct population number.

Best practice: Store this dataset in BigQuery or in a JSON file in Cloud Storage. Treat it like source code. Version it.

Just before we call the agent with these inputs, we'll add one more column with the same value for every input prompt: session_inputs, a simple structure with evaluation user ID, agent name, and an empty state dictionary.

5. The Implementation: Building the Evaluation Engine

When we have a dataset and metrics, we need an engine to drive the tests. A simple Python script isn't enough; we need to replicate the scale of production. To accomplish that, we build an evaluation runner (evaluate_agent.py) that uses the GenAI Client in Vertex AI SDK.

A. Parallel Inference

Agent operations are slow relative to typical API calls. A multi-step reasoning task might take 15 seconds. If our golden dataset has 500 examples, running them sequentially would take 2 hours. We use Python's asyncio to run many concurrent requests against a shadow revision.

This approach reveals another benefit of evaluating agents that are deployed to Cloud Run: unlike your developer machine, it can scale to serve parallel requests. Faster evaluation enables faster iterations.

In the shared/evaluation/evaluate.py module, we implement a throttled parallel runner:

code_block: <ListValue: [StructValue([('code', 'async def run_parallel_inference(client, prompts, shadow_url):\r\n tasks = []\r\n # Semaphore to prevent DDOSing our own service or hitting Rate Limits\r\n # We limit to 10 concurrent requests to match our Cloud Run capacity\r\n sem = asyncio.Semaphore(10)\r\n\r\n for prompt in prompts:\r\n # Each task runs the full HTTP Post -> SSE Stream -> Events Capture\r\n tasks.append(_run_inference(sem, client, shadow_url, prompt))\r\n\r\n return await asyncio.gather(*tasks)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f3602642d60>)])]>

_run_inference calls the ADK server API endpoint POST /run_sse of the target agent. This call initiates a streaming session where the agent pushes events while it thinks. The events are captured and processed to store the final answer and a list of the intermediate events—the reasoning trace.

B. Reasoning Trace Capture

If the agent fails, why did it fail? In standard evaluation, you only see the final answer. In agentic evaluation, we need the reasoning trace or execution history. It includes a list of events that occurred during the agent's execution. Each event has a type and a payload. The payload contains the event's content and metadata. The most interesting events are the tool calls. They include tool call requests from the LLM (with parameter values), and tool call responses from the tools (with return values).

We pass this entire trace to the Gen AI evaluation service. This allows for questions like: "Did the agent hallucinate the number 14 million, or did the tool actually return it?"

We also use this trace for tool trajectory evaluation, which we describe later in this post.

C. Final Evaluation Dataset

After we run the agent for every prompt, we add two more columns to the evaluation dataset. We already have prompt, reference (optional ground truth answer), reference_trajectory (optional ground truth for tool calls and their parameters), and session_inputs. After inference, we add these columns:

response: The actual final response of the agent.
intermediate_events: All events that preceded the final response, including tool calls.

D. Runtime Schema Integration

To verify whether the agent used tools correctly, the evaluation scorer can leverage the tool definition. If we hardcode this definition in our test suite, it will drift from the actual code. Instead, we expose an /agent-info endpoint on every agent, and the evaluator fetches it at runtime.

code_block: <ListValue: [StructValue([('code', '# Fetch live schema from the running service\r\nagent_info_response = await httpx_client.get(f"{agent_api_server}/apps/{agent_name}/agent-info")\r\nagent_info = types.evals.AgentInfo.model_validate_json(agent_info_response.content)\r\n\r\n# Create the Evaluation Run in Vertex AI\r\nevaluation_run = client.evals.create_evaluation_run(\r\n dataset=agent_dataset_with_inference,\r\n agent_info=agent_info, # Contains the LIVE tool definitions/schema\r\n metrics=metrics,\r\n dest=evaluation_storage_uri\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f3602642190>)])]>

This implementation ensures that if you add a new tool parameter in your code, the evaluation automatically knows about it without manual updates to the test suite.

6. Custom Function Metrics Deep Dive: Tool Trajectory Evaluation

For many agents, tool usage isn't an option, but is a mandatory part of their flow. Evaluating general tool usage quality isn't enough. You need to have strictly defined business rules.

Rule 1: "Wikipedia tool must always be called."
Rule 2: "It must be called with the correct search request."

We can enforce these by using custom metrics in Gen AI evaluation service. We write a Python function, and the service executes it in a secure sandbox against every row of our evaluation.

Implementing Tool Trajectory Metrics

In our shared/evaluation/tool_metrics.py module, we implemented multiple custom metrics for tool trajectory evaluation.

Trajectory precision: The agent called 5 tools. 3 were useful, 2 were noise. Precision = 3/5.
Trajectory recall: The task required checking Database and Wiki. The agent only checked Wiki. Recall = 0.5.
Exact order match: The agent called the right tools in the specified order, without calling any other tools.
In-order match: The agent called the right tools in the specified order, even if other tools were called in between.
Any-order match: The agent called the right tools in any order.

The core logic relies on comparing the predicted trajectory (what the agent did) against the reference trajectory (what we wanted it to do).

1. The Reference (From Dataset)

The reference_trajectory looks like a clean list of expected calls:

code_block: <ListValue: [StructValue([('code', '[\r\n {\r\n "tool_name": "wikipedia_search",\r\n "tool_input": { "query": "History of Rome" }\r\n }\r\n]'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3602642730>)])]>

2. The Event Trace (From Agent)

We use captured intermediate_events to extract the actual function calls so that we can compare them to the reference trajectory.

code_block: <ListValue: [StructValue([('code', '// One event in the stream\r\n{\r\n "content": {\r\n "parts": [{ "function_call": {\r\n "name": "wikipedia_search",\r\n "args": { "query": "History of Rome" }\r\n }}]\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f36026428e0>)])]>

The helper function _get_tool_calls extracts the list from the trace and compares it to the reference.

Executing Custom Function Metrics in Vertex AI

These custom metrics require running Python code. Where does the code run? Aren't we using Vertex AI for the evaluation?

Yes, Gen AI evaluation service takes care of running that code. The service expects Python code with an evaluate function:

code_block: <ListValue: [StructValue([('code', 'def evaluate(\r\n instance: dict\r\n) -> float:\r\n ...'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f3602642580>)])]>

Our functions have other functions that they depend on, so we package the module's source code with them and we construct an extra evaluate function to make a call.

code_block: <ListValue: [StructValue([('code', 'import inspect\r\nmodule_source = inspect.getsource(\r\n inspect.getmodule(metrics_function)\r\n)\r\nmodule_source += (\r\n "\\n\\ndef evaluate(instance: dict) -> float:\\n"\r\n f" return {metrics_function.__name__}(instance)\\n"\r\n)\r\nreturn types.EvaluationRunMetric(\r\n metric=metric_name,\r\n metric_config=types.UnifiedMetric(\r\n custom_code_execution_spec=types.CustomCodeExecutionSpec(\r\n remote_custom_function=module_source\r\n )\r\n )\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f36026420d0>)])]>

We package these functions using CustomCodeExecutionSpec and send them to Vertex AI for sandboxed execution. This approach lets us combine the flexibility of custom Python code with the massive scale of managed evaluation.

7. Strategy: Shadow Deployments & Safe Rollouts

The most common fear in deploying AI agents is: "If I change the prompt, will it break for 10% of users?" This fear can paralyze teams. To solve this issue, we borrow a technique from standard microservice engineering: shadow deployments or dark canaries.

The Concept

Instead of replacing the live version of your agent, you deploy a new version alongside it.

Live revision: Serves 100% of user traffic.
Shadow revision: Serves 0% of user traffic but handles simulated traffic from your evaluation pipeline.

This decoupling of deployment (code on server) from release (users see code) lets you test in the exact production environment—same network, same secrets, same latency characteristics—without risk.

Cloud Run Implementation

Cloud Run makes implementation trivial. Every deployment creates a revision. We can assign a tag to a revision to give it a unique URL. We use the Git commit short SHA hash as the tag (e.g., sha-a1b2c3d). This tag creates an immutable link between your source code and your running service.

In our deploy.sh, we use the following logic to deploy a shadow revision:

code_block: <ListValue: [StructValue([('code', '# 1. Capture the commit SHA\r\nexport COMMIT_SHA=$(git rev-parse --short HEAD)\r\nexport REVISION_TAG="sha-${COMMIT_SHA}"\r\n# 2. Deploy with --no-traffic\r\n# This tells Cloud Run: "Start the container, but don\'t route public requests here."\r\ngcloud run deploy researcher \\\r\n --image gcr.io/${GOOGLE_CLOUD_PROJECT}/researcher:latest \\\r\n --region us-central1 \\\r\n --no-traffic \\\r\n --tag "${REVISION_TAG}"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3602642a60>)])]>

Result:

Public URL: https://researcher-xyz.run.app (unchanged, safe).
Shadow URL: https://sha-a1b2c3d---researcher-xyz.run.app (new, testing ground). Three dashes ---separate the revision part.

When a user hits the service public URL, Cloud Run distributes traffic between revisions that are configured as serving the traffic. This new shadow revision doesn't serve any requests unless it's called with a revision-specific shadow URL. You can view and manage revisions in the Google Cloud console.

Your continuous evaluation pipeline then targets this shadow URL. If the shadow revision metrics pass, we can run a promotion command that makes the successful revision serve the traffic:

code_block: <ListValue: [StructValue([('code', '# Promotion command (only run after evaluation passes)\r\ngcloud run services update-traffic researcher --to-tags ${REVISION_TAG}=100'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3602642d90>)])]>

It doesn't have to always be 100% to a single revision. No matter how much we test and evaluate our code, mistakes happen. Instead of switching all at once, you might want to gradually migrate traffic between revisions.

8. Analyzing Evaluation Results

When the pipeline breaks, developers don't dig through text logs. Using the Run ID from the build log, they can pull the full report.

code_block: <ListValue: [StructValue([('code', 'from google.genai import types as genai_types\r\nfrom vertexai import Client\r\n# Initialize SDK\r\nclient = Client(\r\n project=GOOGLE_CLOUD_PROJECT,\r\n location=GOOGLE_CLOUD_REGION,\r\n http_options=genai_types.HttpOptions(api_version="v1beta1"),\r\n)\r\n\r\nevaluation_run = client.evals.get_evaluation_run(\r\n name=EVAL_RUN_ID,\r\n include_evaluation_items=True\r\n)\r\nevaluation_run.show()'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f36006bc730>)])]>

Acting on Evaluation Results

If the evaluation fails, the build fails. The build fails the pipeline. The pipeline fails the commit. The commit fails the PR. The PR fails the merge. The merge fails the release. The release fails the deployment. The users don't use the failed code.

Now, how can we understand why it failed? Let's take a closer look at an example of the evaluation run.

The request was History of Rome. The researcher provided the content of the Wikipedia page History of Rome. The content seemed good to the judge, but the final hallucination metric was too low.

The reason is because the final course that was built by the content builder contained facts that weren't present in the Wikipedia page. By looking at the reasoning trace, we can see that the researcher used the Wikipedia page as a source of information.

However, the content builder was too creative about the course content. The content builder used a Gemini model that certainly knows a lot about Rome, so it enhanced the course with facts that weren't present in the Wikipedia page.

How do we fix that? Let's tell the content builder to stick to the facts that are provided by the researcher.

And voilà! The very next run produced a perfect evaluation score.

What's important is that the rest of the metrics are still good. We found a problem and we made changes to fix it, but the whole system stayed intact.

9. Automating the Loop: The CI/CD Pipeline

Finally, we operationalize this solution by using Cloud Build. The goal is a quality firewall that helps makes sure that bad code can't physically reach production users. Our .cloudbuild/cloudbuild.yaml orchestrates the lifecycle:

Build:

Docker builds the researcher image.
Push to Artifact Registry.

Deploy shadow:

gcloud run deploy ... --tag=sha-${COMMIT_SHA} --no-traffic

Evaluate:

Run python -m evaluator.evaluate_agent.
The script targets the shadow URL.
It uploads results to Vertex AI.
The gate: It checks if metric_score < THRESHOLD. If true, it exits with error code 1, failing the build.

Promote (only runs if evaluate passes):

gcloud run services update-traffic ... --to-tags sha-${COMMIT_SHA}=100

10. Distributed Tracing with OpenTelemetry

Debugging a single monolithic LLM call might be easy. Debugging a distributed system of multiple agents, each making its own LLM calls and tool executions, is exponentially harder. If the orchestrator gives a wrong answer, was it bad logic in the orchestrator? Did the researcher return bad data? Or did a network timeout cause a fallback? To answer these questions, logging isn't enough. We need distributed tracing.

We use OpenTelemetry (OTel) to instrument every part of the stack, capturing the entire lifecycle of a request as a Trace graph.

The Waterfall View in Cloud Trace

By integrating with Cloud Trace, we get a visual waterfall of every operation.

Root span: The initial request to the web app's backend and to the orchestrator.
Child spans: Cross-service A2A requests to other agents, LLM invocations, and tool executions.

The Trace graph lets us see the system's physical execution alongside the logical reasoning.

Enabling End-to-End Tracing with Shared Components

ADK comes with built-in OpenTelemetry support. However, to get a truly unified view across our microservices, we enhanced it with our shared components:

shared/adk_app.py:

Wraps the standard ADK FastAPI app.
Adds OpenTelemetryMiddleware to automatically start a trace span for every incoming HTTP request.
Correctly extracts the traceparent context from incoming headers, connecting this agent's work to the caller's trace.

shared/traced_authenticated_httpx.py:

When an agent calls another agent (e.g., orchestrator -> researcher), we must propagate the trace ID.
This custom client injects the OTel traceparent header into outgoing requests.

System Traces vs. Reasoning Traces

It's crucial to distinguish between the two types of traces that we discuss in this post:

Feature	Reasoning trace (intermediate events)	OpenTelemetry trace (system trace)
Source	The agent's thought process (SSE stream).	The code's execution path (FastAPI, HTTPX, ADK).
Content	"I should search for ...", tool definition, tool output.	Latency, HTTP status codes, service errors.
Goal	Agent observability: Did the agent make the right plan? What parameters did it use?	System observability: What service was called? Which service failed? Was it slow?
Storage	Vertex AI Gen AI evaluation service (JSON).	Cloud Trace (waterfall UI).

Reasoning traces give you visibility into the cognitive process of your agent, while system traces show how API requests flow through your system.

Debugging Non-Deterministic Systems

The combination of these types of traces is your superpower. When an evaluation fails (e.g., "grounding score < 0.5"), you look at the reasoning trace to see what the model thought. If the reasoning looks correct but the result is wrong (e.g., a tool error), you switch to the OpenTelemetry trace in Cloud Trace. You might find that the wikipedia_search timed out after 5000ms, causing the model to hallucinate an answer because it lacked data.

Without this x-ray vision insight into both the cognitive and physical layers of your system, you're debugging in the dark.

Conclusion

Stop messing with the vibe checks. Instead, use the power of evaluated intelligence. Building reliable AI agents requires a shift in mindset from discovery to defense. By implementing continuous evaluation (CE), we treat agentic systems with the rigor that they deserve.

This post explores concepts from the codelab From "vibe checks" to data-driven Agent Evaluation. To run the code yourself, check out the codelab.

Resources and Links

Cloud Run runs and scales your AI agents, isolates failure domains, and enables zero-risk shadow deployments.
Vertex AI Evaluation provides the managed metrics, the adaptive rubrics, and the compute scaling to run them without managing infrastructure.
Cloud Build for CI/CD creates a quality firewall, which helps guarantee that no regression goes unnoticed.
Cloud Trace provides the ability to capture and visualize the entire request lifecycle, from the initial HTTP request through a cascade of cross-services calls, sub-agent invocations, and LLM calls, to the final response.

Connect with Us

Vlad Kolesnikov → Linkedin, X

Give your agentic chatbots a fast and reliable long-term memory

Fri, 27 Feb 2026 17:00:00 +0000

When scaling conversational agents, the data layer design often determines success or failure. To support millions of users, agents need conversational continuity — the ability to maintain responsive chats while preserving the context backend models need.

This article covers how to use Google Cloud solutions to solve two data challenges in AI: fast context updates for real-time chat, and efficient retrieval for long-term history. We’ll share a polyglot approach using Redis, Bigtable, and BigQuery that ensures your agent retains detail and continuity, from recent interactions to months-old archives.

Polyglot storage approach for short, mid, and long-term history

What is a polyglot approach?

A polyglot approach uses a multi-tiered storage strategy that leverages several specialized data services rather than a single database to manage different data lifecycles. This allows an application to use the specific strengths of various tools—such as in-memory caches for speed, NoSQL databases for scale, blob storage for unstructured artifacts, and data warehousing for analytics—to handle the "temperature" and volume of data effectively.

Define a polyglot approach on Google Cloud for short, mid, and long-term memory

To maintain conversational continuity, you can implement this polyglot approach using Memorystore for Redis for sub-millisecond "hot" context retrieval, Cloud Bigtable as a petabyte-scale system of record for durable history, and BigQuery for long-term archival and analytical insights, with Cloud Storage handling unstructured multimedia and an asynchronous pipeline built using Pub/Sub and Dataflow.

1. Short-term memory: Memorystore for Redis

Users expect chat histories to load instantaneously, whether they are initiating a new chat or continuing a previous conversation. For context of a conversation, Memorystore for Redis serves as the primary cache. As a fully managed in-memory data store, it provides the sub-millisecond latency required to maintain a natural conversational flow. Since chat sessions are incrementally growing lists of messages, we store history using Redis Lists. By using the native RPUSH command, the application transmits only the newest message, avoiding the network-heavy "read-modify-write" cycles found in simpler stores like Memcached.

2. Mid-term memory: Cloud Bigtable

As the conversation grows over time, the agentic applications need to account for larger and longer term storage of a growing chat history. This is where Bigtable acts as the durable mid-term store and the definitive system of record for all chat history. Bigtable is a petabyte-scale NoSQL database designed specifically for high-velocity, write-heavy workloads, making it perfect for capturing millions of simultaneous chat interactions. While it handles massive data volumes, teams can keep the active cluster lean by implementing garbage collection policies — retaining, for example, only the last 60 days of data in the high-performance tier. To make lookups fast, we use a key strategy with a user_id#session_id#reverse_timestamp pattern. This co-locates all messages from a single session, allowing for efficient range scans to retrieve the most recent messages for history reloads.

3. Long-term memory and analytics: BigQuery

For archival and analytics, data moves to BigQuery, representing the long-term memory of the system. While Bigtable is optimized for serving the live application, BigQuery is Google's premier serverless data warehouse designed for complex SQL queries at scale. This allows teams to go beyond simple logging and derive analytical insights. Ultimately, this operational data becomes a feedback loop for improving the agent and user experience without impacting the performance of the user-facing components.

4. Artifact storage: Cloud Storage (GCS)

Unstructured data such as multimedia files — whether uploaded by a user for analysis or generated by a generative model — live in Cloud Storage, which is purpose built for unstructured artifacts. We utilize a pointer strategy where Redis and Bigtable records contain a URI pointer (e.g., gs://bucket/file) to the object. To maintain security, the application serves these files using signed URLs, providing the client with time-limited access without exposing the bucket publicly.

A hybrid sync-async strategy for optimal flow of data

As shown in the sequence diagrams below, the hybrid sync-async strategy utilizes the abovementioned storage solutions to balance high-speed consistency with durable data persistence.

The diagram below shows how a user message and corresponding agent response traverse through the architecture:

The diagram below shows how data flows across the architecture when a user decides to retrieve chat history for a particular session:

Start building now

Ready to build an agent with a robust persistence layer?

Build agents quickly: Start prototyping your agentic workflows on Vertex AI Agent Builder.
Configure your cache: Determine which Memorystore for Redis configuration best suits your latency and availability needs.
Design a robust BigTable schema: Review the schema design best practices.
Bridge to analytics: Use the Bigtable change stream to BigQuery template to ready your live chat logs for actionable business insights.
Bring data to life with analytics: Use Looker Conversational Analytics to drive product decisions through business intelligence.