<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:media="http://search.yahoo.com/mrss/"><channel><title>Developers &amp; Practitioners</title><link>https://cloud.google.com/blog/topics/developers-practitioners/</link><description>Developers &amp; Practitioners</description><atom:link href="https://cloudblog.withgoogle.com/blog/topics/developers-practitioners/rss/" rel="self"></atom:link><language>en</language><lastBuildDate>Fri, 13 Nov 2026 16:49:25 +0000</lastBuildDate><image><url>https://cloud.google.com/blog/topics/developers-practitioners/static/blog/images/google.a51985becaa6.png</url><title>Developers &amp; Practitioners</title><link>https://cloud.google.com/blog/topics/developers-practitioners/</link></image><item><title>Experimenting with ClusterNetworkPolicy in GKE</title><link>https://cloud.google.com/blog/topics/developers-practitioners/experimenting-with-clusternetworkpolicy-in-gke/</link><description>&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Standard &lt;/span&gt;&lt;a href="https://kubernetes.io/docs/concepts/services-networking/network-policies/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Network Policies&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; limit you to applying policies at the namespace level, but &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/kubernetes-engine/docs/how-to/configure-cluster-network-policy"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;ClusterNetworkPolicy (CNP)&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; lets you apply granular networking policies across an entire cluster. When you use CNP, you can easily scale security and governance policies. Recently, I experimented with this functionality in &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/kubernetes-engine/docs/concepts/kubernetes-engine-overview"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Google Kubernetes Engine (GKE)&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; and saw the benefits for myself.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;In this post, we'll explore using CNP to control traffic to an LLM and cluster. If you want to jump right into a hands-on lab, check out this Codelab: &lt;/span&gt;&lt;a href="https://codelabs.developers.google.com/codelabs/inference-gke-clusternetworkpolicy" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Secure AI Inference Workloads with GKE ClusterNetworkPolicy&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;strong style="vertical-align: baseline;"&gt;How CNP works&lt;/strong&gt;&lt;/h2&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;CNP &lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;follows a &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/kubernetes-engine/docs/how-to/configure-cluster-network-policy#policy_evaluation_hierarchy"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;three-tier&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; flow with a precedence match:&lt;/span&gt;&lt;/p&gt;
&lt;ol&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Admin tier (top precedence)&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Cluster administrator enforces mandatory security rules that can't be bypassed (for example, blocking metadata access).&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Network policy tier (middle precedence)&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Standard namespace-scoped policies that are managed by application teams to control ingress and egress for their microservices.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Baseline tier (bottom precedence)&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Cluster administrator establishes fallback defaults (for example, a default-deny posture across namespaces).&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;h2&gt;&lt;strong style="vertical-align: baseline;"&gt;Design pattern example&lt;/strong&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The aim of the &lt;/span&gt;&lt;a href="https://codelabs.developers.google.com/codelabs/inference-gke-clusternetworkpolicy#0" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;experiment in the Codelab&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; is to restrict traffic to the LLM namespace, allow only authorized clients, and prevent cluster-wide access to the metadata API except for DNS port 53. It uses these services:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://docs.cloud.google.com/vpc/docs/create-modify-vpc-networks"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;Virtual Private Cloud (VPC)&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;: Provides private networking, subnets for pods and services, and secure outbound egress via Cloud NAT.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://docs.cloud.google.com/tpu/docs/v5e"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;TPU v5e&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;: High-efficiency accelerator hardware (2x2 topology) running optimized model inference.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;GKE&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Managed Kubernetes platform that's powered by GKE Dataplane V2 (eBPF) to enforce multi-tier networking policies.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://huggingface.co/google/gemma-4-E2B-it" rel="noopener" target="_blank"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;Gemma 4 (&lt;/strong&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;google/gemma-4-E2B-it&lt;/strong&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;)&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;: Lightweight open model served with vLLM to act as a sensitive AI workload.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;ClusterNetworkPolicy (CNP)&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Cluster-wide policies that let administrators enforce mandatory administrator and baseline guardrails across all namespaces.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://docs.cloud.google.com/kubernetes-engine/docs/tutorials/network-policy"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;NetworkPolicies&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;: Standard namespace-scoped rules that let developers grant access to authorized clients.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Cloud Logging / &lt;/strong&gt;&lt;a href="https://docs.cloud.google.com/kubernetes-engine/docs/how-to/network-policy-logging?utm_campaign=CDR_0x78e4a6e6_default_b565725300&amp;amp;utm_medium=external&amp;amp;utm_source=lab"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;GKE Network Policy Logging&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;: Captures real-time packet verdicts (&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;allow&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;/&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;deny&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;) and policy audit trails.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/clusternetworkpolicy-flow.max-1000x1000.png"
        
          alt="clusternetworkpolicy-flow"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h2&gt;&lt;strong style="vertical-align: baseline;"&gt;Putting it together: Setup overview&lt;/strong&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Setting up this environment begins with building a custom private network infrastructure and deploying your TPU-enabled clusters:&lt;/span&gt;&lt;/p&gt;
&lt;ol&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Custom VPC &amp;amp; Cloud NAT&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Establish a custom VPC with a dedicated subnet that hosts primary nodes, plus secondary CIDR IP address ranges for GKE pods (&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;10.20.0.0/16&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;) and services (&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;10.30.0.0/20&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;). A Cloud Router and Cloud NAT are configured to handle secure outbound egress for nodes while blocking inbound internet exposure.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;GKE Standard cluster&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Spin up a Standard GKE cluster running GKE Dataplane V2 (&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;ADVANCED_DATAPATH&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;). GKE Dataplane V2 acts as the core engine that leverages eBPF under the hood to enforce both standard and cluster-scoped policies dynamically.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Dedicated TPU v5e node pool&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Provision a node pool featuring &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;ct5lp-hightpu-4t&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; machines arranged in a 2x2 topology slice. The nodes carry a dedicated taint (&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;google.com/tpu=present:NoSchedule&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;) to ensure that only accelerator workloads run on them.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Deploying the Gemma 4 workload&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Isolate the pods by creating three distinct namespaces:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li role="presentation"&gt;&lt;code style="vertical-align: baseline;"&gt;gemma-llm&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; (&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;tier: ai-core&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;): Serves the lightweight Gemma 4 model using the high-performance &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;vllm/vllm-tpu:latest&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; engine on port 8000.&lt;/span&gt;&lt;/li&gt;
&lt;li role="presentation"&gt;&lt;code style="vertical-align: baseline;"&gt;client-ns&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; (&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;tier: app-frontend&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;): Runs the authorized chat client that must query the model.&lt;/span&gt;&lt;/li&gt;
&lt;li role="presentation"&gt;&lt;code style="vertical-align: baseline;"&gt;untrusted-tenant&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; (&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;tier: untrusted&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;): Represents an unauthorized probe container simulating a rogue client.&lt;/span&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;h2&gt;&lt;strong style="vertical-align: baseline;"&gt;Securing the LLM and metadata: Code walkthrough&lt;/strong&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;By default in Kubernetes, all cross-namespace and egress traffic is open. To enforce a defense-in-depth posture, we apply network policies across the GKE evaluation tiers.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;Step 1: Establish zero-trust posture (baseline tier)&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;To deny all unauthorized ingress cluster-wide, the cluster administrator applies a ClusterNetworkPolicy at the baseline tier. This tier acts as a fallback safety net.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;apiVersion: policy.networking.k8s.io/v1alpha2\r\nkind: ClusterNetworkPolicy\r\nmetadata:\r\n  name: baseline-default-deny\r\nspec:\r\n  tier: Baseline\r\n  priority: 1000\r\n  subject:\r\n    namespaces: {}\r\n  ingress:\r\n    - name: allow-dns-ingress\r\n      action: Accept\r\n      from:\r\n        - namespaces: {}\r\n      protocols:\r\n        - udp:\r\n            destinationPort:\r\n              number: 53\r\n        - tcp:\r\n            destinationPort:\r\n              number: 53\r\n    - name: baseline-deny-cross-ns\r\n      action: Deny\r\n      from:\r\n        - namespaces: {}&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba882290&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h4&gt;&lt;strong style="vertical-align: baseline;"&gt;How it works&lt;/strong&gt;&lt;/h4&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Precedence placement&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Setting &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;tier: Baseline&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; places this policy at the very bottom of the GKE evaluation chain, and the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;priority: 1000&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; value orders it within that tier. It only processes traffic that hasn't matched a rule in any higher tier.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Global target&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Specifying &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;subject.namespaces: {}&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; ensures that this fallback applies globally to every pod in every namespace.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;DNS isolation exemption&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The first ingress rule accepts incoming DNS traffic on TCP and UDP port 53. Without this explicit exemption, cluster-wide DNS resolution would fail.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Fallback deny&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The second rule matches any incoming pod-to-pod connection attempt and drops it. At this stage, both the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;client-ns&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; and &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;untrusted-tenant&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; namespaces are blocked from contacting Gemma.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;Step 2: Delegate access to developers (networkpolicy tier)&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;With the baseline default-deny active, the application team that manages &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;gemma-llm&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; can safely delegate access to legitimate clients. They define a standard namespace-scoped &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;NetworkPolicy&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; to allow incoming port 8000 traffic from &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;rag-agent&lt;/code&gt; &lt;span style="vertical-align: baseline;"&gt;pods in the&lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt; &lt;/strong&gt;&lt;code style="vertical-align: baseline;"&gt;client-ns&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; namespace.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;apiVersion: networking.k8s.io/v1\r\nkind: NetworkPolicy\r\nmetadata:\r\n  name: allow-rag-client\r\n  namespace: gemma-llm\r\nspec:\r\n  podSelector:\r\n    matchLabels:\r\n      app: gemma-service\r\n  policyTypes:\r\n    - Ingress\r\n  ingress:\r\n    # Explicitly permit ingress only from pods matching app: rag-agent in client-ns\r\n    - from:\r\n        - namespaceSelector:\r\n            matchLabels:\r\n              kubernetes.io/metadata.name: client-ns\r\n          podSelector:\r\n            matchLabels:\r\n              app: rag-agent\r\n      ports:\r\n        - protocol: TCP\r\n          port: 8000&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba881fd0&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h4&gt;&lt;strong style="vertical-align: baseline;"&gt;How it works&lt;/strong&gt;&lt;/h4&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Developer control&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: This resource resides within the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;gemma-llm&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; namespace and it's managed directly by the developer, matching the GKE focus on operational separation.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Middle precedence matching&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Standard Kubernetes &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;NetworkPolicies&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; map directly to the NetworkPolicy Tier. Because this tier is evaluated before the baseline tier, incoming TCP packets from the authorized client to port 8000 match this rule and are immediately accepted.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Fall-through block&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Connections that originate from the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;untrusted-tenant&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; namespace fail to match this rule, pass through to the baseline tier, and are dropped.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;Step 3: Enforce global egress guardrails (admin tier)&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;By default, Pods can query the GKE link-local Compute Engine metadata server (&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;169.254.169.254&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;). Those queries expose instance metadata and potentially high-privilege service account tokens. To lock this down, the cluster administrator applies an egress guardrail at the highest tier in the GKE hierarchy.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;apiVersion: policy.networking.k8s.io/v1alpha2\r\nkind: ClusterNetworkPolicy\r\nmetadata:\r\n  name: admin-global-guardrails\r\nspec:\r\n  tier: Admin\r\n  priority: 10\r\n  subject:\r\n    namespaces: {}  # Targets every namespace in the cluster\r\n  ingress:\r\n    # Delegate all ingress decisions down to developer and baseline tiers\r\n    - name: delegate-all-ingress\r\n      action: Pass\r\n      from:\r\n        - namespaces: {}\r\n  egress:\r\n    # Allow DNS resolution so pods can locate endpoints\r\n    - name: allow-all-dns\r\n      action: Accept\r\n      to:\r\n        - networks:\r\n            - 0.0.0.0/0\r\n      protocols:\r\n        - udp:\r\n            destinationPort:\r\n              number: 53\r\n        - tcp:\r\n            destinationPort:\r\n              number: 53\r\n    # Hard deny on the Compute Engine metadata server\r\n    - name: block-metadata-server-global\r\n      action: Deny\r\n      to:\r\n        - networks:\r\n            - 169.254.169.254/32\r\n    # Delegate remaining egress traffic to lower tiers\r\n    - name: delegate-other-egress\r\n      action: Pass\r\n      to:\r\n        - namespaces: {}&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba883250&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h4&gt;&lt;strong style="vertical-align: baseline;"&gt;How it works&lt;/strong&gt;&lt;/h4&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Admin tier precedence&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Deploying this policy in the admin tier ensures that it's evaluated first, and priority 10 orders it within that tier. GKE processes this policy before standard NetworkPolicies and baseline rules.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Controlled delegation&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;delegate-all-ingress&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; and &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;delegate-other-egress&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; rules have an &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;action: Pass&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; directive that instructs the engine to forward evaluation down to the next tier for remaining in-cluster traffic.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Inviolable guardrail&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: When any pod attempts to query &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;169.254.169.254/32&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; on non-DNS ports, the query matches the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;block-metadata-server-global&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; rule and it's dropped. Because this drop occurs at the top admin tier, it can't be bypassed or overridden, even if a namespace developer deploys an allow-all egress policy.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;&lt;strong style="vertical-align: baseline;"&gt;Verifying with Observability for GKE&lt;/strong&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Verifying and auditing security policies is straightforward using the built-in telemetry capabilities of GKE Dataplane V2:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Hubble eBPF streams&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;:&lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt; &lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;Provides real-time, kernel-level packet inspection without injecting sidecars. Administrators can stream live network flows to observe policy verdicts as they happen, making it easy to validate allowed communication paths and identify dropped packets immediately.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;GKE Network Logging and Cloud Logging&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Automatically exports structured connection telemetry and policy enforcement verdicts directly into Cloud Logging. Teams can filter and alert on policy decisions using attributes such as &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;disposition="deny"&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, source/destination namespaces, and &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;jsonPayload.policies[0].name&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; to maintain audit compliance and investigate blocked connection attempts.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;CNP can be useful and, as always, when you use it in your production environment, you'll need to plan and design the rules based on your setup.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;strong style="vertical-align: baseline;"&gt;Next steps&lt;/strong&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Take a deeper dive and find more information on these features:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://cloud.google.com/blog/products/networking/new-clusternetworkpolicy-in-gke"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;ClusterNetworkPolicy in GKE: Balancing control and autonomy for your microservices&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://cloud.google.com/kubernetes-engine/docs/how-to/configure-cluster-network-policy"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Configuring GKE ClusterNetworkPolicy&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Want to ask a question, find out more, or share a thought? Connect with me on&lt;/span&gt;&lt;a href="https://www.linkedin.com/in/ammett/" rel="noopener" target="_blank"&gt;&lt;span style="vertical-align: baseline;"&gt; &lt;/span&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;LinkedIn&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;!&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;</description><pubDate>Tue, 10 Nov 2026 07:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/topics/developers-practitioners/experimenting-with-clusternetworkpolicy-in-gke/</guid><category>Networking</category><category>Developers &amp; Practitioners</category><media:content height="540" url="https://storage.googleapis.com/gweb-cloudblog-publish/images/clusternetworkpolicy-hero_TEG7Wrh.max-600x600.png" width="540"></media:content><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Experimenting with ClusterNetworkPolicy in GKE</title><description></description><image>https://storage.googleapis.com/gweb-cloudblog-publish/images/clusternetworkpolicy-hero_TEG7Wrh.max-600x600.png</image><site_name>Google</site_name><url>https://cloud.google.com/blog/topics/developers-practitioners/experimenting-with-clusternetworkpolicy-in-gke/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Ammett Williams</name><title>Developer Relations Engineer</title><department></department><company></company></author></item><item><title>The open AI-native stack on Kubernetes: From open models to distributed inference and agents</title><link>https://cloud.google.com/blog/topics/developers-practitioners/the-open-ai-native-stack-on-kubernetes-from-open-models-to-distributed-inference-and-agents/</link><description>&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Over the past decade, Kubernetes established itself as the standard for orchestrating microservices and containerized applications. Today, we're witnessing the next major architectural shift in distributed computing: &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;AI workloads moving from isolated research prototypes into large-scale, multi-tenant production fleets&lt;/strong&gt;. &lt;span style="vertical-align: baseline;"&gt;In this post, we'll walk through the open-source stack that powers this shift and show you the sample manifests that you can try on your own Kubernetes clusters. You can also join us at KubeCon + CloudNativeCon North America 2026 to see these projects in action through live demos and technical sessions.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;For modern engineering teams, production AI is rapidly moving beyond generic calls to closed, black-box APIs. Organizations are realizing that their long-term competitive advantage lies in domain specialization. Platform engineers, ML researchers, and AI builders want the autonomy to run high-performing &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;open-weight models&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; (such as the &lt;/span&gt;&lt;a href="https://ai.google.dev/gemma/docs" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Gemma&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; family). They want to adapt them to proprietary enterprise datasets by using &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;reinforcement learning (RL)&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; and serve them with predictable latency, high throughput, and maximum hardware efficiency.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Whether it's scientific initiatives training domain models like &lt;/span&gt;&lt;a href="https://blog.google/innovation-and-ai/products/dolphingemma/" rel="noopener" target="_blank"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;DolphinGemma&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; (an LLM helping scientists study how dolphins communicate) on acoustic data, legal platforms evaluating complex contracts, or engineering teams building autonomous coding agents, the objective is the same: &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;achieving frontier-level task performance at a fraction of the serving cost and latency.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Yet, scaling RL post-training, stateful distributed inference, and autonomous agent execution on traditional container infrastructure introduces substantial hurdles:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Accelerator underutilization in RL&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: When you alternate between training and rollout generation, expensive GPUs and TPUs remain idle for most of the training run.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Cache-blind load balancing&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: When you use standard Kubernetes round-robin and least-connections routing, multi-turn agent calls are scattered across arbitrary pods and wipe out key-value (KV) cache locality.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Framework fragmentation&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: When you operate parallel, siloed compute clusters (such as bespoke &lt;/span&gt;&lt;a href="https://www.ray.io/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Ray&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; or &lt;/span&gt;&lt;a href="https://slurm.schedmd.com/overview.html" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Slurm&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; setups) alongside Kubernetes, you fragment capacity and multiply maintenance toil.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Agent security in multi-tenant clusters:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; When you execute untrusted, model-generated shell commands and scripts in shared environments, you expose clusters to sandbox escapes and privilege escalation.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;At &lt;/span&gt;&lt;a href="https://rsvp.withgoogle.com/events/google-cloud-at-kubecon-north-america-2026?pli=1" rel="noopener" target="_blank"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;KubeCon + CloudNativeCon North America 2026&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, Google Cloud is addressing these challenges by introducing and expanding a unified, open-source AI-native stack across Kubernetes and &lt;/span&gt;&lt;a href="https://cloud.google.com/kubernetes-engine"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Google Kubernetes Engine (GKE)&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; based on these components:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="http://openrl.dev" rel="noopener" target="_blank"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;OpenRL&lt;/strong&gt;&lt;/a&gt;&lt;strong style="vertical-align: baseline;"&gt; (CNCF Sandbox):&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; A Kubernetes-native post training service that simplifies RL post-training and maximizes accelerator utilization.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://llm-d.ai/" rel="noopener" target="_blank"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;llm-d 1.0&lt;/strong&gt;&lt;/a&gt;&lt;strong style="vertical-align: baseline;"&gt; (CNCF Sandbox):&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; An open distributed inference control plane that provides prefix-cache-aware routing, disaggregated serving, and multi-tier KV cache management.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://docs.cloud.google.com/kubernetes-engine/docs/add-on/ray-on-gke/concepts/overview"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;Ray on GKE with KubeRay&lt;/strong&gt;&lt;/a&gt;&lt;strong style="vertical-align: baseline;"&gt; (CNCF Sandbox)&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: A framework that provides enterprise lifecycle management, elastic autoscaling, and &lt;/span&gt;&lt;a href="https://kueue.sigs.k8s.io/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Kueue&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;-driven scheduling for distributed Ray compute and data processing.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://ate.dev/" rel="noopener" target="_blank"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;Agent Substrate&lt;/strong&gt;&lt;/a&gt;&lt;strong style="vertical-align: baseline;"&gt; (CNCF Sandbox)&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: An open orchestration runtime that provides lifecycle management and execution targets for multi-turn AI agent workflows.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://gvisor.dev/" rel="noopener" target="_blank"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;gVisor&lt;/strong&gt;&lt;/a&gt;&lt;strong style="vertical-align: baseline;"&gt; (CNCF Sandbox)&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: A kernel-level container sandbox that delivers strong system call isolation when you execute untrusted agent code in multi-tenant clusters.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://pytorch.org/blog/introducing-torchtpu/" rel="noopener" target="_blank"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;TorchTPU&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;: A backend that provides first-class, native PyTorch support across Cloud TPUs and GPUs inside upstream &lt;/span&gt;&lt;a href="https://vllm.ai/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;vLLM&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; and &lt;/span&gt;&lt;a href="https://docs.sglang.io/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;SGLang&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Bridge reinforcement learning and Kubernetes with OpenRL&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;While supervised fine-tuning (SFT) has become standard practice, domain excellence especially in multi-step reasoning, mathematical proofing, and autonomous tool use requires RL.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Historically, it's been difficult to bridge the world of ML experimentation with production Kubernetes fleets. ML researchers want to iterate rapidly in Python and PyTorch by using standard reward functions, while platform teams need multi-tenant governance, quota enforcement, and predictable scheduling.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;To eliminate this friction, Google created &lt;/span&gt;&lt;a href="https://openrl.dev" rel="noopener" target="_blank"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;OpenRL&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; and contributed it to the &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;CNCF Sandbox&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Architectural highlights of OpenRL&lt;/span&gt;&lt;/h3&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Clean 4-primitive Python abstraction&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Researchers write standard Python training code that's centered around four core primitives (&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;forward_backward&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;optim_step&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;sample&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, and &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;save_state&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;). OpenRL translates these primitives into Kubernetes scheduling workflows without requiring developers to learn Kubernetes internals or manage infrastructure code.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Cooperative accelerator time-slicing&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: OpenRL interleaves training steps and rollout inferences on the same physical accelerators. This approach avoids the need to dedicate separate, static clusters to compute-bound training and latency-sensitive rollout generation.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Delta-weight synchronization&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Shipping full multi-gigabyte model weights over cluster networking after every optimization step creates massive network saturation. OpenRL transmits only parameter deltas to rollout workers, eliminating rollout cold starts.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Zero-config dynamic resource allocation (&lt;/strong&gt;&lt;a href="https://kubernetes.io/docs/concepts/resource-management/dynamic-resource-allocation/" rel="noopener" target="_blank"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;DRA&lt;/strong&gt;&lt;/a&gt;&lt;strong style="vertical-align: baseline;"&gt;)&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: OpenRL automatically profiles model context lengths, batch dimensions, and LoRA versus full fine-tuning configurations. It then dynamically bin-packs workloads across a heterogenous accelerator fleet by using native Kubernetes DRA claims.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Distributed inference at scale with llm-d 1.0&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;After you post-train a model, serving it serving it under interactive, bursty agent traffic presents a new set of distributed systems challenges. Standard ingress controllers treat LLM traffic as stateless HTTP requests, scattering sequential turns across different replicas and constantly rebuilding KV cache.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Google Cloud, Red Hat, IBM Research, CoreWeave, and NVIDIA co-founded the project, with contributions from over 500 engineers across 25+ organizations. Today, &lt;/span&gt;&lt;a href="https://llm-d.ai" rel="noopener" target="_blank"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;llm-d 1.0&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; is the CNCF Sandbox distributed inference control plane that transforms standalone inference engines (like vLLM and SGLang) into a coordinated, cluster-wide serving fabric.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;apiVersion: inference.networking.x-k8s.io/v1alpha1\r\nkind: InferencePool\r\nmetadata:\r\n  name: gemma-serving-pool\r\n  namespace: ai-serving\r\nspec:\r\n  targetRef:\r\n    group: apps\r\n    kind: Deployment\r\n    name: vllm-gemma-27b\r\n  scheduling:\r\n    # Routes multi-turn requests to replicas holding matching KV context\r\n    strategy: PrefixCacheAware\r\n    fallback: LeastQueueDepth\r\n  observability:\r\n    kvCacheSyncInterval: &amp;quot;50ms&amp;quot;&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba93ae50&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Core capabilities of llm-d 1.0&lt;/span&gt;&lt;/h3&gt;
&lt;ol&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Prefix-cache-aware routing (&lt;/strong&gt;&lt;a href="https://llm-d.ai/docs/dev/architecture/router/epp" rel="noopener" target="_blank"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;Endpoint Picker&lt;/strong&gt;&lt;/a&gt;&lt;strong style="vertical-align: baseline;"&gt;)&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The Endpoint Picker integrates into the &lt;/span&gt;&lt;a href="https://gateway-api-inference-extension.sigs.k8s.io/" rel="noopener" target="_blank"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;Kubernetes Gateway API Inference Extension,&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; and GKE manages it natively through &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/kubernetes-engine/docs/concepts/about-gke-inference-gateway"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;GKE Inference Gateway&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;. The Endpoint Picker evaluates real-time KV cache residency across worker pods. In production deployments, it's &lt;/span&gt;&lt;a href="https://llm-d.ai/blog/llm-d-v1.0-state-of-the-art-inference-ready-for-day-2" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;doubled KV cache hit rates&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; from 35% to 70% and &lt;/span&gt;&lt;a href="https://cloud.google.com/blog/topics/ai-infrastructure/llm-d-1-0-an-open-source-stack-for-distributed-inference"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;cut median time-to-first-token (TTFT)&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; by 70-85% on multi-turn agent workloads.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;3-Tier KV cache offload&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: When accelerator high-bandwidth memory (HBM) nears capacity during long-context interactions, llm-d cascades KV cache blocks from &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;HBM → host CPU RAM → local NVMe SSD / Google Cloud Managed Lustre / Cloud Storage.&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; This offload prevents expensive context recalculations.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;EPD Disaggregation with &lt;/strong&gt;&lt;strong style="vertical-align: baseline;"&gt;DisaggregatedSet&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Decouples &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Encoding, Prefill, and Decode (EPD)&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; onto specialized hardware pools that are connected by high-speed interconnects. &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;DisaggregatedSet is built &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;built on Kubernetes &lt;/span&gt;&lt;a href="https://lws.sigs.k8s.io/stable/docs/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;LeaderWorkerSet (LWS)&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;. It enables independent horizontal scaling and coordinated, zero-downtime rolling upgrades.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Native TorchTPU upstream support&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: PyTorch models run natively on Google Cloud TPUs inside upstream vLLM and SGLang through TorchTPU. Developers get day-zero parity across TPU v5e, v6e, and v7x alongside NVIDIA GPUs without the need to maintain private forks or rewrite pipelines.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Latency-tolerant async processing &amp;amp; flow control&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;llm-d-async-processor component&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; pulls background, latency-tolerant tasks from Pub/Sub or Redis into cluster slack capacity. It applies strict traffic prioritization so background processing never degrades interactive user experiences.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Fleet observability with Prism&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Track cluster-wide throughput, inter-token latency (ITL), TTFT, and cache hit ratios by using &lt;/span&gt;&lt;a href="https://github.com/llm-d/llm-d-prism" rel="noopener" target="_blank"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;Prism&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, the llm-d open-source benchmarking and performance dashboard.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Distributed Python &amp;amp; ML orchestration with KubeRay&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;While specialized runtimes handle inference and RL loops, distributed data preparation, embedding generation, and distributed PyTorch training depend heavily on &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Ray&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;KubeRay&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; provides an open-source, Kubernetes-native operator that manages the complete lifecycle of Ray clusters, jobs, and services (&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;RayCluster&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;RayJob&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;, and &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;RayService&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; custom resources) on Kubernetes.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;apiVersion: ray.io/v1\r\nkind: RayJob\r\nmetadata:\r\n  name: gemma-fine-tune-job\r\n  namespace: ai-workloads\r\n  labels:\r\n    kueue.x-k8s.io/queue-name: ml-team-queue\r\nspec:\r\n  shutdownAfterJobFinishes: true\r\n  ttlSecondsAfterFinished: 60\r\n  entrypoint: python fine_tune_gemma.py --model-id google/gemma-2-27b-it\r\n  runtimeEnvYAML: |\r\n    pip:\r\n      - datasets==2.18.0\r\n      - transformers==4.40.0\r\n      - torch==2.2.0\r\n  rayClusterSpec:\r\n    rayVersion: &amp;quot;2.35.0&amp;quot;\r\n    headGroupSpec:\r\n      rayStartParams:\r\n        dashboard-host: &amp;quot;0.0.0.0&amp;quot;\r\n      template:\r\n        spec:\r\n          containers:\r\n            - name: ray-head\r\n              image: rayproject/ray-ml:2.35.0\r\n              resources:\r\n                limits:\r\n                  cpu: &amp;quot;4&amp;quot;\r\n                  memory: &amp;quot;16Gi&amp;quot;\r\n                requests:\r\n                  cpu: &amp;quot;2&amp;quot;\r\n                  memory: &amp;quot;8Gi&amp;quot;\r\n    workerGroupSpecs:\r\n      - groupName: gpu-workers\r\n        replicas: 4\r\n        minReplicas: 1\r\n        maxReplicas: 8\r\n        rayStartParams: {}\r\n        template:\r\n          spec:\r\n            containers:\r\n              - name: ray-worker\r\n                image: rayproject/ray-ml:2.35.0\r\n                resources:\r\n                  limits:\r\n                    nvidia.com/gpu: &amp;quot;1&amp;quot;\r\n                    cpu: &amp;quot;8&amp;quot;\r\n                    memory: &amp;quot;32Gi&amp;quot;\r\n                  requests:\r\n                    nvidia.com/gpu: &amp;quot;1&amp;quot;\r\n                    cpu: &amp;quot;4&amp;quot;\r\n                    memory: &amp;quot;16Gi&amp;quot;&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba939690&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Why KubeRay matters for platform teams&lt;/span&gt;&lt;/h3&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Declarative cluster lifecycle&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Platform engineers can spin up ephemeral, self-healing Ray clusters on demand for ad-hoc training runs, or run long-lived clusters for batch pipelines.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Native Kueue queueing &amp;amp; multi-tenancy&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: KubeRay integrates directly with &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Kueue&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; for multi-tenant fair sharing, priority preemption, and quota management across shared GPU and TPU capacity.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Elastic autoscaling&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: KubeRay dynamically scales worker pods based on active Ray tasks and actor placement, preventing idle resource consumption.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Strong isolation for AI agents: Agent Substrate &amp;amp; gVisor&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;As multi-turn AI agents advance from conversational assistants to autonomous systems that generate code, execute shell commands, and orchestrate external APIs running untrusted, model-generated code in shared clusters introduces serious security risks.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Google Cloud is contributing &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Agent Substrate&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; and continuing core contributions to &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;gVisor&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; in the CNCF Sandbox:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Kernel-level sandboxing&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: gVisor provides a user-space kernel that intercepts and virtualizes system calls, which helps ensure that untrusted agent code can't compromise the host kernel or access neighboring tenant workloads.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Sub-second ephemeral execution&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Sandboxes spin up in milliseconds per agent turn, providing isolated execution environments for both RL rollout generation in OpenRL and live production agent workflows.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Proven in practice: Mastering domain reasoning&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;To evaluate this complete open-source stack on a rigorous enterprise workload, we tested &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Gemma 4&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; against the open-source &lt;/span&gt;&lt;a href="https://github.com/harveyai/harvey-labs" rel="noopener" target="_blank"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;Legal Agent Benchmark (LAB)&lt;/strong&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt; &lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;developed by &lt;/span&gt;&lt;a href="http://harvey.ai" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Harvey.ai&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;. This benchmark spans 24 complex legal practice areas from mergers and acquisitions (M&amp;amp;A)M&amp;amp;A due diligence to contract dispute analysis. The evaluation produced these results:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Benchmark results&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: While standard SFT plateaued around 50% accuracy on multi-step reasoning rubrics, when we applied reproducible RL loops with &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;OpenRL, we lifted model accuracy to 70%&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Operational efficiency&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The entire training pipeline dynamically shared GPU and TPU capacity with concurrent LoRA and inference jobs on GKE without manual partition management.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;How teams are adopting this stack&lt;/span&gt;&lt;/h2&gt;
&lt;div align="left"&gt;
&lt;div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;"&gt;
&lt;div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;"&gt;
&lt;div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;"&gt;
&lt;div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;"&gt;
&lt;div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;"&gt;
&lt;div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;"&gt;
&lt;div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;"&gt;
&lt;div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;"&gt;
&lt;div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;"&gt;&lt;table&gt;&lt;colgroup&gt;&lt;col/&gt;&lt;col/&gt;&lt;col/&gt;&lt;/colgroup&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th scope="col" style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Organization persona&lt;/strong&gt;&lt;/p&gt;
&lt;/th&gt;
&lt;th scope="col" style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Core challenge&lt;/strong&gt;&lt;/p&gt;
&lt;/th&gt;
&lt;th scope="col" style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Architecture with OpenRL + llm-d + KubeRay + Gemma&lt;/strong&gt;&lt;/p&gt;
&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Enterprise platform teams&lt;/strong&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Manage multiple internal ML squads without fragmented Slurm/Ray clusters or runaway hardware bills.&lt;/span&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Transform existing GKE clusters into self-hosted &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Reinforcement Learning-as-a-Service (RLaaS)&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; by using DRA, Kueue, and accelerator time-slicing.&lt;/span&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Regulated enterprises (finance, healthcare, legal)&lt;/strong&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Intellectual property (IP), compliance, and privacy mandates prohibit sending proprietary data to third-party public APIs.&lt;/span&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Post-train and serve 1B-30B Gemma models completely inside a private Virtual Private Cloud (VPC) with domain-specific reward functions for superior task accuracy and lower total cost of ownership (TCO).&lt;/span&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;AI-native builders &amp;amp; agent developers&lt;/strong&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;High serving costs and latency bottlenecks on long-context, multi-turn agent interactions.&lt;/span&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Deploy llm-d prefix-cache routing that's paired with gVisor-isolated &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Agent Sandbox&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; execution targets.&lt;/span&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Model-as-a-Service &amp;amp; neocloud providers&lt;/strong&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Deliver high-margin, multi-tenant post-training and inference across diverse hardware fleets.&lt;/span&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Expose OpenRL’s multi-tenant APIs and llm-d’s multi-accelerator control plane across mixed GPU and TPU clusters.&lt;/span&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Connect with us at KubeCon North America 2026&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Attending KubeCon NA in Salt Lake City? Come see these open-source technologies in action:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Join Google Cloud at KubeCon + CloudNativeCon North America 2026:&lt;/strong&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;ul&gt;
&lt;li aria-level="2" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Come say hi to us at booth #122 in Hall B.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="2" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Discover Googlers talks on our &lt;/span&gt;&lt;a href="https://rsvp.withgoogle.com/events/google-cloud-at-kubecon-north-america-2026?pli=1" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Google Cloud at KubeCon North America 2026&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; event site.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Attend featured sessions &amp;amp; keynotes:&lt;/strong&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;ul&gt;
&lt;li aria-level="2" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Kubernetes with Conviction: How Customization is Shaping the OSS AI Ecosystem's Future - &lt;/span&gt;&lt;a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/program/schedule/?id=1301727" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Salt Palace | Level 1 | Hall DE&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="2" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;The Latest in GPU, TPU, NIC and Other Device Support (WG Device Management) - &lt;/span&gt;&lt;a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/program/schedule/?id=1289704" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Salt Palace | Level 2 | 250 D-F&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="2" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Kubernetes Solutions for Agent-Shaped Problems - &lt;/span&gt;&lt;a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/program/schedule/?id=1248040" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Salt Palace | Level 1 | Hall DE&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="2" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;How Anthropic Almost Melted GKE (And the PRs That Saved It) - &lt;/span&gt;&lt;a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/program/schedule/?id=1247612" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Salt Palace | Level 1 | Ballroom H&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="2" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Gateway API Project Update: Building the Future of Traffic - &lt;/span&gt;&lt;a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/program/schedule/?id=1247612" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Salt Palace | Level 2 | 250 D-F&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Watch live demos at the Google Cloud booth:&lt;/strong&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;ul&gt;
&lt;li aria-level="2" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Demo | Inferencing with ARM processors on GKE (Tue 1:30-4:30PM MST)&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="2" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Sandbox your agent workflows with gVisor (Tue 4:30-7:30PM MST)&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="2" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Securely running fleets of agents with OpenShell (Wed 10:30-1:30PM MST)&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="2" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Cost optimization via GKE spot nodes &amp;amp; node auto-provisioning (Wed 12:30-2:30PM MST)&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="2" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;From Reasoning to Action: Arm-Powered Coding Agents with GKE Agent Sandbox and Axion (Thu 10:30-1:30PM MST)&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;/ul&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Get started&lt;/span&gt;&lt;/h2&gt;
&lt;ol&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Explore OpenRL&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Visit &lt;/span&gt;&lt;a href="https://openrl.dev" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;openrl.dev&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; and check out the &lt;/span&gt;&lt;a href="https://github.com/openrl-dev" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;OpenRL GitHub repository&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; for Kubernetes-native RL fine-tuning recipes.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Deploy llm-d 1.0&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Head to &lt;/span&gt;&lt;a href="https://llm-d.ai" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;llm-d.ai&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; to test Gateway API Inference extensions and deploy prefix-aware inference pools on your clusters.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Run KubeRay&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Follow the &lt;/span&gt;&lt;a href="https://docs.ray.io/en/latest/cluster/kubernetes/getting-started.html" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;KubeRay documentation&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; to manage distributed Ray workloads alongside Kueue.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Deploy Gemma on GKE&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Explore step-by-step guides for deploying Gemma models on TPUs and GPUs in the &lt;/span&gt;&lt;a href="https://cloud.google.com/kubernetes-engine/docs"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Google Kubernetes Engine documentation&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;&lt;/div&gt;</description><pubDate>Tue, 10 Nov 2026 01:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/topics/developers-practitioners/the-open-ai-native-stack-on-kubernetes-from-open-models-to-distributed-inference-and-agents/</guid><category>Developers &amp; Practitioners</category><media:content height="540" url="https://storage.googleapis.com/gweb-cloudblog-publish/images/kubena26-hero.max-600x600.jpg" width="540"></media:content><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>The open AI-native stack on Kubernetes: From open models to distributed inference and agents</title><description></description><image>https://storage.googleapis.com/gweb-cloudblog-publish/images/kubena26-hero.max-600x600.jpg</image><site_name>Google</site_name><url>https://cloud.google.com/blog/topics/developers-practitioners/the-open-ai-native-stack-on-kubernetes-from-open-models-to-distributed-inference-and-agents/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Abdelfettah Sghiouar</name><title>Cloud Developer Advocate</title><department>Google Cloud</department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Allan Naim</name><title>Director of Product Management GKE</title><department></department><company></company></author></item><item><title>Building an agent skill optimizer with AlphaEvolve, Harbor and Gemini 4 Argon</title><link>https://cloud.google.com/blog/topics/developers-practitioners/how-to-build-an-agent-skill-optimizer-with-alphaevolve-and-harbor/</link><description>&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Every day, more and more people are using coding agents like &lt;/span&gt;&lt;a href="https://antigravity.google/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Google Antigravity&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; and &lt;/span&gt;&lt;a href="https://opencode.ai/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;OpenCode&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; to complete software engineering tasks. But those tools need instructions to operate and their output is only as good as the inputs they get. By utilizing &lt;/span&gt;&lt;a href="https://agentskills.io" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;agent skills&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; that direct the agents through building, testing, and deploying code, developers have a powerful way to automate their workflows. &lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;But most skills are written manually, so it’s hard to know which instructions are effective without assessing their output.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;To measure the effectiveness of our skills, we ran OpenCode with &lt;/span&gt;&lt;a href="https://blog.google/innovation-and-ai/models-and-research/gemini-models/3-8-flash-and-3-8-flash-cyber/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Gemini 3.8 Flash&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; on tasks from &lt;/span&gt;&lt;a href="https://github.com/harbor-framework/terminal-bench-2" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Terminal-Bench 2.0&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, a public benchmark. The results were inconclusive: with our baseline skill, the agent passed about two-thirds of its attempts, but the outcome was about the same even without the skill. We needed better results.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;To achieve them, we turned to &lt;/span&gt;&lt;a href="https://cloud.google.com/blog/products/ai-machine-learning/alphaevolve-is-available-for-everyone"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;AlphaEvolve&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, which optimizes skills the way it optimizes code: Gemini proposes a new version, an evaluator scores it, and the search retains the higher-scoring version. And with &lt;/span&gt;&lt;a href="https://harborframework.com" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Harbor&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, that evaluator can be a real coding agent working on real tasks. We saw improvements of 5% with just a few iterations with alphaevolve&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;This is how you can build the same loop:&lt;/span&gt;&lt;/p&gt;
&lt;ol&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Treat the skill as the program:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; AlphaEvolve can rewrite your skill's text.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Let the agent judge:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Harbor runs the agent with the skill on real tasks.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Make the score trustworthy:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Compare each version to the baseline skill and run them both at the same time.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Check the winner on unseen tasks:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Save some tasks the search never sees for your final check.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Treat the skill as the program&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;AlphaEvolve is designed to advance the code between two markers in a seed program. Skills are SKILL.md files with names, descriptions, and markdown instructions, so we put their text between the markers and let Gemini rewrite them:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;def build_skill() -&amp;gt; tuple[str, str]:\r\n  &amp;quot;&amp;quot;&amp;quot;Return the evolvable (description, body) pair for the skill.&amp;quot;&amp;quot;&amp;quot;\r\n  # EVOLVE-BLOCK-START\r\n  description = (\r\n      &amp;quot;Solve engineering tasks in a Linux terminal: building and compiling &amp;quot;\r\n      &amp;quot;software, fixing broken repositories and configuration, ...&amp;quot;\r\n  )\r\n  body = r&amp;quot;&amp;quot;&amp;quot;# Terminal task solver\r\nYou are working in a Linux container. The task has a specific, checkable end\r\nstate, and an automated test suite will decide whether you achieved it. ...\r\n&amp;quot;&amp;quot;&amp;quot;\r\n  # EVOLVE-BLOCK-END\r\n  return description, body&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba93c050&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;AlphaEvolve operates in Google Cloud and doesn’t run your code. Its feedback is the scores and notes your evaluator sends back about what "better" means.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/2_-_Setup.max-1000x1000.png"
        
          alt="2 - Setup"&gt;
        
        &lt;/a&gt;
      
        &lt;figcaption class="article-image__caption "&gt;&lt;p data-block-key="j89p4"&gt;Image 1. AlphaEvolve proposes a skill, Harbor runs the agent with it next to the baseline skill, and the score and notes go back to AlphaEvolve.&lt;/p&gt;&lt;/figcaption&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Let the agent judge&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Harbor runs agents on benchmark tasks, with each trial in its own container. We ran it on &lt;/span&gt;&lt;a href="https://cloud.google.com/kubernetes-engine"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Google Kubernetes Engine&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; (GKE) with one pod per trial, and a single search ran thousands of agent trials.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;For each candidate, the evaluator returns a score for passes and failures, as well as  short notes on tasks that failed or timed-out. AlphaEvolve then sends those notes to Gemini for the next version.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Make the score trustworthy&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Agent scores can be noisy. In one of our early runs on the same benchmark, our model’s latency explained much of the variance in candidate scores. With each candidate running next to the baseline skill on the same tasks at the same time,  their scores are the difference.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Check the winner on unseen tasks&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The best skills read differently than the baseline, which is a careful checklist: read first, reproduce the failure, and work in small verified steps. AlphaEvolve's version is a protocol built around the time limit:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;## 2. Ship a Working Baseline to the Target Path Early\r\n(...)\r\n- Having a valid artifact on disk protects you from timeouts: if you refine or\r\n  golf code (`gpt2.c`, `eigen.py`, `my_warrior.red`), keep the last passing\r\n  version written to the target path at all times.\r\n\r\n\r\n(...)\r\n\r\n\r\n## 5. Hard Stop When Green\r\nThe moment `eval.py`, the test suite, or your acceptance check passes and the\r\nrequired files exist with exact formatting (...): **STOP IMMEDIATELY**. (...)&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba93df90&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;What we learned&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;AlphaEvolve focused on the most common failure: running out of time. About a third of the search's trials hit their time limit, as reported in the evaluator's notes, and AlphaEvolve's best skill is built around that limit.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;We also found that running the search is harder than building the loop. After about 10 hours, the search scored no new candidate; our latency check held back every evaluation while model responses were slow, and full agent logs filled the controller's disk. It’s important to plan for slow model windows and for log storage before a long search.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;In addition, skills paid off when they added what the agent lacks, as advised by the &lt;/span&gt;&lt;a href="https://agentskills.io/skill-creation/best-practices" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Agent Skills best practices&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;. Our baseline skill made no measurable difference, while the skill holding the answers added 33 points on the dev tasks.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Our hypothesis: today's models and coding agents are already tuned for coding work, so general advice has little left to add. If so, the room is in specifics, which a search can find: AlphaEvolve's best skill is built around the agent's most common failure, running out of time.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;What's next&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;We’re testing AlphaEvolve on enterprise agent harnesses — system prompt, policy instructions, and tools —  where the agent lacks business rules that a search can add. In one small study on customer service test cases for a fictional bank, AlphaEvolve found and fixed a wrong policy rule we had planted, and carried the fix over to cases it never saw.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Other small studies were mixed, and we’ll share our final results when we’ve completed the full study. &lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Get started&lt;/span&gt;&lt;/h3&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;AlphaEvolve on Google Cloud:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; the &lt;/span&gt;&lt;a href="https://github.com/Google-Cloud-AI/alphaevolve-on-googlecloud" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;GitHub repository&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Benchmark and harness:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; &lt;/span&gt;&lt;a href="https://github.com/harbor-framework/terminal-bench-2" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Terminal-Bench 2.0&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; and &lt;/span&gt;&lt;a href="https://harborframework.com" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Harbor&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Agent skills:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; the format at &lt;/span&gt;&lt;a href="https://agentskills.io" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;agentskills.io&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; and the &lt;/span&gt;&lt;a href="https://opencode.ai/docs/skills/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;opencode skills docs&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Thanks for reading! Tell us what your own tests show on LinkedIn or X.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;</description><pubDate>Thu, 05 Nov 2026 17:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/topics/developers-practitioners/how-to-build-an-agent-skill-optimizer-with-alphaevolve-and-harbor/</guid><category>Developers &amp; Practitioners</category><media:content height="540" url="https://storage.googleapis.com/gweb-cloudblog-publish/images/1_-_Hero_tsgSDxO.max-600x600.jpg" width="540"></media:content><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Building an agent skill optimizer with AlphaEvolve, Harbor and Gemini 4 Argon</title><description></description><image>https://storage.googleapis.com/gweb-cloudblog-publish/images/1_-_Hero_tsgSDxO.max-600x600.jpg</image><site_name>Google</site_name><url>https://cloud.google.com/blog/topics/developers-practitioners/how-to-build-an-agent-skill-optimizer-with-alphaevolve-and-harbor/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Ivan Nardini</name><title>AI Developer Relations Engineer</title><department></department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Anant Nawalgaria</name><title>Group Product Manager &amp; Sr. Staff Engineer</title><department></department><company></company></author></item><item><title>Introducing Compute Engine API multi-versioning: Predictable evolution without breaking production</title><link>https://cloud.google.com/blog/topics/developers-practitioners/introducing-compute-engine-api-multi-versioning-predictable-evolution-without-breaking-production/</link><description>&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Ready to modernize your infrastructure without risking production stability? API multi-versioning for Compute Engine, now in general availability, helps you do just that. With date-based API versions, Compute Engine can now regularly deliver modern API semantics, stronger resilience defaults, and improved developer experience while preserving the rock-solid stability of the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;/v1&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; API.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;For more than a decade, the Google Compute Engine &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;/v1&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; API has served as the foundational control plane for millions of virtual machines, Google Kubernetes Engine (GKE) clusters, and large-scale AI infrastructure. For platform teams and developers who build on Google Cloud, that single &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;/v1&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; endpoint delivered an extraordinary promise: write your integration once, and trust that it will keep running year after year.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Yet as cloud architectures have grown in scale and sophistication, infrastructure teams have faced a growing tension. On one hand, mission-critical production workloads require absolute API stability and strict backward compatibility. On the other hand, customers want faster access to safer security defaults, stronger regional resilience, machine-readable error models, and modern, declarative-friendly APIs, but introducing these improvements in place risks disrupting existing automation. The new versioning model helps avoid that disruption by giving you the flexibility to adopt modern semantics on your own schedule without touching your production workloads.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Date-based API versions for Compute Engine&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Historically, Compute Engine relied on &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;channel-based versioning (CBV)&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; with three long-lived channels: &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;/v1&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; (generally available), &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;/beta&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, and &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;/alpha&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;. Because &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;/v1&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; was a single, mutable production surface shared by every customer worldwide, any behavioral improvement–even changing a counter-intuitive default or modernizing an error format–carried the risk of breaking existing scripts or required multi-year migrations.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;With &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;interface-based versioning (IBV)&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;, aligned with Google &lt;/span&gt;&lt;a href="https://google.aip.dev/185" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;AIP-185&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, Compute Engine introduces immutable, date-based API snapshots alongside &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;/v1&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Stable versions (&lt;/strong&gt;&lt;code&gt;&lt;strong style="vertical-align: baseline;"&gt;YYYY-MM-DD&lt;/strong&gt;&lt;/code&gt;&lt;strong style="vertical-align: baseline;"&gt;)&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Released on a predictable &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;quarterly cadence&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; (starting with &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;2026-09-01&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;), each stable version represents a locked, point-in-time contract of all generally available Compute Engine features and behavioral defaults. Stable versions don't receive breaking changes after they're published. They have no end-of-support date today.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Preview versions (&lt;/strong&gt;&lt;code&gt;&lt;strong style="vertical-align: baseline;"&gt;YYYY-MM-DD-preview&lt;/strong&gt;&lt;/code&gt;&lt;strong style="vertical-align: baseline;"&gt;)&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Released on a rapid cadence (such as &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;2026-10-01-preview&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;), preview versions branch from a stable parent to give developers early access to upcoming features and API designs, carrying a predictable one-year lifespan for testing and feedback.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Uninterrupted &lt;/strong&gt;&lt;code&gt;&lt;strong style="vertical-align: baseline;"&gt;/v1&lt;/strong&gt;&lt;/code&gt;&lt;strong style="vertical-align: baseline;"&gt; support&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The existing &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;/v1&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; API remains fully supported and it continues to serve as the default if no date-based version is specified. Your existing applications, scripts, and integrations continue to work without modification.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Primary benefits for builders&lt;/span&gt;&lt;/h2&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;1. Stability and customer-controlled upgrades&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;In a traditional single-version API, changing a default behavior forces a trade-off between platform progress and customer stability. But date-based multi-versioning puts upgrade timing entirely in your hands:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Pin to a known contract&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Direct REST callers and SDK users can lock onto a specific version identifier (such as &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;2026-09-01&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;) knowing its request and response schemas are frozen.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Upgrade on your schedule&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: When a new quarterly stable version introduces updated defaults or modified resource schemas, your team can evaluate the release notes, test in staging by changing a single version string, and roll out to production at your own pace.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Consistent resource URIs&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Date-based versions continue to use the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;/compute/v1/&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; base URL path. You specify the target version via the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;X-Goog-Api-Version&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; HTTP header or the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;$apiVersion&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; query parameter. This means that resource &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;selfLink&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; URIs remain uniform across tools and services.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;# Request a specific date-based stable version using the HTTP header\r\ncurl -X GET \\\r\n  &amp;quot;https://compute.googleapis.com/compute/v1/projects/my-project/zones/us-central1-a/instances&amp;quot; \\\r\n  -H &amp;quot;Authorization: Bearer $(gcloud auth print-access-token)&amp;quot; \\\r\n  -H &amp;quot;X-Goog-Api-Version: 2026-09-01&amp;quot;&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9e6d50&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;2. Safer defaults, stronger resilience, and modern API standards&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;By isolating behavioral changes to new date-based versions, Compute Engine is delivering improvements that make cloud applications more resilient, more secure, and easier to automate:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Resilient global queries by default&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: In legacy &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;/v1&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, calling &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;AggregatedList&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; across all zones and regions defaults &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;returnPartialSuccess&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; to &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;false&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;. This means a transient issue in a single zone could cause a global list request to fail unless callers explicitly opted in. In new date-based versions, partial success is enabled by default so global fleet queries remain resilient during localized maintenance or disruptions.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Actionable, machine-readable errors (AIP-193)&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Date-based versions adopt standardized &lt;/span&gt;&lt;a href="https://google.aip.dev/193" rel="noopener" target="_blank"&gt;&lt;code style="text-decoration: underline; vertical-align: baseline;"&gt;google.rpc.ErrorInfo&lt;/code&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; payloads with structured &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;reason&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;domain&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, and contextual &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;metadata&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; (such as returning the list of &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;supported_versions&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; if an invalid version identifier is requested), eliminating brittle regex parsing of error strings.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Stronger security and least-privilege hygiene&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: New API versions allow Compute Engine to tighten overly broad legacy IAM permission checks and introduce safer configuration defaults without disrupting legacy automation running on &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;/v1&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Cleaner, declarative-friendly API design&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Multi-versioning provides a clean path to evolve complex, decade-old resource schemas toward modern Google API Improvement Proposals (AIPs).&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;3. Seamless adoption across client libraries, &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;gcloud&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, and Terraform&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Whether you call the REST API directly or manage infrastructure through higher-level tools, Compute Engine API Multi-Versioning is designed to work without friction:&lt;/span&gt;&lt;/p&gt;
&lt;div align="center"&gt;
&lt;div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;"&gt;
&lt;div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;"&gt;
&lt;div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;"&gt;
&lt;div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;"&gt;
&lt;div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;"&gt;&lt;table&gt;&lt;colgroup&gt;&lt;col/&gt;&lt;col/&gt;&lt;col/&gt;&lt;/colgroup&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Surface&lt;/strong&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;How multi-versioning works&lt;/strong&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Your responsibility&lt;/strong&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Client libraries (Python, Go, Java, Node.js, C#, Ruby, PHP)&lt;/strong&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Each SDK release automatically binds to a specific date-based API version while &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;preserving the &lt;/strong&gt;&lt;code&gt;&lt;strong style="vertical-align: baseline;"&gt;/v1&lt;/strong&gt;&lt;/code&gt;&lt;strong style="vertical-align: baseline;"&gt; package namespace&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; (&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;compute_v1&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;apiv1&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;). Enums are represented as strings for seamless forward and backward compatibility.&lt;/span&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Upgrade your client library dependency as usual–&lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;no import path rewrites required&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;. Use &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;-preview&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; SDK releases to test preview features.&lt;/span&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Google Cloud CLI (&lt;/strong&gt;&lt;strong style="vertical-align: baseline;"&gt;gcloud&lt;/strong&gt;&lt;strong style="vertical-align: baseline;"&gt;)&lt;/strong&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;code style="vertical-align: baseline;"&gt;gcloud compute&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; targets the bundled stable API version. The new &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;gcloud preview compute&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; command group targets the bundled preview API version.&lt;/span&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Keep &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;gcloud&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; updated. Existing &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;gcloud compute&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;beta&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, and &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;alpha&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; scripts continue to work as-is.&lt;/span&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Terraform (&lt;/strong&gt;&lt;code&gt;&lt;strong style="vertical-align: baseline;"&gt;google&lt;/strong&gt;&lt;/code&gt;&lt;strong style="vertical-align: baseline;"&gt; &amp;amp; &lt;/strong&gt;&lt;code&gt;&lt;strong style="vertical-align: baseline;"&gt;google-beta&lt;/strong&gt;&lt;/code&gt;&lt;strong style="vertical-align: baseline;"&gt;)&lt;/strong&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;google&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; provider targets stable date-based APIs, while &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;google-beta&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; targets preview APIs under the hood.&lt;/span&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Upgrade your Terraform provider version to adopt new features; no manual API version configuration needed in HCL scripts.&lt;/span&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Direct HTTP / REST&lt;/strong&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Select any active version via &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;-H "X-Goog-Api-Version: YYYY-MM-DD"&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; or &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;?$apiVersion=YYYY-MM-DD&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, and fetch version-specific &lt;/span&gt;&lt;a href="https://www.googleapis.com/discovery/v1/apis/compute/2026-09-01/rest" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Discovery Documents&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Pass the header or query parameter in your HTTP client to opt into a specific version (e.g., &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;2026-09-01&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;2026-10-01-preview&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;).&lt;/span&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Get started&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Compute Engine API multi-versioning is available today across all Google Cloud regions, with &lt;/span&gt;&lt;code&gt;&lt;strong style="vertical-align: baseline;"&gt;2026-09-01&lt;/strong&gt;&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; as the initial stable version and &lt;/span&gt;&lt;code&gt;&lt;strong style="vertical-align: baseline;"&gt;2026-10-01-preview&lt;/strong&gt;&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; available for preview testing.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Start exploring and adopting date-based versions in your workflows:&lt;/span&gt;&lt;/p&gt;
&lt;ol&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Explore the documentation&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;:&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;ul&gt;
&lt;li aria-level="2" style="list-style-type: circle; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://docs.cloud.google.com/compute/docs/api/how-tos/api-versioning-guide"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Compute Engine API Versioning Guide&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="2" style="list-style-type: circle; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://docs.cloud.google.com/compute/docs/api/how-tos/api-requests-responses"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Creating API Requests and Handling Responses&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="2" style="list-style-type: circle; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://docs.cloud.google.com/compute/docs/reference/rest/v1"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Compute Engine REST API Reference&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; (use the version selector to inspect &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;2026-09-01&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; and &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;2026-10-01-preview&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; schemas)&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Try &lt;/strong&gt;&lt;code&gt;&lt;strong style="vertical-align: baseline;"&gt;gcloud preview compute&lt;/strong&gt;&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;: Run &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;gcloud components update&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; and test upcoming preview capabilities using the new &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;gcloud preview compute&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; surface.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;&lt;/div&gt;</description><pubDate>Wed, 04 Nov 2026 16:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/topics/developers-practitioners/introducing-compute-engine-api-multi-versioning-predictable-evolution-without-breaking-production/</guid><category>Developers &amp; Practitioners</category><media:content height="540" url="https://storage.googleapis.com/gweb-cloudblog-publish/images/compute-engine-multi-version-hero.max-600x600.jpg" width="540"></media:content><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Introducing Compute Engine API multi-versioning: Predictable evolution without breaking production</title><description></description><image>https://storage.googleapis.com/gweb-cloudblog-publish/images/compute-engine-multi-version-hero.max-600x600.jpg</image><site_name>Google</site_name><url>https://cloud.google.com/blog/topics/developers-practitioners/introducing-compute-engine-api-multi-versioning-predictable-evolution-without-breaking-production/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Michael Murstein</name><title>Software Engineer</title><department></department><company></company></author></item><item><title>Five rules for evaluating agent skills before they bloat your context window</title><link>https://cloud.google.com/blog/topics/developers-practitioners/evaluate-agent-skills-before-they-bloat-your-context-window/</link><description>&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;You’re probably hearing more about SKILL.md files, and you’ll see that a lot of people test them the same way: paste a prompt into chat, watch it pass once on a laptop, and merge to main. But when you paste three pages of CLI docs into a markdown file without an evaluation loop, you are just as likely to confuse a frontier model as you are to help it.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;For the past weeks, we have been diligently measuring and optimizing our own &lt;/span&gt;&lt;a href="https://cloud.google.com/blog/topics/developers-practitioners/introducing-the-google-cloud-developer-plugin-for-ai-coding-agents"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Google Cloud Developer Plugin&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, with the most essential skills and the Developer Knowledge MCP server to ground responses in the documentation.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;We &lt;/span&gt;&lt;a href="https://x.com/kweinmeister/status/2099962972798153194" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;ran dozens of scenarios&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; with Gemini, Antigravity, and other agents. We used &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Claude Sonnet 5 on Gemini platform&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; inside isolated &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Harbor&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; pods on &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;GKE Autopilot&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;. And, we contributed Kubernetes and Gemini platform capabilities upstream to NVIDIA's open-source evaluator. &lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Here's the five-rule engineering playbook we use to verify a skill is worth its tokens.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;1. Gate trigger collisions and security risks before spinning up containers&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;When an evaluation run shows zero improvement between your plugin arm and your baseline arm (&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;Δ = 0&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;), the natural urge is to rewrite the body of your SKILL.md instructions. Hold off on that. Nine times out of ten, &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;the agent never loaded your skill in the first place&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Coding agents do not read full SKILL.md bodies at turn zero. The router only inspects the short name and description in your YAML frontmatter. Once a workspace grows past a dozen skills, two failure modes collide: under-triggering (your description describes internal implementation rather than developer intent) and sibling hijacking (a broad phrase like "helps with cloud deployments" in a neighboring skill intercepts the prompt first).&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;We recently open-sourced &lt;/span&gt;&lt;a href="https://google.github.io/skill-reach/" rel="noopener" target="_blank"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;skill-reach&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; so you can catch frontmatter collisions before spending on container sandboxes. It validates manifests, computes vocabulary overlap against nearest rival skills, and runs empirical routing probes across Claude Code, Google Antigravity, and OpenAI Codex:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;uv tool install skill-reach\r\n\r\n# 1. Validate YAML frontmatter &amp;amp; token budgets (zero API calls)\r\nreach lint ./skills\r\n\r\n# 2. Detect vocabulary collisions &amp;amp; nearest rival skills (e.g. 0.69 overlap)\r\nreach overlap ./skills\r\n\r\n# 3. Run cross-harness routing probes and auto-tune weak descriptions\r\nreach eval ./skills/cloud-run --runs 3\r\nreach optimize ./skills/cloud-run --apply&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9e6590&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Pair skill-reach with &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;NVIDIA's SkillSpector&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; (Tier 1 and Tier 2 of &lt;/span&gt;&lt;a href="https://github.com/NVIDIA/SkillEvaluator" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;SkillEvaluator&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;). As the Nemotron Labs team details in their &lt;/span&gt;&lt;a href="https://www.youtube.com/watch?v=4rI0zATFxYI" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Evaluating Agent Skills deep dive&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, standard code linters miss &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;agent-native risks&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; buried in markdown prose: prompt injection, silent data exfiltration, tool poisoning, and trigger abuse. &lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;2. Never trust an eval that runs on a developer laptop&lt;/span&gt;&lt;/h2&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/1_dRIBHtV.max-1000x1000.jpg"
        
          alt="1"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;If you run skill benchmarks directly on your workstation, your numbers are lying to you. A developer laptop is packed with ambient state that contaminates the baseline arm:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Cached Google Cloud credentials&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Exported project variables and warm bash history&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Global environment rules&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;An unassisted baseline agent can pass a cloud deployment test by inheriting a default project ID or token sitting in your local shell profile, masking the exact failure your skill was written to solve.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Host resource contention also warps agent trajectories. While hill climbing code-retrieval agents with AlphaEvolve, our internal testing showed that &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;running the exact same agent benchmark while a workstation was under CPU load dropped the score&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;. Slower file scans and subprocess timeouts cause the agent to abandon search paths prematurely.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;3. Measure paired &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;WITH&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; vs. &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;W/OUT&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; ablation (&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;Pass@3&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;), not vibes&lt;/span&gt;&lt;/h2&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/2_UtgDTjb.max-1000x1000.jpg"
        
          alt="2"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Running prompts a single time and screenshotting the green checkmarks is not a real evaluation. Multi-turn tool-using agents branch on every shell exit code and stdout snippet. In our retrieval study, &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;more than a third of benchmark tasks produced different scores across consecutive runs of the exact same code and prompt&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;A perfect score is equally meaningless without a paired baseline for comparison. Both Claude's plugin eval tool and NVIDIA's SkillEvaluator run &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;paired ablation across multiple repetitions&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Arm A runs the base model in a clean sandbox without the skill, while Arm B runs the identical prompt and sandbox with the skill mounted.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Comparing the score improvements across our sandbox trials exposed three clear patterns that single-shot testing hides. Without the plugin, the base model still stumbled into a working deployment on most runs, but only after burning minutes of wall-clock time guessing deprecated flags and retrying bad requests. Conversely, when a test case shows identical success with zero efficiency gain, the base model already owns that capability natively. Your skill is pure context tax — delete it.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;# Scaffold and execute a k=3 paired WITH vs W/OUT ablation locally\r\nclaude plugin eval init ./my-plugin\r\nclaude plugin eval ./my-plugin --runs 3&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba93e090&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;4. Separate efficiency and security from correctness so agents can't reward-hack&lt;/span&gt;&lt;/h2&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/3_sXliN0n.max-1000x1000.jpg"
        
          alt="3"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;When you connect an evaluation harness to an automated optimizer (like &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/gemini/enterprise/docs/alphaevolve/developer-guide/overview"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;AlphaEvolve&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; or &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;reach optimize&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;), it is a surprisingly common pattern for the &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;agent to cheat the grader&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;In our AlphaEvolve retrieval experiments, evolved candidates discovered &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;four distinct ways to game the benchmark without solving the task&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;. When the grading harness blocked specific expected substring checks in source code, an evolved candidate dynamically concatenated strings at runtime, traversed up the directory tree, and read the ground-truth answer file directly off disk. Another candidate spawned subprocesses to bypass AST checks.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The evolutionary loop favored those cheaters because the fitness formula blended correctness (0.95 weight) and token efficiency (0.05 weight) into a single score. Reading the answer file from disk takes &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;no LLM turns&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;, earning an efficiency bonus that gives cheaters a &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;survival advantage&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; over honest agents that spent tokens searching the repository.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Hardening your skill evaluator against reward-hacking takes three structural guardrails. First, configure Harbor so your test fixtures, grading scripts, and answer keys live inside a dedicated verifier container that mounts workspace artifacts only after the agent container exits. Second, keep SkillEvaluator's five dimensions (&lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Correctness&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Discoverability&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Efficiency&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Effectiveness&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;, and &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Security&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;) strictly orthogonal; never let token thrift compensate for a failed test or a security violation. Third, in &lt;/span&gt;&lt;a href="https://github.com/NVIDIA/SkillEvaluator/pull/140" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;NVIDIA/SkillEvaluator#140&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, we extended security checks to fail any trial that reads Kubernetes service account tokens or queries cloud metadata endpoints, while redacting live OAuth tokens from saved trajectories.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;5. Encode operational habits, not documentation dumps&lt;/span&gt;&lt;/h2&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/4_jsRzzuO.max-1000x1000.jpg"
        
          alt="4"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Looking across all of our trials, the difference between the skills that delivered a&lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt; score improvement in our sandbox benchmark&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; and the &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;community skills that failed&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; came down to what was inside the markdown.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Skills that copy-paste public API reference tables into skill files yield almost no lift. Frontier models already have public cloud docs in their pretraining weights; dumping thousands of words of static flags into the prompt just inflates time-to-first-token and buries actionable constraints.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The skills that increased our &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Efficiency score &lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;in our test suite taught compact &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;operational habits and verification reflexes&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; instead of encyclopedic reference text:&lt;/span&gt;&lt;/p&gt;
&lt;ol&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Verify CLI flags at the leaf subcommand before mutating state:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Rather than listing dozens of flags in markdown, the skill teaches a simple reflex: verify help output at the command line before executing a state-changing deployment.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Route through a lightweight catalog index on demand:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Rather than loading fourteen heavy cloud skills into context at once, a thin root router checks a curated remote index and pulls only the specific recipe needed for the task.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Enforce least-privilege auth defaults:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Explicit guardrails instruct agents to authenticate via Application Default Credentials rather than downloading static service-account JSON keys, while enforcing unauthenticated access restrictions alongside explicit invoker IAM bindings.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Because our evaluation stack runs in CI, upgrading models takes one flag change. We re-run the paired WITH vs. W/OUT matrix against the new model and delete every skill whose Delta drops to zero. Deleting a skill because the new model graduated past it is one of the cleanest high-fives in software engineering.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Get started&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Ready to equip your coding agents with production-grade skills? Try the &lt;/span&gt;&lt;a href="https://cloud.google.com/blog/topics/developers-practitioners/introducing-the-google-cloud-developer-plugin-for-ai-coding-agents"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Google Cloud Developer Plugin for AI Coding Agents&lt;/span&gt;&lt;/a&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;</description><pubDate>Tue, 03 Nov 2026 08:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/topics/developers-practitioners/evaluate-agent-skills-before-they-bloat-your-context-window/</guid><category>Developers &amp; Practitioners</category><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Five rules for evaluating agent skills before they bloat your context window</title><description></description><site_name>Google</site_name><url>https://cloud.google.com/blog/topics/developers-practitioners/evaluate-agent-skills-before-they-bloat-your-context-window/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Karl Weinmeister</name><title>Director, Developer Relations</title><department></department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Alan Blount</name><title>Product Manager</title><department></department><company></company></author></item><item><title>Google Cloud Developer plugin now available in Antigravity Marketplace</title><link>https://cloud.google.com/blog/topics/developers-practitioners/google-cloud-developer-plugin-now-available-in-antigravity-marketplace/</link><description>&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Building cloud solutions with autonomous AI agents should feel fast, safe, and reliable. Without cloud-specific context, coding agents often struggle. They might invent deprecated SDK syntax, guess invalid command flags, or attempt operations without the required Identity and Access Management (IAM) permissions. Even worse, an unguided agent can enable billable APIs or provision cloud resources without your explicit permission.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Starting today, you can install the &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;google-cloud-developer&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; plugin directly from the Antigravity Marketplace in the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;Build with Google&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; section. This plugin equips Antigravity with foundational skills, safety guardrails, and live documentation grounding for building on Google Cloud. If you've never installed a plugin using the marketplace before, you can refer to the &lt;/span&gt;&lt;a href="https://antigravity.google/docs/marketplace/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Antigravity documentation&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Configuration is straightforward. The bundled Developer Knowledge Model Context Protocol (MCP) server is pre-authorized by default. You don't need to configure API keys, exchange OAuth tokens, or edit any configuration files. You install the plugin, and your agent connects to official Google developer documentation immediately.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;In this post, we'll walk through what's included in the plugin, how to install it with one click, and how it grounds your agent in live Google Cloud documentation.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Why use the Google Cloud Developer plugin?&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Coding agents often rely on static training data. Cloud platforms update quickly, and models can produce code with outdated client libraries or invalid parameters.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;google-cloud-developer&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; plugin solves this issue. It bundles curated skills, intelligent routing rules, and the Developer Knowledge MCP server into a single portable package.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The plugin provides three key benefits:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Live documentation grounding&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The agent queries official Google Cloud documentation over streamable HTTP.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Safety and cost guardrails&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The agent enforces least-privilege IAM patterns and it requires your consent before it enables billable APIs or modifying cloud infrastructure.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Environment awareness&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The agent inspects your local environment, detects existing CLI configurations, and designs safe execution roadmaps before it runs commands.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;What's included in the plugin&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The plugin pairs core baseline skills and live documentation grounding with an intelligent skill routing architecture. Rather than limiting your agent to a fixed set of capabilities, it acts as an extensible gateway that discovers and pulls specialized skills from Google on demand.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Bundled agent skills&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The bundled skills provide essential day-one guardrails, live documentation grounding, and the dynamic discovery engine that unlocks the broader catalog:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://github.com/google/skills/tree/main/plugins/cloud/google-cloud-developer/skills/finding-google-skills" rel="noopener" target="_blank"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;&lt;code&gt;finding-google-skills&lt;/code&gt;&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;: Powers flexible skill routing by discovering and loading skills on demand from the &lt;/span&gt;&lt;a href="https://g.dev/ai/skills" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;&lt;code&gt;google/skills&lt;/code&gt; repository on GitHub&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;. Rather than requiring you to research, install, and manage individual skills, it dynamically retrieves the exact instructions your agent needs for specialized tasks in real time.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;code&gt;&lt;a href="https://github.com/google/skills/tree/main/plugins/cloud/google-cloud-developer/skills/retrieving-developer-knowledge" rel="noopener" target="_blank"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;retrieving-developer-knowledge&lt;/strong&gt;&lt;/a&gt;&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;: Provides answers and code samples that are grounded in official Google documentation across Google Cloud, Firebase, Android, Web, and Gemini.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://github.com/google/skills/tree/main/plugins/cloud/google-cloud-developer/skills/gcloud" rel="noopener" target="_blank"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;&lt;code&gt;gcloud&lt;/code&gt;&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;: Enforces execution guardrails, validates syntax, and applies safety checks to Google Cloud CLI (&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;gcloud&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;) commands. It blocks accidental destructive operations and it keeps you in control.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;code&gt;&lt;a href="https://github.com/google/skills/tree/main/plugins/cloud/google-cloud-developer/skills/google-cloud-recipe-auth" rel="noopener" target="_blank"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;google-cloud-recipe-auth&lt;/strong&gt;&lt;/a&gt;&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;: Steers your agent toward secure authentication patterns. It enforces service identities and Application Default Credentials (ADC) while it blocks risky practices like downloading service account JSON keys.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;code&gt;&lt;a href="https://github.com/google/skills/tree/main/plugins/cloud/google-cloud-developer/skills/google-cloud-recipe-onboarding" rel="noopener" target="_blank"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;google-cloud-recipe-onboarding&lt;/strong&gt;&lt;/a&gt;&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;: Guides initial project setup, account onboarding, and billing verification. It audits project states before it attempts to activate services.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Pre-authorized Developer Knowledge MCP server&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The plugin connects your agent to the &lt;/span&gt;&lt;a href="https://developers.google.com/knowledge/mcp" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Developer Knowledge MCP server&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;. Typically, adding an MCP server requires manual API key handling. With the Antigravity Marketplace integration in the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;Build with Google&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; catalog, this MCP server is pre-authorized: Antigravity manages the authentication transparently. Your agent connects to Google's documentation server immediately after installation.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Smart routing rules: on-demand capability expansion&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The plugin includes &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;google-cloud-discovery.md&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, an active routing rule that works together with the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;finding-google-skills&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; bundled skill. When your prompt touches specialized topics, the routing rule directs the agent to fetch the relevant skills from the public catalog as needed.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;This dynamic routing keeps your agent fast and token-efficient. Instead of preloading dozens of niche skills that clutter the context window, your agent pulls in deep, specialized guidance only when your task requires it.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Install the plugin with one click&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;You can install the plugin through the Antigravity Marketplace UI or through your terminal.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/original_images/google-cloud-developer-plugin-now-available-in-antigravity-marketplace-agy.gif"
        
          alt="google-cloud-developer-plugin-now-available-in-antigravity-marketplace-agy-install"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;If you prefer the command line, install the plugin with a single command:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;agy plugin install google-cloud-developer&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9e7050&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The CLI registers the skills and connects the Developer Knowledge MCP server immediately. You can also explore the plugin and skills directly on &lt;/span&gt;&lt;a href="https://g.dev/ai/skills" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;GitHub&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;The plugin in action: Querying live documentation&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;After the plugin is installed, your agent automatically grounds its answers in current documentation. You don't need to prompt it with special syntax. Consider a practical query:&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;"Based on the documentation what is the main difference between GKE and Cloud Run? When should I use one over the other?"&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Without the plugin, an agent might guess or provide outdated information. With the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;google-cloud-developer&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; plugin active, the agent queries the pre-authorized Developer Knowledge MCP server:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/original_images/google-cloud-developer-plugin-now-available-in-antigravity-marketplace-mcp.gif"
        
          alt="google-cloud-developer-plugin-now-available-in-antigravity-marketplace-mcp-usage"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The agent produces a clean, verified answer based on official documentation.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Connect with me&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;I'd love to hear how you use the plugin in your daily workflow. Share your feedback and questions with me on &lt;/span&gt;&lt;a href="https://www.linkedin.com/in/remigiusz-samborski/" rel="noopener" style="font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Open Sans', 'Helvetica Neue', sans-serif;" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;LinkedIn&lt;/span&gt;&lt;/a&gt;, &lt;a href="https://x.com/RemikSamborski" rel="noopener" style="font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Open Sans', 'Helvetica Neue', sans-serif;" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;X&lt;/span&gt;&lt;/a&gt;, or &lt;a href="https://bsky.app/profile/rsamborski.bsky.social" rel="noopener" style="font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Open Sans', 'Helvetica Neue', sans-serif;" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Bluesky&lt;/span&gt;&lt;/a&gt;.&lt;/p&gt;&lt;/div&gt;</description><pubDate>Mon, 02 Nov 2026 18:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/topics/developers-practitioners/google-cloud-developer-plugin-now-available-in-antigravity-marketplace/</guid><category>Developers &amp; Practitioners</category><media:content height="540" url="https://storage.googleapis.com/gweb-cloudblog-publish/images/google-cloud-developer-plugin-now-available-.max-600x600.jpg" width="540"></media:content><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Google Cloud Developer plugin now available in Antigravity Marketplace</title><description></description><image>https://storage.googleapis.com/gweb-cloudblog-publish/images/google-cloud-developer-plugin-now-available-.max-600x600.jpg</image><site_name>Google</site_name><url>https://cloud.google.com/blog/topics/developers-practitioners/google-cloud-developer-plugin-now-available-in-antigravity-marketplace/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Remigiusz Samborski</name><title>Lead Developer Relations Engineer</title><department></department><company></company></author></item><item><title>When the Kubernetes API went silent: A three-layer networking mystery</title><link>https://cloud.google.com/blog/products/networking/the-three-layer-gke-network-mystery/</link><description>&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Every support engineer has a case that stays with them. The one that comes up at team lunches. The one where, even after it's resolved, you find yourself staring at a packet capture at midnight just to make sure you &lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;really&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; understand what happened. This is that case.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;It started in January 2026 with a support ticket that read deceptively simply: pods are intermittently failing to reach the Kubernetes API server. The error was a familiar-looking timeout. The customer had already ruled out their own operators. What none of us knew yet was that we were looking at three separate bugs — in three different layers of the stack — all conspiring to produce the same symptom at the same time.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;By the time it was over, around 800 developers had been blocked from deploying to production for weeks on end, and GKE clusters across europe-west had ground to a halt. In a race to fix the problem, we involved engineers across Google Kubernetes Engine (GKE), Private Service Connect, networking, hostnet, kernel, and the Linux upstream community, who together tracked down not one but three distinct root causes. And the final fix landed not as a config change or a feature flag, but as a patch to the Linux kernel itself.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;This is the story of how we got there.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;Part one: The alarm bell that wouldn't stop&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The customer ran a large multi-cluster GKE environment in Google Cloud's europe-west region. Their clusters used Private Service Connect — Google Cloud's mechanism for exposing the GKE control plane privately — and had a rich ecosystem of system-level workloads: Twistlock security scanners, External Secrets Operator for secrets management, Cilium for network policy, as well as several internal Kubernetes operators.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The first errors appeared in one of its user acceptance testing (UAT) clusters in europe-west, and a few days later in a second cluster in the same region. They looked like this:&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Application logs — pods across multiple namespaces&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;quot;TimeoutError: Connect call failed (&amp;#x27;1.2.3.4&amp;#x27;, 443)\r\ndial tcp 1.2.3.4:443: i/o timeout\r\n\r\n# 1.2.3.4 is the cluster&amp;#x27;s internal VIP for the kube-apiserver,\r\n# routed through Private Service Connect&amp;quot;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba98ee90&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Initially the errors were intermittent — spikes in the logs, not a hard outage. The customer suspected their own code. Their team had recently deployed a custom operator built with the &lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;kopf&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; framework, and it was the first thing they looked at. But the logs showed the errors had started eight days before the operator was ever deployed. When they shut it down completely, the timeouts kept coming.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-pull_quote"&gt;&lt;div class="uni-pull-quote h-c-page"&gt;
  &lt;section class="h-c-grid"&gt;
    &lt;div class="uni-pull-quote__wrapper h-c-grid__col h-c-grid__col--8 h-c-grid__col-m--6 h-c-grid__col-l--6
      h-c-grid__col--offset-2 h-c-grid__col-m--offset-3 h-c-grid__col-l--offset-3"&gt;
      &lt;div class="uni-pull-quote__inner-wrapper h-c-copy h-c-copy"&gt;
        &lt;q class="uni-pull-quote__text"&gt;The issue persists even after completely scaling down the External Secrets Operator. The errors are appearing from customer workloads, and system-level pods we haven&amp;#x27;t touched.&lt;/q&gt;

        
      &lt;/div&gt;
    &lt;/div&gt;
  &lt;/section&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h4&gt;&lt;span style="vertical-align: baseline;"&gt;The growing list of suspects&lt;/span&gt;&lt;/h4&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;When a case like this arrives, the first instinct is to ask: what changed? The customer's timeline gave us several candidates, and we worked through them methodically.&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;The custom operator (kopf):&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Ruled out immediately — errors predated deployment by a week and continued after shutdown.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;External Secrets Operator:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Removed entirely on customer request. Timeouts continued in customer workloads that relied on Kubernetes control plane (KCP) connectivity, and cluster-internal system pods.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Security agents:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Uninstalled. Same result. We were now watching &lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;system-level Kubernetes components&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; fail to reach their own control plane.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;conntrack table exhaustion:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; One theory was that too many half-open connections were filling the OS connection tracking table on the control plane nodes. We checked. The conntrack table was at roughly 50% capacity — of a default maximum of 1.3M entries. Not even close to exhaustion.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Control plane overload:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; We scaled up the KCP VM size. kube-apiserver latencies looked healthy. No improvement on the timeout side.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;What made this case genuinely strange was the scope. We confirmed the timeouts were happening across five different clusters, affecting DaemonSets, Deployments, and system pods alike. External clients — developer laptops, GitHub Actions runners — were also hitting the same wall. This was not a broken workload. This was something wrong with the path between &lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;everything&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; and the API server.&lt;/span&gt;&lt;/p&gt;
&lt;h4&gt;&lt;span style="vertical-align: baseline;"&gt;The timeline so far&lt;/span&gt;&lt;/h4&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Day 1:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; First I/O timeout errors appear (One of the UAT clusters in europe-west3. Intermittent, easy to dismiss.)&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Day 10:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Errors become persistent (Frequency increases significantly. The issue is no longer dismissible.)&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Day 15:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Case escalated — P0 bug filed (Multiple clusters confirmed affected. ~800 developers blocked. Production releases are frozen. Formal escalation begins.)&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Day 20:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; All obvious suspects ruled out (Custom operator, security agents, conntrack exhaustion, control plane resizing — all cleared.)&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Day 22:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Packet captures collected (Captures taken simultaneously at the control-plane node NIC and from inside the kube-api container. The investigation enters its most critical phase.)&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;With the obvious suspects cleared, and the scope undeniably wide, we did what you do when the application layer has nothing left to say: We went to the packets.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Part two: Tracing the ghost in the wire&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;To understand what the packet captures revealed, you first need to understand the architecture that a GKE pod traverses when it makes an API call. In a Private Service Connect-based GKE cluster, the path is deceptively long for something that looks like a local connection.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/updated_architecture_official_icon.max-1000x1000.png"
        
          alt="image1"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The key component here is Private Service Connect. When a pod dials &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;1.2.3.4:443&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;, it is not actually going directly to the control plane; that IP is a virtual endpoint. The connection is intercepted by Google's networking fabric, NAT'd through a Private Service Connect endpoint, passed through an internal passthrough Network Load Balancer, and ultimately delivered to the KCP VM, where Envoy acts as a local proxy in front of the actual kube-apiserver process.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Private Service Connect adds a proxy protocol header to each new connection, so the kube-apiserver can see the real client IP. &lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;This will turn out to be important later.&lt;/span&gt;&lt;/p&gt;
&lt;h4&gt;&lt;span style="vertical-align: baseline;"&gt;What the packets revealed&lt;/span&gt;&lt;/h4&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;When we captured simultaneously at the KCP VM's NIC (the PIGv interface, Google's programmable virtual NIC) and from inside the kube-apiserver container, we saw something striking.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;SYN packets from the pods &lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;were&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; arriving at the KCP VM's NIC. They were getting that far. But instead of receiving a SYN-ACK in response — the expected second step of a TCP handshake — the client was getting back an out-of-place ACK. Not an RST. Not silence. An ACK that logically belonged to a different, earlier connection.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Wireshark, the network protocol analyzer, was flagging these streams with a label that initially seemed innocuous: "TCP Port numbers reused."&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Wireshark — packet stream showing the failed handshake&lt;/strong&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;Time      Source        Port  →  Destination   Port  Info\r\n09:13:41  10.0.13.77    53426 →  10.117.0.2    443   [SYN] Seq=0 Win=42600\r\n09:13:41  10.117.0.2    443   →  10.0.13.77    53426 [SYN, ACK] Seq=0 Ack=1 Win=64768\r\n   ...    (data exchange, connection completes)\r\n09:13:42  10.117.0.2    443   →  10.0.13.77    53426 [TCP Retransmission]\r\n09:13:42  10.117.0.2    443   →  10.0.13.77    53426 [TCP Retransmission]\r\n   ...    (many retransmissions of the same ACK)\r\n\r\n# 2 minutes later, same port 53426 is reused for a new connection\r\n09:13:44  10.0.13.77    53426 →  10.117.0.2    443   [SYN] Seq=1498 ← NEW connection\r\n           (expected: SYN-ACK)\r\n           (received: ACK Seq=1498 Ack=1025 — belongs to PREVIOUS stream)\r\n09:13:44  10.0.13.77    53426 →  10.117.0.2    443   [SYN] Retransmit...\r\n09:13:44  10.0.13.77    53426 →  10.117.0.2    443   [SYN] Retransmit...\r\n                                                    → eventually: i/o timeout&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba98ecd0&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The pattern was clear. Something was causing the KCP VM to respond to a fresh SYN with an ACK that belonged to a &lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;different connection that had ended minutes ago&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;. The old connection had used the same source port (53426 in the example above), and the KCP VM had never fully let go of it.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;"The KCP server sends FIN,ACK; client responds with FIN,ACK and then ACK. But KCP receives ACK before FIN,ACK. KCP server never ACKs the FIN,ACK — continuous retransmission of ACK. RST does not clear state."&lt;/span&gt;&lt;/p&gt;
&lt;h4&gt;&lt;strong style="vertical-align: baseline;"&gt;The "Invalid ACK" spike&lt;/strong&gt;&lt;/h4&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;While one thread of the investigation was analyzing packet captures, a.k.a., pcaps, another team was digging into internal metrics for the Private Service Connect infrastructure — specifically the user space packet switch (USPS), Google's software switching layer that handles NAT and proxy protocol header injection for Private Service Connect connections.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;They found something alarming in the metrics for Andromeda, Google's networking fabric. A counter called "Proxy protocol drop: invalid ACK packet" had been flat for months — and then, in February 2026, it spiked sharply upward to several drops per second across the affected europe-west regions.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;That date matched a USPS version rollout. This version included a change which was a fix for a &lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;different&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; bug but something about it had changed the behavior of how Private Service Connect handled a specific class of packet.&lt;/span&gt;&lt;/p&gt;
&lt;h4&gt;&lt;strong style="vertical-align: baseline;"&gt;Challenge ACKs: The RFC 5961 mechanism&lt;/strong&gt;&lt;/h4&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;To understand the next piece of the puzzle, we need to talk about a TCP security mechanism called the challenge ACK, defined in RFC 5961.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Imagine a TCP connection has closed — gone through the FIN/ACK dance and terminated. The server-side (here, the KCP VM) may still briefly remember the connection's sequence numbers in its TCP state machine. Now imagine that shortly afterward, a new client connection arrives on the same source port and destination port (the same 5-tuple). From the kernel's perspective, this &lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;could&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; be a stale packet from the old connection, or it could be an attacker trying to inject data into what the kernel thinks is still an open session.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;RFC 5961 says: when you receive a SYN that matches an existing or recently-closed connection's remembered state, don't accept it blindly. Send a challenge ACK — an acknowledgment that carries the &lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;last known sequence number&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; of the old connection. If the new client is legitimate, it will see this unexpected ACK, recognize the connection is stale, and send a RST to clear the state. The server then accepts the next SYN normally.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;The challenge ACK flow — how it's supposed to work&lt;/strong&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;# Step 1: New client sends SYN on a reused port\r\nClient  →  Server:  SYN  (seq=X, same 5-tuple as an old connection)\r\n\r\n# Step 2: Server is in FIN_WAIT_2, sends challenge ACK\r\nServer  →  Client:  ACK  (ack=old_seq — &amp;quot;is this really you?&amp;quot;)\r\n\r\n# Step 3: Client recognizes the stale state, resets it\r\nClient  →  Server:  RST\r\n\r\n# Step 4: State is cleared. Server accepts the next SYN normally.\r\nClient  →  Server:  SYN  (new connection)\r\nServer  →  Client:  SYN-ACK  ← handshake succeeds&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba98cb10&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;This mechanism works perfectly — &lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;when the challenge ACK is allowed through&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;. But that was exactly what the new USPS change was blocking.&lt;/span&gt;&lt;/p&gt;
&lt;h4&gt;&lt;span style="vertical-align: baseline;"&gt;The culprit: A well-intentioned fix&lt;/span&gt;&lt;/h4&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The change in &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;USPS&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; had been designed to fix a legitimate security issue in Private Service Connect's proxy protocol handling. Part of its effect was to drop packets it classified as "invalid ACKs" during the connection handshake phase — specifically, ACKs that arrived before the initial SYN-ACK had been sent, which in normal circumstances would be anomalous.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Unfortunately, challenge ACKs look exactly like this: They arrive from the server before the new handshake has completed, in response to a SYN. From the USPS's new perspective, they looked like invalid packets. They were dropped silently.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Why this broke GKE specifically: Private Service Connect uses NAT to share a single public endpoint across many client pods. This means many different pods — running on different nodes, from different namespaces — can all end up using the same source port after NAT, creating a high rate of 5-tuple collisions. The challenge ACK mechanism was being triggered far more often than it would be in a normal direct-connection scenario. The USPS fix hadn't &lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;introduced&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; challenge ACKs — they had always been there. It had introduced the &lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;dropping&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; of them, which broke a flow that GKE's TCP stack had silently depended on to clear stale state.&lt;/span&gt;&lt;/p&gt;
&lt;h4&gt;&lt;span style="vertical-align: baseline;"&gt;Mitigation: Rolling back the change&lt;/span&gt;&lt;/h4&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;In February, we rolled back the &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;USPS change&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;, restoring the old behavior where challenge ACKs were passed through instead of dropped. The rate of "Proxy protocol drop: invalid ACK packet" dropped back toward baseline almost immediately.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;By early March , 2026, the customer confirmed that all affected clusters had recovered. No more timeout errors.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;But the rollback wasn't a fix. It was a band-aid. It restored the behavior the system had &lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;accidentally depended on&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;. The deeper question — why was the KCP VM in FIN_WAIT_2 for over two minutes in the first place? — was still completely unanswered.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;That question led to Part 3. And it's where things got genuinely strange.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;Part three: Three bugs, one crime scene&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The mitigation bought us time, but it left a question hanging in the air: Why did the KCP VM keep connections in &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;FIN_WAIT_2&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; for so long? A well-behaved TCP stack processes a FIN, acknowledges it, and closes the connection. The KCP VM was failing to do this, leaving half-open connections that persisted for more than two minutes — long enough for the port to be reused by Private Service Connect's NAT layer, long enough to cause the challenge ACK scenario we'd uncovered.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;To answer this, we needed to understand not just one bug, but three distinct failure modes that had intersected at the worst possible moment.&lt;/span&gt;&lt;/p&gt;
&lt;h4&gt;&lt;span style="vertical-align: baseline;"&gt;Bug #1: Packet reordering at the PIGv&lt;/span&gt;&lt;/h4&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The first piece was discovered through careful analysis of simultaneous captures at two points: the KCP VM's host NIC (inside the physical host running the VM) versus the PIGv — the virtual NIC as seen by the VM.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;At the host NIC level, packets arrived in perfect order. But by the time they crossed through USPS and appeared at the PIGv, they had been subtly reordered.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;This happens because of how Private Service Connect's &lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;miss-thread processing&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; works. When Private Service Connect intercepts a new flow it hasn't seen before (a "miss"), it processes that packet through a slower path that adds the proxy protocol header and sets up NAT state. Subsequent packets in the same flow take a fast path. The consequence: a FIN or RST packet belonging to a flow in mid-close might arrive slightly out of order relative to the ACKs that followed it.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;What the KCP VM actually received (reordered sequence)&lt;/strong&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;quot;# What should have happened:\r\nClient  →  Server: FIN, ACK  (client initiates close)\r\nServer  →  Client: ACK       (server acknowledges client FIN)\r\nServer  →  Client: FIN, ACK  (server initiates its own close)\r\nClient  →  Server: ACK       (client acknowledges server FIN)\r\n                             → Connection fully closed. Both sides free.\r\n\r\n# What the KCP VM actually received (reordered by PIGv/USPS):\r\nClient  →  Server: ACK        (arrives first — out of order)\r\nClient  →  Server: FIN, ACK   (arrives second, but kernel sees ACK first)\r\nServer  →  Client: FIN, ACK   (server sends close, enters FIN_WAIT_1)\r\nClient  →  Server: FIN, ACK   (old, now-reordered FIN from client)\r\n                             → Server should ACK this. But it doesn&amp;#x27;t...&amp;quot;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba98ef50&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The reordering put the client's FIN after the server had already sent its own FIN. The server entered FIN_WAIT_1, then FIN_WAIT_2 after seeing the client's ACK of the server's FIN. But then the client's &lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;own&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; FIN arrived — and the server's kernel ignored it.&lt;/span&gt;&lt;/p&gt;
&lt;h4&gt;&lt;span style="vertical-align: baseline;"&gt;Bug #2: Envoy's half-close and the frozen timer&lt;/span&gt;&lt;/h4&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Why couldn't the operating system clean up on its own? There is a Linux kernel parameter called &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;tcp_fin_timeout&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; that governs exactly this: how long a socket in &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;FIN_WAIT_2&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; waits before being forcibly closed. On the KCP VM, this was set to 60 seconds.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;But &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;tcp_fin_timeout&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; only applies to &lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;orphaned&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; sockets — sockets that no process is currently holding open. Envoy, the proxy that sits in front of kube-apiserver on the KCP VM, was performing a half-close: it called &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;shutdown(SHUT_WR)&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;, which closes the write side of the connection but keeps the socket open so it can still receive data. This is perfectly valid TCP behavior and is how graceful half-duplex termination is supposed to work.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The side effect: As long as Envoy held the socket with &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;SHUT_WR&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;, the kernel did not consider it orphaned. &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;tcp_fin_timeout&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; never started. The socket sat in &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;FIN_WAIT_2&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; indefinitely, waiting for a client FIN it would never be able to process correctly.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Kernel socket state — captured via netstat just before a new connection&lt;/strong&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;Proto  Local Address         Foreign Address       State\r\ntcp    10.128.0.5:443        10.0.13.77:53426      FIN_WAIT_2\r\n# This socket has been in FIN_WAIT_2 for &amp;gt; 120 seconds.\r\n# tcp_fin_timeout=60 has not fired because the socket is not orphaned.\r\n# Envoy holds it open with SHUT_WR.\r\n\r\n# 2 minutes later, Private Service Connect NAT reuses port 53426 for a new pod connection.\r\n# The kernel sees the new SYN on this 5-tuple and sends a challenge ACK.\r\n# USPS drops the challenge ACK. → i/o timeout.&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba98d8d0&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h4&gt;&lt;span style="vertical-align: baseline;"&gt;Bug #3: The Linux kernel's silent failure&lt;/span&gt;&lt;/h4&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;But there was still a missing piece. Even with the reordered FIN, a healthy kernel should still process a received FIN in &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;FIN_WAIT_2&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; and move to &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;TIME_WAIT&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;. The socket should eventually close. Why wasn't it?&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;This led us deep into the Linux TCP state-machine implementation. When the reordered client FIN arrived at the KCP VM, the kernel's &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;tcp_ack()&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; function was called to process it. And here, in kernel version 6.1 (which underpinned the &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/container-optimized-os/docs"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Container-Optimized OS&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; (COS) running on the KCP VMs), there was a bug.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;RFC 5961 added stricter validation for acceptable ACK numbers — specifically, an ACK could be considered valid even if it was slightly "old" (within a sequence number window), to handle network reordering. But in kernel 6.1, there was a code path where &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;tcp_ack()&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; returned zero — indicating "this packet was not fully processed" — when it encountered an ACK that was old but within the RFC 5961 acceptable window. The caller of &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;tcp_ack()&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; in the FIN processing path checked this return value, saw zero, and did… nothing. The FIN was received and silently discarded.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;The Linux kernel commit that fixed it (upstream)&lt;/strong&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;commit 795a7dfbc3d9\r\nAuthor: Eric Dumazet &amp;lt;edumazet@google.com&amp;gt;\r\nnet: tcp: accept old ack during closing\r\n\r\ntcp_ack() returns 0 when ack is old but acceptable wrt rfc5961.\r\nIn FIN_WAIT_2, this caused tcp_fin() to never be called for a\r\nvalid received FIN, leaving the connection stuck indefinitely.\r\n\r\nFixed: properly handle old-but-acceptable ACKs during connection close.\r\nIntroduced in 6.1, fixed in 6.9.7. LTS backports: 6.6.y and 6.1.y.&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba98d4d0&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The bug had existed since kernel 6.1 and was already fixed in kernel 6.9.7 — but GKE's KCP VMs were running COS builds based on 6.1 and 6.6. The fix needed to be backported and shipped as a COS image update.&lt;/span&gt;&lt;/p&gt;
&lt;h4&gt;&lt;span style="vertical-align: baseline;"&gt;How the three bugs combined&lt;/span&gt;&lt;/h4&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Individually, none of these three issues would have caused an outage of this magnitude. It took all three acting together to create the failure:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Bug #1 alone:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Packet reordering is mildly disruptive, but the kernel would normally still eventually process the FIN and clean up, just slower than ideal.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Bug #2 alone: &lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;Envoy's half-close is perfectly correct behavior. It only becomes a problem if the FIN is never properly processed.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Bug #3 alone:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; The kernel's failure to process old-but-acceptable ACKs would cause some connections to linger, but the impact would be limited without the port-reuse pressure from Private Service Connect NAT.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;All three together:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Reordering causes the FIN to arrive out of sequence; the kernel bug prevents it from being processed; Envoy's SHUT_WR prevents the timeout from firing; the socket stays in FIN_WAIT_2 indefinitely. Private Service Connect NAT reuses the port. challenge ACK is sent. The USPS fix drops it. Connection fails.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h4&gt;&lt;span style="vertical-align: baseline;"&gt;The resolution: A three-pronged fix&lt;/span&gt;&lt;/h4&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Resolving this required action at three different levels, coordinated across multiple teams over several weeks:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Immediate mitigation (Feb, 2026):  &lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;Rollback of change in USPS. Restored the previous behavior where challenge ACKs were forwarded rather than dropped. Customer clusters recovered by early March.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Correctness fix:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; A replacement change to handle the challenge ACK scenario properly in the Private Service Connect proxy protocol layer, rather than simply restoring the old drop behavior. A PacketDrill test was added to the test suite to prevent regression.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Root cause fix — Linux kernel patch:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Upstream patch ("net: tcp: accept old ack during closing"), authored by Eric Dumazet. Merged into Linux 6.9.7. Backported to 6.6.y and 6.1.y LTS branches. — shipping in GKE 1.33.5 and later. The bug is fully resolved at the OS level for all GKE versions running these COS images.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;What made this case different&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Most networking cases resolve at one layer. You find the misconfigured firewall rule, the exhausted connection pool, the miscalculated MTU. You fix it and move on. This one was different because the symptom — &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;i/o timeout&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; on a connection that never completed — was the same whether you were looking at the Private Service Connect layer, the kernel's TCP state machine, or Envoy's connection management. Every layer had a plausible explanation, and every explanation was incomplete.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;What finally cracked it was the discipline of simultaneous packet capture at multiple points in the path. Without captures at the host NIC, the PIGv, and inside the container at the same time, we would never have seen the reordering. Without correlating those pcaps to the USPS metric spike on February 18, we would never have found the dropped challenge ACKs. And without reproducing the issue in a lab environment, we would never have found the kernel bug that made the FIN_WAIT_2 state permanent.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The other thing that made this work was the team. Over the course of two weeks, engineers from seven or eight different specialties contributed: GKE networking, Private Service Connect infrastructure, Hostnet, Andromeda, KCP, and ultimately the kernel community. No single person could have held all of this in their head. The case notes, the shared bug tracker, the meticulously collected pcaps — they were the connective tissue that let a distributed team work as one.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;In the end, the customer got back to production. The kernel got a better bug fix. And the rest of us got a very good story about why packet reordering, half-closed sockets, and a well-intentioned security patch should never be allowed in the same room at the same time.&lt;/span&gt;&lt;/p&gt;
&lt;hr/&gt;
&lt;p&gt;&lt;sub&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;This blog is based on real case data from an anonymized customer support engagement and internal bugs. The Linux kernel fix referenced is commit &lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;795a7dfbc3d9&lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;, authored by Eric Dumazet. Special thanks to the engineers across GKE, Private Service Connect, Hostnet, Andromeda, COS, and the Linux networking community whose work made this resolution possible.&lt;/span&gt;&lt;/sub&gt;&lt;/p&gt;&lt;/div&gt;</description><pubDate>Fri, 30 Oct 2026 16:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/networking/the-three-layer-gke-network-mystery/</guid><category>Containers &amp; Kubernetes</category><category>GKE</category><category>DevOps &amp; SRE</category><category>Management Tools</category><category>Developers &amp; Practitioners</category><category>Networking</category><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>When the Kubernetes API went silent: A three-layer networking mystery</title><description></description><site_name>Google</site_name><url>https://cloud.google.com/blog/products/networking/the-three-layer-gke-network-mystery/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Akash Thakur</name><title>Sr. Technical Solutions Engineer</title><department></department><company></company></author></item><item><title>Behind the scenes, two months later: How we ship Google Agent Skills like products</title><link>https://cloud.google.com/blog/topics/developers-practitioners/behind-the-scenes-two-months-later-how-we-ship-google-agent-skills-like-products/</link><description>&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;In my previous post, I took you &lt;/span&gt;&lt;a href="https://cloud.google.com/blog/topics/developers-practitioners/behind-the-scenes-how-we-build-test-and-scale-google-agent-skills"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;behind the scenes of Google Agent Skills&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, covering how the project started, how we maintain quality at scale, and how we handle governance. In this post, I'll share what we've built since then: the automated quality gates that every change must pass, the AI review agent that inspects skill quality, and the lessons that we learned along the way. You can adopt similar gates and review rules in your own skill repository.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Since my last post, the &lt;/span&gt;&lt;a href="https://g.dev/ai/skills" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Google Agent Skills repository&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; has passed 21,000 GitHub stars, up from 15,000. It now hosts over 170 skills from teams across Google, and we distribute them across multiple AI platforms.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/google_skills_star_history_CVeg34U.max-1000x1000.png"
        
          alt="ship-agent-skills-like-products-star-history"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Building agent skills for a single developer or a small project is straightforward. You write a &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;SKILL.md&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; file, define a few instructions, test it manually, and it works. At our scale, manual workflows stop being effective. Unvetted edits can introduce subtle bugs, and it becomes difficult to catch issues before they reach users.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;strong style="vertical-align: baseline;"&gt;The core philosophy: Ship skills like products&lt;/strong&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;One key lesson from my previous post: a skill is a living product, not a one-off document. In practice, that means we ship every skill update like a product release.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;It's tempting to treat a skill as documentation. But documentation explains concepts to a human reader, while a skill steers an AI agent's reasoning loop, tool selection, and execution path. A misplaced phrase in a skill can cause an agent to invoke invalid command flags, enter an infinite loop, or waste tokens.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;That makes every instruction edit a behavioral change. You wouldn't ship a product update without testing it, and the same goes for a skill. This principle drives three core requirements:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Strict semantic versioning&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Every behavioral change requires an explicit version bump.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Deterministic automated validation&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Linters and validators check every proposed change before a human or AI reviewer sees it.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Mandatory evaluation evidence&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Every change includes an evaluation report. It compares the skill against its previous version, or—if the skill is new—against an agent without the skill.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;&lt;strong style="vertical-align: baseline;"&gt;Automated conformance gates: The pull request pipeline&lt;/strong&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;In my previous post, I briefly listed the checks that every skill passes on check-in: linters, link checkers, and AI-assisted checklists. Here's what's inside them, plus two gates that I haven't covered before: semantic versioning and evaluation suite validation.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;When an author submits a change to our internal skills repository, an automated continuous integration and continuous delivery (CI/CD) pipeline triggers immediately. Before a skill is exported to GitHub, it must pass &lt;span style="vertical-align: baseline;"&gt;these&lt;/span&gt; six automated gates and receive human approval:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Structural conformance&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Directory layout, allowed files, and taxonomy.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Skill linters&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Frontmatter, folder naming, and Markdown rules.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Semantic versioning&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Version bump for every behavioral change.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Evaluation suite&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Valid eval suite and a fresh eval report.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Link validation&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Links are valid.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;AI review agent&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Skill quality and design standards.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Human approval&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Final approval from a human reviewer (a subject matter expert)&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;Structural conformance and skill linters&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Our automated validation suite inspects every pull request for structural and syntactical correctness. Some of the rules include these requirements:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Folder name and metadata name must match.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;code style="vertical-align: baseline;"&gt;SKILL.md&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; and description length limits are enforced.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;All required fields in the frontmatter &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;metadata&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; are present.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Only a curated list of files and directories is allowed within a skill directory.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;All Markdown files follow formatting rules.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;The main &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;README.md&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; file and the machine-readable &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;index.json&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; registry are up to date.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;Semantic versioning&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Every skill declares a semantic version in its &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;SKILL.md&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; frontmatter metadata:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;---\r\nname: spanner-basics\r\nmetadata:\r\n  version: &amp;quot;1.1.1&amp;quot;\r\n  category: Databases\r\ndescription: ...\r\n---&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9e5210&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;First, our automated checks verify that the version is a quoted string. YAML parses unquoted values like &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;2.0&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; as numbers, which causes unexpected parsing errors in tools that expect a string.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Second, the validator checks the version transition itself. New skills must start at &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;1.0.0&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, and any change to an existing skill's instructions, scripts, or references requires a version bump or the pull request is blocked. Standard reset rules apply: bumping &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;MAJOR&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; (for example, &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;1.4.2&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; to &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;2.0.0&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;) resets &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;MINOR&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; and &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;PATCH&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; to &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;0&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, and bumping &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;MINOR&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; (for example, &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;1.4.2&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; to &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;1.5.0&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;) resets &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;PATCH&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; to &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;0&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Why enforce strict versioning if the agent only reads the Markdown instructions? The version number is for the developer and their package tooling, not for the model at runtime. Skills don't auto-update in the background, which means there's no network check or latency overhead each time an agent invokes a skill. Instead, explicit versions let developers compare their locally installed skills against the upstream catalog and decide when to pull changes—for example, by running the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;npx skills update&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; command. This versioning also supports workflows where teams want to lock a critical skill to a known-good version to guarantee reproducible agent behavior across their projects.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;Evaluation suite validation&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Our continuous evaluations (evals) compare agent performance with and without each skill. Those results are only as reliable as the evaluation suites behind them. If a suite contains broken file references or compound expectations, test results become noisy and untrustworthy.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Our validator parses each evaluation suite and the pull request description. Then it checks several strict invariants:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Resolution of file-based prompts&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: When a test prompt references a file (such as &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;@file:test_prompt.txt&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;), the validator confirms that the target file exists and that it isn't empty.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Atomic expectations&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Each expectation must check exactly one criterion. The validator blocks compound expectations until authors split them. Lesson 2 later in this post shows why.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Duplicate detection&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The validator flags duplicate test prompts and conflicting expectations within the same evaluation suite.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Mandatory evaluation report&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Every pull request that modifies a skill must link to an evaluation report in its description.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Eval staleness check&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Authors usually iterate on a pull request as reviewers leave feedback. The validator confirms that the linked report ran against the latest commit.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/eval_staleness_check_whiteboard.max-1000x1000.jpg"
        
          alt="ship-agent-skills-like-products-eval-staleness-check"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;Link validation&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Over 170 skills link to documentation for fast-evolving Google products, and those pages can move long after a skill passes review. Rather than waiting for users to report broken links, we validate links at two stages. Before merge, pull request checks verify all relative links, such as files in &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;references/&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Then, every Monday, a &lt;/span&gt;&lt;a href="https://github.com/google/skills/blob/main/.github/workflows/link-checker.yml" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;GitHub Actions workflow&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; runs &lt;/span&gt;&lt;a href="https://github.com/lycheeverse/lychee" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;lychee&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; across all external links in the public repository and it reports any broken links to skill owners for remediation.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;AI review agent&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Linters and schema validators catch syntactic errors, but they can't evaluate overall quality. They can't tell you if an instruction is confusing, if a reference guide is too verbose, or if a task should have been a bash script instead of a large language model (LLM) instruction.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;To solve this, we integrated an AI review agent into our review pipeline. When a change affects a skill, the AI reviewer inspects it using a dedicated review skill. Yes, we use a skill to review skills.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Here are the core rules the AI reviewer evaluates. You can adopt these rules in your own skill development workflows.&lt;/span&gt;&lt;/p&gt;
&lt;h4&gt;&lt;strong style="vertical-align: baseline;"&gt;1. Skill suitability and the deterministic trap&lt;/strong&gt;&lt;/h4&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The first question our AI reviewer asks is simple: &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Does this task belong in an instruction?&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;We often see authors fall into the &lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;deterministic trap&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;. They write rigid, numbered sequences of command-line interface (CLI) commands directly into Markdown text.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;LLMs are probabilistic engines. If an operation requires a strict sequence of commands where a variation can cause an error, embedding those commands in the instruction text invites hallucinations.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Our AI reviewer enforces a clear boundary between deterministic and probabilistic logic. Rigid CLI sequences, JSON parsers, and deployment steps must live in executable scripts under &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;scripts/&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;. Markdown instructions are reserved for tasks that require judgment, such as choosing an architecture, diagnosing error logs, or synthesizing code.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;This rule complements the Model Context Protocol (MCP) guidance that I shared last time: prefer remote MCP tools whenever possible. When you must fall back to CLI or API calls, wrap them in a script instead of spelling them out in the instructions.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/deterministic_trap_decision_tree_whiteboar.max-1000x1000.jpg"
        
          alt="ship-agent-skills-like-products-deterministic-trap"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h4&gt;&lt;strong style="vertical-align: baseline;"&gt;2. The L1/L2/L3 modular architecture&lt;/strong&gt;&lt;/h4&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Context windows are valuable. Loading a 2,000-line skill into an agent consumes context and degrades reasoning.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Our AI reviewer enforces a three-tier information hierarchy that follows the progressive disclosure model of Agent Skills:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Level 1 (Metadata)&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The YAML frontmatter (&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;name&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;description&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;). The agent uses this concise summary for discovery and routing.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Level 2 (Core instructions)&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The main &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;SKILL.md&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; file. The body must stay compact (under 500 lines). It provides workflow overviews, decision trees, and guardrails.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Level 3 (Reference docs)&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Detailed API references, schemas, and other reference tables live in a &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;references/&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; directory.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Our AI reviewer enforces two distinct rules for Level 3: &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;flat directory structure&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; and &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;single-hop indirection&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;. Keeping &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;references/&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; flat (no subdirectories) is a repository convention that discourages sprawling documentation trees and makes file discovery simpler. More importantly for runtime performance, the reviewer blocks &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;chained references&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;, where &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;SKILL.md&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; links to a reference file that links to yet another file. Each hop requires an extra tool call, adding latency and cost, and our evaluations showed that agents frequently stop before following a second level of chained references.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/skills_progressive_disclosure_whiteboard.max-1000x1000.jpg"
        
          alt="ship-agent-skills-like-products-progressive-disclosure"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h4&gt;&lt;strong style="vertical-align: baseline;"&gt;3. Non-interactive script design&lt;/strong&gt;&lt;/h4&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;When skills bundle helper scripts in &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;scripts/&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, we expect agents to execute them directly in terminal sandboxes.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Our AI reviewer flags any script that blocks on interactive prompts. Every script must accept inputs through command-line flags or environment variables, provide &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;--help&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; documentation, and include a &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;--dry-run&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; or confirmation flag for destructive operations. Scripts also need to separate output cleanly, writing structured data (usually JSON) to standard output and writing actionable error messages to standard error.&lt;/span&gt;&lt;/p&gt;
&lt;h4&gt;&lt;strong style="vertical-align: baseline;"&gt;Example AI review checklist snippet&lt;/strong&gt;&lt;/h4&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The review skill also checks frontmatter for spec compliance and discovery, and it turns the L1/L2/L3 architecture into concrete structure rules. Here's an excerpt from those criteria:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;### Front Matter &amp;amp; Discovery\r\n\r\n-   **Spec Compliance**: YAML front matter MUST strictly follow the Agent Skills\r\n    Reference Specification (https://agentskills.io/specification). Use `&amp;gt;-`\r\n    (folded strip-chomping) format for multi-line descriptions.\r\n-   **Description Quality**: The `description` field must follow the\r\n    [Description Guide](references/description_guide.md).\r\n\r\n### Content Structure &amp;amp; Authoring Quality\r\n\r\n-   **L1/L2/L3 Architecture**: Enforce modular structure. The main `SKILL.md`\r\n    body must stay under 500 lines. Move deep domain reference material to\r\n    `references/` (see [Skill Structure](references/skill_structure.md)).\r\n-   **Flat References Directory**: `references/` must be a flat directory (one\r\n    level deep, no nested subdirectories). References &amp;gt; 100 lines must include a\r\n    Table of Contents with line hints (see\r\n    [Skill Structure](references/skill_structure.md)).\r\n-   **Avoid Deeply-Nested References**: Avoid deeply-nested references (e.g.,\r\n    `SKILL.md` -&amp;gt; `references/guides.md` -&amp;gt; `assets/output.tmpl`). Keep\r\n    reference structures shallow (one level of indirection from `SKILL.md`)\r\n    because agents might not load such link targets fully.&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9323d0&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;Human approval&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;After it passes all automated checks, every pull request must receive approval from a human reviewer. The reviewer is typically the skill's subject matter expert and, for new skills, a member of the Central Skills team.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;strong style="vertical-align: baseline;"&gt;Lessons learned from repository history&lt;/strong&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Six months of commit history and pull request discussions reveal several practical takeaways.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;1. Scope CI checks carefully&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Early in the project, we configured automated pull request checks with broad directory patterns.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;One check required an approved bug tracking ticket on every change. Because the pattern matched too broadly, engineers making a one-line typo fix were blocked for missing a tracking ticket. Meanwhile, newly added skills in deeper subdirectories went uninspected.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Takeaway&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Match automated check rules to specific file operations, such as added versus modified files, and verify exact path depths. Treat minor edits to existing skills differently from new skills.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;2. Atomic expectations&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;When we started writing evaluation suites, we saw authors write compound expectations like this:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;# Bad: Compound expectation\r\n- &amp;quot;The agent must create a Cloud Storage bucket in us-central1, enable\r\n  uniform bucket-level access, and apply lifecycle rules.&amp;quot;&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba933010&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;If the agent configured the region and lifecycle rules but missed the uniform access flag, the LLM-as-a-judge scorer returned inconsistent ratings.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Migrating to atomic expectations resolved the inconsistent ratings:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;# Good: Atomic expectations\r\n- &amp;quot;The agent specifies us-central1 for the bucket location.&amp;quot;\r\n- &amp;quot;The agent enables uniform bucket-level access on the created bucket.&amp;quot;\r\n- &amp;quot;The agent defines a lifecycle configuration rule deleting objects\r\n  older than 30 days.&amp;quot;&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba933a90&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Takeaway&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Check exactly one criterion per expectation. Atomic expectations make evaluation results clear, reliable, and debuggable.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;3. Documentation drift happens fast&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;In a multi-contributor repository, documentation tables drift within days if maintained manually.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Takeaway&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: If you publish an index or catalog, never rely on contributors to update it manually. Instead, write a generator script and enforce it with a CI unit test. That's the origin of the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;README.md&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; and &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;index.json&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; checks in our linters. Our &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;index.json&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; even opens with a warning: "This file is generated. Do not edit it by hand."&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;4. Gate expensive LLM checks behind fast deterministic ones&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Running full evaluation suites and an AI review agent on every change can get expensive fast, both in token usage and CI runtime—especially when authors push frequent small commits during review.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;In our pipeline, fast deterministic checks run first. Structural conformance, skill linters, and semantic versioning act as blocking gates: if any of them fail, the pipeline stops immediately and it won't trigger the AI reviewer or evaluation checks until those issues are fixed.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Takeaway&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Stage your pipeline so cheap, deterministic checks block broken changes before LLM-powered gates run. When you build a similar system, consider adding further cost guardrails, such as manual triggers for heavy checks (for example, a &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;/run-evals&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; pull request comment) or only triggering automatic re-runs when the diff exceeds a specific threshold.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;strong style="vertical-align: baseline;"&gt;Conclusion&lt;/strong&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Building a scalable Agent Skills repository requires more than good prompt engineering. It requires solid developer infrastructure:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Ship skills like products: version, validate, and evaluate every change.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Enforce semantic versioning and atomic expectations with automated CI gates.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Use an AI reviewer to catch design anti-patterns, such as the deterministic trap.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Check links before merge and on a schedule.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;&lt;strong style="vertical-align: baseline;"&gt;Links and further reading&lt;/strong&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;To get started with Google Agent Skills, check out these resources:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Google Agent Skills repo&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: &lt;/span&gt;&lt;a href="https://g.dev/ai/skills" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;github.com/google/skills&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Previous post&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: &lt;/span&gt;&lt;a href="https://cloud.google.com/blog/topics/developers-practitioners/behind-the-scenes-how-we-build-test-and-scale-google-agent-skills"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Behind the scenes: How we build, test, and scale Google Agent Skills&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Launch announcement&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: &lt;/span&gt;&lt;a href="https://cloud.google.com/blog/topics/developers-practitioners/level-up-your-agents-announcing-googles-official-skills-repository"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Level Up Your Agents: Announcing Google's Official Skills Repository&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Skills tutorial&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: &lt;/span&gt;&lt;a href="https://medium.com/google-cloud/google-cloud-skills-tutorial-the-complete-guide-to-ai-powered-cloud-operations-7838fcc9541a" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Introduction: What Are Google Cloud Agent Skills?&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;&lt;strong style="vertical-align: baseline;"&gt;What's next?&lt;/strong&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;If you found this post helpful, share it with your friends on social media and connect with me on &lt;/span&gt;&lt;a href="https://www.linkedin.com/in/remigiusz-samborski/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;LinkedIn&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;a href="https://x.com/RemikSamborski" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;X&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, or &lt;/span&gt;&lt;a href="https://bsky.app/profile/rsamborski.bsky.social" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Bluesky&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Thanks for reading!&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;</description><pubDate>Wed, 28 Oct 2026 16:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/topics/developers-practitioners/behind-the-scenes-two-months-later-how-we-ship-google-agent-skills-like-products/</guid><category>Developers &amp; Practitioners</category><media:content height="540" url="https://storage.googleapis.com/gweb-cloudblog-publish/images/hero_pdfIUE4.max-600x600.jpg" width="540"></media:content><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Behind the scenes, two months later: How we ship Google Agent Skills like products</title><description></description><image>https://storage.googleapis.com/gweb-cloudblog-publish/images/hero_pdfIUE4.max-600x600.jpg</image><site_name>Google</site_name><url>https://cloud.google.com/blog/topics/developers-practitioners/behind-the-scenes-two-months-later-how-we-ship-google-agent-skills-like-products/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Remigiusz Samborski</name><title>Lead Developer Relations Engineer</title><department></department><company></company></author></item><item><title>How to troubleshoot GKE workloads faster with VPC Flow Logs</title><link>https://cloud.google.com/blog/products/containers-kubernetes/using-vpc-flow-logs-to-troubleshoot-gke-workloads/</link><description>&lt;div class="block-paragraph_advanced"&gt;&lt;p style="text-align: justify;"&gt;&lt;span style="vertical-align: baseline;"&gt;Having visibility into your network traffic is essential to operating Kubernetes environments effectively. When a service slows down or loses connectivity, knowing who is talking to whom lets you narrow the problem down quickly. In a standard cloud network, flow logs provide that view by recording connections between IP addresses, ports, and protocols. In Kubernetes, however, pods are temporary. Their IP addresses change constantly as pods scale, crash, or move between nodes, so a raw log entry showing 10.4.2.18 talking to 10.4.8.44 tells you little about which workload (and pod) owned that address, what namespace it was in, or what application it was trying to reach. By the time you search your cluster, the pod may already be gone!&lt;/span&gt;&lt;/p&gt;
&lt;p style="text-align: justify;"&gt;&lt;span style="vertical-align: baseline;"&gt;In a Google Cloud environment, &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/vpc/docs/using-flow-logs"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;VPC Flow Logs&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; closes this gap at the network layer. Google Cloud's virtual network already sees every connection a pod makes, so each sampled flow is recorded with the &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/kubernetes-engine/docs/concepts/kubernetes-engine-overview"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Google Kubernetes Engine&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; (GKE) metadata behind the IP address: pod, workload, namespace, and service. You get workload-level attribution for L3/L4 traffic without deploying extra agents or sidecars, and it complements in-cluster Container Network Interface (CNI) and dataplane observability tools you may already run when you need deeper detail. Let’s take a deeper look at how VPC Flow Logs and GKE work together, and how to get started with it.&lt;/span&gt;&lt;/p&gt;
&lt;h3 style="text-align: justify;"&gt;&lt;strong style="vertical-align: baseline;"&gt;How GKE metadata enriches VPC Flow Logs&lt;/strong&gt;&lt;/h3&gt;
&lt;p style="text-align: justify;"&gt;&lt;span style="vertical-align: baseline;"&gt;VPC Flow Logs samples traffic directly inside Google Cloud's virtual network. When flow logs are enabled for a subnet that hosts a GKE cluster, every record of pod traffic that passes through the VPC network is annotated with the Kubernetes context that owned the IP address at the time of sampling. This helps ensure the context is preserved even after the pod is gone:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation" style="text-align: justify;"&gt;&lt;strong style="vertical-align: baseline;"&gt;Cluster: &lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;The cluster name and location.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation" style="text-align: justify;"&gt;&lt;strong style="vertical-align: baseline;"&gt;Workload: &lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;The pod name and namespace, along with the controller that owns the pod, such as a deployment, StatefulSet, or DaemonSet.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation" style="text-align: justify;"&gt;&lt;strong style="vertical-align: baseline;"&gt;Service name: &lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;The Kubernetes service that received the traffic.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation" style="text-align: justify;"&gt;&lt;strong style="vertical-align: baseline;"&gt;Node and zone:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; The standard VPC fields, including the node VM and its zone on both ends of the flow.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p style="text-align: justify;"&gt;&lt;span style="vertical-align: baseline;"&gt;If you register your GKE workloads in &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/app-hub/docs/overview?e=48754805"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;App Hub&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, each record also carries application-level labels (&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;apphubSource&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; and &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;apphubDestination&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;) that identify the business application, environment, and criticality of each endpoint. This is especially useful when GKE microservices talk to App Hub-registered resources outside the cluster, such as Cloud Run services.&lt;/span&gt;&lt;/p&gt;
&lt;h3 style="text-align: justify;"&gt;&lt;strong style="vertical-align: baseline;"&gt;Before you start&lt;/strong&gt;&lt;/h3&gt;
&lt;p style="text-align: justify;"&gt;&lt;strong style="vertical-align: baseline;"&gt;Upgrade your log bucket to &lt;/strong&gt;&lt;a href="https://cloud.google.com/network-intelligence-center/docs/flow-analyzer/enable-log-analytics"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;Observability Analytics&lt;/strong&gt;&lt;/a&gt;&lt;strong style="vertical-align: baseline;"&gt; (formerly Log Analytics): &lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;Flow Analyzer reads only from an upgraded bucket, typically &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;_Default&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;. &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Note:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; This upgrade is permanent.&lt;br/&gt;&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;gcloud logging buckets update _Default \&lt;br/&gt;&lt;/code&gt;&lt;code style="vertical-align: baseline;"&gt;    --location=global \&lt;br/&gt;&lt;/code&gt;&lt;code style="vertical-align: baseline;"&gt;    --enable-analytics&lt;/code&gt;&lt;/p&gt;
&lt;p style="text-align: justify;"&gt;&lt;strong style="vertical-align: baseline;"&gt;Enable VPC Flow Logs organization-wide:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; One organization-level configuration covers every subnet in every VPC network, including future clusters (requires Network Management Admin role). Keep &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;include-all-metadata&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; on so every record carries namespace and pod details. To balance visibility against cost, you can add a secondary sampling rate (&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;--flow-sampling&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;), an aggregation interval (&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;--aggregation-interval&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;), or a filter expression (&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;--filter-expr&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;) to view only the traffic of interest.&lt;br/&gt;&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;gcloud network-management vpc-flow-logs-configs create CONFIG_NAME \&lt;br/&gt;&lt;/code&gt;&lt;code style="vertical-align: baseline;"&gt;    --location=global \&lt;br/&gt;&lt;/code&gt;&lt;code style="vertical-align: baseline;"&gt;    --organization=ORGANIZATION_ID \&lt;br/&gt;&lt;/code&gt;&lt;code style="vertical-align: baseline;"&gt;    --metadata=include-all-metadata&lt;/code&gt;&lt;/p&gt;
&lt;p style="text-align: justify;"&gt;&lt;strong style="vertical-align: baseline;"&gt;Turn on &lt;/strong&gt;&lt;a href="https://cloud.google.com/kubernetes-engine/docs/how-to/intranode-visibility"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;intranode visibility&lt;/strong&gt;&lt;/a&gt;&lt;strong style="vertical-align: baseline;"&gt;:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Autopilot clusters already have it, but you need to enable it on Standard clusters. &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Note: &lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;Enabling it restarts control plane and node components and adds some per-node overhead, so test it on a non-production cluster first.&lt;br/&gt;&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;gcloud container clusters update CLUSTER_NAME \&lt;br/&gt;&lt;/code&gt;&lt;code style="vertical-align: baseline;"&gt;    --location=LOCATION \&lt;br/&gt;&lt;/code&gt;&lt;code style="vertical-align: baseline;"&gt;    --enable-intra-node-visibility&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Now you’re ready to get started.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;Analyze traffic visually in Flow Analyzer&lt;/strong&gt;&lt;/h3&gt;
&lt;p style="text-align: justify;"&gt;&lt;span style="vertical-align: baseline;"&gt;VPC Flow Logs write every record to Cloud Logging, you can search and query it in Logs Explorer or Log Analytics. For day-to-day work, &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/network-intelligence-center/docs/flow-analyzer/overview"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Flow Analyzer&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, part of Network Intelligence Center, turns the same records into an interactive chart and table, so you can group and filter traffic by cluster, namespace, workload, pod, or service in a few clicks. &lt;/span&gt;&lt;/p&gt;
&lt;p style="text-align: justify;"&gt;&lt;span style="vertical-align: baseline;"&gt;The walkthroughs below use &lt;/span&gt;&lt;a href="https://github.com/GoogleCloudPlatform/microservices-demo" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Online Boutique&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, an open-source microservices demo, on a GKE Autopilot cluster in us-central1. A &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;loadgenerator&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; deployment sends synthetic shopper traffic to the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;frontend&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, which calls &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;productcatalogservice&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;cartservice&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;currencyservice&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;recommendationservice&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;shippingservice&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;adservice&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, and &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;checkoutservice&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;. &lt;/span&gt;&lt;/p&gt;
&lt;p style="text-align: justify;"&gt;&lt;strong style="vertical-align: baseline;"&gt;Case 1: Turn IP addresses into workload names&lt;br/&gt;&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;Whether you’re investigating an incident or assessing an architecture change, the first question is always the same: Which endpoints are communicating, and at what volume? &lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Rather than presenting ephemeral IP address pairs, GKE metadata allows Flow Analyzer to answer this question directly using Kubernetes workload names.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Steps in Flow Analyzer:&lt;/span&gt;&lt;/p&gt;
&lt;ol&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Open &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Network Intelligence Center&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; &amp;gt; &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Flow Analyzer&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; and set the time range to &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;last 1 hour&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Under &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Source&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; and &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Destination&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;, set &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Organize flows&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;by&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; to GKE workload.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;In the filter bar, add the &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;GKE workload is known&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; filter so only pod-to-pod traffic is shown.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Click &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Run new query&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/original_images/1_HzpvhDD.gif"
        
          alt="1"&gt;
        
        &lt;/a&gt;
      
        &lt;figcaption class="article-image__caption "&gt;&lt;p data-block-key="psylm"&gt;Grouping traffic by Source GKE workload and Destination GKE workload in Flow Analyzer to map microservice dependencies without raw IP lookups.&lt;/p&gt;&lt;/figcaption&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p style="text-align: justify;"&gt;&lt;strong style="vertical-align: baseline;"&gt;Key insights: &lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;Sorted by volume, the table is a ready-made application dependency map: &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;frontend&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; handles incoming requests from &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;loadgenerator&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; (representing user traffic), while &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;productcatalogservice&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; answers calls from both &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;recommendationservice&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; and &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;frontend&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;. Because each row shows workload names instead of raw IP addresses, any unauthorized or unexpected workload-to-workload communication immediately becomes apparent.&lt;/span&gt;&lt;/p&gt;
&lt;p style="text-align: justify;"&gt;&lt;strong style="vertical-align: baseline;"&gt;Case 2: Drill into a single service&lt;br/&gt;&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;When a service is slow or about to change, you need to know who depends on it and how heavily.&lt;/span&gt;&lt;/p&gt;
&lt;p style="text-align: justify;"&gt;&lt;span style="vertical-align: baseline;"&gt;Steps in Flow Analyzer:&lt;/span&gt;&lt;/p&gt;
&lt;ol&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation" style="text-align: justify;"&gt;&lt;span style="vertical-align: baseline;"&gt;Change &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Traffic aggregation&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; to &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Client&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; ↔&lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt; server (bidirectional)&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;, so requests and responses between a pair are counted together.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation" style="text-align: justify;"&gt;&lt;span style="vertical-align: baseline;"&gt;Under &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Server&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;, add the filter &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;GKE workload&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; = &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;productcatalogservice&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation" style="text-align: justify;"&gt;&lt;span style="vertical-align: baseline;"&gt;Click&lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt; Run new query&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;, then click &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Details&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; on the top row.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/original_images/2_H9re5kG.gif"
        
          alt="2"&gt;
        
        &lt;/a&gt;
      
        &lt;figcaption class="article-image__caption "&gt;&lt;p data-block-key="psylm"&gt;Filtering on Server GKE workload (productcatalogservice) with Client ↔ server (bidirectional) aggregation to view caller share and the Details timeline.&lt;/p&gt;&lt;/figcaption&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p style="text-align: justify;"&gt;&lt;strong style="vertical-align: baseline;"&gt;Key insights:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; You get every caller of the catalog service, each caller's share of its traffic, and a timeline for every pair, so if one caller's line climbs while the others stay flat, you know where to look first. When the question is latency rather than volume, switch the display mode to latency to rank the same callers by round-trip time. &lt;/span&gt;&lt;/p&gt;
&lt;p style="text-align: justify;"&gt;&lt;strong style="vertical-align: baseline;"&gt;Case 3: Find the pods that talk across zones&lt;br/&gt;&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;In a regional GKE cluster, chatty services that land in different zones send every request across a zone boundary, adding inter-zone data transfer charges. Because VPC Flow Logs record the zone at both ends of every flow, the pod pairs driving that traffic are easy to spot.&lt;/span&gt;&lt;/p&gt;
&lt;p style="text-align: justify;"&gt;&lt;span style="vertical-align: baseline;"&gt;Steps in Flow Analyzer:&lt;/span&gt;&lt;/p&gt;
&lt;ol&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation" style="text-align: justify;"&gt;&lt;span style="vertical-align: baseline;"&gt;Switch from &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Basic filters&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; to &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;SQL filters&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;, which let you compare two fields of the same flow.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation" style="text-align: justify;"&gt;&lt;span style="vertical-align: baseline;"&gt;In the &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;WHERE&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; box, enter &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;src_gcp_zone &amp;lt;&amp;gt; dest_gcp_zone&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation" style="text-align: justify;"&gt;&lt;span style="vertical-align: baseline;"&gt;Set &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Organize flows&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; by to &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Source GCP zone, Source GKE pod, Destination GCP zone&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;, and &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Destination GKE pod&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation" style="text-align: justify;"&gt;&lt;span style="vertical-align: baseline;"&gt;Click &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Run new query&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/original_images/3_3n9hbpu.gif"
        
          alt="3"&gt;
        
        &lt;/a&gt;
      
        &lt;figcaption class="article-image__caption "&gt;&lt;p data-block-key="psylm"&gt;Using SQL filters with src_gcp_zone &amp;lt;&amp;gt; dest_gcp_zone and multi-attribute grouping to identify pod pairs driving cross-zone network traffic.&lt;/p&gt;&lt;/figcaption&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p style="text-align: justify;"&gt;&lt;strong style="vertical-align: baseline;"&gt;Key insights:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; The table displays cross-zone traffic ranked by volume, identifying the exact pods and zones on each end. In our cluster, the top flows occur between a &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;frontend&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; pod in us-central1-b and a &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;currencyservice&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; pod in us-central1-c, followed by &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;checkoutservice&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;shippingservice&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, and &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;productcatalogservice&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; pairs. To investigate further, click Explore in Log Analytics to open the underlying SQL query for additional customization, such as adding joins or time buckets.&lt;/span&gt;&lt;/p&gt;
&lt;p style="text-align: justify;"&gt;&lt;span style="vertical-align: baseline;"&gt;GKE has a built-in answer: set &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;trafficDistribution: PreferSameZone&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; (&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;PreferClose&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; on older versions) on the Service to prefer healthy endpoints in the caller's zone, and make sure each zone has enough replicas to serve its own traffic.&lt;/span&gt;&lt;/p&gt;
&lt;h3 style="text-align: justify;"&gt;&lt;strong style="vertical-align: baseline;"&gt;What to keep in mind&lt;/strong&gt;&lt;/h3&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation" style="text-align: justify;"&gt;&lt;strong style="vertical-align: baseline;"&gt;Pod-to-internet traffic:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; VPC Flow Logs carry no pod annotations, because the masquerade agent replaces the pod IP with the node IP before the flow is sampled. Keeping internal destinations in the non-masquerade list preserves pod names for those flows, and if you run &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;ip-masq-agent&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; with intranode visibility, that list must also include the cluster's node and pod IP ranges to avoid connectivity problems.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation" style="text-align: justify;"&gt;&lt;strong style="vertical-align: baseline;"&gt;Blocked traffic:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Traffic blocked by a network policy never reaches the VPC network, so use &lt;/span&gt;&lt;a href="https://cloud.google.com/kubernetes-engine/docs/how-to/network-policy-logging"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;NetworkPolicy logging&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; to investigate it.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation" style="text-align: justify;"&gt;&lt;strong style="vertical-align: baseline;"&gt;Sampling:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Flow logs are sampled, which makes &lt;/span&gt;&lt;a href="https://services.google.com/fh/files/misc/vpc_flow_logs_understanding_byte_and_packet_counts.pdf" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;byte and packet counts&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; well suited for ranking and trending rather than per-connection accounting.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;Next steps&lt;/strong&gt;&lt;/h3&gt;
&lt;p style="text-align: justify;"&gt;&lt;span style="vertical-align: baseline;"&gt;If your clusters already have VPC Flow Logs turned on, upgrading the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;_Default&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; bucket and opening Flow Analyzer is all it takes to reproduce the three walkthroughs above. To learn more, see the documentation:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://cloud.google.com/vpc/docs/using-flow-logs#gke"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Using VPC Flow Logs&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://cloud.google.com/vpc/docs/about-flow-logs-records#gke-annotations"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;VPC Flow Logs record format for GKE&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://cloud.google.com/network-intelligence-center/docs/flow-analyzer/analyze-traffic-flows"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Analyzing network traffic with Flow Analyzer&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://cloud.google.com/kubernetes-engine/docs/best-practices/network-observability"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Best practices for GKE network observability&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://cloud.google.com/kubernetes-engine/docs/troubleshooting/network-observability"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Troubleshooting GKE network observability&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;&lt;/div&gt;</description><pubDate>Wed, 28 Oct 2026 16:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/containers-kubernetes/using-vpc-flow-logs-to-troubleshoot-gke-workloads/</guid><category>Networking</category><category>GKE</category><category>Developers &amp; Practitioners</category><category>Containers &amp; Kubernetes</category><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>How to troubleshoot GKE workloads faster with VPC Flow Logs</title><description></description><site_name>Google</site_name><url>https://cloud.google.com/blog/products/containers-kubernetes/using-vpc-flow-logs-to-troubleshoot-gke-workloads/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Nishant Kumar</name><title>Product Manager</title><department></department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Whitney Jenkins</name><title>Product Manager</title><department></department><company></company></author></item><item><title>Claude Opus 5.5 and Claude Sonnet 5.5 are now available in Google Antigravity</title><link>https://cloud.google.com/blog/products/ai-machine-learning/claude-opus-55-and-claude-sonnet-55-available-in-antigravity/</link><description>&lt;div class="block-paragraph"&gt;&lt;p data-block-key="7jvio"&gt;Whether you are iterating on a quick bug fix or planning a complex migration, different tasks call for different models.&lt;/p&gt;&lt;p data-block-key="clmp"&gt;In keeping with Google Cloud's commitment to an open ecosystem and to give developers the right model for every job, &lt;b&gt;Claude Opus 5.5 and Claude Sonnet 5.5 are now available in Google Antigravity for Google Cloud customers.&lt;/b&gt; Claude Opus 5.5 is built for complex work while Claude Sonnet 5.5 is a faster, lower-cost complement.&lt;/p&gt;&lt;p data-block-key="prmr"&gt;Once enabled by an administrator, developers can select Claude Opus 5.5 and Sonnet 5.5 directly from the model picker. Administrators can also select Google’s Gemini models, including Gemini 3.8 Flash and Gemini 4 Argon. The models apply across Antigravity 2.0, Antigravity CLI and our recently released &lt;a href="https://antigravity.google/docs/ide/extensions/" target="_blank"&gt;Antigravity IDE extensions&lt;/a&gt; for VS Code, JetBrains IntelliJ, and Visual Studio. This lets you choose the right model for the job, under a unified enterprise billing and security boundary.&lt;/p&gt;&lt;p data-block-key="dj20l"&gt;In this post, we’ll share two engineering patterns for building in Antigravity using these models. To view the entire list of models supported in Antigravity across subscriptions, see our documentation &lt;a href="https://antigravity.google/docs/models/" target="_blank"&gt;here&lt;/a&gt;.&lt;/p&gt;&lt;h3 data-block-key="d8trn"&gt;Multi-model engineering patterns in Antigravity&lt;/h3&gt;&lt;p data-block-key="2ach2"&gt;Having multiple model families available inside the same agent harness unlocks practical multi-model workflows that improve code quality while controlling latency and cost.&lt;/p&gt;&lt;h3 data-block-key="a8ic8"&gt;&lt;b&gt;Pattern 1: Cross-model verification ("LLM-as-a-Judge") in loop and graph workflows&lt;/b&gt;&lt;/h3&gt;&lt;p data-block-key="a53gs"&gt;When building agentic loops or multi-step graph workflows, using the same model to both write code and review its own output can reinforce model-specific blind spots. Pairing two distinct model families, one as the implementer and another as the validator (judge), provides independent verification before moving to the next stage of a workflow.&lt;/p&gt;&lt;p data-block-key="eca00"&gt;In Antigravity, you can run this pattern either interactively or via automated subagents:&lt;/p&gt;&lt;ol&gt;&lt;li data-block-key="enlkk"&gt;&lt;b&gt;Plan and implement with Model A:&lt;/b&gt; Start in planning mode using &lt;code&gt;Claude Opus 5.5&lt;/code&gt; or &lt;code&gt;Gemini 4 Argon&lt;/code&gt; to design and implement a multi-file change.&lt;/li&gt;&lt;li data-block-key="c4jg9"&gt;&lt;b&gt;Validate with Model B:&lt;/b&gt; Before committing or advancing the loop, switch models, or invoke a one-shot &lt;code&gt;/model&lt;/code&gt; check in the CLI, to have a different model family critique the implementation against your test suite and requirements.&lt;/li&gt;&lt;/ol&gt;&lt;h3 data-block-key="fp277"&gt;&lt;b&gt;Pattern 2: Pairing reasoning models with Flash subagents&lt;/b&gt;&lt;/h3&gt;&lt;p data-block-key="2ce0q"&gt;Complex engineering tasks often combine a few high-reasoning decisions with dozens of high-volume, mechanical lookups (searching files, grepping logs, checking lint output, or classifying test failures). Running every lookup on a frontier model can increase both latency and spend.&lt;/p&gt;&lt;p data-block-key="ckd6t"&gt;Antigravity's Subagent customization system (&lt;code&gt;.agents/agents/*.md&lt;/code&gt;) lets you assign specific model tiers (&lt;code&gt;inherit&lt;/code&gt;, &lt;code&gt;flash&lt;/code&gt;, or &lt;code&gt;pro&lt;/code&gt;) to specialized subagents. When your primary session is running a more powerful model.&lt;/p&gt;&lt;p data-block-key="2l8u2"&gt;Subagents configured with &lt;code&gt;model: inherit&lt;/code&gt; use your active session model for deep reasoning subtasks.&lt;/p&gt;&lt;p data-block-key="cb05p"&gt;Subagents configured with &lt;code&gt;model: flash&lt;/code&gt; offload high-speed search, classification, and triage work to Gemini Flash models.&lt;/p&gt;&lt;p data-block-key="1fr9p"&gt;For example, you can define a fast log-triage subagent in &lt;code&gt;.agents/agents/test-triager.md&lt;/code&gt;:&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;---\r\nname: test-triager\r\ndescription: Runs unit tests, classifies failures, and returns a concise summary of failing assertions.\r\nmodel: flash\r\ntools:\r\n  - read\r\n  - edit\r\n---\r\n\r\nRun the requested test targets, classify any failures by root cause (compilation error, assertion failure, or timeout), and return a structured summary to the parent agent.&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba98ff50&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph"&gt;&lt;p data-block-key="5p921"&gt;And a cross-model code reviewer subagent in &lt;code&gt;.agents/agents/gemini-judge.md&lt;/code&gt;:&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;quot;---\r\nname:cross-model-judge\r\ndescription: Independent LLM-as-a-judge verifier that audits diffs and edge cases using Gemini Pro.\r\nmodel: pro\r\ntools:\r\n  - read\r\n---\r\n\r\nReview the current workspace diff against the user&amp;#x27;s requirements. Check for regressions, missing edge-case tests, and security issues. Output a pass/fail verdict with specific line references.&amp;quot;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba997350&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph"&gt;&lt;p data-block-key="5p921"&gt;With this setup, you can delegate rapid test-and-classify loops to &lt;code&gt;test-triager&lt;/code&gt; and trigger &lt;code&gt;cross-model-judge&lt;/code&gt; to validate the final diff, combining the strengths of each model in a single workflow.&lt;/p&gt;&lt;h3 data-block-key="fj5e3"&gt;Getting started&lt;/h3&gt;&lt;p data-block-key="eudoh"&gt;Claude Opus 5.5 and Claude Sonnet 5.5 are available now for Google Cloud customers using Antigravity:&lt;/p&gt;&lt;ol&gt;&lt;li data-block-key="19dsb"&gt;&lt;b&gt;Administrators (Gemini for business):&lt;/b&gt; Third-party partner models do not draw from the pooled Gemini Enterprise Standard and Plus edition monthly token quota. Instead, usage is billed on a pay-as-you-go basis through your existing Google Cloud Billing account, so overages must be enabled. Visit “Gemini Enterprise” &amp;gt; “Settings” &amp;gt; “AI developer tools” in the Google Cloud Console to configure overage billing, spend limits and authorize partner models for your organization. See &lt;a href="https://cloud.google.com/gemini/enterprise/docs/ai-developer-tools-settings"&gt;Configure AI developer tools settings&lt;/a&gt; for step-by-step instructions.&lt;/li&gt;&lt;li data-block-key="evit9"&gt;&lt;b&gt;Administrators (Gemini platform):&lt;/b&gt; You can also enable partner models if you are using Antigravity through the API in Gemini platform. Enable Claude models in Model Garden and complete the standard Anthropic terms.&lt;/li&gt;&lt;li data-block-key="1shb0"&gt;&lt;b&gt;Developers:&lt;/b&gt; Update to the latest version of Antigravity 2.0, the Antigravity CLI, or the Antigravity IDE extensions, and open the model picker (or run &lt;code&gt;/model&lt;/code&gt;) to select the right model for your task. To learn more about enterprise governance and multi-agent configuration, visit the &lt;a href="https://antigravity.google/docs/enterprise/" target="_blank"&gt;Google Antigravity Enterprise documentation&lt;/a&gt;. You can switch the active model between turns in the same conversation thread while preserving workspace state, open artifacts, and terminal context.&lt;/li&gt;&lt;/ol&gt;&lt;p data-block-key="c19fm"&gt;Note: Lightweight first-party Gemini models continue to run in the background to handle high-frequency client-side coordination, such as context management and summaries. They consume first-party tokens and can incur charges.&lt;/p&gt;&lt;/div&gt;</description><pubDate>Tue, 20 Oct 2026 16:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/ai-machine-learning/claude-opus-55-and-claude-sonnet-55-available-in-antigravity/</guid><category>Developers &amp; Practitioners</category><category>AI &amp; Machine Learning</category><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Claude Opus 5.5 and Claude Sonnet 5.5 are now available in Google Antigravity</title><description></description><site_name>Google</site_name><url>https://cloud.google.com/blog/products/ai-machine-learning/claude-opus-55-and-claude-sonnet-55-available-in-antigravity/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Marialaura Garcia</name><title>Product Marketing Manager</title><department>Google Cloud</department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Rich Hyndman</name><title>Developer Relations</title><department>Google Antigravity</department><company></company></author></item><item><title>Developer’s guide to implementing MCP apps in Gemini</title><link>https://cloud.google.com/blog/topics/developers-practitioners/devs-guide-to-mcp-apps-in-gemini/</link><description>&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Enterprise assistants still respond mostly in text. However, that can break down for visual, multi-step work, where every adjustment costs another slow chat turn.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;a href="https://github.com/modelcontextprotocol/ext-apps" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;MCP Apps&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; (&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;SEP-1865&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;) solves this. This open Model Context Protocol (MCP) extension pairs a tool with a live UI. In &lt;/span&gt;&lt;a href="https://business.gemini.google/?utm_source=cloud.google.com/blog"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Gemini Enterprise&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, that UI renders directly in the chat flow inside a sandboxed iframe, and can expand to fullscreen or a side panel. Edits run locally while the workspace and assistant stay synchronized.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;You also avoid rebuilding core foundations: Gemini provides workforce sign-in, governed data access, and the Default Assistant, while you host your MCP server on a private &lt;/span&gt;&lt;a href="https://cloud.google.com/run"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Cloud Run&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; service.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Why we need MCP Apps&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Enterprise assistants have evolved through three generations:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Generation 1 (Text chatbots):&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Returns static markdown with no interactive controls.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Generation 2 (Standard MCP tools):&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Invokes live APIs, but outputs static text. Every change requires another model turn.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Generation 3 (Interactive MCP Apps):&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Pairs each tool with a live workspace. Adjustments happen in the widget without a model turn, while the assistant stays in sync.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Building MCP Apps in Gemini delivers six benefits:&lt;/span&gt;&lt;/h3&gt;
&lt;ol&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Responsive user experience:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Single-click actions execute directly in the panel without round-tripping through the chat.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Purpose-built workflows:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Deliver visual redaction views for legal review, 3D viewers for CAD, or sensitivity models for finance.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Enterprise security:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Workforce OAuth sign-in, private Cloud Run ingress, and a sandboxed iframe protect every layer.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Human-in-the-loop governance:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Experts inspect and approve high-stakes actions before execution.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Open standard:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Portable across compatible hosts, with automatic text fallback.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Broad connectivity:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Connect to BigQuery, Agent Platform, Cloud Storage, or internal ERP and CRM systems.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;High-level architecture&lt;/span&gt;&lt;/h3&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/2-high_level_architecture.max-1000x1000.png"
        
          alt="2-high level architecture"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Three decoupled tiers work together:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Client browser:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Shows the chat alongside the MCP App widget in the side panel.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Gemini Enterprise:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; The default assistant calls tools, the AppBridge proxy relays widget messages, and the connector manages OAuth.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Google Cloud (your project):&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; You deploy and host the MCP server on Cloud Run, which exposes one JSON-RPC endpoint (/mcp), runs background jobs, and serves the single-file HTML widget.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Each tier talks only to its neighbor, letting Gemini Enterprise enforce security and routing.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Authentication with two independent layers&lt;/span&gt;&lt;/h3&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/3-Authentication.max-1000x1000.png"
        
          alt="3-Authentication"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Each user signs in through OAuth 2.0. On every call, Gemini Enterprise sends its service agent token alongside the user's OAuth token:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Layer 1 (Cloud Run IAM):&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Enforced at Google's edge, this answers &lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;"which service may call?"&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; Public access is disabled; unauthorized callers receive &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;HTTP 403&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Layer 2 (In-App OAuth):&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Enforced inside the application, this answers &lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;"which user is calling?"&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; Failed verification returns &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;HTTP 401&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;, prompting the user to sign in again.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Cloud Run IAM cannot distinguish end users, so both layers are required. Outbound calls use a least-privilege service account.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Build, deploy on Google Cloud, and register in Gemini Enterprise&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;First, &lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;build your custom MCP server (exposing POST /mcp and a bundled ui:// HTML widget) using the&lt;/span&gt;&lt;a href="https://github.com/modelcontextprotocol/ext-apps" rel="noopener" target="_blank"&gt;&lt;span style="vertical-align: baseline;"&gt; &lt;/span&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;MCP Apps SDK&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, or use &lt;/span&gt;&lt;a href="https://antigravity.google/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Google Antigravity&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;  to scaffold and deploy your MCP App with this starter prompt:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Build an MCP App for Gemini Enterprise using the create-mcp-app skill (https://github.com/modelcontextprotocol/ext-apps/tree/main/plugins/mcp-apps/skills/create-mcp-app).&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Workflow: [describe your input data, side-panel UI controls, and approval or export actions].&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Generate a single-file HTML widget, a Streamable HTTP backend (POST /mcp) with &amp;lt;2s async tool responses and get_app_state polling, and Cloud Run deployment commands (--no-allow-unauthenticated, --no-cpu-throttling, --min-instances=1).&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Once your MCP server code is ready, deploy it to Cloud Run and connect it to Gemini Enterprise&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Step 1: Enable APIs&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; for Cloud Run, Discovery Engine, Agent Platform, and Organization Policy.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Step 2: Deploy your MCP server container to Cloud Run from source &lt;/strong&gt;&lt;code style="vertical-align: baseline;"&gt;(gcloud run deploy &amp;lt;service&amp;gt; --source .&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;)&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; with &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;--no-allow-unauthenticated, --no-cpu-throttling&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; so background work does not freeze between polls, and &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;--min-instances=1&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Step 3: Grant the invoker role&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; (&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;roles/run.invoker&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;) to your Discovery Engine service agent (Layer 1).&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;&lt;strong style="vertical-align: baseline;"&gt;Step 4: Create an OAuth 2.0 Web Client&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; with redirect URI &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;https://vertexaisearch.cloud.google.com/oauth-redirect&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;(Layer 2).&lt;/span&gt;&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Step 5: &lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;Override the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;discoveryengine.managed.disableCustomMcpServerConnector&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; constraint and complete the&lt;/span&gt;&lt;a href="https://cloud.google.com/gemini/enterprise/docs/connectors/custom-mcp-server/set-up-custom-mcp-server#before-you-begin"&gt;&lt;span style="vertical-align: baseline;"&gt; &lt;/span&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Before you begin&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;prerequisites &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;(roles/discoveryengine.editor&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; for the admin,&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; allowedEgressFqdns&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; for your server and OAuth hosts, and&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;custom_mcp&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;in&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;allowedDataSources&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; if enforced). Then, in the Google Cloud Console under Gemini Enterprise &amp;gt; Data stores &amp;gt; Create data store, select Custom MCP Server and register the connector:&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;div align="left"&gt;
&lt;div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;"&gt;
&lt;div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;"&gt;
&lt;div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;"&gt;
&lt;div style="color: #5f6368; overflow-x: auto; overflow-y: hidden; width: 100%;"&gt;&lt;table style="width: 98.1723%;"&gt;&lt;colgroup&gt;&lt;col style="width: 34.0211%;"/&gt;&lt;col style="width: 65.9803%;"/&gt;&lt;/colgroup&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th scope="col" style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Configuration field&lt;/strong&gt;&lt;/p&gt;
&lt;/th&gt;
&lt;th scope="col" style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Value&lt;/strong&gt;&lt;/p&gt;
&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;MCP Server URL&lt;/strong&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;code style="vertical-align: baseline;"&gt;https://&amp;lt;your-cloud-run-service&amp;gt;.run.app/mcp&lt;/code&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Authorization URL&lt;/strong&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;code style="vertical-align: baseline;"&gt;https://accounts.google.com/o/oauth2/v2/auth&lt;/code&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Token URL&lt;/strong&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;code style="vertical-align: baseline;"&gt;https://oauth2.googleapis.com/token&lt;/code&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Client ID and Secret&lt;/strong&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Your Web OAuth 2.0 Client ID and Secret&lt;/span&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Scopes&lt;/strong&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;code style="vertical-align: baseline;"&gt;openid email profile&lt;/code&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Enable PKCE Support&lt;/strong&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Checked (recommended)&lt;/strong&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Use HTTP Basic Auth&lt;/strong&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;td style="vertical-align: top; border: 1px solid #000000; padding: 16px;"&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Unchecked&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; (avoids &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;invalid_client&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; errors)&lt;/span&gt;&lt;/p&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Step 6: &lt;/strong&gt;&lt;a href="https://cloud.google.com/gemini/enterprise/docs/connectors/custom-mcp-server/set-up-custom-mcp-server#enable-actions"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;Enable actions&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; on your data store. Gemini Enterprise imports your MCP server's tools as actions and disables them by default. In the Cloud Console, open Gemini Enterprise &amp;gt; Data stores &amp;gt; [Your MCP Data Store] &amp;gt; Actions, click Reload custom actions (which calls&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;tools/list&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; on your server), select your model-visible tools, and click Enable actions.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/4-Step_6_enable_actions.max-1000x1000.png"
        
          alt="4-Step 6 enable actions"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;ul&gt;
&lt;li&gt;&lt;strong style="vertical-align: baseline;"&gt;Step 7: &lt;/strong&gt;&lt;a href="https://cloud.google.com/gemini/enterprise/docs/connectors/connect-existing-data-store"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;Connect the data store to your app&lt;/strong&gt;&lt;/a&gt;&lt;strong style="vertical-align: baseline;"&gt;. &lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;Open &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Gemini Enterprise &amp;gt; Apps &amp;gt; [Your App] &amp;gt; Connected data stores &amp;gt; Add existing data stores&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;, and select your MCP data store. Connecting it as a data store gives the built-in Default Assistant access to your MCP tools and UI widget directly in main chat ( do not use "Add Agent,"  which is for external sub-agents rather than MCP data stores).&lt;/span&gt;&lt;/li&gt;
&lt;/ul&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/5-Step_7_connect_datastore.max-1000x1000.png"
        
          alt="5-Step 7 connect datastore"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Build your own&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;An MCP App pairs backend tools with a live workspace in Gemini Enterprise. You do not build a chat frontend or identity pipeline; you deploy one Cloud Run container and attach it as a Data Store, making Gemini Enterprise where enterprise work gets done.&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Try Gemini &lt;/strong&gt;&lt;a href="https://auth.business.gemini.google/login?continueUrl=https://business.gemini.google/?utm_source%3Dcloud.google.com/blog%26referrer%3Dhttps://www.google.com/&amp;amp;wiffid=CAoSJGE0ZDBiMWMwLWVmYTUtNGExNy1iNTUzLWM0ODFmNGY4NmJjNQ"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;here&lt;/strong&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Co-build with Google Antigravity &lt;/strong&gt;&lt;a href="https://antigravity.google/" rel="noopener" target="_blank"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;here&lt;/strong&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Build on Google Cloud:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; &lt;/span&gt;&lt;a href="https://github.com/modelcontextprotocol/ext-apps" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;MCP Apps Specification (SEP-1865)&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;a href="https://cloud.google.com/run"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Cloud Run&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, and &lt;/span&gt;&lt;a href="https://cloud.google.com/ai"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Google Cloud AI&lt;/span&gt;&lt;/a&gt;&lt;strong style="vertical-align: baseline;"&gt;.&lt;/strong&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;&lt;/div&gt;</description><pubDate>Mon, 19 Oct 2026 16:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/topics/developers-practitioners/devs-guide-to-mcp-apps-in-gemini/</guid><category>Developers &amp; Practitioners</category><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Developer’s guide to implementing MCP apps in Gemini</title><description></description><site_name>Google</site_name><url>https://cloud.google.com/blog/topics/developers-practitioners/devs-guide-to-mcp-apps-in-gemini/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Dave Wang</name><title>Forward Deployed Engineer, Google Cloud</title><department></department><company></company></author></item><item><title>Four evaluation engineering lessons learned from building agent plugins</title><link>https://cloud.google.com/blog/topics/developers-practitioners/four-eval-engineering-lessons/</link><description>&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The real challenge of building an agentic system isn't the construction, it's the measurement.&lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt; &lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Evaluation engineering is a critical foundation for building reliable AI systems. Setting up systematic, repeatable tests to measure your agent’s true behavior can be the difference between shipping a tool you can trust and one that silently breaks, fails to call APIs, or makes mistakes. &lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;I’ve run evaluation engineering initiatives for years at Google and d&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;eveloping the newly released&lt;/span&gt;&lt;a href="https://cloud.google.com/blog/topics/developers-practitioners/introducing-the-google-cloud-developer-plugin-for-ai-coding-agents?e=48754805"&gt;&lt;span style="vertical-align: baseline;"&gt; &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;Google Cloud Developer plugin&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; served as a live testbed to stress-test these key principles across real-world developer workflows.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Here are four eval lessons we learned from building the new plugin that you can apply to your own builds.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;1. Grade the execution steps instead of the summary&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;When we first evaluated agents, we set up an LLM-as-judge to inspect the agent’s final text reply and scored it against a checklist. However, we quickly realized it’s fundamentally flawed because the grader only sees what the agent wrote, not what it actually did. &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;When we ran our initial 11-prompt evaluation suite for the Google Cloud Developer plugin, the automated grader marked it as failing three prompts. But when we opened the raw tool-call logs and read the transcripts, the plugin had actually passed all three. On a Cloud Run cleanup prompt, for instance, our rubric required checking command syntax before running anything. &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;The agent ran &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;gcloud help run services list&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; as its very first step. It did the work, but its final message was focused on warning the user about permanent data loss rather than reciting that it had checked the help docs.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/Image_1_XmJ1pt2.max-1000x1000.png"
        
          alt="Image 1"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The grader penalized the agent for doing the right thing simply because it didn’t brag about it in the summary. If your rubric line says "did it run X?", a LLM judge is guessing unless you evaluate the actual tool-call trajectory.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Takeaway&lt;/strong&gt;&lt;strong style="vertical-align: baseline;"&gt;:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Don't grade an agent on its closing pitch. Score the raw execution trajectory to verify what commands were actually executed, validated, and scoped.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;2. Design evals for asynchronous operations&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;When you start reading tool-call trajectories, how you extract that telemetry matters just as much as what you're reading. Extracting agent telemetry into a flattened, sequential list of events obscures concurrent tool dispatch, which is one of the critical causes of behavioral failure. Modern agent harnesses often batch multiple tool invocations into a single assistant turn using an array of calls.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;In our evaluations, when an agent was asked to inspect an environment and look up documentation, the harness dispatched the skill-loading call and two documentation searches simultaneously within a single tool_calls payload. Because activating a skill is a multi-stage process, where the initial call returns a short launch stub and the subsequent turn injects the full instruction body, the instructions meant to govern the search arrived in the prompt context after the system had already executed the searches.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/image_2_CHyk3qC.max-1000x1000.png"
        
          alt="image 2"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The system never gave the skill a dedicated turn to let its guidance influence the agent's actions. If telemetry pipelines strip out turn structures and merge calls into a flat timeline, engineers will waste days wondering why an agent disregarded unambiguous instructions that a developer embedded in a skill.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Takeaway&lt;/strong&gt;&lt;strong style="vertical-align: baseline;"&gt;: &lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;Preserve turn structure in telemetry extractors so you can identify when client-side parallel dispatch issues tool calls before injected guidance enters the context window. &lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;3. Test live runs and "dry runs" side-by-side&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;You need to inspect real API calls to catch harness quirks, like parallel dispatch. But if you only test your agent in one environment condition, you're flying half-blind. From day one, you need to run two conditions side-by-side: a live run and a plan-only run. &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;In the live condition, the agent has real, sandboxed credentials and actually executes commands. This is how you see if syntax works, parameters serialize cleanly, and the agent exercises restraint before doing something destructive. &lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Live runs frequently hit an unexpected permission error on step two and abort before you ever reach the behavior you’re trying to test. That's why you need the plan-only condition, where you strip credentials and tell the agent to explain what it would do without running anything. This strips out permission roadblocks and shows you the model’s pure reasoning and routing intent. Roughly half our most important findings came from the delta between the two.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/image_3_OixTmu5.max-1000x1000.png"
        
          alt="image 3"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;For example, on one of our onboarding prompts, a live run was blocked immediately on a project list call. The plan-only run sailed right through and produced an 8,000-character plan that completely ignored organization policies, admin approvals, and governance. The live run masked a major cognitive flaw that the plan-only run exposed in seconds.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Takeaway&lt;/strong&gt;&lt;strong style="vertical-align: baseline;"&gt;:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Run live and plan-only evaluations side by side. Live runs test actual execution and safety restraint; plan-only runs expose the model's raw reasoning and routing without getting tripped up by permission errors.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;4. Use “golden sets” as durable baselines&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Frontier model architectures change every few months, agent harnesses improve, and plugin manifest standards will inevitably evolve. When building the Google Cloud Developer plugin, we treated evaluations as continuous integration testing for these non-deterministic environments. That means we stress-tested the composite stack, including the harness, the live model, and the tools running together on realistic tasks.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Instead of guessing from the final text output, this approach captures the whole trajectory: what the agent thought, what commands it actually ran, and whether it solved the problem without breaking anything. Then you score that run against golden sets curated by real human experts who know what the ideal path looks like.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Takeaway&lt;/strong&gt;&lt;strong style="vertical-align: baseline;"&gt;: &lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;Build golden sets based on what the entire system should actually do in the wild, including your harness, model, and plugins. Models change, but those baseline runs are your safety net.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Reminder: Use evaluation suites over “one-shot” demos&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;A common mistake I see among engineers is what I like to call the one-shot demo. This is when someone feeds a prompt into a LLM, watches it build a complete 3D game or solves a Rubik’s cube in a single pass and calls it an eval. But in reality, this is just a vanity metric. &lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;To do quality work with AI, I truly believe that you need evaluation suites. Without evaluations, teams are like literally running into a dark cave blind. Engineering an enterprise-grade agent means testing the system's ability to handle multi-step interactions, manage permissions, and avoid destructive outcomes like accidentally deleting a project because you asked it to "find opportunities to save costs."&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Where to go from here&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Building an agent plugin or MCP server is the beginning. Shipping something you can actually trust with real cloud infrastructure requires significantly more work in evaluations to ensure it’s doing what you expect it to do. At the end of the day, the only thing that outlasts model churn is a stress-tested eval suite. &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;Check out the Google Cloud Developer plugin &lt;/span&gt;&lt;a href="https://g.dev/cloud/dev-setup" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;documentation&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; or explore this &lt;/span&gt;&lt;a href="https://g.dev/cloud/agent-plugins-codelab-agy" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;codelab&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; to get started with the plugin in Antigravity.&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; &lt;/span&gt;&lt;/p&gt;&lt;/div&gt;</description><pubDate>Fri, 16 Oct 2026 16:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/topics/developers-practitioners/four-eval-engineering-lessons/</guid><category>AI &amp; Machine Learning</category><category>Developers &amp; Practitioners</category><media:content height="540" url="https://storage.googleapis.com/gweb-cloudblog-publish/original_images/blog-post-banner_4eel_v2_2.png.jpg" width="540"></media:content><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Four evaluation engineering lessons learned from building agent plugins</title><description></description><image>https://storage.googleapis.com/gweb-cloudblog-publish/original_images/blog-post-banner_4eel_v2_2.png.jpg</image><site_name>Google</site_name><url>https://cloud.google.com/blog/topics/developers-practitioners/four-eval-engineering-lessons/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Casey West</name><title>Developer Relations Lead, AI Infrastructure &amp; Applied Engineering, Google Cloud</title><department></department><company></company></author></item><item><title>One Agent Development Kit, five languages</title><link>https://cloud.google.com/blog/topics/developers-practitioners/one-agent-development-kit-five-languages/</link><description>&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;When we talk about agents, that doesn't necessarily mean we're talking about Python. In today's modern ecosystem, an agent might run as a high-concurrency Go microservice on &lt;/span&gt;&lt;a href="https://cloud.google.com/run"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Cloud Run&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, or &lt;/span&gt;&lt;a href="https://github.com/google/adk-js/blob/main/core/src/models/chrome_prompt_llm.ts" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;live inside a browser tab&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; calling Chrome's on-device model. Yet many agent frameworks assume a Python-only stack. That assumption forces teams to either spin up a Python service just for their agent workloads, or to build agent orchestration logic from scratch in their own language.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Google's &lt;/span&gt;&lt;a href="https://adk.dev/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Agent Development Kit (ADK)&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; &lt;span style="vertical-align: baseline;"&gt;removes that trade-off with &lt;/span&gt;coverage for &lt;/span&gt;&lt;a href="https://github.com/google/adk-python" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Python&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;a href="https://github.com/google/adk-go" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Go&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;a href="https://github.com/google/adk-js" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;TypeScript&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;a href="https://github.com/google/adk-java" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Java&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, and &lt;/span&gt;&lt;a href="https://github.com/google/adk-kotlin" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Kotlin&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;. Each version of the ADK is built from the ground up for its own runtime and type system, while sharing a unified architecture with interoperability through open protocols.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;A shared foundation&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Picking a language doesn't lock in your architecture: all five support the same open protocols. &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;ADK includes a &lt;/span&gt;&lt;a href="https://modelcontextprotocol.io/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Model Context Protocol&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; client in every language, so a custom tool exposed as an MCP server can be called from any of them immediately.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;For multi-agent systems, ADK supports the &lt;/span&gt;&lt;a href="https://github.com/a2aproject/A2A" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Agent2Agent (A2A)&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; protocol, allowing an agent to communicate with a remote agent over HTTP. Through A2A, agents written in different languages or frameworks are interoperable.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Starting agent-first&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;To get started, run the one-time setup command to check your Google Cloud authentication, install the CLI, and then set up first-party ADK skills in coding agents like &lt;/span&gt;&lt;a href="https://antigravity.google/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Antigravity&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;a href="https://docs.anthropic.com/en/docs/claude-code" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Claude Code&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, or &lt;/span&gt;&lt;a href="https://github.com/openai/codex" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Codex&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;uvx google-agents-cli setup&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9bbd90&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Then, run the scaffold command to generate a working project with A2A endpoints and deployment config for Cloud Run, &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/gemini-enterprise-agent-platform/build/runtime"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Gemini Enterprise Agent Runtime&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, or &lt;/span&gt;&lt;a href="https://cloud.google.com/kubernetes-engine"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Google Kubernetes Engine (GKE)&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;# Scaffold an ADK project (use --agent adk for Python or --agent adk_go for Go)\r\nagents-cli scaffold create my-go-agent \\\r\n  --agent adk_go \\\r\n  --deployment-target cloud_run \\\r\n  --region us-central1&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9bb1d0&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Both the &lt;/span&gt;&lt;a href="https://github.com/google/adk-python/tree/main/.agents/skills/adk-agent-builder" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;ADK for Python&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; and &lt;/span&gt;&lt;a href="https://github.com/google/adk-kotlin/tree/main/.agents/skills/adk-kotlin-agent-builder" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;ADK for Kotlin&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; repositories also ship agent builder skills in their source trees, and the &lt;/span&gt;&lt;a href="https://adk.dev/llms.txt" rel="noopener" target="_blank"&gt;&lt;code style="text-decoration: underline; vertical-align: baseline;"&gt;adk.dev/llms.txt&lt;/code&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; resource gives any coding agent on-demand API documentation across all five languages.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/one-agent-development-kit-five-languages-a.max-1000x1000.jpg"
        
          alt="one-agent-development-kit-five-languages-architecture"&gt;
        
        &lt;/a&gt;
      
        &lt;figcaption class="article-image__caption "&gt;&lt;p data-block-key="9ieq2"&gt;Coding-agent skills scaffold your project while MCP and A2A connect all five language runtimes.&lt;/p&gt;&lt;/figcaption&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Hands-on view of each ADK&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The clearest way to feel the difference between the five languages is to define the same weather tool in each one and see how its design fits the runtime.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Python&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;In &lt;/span&gt;&lt;a href="https://github.com/google/adk-python/blob/main/contributing/samples/core/quickstart/agent.py" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;ADK for Python&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, you don't need a decorator or a schema builder class. You pass a plain Python function into the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;tools&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; parameter, and ADK inspects its signature, type hints, and docstring at startup to automatically construct the tool declaration and &lt;/span&gt;&lt;a href="https://docs.pydantic.dev/latest/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Pydantic&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; parameter schema:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;from google.adk.agents.llm_agent import Agent\r\n\r\n\r\ndef get_weather(city: str) -&amp;gt; dict:\r\n  &amp;quot;&amp;quot;&amp;quot;Retrieves the current weather report for a specified city.\r\n\r\n  Args:\r\n      city (str): The name of the city for which to retrieve the weather report.\r\n  &amp;quot;&amp;quot;&amp;quot;\r\n  return {&amp;quot;status&amp;quot;: &amp;quot;success&amp;quot;, &amp;quot;report&amp;quot;: f&amp;quot;Sunny and 72F in {city}.&amp;quot;}\r\n\r\n\r\nroot_agent = Agent(\r\n    name=&amp;quot;weather_agent&amp;quot;,\r\n    model=&amp;quot;gemini-3.8-flash&amp;quot;,\r\n    instruction=&amp;quot;Answer questions about the weather in a city.&amp;quot;,\r\n    tools=[get_weather],\r\n)&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;lang-py&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9bb790&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Beyond fast prototyping, Python is ADK's reference implementation, shipping the broadest set of Google Cloud integrations (&lt;/span&gt;&lt;a href="https://cloud.google.com/bigquery"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;BigQuery&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;a href="https://cloud.google.com/spanner"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Spanner&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;a href="https://cloud.google.com/bigtable"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Bigtable&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;) and model adapters (&lt;/span&gt;&lt;a href="https://ai.google.dev/gemini-api/docs" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Gemini&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, Anthropic, OpenAI, LiteLLM). Paired with the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;adk eval&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; or &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;agents-cli eval run&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; commands&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;, its evaluation tooling scores agent trajectories against golden datasets in continuous integration (CI) to test the whole system end to end, even when your production service runs in another language.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Go&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;While Python leads in experimentation and evaluation, &lt;/span&gt;&lt;a href="https://github.com/google/adk-go/blob/main/examples/quickstart/main.go" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;ADK for Go&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; hits the sweet spot for production backend engineering.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;A Go tool function takes the &lt;/span&gt;&lt;a href="https://pkg.go.dev/google.golang.org/adk/v2/agent#Context" rel="noopener" target="_blank"&gt;&lt;code style="text-decoration: underline; vertical-align: baseline;"&gt;agent.Context&lt;/code&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; configuration and a strongly typed input struct. The &lt;/span&gt;&lt;a href="https://pkg.go.dev/google.golang.org/adk/v2/tool/functiontool#New" rel="noopener" target="_blank"&gt;&lt;code style="text-decoration: underline; vertical-align: baseline;"&gt;functiontool.New&lt;/code&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; function uses Go generics to infer both the input and output schemas from your &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;struct&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; tags:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;type GetWeatherArgs struct {\r\n\tCity string `json:&amp;quot;city&amp;quot; jsonschema:&amp;quot;The city to look up weather for.&amp;quot;`\r\n}\r\n\r\ntype GetWeatherResult struct {\r\n\tReport string `json:&amp;quot;report&amp;quot;`\r\n}\r\n\r\nfunc getWeather(_ agent.Context, args GetWeatherArgs) (GetWeatherResult, error) {\r\n\treturn GetWeatherResult{Report: &amp;quot;Sunny and 72F in &amp;quot; + args.City}, nil\r\n}\r\n\r\nweatherTool, err := functiontool.New(functiontool.Config{\r\n\tName:        &amp;quot;get_weather&amp;quot;,\r\n\tDescription: &amp;quot;Retrieves the current weather report for a city.&amp;quot;,\r\n}, getWeather)&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9b94d0&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Because the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;functiontool.New&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; function is generic over your handler's signature, the Go compiler verifies that your function matches the expected contract before your binary ever runs.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Go was also the first language after Python to ship the &lt;/span&gt;&lt;a href="https://pkg.go.dev/google.golang.org/adk/v2/workflow" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;ADK 2.0 Graph Workflow runtime&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;. When you need deterministic control flow alongside LLM reasoning, the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;workflow.New&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; function wires agents and plain Go functions into a directed graph with conditional routing and parallel steps:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;ag, err := workflowagent.New(&amp;quot;order_pipeline&amp;quot;, []workflow.Edge{\r\n\t{From: workflow.Start, To: triageNode},\r\n\t{From: triageNode, To: fulfillNode, Route: workflow.StringRoute(&amp;quot;fulfill&amp;quot;)},\r\n\t{From: fulfillNode, To: notifyNode},\r\n})&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9baf90&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Where Go really stands out is in production on Cloud Run and GKE. A compiled Go agent packages onto a roughly 2 MiB &lt;/span&gt;&lt;a href="https://github.com/GoogleContainerTools/distroless" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Google distroless static base image&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, with no language VM or heavy dependency tree. A static Go binary binds to its port as soon as the container starts, skipping the module imports of &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/run/docs/tips/general"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;dynamic languages&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; and the JAR unpacking and JIT warmup of &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/run/docs/tips/java"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;JVM services&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;After it's running, Go's concurrency model fits agent workloads naturally. Each goroutine starts with a &lt;/span&gt;&lt;a href="https://go.dev/doc/go1.4#runtime" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;2 KB stack&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; and &lt;/span&gt;&lt;a href="https://go.dev/doc/faq#goroutines" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;minimal scheduling overhead&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, so a single instance can multiplex thousands of concurrent streaming requests and tool calls. ADK for Go also includes &lt;/span&gt;&lt;a href="https://pkg.go.dev/google.golang.org/adk/v2/server/adka2a/v2" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;built-in A2A server handlers&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; and OpenTelemetry tracing out of the box.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;TypeScript&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;TypeScript erases type annotations when compiling to JavaScript, so runtime reflection isn't an option. Instead, &lt;/span&gt;&lt;a href="https://github.com/google/adk-js/blob/main/core/src/tools/function_tool.ts" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;ADK for TypeScript&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; pairs with &lt;/span&gt;&lt;a href="https://zod.dev/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Zod&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;. You pass a &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;z.object(...)&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; schema to the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;FunctionTool&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; function, and TypeScript statically infers the parameter types of your &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;execute&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; callback from that single declaration:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;import { FunctionTool, LlmAgent } from &amp;quot;@google/adk&amp;quot;;\r\nimport { z } from &amp;quot;zod&amp;quot;;\r\n\r\nconst getWeatherTool = new FunctionTool({\r\n  name: &amp;quot;get_weather&amp;quot;,\r\n  description: &amp;quot;Retrieves the current weather report for a city.&amp;quot;,\r\n  parameters: z.object({\r\n    city: z.string().describe(&amp;quot;The name of the city.&amp;quot;),\r\n  }),\r\n  execute: async ({ city }) =&amp;gt; {\r\n    return { status: &amp;quot;success&amp;quot;, report: `Sunny and 72F in ${city}.` };\r\n  },\r\n});\r\n\r\nexport const rootAgent = new LlmAgent({\r\n  name: &amp;quot;weather_agent&amp;quot;,\r\n  model: &amp;quot;gemini-3.8-flash&amp;quot;,\r\n  instruction: &amp;quot;Answer questions about the weather in a city.&amp;quot;,\r\n  tools: [getWeatherTool],\r\n});&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9bb750&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Your runtime input validation, the JSON Schema sent to the model, and your IDE autocomplete inside the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;execute&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;callback &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;all flow from one source of truth.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;TypeScript also runs on the ADK 2.0 Graph Workflow engine and ships a &lt;/span&gt;&lt;a href="https://github.com/google/adk-js/blob/main/core/src/index_web.ts" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;standalone browser bundle&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;. Beyond Node.js services, you can run an ADK agent directly in a browser tab using &lt;/span&gt;&lt;a href="https://github.com/google/adk-js/blob/main/core/src/tools/webmcp/webmcp_toolset.ts" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;WebMCP&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; for page-level tools and Chrome's on-device &lt;/span&gt;&lt;a href="https://developer.chrome.com/docs/ai/prompt-api" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Prompt API&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Java&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;a href="https://github.com/google/adk-java/blob/main/tutorials/city-time-weather/src/main/java/com/google/adk/tutorials/CityTimeWeather.java" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;ADK for Java&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; snaps right in for teams who have standardized on Maven, Gradle, and Spring Boot. You add&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;@Schema&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; annotations to method parameters and assemble agents by using a fluent &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;LlmAgent.builder()&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; API:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;public class WeatherTools {\r\n  public static Map&amp;lt;String, String&amp;gt; getWeather(\r\n      @Schema(name = &amp;quot;city&amp;quot;, description = &amp;quot;The name of the city&amp;quot;)\r\n      String city) {\r\n    return Map.of(&amp;quot;status&amp;quot;, &amp;quot;success&amp;quot;, &amp;quot;report&amp;quot;, &amp;quot;Sunny and 72F in &amp;quot; + city);\r\n  }\r\n}\r\n\r\nLlmAgent agent = LlmAgent.builder()\r\n    .name(&amp;quot;weather_agent&amp;quot;)\r\n    .model(&amp;quot;gemini-3.8-flash&amp;quot;)\r\n    .instruction(&amp;quot;Answer questions about the weather in a city.&amp;quot;)\r\n    .tools(FunctionTool.create(WeatherTools.class, &amp;quot;getWeather&amp;quot;))\r\n    .build();&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9b8ed0&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;At runtime, the runner streams agent events reactively through &lt;/span&gt;&lt;a href="https://github.com/ReactiveX/RxJava" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;RxJava&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;. For local testing and existing JVM apps, ADK for Java ships a Maven Dev UI plugin alongside first-party bridges for Spring AI and LangChain4j.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Kotlin&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;a href="https://github.com/google/adk-kotlin/blob/main/examples/src/main/kotlin/com/google/adk/kt/examples/tools/FunctionToolDemoAgent.kt" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;ADK for Kotlin&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; takes a strong stance on build-time safety. Its &lt;/span&gt;&lt;a href="https://kotlinlang.org/docs/ksp-overview.html" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Kotlin Symbol Processing (KSP)&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; compiler plugin reads &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;@Tool&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; and &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;@Param&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; annotations at build time and emits the tool schema directly into bytecode:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;class WeatherTools {\r\n  @Tool(\r\n    name = &amp;quot;get_weather&amp;quot;,\r\n    description = &amp;quot;Retrieves the current weather report for a city.&amp;quot;,\r\n  )\r\n  suspend fun getWeather(\r\n    @Param(&amp;quot;The name of the city.&amp;quot;) city: String\r\n  ): Map&amp;lt;String, String&amp;gt; = mapOf(&amp;quot;status&amp;quot; to &amp;quot;success&amp;quot;, &amp;quot;report&amp;quot; to &amp;quot;Sunny and 72F in $city&amp;quot;)\r\n}\r\n\r\nval agent = LlmAgent(\r\n  model = Gemini(&amp;quot;gemini-3.8-flash&amp;quot;),\r\n  name = &amp;quot;weather_agent&amp;quot;,\r\n  instruction = Instruction(&amp;quot;Answer questions about the weather in a city.&amp;quot;),\r\n  tools = WeatherTools().generatedTools(),\r\n)&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9bb850&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;If a tool signature uses an unsupported type or declares a default value on a non-nullable parameter, your Gradle build fails immediately instead of breaking in production. Because the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;.generatedTools()&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; function uses zero runtime reflection, it survives &lt;/span&gt;&lt;a href="https://developer.android.com/topic/performance/app-optimization/enable-app-optimization" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Android R8 shrinking&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;That zero-reflection design makes it possible to run the entire agent loop locally on Android. It offers JVM server support, including a Spring AI bridge and experimental ADK 2.0 graph workflows. On-device, ADK for Kotlin integrates with &lt;/span&gt;&lt;a href="https://ai.google.dev/edge/litert-lm" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;LiteRT-LM&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; for tool calling and &lt;/span&gt;&lt;a href="https://developers.google.com/ml-kit/genai" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;ML Kit&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; (Gemini Nano) for text generation. It also works with Room and AppSearch for local state, and Android AppFunctions for OS-level actions, all without requiring a network connection.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Choice without compromise&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Having five native languages in ADK means you don't have to pick a single runtime for your entire organization. Because every version of ADK speaks MCP and A2A, each component of your system can use the language that fits it best.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;What languages have you used to build agents with Google's ADK? Let me know what stack you're running on &lt;/span&gt;&lt;a href="https://x.com/kweinmeister" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;X&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;a href="https://www.linkedin.com/in/karlweinmeister/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;LinkedIn&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, or &lt;/span&gt;&lt;a href="https://bsky.app/profile/kweinmeister.bsky.social" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Bluesky&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;</description><pubDate>Tue, 13 Oct 2026 12:22:00 +0000</pubDate><guid>https://cloud.google.com/blog/topics/developers-practitioners/one-agent-development-kit-five-languages/</guid><category>Developers &amp; Practitioners</category><media:content height="540" url="https://storage.googleapis.com/gweb-cloudblog-publish/images/one-agent-development-kit-five-languages-her.max-600x600.jpg" width="540"></media:content><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>One Agent Development Kit, five languages</title><description></description><image>https://storage.googleapis.com/gweb-cloudblog-publish/images/one-agent-development-kit-five-languages-her.max-600x600.jpg</image><site_name>Google</site_name><url>https://cloud.google.com/blog/topics/developers-practitioners/one-agent-development-kit-five-languages/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Karl Weinmeister</name><title>Director, Developer Relations</title><department></department><company></company></author></item><item><title>10 Years of the SRE book: Announcing the second edition of "Site Reliability Engineering”</title><link>https://cloud.google.com/blog/products/devops-sre/sre-book-second-edition/</link><description>&lt;div class="block-paragraph_advanced"&gt;&lt;div data-draftjs-conductor-fragment="{&amp;quot;blocks&amp;quot;:[{&amp;quot;key&amp;quot;:&amp;quot;3j2ba&amp;quot;,&amp;quot;text&amp;quot;:&amp;quot;A decade ago, Google released Site Reliability Engineering: How Google Runs Production Systems with O'Reilly Media. What began as a collection of our practices for managing large-scale infrastructure became a bestselling book, and inspired many organizations to adopt the reliability practices it outlined: applying software engineering to operations to build systems that are scalable, reliable, and sustainable.&amp;quot;,&amp;quot;type&amp;quot;:&amp;quot;unstyled&amp;quot;,&amp;quot;depth&amp;quot;:0,&amp;quot;inlineStyleRanges&amp;quot;:[{&amp;quot;offset&amp;quot;:30,&amp;quot;length&amp;quot;:64,&amp;quot;style&amp;quot;:&amp;quot;ITALIC&amp;quot;}],&amp;quot;entityRanges&amp;quot;:[],&amp;quot;data&amp;quot;:{}},{&amp;quot;key&amp;quot;:&amp;quot;6v23u&amp;quot;,&amp;quot;text&amp;quot;:&amp;quot;Over the past ten years, computing landscapes have undergone profound shifts. AI, microservices, multi-cloud deployments, serverless paradigms, and platform engineering have all reshaped how we build and operate software. To mark the 10th anniversary of the book's release, we are excited to announce the completely rewritten second edition of the Google SRE book, reflecting a decade of lessons learned both inside Google and across the broader industry, to offer fresh guidance tailored for today's operational challenges.&amp;quot;,&amp;quot;type&amp;quot;:&amp;quot;unstyled&amp;quot;,&amp;quot;depth&amp;quot;:0,&amp;quot;inlineStyleRanges&amp;quot;:[{&amp;quot;offset&amp;quot;:305,&amp;quot;length&amp;quot;:20,&amp;quot;style&amp;quot;:&amp;quot;ITALIC&amp;quot;}],&amp;quot;entityRanges&amp;quot;:[],&amp;quot;data&amp;quot;:{}},{&amp;quot;key&amp;quot;:&amp;quot;37phi&amp;quot;,&amp;quot;text&amp;quot;:&amp;quot;&amp;quot;,&amp;quot;type&amp;quot;:&amp;quot;header-three&amp;quot;,&amp;quot;depth&amp;quot;:0,&amp;quot;inlineStyleRanges&amp;quot;:[],&amp;quot;entityRanges&amp;quot;:[],&amp;quot;data&amp;quot;:{}}],&amp;quot;entityMap&amp;quot;:{}}"&gt;
&lt;div class="Draftail-block--unstyled" data-block="true" data-editor="12ehc" data-offset-key="1woq2-0-0"&gt;
&lt;div class="public-DraftStyleDefault-block public-DraftStyleDefault-ltr" data-offset-key="1woq2-0-0"&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;A decade ago, Google released &lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;Site Reliability Engineering: How Google Runs Production Systems&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; with O'Reilly Media. What began as a collection of our practices for managing large-scale infrastructure became a bestselling book, and inspired many organizations to adopt the reliability practices it outlined: applying software engineering to operations to build systems that are scalable, reliable, and sustainable.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Over the past ten years, computing landscapes have undergone profound shifts. AI, microservices, multi-cloud deployments, serverless paradigms, and platform engineering have all reshaped how we build and operate software. To mark the 10th anniversary of the book's release, we are excited to announce the &lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;completely rewritten&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; second edition of the Google SRE book, reflecting a decade of lessons learned both inside Google and across the broader industry, to offer fresh guidance tailored for today's operational challenges.&lt;/span&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;/div&gt;&lt;/div&gt;
&lt;div class="block-paragraph_with_image"&gt;&lt;div class="article-module h-c-page"&gt;
  &lt;div class="h-c-grid uni-paragraph-wrap"&gt;
    &lt;div class="uni-paragraph
      h-c-grid__col h-c-grid__col--8 h-c-grid__col-m--6 h-c-grid__col-l--6
      h-c-grid__col--offset-2 h-c-grid__col-m--offset-3 h-c-grid__col-l--offset-3"&gt;

      






  

    &lt;figure class="article-image--wrap-small
      
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/SRE_Book_2e_Front_Cover.max-1000x1000.jpg"
        
          alt="SRE Book 2e Front Cover"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  





      &lt;h3 data-block-key="1woq2"&gt;&lt;b&gt;Why update the book?&lt;/b&gt;&lt;/h3&gt;&lt;p data-block-key="4ebln"&gt;When we published the first edition, many organizations were focused on early cloud migrations and just starting to transition from monolithic architectures. Today, with organizations delivering software at a high velocity and at global scale, the challenge is no longer adopting cloud infrastructure, but maintaining high reliability in the face of the immense complexity.&lt;/p&gt;&lt;p data-block-key="fov6p"&gt;Modern production environments operate under constant change. System boundaries are more complex, dependencies are deep, and failure modes are more complex and damaging than ever before. The explosion of resource-intensive workloads like generative AI has changed the economics and operational requirements of infrastructure management.&lt;/p&gt;&lt;p data-block-key="au36"&gt;Writing this second edition required us to take a step back and evaluate what it takes to successfully implement SRE today.&lt;/p&gt;&lt;h3 data-block-key="2ne5h"&gt;&lt;b&gt;A decade of evolution: How SRE practices have progressed&lt;/b&gt;&lt;/h3&gt;&lt;p data-block-key="biefd"&gt;The second edition takes a deep dive into how foundational site reliability engineering practices have matured over the last ten years, shifting from manual, task-specific work toward automated, platform-wide capabilities. Consider, for example:&lt;/p&gt;&lt;p data-block-key="aset9"&gt;&lt;b&gt;SLOs and error budgets:&lt;/b&gt; In 2016, teams relied on basic availability metrics and manual error budget policies. Today, practices have matured into user-centric, multi-tiered SLOs with dynamic burn-rate alerting and automated policy enforcement.&lt;/p&gt;&lt;p data-block-key="7all2"&gt;&lt;b&gt;Monitoring and observability:&lt;/b&gt; Early setups centered on static dashboards, basic logs, and manual alert triage. Modern observability leverages OpenTelemetry, distributed tracing, and automated anomaly correlation to find root causes quickly.&lt;/p&gt;&lt;p data-block-key="fh1tc"&gt;&lt;b&gt;Automation and platforms:&lt;/b&gt; Operations have shifted from writing custom scripts for isolated tasks to using declarative infrastructure as code, internal developer platforms, and resilient, self-healing systems.&lt;/p&gt;&lt;p data-block-key="474b5"&gt;&lt;b&gt;Reliability testing:&lt;/b&gt; Rather than relying solely on scheduled game days and synthetic load tests, modern engineering embeds continuous resilience checks, automated pipeline testing, and live production canary verification.&lt;/p&gt;&lt;p data-block-key="cdtqm"&gt;&lt;b&gt;Capacity and resource management:&lt;/b&gt; Manual provisioning and static forecasting have been replaced by dynamic auto-scaling, cloud FinOps alignment, and dedicated cost and cluster optimization for GPU and TPU workloads.&lt;/p&gt;&lt;p data-block-key="6912j"&gt;&lt;b&gt;Organizational models:&lt;/b&gt; SRE has expanded beyond dedicated teams assigned to specific applications. Today's organizations use hybrid models, platform engineering integrations, and reliability champions embedded across product teams.&lt;/p&gt;&lt;h3 data-block-key="c90ef"&gt;&lt;b&gt;SRE in the age of AI&lt;/b&gt;&lt;/h3&gt;&lt;p data-block-key="4j8ge"&gt;Then there’s the dual relationship between SRE and artificial Intelligence: how SRE supports AI infrastructure, and how SRE teams use AI to enhance operational reliability.&lt;/p&gt;&lt;p data-block-key="4lti8"&gt;Running large language models (LLMs) and foundation models introduces novel reliability challenges. The book outlines methodologies for operating AI infrastructure at scale:&lt;/p&gt;&lt;ul&gt;&lt;li data-block-key="evk96"&gt;Managing massive GPU and TPU hardware cluster reliability and network topologies&lt;/li&gt;&lt;li data-block-key="c2kaq"&gt;Monitoring non-deterministic outputs and latency characteristics unique to AI&lt;/li&gt;&lt;li data-block-key="ejrm4"&gt;Ensuring pipeline reliability for continuous model training, fine-tuning, and data processing&lt;/li&gt;&lt;/ul&gt;&lt;p data-block-key="d09lr"&gt;AI is also rapidly becoming a force multiplier for reliability engineers. The second edition shares practical frameworks for enhancing human operational efficiency:&lt;/p&gt;&lt;ul&gt;&lt;li data-block-key="eb3ne"&gt;Using generative AI tools to assist in real-time incident triage and postmortem generation&lt;/li&gt;&lt;li data-block-key="71ae0"&gt;Interrogating complex system telemetry using natural language queries&lt;/li&gt;&lt;li data-block-key="672km"&gt;Using predictive machine learning models to identify resource bottlenecks before outages occur&lt;/li&gt;&lt;/ul&gt;&lt;h3 data-block-key="312c"&gt;&lt;b&gt;What hasn’t changed? The importance of culture&lt;/b&gt;&lt;/h3&gt;&lt;p data-block-key="4omqu"&gt;While technologies and architectures continuously evolve, the core principles of SRE remain intact, and are key to establishing a reliability-focused culture. Accordingly, the second edition re-emphasizes and expands upon the human elements of system engineering, and how to get there.&lt;/p&gt;&lt;p data-block-key="3rlcm"&gt;Whether you’re new to the field, a practitioner, an engineering manager, or a technology executive, the rewritten second edition of &lt;i&gt;Site Reliability Engineering&lt;/i&gt; serves as your definitive roadmap for the next era of production systems.&lt;/p&gt;&lt;p data-block-key="23tuu"&gt;You can order your copy today in physical format (see &lt;a href="https://www.google.com/books/edition/_/iXnw0QEACAAJ?kptab=getbook"&gt;Google Books&lt;/a&gt;) or digital format from &lt;a href="https://www.oreilly.com/library/view/site-reliability-engineering/9798341607675/"&gt;O'Reilly&lt;/a&gt;.&lt;/p&gt;
    &lt;/div&gt;
  &lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;</description><pubDate>Mon, 12 Oct 2026 16:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/devops-sre/sre-book-second-edition/</guid><category>Developers &amp; Practitioners</category><category>DevOps &amp; SRE</category><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>10 Years of the SRE book: Announcing the second edition of "Site Reliability Engineering”</title><description></description><site_name>Google</site_name><url>https://cloud.google.com/blog/products/devops-sre/sre-book-second-edition/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Betsy Beyer</name><title>SRE Program Manager</title><department></department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>David Huska</name><title>Software Engineer, Site Reliability Engineer</title><department></department><company></company></author></item><item><title>Announcing MCP Toolbox Java SDK v1.0: Agentic data access for the enterprise</title><link>https://cloud.google.com/blog/topics/developers-practitioners/announcing-mcp-toolbox-java-sdk-v10-agentic-data-access-for-the-enterprise/</link><description>&lt;div class="block-paragraph_advanced"&gt;&lt;p style="text-align: center; font-size: 0.8rem;"&gt;&lt;em&gt;&lt;span style="vertical-align: baseline;"&gt;Co-authors:&lt;/span&gt;&lt;/em&gt;&lt;br/&gt;&lt;em&gt;&lt;span style="vertical-align: baseline;"&gt;&lt;strong&gt;Stenal Jolly&lt;/strong&gt;, Strategic Cloud Engineer, Google&lt;/span&gt;&lt;/em&gt;&lt;br/&gt;&lt;em&gt;&lt;span style="vertical-align: baseline;"&gt;&lt;strong&gt;Anubhav Dhawan&lt;/strong&gt;, Software Engineer, Google&lt;/span&gt;&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Following the &lt;/span&gt;&lt;a href="https://medium.com/google-cloud/mcp-toolbox-v1-0-the-open-source-framework-for-secure-agentic-data-access-3c2199546ba8" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;landmark announcement&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; of MCP Toolbox v1.0, we're thrilled to announce that the &lt;a href="https://github.com/googleapis/mcp-toolbox-sdk-java" rel="noopener" target="_blank"&gt;MCP Toolbox Java SDK&lt;/a&gt; has officially reached version 1.0.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;This release brings first-class, type-safe agent orchestration to one of the world's most widely adopted enterprise ecosystems. Java's mature architecture is purpose-built for rigorous demands, providing the high concurrency, strict transactional integrity, and robust state management required to safely scale mission-critical AI agents in production.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;In this post, we'll tell you about what's new in Java SDK v1.0, show you a real-world example, and help you get started with your own implementation.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;MCP: The universal interface&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Today, developers face a compounding integration bottleneck: if you have &lt;em&gt;N&lt;/em&gt; different AI models and &lt;em&gt;M&lt;/em&gt; enterprise data sources, you must build, secure, and maintain &lt;em&gt;N × M&lt;/em&gt; bespoke, custom connections. This lack of a unified integration layer forces engineering teams to rely on a fragmented web of ad-hoc pipelines. As a result, scaling an agentic architecture quickly becomes unsustainable, exposing sensitive enterprise databases to severe security vulnerabilities, fragmented access controls, and massive maintenance overhead.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Eliminating the fragmented web of custom integrations is the core problem solved by the Model Context Protocol (MCP). Acting as a universal interface—the "USB Type-C" for AI orchestration—MCP decouples models from data sources. Instead of writing custom or managed API integration code for every new model or database, developers write to a single, standardized protocol. This approach allows any MCP-compliant agent to securely and immediately interact with any MCP-enabled system. The MCP connection lets developers connect agents to real-world systems without building bespoke integrations for every new model.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;What's new in Java SDK v1.0: Built for production workloads&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;When we announced the &lt;/span&gt;&lt;a href="https://medium.com/google-cloud/announcing-the-mcp-toolbox-java-sdk-2ed3171bbaa0" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;public Beta&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; for the MCP Toolbox Java SDK, our goal was to bring first-class, type-safe agent orchestration to enterprise Java environments. Since then, we've collaborated with developers and open-source contributors to harden our APIs.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The v1.0 release marks a stable, backwards-compatible foundation suitable for enterprise workloads. Here's what's new and hardened since our v0.2 release:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Transport layer abstraction &amp;amp; &lt;/strong&gt;&lt;code&gt;&lt;strong style="vertical-align: baseline;"&gt;HttpMcpTransport&lt;/strong&gt;&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;: We introduced a clean transport layer abstraction alongside &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;HttpMcpTransport&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;. This feature decouples the core protocol logic from underlying HTTP clients, making it easy to swap network implementations or customize connection pooling.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Decoupled client authentication&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: To simplify enterprise security compliance, client authentication is now decoupled using &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;CredentialsProvider&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; and &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;AuthMethods&lt;/code&gt; classes&lt;span style="vertical-align: baseline;"&gt;. Credentials are resolved asynchronously on every request, so teams can refresh tokens dynamically or plug in their own token source (Google OIDC via ADC ships in the box, anything else is a one-method interface).&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Default parameter support&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Native support for default values in tool parameters, reducing prompt payload sizes and enhancing agent reliability.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Pruning bound parameters&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Sensitive parameters that are bound server-side (like &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;tenant_id&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;) are now automatically stripped from exposed tool definitions so the LLM can't manipulate them.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Version selection &amp;amp; session tracking&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Standardized MCP version selection and robust session tracking ensure consistent protocol negotiation and conversation-state lifecycles.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;HTTP credential exposure warnings&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Added built-in detection that warns you at runtime when credentials are about to travel over a plaintext HTTP connection.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Generic client headers map&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Easily attach custom corporate proxy headers, transaction tracing IDs, or correlation metadata to all outgoing requests.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Get started with the Java SDK v1.0&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;We designed the MCP Toolbox Java SDK to be frictionless for enterprise teams. Just add the following dependency to your &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;pom.xml&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;&amp;lt;dependency&amp;gt;\r\n    &amp;lt;groupId&amp;gt;com.google.cloud.mcp&amp;lt;/groupId&amp;gt;\r\n    &amp;lt;artifactId&amp;gt;mcp-toolbox-sdk-java&amp;lt;/artifactId&amp;gt;\r\n    &amp;lt;version&amp;gt;1.0.0&amp;lt;/version&amp;gt;\r\n&amp;lt;/dependency&amp;gt;&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba5e7d50&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Real-world example: The autonomous transit concierge&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;To demonstrate the power of the Java SDK combined with AlloyDB, let's look at an enterprise use case.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Meet&lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt; Cymbal Transit&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;, a fictitious intercity bus network. Customers don't want to click through nested dropdown menus to plan a trip. They want to ask:&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;"I need to get from New York to Boston tomorrow morning. Can I bring my Golden Retriever? If so, book me the fastest trip."&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;To answer this question, an AI agent must cross-reference unstructured data (pet policies) with structured data (schedules and seat availability) and execute a transaction (booking)—all while maintaining the context of the conversation.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;The foundation: AlloyDB schema with native embeddings&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;We used AlloyDB for this implementation because it handles relational data and high-dimensional vectors natively. Set up your database tables with these statements:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;quot;-- Enable necessary extensions for semantic search and embeddings\r\nCREATE EXTENSION IF NOT EXISTS vector;\r\nCREATE EXTENSION IF NOT EXISTS google_ml_integration;\r\n\r\n-- Table 1: Transit Policies (Unstructured Data for RAG)\r\nCREATE TABLE transit_policies (\r\n    policy_id SERIAL PRIMARY KEY,\r\n    category VARCHAR(50),\r\n    policy_text TEXT,\r\n    policy_embedding vector(768)\r\n);\r\n\r\n-- Table 2: Intercity Bus Schedules (Structured Data)\r\nCREATE TABLE bus_schedules (\r\n    trip_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),\r\n    origin_city VARCHAR(100),\r\n    destination_city VARCHAR(100),\r\n    departure_time TIMESTAMP,\r\n    arrival_time TIMESTAMP,\r\n    available_seats INT DEFAULT 50,\r\n    ticket_price DECIMAL(6,2)\r\n);\r\n\r\n-- Table 3: Booking Ledger (Transactional Action Data)\r\nCREATE TABLE bookings (\r\n    booking_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),\r\n    trip_id UUID REFERENCES bus_schedules(trip_id),\r\n    passenger_id VARCHAR(100),\r\n    status VARCHAR(20) DEFAULT &amp;#x27;CONFIRMED&amp;#x27;,\r\n    booking_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP\r\n);&amp;quot;), (&amp;#x27;language&amp;#x27;, &amp;#x27;lang-sql&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba5e76d0&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Mapping intents to SQL: The &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;tools.yaml&lt;/code&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The MCP Toolbox lets you define custom tools securely. Rather than granting the LLM direct database access, a &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;tools.yaml&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; configuration maps natural language intents directly to parameterized, safe queries:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;kind: source\r\nname: alloydb\r\ntype: alloydb-postgres\r\nproject: my-project\r\nregion: us-central1\r\ncluster: my-cluster\r\ninstance: my-instance\r\ndatabase: postgres\r\n---\r\nkind: tool\r\nname: query-schedules\r\ntype: postgres-sql\r\nsource: alloydb\r\ndescription: Find available bus schedules between cities.\r\nparameters:\r\n  - name: origin\r\n    type: string\r\n    description: The departure city name.\r\n  - name: destination\r\n    type: string\r\n    description: The arrival city name.\r\n  - name: limit\r\n    type: integer\r\n    description: Maximum number of schedules to return.\r\n    default: 5\r\nstatement: |\r\n  SELECT CAST(trip_id AS TEXT) AS trip_id, departure_time, ticket_price\r\n  FROM bus_schedules\r\n  WHERE lower(origin_city) = lower($1) AND lower(destination_city) = lower($2)\r\n  ORDER BY departure_time ASC\r\n  LIMIT $3&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba5e79d0&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;For the complete YAML file, see the &lt;/span&gt;&lt;a href="https://github.com/googleapis/mcp-toolbox-sdk-java/blob/main/demo-applications/cymbal-transit/tools.yaml" rel="noopener" target="_blank"&gt;&lt;code style="text-decoration: underline; vertical-align: baseline;"&gt;tools.yaml&lt;/code&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; file in the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;mcp-toolbox-sdk-java&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; repository.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Stateful agent architecture in Spring Boot&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The hardest part of building conversational AI in enterprise applications is managing state: when a user asks, "What times are available?" and follows up with, "Book the 8 AM one," the agent must remember prior context across turns.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Using the Java MCP Toolbox SDK with Spring Boot and LangChain4j, we can cleanly maintain conversational memory in the HTTP Session and we can cleanly separate the agent into two declarative components:&lt;/span&gt;&lt;/p&gt;
&lt;ol&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;A declarative agent interface that manages the prompt, tools, and conversational memory via an HTTP session.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;A tool execution service that routes agent requests directly to the MCP Toolbox server.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;interface TransitAgent {\r\n    @SystemMessage({\r\n        &amp;quot;You are the Cymbal Transit Concierge.&amp;quot;,\r\n        &amp;quot;Use the \&amp;#x27;querySchedules\&amp;#x27; tool for finding schedules.&amp;quot;,\r\n        &amp;quot;Use \&amp;#x27;bookTicket\&amp;#x27; to execute transactions.&amp;quot;,\r\n        &amp;quot;Use \&amp;#x27;searchPolicies\&amp;#x27; to look up luggage and pet rules.&amp;quot;\r\n    })\r\n    String chat(@MemoryId String sessionId, @UserMessage String userMessage);\r\n}\r\n\r\n@Service\r\nclass TransitAgentTools {\r\n    // These methods automatically invoke our MCP Toolbox server!\r\n    @Tool(&amp;quot;Query specific schedules between an origin and destination city.&amp;quot;)\r\n    public String querySchedules(String origin, String destination) { ... }\r\n\r\n    @Tool(&amp;quot;Book a ticket for a passenger.&amp;quot;)\r\n    public String bookTicket(String tripId, String passengerName) { ... }\r\n\r\n    @Tool(&amp;quot;Search transit policies for luggage and pet rules.&amp;quot;)\r\n    public String searchPolicies(String query) { ... }\r\n}&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba5e7390&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Notice how the &lt;code&gt;@MemoryId&lt;/code&gt; annotation abstracts session tracking: Spring Boot automatically correlates conversational context to the user's HTTP session. Meanwhile, LangChain4j and the MCP Toolbox handle schema translation and tool routing behind the scenes—no handwritten if/else intent parsing required.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;By pairing the MCP Toolbox Java SDK with LangChain4j, we achieve clean separation of concerns and effortless state management:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Zero-boilerplate session management&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;@MemoryId String sessionId&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; parameter binds conversation history directly to the user's HTTP session.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Declarative agent contract&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;TransitAgent&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; interface defines the model's persona and system instructions without complex prompt templating.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Type-safe tool execution&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;TransitAgentTools&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; Spring service wraps remote MCP database tools as native Java methods.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;This architecture ensures your agent remains modular: you can refine prompt guidance in the interface, manage user sessions automatically, and execute secure database queries through MCP Toolbox without tight coupling.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Connecting the dots: Listing, invoking, and executing tools in Java v1.0&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Now let's look under the hood of the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;TransitAgentTools&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; interface. Inside those LangChain4j &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;@Tool&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; methods on our Spring &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;@Service&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, the MCP Toolbox Java SDK handles the heavy lifting—bridging your Java service methods to the MCP tools defined in the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;tools.yaml&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; file. In just a few lines of type-safe code, we can initialize our client using the new v1.0 decoupled authentication and headers abstractions:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;// 1. Initialize the Client with Decoupled Auth and Custom Headers (v1.0)\r\nString serviceUrl = &amp;quot;https://toolbox-my-project-uc.a.run.app/mcp&amp;quot;;\r\nMcpToolboxClient mcpClient = McpToolboxClient.builder()\r\n    .baseUrl(serviceUrl)\r\n    .credentialsProvider(new GoogleCredentialsProvider(serviceUrl)) // Decoupled OIDC credentials\r\n    .headers(Map.of( // Generic client headers\r\n        &amp;quot;X-Correlation-ID&amp;quot;, &amp;quot;enterprise-session-abc123&amp;quot;,\r\n        &amp;quot;X-Client-Platform&amp;quot;, &amp;quot;Spring-Boot&amp;quot;\r\n    ))\r\n    .build();\r\n\r\n// 2. Listing Discoverable Tools\r\nmcpClient.listTools().thenAccept(tools -&amp;gt; {\r\n    System.out.println(&amp;quot;Successfully discovered &amp;quot; + tools.size() + &amp;quot; tools.&amp;quot;);\r\n});\r\n\r\n// 3. Invoking a Tool (Read-Only Data with Default Parameter Support)\r\n// &amp;quot;limit&amp;quot; is omitted: the SDK fills it from the default in the tool definition\r\nString schedules = mcpClient.loadTool(&amp;quot;query-schedules&amp;quot;)\r\n    .thenCompose(tool -&amp;gt; tool.execute(Map.of(\r\n        &amp;quot;origin&amp;quot;, &amp;quot;New York&amp;quot;,\r\n        &amp;quot;destination&amp;quot;, &amp;quot;Boston&amp;quot;)))\r\n    .join().text();\r\n\r\n// 4. Executing a Transactional Tool (Using Bound Parameters)\r\nAuthTokenGetter toolAuthGetter = () -&amp;gt; CompletableFuture.completedFuture(myIdToken);\r\n\r\nString bookingConfirmation = mcpClient.loadTool(&amp;quot;book-ticket&amp;quot;, Map.of(&amp;quot;google_auth&amp;quot;, toolAuthGetter))\r\n    // Bind the authenticated user context securely. bindParam returns a new immutable\r\n    // Tool, and the bound parameter is pruned from the definition exposed to the LLM!\r\n    .thenCompose(tool -&amp;gt; tool.bindParam(&amp;quot;passenger_name&amp;quot;, &amp;quot;Jane Doe&amp;quot;)\r\n        // Execute the mutable transaction\r\n        .execute(Map.of(&amp;quot;trip_id&amp;quot;, &amp;quot;123e4567-e89b-12d3-a456-426614174000&amp;quot;)))\r\n    .join().text();&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba5e5b50&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Secure by default: authentication and deployment&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Moving an AI agent to production requires rock-solid credential handling and an infrastructure that scales with demand. Let's look at how you can enforce credential safety across environments and deploy independently on Cloud Run.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Application Default Credentials (ADC) &amp;amp; safety&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;By using the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;GoogleCredentialsProvider&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; service&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;, your Java app inherits its secure identity from its execution environment (whether local or in Google Cloud) through Application Default Credentials (ADC)—no hard-coded keys, with OIDC tokens minted and cached per audience under the hood. Furthermore, v1.0 offers HTTP Credential Exposure Warnings that automatically detect when credentials are about to travel over a plaintext HTTP connection and emit a runtime warning telling you to switch to HTTPS.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Deploying the fleet to Cloud Run&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Because MCP Toolbox and the Spring Boot Agent are fully decoupled, they scale independently on Google Cloud Run to meet high concurrency and stateful conversation requirements.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;To set up and configure Toolbox on Cloud Run, download the open-source MCP Toolbox for Databases and then follow the &lt;/span&gt;&lt;a href="https://mcp-toolbox.dev/documentation/deploy-to/cloud-run" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;deployment guide&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Get started today&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;With MCP Toolbox Java SDK v1.0, enterprise Java teams can wire Spring Boot and LangChain4j agents to the Toolbox server, and through it, to AlloyDB and every other supported data source. When you use the toolbox, arguments are validated against the tool definition before they leave the JVM, authentication is decoupled, and custom headers are attached to every outgoing request. The following &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;implementation steps will help you get started&lt;/strong&gt;.&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Step 1: Add the Dependency&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;To start building with the SDK, add the following dependency to your Maven project's &lt;code&gt;pom.xml&lt;/code&gt; file:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;&amp;lt;dependency&amp;gt;\r\n        &amp;lt;groupId&amp;gt;com.google.cloud.mcp&amp;lt;/groupId&amp;gt;\r\n        &amp;lt;artifactId&amp;gt;mcp-toolbox-sdk-java&amp;lt;/artifactId&amp;gt;\r\n        &amp;lt;version&amp;gt;1.0.0&amp;lt;/version&amp;gt;\r\n &amp;lt;!-- {x-version-update:mcp-toolbox-sdk-java:current} --&amp;gt;\r\n    &amp;lt;/dependency&amp;gt;&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba5e7f50&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Step 2: Explore Resources &amp;amp; Demos&lt;/span&gt;&lt;/h3&gt;
&lt;ul&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;GitHub Repository&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: &lt;/span&gt;&lt;a href="https://github.com/googleapis/mcp-toolbox-sdk-java" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Java SDK for interacting with the MCP Toolbox for Databases&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Official Documentation&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: &lt;/span&gt;&lt;a href="https://mcp-toolbox.dev/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;MCP Toolbox for Databases&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;/ul&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Demo Application&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: To experience the Java SDK V1.0 for MCP Toolbox latest, try the sample application &lt;a href="https://github.com/googleapis/mcp-toolbox-sdk-java/tree/main/demo-applications/cymbal-transit" rel="noopener" target="_blank"&gt;Cymbal transit project&lt;/a&gt;&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;h4&gt;Gradle implementation&lt;/h4&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;If your team uses Gradle instead of Maven, remember they will need to translate this dependency:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;quot;implementation &amp;#x27;com.google.cloud.mcp:mcp-toolbox-sdk-java:1.0.0&amp;#x27;&amp;quot;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02baa4c8d0&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h4&gt;&lt;span style="vertical-align: baseline;"&gt;Automatic version tracking&lt;/span&gt;&lt;/h4&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;I&lt;span style="vertical-align: baseline;"&gt;f you copy this setup into automated internal repositories, k&lt;/span&gt;eep the XML comment &lt;code&gt;&amp;lt;!-- {x-version-update...} --&amp;gt;&lt;/code&gt; intact. It's required by the release manager's deployment scripts to automatically bump versions.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;&lt;span style="vertical-align: baseline;"&gt;Now that you can integrate your modern agentic tools and servers to your enterprise Java applications with a stale MCP Toolbox Java SDK, get started today!&lt;/span&gt;&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;</description><pubDate>Tue, 06 Oct 2026 19:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/topics/developers-practitioners/announcing-mcp-toolbox-java-sdk-v10-agentic-data-access-for-the-enterprise/</guid><category>Developers &amp; Practitioners</category><media:content height="540" url="https://storage.googleapis.com/gweb-cloudblog-publish/images/announcing-mcp-toolbox-java-sdk-v10-agentic-.max-600x600.png" width="540"></media:content><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Announcing MCP Toolbox Java SDK v1.0: Agentic data access for the enterprise</title><description></description><image>https://storage.googleapis.com/gweb-cloudblog-publish/images/announcing-mcp-toolbox-java-sdk-v10-agentic-.max-600x600.png</image><site_name>Google</site_name><url>https://cloud.google.com/blog/topics/developers-practitioners/announcing-mcp-toolbox-java-sdk-v10-agentic-data-access-for-the-enterprise/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Abirami Sukumaran</name><title>Staff Developer Advocate</title><department></department><company></company></author></item><item><title>Networking for AI inference model serving - GKE only and for all other backends</title><link>https://cloud.google.com/blog/topics/developers-practitioners/networking-for-ai-inference-model-serving-gke-only-and-for-all-other-backends/</link><description>&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Enterprises and individual developers frequently run multiple AI inference models. The right architecture can simplify how the models are called while also providing centralized governance. &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;In this post, we'll look at two reference architectures focused on networking AI inference model serving: one for &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/kubernetes-engine/docs/concepts/kubernetes-engine-overview"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Google Kubernetes Engine (GKE)&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; and one all other backend types. First, we'll explore the commonalities between the reference architectures that you'll see later. Then we'll explore unique components of the architecture for GKE backends and finally, we'll go over the elements of the architecture for all backend types.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;strong style="vertical-align: baseline;"&gt;The entry point&lt;/strong&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;You can expose your model deployment behind a stable, secure, and reliable entry point that acts as the front end for inference calls. This entry point also acts as a control zone where policy, security, and logic can be enforced. &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;Both the Cloud Load balancer and the Inference Gateway provide entry point capability. These types of endpoints can terminate secure connections with TLS, integrate with API management components, extend functionally with service extensions, and capitalize on capabilities of Model Armor for added security.  &lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Common services in the designs &lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Both reference designs use these services:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://docs.cloud.google.com/vpc/docs/private-service-connect#endpoints"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;Private Service Connect inference endpoint&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;: Anchors the entry point inside your consumer Virtual Private Cloud (VPC) network. Traffic hits a private internal &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/vpc/docs/subnets#valid-ranges"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;IP address&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, keeping inference calls in your private network.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://docs.cloud.google.com/apigee/docs/api-platform/get-started/what-apigee"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;Apigee API Management&lt;/strong&gt;&lt;/a&gt;&lt;strong style="vertical-align: baseline;"&gt; (Optional)&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Integrates via an Apigee Extension Processor callout to handle client identity verification, rate limits, and quota enforcement before requests ever reach compute resources.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://docs.cloud.google.com/model-armor/overview"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;Model Armor&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;: Serves as an inline AI safety checkpoint, screening prompts and output completions against prompt injection and sensitive data leakage.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;&lt;strong style="vertical-align: baseline;"&gt;Design pattern serving on GKE only&lt;/strong&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;This section focuses on a GKE-only backend design.&lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt; &lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;To understand the full end-to-end concept, please read the entire architecture document &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/architecture/networking-for-ai-inference-gke"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;Networking for AI inference model serving on GKE&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;. &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;The design pattern is based on this diagram:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/networking-ai-inference-model-serving-gke-.max-1000x1000.png"
        
          alt="networking-ai-inference-model-serving-gke-architecture"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;In addition to the common services identified in the previous section, the design for GKE uses these components:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://docs.cloud.google.com/kubernetes-engine/docs/concepts/about-gke-inference-gateway"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;GKE Inference Gateway&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;: Deployed as an internal Application Load Balancer (&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;gke-l7-rilb&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;). It acts as a specialized ingress engine that parses incoming request payloads, evaluates HTTPRoute rules, and steers queries to appropriate model-serving targets.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Inference pools&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: A logical group containing replicas of the same model. When the Gateway receives a prompt, it evaluates HTTPRoute rules to select the appropriate inference pool based on the model identifier. Pools have an initial size and can be configured to autoscale dynamically.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Model replica sets&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Individual model replicas (inference server instances) deployed across single-node or multi-node GPU or TPU node pools. A replica set represents a uniform group of these model replicas.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;The traffic flow GKE example&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;A client application that uses this GKE-based architecture to call a backend model would go through a flow like this:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Ingress&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: A client application in the consumer VPC issues an OpenAI-compatible API call to the local Private Service Connect endpoint, routing directly to the GKE Inference gateway using a regional internal Application Load Balancer.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Payload inspection&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The Gateway reads the target model parameter specified in the request body and adds it to the HTTP headers.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Control plane validation&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: If Apigee is used, it checks client credentials and quotas. Model Armor screens the prompt for policy violations or data leakage.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Backend selection&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The Gateway evaluates HTTPRoute mappings to identify the target pool, matches shared prefix cache context, and routes to the lowest-load GPU or TPU replica based on real-time Prometheus data.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Egress&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The replica runs the inference workload. Output tokens pass through Model Armor for final response verification before streaming back over the private connection.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;&lt;strong style="vertical-align: baseline;"&gt;Design pattern serving on all backends&lt;/strong&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;This section focuses on multiple backend types which can be used for inference, and it provides an overview of the architecture in the following diagram.&lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt; &lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;To understand the full end-to-end concept, please read the entire architecture document &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/architecture/networking-for-ai-inference"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;Networking for AI inference model serving on all backends&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/networking-ai-inference-model-serving-all-.max-1000x1000.png"
        
          alt="networking-ai-inference-model-serving-all-backends"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;For architectures spanning mixed environments such as GKE, Cloud Run, Agent Platform, on-premises data centers, or external clouds, these additional components are used:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://docs.cloud.google.com/load-balancing/docs/l7-internal#load-balancer-mode"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;Regional internal Application Load Balancer&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;: Serves as the central Layer 7 routing proxy that manages routing logic, SSL termination, and Service Extensions callouts.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://github.com/llm-d/llm-d-inference-payload-processor" rel="noopener" target="_blank"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;Inference Payload Processor&lt;/strong&gt;&lt;/a&gt;&lt;strong style="vertical-align: baseline;"&gt; (Service Extensions)&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: This is similar to body-based routing as used in the GKE Inference Gateway, but to enable the functionality on an Application Load Balancer a service extension is needed.&lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt; &lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;A lightweight Cloud Run callout inspects the JSON body of incoming OpenAI API requests, extracts the target model identifier, and writes an X-Gateway-Model-Name header to drive URL map routing.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://docs.cloud.google.com/load-balancing/docs/negs"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;Network Endpoint Group (NEG)&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;: Delivers flexible routing to heterogeneous backends based on the injected model header.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;All-backends traffic flow example&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;A client application that uses this architecture to call a backend model would go through a flow like this:&lt;/span&gt;&lt;/p&gt;
&lt;ol&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Private ingress&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The client application targets the Private Service Connect endpoint over private IP address space. The regional internal Application Load Balancer receives the request.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Model name extraction&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The load balancer sends the payload to the Cloud Run body-based router callout, which inspects the JSON payload and injects the X-Gateway-Model-Name header.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Policy and safety enforcement&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The request passes to Apigee for identity and quota validation, then to Model Armor to scrub sensitive data and block malicious prompts.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;NEG routing&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The load balancer URL map inspects the model header and forwards the request to the matching backend NEG (Agent Platform, GKE, Cloud Run, Hybrid, or Internet).&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Private delivery&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The target backend executes the model prompt, Model Armor screens the completion, and the result returns privately along the ingress path.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/networking-ai-inference-model-serving-flow.max-1000x1000.png"
        
          alt="networking-ai-inference-model-serving-flow"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h2&gt;&lt;strong style="vertical-align: baseline;"&gt;What's next&lt;/strong&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Take a deeper dive into building AI workloads on Google Cloud:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Document set: &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/architecture/agentic-ai-overview"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Agentic AI architecture guides&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Architecture Center: &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/architecture/multi-agent-private-networking-patterns"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Multi-agent private networking patterns in Google Cloud&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Document: &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/general/netsec-overview"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Gemini Enterprise Agent Platform networking access overview&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Want to ask a question, find out more, or share a thought? Please connect with me on &lt;/span&gt;&lt;a href="https://www.linkedin.com/in/ammett/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Linkedin&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;. &lt;/span&gt;&lt;/p&gt;&lt;/div&gt;</description><pubDate>Tue, 06 Oct 2026 12:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/topics/developers-practitioners/networking-for-ai-inference-model-serving-gke-only-and-for-all-other-backends/</guid><category>Networking</category><category>Developers &amp; Practitioners</category><media:content height="540" url="https://storage.googleapis.com/gweb-cloudblog-publish/images/networking-ai-inference-model-serving-hero.max-600x600.png" width="540"></media:content><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Networking for AI inference model serving - GKE only and for all other backends</title><description></description><image>https://storage.googleapis.com/gweb-cloudblog-publish/images/networking-ai-inference-model-serving-hero.max-600x600.png</image><site_name>Google</site_name><url>https://cloud.google.com/blog/topics/developers-practitioners/networking-for-ai-inference-model-serving-gke-only-and-for-all-other-backends/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Ammett Williams</name><title>Developer Relations Engineer</title><department></department><company></company></author></item><item><title>GKE CPU startup boost: Accelerate app starts without over-provisioning</title><link>https://cloud.google.com/blog/products/containers-kubernetes/gke-cpu-startup-boost-faster-pod-starts-lower-costs/</link><description>&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Whether you’re launching microservices in response to sudden traffic spikes, deploying new software releases, or scaling up application replicas, &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;pod startup time&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; is critical to maintaining a fast, responsive user experience for applications running on Google Kubernetes Engine (GKE).&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-paragraph_with_image"&gt;&lt;div class="article-module h-c-page"&gt;
  &lt;div class="h-c-grid uni-paragraph-wrap"&gt;
    &lt;div class="uni-paragraph
      h-c-grid__col h-c-grid__col--8 h-c-grid__col-m--6 h-c-grid__col-l--6
      h-c-grid__col--offset-2 h-c-grid__col-m--offset-3 h-c-grid__col-l--offset-3"&gt;

      






  

    &lt;figure class="article-image--wrap-small
      
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/1_mhC0eeP.max-1000x1000.png"
        
          alt="1"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  





      &lt;p data-block-key="0udcy"&gt;Yet, platform engineers and developers face a persistent dilemma: Applications often demand significantly more CPU power during startup than they do during steady-state operations. Sizing CPU requests for normal, steady-state usage leads to CPU throttling during launch, which can result in sluggish cold starts and readiness probe timeouts. On the flip side, over-provisioning baseline CPU requests to satisfy short-lived startup bursts wastes valuable compute resources, inflating infrastructure bills.&lt;/p&gt;&lt;p data-block-key="ciu5o"&gt;Today, we are excited to announce &lt;a href="https://docs.cloud.google.com/kubernetes-engine/docs/how-to/boost-application-startup"&gt;&lt;b&gt;CPU startup boost&lt;/b&gt;&lt;/a&gt; for GKE in preview. Integrated directly into GKE's Vertical Pod Autoscaler (VPA), CPU startup boost dynamically elevates a container's CPU allocation during initialization and seamlessly scales it back to baseline steady-state levels once the application is ready - &lt;b&gt;all without restarting your containers.&lt;/b&gt;&lt;/p&gt;
    &lt;/div&gt;
  &lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;Why modern applications need extra CPU at boot time&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;When a new container launches, it may perform intensive initialization tasks before it begins serving user requests. Depending on your tech stack, the following startup workloads require substantial CPU cycles:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Java JVM applications&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Frameworks like Spring Boot require high CPU burst capacity for class loading, classpath scanning, instantiating dependency injection containers, and running Just-in-Time (JIT) compilation.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Node.js servers&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Apps parse JavaScript files, build complex module dependency trees (&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;require&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;/&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;import&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;), and execute V8 engine optimization and JIT compilation passes during initial execution.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Python and AI/ML microservices&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: These services spend initial cycles importing heavy libraries (such as PyTorch, NumPy, or LangChain), compiling &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;.pyc&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; bytecode, establishing ORM database schemas, and pre-loading cache structures.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;If you size CPU requests strictly for steady-state performance, these initialization workloads experience CPU throttling on launch, delaying readiness probes. To prevent slow cold starts, teams frequently overprovision CPU requests. However, once the application stabilizes, those extra CPU resources sit idle, increasing your cloud spend without adding value.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-paragraph_with_image"&gt;&lt;div class="article-module h-c-page"&gt;
  &lt;div class="h-c-grid uni-paragraph-wrap"&gt;
    &lt;div class="uni-paragraph
      h-c-grid__col h-c-grid__col--8 h-c-grid__col-m--6 h-c-grid__col-l--6
      h-c-grid__col--offset-2 h-c-grid__col-m--offset-3 h-c-grid__col-l--offset-3"&gt;

      






  

    &lt;figure class="article-image--wrap-small
      
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/2_zZExLw4.max-1000x1000.jpg"
        
          alt="2"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  





      &lt;h3 data-block-key="0udcy"&gt;&lt;b&gt;How CPU startup boost can help&lt;/b&gt;&lt;/h3&gt;&lt;p data-block-key="8k1a3"&gt;CPU Startup Boost solves this by giving your workloads temporary vCPU "boosts" during launch, and automatically returning them to baseline once initialization completes.&lt;/p&gt;&lt;p data-block-key="c9jmj"&gt;Key benefits:&lt;/p&gt;&lt;ul&gt;&lt;li data-block-key="96uhr"&gt;&lt;b&gt;Faster cold starts&lt;/b&gt;: Reduce application initialization times by up to &lt;b&gt;2x&lt;/b&gt;, accelerating auto-scaling responsiveness during unexpected traffic surges.&lt;/li&gt;&lt;li data-block-key="f852p"&gt;&lt;b&gt;Optimized cloud spend&lt;/b&gt;: Right-size steady-state CPU requests to fit actual runtime needs rather than paying for idle startup headroom.&lt;/li&gt;&lt;li data-block-key="5opq6"&gt;&lt;b&gt;Zero pod restarts&lt;/b&gt;: Dynamic resource resizing happens live inside the running container.&lt;/li&gt;&lt;li data-block-key="3r67d"&gt;&lt;b&gt;Flexible policy controls&lt;/b&gt;: Apply simple pod-level multiplier factors (e.g., 2x CPU during startup) or define granular, container-specific rules for complex multi-container pods.&lt;/li&gt;&lt;/ul&gt;
    &lt;/div&gt;
  &lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;Under the hood: Kubernetes In-place Pod Resize&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Historically, changing a pod's resource requests or limits required deleting and recreating the pod. This disruptive process triggered container restarts, cache invalidation, and node rescheduling overhead.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;To fix that, CPU startup boost builds on Kubernetes &lt;/span&gt;&lt;a href="https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;In-place Pod Resize (IPPR)&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Tracked under &lt;/span&gt;&lt;a href="https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/1287-in-place-pod-resize" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;KEP-1287&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, IPPR introduced dynamic, in-place resource mutation. Introduced as Alpha in Kubernetes 1.27, promoted to Beta in v1.33, and graduating to General Availability (GA) in v1.35, IPPR allows the Kubernetes control plane and &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;kubelet&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; to update container CPU and memory requests on running pods &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;without restarting the container process&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;GKE leverages IPPR within the VPA  to apply startup CPU boosts at pod admission and smoothly step them down post-readiness.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;How CPU startup boost works (pod lifecycle overview)&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;CPU startup boost operates across three distinct phases:&lt;/span&gt;&lt;/p&gt;
&lt;ol&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Admission phase&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: When you deploy a pod, the VPA admission webhook intercepts the creation request. It calculates the elevated CPU request based on your policy (e.g., 2x multiplier or +2 vCPUs) and injects the boosted CPU request along with tracking annotations into the pod spec.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Startup phase&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: The pod is scheduled and initialized with the boosted CPU allocation. Your application completes class loading, JIT compilation, or module parsing at top speed without experiencing CPU throttling.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: decimal; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Unboosting phase&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: As soon as the pod's &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;readinessProbe&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; passes (plus any configured &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;durationSeconds&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; cooldown delay), the VPA updater issues an in-place resize request. The CPU request steps back down to your baseline level while the container continues running uninterrupted.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;Prerequisites and availability&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;CPU startup boost is available today in preview on GKE:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;GKE version&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Version &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;1.36.0-gke.4447000&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; or later on Standard and Autopilot clusters.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Cluster Modes&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Enabled natively on GKE Autopilot (VPA is active by default). On GKE Standard, simply ensure Vertical Pod Autoscaling (VPA) is enabled.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Workload Support&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Works with standard Kubernetes controllers, including Deployments and StatefulSets.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;Getting started: Configuring CPU startup boost&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Configuring CPU startup boost is as simple as adding a &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;startupBoost&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; section to your &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;VerticalPodAutoscaler&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; manifest.&lt;/span&gt;&lt;/p&gt;
&lt;h4&gt;&lt;span style="vertical-align: baseline;"&gt;Example 1: Pod-level boost with fixed steady-state (&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;updateMode: "Off"&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;)&lt;/span&gt;&lt;/h4&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;If you want to use VPA purely for startup boost while keeping steady-state CPU requests locked to your manifest definitions, set &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;updateMode&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; to &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;"Off"&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;apiVersion: &amp;quot;autoscaling.k8s.io/v1&amp;quot;\r\nkind: VerticalPodAutoscaler\r\nmetadata:\r\n  name: java-app-startup-boost\r\n  namespace: default\r\nspec:\r\n  targetRef:\r\n    apiVersion: &amp;quot;apps/v1&amp;quot;\r\n    kind: Deployment\r\n    name: java-app\r\n  updatePolicy:\r\n    updateMode: &amp;quot;Off&amp;quot;\r\n  startupBoost:\r\n    cpu:\r\n      type: &amp;quot;Factor&amp;quot;\r\n      factor: 2\r\n      durationSeconds: 10&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9c5e10&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;In this example, GKE doubles the container's CPU request during launch and holds the boosted allocation for 10 seconds after readiness probes pass before scaling back to baseline.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;Example 2: Combining startup boost with continuous VPA auto-scaling&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;If you want GKE to boost CPU during launch &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;and&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; continuously optimize steady-state resources post-startup, set &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;updateMode&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; to &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;"InPlaceOrRecreate"&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;apiVersion: &amp;quot;autoscaling.k8s.io/v1&amp;quot;\r\nkind: VerticalPodAutoscaler\r\nmetadata:\r\n  name: nodejs-app-vpa\r\n  namespace: default\r\nspec:\r\n  targetRef:\r\n    apiVersion: &amp;quot;apps/v1&amp;quot;\r\n    kind: Deployment\r\n    name: nodejs-service\r\n  updatePolicy:\r\n    updateMode: &amp;quot;InPlaceOrRecreate&amp;quot;\r\n  startupBoost:\r\n    cpu:\r\n      type: &amp;quot;Factor&amp;quot;\r\n      factor: 3\r\n      durationSeconds: 15&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9c7010&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h4&gt;&lt;span style="vertical-align: baseline;"&gt;Example 3: Granular container-level boost&lt;/span&gt;&lt;/h4&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;For pods running sidecars (such as logging agents or service mesh proxies) that do not require extra CPU on boot, target specific app containers:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;apiVersion: &amp;quot;autoscaling.k8s.io/v1&amp;quot;\r\nkind: VerticalPodAutoscaler\r\nmetadata:\r\n  name: app-container-boost\r\nspec:\r\n  targetRef:\r\n    apiVersion: &amp;quot;apps/v1&amp;quot;\r\n    kind: Deployment\r\n    name: API-gateway\r\n  updatePolicy:\r\n    updateMode: &amp;quot;Off&amp;quot;\r\n  resourcePolicy:\r\n    containerPolicies:\r\n    - containerName: &amp;quot;web-app&amp;quot;\r\n      mode: &amp;quot;Off&amp;quot;\r\n      startupBoost:\r\n        cpu:\r\n          type: &amp;quot;Quantity&amp;quot;\r\n          quantity: &amp;quot;2&amp;quot;\r\n          durationSeconds: 5&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9c4e10&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h2&gt;&lt;strong style="vertical-align: baseline;"&gt;Verifying startup boost in your cluster&lt;/strong&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;You can verify that GKE applied and downscaled the startup boost using &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;kubectl&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;:&lt;/span&gt;&lt;/p&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;1. &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Inspect pod annotations&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Check for the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;vpaCpuStartupBoost&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; tracking annotation:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;kubectl get pod &amp;lt;POD_NAME&amp;gt; -o yaml&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9c6050&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;Look for annotations indicating the original baseline and boosted CPU requests.&lt;/span&gt;&lt;/p&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;2. &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Monitor in-place resize events&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Confirm that GKE downscaled the CPU request back to baseline after readiness:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;kubectl get events --field-selector reason=InPlaceResizedByVPA&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9c45d0&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;An event with &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;reason=InPlaceResizedByVPA&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; confirms successful in-place downscaling post-startup.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Best practices for production workloads&lt;/span&gt;&lt;/h3&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Pairing with Horizontal Pod Autoscaler (HPA)&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: When using HPA alongside CPU startup boost, ensure a robust &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;readinessProbe&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; is defined and keep &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;durationSeconds&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; short (e.g., &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;0s&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;–&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;10s&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;). This prevents HPA from falsely interpreting initialization CPU spikes as high steady-state load.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Handle traffic spikes with &lt;/strong&gt;&lt;a href="https://docs.cloud.google.com/kubernetes-engine/docs/how-to/configure-capacity-buffer"&gt;&lt;strong style="text-decoration: underline; vertical-align: baseline;"&gt;GKE capacity buffers API&lt;/strong&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;: By reducing the startup tax at scale, you can achieve higher workload density on fewer nodes to improve overall utilization. Consider adopting the GKE capacity buffers API to absorb sudden traffic surges with minimal operational overhead while maintaining strict SLOs. &lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;GKE Autopilot resource ratios&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: On GKE Autopilot, remember that pods must maintain valid CPU-to-memory ratios. Ensure baseline memory allocations accommodate the boosted CPU ratio during startup.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Get started today&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;CPU startup boost gives GKE users the best of both worlds: lightning-fast cold starts for CPU-intensive workloads like Java, Node.js, and Python, paired with maximum resource efficiency and lower cloud costs.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Ready to accelerate your GKE workloads?&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Explore the &lt;/span&gt;&lt;a href="https://cloud.google.com/kubernetes-engine/docs/how-to/cpu-startup-boost"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;GKE CPU startup boost documentation&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Learn more about &lt;/span&gt;&lt;a href="https://cloud.google.com/kubernetes-engine/docs/how-to/vertical-pod-autoscaling"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Vertical Pod Autoscaling on GKE&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Try CPU startup boost on your GKE Standard or Autopilot clusters running GKE &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;1.36.0-gke.4447000&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; or later!&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;&lt;/div&gt;</description><pubDate>Fri, 02 Oct 2026 16:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/containers-kubernetes/gke-cpu-startup-boost-faster-pod-starts-lower-costs/</guid><category>Compute</category><category>GKE</category><category>Developers &amp; Practitioners</category><category>Containers &amp; Kubernetes</category><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>GKE CPU startup boost: Accelerate app starts without over-provisioning</title><description></description><site_name>Google</site_name><url>https://cloud.google.com/blog/products/containers-kubernetes/gke-cpu-startup-boost-faster-pod-starts-lower-costs/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Abdel Sghiouar</name><title>Cloud Developer Advocate</title><department></department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Jakub Pawełczak</name><title>Cloud Product Manager</title><department></department><company></company></author></item><item><title>Introducing the Server Side Cloud Swift SDK</title><link>https://cloud.google.com/blog/topics/developers-practitioners/introducing-the-server-side-cloud-swift-sdk/</link><description>&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;For years, &lt;a href="https://swift.org/" rel="noopener nofollow noreferrer" target="_blank"&gt;Swift&lt;/a&gt; was perceived mainly as a UI language tied to Apple client devices. With Swift 6 and strict concurrency checking, it has matured into a viable systems and cloud language, pairing Rust-like data-race safety with predictable, reference-counted performance.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;To support this ecosystem, Google engineering has launched the official &lt;/span&gt;&lt;a href="https://github.com/googleapis/google-cloud-swift" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Google Cloud API Client Libraries for Swift&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;. Built from the ground up for Swift 6.2+, this new SDK uses the latest non-blocking &lt;/span&gt;&lt;a href="https://github.com/apple/swift-nio" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Swift NIO&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; event loops, HTTP/2 multiplexing, &lt;/span&gt;&lt;a href="https://grpc.io" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;gRPC&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; transport, and zero-cost compile-time data race safety.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;In this article, we'll walk you through all you need to know to get started, and to understand how the Server Side Cloud Swift SDK, or &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;google-cloud-swift&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, works.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;The rise of server-side Swift and cloud-native concurrency&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Open Sans', 'Helvetica Neue', sans-serif;"&gt;Traditional backend development often forces a trade-off between developer ergonomics and resource utilization. While managed runtimes offer rapid development, lower-level systems languages provide finer control over memory and CPU footprint, often at the expense of feature velocity.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;Server-side Swift aims to strike a practical balance. Swift pairs a lightweight runtime and Automatic Reference Counting (ARC) with expressive syntax. More importantly, Swift 6 introduces compile-time concurrency checking. &lt;span style="vertical-align: baseline;"&gt;When you share state between async tasks across a cloud microservice, the compiler enforces that types conform to &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;Sendable&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;. Data races are caught in your editor before a binary ever compiles or reaches production.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;At the network layer, every request to &lt;/span&gt;&lt;a href="https://cloud.google.com"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Google Cloud&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; APIs runs over event-driven, non-blocking sockets that scale across multicore &lt;/span&gt;&lt;a href="https://www.kernel.org" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Linux&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; server environments without spawning system threads per connection.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/sdk-architecture.max-1000x1000.png"
        
          alt="sdk-architecture"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Where to use the Swift SDK&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The Server Side Cloud Swift SDK is engineered for server, container, and automated DevOps environments.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;When you build high-throughput microservices with Swift web frameworks like &lt;/span&gt;&lt;a href="https://hummingbird.codes" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Hummingbird&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; or &lt;/span&gt;&lt;a href="https://vapor.codes" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Vapor&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;google-cloud-swift&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; provides native access to Cloud Storage, AI, Identity and Access Management (IAM), and over one hundred other Google Cloud services. You can containerize your executable on Linux and deploy directly to &lt;/span&gt;&lt;a href="https://cloud.google.com/run"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Cloud Run&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;a href="https://cloud.google.com/kubernetes-engine"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Google Kubernetes Engine (GKE)&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, or &lt;/span&gt;&lt;a href="https://cloud.google.com/compute"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Compute Engine&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; VMs.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Because the SDK compiles on macOS, and Linux, you can develop the backend in your preferred development environment, and then seamlessly deploy to production. And using Swift on both the frontend and backend allows you to share application-specific types across both.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The SDK also excels at platform engineering and DevOps automation. You can author cross-platform CLI utilities and data rotation scripts that run on your developer laptop or inside CI/CD pipelines. These tools authenticate automatically against Google Cloud using &lt;/span&gt;&lt;a href="https://cloud.google.com/docs/authentication/application-default-credentials"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Application Default Credentials (ADC)&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; or &lt;/span&gt;&lt;a href="https://cloud.google.com/iam/docs/workload-identity-federation"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Workload Identity Federation&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;If you're building an &lt;/span&gt;&lt;a href="https://developer.apple.com/ios/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;iOS&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;a href="https://www.apple.com/ipados/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;iPadOS&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, or &lt;/span&gt;&lt;a href="https://www.apple.com/apple-vision-pro/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;visionOS&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; app for the &lt;/span&gt;&lt;a href="https://www.apple.com/app-store/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Apple App Store&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, you should not embed &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;google-cloud-swift&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; directly into your client bundle. Shipping Google Cloud service account keys or administrative credentials inside a client binary creates security risks. For direct client-side features, use the &lt;/span&gt;&lt;a href="https://github.com/firebase/firebase-ios-sdk" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Firebase SDK for Apple Platforms&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; to handle user authentication, real-time &lt;/span&gt;&lt;a href="https://cloud.google.com/firestore"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Firestore&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; sync, and client-side security rules, or route requests through your own Cloud Run backend API.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Getting started with your IDE and packages&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Because &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;google-cloud-swift&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; treats Linux and macOS as first-class citizens, you can develop on Apple hardware with &lt;/span&gt;&lt;a href="https://developer.apple.com/xcode/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Xcode&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; or on Linux workstations with &lt;/span&gt;&lt;a href="https://code.visualstudio.com/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Visual Studio Code&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; and &lt;/span&gt;&lt;a href="https://swift.org/install/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;swiftly&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;To install the official Swift compiler on Linux workstations using the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;swiftly&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; CLI installer, run:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;curl -O https://download.swift.org/swiftly/linux/swiftly-$(uname -m).tar.gz &amp;amp;&amp;amp;\r\ntar zxf swiftly-$(uname -m).tar.gz &amp;amp;&amp;amp;\r\n./swiftly init --quiet-shell-followup &amp;amp;&amp;amp;\r\n. &amp;quot;${SWIFTLY_HOME_DIR:-$HOME/.local/share/swiftly}/env.sh&amp;quot; &amp;amp;&amp;amp;\r\nhash -r&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9e1750&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Alternatively, you can download prebuilt toolchain tarballs directly from official &lt;/span&gt;&lt;a href="https://www.swift.org/download/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Swift Downloads&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; for Ubuntu, Debian, Fedora, or Amazon Linux. Note that &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;google-cloud-swift&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; requires &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;Swift 6.2 or later&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;, so verify your compiler version with &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;swift --version&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; after installation.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;To resolve &lt;/span&gt;&lt;a href="https://swift.org/package-manager/" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Swift Package Manager&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; bare repository trust warnings when cloning across Linux filesystems, configure &lt;/span&gt;&lt;a href="https://git-scm.com" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Git&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; before building with &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;git config --global safe.bareRepository all&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Add the required packages to your &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;Package.swift&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; manifest:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;swift package add-dependency https://github.com/googleapis/swift-google-cloud-language-v2.git --from 0.4.0\r\nswift package add-target-dependency GoogleCloudLanguageV2 CloudBackendService --package swift-google-cloud-language-v2&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9e2310&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;On macOS you need to change the &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;platforms&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; directive:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;// swift-tools-version: 6.2\r\nimport PackageDescription\r\n\r\nlet package = Package(\r\n  name: &amp;quot;CloudBackendService&amp;quot;,\r\n  // Applied when compiling on Darwin/macOS; ignored by SPM on Linux targets\r\n  platforms: [.macOS(.v15)],\r\n ... ...&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9e17d0&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph"&gt;&lt;p data-block-key="bngnr"&gt;In most environments a default-initialized client can make requests:&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;import Foundation\r\nimport GoogleCloudLanguageV2\r\n\r\nfunc analyzeTextSentiment(text: String) async throws {\r\n  // Initialize explicit API key credentials\r\n  let client = try LanguageServiceClient()\r\n\r\n  // Configure request using structured builder closure\r\n  let document = Document().with {\r\n    $0.type = .plainText\r\n    $0.source = .content(text)\r\n  }\r\n\r\n  let response = try await client.analyzeSentiment(\r\n    request: AnalyzeSentimentRequest().with { $0.document = document }\r\n  )\r\n\r\n  if let sentiment = response.documentSentiment {\r\n    print(&amp;quot;Document sentiment score: \\(sentiment.score)&amp;quot;)\r\n  }\r\n}&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9e0c90&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Notice how &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;Document().with { ... }&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; avoids verbose temporary variables or mutating setters by providing a clean, thread-safe configuration closure.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Networking, transport, and authentication&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The repository splits infrastructure primitives into modular packages under &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;packages/&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;code style="vertical-align: baseline;"&gt;swift-google-cloud-auth&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;: Implements &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/docs/authentication/application-default-credentials"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Application Default Credentials&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; discovery, service account JWT signing, external account exchange for Workload Identity Federation, and API keys.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;code style="vertical-align: baseline;"&gt;swift-google-cloud-wkt&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;: Provides idiomatic Swift types for Google Protocol Buffer well-known types, including nanosecond-precision &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;Timestamp&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; representations that bridge cleanly to Swift's &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;Date&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;code style="vertical-align: baseline;"&gt;swift-google-cloud-gax&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;: Handles Google API Extensions such as automated retry loops, exponential backoff, and pagination state machines.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;When you initialize any client library without arguments, &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;Credentials.default()&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; automatically scans your environment (&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;GOOGLE_APPLICATION_CREDENTIALS&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, quota project variables, or the local Google Cloud CLI configuration) and authenticates connections over gRPC or HTTP/2.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;If you need to programmatically override credentials with an API key or attach custom access headers, you can pass explicit configuration options. For example, you could modify the previous example to use the following:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;import Foundation\r\nimport GoogleCloudAuth\r\nimport GoogleCloudGax\r\nimport GoogleCloudLanguageV2\r\n\r\nfunc analyzeTextSentiment(apiKey: String, text: String) async throws {\r\n  // Initialize explicit API key credentials\r\n  let credentials = try Credentials(configuration: .apiKey(apiKey))\r\n  let client = try LanguageServiceClient(\r\n    ClientOptions().with { $0.credentials = credentials }\r\n  )\r\n\r\n  // Configure request using structured builder closure\r\n  let document = Document().with {\r\n    $0.type = .plainText\r\n    $0.source = .content(text)\r\n  }\r\n\r\n  let response = try await client.analyzeSentiment(\r\n    request: AnalyzeSentimentRequest().with { $0.document = document }\r\n  )\r\n\r\n  if let sentiment = response.documentSentiment {\r\n    print(&amp;quot;Document sentiment score: \\(sentiment.score)&amp;quot;)\r\n  }\r\n}&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9e1ed0&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;The autogenerated client ecosystem&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Google Cloud operates a vast ecosystem of APIs whose schemas update regularly. The teams supporting &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;google-cloud-swift&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; use code generators to automatically update the client libraries with the latest features and with new APIs. Using code generators produces stable APIs, without disruptive breaking changes. While the releases are on a fixed cadence, please contact Cloud Customer Care if you need a particular feature or API urgently.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Whether you need to rotate keys in &lt;/span&gt;&lt;a href="https://cloud.google.com/secret-manager"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Secret Manager&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; or invoke multimodal inference models via the &lt;/span&gt;&lt;a href="https://cloud.google.com/vertex-ai/generative-ai/docs/gemini-v1"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Gemini API&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; on &lt;/span&gt;&lt;a href="https://cloud.google.com/vertex-ai"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Gemini Enterprise Agent Platform&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, the generated SDKs follow consistent naming and async/await signatures.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;These generated clients offer more than plain unary RPC wrappers. They also offer wrappers that simplify application development. For example, iterating over long results involves fetching pages of results with one RPC, iterating over the page of results, and then preparing a new request to retrieve the following page. Using the generated clients this becomes an asynchronous iterator. This example shows how to query project secrets using &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;GoogleCloudSecretManagerV1&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;import Foundation\r\nimport GoogleCloudSecretManagerV1\r\n\r\n@main\r\nstruct SecretManagerQuickstart {\r\n  static func main() async throws {\r\n    guard let projectId = CommandLine.arguments.dropFirst().first else {\r\n      print(&amp;quot;Usage: SecretManagerQuickstart &amp;lt;projectId&amp;gt;&amp;quot;)\r\n      exit(1)\r\n    }\r\n\r\n    // Connects using Application Default Credentials automatically\r\n    let client = try SecretManagerServiceClient()\r\n\r\n    let request = ListSecretsRequest().with {\r\n      $0.parent = &amp;quot;projects/\\(projectId)&amp;quot;\r\n    }\r\n\r\n    // Async sequence streams pages of secrets automatically\r\n    print(&amp;quot;Secrets in project \\(projectId):&amp;quot;)\r\n    for try await item in try client.listSecretsByItem(request: request) {\r\n      print(&amp;quot; - \\(item.name)&amp;quot;)\r\n    }\r\n  }\r\n}&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9e2ad0&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The pagination response returns an asynchronous sequence (&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;AsyncSequence&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;). You can iterate over items with &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;for try await&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; while the client library fetches subsequent pages in the background over non-blocking NIO channels.&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;span style="vertical-align: baseline;"&gt;Where to go next&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;With &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;google-cloud-swift&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, server-side Swift developers can write end-to-end cloud infrastructure with compile-time race safety, native async/await ergonomic APIs, and zero OS thread congestion.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;To inspect the source code, open issues, or contribute new veneers, visit the official repository at &lt;/span&gt;&lt;a href="https://github.com/googleapis/google-cloud-swift" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;googleapis/google-cloud-swift&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Want to discuss server-side Swift architectures or Cloud Run containerization? Join the &lt;/span&gt;&lt;a href="https://developers.google.com/program" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Google Developer Program&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; to continue the conversation.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;</description><pubDate>Thu, 01 Oct 2026 04:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/topics/developers-practitioners/introducing-the-server-side-cloud-swift-sdk/</guid><category>Developers &amp; Practitioners</category><media:content height="540" url="https://storage.googleapis.com/gweb-cloudblog-publish/images/introducing-the-server-side-cloud-swift-sdk-.max-600x600.png" width="540"></media:content><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Introducing the Server Side Cloud Swift SDK</title><description></description><image>https://storage.googleapis.com/gweb-cloudblog-publish/images/introducing-the-server-side-cloud-swift-sdk-.max-600x600.png</image><site_name>Google</site_name><url>https://cloud.google.com/blog/topics/developers-practitioners/introducing-the-server-side-cloud-swift-sdk/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Karl Weinmeister</name><title>Director, Developer Relations</title><department></department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Carlos O'Ryan</name><title>Software Engineer</title><department></department><company></company></author></item><item><title>Democratizing Managed Lustre with lower cost and frictionless development</title><link>https://cloud.google.com/blog/topics/developers-practitioners/democratizing-managed-lustre-with-lower-cost-and-frictionless-development/</link><description>&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;This is the first of a two-part series exploring how Google Cloud is bringing the foundational values of a high-performance parallel filesystem–TB/s throughput, sub-ms latency at high client scale, and POSIX support–to a broader set of use cases and users.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Historically, due to the cost and special purpose nature of parallel filesystems, colder data had to be stored outside of the filesystem and AI developers have had to maintain separate, slower environments for writing code, compiling libraries, and managing repositories. This fragmentation increases the toil of manual data staging, dataset copying, and managing disjointed namespaces.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Google Cloud Managed Lustre is solving these problems through our &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;6 cents/GB*month Dynamic Tier&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;and by optimizing Managed Lustre performance for a range of development tasks and workloads – making Managed Lustre a “One-Stop Shop” for high-performance AI and HPC workloads.&lt;/strong&gt;&lt;/p&gt;
&lt;h2&gt;&lt;strong style="vertical-align: baseline;"&gt;Lower Cost: More Lustre for Less with the Dynamic Tier&lt;/strong&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The Managed Lustre Dynamic Tier provides &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;sub-ms latency for hot data, which allows you to store all of your data in a single namespace, and costs only 6 cents/GB*month&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Throughput, capacity scale and client scale:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Throughput scales linearly with capacity up to 80 PB, while sub-ms latency for hot data remains stable as you scale to tens of thousands of clients.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Single-flat fee:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Predictable pricing. No independent charges for disk media types, data movement within the namespace, or metadata IOPS.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Read Latencies:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Sub-ms latencies for High-Performance Cache (SSD).  The Capacity Pool (“HDD”) is built on Google Cloud Hyperdisk throughput, which has an &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/compute/docs/disks/hd-types/hyperdisk-throughput"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;average read latency of 10 to 30 ms&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h4&gt;&lt;strong style="vertical-align: baseline;"&gt;Recommended workloads for Dynamic Tier&lt;/strong&gt;&lt;/h4&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Multi-Epoch Training and/or Training with Optimized Fetch Sizes: &lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;Hot data is promoted to the High Performance Cache (SSD) after the first run. Larger data prefetch will allow you to take advantage of the Dynamic Tier cost structure and gain from low-latency SSD.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Write-Heavy Checkpointing:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Bursty checkpoint writes land directly in the High Performance Cache. Older checkpoints are transparently demoted to the Capacity Pool (HDD).&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Rapid Checkpoint Restore:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;  New checkpoints are written to the High Performance Cache, enabling low-latency checkpoint restores.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Interactive Snappiness for Developers:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Low-latency tasks like git cloning, compiling libraries, or running notebooks benefit from a local-disk feel (~300µs average read latencies) on the same shared workspace hosting large training sets.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;&lt;strong style="vertical-align: baseline;"&gt;Frictionless development: Lustre as a one-stop shop for developer’s workloads&lt;/strong&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;In addition to Managed Lustre’s scalability for large AI and HPC workloads (checkpoint/restart/data-loading), it also meets the demands for interactive work, meaning developers can start on Managed Lustre and stay on Managed Lustre throughout the entire workload lifecycle:&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;Unified Foundation &amp;amp; Interactive Performance&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Consolidates the AI and HPC lifecycle into a single namespace, providing a "local disk" feel for interactive work (Read more about the &lt;/span&gt;&lt;a href="https://cloud.google.com/products/managed-lustre"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;latency benefits of Managed Lustre experienced by Salesforce and others&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;).&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Latency:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; ~300µs average read latency—delivering up to 4x better responsiveness than alternative distributed file systems.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Accelerated Setup:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Untar the Linux kernel in ~2 minutes (4.7x faster than alternative file solutions), run a 20-worker parallel git clone of Python in ~40 seconds, compile Python in ~200s.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/democratizing-lustre-with-lower-cost-and-f.max-1000x1000.jpg"
        
          alt="democratizing-lustre-with-lower-cost-and-frictionless-development-02-managed-lustre-performance-values"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/democratizing-lustre-with-lower-cost-and-f.max-1000x1000_xLOfL6w.jpg"
        
          alt="democratizing-lustre-with-lower-cost-and-frictionless-development-01-performance-comparision"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;High-Concurrency Broadcast &amp;amp; Cluster Startup&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Managed Lustre maximizes GPU ROI by preventing storage bottlenecks during cluster initialization. &lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;When thousands of worker nodes attempt to read the exact same file simultaneously (such as a shared model checkpoint, base weights, or container layer), traditional distributed file systems can choke on localized hotspotting, leaving high-cost GPU clusters idle for minutes.&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Improves Aggregate Throughput for a large number of clients reading the same file:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Demonstrates a 67% improvement over alternative file solutions.&lt;sup&gt;1&lt;/sup&gt;&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Parallel Loading:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Imports libraries like PyTorch across 4,000+ processes in under 60 seconds.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;&lt;sup&gt;&lt;span style="vertical-align: baseline;"&gt;1 &lt;/span&gt;&lt;/sup&gt;&lt;span style="font-style: italic;"&gt;Storage system specs: 500 MBps per TiB tier of Managed Lustre, 108,000 GiB capacity. Zonal Filestore at 102,400 GiB capacity. Average throughput of 36.7 GB/s to 2,048 client VMs reading the same 40 GiB file.&lt;/span&gt;&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/images/democratizing-lustre-with-lower-cost-and-f.max-1000x1000_9kShfM1.jpg"
        
          alt="democratizing-lustre-with-lower-cost-and-frictionless-development-03-performance-advantage"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h2&gt;&lt;strong style="vertical-align: baseline;"&gt;Run One-Stop Shop Workflows for Yourself&lt;/strong&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Here is the code for the tests we’ve run, so that you can perform your own testing.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;Low latency for interactive access&lt;/strong&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;We used &lt;/span&gt;&lt;a href="https://github.com/axboe/fio" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;fio&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; to emulate small, low-concurrency reads and writes:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;# Read workload\r\nfio --ioengine=libaio --filesize=100M --ramp_time=2s \\\r\n    --runtime=2m --time_based --numjobs=1 --direct=1 --verify=0 --randrepeat=0 \\\r\n    --group_reporting --directory=~/LUSTRE_MOUNT \\\r\n    --name=randread --blocksize=4k --iodepth=1 --readwrite=randread \\\r\n    --buffer_compress_percentage=50\r\n\r\n# Write workload\r\nfio --ioengine=libaio --filesize=100M --ramp_time=2s \\\r\n    --runtime=2m --time_based --numjobs=1 --direct=1 --verify=0 --randrepeat=0 \\\r\n    --group_reporting --directory=~/LUSTRE_MOUNT \\\r\n    --name=randwrite --blocksize=4k --iodepth=1 --readwrite=randwrite \\\r\n    --buffer_compress_percentage=50&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9d9ad0&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;Accelerated setup&lt;/strong&gt;&lt;/h3&gt;
&lt;h4&gt;&lt;strong style="vertical-align: baseline;"&gt;How to run Linux untar&lt;/strong&gt;&lt;/h4&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;# Download a kernel tarball\r\nwget -P /tmp https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.18.9.tar.xz\r\n\r\n# Extract the archive to the Lustre mount\r\nmkdir ~/LUSTRE_MOUNT/kernel\r\ntar -C ~/LUSTRE_MOUNT/kernel -xf /tmp/linux-5.18.9.tar.xz&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9db990&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;In the above use case, you will want to take care to avoid the metadata performance tax that can come from running as root (Namely, &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;tar&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; issues&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt; chown&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; and&lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt; chmod&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; calls to make extracted files’ owner+permissions match the ones recorded in the archive.).  If you still wish to run as root (and have verified that this approach is compatible with your setup), you may &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;specify&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;`--no-same-owner --no-same-permissions`&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; &lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;in order to ensure that extracted files maintain&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; root&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt; as owner and have root's default file permissions. In other words, it makes extraction as root behave like extraction as non-root (by ignoring the owner+permissions in the archive).&lt;/span&gt;&lt;/p&gt;
&lt;h4&gt;&lt;strong style="vertical-align: baseline;"&gt;How to run Python gitclone&lt;/strong&gt;&lt;/h4&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;git config --global checkout.workers 20\r\nmkdir ~/LUSTRE_MOUNT/python\r\ngit clone https://github.com/python/cpython.git ~/LUSTRE_MOUNT/python&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9da550&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;How to run Python compile&lt;/strong&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;pushd ~/LUSTRE_MOUNT/python\r\n./configure &amp;gt; /dev/null\r\nmake &amp;gt; /dev/null\r\npopd&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9d9e10&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h3&gt;&lt;strong style="vertical-align: baseline;"&gt;High scale distribution&lt;/strong&gt;&lt;/h3&gt;
&lt;h4&gt;&lt;strong style="vertical-align: baseline;"&gt;Aggregate throughput for distributing one large file to many nodes&lt;/strong&gt;&lt;/h4&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Run the below on each client VM:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;# Start fio in server mode\r\nfio --server&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba66e5d0&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph"&gt;&lt;p data-block-key="iadne"&gt;Run the below on a selected client VM:&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;quot;# Create a 40 GiB file\r\nfio --name=job1 \\\r\n    --ioengine=libaio \\\r\n    --direct=1 \\\r\n    --buffer_compress_percentage=50 \\\r\n    --blocksize=4m \\\r\n    --iodepth=32 \\\r\n    --filesize=40g \\\r\n    --readwrite=write \\\r\n    --filename ~/LUSTRE_MOUNT/40gb_test\r\n\r\n# Create an fio job file for the read workload\r\ncat &amp;lt;&amp;lt;&amp;#x27;EOF&amp;#x27; &amp;gt; /tmp/read.fio\r\n[job1]\r\nfilename=${HOME}/LUSTRE_MOUNT/40gb_test\r\nrw=read\r\nbs=4m\r\nexitall_on_error=1\r\nEOF\r\n\r\n# Run the read workload using all client VMs in ~/hostfile\r\nfio --client ~/hostfile /tmp/read.fio&amp;quot;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba66d8d0&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h4&gt;&lt;strong style="vertical-align: baseline;"&gt;Parallel loading of libraries across many processes&lt;/strong&gt;&lt;/h4&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Run the below on a selected client VM:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;# Install PyTorch in a virtual env\r\npython3 -m venv ~/LUSTRE_MOUNT/env\r\nsource ~/LUSTRE_MOUNT/env/bin/activate\r\npip3 install --upgrade pip\r\npip3 install torch torchvision torchaudio \r\ndeactivate\r\n\r\n# Import PyTorch on all client VMs in ~/hostfile, 4 processes per host\r\nmpirun --allow-run-as-root --oversubscribe --hostfile ~/hostfile -N 4 \\\r\n  bash -c \&amp;#x27;source ~/LUSTRE_MOUNT/env/bin/activate &amp;amp;&amp;amp; python3 -c &amp;quot;import torch&amp;quot;\&amp;#x27;&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba9cf650&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h1&gt;&lt;strong style="vertical-align: baseline;"&gt;Looking ahead and next steps&lt;/strong&gt;&lt;/h1&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;By eliminating the manual data staging tax and lowering entry costs with the Dynamic Tier, Google Cloud Managed Lustre is evolving from an elite, single-purpose engine into a highly versatile, unified storage fabric for the entire AI lifecycle.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;In the second part of this series, we will focus on &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;upcoming object integration features&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;. Stay tuned!&lt;/span&gt;&lt;/p&gt;
&lt;h2&gt;&lt;strong style="vertical-align: baseline;"&gt;Next steps&lt;/strong&gt;&lt;/h2&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Run the benchmarks yourself (if you haven’t already): Deploy a Google Cloud Managed Lustre instance using the &lt;/span&gt;&lt;a href="https://console.cloud.google.com/"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Google Cloud console&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; and run tests provided above to benchmark your own workloads.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Explore the Dynamic Tier: Read the &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/managed-lustre/docs/performance-tiers"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Google Cloud Managed Lustre Documentation&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; to learn more about the Dynamic Tier.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Stay tuned for Part 2: In the next installment of this series, we will dive deep into upcoming object integration features and how they further simplify AI and HPC storage.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;span style="vertical-align: baseline;"&gt;Get started with centralizing your development-to-training lifecycle on Google Cloud Managed Lustre!&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;&lt;/div&gt;</description><pubDate>Thu, 01 Oct 2026 04:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/topics/developers-practitioners/democratizing-managed-lustre-with-lower-cost-and-frictionless-development/</guid><category>Developers &amp; Practitioners</category><media:content height="540" url="https://storage.googleapis.com/gweb-cloudblog-publish/images/democratizing-lustre-with-lower-cost-and-fri.max-600x600_YyI34FQ.jpg" width="540"></media:content><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Democratizing Managed Lustre with lower cost and frictionless development</title><description></description><image>https://storage.googleapis.com/gweb-cloudblog-publish/images/democratizing-lustre-with-lower-cost-and-fri.max-600x600_YyI34FQ.jpg</image><site_name>Google</site_name><url>https://cloud.google.com/blog/topics/developers-practitioners/democratizing-managed-lustre-with-lower-cost-and-frictionless-development/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Barak Epstein </name><title>Senior Product Manager, Google Cloud Managed Lustre</title><department></department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Yuval Ehrental</name><title>Software Engineer, Google Cloud Managed Lustre</title><department></department><company></company></author></item><item><title>Empower your agents with the Google Cloud CLI remote MCP server</title><link>https://cloud.google.com/blog/products/ai-machine-learning/google-cloud-cli-remote-mcp-server-in-preview/</link><description>&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Today, we’re expanding our ecosystem of managed remote MCP servers by introducing the &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/sdk/use-gcloud-mcp"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Google Cloud CLI remote MCP server&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; in preview.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Powered by the popular &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/sdk/gcloud"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;gcloud&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; and &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/bigquery/docs/reference/bq-cli-reference"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;bq (BigQuery)&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; command-line tools, this new server gives AI agents immediate, broad access to command-line operations for managing Google Cloud infrastructure and working with advanced BigQuery workflows securely and seamlessly. &lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Why CLI matters for AI agents&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Agents are increasingly performing complex cloud operations, but standardizing how they interact with backend systems remains a challenge. The Google Cloud CLI remote MCP server bridges this gap by packaging the versatility of hundreds of gcloud and bq commands into one single MCP server. This results in two strong benefits for the agent:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Higher-level abstractions:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; CLI commands package complex multi-step workflows, validation checks, and high-level operations into unified commands rather than requiring multi-step API orchestration.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Leverages model training:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; LLMs are heavily pre-trained on public command-line documentation, syntaxes, and usage examples, making CLI invocation intuitive and highly accurate for models.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;The benefits of putting CLI behind remote MCP&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Managing cloud infrastructure with AI agents traditionally requires installing and maintaining Google Cloud CLI binaries inside agent execution environments. The Cloud CLI remote MCP server bridges CLI capabilities with MCP benefits by providing an isolated execution sandbox on Google Cloud infrastructure. This solves key infrastructure challenges:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Simplified dependency and runtime management:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; For teams building custom agents, maintaining local CLI versions and dependencies across dev, test, and production environments creates operational overhead. Remote MCP eliminates local installations and runtime maintenance.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Access for web-based agent endpoints:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Web-hosted agent platforms and web interfaces (such as Gemini Enterprise and other hosted enterprise agent platforms) run in environments where users cannot control or install local packages. Remote MCP enables secure, managed access to Google Cloud CLI operations directly from these surfaces.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Enterprise-grade security and governance&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Connecting an AI agent to your infrastructure requires strict, enterprise-ready safeguards. This remote server leverages Google Cloud's standard identity and governance frameworks to keep your environments secure:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Zero ambient credentials:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; The server isolates execution in a network-restricted proxy boundary with no ambient credentials. Authentication and authorization are handled through &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/iam/docs/agent-identity-overview"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Agent Identity&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, &lt;/span&gt;&lt;a href="https://developers.google.com/identity/protocols/oauth2" rel="noopener" target="_blank"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;OAuth 2.0&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, and &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/iam/docs"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Identity and Access Management (IAM)&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Strict policy enforcement:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Every command executed through the remote MCP server is run with the permissions of the authenticated caller identity. Both standard IAM permissions and organization policy service constraints are strictly enforced against downstream target resources.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Advanced protection with Model Armor:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; To minimize the risks associated with AI tool calling, the Cloud CLI remote MCP server integrates with &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/model-armor/model-armor-mcp-google-cloud-integration"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Model Armor&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;. You can proactively screen LLM prompts and responses to protect against risks like prompt injection and malicious inputs.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Cloud audit logging:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; The Cloud CLI remote MCP server can be configured to log every tool invocation to Audit Logs (Data Access logs under cloudcli.googleapis.com/mcp). Security teams can gain full visibility into caller identities, OAuth clients, and IAM authorization decisions (mcp.googleapis.com/tools.call) without exposing sensitive command payloads or personally identifiable information (PII).&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Connecting to the Google Cloud CLI Remote MCP Server&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Integrating cloud management into your agents no longer requires packaging Google Cloud CLI binaries, managing local execution runtimes, or maintaining dependencies inside agent container images. Because the Google Cloud CLI remote MCP server implements the standard Model Context Protocol, any MCP-compatible agent platform or orchestration runtime can connect immediately via standard configuration:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;
&lt;div class="block-code"&gt;&lt;dl&gt;
    &lt;dt&gt;code_block&lt;/dt&gt;
    &lt;dd&gt;&amp;lt;ListValue: [StructValue([(&amp;#x27;code&amp;#x27;, &amp;#x27;{\r\n  &amp;quot;mcpServers&amp;quot;: {\r\n    &amp;quot;google-cloud-cli&amp;quot;: {\r\n      &amp;quot;uri&amp;quot;: &amp;quot;https://cloudcli.googleapis.com/mcp&amp;quot;,\r\n      ...\r\n    }\r\n  }\r\n}&amp;#x27;), (&amp;#x27;language&amp;#x27;, &amp;#x27;&amp;#x27;), (&amp;#x27;caption&amp;#x27;, &amp;lt;wagtail.rich_text.RichText object at 0x7f02ba66c690&amp;gt;)])]&amp;gt;&lt;/dd&gt;
&lt;/dl&gt;&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Your agent immediately gains access to execute gcloud and bq commands in a secure, network-isolated cloud sandbox.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;Authentication is handled via keyless Agent Identity for hosted Google Cloud platforms, or standard OAuth 2.0 for external runtimes. For authentication options, see the &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/mcp/set-up-authentication-mcp-servers"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;MCP Authentication Guide&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Bringing infrastructure management to agents&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;a href="https://docs.cloud.google.com/sdk/use-gcloud-mcp"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;The Cloud CLI remote MCP server&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; exposes two powerful tools, &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;run_gcloud_command&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt; and &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;run_bq_command&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, giving your AI agents broad, immediate access to Google Cloud operations through natural language.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong style="vertical-align: baseline;"&gt;Managing cloud infrastructure with &lt;/strong&gt;&lt;code&gt;&lt;strong style="vertical-align: baseline;"&gt;run_gcloud_command&lt;/strong&gt;&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;With &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;run_gcloud_command&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, agents can execute the full breadth of gcloud operations to manage, diagnose, and secure your Google Cloud environment. An example follows:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Observability and incident diagnostics&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: An agent streamlines incident diagnostics by automating command execution and reducing context-switching across tools.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/original_images/1_IPA9ALa.gif"
        
          alt="1"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Extending BigQuery operations with the run_bq_command tool&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;While the &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/bigquery/docs/use-bigquery-mcp"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;BigQuery MCP server&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; already helps organizations analyze and explore data using AI agents, with the introduction of &lt;/span&gt;&lt;code style="vertical-align: baseline;"&gt;run_bq_command&lt;/code&gt;&lt;span style="vertical-align: baseline;"&gt;, agents can now tackle advanced BigQuery tasks such as resource allocation, job monitoring, and task scheduling by unlocking the full scope of &lt;/span&gt;&lt;a href="https://docs.cloud.google.com/sdk/reference/mcp#mcp-tools"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;bq CLI&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt; functionality. Key capabilities include:&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Automating scheduled queries&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: An agent utilizes BigQuery Data Transfer Service configurations to schedule queries automatically. &lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Job and resource management&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt;: Gain deep insight into query execution details, including processed data volume, slot usage, and execution plans, as well as managing reservations.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;strong style="vertical-align: baseline;"&gt;Access and permissions control:&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; Data administrators and owners can inspect and update table permissions directly through the agent.&lt;/span&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;&lt;/div&gt;
&lt;div class="block-image_full_width"&gt;






  
    &lt;div class="article-module h-c-page"&gt;
      &lt;div class="h-c-grid"&gt;
  

    &lt;figure class="article-image--large
      
      
        h-c-grid__col
        h-c-grid__col--6 h-c-grid__col--offset-3
        
        
      "
      &gt;

      
      
        
        &lt;img
            src="https://storage.googleapis.com/gweb-cloudblog-publish/original_images/2_JA5HWbH.gif"
        
          alt="2"&gt;
        
        &lt;/a&gt;
      
    &lt;/figure&gt;

  
      &lt;/div&gt;
    &lt;/div&gt;
  




&lt;/div&gt;
&lt;div class="block-paragraph_advanced"&gt;&lt;h3&gt;&lt;span style="vertical-align: baseline;"&gt;Pricing and availability&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;span style="vertical-align: baseline;"&gt;The Google Cloud CLI MCP server is available today in public preview. There is no additional charge to use the MCP server itself. You pay only for the GCP resources you create and any applicable data transfer costs.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;a href="https://docs.cloud.google.com/sdk/use-gcloud-mcp"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;To get started&lt;/span&gt;&lt;/a&gt;&lt;span style="vertical-align: baseline;"&gt;, enable the Cloud CLI Execution API (`&lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;cloudcli.googleapis.com&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;`) in your Google Cloud project, grant the required &lt;/span&gt;&lt;strong style="vertical-align: baseline;"&gt;MCP Tool User&lt;/strong&gt;&lt;span style="vertical-align: baseline;"&gt; (`&lt;/span&gt;&lt;span style="font-style: italic; vertical-align: baseline;"&gt;roles/mcp.toolUser&lt;/span&gt;&lt;span style="vertical-align: baseline;"&gt;`) IAM role to your agent or user identity, and configure your MCP client to connect to `cloudcli.googleapis.com/mcp`.&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://docs.cloud.google.com/sdk/use-gcloud-mcp"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Use the Google Cloud CLI Remote MCP Server Guide&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://docs.cloud.google.com/sdk/reference/mcp#mcp-tools"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Google Cloud CLI MCP Reference&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://docs.cloud.google.com/mcp/overview"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Google Cloud Remote MCP Servers Overview&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://docs.cloud.google.com/mcp/supported-products"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Full List of Google OneMCP Servers&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://docs.cloud.google.com/mcp/set-up-authentication-mcp-servers"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Set up authentication to Google and Google Cloud MCP servers&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://docs.cloud.google.com/gemini-enterprise-agent-platform/govern/agent-identity-overview"&gt;&lt;span style="text-decoration: underline; vertical-align: baseline;"&gt;Gemini Enterprise Agent Platform &amp;amp; Agent Identity Overview&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li aria-level="1" style="list-style-type: disc; vertical-align: baseline;"&gt;
&lt;p role="presentation"&gt;&lt;a href="https://www.youtube.com/watch?v=-fb0ycu4kiU" rel="noopener" target="_blank"&gt;&lt;span data-rich-links='{"fple-t":"Automate Google Cloud with Cloud CLI Remote MCP Server","fple-u":"https://www.youtube.com/watch?v=-fb0ycu4kiU","fple-mt":null,"type":"first-party-link"}' style="text-decoration: underline; vertical-align: baseline;"&gt;Automate Google Cloud with Cloud CLI Remote MCP Server&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;&lt;/div&gt;</description><pubDate>Wed, 30 Sep 2026 16:00:00 +0000</pubDate><guid>https://cloud.google.com/blog/products/ai-machine-learning/google-cloud-cli-remote-mcp-server-in-preview/</guid><category>Data Analytics</category><category>Developers &amp; Practitioners</category><category>AI &amp; Machine Learning</category><og xmlns:og="http://ogp.me/ns#"><type>article</type><title>Empower your agents with the Google Cloud CLI remote MCP server</title><description></description><site_name>Google</site_name><url>https://cloud.google.com/blog/products/ai-machine-learning/google-cloud-cli-remote-mcp-server-in-preview/</url></og><author xmlns:author="http://www.w3.org/2005/Atom"><name>Prosper Nwankpa</name><title>Senior Engineering Manager, Google Cloud</title><department></department><company></company></author><author xmlns:author="http://www.w3.org/2005/Atom"><name>Adam Hwang</name><title>Software Engineering Manager, Google Cloud</title><department></department><company></company></author></item></channel></rss>