Application Modernization

Cool stuff Google Cloud customers built, April edition: BMW big on SLMs, MLB’s Scout Insights AI, personalized resort experiences

Wed, 15 Apr 2026 16:00:00 +0000

AI and cloud technology are reshaping every corner of every industry around the world. Without our customers, who are building the future on our platform, there would be no Google Cloud. In this regular round-up, we dive into some of the exciting projects redefining businesses, shaping industries, and creating new categories.

For our latest edition, we learn why BMW Group is experimenting with small language models (SLMs); catch AI-powered commentary from Major League Baseball; hit the slopes with Vail Resort’s AI concierge; build an intelligent grid with CTC Global; witness how ID.me created secure global scale; and see how Manhattan Associates supply chain tools now handle 1 billion daily API calls.

Be sure to check back next month to see how more industry leaders and exciting startups are putting Google Cloud technologies to use. And if you haven’t already, please peruse our list of 1,001 real-world gen AI use cases from our customers.

BMW tests the big potential of small models

Who: As one of the world’s leading providers of premium cars and motorcycles, BMW Group is always at the forefront of automotive technology. This ethos pushed the company to test what type of AI language models are ideally suited to driving situations, where access to cloud-based LLMs isn’t always possible.

What they did: BMW Group wanted to explore the potential of small language models (SLMs), which could run within the limited hardware on a vehicle. Finding the right trade-off between size and capability requires careful optimization, and the sheer volume of viable combinations renders manual searches for the optimal configuration an incredibly tedious, if not impossible, undertaking. To overcome this challenge, BMW and Google built automated, reproducible workflows through executable pipelines using Vertex AI.

Why it matters: The path from a general-purpose LLM to a specialized SLM isn’t straightforward. Every choice — from type of quantization to characteristics and contents of the fine-tuning domain-specific dataset — affects the quality and efficiency of the final model. This creates an exponential range of configurations, each with different trade-offs. It’s a great example of using AI to scale an optimization problem for other AI.

Learn from us: “With automated pipelines, we can rapidly adapt models to our domain and rigorously test and evaluate them against domain-specific benchmarks. This allows us to iterate and optimize models in hours rather than days.” – Dr. Céline Laurent-Winter, vice president, Connected Vehicle Platforms at BMW Group

MLB Scout Insights: AI-powered color commentary

Who: Major League Baseball is famous for its colorful announcers. Now, MLB is bringing more baseball color straight to your pocket, and Gemini is helping give it a voice.

What they did: Each season, millions of baseball fans use the MLB app and tap over to the Gameday feature for live, up-to-the-pitch action across more than a dozen games. Starting this season, the league launched MLB Scout Insights in Gameday, which uses Gemini models to quickly scan decades of game and player data, cross-references it with situational game scenarios, and then delivers game-relevant context during key matchups.

Why it matters: Given the sport’s storied history, 162-game regular season, and global reach, baseball fans are among the most sophisticated and passionate out there. To keep them engaged with Gameday and the MLB app, the league wanted to deliver insights that truly felt meaningful and interesting. Building the tool meant answering a rather squishy question: What makes an insight actually insightful, not just an accurate fact, and how can an AI learn that distinction? The answer came from some clever “surprisal” analysis.

Learn from us: "With Scout Insights, every fan can feel like the smartest person in the stands, at the water cooler, or on the couch. It’s about deepening connections to the game, and sharing that passion with others. That’s the magic of sports, and we’re making more of it possible with the magic of AI." – Josh Frost, senior vice president of product & Matt Graser, director of engineering, Major League Baseball

Vail Resorts makes personalized AI assistance easy

Who: Vail Resorts operates some of the most iconic and beloved mountain destinations in the world, including Whistler Blackcomb, Park City Mountain, Stowe, and Crested Butte.

What they did: Vail Resorts launched My Epic Assistant during the 2024-2025 snow season, and expanded it this year to add even more AI-powered chat features powered by Google’s powerful Gemini models. The result is an agentic guide to the slopes that can help skiers and snowboarders decide on the right season pass, share the latest snow report, check on lesson preparations, or suggest a good stop for cocoa.

Why it matters: Vail Resorts wanted more than a chatbot; they wanted a digital concierge that understands the nuance between a powder day at Whistler and a family trip to Beaver Creek. As the company implemented and refined personalization, improved search, summary capabilities, and conversational flow within My Epic Assistant, the app has delivered a 45% reduction in escalation to human agents since launch.

Learn from us: "Utilizing tooling from Google Cloud, we could lean into agentic design patterns that gave us a way to unlock natural, personalized conversations. These boosted customer satisfaction, while reducing the need for manual intent design. These tools also let us combine flexibility and control to enable the assistant to respond fluidly but always within the boundaries of our brand, policies, and product strategy.”— The Vail Resorts technical team

CTC Global turns the smart grid into an intelligent one

Who: CTC Global is a leading manufacturer of advanced transmission conductors and power lines. While many nodes in the grid contain IoT sensors, it recognized a literal gap in the transmission lines themselves.

What they did: CTC’s new GridVista platform threads fiber-optic cable into its high-strength carbon fiber composite core, and connects these to monitoring technology built with AI and monitoring technology from Google Cloud and Tapestry. With GridVista, CTC can turn every inch of transmission into a smart sensor.

Why it matters: GridVista gives CTC grid operators an accurate and reliable view of what’s happening across the entire line — based on actual, real-time data from the entire length of the conductor, not point estimates from a static model of line conditions or the occasional clamped-on sensor. This means they can significantly improve safety, manage costs, increase the line’s capacity to transmit power, and enhance reliability with more precise insights about events that might trigger an outage.

Learn from us: “This awareness allows for a grid that can truly sense its own health in real time and provide unprecedented awareness of conditions on the entire line. Whether that’s real time storm impacts, ice load, wind load, branches on the wire, or temperatures on or under the line. The GridVista system truly represents next generation capabilities. ” — J.D. Sitton, CEO, CTC Global

ID.me reduces risk while scaling past 160 million users

Who: ID.me is transforming digital identity security for the modern era, offering a single login that lets you easily prove you’re you across a wide range of platforms and wallets.

What they did: ID.me currently serves more than 160 million users, including as many as 40,000 at any time, so they can prove their identity online as easily as flashing their driver’s license in person. Over the last two years, ID.me migrated more than 50 terabytes of data across 15 database instances to Google Cloud with minimal downtime. They also introduced a two-tier architecture with Cloud SQL supporting its smaller and more standard services, while AlloyDB runs heavier workflows that form the backbone of the ID.me platform.

Why it matters: AlloyDB AI has allowed ID.me to scale its systems to handle 10X-20X of what was possible before — and at a lower price to boot. That responsiveness and reliability led the U.S. federal government to recognize ID.me for its role in preventing large-scale fraud within national systems.

Learn from us: "We’ve been able to scale both our infrastructure and trust. With a platform that’s faster, smarter, and built to handle portable identity at massive scale, we’re one step closer to our goal: a secure, digital way to prove who you are, wherever you need it, that works everywhere you need it." — Kevin Liu, Cloud Platform Architect, ID.me

Manhattan Associates powers more than a billion daily API calls

Who: Manhattan Associates is a global leader in supply chain and omnichannel commerce solutions, offering tools and platforms that reach more than 2 billion people across 20 billion consumer touchpoints.

What they did: Manhattan Associates modernized its Manhattan Active SaaS platform by migrating from legacy Oracle and DB2 systems to Google Cloud databases. Each capability of Manhattan Active now runs as an independent, containerized service orchestrated by Google Kubernetes Engine (GKE). Data flows through Pub/Sub into BigQuery for real-time analytics, while Cloud Logging and Cloud Monitoring deliver observability at scale.

Why it matters: With its new microservices-first design, Manhattan gained the agility to evolve faster and the confidence that mission-critical operations would remain resilient across regions. With Cloud SQL and BigQuery, the company now processes more than a billion daily API calls with average response times of less than 150 milliseconds. This evolution supports hundreds of thousands of monthly active users across tens of thousands of stores and distribution centers. The new platform also created the foundation for Manhattan’s Agentic AI suite, which includes prebuilt agents — like the Intelligent Store Manager and Labor Optimizer — that coordinate real-time decisions across store and distribution center operations.

Learn from us: "Operationally, the platform has become more elastic and efficient. The system automatically handles hundreds of thousands of scaling events per day, ensuring performance remains consistent during peak surges without expensive overprovisioning." — Narayana Reddy Kothapu, Senior Director, Manhattan Associates & Rajkumar Ramani, Technical Director, Manhattan Associates

How a leading consumer insight brand uses Dataproc to hyper-personalise faster

Mon, 06 Apr 2026 16:00:00 +0000

At RVU, we have a clear and vital mission: empower people, transform industries.

For our market-leading home management and switching brands — Confused.com, Uswitch, Tempcover, Money.co.uk, and Mojo Mortgages — transparency and accurate information are everything. Today’s consumer expects more than a simple comparison table; they want personalized recommendations tailored to their unique circumstances.

Delivering on that promise — building a true personalization engine that powers all our brands — requires a data foundation capable of processing massive, complex datasets for sophisticated ML models. Today, our platform powers hundreds of automated personalization campaigns, optimized with billions of data points from across all our brands. We tackled this challenge using the power of Google Cloud and its two solutions for Apache Spark: Dataproc and Google Cloud Serverless for Apache Spark. Together, we’re making our mission a reality.

The high-speed engine for feature engineering

Our relationship with Google Cloud isn’t new. In fact, we've used BigQuery as our unified data platform for over a decade. Coming from a performance marketing background, we’ve always dealt with a lot of data, but we recognized early on that we’re not a digital infrastructure company. Instead, our focus must always be on where the value is. Managed solutions like BigQuery that eliminate infrastructure and capacity headaches were a natural fit from the start.

The key challenge was stitching together a meaningful and coherent picture of customer behavior across our brands — turning countless fragmented interactions into something that genuinely reflects how a user behaves, clicks, and makes decisions. Instead of relying on isolated events and aggregate views, we’ve had to build a platform capable of connecting these signals into a narrative that works for our machine learning models.

Using Dataproc to support this was a gamechanger. The biggest impact has been its role as our core high-speed Spark processing engine, primarily for feature engineering for our ML model development. Feature engineering, which is the crucial process of shaping all that raw customer data for our data science models, is a real value-driver for us. It’s where we have a marked competitive edge.

The result has been a significant leap in our innovation velocity. With Serverless for Apache Spark, we now have the ability to shape our customer data for feature engineering in just a matter of days. Previously, this would have taken weeks. We’ve also dramatically reduced our time-to-market, which also used to take weeks. Now, a new contractor can join the team and deliver a model, including exploratory data analysis and all feature engineering, in only a week and a half. That’s incredibly fast.

Delivering personalized experiences

By improving our speed of innovation, we’re better positioned to deliver a personalized user experience to our customers and partners.

Our hyper-personalization journey accelerated once we moved to Spark. We can now run heavyweight data processing jobs that crunch vast amounts of behavioral and contextual data, allowing us to build models that generate genuinely meaningful predictions.

These models help us understand not just what to say to a customer, but when and how to say it — selecting the right moment and the right channel to deliver personalized insight that genuinely resonates.

Building a future vision

Google’s Data Cloud directly aligns with our culture of prioritizing value, and its impact on our business is profound. I call it the network effect, where everything seamlessly connects within the same ecosystem: Our data resides in BigQuery, our ability to validate, enrich, and transform that data is tied to Dataproc and Serverless for Apache Spark, and our capacity to deploy the ML models spans the network. It’s all co-located and integrated, powering the real-time accuracy of our consumer brands and giving us a competitive advantage.

For our engineers, the big win is the lack of infrastructure they have to deal with. They can press a button that processes all the data in 10 minutes, rather than having to set up a network of clusters and servers and make them talk to each other. It’s incredibly efficient and frees up time for more valuable work like building and iterating data products.

Dataproc has upped our speed, scale, and agility. It also gives us the tools to innovate with AI as we build the future of hyper-personalization. Today, we’re proud to say RVU’s cutting-edge tech and data are helping millions of UK consumers make smarter, more informed decisions, and truly transforming industries.

Inspired by RVU's success? Whether you need persistent clusters with Dataproc or the agility of Serverless Spark, Google Cloud has a managed solution to help you focus on value, not infrastructure. Discover the right Spark on Google Cloud for your use case.

Centralized policy meets distributed logic: Getting to know Eventarc Advanced

Fri, 27 Feb 2026 17:00:00 +0000

Enterprise architects often face a fundamental dilemma: choosing between developer agility and organizational control. Development teams need to move fast and deploy independent microservices without waiting for permission. Security and compliance teams need to be safe, and ensure that data flow is observable and governed by policies.

That’s why we built Eventarc Advanced, a serverless eventing platform and the evolution of Eventarc Standard. Eventarc Advanced provides an improved architectural pattern for the modern cloud, where centralized policy meets distributed logic. By clearly separating the governance layer (the "bus") from the processing layer (the "pipeline"), Eventarc Advanced gives SecOps teams the visibility and control they demand, while freeing developers to choreograph AI agents and build event-driven applications with the autonomy they want. Eventarc Advanced became generally available in August 2025.

In this blog, we take a deeper look at the evolution of integration architectures — from service buses, to microservices, to where we are today — and go into depth with a real-world example. Let’s jump in.

The evolution of integration architectures

To understand the value of this new pattern, it helps to look at where we came from and why previous architecture patterns forced a compromise.

The centralized bottleneck of the Enterprise Service Bus

One early integration architecture approach was the Enterprise Service Bus (ESB), which prioritized centralized control. The ESB emerged to solve the "spaghetti architecture" of point-to-point integrations by providing a centralized communication layer that standardized how disparate systems interact. However, it often introduced serious pitfalls.

The primary issue was what’s referred to as a centralized logic trap. Organizations frequently embedded complex business logic — transformations and orchestration — directly into the governance layer. The resulting middleware layer was opaque, with critical business rules hidden from the developers who owned the services.

Consequently, integration changes typically required the intervention of a central middleware team. Development teams lost autonomy, forced to queue behind integration specialists to ship even minor features, often waiting weeks for updates.

Microservices’ governance gap

To address this, the industry shifted toward microservices (often described as "smart endpoints and dumb pipes"), distributing logic to give teams the autonomy they were looking for. For synchronous traffic (REST, gRPC), tools like API gateways and service meshes restored a layer of governance by enforcing policies like authentication and rate limiting at the infrastructure level.

However, as architectures shifted to Event-Driven Architecture (EDA) for greater resilience and decoupling, a new gap emerged. In a distributed, asynchronous world, centralized control often vanished. This created a governance gap where SecOps teams struggled to maintain order. Three issues emerged to the forefront:

The visibility void: Without a central policy, shadow IT services could silently subscribe to sensitive events without detection.
The policy problem: Enforcing data residency or PII masking is nearly impossible when the broker treats every message as an opaque blob.
The dependency risk: Without clear contracts, changing an event schema risks silently breaking unknown downstream consumers.

A new pattern: Centralized policy, distributed logic

Eventarc Advanced addresses the trade-off between control and speed with a novel architectural pattern: centralized policy meets distributed logic.

Eventarc Advanced maps these distinct responsibilities to two specific architectural resources that each correspond to a distinct role:

The bus: This governance layer is a managed, centralized hub where platform administrators enforce global constraints before events are routed. It synthesizes the centralized routing of the legacy ESB with the modern security architecture of a service mesh. It handles Identity and Access Management (IAM), including content-based access control, to strictly define who can publish, and integrates with VPC Service Controls to prevent data exfiltration.
The pipeline: Think of this distributed, team-owned resource as developers’ integration logic layer. This is where eventing patterns for AI agents and microservices are unlocked, allowing developers to configure event flow and delivery according to their specific business logic. Unlike many service meshes that treat data as opaque bits, the pipeline understands content. Developers can transform events, convert payloads between formats (like JSON to Avro), and configure retry policies and authentication independently.

In other words, by decoupling these duties, Eventarc Advanced provides the control of an ESB with the agility of microservices and the resilience of modern event-driven architectures.

How it works: A retail event mesh example

A typical Eventarc Advanced solution can be implemented with minimal configuration, providing a streamlined experience for both administrative governance and distributed integration logic. To see this model in practice, let's look at a real-world implementation of a retail event mesh.

Imagine an ecosystem at a global retailer with four autonomous teams in charge of the following services:

Commerce
Finance
Logistics
Intelligence (AI Insights Agent)

In a traditional setup, aligning these teams is difficult. The Intelligence team wants access to everything for their models, Finance wants to lock everything down for compliance, Logistics just needs a stable schema to ship boxes, and Commerce needs to roll out new features at a moment’s notice.

The foundation: Built on CloudEvents

Eventarc Advanced uses a data model based on the open CloudEvents standard, which can carry any type of payload. This helps ensure governance and discoverability while retaining flexibility. In our example, before a single event is published, the platform administrator mandates that every message must contain standard attributes and a specific custom extension for governance.

In this example, every event on the bus must carry the following attributes:

type: Standard identifiers for the event instance (e.g., com.retail.order.created)
source: A standard attribute identifying the producer (e.g., //commerce/frontend)
data_sensitivity: A custom extension attribute to categorize risk

In addition, the organization defines three data sensitivity levels:

restricted (High): Severe risk data like Credit Card Tokens or Tax IDs
confidential (Medium): PII like home addresses
general (Low): Safe operational data like Order IDs

This standardized metadata layer allows the bus to enforce policies based on specific attribute names — checking who sent the data (source) and what kind of data it is (data_sensitivity).

The workflow

With this model, the lifecycle of a single order becomes a secure flow where sensitivity changes at every step.

Order placement: The Commerce service publishes order.created to the Bus. The event’s data sensitivity is tagged as general. The AI Insights Agent service subscribes to analyze market trends.
Payment authorization: The Commerce service publishes payment.authorized tagged as restricted (containing a secure token). The Finance service subscribes to capture the token and executes the charge.
Settlement: The Finance service publishes payment.success tagged as general, signaling the transaction is safe to fulfill without exposing financial secrets. Logistics subscribes to ship the box, and Intelligence AI Insights Agent is triggered to evaluate market trends for the next supply chain cycle.
Fulfillment: The Logistics service publishes shipment.ready tagged as confidential (containing the customer phone number). The Logistics own notification pipeline subscribes to it to trigger an SMS notification.

In a legacy architecture, mixing PCI, PII, and operational data on a single bus would be a compliance nightmare. With Eventarc Advanced, it’s a solved problem.

The bus: the governance layer

The platform administrator implements a secure strategy on the bus. Rather than blindly trusting internal services, they enforce global policies that inspect these CloudEvents attributes using fine-grained access control (FGAC).

Enforcing source integrity

To ensure a compromised service cannot spoof events, the bus administrator enforces the producer's identity to match the source attribute.

For example, a bus policy can state that only the principal sa-commerce@retail.com can publish events that match the expression message.source.startsWith("//commerce/"). If the Intelligence AI Insights Agent service tries to publish an event claiming to be from //commerce/payments, the bus rejects the request.

Enforcing a data classification

To ensure every event is categorized, the bus administrator requires that every payload received by the bus includes a valid sensitivity attribute. A bus policy can check that message.data_sensitivity is one of ['general', 'confidential', 'restricted']. This guarantees that the event mesh contains only classified, governance-ready data.

The Pipeline: the logic layer - autonomous team innovation

With the security posture established on the bus, development teams can then use pipelines to solve complex integration challenges entirely within their own domains. Let’s take a look at a few of these challenges.

Schema-aware formats conversion and payload transformation

The Logistics team decides to upgrade their warehouse robots to use high-efficiency protocol buffers. Instead of forcing the Finance team to change their JSON output (which would break other consumers), Logistics configures a transformation step in their own pipeline.

A typical com.retail.payment.success event from Finance arrives as JSON:

code_block: <ListValue: [StructValue([('code', '{\r\n "id": "89d5663e-789e-4d9f-a65f-f7d83742d987",\r\n "source": "//finance/ledger",\r\n "type": "com.retail.payment.success",\r\n "data_sensitivity": "general",\r\n "datacontenttype": "application/json",\r\n "data": {\r\n "order_number": "ORD-2023-8841",\r\n "total_amount": 249.99,\r\n "currency": "USD",\r\n "transaction_id": "tx_77382910",\r\n "status": "SETTLED"\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f0a233c4730>)])]>

The warehouse robots service expects a binary Protobuf message:

code_block: <ListValue: [StructValue([('code', 'message PaymentConfirmed {\r\n string order_id = 1;\r\n double insured_value = 2;\r\n string currency_code = 3;\r\n string ledger_reference = 4;\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f0a233c4c40>)])]>

The Logistics team configures their pipeline to accept json as input and output to protobuf. To map the data, they use Common Expression Language (CEL) to configure a transformation:

code_block: <ListValue: [StructValue([('code', '// CEL Transformation to Construct the target Protobuf message\r\n{\r\n "order_id": message.data.order_number,\r\n // 110% of total to cover replacement cost\r\n "insured_value": message.data.total_amount * 1.1,\r\n // Standardize currency to uppercase\r\n "currency_code": message.data.currency.upperAscii(),\r\n "ledger_reference": message.data.transaction_id,\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f0a2139c550>)])]>

This transformation not only maps the input but also applies business logic — calculating the insured value and normalizing the currency code. The Logistics team implements this modernization without a single meeting with the Finance team.

Agentic workflows: Filtering and triggering AI agents

Eventarc Advanced enables agentic workflows by allowing pipelines to communicate directly with AI agents using open standard protocols like Agent2Agent (A2A) and Model Context Protocol (MCP), while also offering rich capabilities like filtering to optimize when those agents are invoked.

The Intelligence team uses a pipeline they name ai-insights and the A2A protocol to connect with an AI Insights Agent that proactively analyzes market trends based on placed orders. Because the agent’s processing is resource-intensive, the team uses a filter to only invoke the agent for high-value orders that warrant deeper analysis.

The pipeline filter is configured with the following expression:

code_block: <ListValue: [StructValue([('code', 'message.type == "order.created" && \r\ndouble(message.amount) > 5000.0'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f0a2139ca60>)])]>

When the filter is passed, the pipeline uses a HTTP Message Destination Binding (MDB) expression to directly trigger the agent. By defining a CEL template, the pipeline handles the complexity of constructing a native A2A SendMessage request to the AI strategic insights agent. This allows the agent to receive a conversational prompt derived from technical event data without any manual "glue code":

code_block: <ListValue: [StructValue([('code', '{\r\n "headers": headers.merge({ "Content-Type": "application/json", "A2A-Version": "1.0" }),\r\n "body": {\r\n "jsonrpc": "2.0",\r\n "id": message.id,\r\n "method": "message/send",\r\n "params": {\r\n "message": {\r\n "messageId": message.id,\r\n "role": "ROLE_USER",\r\n "parts": [\r\n { \r\n "text": "Analyze Order " + message.data.order_number + " for market trends." \r\n }\r\n ]\r\n }\r\n }\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f0a2139c790>)])]>

A similar prompt message can be crafted for other popular agentic communication protocols like MCP.

This combination of filtering and agentic protocol translation ensures that AI resources are used precisely where they add value. The Intelligence team implements this independently – without writing ingestion code or coordinating with the Commerce or Admin team. The agent can then publish its own strategic recommendation back to the bus, enabling a choreography of AI experts that turns standard cloud events into competitive intelligence.

Advanced API request modeling

When a shipment is ready, the Logistics team uses a pipeline to send an SMS using a legacy gateway API. Integrating with legacy third-party APIs often requires writing "glue code" services just to format requests.

The Logistics team eliminates this maintenance burden by configuring a dedicated pipeline to fully construct the exact request expected by the legacy service.

They use a HTTP Message Destination Binding CEL expression, which standardizes the phone number and maps it to the X-SMS-To HTTP header required by the API. It also construct the SMS text:

code_block: <ListValue: [StructValue([('code', '{\r\n "headers": { "X-SMS-To", \r\n message.data.phone.matches(\'^\\\\+1\') ?\r\n message.data.phone : \r\n \'+1\' + message.data.phone \r\n },\r\n\r\n "body": {\r\n "sms_text": "Order " + message.data.order_id + " shipped!"\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f0a2139c190>)])]>

Finally, they configure a robust retry policy (linear backoff, max five attempts) directly on the pipeline, so that temporary network interruptions don't result in lost notifications. In addition to HTTP endpoints, the pipeline supports guaranteed delivery and out-of-the-box authentication for destinations like Cloud Run, Pub/Sub, Bus, Workflows, and over 200 Google services.

The future of agile integration

Eventarc Advanced closes an important gap in event-driven architectures: It brings the same level of maturity to asynchronous communication by introducing the pattern of centralized policy, distributed logic.

For the Platform team, Eventarc Advanced provides assurance that a bus can strictly enforce integrity and confidentiality on every message, bringing "service-mesh-like" security to the event layer.
For the developer, it restores autonomy. The pipeline allows teams to filter, transform, convert, and route events to fit their specific needs, enabling them to treat events as first-class products rather than opaque artifacts.

This architecture lays the foundation for the next generation of intelligent applications. A secure, typed, and trustworthy event mesh can serve as the backbone for generative AI agents and real-time analytics, allowing you to safely expose business context to the systems that need it most.

Get started

Don't let governance slow down your innovation. Here are some Eventarc Advanced resources to get you on your way:

Learn more: Dive into the full capabilities of the Bus and Pipeline in the Eventarc Advanced documentation.
Get hands-on: Deploy the "Retail Event Mesh" scenario yourself and explore enterprise patterns with our Quickstarts and Tutorials.
Start building: Go to the Google Cloud console to configure your first bus and pipeline today.
Let's talk: Have a complex enterprise use case? Contact Google Cloud Sales to discuss how Eventarc Advanced fits into your broader integration strategy.

Cool stuff Google Cloud customers built, Feb. edition: Telco data reinvention; Golden State’s “G.O.A.T.T.”; John Lewis explores DORA

Thu, 26 Feb 2026 17:00:00 +0000

AI and cloud technology are reshaping every corner of every industry around the world. Without our customers, there would be no Google Cloud, as they are the ones building the future on our platform. In this regular round-up, we dive into some of the exciting projects redefining businesses, shaping industries, and creating new categories.

For our latest edition, we explore a new data approach for Vodafone and Fastweb; evaluating John Lewis Partnership’s developer platforms; the Golden State Warrior’s AI playbook; healthy, stable networks at Hackensack Meridian Health; and Ab Initio brings better context to data for AI.

Be sure to check back next year to see how more industry leaders and exciting startups are putting Google Cloud technologies to use. And if you haven’t already, please peruse our list of 1,001 real-world gen AI use cases from our customers.

Fastweb + Vodafone reimagined data workflows

Who: Following the acquisition of Vodafone Italy by Swisscom in 2025, these leading European telecom providers wanted to rethink how they serve customers and deliver timely, personalized experiences across mobile, broadband, and digital channels.

What they did: Both companies had already begun modernizing customer data workflows with BigQuery, but combining ecosystems exposed certain limits of the existing setup. In order to give every channel real-time access to accurate customer data, they implemented Spanner as a service and governance layer, delivering low-latency reads, horizontal scalability, high availability, and a fully managed environment with zero ops overhead. The team is also using Gemini to generate clear documentation directly from the code, which saves hours of manual work.

Why it matters: Using Spanner Graph allowed the organization to map lineage in a way that reflects how its platform actually works: which tables drive specific jobs, how transformations cascade, and where dependencies sit. Call centers now see more complete, up-to-date customer information, digital channels can rely on consistent data without custom integrations, and partners can access what they need with low latency through Apigee.

Learn from us: “Rebuilding our Customer 360 platform with Google Cloud services has already changed how Fastweb + Vodafone works. Workflow monitoring is simpler, pipelines are leaner, and real-time serving is now the norm. ” – Vincenzo Forciniti, IT AI Adoption & Platform Engineering Lead, Fastweb + Vodafone

John Lewis measures the value of its developer platform

Who: The John Lewis Partnership is a major UK retailer operating John Lewis department stores and Waitrose supermarkets. To power their digital transformation, they built the John Lewis Digital Platform (JLDP) to support dozens of product teams building high-quality software for johnlewis.com.

What they did: Moving beyond simple usage metrics, John Lewis developed a sophisticated, multi-stage approach to measuring the real value of their platform. They transitioned from initial speed-based metrics (like "Onboarding Lead Time") to a comprehensive model using DORA metrics and subjective engineer feedback via the DX platform. This included a custom "Technical Health" feature that uses small, automated jobs to monitor more than 35 health measures — such as Kubernetes best practices, security, and operational readiness — providing teams with real-time "traffic light" indicators of their service health.

Why it matters: By focusing on value rather than just activity, John Lewis ensured the platform was actually reducing friction for developers rather than just being a mandatory tool. Their automated Technical Health checks allow product teams to manage technical debt and security vulnerabilities proactively. This approach has decoupled centralized operations teams from individual services, leading to faster incident resolution (MTTR), fewer outages, and significant cost savings.

Learn from us: "Measurement is a journey, not a destination. Start by measuring something meaningful to your stakeholders, but be prepared to adapt as your platform evolves. The things that mattered when you were proving out the platform's viability are unlikely to be what are important several years later when your features are mature." – Alex Moss, Principal Platform Engineer, John Lewis Partnership

Hackensack Meridian Health de-risks network migration using VPC Flow Logs

Who: Hackensack Meridian Health is a leading not-for-profit healthcare organization and the largest hospital system in New Jersey. System reliability is a cornerstone value for HMH as they manage a vast network of hospitals, urgent care centers, and physician practices.

What they did: Preparing for a large-scale migration to a new Google Cloud network design, Hackensack Meridian Health used VPC Flow Logs and Flow Analyzer to eliminate the "black box" of hybrid traffic. By enabling logs on their Cloud Interconnect VLAN attachments, they captured granular telemetry — including source/destination IPs, ports, and protocols. They then exported this data to create a visual "who-is-talking-to-what" map. This allowed them to identify critical traffic patterns between on-premises data centers and specific Google Cloud regions, VPCs, and applications.

Why it matters: In a healthcare environment, even minor network disruptions can have major consequences. By mapping traffic proactively, Hudson Meridian Health pinpointed exactly which moments in the cutover carried the highest risk. This preparation allowed them to detect a migration issue in just three minutes and resolve it within five — a process that previously could have taken hours. Beyond migration, this level of visibility enables the organization to better manage capacity planning, cost attribution, and security compliance across their hybrid infrastructure.

Learn from us: "Getting a clear picture of our interconnect traffic always felt like a black box. Enabling VPC Flow Logs and feeding it into Flow Analyzer finally gave us the map we needed. Identifying those critical traffic flows before we changed any routes was key to de-risking the entire migration." — Randall Brokaw, Cloud Engineering Manager, Hackensack Meridian Health

The Golden State Warriors’ AI-powered back office

Who: The Golden State Warriors are one of the NBA’s most successful modern franchises. Behind their on-court wins are a specialized operations team who run what might be called organization’s "G.O.A.T.T." (Greatest of All-Time Technologies), a data and AI platform that helps drive game-time insights, trading decisions, and fan experience enhancements.

What they did: The Warriors transitioned from a "gut-feeling" culture to an "analytics-first" strategy by building an internal "digital brain" on Google Cloud. Using BigQuery and Gemini, the team now automates complex workflows that previously took hours, such as generating pre-game scouting reports. They use machine learning to run thousands of trade simulations that prioritize "team fit" over raw individual stats and employ computer vision to track the "shot quality" of every attempt in the NBA. On the business side, they built a content recommendation engine using the Discovery API to deliver personalized digital experiences to their global fan base.

Why it matters: This AI-driven approach narrows the decision tree for leadership, allowing them to focus human expertise on the most viable options. By automating the “science” of data processing, coaches and scouts have more time for the "art" of face-to-face training, planning, and player development. This integration has not only influenced on-court strategy — like the three-point revolution — but has also improved business efficiency, with employees now proactively bringing AI-driven ideas to the IT team rather than waiting for top-down mandates.

Learn from us: "You can never reach a point where either humans or machines are making all the decisions. The sweet spot is finding that middle ground where intuition and data converge on the same conclusion. Data helps us narrow our decision tree before we even start evaluating specific options." — Nick Manning, Senior Director of Consumer Products & Emerging Technology, Golden State Warriors

Ab Initio unlocks enterprise data for the agentic AI era

Who: Ab Initio is an enterprise software company specializing in high-volume data integration and governance. Their platform is trusted by large-scale organizations to manage complex data lifecycles across hybrid and multi-cloud environments.

What they did: To solve the challenge of grounding AI agents in accurate data, Ab Initio partnered with Google Cloud to integrate its data fabric with BigQuery, Dataplex Universal Catalog, and Gemini. They launched a suite of more than 500 metadata and data connectors that bridge the gap between legacy systems (like mainframes, COBOL, and SAS) and modern cloud environments. This integration provides field-level, end-to-end lineage, allowing Gemini to access well-documented, "AI-ready" data regardless of where it resides.

Why it matters: AI agents are only as effective as the data they can access. By using Ab Initio as a "neutral hub," enterprises can federate data from on-premises and multi-cloud sources into a single unified layer without moving the data itself. This provides the rich semantic context and lineage needed for Gemini to perform grounded, explainable reasoning. For businesses, this means faster transition from experimental AI to production-ready agentic workflows that are auditable, compliant, and capable of making complex, automated decisions.

Learn from us: "Agentic AI requires trusted, AI-ready data and metadata. Understanding the origin, quality, and meaning of information matters as much as the data itself. Gemini serves as a key component of the agentic layer, using this context to make decisions that are explainable and auditable." — Scott Studer, Head of Development, Ab Initio & Chai Pydimukkala, Data Governance, Sharing & Integration Product Lead, Google Cloud

Cool stuff Google Cloud customers built, Dec. edition: AI for better toys, reliable mapping tech, Gemini stumps an all-star & more

Wed, 31 Dec 2025 16:00:00 +0000

For our latest edition, we look into how Waze made its network more reliable; NBA superstar Stephen Curry gets quizzed by Gemini; a financial market transformation at CME Group; a multi-agent business forecasting platform from AppOrchid; Mattel crunches customer feedback with AI; VMO2 uses decentralized contracts for reliable data; Mercado Libre’s strategic use of Spanner; and how Ericsson enhances data governance.

Waze keeps traffic flowing with Memorystore

Who: Waze (a division of Google parent company Alphabet) is a community-driven, crowd-sourced navigation app with tens of millions of users who share real-time data to provide optimal driving routes, traffic updates, and alerts for hazards, police, and more.

What they did: Waze depends on vast volumes of dynamic, real-time user session data to power its core navigation features, but scaling that data to support concurrent users worldwide required a new approach. Their team built a centralized Session Server backed by Memorystore for Redis Cluster, a fully managed service with 99.99% availability that supports partial updates and easily scales to Waze’s use case of over 1 million MGET commands per second with ~1ms latency.

Why it matters: Moving from Memcached’s 99.9% SLA to Memorystore for Redis Cluster’s 99.99% means higher availability and resiliency from the service. And because Memorystore for Redis supports partial updates, Waze can change individual fields within a session object rather than rewriting the entire record. That reduces network traffic, speeds up write performance, and makes the system more efficient overall.

Learn from us: “Real-time data drives the Waze app experience. Our turn-by-turn guidance, accident rerouting, and driver alerts depend on up-to-the-millisecond accuracy. But keeping that experience seamless for millions of concurrent sessions requires robust and battle hardened infrastructure that is built to manage a massive stream of user session data.” – Eden Levin, Waze BE infrastructure developer & Yuval Kamran Waze site reliability engineer

What Stephen Curry learned from a custom Gemini agent

Who: Stephen Curry is arguably of the greatest three-point shooter of all-time in the NBA — as well as Google’s performance advisor and an all-around stats-obsessive.

What they did: For a special engagement with Curry, the Google Cloud team wanted to showcase the power of Gemini for creative thinking, analysis, and data mining. They took every regular season, play-in, and playoff game from Curry’s career (through the end of the 2024-2025 season) and input the data into a custom-built agent using Google Cloud’s Agent Development Kit and Gemini APIs.The system could then be queried for obscure stats, to see if the team could stump Curry and teach him more about his game.

Why it matters: For example, it found that his three-point shooting percentage after more than seven dribbles, with a minimum 105 attempts was 40.2%, and how many points Curry generated for his teammates off of screens since 2013: 1,105. Instead of countless hours of manual research, the team got query results in less than a minute. Some queries were so obscure, the team wouldn’t have reached a valid answer without the ability of the agent to analyze the rich data.

Learn from us: “Gemini is going to be in my head this year, cause I'm going to be looking at all these details.” – Stephen Curry, Golden State Warriors point guard and 4x NBA champ

How CME Group builds a faster, smarter exchange

Who: CME Group has evolved from a nineteenth-century commodities exchange into one of the most advanced financial market infrastructures in the world. To support real-time trading and risk management at a global scale, the company launched a strategic partnership with Google Cloud.

What they did: By migrating to Cloud SQL and adopting AI-powered insights, CME Group empowered developers, paid down technical debt, and unlocked new opportunities for data-driven innovation across financial markets.

Why it matters: Cloud SQL has given CME a foundation for increased developer and team agility. Fewer performance issues mean more time focused on innovation: expanding CME’s analytics capabilities, accelerating AI initiatives, and exploring new ways to commercialize data responsibly. When teams stopped chasing outages, they unlocked more time to take bigger bets and build the future.

Learn from us: “With Cloud SQL, we’ve found a way to keep our data layer as fast and dependable as the markets we serve. Cloud SQL gives our teams real-time visibility into what’s happening inside the database. When an application slows, we can identify the root cause in minutes instead of hours. Those insights are built into the platform, which means we don’t need custom tooling or manual analysis to keep operations steady.” – Kristofer Shane Sikora, Executive Director, Cloud Data Engineering, CME Group

AppOrchid’s multi-agent system for superior business forecasting

Who: App Orchid is an enterprise AI builder and a leader in making data actionable with AI, with a mission to make AI a force for good. Their goal is to empower every employee with trusted, understandable, and accessible data.

What they did: The business forecasting agent is actually built on the foundation of two powerful, specialized AI agents: a prediction agent built by Google Cloud and App Orchid’s Data Agent offering. These agents work in concert to solve complex business problems, acting as complementary specialists. App Orchid’s agent possesses unparalleled understanding of an enterprise's past and present, while Google’s agent brings world-class capabilities in predicting the future.

Why it matters: Adopting a multi-agent approach provides clear, tangible advantages that directly address the forecasting problems that often plague businesses, including improved accuracy; increased operational efficiency; faster insights; and reduced costs and increased revenue; and greater agility and adaptability. Neither of the underlying agents could achieve these results on their own, but working together, this agent is more than the sum of its subagents.

Learn from us: “As the agentic era gets underway, it is evolving quickly. Our multi-agent approach demonstrates both how true agentic systems are most successful when multiple agents are at play, and the importance of finding strong partners with distinct capabilities to help build and assemble these agentic systems.” – Brian Mills, Director, Enterprise AI, Google Cloud & Taka Shinagawa, Gen AI Field Solution Architect, Google Cloud

How Mattel uses AI for real-time product updates

Who: Since its humble beginnings in a garage in 1945, Mattel has consistently been reshaping play for children and families across the globe with iconic franchises like Barbie, Hot Wheels, and Fisher-Price.

What they did: To improve its understanding of consumer sentiment, Mattel developed an AI-powered feedback classification system, which can analyze millions of customer interactions from a diverse range of sources in a matter of seconds. At its core, the system relies on BigQuery for storing and efficiently processing its massive customer datasets and then utilizes Vertex AI and Google’s multimodal Gemini models to refine and train the sophisticated consumer feedback model.

Why it matters: Already, the new AI-powered system has delivered significant wins, delivering a staggering 100x boost in data processing capacity and reducing analysis times from a month to a single minute. By automating the analysis of many processes, analysts are now freed from the noise of everyday tasks, enabling them to focus on deeper research across the company’s iconic portfolio brand.

Learn from us: “Our big motto is ‘From months to minutes,’ but it’s real. We were literally spending months-worth of analysis and just getting data into the place that an analyst could tally up all the sentiment — and now it’s just at our fingertips.” – Shaun Applegate, Director of Product Quality Analytics, Mattel

Virgin Media O2 uses data contracts for scalable AI products

Who: Virgin Media O2 is one of Europe’s largest telecommunications and media providers, with 45.8 million broadband, mobile, phone, and home subscribers across the UK. To build AI products that are adaptable and data-driven, they needed a decentralized system that internal customers could count on for clean, reliable data.

What they did: New decentralized data contracts, built with Dataplex, serve as the data quality and assurance layer for VMO2’s data products; these ensure every dataset they publish is reliable, documented, and ready for consumption. Defined at the asset level, such as individual BigQuery tables or Google Cloud Storage buckets, data contracts are redefining how VMO2 manage and share data, enabling the creation of trusted and scalable AI products across their data mesh.

Why it matters: The power of this approach lies in moving beyond static documentation. Because they are machine-readable, data contracts become living guarantees with continuous enforcement and real-time validation directly within data pipelines. This proactive monitoring allows teams to detect schema changes or SLA breaches early, transforming data quality from a reactive fix into a scalable, automated mechanism.

Learn from us: “By operationalizing trust through data contracts, we are fostering a culture of shared responsibility and data-first thinking. This federated model does more than simply fix pipelines; it builds the trusted foundation needed to scale next-generation AI. It ensures that the resilient AI tools empowering our teams are built on data that is reliable, consistent, and well-defined.” – Chandu Bhuman, Head of Data Strategy, Cloud & Engineering, Virgin Media O2 & Dženan Softić, Data & AI Architect, Google Cloud

Inside Mercado Libre's multi-faceted Spanner architecture

Who: Mercado Libre, an e-commerce and fintech pioneer across Latin America, operates at a staggering scale, demanding an infrastructure that's not just resilient and scalable, but also a catalyst for rapid innovation.

What they did: At the heart of Mercado Libre's strategy is Fury, an in-house middleware platform designed to abstract away the complexities of various backend technologies, providing developers with standardized, simplified interfaces to build applications. Spanner provides Fury with an always-on, globally consistent, multi-model database with virtually unlimited scale. By designating Spanner as a choice within Fury, Mercado Libre ensures that applications built on the platform using Spanner stay consistent globally, scale without breaking, and rarely go down.

Why it matters: The strategic adoption of Spanner, amplified by internal platforms like Fury and sophisticated data workflows, has yielded significant benefits, including: significant cost savings and low total cost of ownership; business impact and agility for developers; and low operational overhead thanks to automation.

Learn from us: “Mercado Libre's adoption of Spanner demonstrates how to use a powerful, globally consistent database not just for its core capabilities, but as a strategic enabler for developer productivity, operational efficiency, advanced analytics, and future AI ambitions.” – Pablo Leopoldo Arrojo, Software Technical Leader, Mercado Libre

Ericsson achieves data integrity and superior governance with Dataplex

Who: Ericsson is one of the world's leading providers of telecommunications and networking technology and solutions. Its Managed Services unit provides network operations and optimization, including field operations, for various telecom and enterprise customers, including outsourcing network performance management, future provisioning, network vulnerability management, and network energy infrastructure management.

What they did: To power the future of its autonomous network operations and deliver on its strategic priorities across a global network of more than 710,000 sites, Ericsson's Managed Services has been on a transformative data journey with governance at the center of its strategy. Ericsson moved from foundational practices to a sophisticated, business-enabling data governance framework using the Dataplex Universal Catalog — turning data from a simple resource into a strategic asset.

Why it matters: With Dataplex as the governance foundation, Ericsson began implementing the core pillars of its governance program, moving from manual processes to an automated, intelligent data fabric. More specifically, Ericsson established a unified business vocabulary within Dataplex, which helped eliminate ambiguity and ensure their teams — from data scientists to data analysts — were speaking the same language.

Learn from us: “Governance is a value enabler, not a blocker. A modern data governance program should focus on business enablement first, driving value and innovation in order to complement policies, rules and risk management. Also remember this work is a journey, not a destination. Be prepared to fail fast, learn, and adapt. The landscape is constantly changing at breakneck speed.” – William McCann Murphy, Head of Data Authority, Ericsson & Akanksha Bhagwanani, EMEA Data Analytics Solution Lead, Google Cloud

How we built a multi-agent system for superior business forecasting

Thu, 11 Dec 2025 17:00:00 +0000

In today's dynamic business environment, accurate forecasting is the bedrock of efficient operations. Yet, businesses across all industries grapple with the constant challenge of predicting future demand, resource needs, and market trends. Nor is this an abstract problem; the cost of miscalculation can be substantial, leading to wasted resources and missed opportunities.

Imagine a large retail chain struggling to predict seasonal demand for its popular clothing lines. A miscalculation means either mountains of unsold inventory and costly markdowns or constant stock-outs that lead to lost sales and frustrated customers. Or consider a manufacturer trying to optimize the procurement of raw materials. Inaccurate forecasts force them into a reactive cycle of expensive rush orders and production delays, or they see their capital tied up in slow-moving inventory.

Whether it’s outdated processes, siloed systems, or missing data semantics slowing down your forecasts — and your business — there’s now a better way. A new era of AI-powered enterprise intelligence means we can move beyond reactive measures and achieve new levels of foresight. Google Cloud and App Orchid have now developed a novel multi-agent application for business forecasting that helps transform the predictive challenges of the past into strategic advantages.

In this post, we will outline the elements of our multi-agent system, the benefits of this innovative approach to agentic frameworks, and how others can benefit from not only our offering but rethink how they create their own.

Transforming enterprise intelligence with AI agents

App Orchid is a leader in making data actionable with AI, with a mission to make AI a force for good. The goal is to empower every employee with trusted, understandable, and accessible data. While enterprise data is now a mission-critical asset for organizations, it’s often underutilized, difficult to access, and buried under layers of complexity. To help, App Orchid partnered with Google to build a multi-agent application that provides a new level of precision in operational forecasting.

This innovative solution combines two powerful, specialized AI agents: a prediction agent built by Google Cloud and App Orchid’s Data Agent offering. These agents work in concert to solve complex business problems, acting as complementary specialists — each an expert in its own domain. App Orchid’s agent possesses unparalleled understanding of an enterprise's past and present, while Google’s agent brings world-class capabilities in predicting the future.

Adopting a multi-agent approach provides clear, tangible advantages that directly address the forecasting problems that often plague businesses, including:

Improved accuracy: Achieve a level of forecasting precision that was previously unattainable, reducing costly errors.
Increased operational efficiency: Automate and streamline the complex processes of data preparation and prediction, freeing up valuable human resources.
Faster insights: Gain real-time, actionable insights, enabling quicker and more informed decision-making.
Reduced costs and increased revenue: Directly impact the bottom line by minimizing inventory waste, reducing stock-outs to maximize sales, and optimizing resource allocation.
Greater agility and adaptability: Rapidly adapt to market shifts and unforeseen disruptions with agile forecasting capabilities.

How it works: Three powerful agents, one seamless solution

To better understand the power of this new agentic framework, it's first essential to understand how these two AI agents work together as complementary specialists under the direction of a third, orchestrator agent.

1. Google prediction agent - The forecasting powerhouse

The prediction agent, which is primarily the custom engineering work of Google Cloud, is the system’s window to the future. It takes rich, contextualized historical data and applies Google's state-of-the-art predictive models to generate highly accurate and actionable forecasts. The agent utilizes specialized foundation models like TimesFM, which is pre-trained on billions of data points for time-series forecasting, and the Population Dynamics Foundation Model (PDFM), which analyzes geospatial data to understand demographic similarities. By combining these powerful models, the Google prediction agent helps businesses anticipate future demand and identify market trends with a new level of precision.

2. App Orchid Data Agent - The enterprise intelligence data expert

Accurate predictions depend on high-quality, AI-ready data, which is where App Orchid’s Data Agent excels. This agent acts as the intelligent query engine for your enterprise data, connecting even disparate and siloed information to draw insights and offer informed feedback. The Data Agent utilizes the Google Cloud Cortex Framework for a unified view of business data and then applies App Orchid’s own powerful semantic-layer technology, which closes the "AI context gap" by transforming raw, siloed data into a unified, trustworthy knowledge foundation that grounds AI models for accuracy and scalability.

Together, this creates a "smart data layer" within the Data Agent that maps an organization’s entire data landscape into a unified knowledge graph. This allows the agent to understand the unique context, relationships, and terminology specific to a business — from internal acronyms to complex operational data — and deliver the comprehensive, time-series datasets that are essential for producing accurate predictions.

3. The combined business forecasting agent

At the heart of the solution is a unified business forecasting agent, which brings together the capabilities of our unique prediction and data agents in a discrete instance for the user. While our multi-agent system delivers an automated, end-to-end tool for next-generation forecasting, it’s the business forecasting agent that the user interacts with — few actually realize there are multiple agents working behind the scenes, nor need to know.

The process begins by having the forecasting agent function as an orchestrator of user queries, directing the App Orchid Data Agent to construct a unified, trustworthy knowledge foundation; this is a crucial step for eliminating data silos and prepares the high-context, time-series data required for accurate AI. This smart data layer is then passed to the Google prediction agent, which applies its specialized foundation models to project future outcomes with impressive levels of precision.

The final result is a single, comprehensive forecast delivered back to the user via the orchestrator agent. And by automating the labor-intensive processes of data wrangling and prediction execution, the combined solution empowers business leaders to make instant, highly accurate decisions that directly reduce costly markdowns, prevent stock-outs, maximize sales, and efficiently optimize resource allocation.

The technical glue: A2A Protocol, Google’s Agent Development Kit (ADK), and Gemini

The seamless collaboration between these two distinct AI agents is made possible by three key technologies: the Agent-to-Agent (A2A) Protocol, Google's Agent Development Kit (ADK), and Google’s industry-leading family of multimodal Gemini models.

The A2A Protocol enables AI agents, even those built by different teams and organizations, to discover, securely communicate, and collaborate across various systems. This allows developers to unite agents from multiple platforms, improving modularity, reducing vendor lock-in, and speeding up innovation. While Model Context Protocol (MCP) allows developers to connect data and APIs to agents, the agents themselves need a communication layer. The A2A protocol enables the bi-directional agentic communication needed to achieve multi-agent systems.

The Google ADK provides a robust framework for building sophisticated, scalable multi-agent applications. It supports both code-first development (Python and Java SDKs) and a no-code (YAML-based) development to define agent behavior and orchestrate workflows. The ADK also utilizes MCP Tools, which provide a standardized interface for agents to interact with external systems.

For instance, the MCP Toolbox for Databases provides out-of-the-box support for agents to easily access and query data from a variety of sources, including BigQuery. Agents built with the ADK can be deployed across virtually any environment — from a fully managed, enterprise-grade runtime like Vertex AI Agent Engine to Google Cloud services like Compute Engine, Cloud Run, or Google Kubernetes Engine (GKE), other cloud providers, or on-premises infrastructure.

At the heart of both agents' intelligence are Google's Gemini models, which are engineered for sophisticated reasoning and lead the industry on long context performance, with many models offering context windows of 1 million tokens or more. This massive context window is critical: it enable the data agent to understand natural language queries, enterprise data, and its underlying schema and relationships simultaneously, while enabling the prediction agent to analyze complex historical data and grasp the nuances of forecasting tasks.

This holistic understanding is what makes it possible for both agents to break down complex problems, identify subtle trends, and follow multi-step instructions without losing the initial context. Additionally, native tool use allows the agents to proficiently interact with external systems — whether querying a database or invoking a predictive model — and interpret the outputs to fulfill user requests.

Solution architecture: A secure, managed platform for AI agents

The Google and App Orchid multi-agent application is accessible through Gemini Enterprise, which provides a fully managed platform for organizations to discover, manage, and interact with AI agents, featuring enterprise-grade security, data privacy, and governance. The user-facing agent is deployed on Vertex AI Agent Engine and registered within this secure environment, providing authorized users with a simple entry point to the powerful multi-agent capabilities while ensuring that all interactions are grounded in the company's private data and adhere to its security and compliance policies.

From an architectural perspective, the business forecasting agent acts as the orchestrator agent of this multi-agent system, managing and directing the entire workflow between the two specialized agents. It uses the A2A protocol to pass instructions and data back and forth, hiding the underlying complexity from the user.

For example, when a user asks the business forecasting agent to predict revenue per marketing channel, the agent:

Calls upon the App Orchid Data Agent to retrieve historical sales data.
Passes that information to the Google prediction agent to run its models and generate the forecast.
Receives the prediction data back and then summarizes it for the user.

The Future is Collaborative

As the agentic era gets underway, it is evolving quickly. Our multi-agent approach demonstrates both how true agentic systems are most successful when multiple agents are at play, and the importance of finding strong partners with distinct capabilities to help build and assemble these agentic systems.

The partnership between App Orchid and Google is a testament to the power of collaboration in the AI space. By combining App Orchid's deep understanding of enterprise data and Google's global leadership in AI, this solution is greater than the sum of its parts. With the complementary strengths of each agent, enabled by the ADK and A2A Protocol, businesses can achieve levels of forecasting accuracy that were previously unattainable at the speed and ease agents can offer.

This innovative multi-agent system not only optimizes business processes but also paves the way for a future where AI agents from across the industry can collaborate to solve the world's most complex challenges.

_{The Google Cloud team would like to thank App Orchid’s CTO Ravi Bommakanti for his contributions to this project.}

Is your DR plan just wishful thinking? Prove your resilience with chaos engineering

Mon, 08 Dec 2025 17:00:00 +0000

When was the last time you knew — not just hoped — that your disaster recovery plan would work perfectly?

For most of us, the answer is unclear. Sure, you may have a DR plan, a meticulously crafted document stored in a wiki or a shared drive, that gets dusted off for compliance audits or the occasional tabletop drill. You assume its procedures are correct, its contact lists are current, and its dependencies are fully mapped, and you certainly hope it works.

But hope is not a strategy.

Why wouldn’t it work? One problem is that systems are rarely static anymore. In a world where you deploy new microservices dozens of times per day, make constant configuration changes, and maintain an ever-growing web of third-party API dependencies, the DR plan you wrote last quarter is probably just as useful as one from 10 years ago.

And if the failover does work, will it work well enough to meet the promises you've made to your customers (or board of directors or regulators)? When a key component fails, could you still even meet your target availability and latency targets, a.k.a., your Service Level Objectives (SLOs)?

So, how do you close this gap between your current aspirational DR plan and a DR plan that you actually have confidence in? The answer isn't to write more documents or run more theatrical drills. The answer is to stop assuming and start proving.

This is where chaos engineering comes in. Unlike what the name might imply, chaos engineering isn’t a tool for recklessly breaking things. Instead, it’s a framework that provides data-driven confidence in your SLOs under stress. By running controlled experiments that simulate real-world disasters like a database failover or a regional outage, you can quantitatively measure the impact of those failures on your systems’ performance. Chaos engineering is how you transform your DR hypotheses into a proven method to ensure resilience. By validating your plan through experimentation, you create tangible evidence, verifying that your plan will safeguard your infrastructure and keep your promises to customers.

Demystifying chaos engineering

In a nutshell, chaos engineering is the practice of running controlled, scientific experiments to find weaknesses in your system before they cause a real outage.

At its core, it’s about building confidence in your system’s resilience. The process starts with understanding your system's steady state, which is its normal, measurable, and healthy output. You can't know the true impact of a failure without first defining what "good" looks like. This understanding allows you to form a clear, testable hypothesis: a statement of belief that your system's steady state will persist even when a specific, turbulent condition is introduced.

To test this hypothesis, you then execute a controlled action, which is a precise and targeted failure injected into the system. This isn't random mischief; it's a specific simulation of real-world failures, such as consuming all CPU on a host (resource exhaustion), adding network latency (network failure), or terminating a virtual machine (state failure). While this action is running, automated probes act as your scientific instruments, continuously monitoring the system's state to measure the effect.

Together, these components form a complete scientific loop: you use a hypothesis to predict resilience, run an experiment by applying an action to simulate adversity, and use probes to measure the impact, turning uncertainty into hard data.

Using chaos to validate disaster recovery plans

Now that you understand the building blocks of a chaos experiment, you can build the bridge to your ultimate goal: transforming your DR plan from a document of hope into an evidence-based procedure. The key is to stop seeing your DR plan as a set of instructions and start seeing it for what it truly is: a collection of unproven hypotheses.

When you think about it, every significant statement in your DR document is a claim waiting to be tested. When your plan states, "The database will failover to the replica in under 5 minutes," that isn't a fact, it's a hypothesis. When it says, "In the event of a regional outage, traffic will be successfully rerouted to the secondary region," that's another hypothesis. Your DR plan is filled with these critical assumptions about how your system should behave under duress. Until you test them, they remain nothing more than educated guesses.

Chaos experiments are the ultimate validation tools, live-fire drills that put your DR hypotheses to a real, empirical test. Instead of just talking through a scenario, you use controlled actions to safely and precisely simulate the disaster. You're no longer asking "what if?"; you're actively measuring "what happens when."

For example, imagine you have a DR plan for a regional outage. When you adopt chaos engineering, you break down that plan into a hypothesis and an experiment. For example:

The hypothesis: "In case our primary region us-central1 becomes unreachable, the load balancers will failover all traffic to us-east1 within 3 minutes, with an error rate below 1%."
The chaos experiment: Run an action that simulates a regional outage by injecting a "blackhole" that drops all network traffic to and from us-central1 for a limited time. Your probes then measure the actual failover time and error rates to validate the hypothesis.

In other words, by applying the chaos engineering methodology, you systematically move through your DR plan, turning each assumption into a proven fact. You're not just testing your plan; you're forging it in a controlled fire.

Connecting chaos readiness to your SLOs

Beyond simply proving system availability, chaos engineering builds trust in your reliability metrics, ensuring that you meet your SLOs even when services become unavailable. An SLO is a specific, acceptable target level of your service's performance measured over a specified period that reflects the user's experience. SLOs aren't just internal goals; they are the bedrock of customer trust and the foundation of your contractual service level agreements (SLAs).

A traditional DR drill might get a "pass" because the backup system came online. But what if it took 20 minutes to fail over, during which every user saw errors? What if the backup region was under-provisioned, and performance became so slow that the service was unusable? From a technical perspective, you "recovered." But from a customer's perspective, you were down.

A chaos experiment, however, can help you answer a critical question: "During a failover, did we still meet our SLOs?” Because your probes are constantly measuring performance against your SLOs, you get the full picture. You don't just see that the database failed over; you see that it took 7 minutes, during which your latency SLO was breached and your error budget was completely burned. This is the crucial, game-changing insight. It shifts the entire goal from simple disaster recovery to SLO preservation, which is what actually determines if a failure was a minor hiccup or a major business-impacting incident. It also provides the data necessary to set goals for system improvement. So the next time you run this experiment, you can measure if and how much your system resilience has improved, and ultimately if you can maintain your SLO during the disaster event.

Build a culture of confidence

The journey to resilience doesn't start by simulating a full regional failover. It starts with a single, small experiment. The goal is not to boil the ocean; it's to build momentum. Test one timeout, one retry mechanism, or one graceful error message.

The biggest win from your first successful experiment won't be the technical data you gather. It will be the confidence you build. When your team sees that they can safely inject failure, learn from it, and improve the system, their entire relationship with failure changes. Fear is replaced by curiosity. That confidence is the catalyst for building a true, enduring culture of resilience. To learn more and get started with chaos engineering, check out this blog and this podcast. And if you’re ready to get started, but unsure how, reach out to Google Cloud professional services to discuss how we can help.

Streamline the design and deployment of application infrastructure with Application Design Center, now GA

Mon, 08 Dec 2025 17:00:00 +0000

Earlier this year, we unveiled a big investment in platform and developer team productivity, with the launch of Application Design Center, helping them streamline the design and deployment of cloud application infrastructure, while ensuring applications are secure, reliable, and aligned with best practices. And today, Application Design Center is generally available.

We built Application Design Center to put applications at the center of your cloud experience, with a visual, canvas-style and AI-powered approach to design and modify Terraform-backed application templates. It also offers full lifecycle management that’s aligned with DevOps best practices across application design and deployment.

Application Design Center is a core component of our application-centric cloud experience. When you use Application Design Center to design and deploy your application infrastructure, your applications are easily discoverable, observable, and manageable. Application Design Center works in concert with App Hub to automatically register application deployments, enabling a unified view and control plane for your application portfolio, and Cloud Hub, to provide operational insights for your applications.

“Google Application Design Center is a valuable enabler for Platform Engineering, providing a structured approach to harmonizing resource creation in Google Cloud Platform. By aligning tools, processes, and technologies, it streamlines workflows, reducing friction between development, operations, and other teams. This harmonization enhances collaboration, accelerates delivery, and ensures consistency across Google Cloud environments.” - Ervis Duraj, Principal Engineer, MediaMarktSaturn Technology

The gateway to an app-centric cloud

Our goal with Application Design Center is for you to innovate more, and administer less. It consists of four key elements to help you minimize administrative overhead and maximize efficiency, so you can design and deploy applications with integrated best practices and essential guardrails. Let’s take a closer look.

1. Terraform components and application templates
Develop applications faster with our growing library of opinionated application templates. These provide well-architected patterns and pre-built components, including innovative "AI inference templates" to help you leverage AI to create dynamic and intelligent application foundations. As an example, at launch, Application Design Center provides opinionated templates for Google Kubernetes Engine (GKE) clusters (Standard, Autopilot and NodePool) to run AI inference workloads using a variety of LLM models, as well as for enterprise-grade production clusters or single-region web app clusters.

You can also ingest and manage your existing Terraform configurations (“Bring your own Terraform”) directly from Git repositories. Once imported, you can use Application Design Center to design with your own Terraform, or in combination with Google-provided Terraform, to create standardized, opinionated infrastructure patterns for sharing and reuse across your application teams.

2. AI-powered design for rapid application designing and prototyping
Application Design Center integrates with Google's Gemini Cloud Assist Design Agent, empowering you to design actual, deployable application infrastructure application templates on Google Cloud that you can export as Terraform infrastructure-as-code.

With Gemini Cloud Assist, you can describe your application design intents using natural language. In return, Gemini interactively generates multi-product application template suggestions, complete with visual architecture diagrams and summarized benefits. You can then refine these proposals through multi-turn reasoning or by directly manipulating the architecture within the Application Design Center canvas.

Additionally, all designs that you create with Gemini are automatically observable, optimizable, and enabled for troubleshooting assistance during runtime, thanks to their tight integration with Gemini Cloud Assist.

3. A secure, sharable catalog of application templates with full lifecycle management
Platform admins can curate a collection of application templates built from Google's best-practice components. This provides developers a trusted, self-service experience from which they can quickly discover and deploy compliant applications. Tight integration with Cloud Hub transforms these governed templates into a live operational command center, complete with unified visibility into the health and deployment status of the resulting applications. This closes the critical loop between design and runtime, so that your production environments reflect your organization’s approved architectural standards.

Also, Application Design Center’s robust application template revisions serve as an immutable audit trail. It automatically detects and flags configuration drift between your intended designs and deployed applications, so that developers can remediate unauthorized changes or safely push approved configuration updates. This helps ensure continuous state consistency and compliance from Day 1 and through the subsequent evolution of your application.

4. GitOps integration automating developers’ day-to-day software design lifecycle tasks
By integrating Application Design Center into existing CI/CD workflows, platform teams empower developers to own the complete software delivery lifecycle right from their IDE. Developers can leverage compliant application and infrastructure (IaC) code using Application Design Center application templates.

Further, every infrastructure decision made through Application Design Center is committed to code, versioned, and auditable. Specifically, developers can download the application IaC template from Application Design Center and import it into their app repos (the single source of truth), clone their repo, and edit the Terraform directly in their local IDEs. Any modifications go through a Git pull request for review. Once approved, this automatically triggers the existing CI/CD setup to build, test, and deploy both app and infra changes in lockstep. This unified approach minimizes friction, enforcing "golden paths" and providing an end-to-end automated pathway from a line of code in the IDE to a fully deployed change in production.

What's new since preview

This GA launch is packed with features that users have been asking for. We’re excited to share powerful new capabilities: enterprise-grade governance and security with public APIs and gcloud CLI support; full compatibility with VPC service controls; bring your own Terraform and GitOps support for integration with your existing application patterns and automation pipelines; agentic application patterns using GKE templates (Standard, Autopilot and NodePool); and finally, a simplified onboarding experience with app-managed project support, making Application Design Center an AI-powered engine for your applications on Google Cloud.

Get started today

To help you get started, Google provides a growing library of curated Google application templates built by experts. These templates combine multiple Google Cloud products and best practices to serve common use cases, which you can configure for deployment, and view as infrastructure as code in-line. Platform teams can then create and securely share the catalogs and collaborate with teammates on designs and self-service deployment for developers. For enterprises with existing Terraform patterns and assets, Application Design Center interoperates by enabling their import and reuse within its native design and configuration experience.

Ready to experience the power of Application Design Center? You can learn more about ADC and get started building in minutes using the quickstart. You can start building your first AI-powered application template in minutes, free of cost, and quickly deploy applications with working code. For deeper insights, explore the comprehensive public documentation here. We can't wait to see how you innovate with the Application Design Center!

Replit is delivering enterprise-grade vibe coding with Google Cloud

Thu, 04 Dec 2025 17:00:00 +0000

Vibe coding has been all the rage this year. Using AI to build novel apps and create dynamic websites simply by typing conversationally into a chat interface can seem magical — yet it’s largely remained the domain of individual developers, not larger business teams.

Today, Replit and Google Cloud are expanding their strategic partnership to bring vibe coding capabilities to enterprise developers and teams.

Google Cloud was Replit’s first cloud partner, and its popular platform for AI coding continues to run on top of Google Cloud’s infrastructure, utilizing multiple Google Cloud services and integrating Google’s Gemini models through Vertex AI.

Now, Replit is extending its partnership with Google Cloud with a new, multi-year agreement that will grow its usage of infrastructure and cloud services; further integrate Google’s models into its platform; and jointly support vibe coding use cases for enterprise customers. Just last month, Replit integrated Gemini 3 into its new Design mode to positive reception. Under the terms of the expanded deal:

Google Cloud will continue to be the primary cloud provider for Replit, and services like Google Cloud Run, Google Kubernetes Engine, and BigQuery will underpin its applications and enable further scale as the company grows.
Google models, including Gemini 3, 2.5 Flash Lite, 2.5 Flash, and Imagen 4, are now supported on Replit, powering both coding and multimodal use cases — and driving significant token usage to Google Cloud.
Replit and Google Cloud will partner to help enterprise customers embrace vibe coding and help their developers be more productive — through joint go-to-market on Google Cloud Marketplace and through Google Cloud’s extensive co-sell programs.

“Our growing partnership will deliver more capabilities to Replit’s users through deeper integrations with our AI and cloud services,” Google Cloud CEO Thomas Kurian said, “and will accelerate the adoption of vibe coding in the enterprise by bringing Replit's easy-to-use AI tools, powered by Google Cloud AI, to more organizations."

Amjad Masad, co-founder and CEO of Replit, said: “Our mission is to enable the next billion software creators — from hobbyists to entrepreneurs to enterprises. Over the last few months, we have seen fantastic adoption in businesses, especially in the Fortune 1000. Today’s expanded partnership with Google will enable us to scale faster and more deeply as we integrate Google’s offerings with ours – the work is just beginning.”

You can read more about how customers are using Google’s AI models for coding use cases here, and join Replit and other customers in building with Gemini 3 via Vertex AI.

Chaos engineering on Google Cloud: Principles, practices, and getting started

Mon, 13 Oct 2025 16:00:00 +0000

As engineers, we all dream of perfectly resilient systems — ones that scale perfectly, provide a great user experience, and never ever go down. What if we told you the key to building these kinds of resilient systems isn't avoiding failures, but deliberately causing them? Welcome to the world of chaos engineering, where you stress test your systems by introducing chaos, i.e., failures, into a system under a controlled environment. In an era where downtime can cost millions and destroy reputations in minutes, the most innovative companies aren't just waiting for disasters to happen — they're causing them and learning from the resulting failures, so they can build immunity to chaos before it strikes in production.

Chaos engineering is useful for all kinds of systems, but particularly for cloud-based distributed ones. Modern architectures have evolved from monolithic to microservices-based systems, often comprising hundreds or thousands of services. These complex service dependencies introduce multiple points of failure, and it’s difficult if not impossible to predict all the possible failure modes through traditional testing methods. When these applications are deployed on the cloud, they are deployed across multiple availability zones and regions. This increases the likelihood of failure due to the highly distributed nature of cloud environments and the large number of services that coexist within them.

A common misconception is that cloud environments automatically provide application resiliency, eliminating the need for testing. Although cloud providers do offer various levels of resiliency and SLAs for their cloud products, these alone do not guarantee that your business applications are protected. If applications are not designed to be fault-tolerant or if they assume constant availability of cloud services, they will fail when a particular cloud service they depend on is not available.

In short, chaos engineering can take a team's worst "what if?" scenarios and transform them into well-rehearsed responses. Chaos engineering isn’t about breaking systems — engineering chaotically, as it were — it's about building teams that face production incidents with the calm confidence that only comes from having weathered that chaos before, albeit in controlled conditions.

Google Cloud’s Professional Service Organization (PSO) Enterprise Architecture team consults on and provides hands-on expertise on customers’ cloud transformation journeys, including application development, cloud migrations, and enterprise architecture. And when advising on designing resilient architecture for cloud environments, we routinely introduce the principles and practices of chaos engineering and Site Reliability Engineering (SRE) practices.

In this first blog post in a series, we explain the basics of chaos engineering — what it is and its core principles and elements. We then explore how chaos engineering is particularly helpful and important for teams running distributed applications in the cloud. Finally, we’ll talk about how to get started, and point you to further resources.

Understanding chaos engineering

Chaos engineering is a methodology invented by Netflix in 2010 when it created and popularized ‘Chaos Monkey’ to address the need to build more resilient and reliable systems in the face of increasing complexity in their AWS environment. Around the same time, Google introduced Disaster Resilience Testing, or DiRT, which enabled continuous and automated disaster readiness, response, and recovery of Google’s business, systems, and data. Here on Google Cloud’s PSO team, we offer various services to help customers implement DiRT as part of SRE practices. These offerings also include training on how to perform DiRT on applications and systems operating on Google Cloud. The central concept is straightforward: deliberately introduce controlled disruptions into a system to identify vulnerabilities, evaluate its resilience, and enhance its overall reliability.

As a proactive discipline, chaos engineering enables organizations to identify weaknesses in their systems before they lead to significant outages or failures, where a system includes not only the technology components but also the people and processes of an organization. By introducing controlled, real-world disruptions, chaos engineering helps test a system's robustness, recoverability, and fault tolerance. This approach allows teams to uncover potential vulnerabilities, so that systems are better equipped to handle unexpected events and continue functioning smoothly under stress.

Principles and practices of chaos engineering

Chaos engineering is guided by a set of core principles about why it should be done, while practices define what needs to be done.

Below are the principles of chaos engineering:

Build a hypothesis around steady state: Prior to initiating any disruptive actions, you need to define what "normal" looks like for your system, commonly referred to as the "steady state hypothesis."
Replicate real-world conditions: Chaos experiments should emulate realistic failure scenarios that the system might encounter in a production environment.
Run experiments in production: Chaos engineering is firmly rooted in the belief that only a production environment with real traffic and dependencies can provide an accurate picture of resiliency. This is what separates chaos engineering from traditional testing.
Automate experiments: Make resiliency testing part of a continuous ongoing process rather than a one-off test.
Determine the blast radius: Experiments should be meticulously designed to minimize adverse impacts on production systems. This requires categorizing applications and services in different tiers based on the impact the experiments can have on customers and other applications and services.

With these principles established, follow these practices when conducting a chaos engineering experiment:

Define steady state: Identifies the specific metrics (e.g., latency, throughput) that you will look at and establish a baseline for them.
Formulate a hypothesis: This is the practice of creating a single testable statement, for example, ‘By deleting this container pod, user login will not be affected’. Hypotheses are generally created by identifying customer user journeys and deriving test scenarios from them.
Use a controlled environment: While one chaos engineering principle states that experiments need to run in production, you should still start small and run your experiment in a non-production environment first, learn and adjust, and then gradually expand the scope to production environment.
Inject failures: This is the practice of causing disruption by injecting failures either directly into the system (e.g., deleting a VM, stopping a database instance) or indirectly by injecting failures in the environment (e.g. deleting a network route, adding a firewall rule).
Automate experimental execution: Automation is crucial for establishing chaos engineering as a repeatable and scalable practice. This includes using automated tools for fault injection (e.g., making it part of a CI/CD pipeline) and automated rollback mechanisms.
Derive actionable insights: The primary objective of using chaos engineering is to gain insights into system vulnerabilities, thereby enhancing resilience. This involves rigorous analysis of experimental results; identifying weaknesses and areas for improvement; and disseminating findings to relevant teams to inform subsequent experimental design and system enhancements.

In other words, chaos engineering isn't about breaking things for the sake of it, but about building more resilient systems by understanding their limitations and addressing them proactively.

Elements of chaos engineering

Here are the core elements you'll use in a chaos engineering experiment, derived from these five principles:

Experiments: A chaos experiment constitutes a deliberate, pre-planned procedure wherein faults are introduced into a system to ascertain its response.
Steady-state hypotheses: A steady-state hypothesis defines the baseline operational state, or "normal" behavior, of the system under evaluation.
Actions: An action represents a specific operation executed upon the system being experimented on.
Probes: A probe provides a mechanism for observing defined conditions within the system during experimentation.
Rollbacks: An experiment may incorporate a sequence of actions designed to reverse any modifications implemented during the experiment.

Getting started with chaos engineering

Now that you have a good understanding of chaos engineering and why to use it in your cloud environment, the next step is to try it out for yourself in your own development environment.

There are multiple chaos engineering solutions in the market; some are paid products and some are open-source frameworks. To get started quickly, we recommend that you use Chaos Toolkit as your chaos engineering framework.

Chaos Toolkit is an open-source framework written in Python that provides a modular architecture where you can plug in other libraries (also known as ‘drivers’) to extend your chaos engineering experiments. For example, there are extension libraries for Google Cloud, Kubernetes, and many other technologies. Since Chaos Toolkit is a Python-based developer tool, you can begin by configuring your Python environment. You can find a good example of a Chaos Toolkit experiment and step-by-step explanation here.

Finally, to enable Google Cloud customers and engineers to introduce chaos testing in their applications, we’ve created a series of Google Cloud-specific chaos engineering recipes. Each recipe covers a specific scenario to introduce chaos in a particular Google Cloud service. For example, one recipe covers introducing chaos in an application/service running behind a Google Cloud internal or external application load balancer; another recipe covers simulating a network outage between an application running on Cloud Run and connecting to a Cloud SQL database by leveraging another Chaos Toolkit extension named ToxiProxy.

You can find a complete collection of recipes, including step-by-step instructions, scripts, and sample code, to learn how to introduce chaos engineering in your Google Cloud environment on GitHub. Then, stay tuned for subsequent posts, where we’ll talk about chaos engineering techniques, such as how to introduce faults into your Google Cloud environment.

Inside the AI-powered assistant helping doctors work faster and better at Seattle Children’s Hospital

Thu, 18 Sep 2025 16:00:00 +0000

Though its name may suggest otherwise, Seattle Children’s is the largest pediatric healthcare system in the world.

While its main campus is in its namesake city, Seattle Children’s also encompasses 47 satellite hospitals across Alaska, Montana, Idaho, and Washington, and patients come from as far away as Hawaii for treatment. For more than 100 years, Seattle Children’s has helped kids across the Western U.S. get healthy and stay healthy, regardless of the ability to pay.

With so much ground to cover and diverse patient populations to treat, Seattle Children's has always looked to new technologies to bring improved, consistent care to its patients and providers. Generative AI is now the latest advance in their medical toolkit.

It started roughly two decades ago, when Seattle Children’s created its pediatric clinical pathways, a set of standardized protocols designed to help clinicians make quicker and more reliable decisions to address dozens of medical conditions. Such pathways were becoming commonplace across medicine, and Seattle Children’s had developed some of the first for children’s unique medical needs.

Innovative as these were, they still required clinicians to thumb through indexes and long binders of information to find what they needed for a given ailment. And in healthcare, it’s often the case that every second counts.

Seattle Children’s was already working with Google Cloud on a number of projects, and as we began to explore the potential for generative AI to make the work of our clinicians easier, the clinical pathways seemed like an obvious place to start. Using Vertex AI and Gemini, we were able to quickly develop our Pathways Assistant, which took training from the clinical pathways documentation and supercharged it with not just searchability but conversationality.

Instead of flipping pages, we’d flipped the script on how quickly and reliably clinicians could find the lifesaving information they needed.

aside_block: <ListValue: [StructValue([('title', '2025 ROI of AI in Healthcare and Life Sciences'), ('body', <wagtail.rich_text.RichText object at 0x7f0a233cf7c0>), ('btn_text', 'Read now'), ('href', 'https://cloud.google.com/resources/content/roi-of-ai-healthcare-life-sciences'), ('image', None)])]>

The pathways to improved healthcare run through Gemini

“Clinical pathways” are end-to-end treatment protocols for a specific condition or illness. Seattle Children’s pediatric clinical pathways are widely respected and used by hospitals around the globe, providing information on everything from diagnostic criteria to testing protocols to medication recommendations.

Previously, these clinical pathways were documented exclusively in PDFs — hundreds of thousands of pages of them. Performing a traditional search of their contents for the answers clinicians needed delayed their ability to provide treatment in an environment where minutes or even seconds can be critical.

Google Cloud engineers worked with Seattle Children’s informatics physicians, who straddle the worlds of healthcare and technology, to create Pathway Assistant. The new multimodal AI chatbot that responds to spoken or written natural-language queries using the information in those PDFs.

After processing a question, Pathway Assistant searches each PDF’s metadata, which contains semi-structured data in JSON format that’s been extracted from the PDFs by Gemini and curated by clinicians. It then selects the most relevant PDFs, parses the information — including any complex flowcharts, diagrams, and illustrations embedded in them — and answers the clinician’s question in just a few seconds.

Interactive information-finding for accurate decision-making

Pathway Assistant becomes more accurate with use. Healthcare providers can “discuss” clinical pathways with the chatbot, which, instead of answering a question, poses questions of its own if it needs clarification, going back and forth until it’s confident it can answer accurately. The chatbot always displays the specific sections of each PDF that was the source for formulating its answers, helping clinicians confirm the veracity of responses.

The interface also includes a way for users to provide feedback about the accuracy and appropriateness of the chatbot’s analysis and answers. The feedback is then logged in a BigQuery table for future forensic analysis — both by clinicians, who can query the information using natural language, and by the built-in Gemini models, which processes the feedback and summarizes what clinicians found confusing or how to improve the accuracy of future answers.

This reflexive capability enables Pathway Assistant to update the PDFs based on clinicians’ feedback if the inaccuracy stemmed from the PDF’s content. Clinicians are also finding that the metadata is becoming more accurate and requiring less curation. Pathway Assistant even corrects typos in the documentation automatically. And as new clinical pathways are developed, PDFs containing the latest information are added to the PDF library.

This growing collection is housed securely in Google Cloud Storage, and the bigger it gets, the more useful it becomes — which wasn't always the case. Whereas an expanding paper-based collection contained more information, it was also more material to wade through, which is especially challenging in emergency medical situations. Pathway Assistant almost entirely relieves this burden, synthesizing and delivering the most complete information at any time in a matter of seconds.

Ultimately, Pathway Assistant is not a decision-making tool but rather an information-finding tool. Research into critical, evidence-based guidelines that used to take hours now takes minutes.

This speed and effectiveness helps clinicians make the right decisions more quickly at the point of care, drastically reducing research time and improving patient safety and outcomes. Ultimately, clinicians can spend more time with more patients, not with more PDFs.

Ask any physician, they’ll tell you that’s what the best medical technology enables them to do — focus on the patient, not paperwork.

How Rent the Runway supercharges developer speed and insights with Cloud SQL

Fri, 12 Sep 2025 16:00:00 +0000

Editor’s note: Rent the Runway is redefining how consumers engage with fashion, offering on-demand access to designer clothing through a unique blend of e-commerce and reverse logistics. As customer expectations around speed, personalization, and reliability continue to rise, Rent the Runway turned to Google Cloud’s fully managed database services to modernize its data infrastructure. By migrating from a complex, self-managed MySQL environment to Cloud SQL, the company saved more than $180,000 annually in operational overhead. With Cloud SQL now supporting real-time inventory decisions, Rent the Runway has built a scalable foundation for its next chapter of growth and innovation.

Closet in the cloud, clutter in the stack

Rent the Runway gives customers a “Closet in the Cloud” — on-demand access to designer clothing without the need for ownership. We offer both a subscription-based model and one-time rentals, providing flexibility to serve a broad range of needs and lifestyles. We like to say Rent the Runway runs on two things: fashion and logistics. When a customer rents one of our items, it kicks off a chain of events most online retailers never have to think about.

That complex e-commerce and reverse logistics model involves cleaning, repairing, and restocking garments between uses. To address both our operational complexity and the high expectations of our customers, we’re investing heavily in building modern, data-driven services that support every touchpoint.

Out: Manual legacy processes

Our legacy database setup couldn’t keep up with our goals. We were running self-managed MySQL, and the environment had grown… let’s say organically. Disaster recovery relied on custom scripts. Monitoring was patchy. Performance tuning and scaling were manual, time-consuming, and error-prone.

Supporting it all required a dedicated DBA team and a third-party vendor providing 24/7 coverage — an expensive and brittle arrangement. Engineers didn’t have access to query performance metrics, which meant they were often flying blind during development and testing. Even small changes could feel risky.

All that friction added up. As our engineering teams moved faster, the database dragged behind. We needed something with more scale and visibility, something a lot less hands-on.

It’s called Cloud SQL, look it up

When we started looking for alternatives, we weren’t trying to modernize our database: We were trying to modernize how our teams worked with data. We wanted to reduce operational load, yes, but also to give our engineers more control and fewer blockers when building new services.

Cloud SQL checked all the boxes. It gave us the benefits of a managed service — automated backups, simplified disaster recovery, no more patching – while preserving compatibility with the MySQL stack our systems already relied on.

But the real win was the developer experience. With built-in query insights and tight integration with Google Cloud, Cloud SQL made it easier for engineers to own what they built. It fit perfectly with where we were headed: faster cycles; infrastructure as code; and a platform that let us scale up and out, team by team.

aside_block: <ListValue: [StructValue([('title', 'Build smarter with Google Cloud databases!'), ('body', <wagtail.rich_text.RichText object at 0x7f0a2166dac0>), ('btn_text', ''), ('href', ''), ('image', None)])]>

Measure twice and cut over once

We knew the migration had to be smooth because our platform didn’t exactly have quiet hours. (Fashion never sleeps, apparently.) So we approached it as a full-scale engineering project, with phases for everything from testing to dry runs.

While we planned to use Google’s Database Migration Service across the board, one database required a manual workaround. That turned out to be a good thing, though: It pushed us to document and validate every step. We spent weeks running Cloud SQL in lower environments, stress-testing setups, refining our rollout plan, and simulating rollback scenarios. When it came time to cut over, the entire process fit within a tight three-hour window. One issue did come up – a configuration flag that affected performance — but Google’s support team jumped in fast, and we were able to fix it on the spot.

Overall, minimal downtime and no surprises. Our kind of boring.

Less queues and more runway

One of the biggest changes we’ve noticed post-migration was how our teams worked. With our old setup, making a schema change meant routing through DBAs, coordinating with external support, then holding your breath during rollout. Now, engineers can breathe a little easier and own those changes end to end. Cloud SQL gives our teams access to IAM-controlled environments where they can safely test and deploy. That has let us move toward real CI/CD for database changes — no bottlenecks or surprises.

With better visibility and guardrails, our teams are shipping higher-quality code and catching issues earlier in the lifecycle. Meanwhile, our DBAs can focus on strategic initiatives — things like automation and platform-wide improvements — rather than being stuck in a ticket queue.

Our most cost-effective look yet

Within weeks of moving to Cloud SQL, we were able to offboard our third-party MySQL support vendor, cutting over $180,000 in annual costs. Even better, our internal DBA team got time back to work on higher-value initiatives instead of handling every performance issue and schema request.

Cloud SQL also gave us a clearer picture of how our systems were running. Engineers started identifying slow queries and fixing them proactively, something that used to be reactive and time-consuming. With tuning and observability included, we optimized instance sizing and reduced infrastructure spend, all without compromising performance. And with regional DR configurations now baked in, we’ve simplified our disaster recovery setup while improving resilience.

In: Cloud SQL

We’re building toward a platform where engineering teams can move fast without trading off safety or quality. That means more automation, more ownership, and fewer handoffs. With Cloud SQL, we’re aiming for a world where schema updates are rolled out as seamlessly as application code.

This shift is technical, but it’s also cultural. And it’s a big part of how we’ll continue to scale the business and support our expansion into AI. The foundation is there. Now, we’re just dressing it up.

Learn more:

Discover how Cloud SQL can transform your business! Start a free trial today!
Download this IDC report to learn how migrating to Cloud SQL can lower costs, boost agility, and speed up deployments.
Learn how Ford and Yahoo gained high performance and cut costs by modernizing with Cloud SQL.

Driving for the Horizon: New Android Automotive solution on cloud offers faster builds

Mon, 08 Sep 2025 06:00:00 +0000

The automotive industry is in the midst of a profound transformation, accelerating towards an era of software-defined vehicles (SDVs). This shift, however, presents significant challenges for manufacturers and suppliers alike. Their priority is making great vehicles, not great software, though the latter now contributes — and is increasingly a necessity — to achieve the former. These OEMs must find ways to bring greater efficiency and quality to their software delivery and establish new collaboration models, among other hurdles to achieving their visions for SDVs.

To help meet this moment, we’ve created Horizon, a new open-source software factory for platform development with Android Automotive OS — and beyond. With Horizon, we aim to support the software transformation of the automotive industry and tackle its most pressing challenges by providing a standardized development toolchain so OEMs can generate value by focussing on building products and experiences.

In early deployments at a half-dozen automotive partners, we’ve already seen between 10x to 50x faster feedback for developers, leading to high-frequency releases and higher build quality. In this post we will outline how Horizon helps overcome the key impediments to automotive software transformation.

The Roadblocks to Innovation in Automotive Software Development

Today, traditional automotive manufacturers (OEMs) often approach software development from a hardware-centric perspective that lacks agility and oftentimes struggles to scale. This approach makes software lifecycle support burdensome and is often accompanied by inconsistent and unreliable tools, slowing down development.

OEMs face exploding development costs, quality issues and slow innovation, making it difficult to keep pace with new market entrants and the increasing demand for advanced features. Furthermore, most customers expect frequent, high-quality over-the-air (OTA) software updates similar to what they receive on other devices such as on their smartphones, forcing most OEMs to mirror the consumer electronics experience.

But a car is not a television or refrigerator or even a rolling computer, as many now describe them. Vehicles are made up of many separate, highly complex systems, which typically require the integration of numerous components from multiple suppliers who often provide "closed box" solutions. Even as vehicles have become more connected, and dependent on these connective systems for everything from basic to advanced operations, the vehicle platform has actually become harder, not easier, to integrate and innovate with.

We knew there had to be a better way to keep up with the pace necessary to provide a great customer experience.

Introducing HORIZON: A Collaborative Path Forward

To tackle these pressing industry challenges, Google and Accenture have initiated Horizon. It is an open-source reference development platform designed to transform the automotive industry into a software-driven innovation market.

Our vision for Horizon is enabling automakers and OEMs to greatly accelerate their time to market and increase the agility of their teams while significantly reducing development costs. Horizon provides a holistic platform for the future of automotive software, enabling OEMs to invest more in innovation rather than just integration.

Key Capabilities Driving Software Excellence

Horizon offers a comprehensive suite of capabilities, establishing a developer-centric, cloud-powered, and easy-to-adopt open industry standard for embedded software.

1. Software-First Development with AAOS

Horizon champions a virtual-first approach to product design, deeply integrating with Android Automotive OS (AAOS) to empower software-led development cycles. This involves the effective use of the vehicle hardware abstraction layer (VHAL), virtio, and high-fidelity cloud-based virtual devices like Cuttlefish, which can scale to thousands of instances on demand. This approach allows for scalable automated software regression tests, elastic direct developer testing strategies, and can be seen as the initial step towards creating a complete digital twin of the vehicle.

2. Streamlined Code-Build-Test Pipeline

Horizon aims to introduce a standard for the entire software development lifecycle:

Code: It supports flexible and configurable code management using Gerrit, with the option to use GerritForge managed service via our Google Cloud Marketplace for productive deployments. With Gemini Code Assist, integrated in Cloud Workstations, you can supercharge development by leveraging code completion, bug identification, and test generation, while also aiding in explaining Android APIs.
Build: The platform features a scaled build process that leverages intelligent cloud usage and dynamic scaling. Key to this is the caching for AAOS platform builds based on warmed-up environments and the integration of the optimized Android Build File System (ABFS), which can reduce build times by more than 95% and allow full builds from scratch in one to two minutes with up to 100% cache hits. Horizon supports a wide variety of build targets, including Android 14 and 15, Cuttlefish, AVD, Raspberry Pi devices, and the Google Pixel Tablet. Build environments are containerized, ensuring reproducibility.
Test: Horizon enables scalable testing in Google Cloud with Android’s Compatibility Test Suite (CTS), utilizing Cuttlefish for virtualized runtime environments. Remote access to multiple physical build farms is facilitated by MTK Connect, which allows secure, low-latency interaction with hardware via a web browser, eliminating the need for hardware to be shipped to developers.

3. Cloud-Powered Infrastructure

Built on Google Cloud, Horizon ensures scalability and reliability. Deployment is simplified through tools like Terraform, GitOps and Helm charts, offering a plug-and-play toolchain and allowing for tracking the deployment of tools and applications to Kubernetes.

Unlocking Value for Auto OEMs and the Broader Industry

The Horizon reference platform delivers significant benefits for Auto OEMs:

Reduced costs: Horizon offers a reduction in hardware-related development costs and an overall decrease in rising development expenses.
Faster time to market: By accelerating development and enabling faster innovation cycles, Horizon helps OEMs reduce their time to market and feature cycle time.
Increased quality and productivity: The platform enables stable quality and boosts team productivity by providing standardized toolsets and fostering more effective team collaboration.
Enhanced customer experience: By enabling faster, more frequent and higher-quality builds, OEMs can change the way they develop vehicle software, thus offering enhanced customer experiences and unlocking new revenue streams through software-driven services.
Strategic focus: Horizon underpins the belief that efficient software development platforms should not be a point of differentiation for OEMs; instead, their innovation should be focused on the product itself. This allows OEMs to devote more time and resources to software development with greater quality, efficiency, and flexibility.
Robust ecosystem: To ensure scalable, secure, and future-ready deployments across diverse vehicle platforms, Horizon aims to foster collaboration between Google, integration partners, and platform adopters. While advancing the reference platform capabilities, Horizon also allows for tailored integration and compatibility with vehicle hardware, legacy systems and compliance standards.

The Horizon ecosystem

It’s been said that the best software is the one you don’t notice, so seamless and flawless is its functioning. This is especially true when it comes to the software-defined vehicle, where the focus should be on the road and the joy of the trip.

This is why we believe the platforms enabling efficient software development shouldn’t be differentiating for automakers — their vehicles should be. Like a solid set of tires or a good sound system, software is now essential, but it’s not the product itself. That is the full package put together by the combination of design, engineering, development, and production.

Because software development is now such an integral part of that process, we believe it should be an enabler, not a hindrance, for automakers. To that end, the Google Cloud, Android, and Accenture teams have continuously aimed to simplify access and the use of relevant toolchain components. The integration of OpenBSW and the Android Build File System (ABFS) are just the latest waypoints in a journey that started with GerritForge as providing a managed Gerrit offering, and continuing with additional partners in upcoming releases.

Please, join us on this journey. We invite you to become a part of the community to receive early insights, provide feedback, and actively participate in shaping the future direction of Horizon. You can also explore our open-source releases on Github to evaluate and customize the Horizon platform by deploying it in your Google Cloud environment and running reference workloads.

Horizon is a new dawn for the future of automotive software, though we can only get there together, through open collaboration and cloud-powered innovation.

^{A special thanks to a village of Googlers and Accenture who delivered this,}^{Mike Annau, Ulrich Gersch, Steve Basra, Taylor Santiago, Haamed Gheibi, James Brook, Ta’id Holmes, Sebastian Kunze, Philip Chen, Alistair Delva, Sam Lin, Femi Akinde, Casey Flynn, Milan Wiezorek, Marcel Gotza, Ram Krishnamoorthy, Achim Ramesohl, Olive Power,}^{Christoph Horn, Liam Friel, Stefan Beer, Colm Murphy, Robert Colbert, Sarah Kern, Wojciech Kowalski, Wojciech Kobryn, Dave M. Smith, Konstantin Weber, Claudine Laukant, Lisa Unterhauser}

^—

^{Opening image created using Imagen 4 with the prompt: Generate a blog post header image for the following blog post, illustrating the concept of a software-defined vehicle <insert the first six paragraphs>.}

Simplify complex eventing at Scale with Eventarc Advanced

Fri, 29 Aug 2025 16:00:00 +0000

Modern application development requires organizations to invest not only in scale but also in simplification and central governance. This means more than message routing; it requires a simple, unified messaging platform that can intelligently filter, transform, and govern the flow of information in real-time, taming complexity all in one place.

Today, we are excited to announce the general availability of Eventarc Advanced, a unified, serverless eventing platform that goes beyond simple routing by combining real-time filtering, transformation, management, and delivery in one place — for a complex, multi-source event-driven architecture.

Evolving Eventarc to handle complexity

Eventarc Advanced is an evolution of Eventarc Standard and offers out-of-the-box integration patterns to simplify your eventing needs.

With Eventarc Advanced, organizations can

Integrate existing services using Publish API and leverage Google Cloud events to build sophisticated event-driven applications.
Centrally manage, secure, and observe the flow of messages across services with support for per-message fine-grained access control.
Intelligently route messages to appropriate destinations based on flexible message criteria.
Transform and convert events in real-time, with support for multiple payload formats and built-in capability to transform event attributes.
Publish to Google Cloud services using HTTP binding.

With Eventarc Advanced, you can build sophisticated eventing systems. In contrast, Eventarc Standard is best for simple one-to-one eventing needs involving Google Cloud events (comparison).

Eventarc Advanced’s key technical features include:

Publish API to ingest custom and third-party messages using CloudEvents format (details).
Message bus that acts as the central nervous system of your event-driven architecture, providing centralized observability, security and management. Message bus is based on Envoy and uses the policy engine of Cloud Load Balancers and Cloud Service Mesh.

Your existing systems can publish messages to a central message bus that can be intelligently routed to appropriate consumers based on flexible criteria. The message bus simplifies event management and reduces operational overhead.
You can gain insights into your message flows with centralized monitoring, logging, and tracing capabilities. Logs are captured in Cloud Logging, providing detailed information about event processing and errors.

Out-of-the-box event mediation capabilities to adapt messages on the fly without modifying your source or destination services, and to handle different events through support for multiple payload formats (Avro, JSON, Protobuf) and built-in capability to transform event attributes.

Eventarc Advanced incorporates error-handling by offering reliable event delivery and graceful recovery from transient failures.

Empowering developers and operators

We designed Eventarc Advanced to cater to the needs of both developers and operators:

“Simplicity” for developers: Focus on building your core application features, not on complex event routing logic. Eventarc Advanced provides a unified API and a consistent experience, letting you build decoupled, reliable, and scalable services including real-time transformations.
“Centralized governance” for platform operators: Simplify the setup and management of your eventing infrastructure. Centralized governance across projects / teams, plus monitoring and logging make it easier to identify and resolve issues, reducing operational overhead.

How Eventarc Advanced works

Imagine an order processing system where orders are created, payments are processed, and items are shipped. Each action is an "event," and in a complex system, managing this flow can be challenging. This is where Eventarc Advanced comes in. It provides a centralized way to manage, observe, and route all your application's events. Let's explore how it works.

Set up your message bus
At the heart of Eventarc Advanced is a message bus that acts as the central nervous system for your event-driven application. Every event, regardless of its origin, is sent to the message bus to be analyzed and routed. This central hub is where you can define security policies, controlling exactly who can send events and what kind are allowed.

In our example, you would create a message bus to receive all order-related events. Whether an order is newly created, its payment is confirmed, or its status changes to "shipped," the events land here.

Connect your event sources
Next, connect your sources that generate order events. Event sources are the services and applications that generate events and feed them into your message bus. Eventarc Advanced makes this easy, supporting a wide range of sources, including:

Google API events
External apps or custom systems via Publish API

In our example, the event source could be a custom service using the Publish API. Every time a new order is saved or an existing one is updated, it automatically sends an event to your message bus.

Configure pipelines and destinations
This is another area where Eventarc Advanced shines. With events flowing into your message bus, you can configure pipelines to intelligently route them to the correct destinations, allowing you to filter, transform, and direct events with precision.

In the above example,

New order notification: You can set up a filter that looks for events with status: "new". This pipeline routes these events to a notification service that sends an order confirmation email to the customer.
Fraud detection: For high-value orders (e.g., amount > $1000), you can apply a transformation and route it to a specialized fraud detection service for analysis.

Unlocking new possibilities

Eventarc Advanced opens up new possibilities for your applications and workflows:

Large-scale application integration: Connect numerous services and agents, enabling them to communicate asynchronously and reliably, even across different event formats and schemas.
Event streaming for AI and analytics: Handle the influx of data from IoT devices and AI workloads by filtering and transforming them before feeding them into your analytics pipelines.
Hybrid and multi-cloud deployments: Extend your event-driven architectures beyond Google Cloud, integrating with on-premises systems and other cloud providers.

What's next

As today’s applications become increasingly agentic, distributed and data-driven, the need for efficient and secure event orchestration is more critical than ever. With upcoming native support for Service Extensions to insert custom code into the data path and services like Model Armor, Eventarc Advanced’s message bus provides security and networking controls for agent communications.

Eventarc Advanced is available today. To learn more about Eventarc Advanced, see the documentation. To learn more about event-driven architectures, visit our Architecture Center based on Google Cloud best practices. Get ready to take your event-driven architectures to the next level!

Beyond guardrails: A taxonomy of platform engineering control mechanisms

Fri, 15 Aug 2025 16:00:00 +0000

The promise of platform engineering is to accelerate software delivery by empowering developers with self-service capabilities. However, this must be balanced with security, compliance, and operational stability, and for this, you need robust controls. But all too frequently, people talk about "guardrails" — a term whose meaning is often ambiguous, leading to confusion, or worse, disdain. A platform with too many guardrails can feel like a maze of restrictions, turning off the very developers it is trying to recruit.

In order to build a governance framework that enables both fast and safe software delivery, we need to move beyond generic guardrails. In this article, we introduce a practical taxonomy of four distinct platform engineering concepts: golden paths to steer developers; guardrails that act as emergency stops; safety nets, which help ensure recovery from failure; and lastly, manual checkpoints and reviews, which introduce human judgment, oversight, and intervention into the application lifecycle. Once you understand the distinctions between these concepts, you’ll be better equipped to select the right tools and strategies for safely advancing your application through its lifecycle.

At the heart of any platform are its tenants — the development teams, applications, and services that depend on it. These tenants often operate on a shared compute platform, which makes the need for strong controls even more critical. Tenants are the "why" behind the platform controls. While we focus on building robust controls, it's crucial to understand that these mechanisms aren't just abstract rules; they are tools that enable our tenants to innovate safely and autonomously. The true value of a platform lies in how these controls are applied to provide a secure and efficient environment for each tenant, ensuring their velocity is balanced with the overall stability and security of the platform.

A modern taxonomy for platform controls

1. Golden paths: Well-paved roads that guide you

The best platforms don't block developers; they steer them. A golden path (sometimes referred to as a paved road) is a proactive, guiding track that makes the right choice the easy choice. The goal is to accelerate development by providing pre-configured, secure, and efficient patterns that developers want to use. Golden paths aren’t about preventing bad behavior with a wall, but about encouraging good behavior via a well-paved, high-speed lane. Examples include pre-approved Terraform modules that build secure infrastructure by default, standardized CI/CD pipeline templates, or internal developer portals that offer curated, one-click services.

Here are some tools you can use when creating golden paths for developers.

Custom Terraform Modules /Infrastructure Manager: Pre-approved, secure infrastructure patterns.
Internal Developer Platforms (IDPs): Simplified, curated self-service platform for developers.
Standardized CI/CD pipeline templates (in Cloud Build, ArgoCD, GitLab CI): Pre-defined, secure path for code to get to production.
Cloud Code IDE extensions (for VS Code & IntelliJ): Simplified and standardized developer interaction with Google Cloud.
Gemini Code Assist: An SDLC AI assistant that can be customized with code and rules to follow company best-practices.
Cloud Shell: A standardized, pre-configured command-line environment.
Cloud Workstations: Fully managed, secure, and pre-configured development environments.
Cloud Foundation Toolkit: Ready-made, best-practice blueprints for Terraform.

2. Guardrails: The crash barriers

In platform engineering, a guardrail is a hard stop that prevents a platform tenant from taking an action that could compromise the security or stability of their environment or the entire platform. Guardrails are the hard, non-negotiable backstops designed to protect the fundamental integrity of a platform — its security, compliance, and operational stability. While low-friction golden paths guide a developer's journey, guardrails act as the high-friction, non-negotiable last line of defense. A guardrail is not a guide rail; its purpose is to prevent a catastrophic event, not to direct the workflow. It functions like an emergency brake, not a steering wheel.Think of it as a crash barrier like in the picture that prevents a catastrophic accident — developers should rarely encounter a guardrail, and when they do, only when a significant deviation from safe practice has occurred. A guardrail doesn't consider a developer's immediate goal or speed; it only cares about preventing an action that could compromise the entire system.

In a multi-tenant platform, guardrails are essential to prevent one tenant from negatively affecting another. Tools like VPC Service Controls create an impassable perimeter around shared data and services, ensuring that a misconfigured or malicious action from one tenant cannot lead to data exfiltration for the entire platform. Similarly, in a shared GKE cluster, Gatekeeper policies enforce rules on every deployed workload, preventing a single tenant from deploying a container that could compromise the shared cluster's integrity. These controls provide the hard stops that protect all tenants from the actions of any one.

Prime examples of guardrails on Google Cloud include an Organization Policy that unconditionally blocks the creation of public storage buckets, or a Binary Authorization policy that rejects any container deployment whose image isn't cryptographically signed by a trusted source.

The following tools act as guardrails to block potentially catastrophic events.

Organization Policies: Functions as the primary service for setting non-negotiable constraints e.g., blocking public IPs, restricting resource locations, so the constraint itself is the guardrail. Organization policies establish the guardrails, and Google Services provide the means to work effectively within those guardrails.
Binary Authorization: Acts as a strict, non-negotiable gatekeeper, blocking unapproved container deployments in Google Kubernetes Engine (GKE) and Cloud Run.
VPC Service Controls: Creates an impassable network perimeter to prevent data exfiltration.
IAM Conditions and Roles: Enforces strict, context-aware access controls at runtime.
Gatekeeper: Enforces non-negotiable security profiles on pods at creation time in GKE.
Kubernetes Network Policies: Lets you control which pods can send and receive network traffic.
Container sandboxing with gVisor: Provides hard isolation between a container and the host kernel, preventing container escapes.
Vertex AI safety filters: Unconditionally blocks the generation of harmful content from AI models.
Google Cloud Firewall: A globally distributed, stateful service that allows you to enforce granular, layer 4 traffic-filtering policies for your Virtual Private Cloud (VPC) networks.
Google Cloud Armor (WAF & DDoS Mitigation): Acts as a hard shield, blocking malicious web traffic and DDoS attacks before they reach the application.
Shielded GKE Nodes / Shielded VMs: Enforces secure boot and integrity checks, preventing the node from starting if its boot sequence is compromised.
Policy-as-code tools (Open Policy Agent - OPA, Terraform Validator): Validate IaC definitions and block non-compliant changes before deployment.
Artifact Registry (when used to block vulnerable dependencies): Can be configured to block builds if dependencies with critical vulnerabilities are found.

3. Safety nets: Detection and response airbags

These are reactive controls designed to help a platform tenant quickly detect a failure or threat within their application and recover from it with minimal impact. Because failures and threats are inevitable, we need safety nets. A safety net is a reactive control that activates after an error or failure has already occurred. Its purpose is not to prevent the initial event, but to detect the problem, mitigate its impact, and facilitate a swift recovery. Continuing with the car analogy, if a golden path is the well-marked road and a guardrail is the concrete barrier, the safety net is the airbag and seatbelt — it doesn’t prevent the crash, but it dramatically reduces the harm.This category includes monitoring systems that alert on failures, automated rollback mechanisms, backup and restore procedures, and security systems that detect intrusions. The focus is on resilience and damage limitation.

These tools are used to detect and mitigate failures or threats after they have occurred.

Cloud Monitoring: Detects performance degradation, failures, and anomalies and sends alerts.
Cloud Logging: Provides the raw data to detect and investigate incidents after they happen.
Security Command Center (SCC): Acts as the central hub for detecting and viewing existing misconfigurations, vulnerabilities, and threats across Google Cloud.
Chronicle Security Operations (SIEM/SOAR): Ingests telemetry to detect complex threats and orchestrate automated responses after an event.
Cloud Trace: Helps diagnose latency issues in distributed systems after they have been detected.
Automated rollback mechanisms (in Cloud Run and GKE): Reverts a failed deployment to a last known good state.
Backup and restore procedures (Cloud Storage Example, Cloud SQL Example): Allows recovery from data loss or corruption after it has happened.
Static/Dynamic Analysis Tools (SAST/DAST - SonarQube, OWASP ZAP): Used to detect existing vulnerabilities in code.
Artifact registry vulnerability scanning: Detects known CVEs in stored container images and packages.
Firebase Test Lab: Detects issues in mobile applications by running tests on real and virtual devices.

Understanding the unique purpose of these three automated control mechanisms — golden paths (steering), guardrails (prevention), and safety nets (reacts or detects post event) — clarifies the intent behind every tool we implement and empowers us to build a platform that is both fast and safe.

Beyond automated controls: Manual checkpoints and reviews

Everything that we’ve discussed thus far — golden paths, guardrails, and safety nets — almost always refers to automated controls, which are a type of control point programmatically integrated into the platform's workflow, providing speed, consistency, and efficiency.However, other control points inherently require human judgment, oversight, and intervention — think budget approval, architecture reviews, or security post–mortems. Manual checkpoints are processes that require human oversight to ensure a platform tenant's activities, such as a major deployment or new architectural design, align with compliance and governance requirements. In a shared compute environment, manual checkpoints gain even greater importance. An architectural review is no longer just about a single application's design; it's about ensuring a tenant's proposed solution will not cause resource contention, introduce new security risks, or violate compliance requirements for the entire platform. These human-driven reviews act as a crucial final check to protect the shared foundation that all tenants depend on.

As such, manual processes are still a crucial component of a comprehensive governance framework, allowing people to judge complex scenarios. Manual checkpoints and reviews help provide accountability, holistic risk assessments, and audit trails in ways that automated systems alone cannot guarantee (albeit frequently generating a high amount of friction).

Here are some examples of scenarios where you may want to implement manual checkpoints and reviews:

FinOps cost visibility and allocation: Using tools to track cloud spending and allocate costs to specific teams or projects. Here, the Google Cloud FinOps Hub can serve as a centralized dashboard.
FinOps budgeting and forecasting: Setting budgets and forecasting future cloud costs to prevent overspending.
FinOps cost optimization: Implementing strategies to reduce cloud costs, such as rightsizing resources, using reserved instances, and automating a "lights on/lights off" approach to your cloud infrastructure.
Architectural reviews: Formal sessions where architects and senior engineers review proposed system designs. To provide a structured approach, these reviews are often guided by the Google Cloud Well-Architected Framework, where reviewers assess the design against its core pillars: security, reliability, cost optimization, performance, and operational excellence. This involves validating specific aspects, such as the design of air-gapped environments, ensuring reliability requirements are met, and confirming cost-effectiveness. These sessions provide a critical check for complex system interactions that automated tools might miss.
Code reviews (manual): While automated tools catch many issues, it’s critical for a real person to review code changes. Reviewers can identify subtle logic errors, potential race conditions, adherence to non-automatable coding standards or architectural patterns, and opportunities for knowledge sharing and mentoring.
Security assessments: Activities like manual penetration testing, targeted vulnerability assessments, and threat modeling performed by specialized security teams or third-party experts. These assessments simulate real-world attacks and probe for weaknesses that automated scanners might overlook, providing deep insights into the platform's security posture.
Change management: Formal processes for reviewing, approving, and scheduling significant changes to production environments, often involving a Change Advisory Board (CAB). The process includes assessing the potential risk and impact of changes, ensuring rollback plans are in place, and coordinating deployments. Backlog review and prioritization also fall into this category, as they involve human judgment on strategic direction.
Compliance audits: Verifying adherence to regulatory requirements (like PCI-DSS or HIPAA), which often involves manual inspection of configurations, processes, and collected evidence by internal or external auditors. Even if data gathering is automated via tools like Security Command Center, interpretation and sign-off typically require human auditors.
License management: Ensuring compliance with third-party software licenses, which can involve manual tracking, inventory management, and validation processes (although tools can assist).

The challenge lies in balancing these manual processes with the need for agility. Overly burdensome manual gates can become significant bottlenecks, slowing down delivery pipelines. Platform teams should continuously evaluate manual processes, seeking opportunities for streamlining or partial automation, all while ensuring they still provide their intended value in risk mitigation and governance.

From theory to practice: GCP primitives for tenant boundaries

Establishing strong tenant boundaries is foundational to a secure and scalable platform. On Google Cloud, you can use several primitives to create and enforce this isolation. Choosing the right one or a combination of them is a critical design decision.

Organizations, Folders, and Projects: These are fundamental to managing a shared, multi-tenant environment. They provide the logical isolation necessary when multiple development teams or applications operate on the same underlying infrastructure. Even if the hardware is common, a Project provides a strong boundary for a tenant's resources, billing, and permissions.

A primary example of this is a multi-tenant GKE cluster. In this common scenario, a single GKE cluster is shared by multiple tenants. Here, isolation isn't just a best practice — it's a necessity. We achieve this by giving each tenant their own Namespace within the cluster and using Kubernetes Network Policies and Role-Based Access Control (RBAC) to ensure they can only access their own resources and communicate as intended. This use of logical primitives is what makes a shared compute platform both efficient and secure.
GKE team scopes: These are dynamic abstractions that redistribute tenants across multiple clusters, moving beyond static boundaries. They act as intermediaries between tenant models and infrastructure, allowing platform teams to evolve systems independently. This decoupling helps enable optimal tenant bin-packing while maintaining isolation guarantees. In essence, you can create logical groupings by functional teams, departments or workload characteristics rather than being constrained by infrastructure boundaries.
Google Cloud Projects: This is the most fundamental unit of isolation in Google Cloud. Each project acts as a separate, self-contained environment with its own set of resources, permissions, and billing.
Shared Virtual Private Cloud (VPC): This primitive allows different projects (tenants) to connect to a common, centralized VPC network. It's often used when multiple teams need to access shared network resources or communicate with each other.
Identity and Access Management (IAM): IAM is the core primitive for enforcing access control within and across tenant boundaries. You can use it to define granular permissions that dictate who can do what with which resources.

These primitives can be combined to build a robust governance model. For example, you might use Organizations and Folders to group teams, Projects to provide billing and resource isolation for each tenant, Shared VPC to manage their network connectivity, and IAM to define specific permissions within each project.

Ultimately, platform engineering is about balancing developer velocity with robust governance. A successful strategy on Google Cloud depends not on a single type of control, but on a thoughtful blend of different mechanisms. By implementing low-friction golden paths to steer developers, hard-stop guardrails to prevent disaster, and resilient safety nets for swift recovery, we create a layered and effective platform-control framework. By thoughtfully combining these automated and manual controls on Google Cloud, we can build a platform that truly empowers developers without sacrificing security or stability.

In the meantime, consider these strategies for adding extra layers of control to your platform — without placing an undue burden on developers.

Adopt the new vocabulary: Before using the term "guardrail", stop and consider if you're using it as a catch-all term, or if you need to start using the more precise taxonomy of golden paths, guardrails, safety nets, or manual checkpoints correctly.
Audit your existing controls: Use this new framework as a lens to evaluate your current platform.
Build with intent: Consciously decide which type of control is most appropriate for each situation.
Balance and optimize: Continuously evaluate the balance between automated controls and manual checkpoints. Strive to build a platform that empowers developers through the software lifecycle with self-service and speed, rather than putting up yet another wall.

To learn more about platform engineering on Google Cloud, you can find more information here. Also, check out some of our other articles: 5 myths about platform engineering: what it is and what it isn’t, Another five myths about platform engineering, and Light the way ahead: Platform Engineering, Golden Paths, and the power of self-service.

Designing a multi-tenant GKE platform for Yahoo Mail's migration journey

Wed, 13 Aug 2025 16:00:00 +0000

Yahoo is in the midst of a multi-year journey to migrate its renowned Yahoo Mail application onto Google Cloud. With more than 100 services and middleware components in the application, Yahoo Mail is primarily taking a lift-and-shift approach for its on-premises infrastructure, and strategically transforming and replatforming key components and middleware to leverage cloud-native capabilities.

To ensure a successful migration, Google and The Yahoo Mail team collaborated extensively, collecting information around the current architecture, and making decisions regarding project boundaries, network architecture, and how to configure Google Kubernetes Engine (GKE) clusters. These decisions were critical due to the global nature of the application, which needs to be highly available and redundant to provide uninterrupted service to users worldwide.

As Yahoo Mail progresses towards migrating the first wave of workloads onto the production environment, we are highlighting key elements of the design, in particular its multi-tenant GKE platform, which was instrumental in establishing a robust foundation for the migration. With multi-tenancy, the Yahoo Mail application can operate efficiently on Google Cloud, helping to meet the diverse requirements of its various application tenants.

aside_block: <ListValue: [StructValue([('title', '$300 in free credit to try Google Cloud containers and Kubernetes'), ('body', <wagtail.rich_text.RichText object at 0x7f0a20e43f40>), ('btn_text', 'Start building for free'), ('href', 'http://console.cloud.google.com/freetrial?redirectpath=/marketplace/product/google/container.googleapis.com'), ('image', None)])]>

Design process

Google’s professional services organization (PSO) began the design process by taking a detailed analysis of current system usage and capacity requirements. This involved running benchmarks and collecting baseline data of existing workloads, and estimating the number of nodes that would be required based on machine types that would best suit the performance and resource demands of those workloads. Simultaneously, we discussed the optimal number of GKE clusters and cluster types, and how to best position and organize the workloads across the clusters. Another aspect of the design process was defining the number of environments (on top of the production environment). We sought to strike a balance between operational complexity, dependency proliferation to on-prem services at rollouts, and mitigating potential risks due to application defects and bugs.

But prior to making decisions about GKE clusters, we needed to determine the number of projects and VPCs. These decisions were influenced by various factors, including the customer's workload requirements and scalability objectives, and Google Cloud's service and quota limitations. At the same time, we wanted to minimize operational overhead. The analysis around the number of GKE clusters, VPCs, etc., was fairly straightforward: simply document the pros and cons of each approach. Nevertheless, we meticulously followed an extensive process, on the basis of the significant and far-reaching impact these decisions would have on the overall architecture.

We used multiple criteria to determine the number of GKE clusters we needed and how to organize workloads across them. The main characteristics we considered were resource consumption requirements of a variety of workloads, inter-services connectivity patterns, and how to strike a balance between operational efficiency and minimizing the blast radius of an outage.

Architecture diagram

Below is a simplified view of the current architecture. The core architecture consists of four GKE clusters per region group (for example, in the us-east regions in one VPC) and per environment. The architecture spans multiple regions to provide fault tolerance and high availability at the service layer. There are four region groups, and thus, there are four VPCs. External user traffic is routed to the closest region via a geolocation policy configured in Cloud DNS public zones. Internally, traffic is routed across region groups via a proxy application while traffic between GKE clusters is routed via an internal load balancer (ILB), where each ILB has a private DNS record.

Multi-tenant GKE platform characteristics

As outlined in Google's public documentation, there are multiple considerations that need to be addressed when running and operating a GKE cluster. The platform team worked diligently to address the following considerations and satisfy various tenant team's requirements:

Workload placement: The platform team assigns labels to node pools to support workload affinity, and tenants use a combination of taints and tolerations to ensure workloads are scheduled on the right node pools. This is necessary as each node pool has distinct firewall requirements based on the type of traffic it handles (HTTPS, POPS, etc). Additionally, Resource Manager Tags are used to govern firewall rules and associate the node pool with the applicable firewall rule.
Access control: Kubernetes Role-based Access Control (RBAC) is the primary mode of governing and restricting user access to cluster resources. Each tenant has one or more namespaces within their cluster and the cluster is bootstrapped with standard policies during the tenant onboarding process.
Network policies: All clusters are provisioned under dataplane v2, and Yahoo Mail uses the standard Kubernetes Network Policy to control traffic flow. Under the hood, this uses Cilium, an open-source solution for providing, securing, and observing network connectivity between workloads.
Resource quotas: To optimize resource utilization and prevent overconsumption, the platform team enforces resource quotas for cpu/memory within a namespace.
Scaling: This is determined by the platform team based on the tenant's quota requests during onboarding. Due to certain feature limitations associated with node auto-provisioning around usage of secure tags for firewall rules and defining a specific machine type and shapes, it could not be utilized, but we worked with the engineering team to develop feature requests to address this gap.

Challenges

Through the course of this migration, both Yahoo Mail and Google faced technical constraints and challenges. Below, we outline some of the key challenges faced and the approaches taken to address them:

Connecting to the control plane: As with most enterprise customers, Yahoo Mail provisioned private GKE clusters and needed connectivity between the control plane and its CI/CD tool (screwdriver) from outside the VPC network. The platform team deployed bastion hosts that were used to proxy the connection to the control plane, but it faced scalability challenges. Google worked with the customer to test out two solutions using Connect gateway and DNS-based endpoint to obviate the need for a bastion host.
End-to-end mTLS: One of Yahoo Mail key security tenets was to ensure end-to-end mTLS, which the architecture’s overall design and underlying Google Cloud services should be able to accommodate. This resulted in significant problems as one of the key load balancing products (Application Load Balancer) did not offer end-to-end mTLS at the time. We explored alternative measures such as implementing a bespoke proxy application and using Layer 4 load balancers throughout the stack. Professional services also worked with Google Cloud engineering to define the requirements for mTLS support as part of Application Load Balancer.
Integration with Athenz: Yahoo Mail used an internal tool for identity/access management, Athenz that all Google Cloud services needed to integrate with to perform workload identity federation. Within the Kubernetes context, this meant that users still needed to be authenticated via Athenz, but use workload identity federation as a mediator. As workload identity federation was also a fairly new feature, we needed to collaborate closely with Google Cloud engineering to implement successfully in the Yahoo Mail environment.
Kubernetes externalTrafficPolicy: One of the distinct features that Yahoo Mail had been most vocal and excited for was weighted routing for load balancers. This feature would allow for optimal routing of incoming traffic to the backends. While this was supported for managed instance groups, there was no native integration with GKE at the time. In its absence, the platform team had to explore and experiment with externalTrafficPolicy modes such as local and cluster mode to determine its performance impact/limitations.
Capacity planning: Last but not least, Google Professional Services performed capacity planning, a cornerstone of a successful cloud migration. Here, it entailed collaborating across multiple application teams to establish a baseline for capacity needs, making key assumptions, and estimating the resources required to meet both current and future demands. Capacity planning is a highly iterative activity that needs to evolve as workloads and requirements change. Thus, conducting regular reviews and maintaining clear communication with Google was paramount to ensure that the cloud provider could adapt to the customer's needs.

Yahoo Mail’s next milestone

Migrating an application as big as the Yahoo Mail application is a huge endeavor. With its two-pronged approach — lift and shift for most services, and strategic rearchitecting for some — Yahoo is well on its way to setting its mail system up for the next generation of customers. While a small portion of the Yahoo Mail system is in production, the majority of its services are expected to be onboarded over the next year. For more information on how Google PSO can assist with other similar services, please refer to this page.

We would like to express our gratitude to the Yahoo Mail Compute Infra team for their cooperation in sharing details and collaborating with us on this blog post.

How Google does it: Your guide to platform engineering

Wed, 13 Aug 2025 16:00:00 +0000

What guides your approach to software development? In our roles at Google, we’re constantly working to build better software, faster. Within Google, our Developer Platform team and Google Cloud have a strategic partnership and a shared strategy: together, we take our internal capabilities and engineering tools and package them up for Google Cloud customers.

At the heart of this is understanding the many ways that software teams, big and small, need to balance efficiency, quality, and cost, all while delivering value. In our recent talk at PlatformCon 2025, we shared key parts of our platform strategy, which we call “shift down.”

Shift down is an approach that advocates for embedding decisions and responsibilities into underlying internal developer platforms (IDPs), thereby reducing the operational burden on developers. This contrasts with the DevOps trend of "shift left," which pushes more effort earlier into the development cycle, a method that is proving difficult at scale due to the sheer volume and rate of change in requirements. Our shift down strategy helps us maximize value with existing resources so businesses can achieve high innovation velocity with acceptable quality, acceptable risk, and sustainable costs across a diverse range of business models. In the talk, we share learnings that have been really helpful to us in our software and platform engineering journey:

aside_block: <ListValue: []>

Work backwards from the business model: By starting with the business model, organizations can intentionally guide platform evolution and investment to align with desired margins, risk tolerance, and quality requirements. At Google, our central platform must support diverse business models, necessitating continuous strategic refinement and adaptation.
Focus on quality attributes for central software control: Quality attributes, such as reliability, security, efficiency, and performance, are emergent properties of software systems and are important for creating business value and managing risk. These are often referred to as “non-functional requirements” because they define how our software behaves, not what it functionally does. With a shift down strategy, we can embed the responsibility for assuring quality attributes directly into the underlying platform systems and infrastructure, thereby significantly reducing the operational burden on individual developers.
Abstractions and coupling are key technical tools to gain control of quality attributes: We define two key technical components in the way we build platforms: abstractions and coupling. In a shift down strategy, abstractions provide understandability, risk management levers, accountability, and cost control by encapsulating complexity. Coupling refers to the interconnectedness and interdependence of components within a system or development ecosystem. For a successful shift down strategy, the right degree of coupling is crucial because it allows the development platform and ecosystem design to directly implement and influence quality attributes. In fact, coupling is how we offer entire infrastructure and platform solutions as coherent services like Google Kubernetes Engine (GKE).
Shared responsibility, education, and policy are equally important social tools: Shared responsibility is a crucial social tool within software at scale. This is actively cultivated through education, such as training engineers on platform and AI usage, and fostering a "one team" culture that encourages a shift from artifact-bound identities to overarching mission goals and client-focused engagement. Furthermore, explicit policies like centrally enforced style guides and secure-by-design APIs are fundamental for embedding quality attribute assurance directly into the platform and infrastructure, significantly reducing the operational burden on individual developers by ensuring consistency and automated controls at scale.
Use a map. Supporting many business units with one platform is a vast and complex problem; we need a map. The ecosystem model is a framework that categorizes different types of software development environments, ranging from highly flexible, developer-controlled systems to highly opinionated, vertically integrated ones where the ecosystem itself assures quality attributes. Its critical purpose is to provide a visual and conceptual tool for evaluating how well our ecosystem controls match our business risk. This helps us ensure that the level of oversight and assurance of quality attributes aligns with the potential cost of mistakes. The goal is to be in the "ecosystem effectiveness zone," where controls are balanced to mitigate significant risks from human error without imposing overly restrictive systems that negatively impact velocity and developer satisfaction.

6. Divide up the problem space by identifying different platform and ecosystem types.

Because the developer experience and platform infrastructure change with scale and degree of shifting down, it’s not enough to just know where the ecosystem effectiveness zone is — you have to identify the ecosystem by type. We differentiate ecosystem types by the degree of oversight and assurance for quality attributes. As an ecosystem becomes more vertically integrated, such as Google's highly optimized "Assured" (Type 4) ecosystem, the platform itself assumes increasing responsibility for vital quality attributes, allowing specialists like site reliability engineers (SRE) and security teams to have full ownership in taking action through large-scale observability and embedded capabilities. Conversely, in less uniform "YOLO," "AdHoc," or "Guided" (Type 0-2) ecosystems, developers have more responsibility for assuring these attributes, while central specialist teams have less direct control and enforcement mechanisms are less pervasive. It’s really important to note here that this is not a maturity model — the best ecosystem and platform type is the one that best fits your business need (see point #1 above!).

Intentional choices in platform engineering

The most important takeaway is to make active choices. Tailor platform engineering for each business unit and application to achieve the best outcomes. Place critical emphasis on identifying and solving stable sub-problems in reliable, reusable ways across various business problems. This approach directly underpins our "shift down" strategy, moving toward composable platforms that embed decisions and responsibilities for software quality directly into the underlying platform infrastructure, thereby improving our ability to maximize business value with the right resources, at the right quality level, and with sustainable costs.

Watch our full discussion for more insights on effective platform engineering.

How Renault Group is using Google’s software-defined vehicle industry solution

Wed, 16 Jul 2025 16:00:00 +0000

It’s funny to think of Renault Group, the massive European car manufacturer, as a software company, but in many ways, it is. Renault Group subsidiary Ampere Software Technology is dedicated to developing and integrating advanced software solutions for intelligent electric vehicles, aiming to create software-defined vehicles (SDVs) with enhanced customer experiences and new services.

Ampere develops Renault Group’s software-defined vehicle based on Google’s AAOS SDV solution. But like all software companies, it struggled to contain costs, sync code bases, maintain adequate testing regimens, and onboard new talent.

Building on the existing partnership between Google Cloud and Renault Group, Ampere chose a Google Cloud solution for its software-defined vehicle development. This solution, leveraging Google Cloud Workstations and Gemini Code Assist, effectively streamlined the process, making it more secure and productive by eliminating many common development hurdles.

For security-conscious enterprises, Cloud Workstations offer fully managed development environments. Concurrently, Gemini Code Assist, driven by Gemini 2.5, provides secure generative AI coding assistance and agents across the entire software development lifecycle. And by utilizing Google's virtual twin technology, specifically developed for Google’s AAOS SDV and AAOS IVI (Android Automotive OS for IVI), Ampere constructed full digital counterparts to their automobiles.

Let’s take a closer look at the components in this solution:

Google Cloud Workstations

Google Cloud Workstations significantly boosts Android Open Source Project (AOSP) developer productivity in general. In the context of Ampere, it offers on-demand development environments with persistent disks, pre-synced with the customer’s AAOS SDV and AAOS IVI repositories. This eliminates lengthy sync and build times, allowing developers to access their work from anywhere. Ampere's Platform Admins provision these workstations, drastically cutting down the time it takes for developers to become productive. Developers have instant access to powerful virtual machines with ample vCPUs, RAM, and fast SSD storage — important for the demanding emulators that they use. This resource elasticity prevents bottlenecks and accelerates development. Then, secure, authenticated cloud access and Google Cloud security tools helps to significantly reduce IP leaks and unauthorized access. Finally, having a consistent development environment prevents "works on my machine" problems and reduces debugging time, while flexible access and disk configurations enhance AAOS SDV and AAOS IVI developers’ productivity by enabling workstation access from anywhere, and preserving codebases, configurations, and build artifacts across sessions via persistent disks, eliminating repeated repo syncs.

aside_block: <ListValue: [StructValue([('title', 'Try Google Cloud for free'), ('body', <wagtail.rich_text.RichText object at 0x7f09ee603970>), ('btn_text', 'Get started for free'), ('href', 'https://console.cloud.google.com/freetrial?redirectPath=/welcome'), ('image', None)])]>

Gemini Code Assist for AI-powered Android development

To help, Ampere offered their developers Android Studio and Code OSS IDEs integrated with Gemini Code Assist, helping to address code management complexity, reduce steep learning curves, and prevent errors. Gemini Code Assist uses retrieval-augmented generation (RAG) to access Ampere’s private codebases and documentation, providing relevant and accurate code suggestions tailored to their Android development standards and conventions. It sped up understanding of their vast codebases by explaining functions, summarizing modules, and suggesting next steps, benefiting new developers and those working on different parts of the SDV software. It also helped boost their Android development productivity by automating boilerplate code, suggesting APIs, and finding potential problems, letting developers concentrate on core SDV logic instead of repetitive tasks.

The virtual twin

Google Cloud enabled Ampere’s AAOS SDV developers and testers to use a "virtual twin" of a car, resolving resource and complexity issues associated with physical or poorly managed virtual testing. Developers can use powerful Compute Engine instances and specialized Android emulators like Cuttlefish to create accurate virtual vehicle embedded systems. This enables rigorous software testing with virtual hardware, helping to ensure robust performance before building physical prototypes. AAOS SDV developers can also use scalable virtual devices for parallel testing, comprehensive regression suites, and simulations, accelerating the "test" phase of the CI/CD pipeline and improving the SDV lifecycle. The virtual twin is integrated with the Cloud Workstations development environment and the customer’s CI/CD pipelines (e.g., powered by GKE and GitLab), allowing developers to build their AAOS SDV changes on their workstation, trigger automated tests on a fleet of virtual twins, and get immediate feedback on their code.

Tangible returns of modernized SDV development

By combining the robust, managed infrastructure of Cloud Workstations, Gemini Code Assist’s intelligent assistance, and virtual twins, Google Cloud is helping Renault modernize automotive software development, accelerate innovation, and bring new features to market at unprecedented speeds.

“...to invest and build the software platform for the software defined vehicle in Europe ..you need the tools and this is where Google shines ... At the heart of it is AI and when we talk about code generation, instantiation of things that you need to be running immediately versus waiting for the thing to compile and having be available to the developer … to make our engineers more efficient so we can do more with less time because we are challenged with time.” - Henry Bzeih, Ampere Chief Software Officer (Renault Group)

Google Cloud and Gemini Code Assist offer automotive OEMs a transformational shift, extending beyond mere tool adoption to significantly impacting business results. This transition enhances competitiveness, profitability, and innovation speed.

Traditionally, onboarding new developers takes days and is costly. AAOS SDV development often involves time-consuming setup, dependency management, and build system troubleshooting. By reducing environment setup time from days to minutes — including repository syncing and toolchain configuration — and utilizing AI assistance, the development process is vastly accelerated.

OEMs worry about intellectual property leaks from local devices. Cloud Workstations addresses this concern by operating within the customer's Virtual Private Cloud. This approach prevents source code from being synced locally and exposed on endpoints, reducing security risks.

While cloud infrastructure has associated costs, it yields substantial cost optimization. Eliminating high-end local machines, minimizing wasted developer time on environment management, and speeding up timelines all lower total development costs. The ability to quickly adjust cloud resources ensures payment only for active usage, avoiding idle hardware expenses, and a better return on engineering investment.

Finally, having a virtual twin of the car improves quality assurance and validation. Instead of relying on limited prototypes or unreliable local emulators, developers can use detailed virtual car models, facilitating faster iteration, scalable testing, early bug detection, and advanced scenario simulation.

Automotive companies are not only adopting new technologies but are also reshaping their development capabilities by utilizing Cloud Workstations and Gemini Code Assist. For more, watch the fireside chat with Henry Bzeih, Ampere Chief Software Officer (Renault Group) on their success with this SDV Industry solution.

And if you’re in the automotive industry, you can get started on setting up custom AAOS SDV or AAOS IVI development environments with Gemini Code Assist by referring to our GitHub repository. And of course, your Google Partner Engineering or Customer Engineering contacts are ready to help!

From localhost to launch: Simplify AI app deployment with Cloud Run and Docker Compose

Thu, 10 Jul 2025 09:30:00 +0000

At Google Cloud, we are committed to making it as seamless as possible for you to build and deploy the next generation of AI and agentic applications. Today, we’re thrilled to announce that we are collaborating with Docker to drastically simplify your deployment workflows, enabling you to bring your sophisticated AI applications from local development to Cloud Run with ease.

Deploy your compose.yaml directly to Cloud Run

Previously, bridging the gap between your development environment and managed platforms like Cloud Run required you to manually translate and configure your infrastructure. Agentic applications that use MCP servers and self-hosted models added additional complexity.

The open-source Compose Specification is one of the most popular ways for developers to iterate on complex applications in their local environment, and is the basis of Docker Compose. And now, gcloud run compose up brings the simplicity of Docker Compose to Cloud Run, automating this entire process. Now in private preview, you can deploy your existing compose.yaml file to Cloud Run with a single command, including building containers from source and leveraging Cloud Run’s volume mounts for data persistence.

Supporting the Compose Specification with Cloud Run makes for easy transitions across your local and cloud deployments, where you can keep the same configuration format, ensuring consistency and accelerating your dev cycle.

“We’ve recently evolved Docker Compose to support agentic applications, and we’re excited to see that innovation extend to Google Cloud Run with support for GPU-backed execution. Using Docker and Cloud Run, developers can now iterate locally and deploy intelligent agents to production at scale with a single command. It’s a major step forward in making AI-native development accessible and composable. We’re looking forward to continuing our close collaboration with Google Cloud to simplify how developers build and run the next generation of intelligent applications.” - Tushar Jain, EVP Engineering and Product, Docker

Cloud Run, your home for AI applications

Support for the compose spec isn’t the only AI-friendly innovation you’ll find in Cloud Run. We recently announced general availability of Cloud Run GPUs, removing a significant barrier to entry for developers who want access to GPUs for AI workloads. With its pay-per-second billing, scale to zero, and rapid scaling (which takes approximately 19 seconds for a gemma3:4b model for time-to-first-token), Cloud Run is a great hosting solution for deploying and serving LLMs.

This also makes Cloud Run a strong solution for Docker’s recently announced OSS MCP Gateway and Model Runner, making it easy for developers to take the AI applications locally to production in the cloud seamlessly. By supporting Docker’s recent addition of ‘models’ to the open Compose Spec, you can deploy these complex solutions to the cloud with a single command.

Bringing it all together

Let's review the compose file for the above demo. It consists of a multi-container application (defined in services) built from sources and leveraging a storage volume (defined in volumes). It also uses the new models attribute to define AI models and a Cloud Run-extension defining the runtime image to use:

code_block: <ListValue: [StructValue([('code', 'name: agent\r\nservices:\r\n webapp:\r\n build: .\r\n ports:\r\n - "8080:8080"\r\n volumes:\r\n - web_images:/assets/images\r\n depends_on:\r\n - adk\r\n\r\n adk:\r\n image: us-central1-docker.pkg.dev/jmahood-demo/adk:latest\r\n ports:\r\n - "3000:3000"\r\n models:\r\n - ai-model\r\n\r\nmodels:\r\n ai-model:\r\n model: ai/gemma3-qat:4B-Q4_K_M\r\n x-google-cloudrun:\r\n inference-endpoint: docker/model-runner:latest-cuda12.2.2\r\n\r\nvolumes:\r\n web_images:'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f09ee5c9f40>)])]>

Building the future of AI

We’re committed to offering developers maximum flexibility and choice by adopting open standards and supporting various agent frameworks. This collaboration on Cloud Run and Docker is another example of how we aim to simplify the process for developers to build and deploy intelligent applications.

Compose Specification support is available for our trusted users — sign up here for the private preview.

Using Platform Engineering to simplify the developer experience - part one

Thu, 26 Jun 2025 16:00:00 +0000

Editor's note: This is part one of the story. After you’re finished reading, head over to part two.

In 2017, John Lewis, a major UK retailer with a £2.5bn annual online turnover, was hampered by its monolithic e-commerce platform. This outdated approach led to significant cross-team dependencies, cumbersome and infrequent releases (monthly at best), and excessive manual testing, all further hindered by complex on-premises infrastructure. What was needed were some bold decisions to drive a quick and significant transformation.

The John Lewis engineers knew there was a better way. Working with Google Cloud, they modernized their e-commerce operations with Google Kubernetes Engine. They started with the frontend, and started to see results fast: the frontend was moved onto Google Cloud in mere months, releases to the frontend browser journey started to happen weekly, and the business gladly backed expansion into other areas.

At the same time, the team had a broader strategy in mind: to take a platform engineering approach, creating many product teams who built their own microservices to replace the functionality of the legacy commerce engine, as well as creating brand new experiences for customers.

And so The John Lewis Digital Platform was born. The vision was to empower development teams and arm them with the tools and processes they needed to go to market fast, with full ownership of their own business services. The team’s motto? "You Build It. You Run It. You Own It." This decentralization of development and operational responsibilities would also enable the team to scale.

This article features insights from Principal Platform Engineer Alex Moss, who delves into their strategy, platform build, and key learnings of John Lewis’ journey to modernize and streamline its operations with platform engineering — so you can begin to think about how you might apply platform engineering to your own organization.

aside_block: <ListValue: [StructValue([('title', 'Try Google Cloud for free'), ('body', <wagtail.rich_text.RichText object at 0x7f0a212ffc10>), ('btn_text', 'Get started for free'), ('href', 'https://console.cloud.google.com/freetrial?redirectPath=/welcome'), ('image', None)])]>

Step 1: From monolithic to multi-tenant

In order to make this happen, John Lewis needed to adopt a multi-tenant architecture — one tenant for each business service, allowing each owning team to work independently without risk to others -- and thereby permitting the Platform team to give the team a greater degree of freedom.

Knowing that the business' primary objective was to greatly increase the number of product teams helped inform our initial design thinking, positioning ourselves to enable many independent teams even though we only had a handful of tenants.

This foundational design has served us very well and is largely unchanged now, seven years later. Central to the multi-tenant concept is what we chose to term a "Service" — a logical business application, usually composed of several microservices plus components for storing data.

We largely position our platform as a “bring your own container” experience, but encourage teams to make use of other Google Cloud services — particularly for handling state. Adopting services like Firestore and Pub/Sub reduces the complexity that our platform team has to work with, particularly for areas like resilience and disaster recovery. We also favor Kubernetes over compute products like Cloud Run because it strikes the right balance for us between enabling development teams to have freedom whilst allowing our platform to drive certain certain behaviours, e.g., the right level of guardrails, without introducing too much friction.

On our platform, Product Teams (i.e., tenants) have a large amount of control over their own Namespaces and Projects. This allows them to prototype, build, and ultimately operate, their workloads without dependency on others — a crucial element of enabling scale.

Our early-adopter teams were extremely helpful in helping evolve the platform; they were accepting of the lack of features and willing to develop their own solutions, and provided very rich feedback on whether we were building something that met their needs.

The first tenant to adopt the platform was rebuilding the johnlewis.com, search capability, replacing a commercial-off-the-shelf solution. This team was staffed with experienced engineers familiar with modern software development and the advantages of a microservice-based architecture. They quickly identified the need for supporting services for their application to store data and asynchronously communicate between their components. They worked with the Platform Team to identify options, and were onboard with our desire to lean into Google Cloud native services to avoid running our own databases or messaging. This led to us adopting Cloud Datastore and Pub/Sub for our first features that extended beyond Google Kubernetes Engine.

All roads lead to success

A risk with a platform that allows very high team autonomy is that it can turn into a bit of a wild-west of technology choices and implementation patterns. To handle this, but to do so in a way that remained developer-centric, we adopted the concept of a paved road, analogous to a “golden path.”

We found that the paved road approach made it easier to:

build useful platform features to help developers do things rapidly and safely
share approaches and techniques, and engineers to move between teams
demonstrate to the wider organisation that teams are following required practices (which we do by building assurance capabilities, not by gating release)

The concept of the paved road permeates most of what the platform builds, and has inspired other areas of the John Lewis Partnership beyond the John Lewis Digital space.

Our paved road is powered by two key features to enable simplification for teams:

The Paved Road Pipeline. This operates on the whole Service and drives capabilities such as Google Cloud resource provisioning and observability tools.
The Microservice CRD. As the name implies, this is an abstraction at the microservice level. The majority of the benefit here is in making it easier for teams to work with Kubernetes.

Whilst both features were created with the developer experience in mind, we discovered that they also hold a number of benefits for the platform team too.

The Paved Road Pipeline is driven by a configuration file — in yaml (of course!) — which we call the Service Definition. This allows the team that owns the tenancy to describe, through easy-to-reason-about configuration, what they would like the platform to provide for them. Supporting documentation and examples help them understand what can be achieved. Pushes to this file then drive a CI/CD pipeline for a number of platform-owned jobs, which we refer to as provisioners. These provisioners are microservices-like themselves in that they are independently releasable and generally focus on performing one task well. Here are some examples of our provisioners and what they can do:

Create Google Cloud resources in a tenant’s Project. For example, Buckets, PubSub, and Firestore — amongst many others
Configure platform-provided dashboards and custom dashboards based on golden-signal and self-instrumented metrics
Tune alert configurations for a given microservice’s SLOs, and the incident response behaviour for those alerts

Our product teams are therefore freed from the need to familiarize themselves deeply with how Google Cloud resource provisioning works, or Infrastructure-as-Code (IaC) tooling for that matter. Our preferred technologies and good practices can be curated by our experts, and developers can focus on building differentiating software for the business, while remaining fully in control of what is provisioned and when.

Earlier, we mentioned that this approach has the added benefit of being something that the platform team can rely upon to build their own features. The configuration updated by teams for their Service can be combined with metadata about their team and surfaced via an API and events published to Pub/Sub. This can then drive updates to other features like incident response and security tooling, pre-provision documentation repositories, and more. This is an example of how something that was originally intended as a means to help teams avoid writing their own IaC can also be used to make it easier for us to build platform features, further improving the value-add — without the developer even needing to be aware of it!

We think this approach is also more scalable than providing pre-built Terraform modules for teams to use. That approach still burdens teams with being familiar with Terraform, and versioning and dependency complexities can create maintenance headaches for platform engineers. Instead, we provide an easy-to-reason-about API and deliberately burden the platform team, ensuring that the Service provides all the functionality our tenants require. This abstraction also means we can make significant refactoring choices if we need to.

Adopting this approach also results in a broad consistency in technologies across our platform. For example, why would a team implement Kafka when the platform makes creating resources in Pub/Sub so easy? When you consider that this spans not just the runtime components that assemble into a working business service, but also all the ancillary needs for operating that software — resilience engineering, monitoring & alerting, incident response, security tooling, service management, and so on— this has a massive amplifying effect on our engineers’ productivity. All of these areas have full paved road capabilities on the John Lewis Digital Platform, reducing the cognitive load for teams in recognizing the need for, identifying appropriate options, and then implementing technology or processes to use them.

That being said, one of the reasons we particularly like the paved road concept is because it doesn't preclude teams choosing to "go off-road." A paved road shouldn’t be mandatory, but it should be compelling to use, so that engineers aren’t tempted to do something else. Preventing use of other approaches risks stifling innovation and the temptation to think the features you've built are "good enough." The paved road challenges our Platform Engineers to keep improving their product so that it continues to meet our Developers' changing needs. Likewise, development teams tempted to go off-road are put off by the increasing burden of replicating powerful platform features.

The needs of our Engineers don’t remain fixed, and Google Cloud are of course releasing new capabilities all the time, so we have extended the analogy to include a “dusty path” representing brand new platform features that aren’t as feature-rich as we’d like (perhaps they lack self-service provisioning or out-the-box observability). Teams are trusted to try different options and make use of Google Cloud products that we haven't yet paved. The Paved Road Pipeline allows for this experimentation - what we term "snowflaking". We then have an unofficial "rule of three", whereby if we notice at least 3 teams requesting the same feature, we move to make the use of it self-service.

At the other end of the scale, teams can go completely solo — which we refer to as “crazy paving” — and might be needed to support wild experimentation or to accommodate a workload which cannot comply with the platform’s expectations for safe operation. Solutions in this space are generally not long-lived.

In this article, we've covered how John Lewis revolutionized its e-commerce operations by adopting a multi-tenant, "paved road" approach to platform engineering. We explored how this strategy empowered development teams and streamlined their ability to provision Google Cloud resources and deploy operational and security features.

In part 2 of this series, we'll dive deeper into how John Lewis further simplified the developer experience by introducing the Microservice CRD. You'll discover how this custom Kubernetes abstraction significantly reduced the complexity of working with Kubernetes at the component level, leading to faster development cycles and enhanced operational efficiency.

To learn more about shifting down with platform engineering on Google Cloud, you can find more information available here. To learn more about how Google Kubernetes Engine (GKE) empowers developers to effortlessly deploy, scale, and manage containerized applications with its fully managed, robust, and intelligent Kubernetes service, you can find more information here.