Data Analytics

The future of orchestration: Pine59’s journey to Airflow 3 on Google Cloud

Thu, 17 Sep 2026 17:00:00 +0000

Operating large data pipelines requires an orchestration layer that scales smoothly as workloads expand. When your pipelines process millions of complex data points every day to feed predictive models, staying up-to-date with your technology stack is a strategic necessity.

Pine59 provides location intelligence data through data pipelines that produce analytical metrics on cadences ranging from hourly to quarterly. One of the company’s most data-intensive metrics, Daily Foot Traffic, computes data for as many as 14 million distinct locations in a single job. To handle this massive volume, Pine59’s system runs entirely on Google Cloud, with the heavy lifting in BigQuery and all of it orchestrated by Managed Service for Apache Airflow (formerly Cloud Composer) running Apache Airflow 3.

As the company’s volume of data and number of machine learning workloads scaled up, Pine59 decided to modernize its monorepo, which contains hundreds of directed acyclic graphs (DAGs). Here is a look at how that transition improved Pine59’s MLOps capabilities, developer workflow, and pipeline speed.

Proactive modernization for growth

Pine59 has long relied on a shared monorepo with code and tooling spanning multiple projects to run its metric production pipelines. As it considered its infrastructure’s future, the company wanted to help its data pipelines run faster and more reliably.

That’s why it decided to stress-test production workloads against the newly available Managed Airflow (Gen 3) architecture running Airflow 3. The initial results were unambiguous: the Gen 3 environment delivered immediate and significant processing speed, task scheduling, and overall stability improvements. Recognizing the clear potential for performance gains, Pine59 initiated a full transition to the new environment.

Orchestrating advanced MLOps

Pine59’s pipelines don’t just move data; they drive complex ML models, so a core aspect of its migration was optimizing the orchestration of its ML inference workloads.

Previously, Pine59 had used standard Kubernetes operators for these tasks. By moving to Managed Airflow (Gen 3), which features a highly optimized and abstracted infrastructure layer, the company’s engineering team refined its MLOps architecture. They did so by setting up a dedicated Google Kubernetes Engine (GKE) cluster that was specifically optimized for model inference and integrated it into the Pine59 pipelines.

This clear separation of orchestration and heavy ML execution compute allows data processing and model inference to run efficiently, showcasing Managed Airflow as a resilient, scalable backbone for enterprise MLOps.

Supporting developers with custom extensibility

Beyond infrastructure improvements, Pine59 was also able to immediately capitalize on Airflow 3’s delivery of a vastly improved developer workflow and user interface. Indeed, managing hundreds of interconnected DAGs requires excellent observability, and Pine59 found Airflow 3’s plugin authoring system remarkably easy to use.

To improve internal developer velocity, the company quickly built a number of custom plugins that it integrated directly into its new Airflow UI:

BigQuery Auto-linkify: A tool that automatically detects internal BigQuery table references within the Airflow Logs and XCom tabs, dynamically generating direct links to BigQuery Studio for faster debugging (available as a public GitHub gist)
DAG Run Configuration Search: A custom search form added directly to the DAG overview page. It allows Pine59 engineers to query specific key-value pairs within DAG run payloads (configs) and instantly surface matching runs. This in turn drastically reduces troubleshooting time.

In addition, the team also deployed a compatibility shim layer within its monorepo. This “compat” module dynamically abstracts logic between Airflow versions, streamlining operator migration across versions.

Faster, more reliable pipelines

For Pine59, migrating to Managed Airflow (Gen 3) with Airflow 3 has yielded clear, quantifiable results.

The most important improvement was the speed of its DAG runs. In the company’s previous setup, tasks often got stuck in a queued state during peak processing surges. With Gen 3, queue latency has dropped dramatically, allowing tasks to start running almost immediately.

Consider the comparison below of total aggregated “queued” & “running” time of more than 300 runs of the same DAG between Managed Airflow (Gen2) with Airflow 2.11 vs. Managed Airflow (Gen3) with Airflow 3.1 below. As we can readily see, the difference in queued time is significant.

Coupled with internal DAG optimizations made during the transition, the performance gains are also highly tangible. For example, the Daily Foot Traffic pipeline previously took nearly 38 minutes to complete. With the new instance, the same workload now takes less than 26 minutes —nearly 32% less processing time.

Today, Pine59 processes all its production workloads on its new Managed Airflow (Gen 3) instance. By moving to this next generation orchestration, the company improved its MLOps capabilities, equipped its developers with better tools, and built a faster, more resilient foundation for future workloads.

If your engineering team spends more time managing infrastructure than delivering value, consider a similar transition and discover how it can help you move from maintaining servers to building the future of your data and AI pipelines today.

^{Special thanks to the following contributor to this post: Alexandre Crespo-Perez}

What’s new with Google Data Cloud

Thu, 17 Sep 2026 16:00:00 +0000

September 14 - September 17

Managed Service for Apache Kafka supports clusters with public Internet access!
With Managed Kafka public clusters, you can now produce and consume messages from clients outside your VPC—including your local machine, for faster, frictionless testing. Public clusters unlock use cases like IoT devices, retail storefronts, and telco network towers. Enable public access on new or existing clusters via the Google Cloud console, gcloud CLI, or REST API. Spin up your first public cluster, or reach out to kafka-hotline@google.com with questions.
Stream data directly into Bigtable using Bigtable subscriptions, now in Preview!
You can write Pub/Sub messages to a Bigtable table with zero ETL with Bigtable subscriptions. No pipelines, no code, delivered by the serverless, zero-ops experience you already know with Pub/Sub. Power your AI workloads, from model telemetry to real-time context engineering, without the overhead of managing complicated ETL pipelines. Built to be dependable, with native support for dead-letter topics. Try the feature today!

September 7 - September 10

Pub/Sub SMTs can now AI Inference your Gemini Enterprise Agent Platform models!
Pub/Sub AI Inference SMTs allow you to apply inference on an incoming stream of events using models hosted in Gemini Enterprise Agent Platform. The model’s prediction is appended to your event, making it available for downstream processing in your data warehouse (like BigQuery) or operational database (like BigTable). This feature, now generally available, can dramatically simplify or enhance anomaly detection systems you are operating.
PostgreSQL Source Connector is now generally available in Managed Service for Apache Kafka!
Managed Service for Apache Kafka’s PostgreSQL connector allows customers to capture changes from their PostgreSQL database and ingest them into their Kafka infrastructure with low latency. This source connector is compatible with Cloud SQL for Postgres, AlloyDB, and self-managed PostgreSQL databases. Try this along with our entire portfolio of managed connectors, including MirrorMaker 2.0, BigQuery, Cloud Storage, and Pub/Sub! E-mail kafka-hotline@google.com if you have questions or feedback!

Pause-on-failure for Dataflow batch jobs is GA
Dataflow pause-on-failure enables you to preserve the state of a batch Dataflow job before it fails. By pausing your Dataflow job, you can address issues that are external to the pipeline and resume processing without losing completed work. This helps you better manage resource costs and improve job reliability when you face temporary outages or capacity constraints.

The insertAll API is now the BigQuery Storage Write API (REST)
The legacy insertAll streaming API is now rebranded as the BigQuery Storage Write API (REST). By dropping the "legacy" label, developers can confidently build long-term HTTP-based streaming workflows. This stateless JSON-over-HTTPS endpoint offers a lightweight alternative to heavy gRPC libraries—ideal for serverless web apps, IoT telemetry, and AI logging. The transition is seamless for existing users, requiring zero code changes and offering 100% backward compatibility. However, the Storage Write API (gRPC) version remains the recommended standard for high-throughput, continuous pipelines.

August 31 - September 4

Stateful processing is available in BigQuery continuous queries in Preview
Stateful operations significantly expand what’s possible with BigQuery continuous queries. This feature allows users to leverage functions like JOINs, aggregations, and windowing functions directly in their streaming queries. Now you can calculate metrics over time (for example, a 30-minute average) to power your downstream applications and AI agents with much richer, real-time signals.

Try out our feature here and share your feedback with bq-continuous-queries-feedback@google.com!
Synthetic data generator tool is available for Managed Service for Kafka
You’ve launched your first Kafka cluster. Now what? The next thing to do is to produce some data to the cluster, but that involves modifying a client application somewhere or spinning up a virtual machine. The synthetic data generator tool, now generally available, can start sending mock data to your cluster in 3 clicks, and will get data streaming into your cluster in less than two minutes. The perfect utility for those moments you just want to test your cluster and new features. Try our quickstart today!

Dataflow pipeline updates are faster & more flexible
Dataflow pipeline updates can now stop-and-replace pipelines, a major addition to the existing in-place-update feature. The new parallel pipeline option accelerates the migration between the old & new pipeline, resulting in reduced disruption to your business. You can also set a timeout on drains that prevents runaway costs for your pipeliness in the event of stuck processing. This feature is generally available. Try it here!

July 6 - July 10

New Lakehouse managed tables now in preview
Lakehouse tables for Apache Iceberg are now in preview and available in the console. By using Google-managed Apache Iceberg tables in Lakehouse, you can eliminate the overhead of maintaining duplicate data pipelines and complex synchronization logic between BigQuery and open-source engines. This unified table format delivers native, multi-engine read and write interoperability, allowing you to run concurrent DML/DDL operations across diverse analytics tools on a single, shared storage layer. Built-in automated table management handles painful background optimization tasks like compaction and partition tuning, freeing up your team to focus on building rather than managing storage maintenance.

June 1 - June 5

Beyond the Query: Powering AI Agents with Bigtable, Firestore & Memorystore
Discover the latest advancements in Google Cloud's NoSQL Database portfolio, including Bigtable, Firestore, and Memorystore. This series is designed for a broad audience: whether you are exploring these databases for the first time or are an existing user looking to leverage the new capabilities announced at Next '26.

Register here to secure your spot!

Cloud Engineer's AI Toolkit Workshops: Solve data-driven challenges with BigQuery, AlloyDB, Gemini and more. Hosted by Google Cloud Labs, this highly technical event is built specifically for Platform Engineers, SREs, and cloud infrastructure teams ready to bridge the gap between AI prototypes and production-grade deployments. Look out for more locations coming soon

Toronto - June 25 (Data Cloud) | RSVP Here
Chicago - June 30 (Data Cloud) | RSVP Here
Start a 10-day Bigtable free trial with a 1 node SSD cluster and up to 500GB of storage capacity. With no credit card required to start, you can easily ingest workloads and manage workloads that require low-latency, high-throughput, and predictable access. Plus, new Google Cloud customers get $300 in free credits on signup.

May 11 - May 15

Managed Service for Apache Airflow has launched a wave of new features, including the general availability of Airflow 3.1, AI-powered agentic troubleshooting, a new managed Airflow MCP Server for custom agent integration, and declarative YAML-based orchestration pipelines—discover all the details in the full blog post.

April 20 - April 24

Google-built ODBC Driver for BigQuery is now available in Preview
We are excited to announce the launch of the new, Google-built ODBC driver for BigQuery. This new open-source driver provides a direct, high-performance connection for applications to BigQuery and is developed entirely in-house by Google. Download a new driver and connect your application to BigQuery.

April 13 - April 17

We announced we are reintroducing Data Studio to play a significant role in the AI era, expanding from data visualizations and reports to host BigQuery conversational agents and data apps built in Colab notebooks.
We announced BigQuery Graph is now available in preview, offering an easy-to-use, highly scalable graph analytics solution, empowering data professionals to model, analyze and visualize massive-scale relationships in an entirely new way.

April 6 - April 10

We introduced Conversational Analytics for Looker Embedded environments, enabling users to add natural language experiences to their own custom data-driven applications, powered by Gemini.
We expanded Looker’s capabilities for faster ad-hoc analysis, with the introduction of self-service Explores, enabling you to bring your own data to Looker’s semantic layer and gain instant access to insights in a governed data environment.

March 23 - March 27

We showed you how you can scale your reads with Cloud SQL autoscaling read pools. This feature allows you to provision multiple read replicas that are accessible via a single read endpoint and to dynamically adjust your read capability based on real-time application needs.
Our customers are leveraging the full power of Conversational Analytics and Looker to drive major business and technical breakthroughs in the AI era. Companies like Telenor, Pet Circle, Fluent Commerce, Lighthouse Intelligence, Wego, and ROLLER are turning data into insights and actions, grounded by Looker’s semantic layer.

March 16 - March 20

We introduced an enhanced Gemini assistant in BigQuery Studio, transforming the agent from a code assistant into a fully context-aware analytics partner.

February 23 - February 27

We introduced managed and remote MCP support for Google Cloud databases, including AlloyDB, Spanner, Cloud SQL, Bigtable and Firestore, to power the next generation of agents. This announcement extends the ability for AI models to plan, build, and solve complex problems, connecting to the database tools our customers leverage daily as the backbone of their work environment.
We outlined how you can build a conversational agent in BigQuery using the Conversational Analytics API to help you build context-aware agents that can understand natural language, query your BigQuery data, and deliver answers in text, tables, and visual charts.

February 16 - February 20

Our customers are leveraging the full power of Looker to drive major business and technical breakthroughs. Companies like Arrive, Audika, Carousell, Framebridge, GumGum, Intel, Overdose Digital, Ocean Network Express, Subskribe and Promevo are leveraging Looker’s newest AI-driven capabilities, including Conversational Analytics, to transform data to insights and actions, and empower their entire organization with a single source of truth, powered by Looker’s semantic layer.

February 2 - February 6

Join us on March 4 for our webinar, Win Your AI Strategy with Cloud SQL Enterprise Plus, to learn how to power your generative AI workloads with 3x higher performance and 99.99% availability. Register today to discover how to build a scalable, enterprise-grade foundation for your most demanding AI applications.

January 26 - January 30

We introduced Conversational Analytics in BigQuery, which allows users to analyze data using natural language. Conversational Analytics in BigQuery is an intelligent agent that generates, executes and visualizes answers grounded in your business context directly in BigQuery Studio, making data insights for data professionals more conversational.
We outlined how data products have become the foundation for AI agents, providing the context needed to make autonomous agents reliable and trusted for real business use, backed by organized business logic and semantic understanding.
We highlighted how you can supercharge data analytics workflows, and outlined Google Cloud’s AI agent offerings for data engineering, data science, and development tools, so you can integrate agentic workflows in your applications, empower your teams and speed discovery.

January 19 - January 23

We have fundamentally reimagined Firestore with pipeline operations for Enterprise edition. Experience a powerful new engine featuring over a hundred new query features, index-less queries, new index types, and observability tooling to improve query performance. Seamlessly migrate using built-in tools and leverage Firestore’s existing differentiated serverless foundation, virtually unlimited scale, and industry-leading SLA. Join a community of 600K developers to craft expressive applications that maximize the benefits of rich queryability, real-time listen queries, robust offline caching, and cutting-edge AI-assistive coding integrations.
Introducing Google Cloud SQL on MSSQLTips: We are highlighting a new technical guide published on MSSQLTips titled "Introducing Google Cloud SQL." This article serves as an essential resource for SQL Server administrators and developers exploring Google Cloud's fully managed database service. It provides a detailed overview of Cloud SQL capabilities, including high availability, security integration, and the seamless transition of on-premises SQL Server workloads to the cloud, making it an ideal resource for those planning their migration strategy.
We are excited to announce the Public Preview of Microsoft Entra ID (formerly Azure Active Directory) integration with Cloud SQL for SQL Server. Designed to tackle the challenge of identity sprawl in multi-cloud environments, this integration allows organizations to govern database access using their existing Microsoft identity infrastructure. Key benefits include centralized identity management, enhanced security features like Multi-Factor Authentication (MFA), and simplified user administration through direct group mapping. This feature is available for SQL Server 2022 and supports both public and private IP configurations.

January 12 - January 16

Google-built JDBC Driver for BigQuery is now available in Preview
We are excited to announce the launch of the new, Google-built JDBC driver for BigQuery. This new open-source driver provides a direct, high-performance connection for Java applications to BigQuery and is developed entirely in-house by Google. Download a new driver and connect your Java application to BigQuery.
Troubleshoot Airflow tasks instantly with Gemini Cloud Assist investigations: Cloud Composer just got smarter. We are excited to announce that Gemini Cloud Assist investigations are now available directly within Cloud Composer 3. Instead of manually sifting through raw logs, you can now simply click "Investigate" on a failed Airflow task. Gemini analyzes logs and task metadata to identify failure patterns—such as resource exhaustion or timeouts—and provides actionable recommendations driven by Gemini Cloud Assist to resolve the issue. This integration shifts the debugging experience from manual toil to automated root cause analysis, significantly reducing the time required to restore your pipelines. Learn more about AI-assisted troubleshooting.

Scaling Telco Autonomy: Leveraging GNNs with Distributed GraphFlow

Tue, 15 Sep 2026 16:00:00 +0000

The telecommunications industry is currently undergoing a paradigm shift, moving from traditional manual human-driven operations to fully Autonomous Network Operations. Modern networks have grown increasingly complex, heterogeneous, and large-scale, making handcrafted rules-based methods and traditional Machine Learning (ML) approaches alone insufficient to automate network operations. While ML methods can identify subtle patterns and make fine predictions from large amounts of structured data, they lack the ability to understand, reason about the data and the system it represents, and ultimately make the kind of decision a human operator would.

The growth of AI agents and their ability to reason is a promising solution to this shortcoming. However, in the same way a human operator is not capable of directly ingesting the statistical information spread across the billions of data points created in a large network, AI agents also lack the ability to operate at this scale. To address this challenge, telecommunications companies are adopting Graph Neural Networks (GNNs), a modern form of machine learning designed to operate natively on massive volumes of temporal and relational data. By integrating GNNs with AI agents, operators can combine advanced diagnostics such as root cause analysis, capacity planning, traffic forecasting, what-if simulations, and real-time anomaly detection with the reasoning power required to interpret these insights and execute justified actions. This powerful combination enables networks to safely move towards Level 5 Autonomy as defined by TM Forum, where the system operates autonomously.

In this post, we present the three components (Data, ML, and AI) that will power Google Cloud’s Autonomous Network Operations framework.

Google Autonomous Network Operations framework architecture

Foundation: Digital Twin on Spanner Graph

At the heart of Google Cloud’s Autonomous Network Operations framework is the network digital twin: a highly detailed, virtual replica that continuously mirrors its living telecommunications network in real time. Rather than being a static model, it is represented as a dynamic, temporal network graph that captures the evolving state and relations of its components over time. This architectural approach allows operators to "go back" in time to train and evaluate ML models on historical data, while providing AI agents with the foundational operational knowledge required to achieve Level 5 Autonomy. By simulating the impact of proposed network changes within this digital environment, the Digital Twin establishes a critical layer of trust, enabling AI agents to confidently design future states and automatically resolve network issues.

Google Cloud’s Spanner Graph is well suited to host this digital twin:

Scalability and Availability: Spanner Graph provides a no compromise foundation for modern applications, offering virtually unlimited scaling that grows as the network grows, along with 0-RPO/0-RTO and five 9s of availability.
Multi-Model Support: Supports multiple data models (Relational, Graph, Vector, and Full-Text Search) in a single platform allowing developers to build complex compositions such as graph transversals combined with nearest neighbor vector search.
Global Consistency: Spanner provides a globally consistent view of the network, simplifying system development.

The next figure illustrates a network topology with four node types: routers, interfaces (the physical ports), VPNs (L3VPN service instances), and flows (active traffic sessions). These are connected by directed edge types capturing the full network stack: physical containment (router-interface), physical links (interface-interface), control-plane peering (router-router via OSPF/iBGP), service membership (router-VPN), and traffic anchoring (flow-interface, flow-VPN).

High Level network topology

The ML layer: Distributed Graph Flow (DGF)

To predict how a network will behave and react, the digital twin leverages an ML layer powered by Distributed Graph Flow (DGF). By training on the vast volumes of structured historical data hosted within Spanner Graph, this layer uncovers critical predictive insights that enable human operators and AI agents to manage networks proactively rather than reactively.

DGF is a recently open-sourced Python library designed to manage the entire end-to-end lifecycle of GNN modeling. Developed by Google CoreML and Google Research, it brings a decade of internal Google-scale tools and expertise directly to Google Cloud enterprise clients. To accommodate different engineering needs, the library offers high-performance, composable, low-level primitives for advanced teams, alongside a simple API for rapid development that requires no prior GNN expertise.

For instance, training and evaluate a GNN model in GraphFlow with the high level API can be as simple as writing 5 lines of code:

code_block: <ListValue: [StructValue([('code', 'import dgf\r\n\r\n# Fetch the data from Spanner Graph\r\ngraph, schema = dgf.io.read_spanner_graph(...)\r\n\r\n# Train a node attribute prediction model\r\nmodel = dgf.learning.train_node_model(graph, schema, target_column="risk_score")\r\n\r\n# Evaluate the model\r\nmodel.evaluate()\r\n# Make predictions\r\nmodel.predict(graph, seed_node_idxs=[0, 1, 2])\r\n\r\n# Save the model for later\r\nmodel.save("/tmp/model")'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc3b4c2e90>)])]>

The DGF provides high-level concepts that map directly to Autonomous Network Operations requirements:

Use cases

By leveraging DGF and GNNs, telcos can move from reactive maintenance to proactive prevention through several advanced use cases:

Anomaly detection: GNNs generate node and edge embeddings that encapsulate historical patterns and current health. Any anomalous embeddings are flagged for review before they lead to service degradation.
Root cause analysis (RCA): DGF can output specific subgraphs containing only the relevant network instances related to an incident, such as "Attach Failures" in a specific ZIP code. This allows troubleshooting agents to perform high-speed analysis without scanning the entire global network.
Predictive maintenance: The system can predict the likelihood of device failures or edge breaks, such as "handover failures" for fast-moving equipment, enabling proactive load balancing or rerouting. Furthermore, by combining agents, remedial actions can be automated by adopting a ‘human-on-the-loop’/’human-in-the-loop’.
What-if analysis: GNNs enable Telcos to simulate scenarios like fiber cuts, or traffic surges or device configuration changes. By modeling topological dependencies, GNNs can predict how these local changes propagate across the entire network, allowing engineers to test resilience and evaluate mitigation strategies in a risk-free digital environment.

Scenario: Root cause analysis with GNNs and DGF

Once you have created a digital twin (example code), a straight-forward 5-step process can be used to implement Root Cause Analysis(RCA) detection using GNNs and DGF.

Connect to the Digital Twin: Use the DGF Spanner Graph connector (dgf.io.read_spanner_graph) to load the network topology directly from Spanner Graph's Digital Twin into the DGF environment.
Train a Supervised Node (or Edge) Prediction model: Depending on the training data and objective, you will train a supervised node prediction model to predict a target node feature or an edge prediction model to predict an edge between the root cause entity node and the affected entity node. For the given sample data you will use the high-level dgf.learning.train_node_model API to train a supervised node prediction model.
Use the node prediction model to predict root cause node: The node prediction model can be directly used to predict the impact score on the node with the anomaly. Entity nodes affected by the anomaly with highest predicted impact score will be the top candidates for root cause.
Deploy to Gemini Enterprise Agent Platform (formerly Vertex AI): Export the model and host it on a Gemini Enterprise endpoint to enable scalable, low-latency predictions.
Real-time Inference: Make prediction calls to the inference endpoint with the anomaly date as input. The endpoint will return the predicted root cause Entity nodes.

Get started today

The integration of GNN using Distributed Graph Flow into network operations is more than just a technical upgrade; it is a critical evolution for the telco industry. By moving towards a GNN-powered autonomous framework, operators can significantly shorten outage times, optimize capacity in real-time, and ultimately deliver a superior customer experience through improved operational efficiency.

To start building your own intelligent network applications, check out the Distributed GraphFlow (DGF) library, which provides the essential primitives for scalable GNN training and inference. For a hands-on experience, follow our step-by-step code sample. You can also explore our recent award-winning Moonshot project on Business-aware GNN-healing networks, and dive deeper into our approach on self-optimizing autonomous networks by reviewing this whitepaper.

Agent-ready analytics: Unlocking insights with BigQuery augmented analytics

Mon, 14 Sep 2026 16:00:00 +0000

BigQuery now features a suite of augmented analytics Table-Valued Functions (TVFs) designed to automate complex data analysis at scale. Augmented analytics combines AI, ML and statistical methods to automate insight discovery and pattern explanation. These functions allow you to diagnose why metrics changed, uncover underlying trends and relationships across the data, and even isolate the true impact of business decisions.

These TVFs run directly where your data lives, which helps speed up analysis and reduces the need to export data into external tools. In addition, since these functions are compact and yield structured SQL outputs, they can easily be integrated as skills for AI agents, which easily enables automated, conversational data investigation workflows.

We are introducing six new augmented analytics functions in BigQuery, each created to address a specific analytical challenge:

TVF Function	What It Helps You Find	Real World Question It Answers
AI.KEY_DRIVERS	Identifies the top drivers behind an increase or drop in a metric between two time periods or groups.	Why did revenue spike this quarter compared to last quarter?
AI.CAUSAL_EFFECT	Quantifies the impact of an action or event by comparing the observed results to an expected baseline.	How much of the revenue lift came from our pricing update rather than organic growth?
ML.CORRELATION	Evaluates the direction and strength of the relationship between pairs of numeric metrics.	Does increased user session duration correlate with higher lifetime customer value?
ML.DETECT_CHANGE_POINTS	Identifies specific dates or intervals where a metric experiences a shift compared to surrounding patterns.	During which time periods did our platform latency experience persistent, structural shifts?
ML.TREND	Separates the underlying growth or decline from short-term fluctuations or noise.	What are the underlying trends of my revenue over the past year, abstracting away the outlying spikes and drops?
ML.SEASONALITY	Discovers predicable repeated cycles across hours, days, weeks, months or quarters.	Which days of the week consistently experience the highest server load?

As we show in the next section, these functions can be easily chained together. The output of one function, such as a detected time window, can directly parameterize the next analytical step.

A step-by-step example of chaining insights

Consider a case where there is a shift in a metric, and you need to diagnose the underlying cause and measure the business lift.

To diagnose, we can chain ML.DETECT_CHANGE_POINTS, AI.KEY_DRIVERS and AI.CAUSAL_EFFECT using the Austin Bikeshare sample dataset (bigquery-public-data.austin_bikeshare.bikeshare_trips). This dataset contains historical trip volume and demographic data for the city’s bikesharing program.

Step 1: Detect change points

ML.DETECT_CHANGE_POINTS automatically identifies statistically significant structural shifts or level changes in your time-series data. While this example demonstrates the analysis in a single aggregate metric, this function is highly scalable and is capable of running across millions of individual time series.

To find these shifts, we run the following query across the daily baseline:

code_block: <ListValue: [StructValue([('code', "WITH daily_trips AS (\r\n SELECT\r\n TIMESTAMP_TRUNC(start_time, DAY) AS trip_day,\r\n COUNT(*) AS total_trips\r\n FROM `bigquery-public-data.austin_bikeshare.bikeshare_trips`\r\n GROUP BY 1\r\n)\r\nSELECT\r\n begin_timestamp,\r\n end_timestamp,\r\n metrics.avg AS avg_daily_trips,\r\n metrics.min AS min_daily_trips,\r\n metrics.max AS max_daily_trips,\r\n metrics.count AS duration_days\r\nFROM ML.DETECT_CHANGE_POINTS(\r\n (SELECT * FROM daily_trips),\r\n data_col => 'total_trips',\r\n timestamp_col => 'trip_day'\r\n);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc484ef650>)])]>

The output identifies the exact time intervals where the baselines have shifted over the company’s history:

If we look at the raw daily session counts, this aligns with shifts over time. We highlight the two change points with the longest durations below:

The shift in February 2018 aligns with the day the Austin City Council passed the “Dockless Mobility Pilot Program”, to transform the transit ecosystem, integrating shared electric scooters and bikes into the public.

Step 2: Key drivers attribution

We can input the February 2018 slice found directly to AI.KEY_DRIVERS to determine the particular factors (i.e. bike_type, subscriber_type, etc) driving the surge. AI.KEY_DRIVERS can scan through millions of rows of multi-dimensional data in seconds.

We define the interest group as the slice of time after the shift occurs and compare it against the time period before the shift as the reference group.

code_block: <ListValue: [StructValue([('code', "WITH daily_segments AS (\r\n SELECT \r\n start_station_name,\r\n end_station_name,\r\n subscriber_type,\r\n bike_type,\r\n 1 AS trip_count,\r\n -- We use the precise breakpoint identified by Change Points\r\n IF(EXTRACT(DATE FROM start_time) >= '2018-02-11', TRUE, FALSE) AS after_shift\r\n FROM `bigquery-public-data.austin_bikeshare.bikeshare_trips`\r\n -- Equidistant ~30 day window around the event\r\n WHERE start_time BETWEEN '2018-01-12' AND '2018-03-13'\r\n)\r\nSELECT \r\n drivers,\r\n metric_interest,\r\n metric_reference,\r\n difference,\r\n relative_difference,\r\n unexpected_difference,\r\n contribution\r\nFROM AI.KEY_DRIVERS(\r\n (SELECT * FROM daily_segments),\r\n metric_col => 'trip_count',\r\n interest_label_col => 'after_shift',\r\n dimension_cols => ['start_station_name', \r\n 'end_station_name', \r\n 'subscriber_type', \r\n 'bike_type'],\r\n top_k => 10\r\n);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc3bba8190>)])]>

AI.KEY_DRIVERS isolates the top contributing dimension values. Each row contains a segment, which represents a slice of data identified by a specific combination of dimension values (e.g., subscriber_type = 'UT Student' and bike_type = 'classic').

The analysis reveals that the overall trip count increased +374.7% (+40,159 trips) between the reference and interest time windows. The massive growth was overwhelmingly concentrated in U.T. Student Memberships (+7,167.1%) and trips ending at the 21st & Speedway @PCL station (+20,739.1%).

This aligns with Austin Bikeshare’s response to the Dockless Mobility Pilot Program. In early February, the bikeshare program launched a large promotional partnership with the University of Texas that offered free annual memberships to all UT students.

Step 3: Causal effect

While we know what drove the surge and when it started, we need to isolate the true return on investment over organic expectations. AI.CAUSAL_EFFECT can construct an ARIMA_PLUS counterfactual to measure what the volume would have been had the program never launched.

code_block: <ListValue: [StructValue([('code', "WITH daily_trips AS (\r\n SELECT \r\n TIMESTAMP_TRUNC(start_time, DAY) AS trip_day, \r\n COUNT(*) AS total_trips\r\n FROM `bigquery-public-data.austin_bikeshare.bikeshare_trips`\r\n -- Training on the 6-month baseline leading up to the intervention\r\n WHERE start_time BETWEEN '2017-08-11' AND '2018-04-11'\r\n GROUP BY 1\r\n)\r\nSELECT \r\n *\r\nFROM AI.CAUSAL_EFFECT(\r\n (SELECT * FROM daily_trips),\r\n data_col => 'total_trips',\r\n timestamp_col => 'trip_day',\r\n -- We inject the breakpoint found in Step 1 as our intervention\r\n intervention_timestamp => '2018-02-11 00:00:00',\r\n output_time_series => TRUE\r\n);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc3bbab990>)])]>

If we graph the predicted and actual trips per day, we can see the surge compared to the counterfactual.

If we set the output_time_series => FALSE, we can see a summary of the lift

AI.CAUSAL_EFFECT reveals that the program caused a +358% volume surge above organic baseline projections, resulting in an estimated 89,775 incremental trips (with 99.9% probability of causal effect).

Connecting augmented analytics to Conversational Analytics

Conversational Analytics lets you chat with agents about your data using natural language. All new BigQuery augmented analytical functions are now available in Conversational Analytics. Since these TVFs can execute complex analytics at BigQuery-scale in seconds, Conversational Analytics can orchestrate multi-step investigative workflows based on a given prompt. Below we show two examples:

Example 1: Chicago taxi trips

Here is an example using the Chicago Taxi Trips (`bigquery-public-data.chicago_taxi_trips.taxi_trips`).

Prompt: What metric has the strongest correlation with drivers getting tipped? Then run an attribution analysis to tell me which categorical dimensions (like location and payment type) most disproportionately drive that specific metric.

The results here used ML.CORRELATION in combination with AI.KEY_DRIVERS.

Credit card payments serve as the primary positive driver of trip distance, adding +1.65M due to longer travel routes and automated digital tip tracking. Trips originating from O'Hare International Airport (Community Area 76) represent another major positive factor, contributing an additional +1.10M miles among tipped credit card rides. In contrast, cash transactions act as a significant negative driver (-652.96K miles), reflecting that cash is predominantly used for shorter journeys rather than extended airport travel.

Example 2: Iowa liquor dataset

Here is an example using the Iowa liquor dataset (`bigquery-public-data.iowa_liquor_sales.sales`) that uses both ML.TREND in combination with ML.SEASONALITY.

Prompt: Find the historical trend for bottles sold. Then, describe the yearly seasonality patterns.

The results show that liquor sales in Iowa show persistent long-term growth, rising from 1.3–1.5 million bottles in 2012 before stabilizing around 2.6 million in recent years. There are strong seasonal cycles, particularly during October and December as well as May and June. There is a drop in sales around January and February.

The skills for these TVFs are now available at the Google Skills Github repository. The BQ AI/ML skills can be found here.

Take the next step

Documentation:

BigQuery AI/ML support in Conversational Analytics

_{We would like to extend our sincere thanks to Katelin Amann, Shirley Fu, Chaoyi Shen, Haiyang Qi, Zheng Zhang, Xi Cheng and the wider engineering team for their feedback and contributions of this work.}

Announcing Pause/Resume and NVIDIA RTX PRO 6000 Blackwell GPU support in Dataflow

Mon, 14 Sep 2026 16:00:00 +0000

Overview
As enterprises scale their AI and agentic workflows, they require serverless platforms that make data preparation for model training, evaluation, and inference effortless and efficient. Dataflow is a critical component of Google Cloud’s AI stack. It enables our customers to create batch and streaming pipelines that support a variety of analytics and AI use cases.

Today, we’re delivering significant enhancements to Dataflow that directly address your top challenges: maximizing compute efficiency for long-running batch jobs and delivering extra inference power for your most demanding AI workloads. We’re thrilled to announce the general availability of Pause/Resume for Dataflow batch jobs as well as support for G4 VMs powered by NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs. With these features, you can accelerate your AI development lifecycle and optimize your costs.

Recover wasted compute and increase developer productivity with Pause/Resume for Dataflow batch jobs
Dataflow customers frequently run large batch workloads that sometimes run for a few days. When these jobs fail, Dataflow users currently cannot access the data that was already processed before the job failure. Instead, they have to retry the entire job, leading to wasted compute resources and decreased engineering productivity.

In addition to addressing failures from large jobs, Dataflow customers with AI workloads sometimes want to increase the utilization of accelerated compute resources like GPUs and TPUs by dynamically re-allocating them from already running, lower priority Dataflow batch jobs to higher priority workloads like feature engineering and AI inference.

To better support these use cases, we are announcing the GA launch of Pause/Resume for Dataflow batch jobs. Powered by internal Google innovation, this feature enables Dataflow customers to resume their failed long running jobs instead of starting from scratch. It also allows customers to pause and resume their Dataflow batch jobs based on their respective business requirements.

For more details, see manually pause a Dataflow job.

Accelerate AI inference workloads with NVIDIA RTX PRO 6000 GPUs
While Dataflow already supports a wide variety of GPUs and TPUs for accelerating AI inference workloads, we’re taking things a step further by announcing support for G4 VMs powered by NVIDIA RTX PRO 6000 Blackwell GPUs.

The NVIDIA RTX PRO 6000 Blackwell GPU delivers significant performance gains compared to the NVIDIA L4 GPU, bringing 96GB vGPU memory and 1.6 TB/s of bandwidth. This means that you can perform AI inference right within your Dataflow job using up to 70B+ parameter models. You can do this while continuing to take advantage of native Dataflow ML capabilities like RunInference, right fitting and GPU-enabled autoscaling which make it easy for you to onboard and scale your AI inference jobs without having to manage underlying infrastructure or manually deal with hard problems like tuning and autoscaling.

Take the next step
Together, Pause/Resume and RTX PRO 6000 Blackwell GPUs help you optimize your batch job costs while running demanding AI workloads. We’re incredibly excited about Dataflow’s capabilities and the possibilities they unlock for our customers. Get started with Dataflow today and use these features to solve your hardest AI challenges. We cannot wait to see what you build.

Agentic analytics with the Data Agent Kit

Tue, 08 Sep 2026 16:00:00 +0000

Imagine your director sends you a chat message Monday morning: Our average order value dropped 7% in January, but total revenue stayed flat. Why?

If you’re a data practitioner, you know why these types of questions can be tough. They’re totally open ended. There’s not a single root cause dashboard you can open. Was there an error in the web logs? Was a promo code misconfigured? You won’t know until you start digging, and you rarely find the answer in just one place.

Each piece of the answer lives somewhere different in your environment:

Sales history (orders and line items) sits in a data warehouse
Live customer records are in a production PostgreSQL instance
Marketing campaign rules are raw JSON files in an object store

Writing any one of these queries is easy. You’ll write the same one a dozen times, tweaking WHERE clauses or adding subqueries to find the answer. Then you’ll bounce to the next system and start again with a different dialect. Before you know it, you have ten browser tabs open and a whole afternoon gone, all to answer one question.

Data Agent Kit

The Data Agent Kit is built to solve this issue. It is a set of MCP servers and agent skills that helps data developers run data workflows from their IDEs. It’s available both as an extension for VS Code forks (Antigravity IDE, Cursor) and as a plugin for other tools (Antigravity 2.0, Antigravity CLI, Claude Code, Codex), so you don’t need to leave your IDE to get answers.

The Data Agent Kit relies on two core mechanisms:

Model Context Protocol (MCP): an open standard that connects your agent to tools, databases, and remote cloud infrastructure.
Skills: markdown files that augment your agent’s knowledge, teaching it how to interact with your specific stack.

Instead of generating SQL snippets and copy-pasting them into a console, Data Agent Kit lets agents run the queries and read the results on your behalf.

Let’s see what this looks like in practice applied to the average order value scenario. In this setup, the data warehouse is BigQuery, the Postgres instance is Cloud SQL, and the campaign rules sit in Cloud Storage.

Data Agent Kit sample architecture

Finding out what happened

The investigation begins in the IDE’s chat pane with the following natural language prompt to confirm the baseline numbers:

code_block: <ListValue: [StructValue([('code', 'Calculate our monthly average order value from August 2025 through January 2026 using the orders and order items tables in BigQuery.'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc3bfb5c90>)])]>

Checking its work

The agent processes your prompt, invokes relevant skills, and prepares to start querying your data. But before it can execute anything, the IDE pauses to ask for permissions to use the necessary MCP tools (e.g. execute_sql_readonly). You can allow it once for auditing, or select “always allow” to keep the workflow moving. Once approved, the agent sends off the queries.

Invoking skills and BigQuery MCP from chat

Agentic IDEs allow you to inspect the execution trail, which reveals items like each MCP tool call or the raw SQL sent to BigQuery. It’s important to keep an eye on generated code, though reading a query can take much less time than writing one against schemas you’re unfamiliar with.

Breaking down the numbers

The numbers showed that average order value remained around $110 from August to December, but dropped to $103 in January. To find out why, ask the agent to drill down:

code_block: <ListValue: [StructValue([('code', "Break down January's AOV by order type to see what's going on"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc3be7a510>)])]>

The results point to a skewed average instead of a business decline. Online and Offline orders stayed healthy (~$110). A new channel called B2B-Wholesale appeared in January with an AOV of just ~$75. Nothing declined, but the product mix changed.

Crossing into Cloud SQL

You know what led to lower AOV. Next, you need to figure out who the wholesale buyers are. The customer records are stored in a Cloud SQL Postgres operational database, and you can continue in the same chat thread:

code_block: <ListValue: [StructValue([('code', 'Who are these B2B customers? Check our Cloud SQL database for their account details and creation dates.'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc3bb4b750>)])]>

The agent switches to the Cloud SQL MCP and inspects the customers table for you. All 100 wholesale accounts are brand-new business entities created within the last 30 days. None of them existed in December.

Querying operational customer records in Cloud SQL

Dropping into the terminal

A quick glance at the B2B orders in BigQuery shows that 92% applied promo_code = BIGORDER25. You can then ask the agent to track that code back to the campaign files, and it will use the Google Cloud Storage MCP server to access the file.

The marketing campaign shows a 25% discount code led to a huge number of low-priced wholesale orders, which reduced the blended AOV while total revenue remained flat.

In a single chat session, the agent queried analytical data (BigQuery), operational records (Cloud SQL), and unstructured metadata (Cloud Storage) to find the root cause.

Updating the director

Now, you can prompt the agent to return a short executive summary for your director.

Agent-generated executive summary

And voilà! With a few natural language prompts straight from your IDE, you've answered the director's open ended question.

Root cause analysis is only part of the job. The next time this issue occurs, you won't want to run through the same situation. Instead, you can turn this investigation into a reproducible data model.

Build a reproducible pipeline

Ask the agent to turn your ad-hoc analysis into a persistent dbt project:

code_block: <ListValue: [StructValue([('code', 'Build a dbt project that joins our BigQuery staging models with our Cloud SQL customer and pet profile attributes. Add a uniqueness test on order_id and run dbt build.'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc49c87d90>)])]>

From a single prompt, the agent creates a virtual Python environment with dbt-bigquery and writes project models and tests. But then dbt build fails. The uniqueness test catches duplicates on order_id.

Customers can own more than one pet. The first version of the model attached those profiles directly to each order, so an order from a three-pet household became three rows (not unique).

The agent reads its own terminal output and catches the failure. It then rewrites the dbt logic and reruns it until the build passes.

This introduces an important note about agentic workflows. Agents are capable of writing mountains of code - but you'll still need to apply data quality checks to your pipeline (fortunately, an agent can write those too).

The next time leadership asks why average order value moved, you'll have a dbt model ready to answer it.

Wrap up

An agentic IDE keeps you from bouncing between your warehouse, your databases, your object store, and your terminal.

By pairing open standards like MCP and modular (and editable!) agent skills, the Data Agent Kit removes the friction between question and answer. Combing through unfamiliar schemas, translating between dialects, writing the joins you’ve written a hundred times: that becomes the agent’s job. You’re in charge of directing the investigation.

Try it yourself

The Data Agent Kit is in preview and works natively in Antigravity (2.0, CLI, IDE), Claude Code, Codex, Cursor, and other popular tools.

Try the Scenario: walk through the full setup in the Analytics with Data Agent Kit and Antigravity IDE Codelab.
Read the Docs: learn more at the Google Cloud Data Agent extension documentation.
Explore the Plugin: check out the skills and tools in the open-source repository on GitHub.

How Yahoo optimizes resources with flexible VMs in Managed Service for Apache Spark

Fri, 04 Sep 2026 16:00:00 +0000

As a global media and technology company connecting hundreds of millions of users to finance, sports, and entertainment platforms, Yahoo operates a massive data infrastructure where analytics workloads must run continuously at high speed. In deadline-driven data environments, relying on fixed virtual machine (VM) configurations creates a brittle system; if a specific machine shape faces a regional capacity constraint, cluster provisioning in Managed Service for Apache Spark (formerly Dataproc) can experience delays and stall critical data pipelines.

Yahoo utilizes flexible VMs in Managed Service for Apache Spark clusters to automatically absorb these resource fluctuations by defining a ranked list of acceptable VM shapes. This allows the system to dynamically search regional zones and maintain pipeline execution without manual intervention. To search for capacity across a region, teams must also enable Auto-Zone placement.

This optimization builds on Yahoo's broader data modernization journey, which involved migrating on-premises Hadoop and big data estates directly to Google Cloud. By transitioning those legacy workloads, the team established a cloud foundation capable of running high-scale batch and streaming analytics with dynamic resource flexibility.

This post provides a technical blueprint for configuring flexible VM instance rankings in Managed Service for Apache Spark to automatically manage capacity constraints and maintain pipeline execution.

Operational trade-offs of static configurations

Configuring clusters with a single, fixed machine type in a specific zone introduces constraints when regional zonal capacity fluctuations occur, potentially impacting cluster provisioning. Rather than manage these capacity variations through custom retry logic or manual intervention, using flexible configurations allows your infrastructure to automatically adapt. By accepting multiple VM shapes and searching across zones in the selected region, flexible configurations help streamline provisioning to better support high-scale analytics workloads.

Rules for configuring flexible clusters

Deploying flexible configurations requires aligning several connected design choices:

Enable auto-zone placement: You must pass a region(--region=${REGION}) or an empty zone string (--zone="") so Managed Spark can search for available capacity across the entire region.
Maintain core and memory symmetry: If your Managed Spark cluster uses autoscaling, all machine types in your flexible list must share a similar core count and memory size, even if they come from different VM families. A uniform CPU-to-memory ratio across primary and secondary workers prevents performance degradation, as the smallest ratio determines your effective container sizing.
Align component properties: Managed Spark calculates system properties based on VM cores and memory. When mixing machine shapes, you may need explicit property overrides to keep YARN and Spark resource allocations aligned with your expected worker behavior.

Two ways flexible VMs support massive workloads

For large-scale data environments, flexible configurations support operations in two ways:

Higher cluster creation success: Instead of failing when a preferred VM type is out of stock, Managed Spark selects from a ranked list to keep provisioning moving.
Better regional resource use: Auto-zone placement searches the entire region to find capacity, which reduces provisioning friction during high-demand periods.

gcloud example

code_block: <ListValue: [StructValue([('code', 'gcloud dataproc clusters create analytics-cluster \\\r\n --region=us-central1 \\\r\n --zone="" \\\r\n --num-workers=10 \\\r\n --master-instance-selection=\'{"machineTypes":["e2-standard-8"],"rank":0}\' \\\r\n --master-instance-selection=\'{"machineTypes":["n2-standard-8"],"rank":1}\' \\\r\n --worker-instance-selection=\'{"machineTypes":["e2-standard-8"],"rank":0}\' \\\r\n --worker-instance-selection=\'{"machineTypes":["n2-standard-8"],"rank":1}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc48141f10>)])]>

API example

You can also build this capacity policy into your automated pipelines or Managed Service for Apache Airflow DAGS using the instanceFlexibilityPolicy field in the ‘Dataproc’ API:

code_block: <ListValue: [StructValue([('code', '{\r\n "projectId": "PROJECT_ID",\r\n "clusterName": "analytics-cluster",\r\n "config": {\r\n "gceClusterConfig": {\r\n "zoneUri": ""\r\n },\r\n "secondaryWorkerConfig": {\r\n "numInstances": 8,\r\n "instanceFlexibilityPolicy": {\r\n "instanceSelectionList": [\r\n {\r\n "machineTypes": ["n2-standard-8"],\r\n "rank": 0\r\n },\r\n {\r\n "machineTypes": ["e2-standard-8", "t2d-standard-8"],\r\n "rank": 1\r\n }\r\n ]\r\n }\r\n }\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc3bf1bc50>)])]>

This API policy achieves the same goal: it establishes your preferred shape, documents valid fallbacks, and lets Managed Spark resolve resource constraints without breaking your automation scripts.

Establishing an infrastructure policy

Managing data at this scale requires standardizing a clear resource policy rather than relying on a single rigid machine type. Your configuration standards should outline:

Preferred and fallback VM families for secondary workers.
Default auto-zone placement to enable flexible provisioning.
Identical core and memory configurations when using autoscaling.
Uniform CPU-to-memory ratios across all worker groups to maintain predictable container sizing.
Explicit YARN or Spark property overrides to guarantee consistent runtime behavior across different machine lines.
Shuffle-safe patterns for Spark workloads running on Spot or highly elastic capacity.

By adopting flexible configurations, you turn infrastructure scarcity into a predictable fallback plan, keeping your critical data pipelines up and running.

Yahoo impact and results

By implementing flexible VMs in Managed Service for Apache Spark, Yahoo successfully reduced cluster provisioning failures by 85% which were caused by regional capacity stockouts. This flexible configuration allows their data infrastructure to automatically handle capacity constraints and successfully provision resources without requiring manual intervention. As a result, Yahoo ensures continuous workload execution and prevents downstream processing delays across their massive data pipelines.

"Managing high-scale data analytics at Yahoo requires resilient, automated infrastructure. Moving to flexible VMs in Managed Service for Apache Spark has transformed our approach; instead of stalling when a specific machine shape faces capacity constraints, our clusters now automatically pivot to our ranked fallback options. This has helped us reduce provisioning failures by 85%, providing the reliability we need to keep our global media platforms running smoothly." - Akshay Jain, Senior Software Developer Engineer, Yahoo!

Strategic benefits of flexible infrastructure

Adopting a flexible compute stack transforms your environment into a dynamic pool of resources that adapts to your operational needs. By moving away from rigid, single-machine type configurations, you ensure that your workloads reliably access the compute they need, regardless of supply fluctuations. This shift not only maximizes workload obtainability and reliability but also facilitates seamless hardware modernization by allowing you to prioritize newer VM generations while maintaining older types as reliable fallback options.

Build your resilient data pipeline

Transitioning to a fluid compute strategy ensures your critical analytics remain operational despite regional resource shifts. Here is how you can begin optimizing your infrastructure today:

Audit your workloads: Identify applications tightly coupled to specific VM families or zones and map out viable alternative hardware shapes.
Standardize resource policies: Explore the documentation for Managed Spark flexible VMs to establish your preferred and fallback VM families.
Align financial strategy: Utilize Flexible Committed Use Discounts (Flex CUDs) to maintain cost predictability when workloads dynamically pivot to alternative machine types.
Claim your credits: New customers may be eligible for $300 in credits to try Managed Service for Apache Spark and other Google Cloud products at no cost.

Simplify pipelines with new BigQuery identity columns

Wed, 02 Sep 2026 16:00:00 +0000

To further empower our customers in their data journey, we are excited to announce the launch of identity columns in BigQuery. This new feature allows users to define columns that automatically generate sequential 64-bit integer values, simplifying the way you manage unique identifiers within your tables.

Data engineers are always looking for ways to make data ingestion smoother and more reliable. BigQuery identity columns offer a powerful, built-in mechanism to automatically generate unique numerical values for your tables. By shifting the responsibility of ID generation to BigQuery, you can significantly reduce the complexity of your data pipelines and focus on delivering insights.

Key benefits for your data pipelines

Implementing identity columns provides several advantages that help streamline the development and maintenance of your data architecture.

Streamlined ingestion: You can now ingest data without needing to pre-calculate unique keys in your application logic or ETL tools.
Reduced boilerplate: By using auto-generated sequences, your SQL code becomes cleaner and easier to maintain, as the database handles key management natively.
Integrated automation: Identity columns work harmoniously with standard DML operations, ensuring that every new row receives a unique identifier automatically.
Flexible integration: Whether you are using INSERT or MERGE statements, identity columns adapt to your existing workflow.

How to implement identity columns

Setting up an identity column is simple and can be done directly within your CREATE TABLE statement. You have two primary ways to define how these values are handled.

Definition options

Clause	Description
`GENERATED ALWAYS AS IDENTITY`	BigQuery automatically manages and ensures the uniqueness of the values.
`GENERATED BY DEFAULT AS IDENTITY`	Provides an automatic value but still allows for manual overrides when necessary.

Example usage
The following SQL statement demonstrates how to create a table that automatically increments IDs, starting at 1 and increasing by one for each new entry.

code_block: <ListValue: [StructValue([('code', "CREATE TABLE my_project.my_dataset.orders (\r\n order_id INT64 GENERATED ALWAYS AS IDENTITY (START WITH 1 INCREMENT BY 1),\r\n customer_name STRING,\r\n order_date DATE\r\n);\r\n\r\n-- Ingesting data is now simpler:\r\nINSERT INTO my_project.my_dataset.orders (customer_name, order_date)\r\nVALUES ('Joe Doe', CURRENT_DATE());"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc3bba8e50>)])]>

Get started today

Identity columns represent our ongoing commitment to providing a flexible, high-performance, and standards-compliant data platform. By automating the generation of surrogate keys, we are making it easier for you to build scalable and maintainable data architecture.

To learn more about how to implement this feature in your projects, please visit the BigQuery identity columns documentation.

Introducing TabFM in BigQuery: Predictive analytics reimagined

Tue, 01 Sep 2026 16:00:00 +0000

Historically, enterprise predictive analytics tasks such as predicting churn, purchase intent, or fraud scoring have meant building custom models using libraries like XGBoost, Random Forest, or Deep Neural Networks (DNNs). While effective, the traditional train-tune-deploy-retrain cycle can be complex and time-consuming. Additionally, the overhead of manual feature engineering, hyperparameter tuning, lengthy and expensive training, and the need for specialized data science skills can lead businesses to underutilize predictive models in their decision-making.

Today, we are announcing the TabFM model in BigQuery. Developed by Google Research, TabFM is a state-of-the-art, pre-trained foundation model for regression and classification on tabular data. It leverages in-context learning (ICL) to deliver highly accurate predictions on your tabular datasets instantly via a single SQL statement, removing the separate training and deployment steps. TabFM on BigQuery is currently in preview.

Here is what TabFM brings to your BigQuery analytics:

Zero-shot predictions: Skip model training, tuning, and artifact deployment. Simply pass your labeled historical data and new prediction tables into a single SQL function to get instant, high-quality predictions.
Predictive ML for your agentic applications: Building an agent for your business use? Add predictive powers to it with TabFM plus BigQuery MCP server. No runtimes or infrastructure to manage, just data in and predictions out.
State-of-the-art accuracy: Outperforms custom-trained, out-of-the-box traditional models on complex datasets, achieving superior accuracy scores on industry benchmarks.
Simple developer experience: Runs natively in BigQuery and is accessible via simple SQL syntax. Automatically handles featurization tasks such as missing values, categorical encoding, etc., with no complex feature engineering pipelines to manage.
Scalability: Processes massive inference tables (up to millions of rows) in minutes using BigQuery’s distributed inference architecture.

The leading model for tabular predictions

Google’s TabFM delivers industry-leading accuracy across a wide range of tabular data. In evaluations on the TabArena benchmark, TabFM consistently outperforms both classic machine learning models and other tabular foundation models.

ELO ratings (↑) for the top 10 models across TabArena classification (upper) and regression (lower). (D) = default; (T+E) = tuned + ensemble. Higher scores denote superior performance.

Learn more about the TabFM model here.

Getting started with TabFM in BigQuery

Using TabFM is straightforward. It is exposed directly through new, built-in SQL functions: AI.PREDICT and AI.EVALUATE.

1. Get instant predictions with AI.PREDICT
To make predictions, you write a single query that passes your training data and prediction data. The model automatically infers whether the task is a classification or regression problem based on the data type of your target label.

code_block: <ListValue: [StructValue([('code', "-- Classifying transactions as fraudulent or not\r\nSELECT *\r\nFROM AI.PREDICT(\r\n TABLE `my_project.my_dataset.historical_transactions`, -- Training data (in-context examples)\r\n TABLE `my_project.my_dataset.new_transactions`, -- Prediction data\r\n label_col => 'is_fraud'-- Target column to predict\r\n);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc3b919610>)])]>

In this example, the output contains all original columns from your prediction table plus predicted label and probability columns (e.g. predicted_is_fraud). No manual feature engineering or model creation was required.

2. Evaluate models with AI.EVALUATE
You can quickly check prediction performance against a test set using the AI.EVALUATE function. This allows you to generate standard evaluation metrics in a single step.

code_block: <ListValue: [StructValue([('code', "-- Regression Evaluation for Customer Lifetime Value (LTV)\r\nSELECT *\r\nFROM AI.EVALUATE(\r\n TABLE `my_project.my_dataset.historical_customer_ltv`,\r\n TABLE `my_project.my_dataset.test_customer_ltv`,\r\n label_col => 'ltv'\r\n);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc3b860c50>)])]>

AI.EVALUATE returns a robust set of metrics such as r2_score, mean_absolute_error etc. for regression problems and metrics such as precision, recall, and f1 for classification problems.

TabFM in BigQuery under the hood

Traditional machine learning requires fitting model parameters to a training dataset. TabFM, in contrast, uses in-context learning. Similar to how large language models (LLMs) learn a task from few-shot examples in a prompt, TabFM reads your training table as in-context examples and generates predictions for your target table in a single forward pass.

To handle the computational complexity and memory footprint of tabular foundation models, BigQuery performs distributed, parallelized inference on your data. Further, to optimize performance and resource utilization, it uses intelligent training-data sampling as well as distributed execution. This allows BigQuery to handle large input rows for training data while executing predictions quickly and efficiently across millions of rows of inference data.

Choosing the right tool for the job

TabFM introduces groundbreaking zero-shot capabilities to BigQuery, and complements existing offerings such as XGBoost models. Here’s how to choose between TabFM and other models:

Use TabFM when you need rapid, high-quality predictive insights without machine learning expertise, when historical datasets are small-to-medium sized, when data changes frequently, and when you need to retrain your models frequently to maintain accuracy. It is also a great fit for conversational or agentic workflows where you need predictive analysis on demand.
Use traditional models like XGBoost when you have very large historical datasets, require complete control over custom hyperparameter tuning, have a high number of features that exceed current limits of TabFM, or need feature-importance explainability, i.e., which of the input features contributed most to the prediction.

Predictive machine learning made easy

With TabFM natively integrated into BigQuery, predictive ML is now as easy as running a standard SELECT query. By eliminating the manual overhead of model training, tuning, and management, TabFM lets developers, data scientists and analysts go from raw data to rich predictive insights in seconds.

To get started today, check out the public documentation. For questions or feedback reach out to our team at bqml_feedback@google.com. We look forward to seeing what you build!

BigQuery Graph is now GA: the knowledge foundation for the agentic era

Mon, 31 Aug 2026 23:00:00 +0000

Many of the questions that matter in enterprise data aren't just about individual rows — they're about how things connect: how two accounts are linked, what path a payment took, what context grounds an AI agent's answer. That’s what a graph is built to solve. Historically, unlocking these insights meant extracting data into standalone graph databases, creating silos and operational overhead. To remove these barriers, we brought native graph capabilities directly to the data warehouse. Today, we are announcing the general availability of BigQuery Graph.

We introduced BigQuery Graph in preview to unify graph and relational analytics. ISO-standard Graph Query Language (GQL) sits alongside SQL, traversals run natively, and there’s no ETL. And because it’s built on BigQuery, BigQuery Graph inherits and expands its capabilities: It reaches petabyte-scale without the memory bottlenecks of a scale-up database, runs under your existing row- and column-level security, and calls BigQuery ML and AI functions in the same query. One engine, two jobs — large-scale graph analytics, and connected context for AI agents.

"BigQuery Graph has been a game-changer for our threat detection pipeline, allowing us to move beyond simple, siloed alerts. By modeling our security signal data as a property graph, we can now perform complex, multi-hop traversals in seconds - something that was previously computationally prohibitive. This graph-centric approach automatically clusters anomalies into coherent attack stories, which, combined with the seamless integration of Gemini models, helps us generate actionable threat narratives. We look forward to integrating native BigQuery Graph algorithms to further streamline our workflows." - Pete Rubio, VP of Global engineering at Thales Cybersecurity Products

Since preview, we saw data teams across industries adopt BigQuery Graph for both analytical and agentic workflows:

Threat and fraud detection: Security and financial organizations correlate signals across event logs to uncover multi-hop attack paths, fraud networks, and suspicious transaction loops.
Supply chain digital twins: Manufacturing and logistics organizations map dependencies across suppliers, parts, and distribution routes to simulate disruptions and optimize fulfillment.
Identity resolution and Customer 360: Ad-tech and retail platforms stitch fragmented user identifiers and behavioral touchpoints into unified customer profiles across channels.
Knowledge graphs and AI agent grounding: Enterprise AI teams build structured knowledge graphs from unstructured documents, providing domain context to ground Gemini models and GraphRAG workflows.
Network lineage and infrastructure management: Telecommunications and enterprise IT teams track complex network topologies, service dependencies, and data lineage across multi-hop paths.

What’s new in BigQuery Graph

Reaching GA is more than a stability milestone. The work fell into two movements: we made the graph engine itself faster and broader, and we built an agentic ecosystem around it — so agents can build a graph, chat with it, and keep an auditable memory on it. Some of what follows is generally available today; some is in preview or rolling out over the coming weeks.

A faster, broader graph engine

“Advertising has spent decades optimizing individual events; the agentic era will optimize the relationships between them. At Yahoo, BigQuery Graph gives our AI agents connected context - campaigns, audiences, exposures, and outcomes, traversable with standard GQL right where our monetization data already lives, with no separate graph engine and no data movement. Our agents don't just read the graph; they reason over it and write their conclusions back as new relationships. That's how monetization moves beyond automation, to autonomous systems we can trust to act.” - Mikul Bhatt, Director of Engineering, Monetization Platform at Yahoo

Borderless graph Lakehouse

Agents are only as good as the context they can reason over, and that context is rarely in one place. With borderless Lakehouse, a single BigQuery Graph can span native BigQuery tables and open Iceberg tables in other clouds — through Databricks Unity Catalog, AWS Glue, or Snowflake — traversed in place, without copying data or building ETL pipelines.

Say a support agent needs to answer, "who supplies the product behind this customer's delayed order, and where are they based?" The customer data sits in an Iceberg lakehouse on Google Cloud, the product and supplier records in a Databricks catalog on AWS. Instead of stitching the sources together per request, the agent traverses one virtual knowledge graph that already connects them — over data that never moved.

Figure 1: A diagram illustrating a virtual knowledge graph spanning across Google Cloud (blue nodes), AWS (yellow nodes), and other clouds (green nodes) without data movement.

The following DDL statement shows how you can define this virtual graph, mapping your node and edge tables directly across both cloud environments:

code_block: <ListValue: [StructValue([('code', '-- A virtual knowledge graph spanning two clouds - no data movement\r\nCREATE OR REPLACE PROPERTY GRAPH `my_project.retail.virtual_kg`\r\n NODE TABLES (\r\n -- Google Cloud\r\n `my_project.gcs_lake.retail.customers` AS Customer KEY (customer_id),\r\n -- AWS\r\n `my_project.dbx_fed_catalog.retail.products` AS Product KEY (product_id),\r\n `my_project.dbx_fed_catalog.retail.suppliers` AS Supplier KEY (supplier_id)\r\n )\r\n EDGE TABLES (\r\n `my_project.gcs_lake.retail.purchases` AS Bought KEY (purchase_id)\r\n SOURCE KEY (customer_id) REFERENCES Customer (customer_id)\r\n DESTINATION KEY (product_id) REFERENCES Product (product_id),\r\n `my_project.dbx_fed_catalog.retail.products` AS Supplied_By KEY (product_id)\r\n SOURCE KEY (product_id) REFERENCES Product (product_id)\r\n DESTINATION KEY (supplier_id) REFERENCES Supplier (supplier_id)\r\n );'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc3b862150>)])]>

With that, the agent gets a grounded, multi-hop answer assembled across two clouds in a single traversal:

code_block: <ListValue: [StructValue([('code', "-- Agent grounding: trace a customer to the supplier behind their product, across clouds\r\nGRAPH `my_project.retail.virtual_kg`\r\nMATCH (c:Customer {customer_id: 'C1'})-[:Bought]->\r\n (:Product)-[:Supplied_By]->(s:Supplier)\r\nRETURN s.name AS supplier, s.country AS supplier_country"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc4a25f510>)])]>

Faster and more expressive GQL

BigQuery Graph is built for questions about connection: how two accounts are linked, what path a payment took, which entities sit within a few hops of a flagged one. These are the questions SQL joins struggle to express, and they're where a graph engine earns its place. At GA, we've made them both faster to run and easier to write:

Faster execution. GA optimizes path-finding for acyclic and undirected traversals: against public benchmarks, GQL is 2x faster since preview and undirected traversal 100x, with faster, more resource-efficient cycle detection in ACYCLIC and TRAIL path modes. Lower query latency keeps the neighborhood and path lookups that ground an agent's answer responsive under frequent, interactive access.
More expressive queries. With the new CALL statement and extended subquery support, you can run a graph subquery for each entity in a result, or invoke a reusable named function, so a complex question breaks into parts instead of one sprawling pattern. The same functions an analyst writes become the building blocks an agent calls as a tool.

Built for the agentic era

“Companies have plenty of workforce data, but very little shared understanding of what their people can do or where they fit. BigQuery Graph lets us turn that scattered information into a reusable property graph and traverse connections across people, roles, capabilities, and evidence at scale, so the same connected workforce context can support thousands of decisions instead of being recreated one decision at a time. That gives AI a stronger foundation for much harder questions about how work should get done.” - Heiko Roth, Founder & CEO, Workerbee

Chat with your graphs

You don't have to write GQL to explore a graph. BigQuery conversational analytics lets you chat with your graph directly in natural language: it reads the relationships in your schema to translate a question into SQL or GQL, and visualizes the traversal for path-based answers. The agent draws on graph metadata like descriptions and synonyms to keep results grounded — the relationships that make a graph a graph are exactly what cut the ambiguity and hallucination that plague free-form natural language querying. You can also connect Gemini Enterprise to BigQuery Graph through an MCP server, or publish the conversational data agent to it directly.

Build a graph with an agent

Standing up a graph — modeling tables into nodes and edges, then writing GQL against them — is work you can hand to the data agent you already use. We've packaged BigQuery Graph expertise into an agent skill that makes your agent fluent in graph: GQL pattern matching, blending graph and SQL, and schema design that follows our recommended practices. The capabilities are accessible out of the box in your preferred agentic coding tool, such as Antigravity, Visual Studio Code, Claude Code, and Codex, with the Google Cloud Data Agent Kit extension.

The skill is also learning to author, not just advise — a capability rolling out soon. Point it at a dataset, a model document, or an ER diagram and it proposes the nodes and edges, then verifies each relationship against your data before building, showing you the match rates: this one resolves at, say, 98%, that one 56%. You get a graph you can trust from day one.

Give your agents an auditable memory

Grounding an agent is half the job; the other half is remembering what it did. As agents move from advising to acting, every decision has to be explainable after the fact — which option was chosen, which policy applied, which alternatives were rejected. With context graph in BigQuery Agent Analytics, each action an agent takes is captured and shaped into a context graph: a typed, queryable trace of the agent's reasoning, stored right in BigQuery Graph. Because the trace is itself a graph, "why did the agent do this?" is a single traversal — and the outcomes you join back to those decisions become the data that improves the next one.

Get started with BigQuery Graph today

BigQuery Graph runs graph analytics and grounds AI agents on your data, across clouds. To get started, check out the overview and data model to see how GQL, node tables, and edge tables fit together, then put them to work on your team’s common patterns. Trace suspicious money movement and synthetic identities in the fraud detection codelab, stitch fragmented emails, devices, and cookies into one customer in the identity resolution codelab, or model a supply chain as a digital twin you can query for hidden dependencies when disruption hits.

From there, take it toward agents. The agent context graph codelab turns raw event logs into a graph that audits, explains, and traces what your autonomous agents actually did — the connected memory behind a system you can trust to act. If your workloads span both real-time operational transactions and massive-scale analytics, explore our unified graph solution to see how Spanner Graph and BigQuery Graph work together. And when you are ready to go deeper — our ebook walks the journey end-to-end.

From weeks to minutes: The new agentic era of data pipelines

Mon, 31 Aug 2026 16:00:00 +0000

Data pipelines are the backbone of the modern enterprise, yet a barrier to entry exists for orchestrating them, making this critical capability unavailable to many data professionals. Following our announcements at Google Cloud NEXT ’26, where we introduced the Orchestration Pipelines framework, we are fundamentally changing this dynamic.

To bring this powerful framework directly to practitioners, we offer the Data Agent Kit — a unified, freely available, and open-source collection of data engineering and data science tools that integrate directly into your preferred IDE or CLI (such as VS Code, Claude Code, or Codex).

The Data Agent Kit seamlessly embeds the Orchestration Pipelines framework into your workflow in two distinct ways. First, it provides a dedicated Data Engineering tab for comprehensive pipeline management. Second, it includes a specialized agentic skill designed to author, deploy, and troubleshoot production-grade Apache Airflow® DAGs using natural language.

By pairing these specialized agent skills with a declarative YAML DSL, all data personas — from analysts to ML engineers — can bypass complex Python Airflow boilerplate. This framework decouples high-level orchestration logic from underlying compute execution, democratizing access to powerful MLOps capabilities across your entire data organization.

In this post, we will walk through an exemplary MLOps use case to demonstrate how easily this can be achieved.

Setting up your environment

Before authoring your first Orchestration Pipeline, you need to set up your local development environment. Getting started takes less than two minutes.

1. Install and configure the extension

To install the extension in your preferred IDE or CLI — such as VS Code, VS Code forks, Antigravity, Claude Code, Antigravity CLI, or Codex — and authenticate it with your Google Cloud account, follow the step-by-step setup guide in the official documentation: Google Cloud Data Agent Kit installation guide

2. Verify orchestration pipeline skills

Once installed, verify that the required agent skills are active:

Open the ‘Google Cloud Data Agent Kit’ panel on the VS Code activity bar.
Navigate to ‘Settings’ then ‘Skills’.
Ensure the ‘gcp-pipelines-orchestration’ skill is enabled.

This skill provides the agent with deep contextual knowledge of pipeline syntax, variable substitution, secret management, and automated incident diagnosis for Airflow runs.

3. Building your first pipeline

To start authoring, building, and validating orchestration pipelines directly inside the any VS Code compatible IDE using natural language prompts, follow the official building guide: Build pipelines guide

An example business problem: Proactive supply chain management

Let’s walk through an example business problem. In the logistics and retail sector, customer satisfaction hinges on accurate delivery estimates. When an order is delayed without warning, customer churn can spike and support costs can escalate.

To address this, we are building an end-to-end MLOps architecture that predicts the exact transit time (in days) based on warehouse location, customer location, and order characteristics. By predicting these delays before shipping, operations teams can proactively notify customers or automatically upgrade shipping tiers before Service Level Agreements (SLAs) are breached.

To make this architecture fully reproducible, we use the bigquery-public-data.thelook_ecommerce public dataset in BigQuery. For demo purposes, we split this static dataset into training and inference sets. In a real-life scenario, inference would be performed on new, incoming data. This dataset provides authentic operational complexity:

Geographical data: Latitude and longitude for both customer addresses (users) and distribution centers (distribution_centers).
Temporal data: Granular order lifecycle timestamps (created_at, shipped_at, delivered_at).
Order attributes: Product categories, pricing, and fulfillment status (orders, order_items).

By combining this dataset with BigQuery, Managed Service for Apache Spark serverless, Gemini Enterprise Agent Platform, and dbt, we will demonstrate how to build an automated, self-healing MLOps loop that handles training, daily batch inference, and model drift evaluation.

The agentic workflow: From prompt to pipeline in minutes

With the extension configured, we can bypass boilerplate Python for DAG authoring entirely. Inside VS Code, we opened the Data Agent Kit chat and provided a single natural language prompt to define our continuous MLOps feedback loop:

Note: The detailed prompt was crafted with repeatability in mind specifically for this blog post. In real-life scenarios, you can achieve the same result in a more conversational way, pipeline by pipeline. The complete prompt and all generated files are available in the Orchestration-pipelines GitHub repository.

Note: While frontier models equipped with the Orchestration Pipelines skill can often scaffold complete workflows in a single step, LLM responses naturally vary based on model versions, workspace context, and token depth. If a specific parameter, dataset path, or dependency is omitted in the initial pass, simply provide a short follow-up prompt.

Within minutes, the Data Agent Kit generated the underlying PySpark scripts, dbt configurations, and the three declarative YAML pipelines.

Please find below the generated YAML pipelines and a visual diagram of them. This pipeline is a simplified example designed to showcase Orchestration Pipelines capabilities. In practice, recommended production MLOps setups will vary depending on your specific use cases and operational needs.

Pipeline 1: The training engine
This pipeline serves as our heavy-compute engine. The agent generated a YAML definition that first queries BigQuery to extract historical completed orders. It then dynamically provisions a Managed Spark serverless cluster to calculate geographical distances and train a model for production use. Finally, it pushes the trained model to Gemini Enterprise Agent Platform Model Registry.

code_block: <ListValue: [StructValue([('code', 'modelVersion: "1.0"\r\npipelineId: "training-pipeline"\r\nrunner: airflow\r\nowner: "mlops"\r\ntags:\r\n - "job:datacloud:antigravity"\r\ndefaults:\r\n projectId: "your-project-id"\r\n location: "us-central1"\r\n executionConfig:\r\n retries: 0\r\n\r\nactions:\r\n - sql:\r\n name: "extract_training_data"\r\n engine:\r\n bigquery:\r\n location: "US"\r\n destinationTable: "your-project-id.mlops.training_dataset"\r\n query:\r\n path: "blogpostdemo/training_query.sql"\r\n\r\n - pyspark:\r\n name: "train_model_dataproc"\r\n dependsOn:\r\n - "extract_training_data"\r\n engine:\r\n dataprocServerless:\r\n location: "us-central1"\r\n resourceProfile:\r\n inline:\r\n runtimeConfig:\r\n version: "2.3"\r\n properties:\r\n "spark.dataproc.driverEnv.PYTHONPATH": "./libs/lib/python3.11/site-packages"\r\n "spark.executorEnv.PYTHONPATH": "./libs/lib/python3.11/site-packages"\r\n mainFilePath: "blogpostdemo/train_model.py"\r\n environment:\r\n requirements:\r\n inline:\r\n list:\r\n - "tensorflow==2.14.1"\r\n - "numpy<2.0.0"\r\n - "protobuf<5.0.0dev"\r\n - "google-cloud-storage"\r\n\r\n - ai:\r\n name: "upload_model_vertex"\r\n dependsOn:\r\n - "train_model_dataproc"\r\n agentPlatform:\r\n projectId: "your-project-id"\r\n location: "us-central1"\r\n modelUpload:\r\n modelName: "transit_days_predictor"\r\n modelArtifactUri: "gs://your-bucket-name/models/tf_transit_days_model"\r\n servingContainerImageUri: "us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-14:latest"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc3bfb6690>)])]>

Pipeline 2: Daily inference
For our daily operational workflow, this lightweight pipeline applies the trained model to all currently in-transit orders. It queries the dataset via BigQuery job, executes inference job via Gemini Enterprise Agent Platform, and writes the results back to a BigQuery table to flag potential SLA breaches for the customer support team.

code_block: <ListValue: [StructValue([('code', 'modelVersion: "1.0"\r\npipelineId: "inference-pipeline"\r\nrunner: airflow\r\nowner: "mlops"\r\ntags:\r\n - "job:datacloud:antigravity"\r\ndefaults:\r\n projectId: "your-project-id"\r\n location: "us-central1"\r\n executionConfig:\r\n retries: 0\r\n\r\nactions:\r\n - sql:\r\n name: "extract_inference_data"\r\n engine:\r\n bigquery:\r\n location: "US"\r\n destinationTable: "your-project-id.mlops.inference_dataset"\r\n query:\r\n path: "blogpostdemo/inference_query.sql"\r\n\r\n - ai:\r\n name: "run_vertex_batch_prediction"\r\n dependsOn:\r\n - "extract_inference_data"\r\n agentPlatform:\r\n projectId: "your-project-id"\r\n location: "us-central1"\r\n batchInference:\r\n jobDisplayName: "inference_job"\r\n modelName: "projects/your-project-id/locations/us-central1/models/your-model-id"\r\n bigquerySource: "bq://your-project-id.mlops.inference_dataset"\r\n bigqueryDestinationPrefix: "bq://your-project-id.mlops"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc484ee0d0>)])]>

Pipeline 3: Automated evaluation and branching
The daily evaluation pipeline acts as our automated quality gate. It triggers dbt models to join our predictions with actual delivery timestamps, calculating absolute errors and SLA breaches.

Using built-in logic, the pipeline automatically evaluates these metrics. If the model’s error rate exceeds our acceptable threshold, it conditionally triggers the ‘training-pipeline’ to generate a fresh model.

code_block: <ListValue: [StructValue([('code', 'modelVersion: "1.0"\r\npipelineId: "evaluation-pipeline"\r\nrunner: airflow\r\nowner: "mlops"\r\ntags:\r\n - "job:datacloud:antigravity"\r\ndefaults:\r\n projectId: "your-project-id"\r\n location: "us-central1"\r\n executionConfig:\r\n retries: 0\r\n\r\nactions:\r\n - pipeline:\r\n name: "run_dbt_models"\r\n framework:\r\n dbt:\r\n airflowWorker:\r\n projectDirectoryPath: "blogpostdemo/dbt_project"\r\n\r\n - python:\r\n name: "check_retraining_condition"\r\n dependsOn:\r\n - "run_dbt_models"\r\n mainFilePath: "blogpostdemo/evaluate_drift.py"\r\n pythonCallable: "check_drift"\r\n engine:\r\n local: {}\r\n\r\n - orchestrationPipeline:\r\n name: "trigger_retraining_pipeline"\r\n dependsOn:\r\n - "check_retraining_condition"\r\n pipelineId: "training-pipeline"\r\n bundleId: "my-first-bundle"\r\n waitForCompletion: false'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc49a90810>)])]>

Automated deployment to Managed Service for Apache Airflow

Authoring pipeline logic is only half the battle; deploying it securely and reliably to production is where data teams historically lose valuable time.

With Orchestration Pipelines, deployment is streamlined through standard CI/CD practices. Rather than manually writing deployment scripts or configuring complex environment boundaries, the Data Agent Kit automatically generates the necessary continuous integration workflows (such as GitHub Actions) for your workspace.

This means you can simply click commit, and the framework will seamlessly package and deploy your Orchestration Pipeline bundle directly to your Managed Airflow environment.

For a comprehensive guide on integrating these automated workflows into your existing CI/CD pipelines, review the official guide: Deploying Orchestration Pipelines.

Day-two operations: Monitoring and agentic troubleshooting

Maintaining these pipelines is just as intuitive as building them. By bringing the orchestration control plane directly into your IDE, the Data Agent Kit provides real-time monitoring of your Managed Airflow runs without requiring you to constantly context-switch between browser tabs.

The Data Agent Kit provides real-time monitoring of your Managed Airflow runs directly within your IDE.

The Data Agent Kit visualises the created pipeline.

Inevitably, infrastructure or data issues occur—perhaps a Managed Spark cluster hits an out-of-memory exception due to a seasonal data spike, or a BigQuery quota is reached. Resolving these issues no longer requires digging through thousands of lines of raw execution logs.

If a pipeline fails, the Data Agent Kit provides out-of-the-box agentic troubleshooting. With the click of a "Troubleshoot" button in your IDE, the Data Engineering Agent analyzes the failure context. It can accurately distinguish between infrastructure quota limits and code-level bugs, instantly providing a root-cause summary and suggesting an inline fix (such as scaling up the compute template).

Agentic troubleshooting instantly diagnoses pipeline failures, identifies infrastructure bottlenecks, and suggests inline fixes.

Summary: Accelerating time to value

Building a resilient MLOps architecture — extracting historical data, executing dbt transformations, provisioning Managed Spark ML compute, integrating Gemini Enterprise Agent Platform for model registry and inference, and configuring cross-DAG conditional triggers — traditionally takes platform engineering teams weeks of writing complex Python Operator logic.

With Orchestration Pipelines and the Data Agent Kit, this entire lifecycle was authored, deployed, and easily maintained in a matter of minutes. By replacing boilerplate infrastructure code with a declarative, agent-ready standard, we are ensuring your data organization spends less time orchestrating pipelines and more time delivering tangible business value.

Get Started Today:

Review the Orchestration Pipelines documentation.
Install the Data Agent Kit in your preferred IDE or CLI and configure your workspace.
Learn more about the broader ecosystem in our recent blog post: Data Agent Kit brings data skills and tools to your IDE or CLI.
Explore reference architectures in the Data Agent Kit documentation.

Using OKF with Knowledge Catalog to serve context for agents

Wed, 26 Aug 2026 16:00:00 +0000

We continue to iterate on the Open Knowledge Format (OKF), an open specification that formalizes the LLM-wiki pattern into a portable, interoperable format. But a big question remains: How can you share and govern access to an OKF bundle across an organization?

OKF v0.1 established a portable format for the context agents need: markdown files with YAML frontmatter, one required field, and five conventions. Then, OKF v0.2 added the trust signals (provenance, verification, freshness, attestation) that a machine-authored bundle requires to be relied on, allowing a team to publish a trustworthy bundle for its own agents.

However, what OKF does not answer is how teams share their bundles across an organization. A git repo per bundle is portable, but it is not searchable alongside the data it describes, it cannot be secured and governed using the same organizational identity and compliance policies, and it does not sit next to the technical metadata (schemas, lineage, ownership) that data teams already work in. Every downstream agent must know where each bundle resides, and that does not scale beyond a small number of bundles.

To scale an OKF bundle across an organization, you can use Knowledge Catalog, Google Cloud's context engine for agents. By mapping the bundle onto Knowledge Catalog's existing types, every concept becomes discoverable, governed, and reachable by any agent already reading from the catalog.

Knowledge Catalog is the context engine for agents

Every agent that queries Knowledge Catalog reads from one governed index over what the organization already has in BigQuery, Cloud Storage, operational databases, and applications. Each entry carries schema, lineage, ownership, and tags, and can be extended with typed aspects that add domain-specific fields. The same catalog exposes search and cross-project lookup to retrieve optimized context for each agentic query. The context retrieval is secure and governed by IAM controls, so agents can only see the entries they have access to based on IAM identity.

Publishing an OKF bundle into Knowledge Catalog takes a one-time setup and a single push. Both use the OKF sample code in the Knowledge Catalog repository, whose wrappers call gcloud dataplex for setup and delegate push to kcmd (the Metadata-as-Code CLI in the same repository).

The setup registers three Knowledge Catalog resources: an EntryGroup to hold the bundle, an EntryType named okf-bundle for its concepts, and an AspectType named okf that carries the OKF signal fields (from the okf-aspect.json schema in the sample code). The push then creates one okf-bundle Entry per concept, each with two Aspects: an overview Aspect for the markdown body, and an okf Aspect for the structured signals. Display name, description, and tags live on the Entry itself. The bundle's index.md navigation files and its root log.md are also published as Entries: index files carry only the overview Aspect (no OKF frontmatter), and log.md carries both Aspects with okf_type: Log.

Everything Knowledge Catalog already does for technical metadata (search, IAM, lineage, cross-project discovery) applies equally to OKF bundles, alongside the data they describe.

The okf AspectType

The okf-aspect.json schema in the sample code defines the AspectType. It carries 13 fields covering the full OKF v0.2 spec:

#	Field	Type	Purpose
1	`okf_type`	string	The OKF document type (freeform, e.g. `BigQuery Table`, `Metric`, `Attested Computation`).
2	`generated`	record `{by, at}`	Actor and timestamp for the last meaningful change.
3	`sources`	array of `{id, resource, title, author, usage_count, last_modified}`	Materials the concept derives from, with credibility signals.
4	`verified`	array of `{by, at}`	Verification events. A `human:` actor marks the highest trust tier.
5	`status`	string	Lifecycle state: `draft`, `stable`, or `deprecated`.
6	`stale_after`	datetime	Absolute point in time (RFC3339 with an explicit offset) on or after which the content is stale.
7	`usage_window`	record `{from, to}`	Period the source usage counts were measured over.
8	`runtime`	string	How an Attested Computation runs (e.g., `bigquery`).
9	`parameters`	array of `{name, type, required}`	Typed named holes a caller may fill. The only surface a caller may vary.
10	`computation`	string	Path to a file holding the computation body.
11	`executor`	record `{resource, receipt[]}`	How the computation runs and what evidence it must return.
12	`attester`	record `{resource}`	Deterministic code that takes a receipt and returns a verdict.
13	`extra`	string	Producer-defined frontmatter the template does not model, as JSON `[path, value]` pairs. Keeps the round-trip lossless.

Every field is annotated with a display name, a description, and a mandatory index. Any top-level scalar field in the okf Aspect (okf_type, status, stale_after, runtime, computation, extra) can drive Knowledge Catalog search predicates directly, so aspect:acme-analytics.us-central1.okf.okf_type=Metric returns every OKF Metric in scope. Scalar subfields of record fields (generated.by, usage_window.from, executor.resource, attester.resource) also drive predicates. The array fields (sources, verified, parameters) are not server-side searchable on their subfields; agents narrow on them client-side after entries.get with view=ALL. One caveat for search predicates on datetime-typed fields (stale_after, generated.at, usage_window.from/.to), use a bare date (stale_after=2026-12-31) or a range comparison (stale_after>2026-01-01), not the full RFC3339 timestamp.

Pushing a bundle

kcmd push reads an OKF bundle from git and writes each concept as an Entry in the target Knowledge Catalog EntryGroup. index.md files become Entries too, and each concept is parented to the index above it, so the bundle's directory structure survives as a browsable hierarchy.

kcmd expects a bundle in the Documents Layout: markdown files under a catalog/ subdirectory, and a catalog.yaml at the bundle root that lists the snapshot's entry and aspect types. The sample code's setup.ts generates catalog.yaml from its --entry-group flag (default okf_demo), so a reader wiring the sample to a new bundle passes the flag rather than editing catalog.yaml by hand.

Here is an end-to-end workflow for the Acme Retail bundle that we introduced in the OKF v0.2 blog:

code_block: <ListValue: [StructValue([('code', '# One-time setup (if required): install bun, clone the repo, build kcmd, configure gcloud\r\ncurl -fsSL https://bun.sh/install | bash\r\nexport BUN_INSTALL="$HOME/.bun" && export PATH="$BUN_INSTALL/bin:$PATH"\r\ngit clone https://github.com/GoogleCloudPlatform/knowledge-catalog\r\ncd knowledge-catalog/toolbox/mdcode && npm install && npm run build\r\n\r\n# Authenticate, set project and enable dataplex apis\r\ngcloud auth login\r\ngcloud config set project <your-project>\r\ngcloud config set compute/region <your-location>\r\ngcloud services enable dataplex.googleapis.com\r\ngcloud auth application-default login\r\n\r\n# Push the Acme Retail bundle\r\ncd demo/okf\r\nbun run setup.ts # creates the EG (default \'okf_demo\')\r\nbun run push.ts # pushes okf/bundles/acme_retail into the EG setup created'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc48362cd0>)])]>

To pick a different EntryGroup name or push a different bundle, pass --entry-group your-name to setup.ts and --bundle path/to/your/bundle to push.ts. For example: bun run setup.ts --entry-group acme-bundle followed by bun run push.ts. This regenerates the manifest, so subsequent push, pull, and cleanup all target the new EG; delete earlier EGs manually with gcloud dataplex entry-groups delete <name> --project <your-project> --location <your-location>.

The Acme Retail bundle is a synthetic OKF bundle for a US retailer's BigQuery estate. It contains nine leaf concepts across six directories (attesters, tables, metrics, computations, policies, skills), each with its own index.md, plus a bundle root with its own index.md and log.md. That's 17 pushed Entries in total; Dataplex auto-creates one <eg>_entry alongside, so gcloud dataplex entries list returns 18 rows.

After the push completes:

Every concept markdown file is a Knowledge Catalog Entry, discoverable by search across the whole project or organization, depending on IAM configuration.
The revenue-ytd Attested Computation appears in the console with its sanctioned SQL, its executor, its attester, its verification history, and the full concept body.
An analyst searching Knowledge Catalog for "revenue" finds Acme Retail's business definition alongside the BigQuery table it computes from, both under one permission model.
A downstream agent that already calls LookupContext for BigQuery table Entries retrieves the bundle's context by adding the OKF entry names to its resources list.

Further, metrics/revenue.md becomes an Entry with two Aspects. The full entries.get response (with view=ALL) looks like:

code_block: <ListValue: [StructValue([('code', '{\r\n "name": "projects/acme-analytics/locations/us-central1/entryGroups/acme-retail/entries/metrics/revenue",\r\n "entryType": "projects/acme-analytics/locations/us-central1/entryTypes/okf-bundle",\r\n "createTime": "2026-08-15T00:48:39.123456Z",\r\n "updateTime": "2026-08-15T00:48:57.234567Z",\r\n "parentEntry": "projects/acme-analytics/locations/us-central1/entryGroups/acme-retail/entries/metrics/index",\r\n "entrySource": {\r\n "displayName": "Revenue",\r\n "description": "Recognized revenue for a period, per Acme\'s FY2026 revenue-recognition policy. Backed by an Attested Computation.",\r\n "labels": {\r\n "finance": "true",\r\n "revenue": "true",\r\n "headline-metric": "true"\r\n },\r\n "location": "us-central1"\r\n },\r\n "aspects": {\r\n "dataplex-types.global.overview": {\r\n "aspectType": "projects/dataplex-types/locations/global/aspectTypes/overview",\r\n "createTime": "2026-08-15T00:48:57.111111Z",\r\n "updateTime": "2026-08-15T00:48:57.111111Z",\r\n "aspectSource": {},\r\n "data": {\r\n "content": "# Definition\\n\\nRevenue for a fiscal year is the sum of `net_amount` over orders that (a) reached `order_status = \'delivered\'`, (b) completed the 30-day return window, and (c) fall in the fiscal year by `order_ts`. Multi-currency orders are converted to USD at the `order_ts` daily reference rate. [^revenue-policy]\\n\\nThe sanctioned computation is [`computations/revenue-ytd.md`](../computations/revenue-ytd.md). Consumers MUST run and attest that computation rather than composing their own SUM. The attester rejects any receipt whose executed SQL does not match the sanctioned form.\\n\\n# Reporting cuts\\n\\n- **By fiscal year:** the sanctioned computation takes `year` as its sole parameter.\\n- **By channel or category:** these are approved narrations, not new metrics. Join the receipt\'s row-level result to `orders.channel` or to `order_lines` × `products.category` client-side. Do NOT rewrite the sanctioned SQL.\\n\\n# Trust and freshness\\n\\n- **Verified:** VP Finance sign-off on 2026-07-01, against the FY2026 policy.\\n- **Stale after 2026-12-31:** Finance re-issues the revenue recognition policy each January. Consumers of this concept after 2027-01-01 MUST re-verify the definition against the new policy before serving.\\n\\n[^revenue-policy]: Revenue Recognition Policy (FY2026)",\r\n "contentType": "MARKDOWN"\r\n }\r\n },\r\n "acme-analytics.us-central1.okf": {\r\n "aspectType": "projects/acme-analytics/locations/us-central1/aspectTypes/okf",\r\n "createTime": "2026-08-15T00:48:57.222222Z",\r\n "updateTime": "2026-08-15T00:48:57.222222Z",\r\n "aspectSource": {},\r\n "data": {\r\n "okf_type": "Metric",\r\n "generated": { "by": "reference_agent/gemini-2.5-pro", "at": "2026-06-30T14:00:00Z" },\r\n "verified": [ { "by": "human:jsmith@acme", "at": "2026-07-01T09:00:00Z" } ],\r\n "status": "stable",\r\n "stale_after": "2026-12-31T00:00:00Z",\r\n "sources": [\r\n {\r\n "id": "revenue-policy",\r\n "resource": "policies/revenue-recognition.md",\r\n "title": "Revenue Recognition Policy (FY2026)",\r\n "author": "human:jsmith@acme",\r\n "last_modified": "2026-06-15T00:00:00Z"\r\n }\r\n ]\r\n }\r\n }\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc3b8ad410>)])]>

The overview Aspect holds the full body of revenue.md. The okf Aspect carries the structured signal fields, so agents get provenance, source, and OKF type in a form they can filter on directly instead of parsing markdown. Server-side searchEntries filters on the top-level scalar fields and on the scalar subfields of record fields; agents narrow further on the array-element subfields client-side after entries.get. (Aspects and EntryTypes are keyed by project number in real API responses and search predicates; the acme-analytics project ID is shown throughout for readability.)

What pushing your OKF to Knowledge Catalog enables

Once the bundle is in Knowledge Catalog, it provides two capabilities to any agent that reads from the catalog:

Discoverability across the organization. Agents find bundle concepts through the same searchEntries and LookupContext APIs they already use for cataloged data, so an OKF bundle appears alongside BigQuery tables and other resources in every query it matches.
Governance. Bundle Entries inherit IAM from the EntryGroup, so a single agent call returns exactly what the caller is permitted to read, with no parallel permission model to maintain.

Discoverability across the organization
OKF bundle Entries appear in searchEntries results alongside BigQuery tables and other cataloged resources, so an agent already querying the catalog picks up new bundles automatically. To retrieve a concept's body, trust signals, or linked concepts from a match, the agent moves to LookupContext and entries.get.

A LookupContext call looks like this:

code_block: <ListValue: [StructValue([('code', 'POST https://dataplex.googleapis.com/v1/projects/acme-analytics/locations/us-central1:lookupContext\r\n{\r\n "resources": [\r\n "projects/acme-analytics/locations/us-central1/entryGroups/acme-retail/entries/metrics/revenue"\r\n ],\r\n "options": { "format": "yaml", "context_budget": "8000" }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc3b8ae190>)])]>

The response is a single context field containing a pre-formatted YAML block. The block carries the entry's catalogEntry, its type, its description, its tags as labels, and its overview: the full markdown body of the concept, including its trust and freshness section. LookupContext does not render custom Aspects, so an agent that needs the structured OKF signal fields (okf_type, generated, sources, and the other ten) reads them with entries.get and view=ALL alongside the LookupContext call.

There is no repository clone, no manual Aspect merging, and no re-parse of frontmatter. The agent uses the same API call any Knowledge Catalog client already makes.

An agent traversing an OKF bundle typically follows a three-step flow. An agent that already knows the specific Entry names it needs skips step 1. An agent that already knows the target EntryGroup and wants to enumerate the bundle exhaustively substitutes entryGroups.entries.list for step 1.

searchEntries returns candidate Entry names and descriptions. Its scope accepts a project or organization; narrowing within that scope happens through query terms, including aspect predicates like aspect:acme-analytics.us-central1.okf.okf_type=Metric.
LookupContext on the top few Entry names (up to ten per call) returns the full concept body as pre-formatted YAML; context_budget caps the response size.
entries.get with view=ALL on any Entry returns its structured OKF signals (okf_type, generated, sources, and the other ten) directly, which the agent can then filter or attest on.

When a concept's sources[] references another concept by path, the agent calls LookupContext on that Entry name to walk the reference.

The full response for the Revenue Entry:

code_block: <ListValue: [StructValue([('code', "resources:\r\n -\r\n catalogEntry: projects/acme-analytics/locations/us-central1/entryGroups/acme-retail/entries/metrics/revenue\r\n type: OKF Document\r\n description: Recognized revenue for a period, per Acme's FY2026 revenue-recognition\r\n policy. Backed by an Attested Computation.\r\n overview: |-\r\n # Definition\r\n\r\n Revenue for a fiscal year is the sum of `net_amount` over orders that (a) reached `order_status = 'delivered'`, (b) completed the 30-day return window, and (c) fall in the fiscal year by `order_ts`. Multi-currency orders are converted to USD at the `order_ts` daily reference rate. [^revenue-policy]\r\n\r\n The sanctioned computation is [`computations/revenue-ytd.md`](../computations/revenue-ytd.md). Consumers MUST run and attest that computation rather than composing their own SUM. The attester rejects any receipt whose executed SQL does not match the sanctioned form.\r\n\r\n # Reporting cuts\r\n\r\n - **By fiscal year:** the sanctioned computation takes `year` as its sole parameter.\r\n - **By channel or category:** these are approved narrations, not new metrics. Join the receipt's row-level result to `orders.channel` or to `order_lines` × `products.category` client-side. Do NOT rewrite the sanctioned SQL.\r\n\r\n # Trust and freshness\r\n\r\n - **Verified:** VP Finance sign-off on 2026-07-01, against the FY2026 policy.\r\n - **Stale after 2026-12-31:** Finance re-issues the revenue recognition policy each January. Consumers of this concept after 2027-01-01 MUST re-verify the definition against the new policy before serving.\r\n\r\n [^revenue-policy]: Revenue Recognition Policy (FY2026)\r\n labels:\r\n finance: 'true'\r\n revenue: 'true'\r\n headline-metric: 'true'"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc3b8ac7d0>)])]>

Governance
Permissions on the EntryGroup use standard Knowledge Catalog IAM. An agent that names both a bundle concept and the BigQuery table it grounds against in one call receives both, each subject to its own existing access control list (ACL), so the response carries only what the caller is already permitted to read. There is no parallel permission model to maintain.

Reading agents use roles/dataplex.catalogViewer, which grants the read paths: entries.get, LookupContext, and searchEntries. The identity that runs kcmd push uses roles/dataplex.catalogEditor, which grants the write paths: entries.create and entries.patch. One EntryGroup per bundle-owning team is the multi-team pattern, and IAM on the EntryGroup cascades to its Entries.

LookupContext resolves the entry names it is given, up to ten per call, within a single location. It does not follow links out of a concept's body, so an agent that wants a referenced concept must name it explicitly. Place the bundle's EntryGroup in the same location as the data it describes to fetch both in one call.

Lifecycle

kcmd push is an idempotent upsert. Re-running is safe (no duplicates, no error), but every push writes every Entry. Concept deletes require an explicit kcmd delete on the Entry, or cleanup.ts to remove the whole EntryGroup at once; cleanup.ts deletes only the EntryGroup and its Entries, so the shared okf AspectType and okf-bundle EntryType stay in place for other bundles that reference them. For continuous ingestion in production, wire a CI job to kcmd push on every commit to the bundle repository, using a service-account credential with roles/dataplex.catalogEditor on the target EntryGroup.

Getting started

OKF defines what a trustworthy bundle looks like. Knowledge Catalog makes it reachable across the organization. To get started, check out the following resources:

Read the OKF v0.2 spec and browse the Acme Retail bundle.
Author a small bundle for one domain your team owns.
Sync it into your Knowledge Catalog project using the sample code's setup.ts (which registers the resources) and push.ts (which delegates to kcmd).
Point your existing agents at Knowledge Catalog. New context becomes reachable through the same LookupContext and searchEntries calls they already use.

Serverless Apache Spark on Google Cloud: Architecture Choices & AI Troubleshooting

Wed, 19 Aug 2026 16:00:00 +0000

In modern enterprise data engineering, Apache Spark remains a cornerstone framework for processing massive datasets at scale. However, managing infrastructure such as provisioning clusters, tuning YARN configurations, and avoiding costs for idle hardware often detracts from what matters most: building resilient data pipelines. Google Cloud addresses this operational overhead via its Managed Service for Apache Spark, offering flexible deployment modes of serverless and managed clusters tailored to specific operational needs.

This technical guide walks through the architectural decision matrix for deploying Spark on Google Cloud, details resource and cost optimization techniques, and demonstrates how to apply built-in Gemini Cloud Assist to rapidly troubleshoot and resolve serverless batch pipeline failures. While there is benefit to reading these three parts in a sequence, each one can be read independently and add value to how you approach Spark development on Google Cloud.

Part 1: Choosing your Apache Spark deployment model

When launching Spark workloads on Managed Service for Apache Spark, the first major decision point is evaluating whether to construct traditional managed clusters or transition to a zero-management, serverless infrastructure footprint.

Decision #1: Managed clusters vs. serverless

*Created using Nano Banana 2 in Gemini Enterprise Agent Platform

Choosing between traditional Managed Spark clusters and serverless depends on ecosystem requirements, infrastructure control needs, and financial utilization patterns:

Workload frequency, latency sensitive workloads & financial fit: For continuous, highly predictable, 24/7 streaming or batch processing pipelines where cluster nodes maintain constant high utilization baselines (80%+) or when the workflow’s accumulated startup time risk meeting SLA target, a permanently running, finely tuned traditional cluster, with custom YARN autoscaling rules, can sometimes be more cost-predictable. Conversely, for intermittent, bursty, ad-hoc, or orchestrator-triggered pipelines, Managed Spark serverless is highly optimal, eliminating operational management, requiring less planning time and ensuring you don’t pay for idle compute time.
Ecosystem & component requirements: Managed Spark serverless is strictly optimized for Apache Spark 3.x+ codebases. If your processing pipeline relies on other ecosystem components such as Apache Flink, Presto/Trino, Hive LLAP, or Apache HBase, or if you are locked into a legacy Spark 2.x codebase, you must use Managed Spark clusters.

Infrastructure customization needs: Managed Spark serverless abstracts away the underlying virtual machine (VM) layer. If your workload mandates deep OS-level hardware tuning, custom OS initialization actions, root SSH access to instances, specific local SSD configurations, or custom machine shapes, a traditional cluster is required. Note that serverless does support custom Docker container images for bundling specific application-level libraries.

Decision #2: Serverless interactive sessions vs. serverless batches

*Created using Nano Banana 2 in Gemini Enterprise Agent Platform

Once you select the serverless deployment mode, you must choose the appropriate execution model based on your development stage and operational requirements. Managed Service for Apache Spark provides two options for running serverless workloads:

Serverless interactive sessions

Interactive sessions are great for iterative and exploratory use cases. You write blocks of code, inspect intermediate DataFrames, modify variables, and generate visualizations with your dataset held warm in-memory.

Primary interface: Designed for human-in-the-loop interaction. Developers execute code cell-by-cell using their IDE of choice, such as Colab, Gemini Enterprise Agent Platform Workbench, Antigravity, Jupyter notebooks, etc.
Idle cost profile: Compute resources remain active to support immediate execution during developer thinking time, which can incur some idle compute charges if sessions are left inactive.

Serverless batches

Batches are useful when you know what you want to run, and need automated, non-interactive execution. The engine runs fully completed, packaged PySpark scripts (.py) or Java/Scala application files (.jar) from start to finish without manual human intervention.

Primary interface: Managed by automated orchestrators, such as Managed Service for Apache Airflow, Cloud Scheduler, or CI/CD pipelines.
Idle cost profile: Billed strictly for the duration of the run. Compute resources are provisioned on-demand, run the script, and immediately shut down upon completion to prevent idle costs.

The development-to-production lifecycle

These execution options are designed to work together as a natural pipeline lifecycle. During the initial development phase, you open a serverless interactive session within your notebook interface to explore datasets, clean schemas, and prototype transformations. Once your logic is validated and the transformations are finalized, you package the code into a Python script and schedule it as a serverless batch job orchestrated by Managed Service for Apache Airflow for production execution. This transition minimizes ongoing development costs while maintaining operational reliability.

Part 2: Advanced performance tuning and DCU cost optimization

While serverless Managed Spark eliminates the operational overhead of cluster maintenance, running production enterprise-grade pipelines on default settings can result in performance bottlenecks or budget waste. Resource allocation must be explicitly declared during submission using runtime configuration properties to maintain an efficient Data Compute Unit (DCU) burn rate.

Google recently introduced history-based autotuning. In the context of serverless, this capability automatically applies optimizations based on best practices and historical execution. It does this by grouping recurring batch workloads into what Google calls cohorts. The autotuner analyzes the telemetry and statistics from previous runs under that same cohort name to figure out where the bottlenecks are.

Customizing driver and executor shapes

By default, serverless batches allocate generic specifications (4 cores and 16,000MB RAM). This can cause critical efficiency issues depending on the nature of the application:

The Memory-Bound job: Pipelines processing highly uncompressed data volumes may hit Out-Of-Memory (OOM) errors and crash. To counter this, increase heap sizing independently using spark.driver.memory and spark.executor.memory.
The Compute-Bound Job: Processing-intensive jobs running mathematical modeling or heavy tokenization might saturate CPUs while leaving expensive RAM sitting idle. Fine-tune processing concurrency per instance by explicitly adjusting spark.driver.cores and spark.executor.cores.

Remember that by default increasing cores, automatically provisions a proportionate baseline of memory to match the vCPU-to-RAM ratio. This is why overriding the values for both cores and memory is critical

Controlling autoscaling boundaries

Managed Spark serverless dynamically scales up and down the number of active executors based on backlogged tasks. However, unconstrained scaling can lead to budget overruns if a rogue code loop or unoptimized cartesian join is introduced.

As a defensive guardrail, always declare an explicit upper limit using spark.dynamicAllocation.maxExecutors. This acts as your budget deadman-switch. By capping this at a reasonable ceiling, you guarantee that even if the code behaves sub-optimally, the job will never scale past a fixed infrastructure footprint.

High priority (SLA-driven): Set maxExecutors to a higher ceiling to allow resource bursting and minimize overall runtime duration.
Low priority (nightly batch): Set maxExecutors to a low, tight ceiling. The workload will run longer but will consume a predictable, flat, cost-efficient stream of DCUs.

Managing shuffle storage efficiency

When execution involves wide transformations like groupBy(), join(), or distinct(), data must be redistributed across the network, generating intermediate disk writes known as shuffle storage.

Spark defaults to a static setting of 200 partitions (spark.sql.shuffle.partitions). If you are processing a massive, multi-gigabyte dataset, 200 partitions means each individual chunk will be too large. When a partition's size exceeds available executor RAM (e.g., a 1GB partition trying to process inside 0.5GB of assigned heap space), data spills onto disk. This slows execution and incurs additional billing fees for premium or standard shuffle storage blocks. A helpful rule of thumb: Dynamically scale your partition parameters based on total data size so that each partition handles roughly 100MB to 200MB of data in memory. This may require a few iterations before the optimal results are achieved.

The above properties are the main tunable properties. Additional Serverless runtime configuration properties can be found in this link

Part 3: Operational diagnosis with Gemini Cloud Assist

When automated data pipelines fail in production, data engineers are traditionally forced to spend hours sifting through verbose, disjointed log files across drivers and executors. Managed Service for Apache Spark addresses this friction by natively integrating Gemini Cloud Assist into the Google Cloud console, allowing engineers to diagnose and resolve failures using natural language.

To illustrate this operational shift, we examine the typical troubleshooting lifecycle for a failed PySpark ETL pipeline that reads customer transaction data from a Google Cloud Storage (GCS) bucket, applies transformations, and encounters unexpected runtime errors.

Stage 1: Diagnosing missing execution parameters

During the initial execution attempt of a new pipeline, the batch job status switches from pending to running, and ultimately ends in a failed state with a generic exit message: Application failed with exit code 1.

Rather than manually querying Cloud Logging or navigating through multiple sections of the console, the engineer can locate the error log and select the ‘Investigate log’ option. This action opens a native conversation pane where Gemini Cloud Assist automatically analyzes the driver telemetry and system logs.

In this scenario, the assistant explains in plain English that the PySpark script failed because required runtime arguments (such as the source GCS bucket path) were omitted during submission. It instantly identifies the exact lines in the script expecting these arguments, eliminating the need to read through the stack trace.

Stage 2: Resolving schema and data type anomalies

Once the missing arguments are resolved and the job is re-submitted, the pipeline runs but encounters a secondary data anomaly. In high-volume ingest pipelines, upstream source files frequently contain corrupted records or formatting inconsistencies.

Upon the second failure, the engineer again prompts Gemini Cloud Assist to investigate the logs. The assistant identifies a TypeError and pinpoints the exact DataFrame transformation causing the crash: a division operation (df['amount'] / df['transaction_id']) that failed because the schema auto-inferred the columns as strings.

Additionally, the assistant scans the underlying GCS file data to identify the root cause: non-numeric anomalies (such as text strings within numerical cells) in the source dataset.

Stage 3: Generating and deploying verified code fixes

Rather than manually rewriting the PySpark logic to cast schema types and catch null values, the engineer can prompt Gemini Cloud Assist directly to generate a resilient solution:

User Prompt: "Suggest how to rewrite the code to divide the amount by quantity instead of transaction_id. In addition, add logic to skip invalid records without failing the process."

The assistant generates the corrected PySpark code block, using resilient casting and null-handling functions (such as coalesce and try_cast).

By implementing this corrected script, the orchestration pipeline can filter out bad source records smoothly without crashing the entire batch run. The subsequent execution completes successfully, preserving data freshness SLAs.

Unlock serverless Apache Spark: Benefits and next steps

Managing data processing pipelines should not require a deep specialization in infrastructure configuration. By pairing the hands-off scale of serverless batches with explicit resource tuning — such as dynamic allocation caps and calculated shuffle sizing — data teams can maintain strict control over performance and cost profiles. When failures do occur, integrating Gemini Cloud Assist directly into your logging workflows transforms complex troubleshooting from a manual log-sifting exercise into a rapid, automated cycle.

To start putting these architectures into practice, you can explore the Managed Service for Apache Spark documentation and execute a serverless batch directly in the Google Cloud console.

For a deep architectural analysis of these concepts, get instant access to A practitioner’s guide to Apache Spark® in the agentic era. This guide includes step-by-step workflows, Codelabs, and runnable PySpark and Terraform templates directly from our GitHub repository. If you are new to Google Cloud, you can test these blueprints on serverless and managed clusters at zero cost by signing up for a free trial with $300 in credits.

How to modernize Apache Hive using Google Cloud’s Lakehouse runtime catalog

Wed, 19 Aug 2026 16:00:00 +0000

For over a decade, the Apache Hive Metastore (HMS) has served as the de facto metadata authority for big data analytics. Whether it was deployed on Hadoop clusters, self-managed Compute Engine VMs backed by MySQL or PostgreSQL, HMS provided the central schema registry that let Apache Spark, Presto, and Hive query raw .parquet and .orc files.

However, as enterprise data architectures scale to petabytes and span multiple query engines (such as Google Cloud Managed Service for Apache Spark, BigQuery, and Trino), legacy Hive Metastores often become critical operational bottlenecks.

In this blog, we explore why legacy metastores struggle in modern cloud environments at agent scale, and show you how the serverless Google Cloud Lakehouse runtime catalog that we introduced last year can help: Built on the open Apache Iceberg REST catalog specification, it is a runnable, zero-data-copy migration solution to help you transition your production Hive tables in minutes.

The challenges of legacy Hive Metastores

When speaking with data engineers and infrastructure leads running production analytics at scale, three core pain points consistently emerge with standalone Hive Metastores:

Architectural and scaling bottlenecks
Standalone HMS deployments rely on relational database backends (such as MySQL or Postgres) to track table schemas, partitions, and storage locations. As data lakes grow to hundreds of thousands of partitioned tables, partition pruning and bulk listing operations lead to key performance bottlenecks on the relational database. A complex Spark job requesting partition metadata can spike metastore CPU to 100%, causing cluster-wide query delays or out-of-memory (OOM) failures.

Siloed identity and security governance
Legacy metastores were designed around perimeter-based Hadoop security models. Enforcing modern granular data governance — such as table-level access control lists (ACLs) — across both Apache Spark compute jobs and enterprise SQL engines like BigQuery requires maintaining fragmented, duplicated security policies across two distinct control planes.

Operational overhead and total cost of ownership (TCO)
Managing high-availability MySQL/Postgres instances, patching HMS daemons, tuning JDBC connection pools, and paying for idle instance-based metastore servers creates unnecessary operational toil for data platform teams, whose time is better spent building high-leverage data products for agents.

The solution: Lakehouse runtime catalog

To solve these architectural bottlenecks without forcing data engineers to rewrite petabytes of existing storage payloads, we built the Lakehouse runtime catalog with support for Iceberg Rest Catalog and Hive Catalog.

The Lakehouse runtime catalog is a fully serverless, highly available, and unified metadata registry designed from the ground up to support both legacy Hive/Parquet tables and modern open table formats like Apache Iceberg. By natively implementing the Apache Iceberg REST Catalog specification, the Lakehouse runtime catalog decouples metadata discovery from compute engines. This decoupling of the catalog and compute engines ensures multiple Iceberg compatible engines can access the same data in a zero copy fashion thereby reducing the need for customers to maintain multiple copies of the data and enables them to take their workloads to production sooner.

This approach offers a number of architectural benefits:

Multi-engine interoperability: Once registered, tables are immediately discoverable and queryable across Google Cloud Managed Spark, BigQuery, and open-source engines via standard REST interfaces.
Open APIs: Supports Iceberg Rest Catalog and Hive Catalog which enables different teams to use their preferred analytics tools on a single, unified dataset.
Zero-data copy: Table definitions point directly to your existing data in Google Cloud Storage. You do not move, rewrite, or duplicate your underlying data.
AI-powered governance, security and trusted context: The Lakehouse runtime catalog integrates directly with Knowledge Catalog and Cloud IAM, allowing you to define trusted context for your agents and table-level security that apply consistently across all compute engines. Further it supports key authorization mechanisms, such as credential vending. This means you can access your tables without needing direct access to the files in the underlying Cloud Storage bucket.
Enterprise-readiness, scale and reduced TCO: Backed by Google’s planet-scale infrastructure and Spanner, enabling your metadata to scale with your data. Support for Cloud Storage dual-region and multi-region buckets enables failover use cases. It also provides reduced TCO due to serverless and no-ops environments, and scalability for any workload size.

Zero-copy migration from legacy Hive Metastore in action

To demonstrate how smooth cutover is in practice, we have provided a capability that lets you modernize your legacy self-managed Hive Metastore to the Google Cloud Lakehouse. This capability connects directly to your legacy Hive Metastore, extracts external table definitions and partition maps, and registers them cleanly into the serverless Lakehouse catalog and then start using the data in Google Managed Spark, BigQuery and Conversational Analytics agents with Gemini.

Modernize to the Lakehouse and immediately tap your data in key agentic journeys

Ready to modernize your data architecture?

Modernizing from a legacy Hive Metastore to Google Cloud’s Lakehouse minimizes data silos across analytics engines and agents, unifies multi-engine governance, provides trusted context to your agents and slashes operational TCO. In other words, it helps prepare your modern cloud environments to operate at agent scale.

Get started and migrate your Apache Hive Metastore tables to Google Cloud today, and get ready for the agentic era. Learn more about Google Cloud Lakehouse here.

How Box is unlocking multimodal enterprise agents with Gemini Embeddings 2

Tue, 18 Aug 2026 16:00:00 +0000

Enterprise content management is experiencing its biggest architectural shift since the cloud migration era.

For years, enterprises have stored trillions of gigabytes of critical data in Box: financial models, clinical trial protocols, M&A due diligence rooms, engineering schematics, and legal compliance playbooks. Up to this point, text-based search and retrieval-augmented generation (RAG) have successfully unlocked the vast narrative knowledge within these repositories, establishing a powerful and highly effective baseline for enterprise AI intelligence.

Traditional RAG architectures have mastered text processing, but the agentic era demands more. The next logical evolution is to extend this framework to capture the inherently multimodal, deeply spatial, and highly structured elements that exist alongside text. While text embeddings excel at indexing prose, multimodal architectures unlock a major new capability: For example, they preserve the strict row-column semantics of financial tables, interpret visual evidence like clinical data, and map the logic of multi-page flowcharts without losing their spatial layout.

To deliver next-generation capabilities that can handle the vast universe of digital content, Google Cloud and Box are integrating advanced multimodal capabilities into Box's Agentic Platform, powered by Gemini Multimodal Embeddings 2 merging Box’s industry-leading Intelligent Content Management platform with Google Cloud’s advanced AI embeddings.

Benefits of improved embedding: Extending the dimensions of document content

Preserving visual and spatial geometry: Complex document elements like multi-column tables or financial matrices rely on their spatial layout to convey meaning. Converting these elements into a flat string of text can disassociate column headers from their corresponding data points. Multimodal embeddings allow systems to interpret the document exactly as a human does, maintaining the integrity of spatial relationships.
Illuminating the visual modality: Enterprise documents are filled with visual indicators: technical charts, process flowcharts, branding assets, and product photography. Multimodal capabilities ensure that these elements are no longer invisible to search systems, allowing users to query images and text simultaneously.
Connecting hybrid file formats: Real-world business workflows rarely live in a single document format. An agent may need to cross-reference a PDF policy, a spreadsheet tracking log, and a presentation deck. Extending RAG with multimodal embeddings creates a unified understanding across these varied formats.

The Architectural Solution: Gemini Multimodal Embeddings 2

Google Cloud’s Gemini Multimodal Embeddings 2 introduces a unified, multimodal vector space capable of embedding text, raster images, document pages, rendered spreadsheet tables, and visual charts into the same semantic representation space.

Key product capabilities unlocked by gemini-embeddings-2:

Crossmodal retrieval (text-to-visual / visual-to-text): Enables natural language queries to retrieve highly specific visual components, such as locating a target chart or diagram within a massive library of slides, without requiring manual tagging.
Layout-aware document embedding: Rather than breaking files into arbitrary text blocks, the system can embed document page renderings directly, preserving visual hierarchies, callout boxes, and structural context.
Heterogeneous format bridging: Native support for seamlessly bridging content across .docx, .xlsx, .pdf, .pptx, .png, and .csv without losing modality-specific structural information.

Three core patterns of multimodal enterprise agents

By leveraging multimodal embeddings within Box, we have identified three uniqueprimary design patterns that illustrate how organizations can extend traditional RAG to support complex, visual workflows.

Pattern 1: Complex financial & analytical reporting

The challenge

Corporate finance, research, and audit teams analyze highly structured documents where vital data resides in embedded tables, growth charts, and footnote annotations. Text-only indexing can separate these numbers from their context, making automated analysis challenging.

The multimodal advantage

Structural alignment: The embedding model captures the physical structure of tables and charts, allowing financial agents to understand that a column header applies to a specific row of metrics.
Visual trend analysis: Agents can cross-reference written summaries with visual trends in accompanying bar or line charts, identifying and pointing out discrepancies between written claims and source data.
Contextual sourcing: Users can query complex portfolios and instantly retrieve the exact page, table, or chart supporting a specific metric.

Pattern 2: Multimodal clinical decision support & assisted diagnosis

The challenge

In healthcare and clinical environments, critical patient data is fragmented across vastly different, unstructured visual and textual formats — ranging from external physical photos (visual evidence) and microscopic pathology slides (lab reports) to structured risk matrices (triage grids). Traditional text-based systems or isolated analysis tools cannot synthesize these cross-modal relationships simultaneously, which can delay critical diagnoses or risk missing immediate, life-threatening procedural complications.

The multimodal advantage

Cross-modal clinical synthesis: Evaluates physical symptoms alongside cellular-level laboratory evidence simultaneously by indexing clinical photos, histopathology imagery, and triage grids into a single space.
Granular anomaly identification: Connects niche visual patterns under a microscope (like parasitic cyst walls) with medical knowledge to rapidly isolate rare conditions.
Risk-aware decision support: Cross-references findings against triage frameworks to deliver instant warnings about immediate patient risks, such as life-threatening anaphylactic shock.

Pattern 3: Cross-document multimodal synthesis & data reconciliation

The challenge

Enterprise information is fragmented across disconnected files and formats (e.g., PDF minutes, Excel charts, PNG flyers, and email threads). Traditional tools analyze these files in isolation, failing to connect the dots when verifying details or resolving data contradictions across independent documents.

The multimodal advantage

Cross-file synthesis: Connects information across entirely different formats (PDFs, spreadsheets, images, emails) simultaneously to answer complex business queries.
Conflict resolution: Flags and resolves contradictions between assets, such as catching outdated pricing on an image by cross-checking it against the latest financial spreadsheets.
Visual-to-text auditing: Audits visual or scanned files against text-based records (e.g., verifying a signed PDF contract against a legal review email) to catch missing clauses or changes.

The future of agentic enterprise content management

The integration of gemini-embeddings-2 into Box’s Agentic Platform is an important new capability to improve the next era of content intelligence. Multimodal embeddings help Box to move beyond basic search to active, intelligent collaboration.Box's Intelligent Content Management platform represents a fundamental shift in enterprise AI infrastructure — moving beyond passive document storage to deliver a governed, semantically indexed reasoning layer where AI agents can interrogate, cross-reference, and act on content with full compliance and security controls already in place.

Powered by multimodal embeddings and a suite of native AI agents spanning search, metadata extraction, research, analysis, and composition, Box enables organizations to proactively surface insights such as flagging stale pricing data, expiring contract clauses, or cross-document contradictions before they become business risks. For high-complexity industries like financial services, life sciences, and legal operations, Box's ability to reason across text, tables, charts, and images makes multimodal understanding a competitive requirement.

Designed to interoperate with the broader enterprise AI ecosystem, Box serves as the single governed content foundation that ensures every AI-driven workflow is grounded in authorized, auditable enterprise data.

When you think about it, the enterprise data landscape was always multimodal. Now we have the technology to make the most of it. By integrating gemini-embeddings-2, Box helps its users unlock unprecedented value from unstructured enterprise content. Product leaders who embrace multimodal-first architectures, rigorous precision benchmarking, and audit-ready grounding will lead the next wave of enterprise productivity and innovation.

The team would like to thank Ken Ikeda, Afshaan Mazagonwalla, and Samip Thakkar for their work on this project.

Building cost-effective, high-throughput gen AI workflows in Google Dataflow

Tue, 18 Aug 2026 16:00:00 +0000

Real-time streaming pipelines are the operational backbone of modern enterprises, continuously processing everything from customer support interactions to transaction logs. Traditionally, streaming DAGs are static; once deployed, their processing logic and execution paths are fixed. However, by integrating generative AI agents, we can move beyond static logic to adaptive execution. This allows streaming workflows to dynamically construct plans, query databases, and trigger custom remediation paths at runtime depending on the content of the data.

For example, when a customer sends an angry message about a damaged order, a pipeline shouldn't just log the error or flag a dashboard. It should look up the order in the database that holds customer order and inventory records, decide on a remediation action (like shipping a replacement or issuing a refund), email the customer, and log the final resolution.

However, streaming systems face a fundamental engineering hurdle when executing gen AI workflows: scale, latency, and cost. Sending every raw event directly to a heavyweight model or multi-step agent equipped with external database and email tools is prohibitively expensive, introduces high latency, and quickly exhausts API rate limits.

This pattern addresses the scale and complexity challenge by combining Google Dataflow, Google Cloud's fully managed, serverless execution service for Apache Beam, and the Agent Development Kit (ADK) to build a hybrid streaming pipeline. By using a lightweight, CPU-bound machine learning model upstream to filter and qualify events, we keep the pipeline highly cost-effective, routing only the complex cases to the downstream agent. There, the agent dynamically decides what actions to take, introducing dynamic branching to the stream without hardcoding thousands of conditional steps into the pipeline's static DAG.

A universal blueprint for high-volume streams

While we use a customer support triage scenario below, this pre-filter + agentic action pattern is a universal paradigm. It applies to any stream where a high volume (>9X%) of events are routine, and only a small number require complex, contextual reasoning.

IT Operations & DevOps: Filtering millions of routine system logs on CPU, and triggering an agent to run diagnostics and open bug tickets only when a critical anomaly is flagged.
Financial Fraud Triaging: Passing millions of transactions through lightweight, local rules, and calling an agent to execute multi-database lookup tools only for highly suspicious patterns.
Industrial IoT: Monitoring normal telemetry on the edge, and routing erratic spikes to an agent to coordinate equipment shutdowns and email field engineers.

The architecture: Why pre-filter streaming events?

In a high-throughput stream, the vast majority of messages do not require complex reasoning or remediation. They might be positive feedback, neutral inquiries, or simple queries.

Routing every single event to a heavyweight LLM workflow creates three primary bottlenecks:

API cost: Frontier models charge per token. Under high throughput, cost scales linearly with stream volume.
Latency: Multi-step workflows (which involve database lookups and external API calls) take seconds, creating a bottleneck in streaming DAGs.
Quotas: External APIs have strict rate limits that streaming workers can easily exhaust.

To prevent this, we build a pre-filtered pipeline in Apache Beam/Dataflow:

Pipeline flow

Ingestion: Read raw customer messages from Google Pub/Sub.
Lightweight sentiment classifier (CPU): Run all messages through a lightweight, CPU-based Hugging Face model (distilbert-base-uncased-finetuned-sst-2-english) using Apache Beam’s RunInference transform. This executes locally on the Dataflow worker CPUs, avoiding external API costs.
Pre-qualification Gate: A simple DoFn filters the stream. Messages with POSITIVE or NEUTRAL sentiment are acknowledged and dropped.
Automated Remediation (ADK): If and only if a message is classified as NEGATIVE, we trigger the gen AI agent backed by gemini-3.5-flash using the ADKAgentModelHandler. The agent uses tools to look up the user in BigQuery, fetch orders, choose a remediation plan, and send a notification email via the Gmail API.

Adaptive execution: Making the Beam DAG dynamic

In traditional streaming architectures, the pipeline's Directed Acyclic Graph (DAG) is rigid. Once deployed to Dataflow, the sequence of transforms is set. If you need to handle new types of alerts or change how specific events are routed, you have to modify, test, and redeploy the entire pipeline.

By placing a gen AI agent downstream of our sentiment pre-filter, we introduce a dynamic, adaptive node inside the static DAG.

For the 95% of records that are positive or neutral, the pipeline runs along a fast, static path. But when the filter gates a negative record, the agent evaluates the payload and dynamically selects the correct sequence of API tools (e.g., database query, inventory check, or email notification) at runtime. This allows the pipeline to execute complex decision trees dynamically, eliminating the need to build and maintain thousands of hardcoded conditional branches in the static Apache Beam code.

Implementing the pipeline

Here is an example implementation in Apache Beam using the Google Agent Development Kit (ADK) and the RunInference framework.

1. Defining the lightweight sentiment model

We define the upstream CPU model using HuggingFacePipelineModelHandler. This model classifies sentiment into POSITIVE, NEUTRAL, or NEGATIVE on the worker instance.

code_block: <ListValue: [StructValue([('code', 'model_handler = HuggingFacePipelineModelHandler(\r\n task="sentiment-analysis",\r\n model="distilbert-base-uncased-finetuned-sst-2-english"\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc483612d0>)])]>

2. Building the heavyweight ADK agent

The ADK agent acts as our remediation assistant. We equip it with three tools:

lookup_user: Queries BigQuery for the customer's email.
lookup_orders: Queries BigQuery for the customer's orders and current product inventory.
send_email: Sends a remediation email to the customer using the Gmail API.

code_block: <ListValue: [StructValue([('code', 'def make_adk_tools(project: str, dataset: str = "sentiment_demo"):\r\n def lookup_user(user_id: int) -> dict:\r\n """Look up user information (email address) from BigQuery by user ID."""\r\n from google.cloud import bigquery\r\n\r\n client = bigquery.Client(project=project)\r\n query = (\r\n f"SELECT user_id, user_email "\r\n f"FROM `{project}.{dataset}.users` "\r\n f"WHERE user_id = @user_id"\r\n )\r\n job_config = bigquery.QueryJobConfig(\r\n query_parameters=[bigquery.ScalarQueryParameter("user_id", "INT64", user_id)]\r\n )\r\n try:\r\n results = list(client.query(query, job_config=job_config).result())\r\n if results:\r\n row = results[0]\r\n return {"user_id": row.user_id, "user_email": row.user_email}\r\n return {"error": f"No user found with user_id={user_id}"}\r\n except Exception as exc:\r\n return {"error": str(exc)}\r\n\r\n def lookup_orders(user_id: int) -> dict:\r\n """Look up a user\'s orders and current product inventory from BigQuery."""\r\n from google.cloud import bigquery\r\n\r\n client = bigquery.Client(project=project)\r\n query = (\r\n f"SELECT p.order_id, p.product_id, pr.remaining_inventory, pr.price "\r\n f"FROM `{project}.{dataset}.purchases` p "\r\n f"JOIN `{project}.{dataset}.products` pr ON p.product_id = pr.product_id "\r\n f"WHERE p.user_id = @user_id"\r\n )\r\n job_config = bigquery.QueryJobConfig(\r\n query_parameters=[bigquery.ScalarQueryParameter("user_id", "INT64", user_id)]\r\n )\r\n try:\r\n results = list(client.query(query, job_config=job_config).result())\r\n orders = [\r\n {\r\n "order_id": row.order_id,\r\n "product_id": row.product_id,\r\n "remaining_inventory": row.remaining_inventory,\r\n "price": float(row.price),\r\n }\r\n for row in results\r\n ]\r\n return {"orders": orders}\r\n except Exception as exc:\r\n return {"error": str(exc)}\r\n\r\n def send_email(to_address: str, subject: str, body: str) -> str:\r\n """Send a plain-text email to the customer via the Gmail API."""\r\n import google.auth\r\n import googleapiclient.discovery\r\n import email.mime.text\r\n import base64\r\n\r\n try:\r\n creds, _ = google.auth.default(\r\n scopes=["https://www.googleapis.com/auth/gmail.send"]\r\n )\r\n service = googleapiclient.discovery.build("gmail", "v1", credentials=creds)\r\n\r\n mime_msg = email.mime.text.MIMEText(body)\r\n mime_msg["to"] = to_address\r\n mime_msg["subject"] = subject\r\n raw = base64.urlsafe_b64encode(mime_msg.as_bytes()).decode("utf-8")\r\n service.users().messages().send(userId="me", body={"raw": raw}).execute()\r\n return "Email sent successfully"\r\n except Exception as exc:\r\n return f"Failed to send email: {exc}"\r\n\r\n return [lookup_user, lookup_orders, send_email]'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc48363d10>)])]>

We configure the LlmAgent and package it in the ADKAgentModelHandler:

code_block: <ListValue: [StructValue([('code', 'adk_agent = LlmAgent(\r\n name="remediation_agent",\r\n model="gemini-3.5-flash",\r\n instruction=(\r\n "You are a customer service remediation assistant with access to "\r\n "BigQuery lookup tools and an email sending tool. "\r\n "When given a prompt describing a customer situation, follow the "\r\n "numbered steps exactly and use your tools to complete the task."\r\n ),\r\n tools=adk_tools,\r\n)\r\n\r\n# RunInference handler for the ADK agent\r\nadk_handler = ADKAgentModelHandler(agent=adk_agent)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc4a4b9290>)])]>

3. Assembling the Dataflow DAG

The entire pipeline is declared cleanly. The upstream sentiment inference feeds directly into the filtering step (FilterNegativeADK), which then conditionally executes the downstream ADKInference:

code_block: <ListValue: [StructValue([('code', 'with beam.Pipeline(options=pipeline_options) as p:\r\n # 1. Read from Pub/Sub and classify sentiment on CPU\r\n sentiment_results = (\r\n p\r\n | "ReadFromPubSub" >> beam.io.ReadFromPubSub(topic=known_args.input_topic)\r\n | "DecodeMessages" >> beam.Map(lambda x: x.decode(\'utf-8\'))\r\n | "SentimentInference" >> RunInference(model_handler)\r\n )\r\n\r\n # 2. Filter out non-negative sentiment and invoke the ADK Agent\r\n _ = (\r\n sentiment_results\r\n | "FilterNegativeADK" >> beam.ParDo(FilterNegativeAndPromptADK())\r\n | "ADKInference" >> RunInference(adk_handler)\r\n | "LogADKResults" >> beam.ParDo(LogADKResponse())\r\n )'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc4a4b9850>)])]>

Cost and performance advantages

By introducing this filtering step, we gain major engineering and operational advantages:

1. Significant cost reductions

Instead of paying for Gemini input/output tokens on 100% of incoming events, we pay only for the fraction that represent negative customer sentiment (typically < 5% of messages). The other 95% are classified locally on CPU instances at zero incremental API cost.

2. High streaming throughput

Dataflow distributes the CPU classification workload across many instances. Since CPU inference takes milliseconds, the pipeline scales horizontally to handle high-throughput event streams. The heavyweight LLM agent, which can take seconds per request due to tool execution, is called sparingly, preventing backlog.

3. Native Apache Beam integration

Adding the agent into the DAG requires no complex orchestration logic or manual thread pools. Using ADKAgentModelHandler with Beam's native RunInference transform handles parallel worker threads, batching, and integration automatically, keeping the codebase maintainable and clean.

Key takeaways

Streaming data is fast and high-volume, while heavyweight generative AI reasoning is slow and costly.

By building a pre-filtered pipeline with Google Dataflow and the ADK, you get the best of both worlds: the cost and speed of local CPU-based models, and the deep, automated capabilities of Gemini-backed agents.

To see the complete codebase and deploy this yourself, check out the next-2026-demo GitHub repository.

_{Apache Beam is a trademark of the Apache Software Foundation}

Governance on autopilot, minus the turbulence

Tue, 18 Aug 2026 16:00:00 +0000

Every data team knows the moment. Someone opens a table, sees a column called cust_seg_flg, and has to go ask around to find out what it means, whether it's safe to use, and whether anyone has already answered that question in another dashboard three teams over. Multiply that by thousands of tables and views, and you get the real cost of governance debt: not a compliance failure, but a daily tax on every person trying to do honest work with your data.

Most governance tooling today is reactive. You scan for problems, you get a report, someone opens a ticket, and three weeks later a column gets a description. The Governance Agent project (built on Google Cloud Knowledge Catalog, BigQuery, and column-level lineage) takes a different starting point: if a table upstream is already documented, tagged, and trusted, why should every downstream view have to earn that trust from scratch, by hand, every time?

This post is about that shift, from governance as an audit you dread to governance that keeps itself current in the background.

The problem in plain terms

Data estates grow through pipelines. Raw tables get joined, filtered, and reshaped into views, and those views feed more views. Somewhere in that chain, the original context (what a column means, whether it's PII, what quality bar it's held to) tends to get lost. It just doesn't travel.

The result is a familiar pattern: a handful of gold tables are well governed because someone invested real time in them, and everything built downstream of them is progressively less documented, less tagged, and less trustworthy, even when the underlying data hasn't actually gotten worse. The governance quality of a table ends up depending on how long ago someone cared about it, not on how the data is actually being used today.

As data flows through a company, it gets combined, filtered, and reshaped for different teams to use. But somewhere along that journey, the important context—like what a piece of information means, whether it contains private details, or if it’s accurate—gets left behind. The metadata simply doesn't travel with the data through the progression of data assets within the ecosystem.

The result is a familiar pattern: a company will have a few perfectly documented "core" datasets because someone invested time in them, but everything built on top of them becomes a mystery. The data itself hasn't gone bad, but without the original context, people stop trusting it. Ultimately, data is only considered reliable if someone manually updated its metadata recently, rather than because of what it actually contains.

What the agent actually does

The core idea is straightforward: use column-level lineage to figure out where a column came from, and propagate the governance metadata that already exists upstream, rather than asking a human to re-derive it.

Concretely, it handles four things:

Descriptions. If transactions.customer_id has a clear description upstream, and a downstream view pulls that column through two or three hops of joins, the agent traces that lineage and proposes the same description downstream. When a column isn't a straight passthrough (it's a SUM(), a CASE WHEN, a COALESCE), the agent reads the actual SQL that generated it and writes a description that reflects the transformation, instead of copying an upstream description that no longer applies.

Business glossary terms. Technical column names rarely match the business language people actually use. The agent uses semantic similarity to map columns to a controlled glossary, and can also read unstructured documents (a PDF policy, a product spec, a markdown design doc) to find explicit definitions rather than guessing from column names alone.

Policy tags. This is the one that matters most for risk. If a column is tagged as PII upstream, the agent traces where that data flows and recommends the same tag downstream, along with a summary of who currently has read access and what masking rules apply. It also checks whether a transformation looks like a "straight pull" (the sensitive value passed through unchanged) versus something that's been aggregated or anonymized, so it isn't blindly stamping PII tags on data that no longer carries the risk.

Trust and data quality scores. Rather than treating every view as an unknown, the agent derives a trust score based on the Data Quality and Profiling results of its upstream sources, and gives credit when it detects that a transformation actually improved data quality (deduplication, null handling, and so on).

Every one of these runs through a confidence threshold before anything gets applied. The system is explicit about not inferring PII status or glossary mappings without solid grounding. If the evidence is weak, the propagation doesn't happen automatically. That's a deliberate design choice: the agent is meant to close obvious gaps quickly, not make judgment calls that a person should be making.

Why proactive is the right word, and not a stretch

Proactive governance doesn't mean predicting the future. It means the governance work happens as data moves, instead of waiting for a scheduled review or a compliance incident to trigger it. In practice, that shows up in three ways:

New views inherit context automatically, instead of starting undocumented and waiting for someone to notice.
Sensitive data is flagged as it flows, not discovered after it's already been queried by twelve people who didn't know they needed a masking policy.
Stewards spend their time on judgment calls, like ambiguous mappings or new glossary terms, instead of repetitive column-by-column tagging that a lineage graph could have told you.

None of this replaces a data steward. It changes what a data steward's day looks like: fewer hours spent typing descriptions into a UI, more hours spent deciding what should count as a business term or whether an edge case needs a policy exception.

When lineage runs out, bring your own context

Lineage is powerful, but it isn't complete. Plenty of tables have no clean upstream source to inherit from: a newly ingested dataset, a one-off import, a table that predates whatever lineage tracking you have in place. For those, the agent doesn't just shrug. It lets you point it at your own documents (a PDF policy, a product spec, a markdown design doc, even a spreadsheet or a screenshot of a data dictionary) and uses that as grounding instead.

There are three ways to feed it context, and which one you pick depends on the size of what you're handing over. For a short document, you can inject the full text directly into the prompt. For something long, like a fifty-page data classification policy, the agent chunks it, embeds it, and retrieves only the passage relevant to the specific column it's describing, so you're not paying to re-read the whole document for every field. And if your organization already has a proper document repository indexed in Vertex AI Search, the agent can query that directly instead of re-processing files every time.

The part worth calling out is how conservative the grounding is. This isn't "read the doc and take a guess." The instructions given to the model are explicit: if a column isn't clearly defined in the document you provided, it has to say so and stop, not fill in the gap with a plausible-sounding guess. That rule is strict for policy tags and glossary terms in particular. A column only gets marked as PII if the document says so in plain language, like an explicit "PII: Y" flag or a named sensitivity section. No inference from column names, no "this sounds like it might be sensitive."

That distinction matters more than it sounds like it should. A tool that infers PII status when it's uncertain is a tool that will eventually mask a column that didn't need it, or worse, wave through one that did. Making "I don't know" a valid answer is what makes the automation trustworthy enough to run without someone re-checking every single output.

Two signals, not one: lineage plus insights

Lineage is the primary signal, but it isn't the only one the agent listens to, and it's worth being precise about why. The Data Lineage API only knows what a job explicitly recorded. If a table was built through a well-instrumented pipeline, that's a clean, high-confidence trail. But plenty of real estates have gaps: a table that predates good lineage capture, a transformation that ran outside the tracked jobs, a relationship that technically exists but was never logged as such.

For those gaps, the agent has a second pass. It can trigger a Knowledge Catalog Data Documentation scan (an AI-driven analysis Gemini runs over a table or dataset) which infers relationships and column meaning even without a clean SQL trail behind them. Those inferred relationships get extracted and cached locally, then loaded into the same traversal engine that handles lineage, so both signals are checked together rather than living in separate systems a steward has to reconcile by hand.

The order matters. Standard lineage runs first, since it's grounded in an actual recorded job. The Insights pass runs second, filling in only what lineage didn't find, and anything it contributes is explicitly tagged as coming from that source rather than blended in silently. If you're auditing a propagated description or tag later, you can tell whether it came from a hard lineage link or an inferred one. There's a dedicated end-to-end flow for this path (trigger the scan, wait for it, extract the results, apply them) so it isn't a manual side-quest bolted onto the main workflow.

The practical effect: an incomplete or newly onboarded pipeline still gets useful propagation on day one, instead of waiting until lineage coverage catches up.

What it looks like day to day

The project ships with both a Gradio-based dashboard and a CLI, which matters more than it sounds like it should. A steward reviewing a handful of tables before a demo will want the dashboard: run a scan, see which tables have metadata gaps, preview a proposed description or tag, and approve it with a click. A platform team that wants this running as part of a nightly job or a CI/CD pipeline will use the CLI, scripting steward_cli scan, apply, and policy-propagate commands the same way they'd script any other pipeline step.

That dual interface reflects a real operational choice: governance tooling that only works from a UI never gets automated, and governance tooling that only works from a CLI never gets adopted by the people closest to the data.

Where this needs a human in the loop

Worth saying plainly: this is not a "set it and forget it" system, and it shouldn't be treated as one. Lineage confidence scoring can still get things wrong, especially across renamed columns or unusual joins. Semantic mismatch checks catch obvious errors (a date column shouldn't inherit a description from an id column) but they're heuristics, not guarantees. Every propagation is designed to be previewed before it's applied, and that preview step isn't a formality, it's the actual safety mechanism.

The honest pitch here isn't "governance without effort." It's "governance where the effort goes to the right five percent of decisions instead of the repetitive ninety-five percent."

Customer Testimonial

“As custodians of the VodafoneThree UK Datahub, one of our biggest challenges is that a significant proportion of our data estate remains undocumented or inconsistently labelled. This creates friction for data discovery, slows down delivery teams, and limits the value we can unlock from AI solutions built on top of our data. The Data Steward Agent changes that. By combining cataloguing, lineage, and automated metadata propagation, it enables us to focus governance effort where it adds the most value while automatically carrying trusted context downstream. Rather than manually reviewing thousands of tables, we can concentrate on governing source datasets and allow lineage to scale that knowledge across the platform. We estimate this approach can reduce cataloguing effort by up to 75%, while significantly improving data discoverability, trust, and AI readiness across the UK Datahub.” - Radina-Paola Ivanova, GenAI Engineer, VodafoneThree UK Datahub

The takeaway

Governance debt compounds the same way technical debt does: quietly, until someone downstream hits it at the worst possible time. The value of an approach like this isn't that it makes governance disappear as a concern. It's that it moves the work upstream, literally, so that context and controls travel with the data instead of being reconstructed from scratch every time someone builds a new view.

For teams sitting on years of undocumented BigQuery estates, that's not a nice-to-have. It's the difference between governance being something you catch up on twice a year, and governance being something that just keeps pace with how fast your data actually moves.

Using BigQuery Graphs with measures for trusted agentic workloads

Thu, 13 Aug 2026 17:00:00 +0000

When enterprises transition from using simple chat assistants to autonomous, agentic workloads, they quickly run into a hard truth: Agents are prone to inaccurate insights when working with directly raw tables.

BigQuery Graph helps organizations move beyond flat, static tables to represent enterprises exactly how they exist in the physical world: as interconnected business entities with real-world dependencies. With the support of measures in BigQuery Graph (preview), we are unifying governed metrics with relationship mapping. This allows your agents to reason across complex dependencies captured in graphs with precision of measures.

Why relationships matter

Traditional data structures are blind to multi-hop business context, causing AI agents to make incorrect operational decisions:

The concrete problem: If a retailer has an agent who is asked why winter jacket sales dropped 12% in Seattle, it can query flat tables to report the what (the 12% dip). But it fails at the why because it cannot trace the relational path: Seattle orders ➔ distribution centers ➔ suppliers delayed by regional storms.
The risk of disjointed systems: Lacking relationship context, the agent suggests an irrelevant 15% markdown campaign, needlessly eroding margins. Furthermore, maintaining separate systems - where one team maps supplier relationships in a separate graph database while another maintains SQL metrics - forces your agent to stitch these stacks together at runtime. This process is slow, expensive, and leads to inconsistent KPI calculations.

Measures in BigQuery Graph solves this by letting you map existing tables to a property graph in-place with zero ETL. This unified setup enables a logical evolution of inquiry:

Metadata grounding establishes what data you have.
Business metrics (measures) calculate how your business performed.
Relationship mapping (graph) uncovers why it happened.

Under the hood

Historically, standard SQL joins during graph traversals duplicate rows, leading to incorrect aggregation calculations. BigQuery Graph solves this natively.

Data modelers define a MEASURE (like SUM or AVG) directly within the Property Graph DDL. Using standard SQL via the GRAPH_EXPAND function and the AGG aggregator, the engine resolves the structural graph paths before evaluating metrics. This ensures your agent is smart enough to know when it needs a calculator (SQL) and when it needs a map (graph).

Because public projects like bigquery-public-data are strictly read-only, you must map the logical property graph inside your own project using a placeholder variable (YOUR_PROJECT_ID), while directly referencing the read-only public tables as nodes and edges.

code_block: <ListValue: [StructValue([('code', '-- 1. Map the graph inside YOUR project \r\n\r\n\r\nCREATE OR REPLACE PROPERTY GRAPH `YOUR_PROJECT_ID.YOUR_DATASET.thelook_ecommerce_graph`\r\nNODE TABLES(\r\n `bigquery-public-data.thelook_ecommerce.users` AS User\r\n KEY(id)\r\n LABEL User PROPERTIES(id, city, country),\r\n `bigquery-public-data.thelook_ecommerce.orders` AS Order\r\n KEY(order_id)\r\n LABEL Order PROPERTIES(\r\n order_id, \r\n MEASURE(AVG(num_of_item)) AS avg_items_per_order,\r\n MEASURE(SUM(num_of_item)) AS total_items\r\n )\r\n)\r\nEDGE TABLES(\r\n `bigquery-public-data.thelook_ecommerce.orders` AS OrderedBy\r\n SOURCE KEY(order_id) REFERENCES Order(order_id)\r\n DESTINATION KEY(user_id) REFERENCES User(id)\r\n LABEL ORDERED_BY\r\n);\r\n\r\n-- 2. Query your new graph with standard SQL—using standard {Label}_{Property} column outputs\r\nSELECT\r\n User_city AS city,\r\n ROUND(AGG(Order_avg_items_per_order), 2) AS agg_avg_items,\r\n ROUND(AGG(Order_total_items), 2) AS agg_total_items\r\nFROM GRAPH_EXPAND("YOUR_PROJECT_ID.YOUR_DATASET.thelook_ecommerce_graph")\r\nGROUP BY User_city\r\nORDER BY agg_total_items DESC\r\nLIMIT 10;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc48363990>)])]>

Democratizing graph intelligence in BigQuery Studio

To make managing and deploying these relationship networks frictionless for both developers and business users, we have built native, intuitive operational tools directly into BigQuery Studio:

Visual graph modeler: A no-code, drag-and-drop interface inside BigQuery Studio that lets you visually build, edit, and map property graphs, nodes, and edges without writing complex DDL scripts manually.

Conversational Analytics (CA) integration: Users can interact with the graph naturally. Instead of guessing table joins, Conversational Analytics agents navigate the deterministic, relationship-aware map of the graph, converting natural language questions into precise, boundary-constrained GoogleSQL or ISO GQL queries. This prevents model hallucinations and enforces semantic consistency.

Unified semantics: Native Looker integration

To avoid maintaining fragmented logic stacks, business metrics must live at the data layer. By integrating Looker (LookML) natively with BigQuery Graphs as in-database analytic models, you define logic once at the core:

Database-managed models (sql_analytic_model_name): Point Looker directly to your database-defined BigQuery Graph using sql_analytic_model_name to map standard LookML dimensions and measures directly to your graph properties.
Looker-managed models (derived_analytic_model): Define your BigQuery Graph schema directly inside your LookML view using derived_analytic_model. Looker will dynamically generate and execute the SQL DDL statements to maintain the graph inside BigQuery.
Enterprise DevOps workflows: Manage your graph's entire lifecycle using the Looker IDE, Git-based version control, and Continuous Integration (CI). Core KPIs (like Churn Rate) remain completely identical, verified, and trusted.

Looker’s semantic layer governs Gemini Enterprise data for user trust

Tue, 11 Aug 2026 16:00:00 +0000

For organizations deploying AI agents at scale, there’s often a critical divide between structured and unstructured data. While large language models (LLMs) excel at parsing text documents, emails, and PDFs, they can struggle when presented with raw enterprise databases. Meanwhile, standard natural-language-to-SQL (NL2SQL) models often guess how database schemas fit together, which can lead to unpredictable queries, inconsistent metrics, and AI hallucinations that erode user trust.

Gemini Enterprise brings the best of Google AI to every employee through an intuitive chat interface that acts as a single front door for AI in the workplace. And now, Looker’s governed semantic layer serves as the trusted foundation for structured data within Gemini Enterprise, enabling trusted self-service business intelligence for all Gemini Enterprise users. With this integration, Looker analysts and admins can publish conversational agents natively into their Gemini Enterprise environments via the Agent-to-Agent (A2A) protocol. Now, organizations can provide their AI-accelerated taskforce with robust and trusted tools, powered by real-time analytics, that they can explore in natural language in addition to their daily workspace workflows. Making it easy to offer conversational agents in Gemini Enterprise expands discoverability and promotes a data-driven culture, while reducing friction to adoption.

Bringing a semantic foundation to structured and unstructured data

By combining Looker’s semantic layer with Gemini Enterprise, you can query both structured databases and unstructured documents in plain English, all in one place. Instead of jumping between dashboards and other tools to understand your numbers, teams can instantly connect hard metrics with real-world context to solve problems and make decisions faster.

Publishing Looker agents for consumption in Gemini Enterprise

Minimize AI hallucinations

If you ask the typical AI chatbot to calculate "revenue" or "churn rate" against an unstructured cloud database, it has to guess which tables to join, which filters to apply, and which timestamps to trust. This can result in different people asking the same question, only to get completely different answers.

Looker’s semantic layer eliminates this guesswork, serving critical context to Gemini Enterprise in the form of codified data, allowing the agent to give deterministic, predictable responses.

code_block: <ListValue: [StructValue([('code', '[ Gemini Enterprise Chat UI ] \r\n │\r\n (A2A Protocol / NLP)\r\n ▼\r\n [ Looker Governed Agent ] ──► Generates Deterministic SQL\r\n │\r\n [ Looker Semantic Layer ] ──► Business-Approved Definitions & Logic\r\n │\r\n ▼\r\n [ Enterprise Data Cloud ] ──► (BigQuery, AlloyDB, Spanner, etc.)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7efc483c1c50>)])]>

When a Gemini Enterprise user requests a business KPI in Gemini Enterprise, the request is routed directly to a Looker agent. The semantic layer generates deterministic, precise SQL based on version-controlled business logic. This helps ensure when an executive asks for "Revenue," they get the exact, governed enterprise metric — not a guess.

Robust governance and secure access management

Data governance and security are critical when introducing AI to enterprise data warehouses. Organizations can’t risk corporate information being loosely ingested, indexed, or exposed outside of strict permissions.

Looker’s integration with Gemini Enterprise is built on a zero-risk pass-through architecture, processing the data, but not writing to persistent storage. Gemini Enterprise does not ingest, replicate, or store your underlying database records. Instead, the integration operates safely and securely over the A2A protocol, following these core tenets:

OAuth authorization: In order to interact with a Looker agent within Gemini Enterprise, end users provide a secure, one-time OAuth consent. This binds their Gemini session to their specific Looker credentials.
Strong governance enforcement: Because the architecture relies on live pass-through queries, Looker’s existing row-level and column-level access controls are maintained.
Strict security isolation: If a user does not have permission to view, say, sensitive regional payroll or financial rows within the Looker platform, the Looker agent actively restricts that data in the Gemini environment. Should an agent be published to the Agent Gallery to simplify discovery, it still does not bypass the security controls that you established.

Technical capabilities and enterprise readiness

Deploying Looker agents natively into Gemini Enterprise via the A2A protocol doesn't just make it smarter — it makes it more interactive and interoperable, without sacrificing security. Here are some of the features you’ll find in this release.

Rich visual interactivity: support for charts

They say a picture is worth a thousand words. When users interact with Looker agents inside Gemini Enterprise, the platform goes beyond textual explanations and provides native, interactive data charts. If a user asks for a visual trend—such as monthly sales performance or regional distribution—the Looker agent maps the database response with rich, presentation-ready visualizations directly inside the universal chat box.

Note: If you published Looker agents in Gemini Enterprise prior to Looker release 26.12, we recommend updating or refreshing them to take advantage of these enhanced visualization capabilities.

Interoperability with first- and third-party agents

Looker agents published to Gemini Enterprise can understand context across different agents and data sources. Leveraging standard communication frameworks, these agents can securely share structured, governed insights with other first-party Google Cloud agents like the Deep Research Agent or external third-party agents to create structured workflows. This enables complex multi-agent orchestration, where an enterprise operational agent can pull data from a Looker agent to feed into a separate productivity or supply-chain workflow.

Looker-based user authentication

To preserve enterprise governance, this integration implements a robust, identity-centric authentication model. Users are required to provide a one-time OAuth consent, binding their active Gemini Enterprise session securely to their underlying Looker credentials. This helps ensure that every conversational query hitting your databases is authenticated at the user level, enforcing pre-existing Looker permission structures, row-level data access filters, and column-level masking rules — no exceptions.

Trusted data in Gemini Enterprise

The future of work is agentic. Gemini Enterprise provides a single, secure architecture to deploy a global digital task force,empowering your business with the best of Google AI for developers, employees, and customers.

The integration of Looker with Gemini Enterprise not only brings trusted data analytics to business users but also adds rich interactivity, visual charts, and data storytelling directly into their everyday workspace. As business users embrace this agentic new way of working, they aren't just getting text answers; they are getting presentation-ready visualizations that bring operational metrics to life and deliver complex insights.

To get started, learn how to publish your data agents in Gemini Enterprise to make your agent’s predefined context and analytics available to your entire organization.

How Malachyte solves retail’s cold-start problem with managed real-time AI

Mon, 10 Aug 2026 16:00:00 +0000

What’s the best way to recommend products to little-known users?

We’ve spent our careers trying to solve this problem for major companies like Spotify and Priceline, and it’s why Sidd founded Malachyte, an AI-powered ecommerce recommendation platform. These days, consumers have come to expect content that feels personalized and relevant, and online services competing for their attention have no choice but to do this exceptionally well.

Malachyte was inspired by some unique insights into how advanced AI models, and large language models in particular, could be applied in new ways to old challenges like personalization and recommendations.

As Malachyte set out to win potential customers’ business, we needed secure, scalable, reliable and, above all, leading-edge AI infrastructure to continue building the personalization algorithm we had always envisioned. By utilizing Google Cloud tools like Bigtable and Managed Service for Apache Kafka, Malachyte has been able to help some of its retailers double and sometimes even triple their sales.

This is the story of how we built it, and the ways any founder can use services like these to start deploying AI foundation models in new ways.

How Malachyte lifted sales for their users

For Malachyte, the aha moment was discovering that it could use neural networks with attention mechanisms — the same concept powering large language models — to personalize retail search and product pages. This approach is what enables LLMs to derive meaning from the relative order of items in a sequence, in their case the order of words and syllables in a sentence. When it comes to a retail website or app, what Malachyte wanted to capture was the sequence of customer interactions with the site.

What if we predicted the next thing a user wants on an ecommerce website just like LLMs predict the next word in a sentence?

A pre-GPT language model might have tried to look at a specific sequence of words or even fragments of words (what we now know of as tokens), but those earlier models wouldn’t examine what happens if the words were in the comparable order but weren’t contiguous or were re-arranged. The breakthrough came — in part through Google’s work on transformers — when LLMs gained the ability to understand complex and long-range dependencies within a sequence of items.

This more sophisticated method has delivered dramatic results — both for the proliferation of gen AI in general, and for Malachyte’s application of the technology.

To make this work in practice, Malachyte creates a vector of everything known about a visitor when they arrive on a site. Most users are visiting for the first time, so little is known about them. This is what’s known as the “cold start” problem. The trick is to use every interaction with a user to refine this vector. Each new addition to the vector, like a click or a query, does two things: it drives a prediction about the next thing the user wants, and it provides more information about the user.

Malachyte’s platform then updates the user vector and the prediction at the same time. This not only enhances the understanding of the individual user and their preferences, it also improves the overall model with the anonymized user data. With every inference, the context of both the average and the specific shopper grows.

The company further innovates by not just using attention-based neural networks but combining that with updating user profiles 100 milliseconds at time.

Malachyte’s recommendation and search agents populate the next page’s search results or recommendation carousels based on what users clicked on previous pages.

To be sure, this idea isn’t in itself new. Retailers have long used collaborative filtering recommendations systems to identify similar users and items that required massive sets of interaction history. These models typically required a lot of data, including third-party cookie-based profiles and demographics.

By focusing on the sequence of interactions in a session, retailers can achieve far more personalization — with less required data or spend — than by focusing only on a user’s profile. As a bonus, retailers can now offer their users more privacy by not relying on long-term cookie data.

This works because of the model structure and multimodal vectors that encode everything they know about a user, including browser data, click history and searches. The output, too, is multimodal: The same model can be applied to on-site search product pages, category pages, and add-to-cart carousels.

To make this work, each product in the catalog is embedded into the same space as the user vector, which gets updated and subsequently moves the vector closer to relevant products and further from those that aren’t. The neural network computing the embedding is being continuously trained across retailers who work with Malachyte, improving the quality for everyone. The system effectively becomes a data cooperative with each retailer's user helping make the model smarter for everyone.

A user session represented as a vector in a space of products.

To make this delivery for every user at every inference in 100 milliseconds, Malachite found real benefits in building onGoogle Cloud’s real-time AI stack.

With this system, every behavioral event streams into a Managed Service for Apache Kafka cluster. Rather than queuing for a future training job, each event immediately becomes an update to the user’s profile in Bigtable. The Kafka cluster allows the customer’s front-end to persist, so the user session signals quickly with little worry about how they fit into the user vector.

Bigtable allows Malachyte’s services to look up and update the right user vectors, and it and Kafka operate at the order of 10 milliseconds per step, which allows the entire recommendation loop to complete with no disruption to the user experience.

The three layer real-time AI architecture: a retailer’s website, Malachyte’s AI models and serving front ends, and context management infrastructure.

In addition to a fast core, a second layer of product catalog updates, inventory signals, and retailer-specific dimensional data keeps product data up to date. This flows through Cloud Pub/Sub, which offers globally accessible REST APIs that enable connections retailers can use without deep integration work. Malachyte agents run on Google Kubernetes Engine (GKE), with model inference on Google Compute Engine (GCE).

Continuous ingestion of external data, such as product catalog updates, operates through Pub/Sub’s global messaging system.

With its migration to Google Cloud’s AI architecture, Malachyte demonstrated that production AI inference and training are about more than GPUs and storage. They require real-time continuous learning infrastructure that includes a fast key-value store, a streaming layer, and a managed messaging system, all integrated with the foundation model architecture.

This approach also shows that even a small team like Malachyte’s can have a big impact in an industry. It just needs access to powerful infrastructure and core AI managed services.

Try it for yourself

Looking to shake up your industry or stay ahead of the competition like Malachyte? Try Managed Service for Apache Kafka, Cloud Pub/Sub, and Bigtable. New customers can receive $300 in Google Cloud credits.

Data Analytics

The future of orchestration: Pine59’s journey to Airflow 3 on Google Cloud

Proactive modernization for growth

Orchestrating advanced MLOps

Supporting developers with custom extensibility

Faster, more reliable pipelines

What’s new with Google Data Cloud

September 14 - September 17

September 7 - September 10

August 31 - September 4

July 6 - July 10

June 1 - June 5

May 11 - May 15

April 20 - April 24

April 13 - April 17

April 6 - April 10

March 23 - March 27

March 16 - March 20

February 23 - February 27

February 16 - February 20

February 2 - February 6

January 26 - January 30

January 19 - January 23

January 12 - January 16

What’s new with Google Data Cloud - 2025

Scaling Telco Autonomy: Leveraging GNNs with Distributed GraphFlow

Foundation: Digital Twin on Spanner Graph

The ML layer: Distributed Graph Flow (DGF)

Use cases

Scenario: Root cause analysis with GNNs and DGF

Get started today

Agent-ready analytics: Unlocking insights with BigQuery augmented analytics

A step-by-step example of chaining insights

Step 1: Detect change points

Step 2: Key drivers attribution

Step 3: Causal effect

Connecting augmented analytics to Conversational Analytics

Example 1: Chicago taxi trips

Example 2: Iowa liquor dataset

Take the next step

Announcing Pause/Resume and NVIDIA RTX PRO 6000 Blackwell GPU support in Dataflow

Agentic analytics with the Data Agent Kit

Data Agent Kit

Finding out what happened

Checking its work

Breaking down the numbers

Crossing into Cloud SQL

Dropping into the terminal

Updating the director

Build a reproducible pipeline

Wrap up

Try it yourself

How Yahoo optimizes resources with flexible VMs in Managed Service for Apache Spark

Operational trade-offs of static configurations

Rules for configuring flexible clusters

Two ways flexible VMs support massive workloads

gcloud example

API example

Establishing an infrastructure policy

Yahoo impact and results

Strategic benefits of flexible infrastructure

Build your resilient data pipeline

Simplify pipelines with new BigQuery identity columns

Key benefits for your data pipelines

How to implement identity columns

Get started today

Introducing TabFM in BigQuery: Predictive analytics reimagined

The leading model for tabular predictions

Getting started with TabFM in BigQuery

TabFM in BigQuery under the hood

Choosing the right tool for the job

Predictive machine learning made easy

BigQuery Graph is now GA: the knowledge foundation for the agentic era

What’s new in BigQuery Graph

A faster, broader graph engine

Borderless graph Lakehouse

Faster and more expressive GQL

Built for the agentic era

Chat with your graphs

Build a graph with an agent