Data Analytics

Querying Databricks Apache Iceberg Tables with the borderless Lakehouse

Thu, 29 Oct 2026 16:00:00 +0000

Enterprise organizations often leverage multiple data platforms, including Databricks, to execute heavy Apache Spark jobs alongside BigQuery to power highly concurrent analytics workloads. Historically, bridging these systems often meant building complex pipelines to copy data between environments. This approach increases storage costs, introduces latency, and fragments security controls.

With Google Cloud’s borderless Lakehouse, you can now query Databricks-managed Apache Iceberg tables straight from BigQuery. This zero-copy approach removes the need to duplicate data: BigQuery reads the same files Databricks writes, and new changes appear after Lakehouse's next scheduled metadata refresh (as often as every 5 minutes).

How Lakehouse and Databricks work together

Managed Iceberg tables automatically generates Iceberg-compatible metadata alongside native Delta logs. Data engineering teams can store Parquet files once in Google Cloud Storage (GCS) using Databricks. Rather than loading this information into BigQuery, Lakehouse reads the Iceberg metadata directly, applying BigQuery’s vectorized engine directly to open-format files in GCS.

Although Google Cloud Storage (GCS) serves as the primary storage layer in the above example, Lakehouse extends this borderless federation directly to external cloud environments like Amazon S3 and Azure Blob Storage. By pairing the open Apache Iceberg REST Catalog with cross-cloud connections, Lakehouse lets BigQuery query data across clouds without moving it. Each federated catalog lives in a Google Cloud location you choose, ideally the region closest to your data. This lets BigQuery read open-format Parquet files where they live, delivering multi-cloud analytics without replication pipelines. Remote-cloud egress charges still apply, but Cross-Cloud Interconnect (AWS) and Lakehouse caching can reduce them.

Beyond simply querying external data, Lakehouse framework establishes true read and write (R/W) interoperability across engines. Because the platform is built on an open Apache Iceberg foundation, data teams can effortlessly execute multi-statement SQL commands directly inside BigQuery.

Centralized security and governance

Handling access controls across disparate systems can introduce vulnerabilities. Lakehouse addresses this through delegated access. Administrators can configure a BigQuery Cloud Resource Connection tied to a service account that manages IAM rights for GCS buckets. Granting users access to the Lakehouse table entity lets you enforce BigQuery controls including row-level, column-level security, and dynamic data masking on external Iceberg tables while Databricks handles ingestion.

Catalog federation and credential vending

Previously, connecting these platforms meant manually configuring BigQuery queries to point at static metadata paths. Modern setups leverage the open Apache Iceberg REST Catalog API. Databricks Unity Catalog functions as the central metadata layer, allowing a BigQuery administrator to establish a Lakehouse runtime catalog with a federated connection. When running a query, Lakehouse contacts Unity Catalog for current table state, and short-lived tokens are issued through credential vending to grant BigQuery temporary read access to GCS.

Partition pruning

BigQuery uses each table's Iceberg metadata to plan the query, then reads only the relevant data files in place. Iceberg's hidden partitioning lets you filter on logical columns, such as timestamps, without knowing the directory layout. Note that Databricks managed Iceberg tables don't support partition transforms; Databricks recommends liquid clustering instead.

Filters in a standard WHERE clause let BigQuery use the Iceberg manifests to skip partitions that can't match. Compare partition columns with literal values so pruning can apply. These manifests maintain detailed column-level statistical bounds, enabling BigQuery to bypass irrelevant manifest records and unneeded Parquet files immediately. Consequently, queries scan only necessary byte ranges, accelerating query execution while significantly reducing on-demand slot consumption and storage scan costs.

Fast queries with Iceberg metadata and Lakehouse caching

Iceberg manifests already identify which data files to read, so BigQuery doesn't need a metadata cache for Iceberg tables. For cross-cloud tables, such as those in Amazon S3, Lakehouse caching (Preview) keeps retrieved data blocks in Google Cloud so repeat queries read them locally. BigQuery reads the Iceberg manifest files directly when it plans each query; there's no metadata caching to enable. This dynamic partition pruning bypasses unneeded files, significantly reducing runtime and bytes scanned.

Maintenance and lifecycle management

Modern lakehouses demand active upkeep; without routine care, performance degrades into an inefficient repository. Fortunately, both BigQuery and Databricks supply native capabilities to preserve table hygiene and accelerate analytical workloads.

Keeping your tables optimized

To maintain optimal query performance, active maintenance of your lakehouse storage footprint remains necessary.

Trigger regular compaction pipelines via Databricks APIs to merge tiny Parquet files into larger, optimized layouts and use Predictive Optimization.
Execute the Databricks VACUUM command to clear unreferenced files outside active snapshots, avoiding extra GCS storage costs.

Compaction and the small file problem

Frequent write operations, continuous ingestion jobs, and streaming workloads continuously produce thousands of small Parquet files. This fragmentation creates a storage bottleneck that can limit BigQuery’s reading efficiency. Establishing thousands of network connections to GCS and processing metadata footers across numerous tiny files consumes significant compute time, diminishing the performance advantages of vectorized query execution.

Running routine compaction within Iceberg resolves these inefficiencies. Databricks manages this automatically using Liquid Clustering which dynamically adjusts file layouts based on access patterns or through native table management utilities.

To compact small files manually on Databricks, run OPTIMIZE:

code_block: <ListValue: [StructValue([('code', "-- Run on Databricks to compact small files\r\n-- On Databricks:\r\nOPTIMIZE customer_events;\r\n-- For Iceberg tables you manage outside Databricks:\r\nCALL my_catalog.system.rewrite_data_files(\r\ntable => 'default.customer_events',\r\nstrategy => 'binpack',\r\noptions => map(\r\n'target-file-size-bytes',\r\n'268435456',\r\n'min-file-size-bytes',\r\n'67108864'\r\n)\r\n);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb8a8a6e650>)])]>

Executing this command aggregates scattered file segments into larger, structured blocks typically between 256MB and 512MB that can help improve BigQuery’s scan throughput. The binpack approach operates rapidly by prioritizing total file size balancing, though alternative sorting strategies can organize dataset ordering along specific column keys.

Managing orphan files and snapshot retention (VACUUM)

Every transactional update, record modification, and table compaction creates fresh snapshots within Iceberg to preserve dataset integrity. Over time, historical Parquet objects detached from active table states become orphaned. Although these legacy files facilitate time-travel analytics, neglecting them can drive up GCS storage charges over time.

Data teams can use the Databricks VACUUM operation to scan and purge abandoned storage files.

code_block: <ListValue: [StructValue([('code', "ALTER TABLE customer_events SET TBLPROPERTIES ('delta.deletedFileRetentionDuration' = '7 days');\r\nVACUUM customer_events FULL;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb8a9195490>)])]>

Standard configurations apply a mandatory seven-day threshold (deletedFileRetentionDuration) before executing VACUUM routines. Preserving this delay prevents active analytical jobs from encountering missing file errors while keeping historical recovery options available. Scheduling routine VACUUM jobs, or letting predictive optimization run them, removes unreferenced data files and, for tables with Iceberg reads, old Iceberg metadata. That keeps storage costs low and performance high across your lakehouse.

Conclusion

Connecting Databricks Apache Iceberg datasets to Lakehouse can help you activate your data with BigQuery. By separating processing from underlying storage, embracing open formats, and using standard REST APIs with credential vending, you can run scalable engineering workloads and concurrent analytics without migrating data or sacrificing performance

Pairing Databricks Iceberg reads (UniForm) with Lakehouse federation, Lakehouse caching, and regular OPTIMIZE and VACUUM maintenance helps you build an efficient, zero-copy lakehouse. This approach can help lower storage spend, avoids fragile pipeline maintenance, and prepares your data environment for modern analytics and emerging AI workloads.

Constructing a unified architecture no longer requires moving data files across platforms. You can begin querying Databricks Iceberg tables by registering them with the Lakehouse runtime catalog today for bi-directional federation. Learn more by reading our documentation to get started.

Grounding agents in one context graph connecting semantic models you already trust

Thu, 29 Oct 2026 16:00:00 +0000

Picture the scene: Before an executive business review, your VP of Sales asks an AI assistant, "What was net revenue for our top enterprise accounts last quarter?" In seconds, the agent returns a polished chart! Unfortunately, the numbers don't match the finance report. Why? Because the agent counted cancelled orders as revenue and pulled from an unvalidated staging table.

The agent didn't fail because of the model, or because it couldn't translate a user question into a query. It failed because it didn't understand your business. The real kicker is that understanding usually already exists in the form of a semantic model in a Looker, dbt, Snowflake, Databricks, or SAP system. But because each definition stays locked inside its own tool, every agent guesses at it… and often gets it wrong.

This is where Knowledge Catalog comes in. It brings the semantic models from all of those tools into one context graph, so every agent works from the same trusted definitions. And when two tools define the same metric differently, both definitions appear with their provenance, so data stewards can reconcile the conflict before it reaches end users.

Today, we're announcing the general availability of expanded semantic model support. Knowledge Catalog now natively ingests semantic models and metric definitions from Looker LookML, BigQuery Graph (including BigQuery measures), dbt MetricFlow, Snowflake Semantic Views, Databricks Metric Views, and SAP Core Schema Notation (CSN), and connects them with schema, lineage, data profiles, business glossary and other key metadata in a single context graph, without moving data or rewriting business logic in a new language.

By grounding your AI agents in those models, they get the same business definitions your analysts rely on, so their answers line up with the numbers your business trusts. In fact, based on internal benchmarks, grounding agents in Knowledge Catalog makes them nearly 4X more accurate, with up to 33% lower latency, and using up to 75% fewer tokens.

The unified context graph enables you to:

Reuse the semantic models you already trust: Knowledge Catalog grounds Gemini agent, BigQuery Conversational Analytics, and your custom agents in the metrics, filters, and join paths your teams have already defined, wherever they live.
Go beyond hand-curated definitions with self-learned context: Knowledge Catalog pairs the well-defined metrics and KPI definitions in your semantic models with relevant context it extracts from unstructured knowledge sources, such as operational playbooks, meeting notes, chat threads, code, and more. Powered by Gemini, self-learning enrichment continuously finds missing or outdated context and turns it into governed, reusable context sets, ensuring your agents' context stays current as definitions change, without relying only on manual curation.
Pair business logic with enterprise trust: Even the right formula produces the wrong answer when it runs on stale or uncertified data. That's why Knowledge Catalog can connect semantic definition to data quality scores, freshness signals, and lineage, so an agent knows what it can trust before it writes a line of SQL.

Side-by-side comparison of the same question in Gemini without Knowledge Catalog (left) and grounded in Knowledge Catalog (right). Highlights the grounded agent (right) giving an accurate response with less token and lower latency, compared to the agent without Knowledge Catalog (left) hallucinating and using more tokens with higher latency.

One context graph across your data estate

The context graph that Knowledge Catalog creates is a connected map of your data and semantic estate: tables, semantic models, glossary terms, and data products are nodes; lineage, associations, schema keys, joins mined from real query history, and the join paths declared in your semantic models are the edges. Ingested semantics land in that same graph, next to pre-existing governance and usage signals, so an agent moves from business concept to trusted formula and verified join paths in one traversal. Each source adds its own perspective on your business:

Looker LookML (GA): Indexes models, explores, views, and dashboards using Live sync from Looker Core, capturing compiled metric SQL, fan-out-safe join conditions, mandatory filters, and lineage from source tables to executive dashboards.
BigQuery Graph (GA): Indexes native property-graph definitions, node and edge tables, graph measures, and business synonyms, so agents reason over connected entities with Graph Query Language (GQL) traversals instead of brittle multi-table self-joins.
dbt MetricFlow (GA): Links source objects to dbt models and MetricFlow metrics, alongside dbt data quality test results, filters, saved queries, freshness SLAs, and downstream exposures.
Snowflake Semantic Views via Horizon Catalog (GA): Pairs federated BigQuery or Google Cloud Lakehouse tables with Snowflake Horizon metadata, bringing in Semantic View dimensions, facts, metric formulas, relationships, and synonyms, plus Data Metric Function quality scorecards and PII masking tags.
Databricks Metric Views via Unity Catalog (GA): Ingests metric view definitions, measure formulas, scoping filters, and join cardinalities from Unity Catalog, alongside table certification badges and Lakehouse Monitoring profiling results.
SAP Core Schema Notation (CSN) via SAP Business Data Cloud (BDC) (GA): Captures business labels, dimension and cube modeling patterns, default aggregations, currency and unit bindings, key and composite join associations, and data sensitivity flags when SAP BDC Data Products are federated through Google Cloud Lakehouse.

Because every source lands as a node or an edge in the same graph, a LookML join, a dbt exposure, a Snowflake relationship, or an SAP association are traversable side by side, with links tying each one back to the physical tables it depends on. And nothing is silently overwritten: overlapping definitions stay side by side, each labeled with its source, giving data stewards the full picture they need to settle on a single version.

Navigating semantic models, join paths, and lineage on the context graph in Knowledge Studio.

You can explore and govern all of this in Knowledge Studio (preview), your command center to explore and manage your context. Explore your context graph, including your semantic definitions, using natural language prompts and traverse connection paths using the context graph canvas. Then, once you've validated the right assets and definitions, you can package them into reusable context sets that map to specific use cases and domains in your organization, which Gemini and custom agents can consume in production.

Get started

Let’s get back to that net revenue question. With Knowledge Catalog, the VP of Sales’ agent issues a single context query to discover the trusted asset, applying the exact metric formula and filters that your domain experts defined, all while respecting join keys across systems, and confirming that the underlying data passed its quality checks. Better yet, the numbers in the report are correct!

To get started, explore your context graph in Knowledge Studio and visit the Knowledge Catalog documentation to connect to more sources.

What’s new with Google Data Cloud

Thu, 29 Oct 2026 16:00:00 +0000

October 19 - October 23

BigQuery introduces column aliases in GoogleSQL and cross-region table copy GA
GoogleSQL in BigQuery now supports column aliases in SELECT and WHERE clauses (GA opt-in), avoiding nested subqueries and simplifying migrations from Snowflake and Teradata. Cross-region table copying is also now generally available to streamline regional data replication and disaster recovery. Review the BigQuery release notes.
Conversational analytics in BigQuery expands to Canada and adds CSV/PDF export
Conversational analytics in BigQuery is now available in Montréal (northamerica-northeast1) and Toronto (northamerica-northeast2) to support Canadian data residency requirements. You can also now export agent responses — including text, tables, charts, and citations — directly to CSV or PDF (GA). Learn more about conversational analytics in BigQuery.
AlloyDB Omni 18.4.0 is now GA with Advanced Query Insights and external KMS support
AlloyDB Omni 18.4.0 brings PostgreSQL 18.4 to self-hosted and multi-cloud environments, adding the google_insights_v2 extension for granular query execution and wait-event diagnostics. It also adds Thales CipherTrust Manager support for Transparent Data Encryption (TDE) alongside HashiCorp Vault. Explore the AlloyDB Omni release notes.
Cloud SQL for PostgreSQL adds managed connection pooling and Private Service Connect defaults
Cloud SQL for PostgreSQL now supports enabling built-in managed connection pooling on existing Private Service Connect (PSC) instances to scale high-concurrency workloads. PSC is also now the default private IP option for new Cloud SQL instances in new projects to simplify cross-project networking. Review Cloud SQL updates.
Customer momentum: Unico prevents over $5B in fraud on Google's Agentic Data Cloud
Digital identity platform Unico migrated from Databricks Spark/Scala clusters to BigQuery and Spanner at the core of its data architecture. Unifying real-time transaction streams with petabyte-scale analytics enabled low-latency identity verification that has prevented over $5 billion in fraud. Read the Unico case study.
Developer guide: Getting started with Looker Data Apps
Following the debut of Looker Data Apps in preview, a new walkthrough on the Google Developer Community shares practical patterns for building prompt-driven applications inside Looker. Read the guide on the Google Developer Community.

October 12 - October 16

Looker Data Apps are now in Preview to build and host interactive, agent-generated applications natively inside Looker
Generated using natural language prompts, the Antigravity CLI, or Claude Code, these interactive web applications, natively hosted inside Looker, transform business reporting into operational systems of action. Users can test scenarios, adjust parameters, and trigger workflows in downstream platforms while inheriting LookML semantic modeling, row-level security, and access controls with no runtime LLM token costs. Read the announcement blog.
Build end-to-end data science pipelines with the Data Agent Kit
Following the general availability of the Data Agent Kit, a new end-to-end walkthrough demonstrates how data teams can build, test, and schedule machine learning pipelines directly from their preferred code editors. Explore the walkthrough and codelab.
Manage dashboard agents as code to bring CI/CD and version control to LookML dashboards
Data teams can now declare agent parameters, configurations, and prompt instructions alongside dashboard files in code. This integration enables standard developer workflows for conversational BI, including Git version control, peer review, branching, and automated CI/CD validation before agent modifications reach production end users. Read the technical guide on the Google Developer Community.
Simplify AI search with native AlloyDB hybrid search and Reciprocal Rank Fusion
AlloyDB for PostgreSQL simplifies hybrid search architectures by combining vector similarity search and keyword BM25 retrieval within a unified database engine. Using Reciprocal Rank Fusion (RRF), developers can merge lexical scoring and semantic embeddings into a single ranked result without writing a complex SQL query. This unified approach removes the operational overhead of running and synchronizing external search engines alongside operational databases for retrieval-augmented generation (RAG) and search applications. Explore hybrid search in AlloyDB.

October 5 - October 9

Data Agent Kit is now generally available to bring Google Data Cloud to any coding agent
Data Agent Kit provides a free collection of Model Context Protocol (MCP) tools and Google-authored agent skills that connect more than 15 Google Data Cloud services directly to coding agents across VS Code, Antigravity, Cursor, Claude Code, and Codex. This release expands support for developers and data practitioners to inspect schemas, author queries, and build end-to-end data pipelines in natural language directly from their IDE or CLI. Read the announcement blog.
Spanner Omni is now generally available for deploy-anywhere, multi-model workloads
Spanner Omni brings Spanner's distributed SQL, graph, vector, and key-value capabilities to any environment, including on-premises data centers, other public clouds via Kubernetes or VMs, and local developer laptops. Organizations can build and run multi-model applications with consistent APIs and query engines across hybrid and multi-cloud footprints while seamlessly scaling to managed Spanner on Google Cloud. Read the launch blog and explore how Spanner also removed cumulative mutation limits for DML transactions.
AlloyDB delivers PostgreSQL for agents and native BM25 hybrid search
We announced PostgreSQL for agents in AlloyDB, introducing an agentic database architecture engineered for real-time data access at agent scale with workload isolation. In addition, native BM25 ranking is now in Preview for AlloyDB and Cloud SQL, allowing developers to combine keyword search and vector embeddings directly in PostgreSQL without maintaining an external search engine. For hybrid deployments, the AlloyDB Omni RPM Orchestrator is now generally available, and you can dive deeper in our new Build AI apps and agents with AlloyDB eBook.
Lakehouse runtime catalog adds regional endpoints across 22 regions and GA support for flexible column names
Lakehouse runtime catalog regional endpoints are now live across 22 Google Cloud regions across the Americas, Europe, Asia-Pacific, and the Middle East, supporting both the Apache Iceberg REST catalog endpoint and the Apache Hive catalog endpoint to help customers meet strict data residency and sovereignty requirements. In addition, flexible column names for Apache Iceberg tables are now generally available by default across Iceberg external tables and managed tables, enabling seamless querying of columns containing spaces, hyphens, and Unicode characters. We also announced the Preview of cross-cloud caching and connections for the borderless Lakehouse.
Unlock predictive and agent-ready insights with BigQuery augmented analytics TVFs, TabFM, and identity columns
BigQuery introduced augmented analytics table-valued functions (TVFs): six built-in functions that automate metric diagnostics, trend analysis, and pattern discovery directly where your data lives and integrate as skills for AI agents. We also introduced TabFM in BigQuery to bring tabular foundation model capabilities to predictive analytics, alongside BigQuery identity columns to automatically generate sequential surrogate keys and simplify data warehouse pipelines.
Dataflow adds Pause/Resume and NVIDIA RTX PRO 6000 Blackwell GPU support for large-scale AI workloads
New Dataflow capabilities make it easier to run high-throughput streaming and batch AI pipelines at scale, including Pause/Resume job controls and support for NVIDIA RTX PRO 6000 Blackwell GPUs. And for teams running Apache Spark workloads, learn how to mitigate compute stockouts and maximize availability using flexible VMs.
Unlock up to 3x higher QPS and microsecond latency with Memorystore for Valkey 9.1
Memorystore for Valkey 9.1 is now available, delivering up to 3x higher queries per second (QPS) and microsecond-level caching latencies to accelerate high-throughput applications and real-time AI agent state management. Read the blog.
Customer momentum across Google Data Cloud — Yahoo, PayPal, Surescripts, Airwallex, and Lucius AI
Organizations across industries are scaling their data and AI foundations on Google Data Cloud. Yahoo uses flexible VMs in Managed Service for Apache Spark to reduce provisioning failures by 85%, while PayPal migrated mission-critical analytics workloads to Managed Service for Apache Spark to scale and reduce operational overhead. On the database front, Surescripts migrated 30.5 billion annual healthcare transactions to AlloyDB with 99.998% uptime, Airwallex manages more than 5,000 databases with 99.99% availability on Cloud SQL with a team of four DBAs, and Lucius AI runs a 210,000-tender global platform on AlloyDB and MCP with 47x faster ScaNN vector search.

September 7 - September 10

Pub/Sub SMTs can now AI Inference your Gemini Enterprise Agent Platform models!
Pub/Sub AI Inference SMTs allow you to apply inference on an incoming stream of events using models hosted in Gemini Enterprise Agent Platform. The model’s prediction is appended to your event, making it available for downstream processing in your data warehouse (like BigQuery) or operational database (like BigTable). This feature, now generally available, can dramatically simplify or enhance anomaly detection systems you are operating.
PostgreSQL Source Connector is now generally available in Managed Service for Apache Kafka!
Managed Service for Apache Kafka’s PostgreSQL connector allows customers to capture changes from their PostgreSQL database and ingest them into their Kafka infrastructure with low latency. This source connector is compatible with Cloud SQL for Postgres, AlloyDB, and self-managed PostgreSQL databases. Try this along with our entire portfolio of managed connectors, including MirrorMaker 2.0, BigQuery, Cloud Storage, and Pub/Sub! E-mail kafka-hotline@google.com if you have questions or feedback!

Pause-on-failure for Dataflow batch jobs is GA
Dataflow pause-on-failure enables you to preserve the state of a batch Dataflow job before it fails. By pausing your Dataflow job, you can address issues that are external to the pipeline and resume processing without losing completed work. This helps you better manage resource costs and improve job reliability when you face temporary outages or capacity constraints.

The insertAll API is now the BigQuery Storage Write API (REST)
The legacy insertAll streaming API is now rebranded as the BigQuery Storage Write API (REST). By dropping the "legacy" label, developers can confidently build long-term HTTP-based streaming workflows. This stateless JSON-over-HTTPS endpoint offers a lightweight alternative to heavy gRPC libraries—ideal for serverless web apps, IoT telemetry, and AI logging. The transition is seamless for existing users, requiring zero code changes and offering 100% backward compatibility. However, the Storage Write API (gRPC) version remains the recommended standard for high-throughput, continuous pipelines.

August 31 - September 4

Stateful processing is available in BigQuery continuous queries in Preview
Stateful operations significantly expand what’s possible with BigQuery continuous queries. This feature allows users to leverage functions like JOINs, aggregations, and windowing functions directly in their streaming queries. Now you can calculate metrics over time (for example, a 30-minute average) to power your downstream applications and AI agents with much richer, real-time signals.

Try out our feature here and share your feedback with bq-continuous-queries-feedback@google.com!
Synthetic data generator tool is available for Managed Service for Kafka
You’ve launched your first Kafka cluster. Now what? The next thing to do is to produce some data to the cluster, but that involves modifying a client application somewhere or spinning up a virtual machine. The synthetic data generator tool, now generally available, can start sending mock data to your cluster in 3 clicks, and will get data streaming into your cluster in less than two minutes. The perfect utility for those moments you just want to test your cluster and new features. Try our quickstart today!

Dataflow pipeline updates are faster & more flexible
Dataflow pipeline updates can now stop-and-replace pipelines, a major addition to the existing in-place-update feature. The new parallel pipeline option accelerates the migration between the old & new pipeline, resulting in reduced disruption to your business. You can also set a timeout on drains that prevents runaway costs for your pipeliness in the event of stuck processing. This feature is generally available. Try it here!

July 6 - July 10

New Lakehouse managed tables now in preview
Lakehouse tables for Apache Iceberg are now in preview and available in the console. By using Google-managed Apache Iceberg tables in Lakehouse, you can eliminate the overhead of maintaining duplicate data pipelines and complex synchronization logic between BigQuery and open-source engines. This unified table format delivers native, multi-engine read and write interoperability, allowing you to run concurrent DML/DDL operations across diverse analytics tools on a single, shared storage layer. Built-in automated table management handles painful background optimization tasks like compaction and partition tuning, freeing up your team to focus on building rather than managing storage maintenance.

June 1 - June 5

Beyond the Query: Powering AI Agents with Bigtable, Firestore & Memorystore
Discover the latest advancements in Google Cloud's NoSQL Database portfolio, including Bigtable, Firestore, and Memorystore. This series is designed for a broad audience: whether you are exploring these databases for the first time or are an existing user looking to leverage the new capabilities announced at Next '26.

Register here to secure your spot!

Cloud Engineer's AI Toolkit Workshops: Solve data-driven challenges with BigQuery, AlloyDB, Gemini and more. Hosted by Google Cloud Labs, this highly technical event is built specifically for Platform Engineers, SREs, and cloud infrastructure teams ready to bridge the gap between AI prototypes and production-grade deployments. Look out for more locations coming soon

Toronto - June 25 (Data Cloud) | RSVP Here
Chicago - June 30 (Data Cloud) | RSVP Here
Start a 10-day Bigtable free trial with a 1 node SSD cluster and up to 500GB of storage capacity. With no credit card required to start, you can easily ingest workloads and manage workloads that require low-latency, high-throughput, and predictable access. Plus, new Google Cloud customers get $300 in free credits on signup.

May 11 - May 15

Managed Service for Apache Airflow has launched a wave of new features, including the general availability of Airflow 3.1, AI-powered agentic troubleshooting, a new managed Airflow MCP Server for custom agent integration, and declarative YAML-based orchestration pipelines—discover all the details in the full blog post.

April 20 - April 24

Google-built ODBC Driver for BigQuery is now available in Preview
We are excited to announce the launch of the new, Google-built ODBC driver for BigQuery. This new open-source driver provides a direct, high-performance connection for applications to BigQuery and is developed entirely in-house by Google. Download a new driver and connect your application to BigQuery.

April 13 - April 17

We announced we are reintroducing Data Studio to play a significant role in the AI era, expanding from data visualizations and reports to host BigQuery conversational agents and data apps built in Colab notebooks.
We announced BigQuery Graph is now available in preview, offering an easy-to-use, highly scalable graph analytics solution, empowering data professionals to model, analyze and visualize massive-scale relationships in an entirely new way.

April 6 - April 10

We introduced Conversational Analytics for Looker Embedded environments, enabling users to add natural language experiences to their own custom data-driven applications, powered by Gemini.
We expanded Looker’s capabilities for faster ad-hoc analysis, with the introduction of self-service Explores, enabling you to bring your own data to Looker’s semantic layer and gain instant access to insights in a governed data environment.

March 23 - March 27

We showed you how you can scale your reads with Cloud SQL autoscaling read pools. This feature allows you to provision multiple read replicas that are accessible via a single read endpoint and to dynamically adjust your read capability based on real-time application needs.
Our customers are leveraging the full power of Conversational Analytics and Looker to drive major business and technical breakthroughs in the AI era. Companies like Telenor, Pet Circle, Fluent Commerce, Lighthouse Intelligence, Wego, and ROLLER are turning data into insights and actions, grounded by Looker’s semantic layer.

March 16 - March 20

We introduced an enhanced Gemini assistant in BigQuery Studio, transforming the agent from a code assistant into a fully context-aware analytics partner.

February 23 - February 27

We introduced managed and remote MCP support for Google Cloud databases, including AlloyDB, Spanner, Cloud SQL, Bigtable and Firestore, to power the next generation of agents. This announcement extends the ability for AI models to plan, build, and solve complex problems, connecting to the database tools our customers leverage daily as the backbone of their work environment.
We outlined how you can build a conversational agent in BigQuery using the Conversational Analytics API to help you build context-aware agents that can understand natural language, query your BigQuery data, and deliver answers in text, tables, and visual charts.

February 16 - February 20

Our customers are leveraging the full power of Looker to drive major business and technical breakthroughs. Companies like Arrive, Audika, Carousell, Framebridge, GumGum, Intel, Overdose Digital, Ocean Network Express, Subskribe and Promevo are leveraging Looker’s newest AI-driven capabilities, including Conversational Analytics, to transform data to insights and actions, and empower their entire organization with a single source of truth, powered by Looker’s semantic layer.

February 2 - February 6

Join us on March 4 for our webinar, Win Your AI Strategy with Cloud SQL Enterprise Plus, to learn how to power your generative AI workloads with 3x higher performance and 99.99% availability. Register today to discover how to build a scalable, enterprise-grade foundation for your most demanding AI applications.

January 26 - January 30

We introduced Conversational Analytics in BigQuery, which allows users to analyze data using natural language. Conversational Analytics in BigQuery is an intelligent agent that generates, executes and visualizes answers grounded in your business context directly in BigQuery Studio, making data insights for data professionals more conversational.
We outlined how data products have become the foundation for AI agents, providing the context needed to make autonomous agents reliable and trusted for real business use, backed by organized business logic and semantic understanding.
We highlighted how you can supercharge data analytics workflows, and outlined Google Cloud’s AI agent offerings for data engineering, data science, and development tools, so you can integrate agentic workflows in your applications, empower your teams and speed discovery.

January 19 - January 23

We have fundamentally reimagined Firestore with pipeline operations for Enterprise edition. Experience a powerful new engine featuring over a hundred new query features, index-less queries, new index types, and observability tooling to improve query performance. Seamlessly migrate using built-in tools and leverage Firestore’s existing differentiated serverless foundation, virtually unlimited scale, and industry-leading SLA. Join a community of 600K developers to craft expressive applications that maximize the benefits of rich queryability, real-time listen queries, robust offline caching, and cutting-edge AI-assistive coding integrations.
Introducing Google Cloud SQL on MSSQLTips: We are highlighting a new technical guide published on MSSQLTips titled "Introducing Google Cloud SQL." This article serves as an essential resource for SQL Server administrators and developers exploring Google Cloud's fully managed database service. It provides a detailed overview of Cloud SQL capabilities, including high availability, security integration, and the seamless transition of on-premises SQL Server workloads to the cloud, making it an ideal resource for those planning their migration strategy.
We are excited to announce the Public Preview of Microsoft Entra ID (formerly Azure Active Directory) integration with Cloud SQL for SQL Server. Designed to tackle the challenge of identity sprawl in multi-cloud environments, this integration allows organizations to govern database access using their existing Microsoft identity infrastructure. Key benefits include centralized identity management, enhanced security features like Multi-Factor Authentication (MFA), and simplified user administration through direct group mapping. This feature is available for SQL Server 2022 and supports both public and private IP configurations.

January 12 - January 16

Google-built JDBC Driver for BigQuery is now available in Preview
We are excited to announce the launch of the new, Google-built JDBC driver for BigQuery. This new open-source driver provides a direct, high-performance connection for Java applications to BigQuery and is developed entirely in-house by Google. Download a new driver and connect your Java application to BigQuery.
Troubleshoot Airflow tasks instantly with Gemini Cloud Assist investigations: Cloud Composer just got smarter. We are excited to announce that Gemini Cloud Assist investigations are now available directly within Cloud Composer 3. Instead of manually sifting through raw logs, you can now simply click "Investigate" on a failed Airflow task. Gemini analyzes logs and task metadata to identify failure patterns—such as resource exhaustion or timeouts—and provides actionable recommendations driven by Gemini Cloud Assist to resolve the issue. This integration shifts the debugging experience from manual toil to automated root cause analysis, significantly reducing the time required to restore your pipelines. Learn more about AI-assisted troubleshooting.

Securely connecting Gemini Enterprise to Snowflake data

Tue, 27 Oct 2026 16:00:00 +0000

Enterprise AI needs to go beyond simple chat interfaces. To provide real value, AI agents must access your data warehouses to generate actionable insights, all while maintaining strict security and data governance.

This guide explores how to build a secure bridge between Google Cloud’s Gemini Enterprise and Snowflake. By using the Model Context Protocol (MCP) and the Google Agent Development Kit (ADK), you can empower your agent to act as an assistant that understands natural language, generates precise SQL via Snowflake Cortex and executes queries with user-level permissions.

Choosing the right integration pattern

Direct Connection vs. custom orchestration

The Google ADK architecture demonstrated in this post represents the advanced / custom orchestration pattern. It is designed for developers building complex, multi-agent systems on Google Cloud, such as workflows that combine Snowflake insights with external enterprise tools, custom reasoning loops, or cross-platform state management in Gemini Agent Engine.

For standard enterprise basic use cases where you want to connect Gemini Enterprise directly to Snowflake use the joint solutions mentioned in this link.

The architectural approach

This architecture allows a business user to ask questions like, "What were our top 5 selling items in Texas last quarter?"

Rather than exposing raw SQL execution to the client, the agent delegates both query generation and execution to a governed Snowflake Cortex Agent exposed through the Snowflake MCP server:

Governed translation: The ADK agent passes the user's natural language question to the Snowflake MCP server, which invokes the Cortex Agent (CORTEX_AGENT_RUN) backed by Cortex Analyst semantic view.
Direct execution within the governed boundary: The Cortex Agent translates the question into SQL and executes it directly inside Snowflake using the authenticated user's OAuth session and role privileges, returning the structured results to the ADK agent.

This approach provides deterministic text-to-SQL translation mapped to your corporate semantic definitions, keeps query execution inside Snowflake's governed boundary, and reduces the risk of schema hallucinations or unvalidated SQL execution.

System architecture flow

The diagram below illustrates the trust boundaries and the numbered flow of execution between the Gemini Enterprise app and Snowflake ecosystems.

Detailed execution sequence

To highlight how the agent delegates natural language query translation and governed execution to the Snowflake Cortex Agent over MCP, see the sequence diagram below:

Step 1: Setting up the Snowflake security integration

A major risk in enterprise AI is granting the model broad access via a generic service account, which could allow users to bypass security restrictions. We solve this by using OAuth 2.0 to perform user impersonation.

First, establish a confidential OAuth client in Snowflake to protect the client secret. The snippet below configures the security integration inside Snowflake:

code_block: <ListValue: [StructValue([('code', "USE ROLE ACCOUNTADMIN; \r\nCREATE OR REPLACE SECURITY INTEGRATION <SECURITY_INTEGRATION_NAME> \r\nTYPE = OAUTH \r\nENABLED = TRUE \r\nOAUTH_CLIENT = CUSTOM \r\nOAUTH_CLIENT_TYPE = 'CONFIDENTIAL' \r\n-- Google Cloud's official handshake receiver address: \r\nOAUTH_REDIRECT_URI = 'https://vertexaisearch.cloud.google.com/oauth-redirect' \r\nOAUTH_ISSUE_REFRESH_TOKENS = TRUE \r\n-- Let the connection stay alive for 90 days (max Snowflake allowed limit) \r\nOAUTH_REFRESH_TOKEN_VALIDITY = 7776000;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb8a81dbe90>)])]>

To retrieve the generated client credentials needed for the Google Cloud configuration, run the following command in Snowflake:

code_block: <ListValue: [StructValue([('code', "-- Retrieve OAuth Client ID and Client Secret\r\nSELECT SYSTEM$SHOW_OAUTH_CLIENT_SECRETS('<SECURITY_INTEGRATION_NAME>');"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb897ae4410>)])]>

Step 2: Data governance and access control

Apply the principle of least privilege by creating a dedicated role for your agent inside Snowflake.

code_block: <ListValue: [StructValue([('code', 'USE ROLE SECURITYADMIN;\r\n-- 1. Create a dedicated role\r\nCREATE ROLE IF NOT EXISTS <GENAI_ROLE_NAME>;\r\nGRANT ROLE <GENAI_ROLE_NAME> TO USER <BUSINESS_USER_NAME>; \r\nUSE ROLE SYSADMIN;\r\n-- 2. Grant structural access\r\nGRANT USAGE ON DATABASE <DATABASE_NAME> TO ROLE <GENAI_ROLE_NAME>;\r\nGRANT USAGE ON SCHEMA <DATABASE_NAME>.<SCHEMA_NAME> TO ROLE <GENAI_ROLE_NAME>;\r\n-- 3. Grant compute access\r\nGRANT USAGE ON WAREHOUSE <WAREHOUSE_NAME> TO ROLE <GENAI_ROLE_NAME>;\r\n-- 4 .Grant access to the semantic layer\r\nGRANT SELECT, REFERENCES ON SEMANTIC VIEW <DATABASE_NAME>.<SCHEMA_NAME>.<SEMANTIC_VIEW_NAME> TO ROLE <GENAI_ROLE_NAME>;\r\n-- 5. Grant select privileges on underlying raw data tables\r\nGRANT SELECT ON ALL TABLES IN SCHEMA <DATABASE_NAME>.<SCHEMA_NAME> TO ROLE <GENAI_ROLE_NAME>;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb897ae7fd0>)])]>

IMPORTANT

The user context must have SELECT privileges on the physical tables backing the semantic view. Granting access only to the semantic view will cause runtime errors during query execution.
If granting direct access to base tables is not permitted by your security policy, place a Snowflake Secure View over the physical tables and build the Semantic View on top of it. Because Snowflake views execute with Owner's Rights, the user's GenAI role only needs SELECT privileges on the Secure View and Semantic View rather than the raw tables.
Snowflake OAuth tokens capture and bind to the user's DEFAULT_ROLE and DEFAULT_WAREHOUSE at the exact moment of authorization. If a user authorizes the connection before these defaults are configured, the session will be locked to the wrong role(role available at that time) and query execution will fail if the intended role will not have required privileges.
- Set defaults(as shown below) prior to authorizing in Gemini Enterprise.
- Authorize connection: Have the user perform the one-time OAuth authorization in Gemini Enterprise.
- (Optional) Revert defaults: Once authorized, the user can safely revert their default role and warehouse in Snowflake back to their standard operational preferences.

code_block: <ListValue: [StructValue([('code', "ALTER USER <BUSINESS_USER_NAME> SET DEFAULT_ROLE = '<GENAI_ROLE_NAME>';\r\nALTER USER <BUSINESS_USER_NAME> SET DEFAULT_WAREHOUSE = '<WAREHOUSE_NAME>';"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb897ae4850>)])]>

Step 3: Provisioning the Cortex Agent and MCP server inside Snowflake

Next, create a governed Cortex Agent in Snowflake using your semantic view, then provision an MCP server that exposes the agent via the CORTEX_AGENT_RUN tool type. This approach delegates both SQL generation and query execution to Snowflake while enforcing the caller's OAuth identity and role permissions.

code_block: <ListValue: [StructValue([('code', 'USE DATABASE <DATABASE_NAME>;\r\nUSE SCHEMA <SCHEMA_NAME>;\r\nUSE ROLE SYSADMIN;\r\n\r\n-- 1. Create the Cortex Agent over your semantic view\r\nCREATE OR REPLACE AGENT <AGENT_NAME>\r\n COMMENT = \'Governed data assistant for business queries\'\r\n PROFILE = \'{"cortex_analyst": {"semantic_models": ["<DATABASE_NAME>.<SCHEMA_NAME>.<SEMANTIC_VIEW_NAME>"]}}\';\r\n\r\n-- 2. Provision the MCP Server exposing the Cortex Agent\r\nCREATE OR REPLACE MCP SERVER <SERVER_NAME>\r\nFROM SPECIFICATION $$\r\ntools:\r\n - name: "run_cortex_agent"\r\n type: "CORTEX_AGENT_RUN"\r\n identifier: "<DATABASE_NAME>.<SCHEMA_NAME>.<AGENT_NAME>"\r\n title: "Cortex Agent Assistant"\r\n description: "Converts natural language questions into SQL and executes them directly within Snowflake using the caller\'s OAuth session permissions."\r\n$$;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb897ae7710>)])]>

ISOLATION & GOVERNANCE NOTE: By using CORTEX_AGENT_RUN, query generation and execution remain fully contained within Snowflake's governed security boundary. Avoid exposing raw execution tools like SYSTEM_EXECUTE_SQL to external clients, as doing so introduces unnecessary security surface area and bypasses centralized Cortex Agent governance.

Step 4: Locating and constructing the MCP server URL

The ADK code requires the endpoint URL of your Snowflake-managed MCP server. Now that the server is provisioned, you must first grant access to the server, and then you can retrieve its exact endpoint URL using the following query in a Snowflake worksheet:

code_block: <ListValue: [StructValue([('code', '-- 1. Grant access to the Cortex Agent and the MCP server\r\nUSE ROLE SECURITYADMIN;\r\nGRANT USAGE ON AGENT <DATABASE_NAME>.<SCHEMA_NAME>.<AGENT_NAME> TO ROLE <GENAI_ROLE_NAME>;\r\nGRANT USAGE ON MCP SERVER <DATABASE_NAME>.<SCHEMA_NAME>.<SERVER_NAME> TO ROLE <GENAI_ROLE_NAME>;\r\n-- 2. View the MCP URL\r\nDESCRIBE MCP SERVER <DATABASE_NAME>.<SCHEMA_NAME>.<SERVER_NAME>;\r\n-- Or:\r\nDESC MCP SERVER <DATABASE_NAME>.<SCHEMA_NAME>.<SERVER_NAME>;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb897ae6f50>)])]>

In the command output, inspect the metadata properties to locate the endpoint link. The URL is structured as follows:

code_block: <ListValue: [StructValue([('code', 'https://<org_name>-<account_name>.snowflakecomputing.com/api/v2/databases/<database_name>/schemas/<schema_name>/mcp-servers/<server_name>'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb897ae4ad0>)])]>

Step 5: Connecting via Google ADK

With the MCP Server created and its URL resolved, write your agent code using the Google Agent Development Kit (ADK). In your Python code, define the unique identifier AUTH_ID (e.g. snowflake_oauth_conn) that the agent will use to extract user credentials.

code_block: <ListValue: [StructValue([('code', 'import os\r\nimport re\r\nfrom typing import Optional\r\nfrom google.adk.agents.llm_agent import LlmAgent\r\nfrom google.adk.tools.mcp_tool.mcp_toolset import McpToolset\r\nfrom google.adk.tools.mcp_tool.mcp_session_manager import StreamableHTTPConnectionParams\r\nfrom google.adk.tools.tool_context import ToolContext\r\n\r\n# Define the Auth ID first in your code. \r\n# You will use this exact name when registering the agent in the Google Cloud Console.\r\n# Gemini Enterprise maps the authorization token to this identifier in the ToolContext.\r\nAUTH_ID = "snowflake_oauth_conn" \r\n\r\n# 1. OAuth Header Extraction:\r\n# Define a callback to dynamically extract the user\'s OAuth token injected by Gemini Enterprise \r\n# and generate the Authorization headers required to authenticate with the Snowflake MCP.\r\ndef get_auth_headers(tool_context: Optional[ToolContext] = None) -> dict: \r\n """ \r\n OAuth Handling Callback: Dynamically extracts the user\'s OAuth token injected by Gemini Enterprise and generates the headers required to authenticate with the Snowflake MCP. \r\n """ \r\n token = None \r\n if tool_context and hasattr(tool_context, "state"): \r\n try: \r\n # Parse the state dictionary injected by Gemini Enterprise containing authorization states \r\n state_dict = tool_context.state.to_dict() if hasattr(tool_context.state, \'to_dict\') else tool_context.state \r\n pattern = re.compile(fr".*{re.escape(AUTH_ID)}.*", re.IGNORECASE) \r\n # Find the active token matching our Auth ID (snowflake_oauth_conn)\r\n matching_keys = [k for k in state_dict.keys() if pattern.match(k)] \r\n if matching_keys: \r\n token = state_dict.get(matching_keys[0]) \r\n except Exception as e: \r\n print(f"Warning: Failed to read ToolContext OAuth state: {e}") \r\n # Return authorization headers. If token is None, requests to MCP will fail.\r\n return { \r\n "Authorization": f"Bearer {token}", \r\n "Content-Type": "application/json", \r\n "Accept": "application/json, text/event-stream", \r\n } \r\n\r\n# 2. MCP Initialization:\r\n# Define the Snowflake MCP URL retrieved from \'DESCRIBE MCP SERVER\' directly in the code:\r\nMCP_SERVER_URL = "<YOUR_MCP_SERVER_URL>" # Replace with the URL retrieved from \'DESCRIBE MCP SERVER\' \r\n\r\n# Initialize the Snowflake MCP Toolset using this URL.\r\nsnowflake_mcp = McpToolset( \r\n connection_params=StreamableHTTPConnectionParams( \r\n url=MCP_SERVER_URL \r\n ), \r\n header_provider=get_auth_headers\r\n) \r\n\r\n# 3. Build the Agent:\r\n# Construct the LlmAgent using the Gemini 2.5 Flash model and attach the Snowflake MCP Toolset.\r\nroot_agent = LlmAgent( \r\n model="gemini-2.5-flash", \r\n name="enterprise_data_analyst", \r\n instruction=( \r\n "You are an enterprise data analyst. Use the `enterprise_data_agent` tool " \r\n "on the Snowflake MCP server to answer business questions. Pass the user\'s " \r\n "natural language question to the tool, which translates it into SQL and " \r\n "executes it directly within Snowflake\'s governed boundary, then present " \r\n "the results clearly to the user." \r\n ), \r\n tools=[snowflake_mcp]\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb897ae7450>)])]>

NOTE: The Python snippet above represents a plain/vanilla version of the agent setup. Depending on your organization's specific requirements, you should add robustness on top of this template (e.g., custom error handling, structured schemas, custom prompts, configurations, and additional tool wrappers).

Step 6: Project structure and deployment

To deploy your ADK agent to Agent Runtime in Agent Platform, organize your files in the following directory layout:

code_block: <ListValue: [StructValue([('code', 'snowflake_agent/\r\n├── agent.py # The agent definition and MCP connection code (from Step 5)\r\n├── __init__.py # Package initializer exposing the agent\r\n├── .env # Environment configurations (Google Cloud Project, etc.)\r\n└── requirements.txt # List of project dependencies (google-adk, etc.)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb897ae6bd0>)])]>

File details

__init__.py: Exposes the root_agent object to the deployment CLI:

code_block: <ListValue: [StructValue([('code', '# Snowflake ADK Agent package\r\nfrom .agent import root_agent'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb897ae44d0>)])]>

.env: Declares environment configuration:

code_block: <ListValue: [StructValue([('code', 'GOOGLE_CLOUD_PROJECT="<YOUR_GCP_PROJECT_ID>"\r\nGOOGLE_CLOUD_LOCATION="<LOCATION>"\r\nGOOGLE_GENAI_USE_VERTEXAI="TRUE"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb897ae6f90>)])]>

requirements.txt: Specifies dependencies:

code_block: <ListValue: [StructValue([('code', 'google-adk[mcp]==2.3.0\r\ngoogle-genai==2.10.0\r\npydantic-settings==2.14.2\r\ngoogle-cloud-aiplatform[agent_engines]>=1.50.0'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb897ae6810>)])]>

2. Deploying via the CLI

Rather than running complex custom Python scripts or manual container builds, you can deploy using the official Google Agents CLI (agents-cli). The CLI packages your code, sets up the server, and handles the deployment pipeline:

code_block: <ListValue: [StructValue([('code', '# Deploy the agent to the Google Cloud Agent Runtime\r\nagents-cli deploy --deployment-target agent_runtime'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb897ae5390>)])]>

The CLI automatically handles the deployment lifecycle. Once complete, it outputs the unique project resource path (e.g., projects/PROJECT_ID/locations/LOCATION/reasoningEngines/RESOURCE_ID) which you will use to register the agent in the next step.

TIP: For more information and advanced configuration options for ADK agent deployment, refer to:

Step 7: Registering the custom agent in Gemini Enterprise

Once the agent code is ready and deployed, register the agent in the Google Cloud Console. Here, you will link the agent configuration and map the OAuth parameters retrieved from your Snowflake Security Integration.

Extracting the OAuth Endpoints from Snowflake

Run this query in Snowflake to extract the precise Token and Authorization endpoints:

code_block: <ListValue: [StructValue([('code', 'DESCRIBE SECURITY INTEGRATION <SECURITY_INTEGRATION_NAME>;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb897ae5f10>)])]>

In the results, locate the values for the following properties:

OAUTH_AUTHORIZATION_ENDPOINT: Your Snowflake Authorization URI.
OAUTH_TOKEN_ENDPOINT: Your Snowflake Token URI.

2. Registration steps in Google Cloud console

Go to the Gemini Enterprise page in the console.
Select the application you want to register the agent with.
In the left navigation, click Agents, then click + Add agent.
Select Add for Custom agent via Agent Platform. The Authorizations setup displays.
Set up the Snowflake OAuth connection:
- Click Add authorization.
- Authorization name: Enter the exact ID you defined in your ADK code (e.g., snowflake_oauth_conn). This ensures the credentials map correctly to the code's AUTH_ID.
- Client ID & Client secret: Input the credentials you retrieved from Snowflake using the SYSTEM$SHOW_OAUTH_CLIENT_SECRETS query.
- Token URI: Input the exact token endpoint value extracted from DESCRIBE SECURITY INTEGRATION.
- Authorization URI: Take the authorization endpoint value extracted from DESCRIBE SECURITY INTEGRATION and append ?response_type=code at the end (along with your client ID and scope query parameters, for example):

code_block: <ListValue: [StructValue([('code', 'https://<org_name>-<account_name>.snowflakecomputing.com/oauth/authorize?client_id=YOUR_CLIENT_ID&redirect_uri=https%3A%2F%2Fvertexaisearch.cloud.google.com%2Foauth-redirect&scope=session%3Arole-any&response_type=code'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb8a8af0d10>)])]>

IMPORTANT
Ensure the Authorization URI ends with &response_type=code (or ?response_type=code if there are no other parameters). This parameter is required by Google Cloud to trigger the OAuth 2.0 authorization code flow.

Click Done, then click Next.

6. Configure the agent properties:

7. Click Create to complete registration.

Understanding identifier alignment

ADK Code AUTH_ID <==> Google Cloud Authorization Name: The AUTH_ID in your Python script must exactly match the Authorization Name/ID you input during console registration. Because we write code first, we reuse the code-defined identifier during registration.
Snowflake Security Integration Name (e.g. <SECURITY_INTEGRATION_NAME>): This is purely a Snowflake-side database object name. It does not need to match the Google Cloud Authorization Name or the AUTH_ID in your ADK Python code.

Get started

By uniting the Gemini model’s reasoning capabilities, ADK's orchestration, and Snowflake's governed data layer, you can safely bring AI to your data. This architecture provides an agentic assistant that respects your security perimeters while delivering business value.

Unifying enterprise context and semantics: LookML in Knowledge Catalog is now GA

Mon, 26 Oct 2026 16:00:00 +0000

As enterprises go from experimental AI agent prototypes to using them in production, data teams are realizing that access to data is not the same as understanding that data. To help bridge that gap, today, we are announcing the general availability (GA) of LookML semantic language indexing in Knowledge Catalog for Looker Core on Google Cloud.

To answer a complex business question accurately, an AI agent needs context — table schemas, data lineage, governance policies, business definitions, etc.. However, context is not a monolith; knowing where data lives is very different from knowing how to calculate metrics with it. In the enterprise, teams often conflate broad organizational context (usually in the form of metadata from structured and unstructured documents) with deterministic business semantics (the exact SQL formulas, join paths, and filtering rules that define metrics like net revenue or churn), assuming that a single data catalog or a business intelligence (BI) tool handling both enterprise-wide asset discovery and precise metric calculation.

In reality, no one tool can carry the entire load for agentic AI, as enterprise agents require both organizational context and deterministic business semantics. In Google Cloud environments, those are provided by Knowledge Catalog and LookML:

Broad enterprise context with Knowledge Catalog: Agents need a unified context graph that aggregates metadata across structured tables and unstructured documents. Through Knowledge Catalog, agents discover assets, inspect data quality, trace end-to-end lineage, respect column-level privacy tags, and surface corporate policy documents. The context graph tells an agent what data exists, how it is governed, and how it is connected.
Deterministic mathematical semantics with LookML: What a catalog does not author on its own is how the numbers are computed. Describing a table is not the same as defining how tables join or how metrics are calculated. Looker's modeling language, LookML, encodes production-tested semantic logic — multi-grain join protections, conditional filters, and exact KPI formulas — so agents don't guess SQL math at runtime.

With this native integration, the LookML business logic that powers your trusted BI reporting is automatically indexed into Knowledge Catalog — giving Gemini Enterprise and custom AI agents both enterprise-wide discovery and deterministic BI semantics.

Why a context graph needs deterministic semantics

When an LLM agent connects directly to raw database tables — even when armed with basic table descriptions or a business glossary — it still has to infer mathematical relationships on the fly. Without explicit semantic rules, agents routinely fall into two traps:

Metric definition drift: Standard text-to-SQL agents can invent conflicting formulas for foundational KPIs like net revenue or customer acquisition cost, failing to apply domain-specific accounting rules (such as recognizing revenue only when order_status = 'Complete' or prorating annual contracts across active service days).
Naive join fan-outs: Complex schemas often contain one-to-many (1:N) and zero-to-many (0:N) relationships across invoices, line items, and refunds. Without pre-modeled join relationships and symmetric aggregation rules, agent-generated SQL can multiply rows and inflate financial totals.

Knowledge Catalog and Looker solve this together, enabling you to model once in LookML and access those semantics everywhere across Google Cloud's AI ecosystem. Knowledge Catalog enforces semantic math when that logic is defined in a semantic modeling tool; LookML provides a mature, production-proven language for defining those exact mathematical relationships.

What's new: LookML in Knowledge Catalog

Previously, catalog integrations only captured shallow, file-level pointers and basic dashboard inventories. With this release, LookML's deterministic semantics become a first-class layer in the Knowledge Catalog context graph. An agent that finds a metric in the graph now also finds its exact definition, its join path, and its lineage back to source tables — in the same place, under the same governance. Synchronization is always on by default with no action required from end users, and with live sync, any update pushed to production LookML is reflected in Knowledge Catalog in near-real-time so agents operate on the freshest business logic.

Because this metadata is indexed directly into Knowledge Catalog by default, it is immediately available to Gemini Enterprise, without custom agent code or separate catalog data stores. When connected to Knowledge Catalog, Gemini Enterprise uses built-in read-only skills and hybrid search to look up technical schemas, business glossaries, and indexed LookML logic in a single step, passing that rich context directly to your federated BigQuery connector to generate accurate SQL. (For developers building custom agents, the same context bundles are also accessible via the LookupContext API.)

Knowledge Catalog now captures and links the following entity levels across Looker Core on Google Cloud:

LookML models (looker-model): Captures descriptive metadata, database connection routing, include hierarchies, data validation tests (assert), and parent-child EntryLinks mapping models to their associated Explores, Views, and LookML dashboards.
LookML Explores (looker-explore): Indexes full relational join logic (type, sql_on, relationship cardinality such as many_to_one fan-out protection, foreign_key, required_joins, and sql_where) as Schema-Join relationships. It also captures mandatory filter guardrails (always_filter, access_filter), inherited dimensions and measures, and predefined starter queries that help agents and analysts overcome the cold-start problem.
LookML Views and fields (looker-view): Extracts column-level compiled LookML SQL expressions (for example, ${total_revenue} / NULLIF(${count}, 0)), primary keys, dimension group timeframes, field synonyms, and derived table transformation logic (including SQL and native derived tables, partition keys, and cluster keys).
Dashboards, Looks, and tiles (looker-dashboard, looker-dashboard-element, looker-look): Catalogs both UI-created and LookML-defined dashboards and visualization tiles, complete with dashboard filters, chart types, ownership, and usage statistics. Users in Knowledge Catalog are able to see table- and column-level lineage from underlying BigQuery tables all the way to individual dashboard tiles, allowing data engineers to evaluate schema changes proactively, before disruptions occur.

Looker Core on Google Cloud automatically indexes resolved LookML models, Explores, Views, lineage, and dashboards directly into Knowledge Catalog.

Three patterns for data agents

Bringing LookML into Knowledge Catalog gives Gemini Enterprise (and your broader agent ecosystem) a single place to retrieve both broad organizational context and governance from Knowledge Catalog and explicit mathematical relationships and SQL definitions authored in LookML.

To see the impact of combining these two layers in Gemini Enterprise, compare how three architectures handle a complex cross-domain prompt: "What was our total revenue last fiscal quarter by product category, and which regional promotional policy limits apply?"

Agent architecture

System mechanics

Data logic and governance resolution

User experience and token efficiency

Pattern 1: Baseline database agent

Data warehouse only

Direct LLM writing raw SQL against warehouse tables with no semantic or catalog context.

Ungoverned and non-deterministic

Has no visibility into data quality, PII tags, or corporate documents, and guesses SQL metric logic blindly — assuming a standard January fiscal calendar, summing uncompleted orders, and exposing restricted customer PII.

Fragile and high-risk

Prone to schema hallucinations, fails on questions requiring unstructured policy context, and burns tokens in multi-turn SQL debugging loops.

Pattern 2: Context-grounded agent

Knowledge Catalog without indexed semantics

Gemini Enterprise or another agent platform uses Knowledge Catalog to look up table descriptions, glossary terms, PII governance aspects, and unstructured policy PDFs, then writes SQL via the warehouse connector.

Context-grounded, but lacks semantic SQL logic

Enforces Knowledge Catalog governance and retrieves unstructured promotional policy rules, but lacks semantic definitions for SQL metrics and table relationships. This approach can miss conditional accounting rules (like order_status = 'Complete') or cause join fan-outs across multi-grain tables.

Governed discovery, inconsistent math

Solves asset discovery, compliance, and unstructured context, but still suffers from text-to-SQL calculation drift and multi-turn SQL retries.

Pattern 3: Context + semantic-grounded agent

Knowledge Catalog + Looker Core on Google Cloud with indexed LookML

Gemini Enterprise or another agent platform connects to Knowledge Catalog enriched with indexed LookML metadata, automatically retrieving both broad catalog context and LookML join paths and field formulas to ground its SQL generation.

Governed context plus semantic SQL definitions

Enforces Knowledge Catalog PII aspects, retrieves unstructured policy PDFs, and grounds its SQL in LookML's exact mathematical definitions indexed in Knowledge Catalog, applying status = 'Complete', the May–July fiscal calendar, and predefined join conditions across warehouse tables.

Accurate and token-efficient:

Pre-mapped LookML join blueprints and compiled SQL expressions replace guessing, prevent join fan-outs, reduce SQL retry loops, and let the agent point users to relevant Looker dashboards and Explores.

When you follow pattern 3 and pair Knowledge Catalog with Looker's semantic layer in an enterprise agent workflow, the agent moves from guessing at raw SQL to a governed, two-stage execution flow:

Context and governance discovery in Knowledge Catalog: Before touching any data, the agent searches Knowledge Catalog, verifies that the thelook_ecommerce LookML model is gold-certified, checks column-level governance tags, and maps the prompt to exact LookML dimensions and measures, converting natural language into structured business intent.
Deterministic semantic execution: Rather than having the LLM invent SQL on the fly, the agent grounds its SQL in LookML's exact mathematical definitions indexed in Knowledge Catalog and returns trusted, repeatable figures.

Alongside the answer, the agent returns an interactive Open in Looker Explore link with the exact measures and filters pre-populated, letting users slice, dice, and inspect row-level line items in a single click.

Side-by-side comparison of a baseline agent querying BigQuery directly (left), which guesses at raw SQL and fiscal dates, versus an agent governed by Knowledge Catalog and Looker (right), which applies indexed LookML definitions and returns an interactive "Open in Looker Explore" link.

Scaling agents for BI and the enterprise

A common question from data leaders is: "If we use Looker Conversational Analytics, when do we use an enterprise agent grounded in Knowledge Catalog?"

The two experiences are designed for complementary jobs across your organization:

Looker Conversational Analytics is your dedicated BI assistant: When business users are working inside BI workflows and asking questions scoped to a focused set of curated Looker Explores, Conversational Analytics delivers immediate, self-service data exploration and visualization directly within Looker.
Knowledge Catalog + Looker powers your enterprise-wide agents: When users ask questions in Gemini Enterprise across federated Data Cloud sources like BigQuery, they often span structured tables, BI metrics, and unstructured documents. Enabling Knowledge Catalog in Gemini Enterprise gives the assistant built-in skills to dynamically discover relevant semantics across hundreds of LookML models, pass indexed LookML join definitions, metric SQL formulas, and starter queries to the BigQuery connector to prevent join hallucinations, and enforce centralized governance aspects and IAM.

Getting started

Because Looker Core on Google Cloud, Knowledge Catalog, and Gemini Enterprise are natively integrated, you can connect your existing LookML business logic to your enterprise AI assistant in just a few clicks, with no custom agent code required.

To start discovering Looker semantics and grounding your agents:

Confirm first-party metadata indexing is enabled for Looker Core on Google Cloud. This is enabled by default for new instances, with no additional charge for storing Looker technical metadata. Once enabled, metadata synchronization into Knowledge Catalog is always on by default (across both initial indexing and ongoing live updates). Enable the Cloud Dataplex API and Data Lineage API in your project and grant users the required view roles (roles/dataplex.viewer, roles/looker.schemaViewer, and roles/datalineage.viewer) to have your resolved LookML models, Explores, Views, join paths, and dashboards automatically sync into Knowledge Catalog.
Enable Knowledge Catalog in Gemini Enterprise. When you attach a BigQuery (or Spanner, Cloud SQL, or AlloyDB) data store in federated query mode to your Gemini Enterprise app, the Knowledge Catalog integration is enabled automatically. You can also turn it on or off at any time in the Google Cloud console by opening your Gemini Enterprise app, navigating to Configurations > Assistant, toggling on Enable Knowledge Catalog, and clicking Save and publish.
Ask questions. Gemini Enterprise connects to Knowledge Catalog using end-user credentials, meaning users only discover and query data assets they have IAM permission to view (such as roles/bigquery.dataViewer). Behind the scenes, the assistant automatically looks up your indexed LookML definitions, join logic, and starter queries in Knowledge Catalog and passes them to your federated connector to deliver accurate, governed answers.

To dive deeper, read the Manage Looker Core on Google Cloud with Knowledge Catalog documentation and learn how to Connect to Knowledge Catalog in Gemini Enterprise. To build a more customized experience, explore our hands-on ADK agent codelab on GitHub.

Introducing new self-learning context enrichment in Knowledge Catalog

Thu, 22 Oct 2026 16:00:00 +0000

Today, we’re launching self-learning context enrichment (preview) and a reimagined Knowledge Studio experience (preview) in Knowledge Catalog, Google Cloud’s context engine for agents. Building on Knowledge Catalog's existing Gemini-powered enrichment, self-learning enrichment analyzes agent and user interactions to find gaps and keep your business context accurate and current. Knowledge Studio is the new command center for context engineers and data stewards to explore their data estate conversationally, navigate a visual context graph, and review proposed changes to the context graph before they are published to their users and agents.

An agent is only as good as the organization-wide definitions, rules, and relationships it reasons over — the context, as it were. That’s why curating context has become one of the most important disciplines in enterprise agentic capabilities. Today, most of that work happens by hand. Context engineers write prompts and mappings agent by agent, while data stewards try to manually keep definitions current across thousands of assets. But this breaks down as organizations move from a handful of agents to thousands, and eventually millions.

Knowledge Catalog’s Gemini-powered self-learning enrichment turns that manual upkeep into a continuous feedback loop. By analyzing agent and user interactions, it pinpoints missing or outdated context and pulls verified definitions from playbooks, code repositories, and business documents into governed, reusable context sets. Then, with optimized search, every agent gets precisely the right context it needs instead of reasoning over bloated prompts and raw schemas, and benefits from shared learning across your organization. Knowledge Catalog makes AI agents nearly 4X more accurate, with up to 33% lower latency, and using up to 75% fewer tokens, based on internal benchmarks.

Keep context current with Gemini-powered self-learning enrichment

Consider a simple question to a Gemini agent: "What was the net revenue in EMEA last quarter?" Answering it correctly requires knowing how your company defines net revenue, which tables hold the bookings, and how they join, yet that knowledge is typically scattered across a finance playbook in Google Docs, a data model in GitHub, and a Confluence page. Static catalogs expect subject matter experts to manually capture and update business definitions, metrics, and other context upfront so that an agent can effectively query the right data. However, in reality, experts rarely have the time to document every asset, and the moment the finance team revises how partner credits are treated next quarter, static definitions fall behind and every agent with a hardcoded formula starts giving the wrong answer.

Knowledge Catalog’s enrichment capability acts where you already keep your business knowledge: across your team’s every doc, repo, and table. It connects to your structured data across first-party and third-party sources and unstructured data in Cloud Storage, Google Workspace, GitHub, Confluence, and SharePoint. It finds, extracts, assembles, and maintains that business knowledge for you, linking those unstructured sources like playbooks and wikis and their context to your structured metadata in Knowledge Catalog like tables and schemas to make your structured data far more discoverable and actionable. On demand or on a schedule, it proposes business definitions, table and column descriptions, join relationships, golden queries, and task-specific agent instructions, each with source citations and a confidence score. On every run, it detects what changed at the source, retires obsolete definitions, and flags conflicts for review.

It learns from both your user and agent interactions. You can configure enrichment to analyze interaction traces from Gemini agents, Conversational Analytics in BigQuery, or your custom agents while redacting PII, and it pinpoints where agents struggled, such as ambiguous column lookups, ungrounded metrics, missing definitions, or users rephrasing the same question. It then finds the authoritative answer and stages a fix, so the same mistake doesn't happen twice. That's how context engineering keeps pace with millions of agent interactions, freeing experts to review changes instead of writing them from scratch.

Configuring an enrichment task with connected sources, evaluation, and self-learning

Automation doesn't mean giving up control. You can choose to enable or disable auto-publishing at any time, and set the exact confidence score (for example, 85%) at which proposed changes are automatically approved. Anything below your threshold, along with sensitive updates such as deletions, become a Change Request for a steward to approve, edit, or reject. You can also provide your own evaluation set to test proposed updates end to end. Every change is logged with full provenance and can be reverted. And of course, Google never uses your data to train its models.

Reviewing enrichment proposals, confidence scores, and change diffs before publishing

Set the right context for each audience with context sets

By design, an enterprise context graph is quite broad, but for a given task, both agents and users often only need the right, trusted segment of it. Knowledge Catalog achieves that with context sets — curated, governed subsets of the context graph that contains the definitions, metrics, join logic, and golden queries for a specific domain, team, agent, or group of end users. For instance, a sales agent can be bound to the Sales context set, while a finance analyst's assistant can be bound to the Finance context set. Because "net revenue" can legitimately mean different things to each group, every domain maintains its own definitions in the same catalog. However, agents don't have to be bound to a single set; with search in Knowledge Catalog, an agent can also dynamically search and find the right context set for any given task.

Context sets also provide built-in data governance and fine-grained access control. Access to each context set is managed with IAM, so stewards decide which agents and users can read, use, or edit it. Agents only retrieve context from the sets they're authorized to use, and access to the underlying data continues to follow your existing permissions. Every update is traceable, and reversible, and because context sets build on the Open Knowledge Format (OKF), you can import an OKF bundle to become a context set and export it back out as OKF so your context stays portable.

Approved enrichments published to a governed context set

Explore and govern it all in Knowledge Studio

All of these capabilities come together in the reimagined Knowledge Catalog experience, built for context engineers and data stewards who curate and govern context for agents.

Navigating the context graph visually in Knowledge Studio

The new Knowledge Studio turns Knowledge Catalog from a repository of static inventory into an interactive command center, allowing you to:

See context coverage at a glance: Navigate a visual map of your cross-cloud data estate, from operational databases and warehouses to semantic models, context sets, and documentation.
Explore context conversationally: Ask business questions in natural language, traverse relationships across your enterprise context graph, and see exactly how agents reason over your metadata.
Review and validate enrichments in one place: Inspect each proposed change, source citations, and confidence score. Test context sets against real questions, and approve, edit, or roll back change requests with software-style version control.
Monitor enrichment runs: Track each enrichment task's schedule, results, token usage, and run duration, and get notified by email when a run completes.

Get started

Watch the demo: How Knowledge Catalog’s self-learning enrichment grounds agents

These features will be rolled out to customers in the next few weeks. Sign up to request immediate access and visit the documentation for step-by-step guidance.

BigQuery ObjectRef now GA: Build multimodal data platforms for conversational analytics entirely in SQL

Wed, 21 Oct 2026 16:00:00 +0000

For data science, data engineering, and machine learning teams, unlocking the value of unstructured data, such as images, videos, audio, and documents, has historically meant managing siloed data pipelines for structured and unstructured data, and for analytics and AI/ML use cases. Even when files remain in Cloud Storage, running inference requires teams to orchestrate external machine learning workloads, only to face the complex, brittle challenge of stitching those structured inference outputs back into the data warehouse for analysis. This post-hoc data-stitching introduces significant operational overhead and database misalignment.

ObjectRef and multimodal data analytics in BigQuery are now in General Availability (GA), establishing BigQuery as a comprehensive, multimodal agentic data platform.

With ObjectRef, BigQuery natively bridges the gap between Cloud Storage and your analytical engine. You can represent unstructured data as native ObjectRef columns directly alongside your traditional relational columns. This unlocks a massive advantage over standalone model APIs by enabling you to pass multimodal, multi-column inputs directly to Gemini models in a single SQL query or natural language query using conversational analytics. ObjectRef integrations with BigQuery AI, conversational analytics in BigQuery and Gemini Enterprise enable you to easily combine Cloud Storage files with rich, structured database context, metadata, timestamps and custom agent instructions.

You can now leverage BigQuery's core strengths (fully serverless compute, low total cost of ownership, and built-in at-scale inference) to build, run, and govern powerful unstructured-to-structured, unstructured-to-unstructured and fully multimodal-to-multimodal pipelines without separate compute clusters. Crucially, the extracted features and annotations can be combined directly with the unstructured object itself by storing the ObjectRef in the same row. Using this combined data directly in conversational analytics and AI agents avoids the loss of nuanced data and context that typically occurs when customers transform unstructured files into purely structured formats. Customers are already using this approach to generate embeddings for multimodal vector search, perform at-scale classification using GenAI, and execute precise data and feature extraction from vast archives of documents and media, all via standard SQL.

For example, this single query uses AI.AGG to summarize the product images in a public BigQuery sample dataset:

code_block: <ListValue: [StructValue([('code', 'SELECT\r\nAI.AGG(image_ref, "Summarize the major categories of pictures")\r\nFROM\r\n`bigquery-public-data.cymbal_pets.products`;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb896e58bd0>)])]>

ObjectRef brings structure to the unstructured

ObjectRef is a built-in, predefined STRUCT data type that points to objects outside of BigQuery.

code_block: <ListValue: [StructValue([('code', '-- ObjectRef is a special STRUCT in BigQuery \r\nSTRUCT<\r\n uri STRING,\r\n version STRING,\r\n authorizer STRING,\r\n details JSON'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb896e59bd0>)])]>

By storing these pointers natively in standard BigQuery tables alongside traditional columns, BigQuery treats unstructured files as first-class data citizens. Furthermore, this architecture supports independent data lifecycles: your GCS data and BigQuery tables can change independently without affecting the underlying data in the other system.

This approach serves multiple data personas by offering powerful advantages:

Zero-copy architecture: Keep your high-volume media files securely in Cloud Storage and query them instantly in BigQuery, respecting data gravity and avoiding costly data duplication.
SQL and Python interoperability: You can query unstructured data using standard SQL or BigQuery DataFrames. Crucially, you can pass ObjectRefs into BigQuery Python UDFs, allowing you to bring open-source Python libraries, like OpenCV, directly to your data to execute custom logic without moving it.
Independent lifecycles and versioning: ObjectRefs ensure reproducible inference by tracking specific GCS generation numbers. If the underlying GCS file changes, your query can still reference the exact version of the file that existed when the row was written.
Unified governance: You can secure, audit, and share access to both structured columns and their related unstructured files using BigQuery's existing security policies and IAM controls.

Create full-fidelity ETL with SQL

Since launching ObjectRef in preview, a powerful, highly effective pattern has emerged among our users: building full-fidelity ETL pipelines entirely with SQL.

The diagram below shows the 4 ways to get ObjectRefs for processing unstructured data with BigQuery AI and further querying the multimodal data with AI agents.

Teams are leveraging BigQuery’s serverless architecture to integrate AI/ML directly into their data environments, eliminating the need for separate compute clusters to handle data movement and inference. Managing unstructured data in GCS as standard BigQuery columns allows users to apply core data engineering best practices, such as building idempotent pipelines, ensuring reproducibility, and executing seamless backfills, directly to images, video, audio, and documents.

Once data is robustly prepared, teams use Gemini within BigQuery to execute full-fidelity ETL. They unlock dual-layer analytics by extracting features and annotating unstructured data while keeping the original ObjectRef anchored in the same row. The structured outputs power fast filtering, reporting and BI dashboards, while the attached unstructured objects remain available for deep, context-heavy reasoning. For example, you might extract sentiment scores and topic tags from an archive of customer call recordings to quickly flag at-risk accounts on a BI dashboard (the structured layer). Simultaneously, a conversational analytics agent evaluating escalation processes can seamlessly fall back to the original audio files to analyze the caller and customer service representative interactions, preventing the loss of critical nuance that typically occurs in unstructured-to-structured ETL pipelines.

Proven use cases from industry leaders

“At Snap, the safety of our community is a top priority, and we aim to limit users’ exposure to content that violates our Community Guidelines. We regularly review samples of public posts, and by leveraging ObjectRef with BigQuery AI, we transitioned from simple random sampling to post-stratification sampling. This transition improved the efficiency of our human review processes, enabling us to reinvest reclaimed capacity into new and expanded workflows that further enhance our ability to detect and address potentially harmful content.” - Carlos Ortiz, Data Scientist, Snap

During the preview phase, we saw adoption across a variety of critical workloads. Customers have successfully transitioned these experimental pipelines into production, transforming unstructured data into actionable, structured insights:

Content moderation: Online platforms leverage BigQuery to integrate multimedia assets with standard telemetry data in a single pipeline to fundamentally enhance their moderation and safety outcomes.
Proactive threat detection: Enterprise security teams use BigQuery to analyze visual and structural web components alongside network logs, driving critical advancements in proactive threat identification.
Inventory compliance: Online retailers extract text and context directly from vast image repositories to automatically flag regulatory or brand compliance issues at scale.
Customer experience: Service-oriented enterprises analyze audio interactions between support agents and customers with BigQuery AI to evaluate customer sentiment and improve service delivery.

New in GA

We streamlined the developer experience for the General Availability release, focusing on support for a wide variety of customer use cases:

Flexible authentication:
- In GA, we introduced End User Credentials (EUC) support for ObjectRefs. Users can now quickly query and explore unstructured data using their personal IAM identity directly for ad-hoc analysis.
- In preview, querying unstructured files required setting up explicit BigQuery connections for ObjectRefs. While these connections remain fully supported and are the recommended best practice for robust, service-account-driven production pipelines, we recognize they require additional setup.
Unified support for your Cloud Storage metadata: We know your unstructured data’s metadata lives in various places across Google Cloud, and ObjectRef now supports all of them.
- If it's already in a table: Whether your metadata currently resides in BigQuery object tables, Cloud Storage insights datasets, or simply as raw string URIs within standard BigQuery tables, you can seamlessly create and manage ObjectRefs to analyze the underlying data.
- If it isn't in a table: The new OBJ.LIST function lists files directly from a Cloud Storage path and returns an ObjectRef for each one, so you don't need to create a table first.
Array support for comparative analytics: We introduced native support for processing ObjectRefs contained within arrays. This enables entirely new classes of workloads, such as comparing multiple documents against one another, analyzing sequences of images, or batching items in a single semantic call.

See it in action: End-to-end multimodal analytics

To illustrate the power of these GA updates, let's look at how you can build real-world, multimodal pipelines using our streamlined SQL operators.

1. Full-fidelity ETL: Call center audio analysis

Analyzing customer service calls historically required moving audio files to specialized transcription and natural language processing clusters. With ObjectRef, you can parse audio files, extract structured business intelligence and add them directly as standard relational columns.

Use the GA OBJ.MAKE_REF() function to automatically detect the audio format from a raw string URI and BigQuery’s managed AI functions for well known use-cases. For example, you can use AI.CLASSIFY() to categorize customer call types. By default, BigQuery uses Optimized Mode for AI Functions to reduce the number of tokens consumed.

code_block: <ListValue: [StructValue([('code', 'CREATE OR REPLACE TABLE call_center.annotated_call_logs AS\r\nSELECT \r\n customer_id,\r\n call_date,\r\n OBJ.MAKE_REF(audio_uri) as ref,\r\n AI.CLASSIFY(\r\n OBJ.MAKE_REF(audio_uri), [\r\n "Lost or stolen card", \r\n "Declined card payment", \r\n "Activate my card",\r\n "Order physical card",\r\n "Cancel transfer",\r\n "Password or pin forgotten",\r\n "Virtual card not working",\r\n "Card swallowed by ATM",\r\n "Balance not updated after bank transfer",\r\n "Balance not updated after check or cash deposit",\r\n "Extra charge on statement",\r\n "Transfer fee charged"\r\n ]\r\n ) AS customer_issue_type,\r\n FROM \r\n call_center.raw_call_logs;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb896e593d0>)])]>

This table can serve as a multimodal data source for a BigQuery conversational analytics agent. The agent translates natural language queries into SQL using BigQuery AI functions to analyze audio files when additional information is needed.

Go to the Agents tab in BigQuery to create a conversational analytics agent and select your multimodal table as a source.

2. At-scale gen AI filtering: Product compliance

Data engineers frequently need to filter massive datasets based on the contents of images or documents. Instead of running heavy generative queries and parsing text outputs, you can now use the declarative AI.IF() operator natively alongside your standard SQL in WHERE clauses. The default Optimized Mode reduces the number of tokens consumed to further optimize your workloads.

code_block: <ListValue: [StructValue([('code', 'SELECT \r\n product_id,\r\n seller_id,\r\nFROM \r\n ecommerce.vendor_uploads\r\nWHERE \r\n upload_date >= CURRENT_DATE() - 1 AND\r\n AI.IF(("Does this product image contain a restricted item or policy violation?", OBJ.MAKE_REF(image_uri)))'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb896e58450>)])]>

3. Ad-hoc discovery: Product catalog enrichment

The examples above start from tables that already hold your file URIs. If you don't have one yet, OBJ.LIST turns a Cloud Storage path into queryable rows, each with an ObjectRef in the ref column, so you can list files, send them to Gemini, and get typed columns back in a single query. Try it on the public sample bucket:

code_block: <ListValue: [StructValue([('code', 'SELECT\r\n uri,\r\n size,\r\n updated,\r\n result.animal_type,\r\n result.item_color\r\nFROM (\r\n SELECT\r\n uri,\r\n size,\r\n updated,\r\n AI.GENERATE(\r\n ("What type of animal is this pet product for, and what is its primary color?", ref),\r\n output_schema => \'animal_type STRING, item_color STRING\'\r\n ) AS result\r\n FROM\r\n OBJ.LIST(\'gs://cloud-samples-data/bigquery/tutorials/cymbal-pets/images/*.png\')\r\n);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb896e5b0d0>)])]>

4. Multi-object reasoning: Proactive threat detection

With native array support in GA, you can now pass an array of ObjectRefs into a single Gemini call to analyze multiple files simultaneously using Gemini.

This unlocks complex comparative logic directly in SQL. For example, a security team can pass a screenshot of their official login page and a screenshot of a newly discovered, suspicious webpage to Gemini simultaneously to detect phishing attempts at scale:

code_block: <ListValue: [StructValue([('code', 'SELECT \r\n log_id,\r\n suspect_domain,\r\n assessment.is_phishing,\r\n assessment.visual_discrepancies\r\nFROM (\r\n SELECT \r\n log_id,\r\n suspect_domain,\r\n AI.GENERATE(\r\n (\r\n "Compare these two web UI screenshots. The first is our official, trusted login page. The second is a suspected spoofed page. Identify any visual discrepancies, missing logos, or fake elements, and determine if it is a phishing attempt.", \r\n [OBJ.MAKE_REF(trusted_ui_uri),OBJ.MAKE_REF(suspect_ui_uri)]\r\n ),\r\n output_schema => \'is_phishing BOOLEAN, visual_discrepancies STRING\'\r\n ) AS assessment\r\n FROM \r\n cybersecurity.network_ui_logs\r\n);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb896e58a50>)])]>

Start analyzing today

BigQuery ObjectRef is now generally available on all BigQuery editions and on-demand pricing tiers.

If you are currently managing complex, fragmented infrastructure just to perform machine learning inference on your Cloud Storage files, it's time to consolidate. Bring AI directly to your storage. Start leveraging your existing data engineering practices to build robust, scalable, and secure multimodal analytics pipelines directly within BigQuery.

To get started with ObjectRef and multimodal analytics, check out these resources:

Walk through our hands-on tutorial to generate image captions and assign product categories using synthetic data.
Watch live demos from Cloud Next about unifying unstructured and structured data, and generating text with LLMs and performing vector search.
Learn how to work with ObjectRef in the BigQuery documentation.

How to Use Python UDF Logging for Pipeline Observability

Mon, 19 Oct 2026 16:00:00 +0000

We are announcing a new logging feature that is now available in Python User-Defined Function (UDF) in BigQuery. Developers can now use Python’s standard logging library directly inside BigQuery UDFs to emit structured logs seamlessly into Cloud Logging—with zero external plugins or custom connectors required.

While UDF logging enables a wide range of operational monitoring and debugging workflows, to demonstrate this feature we are showcasing an example for building fault-tolerant, resilient data pipelines. In the example below, we are capturing row-level diagnostic telemetry during query execution, so that teams can easily isolate problematic records, trigger real-time alerts, and triage issues via SQL without halting entire workloads.

In this post, we’ll show you how to get started with Python UDF logging and walk through an e-commerce event ingestion pipeline as a practical, end-to-end example.

Architecture overview

The workflow consists of three stages:

Inline validation: The Python UDF parses incoming JSON payloads. Valid records pass to the destination table; corrupted or non-compliant rows trigger logging.error() and return NULL to prevent query failure.
Real-time alerting: A Cloud Monitoring log-based metric notifies your team when error rates breach SLA thresholds.
SQL triage & replay: Data engineers query the quarantined payloads in BigQuery using standard SQL over the linked _AllLogs view, diagnose the root cause, and re-ingest recovered data.

Although this specific demonstration relies on Python for data validation and transformation, you can seamlessly integrate Python-driven logging and validation into pipelines that perform SQL-based transformations.

Quickstart walkthrough: Step-by-step

Step 1: Create the generic Python UDF

This UDF parses JSON and checks for a required field. If parsing fails or the required key is missing, it logs structured diagnostic JSON and returns NULL.

code_block: <ListValue: [StructValue([('code', 'CREATE OR REPLACE FUNCTION `YOUR_PROJECT.YOUR_DATASET.safe_parse_json`(\r\n raw_payload STRING,\r\n required_key STRING\r\n)\r\nRETURNS STRING\r\nLANGUAGE python\r\nOPTIONS (entry_point=\'safe_parse_json\', runtime_version=\'python-3.11\')\r\nAS r\'\'\'\r\nimport json\r\nimport logging\r\n\r\ndef safe_parse_json(raw_payload, required_key):\r\n if not raw_payload or raw_payload.strip() == "":\r\n logging.warning(json.dumps({"error_type": "EMPTY_PAYLOAD"}))\r\n return None\r\n\r\n try:\r\n data = json.loads(raw_payload)\r\n if required_key and (required_key not in data or data[required_key] is None):\r\n logging.error(json.dumps({\r\n "error_type": "MISSING_REQUIRED_KEY",\r\n "missing_key": required_key,\r\n "raw_payload": raw_payload[:200]\r\n }))\r\n return None\r\n return json.dumps(data)\r\n\r\n except Exception as e:\r\n logging.error(json.dumps({\r\n "error_type": "JSON_SYNTAX_ERROR",\r\n "error_message": str(e),\r\n "raw_payload": raw_payload[:200]\r\n }))\r\n return None\r\n\'\'\';'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb897e504d0>)])]>

Step 2: Run ingestion with logging enabled

In this example, we simulate an e-commerce event ingestion pipeline processing incoming user actions (click, view, checkout). Our goal is to extract valid events into our clean analytics table while gracefully quarantining broken or missing-schema payloads.

Enable logging in your session with SET @@udf.properties:

code_block: <ListValue: [StructValue([('code', 'SET @@udf.properties = JSON \'{"enableDebugOutput": true}\';\r\n\r\nCREATE OR REPLACE TABLE `YOUR_PROJECT.YOUR_DATASET.clean_events` AS\r\nWITH sample_events AS (\r\n SELECT \'evt_001\' AS event_id, \'{"user_id": "usr_101", "event": "click", "amount": 25.5}\' AS raw_data UNION ALL\r\n SELECT \'evt_002\' AS event_id, \'corrupted_json{\' AS raw_data UNION ALL -- Syntax error\r\n SELECT \'evt_003\' AS event_id, \'{"event": "view"}\' AS raw_data UNION ALL -- Missing required user_id\r\n SELECT \'evt_004\' AS event_id, \'{"user_id": "usr_102", "event": "checkout", "amount": 99.0}\' AS raw_data\r\n),\r\nparsed_events AS (\r\n SELECT\r\n event_id,\r\n `YOUR_PROJECT.YOUR_DATASET.safe_parse_json`(raw_data, \'user_id\') AS validated_json\r\n FROM sample_events\r\n)\r\nSELECT\r\n event_id,\r\n JSON_VALUE(validated_json, \'$.user_id\') AS user_id,\r\n JSON_VALUE(validated_json, \'$.event\') AS event_name,\r\n \'VALID\' AS status\r\nFROM parsed_events\r\nWHERE validated_json IS NOT NULL;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb897e52310>)])]>

Ingestion Results

event_id	user_id	event_name	status
`evt_001`	`usr_101`	`click`	`VALID`
`evt_004`	`usr_102`	`checkout`	`VALID`

The query succeeds without interruption while bad rows (2 and 3) are routed to Cloud Logging.

Step 3: Set up real-time alerts on Cloud Logging

You can configure alerts to notify your on-call team as soon as malformed payloads exceed acceptable thresholds.

In the Google Cloud Console (1-Click)

In the Google Cloud Console, open Logs Explorer.
Paste the log filter for your UDF routine:

code_block: <ListValue: [StructValue([('code', 'resource.type="bigquery.googleapis.com/ManagedRoutineInvocation"\r\nseverity>=ERROR'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb897b06250>)])]>

3. Click Create alert in the top action bar.

4. Set your notification channels (e.g., Slack, PagerDuty, or Email) and save the policy.

Step 4: Triage bad rows with BigQuery Log Analytics

"Note: To query your logs via BigQuery SQL, ensure you have enabled BigQuery Log Analytics on your Cloud Logging bucket and created a linked dataset."

Query the quarantined payloads using BigQuery SQL via the linked _AllLogs view:

code_block: <ListValue: [StructValue([('code', "SELECT\r\n timestamp,\r\n severity,\r\n JSON_VALUE(json_payload.error_type) AS error_type,\r\n JSON_VALUE(json_payload.missing_key) AS missing_key,\r\n JSON_VALUE(json_payload.raw_payload) AS raw_payload\r\nFROM `YOUR_PROJECT.YOUR_LOGS_LINKED_DATASET._AllLogs`\r\nWHERE log_name LIKE '%managed_routine_output%'\r\nORDER BY timestamp DESC\r\nLIMIT 10;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb8a8b71e90>)])]>

Log Analytics output

timestamp	severity	error_type	missing_key	raw_payload
`2026-09-17 18:35:50`	`ERROR`	`MISSING_REQUIRED_KEY`	`user_id`	`{"event": "view"}`
`2026-09-17 18:35:50`	`ERROR`	`JSON_SYNTAX_ERROR`	null	`corrupted_json{`

Recovering data

Because full raw payloads are preserved in Cloud Logging, you can recover quarantined records directly with SQL once upstream data is fixed:

code_block: <ListValue: [StructValue([('code', "-- 1. Replay and recover quarantined rows from Cloud Logging sink table\r\nINSERT INTO `YOUR_PROJECT.YOUR_DATASET.clean_events` (event_id, user_id, event_name, status)\r\nSELECT\r\n 'evt_003_recovered' AS event_id,\r\n 'default_user' AS user_id, -- Apply fallback default for the missing field\r\n JSON_VALUE(JSON_VALUE(json_payload.raw_payload), '$.event') AS event_name,\r\n 'RECOVERED' AS status\r\nFROM `YOUR_PROJECT.YOUR_LOGS_LINKED_DATASET._AllLogs`\r\nWHERE JSON_VALUE(json_payload.error_type) = 'MISSING_REQUIRED_KEY'\r\n\r\n-- 2. Inspect the final clean destination table\r\nSELECT event_id, user_id, event_name, status\r\nFROM `YOUR_PROJECT.YOUR_DATASET.clean_events`\r\nORDER BY event_id;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb8a8b727d0>)])]>

Final `clean_events` Table Result:

event_id	user_id	event_name	status
`evt_001`	`usr_101`	`click`	`VALID`
`evt_004`	`usr_102`	`checkout`	`VALID`
evt_003_recovered	default_user	view	RECOVERED

All valid and recovered data is now unified in the production table without any data loss.

Other use cases

Shadow execution & safe migrations: Run legacy and new business logic side-by-side in a UDF. Return the legacy result to production while logging calculation differences to Cloud Logging.
PII redaction auditing: Mask sensitive tokens (such as SSNs or account numbers) before writing to BigQuery, while emitting anonymized audit events to Cloud Logging.
ML inference drift: Log warnings when embedded scoring logic encounters out-of-vocabulary inputs or low model confidence.

Best practices

Enable debug output: Include SET @@udf.properties = JSON '{"enableDebugOutput": true}'; in the session where you want logs captured.
Avoid sensitive data in logs: Cloud Logging is visible to users with Logging Viewer permissions. Do not log credentials or secrets.
Cost management: On high-throughput pipelines, log selectively (e.g., only on errors or via sampling) to manage logging ingestion volume.

Getting started

Bringing the borderless Lakehouse into the agentic era with innovations for Gemini

Thu, 15 Oct 2026 16:00:00 +0000

As enterprises shift their AI usage from experimental chatbots to autonomous agents, teams are hitting a critical bottleneck: accessing and reasoning over data scattered across multiple clouds, SaaS platforms, and on-premises systems. To turn AI agents into true systems of action, organizations need a foundation that can handle petabyte-scale reasoning without the security risks of data movement. Last week at our Gemini at Work event, we announced how Gemini can help you transform business workflows. To help power your agents with your enterprise data, we will dive deeper into the new Gemini integration with borderless Lakehouse to help you build, secure, and scale agents that work across your data estate.

Access data that lives outside of Google Cloud through Gemini

The ultimate value of the borderless Lakehouse is realized through the agents you build to transform your organization, with Gemini as the front door. With the new Gemini integration, data that’s either preconfigured or stored in your borderless Lakehouse is now directly integrated into Gemini. Currently in preview, this means you can set up access to new data sources, such as Databricks Unity Catalog or Snowflake Horizon, through Lakehouse catalog federation directly from Gemini.

Now your business users can use natural language to query records that live across clouds while your AI agents can ground their reasoning and generate code from the same data — all without moving it. By directly contextualizing AI agents across your borderless Lakehouse, you can help accelerate your organization’s AI adoption while simplifying access to critical information.

Gemini agents invoked by business users can access data that lives in BigQuery and Databricks Unity Catalog to provide comprehensive reporting and get insights using natural language.

Borderless Lakehouse advancements

In addition to this new, native integration with Gemini, we’re sharing that several features we introduced earlier this year to power your data foundation with the borderless Lakehouse are now generally available, including:

Support for Cross-Cloud Interconnect and cross-cloud caching for AWS and Azure
BigQuery support for Apache Iceberg V3 specifications
Read-write interoperability on Apache Iceberg tables for both BigQuery and Managed Service for Apache Spark
Bi-directional caching with AWS Glue, Databricks Unity Catalog, and Snowflake Horizon Catalog using Lakehouse runtime federation
Integration with Knowledge Catalog
Zero-copy data integration with Salesforce, SAP and Workday, and now, Confluent Tableflow
Synchronization of Databricks Delta Lake assets through the Lakehouse console using Dataflow

Let’s explore.

Performance and reasoning at petabyte scale

While agent context windows are typically limited to ingesting hundreds of kilobytes at a time, enterprise workflows sometimes join petabytes of data across clouds. An agent-native data platform that needs to combine data across sources, like product and customer data, to your CRM cloud and warehouse telemetry needs to be able to process data at scale. When you do this over MCP, you are limited to megabyte scale because of context windows and payload size. By integrating at every layer of the data and AI stack, the borderless Lakehouse provides enterprise-grade scalability, enabling agents to reason over petabytes of data with similar performance to it being stored locally.

At the networking layer, Cross-Cloud Interconnect and cross-cloud caching provide high-throughput, low-latency paths to AWS and Azure, eliminating variable egress fees. When an agent joins Iceberg tables in Amazon Azure Data Lake Storage with data in BigQuery, every uncached byte it scans crosses a cloud boundary, so the network sets the pace. Cross-Cloud Interconnect gives the Lakehouse a dedicated, private connection between Google Cloud and AWS or Azure, at 10 Gbps or 100 Gbps per connection. Lakehouse queries travel over that connection rather than the public internet, so latency stays low and predictable. Cross-cloud caching serves repeat reads from inside Google Cloud, so agents can query data in other clouds without migrating it.

The borderless Lakehouse includes several key optimizations to integrate Iceberg metadata within the query engine. BigQuery now supports Apache Iceberg V3 specifications, including binary deletion vectors and sort orders. These features allow the engine to handle row-level updates and deletes more efficiently by eliminating the need to rewrite entire data files. We also upgraded BigQuery’s execution engine with a new FlatBuffers-based Parquet reader and parallelized I/O to help reduce CPU overhead and serialization bottlenecks and narrow the performance gap between open formats and native storage.

This builds upon recent performance innovations to the query engine designed to support agent workloads, which often come in herds and need to process small queries quickly and cost effectively. With BigQuery fluid scaling and history based optimizations, you can run scalable and efficient agent workloads while maintaining openness with Apache Iceberg.

Meanwhile, read-write interoperability on Apache Iceberg tables for both BigQuery and Managed Service for Apache Spark makes it easy for all of your data teams to work off the same Iceberg tables in Google Cloud Storage. This vertically optimized stack helps make your agents faster and more cost-effective, leveraging the power of BigQuery and Lightning Engine to activate data without moving it. To expand the analytical depth of the lakehouse, graph support for Iceberg in BigQuery lets you analyze relationship networks, such as supply chains, directly on open table datasets.

Comprehensive context and data access for your agents

An AI agent is only as smart as the context it can access. To help prevent hallucinations and improve accuracy at the metadata layer, the borderless Lakehouse unifies your enterprise data using the Lakehouse runtime catalog. Based on the Apache Iceberg REST Catalog, it provides bi-directional catalog federation with AWS Glue, Databricks Unity Catalog, and Snowflake Horizon Catalog. The Lakehouse runtime catalog also includes a new standard Hive Metastore-compatible endpoint, allowing engines like Apache Spark, Trino, and Presto to discover metadata without architectural changes.

The borderless Lakehouse also integrates directly with Knowledge Catalog, which automatically indexes your distributed data assets, to provide the precise business metadata and semantic definitions agents need to reason over your entire enterprise corpus — including unstructured data in Cloud Storage. This gives agents just enough context to be accurate while controlling token costs, helping you develop an intelligent and sustainable agentic strategy.

The borderless Lakehouse provides zero-copy data integration with your SaaS applications like Salesforce, SAP and Workday and integrates with a larger partner ecosystem, capturing enterprise data in Knowledge Catalog and acting as a single context engine for your agents. Today, we’re expanding the partner ecosystem with new integrations for Confluent Tableflow, now generally available, which syncs to Lakehouse runtime catalog so the Iceberg tables it produces show up directly in BigQuery, with no copying and no extra ETL.

Finally, organizations with Databricks-centric repositories can now easily import, transform and synchronize their Delta Lake assets into Lakehouse through the Lakehouse console using Dataflow.

Modernize with ease

Customer use of Apache Iceberg has more than tripled since the beginning of 2025, and enterprises building with Google Cloud’s borderless Lakehouse are seeing significant impact from it:

Etsy unified three petabytes with borderless Lakehouse, joining data in place to accelerate pipeline queries by 60%.
Commerzbank transforms raw distributed data into governed, agent-ready access.
Hightouch integrates directly with borderless Lakehouse so teams can power their marketing directly on an open Iceberg lakehouse and activate that data across the Google Cloud ecosystem.
Roundel modernized its Target Product Ads technology stack from the ground up with borderless Lakehouse, completely rebuilding its data ingestion, processing, and measurement pipelines.

If you are ready to get started with the transition to borderless Lakehouse for your organization, it doesn’t need to be difficult. For organizations currently using legacy Dataproc Metastore, we now offer an automated, zero-downtime migration path to the serverless Lakehouse runtime catalog. This preserves your existing schemas and lineage while unlocking the AI-ready features of the borderless Lakehouse, so you can stop managing infrastructure and start activating agents.

Get started today with the Build a borderless Lakehouse codelab or visit our website to learn more.

Predictable AI spend in BigQuery: Introducing cost management for generative AI functions

Wed, 14 Oct 2026 16:00:00 +0000

The generative AI functions in BigQuery (AI.GENERATE, AI.IF, AI.CLASSIFY, AI.SCORE, and more) bring large language models (LLMs) directly into your SQL, so a single query can classify millions of support tickets or score each product review for sentiment. However, because these functions are billed by the token, scaling them across massive datasets highlights the importance of managing your budget proactively.

We recently announced the general availability (GA) of token-based cost management for BigQuery generative AI functions: daily quotas that let administrators set self-serve guardrails on AI token consumption, so teams can innovate while staying within your budget. In this blog, we provide an overview of cost management for BigQuery generative AI functions, walk through a few practical examples, and share resources to help you get started. Let’s jump in.

Quotas for managing costs

Four daily quotas, managed through the Google Cloud Identity and Access Management (IAM) quota system, provide controls at the project level and the per-user level. Per-user limits apply to each user within a project, not across your organization.

Quota name	Metric	Scope	Default value
AI input tokens per day	Input tokens used by the LLM	Per day, per project	200 billion
AI input tokens per day per user	Input tokens used by the LLM	Per day, per project, per user	150 billion
AI output tokens per day	Output and thinking tokens used by the LLM	Per day, per project, per project	20 billion
AI output tokens per day per user	Output and thinking tokens used by the LLM	Per day, per project, per user	15 billion

Key details:

The quotas are global. Usage is summed across all regions, so costs can't pile up unnoticed in a region you weren't watching.
Output quotas include "thinking" tokens. Reasoning tokens count toward your output quota, just as they count toward your Gemini platform bill.

Which functions are covered?

The quotas apply to these BigQuery functions that perform LLM-backed generation:

AI.GENERATE
AI.GENERATE_TABLE
AI.GENERATE_TEXT (and its alias ML.GENERATE_TEXT)
AI.CLASSIFY
AI.IF
AI.SCORE
AI.AGG

What's not covered: Embedding functions such as AI.GENERATE_EMBEDDING (also available as ML.GENERATE_EMBEDDING) and AI.EMBED are not subject to these quotas, because they run on different, substantially cheaper models.

Managing your quotas

Administrators can view and adjust the quotas directly in the Google Cloud console as workloads change, with no support case required.

Navigate to IAM & Admin → Quotas & System Limits.
Filter by service and select BigQuery API.
Sort by the Name column to bring the four AI quotas to the top.

Locating the gen AI token quotas in IAM & Admin → Quotas & System Limits.

4. Find the quota, for example AI input tokens per day.

5. Open the three-dot menu and choose Edit quota.

From there:

To tighten cost control, lower a quota. This takes effect immediately with no approval needed.
To support higher usage, request an increase from the same page. Many requests are auto-approved; larger increases may go through a quick review.

Editing a gen AI token quota in IAM & Admin → Quotas & System Limits.

You'll need the right permissions. You need one of: Owner, Editor, Quota Administrator (roles/servicemanagement.quotaAdmin), or Service Usage Admin (roles/serviceusage.serviceUsageAdmin).

Prefer automation? You can also manage these quotas programmatically through the Cloud Quotas API, the gcloud CLI, and Terraform to apply consistent guardrails across projects.

Monitoring usage

Beyond setting limits, the Quotas & System Limits page displays your historical token consumption so you can quickly detect spikes and right-size your budgets. Use these trends to see how new generative AI workloads and data pipelines affect your overall token usage.

Daily project-level token usage for a gen AI token quota in IAM & Admin → Quotas & System Limits.

Longer-term usage graph with selectable 7-day, 14-day, 30-day, and custom ranges.

For a step-by-step walkthrough, see the BigQuery documentation on controlling costs with token quotas.

Token quotas vs. billing spend caps

Token quotas aren't the only cost guardrail in Google Cloud.

Token quotas (this feature) are preventive and AI-specific. BigQuery tracks token usage during query execution and stops work close to the limit, with separate limits per project and per user.
Cloud Billing spend caps operate on dollars at the project and service level and can automatically block usage when a threshold is reached. They're a broad, financial backstop rather than a precise usage control, and because enforcement is not instant, some overages are incurred.

Use token quotas as your first line of defense for AI spend, and a spend cap as a project-wide safety net.

How enforcement works

Quotas are enforced by stopping work rather than letting a query run past your limit. A query is rejected up front with a QuotaExceeded error only if a quota it needs is already exhausted. Otherwise, it starts, and if a quota runs out mid-query, BigQuery stops sending new LLM calls:

Functions that support max_error_ratio (such as AI.IF) return partial results if unprocessed rows stay within the ratio, and otherwise fail.
General-purpose functions (such as AI.GENERATE) still succeed. Skipped rows have no generated value, and their status field contains the quota-exceeded error.

For details, see quota enforcement behavior in the documentation.

Sizing your input data: `AI.COUNT_TOKENS`

To choose quota values, it helps to know how much text your workloads send to a model. The AI.COUNT_TOKENS function helps with this.

AI.COUNT_TOKENS returns the number of tokens a model's tokenizer produces for a text string, without making a billable generation call. Run it over a column to measure your input data:

code_block: <ListValue: [StructValue([('code', "-- Count the input text tokens for each product review\r\nSELECT\r\n SUM(\r\n AI.COUNT_TOKENS(\r\n CONCAT('Classify the sentiment of this review as positive, negative, or neutral: ', review_text),\r\n\r\n endpoint => 'gemini-3.1-flash-lite').result) AS total_input_tokens\r\nFROM\r\n `my_project.my_dataset.product_reviews`;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb8a8c66f90>)])]>

The count is a lower bound on input tokens. It doesn't include output or thinking tokens, the instructions each AI function adds around your input, or non-text inputs such as images. Use it to:

Gauge the input side of a workload before you launch it.
Right-size your quotas together with a small sample run that measures actual input and output usage.
Translate tokens into dollars using the published Gemini platform pricing for your model.

From tokens to budget: a worked example

Suppose you want to keep a project's daily AI spend around $100 with the Gemini 3.1 Flash-Lite model. As of today's published pricing (BigQuery processes in batch which provides a 50% discount):

Input tokens: $0.125 per 1 million tokens
Output tokens (response + reasoning): $0.75 per 1 million tokens

If a small sample run shows an average 100:1 input-to-output token ratio, you can solve for a daily budget:

code_block: <ListValue: [StructValue([('code', 'Cost: 0.125 × I + 0.75 × O = 100 (I, O in millions of tokens)\r\nRatio: I = 100 × O'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb897de48d0>)])]>

Solving gives about 754 million input tokens and 7.54 million output tokens per day. Set AI input tokens per day and AI output tokens per day to these values to hold the project near $100/day. If a project uses models with different prices, budget per model or blend the rates.

Best practices for managing AI cost

Start with per-user quotas. They stop one user or runaway experiment from draining the project's budget.
Measure before you scale. Use AI.COUNT_TOKENS to size your input, then check a small sample run in the usage chart before setting limits.
Mind the output side. Output tokens cost more. Where a function accepts max_output_tokens, set it to bound generation length.
Monitor and iterate. Adjust limits as real usage and budgets change.

Get started

Token-based cost management is available now. To begin:

Open IAM & Admin → Quotas & System Limits in the Google Cloud Console and filter by BigQuery API.
Review the four AI-related token quotas and set limits that match your budget.
Use AI.COUNT_TOKENS to size your input data.

For the full details, see the documentation on controlling generative AI costs.

With token-based guardrails in place, your teams can use generative AI in BigQuery with the cost predictability your organization needs.

Cross-cloud conversational analytics made simple with open standards

Tue, 13 Oct 2026 16:00:00 +0000

The reality for most organizations is that data lives everywhere. As engineering teams expand and adopt specialized platforms for their workloads, enterprise data naturally disperses across multiple clouds and operational systems. When analytical and AI initiatives require context from across the entire organization,these teams traditionally face a difficult trade-off: build complex extraction pipelines to copy data into a central repository or accept fragmented reporting.

Traditional architectures address this fragmentation with scheduled extraction pipelines that copy remote datasets into a central staging warehouse. This approach creates synchronization lag, adds recurring storage costs, and triggers variable network egress fees. When schemas change upstream, pipelines break, and teams spend hours reconciling metrics across out-of-sync dashboards.

BigQuery and Google Cloud’s borderless Lakehouse offer a streamlined way to solve this challenge with a foundation based on the open Apache Iceberg REST catalog specification. Rather than moving data to the compute engine, you can bring Google Cloud compute and generative AI directly to your data, querying it in place across Databricks, Snowflake, and AWS Glue.

In this post, we walk through how to configure Databricks Unity Catalog for federation into BigQuery (AWS Glue and Snowflake set up in appendix), run a single query joining data across all three external catalogs, and use conversational analytics in BigQuery to explore the consolidated data with natural language.

Figure 1: BigQuery federates metadata across external Apache Iceberg REST catalogs through Lakehouse, enabling in-place conversational analytics without data migration.

The borderless lakehouse architecture

This solution uses Lakehouse runtime catalog to synchronize metadata from other cloud providers, allowing you to query data with BigQuery or external open source engines through the Apache Iceberg REST catalog endpoint, all without migrating the data. Because Databricks Unity Catalog, Snowflake Horizon Catalog, and AWS Glue Data Catalog support the open Apache Iceberg REST catalog specification, you can apply a consistent connection pattern across all three platforms.

When you run a federated query, Lakehouse runtime catalog separates metadata synchronization from storage access:

Metadata synchronization: Lakehouse runtime catalog periodically syncs schemas, snapshot pointers, and table definitions from the remote catalog based on a refresh schedule you define.
Direct storage access: During query runtime, a compatible engine reads Parquet data files directly from remote object storage, such as Amazon S3, using temporary credentials issued by the remote platform.

All query processing runs within Google cloud compute such as BigQuery or Managed Service for Apache Spark. Remote Databricks clusters, Snowflake virtual warehouses, and AWS compute instances are not invoked during execution.

Authentication follows an explicit three-step deployment pattern: create the federated catalog in Google Cloud, inspect its configuration to retrieve its automatically generated service account, and grant access permissions within the remote platform.

You can use AI to generate sample data. The schema details for tables are provided below.

Configure Databricks Unity Catalog

We begin in Databricks by preparing the catalog and table. In Unity Catalog, federated access requires that your target schema be associated with an external location registered in Databricks:

code_block: <ListValue: [StructValue([('code', "-- Run inside Databricks SQL workspace\r\nCREATE CATALOG IF NOT EXISTS dbx_retail_db;\r\n\r\n-- Declare the managed target location at the schema level\r\nCREATE SCHEMA IF NOT EXISTS dbx_retail_db.dbx_core \r\n MANAGED LOCATION 's3://my-databricks-bucket/retail/data/';\r\n\r\n-- Create the Iceberg dimension table\r\nCREATE TABLE IF NOT EXISTS dbx_retail_db.dbx_core.dim_products (\r\n product_key INT,\r\n product_id STRING,\r\n product_name STRING,\r\n category STRING,\r\n subcategory STRING,\r\n brand STRING,\r\n price DOUBLE,\r\n cost DOUBLE\r\n)\r\nUSING iceberg;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb897d437d0>)])]>

You can use AI to generate sample data. The schema details for tables are provided in each section.

Before external engines can read this table, a Databricks metastore administrator must enable External data access on the metastore details tab.

Go to Databricks Admin account and under User Management, create a Databricks Service Principal and make a note of its UUID "6ea7b861-c5a5-xxxx-xxxx-xxxxxxxxxxxx"

Connect Databricks to BigQuery

Next, create the federated catalog inside Google Cloud using the gcloud command-line tool. You will need your Databricks workspace URL and the application ID of your Databricks service principal:

code_block: <ListValue: [StructValue([('code', 'export GCP_PROJECT_ID="my-lakehouse-project"\r\nexport GCP_REGION="us-central1"\r\n\r\ngcloud alpha biglake iceberg catalogs create dbx_retail_db \\\r\n --project="$GCP_PROJECT_ID" \\\r\n --primary-location="$GCP_REGION" \\\r\n --catalog-type="federated" \\\r\n --federated-catalog-type="unity" \\\r\n --unity-instance-name="dbc-xxxxxxxx-xxxx.cloud.databricks.com" \\\r\n --unity-catalog-name="dbx_retail_db" \\\r\n --unity-service-principal-application-id="6ea7b861-c5a5-xxxx-xxxx-xxxxxxxxxxxx" \\\r\n --refresh-interval="300s"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb8a8b4e250>)])]>

Once created, describe the catalog to view its assigned service account:

code_block: <ListValue: [StructValue([('code', 'gcloud alpha biglake iceberg catalogs describe dbx_retail_db \\\r\n --project="$GCP_PROJECT_ID"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb896d89950>)])]>

Return to Databricks to assign permissions to your service principal:

code_block: <ListValue: [StructValue([('code', '-- Grant catalog traversal permission\r\nGRANT USE CATALOG ON CATALOG dbx_retail_db \r\n TO `6ea7b861-c5a5-xxxx-xxxx-xxxxxxxxxxxx`;\r\n\r\n-- Grant schema and read permissions\r\nGRANT USE SCHEMA,EXTERNAL USE SCHEMA, SELECT ON SCHEMA dbx_retail_db.dbx_core \r\n TO `6ea7b861-c5a5-xxxx-xxxx-xxxxxxxxxxxx`;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb896d890d0>)])]>

You must grant the EXTERNAL USE SCHEMA privilege to the Databricks Service Principal on the target schemas or catalog to enable credential vending for external data access.

Finally, to complete the connection, configure the federation policy on the Databricks service principal with the following details -

Issuer URL* - https://accounts.google.com
Subject* - numeric subject ID returned by the catalog describe command eg 1124251400450xxxxxxxx
Audiences - https://accounts.cloud.databricks.com

Run one query across three clouds

Federated Iceberg tables are addressed in BigQuery using four-part paths: project.catalog.namespace.table.

In our environment, sales transactions reside natively in BigQuery:

code_block: <ListValue: [StructValue([('code', 'CREATE SCHEMA IF NOT EXISTS local_retail_dataset\r\nOPTIONS(location="us-central1");\r\n\r\nCREATE OR REPLACE TABLE `my-lakehouse-project.local_retail_dataset.fact_sales` (\r\n sale_id STRING,\r\n sale_date DATE,\r\n customer_key INT64,\r\n product_key INT64,\r\n store_key INT64,\r\n quantity INT64,\r\n unit_price NUMERIC,\r\n discount NUMERIC,\r\n total_revenue NUMERIC,\r\n total_cost NUMERIC,\r\n margin NUMERIC\r\n)\r\nPARTITION BY sale_date\r\nCLUSTER BY store_key, product_key;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb8a8a9d290>)])]>

We can now run a single SQL query joining our native sales transactions with customer records in Snowflake, product lines in Databricks, and store locations in AWS Glue:

code_block: <ListValue: [StructValue([('code', "SELECT \r\n c.first_name || ' ' || c.last_name AS customer_name, -- From Snowflake Iceberg\r\n p.product_name, -- From Databricks Iceberg\r\n s.store_name, -- From AWS S3 Iceberg\r\n SUM(f.quantity) AS total_units_sold, -- From BigQuery native fact\r\n SUM(f.total_revenue) AS total_sales_revenue -- From BigQuery native fact\r\nFROM `my-lakehouse-project.local_retail_dataset.fact_sales` f\r\nINNER JOIN `my-lakehouse-project.sf_retail_db.CORE.DIM_CUSTOMERS` c \r\n ON f.customer_key = c.customer_key\r\nINNER JOIN `my-lakehouse-project.dbx_retail_db.dbx_core.dim_products` p \r\n ON f.product_key = p.product_key\r\nINNER JOIN `my-lakehouse-project.glue_retail_db.glue_retail_db.dim_stores` s \r\n ON f.store_key = s.store_key\r\nGROUP BY 1, 2, 3\r\nORDER BY total_sales_revenue DESC\r\nLIMIT 10;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb896ae0610>)])]>

BigQuery processes the joins in place. Notice that Snowflake identifiers appear in uppercase (CORE.DIM_CUSTOMERS), while Databricks and AWS Glue maintain lowercase table paths.

Figure 2 - Query results combining data across Snowflake, Databricks, and AWS Glue.

Explore cross-cloud data with natural language

Because federated Iceberg tables behave like native tables within BigQuery, downstream analytical features inherit access directly. You can point conversational analytics in BigQuery at these federated schemas to let team members explore cross-cloud data using natural language prompts.

To give conversational analytics full operational context, we save our federated join as a logical view:

code_block: <ListValue: [StructValue([('code', "CREATE OR REPLACE VIEW `my-lakehouse-project.local_retail_dataset.v_omnichannel_sales_pulse` AS\r\nSELECT \r\n f.sale_id,\r\n f.sale_date,\r\n c.first_name || ' ' || c.last_name AS customer_name,\r\n c.country AS customer_country,\r\n p.product_name,\r\n p.category AS product_category,\r\n p.brand AS product_brand,\r\n s.store_name,\r\n s.city AS store_city,\r\n s.region AS store_region,\r\n f.quantity,\r\n f.total_revenue,\r\n f.margin\r\nFROM `my-lakehouse-project.local_retail_dataset.fact_sales` f\r\nINNER JOIN `my-lakehouse-project.sf_retail_db.CORE.DIM_CUSTOMERS` c \r\n ON f.customer_key = c.customer_key\r\nINNER JOIN `my-lakehouse-project.dbx_retail_db.dbx_core.dim_products` p \r\n ON f.product_key = p.product_key\r\nINNER JOIN `my-lakehouse-project.glue_retail_db.glue_retail_db.dim_stores` s \r\n ON f.store_key = s.store_key;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb896ae0c90>)])]>

Analyze sales performance in natural language

When you ask conversational analytics in BigQuery:

"Who are our top customers by total sales revenue, what products did they buy, and which stores did they shop at?"

Conversational analytics evaluates the schema across all three remote catalogs, generates the SQL statement, executes the query, and presents the response.

Figure 3: Conversational analytics in BigQuery translates plain-language prompts into federated queries across Snowflake, Databricks, and AWS Glue.

Plan for caching and data residency

Before deploying federated catalogs in production, consider these operational behaviors:

Background refresh timing: The --refresh-interval flag controls metadata synchronization. The minimum interval is 300 seconds. For larger catalogs containing hundreds of tables, metadata retrieval can take longer. If a sync cycle overruns, BigQuery skips the next scheduled run to avoid task queuing.
Automatic query caching: BigQuery caches retrieved cross-cloud data blocks in the region where your federated catalog resides (e.g. us-central1). This helps reduce network transfer on repeated queries. Because cached blocks reside within Google Cloud storage in that region, confirm that this aligns with your organization's data residency policies.
Encryption controls: Cross-cloud cache blocks use Google-managed encryption keys. If your organization enforces the constraints/gcp.restrictNonCmekServices policy constraint, BigQuery bypasses the cache and reads directly from remote storage for every query.
Monitoring access metrics: You can evaluate cache utilization in your query execution plans. Check the objectStorageStats section in BigQuery job statistics to compare cacheBytesRead against objectStorageBytesRead.

Next steps

In summary, Google Cloud’s borderless Lakehouse helps you activate your data regardless of where it lives without costly migrations or building additional data pipelines. Get started today by reviewing the borderless Lakehouse architecture documentation to learn more about supported regions and credential configurations. You can work through the borderless Lakehouse codelab to practice federating external catalogs step-by-step. Finally, connect with your Google Cloud account team or customer engineer to evaluate borderless lakehouse patterns for your specific data platforms or industries.

Appendix

Setting up Snowflake and AWS Glue cross cloud data access

Connect Snowflake and Big Query

Configure Snowflake Horizon Catalog

We begin in Snowflake by creating the database, schema, and Iceberg table. In Snowflake Horizon, managed Iceberg tables expose an open Apache Iceberg REST catalog interface automatically:

code_block: <ListValue: [StructValue([('code', "-- Run inside Snowflake SQL worksheet\r\nCREATE DATABASE IF NOT EXISTS sf_retail_db;\r\nCREATE SCHEMA IF NOT EXISTS sf_retail_db.core;\r\n\r\n-- Create the Iceberg dimension table\r\nCREATE OR REPLACE ICEBERG TABLE sf_retail_db.core.dim_customers (\r\n customer_key INTEGER,\r\n customer_id VARCHAR,\r\n first_name VARCHAR,\r\n last_name VARCHAR,\r\n email VARCHAR,\r\n gender VARCHAR,\r\n country VARCHAR,\r\n signup_date DATE\r\n\r\n)\r\nCATALOG = 'SNOWFLAKE'\r\nEXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED';"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb896ae2810>)])]>

Before external engines can read this catalog, configure a dedicated reader role and grant traversal permissions across the database, schema, and table.

SQL

code_block: <ListValue: [StructValue([('code', '-- Create a dedicated reader role\r\nCREATE ROLE IF NOT EXISTS gcp_lakehouse_reader_role;\r\n\r\n-- Grant database and schema traversal permissions\r\nGRANT USAGE ON DATABASE sf_retail_db TO ROLE gcp_lakehouse_reader_role;\r\nGRANT USAGE ON SCHEMA sf_retail_db.core TO ROLE gcp_lakehouse_reader_role;\r\n\r\n-- Grant read access on the target table\r\nGRANT SELECT ON TABLE sf_retail_db.core.dim_customers TO ROLE gcp_lakehouse_reader_role;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb896ff1ad0>)])]>

Next, create the federated catalog inside Google Cloud using the gcloud command-line tool. You will need your Snowflake account identifier, the reader role, and the root Iceberg database name:

bash

code_block: <ListValue: [StructValue([('code', 'export GCP_PROJECT_ID="my-lakehouse-project"\r\nexport GCP_REGION="us-central1"\r\n\r\ngcloud alpha biglake iceberg catalogs create sf_retail_db \\\r\n --project="$GCP_PROJECT_ID" \\\r\n --primary-location="$GCP_REGION" \\\r\n --catalog-type="federated" \\\r\n --federated-catalog-type="snowflake" \\\r\n --snowflake-account-identifier="xy12345-ab67890" \\\r\n --snowflake-warehouse="SF_RETAIL_DB" \\\r\n --snowflake-role="gcp_lakehouse_reader_role" \\\r\n --refresh-interval="300s"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb8a8a9d790>)])]>

In the Apache Iceberg REST specification, --snowflake-warehouse references the catalog root database name (SF_RETAIL_DB), not a virtual compute warehouse.

Once created, describe the catalog to view its assigned numeric service account ID:

bash

code_block: <ListValue: [StructValue([('code', 'gcloud alpha biglake iceberg catalogs describe sf_retail_db \\\r\n --project="$GCP_PROJECT_ID" \\\r\n --format=\'value(biglake-service-account-id)\''), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb896ff3c50>)])]>

Return to Snowflake to bind Google Cloud's numeric identity to a passwordless service user via Workload Identity Federation (WIF):

SQL

code_block: <ListValue: [StructValue([('code', "-- Run inside Snowflake SQL worksheet\r\nUSE ROLE ACCOUNTADMIN;\r\n\r\n-- Bind Google Cloud's numeric identity to a passwordless service user\r\nCREATE OR REPLACE USER snowflake_service_user\r\n TYPE = SERVICE\r\n WORKLOAD_IDENTITY = (\r\n TYPE = GCP\r\n SUBJECT = '102669xxxxxxxxxxxxxxxxx' -- Numeric subject ID returned by gcloud describe\r\n )\r\n DEFAULT_ROLE = gcp_lakehouse_reader_role\r\n COMMENT = 'Service user for Lakehouse catalog federation over WIF';\r\n\r\n-- Grant the reader role to this service user\r\nGRANT ROLE gcp_lakehouse_reader_role TO USER snowflake_service_user;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb896ff3a10>)])]>

This establishes passwordless authentication over OpenID Connect, allowing BigQuery to discover table metadata and vend temporary query-time credentials directly from Snowflake Horizon.

Configure AWS Glue Data Catalog

We begin in AWS by preparing the IAM identity, establishing access permissions across AWS Glue and Amazon S3, and registering the Iceberg table.

First, create a placeholder trust policy named trust_policy.json. This allows you to provision the AWS IAM role before Google Cloud assigns the unique catalog service agent:

code_block: <ListValue: [StructValue([('code', '{\r\n "Version": "2012-10-17",\r\n "Statement": [\r\n {\r\n "Effect": "Allow",\r\n "Principal": {\r\n "Federated": "accounts.google.com"\r\n },\r\n "Action": "sts:AssumeRoleWithWebIdentity",\r\n "Condition": {\r\n "StringEquals": {\r\n "accounts.google.com:aud": ["PLACEHOLDER_VALUE"],\r\n "accounts.google.com:sub": ["PLACEHOLDER_VALUE"]\r\n }\r\n }\r\n }\r\n ]\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb896ff3210>)])]>

Create the IAM role using the AWS CLI. Setting a 12-hour session duration (43200 seconds) prevents credential expiration during long-running analytical queries:

code_block: <ListValue: [StructValue([('code', 'export AWS_ROLE_NAME="lakehouse_glue_federation_role"\r\n\r\naws iam create-role \\\r\n --role-name "$AWS_ROLE_NAME" \\\r\n --assume-role-policy-document file://trust_policy.json \\\r\n --max-session-duration 43200'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb896ff3e10>)])]>

Next, define the permissions policy in a file named permissions_policy.json. This policy grants BigQuery permission to read the AWS Glue catalog metadata and access the underlying Parquet data files in Amazon S3:

code_block: <ListValue: [StructValue([('code', '{\r\n "Version": "2012-10-17",\r\n "Statement": [\r\n {\r\n "Sid": "GlueCatalogRead",\r\n "Effect": "Allow",\r\n "Action": [\r\n "glue:GetCatalog",\r\n "glue:GetDatabase",\r\n "glue:GetDatabases",\r\n "glue:GetTable",\r\n "glue:GetTables"\r\n ],\r\n "Resource": "arn:aws:glue:us-east-2:123456789012:*"\r\n },\r\n {\r\n "Sid": "S3BucketAndObjectRead",\r\n "Effect": "Allow",\r\n "Action": [\r\n "s3:ListBucket",\r\n "s3:GetObject"\r\n ],\r\n "Resource": [\r\n "arn:aws:s3:::my-glue-iceberg-bucket",\r\n "arn:aws:s3:::my-glue-iceberg-bucket/*"\r\n ]\r\n }\r\n ]\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb896ff1e50>)])]>

Attach this permissions policy to your newly created role:

code_block: <ListValue: [StructValue([('code', 'aws iam put-role-policy \\\r\n --role-name "$AWS_ROLE_NAME" \\\r\n --policy-name "lakehouse_glue_federation_policy" \\\r\n --policy-document file://permissions_policy.json'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb896ff1650>)])]>

Important (AWS Lake Formation): If your AWS environment uses AWS Lake Formation, verify that the IAMAllowedPrincipals group has access to the target database and table (or grant explicit Lake Formation permissions to lakehouse_glue_federation_role if the bucket is registered with AWS Lake Formation). Without this, Lake Formation security settings will block the IAM role from accessing catalog metadata.

With permissions in place, create the database and Iceberg table in AWS Glue using Amazon Athena:

code_block: <ListValue: [StructValue([('code', "-- Run inside Amazon Athena\r\nCREATE DATABASE IF NOT EXISTS glue_retail_db;\r\n\r\nCREATE TABLE IF NOT EXISTS glue_retail_db.dim_stores (\r\n store_key INT,\r\n store_id STRING,\r\n store_name STRING,\r\n city STRING,\r\n state STRING,\r\n region STRING,\r\n manager STRING,\r\n opened_date DATE\r\n\r\n)\r\nLOCATION 's3://my-glue-iceberg-bucket/retail/dim_stores/'\r\nTBLPROPERTIES (\r\n 'table_type'='ICEBERG', \r\n 'format'='parquet', \r\n 'write_compression'='zstd' \r\n\r\n);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb8971b0b90>)])]>

Connect AWS Glue to BigQuery

Next, register the federated catalog inside Google Cloud using the gcloud command-line tool. You will need your 12-digit AWS Account ID, AWS Region, and the role ARN created above:

code_block: <ListValue: [StructValue([('code', 'export GCP_PROJECT_ID="my-lakehouse-project"\r\nexport GCP_REGION="us-central1"\r\nexport AWS_REGION="us-east-2"\r\nexport AWS_ACCOUNT_ID="123456789012"\r\n\r\ngcloud alpha biglake iceberg catalogs create glue_retail_db \\\r\n --project="$GCP_PROJECT_ID" \\\r\n --primary-location="$GCP_REGION" \\\r\n --catalog-type="federated" \\\r\n --federated-catalog-type="glue" \\\r\n --glue-warehouse="$AWS_ACCOUNT_ID" \\\r\n --glue-aws-region="$AWS_REGION" \\\r\n --glue-aws-role-arn="arn:aws:iam::${AWS_ACCOUNT_ID}:role/lakehouse_glue_federation_role"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb897e27390>)])]>

To prevent premature background refresh failures while the AWS trust policy propagates, initialize the catalog without setting a refresh schedule.

Next, extract the unique numeric subject ID assigned to the catalog by Google Cloud:

code_block: <ListValue: [StructValue([('code', 'export LAKEHOUSE_SA_ID=$(gcloud alpha biglake iceberg catalogs describe glue_retail_db \\\r\n --project="$GCP_PROJECT_ID" \\\r\n --format="value(biglake-service-account-id)")\r\n\r\necho "Assigned Lakehouse Service Account ID: $LAKEHOUSE_SA_ID"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb897e26ed0>)])]>

Return to the AWS CLI and update your IAM role's trust policy, replacing the placeholder with the verified Google Cloud identity:

code_block: <ListValue: [StructValue([('code', 'cat > trust_policy_final.json << EOF\r\n{\r\n "Version": "2012-10-17",\r\n "Statement": [\r\n {\r\n "Effect": "Allow",\r\n "Principal": {\r\n "Federated": "accounts.google.com"\r\n },\r\n "Action": "sts:AssumeRoleWithWebIdentity",\r\n "Condition": {\r\n "StringEquals": {\r\n "accounts.google.com:aud": [\r\n "$LAKEHOUSE_SA_ID"\r\n ],\r\n "accounts.google.com:sub": [\r\n "$LAKEHOUSE_SA_ID"\r\n ]\r\n }\r\n }\r\n }\r\n ]\r\n}\r\nEOF\r\n\r\naws iam update-assume-role-policy \\\r\n --role-name "$AWS_ROLE_NAME" \\\r\n --policy-document file://trust_policy_final.json'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb8968f6f50>)])]>

5. Enable metadata synchronization and verify

bash

code_block: <ListValue: [StructValue([('code', 'gcloud alpha biglake iceberg catalogs update glue_retail_db \\\r\n --project="$GCP_PROJECT_ID" \\\r\n --refresh-interval="300s"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb8968f7e50>)])]>

Verify that the namespaces are discovered and synchronized:

bash

code_block: <ListValue: [StructValue([('code', '# Verify namespaces synchronized from AWS Glue\r\ngcloud alpha biglake iceberg namespaces list \\\r\n --project="$GCP_PROJECT_ID" \\\r\n --catalog="glue_retail_db"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb8968f51d0>)])]>

Introducing TimesFM 3.0 with multivariate time-series forecasting in BigQuery

Mon, 12 Oct 2026 16:00:00 +0000

Overview

Since introducing zero-shot time-series forecasting in BigQuery with AI.FORECAST — initially powered by TimesFM 2.0 and later upgraded to TimesFM 2.5 — you have been able to generate forecasts directly in SQL without training or managing custom models. However, up until TimesFM 2.5, these foundation models were limited to univariate forecasting, predicting future values from the history of a single time series alone. In practice, most real-world forecasting problems are multivariate, where related targets and external drivers like weather, promotions, and holidays jointly shape future outcomes.

Today, we are thrilled to announce the preview of TimesFM 3.0 in BigQuery. Developed by Google Research, TimesFM 3.0 is a 330-million parameter state-of-the-art foundation model designed to unlock native multivariate time-series forecasting directly inside BigQuery SQL. TimesFM 3.0 ranks #1 across all three major public time-series leaderboards — GIFT-Eval, FEV-Bench, and TIME — achieving top zero-shot accuracy on both covariate and multivariate forecasting tasks compared to existing foundation models.

Alongside its new multivariate capabilities, TimesFM 3.0 also serves as a drop-in upgrade for univariate forecasting via AI.FORECAST and anomaly detection via AI.DETECT_ANOMALIES. Existing workflows can immediately benefit from higher benchmark accuracy simply by specifying model => 'TimesFM 3.0'.

What’s new: Multivariate foundation modeling

With TimesFM 3.0, BigQuery brings enterprise-grade multivariate modeling into familiar SQL workflows via AI.FORECAST:

Capability	TimesFM 2.0 / 2.5	TimesFM 3.0
Forecasting scope	Univariate only	Univariate and multivariate
Target columns	Single target	Single or multiple targets
Covariate columns with past-only values	Not supported	Supported
Covariate columns with past and future values	Not supported	Supported

Multivariate forecasting with BigQuery SQL

To see how these capabilities work together, let’s forecast daily commuter (subscriber_trips) and leisure (casual_trips) bike rides over a seven-day horizon using the NYC Citi Bike public dataset (bigquery-public-data.new_york.citibike_trips).

BigQuery uses a single-table input format for multivariate forecasting that combines two sets of rows:

Historical observation rows: timestamps between start time (2016-01-01) and the forecast cutoff date (2016-06-01), fully populated with target metrics, past covariates, and known future covariates
Future horizon rows: timestamps across the seven-day forecast horizon (2016-06-01 to 2016-06-07) populated with known future covariates, while target columns and past-only covariates are set to NULL

code_block: <ListValue: [StructValue([('code', "WITH data AS (\r\n SELECT\r\n DATE(starttime) AS date,\r\n IF(MIN(starttime) < '2016-06-01', COUNTIF(usertype = 'Subscriber'), NULL) AS subscriber_trips,\r\n IF(MIN(starttime) < '2016-06-01', COUNTIF(usertype = 'Customer'), NULL) AS casual_trips,\r\n IF(MIN(starttime) < '2016-06-01', AVG(tripduration / 60.0), NULL) AS avg_duration_min,\r\n IF(EXTRACT(DAYOFWEEK FROM MIN(starttime)) IN (1, 7), 1.0, 0.0) AS is_weekend\r\n FROM `bigquery-public-data.new_york.citibike_trips`\r\n WHERE starttime >= '2016-01-01' AND starttime < '2016-06-08'\r\n GROUP BY date\r\n)\r\nSELECT\r\n *\r\nFROM\r\n AI.FORECAST(\r\n TABLE data,\r\n MODEL => 'TimesFM 3.0',\r\n TIMESTAMP_COL => 'date',\r\n TARGET_COLS => ['subscriber_trips', 'casual_trips'],\r\n PAST_COVARIATE_COLS => ['avg_duration_min'],\r\n FUTURE_COVARIATE_COLS => ['is_weekend']\r\n );"), ('language', 'lang-sql'), ('caption', <wagtail.rich_text.RichText object at 0x7fb896d162d0>)])]>

This query highlights the three multivariate dimensions supported in TimesFM 3.0:

Forecasting correlated targets (target_cols) Instead of running separate models for each rider segment, passing TARGET_COLS => ['subscriber_trips', 'casual_trips'] forecasts both metrics jointly in a single call. TimesFM 3.0 captures the cross-series dynamics between commuter and casual riders — such as shared weather sensitivity and diverging weekend ridership patterns.
Using past covariates (past_covariate_cols) Some signals can only be observed historically and are not known in advance, such as average ride duration (avg_duration_min), historical foot traffic, or unplanned service disruptions. Passing PAST_COVARIATE_COLS => ['avg_duration_min'] (with future horizon rows set to NULL) gives the model historical context to distinguish temporary shocks from underlying trends without requiring future values for that feature.
Using future covariates (future_covariate_cols) Other external drivers are known ahead of time across both the historical window and the forecast horizon — such as weekends (is_weekend), holidays, or scheduled promotions. Passing FUTURE_COVARIATE_COLS => ['is_weekend'] (populated across all rows through 2016-06-07) allows TimesFM 3.0 to condition its seven-day predictions directly on upcoming calendar events.

Understanding the output

The function returns structured predictions for each target column specified in target_cols. For each column in target_cols, AI.FORECAST returns a struct containing the point forecast (value) and the lower and upper prediction interval bounds:

date	subscriber_ trips.value	subscriber_trips.prediction_ interval_lower_bound	subscriber_trips.prediction_ interval_upper_bound	casual_trips.value	casual_trips.prediction_ interval_lower_bound	casual_trips.prediction_ interval_upper_bound	time_series_type	confidence_level	ai_forecast_status
2016-06-01 00:00:00 UTC	44699.64	32590.76	55723.44	4581.32	1727.91	9189.31	forecast	0.95
2016-06-02 00:00:00 UTC	44611.75	30413.05	57209.19	4647.17	1466.23	10273.55	forecast	0.95
2016-06-03 00:00:00 UTC	41379.15	26218.65	54940.09	5341.36	1568.23	11805.86	forecast	0.95
2016-06-04 00:00:00 UTC	30164.34	14266.43	45075.92	9097.74	2558.27	18452.52	forecast	0.95
2016-06-05 00:00:00 UTC	26727.27	10101.44	42239.33	8398.50	1692.15	17768.99	forecast	0.95
2016-06-06 00:00:00 UTC	36713.84	19493.00	52679.59	4723.08	400.08	11710.73	forecast	0.95
2016-06-07 00:00:00 UTC	39749.97	22157.42	56119.86	4180.97	95.70	10845.74	forecast	0.95

TimesFM 3.0 in BigQuery Conversational Analytics

You can also run multivariate forecasts using natural language in BigQuery Conversational Analytics. On the same bigquery-public-data.new_york.citibike_trips dataset, you can simply ask "Forecast both daily subscriber and casual trips for June 1–7, 2016 using two month historical data, with average trip duration as a past covariate and weekend indicator as a future covariate." Conversational Analytics automatically generates and executes the underlying AI.FORECAST query with TimesFM 3.0, returning a summary of the results, interactive forecast charts with prediction intervals for each target, and key forecast insights.

Multivariate forecasting with TimesFM 3.0 in BigQuery Conversational Analytics, generating 7-day predictions, interactive charts, and insights from a natural-language prompt.

Summary and next steps

With native multivariate support in TimesFM 3.0, BigQuery brings state-of-the-art foundation model forecasting to complex, interconnected enterprise datasets without requiring data movement, custom feature engineering pipelines, or dedicated model hosting infrastructure.

To start forecasting with TimesFM 3.0 in BigQuery:

Read the Google Research blog post to learn more about the TimesFM 3.0 model architecture and capabilities.
Explore the BigQuery TimesFM documentation.
Follow the step-by-step multivariate forecasting tutorial.
Try AI.FORECAST with `model => 'TimesFM 3.0'` directly in the BigQuery Studio SQL workspace.
Try AI.DETECT_ANOMALIES with `model => 'TimesFM 3.0'` for univariate time series anomaly detection in BigQuery.
Try AI.EVALUATE with `model => 'TimesFM 3.0'` for univariate time series evaluation in BigQuery.

Agentic data science with the Data Agent Kit

Mon, 12 Oct 2026 16:00:00 +0000

Imagine your compliance team notices that payments are taking longer than expected to clear. They suspect fraud, and you’re the data scientist helping figure out which transactions to investigate first. Assembling that shortlist requires ingesting and cleaning raw transaction logs, training a model on past fraud, scoring new records, and scheduling the pipeline to run daily. Here’s how you can build that workflow end to end with an agent, using the Data Agent Kit inside the Antigravity IDE.

Introducing the Data Agent Kit

Data Agent Kit helps you work with Google Cloud data through a plugin for coding agents or an extension for Visual Studio Code-based IDEs. It brings two significant additions to the workflow:

Help for your agent: MCP tools connect your agent to data services. Skills are Markdown files that give it instructions and best practices for working with your data stack
Tools in your IDE: work with Google’s Agentic Data Cloud, from querying databases to deploying an Airflow pipeline, without leaving your editor

Loading the data

Let’s start with the raw transaction logs in Google Cloud Storage. You can ask the agent to create a notebook that loads them into BigQuery using Spark:

code_block: <ListValue: [StructValue([('code', 'Create a PySpark notebook to ingest JSON transaction logs from gs://financials-clearing-raw/ into the BigQuery table transactions.raw_transactions.'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb8a8db8a50>)])]>

The agent generates a Jupyter notebook that loads the JSON logs into a BigQuery table named raw_transactions. You can open the notebook, review the code, and run it using a preconfigured Spark Serverless runtime from the IDE.

Running a generated PySpark notebook

After the notebook runs, you can open the raw_transactions table with the Data Agent Kit. The extension allows you to browse the table and inspect its metadata right from the IDE.

Inspecting a BigQuery table in the IDE

The records are now in BigQuery, but they aren’t ready for model training yet. First, we’ll handle duplicate and incomplete records, then join the remaining records with info about who sent and received each payment.

Cleaning it up

Let’s put the cleanup logic into a dbt project so we can run it on each new batch of data. We’ll also test that each transaction appears only once. Prompt the agent:

code_block: <ListValue: [StructValue([('code', 'Create a dbt project in dbt_project/ for the BigQuery dataset transactions.\r\n\r\nDeduplicate raw_transactions by transaction_id in staging. Separate records with null IDs into invalid_transactions. Join valid records with dim_payers and dim_payees to create enriched_transactions, preserving the is_fraud label. Add a transaction uniqueness test.'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb8a8dba8d0>)])]>

The agent generates a dbt project with SQL models for cleaning and enriching the data. Take a look at the generated code to check how it handles duplicates and joins reference tables. Then run dbt build in the terminal. This builds the staging and enriched tables in BigQuery, along with a separate table for records with missing transaction IDs. dbt also runs the uniqueness test.

Validating dbt models in the integrated terminal

We also keep the is_fraud column, which indicates whether a past transaction was fraudulent. We’ll train the model on records with known labels and use it to score new transactions, where we’re unsure yet whether fraud occurred.

Building the review queue

With the data prepared, let’s ask the agent to create a training notebook using a Random Forest classifier running on Spark Serverless.

code_block: <ListValue: [StructValue([('code', 'Create a PySpark notebook to train a RandomForestClassifier\r\non table transactions.enriched_transactions.\r\n\r\nPredict is_fraud, training only on records where the label\r\nis not null. Encode categorical features, scale amounts,\r\nand evaluate the model using ROC AUC. Save the evaluated model to gs://models/fraud_model.'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb8a8db9590>)])]>

Review and run the notebook, then check its evaluation output to see how well the model separates fraudulent transactions from legitimate ones. You can also use the Data Agent Kit extension to verify that the trained model artifacts were saved to Google Cloud Storage.

Verifying the trained model artifacts in Google Cloud Storage

Now we can use the saved model to score transactions that don’t have an is_fraud value yet:

code_block: <ListValue: [StructValue([('code', 'Create a batch inference notebook that loads the saved model from gs://my_models/fraud_model and scores records in transactions.enriched_transactions where is_fraud is null.\r\n\r\nWrite transactions with a fraud probability score >= 0.50 to the Spanner table SparkEvalFraudReviewQueue, in the instance cymbal-fraud, database fraud-db.'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb8a8db8290>)])]>

In this example, the inference notebook scores the unlabeled transactions in BigQuery and sends those with an estimated fraud probability of 50% or higher to the Spanner review queue. The flagged records help the team decide where to investigate, but they still need to determine whether fraud occurred.

Look at how the inference notebook filters and writes flagged records, then run it. Once it finishes, query the Spanner table in the IDE to inspect the results.

Querying flagged transactions in the Spanner table

This table is the list we set out to build. The compliance team can now start reviewing transactions, and we have a workflow that gets records from raw logs into the queue.

So far, we have been kicking off each step ourselves. We can now connect them so the workflow runs automatically each day.

Running it again tomorrow

Each day we will load the latest logs, apply the dbt transformations, and score unlabeled transactions using the saved model. We’ll use the Managed Service for Apache Airflow to run these steps in order. Ask the agent:

code_block: <ListValue: [StructValue([('code', 'Create an orchestration pipeline named fraud_analysis_pipeline that runs ingestion, the dbt project, and batch inference in that order.\r\n\r\nSchedule it daily at midnight. Use Spark Serverless for the notebooks, including the Spanner connector for inference, and gs://YOUR_ARTIFACT_BUCKET for artifact storage.'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb8a8dbaf10>)])]>

The agent generates a YAML pipeline definition and deployment configuration. The pipeline definition sets the schedule and dependencies between steps. Meanwhile, the deployment configuration specifies the Airflow environment where it will run.

Data Agent Kit then turns the definition into an Airflow DAG, which you can explore as an interactive graph in the IDE. Open the pipeline under “Orchestration Pipelines” and click a task to inspect it. For notebook tasks, you can follow the link straight back to the code.

Inspecting a data pipeline

With the Airflow environment configured, click “Run pipeline” in the canvas. Data Agent Kit packages the notebooks and dbt project, deploys the DAG, and triggers a run. Open the pipeline’s execution history to follow its progress. You can also select individual tasks to inspect their status, duration, and logs.

Kicking off an Airflow DAG and inspecting the logs

Airflow will run the pipeline every day, taking new batches of data from raw logs to the review queue. The compliance team can now focus on flagged transactions, and you’ve automated the work of getting those records to them!

Try it yourself

If you are ready to try this for yourself, we have designed a codelab that walks you through the full setup so you can build and run this workflow. You can also read our documentation or check out the skills and tools in the open-source repository on GitHub.

Trusted Data Apps for the Agentic Data Cloud, powered by Looker

Mon, 12 Oct 2026 16:00:00 +0000

As a business intelligence user, this probably sounds familiar: Most dashboards only provide you with a standardized canvas and a fixed library of charts, and the “analysis” consists of a CSV export that takes you to another system. Or sometimes it's designing the dashboard that’s the challenge: You can picture the executive briefing, the customer-facing portal, or the interactive explainer that you want, but can’t assemble it out of the available grid tiles.

Either way, the solution has always been to build a custom application. The advent of AI has sped up the process — a working prototype now takes an afternoon — but the excitement of that first demo fades fast. Without access to governed data, row-level security, an infrastructure owner, or a clear path through security review, a tool written by an LLM is simply a prototype that can never go to production.

Today, to bridge this gap, we are announcing Looker Data Apps (in preview) — pairing the speed of agent-generated applications with the governance, security, and trust of your existing Looker governance framework. Describe what you need in plain language, and the agent builds it as a real, production-grade application hosted natively inside Looker!

What is a data app?

A data app is a bespoke, interactive web application generated by an AI agent and hosted natively within Looker. While dashboards provide top-down visibility, data apps serve as an active system of action designed around a specific operational workflow. A data app:

Goes beyond standard grid tiles with pixel-perfect, bespoke layouts configured for a specific operational role or job-to-be-done.
Enables business users to input parameters, test scenarios, run live simulations, and update underlying records.
Connects end-to-end business processes by connecting data analysis to business execution tasks like updating source records or dispatching events to tools like Salesforce and Jira.

Build pixel-perfect interfaces using AI developer tools like Claude Code or Antigravity CLI, grounded directly by Looker’s semantic layer.

Expanding your semantic layer into an application platform

Data apps do not replace dashboards; they expand the boundaries of what you can build on top of your existing dashboards and semantic layer.

Dashboards excel at aggregate visibility and passive reporting. Data apps handle active execution, letting users modify parameters and trigger downstream business tasks.
Dashboards provide consistency using standardized chart libraries. Data apps deliver broad flexibility, unlocking custom web components and specialized interactions.
Data apps are grounded in governed data to ensure accuracy. Whether users are reviewing a KPI dashboard or adjusting budgets in a data app, all logic is governed centrally with the LookML semantic layer to eliminate metric discrepancies and hallucinations.

Transform analysis into action with customizable, multi-page web app experiences and embedded operational workflows.

Built for enterprise scale

We designed Looker Data Apps for demanding, governed enterprises.

Every measure is grounded in the semantic layer. Every app you build directly inherits your existing semantic models, row-level security, and access controls. Calculations stay consistent across the enterprise, and built-in query inspection lets teams audit every line of generated SQL to eliminate metric hallucinations.
Zero-code by default. Deploy agent-generated React applications directly inside Looker with no code. Or users can use LLMs like Claude Code and Antigravity CLI to build interactive apps and deploy them as data apps in minutes.
No runtime token costs. LLM tokens are only consumed when you’re designing or generating the app. After that, every end-user interaction runs on native Looker APIs and models. Put a data app in front of 50 people or 5,000 and the marginal cost doesn't change.
As easy as embedding a dashboard. You can deploy your data apps into external portals using the same workflows you use to embed a dashboard. No new pipelines, no custom application code, no second permission models.

From prototype to production in minutes

The days of having to choose between basic dashboards and investing valuable engineering time to build bespoke systems are coming to an end. Instead of letting prototypes stall in sandboxes due to compliance and security barriers, Looker Data Apps allows teams to deploy governed, production-grade tools in minutes.

Looker Data Apps are now in preview. Get started today.

Modernizing Unstructured Data Workflows: Alteryx Live Query meets Google Cloud BigQuery

Fri, 09 Oct 2026 16:00:00 +0000

Alteryx One Live Query and Google Cloud BigQuery redefine how enterprises handle complex, unstructured data at scale by reducing reliance on disconnected tools, minimizing data movement, and enabling warehouse-native execution.

Live Query provides an intuitive, browser-based environment that allows business users to build sophisticated data pipelines without writing a single line of code. When paired with the scale and performance of BigQuery, this duo enables secure SQL pushdown, meaning data transformation logic is executed directly within the warehouse rather than moving it across the network. This synergy is particularly potent for document-heavy workflows, where Live Query can orchestrate Google’s advanced AI and machine learning models — like Gemini — to extract intelligence from PDFs and images while maintaining the strict data governance and security of the BigQuery environment.

From invoice PDFs to govern exception reporting in BigQuery with Live Query

Finance and operations teams often find themselves buried under a mountain of vendor invoices. These documents come as PDFs with inconsistent data, varying invoice number formats, and a high risk of manual error. Traditionally, data teams have had to resort to manual data extraction or use a fragmented series of tools to move data into a warehouse — a process that is both time-consuming and prone to duplicates or amount mismatches.

With the integration of Live Query for BigQuery, you can now automate the entire lifecycle of an invoice. By processing PDFs, extracting key fields, and standardizing results directly within the BigQuery ecosystem, data teams can identify exceptions like anomalies or data issues within the documents faster while ensuring their governed enterprise data configured through the BigQuery fine grained access policies never has to leave the warehouse.

Solution overview: Extract, classify, validate, route, and post

Let’s dive deeper into how the Live Query solution works. Invoice PDFs arrive as unstructured documents in Google Cloud Storage. In the Live Query workflow, the Document Extract tool extracts structured fields using the model selected by the user, such as Gemini or Document AI. The results are then standardized for reliable matching and compared against historical invoice records already stored and governed in BigQuery.

This automated workflow is designed to move from document to decision with minimal manual intervention. At enterprise scale, that pattern is not just about processing a handful of PDFs — it is about enabling analytics and AI over millions of unstructured documents while keeping reconciliation logic aligned with BigQuery.

By combining AI-driven extraction with BigQuery-native execution, the solution provides:

Higher straight-through processing: Reducing manual handling and allowing more invoices to move through the process automatically.
Faster reconciliation and fewer manual exceptions: Comparing extracted invoices against BigQuery data earlier in the process so teams can identify duplicates and mismatches sooner, reducing the number of invoices that require manual review.
Stronger auditability: Maintaining a clear trail of data from source to operational output.
Reduced data movement: Keeping data resident in BigQuery and leverage its built-in capabilities for enhanced security, governance, and control.

Architecture: A Google Cloud-native pattern

At a high level, invoice PDFs are staged in Cloud Storage, processed through a Live Query workflow in the browser, reconciled against governed invoice history already resident in BigQuery, and written back as operational outputs for finance teams. The key architectural advantage of this pattern is that workflow authoring happens in the browser, while transformation and reconciliation logic are executed through platform services against the governed data environment.

The Live Query architecture is designed for warehouse-scale processing and governed execution. It uses secure SQL pushdown to execute transformation and reconciliation logic directly in BigQuery, and integrates with Gemini for advanced AI capabilities such as document extraction, classification, and summarization.

Live Query (Browser): Users author workflows visually in the browser, configuring document extraction, classification, cleansing, validation, and routing logic.
Alteryx One Platform Services: The Transformation Service coordinates request handling, execution planning, and result retrieval between the browser workflow and the customer data environment. The services also add traceability and observability across workflow execution, strengthening both operational monitoring and auditability.
Customer Google Cloud: Invoice PDFs are staged in Google Cloud Storage (GCS), historical invoice data remains governed in BigQuery, AI-backed extraction runs against that environment, and the workflow outputs are written back into BigQuery.

In this pattern, the browser is the authoring layer, Alteryx One Platform Services are the coordination layer, and Google Cloud is the governed data and execution layer. It allows workflow logic to be authored visually in Live Query while execution remains aligned with governed data and warehouse-scale processing in BigQuery.

Live query workflow walkthrough: From PDF to insight

Let’s take a closer look at how the Live Query workflow handles unstructured PDFs, using an invoice as the example.

Ingest invoice PDFs
The workflow begins by pointing to a directory input in Google Cloud Storage, where the raw PDF files are stored. In practice, this step does more than just identify a folder: it enumerates the PDF locations and returns file-level metadata — such as absolute file paths, size, and created / updated timestamps — so the workflow can reliably pass the right document set into downstream extraction steps.

Extract key fields
Using Alteryx’s Document Extract tool, the workflow processes PDF invoices stored in Google Cloud Storage and extracts structured or semi-structured results directly in BigQuery. The extraction step uses the user-selected model, either Document AI or Gemini AI, depending on the workflow configuration. Under the hood, the tool builds a temporary external object table from the input document URIs and executes the appropriate function — ML.PROCESS_DOCUMENT for Document AI model or AI.GENERATE for Gemini AI model. For this workflow, the Gemini path is used, with AI.GENERATE returning the requested invoice fields into the downstream Live Query workflow.

During interactive design, Live Query limits processing to a small subset of document URIs for preview, while runtime execution processes the full set of invoice PDFs.

code_block: <ListValue: [StructValue([('code', "-- illustrative pattern only\r\nSELECT *\r\nFROM AI.GENERATE(\r\n MODEL project.dataset.gemini_model,\r\n (\r\n SELECT\r\n uri,\r\n 'Extract invoice_number, invoice_amount, vendor_name, net_amount, tax_amount, and line_items.' AS prompt\r\n FROM project.dataset.temp_invoice_object_table\r\n )\r\n);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb8a8a1b690>)])]>

Classify extracted content
After extraction, the workflow uses Alteryx’s Classify tool to apply zero-shot classification to the extracted invoice text. In this use case, line-item content is labeled into predefined business categories — such as industrial or office supplies — using AI.CLASSIFY in BigQuery, which returns a new predicted-label column into the downstream workflow. This business context makes the extracted data more useful for downstream reconciliation, exception handling, and reporting.

code_block: <ListValue: [StructValue([('code', "-- illustrative pattern only\r\nSELECT *,\r\n AI.CLASSIFY(line_item_text, categories => ['industrial', 'office']) AS line_item_classify\r\nFROM upstream"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb8a90ca090>)])]>

Normalize data
The extracted invoice numbers are cleansed and standardized so that minor formatting differences — such as whitespace or inconsistent formatting — do not hide potential duplicates during reconciliation.

Compare against history
The workflow joins the normalized current invoice records against historical invoice data already resident in BigQuery. This reconciliation step identifies records with matching invoice numbers already present in history and flags them as potential duplicates.

Flag exceptions
A formula step then applies business-rule logic to the remaining records. Those rules can vary by workflow and may include checks for amount mismatches, missing required fields, invalid totals, or other finance-specific exceptions. In this example, one rule checks whether invoice_amount matches net_amount + tax_amount and flags mismatches for review.

Produce operational outputs
The workflow produces two operational outputs: one for invoices matched to historical records and flagged as potential duplicates, and another for unmatched current invoices that have been validated, labeled for exceptions where needed, and prepared for unpaid-invoice processing.

The result is not just extracted data, but governed operational outputs that finance teams can review and act on immediately.

Summary: A governed document-to-decision workflow

Our joint customers are excited for these new capabilities that will help improve business processes.

“What stood out to me about Alteryx One: Google Edition was how naturally it fits into the Google Cloud experience. I'm excited about the opportunity to make analytics more accessible by enabling business users to work with BigQuery data through an intuitive, governed experience.” - Michael Wyant, Vice President, Enterprise Data & Corporate Solutions, Papa Johns

This use case is more than just a simple extraction tool; it represents a strong Google Cloud-native pattern for transforming unstructured documents into actionable operational outputs. Instead of moving warehouse data into disconnected preparation tools, the workflow keeps storage, extraction, reconciliation, and output generation aligned with governed enterprise data already resident in BigQuery. By combining the intuitive design of Alteryx One Live Query with the scale and AI capabilities of BigQuery, finance teams can reduce manual effort, accelerate exception handling, and focus on higher-value strategic initiatives.

Managed Apache Iceberg at scale: How Spanner powers Lakehouse runtime catalog

Tue, 06 Oct 2026 16:00:00 +0000

As customers modernize to lakehouse architectures, they are standardizing on open formats such as Apache Iceberg to create a shared data estate across compatible engines. This enables you to build AI-native lakehouses that turn data to semantic knowledge, enable proactive action, and operate at an agentic scale.

One of the key components of a lakehouse is the catalog, and in the Apache Iceberg environment, that usually means the Iceberg REST Catalog. An Apache Iceberg catalog is responsible for maintaining table pointers, handling atomic commits, and serving as the single source of truth for table locations. But as large enterprise organizations modernize to lakehouses, they have begun to realize that they need a highly scalable and available managed catalog as a part of their lakehouse architecture. This becomes even more important as querying scales with agents. To support a large amount of repeated small queries from agents, you will need to build on top of a managed catalog that provides atomicity, consistency, availability and concurrency at massive scale.

In this blog, we explore the challenges a managed catalog faces in modern cloud environments at agent scale, and show you how Google Cloud’s serverless Lakehouse runtime catalog can help address them. Powered by Spanner, Google Cloud’s always-on database with virtually unlimited scale, and built to meet the open Apache Iceberg REST catalog specification, the Lakehouse runtime catalog is the highly scalable and available foundation you need for the agentic era.

Challenges a managed catalog needs to solve in the Lakehouse

When speaking with data engineers and infrastructure leads running production analytics at scale in a Lakehouse, the following core pain points consistently emerge and need to be solved by a managed catalog:

Atomic commits and concurrency control: Iceberg guarantees ACID transactions via optimistic concurrency control (OCC). A managed catalog must implement a bulletproof atomic compare-and-swap (CAS) operation to swap the current metadata pointer.

High availability and operational maintenance: Because queries fail immediately if the catalog is down, a managed catalog becomes a critical Tier-1 service.

Scaling the database backing the catalog: Catalog architects typically face a difficult trade-off when choosing a backing database for table metadata and state. Traditional scale-up relational databases provide SQL and ACID transactions, but hit vertical CPU, memory, storage and connection limits under heavy concurrent read/write loads unless manually sharded which incurs a huge operational overhead; while scale-out database systems are either eventually consistent, hard to manage, not enterprise-ready, or all of the above.

Table maintenance coordination: A catalog alone does not optimize data; you must build and operate ancillary pipelines for compaction, snapshot expiration, manifest rewriting, and orphan file cleanup.

Governance and security: A catalog acts as the security gatekeeper. The catalog must implement and maintain:

Authentication protocols (e.g., OAuth2 token exchange, IAM federation)
Access control down to namespace and table levels
Vended storage credentials (e.g., generating short-lived tokens so query engines don't need broad, direct storage credentials)

Lakehouse runtime catalog

To solve these challenges, we built the Lakehouse runtime catalog (GA) with support for Iceberg Rest Catalog. The Lakehouse runtime catalog is a fully serverless, highly available, and unified metadata registry designed from the ground up to support modern open table formats like Apache Iceberg. By natively implementing the Apache Iceberg REST Catalog specification, the Lakehouse runtime catalog decouples metadata discovery from compute engines, helping ensure multiple Iceberg-compatible engines can access a shared data estate and enabling you to take your workloads to production sooner. We’ve helped many customers streamline the migration of their managed catalogs. For example, Etsy migrated its catalog to Lakehouse runtime catalog, joining data in place to accelerate pipeline queries by 60%.

This approach offers a number of architectural benefits:

Open APIs: Support for Iceberg Rest Catalog enables different teams to use their preferred analytics tools on a single, unified dataset.
Multi-engine interoperability: Once registered, tables are immediately discoverable and queryable across Google Cloud Managed Service for Apache Spark, BigQuery, and open-source engines via standard REST interfaces.
Read/write interoperability for Iceberg tables: Leverage Iceberg-compatible engines such as BigQuery, Managed Spark to write to Iceberg tables registered in the Lakehouse runtime catalog. Customers can also use Managed Spark to write to Iceberg tables in external catalogs.
Fully managed Iceberg storage with enterprise-grade features: Use Google's differentiated infrastructure to run analytics with performance on Iceberg tables. This gives you the benefits of open-source flexibility plus performance, scale, governance, and multimodal processing.
Zero data copy: Table definitions point directly to your existing data in the underlying object store. You do not move, rewrite, or duplicate your underlying data.
Bi-directional catalog federation across clouds: Access data from Databricks Unity, Snowflake Horizon and AWS Glue with support for vended credentials and OIDC token exchange. This lets you bring Google AI directly to your AWS and Azure data.
Secure access using credential vending: The catalog supports multiple authorization mechanisms, letting you choose between credential vending and end-user credentials. This means that you can access tables with modern mechanisms such as credential vending without needing direct access to the files in the underlying object store (Cloud Storage, AWS S3, Azure Blob Storage).
AI-powered context and governance: The Lakehouse runtime catalog integrates directly with Knowledge Catalog and Cloud IAM, allowing you to define trusted context for your agents, and apply table-level security consistently across all compute engines. Get out-of-the-box search, lineage, and insights for Iceberg tables in the catalog.
Atomic commits and concurrency control, high availability and scalability: Backed by Google’s planet-scale infrastructure and Spanner, you get the high availability, concurrency and scale you need for your metadata to scale with your data. Support for Cloud Storage dual-region and multi-region buckets enables failover use cases. It also provides reduced TCO due to serverless and no-ops environments, and scalability for any workload size.

What powers the Lakehouse runtime catalog?

The Lakehouse runtime catalog is a highly available, concurrent and scalable catalog with strong consistency guarantees because it is built on top of Spanner. Unlike traditional scale-up relational databases that hit vertical single-node ceilings, Spanner combines full relational SQL semantics and multi-table ACID transactions with the horizontal scale-out elasticity for both reads and writes of a NoSQL system.

Spanner makes the Lakehouse runtime catalog highly available through Spanner’s regional configurations with up to 99.99% availability. Spanner also delivers out-of-the-box scalability for Lakehouse runtime catalog: As a horizontally scalable database, Spanner does not require manual sharding and scales compute and storage independently and transparently. Spanner dynamically monitors data volume and query load, splitting and redistributing data ranges across nodes. Compute nodes scale dynamically based on CPU utilization and storage thresholds. Spanner automatically detects split-level overload and moves heavy splits away from overloaded nodes. Spanner also lets Lakehouse runtime catalog users eliminate the traditional trade-offs between relational consistency and distributed scalability, delivering Lakehouse runtime catalog’s industry-leading consistency guarantees. Lakehouse transactions are serializable — the order of transactions within the database is the same as the order in which clients observe the transactions to have been committed. This foundation allows Lakehouse users to operate at agent-scale.

Then, to further power agentic use cases, the Lakehouse runtime catalog integrates directly with Knowledge Catalog to easily discover lakehouse Iceberg tables and provide trusted context to agents. Knowledge Catalog leverages an efficient combination of full-text search and native vector search provided by Spanner; this approach enables better recall for search retrieval, pairing lexical keyword searches with semantic embeddings in a single query. Because both index types are built on the identical base dataset, they update with strict, transactional ACID consistency alongside base table DML operations. This removes operational overhead such as managing sync pipelines, and external-vector and full-text search systems. In short, the Lakehouse runtime catalog provides faster time-to-market for your agentic use cases.

Combine analytical and operational workloads

With Google Cloud’s borderless Lakehouse based on Apache Iceberg, you can combine your analytical data with your operational workloads. Use cases span combining data assets from your Lakehouse with OLTP data (from Spanner) for analytics, to low-latency serving applications where your Lakehouse assets are accessible in an operational database such as Spanner, to conversational analytics in first-party and third-party agents.

Below, in an example, you can see the Lakehouse runtime catalog with Spanner in action. Here, we combine analytical data for taxi trips in Manhattan (backed by Apache Iceberg) with operational data for taxi zones in Spanner to find the most congested traffic routes. The example also shows that you can also use a Conversational Analytics Agent to access the same data and get second-order insights.

Find out most congested traffic routes in Manhattan

Modernize to the borderless Lakehouse

Modernizing to Google Cloud’s Lakehouse minimizes data silos across analytics engines and agents, unifies multi-engine governance, provides trusted context to your agents and slashes operational TCO. Leveraging the Spanner-based Lakehouse runtime catalog helps prepare your modern cloud environments to operate at agent-scale. To learn more and get started with a free trial, visit the

Lakehouse web page and learn more about Spanner here.

Accelerating analytics: PayPal’s journey with Managed Service for Apache Spark

Thu, 01 Oct 2026 16:00:00 +0000

In a data-driven world, PayPal’s ability to deliver timely and actionable insights is central to staying ahead. At PayPal, data powers everything from fraud detection to user experience enhancements. Data is also central to unleashing the potential of agentic solutions and experiences.

Over time, though, our analytics environment had become a complex ecosystem of various technologies and solutions assembled on-premise to address growing demands. While this approach supported our needs at the time, it began presenting new challenges to scale and maintain.

Navigating a challenging analytics landscape

Due to expedited growth and acquisitions, our data analytics platform gradually turned into an uneven landscape. Each new platform or integration addressed a specific business need, but together, they increased operational overhead and introduced performance blockages. Scalability became increasingly difficult, and time-to-insight slowed as processes grew more complex.

Complexity breeds stagnation

PayPal’s legacy data analytics platform was powerful—handling petabytes daily—but it was also increasingly rigid following rapid growth. Scaling up during peak retail events or global launches meant months of planning, slow manual provisioning of hardware, and too often, a compromise between speed and cost.

As PayPal continued to scale globally, we recognized the need for a streamlined, unified infrastructure to drive data efficiency and accelerate innovation.

The solution: Unified, cloud-native analytics

To overcome these obstacles, we migrated our analytics workloads from legacy Hadoop on-premise platforms to Google’s Managed Service for Apache Spark.

Key reasons for this choice included:

Rapid provisioning and elastic scaling: Managed Spark enabled us to deploy clusters in minutes and scale based on processing needs, eliminating lengthy setup and idle resource costs.
Unified infrastructure: Standardizing on Apache Spark created consistency across teams while leveraging Managed Service for Apache Spark and other managed services reduced operational complexity.
Seamless integration: Native hooks into Google Cloud Storage (GCS), BigQuery, and other Google Cloud services streamlined end-to-end data movement.

This move enabled PayPal to modernize our data processing capabilities, leveraging the flexibility, scalability, and reliability of cloud-native solutions. By consolidating previously disparate workflows and batch jobs that run on multiple platforms onto a single cloud-based analytics platform, we reduced data silos and built a unified data foundation that provides faster, richer insights.

This empowered developers and application teams to focus on delivering business value rather than being limited by infrastructure. Crucially, this shift was about more than re-platforming. We fostered a new culture of experimentation, enabling teams to test, tune, and deploy analytics workloads quickly in response to changing business needs.

The results: Faster insights, lower overhead

The impact of our modernized Google Cloud-based ecosystem leveraging Managed Spark has been profound:

Processing times for core analytics workloads improved by 25%, enabling near real-time insights for key business operations.
SLA adherence rose substantially by 30%, even during traffic surges such as seasonal sales events.
Operational costs dropped as we consolidated tooling and reduced manual maintenance.

But perhaps most importantly, our engineers now spend less time firefighting and more time innovating, rapidly prototyping new analytics capabilities that deliver value to customers and partners.

Transitioning from a fragmented environment to a cohesive, cloud-native platform has fundamentally strengthened PayPal’s analytics capabilities. As business needs evolve, investing in a scalable, unified data foundation ensures that we can deliver insights with speed, precision, and impact—driving continued innovation for customers worldwide. Our journey with Managed Service for Apache Spark is an important step in building that modern analytics foundation.

Learn more about how you can get started with Managed Service for Apache Spark and BigQuery to build your Agentic Data Cloud today.

Empower your agents with the Google Cloud CLI remote MCP server

Wed, 30 Sep 2026 16:00:00 +0000

Today, we’re expanding our ecosystem of managed remote MCP servers by introducing the Google Cloud CLI remote MCP server in preview.

Powered by the popular gcloud and bq (BigQuery) command-line tools, this new server gives AI agents immediate, broad access to command-line operations for managing Google Cloud infrastructure and working with advanced BigQuery workflows securely and seamlessly.

Why CLI matters for AI agents

Agents are increasingly performing complex cloud operations, but standardizing how they interact with backend systems remains a challenge. The Google Cloud CLI remote MCP server bridges this gap by packaging the versatility of hundreds of gcloud and bq commands into one single MCP server. This results in two strong benefits for the agent:

Higher-level abstractions: CLI commands package complex multi-step workflows, validation checks, and high-level operations into unified commands rather than requiring multi-step API orchestration.
Leverages model training: LLMs are heavily pre-trained on public command-line documentation, syntaxes, and usage examples, making CLI invocation intuitive and highly accurate for models.

The benefits of putting CLI behind remote MCP

Managing cloud infrastructure with AI agents traditionally requires installing and maintaining Google Cloud CLI binaries inside agent execution environments. The Cloud CLI remote MCP server bridges CLI capabilities with MCP benefits by providing an isolated execution sandbox on Google Cloud infrastructure. This solves key infrastructure challenges:

Simplified dependency and runtime management: For teams building custom agents, maintaining local CLI versions and dependencies across dev, test, and production environments creates operational overhead. Remote MCP eliminates local installations and runtime maintenance.
Access for web-based agent endpoints: Web-hosted agent platforms and web interfaces (such as Gemini Enterprise and other hosted enterprise agent platforms) run in environments where users cannot control or install local packages. Remote MCP enables secure, managed access to Google Cloud CLI operations directly from these surfaces.

Enterprise-grade security and governance

Connecting an AI agent to your infrastructure requires strict, enterprise-ready safeguards. This remote server leverages Google Cloud's standard identity and governance frameworks to keep your environments secure:

Zero ambient credentials: The server isolates execution in a network-restricted proxy boundary with no ambient credentials. Authentication and authorization are handled through Agent Identity, OAuth 2.0, and Identity and Access Management (IAM).
Strict policy enforcement: Every command executed through the remote MCP server is run with the permissions of the authenticated caller identity. Both standard IAM permissions and organization policy service constraints are strictly enforced against downstream target resources.
Advanced protection with Model Armor: To minimize the risks associated with AI tool calling, the Cloud CLI remote MCP server integrates with Model Armor. You can proactively screen LLM prompts and responses to protect against risks like prompt injection and malicious inputs.
Cloud audit logging: The Cloud CLI remote MCP server can be configured to log every tool invocation to Audit Logs (Data Access logs under cloudcli.googleapis.com/mcp). Security teams can gain full visibility into caller identities, OAuth clients, and IAM authorization decisions (mcp.googleapis.com/tools.call) without exposing sensitive command payloads or personally identifiable information (PII).

Connecting to the Google Cloud CLI Remote MCP Server

Integrating cloud management into your agents no longer requires packaging Google Cloud CLI binaries, managing local execution runtimes, or maintaining dependencies inside agent container images. Because the Google Cloud CLI remote MCP server implements the standard Model Context Protocol, any MCP-compatible agent platform or orchestration runtime can connect immediately via standard configuration:

code_block: <ListValue: [StructValue([('code', '{\r\n "mcpServers": {\r\n "google-cloud-cli": {\r\n "uri": "https://cloudcli.googleapis.com/mcp",\r\n ...\r\n }\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb896eee150>)])]>

Your agent immediately gains access to execute gcloud and bq commands in a secure, network-isolated cloud sandbox.

Authentication is handled via keyless Agent Identity for hosted Google Cloud platforms, or standard OAuth 2.0 for external runtimes. For authentication options, see the MCP Authentication Guide.

Bringing infrastructure management to agents

The Cloud CLI remote MCP server exposes two powerful tools, run_gcloud_command and run_bq_command, giving your AI agents broad, immediate access to Google Cloud operations through natural language.

Managing cloud infrastructure with run_gcloud_command

With run_gcloud_command, agents can execute the full breadth of gcloud operations to manage, diagnose, and secure your Google Cloud environment. An example follows:

Observability and incident diagnostics: An agent streamlines incident diagnostics by automating command execution and reducing context-switching across tools.

Extending BigQuery operations with the run_bq_command tool

While the BigQuery MCP server already helps organizations analyze and explore data using AI agents, with the introduction of run_bq_command, agents can now tackle advanced BigQuery tasks such as resource allocation, job monitoring, and task scheduling by unlocking the full scope of bq CLI functionality. Key capabilities include:

Automating scheduled queries: An agent utilizes BigQuery Data Transfer Service configurations to schedule queries automatically.
Job and resource management: Gain deep insight into query execution details, including processed data volume, slot usage, and execution plans, as well as managing reservations.
Access and permissions control: Data administrators and owners can inspect and update table permissions directly through the agent.

Pricing and availability

The Google Cloud CLI MCP server is available today in public preview. There is no additional charge to use the MCP server itself. You pay only for the GCP resources you create and any applicable data transfer costs.

To get started, enable the Cloud CLI Execution API (`cloudcli.googleapis.com`) in your Google Cloud project, grant the required MCP Tool User (`roles/mcp.toolUser`) IAM role to your agent or user identity, and configure your MCP client to connect to `cloudcli.googleapis.com/mcp`.

Introducing Ask, a new Google Earth Engine feature to accelerate geospatial coding

Mon, 28 Sep 2026 16:00:00 +0000

Whether you are mapping global forest cover, detecting changes in the built environment, or monitoring agricultural yields, writing scripts in Google Earth Engine is a powerful way to develop these insights. This platform is part of Google Earth AI, our collection of geospatial models and datasets designed to help you transform planetary information into actionable intelligence, and we’ve launched a new feature, Ask, that makes accessing that planetary intelligence even easier.

We know that translating complex geospatial logic into code takes time, and memorizing specific Google Earth Engine API functions, searching documentation, and debugging syntax or memory errors can interrupt your flow. Ask is designed to help you get to actionable insights faster, by integrating Gemini capabilities directly into the Google Earth Engine Code Editor.

Starting today, you can use your own Gemini API key to write, debug, understand, and optimize geospatial queries without ever leaving the Code Editor.

Figure 1: The new Ask panel resides on the right side of the Code Editor, providing chat-based AI assistance tailored to your active script.

How it works: Context-aware assistance

You can ask a question directly in the Code Editor, and get a response based on context from your workspace, such as:

The full text of your active script
Your imported assets and geometries
Your active session chat history

Because Gemini capabilities in Google Earth Engine automatically understand your work, you don’t have to add code or explain your datasets. It already knows these details, so it can provide more relevant, helpful answers. Additionally, you can automatically view the diff and merge it into your script. No more copy and pasting!

You can also personalize Ask to make responses more comprehensive and suited to your needs:

Access the latest Gemini models: Choose the Gemini model that meets your needs — at launch, the available models are Gemini 3 Flash Preview, Gemini 3.1 Pro Preview, and Gemini 3.5 Flash.
Docs search: Use AI to search the official documentation for up-to-date syntax and functions.
Dataset search: Allow AI to search the Earth Engine Data Catalog to find and reference the exact datasets you need.
Google Search: Grounds response in the latest public web results using Grounding with Google Search (mutually exclusive with docs search and dataset search).

Quick-start examples

Not sure where to start? Here are four ways you can use Ask in your workflows:

1. Generate code from natural language

Need to calculate NDVI (“Normalized Difference Vegetation Index”), perform a cloud mask, or build a chart, but can't remember the exact syntax? Just ask and Earth Engine can generate JavaScript code for you. You can review the code directly in the chat and click Insert to add it to your script, or Copy it to your clipboard. If your editor is not empty, inserting code displays a side-by-side diff view so you can review changes before accepting them.

Prompt example: "Write a script to load Sentinel-2 imagery for 2025 over Boulder, CO, apply a cloud mask, calculate NDVI, and add the median composite to the map."

2. Explain complex code

If you’re working with a script written by a colleague or adapting an example from the community, you can use Gemini capabilities to explain it. Simply ask, "Explain what this script does," and receive a step-by-step breakdown of the logic and Earth Engine functions being used.

3. Perform one-click troubleshooting and debugging

Debugging is an inevitable part of coding. When your script throws an error in Google Earth Engine, the Console displays an error message. Now, those messages include a "Troubleshoot" button. Clicking it automatically populates the Ask panel with a prompt that contains the error message and a request for help to diagnose the issue and suggest a fix.

Figure 2: The "Troubleshoot" button in the Console makes debugging errors fast and frictionless.

4. Optimize queries

If your script is, for example, running slowly, consuming resources inefficiently, throwing "computation timed out" or "too many concurrent aggregations" errors, ask for optimization tips. Gemini capabilities can suggest best practices like early filtering, reducing computation steps, or converting client-side loops into server-side operations.

Get started in two steps

Ask is available globally today. To get started, you just need a Gemini API key:

Get an API key: If you don’t already have a Gemini API key, head to Google AI Studio and create one (there are free options; if you choose a paid-tier API key, you will be billed for your usage). Learn more about Gemini API terms.
Add the API key to GEE: Open the Google Earth Engine Code Editor. On the right-hand panel, click the Ask panel. Click the key icon in the bottom left corner and enter your API key.

We want to hear from you! Please use the Code Editor Feedback button to share your feedback and help us improve the experience.

Maximizing Apache Spark availability: Mitigating compute stockouts with flexible VMs and other best practices

Mon, 21 Sep 2026 16:00:00 +0000

The surge in AI development has created unprecedented demand for compute capacity around the globe. This can have negative implications for data processing and pipelines with Apache Spark. Whether you are managing your own Spark infrastructure or using a managed service, you can face availability constraints. However, a significant advantage of using Google’s Managed Service for Apache Spark is the availability of flexible VMs, which provide a targeted mechanism to adopt a dynamic, resource-agnostic philosophy and ensure your pipelines remain operational, even during regional or zonal capacity stockouts.

Understanding capacity stockouts

Capacity stockouts occur when demand for a specific machine family (such as N2 or N2D) exceeds available capacity in a target zone or region. For time-sensitive analytics pipelines, rigid single-VM requirements transform standard provisioning into a single point of failure which can result in cluster creation delays, failed executions, and potentially compromised business SLAs.

Flexible VMs

Flexible VMs fundamentally overhaul how a Managed Spark cluster requests compute resources. Rather than binding a cluster to a rigid instance type, flexible VMs allow teams to establish an ordered list of acceptable machine families for master, primary worker, and secondary worker nodes.

Key features

Multi-family blending: Mix nodes across diverse machine types and generations, combining Gen2 families (e.g., N2, N2D) with Gen4 families (e.g., N4, C4) in a single configuration.
Mixed storage support: Broaden available capacity pools by allowing storage options to dynamically adapt to the underlying host family's supported disk types.
Comprehensive cluster coverage: Apply flexible rules to primary workers, secondary (preemptible/spot) workers, and master nodes to guarantee cluster provisioning end-to-end.

Ranked configuration: A strategy for success

A successful flexible VM implementation relies on intentional ranking. By defining a clear hierarchy of options, Managed Spark clusters automatically attempt provisioning, systematically mitigating stockout risks without requiring manual intervention. To improve the availability of suitable VMs, we recommend specifying at least two machine families in the highest priority (Rank 0) flexible VM list.

As an example, for production pipelines standardizing on n2d-standard-16 shapes, the following tiering strategy provides robust resilience against capacity constraints:

Rank	Machine family examples	Storage recommendation
Rank 0 (Primary)	n2d-standard-16, n2-standard-16	Standard Local SSD or PD
Rank 1	n4-standard-16, n4d-standard-16	Hyperdisk Balanced
Rank 2	c4-standard-16, c3-standard-22	Hyperdisk Balanced
Rank 3	e2-standard-16	Standard PD

code_block: <ListValue: [StructValue([('code', 'gcloud dataproc clusters create $CLUSTER_NAME \\\r\n--num-workers=10 \\\r\n--zone="" \\\r\n--region=us-east1 \\\r\n--worker-instance-selection=\'{"machineTypes":["n2d-standard-16","n2-standard-16"],"rank":0,"diskConfig":{"bootDiskType":"pd-standard","bootDiskSizeGb":400}}\' \\\r\n--worker-instance-selection=\'{"machineTypes":["n4-standard-16","n4d-standard-16"],"rank":1,"diskConfig":{"bootDiskType":"hyperdisk-balanced","bootDiskSizeGb":400}}\' \\\r\n--worker-instance-selection=\'{"machineTypes":["c4-standard-16","c3-standard-22"],"rank":2,"diskConfig":{"bootDiskType":"hyperdisk-balanced","bootDiskSizeGb":400}}\' \\\r\n--worker-instance-selection=\'{"machineTypes":["e2-standard-16"],"rank":3, "diskConfig":{"bootDiskType":"pd-ssd","bootDiskSizeGb":400}}\' \\\r\n--master-instance-selection=\'{"machineTypes":["n4-standard-16","n4d-standard-16"],"rank":0,"diskConfig":{"bootDiskType":"hyperdisk-balanced","bootDiskSizeGb":400}}\''), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb8a8b859d0>)])]>

For pipelines standardizing on legacy n1-standard-16 shapes, the following tiering strategy helps transition workloads toward newer, more available architectures while preserving operational stability:

Rank	Machine family examples	Storage recommendation
Rank 0 (Primary)	n1-standard-16 n2-standard-16	Standard Local SSD or PD
Rank 1	n2d-standard-16	Standard Local SSD or PD
Rank 2	n4-standard-16 n4d-standard-16	Hyperdisk Balanced
Rank 3	e2-standard-16	Standard PD

Leveraging Hyperdisk Balanced

Unlocking maximum availability with flexible VMs often requires adopting modern storage architectures like Hyperdisk Balanced. Newer instance families (including N4 and C4) rely on Hyperdisk to deliver predictable performance across variable VM sizes. Starting with default IOPS and throughput settings typically provides a reliable baseline for the majority of distributed Spark jobs.

Trade-offs and key considerations

While flexible VMs dramatically improve cluster provisioning success, aligning them with enterprise requirements involves evaluating several architectural and financial factors:

1. Resource quotas

It is no longer enough to have one specific machine (e.g., N2) quota. You need to ensure you have sufficient compute and disk quotas allocated for all specific machine types and disks (including Hyperdisk) defined in their flexible VM lists.

2. Compute flexible Committed Use Discounts (CUDs)

Traditional, resource-based CUDs are tied to specific machine families, which limits flexibility. Adopt Compute flexible Committed Use Discounts (CUDs) to apply savings across multiple VM families and regions.

3. Performance Characteristics

Performance can vary between machine generations, as well as between Local SSD and Hyperdisk. While the Managed Spark team maintains internal benchmarks for these comparisons, actual outcomes are workload-dependent. Testing your specific Spark jobs across these families is essential for understanding SLA impacts.

Additional recommendations

In addition to implementing flexible VMs, there are several other key architectural and scheduling strategies to improve resource availability and workload stability:

AutoZone: Implement AutoZone routing to allow Managed Spark to automatically select the zone best suited to execute the job based on current capacity.
Smaller machine shapes: Avoid high in demand, large-core shapes. Design workloads and YARN containers to utilize smaller machine shapes (such as 4, 8, or 16 cores). These smaller shapes are much easier to fulfill from the available GCE on-demand pool.
Autoscaling: Deploy cluster autoscaling with reasonable maxInstances to manage capacity effectively for bursty or unpredictable workloads without relying on rigid, massive upfront provisioning.
Partial cluster creation: Configure a minimum acceptable number of primary workers. This allows clusters to spin up under resource constraints and begin executing, while autoscaling can dynamically add remaining workers as resources become available.
Establish regional fallbacks: Some regions, such as us-central1, can experience high demand. Setting up fallbacks to other regions reduces capacity stockout risks.

Keep your Spark jobs running with flexible VMs

Managing your own Apache Spark infrastructure can be complex, especially when capacity stockouts disrupt your data processing. Utilizing a managed service like Managed Service for Apache Spark provides unique advantages — including built-in platform resilience and access to flexible VMs. By adopting a prioritized fallback strategy with flexible VMs, you can protect your workloads from regional hardware shortages and keep your critical pipelines running.

Ready to improve your Spark workload resilience? Start configuring flexible VMs for your Managed Spark clusters today.

Data Analytics

Querying Databricks Apache Iceberg Tables with the borderless Lakehouse

How Lakehouse and Databricks work together

Centralized security and governance

Catalog federation and credential vending

Partition pruning

Fast queries with Iceberg metadata and Lakehouse caching

Maintenance and lifecycle management

Keeping your tables optimized

Compaction and the small file problem

Managing orphan files and snapshot retention (VACUUM)

Conclusion

Grounding agents in one context graph connecting semantic models you already trust

One context graph across your data estate

Get started

What’s new with Google Data Cloud

October 19 - October 23

October 12 - October 16

October 5 - October 9

September 7 - September 10

August 31 - September 4

July 6 - July 10

June 1 - June 5

May 11 - May 15

April 20 - April 24

April 13 - April 17

April 6 - April 10

March 23 - March 27

March 16 - March 20

February 23 - February 27

February 16 - February 20

February 2 - February 6

January 26 - January 30

January 19 - January 23

January 12 - January 16

What’s new with Google Data Cloud - 2025

Securely connecting Gemini Enterprise to Snowflake data

Choosing the right integration pattern

The architectural approach

System architecture flow

Detailed execution sequence

Step 1: Setting up the Snowflake security integration

Step 2: Data governance and access control

Step 3: Provisioning the Cortex Agent and MCP server inside Snowflake

Step 4: Locating and constructing the MCP server URL

Step 5: Connecting via Google ADK

Step 6: Project structure and deployment

Step 7: Registering the custom agent in Gemini Enterprise

Get started

Unifying enterprise context and semantics: LookML in Knowledge Catalog is now GA

Why a context graph needs deterministic semantics

What's new: LookML in Knowledge Catalog

Three patterns for data agents

Scaling agents for BI and the enterprise

Getting started

Introducing new self-learning context enrichment in Knowledge Catalog

Keep context current with Gemini-powered self-learning enrichment

Set the right context for each audience with context sets

Explore and govern it all in Knowledge Studio

Get started

Watch the demo: How Knowledge Catalog’s self-learning enrichment grounds agents

BigQuery ObjectRef now GA: Build multimodal data platforms for conversational analytics entirely in SQL

ObjectRef brings structure to the unstructured

Create full-fidelity ETL with SQL

Proven use cases from industry leaders

New in GA

See it in action: End-to-end multimodal analytics

1. Full-fidelity ETL: Call center audio analysis

2. At-scale gen AI filtering: Product compliance

3. Ad-hoc discovery: Product catalog enrichment

4. Multi-object reasoning: Proactive threat detection

Start analyzing today

How to Use Python UDF Logging for Pipeline Observability

Architecture overview

Quickstart walkthrough: Step-by-step

Step 1: Create the generic Python UDF

Step 2: Run ingestion with logging enabled

Ingestion Results

Step 3: Set up real-time alerts on Cloud Logging

In the Google Cloud Console (1-Click)

Final `clean_events` Table Result:

Sizing your input data: `AI.COUNT_TOKENS`