Data Analytics

Agentic Future Ready With BigQuery: Continually Improving Price-Performance, Zero Effort

Thu, 06 Aug 2026 16:00:00 +0000

In the modern data landscape, query performance tuning and managing system price-performance is challenging, especially as the number of agentic workloads increase. Even for experienced developers and DBAs, constantly analyzing query execution plans, tweaking schemas, and adding query hints with ever exploding volume, variety, and velocity of data is a never-ending cycle that drains business velocity.

While performance tuning is a common practice, a modern data platform should do more. As data platforms evolve from systems of intelligence to systems of action, and analytics workloads shift from humans running a few queries per day to countless agents running many thousands of queries per minute, the old way of manual query tuning doesn’t work. When queries are generated by agents and applications automatically based on user actions, manual optimization becomes practically impossible.

BigQuery has evolved from a data warehouse to the primary engine for the Agentic AI era. Building on a unique, truly disaggregated storage and compute architecture, serverless processing, and fine grained compute management, BigQuery continues to push the boundaries of autonomous query processing. Our North Star is an autonomous query processor powering both humans and agents for hands-free optimum price-performance regardless of query, schema, data, or workloads. Just in 2025, we delivered up to 35% better query performance and as much as 40% reduction in query processing costs (slot usage).

Figure 1. Summary of BigQuery price-performance improvements throughout 2025 based on industry standard benchmarks.

The following are some of the major innovations contributing to these improvements in performance and total cost of ownership (TCO), including the built-in guardrails against regression.

BigQuery’s Self-Learning Engine: History-Based Optimizations (HBO)

One of the foundational capabilities of BigQuery’s autonomous query processor is history-based optimizations.

Traditional query optimizers rely on static statistics, metadata and cardinality estimates, which can be wildly inaccurate when faced with highly complex, multi-table joins and rapidly growing/changing data. Managing these is part of the critical path for administrators and automation has to be tailored to individual workloads to be effective and efficient. Even when everything is up-to-date and correct, queries can still have vastly different behavior due to natural data skew or changes in available compute resources. History-based optimizations change the paradigm: in addition to BigQuery’s already adaptive query execution that can change plans and resource allocations while a query is running, it learns from past executions and automatically applies additional optimizations for future executions.

It tracks runtime statistics of past queries to "remember" which optimizations were beneficial and continue to apply them, and learn from prior mistakes to ensure they are not repeated. When the same or similar query runs again, BigQuery automatically applies any additional optimization technique that is known to be beneficial and avoids those that can cause regressions.

No User Action Required & Built-in Safety Guardrails
History-based optimizations require no application rewrites, SQL modifications, or schema changes. Users literally do nothing, and their recurring dashboards, ELT pipelines, agentic workloads, or line of business modules run faster.

Crucially, this is a self-maintaining, self-improving closed-loop system with built-in safety guardrails. History-based optimizations only apply an optimization when there is high confidence it will improve performance. What if an optimization makes the wrong decision? This capability is inherently self-correcting. When an optimization is applied, the system measures the result. If the expected improvement is observed, the optimization is accepted. If it does not significantly improve performance—or worse, regresses or leads to failure—the optimization is immediately rejected, revoked, and never tried again for that query. This includes detecting data skew so queries that have parameter sensitive plans do not run into major performance issues when just a single parameter (aka WHERE clause) is changed.

The result? A decrease in execution times and a reduction of slots. One enterprise customer reported P90 execution times drop by up to 50%, with slot usage falling by up to 15% resulting in substantial price-performance improvements and no regressions.

Figure 2. One example of History Based Optimization performance improvement reported by an enterprise customer.

Teaching the BigQuery execution engine newer tricks: Advanced runtime

BigQuery’s autonomous capabilities extend deep into the execution layer with BigQuery advanced runtime. This engine upgrade automatically determines the best physical execution path for a query without any manual knob-turning.

1. Enhanced vectorization
While vectorized execution is not a new concept, BigQuery enhanced its implementation by taking advantage of the newer processor efficiencies in Single Instruction Multiple Data (SIMD) instructions. The engine autonomously identifies opportunities to avoid duplicate computations, processing data in dictionary and run-length encodings natively. It couples with state-of-the-art parallel algorithms and is applied at eligible query stages to increase the opportunities for acceleration. The Impact: Fully automated, safely accelerating qualifying queries by up to 10x, yielding up to a 40% overall slot time reduction.

Figure 3. Advanced runtime - Enhanced vectorization

2. Short query optimizations
For high-concurrency and low latency BI dashboards or agentic applications generating thousands of queries that require sub-second latency, distributed processing overhead can be a bottleneck. BigQuery now autonomously accelerates eligible "short" queries without impacting other queries running concurrently. By reducing the number of stages hence reducing data shuffling, BigQuery improves overall query latency and resource efficiency. BigQuery has effectively implemented an efficient symmetric multi-processing (SMP) query path transparently within a scalable massively parallel processing (MPP), distributed databases. The Impact: up to 10x lower slot usage for short queries, with P99 sub-second query latencies. Because each query uses slots for shorter periods, we observed some customer workloads having up to 3x higher throughput—all completely transparent to the end-user.

Figure 4. Advanced runtime - Short query optimizations

Same Benefits Regardless of Data Formats

We believe you shouldn't have to sacrifice autonomous performance when adopting open lakehouse architectures using open table formats like Apache Iceberg. BigQuery performance improvements work the same regardless of the underlying table or data format. That means you still get the same benefits whether you’re using BigQuery’s native capacitor storage format or Iceberg tables where the underlying data is in parquet format.

Whether it’s automatically pushing down filters, employing Column Metadata Index (CMETA) pruning, and optimizing the I/O layer with page skipping and asynchronous read or employing the same enhanced vectorization, open formats benefit from the same hands-free acceleration as native tables.

Figure 5. BigQuery on Lakehouse Iceberg tables performance and costs tested internally against a popular open source, distributed SQL query engine using a benchmark derived from TPC-DS (10TB)

Fluid Scaling: The True Only-Pay-For-What-You-Use Autoscaler

Because BigQuery’s compute models are tied directly to slot-seconds consumed rather than a slice of nodes or clusters, autonomous performance gains can translate directly to cost savings. BigQuery autoscaler enhanced with fluid scaling enables you to run any mix of highly variable workloads with a premier autoscaling model that does not require a cost-and-performance trade-off. Fluid scaling in BigQuery enables true per-second billing for compute resources (slots) consumed and lowers costs by up to 34% on average for autoscaling workloads.

Figure 6. Fluid scaling enhanced autoscaler

On the price-performance benefits of fluid scaling, Chen Shalit, the CEO and Co-Founder at RISE, a leading AdTech company that processes 1 PB+ data per day and manages 3 trillion+ monthly bids across publishers and advertisers, summed it best. “In the high-stakes world of advertising, every cent counts. BigQuery’s Fluid Scaling has reduced our infrastructure cost by 25% allowing us to scale our slot consumption and accelerate our entire workflow. By delivering faster model results and increasing our hourly data processing throughput, we’re providing our customers with the 'freshest' data possible.”

The foundation for the agentic future

These autonomous capabilities are not just about making today's business intelligence dashboards and analytics queries run faster; they are foundational to the future of modern data platforms. As organizations rapidly deploy AI and machine learning, they are shifting toward agentic workloads.

AI agents do not interact with data the way humans do. Their query latency and concurrency (QPS) requirements are orders of magnitude more demanding than what human users generate—which is the limit of what most traditional analytics platforms were designed for. When autonomous agents are firing off thousands of distinct queries per second to make real-time decisions, human-in-the-loop query tuning is simply impossible.

BigQuery's differentiated abilities in a self-tuning and self-learning system include built-in safety guardrails against regression. With this standard of autonomy, you can spend your time and effort where it truly matters: delivering the best experiences for your users with advanced agentic applications, rather than wasting critical engineering cycles on manual infrastructure management, reactive query optimization, or trying to rein in spiraling compute costs.

BigQuery’s aggregate TCO impact: Faster and more cost-effective price-performance

When you combine a self-learning history-based optimizer with an auto-scaling advanced runtime engine supported by a highly scalable metadata platform (CMETA), you can see real impact on performance and total cost of ownership (TCO).

Conclusion

Our goal is for BigQuery to automatically and safely optimize queries without user action so our users can “just come in to work and BigQuery is faster than it was the day before."

You no longer need to worry about data skew or stale statistics causing performance regressions. No more trying to figure out which plan guide works best for which queries at what time of the day or month. No more hoping you won’t run out of budget because there aren’t enough low priced spot instances. Just focus on your organization’s goals, and your BigQuery data platform can deliver optimized price-performance for both agent and human workloads autonomously; it just works.

Learn more about BigQuery’s data and AI capabilities and get started with a free trial.

Solving the "Noisy Neighbor": How Sharded Architecture Protects Multi-Tenant Platforms

Wed, 05 Aug 2026 16:00:00 +0000

Whether you are a multi-tenant SaaS provider, a large enterprise managing internal data platforms, or a company handling mixed-workload data processing, managing a shared infrastructure environment means facing a common threat: the 'noisy neighbor'. A single tenant with a massive data burst or a failing database instance can bring down the entire neighborhood, manifesting as significant backlog accumulation and global Service Level Agreement (SLA) violations across critical data pipelines

Here is how to transition from a monolithic architecture to a sharded hub-and-spoke pattern to ensure platform resilience.

The problem: The monolithic bottleneck

A typical legacy architecture processes data for all tenants and business domains through a single, massive stream. Because the pipeline is unified, a performance issue with one specific database tenant instance creates back pressure that degrades performance for every other tenant on the platform.

Painful effects:

100% Blast radius: One database failure can stop all processing.
Inefficient scaling: Resources often have to be scaled for the "worst-case" tenant, leading to significant wasted spend.
SLA instability: Maintaining a global SLA is nearly impossible when one high-volume tenant can lag the entire system.

The solution: sharded hub-and-spoke architecture

To solve this, processing is decoupled into a "hub" for routing and "spokes" for isolated execution.

1. The hub: The router pipeline

The Hub is a lightweight Dataflow job that acts as a traffic controller. It reads from unified source topics, parses the Tenant ID or Business Domain, and fans the data out into isolated buffers. This keeps the entry point simple and robust.

2. The buffer: Durable isolation

Pub/Sub topics are introduced between the Hub and the Spokes. These act as a durable shock absorber, preventing a slow downstream sink from backing up the original source.

3. The spokes: Isolated execution

Instead of one giant pipeline, multiple, smaller Dataflow instances are deployed and categorized by workload:

Tier 1 (high-priority): Dedicated pipelines with high resource allocation for critical tenants.
Shared tiers: Grouped pipelines for smaller tenants to optimize costs.
Domain specific: Specialized pipelines for complex logic (e.g., separating distinct business domains) to isolate code complexity.

Comparative Benefits at a Glance

Feature	Monolithic (legacy)	Hub-and-spoke
Fault tolerance	One failing DB stops everything	Failures isolated to specific spoke
Blast radius	100%	< 5% (Isolated to one spoke)
Resource scaling	Scaled for "worst-case"	Independent scaling per tenant load
Maintenance	Global updates affect everyone	Update one domain without touching others

Pro-Tips for Implementation

Moving to this architecture is more than just shifting boxes on a diagram. Additional "Spoke" level optimizations are recommended for maximum stability:

Implement Dead Letter Queues (DLQ): Do not let a single SQL exception stall the pipeline. Route failed records to storage (like BigQuery or Google Cloud Storage) for later investigation.
Strict connection pooling: Databases have connection limits. Use a thread-safe singleton pattern and set a low MaximumPoolSize (e.g., 1-2) per worker to avoid exhausting the database during autoscaling.
Asynchronous I/O: Use the GroupIntoBatches transform to buffer writes, which reduces the connection overhead that often triggers database-induced latency.

Conclusion

By adopting a sharded approach, platforms can guarantee that a "noisy neighbor" is no longer a threat to the neighborhood. This architecture provides the isolation needed to maintain strict SLAs while allowing for independent scaling and safer deployments

To learn more about implementing a sharded hub-and-spoke architecture, explore the Dataflow documentation.

How Target is enhancing retail discovery and cutting database maintenance by 50% with Spanner Graph

Tue, 04 Aug 2026 16:00:00 +0000

In today’s retail environment, shoppers expect highly personalized product discovery experiences and conversational assistance that feels genuine, natural, and genuinely helpful. Today, successful product discovery is about understanding semantic meaning and the rich, connected relationships between products, categories, and guest intent. It is no longer just about keywords and basic browsing.

At Target, this work is handled by our Guest Product Confidence platform team. They are responsible for building the features that establish trust and guide purchasing decisions, such as ratings, reviews, and AI-driven digital shopping assistants. An exciting example of this is our Gift Finder chat agent, which we launched during the 2025 holiday season online and in the Target app to help shoppers discover the perfect items through friendly, conversational dialogue.

To deliver real-time personalization and context-rich semantic responses like these at global scale, we identified a critical architectural need to move away from a fragmented data ecosystem toward a unified data platform. We needed a solution capable of supporting high-throughput transactional workloads, highly connected graph relationships, vector similarity search, and full-text keyword search all at once.

In this post, we’ll explore how we achieved all four with Spanner.

Overcoming fragmented architecture

Previously, Target’s discovery data ecosystem relied on a combination of Elasticsearch clusters for search and inverted indexes, alongside separate NoSQL datastores for our transactional data. While functional, this fragmented architecture presented significant operational and technical challenges.

Disconnected context: Keeping separate search, vector, and transactional databases in perfect sync was a constant challenge. Siloed information led to missing context, disconnected attribute relationships, and inconsistent query results.
High operational overhead: Managing independent clusters, tuning search indexes, and handling complex, custom synchronization and aggregation logic required intensive manual intervention from our engineering teams.
Expansion bottlenecks: Expanding our retail data domains required adding new database collections, maintaining complex joins, and navigating weak transactional guarantees across our discovery and core transactional systems.
Siloed intelligence: We lacked the ability to query graph relationships, vector similarity, and keyword search indexes in a single transaction.

To build the next generation of AI-driven guest experiences, we needed to consolidate on one platform.

Building the enterprise ontology on Spanner Graph

We evaluated multiple specialized technologies, including standalone vector databases and niche graph databases. However, adding more single-purpose databases would have only worsened our operational complexity and data synchronization pipelines.

We ultimately chose Spanner Graph to build our enterprise ontology, which is a "graph-of-graphs" paradigm that allows us to construct a massive, generative AI-powered shopping graph.

By unifying our data, we bring semantic data, graph relationships, vector embeddings, and operational transactions under one roof. This establishes Spanner as our single authoritative source of truth for both transactional state and semantic intelligence.

Our high-level architecture now consists of three core pillars:

1. Enterprise augmentation
This layer captures our enterprise retail catalog, aggregates relevant metadata from multiple backend sources, and utilizes generative AI for agentic data enrichment to dramatically improve the quality and depth of the product data we ingest.

2. Unified graph, vector, and search store
Instead of shifting data across multiple databases, Spanner Graph stores our entity nodes, relationship edges, and vector embeddings in the same database engine. Spanner Graph natively supports multi-hop graph traversals, semantic vector similarity, and full-text keyword queries over our relational tables. Because this multi-model synergy is native, we get strict ACID transactions for absolute correctness across distributed workloads without the need for fragile external sync pipelines.

3. Orchestration and AI layer
This layer powers our conversational guest interfaces, utilizing rich, structured context fed directly from Spanner Graph to ground our LLMs. It extracts highly specific product relationships to power tools like the Gift Finder while governing responsible AI processes and evaluating generated outputs.

A smooth, zero-downtime incremental migration

Transitioning critical search and discovery infrastructure that millions of guests rely on required a cautious, zero-downtime approach. We executed this migration in four structured phases.

Schema and ontology mapping: We defined the specific retail entities, such as products, categories, brands, and guest preferences, and their corresponding relationships within the Spanner Graph schema.
Data integration and parallel replay: We built mutation-based data integrations in a parallel pipeline. This allowed us to continuously replay live transactional updates, apply schema transformations, generate embeddings, and write them directly into Spanner Graph in real-time.
Canary deployment: We gradually shifted live read traffic to the new Spanner Graph-backed platform, validating query performance, semantic accuracy, and database stability under real retail workloads.
Cutover and cleanup: Once performance was thoroughly verified, we fully transitioned all search and discovery traffic to Spanner and deprecated our legacy Elasticsearch stack, entirely removing the maintenance burden of those clusters.

Business impact

By building directly on Spanner Graph, we unlocked measurable technical and business outcomes:

The ultimate GraphRAG foundation: Traditional RAG relies on flat vector similarity, which often misses the structured associations between products, such as matching a toy with its compatible accessories or age-appropriateness. By combining deep graph traversals with semantic vector search in a unified GraphRAG architecture, we grounded our LLMs with highly precise context. This directly improved our recommendation relevancy, enhanced guest satisfaction, and boosted our Net Promoter Score.

Consolidated SQL + GQL interoperability: With Spanner Graph, our developers query structured relational catalog data and connected graph relationships in a single query using standard SQL and GQL (Graph Query Language). This eliminates the need for data duplication, latency, or complex ETL pipelines to bridge these paradigms.

Serverless scalability with zero growth ceiling: Spanner automatically handled massive, unpredictable traffic spikes during peak retail events like Black Friday and Cyber Monday. Spanner's built-in autoscaler dynamically adjusted computing capacity to handle burst traffic during high-intensity, limited-time promotional offers without sacrificing performance.

50% reduction in infrastructure maintenance: By consolidating our transactional NoSQL and search index databases into a single managed Google Cloud service, we eliminated the operational burden of maintaining separate database clusters. Our developers now spend 50% less time on database administration and infrastructure upkeep, allowing us to build and deploy new, customer-facing AI features much faster.

Migrating to Spanner Graph has accelerated our generative AI roadmap, serving as the ultimate proof of what is possible when you build on the right data foundation.

Want to supercharge your AI apps? It starts with databases with the right graph capabilities at virtually unlimited scale. Discover how Spanner Graph can turn data into action for your organization.

Cortex Framework v7 is GA: Build agentic workflows without disrupting SAP operations

Mon, 03 Aug 2026 16:00:00 +0000

Businesses want to quickly and safely deploy AI agents to drive revenue, mitigate risk, and optimize capital, all without disrupting mission-critical ERP systems. And to power AI agents, you need more than raw data: You need interoperable data products that act as a single, reusable source of truth, turning cryptic source system records into clear business terms.

Today, we are announcing the general availability of Google Cloud Cortex Framework version 7. This release modernizes your data architecture for agent readiness, helping you quickly deploy, customize, and extend robust data products while simplifying orchestration and reducing the infrastructure overhead of traditional approaches.

While ERP systems hold a wealth of foundational data, turning those transactional records into AI-ready data products is difficult, especially without slowing down core operations. Cortex Framework v7 solves this with purpose-built data product accelerators for SAP. Deployed directly in BigQuery and Knowledge Catalog, these data products feed Gemini Enterprise Agent Platform with accurate business context so your AI agents can execute with high fidelity. It also simplifies data orchestration using a modular, scalable deployment architecture powered by Dataform, an end-to-end experience that helps data teams build, version control, and orchestrate workflows in BigQuery.

Read on for more details about what you’ll find in Cortex Framework v7, how it integrates with the recently released SAP Business Data Cloud Connect for BigQuery, what our customers are saying, and how to get started.

What’s new in Cortex Framework v7

1. Deliver agent-ready data products

Traditional business intelligence (BI) dashboards tell you what happened last quarter, but modern AI models and agents help you act in real time. To power this shift, Cortex Framework v7 packages your enterprise data into semantically rich data products that contain AI-friendly metadata. This enables your agents to reason, orchestrate, and execute high-impact workflows.

By translating raw SAP tables into clear business terms, dynamically ingesting custom fields, and natively handling advanced logic (like SAP TCURX currency decimal shifts), the Framework maintains the high data fidelity required for large language model (LLM) interactions and enterprise analytics.

Image showing Cortex Framework v7 deployed data products for SAP ERP inside BigQuery with built-in description metadata for field names and descriptions.

These purpose-built data product accelerators for SAP ERP and SAP Business Data Cloud deploy directly in BigQuery. Once they’re deployed, you can trigger automatic registration in Knowledge Catalog to easily discover data and connect it with your AI agents.

Image showing a Cortex Framework v7 data product registered in Knowledge Catalog with built-in metadata for descriptions and labels.

To accelerate custom AI projects, this release also includes agent skills for an agentic data product builder that helps data and knowledge engineers use natural language to generate custom data products. Using the Framework's provided content, an AI agent can automatically handle the build process, including adapting to specific source data requirements and customizations.

GIF showing Cortex Framework v7 delivered agentic data product builder skills in action using Google Antigravity with Gemini to automate the build and deployment of new data products using natural language prompts.

With v7 you can also extend Cortex Framework to maintain a clean separation between Cortex Framework-delivered content and your custom models. This helps you apply the latest code updates without impacting your custom work.

2. Build faster with modular deployments

Dataform powers the modernized data pipelines in Cortex Framework v7, offering version-controlled SQL and native dependency management. When you select a packaged data product for deployment, Cortex Framework retrieves and processes only the required tables. Dependency resolution automatically handles order-of-operations and generates a dependency graph for efficient processing. You can also add custom fields or logic without breaking standard models.

Image showing Cortex Framework v7 data products in Dataform with automated dependency graph showing data asset and table dependencies as well as built-in metadata like natural language tags.

This modular approach extends natively to complex, multi-system SAP environments. Using built-in logic differentiation, Dataform compiles and deploys Cortex Framework data products for SAP ECC, SAP S/4HANA, and SAP BDC source systems in parallel, while dynamic schema discovery automatically ingests and processes custom SAP fields such as Z-fields. Together, these capabilities help you bring in data from multiple systems simultaneously, reducing the need to manually refactor pipelines.

3. Scale cost-effectively with flexible processing

Cortex Framework now defaults to BigQuery incremental loading and non-destructive schema updates, processing only new or modified data. This approach minimizes compute time and significantly lowers operational expenses. Because orchestration relies on Dataform’s serverless, BigQuery-native execution, data processing scales without additional infrastructure overhead.

You can also use a bring-your-own (BYO) change data capture (CDC) approach. You can use this approach to bypass built-in CDC data processing and connect your existing CDC-processed pipelines directly.

Interoperability with SAP Business Data Cloud

Following the recent general availability of SAP Business Data Cloud Connect for BigQuery, Cortex Framework v7 now natively supports SAP BDC standard and custom data products. By combining Cortex Framework-delivered data products with SAP BDC data products, you can uncover new business opportunities. To accelerate this, v7 includes solution samples across both SAP BDC and SAP ERP data to help you quickly answer a wide range of critical questions such as assessing the health of your sales pipeline or identifying procurement inefficiencies.

Image showing Cortex Framework v7 consumption data product solution samples deployed on SAP BDC sourced data products via SAP Business Data Cloud Connect for BigQuery integration.

Co-innovating with industry leaders

We’ve worked closely with our Professional Services Organization and industry leaders to test these new capabilities. By serving as our lighthouse customers, these organizations provided valuable feedback to help us shape a framework built for enterprise-scale agility and AI readiness.

“As an existing Cortex Framework customer, we are excited about the capabilities introduced with the latest v7 release. Bridging complex SAP ECC and S/4 environments with Google Cloud’s AI capabilities requires a trusted data foundation with accurate business context. Based on the capabilities announced for Cortex Framework v7, including the agentic data product builder, we see significant potential to accelerate and scale the creation of enterprise data products for analytics and agentic AI use cases. The direction of the platform is particularly promising for organizations looking to make SAP data more accessible, semantically meaningful, and ready to support future reporting and AI initiatives.”
- Dr Jwan Ibrahim, Director, Data and Analytics, SLB

Get started

Cortex Framework v7 changes how enterprises operationalize their most critical data for AI agents. By combining serverless execution, AI-ready semantics, and native SAP integration, Google Cloud helps you turn complex enterprise data into a strategic asset. To get started, check out the following resources:

Execute a deployment: Try a Cortex Framework demo deployment (with a predefined sample dataset) to explore your first v7 data products today.
Learn more: Read the comprehensive Cortex Framework v7 technical documentation for deep dives into production deployment steps, configuration, extensibility, source system integration, and packaged data products.
Talk to an expert: Contact your Google Cloud account team to learn how Cortex Framework v7 can accelerate your specific data-to-AI journey.

Unifying public and private data: Scale knowledge graphs with Data Commons on Spanner

Mon, 03 Aug 2026 16:00:00 +0000

To make informed decisions, businesses often need to connect their internal data with public reference data, to create a knowledge graph that connects real-world things and their relationships. However, bridging data from public and private worlds has traditionally been complex. Today, we are streamlining these connections with the general availability of Data Commons on Spanner Graph and the preview of the new Data Commons Platform to unify your private knowledge with knowledge graphs from public datasets.

The overarching Data Commons project supports Google’s mission to organize the world's information and make it universally accessible and useful. Data Commons unifies fragmented public datasets from over 100 authoritative providers, including the United Nations, World Bank, US Census Bureau, Eurostat, WHO, and NOAA, with over 400 billion data points structured using standardized Schema.org definitions. Data Commons provides data exploration tools, MCP tools, and cloud-based APIs to access and integrate the clean datasets.

Data Commons integrates public information across multiple domains, including agriculture, demographics, economy, environment, and health. This standardized approach unlocks powerful use cases, for instance, letting you analyze national GDP trends, map regional smoke pollution levels, track local health equity, or demographic distributions over time, all using data that has already been preprocessed and normalized for you.

Data Commons knowledge graph dimensions

Dimension	Size	Technical description
Statistical observations	400+ billion	Individual metric data points
Graph edges	2.6+ billion	Relationships
Knowledge graph nodes	1.7+ billion	Standardized entities
Data sources	100+ providers	Authoritative institutions

^{Data Commons makes meaningful quantities of public administrative data available to users on readily consumable cloud-based infrastructure.}

A modern infrastructure powered by Spanner Graph

When we first built Data Commons, our goal was to aggregate massive, disparate public datasets using the tools available at the time. The platform relied on Bigtable as a caching layer, which was an effective strategy for handling large-scale lookups in the absence of native graph database technology.

Today, we have transitioned our architecture to a native graph model with Spanner Graph, which brings the convenience of a SQL-like interface and graph expressiveness to Spanner, with its high availability, horizontal scale-out, multi-region transactional consistency, and native ISO/IEC 39075 Graph Query Language (GQL) support.

By adopting a multi-entity Spanner Graph schema, we represent entities as nodes and their domain links as dynamic graph edges, allowing us to move away from pre-computed cache structures and perform complex relationship queries directly within the database using GQL.

This architecture also simplifies our pipelines by removing the need for complex, pre-computed indices that require costly in-memory rebuilds and multiple snapshots. Spanner Graph enables incremental updates to specific datasets without refreshing the entire database, while stale reads maintain consistent data snapshots during ingestion.

Key benefits by moving to Spanner Graph

Unified storage and incremental updates: By utilizing Spanner Graph’s multi-entity schema, the platform replaces complex caches with a model that supports incremental data imports, allowing for targeted updates to specific datasets.
Dynamic graph traversals via GraphRAG: The system executes multi-hop queries such as navigating hierarchies like continent → country → state → county → city on the fly. This removes reliance on static caches and enables GraphRAG workflows, where the database maps natural language queries directly to structured path-matching traversals.
Consistent data snapshots: Leveraging Spanner TrueTime and stale reads, the platform provides you with a version-consistent snapshot of data, maintaining integrity across distributed nodes following batch ingestion cycles.
Operational analytics at scale: Spanner’s columnar engine efficiently scans massive time-series datasets by reading only the necessary fields, while BigQuery federation that leverages Spanner’s Data Boost technology performs complex aggregations via EXTERNAL_QUERY in an isolated environment, helping isolate production traffic.

Bridging systems with SDMX 3.0 interoperability

To facilitate the use of complex statistical data, Data Commons adopts a lean implementation of Statistical Data and Metadata eXchange (SDMX) technical standard. As an ISO specification, SDMX provides a consistent approach for describing and exchanging statistical data along with descriptive statistical meta-information.

In this Data Commons Platform update we added support for the SDMX technical standard version 3.0, providing out-of-the-box integration with third-party tools like Tableau, Flourish, and Observable for multi-dimensional datasets. This is made possible using the API standard SDMX-JSON and SDMX-CSV 2.0 formats across two high-value endpoints:

The availability API: A programmatic discovery mechanism to identify existing dimensions, variables, and date ranges without reading raw values.
The data API: Retrieves actual observations and metadata, using named parameters to help prevent code from breaking when dimensions are added.

Transforming private instances of Data Commons Platform

For organizations that want to build private instances of the Data Commons Platform, this new modern architecture resolves legacy scaling limits and simplifies data schematization. Developers can instantiate a private instance of the Data Commons Platform leveraging the same scalable technology that powers Google’s Data Commons instance. As a private instance, users retain full control of their own data and have the ability to limit access, while enabling natural language queries to blend results from their private data with Google’s public data that is hosted on the Google Data Commons instance. By federating across our public knowledge graph and a private knowledge graph containing your own data, you can light up exciting new use cases, while maintaining data isolation and ensuring no data duplication.

For instance, a retail enterprise can combine public data such as national GDP trends, regional demographic breakdowns, and employment statistics, with their own enterprise data, including sales histories, store performance metrics, and supply chain logistics. This allows analysts to contrast public macroeconomic indicators against their own company transactions to optimize merchandise distribution and identify untapped markets.

Example of a natural language query combining statistical data from the Directorate General of Commercial Intelligence and Statistics (DGCIS) stored in a Data Commons Platform private instance with World Development Indicators from the World Bank stored in the Google Data Commons public instance.

A user is querying a Data Agent for average annual temperature trends in the country. The agent retrieves information from Data Commons, explaining that while historical data is available, it provides projected temperature changes, climate drivers, and CMIP6 climate model scenarios (SSPs), with options to export the generated report.

A user asks the Data Agent to compare the Worker Population Ratio (WPR) of rural versus urban males in a country. Fetching data from Data Commons, the agent defines WPR—the percentage of workers relative to the total population—and outlines the available demographic variables to analyze and compare both groups.

Get started today

Explore Data Commons: Visit datacommons.org to query global statistical knowledge.
Explore Spanner Graph's use cases and setup guide for your knowledge graphs.
Deploy Data Commons Platform: contact support@datacommons.org to request preview access and to review the developer tools.

The borderless Lakehouse: Bring AWS, Databricks and Snowflake data to your AI agents

Wed, 29 Jul 2026 16:00:00 +0000

Today’s data lakehouse is no longer mere data repository, but increasingly a system of action, actively executing tasks via always-on, autonomous AI agents. Rather than waiting for static reports, these agents run continuous, real-time reasoning loops, monitoring supply chains, flagging anomalies, and executing business workflows. To scale this model, AI agents need to access your entire data estate and the right context to understand what data to use and when. However, traditional data architectures, with their high costs, fragmented security, and weak governance, don’t make it easy.

Today at Next Tokyo, we’re introducing enhancements to our borderless Lakehouse. Built on open Apache Iceberg, it connects your on-premises, cross-cloud operational systems, and SaaS application clouds, so you can activate and query your data wherever it lives, without moving it.

Federating multiple catalogs with Iceberg REST

The borderless Lakehouse enables Gemini Enterprise and conversational agents to analyze and act on data regardless of its physical location. Built on the Iceberg REST catalog, it lets you discover and query remote data instantly, eliminating the high costs and delays of building data pipelines. This connectivity is made possible with catalog federation (now in preview) for AWS Glue, Databricks Unity, and Snowflake Horizon, providing secure, bi-directional access via BigQuery, Managed Service for Apache Spark, and any Iceberg-compatible engine.

This open approach also extends to the application layer, with zero copy data integrations to major SaaS applications like SAP, Salesforce, and Workday. BigQuery can now directly query live application data in these platforms without complex ETL pipelines, while they can run BigQuery's AI engines on their own data in-place to easily unify finance, HR, and customer data. These capabilities unlock three significant benefits:

Zero-copy, cross-cloud analytics: Instantly discover and query enterprise data across platforms without duplicating files, allowing various data teams to analyze the exact same copy of Apache Iceberg data.
Bidirectional interoperability: Read and write across environments, querying external tables from BigQuery or Managed Spark, then sharing derived datasets enriched by Google AI back to partner systems for downstream action.
Unified governance and access control: Get out-of-the-box governance with trusted context and Gemini insights for your agents. Secure access control at the table level with support for credential vending, regardless of which platform initiates the query.

The borderless Lakehouse expands to Google Cloud’s database portfolio which lets you integrate transactional systems with data lakehouses: Spanner Omni lets you run the highly scalable database in any environment outside of Google Cloud, and Lakehouse Federation for AlloyDB allows transactional systems to directly query warehouses. By eliminating costly data movement, your teams can securely and efficiently analyze live operational and historical data together in real time. Now your lakehouse is truly borderless.

Bring Google AI directly to your AWS and Azure data

Historically, running advanced analytics or training ML models across clouds meant a cross-cloud tax: high egress fees, network latency, and fragile ETL pipelines. The borderless Lakehouse solves this with Cross-Cloud Interconnects. These private, dedicated links deliver consistent bandwidth and lower latency than the public internet, at a fraction of the cost of traditional cross-cloud connections.

The borderless Lakehouse supports zero variable egress costs when accessing your data from AWS¹. With Partner Cross-Cloud Interconnect pricing, you get predictable monthly costs with an SLA-backed connection. This managed, private connectivity simplifies provisioning from 1G to 100G using a flat-rate, subscription-based model.

In addition, intelligent cross-cloud caching in the borderless Lakehouse securely stores remote data fragments temporarily inside Google Cloud to eliminate repeated, costly transfers for subsequent ad-hoc or BI queries. These capabilities, combined with BigQuery's vectorized processing, BigQuery AI functions for multimodal analysis, Spark's Lightning Engine runtime, and scalable metadata storage, scale to petabytes of data without sacrificing performance and enable:

In-place AI and machine learning: Apply powerful AI models and Gemini directly to AWS data and Azure without migration. Ingesting metadata and context right where it lives helps guarantee high-accuracy grounding and a faster time-to-market.
Avoid the cost and delay of data copying: Query live data stored in other clouds directly and bring your data closer to your agents.
Unified analytics experience: Deliver consistent, hardware-optimized performance across BigQuery, Spark, or open-source engines by centralizing multi-cloud compute back to Google Cloud’s infrastructure.

Bridging the agent trust gap with universal context

To prevent hallucinations, AI agents need more than raw technical metadata; they need deep business context. The borderless Lakehouse relies on Knowledge Catalog, our always-on agentic context engine, to establish a unified view of your enterprise context across clouds, without moving physical files.

To support this, the borderless Lakehouse’s runtime catalog automatically synchronizes with AWS Glue, Databricks Unity Catalog, and Snowflake Horizon, all in preview. Knowledge Catalog then ingests these feeds to aggregate, extract, and index their metadata, translating raw schemas into clear business terminology and searchable column-level lineage. As schemas change, Knowledge Catalog instantly updates their business meaning, delivering:

Lower costs and overhead: Minimize expensive, time-consuming data migration pipelines while gaining a consolidated view of your entire multi-cloud estate.
Trustworthy AI decisions: Provide agents with a clear semantic layer and lineage tracking so they can quickly discover, trust, and accurately interpret data.
Automated governance: Embed security directly into the metadata layer, helping ensure AI agents strictly respect compliance guardrails and access permissions.

Build and scale agents with Gemini Enterprise

By pairing the open-source Google Cloud Data Agent Kit with the Conversational Analytics API, you can build and publish custom data agents that operate on the borderless Lakehouse directly into Gemini Enterprise. This allows business users to talk to their data using natural language. The Data Agent Kit is a full-stack collection of agent building capabilities — meeting developers inside their favorite IDEs (like VS Code) — to package pre-codified analytical skills and Model Context Protocol (MCP) tools. With the borderless Lakehouse, your agents operate across your entire data estate and help with:

Integrated data and agent ecosystem: Built-in MCP tools establish secure, direct connections to BigQuery, Managed Spark, and Cloud Storage, eliminating the need to write complex pipeline code or copy-paste massive table schemas into LLM prompts.
Self-service analytics: Business users bypass static dashboards, querying and visualizing multi-cloud datasets instantly in plain language within the Gemini Enterprise interface.
Grounded, high-accuracy agent results: Agents run on top of the Knowledge Catalog, ensuring that natural-language-to-SQL translations are anchored in curated business schemas, highly secure, and strictly governed.

The economics of the borderless Lakehouse and agents

The borderless Lakehouse redefines enterprise AI economics by delivering compounded savings across data transfer, compute, and token consumption. By utilizing Cross-Cloud Interconnects and zero-copy sharing, you bypass fragile data pipelines and unpredictable egress fees to query remote datasets at a flat, predictable rate. Knowledge Catalog filters and delivers the precise, minimal business context required for each prompt, preventing token bloat and eliminating unnecessary reasoning loops. BigQuery AI prevents runaway agent billing through built-in token controls that let you estimate token usage pre-query, enforce strict limits, and leverage an optimized mode that automatically uses smaller, distilled models. In fact, customers are seeing 230x reduction in token consumption using BigQuery’s cost-optimized, built-in AI functions.

Next steps

The future belongs to the system of action, and the borderless Lakehouse allows your AI agents to query, reason, and act on your data wherever it lives — safely, instantly, and cost-effectively. To start building, check out the official Google Cloud Lakehouse About Guide and explore our Building a borderless Lakehouse codelab.

_{1. Customers are required to pay an hourly fee for interconnection service.}

Bringing Conversational Analytics to your entire data ecosystem

Tue, 28 Jul 2026 17:30:00 +0000

Increasing the adoption of generative AI across the enterprise requires you to do more than deploy a generic chatbot with a custom wrapper. Interacting with business-critical databases demands absolute trust, strict governance, and deep grounding in enterprise semantics.

Over the last year, Conversational Analytics (CA) in Google Cloud has moved from isolated experiments to scaled, enterprise-wide deployments. BigQuery Conversational Analytics and the Conversational Analytics API are now generally available, adding to the general availability of Conversational Analytics in Looker last year. Building on this momentum, Conversational Analytics in Databases are also available in Preview. And so much more has happened — Google Cloud Conversational Analytics is available for more data, across more surfaces, with more enterprise controls, and greater capability than ever before.

Let’s take a deeper look at the state of Conversational Analytics in the Google Data Cloud — what you can do with it, the benefits that it brings, and how to get started with it today.

Query across multi-cloud and database workloads

Conversational Analytics is now generally available for BigQuery and Looker, and in preview for AlloyDB, Cloud SQL, and Spanner. You can also analyze data stored in Lakehouse Managed Service tables, Apache Iceberg REST catalogs, and federated AWS S3 Unity Catalogs. Whether your data resides exclusively in Google Cloud or across multiple cloud providers, your agents can query it natively.

For data practitioners, Conversational Analytics is integrated directly into BigQuery Studio, BigQuery Data Canvas, Database Studio and Data Agent Kit. For business teams, these conversational capabilities extend directly into Looker, Data Studio, and Gemini Enterprise. Data teams can publish Conversational Analytics agents created in BigQuery, Looker, AlloyDB, Spanner, and Cloud SQL directly into Gemini Enterprise, giving business leaders a centralized interface to query complex data safely.

Our APIs and MCP tools let you embed Conversational Analytics wherever your business users work, like custom applications and multi-agent systems, or as slack chatbot that can answer questions across data sources, as we showed at Google Cloud Next.

Enterprise security and governance controlsScaling generative AI to tens of thousands of users requires ironclad governance and transparent cost controls. Conversational Analytics includes Customer Managed Encryption Keys (CMEK), Private IP, and Virtual Private Cloud (VPC) controls.

We guarantee Data Residency (DRZ) at rest and machine learning processing inside multi-region endpoints within the European Union and the United States, along with HIPAA compliance. For data access, role-based controls, including parameterized secure views in AlloyDB for PostgreSQL, help ensure users chatting with an agent only see data they are authorized to view, enforced down to row- and column-level permissions.

Monitoring Conversational Analytics in BigQuery to track agent fleet health, active users, query volumes, and top knowledge sources.

As usage grows, administrators need tools to manage costs, observe system health, and improve accuracy. You can configure native cost controls to define limits on maximum query sizes in bytes, and track usage through BigQuery query labels and Looker system activity logs.

To maintain fleet visibility, agents can also export health, tool usage, latency, and token consumption metrics via OpenTelemetry (OTEL) standards. Integrated feedback loops allow administrators to review agent traces and user feedback, establishing a foundation for continuous evaluation and accuracy improvements over time.

Grounded context through agent and data co-designWrapping a generic LLM around an enterprise database can sometimes lead to hallucinated logic. To minimize this, we co-designed Conversational Analytics agents alongside the data platforms they query.

For instance, agents leverage Knowledge Catalog for data discovery, glossaries, and automated context enrichment like table joins and descriptions. BigQuery Graphs and Spanner Graphs allow agents to query structured and unstructured data across multi-hop relationships. Additionally, Looker’s semantic layer (LookML) grounds agent responses in centrally governed metric definitions, helping ensure answers remain deterministic rather than relying on guessed SQL joins.

Grounding Conversational Analytics across Knowledge Catalog, BigQuery Graph, and Looker’s semantic model helps ensure deterministic, enterprise-governed responses.

Conversational Analytics agents are also co-designed with the data they query. This means their tools are context-aware, to have the best understanding of the metadata. They also benefit from built-in capabilities like multimodal data querying using BigQuery object tables, operating over multimodal data with ai.search, ai.generate_embedding, ai.classify, ai.score and using ai.forecast and ai.detect_anomalies to use the TimesFM foundation for forecasting and anomaly detection.Additionally, ai.key_drivers performs automated contribution analysis to pinpoint exactly what is driving unexpected changes in your data. When integrated with Looker, these agents leverage the semantic layer to ground their responses in centrally governed, deterministic metrics. To avoid AI hallucinations, this API-first approach (using 'Golden Queries') ensures agents retrieve verified business logic rather than guessing at SQL joins. Looker additionally equips the agents to seamlessly navigate high-cardinality datasets with dynamic filtering, automatically enforce row-level security during the chat experience, and surface context-aware suggested questions.

Proactive insights with Agentic Workflows

Analytics is moving beyond reactive question-answering toward proactive intelligence. That is, instead of requiring users to ask the right question at the right time, Conversational Analytics agents can run multidimensional deep dives to analyze 10 to 20 contributing factors behind a change in a metric.

With Agentic Workflows, now in preview, you can schedule automated reporting routines delivered directly into your chat workflow. Agents continuously run anomaly detection across key metrics, sending daily or weekly summaries straight to your team. Streaming anomaly detection can also launch an agent automatically the moment a key metric deviates from baseline thresholds.

Running a multi-step deep dive in Conversational Analytics to automatically investigate complex data relationships across enterprise datasets.

Flexible integration with APIs, SDKs, and MCP

Conversational Analytics is available to developers and business users in their existing environments. The Conversational Analytics API includes native SDKs for Node.js, Java, Go, Python, PHP, Ruby, and .NET and keeps insights where the work happens. We are expanding how and where people use Conversational Analytics, starting with Looker Dashboards and Data Studio, as well as supporting publishing agents to Gemini Enterprise.

You can also add Conversational Analytics to other multi-agent systems. Using the Agent Development Kit (ADK) and Model Context Protocol (MCP), you can integrate Conversational Analytics into custom applications, Slack bots, or multi-agent orchestrators. For example, a supply chain orchestrator agent can query a financial data agent to calculate the margin impact of a shipping delay in real time.

Get started with Conversational Analytics

Google Cloud Conversational Analytics unifies your data estate, security control plane, and developer APIs to deliver proactive data insights wherever your team works. Explore our Conversational Analytics documentation, review our quickstart repositories, and sign up to try our new previews today.

Announcing general availability of SAP Business Data Cloud Connect for BigQuery

Mon, 27 Jul 2026 16:30:00 +0000

Traditional data replication techniques often struggle to deliver the data freshness that modern workflows require. To help organizations overcome this challenge, SAP and Google Cloud are announcing that SAP Business Data Cloud Connect for BigQuery is now generally available.

As the next phase of our long-standing partnership, SAP BDC Connect for BigQuery gives you zero-copy access to data products across SAP Business Data Cloud and BigQuery – reducing silos, preserving valuable business context, and accelerating analytics and AI. This makes current, semantically rich data available across both environments, helping AI agents execute complex business tasks while your teams remain in control.

Stop copying data for AI projects

Built to support a zero-copy architecture, SAP BDC Connect for BigQuery establishes a secure, bi-directional connection between SAP Business Data Cloud and Google’s data and AI ecosystem. Access SAP tables, metadata, and business semantics directly in BigQuery and Knowledge Catalog (formerly Dataplex) and vice versa. Ground your AI agents in operational reality instead of raw, isolated database fields.SAP BDC Connect for BigQuery enables organizations to:

Scale AI on trusted data: Build governed, trustworthy AI grounded on a semantically rich data foundation with live, zero-copy access to data products across SAP Business Data Cloud and BigQuery.
Deploy multi-agent workflows: Orchestrate workflows with intelligent agents across SAP and Google with Gemini Enterprise Agent Platform and SAP Joule.
Discover deeper patterns: Combine mission-critical SAP data with unique datasets, such as Google Trends and Google Maps geospatial data, to create better business context. You can then publish this enriched data directly back to SAP.
Improve business analytics: Empower employees to ask open-ended business questions in natural language using Gemini Enterprise or Joule and receive immediate, context-aware answers grounded in SAP and Google Cloud data.

Scale your data value without inflating costs

SAP BDC Connect for BigQuery’s zero-copy architecture helps lower your total cost of ownership (TCO) compared to alternative data platforms by minimizing the need to replicate, store, and maintain duplicate copies of your data. Because you can query data in place, you don’t incur data sharing fees. This creates a predictable billing model that insulates your organization from unexpected charges.

How customers translate shared data into actions

Organizations with preview access to SAP BDC Connect for BigQuery are already using its zero-copy architecture to solve more complex operational challenges. By speeding up data pipelines, early adopters are breaking down global data silos, predicting supply chain disruptions, and responding to fluctuating customer demands faster. Contrast that with traditional data practices where data connections take weeks to build, stalling enterprise AI initiatives. By interacting directly with SAP and Google data, teams can rapidly build functional prototypes and accelerate return on their AI investments.

ElringKlinger is a global automotive supplier managing complex, multi-country manufacturing and supply chain data, and is using zero-copy to break down data silos. By querying real-time ERP data without having to replicate it, the company aims to reduce the complexity of its data infrastructure while supporting rapid production analytics with localized inventory reporting.

"Operating a global automotive footprint means managing vast, complex supply-chain and production data. Today, the majority of that data sits in SAP BW/4HANA, our established reporting backbone. Testing SAP BDC Connect for BigQuery showed a powerful new path forward: BigQuery as an analytical layer on top of our existing BW/4HANA assets — no data copy, no migration — unlocking AI-based Conversational Analytics while fully preserving our prior investment. This also lays the groundwork for agentic use cases that automate supply chain processes, giving teams real-time visibility into manufacturing signals worldwide." - Cleve Bankston, Project Manager and Dirk Brümmer Head of IT Architecture, ElringKlinger

Unify data to build smart workflows with partners

SAP and Google Cloud’s global partner network is actively building use cases with SAP BDC Connect for BigQuery. These help you move from isolated AI experiments to amplifying your team's ability to handle highly variable tasks that are difficult to manage manually. To make this possible, our partners combine trusted SAP data with broader sources to help you deploy agentic AI and autonomous operations. With this unified approach, your team can connect disparate systems, maintain governance, and drive operations that deliver bottom-line impact.

“As enterprises accelerate their reinvention agenda, the ability to unify data across business systems is foundational to scaling agentic AI. And that’s where SAP Business Data Cloud Connect for BigQuery is quite powerful. It enables enterprises real-time access to trusted financial, operational, and sustainability data, eliminating traditional data silos. This, combined with Gemini Enterprise and Accenture's deep industry expertise, allows organizations to build intelligent agents, drive autonomous operations, and modernize their digital core faster while unlocking new levels of business value.” – Chetna Sehgal, Global Practice Lead, Accenture Google Business Group

“Organizations don’t want a data science project; they want concrete business outcomes like working capital improvement and reduced operational costs. By leveraging this zero-copy integration, we built an autonomous procurement agent for a client. Because the agent reads live SAP inventory levels and market data inside BigQuery simultaneously, it autonomously re-routed orders during a recent supplier shift — saving millions in potential downtime and demonstrating that agentic AI tied to real-time data can deliver bottom-line impact.” - Piyush Bhandari, Managing Director, Deloitte Consulting LLP

“Based on a recent assessment and prototyping experience with a KPMG member firm energy client, we believe there is strong potential for SAP BDC Connect for BigQuery to help organizations modernize enterprise planning, analytics, and data integration on a scalable foundation. KPMG firms’ evaluations indicate that this integration can support higher-volume analytical workloads at enterprise scale, helping to provide a critical data foundation to allow exploration of future AI and agentic use cases. As with any transformation, realizing the full business value can depend on the implementation approach, data quality, and organizational readiness.” - Mark Shank, Principal, Advisory, Banking, Tech & Data Engineering, Google in KPMG US

“Our clients aren't just looking for more information; they’re looking for better business decisions. At PwC, we’re helping organizations unify SAP business data with broader enterprise data sources inside Google Cloud. This creates a single, trusted foundation for AI that strengthens analytics, supports robust governance and compliance, and drives intelligent workflows across the enterprise. This unified approach is how organizations move from isolated AI experiments to true, enterprise-wide business transformation.” - Patrick Pugh, Global and US Alliances and Ecosystem Leader, PwC US

Power of agentic data: Perspectives from SAP and Google Cloud

"Every enterprise is striving to inject intelligence and automation into the core of their business operations. By uniting SAP Business Data Cloud with BigQuery and Gemini, we are breaking down one of the biggest walls in enterprise data. We are going far beyond raw data access by actively sharing rich semantic metadata and agentic context across platforms at zero data-sharing cost. This ensures our joint customers can deploy autonomous AI agents that operate with complete, real-time business awareness — executing complex supply chain and financial processes with absolute precision." - Andi Gutmans, VP/GM, Agentic Data Cloud, Google Cloud

"SAP Business Data Cloud Connect for Google BigQuery delivers real-time, bidirectional, zero‑copy data sharing that brings SAP’s mission‑critical data to Google BigQuery making it accessible to Google Gemini and Gemini Enterprise Agent Platform. With SAP BDC Connect, customers can unify SAP and non‑SAP data without complex extracts or custom APIs, reducing data complexity, strengthening governance, and accelerating trusted, AI powered business outcomes." - Irfan Khan, President & Chief Product Officer, SAP Data & Analytics

Regional availability

SAP Business Data Cloud: Available today in Google Cloud regions in Australia (Sydney), Brazil (São Paulo), India (Mumbai), Israel (Tel Aviv), Germany (Frankfurt), Japan (Osaka), Saudi Arabia (Dammam) and the United States (Iowa).
SAP BDC Connect for BigQuery: Generally available globally for all Google Cloud customers.
Cross-cloud support: SAP BDC Connect for BigQuery supports SAP Business Data Cloud instances hosted on Google Cloud and AWS, with support for SAP Business Data Cloud instances hosted on Microsoft Azure coming soon.

Get started

To learn more, visit our website or speak with your Google Cloud account representative. For details on getting started with SAP BDC connect for BigQuery, visit our documentation page to get started with your zero-copy connection today.

Open Knowledge format v0.2 tackles agentic trust

Fri, 24 Jul 2026 16:00:00 +0000

When we introduced the Open Knowledge Format (OKF) in June 2026, we asserted that the context that agents need (table schemas, metric definitions, runbooks) should live in a format, not in a proprietary service, and not scattered across unstructured text blobs. Accordingly, OKF v0.1 started simple: just markdown, YAML frontmatter, and a handful of conventions.

The strong response and contributions from the developer community informed our focus for the next version. Since launch, contributors opened extension proposals (typed relationship edges, agent-routing hint fields, an optional erasure conformance profile, an .okfignore convention, and more), sent new sample bundles, and begun cataloging OKF ecosystem tools built outside Google. Many of these contributions and contributor feedback reflected a larger concern about OKF: once agents are writing to the corpus, can it really be trusted?

The most valuable OKF bundles won't be written by hand once and then read forever. They're written continuously, by agents, and consumed by a different set of agents. A human-authored wiki page comes with an implicit guarantee: a person wrote it, and you can hold them accountable if it is wrong. When an agent generates ten thousand concepts overnight, that guarantee is gone. To provide accountability, a consumer (often another agent) has to judge each concept on explicit signals instead, and needs to answer five questions:

What was this created from? (provenance)
How much should I trust it? (trust)
Is it still true? (freshness)
Is it the current version? (lifecycle)
Was this number produced the way we said it must be? (attestation)

In OKF v0.2, it is now possible to answer all five of those questions from frontmatter, while the format remains as minimally opinionated as v0.1. It adds vocabulary, not rules: type is still the only always-required field, every new field is opt-in, custom keys are still preserved rather than rejected, and a bundle that adopts none of the additions is exactly as valid as it was under v0.1. Everything new below is optional, but its absence now carries meaning: an unverified concept is distinguishable from a verified one (although never rejected for the difference).

From describing to deciding

v0.1 already kept metadata in frontmatter: type, title, description, resource, tags. Those fields describe a concept: what it is and what it points at. v0.2 adds a second kind of frontmatter field, the kind you use to decide something about a concept before you read it: who produced it, whether it has been verified, whether it is still current, and how a value it reports should be computed.

The reason these fields belong in frontmatter is that most interactions with a concept never actually progress to accessing the information in the body of the file. A consumer, whether a person, deterministic code, or an agent scanning during search and discovery, first has to decide whether a concept is relevant at all. Everything in a concept should be concise, but frontmatter has the narrower job of elevating exactly the signals needed to make decisions about relevance and trustworthiness, so it can be made cheaply and often, without spending tokens on prose. The content that must be read in full stays in the body, accessed only once a concept is chosen. Trust becomes something you can filter on before you commit to reading.

To make the sections that follow concrete, every example below draws from a small OKF v0.2 bundle we've prepared as a companion to this post: acme_retail, a fictional US retail company's shared knowledge for AI-assisted analytics over BigQuery:

code_block: <ListValue: [StructValue([('code', 'acme_retail/\r\n├── index.md, log.md\r\n├── tables/ orders.md\r\n├── metrics/ revenue.md, gross-margin.md, gross-margin-legacy.md\r\n├── computations/ revenue-ytd.md, gross-margin-period.md\r\n├── skills/ run-on-bq.md\r\n├── attesters/ sql_equality.py\r\n└── policies/ revenue-recognition.md, margin-standard.md'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe3122ea190>)])]>

Each section that follows shows the corresponding file. Here is tables/orders.md, a concept carrying the new signals:

code_block: <ListValue: [StructValue([('code', "---\r\ntype: BigQuery Table\r\ntitle: Customer Orders\r\ndescription: One row per completed customer order across web, mobile, and marketplace channels. The grain is the order, not the line item.\r\nresource: https://bigquery.googleapis.com/v2/projects/acme/datasets/sales/tables/orders\r\ntags: [sales, orders, revenue]\r\ngenerated: { by: reference_agent/gemini-2.5-pro, at: 2026-06-30T14:00:00Z }\r\nverified:\r\n - { by: human:kliu@acme, at: 2026-07-01T16:00:00Z }\r\nstatus: stable\r\nstale_after: 2026-12-31\r\nsources:\r\n - id: warehouse-schema\r\n resource: https://wiki.acme.internal/data/warehouse/schemas/sales\r\n title: Acme Retail warehouse schema — sales dataset\r\n author: team:data-platform\r\n usage_count: 1240\r\n last_modified: 2026-06-15\r\n - id: revenue-policy\r\n resource: policies/revenue-recognition.md\r\n title: Revenue Recognition Policy (FY2026)\r\n author: human:jsmith@acme\r\n last_modified: 2026-06-15\r\n---\r\n\r\n# Schema\r\n\r\n| Column | Type | Description |\r\n|----------------|---------------|----------------------------------------------------------------------------------------|\r\n| `order_id` | STRING | Globally unique order id. [^warehouse-schema] |\r\n| `order_ts` | TIMESTAMP | Order placement time in UTC; drives fiscal-year assignment. [^revenue-policy] |\r\n| `order_status` | STRING | Revenue is recognized only at `'delivered'` and after the 30-day return window. [^revenue-policy] |\r\n| `net_amount` | NUMERIC(18,4) | `gross_amount - discount_amount`. The recognized-revenue amount per policy. [^revenue-policy] |"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe3122ea910>)])]>

Each of those families answers one of the five questions.

Provenance: sources, not a score

The new sources field records the materials a concept derives from: an external doc, a bundle-relative path, or even a scope descriptor like "all queries in project X." At the same time, an entry can carry objective credibility signals: author, usage_count, last_modified.

The deliberate choice here is what we didn't add. OKF records the signals, not a credibility score. A score is subjective, doesn't port across consumers, and goes stale the moment it's written. Instead, credibility is inferred from the signals by whoever is consuming (and can be dynamically scored by the consumer, if desired), the same way you'd trust a heavily used, recently updated, authoritatively authored source more than an anonymous one. And when the body cites a specific source, it does so with an ordinary markdown footnote keyed to the source id ([^export-schema]), so attribution is per-claim instead of a dangling list at the bottom.

Trust: `generated`, `verified`

Trust is established using two fields, kept deliberately distinct, because who wrote something need not be who confirmed it:

generated: { by, at }: how the current content was produced, and when it last meaningfully changed.
verified: [ { by, at } ]: a list of independent confirmations against the sources or the underlying resource; a human sign-off, a nightly finance process, or both.

From verified, a consumer derives a trust tier: no verified key is unverified; confirmation by machine actors only is machine-confirmed; confirmation by a human:<id> actor is human-reviewed. Tiers are advisory signals, not access control, but they let a consumer say "only surface human-reviewed metrics in the executive dashboard" as a frontmatter filter.

In acme_retail: metrics/revenue.md is authored by the reference agent and verified by the VP of Finance, which places it in the human-reviewed tier. An executive-dashboard consumer configured with a trust-tier filter surfaces it; a throwaway testing environment can accept lower-tier concepts:

code_block: <ListValue: [StructValue([('code', 'type: Metric\r\ntitle: Revenue\r\ngenerated: { by: reference_agent/gemini-2.5-pro, at: 2026-06-30T14:00:00Z }\r\nverified:\r\n - { by: human:jsmith@acme, at: 2026-07-01T09:00:00Z }'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe3122eaaf0>)])]>

Freshness and lifecycle: stale_after, status

OKF v0.2 establishes freshness and lifecycle with the stale_after and status fields. status moves a concept through draft → stable → deprecated (absent means stable). stale_after is a single absolute date. We chose an absolute date over a relative TTL on purpose: staleness becomes a plain date comparison with no reference to when the concept happened to be read, which is exactly the kind of determinism a non-LLM consumer wants.

In acme_retail: metrics/revenue.md and metrics/gross-margin.md both carry stale_after: 2026-12-31 because Acme's finance team re-approves the underlying policies every January. On 2027-01-01 both concepts require re-verification against the FY2027 policy before serving.

And metrics/gross-margin-legacy.md is status: deprecated. Acme changed its cost allocation standard in Feb 2026 (the old formula excluded shipping and fulfillment from its cost of goods sold). The legacy definition is preserved for historical query reproducibility but not surfaced to new work:

code_block: <ListValue: [StructValue([('code', 'type: Metric\r\ntitle: Gross Margin (legacy, pre-FY2026)\r\nstatus: deprecated'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe3122ea610>)])]>

Attestation: Was this number computed the sanctioned way?

Provenance answers where a claim came from. Attestation answers a harder question that matters the moment an agent reports a dollar figure: was this number produced the way we said it must be, or did the agent improvise its own SQL?

OKF v0.2 introduces a new concept type, Attested Computation. It carries not just what a value means but a sanctioned way to compute it, and the means to check that the sanctioned thing actually ran. Here is acme_retail/computations/revenue-ytd.md:

code_block: <ListValue: [StructValue([('code', "---\r\ntype: Attested Computation\r\ntitle: Revenue for a fiscal year\r\nruntime: bigquery\r\nparameters:\r\n - { name: year, type: integer, required: true }\r\nexecutor:\r\n resource: skills/run-on-bq.md\r\n receipt: [job_id, executed_sql, result]\r\nattester:\r\n resource: attesters/sql_equality.py\r\ngenerated: { by: reference_agent/gemini-2.5-pro, at: 2026-06-30T14:00:00Z }\r\nverified:\r\n - { by: human:jsmith@acme, at: 2026-07-01T09:00:00Z }\r\nstatus: stable\r\nstale_after: 2026-12-31\r\nsources:\r\n - id: revenue-policy\r\n resource: policies/revenue-recognition.md\r\n title: Revenue Recognition Policy (FY2026)\r\n author: human:jsmith@acme\r\n last_modified: 2026-06-15\r\n---\r\n\r\n# Computation\r\n\r\nSELECT\r\n SUM(\r\n CASE\r\n WHEN o.currency = 'USD' THEN o.net_amount\r\n ELSE o.net_amount * fx.rate_to_usd\r\n END\r\n ) AS revenue_usd\r\nFROM `acme.sales.orders` AS o\r\nLEFT JOIN `acme.finance.fx_daily_rates` AS fx\r\n ON fx.currency = o.currency\r\n AND fx.rate_date = DATE(o.order_ts)\r\nWHERE o.order_status = 'delivered'\r\n AND DATE_DIFF(CURRENT_DATE(), DATE(o.order_ts), DAY) >= 30\r\n AND EXTRACT(YEAR FROM o.order_ts) = @year"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe3122ea3d0>)])]>

The agent may only fill the declared parameters; it must never author or edit the computation. A consumer runs the computation through the executor, which returns a receipt (here: a BigQuery job_id, the SQL that was actually executed, and the result). Then a deterministic, no-LLM attester inspects that receipt and returns a verdict: did the query that ran equal the sanctioned computation bound with the claimed parameters, and does the displayed value match the receipt's authoritative source? Because the comparison is mechanical, a rewritten query, a swapped computation file, or a mutated dependency fails the check.

In acme_retail: the attester at attesters/sql_equality.py canonicalizes both SQLs (strips comments, collapses whitespace, uppercases known keywords) and refuses to return ok if the canonical forms differ. A swapped table name, an added filter, a dropped JOIN all fail attestation. The consumer refuses to display the value when the verdict comes back false.

While we used BigQuery (and SQL) in this example, the abstraction is deliberately flexible. Attested computations can correspond to invoking semantic models (in Looker, AtScale, or other systems), querying structured knowledge graphs, or even making arbitrary API calls.

Crucially, OKF records the computation and how to check it; it never executes anything itself. And attestation is distinct from verification: verified confirms the definition still matches policy (slow, doc-level, stored in the bundle); attestation confirms a single run produced the value correctly (per-call, runtime, never stored in the bundle). A stale definition can still attest cleanly; a freshly-verified definition still needs attestation on every run. That's why both exist.

What we're releasing with v0.2

As with v0.1, the reference implementations are deliberately proofs of concept; nothing about OKF requires them. Here’s what’s changing in the Github repo:

reference_agent now emits the provenance and trust families as it generates, so a freshly-minted bundle arrives with generated sources, and citations already in place.
The static visualizer surfaces trust tier, status, and staleness alongside the concept graph, so these signals are visible, not just parseable.
Updated sample bundles (GA4 e-commerce, Stack Overflow, Bitcoin, and the acme retail example used in this blog post) carry the v0.2 fields.
A Knowledge Catalog demo shows a bundle round-tripping through Google Cloud's Knowledge Catalog (formerly Dataplex): clean OKF on disk, trust and provenance signals preserved through the catalog and back.

Compatibility

v0.2 is a minor version bump that is additive, backward-compatible, with two deliberate renames: timestamp is superseded by generated.at, and the body # Citations list is superseded by sources; in both cases a v0.2 consumer can fall back to the v0.1 form. A v0.1 bundle drops in unchanged.

Where we go from here

Read the spec (it's still short). Write a producer that emits trust signals for your source system. Write a consumer that filters on them. Try an Attested Computation against your own finance definitions. File issues, send PRs, propose extensions. A lingua franca is only as good as the number of parties who speak it, and now they can also check each other's work.

OKF v0.2 spec, samples, and reference implementations: github.com/GoogleCloudPlatform/knowledge-catalog/tree/main/okf

Your AI agents are ready. Is your data?

Thu, 23 Jul 2026 16:00:00 +0000

What’s one of the biggest bottlenecks stopping organizations from scaling their AI initiatives? It isn’t the capabilities of today’s models — it’s their access to business context and semantic meaning.

In the agentic era, enterprises need to go beyond simply storing data to activating it with trusted context, moving from passive systems of record to proactive systems of action.

But AI agents operate with nonlinear speed; for example, a single prompt can trigger the agent to independently browse, query, and execute across multiple systems, placing stress on the underlying infrastructure. If the compute, networking, and storage layers aren't optimized for agentic AI, the data platform sitting on top of them will buckle.

It’s no wonder that, according to our State of infrastructure report, 83% of organizations believe they require infrastructure upgrades to support production-grade agentic AI systems.

To solve this problem, we introduced the Agentic Data Cloud at Google Cloud Next 2026; unifying your data, AI models, and operational databases into a single System of Action. To make an Agentic Data Cloud work, it must be AI-native from the chip to the model. The underlying infrastructure must be able to accommodate agentic load.

Google’s Agentic Data Cloud

Let’s explore how the right infrastructure foundation empowers an Agentic Data Cloud to solve the biggest data challenges organizations face today.

Overcoming a lack of context

To be effective, agentic systems require access to context that is often found in fragmented data systems and legacy architectures. This can make it hard for agents to get this context, leading to incomplete, inaccurate results. In fact, our report found that 43% of IT leaders cite “difficulty integrating with legacy APIs and data sources” as their biggest agentic AI infrastructure gap.

But organizations cannot simply move massive datasets and connect them to AI without increasing complexity and cost.

Our Agentic Data Cloud solves this by leveraging a borderless Lakehouse running on open, flexible infrastructure. By accessing powerful native engines like BigQuery and Spanner over open standards (Apache Spark, Apache Iceberg), agents can read, reason over, and activate data across environments as if it were local, bypassing the latency and costs of traditional setups.

Escaping unnecessary manual work

Scaling agents on a patchwork of disconnected systems can create significant bottlenecks. In our research, 81% of leaders called out operational complexity and engineering overhead as top unforeseen expenses when scaling AI, citing the time engineers spend doing manual work to patch together AI agents across disparate systems.

To move from thinking to doing, agents must be able to connect real-time data across both analytical and operational sources. This requires vertical integration. When an Agentic Data Cloud is built on an AI-native infrastructure where the models, data systems, and underlying accelerators are co-designed, there are fewer network hops and tooling is better integrated. This unified system allows an agent to reach an insight and trigger secure transactions without the typical engineering overhead.

Bringing trust and knowledge to the data

It’s not enough for agents to just discover and query data. To take safe, accurate actions, agents also need rich context and business logic. Yet, 36% of leaders cite a lack of specialized, high-throughput vector databases used for AI model grounding, as a key infrastructure gap, hindering their ability to give agents context.

In order to work to their full potential, agents need a foundation which is built to read and write data systems in real-time, including legacy ERPs and third-party CRMs. It also gives them the long-term memory to recall a user’s preference from, say, three weeks ago, while executing a complex task today. And without this real-time automation, agents have to re-process data for every single query.

To provide context for AI, organizations are using Knowledge Catalog to aggregate and enrich data in their data lakes, and enable agentic searches. By extracting meaning from unstructured data and automatically generating semantics, the catalog acts as an active reasoning layer. That catalog in turn, must be backed by high-throughput infrastructure, so that agents can retrieve the right context.

The path forward

To turn AI into a true competitive advantage, it’s time to build a connected, active data ecosystem. Giving your agents seamless access to all of your data is a must to move from pilots to production, and this must be supported by an infrastructure that can handle the demands of the agentic era. The winners in 2026 and beyond won’t necessarily be the ones with the smartest agents. They’ll be the ones who can feed those agents the right knowledge — securely, cost-effectively, and at scale. Is your data ready for the agentic era?

See how leaders are taking an AI-optimized approach to architecture in the State of infrastructure in the agentic AI era report.

From maintenance to innovation: Checkout's migration to Managed Service for Apache Airflow

Wed, 22 Jul 2026 14:00:00 +0000

Data engineering teams often face a “Day 2” operational reality after building a data platform: the ongoing work of maintaining the orchestrator itself.

For the Data Platform team at Checkout.com, managing a self-hosted Apache Airflow environment on another hyperscaler was consuming time the team wanted to spend elsewhere as server management, patching, and incident response were pulling focus from building pipelines.

By migrating to Managed Service for Apache Airflow (Gen 3), Google Cloud’s fully managed Airflow service, Checkout.com transformed its reliability and cost structure. Here’s how they built a more scalable, cost-efficient, and robust data foundation.

The starting point: self-managed Airflow

Before the migration, Checkout.com ran Airflow on self-managed infrastructure. While functional, maintaining the underlying resources required significant attention. Patching, upgrades, and server management created regular interruptions.

Operational data from the past year illustrates some of the challenges the company was navigating:

Reducing operational friction: In its self-managed environment, Checkout.com faced stability challenges, particularly during high-load periods.
Complex dependency management: Upgrading packages and ensuring compatibility was a constant, manual struggle. With Managed Airflow (Gen 3), the company was able to simplify this by handling dependencies at the image level, ensuring seamless compatibility out-of-the-box during routine environment upgrades.
DAG sync time: Syncing DAGs to the scheduler took approximately six minutes after deployment to S3, which affected iteration speed.
Manual processes: Scaling required manual intervention, and onboarding new teams meant manually creating secrets and variables for dbt.

The solution: Managed Service for Apache Airflow (Gen 3)

Checkout.com’s team migrated to Managed Airflow to offload infrastructure responsibility and take advantage of Google Cloud's managed scalability. The results were immediate and measurable across three areas: reliability, cost, and developer velocity.

Dynamic scaling in action

In the previous elastic container service setup, the team allocated the maximum number of workers required for peak loads. This meant paying for peak capacity around the clock, regardless of actual usage.

Managed Airflow provides built-in dynamic scaling, eliminating the need for manual resource management. The environment automatically adjusts the number of workers based specifically on the workload demands. When tasks spike, the system scales up; when they drop, it scales down to save resources. Similarly, moving from fixed provisioning to dynamic scaling reduced monthly costs by an estimated 30%.

Reliability and DAG isolation

Achieving increased stability was a primary driver for Checkout.com’s migration since in the past, a single problematic DAG could affect its entire environment.

Managed Airflow introduced a number of critical architecture improvements:

DAG isolation: Each DAG runs in its own execution environment. If one DAG fails or consumes excessive resources, it doesn’t affect the entire environment.
Managed operations: Google Cloud handles patching and upgrades during scheduled windows, removing the need for manual upgrade management.
Improved visibility: Integration with Google Cloud Monitoring and Cloud Logging provides clear visibility into task execution. Engineers can now debug issues independently without escalating to the platform team.

Faster developer workflows

The migration also improved day-to-day workflows for Checkout.com’s data engineers.

Faster deployments: Using Cloud Storage for DAGs enabled near-instant syncing.
Simpler onboarding: Teams no longer needed platform support to create variables before onboarding.
Modernizing dbt execution: One of the company’s most significant wins was changing how it runs dbt. Previously, its engineers had to manually install and manage complex virtual environments for every supported dbt version. By leveraging containerized dbt runs, Managed Airflow (Gen 3) eliminates dependency bottlenecks. This ensures complete dependency isolation, allowing teams to run any required dbt model with minimal setup and no manual infrastructure overhead.
Environment updates: The company no longer needs to redeploy the entire Airflow environment to add new roles or update Python packages.
AI-powered troubleshooting with Gemini Cloud Assist: In a self-managed environment, a failed task often triggered a frantic hunt through fragmented logs and metrics. With Managed Airflow, Checkout.com can initiate a Gemini investigation directly from its Airflow DAG UI in the Google Cloud console.

Gemini doesn't just provide generic error messages; it generates a scorecard that evaluates different hypotheses with both supporting and contradictory evidence, which can drastically reduce mean time to recovery.

Conclusion

For Checkout.com, the move to Managed Airflow (Gen 3) marked a strategic shift, one that freed its engineers to focus on delivering value.

"With Managed Service for Apache Airflow, we’ve achieved significant improvements in efficiency, scalability, and reliability. Managed infrastructure, automated scaling, faster deployments, and isolated execution environments have transformed how we operate." — Keisi Mancellari, Data Platform Engineer, Checkout.com

With a stable, scalable, and cost-efficient platform in place, Checkout.com is now able to focus on the future of its data pipelines, confident that its orchestration layer is ready for whatever comes next.

Learn more about Managed Service for Apache Airflow and how it can support your data platform.

_{Special thanks to the following contributors to this post: Serge Bouschet and Keisi Mancellari}

Level Up Your Column-level Security: Using IAM Data Governance Tags in BigQuery

Fri, 17 Jul 2026 16:00:00 +0000

Many BigQuery customers rely on policy tags for protecting their sensitive information in BigQuery. Policy tags were the go-to solution for applying column-level access controls, allowing only users with the right permission to view sensitive columns like personally identifiable information (PII). It was a robust and effective system — for its time.

However, data ecosystems have grown in complexity, and the tools we use to help secure them need to evolve with them. New challenges include creating and managing a taxonomy that supports multiple tags across multiple regions and locations, enabling disaster recovery, and integrating with a broad centralized governance strategy.

To help you meet the needs of today’s data ecosystems, we're excited to introduce the preview of data governance tags in BigQuery. Built on Google Cloud's Identity and Access Manager’s (IAM) Resource Manager infrastructure, data governance tags provide a scalable, and robust method to help you manage access controls and protect your BigQuery column data.

What are IAM data governance tags?

Data governance tags are a special type of Resource Manager tags. You can create it by setting the purpose field to DATA_GOVERNANCE when creating a tag key in IAM, you designate it for use in BigQuery column-level security. You can create a hierarchical tree of data governance tags specifically for column-data governance purposes and apply them directly to your BigQuery columns.

Why use data governance tags for column-level security?

Global scope, regional enforcement: Unlike policy tags (which are regional-only), data governance tags are global. You can define a single tag key:value pair (like “data_sensitivity:high”) at the organization level and use it across any project or region in your organization.
Managed disaster recovery: Security policies should persist during a failover. Data governance tags and their associated data policies are automatically replicated to secondary regions. If you need to switch regions, your security posture moves with you automatically.
Hierarchical security: You can now build a tree of tags up to five levels deep. This allows for inheritance and more granular classification (such as PII > Financial > CreditCardNumber).
Decoupled governance: You can tag your data to organize and classify it before you decide to enforce security. Access control only kicks in once you define a data policy for that tag, giving your team more flexibility during data onboarding.

Three steps to column-level security

Step 1: Create the tag key and values

1. Create data governance tag key: First you create an IAM tag key in Console, gcloud CLI, or API. The magic happens when you specify the purpose field as --purpose=DATA_GOVERNANCE for the tag key. This key change tells Google Cloud that this tag will be used for column-level security in BigQuery.

code_block: <ListValue: [StructValue([('code', '# Example: Creating a Data Governance tag key named "data_class"\r\ngcloud resource-manager tags keys create data_class \\\r\n --parent=projects/my-governance-project \\\r\n --purpose=DATA_GOVERNANCE'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe310167820>)])]>

2. Create tag values:

Once your data governance tag key has been created, you need to create specific tag values under the key that you will use to categorize/classify your column data. One of the useful features of data governance tags is the ability to build a hierarchical tree of tag values. The tag-values tree allows you to create broad categories and then drill down into specific categories based on data type. You can go up to five levels deep for granular access control.

code_block: <ListValue: [StructValue([('code', '# Level 1: Create a tag value called "pii"\r\ngcloud resource-manager tags values create pii \\\r\n --parent=my-governance-project/data_class\r\n\r\n\r\n# Level 2: Create a child value under "pii" for "private" data\r\ngcloud resource-manager tags values create private \\\r\n --parent=my-governance-project/data_class/pii\r\n\r\n\r\n# Level 3: Create another child tag value for "email" under "private"\r\n# You can go up to 5 levels deep for granular control\r\ngcloud resource-manager tags values create email \\\r\n --parent=my-governance-project/data_class/private'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe3101678b0>)])]>

Step 2: Attach tags to your columns via JSON schema

1. Export your existing schema

For existing tables, the most efficient way to manage tags is by updating the table schema using a JSON file and using API or BQ CLI because it allows you to tag multiple columns at once.

code_block: <ListValue: [StructValue([('code', '# Save the current table schema to a local JSON file.\r\nbq show --schema --format=prettyjson my_project:my_dataset.my_table > schema.json'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe310167b20>)])]>

2. Add the dataGovernanceTags to your JSON file

Open schema.json and add the tag mapping to your sensitive columns. Note the use of the namespaced key and the short name for the value.

code_block: <ListValue: [StructValue([('code', '[\r\n {\r\n "name": "user_email",\r\n "type": "STRING",\r\n "dataGovernanceTagsInfo": {\r\n "dataGovernanceTags": {\r\n "my-governance-project/data_class": "email" \r\n }\r\n }\r\n },\r\n {\r\n "name": "phone_number",\r\n "type": "STRING",\r\n "dataGovernanceTagsInfo": {\r\n "dataGovernanceTags": {\r\n "my-governance-project/data_class": "private"\r\n }\r\n }\r\n },\r\n {\r\n "name": "government_id",\r\n "type": "STRING",\r\n "dataGovernanceTagsInfo": {\r\n "dataGovernanceTags": {\r\n "my-governance-project/data_class": "pii"\r\n }\r\n }\r\n }\r\n]'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe310167250>)])]>

3. Update the table:

Apply the schema to your BigQuery table.

code_block: <ListValue: [StructValue([('code', '# Overwrite the table schema with your newly tagged JSON file.\r\nbq update --project_id=my-data-project --schema=schema.json my_dataset.my_table'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe3101679a0>)])]>

Alternatively you can also use SQL to bind data governance tags to BigQuery table columns.

code_block: <ListValue: [StructValue([('code', "CREATE OR REPLACE TABLE my_dataset.my_table(\r\n user_email STRING\r\n OPTIONS (\r\n data_governance_tags = [('my-governance-project/data_class', 'email')]),\r\n );\r\nALTER TABLE my_dataset.my_table\r\nALTER COLUMN phone_number\r\n SET OPTIONS (\r\n data_governance_tags = [('my-governance-project/data_class', 'private')]);\r\nALTER TABLE my_dataset.my_table\r\nADD COLUMN government_id\r\n STRING\r\n OPTIONS (\r\n data_governance_tags = [('my-governance-project/data_class', 'pii')]);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe310167f40>)])]>

You can also remove a column tag by setting it to [], for example:

code_block: <ListValue: [StructValue([('code', 'ALTER TABLE my_dataset.my_table\r\nALTER COLUMN phone_number\r\n SET OPTIONS (\r\n data_governance_tags = []\r\n);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe310167700>)])]>

You can use information_schema COLUMNS view to see the columns tags:

code_block: <ListValue: [StructValue([('code', "SELECT\r\n column_name,\r\n data_governance_tags[SAFE_OFFSET(0)].key AS tag_key,\r\n data_governance_tags[SAFE_OFFSET(0)].value AS tag_value,\r\nFROM `my_project.my_dataset.INFORMATION_SCHEMA.COLUMNS`\r\nWHERE table_name = 'my_table'"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe310167520>)])]>

The result is similar to the following:

code_block: <ListValue: [StructValue([('code', '+---------------+----------------------------------+-----------+\r\n| column_name | tag_key | tag_value |\r\n+---------------+----------------------------------+-----------+\r\n| user_email | my-governance-project/data_class | email |\r\n| phone_number | my-governance-project/data_class | private |\r\n| government_id | NULL | NULL |\r\n+---------------+----------------------------------+-----------+'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe3101679d0>)])]>

Step 3: Create data policies

Finally, define a BigQuery data policy to govern access to these tagged columns. These policies explicitly reference the tag values you attached previously. Note that, while data governance tags are global, data policies are regional.

To protect your data, the policy must be created in the same region where your BigQuery table is located. Once the policy is defined, access is only granted to the specified grantees; all others will be denied access to the sensitive column data.

Also, keep in mind that security in BigQuery is layered. For a data policy to be effective, the users (grantees) must first possess base-level access to the table itself (typically via a role like roles/bigquery.dataViewer). Data policy then acts as a second security layer, determining whether they view the raw, sensitive column data or a masked, obfuscated version.

Masking policy for ‘pii’ tagged column-data (SHA256 Masking):

code_block: <ListValue: [StructValue([('code', 'curl --request POST "https://bigquerydatapolicy.googleapis.com/v2/projects/myProject/locations/us-east1/dataPolicies" \\\r\n --header "Authorization: Bearer $(gcloud auth print-access-token)" \\\r\n --header \'Accept: application/json\' \\\r\n --header \'Content-Type: application/json\' \\\r\n --data \'{\r\n "dataPolicy": {\r\n "dataPolicyType": "DATA_MASKING_POLICY",\r\n "dataMaskingPolicy": { "predefinedExpression": "SHA256" },\r\n "grantees": [ "principalSet://goog/group/grp-sales@corp.com" ],\r\n "dataGovernanceTag": { "key": "myProject/data_class", "value": "pii" }\r\n },\r\n "dataPolicyId": "masking_policy_for_data_class_pii"\r\n}\' \\\r\n --compressed'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe31035c460>)])]>

Raw access policy for ‘pii’ tagged column-data

code_block: <ListValue: [StructValue([('code', 'curl --request POST "https://bigquerydatapolicy.googleapis.com/v2/projects/myProject/locations/us-east1/dataPolicies" \\\r\n --header "Authorization: Bearer $(gcloud auth print-access-token)" \\\r\n --header \'Accept: application/json\' \\\r\n --header \'Content-Type: application/json\' \\\r\n --data \'{\r\n "dataPolicy": {\r\n "dataPolicyType": "RAW_DATA_ACCESS_POLICY",\r\n "grantees": [ "principal://goog/subject/abc@xyz.com" ],\r\n "dataGovernanceTag": { "key": "myProject/data_class", "value": "pii" }\r\n },\r\n "dataPolicyId": "raw_access_policy_data_class_pii"\r\n}\' \\\r\n --compressed'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe310a68d90>)])]>

Masking policy for “private” tagged column data (NULL Masking):

code_block: <ListValue: [StructValue([('code', 'curl --request POST "https://bigquerydatapolicy.googleapis.com/v2/projects/myProject/locations/us-east1/dataPolicies" \\\r\n --header "Authorization: Bearer $(gcloud auth print-access-token)" \\\r\n --header \'Accept: application/json\' \\\r\n --header \'Content-Type: application/json\' \\\r\n --data \'{\r\n "dataPolicy": {\r\n "dataPolicyType": "DATA_MASKING_POLICY",\r\n "dataMaskingPolicy": { "predefinedExpression": "ALWAYS_NULL" },\r\n "grantees": [ "principal://goog/subject/abc@xyz.com" ],\r\n "dataGovernanceTag": { "key": "myProject/data_class", "value": "private" }\r\n },\r\n "dataPolicyId": "null_policy_data_class_private"\r\n}\' \\\r\n --compressed'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe310a68e80>)])]>

With these three steps, your column data is now protected. The next time a principal queries your BigQuery table, our authorization engine automatically evaluates their identity against your data policies. If the principal is part of the policy, they get to see the masked or raw data as per the policy; if they are not, then they will be denied access.

Get started today

Data governance tags are a powerful new tool to enhance your data security and governance strategy in BigQuery. We are continuously working to enhance data governance capabilities in BigQuery. Future updates include support for using SQL to create tags and tag based policies, ability to attach multiple tags to a single column, ability to define policies based on combinations of tags, and deeper integrations with services like Knowledge Catalog.

You can start tagging your columns and defining fine-grained access controls at scale. To learn more, dive into the Data Governance Tags documentation.

Bridging the gap between SQL and Python with BigQuery and the %%bqsql magic

Thu, 16 Jul 2026 16:00:00 +0000

Data scientists and data engineers often find themselves caught between two worlds: SQL and Python. Some find SQL more intuitive, especially when combined with a powerful engine like BigQuery to process data at scale. Others find it easier to work in Python with its rich ecosystem of libraries and runtimes. Historically, using these languages together in one notebook required moving data from SQL results to in-memory and writing from Python memory to temporary tables for SQL to access.

To solve this friction, the Google Cloud team introduced SQL cells in Colab Enterprise. Now, we are expanding that seamless experience to the broader open-source ecosystem. With the %%bqsql IPython cell magic, you can now effortlessly chain data processing workloads across SQL and Python code cells.

Thanks to open-source packages like Jupyter, pandas, BigFrames, and the BigQuery sandbox, you can follow all steps in this guide for free* and without a credit card.

*See the BigQuery sandbox documentation for limitations.

Setting up your environment

To get started,

1. Enable the BigQuery sandbox. Make note of your Google Cloud project ID.

2. Set up a local Python development environment, or alternatively, open this notebook in Colab, which has a Python environment already installed.

To set up a local python environment, see the steps on Google Cloud Documentation. Continue with the following steps, if you choose to set up a local python environment, else jump to the next section.

3. Activate the venv you created in the previous step to isolate Python dependencies.

On Linux or macOS, use these commands (update to your preferred Python version):

code_block: <ListValue: [StructValue([('code', '. ./env/bin/activate'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe311402d00>)])]>

4. Install the Jupyter, bigframes, and python-calamine packages.

code_block: <ListValue: [StructValue([('code', 'pip install --upgrade jupyterlab bigframes python-calamine'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe311402ca0>)])]>

5. Start Jupyter Lab.

code_block: <ListValue: [StructValue([('code', 'jupyter lab'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe3114024f0>)])]>

6. Open a web browser to the URL listed in the output. It will be something like http://localhost:8888/lab?token=somesupersecretvaluehere .

7. Create a new notebook using the Jupyter Lab UI (File > New > Notebook). Alternatively, download the notebook associated with this tutorial from the BigQuery DataFrames GitHub repository and open it.

Accessing and preparing local data

In this tutorial, you'll analyze the USDA wheat data. Pandas will download the data, mimicking a typical local data analysis workflow.

code_block: <ListValue: [StructValue([('code', 'url = "https://www.ers.usda.gov/media/5706/wheat-data-all-years.xlsx?v=52690"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe311402670>)])]>

Next, read the data into a local pandas DataFrame. Use the pyarrow dtype_backend when preparing local pandas data for SQL processing. This ensures more consistent handling of NULL values and seamless schema mapping when you hand off the data to the BigQuery SQL engine. For this example, read the 'Table05' sheet, which contains annual wheat supply and disappearance data:

code_block: <ListValue: [StructValue([('code', 'import pandas as pd\r\n\r\ndf = pd.read_excel(\r\n url,\r\n sheet_name="Table05",\r\n dtype_backend="pyarrow",\r\n engine="calamine",\r\n header=1, # Skip the first row.\r\n)\r\ndf'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe311402dc0>)])]>

Before querying the local DataFrame with SQL, ensure that the column names are SQL-friendly. BigQuery supports flexible column names, allowing most unicode characters, but special characters like "/" and "" must be removed or replaced.

code_block: <ListValue: [StructValue([('code', 'df.columns = [name.replace("/", "") for name in df.columns]\r\ndf'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe3114022b0>)])]>

Perform a basic filter using standard Python/pandas syntax to remove rows with missing data. This represents the initial Python-only stage of a processing chain.

code_block: <ListValue: [StructValue([('code', "full_rows = df[~df['Beginning stocks'].isna()]\r\nfull_rows"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe311402190>)])]>

Initializing the BigQuery SQL magic

The BigQuery DataFrames library provides the %%bqsql magic, which acts as the bridge between your Python and SQL environments. It allows the BigQuery query engine to directly reference and query your local pandas DataFrames (by implicitly uploading them as temporary tables) as well as actual BigQuery tables and external tables in GCS (Parquet, Iceberg, CSV).

To enable this integration in your notebook, load the bigframes extension.

code_block: <ListValue: [StructValue([('code', '%load_ext bigframes'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe310a2a130>)])]>

Note: The extension is pre-loaded in BigQuery Studio and Colab environments.

To ensure the correct Google Cloud project is billed for query usage, including free tier usage, configure the project ID used by the magics. Even in the free sandbox tier, a project ID is required to allocate query resources. If you don't set it explicitly, BigFrames will try to discover it from your environment (e.g., your Application Default Credentials).

code_block: <ListValue: [StructValue([('code', 'import bigframes.pandas as bpd\r\n\r\nbpd.options.bigquery.project = "your-project-id-here"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe310a2a430>)])]>

Querying local pandas DataFrames with SQL

With the project configured, you can now run SQL queries directly against your local pandas DataFrame (full_rows) as if it were a table in BigQuery. Simply reference the variable name inside braces {full_rows} in your SQL query. You may be prompted for an authorization code, which you'll obtain by following the link provided as part of the same message.

code_block: <ListValue: [StructValue([('code', '%%bqsql\r\nSELECT * FROM {full_rows}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe3108363d0>)])]>

Chaining SQL and Python: Saving SQL Results

The true power of the %%bqsql magic lies in chaining. By providing a destination variable name as an argument to %%bqsql (e.g., %%bqsql destination_var), the query result is saved as a BigQuery DataFrame to that variable.

This DataFrame lives on the BigQuery engine but behaves like a pandas DataFrame in Python. You can immediately use it in subsequent Python cells, or reference it again in another SQL cell. This allows you to build a multi-step, hybrid processing pipeline.

Filter the data to only yearly entries using SQL, and save the result into a new BigFrames DataFrame named yearly:

code_block: <ListValue: [StructValue([('code', "%%bqsql yearly\r\nSELECT *\r\nFROM {full_rows}\r\nWHERE STARTS_WITH(`Time period`, 'MY')"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe310836100>)])]>

Now, you can chain another SQL operation. Reference the yearly BigFrames DataFrame that you just created, extract the year using SQL regular expressions, cast it to a timestamp, and save the results into a new BigFrames DataFrame named timeseries.

code_block: <ListValue: [StructValue([('code', "%%bqsql timeseries\r\nSELECT\r\n * EXCEPT (`Marketing year 1`),\r\n TIMESTAMP(CONCAT(\r\n REGEXP_EXTRACT(`Marketing year 1`, r'([0-9]+)\\/'),\r\n '-01-01')) AS `year`\r\nFROM {yearly}"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe310836d00>)])]>

Notice how you are building a chain from Python to SQL and back again.

Returning to Python for visualization

Now that you've completed some SQL transformations, you can chain back to Python for visualization. Because BigFrames DataFrames implement the pandas API, you can call standard visualization methods (like .plot.line()) directly on the timeseries DataFrame without downloading the full dataset first. The computations happen in BigQuery, and only the summarized chart data is sent back to the notebook.

code_block: <ListValue: [StructValue([('code', "timeseries.set_index('year').sort_index().plot.line()"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe3108365b0>)])]>

Alternatively, download the time series as a pandas DataFrame to use with your visualization library of choice.

code_block: <ListValue: [StructValue([('code', "pddf = timeseries.set_index('year').sort_index().to_pandas()"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe310836880>)])]>

Why a hybrid pipeline matters

By pairing BigQuery DataFrames with %%bqsql magics, you have built a powerful, interoperable pipeline that seamlessly transitions between SQL and Python.

local pandas df and full_rows DataFrames
to SQL filter
to BigFrames yearly DataFrame
to SQL transform
to BigFrames timeseries DataFrame
to Python data visualization
to local pandas DataFrame.

This architecture offers key advantages:

Optimal tool selection: Use SQL for what it does best (heavy aggregations, window functions, and complex joins) and Python for what it does best (visualization, statistical modeling, and ML orchestration).
Improved code readability: Instead of writing massive SQL queries with dozens of common table expressions (CTEs), or doing complex aggregations using pandas APIs which are often convoluted compared to SQL, you can split your pipeline into logical steps, alternating between SQL and Python.
Seamless scaling: The exact same %%bqsql code can scale from a tiny local pandas DataFrame to billions of rows in a production BigQuery table. You only need to swap the initial local pandas DataFrame with a BigQuery DataFrame reference.

Next steps and scaling up

Check out the other notebooks in the BigFrames API reference site. In addition to the %%bqsql cell magic, BigFrames also registers a BigQuery Accessor on standard pandas DataFrames, allowing you to run SQL scalar functions directly on local pandas data.

For example, you can call powerful Google Cloud community UDFs from BigQuery Utils, BigFunctions, or CARTO Analytics Toolbox for BigQuery using df.bigquery.sql_scalar(...):

code_block: <ListValue: [StructValue([('code', 'import pandas as pd\r\nimport bigframes.pandas as bpd # Registers the accessor\r\n\r\nbpd.options.bigquery.project = "your-project-id"\r\ndf = pd.DataFrame({"x": [1, 2, 3]})\r\npandas_s = df.bigquery.sql_scalar("`bqutil`.fn.cw_setbit({x}, 2)")'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe310836c40>)])]>

While the BigQuery sandbox offers a powerful environment to test these hybrid Python-SQL workflows for free, some advanced features like BigQuery Machine Learning (BQML) are restricted. By connecting a billing account to your Google Cloud project, you can unlock advanced capabilities such as the bigframes.bigquery.ai.forecast function to predict time-series data using Google's state-of-the-art foundational models directly from your SQL/Python chain.

code_block: <ListValue: [StructValue([('code', 'forecasted_pandas_df = (\r\n pddf\r\n .reset_index(drop=False)\r\n .bigquery.ai.forecast(\r\n data_col="Production",\r\n timestamp_col="year",\r\n horizon=10,\r\n )\r\n)\r\n\r\n# Plot the results\r\nforecasted_pandas_df_sorted = forecasted_pandas_df.sort_values(by=\'forecast_timestamp\')\r\nplt.plot(pddf.index, pddf[\'Production\'], label=\'Real Production\', color=\'blue\')\r\nplt.plot(forecasted_pandas_df_sorted[\'forecast_timestamp\'], forecasted_pandas_df_sorted[\'forecast_value\'], label=\'Forecasted Production\', color=\'red\', linestyle=\'--\')\r\nplt.fill_between(\r\n forecasted_pandas_df_sorted[\'forecast_timestamp\'],\r\n forecasted_pandas_df_sorted[\'prediction_interval_lower_bound\'],\r\n forecasted_pandas_df_sorted[\'prediction_interval_upper_bound\'],\r\n color=\'red\',\r\n alpha=0.2,\r\n label=\'Confidence Interval\'\r\n)\r\n# ...\r\nplt.show()'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe310836eb0>)])]>

The BigFrames team would love to hear your feedback on the hybrid Python-SQL experience:

Email: bigframes-feedback@google.com
Issues: File bug reports or feature requests on the open-source BigFrames repository.
Updates: To receive news and updates, subscribe to the BigFrames email list.
Learn more: Read the BigFrames API reference and user guides in the documentation.

How to Analyze and Govern Gemini Enterprise App Usage at Scale with BigQuery

Wed, 15 Jul 2026 16:00:00 +0000

Deploying the Gemini Enterprise app across an organization marks a transformative leap forward in workforce productivity, providing employees with an amazing, high-performance suite of agentic AI tools, search-grounded assistants, and specialized solutions like NotebookLM. As adoption grows to a large scale, it can introduce a critical administrative scale challenge: how to audit, govern, and extract insights from a massive volume of telemetry without getting bogged down in manual overhead. To help administrators succeed, Google Cloud provides comprehensive, out-of-the-box analytics via pre-computed dashboards to track day-to-day adoption, user engagement, and active user metrics. While this provides a product-centric lens to look at Gemini Enterprise app's usage, to understand the impact of agentic AI, administrators might need a more nuanced, organization-centric perspective tailored to their own internal context. This is where using Google BigQuery becomes a crucial tool in the administrator's arsenal to run deep-dive forensics across their organization to analyze and govern the adoption of agentic AI.

Why Gemini Enterprise app + BigQuery is a game-changer

Augmenting the Gemini Enterprise app with BigQuery through log sinks allows a lean administrative team to analyze and govern a large-scale deployment. Specifically, it empowers IT, Data, and Security teams to:

Profile nuanced adoption and behaviors: Segment usage patterns by department to see which teams are building custom agents, track NotebookLM utilization, and calculate agent-to-employee ratios.
Quantify organizational value: Combine conversational logs with HR or line-of-business datasets to calculate actual employee hours saved, trace value creation, and build executive Looker dashboards.
Execute precision compliance audits: Audit grounding queries across Google Drive folders and enterprise directories to prevent data leaks and protect corporate IP.
Investigate safety alerts instantly: Query historical logs when security filters flag a prompt, identifying the exact text that triggered a Model Armor block to resolve compliance alerts.

To support these use cases, the telemetry is partitioned into five distinct log tables in BigQuery, capturing unique data fields:

BigQuery Destination Table	Telemetry Captured
Gen AI User Messages `discoveryengine_googleapis_com_g en_ai_user_message`	Verbatim prompt inputs typed by users
Gen AI Choices `discoveryengine_googleapis_com_g en_ai_choice`	Verbatim model responses, finish reasons, and LLM reasoning steps
User Activity Telemetry `discoveryengine_googleapis_com_g emini_enterprise_user_activity`	Corporate identity (IAM emails) and grounding file access paths
Cloud Audit Activity `cloudaudit_googleapis_com_activity`	Control plane configuration changes and administrative user logs
Cloud Audit Data Access `cloudaudit_googleapis_com_data_ac cess`	High-volume data plane interactions and search queries
Aggregate OOB Metrics (Batch Export Table)	Pre-aggregated seats claimed, seat purchases, and engagement metrics from the past 30 days. To be pulled asynchronously via custom daily batch runs of the `analytics:exportMetrics` API to build high-level adoption and cost dashboards.

Ingestion pipeline and architecture

To implement scale-ready observability, administrators establish an automated telemetry pipeline. Moving your Gemini Enterprise data to BigQuery does not require complex custom software development; instead, it leverages a continuous Cloud Logging Log Router Sink for conversational logs and an asynchronous batch export API for high-level aggregate seat metrics.

The diagram below illustrates the ingestion pipeline and how telemetry is mapped to BigQuery:

Here is your blueprint for connecting Gemini Enterprise to BigQuery to build the ultimate analytics and governance foundation for your organization.

Routing pipelines: Continuous logging and audit sinks

To capture your telemetry, establish log sinks within Cloud Logging to intercept and route runtime events to BigQuery:

The streaming pipeline (detailed logs): Streams row-by-row conversational data (user prompts, model choices, and grounding events). Ensure prompt and response logging is enabled in your Gemini Enterprise Admin Console (see Set Up Usage & Audit Logs).

- Inclusion Filter (replace [PROJECT_ID] with your Google Cloud Project ID):

code_block: <ListValue: [StructValue([('code', 'logName="projects/[PROJECT_ID]/logs/discoveryengine.googleapis.com%2Fgemini_enterprise_user_activity" OR\r\nlogName="projects/[PROJECT_ID]/logs/discoveryengine.googleapis.com%2Fgen_ai.user.message" OR\r\nlogName="projects/[PROJECT_ID]/logs/discoveryengine.googleapis.com%2Fgen_ai.choice"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe303347580>)])]>

The governance pipeline (audit logs): Captures administrative actions (Admin Activity) and data plane operations (Data Access, such as grounding data connector lookups).

- Inclusion Filter (replace [PROJECT_ID] with your Google Cloud Project ID):

code_block: <ListValue: [StructValue([('code', 'logName:"projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com" AND \r\nprotoPayload.serviceName="discoveryengine.googleapis.com"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe303347160>)])]>

- Admin Activity Logs: Always enabled by default; tracks resource changes (e.g., custom agent creation, updates, deletions).
- Data Access Logs: Off by default; must be enabled in GCP IAM settings for the Discovery Engine API to log user-level data read/write interactions during chats.

Unlock advanced intelligence in BigQuery

Transform raw telemetry into insights

BigQuery provides AI-powered analysis tools that make understanding and navigating telemetry effortless. By leveraging Gemini in BigQuery, administrators can translate raw log streams into visual insights and clear documentation without manual guesswork.

No-Code Conversational Analytics (BigQuery CA)
Querying nested JSON schemas is made simple with Conversational Analytics in BigQuery (BQ CA). BQ CA acts as an intelligent agent within BigQuery Studio, automatically generating and executing SQL grounded in your schema, business metadata, and verified queries/UDFs to ensure metrics consistency. It also surfaces its "thinking process" alongside the generated code to build administrative trust.

For example, as shown in the screenshot below, asking "Compare the usage of notebooklm, deep research and custom agents using oob_metrics?" generates the correct SQL, runs the query and outputs the result in seconds:

As shown in the screenshot below, BQ CA goes beyond traditional querying and standard SQL generation by allowing users to execute sophisticated AI and machine learning tasks directly within the console. Administrators can leverage these native capabilities to run advanced analysis, such as classification of user prompt sentiment or forecasting future adoption trends, streamlining the governance process.

Auto-generated schema documentation and insights

Understanding telemetry fields like useriamprincipal, finish_reason, or groundedContent is crucial for extracting the right insights. BigQuery simplifies this through automated schema documentation and AI-powered context:

Automated profiling and metadata: By pairing Knowledge Catalog Data Profiling with Gemini, you can evaluate unique value counts, null rates, and data distributions in raw tables. With a single click, Data Insights generates descriptive metadata for both tables and individual nested columns.
Unified data insights: Gemini leverages this rich context to surface insights across your entire data estate. It automatically recommends queries to find anomalies or safety failures within a single table (like gen_ai_user_message). At the dataset level (Preview), it generates an interactive relationship graph to map cross-table join paths and suggests queries that combine data—like user activity and model outputs—to calculate task complexity.
Seamless agent integration and glossaries: Table and data insights integrate directly into the BigQuery Conversational Analytics (BQ CA) agent UI, giving agents immediate access to enriched metadata and few-shot examples. To ensure agents accurately interpret domain-specific prompts, BQ CA also supports business glossaries. You can define custom terms directly for your agents or import existing glossaries from Knowledge Catalog to establish a standardized vocabulary.

Administrators can then leverage the profiling, enriched metadata and insights to navigate logged fields, understand the telemetry structure, and catalog data for compliance audits. As shown in the screenshots below, the output of Gemini-powered auto generation of schemas, descriptions and linkages makes it easy to make sense of the complex relationships and telemetry data output by agentic interactions on the Gemini Enterprise app.

Visualizing with Data Studio dashboards

For executive stakeholders, raw log tables can be transformed into interactive, high-impact business intelligence dashboards. By connecting Data Studio directly to BigQuery, you can build dashboards that monitor:

User adoption and seat ROI: Segment usage trends by department, highlighting the ratio of custom agents built relative to employee headcount.
Data grounding traffic: Map which enterprise connectors—such as SharePoint, Google Drive, or Gmail—experience the highest utilization.
Content safety and violations: Track Model Armor sanitization blocks and sentiment feedback loops over time to maintain safety standards.
Share BQ Conversational Analytics agent: Share the BQ CA agents you built via Data Studio to give business users the ability to ask more questions of the data.

Empower your organization with the Gemini Enterprise app and your administrators with BigQuery

Deploy the Gemini Enterprise App: Bring the best of Google AI to every employee.
Enable Prompt & Response Logging: Turn on prompt and response logging in the Admin Console to begin recording user activity telemetry.
Configure Log Router Sinks: Establish sinks to stream telemetry into BigQuery.
Track Metrics & Export Analytics: Access pre-computed, out-of-the-box dashboards on the console and export historical aggregate statistics.
Extract Table-Level & Dataset-Level Insights: Explore unfamiliar log tables and discover relationship join paths automatically.
Query with Conversational Analytics: Build data reasoning agents and leverage natural language querying inside BigQuery Studio.
Visualize with Data Studio: Connect Data Studio to BigQuery to build executive-level dashboards & give access to BQCA agents to business users.
Consult a Google Cloud Customer Engineer for the most cost-effective and secure configuration for the analytics setup described above.

The authors would like to acknowledge and thank the Google Forge team, especially Vicky Falconer, Dharini Chandrashekhar and Adhaar Gupta, for contributing to the core work that led to this article.

Frontier and Center: Who evaluates the evaluations?

Fri, 10 Jul 2026 16:00:00 +0000

Editor’s note: Some of the most interesting questions in AI are being asked by information theoreticians, around how to provide context to an emerging class of AI agents. A few weeks ago, we waded into those waters with a blog about the Open Knowledge Format, a specification that formalizes the LLM-wiki pattern into a portable, interoperable format to represent the metadata, context, and curated knowledge that modern AI systems need to operate. That blog generated a ton of interest, so we’ve decided to bring you more of the same, as part of our new “Frontier and Center” series. Today, we hear from two members of Google Data Cloud’s frontier AI team on the recurring challenge of how to systematically evaluate whether or not an agent is able to answer questions effectively based on its context. Read on for more, and watch this space for more blogs from this team.

A passing grade is the least interesting thing an exam can tell you. It says the student cleared the bar; leaving you entirely in the dark about how narrow their failures were, how effortless their passes were, or what to teach next. Yet this is exactly how we evaluate AI agents. We run a fixed benchmark, calculate a score, and declare progress. In doing so, we are handing our agents a pass/fail exam when what we actually need is a map of the agent’s capabilities: a picture of the terrain that shows exactly where capability falls off, and by how much.

For data agents, this map matters a lot for data discovery in search and retrieval — the unglamorous first step where an agent, handed a vague human question and a warehouse or data lake of thousands of tables and files, has to find the right datasets before it can reason over anything. Discovery is a "needle in a haystack" problem. Real users phrase their questions imperfectly, and inferring what datasets to retrieve presents a real challenge to agents. So the interesting question in evaluations is never "can the agent pass?" It is "how vague can the question get before the agent breaks?" An exam cannot easily answer that, but a map can.

Today, we share an approach rooted in information theory that we’ve been leveraging to add detail and nuance, i.e., fidelity, to benchmarks, so we can better understand agents’ performance as a part of their evaluations. Along the way, the added fidelity exposed some deeper issues with the quality of emergent evaluation cases themselves.

Difficulty, measured

When it comes to retrieval, evaluation cases are often stratified into tiers of difficulty. This can happen organically, e.g., pervasive and enduring failure scenarios are deemed difficult. Or it can be from labels applied by humans or machines categorizing some questions as "easy" or "hard" for an agent to answer correctly, e.g., based on the context provided in the query. While this kind of sentiment-based labeling is not the only way to label test cases, it’s frequently used despite its imperfections, such as being challenging to reproduce.

Despite being an industry staple, the approach of assessing every evaluation case by hand is unrealistic at scale. What we need is a rigorous approach that can modulate the difficulty of evaluation cases. We’re iterating on a meta-benchmark we call Discovery Bench: a framework that modulates an evaluation case by generating “easy” and “hard” variations of every case. This allows us to audit how close or how far an agent is from succeeding in those cases.

The lever for modulating the difficulty of an input query comes via a tried-and-trusted concept that’s present across information theory and machine learning: surprisal, or the likelihood of an output given a set of inputs. In our case, a query’s surprisal represents the uncertainty that remains about the correct dataset given the query.

The thinking behind our approach is simple: A term or a phrase in an evaluation query has high informative power when it sharply distinguishes the target from everything else in the corpus. Therefore, we can adjust the difficulty of evaluation cases by adding or removing terms with varying levels of informative power.

Let’s work through a real example from KramaBench, a publicly available benchmark. One of KramaBench’s datasets has information about orbiting satellites, and the example query from the suite includes the following text: "…the total count of satellite major altitude changes for satellite 48445 during 2024 using TLE history."

The token "TLE" is sharply distinguishing; it points almost uniquely at the TLE_____48445 table from the dataset. Strip it, and the query degrades to "the count of satellite altitudes for satellite 48445," whose vague phrasing now matches density tables, precise-orbit files, and decay logs alike. Surprisal makes this quantitative: rare, pointed terms carry more bits than common ones.

The remaining surprisal of a query is how much uncertainty is left about its answer. As surprisal approaches zero, the query has become specific enough to pinpoint exactly one dataset.

The heart of the idea behind Discovery Bench is this refinement loop, which we call iterative surprisal-based query refinement, or iSQR, which generates cases with higher or lower informative power to test where an agent can start successfully answering the query:

Figure 2: The iSQR refinement loop.

The crux is being able to control the challenge embedded into the evaluation case by making adjustments: Instead of one fixed phrasing per question, we generate the same question at three levels of calibrated ambiguity [high, medium, low], with each grounded in bits (not subjective opinion). We can even justify, term by term, why a word was added or removed. Difficulty stops being a property that is attributed by sentiment or classification, and becomes one we engineer.

The cliff you couldn't see

Here is what Discovery Bench’s difficulty dial reveals — and what a single-phrasing benchmark structurally cannot.

We have an F1 agent that's built for recall (on Gemini 3.1 Pro). Running it against KramaBench and across the full sweep of ambiguity levels traces a curve: 0.34 at high ambiguity, 0.76 at neutral, 0.81 at medium, 0.78 at low.

Figure 3: F1 swept across ambiguity — the dot versus the curve.

Two findings fall out immediately (and neither were visible to a conventional eval).

First, the cliffs. This query scores a perfect F1 = 1.00 at neutral phrasing — and 0.00 at high ambiguity. It is the satellite-48445 case from above: drop the distinguishing token "TLE" and the agent loses the table entirely. Same query, same agent, same ground truth; one notch vaguer and it falls off a cliff. A static benchmark tests the neutral phrasing, stamps "solved," and reports flat ground where there is a precipice. Pass/fail was particularly misleading in that it did not just miss the cliff, but it told us the terrain was level.

Second, the sweet spot. For Discovery Agent, medium ambiguity beat neutral, and low ambiguity sometimes underperformed it. More specificity is not monotonically better for the system being evaluated; there is an optimal amount of steering. That is a graded, actionable signal. This is the "how close, how hard" texture we were missing from a scalar. It tells you where to hill-climb, or improve, the agent: in our case, straight at concrete failure modes like time-sharded tables (precision collapsing to ~8% as the agent over-retrieves 21 near-identical shards for a two-table answer) and context blow-up (F1 dropping from 0.75 to 0.32 once a query triggers long search chains). The map did not just say that the agent failed, but it said where, and why. Note that our hypothesis that less ambiguity and more context (via steering terms) should improve retrieval generally holds true, but for the specific Discovery Agent being exercised, the idiosyncratic “sweet spot” meaningfully highlighted trade-offs in its implementation.

We're not alone

The field is converging on meta-benchmarking and exerting greater control of how we challenge and evaluate our agents. A growing body of work uses item response theory, the latent-ability model behind standardized testing, to treat difficulty as a measured quantity rather than a label: tinyBenchmarks and metabench show that a handful of informative items reproduce a model's full score, and PSN-IRT turns the same lens on benchmark quality itself. Others audit the ground truth directly: MMLU-Redux found that 6.49% of Massive Multitask Language Understanding (MMLU) questions are mislabeled, and Platinum Benchmarks re-cleaned ten datasets to minimize both label errors and ambiguity — the same two axes we sweep for. And ambiguity is increasingly treated as intrinsic rather than noise: AmbigQA showed that a large fraction of real questions admit multiple readings, and later work finds that apparent hallucinations often stem from query ambiguity rather than model failure. What we have not seen elsewhere is the combination: information-theoretic ambiguity sweeping applied as a meta-benchmark over live enterprise data.

A benchmark we trusted turned out to be broken

We built our first evaluation on kramabench-astronomy, a benchmark established in the field, and one which other teams had already leaned on for their own evals. Teams derived benchmarks from this dataset, and we hypothesized subtle issues may have been introduced over time. When we actually read the benchmarks used by teams, with Gemini's help, we found it was wrong in meaningful ways: ground-truth tables that did not answer their query, a question whose 124 sharded tables exceeded what some teams’ retrieval APIs could even return, months specified where exact dates were required. Quietly broken ground truth means quietly wrong conclusions not just for us, but for every prior analysis built on it.

This is the generalized crux of the matter: an evaluation is itself an artifact that can be defective, and almost nobody evaluates it. We instrument the agent and trust the ruler, but where do we validate that the measuring stick makes sense?

When two maps disagree

Now the recursive turn: If difficulty is something we generate, then we need to evaluate the generator itself; we should not trust it blindly either.

So we built the same ambiguity sweep two ways: steering terms from a pure-LLM guess, versus terms grounded in TF-IDF surprisal. The two disagreed violently. At high ambiguity, the LLM-built sweep scored the agent at F1 ≈ 0.34; the grounded sweep, ≈ 0.85. One of these maps is badly distorted. The grounded one, predictably, is the more robust: surprisal gives it a footing the free-running LLM lacks.

This is "evaluate your evals," made concrete. The information-theoretic lens does not only grade the agent along a continuous axis; it grades the benchmark's own construction, and adjudicates between the two.

Evaluate your evals

We have spent years optimizing agents against rulers we never measured. The bitter irony is that better models make this worse: as agents clear coarse benchmarks, the score saturates near the top and the exam loses its ability to highlight where the agent can be improved.

So the call to action is uncomfortable and overdue: evaluate your evals. Read your ground truth. Treat difficulty as a measured quantity, not a label: sweep it, plot it, find the bit-width where your system breaks. Ask not just "did it pass?" but "how close was the miss, how hard was the pass, and would a slightly vaguer question have sent it off a cliff?" Build evaluations that produce signals; not just verdicts.

There is a genuine tension to sit with here. Difficulty-as-entropy is only as reliable as the model that estimates the entropy. There's a risk that if we push too hard on a measurable proxy, we optimize the ruler instead of the agent. That is not a reason to retreat to pass/fail; it is a reason to keep the evaluator under the same scrutiny as what it is evaluating. The moment we stop asking who evaluates the evaluators is the moment our maps stop being useful again.

^{1. Maia Polo, F. et al. tinyBenchmarks: Evaluating LLMs with Fewer Examples. ICML 2024. arxiv.org/abs/2402.14992}^{2. Kipnis, A. et al. metabench: A Sparse Benchmark of Reasoning and Knowledge in Large Language Models. ICLR 2025. arxiv.org/abs/2407.12844}^{3. Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory (PSN-IRT). AAAI 2026. arxiv.org/abs/2505.15055}^{4. Gema, A. P. et al. Are We Done with MMLU? (MMLU-Redux). 2024. arxiv.org/abs/2406.04127}^{5. Vendrow, J. et al. Do Large Language Model Benchmarks Test Reliability? (Platinum Benchmarks). 2025. arxiv.org/abs/2502.03461}^{6. White, C., Dooley, S. et al. LiveBench: A Challenging, Contamination-Limited LLM Benchmark. 2024. arxiv.org/abs/2406.19314}^{7. Min, S. et al. AmbigQA: Answering Ambiguous Open-domain Questions. EMNLP 2020. aclanthology.org/2020.emnlp-main.466}^{8. Lai, E., Vitagliano, G. et al. KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes. 2025. arxiv.org/abs/2506.06541}

Shift into high gear with agents: Securing the software-defined vehicle

Mon, 06 Jul 2026 16:00:00 +0000

The automotive industry is at a pivotal crossroads as it hits the gas on adopting new technology. The era of the traditional connected vehicle has shifted into the age of the software-defined vehicle (SDV), notable for rapid innovation with many new capabilities delivered over the air.

By integrating AI and agents, the next generation of SDVs will be capable of turning raw telemetry into actionable insights in real-time, allowing for a fundamental rethink of how vehicles interact with their environment and their users. To better support and secure SDVs, Google Cloud and Valtech have partnered to develop Nexus SDV, a highly-scalable, AI-enabled connected vehicle platform built on Google Cloud. This modular, developer-friendly and open-source solution is designed to manage up to 100 million devices, and features deep integration with Android Automotive OS (AAOS) to streamline data flows and in-vehicle experiences.

We are proud to announce the first release of the Nexus SDV open-source core that showcases how it can reduce total cost of ownership through Arm-based compute and Bigtable, while providing a AI-native environment for building the next era of automotive intelligence.

AI-driven experiences with Nexus SDV

Nexus AI serves as the platform’s intelligent engine, transforming the vehicle from a passive data source into a proactive, agentic partner. Using Gemini models and Gemini Enterprise Agent Platform, Nexus AI can analyze complex telemetry in real-time to provide information for autonomous decision-making and hyper-personalized driver assistance, effectively acting as an intelligent agent that anticipates user needs.

Crucially, this advanced intelligence is paired with a focus on significant total cost of ownership (TCO) reduction. By using high-efficiency Arm-based compute and Bigtable-optimized data storage, the platform lowers the operational costs associated with processing massive data volumes. This modular, AI-native architecture ensures that manufacturers can scale their fleet intelligence rapidly without the prohibitive cloud and development expenses traditionally associated with next-generation vehicle software.

Cloud-native under the hood

The architecture of Nexus SDV is built on a modular, cloud-native foundation designed to bridge the gap between the vehicle edge and the data center. Deep compatibility with AAOS is the keystone of the close integration between the cloud and the vehicle, and will help ensure that high-fidelity telemetry is ingested and synchronized in real-time. This robust data loop allows Nexus AI to quickly push intelligent updates and services back to the vehicle.

By providing this developer-friendly, open framework, Nexus SDV enables manufacturers to manage the entire lifecycle of a SDV with the scalability and reliability of the Google Cloud ecosystem.

Architecture for Nexus SDV.

Defense in depth with Google Cloud Security controls

By building on Google's secure foundations, including secure-by-design and Zero Trust architecture, Nexus SDV supports the heavy lifting of compliance and threat protection. To achieve this, the Nexus SDV architecture implements a comprehensive, defense-in-depth security model across six key elements:

Mutual TLS (mTLS) and public key infrastructure (PKI)
Nexus SDV relies on cryptographic trust chains to authenticate vehicles before any data exchange can occur. The infrastructure uses Google Cloud Certificate Authority Service (CAS) to manage distinct CA pools (server, factory, and registration CAs), ensuring a highly available and secure root of trust.

Specifically, the registration server enforces registration by forcing clients to present a valid "factory-issued" certificate during the initial TLS handshake, extracting and parsing the certificate directly from the connection stream to definitively prove the vehicle's identity. During registration, the server performs Certificate Signing Request (CSR) validation sent by the vehicle before issuing a new operational certificate.

Identity and access management
The system uses identity brokering where Keycloak is deployed as the central OpenID Connect (OIDC) identity provider. Vehicles authenticate against Keycloak using their operational certificate via mTLS to receive a short-lived JSON Web Token (JWT).

For fine-grained access control, a custom NATS Auth Callout service provides dynamic subject permissions: It intercepts all messaging broker connection attempts, validates the Keycloak JWT using public JWK keys, and programmatically maps the vehicle's roles to specific NATS subjects.

For secure service-to-service communication, it uses Workload Identity Federation so pipelines exchange GitHub OIDC tokens for temporary Google Cloud access, removing static credentials, while GKE Workload Identity allows Kubernetes Pods to access backend services like Bigtable by binding Kubernetes service accounts to Google service accounts.

Security is reinforced through restricted IAM scopes, ensuring dedicated service accounts are provisioned with minimal permissions, such as the data API being restricted only to reading from Bigtable. Using VPC-SC, Organization policy constraints and Private Service Connect (PSC) in your deployment context also helps you achieve secure foundations.

Secret management
Nexus SDV relies on centralized secret management to protect sensitive information. All sensitive configurations, database passwords, and cryptographic signing keys are generated dynamically during Terraform infrastructure provisioning and locked inside Google Cloud Secret Manager.

A secret fetching during deployment is used to avoid baking secrets into application code and container images. Instead, services pull signing keys and credentials directly into memory only at runtime, minimizing exposure both at rest and in transit.

Network isolation
To enforce network isolation, the underlying computer infrastructure is heavily shielded. Nexus SDV runs on private GKE clusters where worker nodes have no public IP addresses, preventing direct internet exposure. Additionally, the Keycloak PostgreSQL database uses Cloud SQL IAM Authentication, which allows the Cloud SQL Proxy to connect securely using IAM roles rather than relying on static database passwords or managing IP allowlists.

Secure AI Framework
Google Cloud secures these advanced AI capabilities through a comprehensive, enterprise-grade framework that prioritizes data privacy, model governance, and safe execution, based on guidance from the Secure AI Framework (SAIF). With Gemini Enterprise Agent Platform, security and governance are natively embedded into the machine-learning lifecycle through capabilities, such as dedicated Explainability and Safety controls, continuous Evaluation and Monitoring, and secure model registries.

You can learn more about how we secure AI here.

Data API
Instead of allowing downstream applications and external clients direct access to data stores like Bigtable, Nexus SDV routes data retrieval through a custom Data API. This microservice acts as a secure abstraction layer that translates strictly, such as querying specific vehicle IDs, sensor data types, and predefined time windows, into heavily constrained Bigtable row-range scans and column filters.

By doing so, it serves as a secure gateway that enforces structured data access patterns.

Start your journey with Nexus SDV

Nexus SDV represents a new era of automotive intelligence, delivering an agentic, secure, and cost-efficient platform that empowers manufacturers to harness the full power of AI in an open-source framework. You can learn more about how we are redefining the software-defined vehicle here.

Conversational analytics in BigQuery brings trusted agentic reasoning to everyone

Tue, 30 Jun 2026 18:00:00 +0000

Businesses run on fast decisions, but the teams who hold the answers are often buried under a backlog of routine requests, leaving users waiting in line for insights they need now. Today, we are bringing Conversational Analytics in BigQuery to general availability, so both business and technical teams can query data, run multi-step analyses, and generate visual reports using natural language, right where the data lives. With this release, Conversational Analytics in BigQuery now delivers an agent that behaves like an analyst who knows your business, thinks before it answers, and stands behind its work. Built on Google’s latest Gemini models and BigQuery’s secure, governed foundation, it brings that trusted analyst to everyone in your organization.

Fig 1. Conversational Analytics in BigQuery

Conversational analytics for enterprise data

BigQuery’s conversational capabilities are built-in and available for use instantly, with no setup required.

For deeper, more consistent insights, data professionals can author specialized agents grounded in the exact sources that matter, from projects, datasets, and tables to views, graphs, and user-defined functions. And because your data rarely lives in one place, Conversational Analytics reaches beyond native BigQuery tables to Lakehouse-managed Apache Iceberg tables and cross-cloud Lakehouse sources like Databricks Unity, AWS Glue, SAP and Salesforce, so you can break down data silos and analyze data across clouds from a single conversation.

As a data practitioner, you work with Conversational Analytics right inside BigQuery Studio and Data Canvas, and publish the agents you build to Gemini Enterprise, Data Studio, or your own application through the Conversational Analytics API, putting them in the hands of business users wherever they work.

“At MoneySuperMarket, BigQuery Conversational Analytics has changed how our teams get to insight. Analysis that used to take weeks can now be done in minutes, saving our financial analysts around half a day each week. By making analysis more self-serve, we’re helping teams create faster insight to support better product and commercial decision-making.” - Suzie Millar, Head of Data, Mony Group

Engineered trust and explainability

Accuracy in Conversational Analytics is by design, not aspirational: every agent is grounded in your business context, not a model's assumptions. That context comes from the Knowledge Catalog (glossaries, profile scans, and context bundles), BigQuery Graph for multi-hop queries, and your own verified queries and custom agent instructions. With the new Open Knowledge Format, the wiki your team already maintains can feed straight into Knowledge Catalog. At query time, Conversational Analytics leverages existing embeddings of your column values, generated by AI.GENERATE_EMBEDDINGS, to match your question to the right data, so asking about "Texas" finds rows stored as "TX."

Grounding only earns trust if the user can see it. So every answer is inspectable, providing:

Visible thinking steps: Review the agent's step-by-step reasoning and the exact SQL it generates before it returns an answer.
Context citations: See the precise sources behind every response, including tables, schema definitions, verified queries, and glossary terms used to calculate it.
Proactive disambiguation: When a prompt is vague, the agent asks targeted clarifying questions instead of guessing.
Long-term memory: The agent remembers what your terms and questions mean, so you don't have to disambiguate the same thing twice.

Fig 2. Generating answers that you can trust

Security and governance by design

One common barrier to scaling AI is governance. Reaching tens of thousands of users requires rigorous security, governance, and transparent cost controls. Conversational Analytics inherits BigQuery's governance model, so users only query data they are authorized to see and every query is logged for auditing within the BigQuery compliance framework. On top of that baseline, it supports Access Transparency (AxT), Customer-Managed Encryption Keys (CMEK), Private IP, and VPC Service Controls, and now guarantees data residency for data at rest and for ML processing within EU and US multi-region endpoints.

For your most engaged users, we also deliver the operational controls that scale demands: Configure Google Cloud-native cost controls so no user or project exceeds its allotment, cap an agent's maximum query size in bytes, and track usage through BigQuery labels on jobs.

Fig 3. Agent Observability and Monitoring

The power of BigQuery AI, in plain language

The agent doesn't just retrieve rows, but calls BigQuery's AI functions for you, turning advanced analysis into a question you can ask in plain language.

Find the "why," not just the "what": Ask what drove a change and the agent runs root-cause analysis with AI.KEY_DRIVERS, surfacing the exact segments behind the move.
See what's coming: Move past historical reporting by triggering AI.FORECAST and AI.DETECT_ANOMALIES right in the chat to project trends and flag outliers, with no model to build or manage.
Query your entire data estate: With object tables, the agent reasons over relational data and unstructured files together, PDFs, images, logs, and video, so a single conversation spans your whole estate.

Fig 4. Conversational Analytics leverages BigQuery AI functions

From answering questions to running the investigation

Conversational Analytics agents are moving from human-scale reactive analysis to agent-scale proactive action. You're no longer limited to asking a question and waiting for the answer.

Deep-dive mode: If you ask ‘Why a metric moved?’ the agent will build its own analytical plan, mapping the critical questions, working through a full multi-step investigation with no manual SQL, and minimizing analytical blind spots. The result is a comprehensive report you can download and share.

Fig 5. Deep Dive mode in Conversational Analytics

Agentic workflows: Deploy autonomous agents that monitor your data, reason over events, run multi-step workflows on a schedule, and deliver insights straight to your chat. You can set up a Monday-morning business report or daily anomaly detection across key metrics, each with a custom directive so they investigate only what you care about.

Fig 6. Scheduling Conversational Analytics agent workflows

Start talking to your data today

General availability of Conversational Analytics in BigQuery marks an official exit from the static dashboard era. By embedding Gemini’s deep cognitive reasoning directly into the data warehouse, we are enabling a self-managing environment that transforms raw data into active, corporate knowledge. This delivery is a key component of the Agentic Data Cloud, providing a true system of action that moves past retrospective reporting, incorporates security and governance by design and is engineered for enterprise trust.

If you are ready to get started, learn more from our documentation, reach out to your Google Cloud account representative, or get started in BigQuery Studio today to build and deploy your first agent.

Scaling Network Analysis for Fraud Prevention with BigQuery Graph

Mon, 29 Jun 2026 16:00:00 +0000

Based in the UK, Curve are building a financial super-app, a smart wallet that consolidates all your debit and credit cards into a single app and card, simplifying how millions of users spend, send and save money.

However, operating at this scale means confronting a high-volume, ever-evolving landscape of financial crime. While traditional fraud detection models are excellent at flagging suspicious individual transactions, they often miss the "bigger picture"—the complex networks and hidden relationships that characterize organized fraud rings.

To uncover these connections, we realized we needed to move beyond traditional relational data modeling. By partnering with Google Cloud to implement BigQuery Graph, we’ve been able to conduct deep network analysis at scale, helping us identify hidden fraud networks and achieve significant transaction savings.

The Challenge: The Multi-Hop Problem

Fraudsters rarely operate in isolation. They often share a subset of attributes across multiple accounts—such as a common device, a specific funding card, or shared contact information. In a standard relational database, identifying these links requires complex "multi-hop" analysis.

Attempting to scale this using standard SQL presented two significant hurdles:

Computational complexity: Uncovering a chain of connections (e.g., User A connects to User B, who connects to User C) requires multiple, massive self-joins. At our volume of millions of users and tens of millions of connections, these queries quickly became computationally expensive and difficult to maintain.
Data scale: Our most granular signals involve billions of potential connections. Standard relational approaches struggle to process these relationships without hitting performance bottlenecks or exhausting system resources.

The Solution: Native Graph Analytics in the data platform

We transitioned our network analysis to BigQuery Graph to take advantage of its native Graph Query Language (GQL) support. The primary advantage was the ability to stop moving our data and start connecting it directly within our existing environment. We had previously explored other popular graph databases - however, being able to keep our data within our BigQuery existing data warehouse gave us significant time and cost savings compared to having to migrate to a new graph database.

By modeling our payment ecosystem as a property graph—where users are nodes and their shared identifiers are edges—we simplified our architecture significantly. Instead of writing dozens of lines of complex JOIN logic, we can now use intuitive GQL syntax to "match" patterns of suspicious behavior across our entire dataset. This approach allows us to:

Traverse billions of connections: We can now analyze massive datasets, including user-level, device-level, and card-level connections, with high performance.
Unify our data experience: Because BigQuery Graph is built into the data platform, we can combine graph traversals with standard SQL analysis, search, and machine learning workflows in a single query. We could therefore leverage our existing SQL pipelines to build the nodes and edges tables, switch to GQL for traversing the graph, and then perform final aggregations with standard SQL. This flexibility makes it accessible to more analysts, without having to upskill in a new language.

Impact and Results

Since integrating BigQuery Graph into our fraud mitigation strategy, the impact on our operational efficiency and bottom line has been profound.

Financial impact: We estimate that the automated blocks triggered by these graph-based insights have saved Curve ~$12M in transaction losses in 2025 alone.
Precision and accuracy: Our graph-powered queries have achieved an accuracy of approximately 72% in identifying fraudulent users. This high precision allows our fraud mitigation agents to focus their manual reviews on high-certainty cases rather than chasing false positives.
Operational speed: Moving to GQL allowed us to streamline our graph queries and refresh our fraud rules more frequently. Previously we were limited to one-hop queries in our hourly rules, but GQL allowed us to optimize these slow-running scripts to stay one step ahead of organized crime.
From rules to ML: The faster we can traverse the network, the faster we can serve graph-based features to our machine learning models. While rebuilding and traversing the graph on a daily basis is sufficient for training models, it is simply too slow at inference-time when transactions can be authorised in less than a second. GQL is allowing us to move towards micro-batch or streaming traversals to serve fresh data to our fraud monitoring models.

Looking Ahead

Our success with BigQuery Graph has opened new doors for our data science and security teams. We are currently working on fully incorporating our highest-volume signals—including billions of IP address connections—into our real-time detection loops. We are also exploring native graph visualization to give our analysts a more intuitive way to explore and "see" fraud webs as they form.

By treating our data as a living network of relationships rather than just rows in a table, Curve is ensuring that our security remains as efficient and robust as our customer experience.

Synthesize the big picture and analyze trends with BigQuery's AI.AGG function

Mon, 29 Jun 2026 16:00:00 +0000

We recently announced the preview of the BigQuery AI.AGG() function. With AI.AGG(), you can use natural-language instructions within a single line of SQL to summarize or synthesize information over millions of rows of unstructured or even multimodal data.

While BigQuery already offers powerful AI functions that help you analyze individual rows of data, analyzing unstructured data at scale requires a different approach. AI.AGG() lets you ask questions from unstructured data such as logs and documents, for example:

What are the top three feature requests among the negative product reviews?
What kind of errors are users seeing most frequently, and how should I start investigating them?
In which specific scenarios is our automated agent consistently failing to resolve customer issues?

In this post, we'll dive deeper into the AI.AGG() function and look at a few of the use cases that it unlocks, including how it can be used in combination with BigQuery’s other managed AI functions for complex, intelligent data analysis.

Analyzing system logs with `AI.AGG()`

A great example of the power of AI.AGG() is analyzing system logging. Log messages, warnings, errors, and stack traces can contain extremely useful information for improving your service, but it can be time- and labor-intensive to investigate them manually — especially if you operate at scale and have thousands of them to review.

With AI.AGG(), you can easily analyze many logs at once, grouping and prioritizing them to decide which ones to dig deeper into first. In fact, our BigQuery engineering team used this exact approach while developing AI.AGG() — using the function to help identify edge cases related to input handling for the feature itself!

To demonstrate this, let’s analyze a public dataset of Apache Spark standard INFO logs available from Loghub. Often, clusters can run into issues like memory thrashing, clock drift, or broadcast bottlenecks without ever throwing a FATAL error. You can use AI.AGG() to analyze these seemingly normal logs for hidden inefficiencies. You can load the sample data file into BigQuery using any of the supported methods, such as the UI, CLI, or client libraries. The following example assumes you’ve loaded the log file into a dataset called bq_logs_demo and table named spark_logs_unstructured.

Notice how we construct the prompt here. We explicitly give the model permission to say "everything is fine," which prevents it from hallucinating errors, while instructing it to hunt for specific anomalies:

code_block: <ListValue: [StructValue([('code', "SELECT\r\n Component AS spark_component,\r\n COUNT(*) AS log_count,\r\n AI.AGG(\r\n Content,\r\n 'Analyze these Spark system INFO logs. Provide a 2-sentence summary: First, describe the normal operation of this component. Second, explicitly identify any hidden inefficiencies, latency spikes, repeated retries, or unusual patterns.'\r\n ) AS performance_analysis\r\nFROM\r\n `bq_logs_demo.spark_logs_structured`\r\nGROUP BY\r\n Component\r\nORDER BY\r\n log_count DESC;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe30342dd00>)])]>

You can see in these results that AI.AGG() successfully acknowledges the "operating normally" messages while surfacing the critical diagnostic insights:

The query results pane showing the insights generated by AI.AGG() over the logs dataset.

Extracting categories from unstructured text and image data

Now, let’s look at some more use cases that demonstrate the flexibility of AI.AGG(), using one of BigQuery’s public datasets, cymbal_pets, a fictional pet supply shop. It includes a catalog of products carried by the store, with unstructured data like product names, descriptions, and images, making it a great example of the power of AI functions for handling unstructured data.

For example, let’s say you want to categorize the products in the dataset. The first hurdle in this case isn't applying labels to your products, but discovering what categories exist across the product catalog. With AI.AGG(), you can ask the model to analyze the raw product names and descriptions to identify the overarching categories for you.

code_block: <ListValue: [StructValue([('code', "-- Identify categories of products from product name and description\r\nSELECT\r\n AI.AGG(\r\n ('Product: ', product_name, ' - Description: ', description),\r\n 'What are the major categories of these products?' \r\n ) AS category_description\r\nFROM\r\n `bigquery-public-data.cymbal_pets.products`;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe30342df10>)])]>

This query returns a simple plaintext list of categories:

The plaintext result of categories determined by AI.AGG() over our products dataset.

This initial query is great for discovery, but a simple plaintext string isn't enough to build a reliable, automated data pipeline. To actually tag your data, you need to instruct AI.AGG() to return a structured format, like a JSON array. Then, you can use the structured categories as a parameter within another AI function, AI.CLASSIFY(), to actually label each product with its category.

The following SQL statement completes each of these steps in one script:

code_block: <ListValue: [StructValue([('code', "-- 1. Declare a variable to hold the array of categories\r\nDECLARE generated_labels ARRAY<STRING>;\r\n\r\n-- 2. Create a dataset to store the results\r\nCREATE SCHEMA IF NOT EXISTS categorized_cymbal_pets;\r\n\r\n-- 3. Generate the JSON string with AI.AGG and extract it into the variable\r\nSET generated_labels = (\r\n SELECT \r\n JSON_VALUE_ARRAY(\r\n AI.AGG(\r\n ('Product: ', product_name, ' - Description: ', description), \r\n 'Identify the major product categories. Return exactly one valid JSON array of strings. Do not include markdown code blocks, backticks, or conversational text.'\r\n )\r\n )\r\n FROM `bigquery-public-data.cymbal_pets.products`\r\n);\r\n\r\n-- 4. Feed the variable directly into AI.CLASSIFY\r\nCREATE OR REPLACE TABLE `categorized_cymbal_pets.categorized_products` AS (\r\nSELECT \r\n product_name,\r\n description,\r\n AI.CLASSIFY(\r\n ('Product: ', product_name, ' - Description: ', description),\r\n generated_labels\r\n ) AS assigned_category\r\nFROM \r\n `bigquery-public-data.cymbal_pets.products`\r\n);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe30342dee0>)])]>

You can now view the resulting table, which includes an assigned_category:

A preview of the categorized_products table which includes the new assigned_category column created by AI.AGG() and AI.CLASSIFY().

If you look closely at the intermediate table, you'll notice the structured categories changed slightly from the initial plaintext results. This happens for two reasons: First, LLMs are nondeterministic, meaning that they don't always give the exact same response to the same prompt. Second, the prompt was adjusted to accommodate the new output structure.

The returned product categories are structured as JSON by AI.AGG() as requested as part of the prompt.

With the table now labeled by category, you can group by the categories to do traditional SQL aggregation, or use AI.AGG() to consider each category separately.

For example, the following query fetches traditional metrics (like row counts) right alongside a synthesized AI summary of what those specific grouped products have in common:

code_block: <ListValue: [StructValue([('code', "-- Synthesize insights grouped by our newly assigned categories\r\nSELECT \r\n assigned_category,\r\n COUNT(*) AS item_count,\r\n AI.AGG(\r\n ('Product: ', product_name, ' - Description: ', description),\r\n 'Write a concise, one-sentence summary describing the common characteristics or purpose of the products in this category.'\r\n ) AS category_summary\r\nFROM \r\n `categorized_cymbal_pets.categorized_products`\r\nGROUP BY \r\n assigned_category\r\nORDER BY \r\n item_count DESC;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe30342deb0>)])]>

Query results showing analyzing with AI.AGG() alongside more traditional SQL methods.

Unstructured data isn't limited to text. Because AI.AGG() natively supports multimodal inputs, you can return aggregated insights directly from image files.

The cymbal_pets Google Cloud project also contains a Cloud Storage bucket full of product photos. By creating an external object table, you can securely pass the image URIs directly into AI.AGG() and ask the model to summarize the visual content of the entire collection.

code_block: <ListValue: [StructValue([('code', "-- Summarize content of images in the object table\r\nSELECT\r\n AI.AGG(\r\n STRUCT(OBJ.GET_ACCESS_URL(ref, 'r')),\r\n 'What are the major categories of these images?'\r\n ) AS category_description\r\nFROM\r\n `bigquery-public-data.cymbal_pets.product_images`;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe30342df70>)])]>

Query results showing AI.AGG() surface product categories by analyzing the product images located in Google Cloud Storage.

How AI.AGG() works and best practices

To use AI.AGG() effectively in your own environment, it helps to understand how it processes data behind the scenes. Here’s what you need to know about context windows, error handling, and optimizing your pipelines.

1. Context windows and multi-level aggregation
LLMs have a specific context window and can have a hard time handling massive amounts of input. AI.AGG() solves this problem by automatically dividing your input rows into batches, aggregating those batches, and then aggregating the results of those batches into a final answer. This means you don’t have to worry about manually managing the context window when passing in large numbers of rows. Note that AI.AGG() won’t split up a row of data across batches, so make sure that each individual row is smaller than the context window, to avoid the row being skipped. Many smaller rows will give AI.AGG() more flexibility with how to batch each row.

2. Token usage with multi-level aggregation
Because AI.AGG() uses a multi-level aggregation structure, the total input tokens sent to the model may be higher than the raw tokens in your starting table (depending on how many rounds of aggregation are required). As a best practice, always reduce the number of input tokens by using LIMIT or pre-filtering your data upstream before passing it to AI.AGG().

3. Specifying your model endpoint
If you don’t specify a model endpoint, AI.AGG() will default to a recent model. However, for production pipelines, you often want explicit control:

Short-form names: You can use a short-form endpoint (e.g., gemini-2.5-flash), in which case AI.AGG() will use that model in the query execution region:

code_block: <ListValue: [StructValue([('code', "AI.AGG(\r\n input_data,\r\n instructions => 'Your instructions here.',\r\n endpoint => 'gemini-2.5-flash' \r\n)"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe30342ddf0>)])]>

Fully-qualified names: If the query execution region doesn’t support your desired model, or you prefer to use a global or multiregional endpoint, provide the fully qualified model name:

code_block: <ListValue: [StructValue([('code', "AI.AGG(\r\n input_data,\r\n instructions => 'Your instructions here.',\r\n endpoint => 'https://aiplatform.googleapis.com/v1/projects/[YOUR_PROJECT]/locations/global/publishers/google/models/gemini-3.5-flash'\r\n)"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe30342ddc0>)])]>

4. Input and output modalities

Inputs: AI.AGG() supports text (via strings or references to text files) and image data. It also supports arrays of these types, though you should refer to the known issues documentation for edge cases regarding arrays of images.
Outputs: The function will always return a string. While you can prompt the model in your instructions to format the output as JSON or Markdown, keep in mind that the database engine does not strictly enforce this. Multimodal output (e.g., generating an image) is not currently supported.

5. Treatment of NULLs
AI.AGG() automatically skips NULL input rows without processing them. However, you must be careful when passing structured data. Like other BigQuery AI functions, AI.AGG() concatenates STRUCT fields similarly to the standard CONCAT() function. This means if even one field within your STRUCT is NULL, the entire row is treated as NULL and will be skipped.

Let's revisit our first categorization query. What if several rows of our products table are missing their description? Because of the NULL concatenation rule, those rows would be silently dropped from the analysis entirely. Here is how we can use IFNULL() to provide a fallback string, guaranteeing that every product is taken into account even if its description is blank:

code_block: <ListValue: [StructValue([('code', "-- Identify categories of products from product name and (optional) description\r\nSELECT\r\n AI.AGG(\r\n ('Product: ', product_name, ' - Description: ', IFNULL(description, 'No description provided')),\r\n 'What are the major categories of these products?' \r\n ) AS category_description\r\nFROM\r\n `bigquery-public-data.cymbal_pets.products`;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe303404250>)])]>

6. Error handling
If AI.AGG() receives invalid input, or encounters an error during LLM processing, it will attempt to provide partial results. Rows containing invalid input or which were rejected by the LLM model will not be considered in the final results.

You can review exactly how many rows failed to process by checking your BigQuery job statistics, exactly as you would for scalar managed AI functions like AI.IF().

information showing an example of Gen AI function error details.

Give it a try!

These are just a few examples of the ways AI.AGG() can help analyze unstructured data. The AI.AGG() function is in preview in BigQuery now, so it’s available to all BigQuery users. Try it out on your own use cases!

You may also be interested in checking out BigQuery's other managed AI functions, AI.CLASSIFY(), AI.IF(), and AI.SCORE(), as well as general-purpose functions like AI.GENERATE(). We look forward to seeing what you build with them.

Boost BigQuery with Python: Managed Python UDFs now generally available

Mon, 22 Jun 2026 17:00:00 +0000

SQL is the industry standard for high-performance structured data analysis. However, expressing complex procedural logic, scientific computations, advanced string manipulations, or machine learning workflows in pure SQL can be highly challenging, if not impossible. That kind of work is better done with Python. Data practitioners often take on additional infrastructure management tasks — maintaining custom images and containers, and working with additional compute services — just to run simple helper functions with custom Python code and libraries.

Today, we are thrilled to announce the general availability (GA) of BigQuery Managed Python User-Defined Functions (UDFs).

This launch represents a major milestone in BigQuery’s extensibility strategy, allowing data scientists, engineers, and analysts to execute custom Python code directly and securely inside BigQuery using standard SQL queries or BigQuery DataFrames (BigFrames) in Python. With this release, Python UDFs are fully supported for production enterprise workloads and completely integrated into BigQuery's billing SKUs.

Bridging SQL and the Rich Python Ecosystem

BigQuery Managed Python UDFs run on BigQuery-managed serverless resources that automatically scales to billions of rows, without having to set up infrastructure or manage containers. BigQuery automatically handles the compilation, image building, security patching, deployment, and execution of your Python code, making it super simple to use Python functions in your SQL.

Core benefits

Flexibility: Access the vast Python ecosystem — including top-tier scientific and mathematical libraries like NumPy, SciPy, pandas, scikit-learn and more — directly in your SQL select statements.
Tight external API integration: Clean and enrich your BigQuery tables in real time by calling external web APIs or Google Cloud services such as Cloud Translation, Gemini Enterprise Agent Platform or custom microservices securely within your queries.
Fully managed and serverless: BigQuery handles the underlying container infrastructure and auto-scales performance dynamically.

Code example

Here is an example of a Python UDF that utilizes a popular Python package — beautifulsoup — to remove HTML tags. We use this function to process

StackOverflow answer bodies that are stored in a BigQuery public table:

code_block: <ListValue: [StructValue([('code', 'CREATE OR REPLACE FUNCTION `your_project.your_dataset.clean_html`(html_content STRING)\r\nRETURNS STRING\r\nLANGUAGE python\r\nOPTIONS (\r\n runtime_version = \'python-3.11\',\r\n entry_point = \'strip_tags\',\r\n packages = [\'beautifulsoup4>=4.12.0\']\r\n) AS r\'\'\'\r\nfrom bs4 import BeautifulSoup\r\n\r\ndef strip_tags(html_content):\r\n if not html_content:\r\n return ""\r\n soup = BeautifulSoup(html_content, "html.parser")\r\n return soup.get_text(separator=" ")\r\n\'\'\';'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe31018a280>)])]>

How to query it:

code_block: <ListValue: [StructValue([('code', 'SELECT \r\n id, \r\n `your_project.your_dataset.clean_html`(body) AS cleaned_answer_body\r\nFROM \r\n `bigquery-public-data.stackoverflow.posts_answers`\r\nLIMIT 100'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe31018a2e0>)])]>

Advanced capabilities

For advanced users, Python UDF adds a set of capabilities to tune the performance as well as monitor the usage. Here are some examples.

Vectorized processing with Pandas PyArrow
To maximize throughput, the GA release supports direct processing of vectorized input as PyArrow RecordBatches. By processing columns of data in bulk rather than row-by-row, PyArrow eliminates Python serialization and conversion overhead, boosting performance by up to 10x for data-intensive calculations.

Configurable container resources
For heavy-duty data science and ML data preparation, you can now provision container memory (up to 16 GB) and CPU (up to 4 vCPUs) per function. This enables memory-intensive workloads (such as loading large serialized models or geospatial datasets) to run directly within the sandbox.

Customizable concurrency
Optimize your throughput and resource efficiency by configuring concurrent requests per container (up to 1,000 concurrent operations). This helps ensure that your scale-out execution is highly cost-effective and performs exceptionally well under heavy parallel loads.

Streaming logs and real-time metrics
Easily debug and monitor your production workloads. The BigQuery console now features a direct link from your query results to real-time CPU, memory, and concurrency metrics in Cloud Monitoring.

Billing

BigQuery Managed Python UDF are billed with BigQuery Services SKU. This SKU is fully eligible for BigQuery spend commitment-based usage discounts (CUDs), allowing you to maximize budget efficiency.

You can also get cost observability through INFORMATION_SCHEMA.JOBS as well as using billing labels MANAGED_ROUTINE_EXECUTION and MANAGED_ROUTINE_BUILD).

See more details in the Pricing section of the documentation.

Getting started

To get started with BigQuery Python UDFs, first check out product documentation.

Then, try out the functions published in the public BigQuery dataset. For example, run the following code in a BigQuery project to tokenize country names data from BigQuery public data. Under the hood, the token UDF utilizes the o200k_base tokenizer library.

code_block: <ListValue: [StructValue([('code', 'SELECT \r\n country_code,\r\n country_name,\r\n `bigquery-public-data`.python_udfs.tokenize(country_name) AS name_tokens,\r\n ARRAY_LENGTH(`bigquery-public-data`.python_udfs.tokenize(country_name)) AS token_count\r\nFROM \r\n `bigquery-public-data.census_bureau_international.country_names_area`\r\nORDER BY \r\n country_name'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe31018a340>)])]>

Or, try out this code lab to explore some advanced scenarios.

Then, to learn how to implement other advanced design patterns, we encourage you to explore our official public documentation guides:

Calling Google Cloud or online services (with connections): To connect to first-party Google Cloud services such as Gemini Enterprise Agent Platform or Cloud Translation, or external API endpoints securely using Cloud Resource connections, - check out the Call Google Cloud or online services in Python code guide.
BigQuery DataFrames (BigFrames) Python UDFs:To learn how to write, deploy, and scale custom Python functions natively from standard Jupyter notebook or Colab environments using BigQuery DataFrames, visit the Customize Python functions for BigQuery DataFrames guide.

Bring your Python workflows out of isolation and directly into the heart of your data warehouse today!

Data Analytics

Agentic Future Ready With BigQuery: Continually Improving Price-Performance, Zero Effort

BigQuery’s Self-Learning Engine: History-Based Optimizations (HBO)

Teaching the BigQuery execution engine newer tricks: Advanced runtime

Same Benefits Regardless of Data Formats

The foundation for the agentic future

BigQuery’s aggregate TCO impact: Faster and more cost-effective price-performance

Solving the "Noisy Neighbor": How Sharded Architecture Protects Multi-Tenant Platforms

The problem: The monolithic bottleneck

Painful effects:

The solution: sharded hub-and-spoke architecture

1. The hub: The router pipeline

2. The buffer: Durable isolation

3. The spokes: Isolated execution

Comparative Benefits at a Glance

Pro-Tips for Implementation

Conclusion

How Target is enhancing retail discovery and cutting database maintenance by 50% with Spanner Graph

Overcoming fragmented architecture

Building the enterprise ontology on Spanner Graph

A smooth, zero-downtime incremental migration

Business impact

Cortex Framework v7 is GA: Build agentic workflows without disrupting SAP operations

What’s new in Cortex Framework v7

Interoperability with SAP Business Data Cloud

Co-innovating with industry leaders

Get started

Unifying public and private data: Scale knowledge graphs with Data Commons on Spanner

Data Commons knowledge graph dimensions

A modern infrastructure powered by Spanner Graph

Bridging systems with SDMX 3.0 interoperability

Transforming private instances of Data Commons Platform

Get started today

The borderless Lakehouse: Bring AWS, Databricks and Snowflake data to your AI agents

Federating multiple catalogs with Iceberg REST

Bring Google AI directly to your AWS and Azure data

Bridging the agent trust gap with universal context

Build and scale agents with Gemini Enterprise

The economics of the borderless Lakehouse and agents

Next steps

Bringing Conversational Analytics to your entire data ecosystem

Query across multi-cloud and database workloads

Proactive insights with Agentic Workflows

Flexible integration with APIs, SDKs, and MCP

Get started with Conversational Analytics

Announcing general availability of SAP Business Data Cloud Connect for BigQuery

Stop copying data for AI projects

Scale your data value without inflating costs

How customers translate shared data into actions

Unify data to build smart workflows with partners

Power of agentic data: Perspectives from SAP and Google Cloud

Regional availability

Get started

Open Knowledge format v0.2 tackles agentic trust

From describing to deciding

Provenance: sources, not a score

Trust: generated, verified

Freshness and lifecycle: stale_after, status

Attestation: Was this number computed the sanctioned way?

What we're releasing with v0.2

Compatibility

Where we go from here

Your AI agents are ready. Is your data?

Overcoming a lack of context

Escaping unnecessary manual work

Bringing trust and knowledge to the data

The path forward

Report: 83% of organizations need to upgrade their infrastructure to support agentic AI

From maintenance to innovation: Checkout's migration to Managed Service for Apache Airflow

The starting point: self-managed Airflow

The solution: Managed Service for Apache Airflow (Gen 3)

Dynamic scaling in action

Reliability and DAG isolation

Faster developer workflows

Conclusion

Level Up Your Column-level Security: Using IAM Data Governance Tags in BigQuery

What are IAM data governance tags?

Why use data governance tags for column-level security?

Three steps to column-level security

Get started today

Trust: `generated`, `verified`

Analyzing system logs with `AI.AGG()`