Databases

Bringing conversational analytics to Gemini with Agentic Data Cloud connectors

Thu, 05 Nov 2026 23:00:00 +0000

Data Cloud connectors in Gemini for business, now in preview, give business users and analysts a direct way to query live enterprise data in natural language. Across most organizations, data questions multiply faster than analysts can answer them: Projects stall in ticket queues, leaders fall back on guesswork, and unmanaged spreadsheets spread across teams. Early generative AI tools promised conversational access to data, but they often struggled to generate accurate SQL across complex schemas or to respect governance rules. Data Cloud connectors let you query governed data directly from your workspace while enforcing row- and column-level security and respecting end-user credentials.

A new way to query enterprise data

Gemini brings conversational analytics directly into your day-to-day workflow through two paths: Conversational Analytics agents, which let you create data agents that include context and instructions to effectively answer questions for specific use cases, and Data Cloud connectors, allowing you to query live data directly with Gemini. End users can pull data directly from BigQuery (preview), operational databases like Spanner, AlloyDB, and Cloud SQL (preview), and governed data from Looker (preview). From there, you can move your findings directly into everyday tools, such as turning query results into a Google Doc, an email draft in Gmail, or a deck in Google Slides.

Ask natural-language questions across enterprise data using Data Cloud connectors.

How Data Cloud connectors work

Data Cloud connectors work with Knowledge Catalog to locate relevant data, interpret your question, and execute permission-scoped queries under your identity.

Knowledge Catalog interprets the user prompt and identifies the right data source for the query.

When you ask a question in Gemini, the system coordinates discovery, semantic interpretation, and query execution, using the following capabilities:

Natural-language question: You ask an exploratory question in plain language (for example, "How many sales did we have this month for all cold weather products?"). End users aren’t expected to know the table or project to use.
Semantic discovery: Gemini searches Knowledge Catalog to locate candidate data across your estate, considering key attributes like usage patterns.
Context mapping: The system uses technical metadata, business glossaries, and semantic models to map everyday terms into exact database structures, such as resolving "cold weather products" to specific categories like sweaters, coats, and jackets in an ecommerce catalog.
Permission-aware execution: Gemini generates a SQL query and runs it through the connector under your user credentials, respecting your existing row- and column-level database permissions.

Step-by-step flow of a user prompt from Gemini to query execution.

To accurately navigate an enterprise data estate, language models need business context alongside raw database access. Knowledge Catalog provides a curated context bundle, including technical metadata, business logic, table relationships, and verified queries, that improves query accuracy while reducing latency and token consumption.

Navigating the context graph visually in Knowledge Catalog.

Enterprise analytics has long been fragmented by physical borders: data locked in proprietary silos, open lakehouse tables in cloud object storage, and datasets distributed across multicloud environments. Beyond native BigQuery storage, the BigQuery connector brings conversational analytics to a borderless lakehouse, unifying these environments into a single conversational interface to query open table formats (including Apache Iceberg, Delta Lake, and Parquet) and cross-cloud datasets in AWS or Azure using BigQuery. The connector inherits fine-grained access policies from BigQuery and Knowledge Catalog, so row- and column-level access controls apply whether you query native tables or open lakehouse files.

Gemini accesses data across BigQuery and AWS S3 (through the AWS Glue federated catalog) to answer questions in plain language.

Choosing between connectors and data agents

In addition to using direct Data Cloud connectors to query data in plain language, you can also use Conversational Analytics agents to enable these experiences. While Data Cloud connectors allow you to provide exploratory and ad hoc analysis for users, conversational analytics agents are scoped agents that are built and certified directly by data teams within your organization. These can be built across Data Cloud products such as BigQuery, Looker, and operational databases including Spanner, AlloyDB, and Cloud SQL.

These pre-configured agents allow users to run highly governed, multi-step strategic analyses that are scoped to particular subsets of data, supporting compliance requirements and predictable results for reporting. These data agents can also be published in Gemini for consumption or made available via API for building your own conversational apps in the enterprise.

These approaches are complementary. Gemini direct connectivity also surfaces relevant published agents created by central data teams, so users can decide whether to proceed with ad hoc analysis using the connectors or use the scoped agent. You can choose the right approach for each workload.

Built-in access controls and governance

Data Cloud connector queries execute under the end user’s identity. An employee without access to executive salary data or protected healthcare records cannot query that data through natural-language prompts. The agent only discovers metadata in Knowledge Catalog to which you have access, and only executes queries that your account has permission to run.

How to create Data Cloud connectors

Administrators can configure Data Cloud connectors in the Gemini admin console. From there, admins set up default read-only connections to datasets in BigQuery, operational databases, or Looker, with Knowledge Catalog enabled automatically. Once configured, end users can enable the connector in Gemini to start querying data and building reports in plain language.

Create a connected data store in Gemini and authorize it on first use.

Setup documentation is available for the BigQuery, Spanner, Cloud SQL, and AlloyDB for PostgreSQL connectors, with Looker also in preview.

Unlike data ingestion mode, which copies data into a Gemini data store, Data Cloud connectors use federated query mode to query your data in place without copying or indexing it — using Knowledge Catalog automatically to search accessible metadata and look up business context.

Why use Data Cloud connectors over custom MCP

Compared to connecting to public or self-managed Model Context Protocol (MCP) servers, first-party Data Cloud connectors provide two main advantages:

Built-in skills for higher response quality: Data Cloud connectors include pre-configured agent skills tailored to each data engine, such as advanced SQL and AI function generation for BigQuery and operational databases, to improve query accuracy and response quality.
Guided setup and managed authentication: First-party connectors provide a guided setup experience with managed authentication for Google Cloud databases and BigQuery, removing the need to deploy MCP middleware or manage your own OAuth client configuration for Google Cloud services. In addition, connector-specific settings, such as specifying a default Google Cloud billing project for BigQuery query execution, help administrators enforce governance and manage query costs.

Get started today

Data Cloud connectors help teams find answers from enterprise data sources without waiting on backlog queues. To get started, review the Data Cloud connectors documentation or explore Conversational Analytics agents.

What’s new with Google Data Cloud

Thu, 05 Nov 2026 17:00:00 +0000

November 2 - November 6

BigQuery observability for data agents is now generally available: Data teams can now monitor the performance, adoption, latency, and cost of BigQuery data agents and their multi-turn conversations using Google Cloud Observability. This capability provides production-grade visibility into agent execution traces, usage volume, and query behavior so organizations can scale conversational analytics with confidence. Review the BigQuery release notes.
BigQuery change history and continuous queries for Apache Iceberg managed tables are now GA: BigQuery change history and continuous queries are now generally available for Lakehouse for Apache Iceberg managed tables. You can use the APPENDS table-valued function to track row additions over a specific time range and run continuous queries to read incoming data from and write output rows to Apache Iceberg managed tables. Review the BigQuery release notes.
BigQuery Data Engineering Agent adds support for BigQuery AI and ML functions: The BigQuery Data Engineering Agent now supports BigQuery AI, ML, and object functions and statements directly inside generated Dataform SQLX pipelines and agent instruction files. Paired with new folder-based Git repository integration in BigQuery Studio, data engineers can build, version, and automate AI-enriched data pipelines faster. Review the BigQuery release notes.
AlloyDB for PostgreSQL introduces AI-powered SQL tuning in preview: AlloyDB for PostgreSQL added the ai.tune_sql function in Public Preview. The function combines AI models with query execution plans and runtime statistics to automatically recommend query rewrites, missing indexes, vacuum operations, and database flag adjustments. Review the AlloyDB release notes.
Zero-downtime embedding model upgrades in AlloyDB AI: A new technical guide demonstrates how to upgrade vector embedding models in AlloyDB AI without disrupting production traffic. Using automatic embeddings alongside ScaNN vector indexes, engineering teams can transition to newer embedding models in the background while keeping semantic search and RAG applications online. Read the blog.
Shrink Cloud SQL for SQL Server storage disks with minimal downtime using Database Migration Service: A new technical guide demonstrates how to reclaim underutilized storage and reduce infrastructure costs on Cloud SQL for SQL Server instances using Database Migration Service (DMS). This workflow migrates databases to right-sized target disks with continuous replication so teams can downsize over-provisioned storage with minimal cutover downtime. Read the blog.
Dataflow Streaming Engine adds latency tiers and scheduled autoscaling: Dataflow introduced new horizontal autoscaling controls for Streaming Engine jobs, allowing teams to select low, medium, or high latency tiers to tune backlog sensitivity. Engineers can also configure recurring autoscaling schedules to adjust minimum and maximum worker counts or utilization targets during predictable traffic windows. Review the Dataflow release notes.
Spanner adds ON DELETE SET NULL foreign key actions across GoogleSQL and PostgreSQL: Spanner now supports the ON DELETE SET NULL referential action on foreign keys for both GoogleSQL and PostgreSQL dialects. When a referenced parent row is deleted, Spanner automatically updates the referencing child columns to NULL, simplifying schema migrations and relational integrity management. Review the Spanner release notes.

October 26 - October 30

LookML metadata indexing in Knowledge Catalog is now generally available: LookML models, Explores, Views, and join logic now continuously sync into the Knowledge Catalog context graph. This integration provides turnkey grounding for Gemini Enterprise and custom AI agents with deterministic business math, paired with end-to-end lineage tracing from underlying BigQuery tables directly to individual Looker dashboard tiles. Read the announcement blog.
Best Buy accelerates self-service insights with conversational analytics in BigQuery: Best Buy deployed conversational data agents directly against BigQuery to empower retail teams with multi-turn natural language exploration. The implementation reduces reliance on manual SQL and dedicated engineering queues, accelerating strategic planning and forecasting cycles while preserving enterprise governance and data protections validated against historical sales records. Explore the Best Buy customer story.
Myer accelerates retail insights with BigQuery data agents and Gemini: Leading department store Myer rolled out conversational data agents in BigQuery across 50+ retail analytics projects. Powered by Gemini, the solution enables commercial and merchandising teams to analyze store performance, stock levels, and customer trends using plain English, speeding up decision-making and reducing backlog requests for data teams. Read the Myer case study.
Rightmove speeds real estate analytics and predictive forecasting with Looker: The UK's top property portal unified its business analytics on Looker and BigQuery, delivering 4x faster reporting for 400+ team members and cutting meeting prep from hours to minutes. In addition, Rightmove built forecasting models that predict lead volume within 2–4.5% of actual results, while piloting Conversational Analytics in Looker. Read the Rightmove customer story.
BigQuery adds AI.TRANSLATE and AI.CHUNK_DOCUMENT for in-database text and RAG pipelines: BigQuery introduced two native generative AI capabilities to simplify unstructured data workflows directly within SQL. The AI.TRANSLATE function enables multilingual translation across Neural Machine Translation and Translation LLMs, while the AI.CHUNK_DOCUMENT table-valued function partitions unstructured text into semantically coherent segments optimized for vector embeddings and RAG pipelines. Review the BigQuery release notes.
AlloyDB adds in-database ai.extract and ai.classify functions to accelerate operational AI: AlloyDB for PostgreSQL introduced two new AI functions, ai.extract and ai.classify, in Public Preview, allowing developers to extract structured entities and categorize unstructured text directly inside SQL queries without external inference pipelines. These functions add to existing AlloyDB AI capabilities such as summarizing, generating, and filtering data with AI. Review the AlloyDB release notes.
AlloyDB Omni Web UI is generally available for centralized hybrid database management: The AlloyDB Omni Web UI is now generally available, providing a centralized graphical interface to manage hybrid and on-premises PostgreSQL deployments. Database administrators can monitor multi-cluster topologies, inspect columnar engine metrics, diagnose slow queries with Query Insights, and verify AI model connectivity across their database fleet. Review the AlloyDB Omni release notes.
Cloud Spanner adds approximate aggregate functions for high-speed analytics at scale: Cloud Spanner now supports approximate aggregate functions across both GoogleSQL and PostgreSQL dialects. Powered by HyperLogLog++ sketches, these functions generate fast statistical estimates for distinct count metrics with minimal compute overhead, accelerating analytical queries and reporting dashboards across petabyte-scale operational datasets. Review the Cloud Spanner release notes.
Knowledge Catalog introduces self-learning context enrichment and Knowledge Studio (Preview): Knowledge Catalog's enrichment agent now learns from agent and user interactions and from sources such as playbooks, code repositories, and business documents to find and fix missing or outdated context, which teams can package into reusable context sets. Knowledge Studio, the new studio experience in Knowledge Catalog, lets context engineers and data stewards explore their data estate conversationally, navigate a visual context graph, and review changes before they reach users and agents. Based on internal benchmarks, grounding agents in Knowledge Catalog makes them nearly 4x more accurate, with up to 33% lower latency and using up to 75% fewer tokens. Read the launch blog.
Knowledge Catalog now grounds AI agents in the semantic models you already trust (GA): Knowledge Catalog natively ingests semantic models and metric definitions from Looker LookML, BigQuery Graph, dbt MetricFlow, Snowflake Semantic Views, Databricks Metric Views, and SAP Core Schema Notation, and connects them with lineage, data quality, and access controls in a single context graph. Gemini Enterprise, BigQuery conversational analytics, and custom agents retrieve a metric's trusted formula, verified join path, and backing table in one context call, without moving data or rewriting business logic. Read the launch blog.

October 19 - October 23

BigQuery introduces column aliases in GoogleSQL and cross-region table copy GA
GoogleSQL in BigQuery now supports column aliases in SELECT and WHERE clauses (GA opt-in), avoiding nested subqueries and simplifying migrations from Snowflake and Teradata. Cross-region table copying is also now generally available to streamline regional data replication and disaster recovery. Review the BigQuery release notes.
Conversational analytics in BigQuery expands to Canada and adds CSV/PDF export
Conversational analytics in BigQuery is now available in Montréal (northamerica-northeast1) and Toronto (northamerica-northeast2) to support Canadian data residency requirements. You can also now export agent responses — including text, tables, charts, and citations — directly to CSV or PDF (GA). Learn more about conversational analytics in BigQuery.
AlloyDB Omni 18.4.0 is now GA with Advanced Query Insights and external KMS support
AlloyDB Omni 18.4.0 brings PostgreSQL 18.4 to self-hosted and multi-cloud environments, adding the google_insights_v2 extension for granular query execution and wait-event diagnostics. It also adds Thales CipherTrust Manager support for Transparent Data Encryption (TDE) alongside HashiCorp Vault. Explore the AlloyDB Omni release notes.
Cloud SQL for PostgreSQL adds managed connection pooling and Private Service Connect defaults
Cloud SQL for PostgreSQL now supports enabling built-in managed connection pooling on existing Private Service Connect (PSC) instances to scale high-concurrency workloads. PSC is also now the default private IP option for new Cloud SQL instances in new projects to simplify cross-project networking. Review Cloud SQL updates.
Customer momentum: Unico prevents over $5B in fraud on Google's Agentic Data Cloud
Digital identity platform Unico migrated from Databricks Spark/Scala clusters to BigQuery and Spanner at the core of its data architecture. Unifying real-time transaction streams with petabyte-scale analytics enabled low-latency identity verification that has prevented over $5 billion in fraud. Read the Unico case study.
Developer guide: Getting started with Looker Data Apps
Following the debut of Looker Data Apps in preview, a new walkthrough on the Google Developer Community shares practical patterns for building prompt-driven applications inside Looker. Read the guide on the Google Developer Community.

October 12 - October 16

Looker Data Apps are now in Preview to build and host interactive, agent-generated applications natively inside Looker
Generated using natural language prompts, the Antigravity CLI, or Claude Code, these interactive web applications, natively hosted inside Looker, transform business reporting into operational systems of action. Users can test scenarios, adjust parameters, and trigger workflows in downstream platforms while inheriting LookML semantic modeling, row-level security, and access controls with no runtime LLM token costs. Read the announcement blog.
Build end-to-end data science pipelines with the Data Agent Kit
Following the general availability of the Data Agent Kit, a new end-to-end walkthrough demonstrates how data teams can build, test, and schedule machine learning pipelines directly from their preferred code editors. Explore the walkthrough and codelab.
Manage dashboard agents as code to bring CI/CD and version control to LookML dashboards
Data teams can now declare agent parameters, configurations, and prompt instructions alongside dashboard files in code. This integration enables standard developer workflows for conversational BI, including Git version control, peer review, branching, and automated CI/CD validation before agent modifications reach production end users. Read the technical guide on the Google Developer Community.
Simplify AI search with native AlloyDB hybrid search and Reciprocal Rank Fusion
AlloyDB for PostgreSQL simplifies hybrid search architectures by combining vector similarity search and keyword BM25 retrieval within a unified database engine. Using Reciprocal Rank Fusion (RRF), developers can merge lexical scoring and semantic embeddings into a single ranked result without writing a complex SQL query. This unified approach removes the operational overhead of running and synchronizing external search engines alongside operational databases for retrieval-augmented generation (RAG) and search applications. Explore hybrid search in AlloyDB.

October 5 - October 9

Data Agent Kit is now generally available to bring Google Data Cloud to any coding agent
Data Agent Kit provides a free collection of Model Context Protocol (MCP) tools and Google-authored agent skills that connect more than 15 Google Data Cloud services directly to coding agents across VS Code, Antigravity, Cursor, Claude Code, and Codex. This release expands support for developers and data practitioners to inspect schemas, author queries, and build end-to-end data pipelines in natural language directly from their IDE or CLI. Read the announcement blog.
Spanner Omni is now generally available for deploy-anywhere, multi-model workloads
Spanner Omni brings Spanner's distributed SQL, graph, vector, and key-value capabilities to any environment, including on-premises data centers, other public clouds via Kubernetes or VMs, and local developer laptops. Organizations can build and run multi-model applications with consistent APIs and query engines across hybrid and multi-cloud footprints while seamlessly scaling to managed Spanner on Google Cloud. Read the launch blog and explore how Spanner also removed cumulative mutation limits for DML transactions.
AlloyDB delivers PostgreSQL for agents and native BM25 hybrid search
We announced PostgreSQL for agents in AlloyDB, introducing an agentic database architecture engineered for real-time data access at agent scale with workload isolation. In addition, native BM25 ranking is now in Preview for AlloyDB and Cloud SQL, allowing developers to combine keyword search and vector embeddings directly in PostgreSQL without maintaining an external search engine. For hybrid deployments, the AlloyDB Omni RPM Orchestrator is now generally available, and you can dive deeper in our new Build AI apps and agents with AlloyDB eBook.
Lakehouse runtime catalog adds regional endpoints across 22 regions and GA support for flexible column names
Lakehouse runtime catalog regional endpoints are now live across 22 Google Cloud regions across the Americas, Europe, Asia-Pacific, and the Middle East, supporting both the Apache Iceberg REST catalog endpoint and the Apache Hive catalog endpoint to help customers meet strict data residency and sovereignty requirements. In addition, flexible column names for Apache Iceberg tables are now generally available by default across Iceberg external tables and managed tables, enabling seamless querying of columns containing spaces, hyphens, and Unicode characters. We also announced the Preview of cross-cloud caching and connections for the borderless Lakehouse.
Unlock predictive and agent-ready insights with BigQuery augmented analytics TVFs, TabFM, and identity columns
BigQuery introduced augmented analytics table-valued functions (TVFs): six built-in functions that automate metric diagnostics, trend analysis, and pattern discovery directly where your data lives and integrate as skills for AI agents. We also introduced TabFM in BigQuery to bring tabular foundation model capabilities to predictive analytics, alongside BigQuery identity columns to automatically generate sequential surrogate keys and simplify data warehouse pipelines.
Dataflow adds Pause/Resume and NVIDIA RTX PRO 6000 Blackwell GPU support for large-scale AI workloads
New Dataflow capabilities make it easier to run high-throughput streaming and batch AI pipelines at scale, including Pause/Resume job controls and support for NVIDIA RTX PRO 6000 Blackwell GPUs. And for teams running Apache Spark workloads, learn how to mitigate compute stockouts and maximize availability using flexible VMs.
Unlock up to 3x higher QPS and microsecond latency with Memorystore for Valkey 9.1
Memorystore for Valkey 9.1 is now available, delivering up to 3x higher queries per second (QPS) and microsecond-level caching latencies to accelerate high-throughput applications and real-time AI agent state management. Read the blog.
Customer momentum across Google Data Cloud — Yahoo, PayPal, Surescripts, Airwallex, and Lucius AI
Organizations across industries are scaling their data and AI foundations on Google Data Cloud. Yahoo uses flexible VMs in Managed Service for Apache Spark to reduce provisioning failures by 85%, while PayPal migrated mission-critical analytics workloads to Managed Service for Apache Spark to scale and reduce operational overhead. On the database front, Surescripts migrated 30.5 billion annual healthcare transactions to AlloyDB with 99.998% uptime, Airwallex manages more than 5,000 databases with 99.99% availability on Cloud SQL with a team of four DBAs, and Lucius AI runs a 210,000-tender global platform on AlloyDB and MCP with 47x faster ScaNN vector search.

September 7 - September 10

Pub/Sub SMTs can now AI Inference your Gemini Enterprise Agent Platform models!
Pub/Sub AI Inference SMTs allow you to apply inference on an incoming stream of events using models hosted in Gemini Enterprise Agent Platform. The model’s prediction is appended to your event, making it available for downstream processing in your data warehouse (like BigQuery) or operational database (like BigTable). This feature, now generally available, can dramatically simplify or enhance anomaly detection systems you are operating.
PostgreSQL Source Connector is now generally available in Managed Service for Apache Kafka!
Managed Service for Apache Kafka’s PostgreSQL connector allows customers to capture changes from their PostgreSQL database and ingest them into their Kafka infrastructure with low latency. This source connector is compatible with Cloud SQL for Postgres, AlloyDB, and self-managed PostgreSQL databases. Try this along with our entire portfolio of managed connectors, including MirrorMaker 2.0, BigQuery, Cloud Storage, and Pub/Sub! E-mail kafka-hotline@google.com if you have questions or feedback!

Pause-on-failure for Dataflow batch jobs is GA
Dataflow pause-on-failure enables you to preserve the state of a batch Dataflow job before it fails. By pausing your Dataflow job, you can address issues that are external to the pipeline and resume processing without losing completed work. This helps you better manage resource costs and improve job reliability when you face temporary outages or capacity constraints.

The insertAll API is now the BigQuery Storage Write API (REST)
The legacy insertAll streaming API is now rebranded as the BigQuery Storage Write API (REST). By dropping the "legacy" label, developers can confidently build long-term HTTP-based streaming workflows. This stateless JSON-over-HTTPS endpoint offers a lightweight alternative to heavy gRPC libraries—ideal for serverless web apps, IoT telemetry, and AI logging. The transition is seamless for existing users, requiring zero code changes and offering 100% backward compatibility. However, the Storage Write API (gRPC) version remains the recommended standard for high-throughput, continuous pipelines.

August 31 - September 4

Stateful processing is available in BigQuery continuous queries in Preview
Stateful operations significantly expand what’s possible with BigQuery continuous queries. This feature allows users to leverage functions like JOINs, aggregations, and windowing functions directly in their streaming queries. Now you can calculate metrics over time (for example, a 30-minute average) to power your downstream applications and AI agents with much richer, real-time signals.

Try out our feature here and share your feedback with bq-continuous-queries-feedback@google.com!
Synthetic data generator tool is available for Managed Service for Kafka
You’ve launched your first Kafka cluster. Now what? The next thing to do is to produce some data to the cluster, but that involves modifying a client application somewhere or spinning up a virtual machine. The synthetic data generator tool, now generally available, can start sending mock data to your cluster in 3 clicks, and will get data streaming into your cluster in less than two minutes. The perfect utility for those moments you just want to test your cluster and new features. Try our quickstart today!

Dataflow pipeline updates are faster & more flexible
Dataflow pipeline updates can now stop-and-replace pipelines, a major addition to the existing in-place-update feature. The new parallel pipeline option accelerates the migration between the old & new pipeline, resulting in reduced disruption to your business. You can also set a timeout on drains that prevents runaway costs for your pipeliness in the event of stuck processing. This feature is generally available. Try it here!

July 6 - July 10

New Lakehouse managed tables now in preview
Lakehouse tables for Apache Iceberg are now in preview and available in the console. By using Google-managed Apache Iceberg tables in Lakehouse, you can eliminate the overhead of maintaining duplicate data pipelines and complex synchronization logic between BigQuery and open-source engines. This unified table format delivers native, multi-engine read and write interoperability, allowing you to run concurrent DML/DDL operations across diverse analytics tools on a single, shared storage layer. Built-in automated table management handles painful background optimization tasks like compaction and partition tuning, freeing up your team to focus on building rather than managing storage maintenance.

June 1 - June 5

Beyond the Query: Powering AI Agents with Bigtable, Firestore & Memorystore
Discover the latest advancements in Google Cloud's NoSQL Database portfolio, including Bigtable, Firestore, and Memorystore. This series is designed for a broad audience: whether you are exploring these databases for the first time or are an existing user looking to leverage the new capabilities announced at Next '26.

Register here to secure your spot!

Cloud Engineer's AI Toolkit Workshops: Solve data-driven challenges with BigQuery, AlloyDB, Gemini and more. Hosted by Google Cloud Labs, this highly technical event is built specifically for Platform Engineers, SREs, and cloud infrastructure teams ready to bridge the gap between AI prototypes and production-grade deployments. Look out for more locations coming soon

Toronto - June 25 (Data Cloud) | RSVP Here
Chicago - June 30 (Data Cloud) | RSVP Here
Start a 10-day Bigtable free trial with a 1 node SSD cluster and up to 500GB of storage capacity. With no credit card required to start, you can easily ingest workloads and manage workloads that require low-latency, high-throughput, and predictable access. Plus, new Google Cloud customers get $300 in free credits on signup.

May 11 - May 15

Managed Service for Apache Airflow has launched a wave of new features, including the general availability of Airflow 3.1, AI-powered agentic troubleshooting, a new managed Airflow MCP Server for custom agent integration, and declarative YAML-based orchestration pipelines—discover all the details in the full blog post.

April 20 - April 24

Google-built ODBC Driver for BigQuery is now available in Preview
We are excited to announce the launch of the new, Google-built ODBC driver for BigQuery. This new open-source driver provides a direct, high-performance connection for applications to BigQuery and is developed entirely in-house by Google. Download a new driver and connect your application to BigQuery.

April 13 - April 17

We announced we are reintroducing Data Studio to play a significant role in the AI era, expanding from data visualizations and reports to host BigQuery conversational agents and data apps built in Colab notebooks.
We announced BigQuery Graph is now available in preview, offering an easy-to-use, highly scalable graph analytics solution, empowering data professionals to model, analyze and visualize massive-scale relationships in an entirely new way.

April 6 - April 10

We introduced Conversational Analytics for Looker Embedded environments, enabling users to add natural language experiences to their own custom data-driven applications, powered by Gemini.
We expanded Looker’s capabilities for faster ad-hoc analysis, with the introduction of self-service Explores, enabling you to bring your own data to Looker’s semantic layer and gain instant access to insights in a governed data environment.

March 23 - March 27

We showed you how you can scale your reads with Cloud SQL autoscaling read pools. This feature allows you to provision multiple read replicas that are accessible via a single read endpoint and to dynamically adjust your read capability based on real-time application needs.
Our customers are leveraging the full power of Conversational Analytics and Looker to drive major business and technical breakthroughs in the AI era. Companies like Telenor, Pet Circle, Fluent Commerce, Lighthouse Intelligence, Wego, and ROLLER are turning data into insights and actions, grounded by Looker’s semantic layer.

March 16 - March 20

We introduced an enhanced Gemini assistant in BigQuery Studio, transforming the agent from a code assistant into a fully context-aware analytics partner.

February 23 - February 27

We introduced managed and remote MCP support for Google Cloud databases, including AlloyDB, Spanner, Cloud SQL, Bigtable and Firestore, to power the next generation of agents. This announcement extends the ability for AI models to plan, build, and solve complex problems, connecting to the database tools our customers leverage daily as the backbone of their work environment.
We outlined how you can build a conversational agent in BigQuery using the Conversational Analytics API to help you build context-aware agents that can understand natural language, query your BigQuery data, and deliver answers in text, tables, and visual charts.

February 16 - February 20

Our customers are leveraging the full power of Looker to drive major business and technical breakthroughs. Companies like Arrive, Audika, Carousell, Framebridge, GumGum, Intel, Overdose Digital, Ocean Network Express, Subskribe and Promevo are leveraging Looker’s newest AI-driven capabilities, including Conversational Analytics, to transform data to insights and actions, and empower their entire organization with a single source of truth, powered by Looker’s semantic layer.

February 2 - February 6

Join us on March 4 for our webinar, Win Your AI Strategy with Cloud SQL Enterprise Plus, to learn how to power your generative AI workloads with 3x higher performance and 99.99% availability. Register today to discover how to build a scalable, enterprise-grade foundation for your most demanding AI applications.

January 26 - January 30

We introduced Conversational Analytics in BigQuery, which allows users to analyze data using natural language. Conversational Analytics in BigQuery is an intelligent agent that generates, executes and visualizes answers grounded in your business context directly in BigQuery Studio, making data insights for data professionals more conversational.
We outlined how data products have become the foundation for AI agents, providing the context needed to make autonomous agents reliable and trusted for real business use, backed by organized business logic and semantic understanding.
We highlighted how you can supercharge data analytics workflows, and outlined Google Cloud’s AI agent offerings for data engineering, data science, and development tools, so you can integrate agentic workflows in your applications, empower your teams and speed discovery.

January 19 - January 23

We have fundamentally reimagined Firestore with pipeline operations for Enterprise edition. Experience a powerful new engine featuring over a hundred new query features, index-less queries, new index types, and observability tooling to improve query performance. Seamlessly migrate using built-in tools and leverage Firestore’s existing differentiated serverless foundation, virtually unlimited scale, and industry-leading SLA. Join a community of 600K developers to craft expressive applications that maximize the benefits of rich queryability, real-time listen queries, robust offline caching, and cutting-edge AI-assistive coding integrations.
Introducing Google Cloud SQL on MSSQLTips: We are highlighting a new technical guide published on MSSQLTips titled "Introducing Google Cloud SQL." This article serves as an essential resource for SQL Server administrators and developers exploring Google Cloud's fully managed database service. It provides a detailed overview of Cloud SQL capabilities, including high availability, security integration, and the seamless transition of on-premises SQL Server workloads to the cloud, making it an ideal resource for those planning their migration strategy.
We are excited to announce the Public Preview of Microsoft Entra ID (formerly Azure Active Directory) integration with Cloud SQL for SQL Server. Designed to tackle the challenge of identity sprawl in multi-cloud environments, this integration allows organizations to govern database access using their existing Microsoft identity infrastructure. Key benefits include centralized identity management, enhanced security features like Multi-Factor Authentication (MFA), and simplified user administration through direct group mapping. This feature is available for SQL Server 2022 and supports both public and private IP configurations.

January 12 - January 16

Google-built JDBC Driver for BigQuery is now available in Preview
We are excited to announce the launch of the new, Google-built JDBC driver for BigQuery. This new open-source driver provides a direct, high-performance connection for Java applications to BigQuery and is developed entirely in-house by Google. Download a new driver and connect your Java application to BigQuery.
Troubleshoot Airflow tasks instantly with Gemini Cloud Assist investigations: Cloud Composer just got smarter. We are excited to announce that Gemini Cloud Assist investigations are now available directly within Cloud Composer 3. Instead of manually sifting through raw logs, you can now simply click "Investigate" on a failed Airflow task. Gemini analyzes logs and task metadata to identify failure patterns—such as resource exhaustion or timeouts—and provides actionable recommendations driven by Gemini Cloud Assist to resolve the issue. This integration shifts the debugging experience from manual toil to automated root cause analysis, significantly reducing the time required to restore your pipelines. Learn more about AI-assisted troubleshooting.

New capabilities to power agents at scale with Google's Agentic Data Cloud

Wed, 04 Nov 2026 14:00:00 +0000

Not long ago, we were entering the agentic era; today, organizations are deriving significant value from running agents in production. The opportunity to capture new revenue streams, improve efficiency, and manage risk makes reimagining operations with agents top of mind for enterprise leaders.

That is why we built the Agentic Data Cloud — a System of Action designed to activate all your data by breaking down walled gardens, deliver high accuracy and trust, and operate proactively at low cost. Leading organizations such as PayPal, Siemens, Telus, Vodafone, Walmart, and more are already achieving business impact with us today.

Today, we’re sharing key updates to Google’s Agentic Data Cloud. We’re empowering your agents everywhere, grounding them in your enterprise context to deliver accurate, trusted answers, and providing elevated agentic-first experiences for both practitioners and business users.

Activate all your data for agents with the borderless Lakehouse

Enterprise data lives everywhere — across multi-cloud environments, on-premises systems, and SaaS platforms like Confluent, SAP, Salesforce, and Workday. The borderless Lakehouse, built on the open Apache Iceberg format, lets you activate all of your data, wherever it lives, so that your agents have one governed foundation. Instead of moving data, stitching together connectors, or paying surprise variable egress fees, you get a stack tuned at every layer, from the infrastructure to the model. In fact, Apache Iceberg data under management has more than tripled over the past year.

Today, we are announcing several new enhancements for the borderless Lakehouse. Gemini now works directly on your borderless Lakehouse allowing business users to use natural language to query records that live across clouds while agents ground their reasoning and drive business workflows. With cross-cloud interconnect and caching for AWS and Azure, now generally available, you can query data across clouds with a private, secure connection and zero variable egress fees. And when you query remote cloud data, Lakehouse automatically caches retrieved data blocks locally within Google Cloud. Additionally, customers can use the open Iceberg catalog federation with Databricks Unity, Snowflake Horizon, and AWS Glue as well as zero-copy integrations with SAP, Workday, and Salesforce.

To deliver context for AI, the borderless Lakehouse integrates directly with Knowledge Catalog, to automatically index your distributed and unstructured data, in cloud storage, giving agents the right context to be accurate, while controlling token costs for an intelligent and sustainable agentic strategy.

"By implementing the borderless Lakehouse that federates single-copy Apache Iceberg data from Amazon S3 tables to BigQuery, Delivery Hero's talabat reduced data-movement costs by approximately 40% while making streaming transactional events queryable across both clouds within minutes of production." - Harish Ramesh, Staff Data Engineer, Delivery Hero talabat

To take action in the moment, agents need simultaneous access to both your historical analytical data, and real-time transactional data. That’s why today we're announcing new AlloyDB integrations with the borderless Lakehouse. With direct Iceberg access in AlloyDB, you can now directly query Iceberg tables stored in cloud storage, without any ETL pipelines or copies, and join live operational data with historical lakehouse data in a single PostgreSQL query. By combining AlloyDB's transactional capabilities with direct Iceberg table access, systems can efficiently bridge transactional and analytical data to meet rigorous demands from agents. It is fully integrated with the Lakehouse runtime catalog, allowing you to register Iceberg tables in AlloyDB alongside the rest of your tables. Everything automatically shows up in Knowledge Catalog — so your agents get the same governed context. AlloyDB direct Iceberg access is up to 7x faster on aggregation queries than Amazon’s comparable PostgreSQL-compatible service.

And completing this borderless strategy, Spanner Omni, which is generally available, brings Google-grade consistency, scale, and multi-model capabilities to hybrid agentic applications across on premises, edge nodes, and multi-cloud environments.

Deliver accurate, trusted answers by grounding every agent in enterprise context

Agents are only as accurate as the business context they are grounded in. Knowledge Catalog is Google Cloud’s universal context engine that turns raw data into shared business context for agents to answer and act with higher accuracy, trust, and token efficiency. Today, we’re taking Knowledge Catalog to the next level by extending our AI-driven context curation capabilities with self-learning, reusable context sets, and a reimagined user experience so your context layer is built and evolved automatically with your business while keeping humans in full control of their data governance. Based on our internal benchmarks, Knowledge Catalog makes AI agents nearly 4x more accurate, with up to 33% lower latency, and using up to 75% fewer tokens.

Much of an enterprise’s institutional knowledge lives outside their databases in operational playbooks, documents, and tickets, and context for agents must be curated around the questions people actually ask. With the new self-learning enrichment in Knowledge Catalog, available in preview, agents take on the heavy lifting of context curation. Knowledge Catalog learns from user intent and every human and agent interaction by continuously mining unstructured sources like Google Workspace, GitHub, Jira, Confluence, and SharePoint to close context gaps and evolve context automatically. It also packages domain knowledge into reusable Open Knowledge Format (OKF) context sets, allowing context engineers and data stewards to govern this enriched context effortlessly. For data stewards and data teams, we also introduced, a new studio experience in Knowledge Catalog, in preview, which provides a conversational user interface where you can explore context in natural language, visualize your entire data estate across cloud environments and SaaS applications, and review, audit, and approve proposed enrichment updates with full versioning and rollback.

New studio experience in Knowledge Catalog showcasing conversational context exploration and an interactive graph

Additionally, we announced that Knowledge Catalog is now on by default in Gemini, in preview. Every data question is now grounded in a governed business context automatically. When you connect BigQuery, AlloyDB, Cloud SQL, or Spanner, Knowledge Catalog resolves business terms, locates trusted tables, and determines join paths before generating SQL — under each user’s own access permissions. This gives organizations the confidence to deploy agents at enterprise scale because your agents draw from shared, registered context sets for consistent accuracy, lower latency, and reduced token costs.

Finally, because critical business definitions cannot be left to an LLM’s guesswork, Knowledge Catalog grounds agents in trusted semantic models, now generally available, from BigQuery, Looker (LookML), Databricks, Snowflake, dbt, and SAP, without moving data. Whether an agent is calculating net revenue for finance or customer churn for operations, it applies the exact same governed formulas your business teams rely on. Knowledge Catalog also extends metadata discovery and federation to Oracle AI Database@Google Cloud, Snowflake, Databricks Unity Catalog, and third-party catalogs like Atlan and Collibra — building on the catalog investments you’ve already made so your entire data estate is universally understood by every agent.

Transforming practitioner experiences with agents

We are reimagining how data professionals and business teams work — elevating practitioners from individual contributors supported by co-assistants, into orchestrators of autonomous agents that drive intent-driven business outcomes.

For both technical teams and everyday business users, we introduced two new purpose-built capabilities in Gemini, at the recent Gemini at Work event. For Data and ML engineers, we are extending Gemini with machine learning skills and tools that back every data scientist and data engineer with a team of agents. Engineers describe the outcome they want in plain language, and Gemini generates PySpark code, provides notebooks to edit and test it, trains models, and troubleshoots and fixes pipeline issues on its own. And for business users, anyone in your organization can generate reliable, real-time operational reports simply by asking. Gemini uses operational reporting skills integrated with BigQuery and the Knowledge Catalog to construct and save the query. Once saved, teams can run reports on demand without incurring token costs, guaranteeing consistent, verified answers every time.

Now, for developers who'd rather stay in their own tools, we recently announced the general availability of the Data Agent Kit. It eliminates console hopping and ETL boilerplate by integrating Google Data Cloud tools directly into your developers' preferred IDEs and CLIs — like VS Code, Claude Code, and Antigravity. Armed with native MCP tooling, practitioners define business intent and let agents generate, tune, and test data pipelines.

Engineering a data infrastructure built for agent scale

As organizations scale to millions of agents collaborating across workflows, traditional database architectures break down. We’re introducing tailored infrastructure engineered specifically for agent scale.

To accelerate rapid testing and prototyping, today we’re announcing AlloyDB Express, in preview. This is a new edition of AlloyDB designed for builders and coding agents who need extreme development velocity. It offers sub-2-second instant provisioning, scale-to-zero pause and resume, secure private IP connectivity, integrated observability, and a small footprint with 1–2 vCPUs and 20GB of storage. AlloyDB now provides a developer-first, PostgreSQL environment for agent development and testing.

For production environments, we recently announced PostgreSQL for agents in AlloyDB, in preview, which enables thousands of collaborating AI agents to query live transactional data with complete workload isolation. With the agentic database architecture, agents achieve sub-millisecond I/O, and terabit-per-second aggregated scan throughput, supporting over 3 million queries per second. Compute scales automatically based on active reasoning sessions and scales back to zero when idle without affecting production systems.

For running open-source data infrastructure, Managed Service for Apache Spark extends its price-performance advantage compared to market alternatives by over 3x through further improvements to Lightning Engine. Recent enhancements for Spark 4.0 include faster startup at no additional cost, beneficial for interactive workloads and small batch ETL jobs. In addition, the premium tier provides performance boosts by pairing upgraded hardware with software acceleration from the Lightning Engine, and this is on by default.

Finally, we’ve continued to enhance BigQuery for agent scale. In the past year alone, we've made queries over 40% faster while cutting query processing cost by more than 50% through intelligent optimizations like fluid scaling and built-in workload acceleration. BigQuery Graph is now generally available, bringing petabyte-scale graph analytics directly on relational tables with zero ETL. It allows organizations to model data as nodes and edges to uncover complex, hidden relationships without moving data into a separate graph database. And today, two new models from Google Research run natively in BigQuery. The first is TimesFM 3.0, now in preview, which lets you generate forecasts directly in SQL with no custom models to train or manage and TabFM, also now in preview, which brings predictive machine learning directly into BigQuery.

Get started today

In addition to these announcements, we continue to enhance core performance, cost management, and operational resilience across our broader data ecosystem.

Ready to transform your data estate into an active System of Action? Explore our Agentic Data Cloud page, sign up for a Data Strategy Workshop and start building your agentic enterprise, at no cost, with Google Cloud.

You can also head over to our Trusted Agents, Real Outcomes digital event to catch our Agentic Data Cloud vision and the latest innovations. See the portfolio in action, learn from Yahoo’s real-world blueprint, and unpack deep-dive sessions with product leaders across Databases, Data Analytics, and BI.

Unify operational and Lakehouse data with direct Iceberg access in AlloyDB for PostgreSQL

Wed, 04 Nov 2026 14:00:00 +0000

To make good business decisions, you need two kinds of data: the live state of the business from operational databases, and the historical context stored in an analytical system such as a lakehouse or a data warehouse. Traditionally, those systems have been separate, but they are now converging rapidly. That’s why today, we're announcing direct Iceberg access in AlloyDB for PostgreSQL, in preview. Now, developers can query Apache Iceberg tables stored in a cloud storage bucket directly from AlloyDB and join them with live operational data, all in a single SQL query.

In addition, we’re announcing the general availability of AlloyDB sync and Lakehouse federation. Together, they give AlloyDB three ways to work with the borderless Lakehouse, which keeps one open copy of your data available to any engine. Etsy, for example, unified three petabytes with the borderless Lakehouse, joining data in place to accelerate pipeline queries by 60%. With AlloyDB, which approach to use depends on where the data lives and how your application reads it:

Direct Iceberg access: Read Iceberg tables in place, accelerated by the AlloyDB columnar engine.
AlloyDB sync: Serve lakehouse data from AlloyDB for your most latency-sensitive workloads, refreshed on a schedule you set.
Lakehouse federation: Query native BigQuery tables in place, with BigQuery doing the processing.

With this launch, apps and agents can act on operational and lakehouse data together without building or managing custom pipelines.

AlloyDB and the borderless Lakehouse: Query Apache Iceberg and BigQuery data in place or sync it to AlloyDB, giving real-time applications and AI agents governed access to operational and lakehouse data together.

Decisions at the moment of action

With these integrations between AlloyDB and the borderless Lakehouse, historical context sits next to live transactions, so decisions can happen at the moment of action, without building or managing custom pipelines. For example:

Fraud and risk: Before approving a payment, a fraud agent checks it against five years of customer transaction history in Iceberg, in addition to the last 30 days in the database.
Personalization: A shopping agent joins the live cart with lifetime value and return history in the lakehouse to choose the right offer while the shopper is still on the page.
Dynamic pricing: A pricing agent combines live inventory in AlloyDB with years of demand and weather history in the lakehouse to adjust prices as conditions change.
Data products: A data team publishes curated tables once to Iceberg, and then dozens of applications read them in place instead of each maintaining its own copy.

In each case, the history stays in the lakehouse and the decision happens in the database. Agents act on your behalf using only the tables and permissions you grant them in AlloyDB.

Let’s take a deeper look at each of the three integrations.

Direct Iceberg access in AlloyDB (Preview)

With direct Iceberg access, AlloyDB reads Apache Iceberg tables directly from Cloud Storage, using either a bucket path or the Lakehouse runtime catalog. You register an Iceberg table once and query it like any other PostgreSQL table, including joins with live operational tables.

code_block: <ListValue: [StructValue([('code', "-- Register five years of transaction history stored as Iceberg\r\nCREATE TABLE transaction_history (\r\n customer_id BIGINT, category TEXT, amount NUMERIC, ts TIMESTAMPTZ)\r\nUSING iceberg\r\nWITH (uri = 'gs://my-bucket/warehouse/payments/transaction_history');\r\n\r\n\r\n-- Compare the payment in flight with the customer's history\r\nSELECT t.amount,\r\n AVG(h.amount) AS usual_spend,\r\n MAX(h.amount) AS largest_past_purchase\r\nFROM recent_transactions t -- live, in AlloyDB\r\nJOIN transaction_history h -- five years, in Iceberg\r\n ON h.customer_id = t.customer_id AND h.category = t.category\r\nWHERE t.transaction_id = 'TX-1001'\r\nGROUP BY t.amount;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f0535510>)])]>

AlloyDB reads Iceberg tables in place and joins them with live operational data, using the same copy as the rest of your lakehouse.

Iceberg support is built into AlloyDB and is accelerated by the AlloyDB columnar engine, so queries over lakehouse data are fast. They run on the same database instance as your application, without a separate reader instance to manage, and they read the latest data your team has published to the lakehouse.

In our testing on a 1 TB dataset in Cloud Storage, AlloyDB direct Iceberg access is up to 7x faster on aggregation queries than Amazon's comparable PostgreSQL-compatible service, and up to 3.5x faster on aggregation queries than the open-source PostgreSQL extension.

AlloyDB sync for the lowest-latency reads (GA)

Not every table needs to be read in place. A retail merchant risk score, for example, is checked before every payment but only changes when the data team recomputes it overnight. Data like this, read on every request and refreshed on a schedule, is fastest as a local table in AlloyDB. The challenge is keeping that copy fresh without building and maintaining a pipeline.

AlloyDB sync copies lakehouse data from BigQuery into AlloyDB, once or on a schedule, and keeps it refreshed. The copy is a local AlloyDB table, so your app reads it as fast as any other table, and you can index it, serve it from read pools, or use it with the columnar engine and vector search. You can start a sync with one click from the BigQuery console or from SQL:

code_block: <ListValue: [StructValue([('code', "-- Keep merchant risk scores refreshed every hour\r\nSELECT alloydb_sync.create_bq_sync_table(\r\n 'my-project.risk.merchant_risk_scores',\r\n 'merchant_risk_scores',\r\n INTERVAL '1 hour');"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01eb00d3d0>)])]>

Lakehouse federation (GA)

For organizations with years of analytical history already in BigQuery, Lakehouse federation lets AlloyDB query native BigQuery tables in place and join them with live operational data in a single query. AlloyDB pushes filters, aggregations, and limits down to BigQuery, which does most of the processing and sends back only the results. You can also set which project is billed for these queries and restrict which roles can use them.

Google Cloud customers like Loyal Guru are already pairing AlloyDB and BigQuery in this way.

"AlloyDB, along with other Google Cloud products like BigQuery, provides the agility and performance needed to continually enhance our platform's capabilities and help us anticipate emerging trends rather than merely reacting." - Javi Fernández, Co-founder & CTO, Loyal Guru

Choose the right approach and get started

The convergence of operational and analytical systems is underway; how you choose to approach it in a Google Cloud environment depends on your use case:

Direct Iceberg access: Best for historical data in Iceberg that your app joins with live data, accelerated by the AlloyDB columnar engine.

Try it: Enable direct Iceberg access on a PostgreSQL 18 instance and follow: Access Iceberg data from AlloyDB.

AlloyDB sync: Best for tables your app reads constantly and needs at the lowest latency. The copy is as fresh as its last refresh.

Build: Pick a table in BigQuery and choose to sync to AlloyDB using the Google Cloud console or psql.

Lakehouse federation: Best for history that lives in native BigQuery tables, processed by BigQuery.

Start: Follow Configure AlloyDB access to real-time data in BigQuery.

Read Choose how to access BigQuery data from AlloyDB to learn more.

AlloyDB Express: Instant, scale-to-zero PostgreSQL for developers and coding agents

Wed, 04 Nov 2026 14:00:00 +0000

Databases have struggled to keep up with the speed of AI-enabled software development. Across the engineering and platform teams we talk to, we see coding agents stalled by slow database provisioning, CI/CD test pipelines bottlenecked by complexity, and internal apps hitting a wall as they transition from sandbox to production. Until now, no PostgreSQL service in the market has combined agent-ready capabilities with uncompromised enterprise security and scale.

To solve this, we are excited to announce the preview of AlloyDB Express, a new offering of AlloyDB for PostgreSQL designed specifically for developers and AI coding agents. Capable of spinning up in seconds and automatically pausing when idle, it helps bridge the gap between rapid, AI-driven development and robust enterprise infrastructure. The result is a highly secure, cost-effective database environment that accelerates how you build applications and can easily grow to production scale. Get started with the free tier and pay only for what you use.

AlloyDB Express is an agent-ready database that gives you:

Database creation in under two seconds: Create a ready-to-use PostgreSQL database in under two seconds using gcloud CLI, REST API, MCP, Terraform or Console.
Scale-to-zero: Your database automatically pauses after 15 minutes of inactivity so you do not incur costs for compute while it is idle. When a new query arrives, it wakes up automatically in seconds.
Enterprise-ready with built-in AI and security: Access enterprise AI capabilities including vector, text, and hybrid search to quickly build AI applications, all protected by built-in enterprise security guardrails.
Start for free and pay only for what you use: Get started with our free monthly allowance and as you scale, pay only for what you use.
One-click upgrade to production scale: When your app is ready for production, upgrade your existing database to AlloyDB in one click—keeping data, schemas, users, and connection strings intact while gaining enterprise high availability (HA), read replicas, and disaster recovery (DR).

Accelerate development without compromising security

AlloyDB Express combines self-service speed with built-in enterprise security. For app builders and coding agents this means you can instantly provision fully isolated PostgreSQL databases without manual setup, allowing every test and prototype to run in its own dedicated database. And for platform and security administrators you define IAM, network, and organization policies centrally once, and every AlloyDB Express database created by a developer or coding agent automatically stays within your company's security guardrails.

Whether code is written by developers or AI agents, security remains enforced through built-in protection mechanisms which:

Minimize exposure with safe default connections: Keep your data hidden from the public internet using public DNS endpoints that mask IP addresses and block incoming traffic until explicitly allowed.
Secure internal integrations via private VPC networking: Safely connect your internal workloads using Private Service Connect (PSC) or private DNS.
Enforce least privilege with granular access controls: Confidently restrict agent and user permissions utilizing Google Cloud IAM, VPC Service Controls, and native PostgreSQL roles.

AlloyDB Express brings built-in AI capabilities

Unlike databases that rely on external vector stores or complex setups, AlloyDB Express embeds enterprise AI tools directly into your database, so you can focus on building features. You can accelerate your AI application development, with tools like:

Fast vector search (pgvector HNSW and Google’s ScaNN index): Store and search vector embeddings alongside your operational data for semantic search, recommendation engines, and agent memory (RAG) — without syncing data to a separate vector database.
Direct access to Gemini models within SQL queries: Generate text or image embeddings on the fly and run semantic filtering, ranking, and summarization (ai.if, ai.rank, ai.generate) directly inside standard SELECT and INSERT statements.
Natural language-to-SQL generation: Let developers and AI agents ask questions in plain English to generate, run, and refine queries in seconds using built-in AlloyDB Studio and natural language APIs.

Start with the free tier and pay only for what you use

AlloyDB Express offers simple intuitive pricing and scales to zero when inactive. Data stays safe, and the database wakes up instantly when a new query arrives.Here is a short summary of our pricing:

Active compute: $0.115 per Compute Unit hour (CU-hr), billed by the second.
Paused compute: $0.00 per hour when idle for more than 15 mins.
Used storage: $0.30 per GB-month, billed only for the disk space your data actually uses.

In addition, Google Cloud offers a free tier, making it easier to get started with 50 free CUs every month. For more details see the AlloyDB Express pricing.

Get started today

AlloyDB Express is available today in preview. Try it in the Google Cloud console or read the AlloyDB Express documentation page for more details on how to get started.

Beyond Migration: Using Database Migration Service for disk shrinking

Mon, 02 Nov 2026 17:00:00 +0000

When database engineers hear “database migration", they typically think of a one-time onboarding pathway—moving databases from on-premises environments to the cloud, or between different cloud providers.

However, Google Cloud's Database Migration Service (DMS) supports Cloud SQL for SQL Server as a source. This opens up powerful, non-traditional operational patterns for SQL Server customers within Google Cloud. Because DMS enables continuous, low-downtime database replication between Cloud SQL instances—whether in the same project, across different projects, or across regions—it can serve as an everyday utility for day-two operations like storage management.

In this post, we explore a compelling operational use case for DMS with Cloud SQL for SQL Server: executing disk shrinks with minimal application downtime.

Why should I shrink my disk?

In many production environments, database storage only moves in one direction: up. Features like Cloud SQL automatic storage increase are essential for preventing out-of-space outages during unexpected traffic spikes, large batch imports, index maintenance, or temporary staging loads. Similarly, over the lifecycle of an application, you might archive historical records, purge obsolete data, or drop unused indexes.

When data is deleted, SQL Server marks the freed space inside your data files (.mdf, .ndf) as reusable for future writes; similarly, log truncation frees up space within transaction log files (.ldf). However, the database files themselves continue to hold their peak used space unless AUTO_SHRINK is configured — a setting that is generally not recommended in production due to severe index fragmentation and performance overhead.

With a manual shrink operation on the database files using DBCC SHRINKFILE, you can release that unused space from SQL Server files back to the underlying disk. Even after shrinking the database files, though, the Cloud SQL persistent disk remains at its peak allocated capacity. If you do not expect your database to grow back into that allocated disk space anytime soon, you should consider shrinking the disk itself to reduce ongoing storage costs — especially in scenarios such as:

After major data archiving or purging: You have permanently moved historical data to cold storage or BigQuery and your active dataset is now a fraction of its peak size.

Recovering from one-time storage spikes: A runaway transaction, large data migration, or temporary staging load triggered storage auto-increase, leaving you paying for terabytes of provisioned disk capacity you no longer need.

Post-refactoring or schema optimization: You have split tables out of a monolithic database into separate services, dropped large unused indexes, or applied data compression.

The query below allows you to inspect both the allocated capacity and available free space for each database file fo a database:

code_block: <ListValue: [StructValue([('code', "SELECT \r\n DB_NAME() AS [DatabaseName],\r\n name AS [LogicalName],\r\n type_desc AS [FileType],\r\n -- Allocated size on disk (converted from 8KB pages to MB)\r\n CAST(size / 128.0 AS DECIMAL(18, 2)) AS [AllocatedSize_MB], \r\n -- Actually used space inside the file\r\n CAST(FILEPROPERTY(name, 'SpaceUsed') / 128.0 AS DECIMAL(18, 2)) AS [UsedSpace_MB], \r\n -- Free space left in the allocated file\r\n CAST((size - FILEPROPERTY(name, 'SpaceUsed')) / 128.0 AS DECIMAL(18, 2)) AS [FreeSpace_MB], \r\n -- Percentage of allocated space that is used\r\n CAST((FILEPROPERTY(name, 'SpaceUsed') * 100.0) / size AS DECIMAL(18, 2)) AS [SpaceUsed_Percent]\r\nFROM sys.database_files;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01eaf31fd0>)])]>

You can monitor the following key Cloud SQL metrics in Cloud Monitoring to identify whether your provisioned disk quota significantly exceeds actual disk usage, indicating an opportunity for disk shrinking:

database/disk/bytes_used: Tracks the actual amount of disk space currently consumed by your database files and logs.
database/disk/quota: Represents the total provisioned disk capacity allocated to your instance.

Cloud SQL provides a built-in storage shrink feature to reclaim this unallocated disk space, which works well when your workload can tolerate a planned maintenance window. However, native disk shrinking requires instance downtime proportional to the total allocated disk size — which can mean hours of offline maintenance for multi-terabyte drives.

If your production SLAs cannot accommodate extended downtime, Database Migration Service (DMS) offers a minimal-downtime alternative. By continuously replicating your databases to a newly provisioned Cloud SQL instance with a smaller, right-sized disk, you can reclaim unused storage while your application stays online — requiring only a brief cutover window. To understand why this approach keeps your cutover window so short, let's first look at how DMS replicates data between Cloud SQL for SQL Server instances under the hood.

How DMS works under the hood: Replication between Cloud SQL for SQL Server instances

To leverage DMS effectively, it helps to understand its underlying replication mechanism. When migrating between Cloud SQL for SQL Server instances, DMS coordinates a synchronization process similar to native SQL Server log shipping.

Initial full backup: It captures an initial full backup (.bak) of the selected user databases and restores them on the target instance.
Continuous log backups: It continuously captures and applies transaction log backups (.trn) to keep the target instance synchronized in near-real time.

Because synchronization occurs at the user database level, objects residing outside of it—such as logins, linked servers, and SQL Server Agent jobs—are not automatically replicated to the standby instance. Consequently, server-level objects must be migrated manually.

Disk shrinking with minimal downtime

Using this continuous replication pipeline, you can reclaim unallocated storage without taking a lengthy maintenance window. Instead of shrinking the existing disk in place while the instance is offline, you migrate your workload to a newly provisioned Cloud SQL instance with a right-sized disk while your production application remains online.

The DMS disk shrink workflow

Provision a new target instance with a smaller, appropriately sized disk (leaving sufficient headroom for transaction logs and future data growth).
Start a continuous DMS job using your existing Cloud SQL instance as the source.
Let replication catch up while your production application runs uninterrupted.
Perform a brief cutover: Stop application traffic, let the final transaction log apply, promote the target instance in DMS, and repoint your application.

Comparison: Native shrink vs. DMS migration

As shown in the comparison above, because DMS migrates your workload to a new instance at the user-database level, a smooth cutover requires migrating your server-level objects and user mappings alongside your data. Let's look at the essential pre- and post-cutover steps to include in your runbook.

Pre and post-cutover essentials: Going beyond database replication

Pre-cutover preparation: Migrating server-level objects

Because DMS operates at the user-database level, server-level objects are not automatically replicated. You will need to create them manually in the destination instance before the cutover.

Script out logins from the source using sp_help_revlogin and recreate them on the target instance.

SQL Server Agent jobs & Server-Level objects: Because DMS replicates databases rather than server instance state, objects outside user databases must be scripted from the source and deployed to the destination Cloud SQL instance. In SQL Server Management Studio (SSMS), connect to your source instance, navigate to the object in Object Explorer, right-click and select Script [Object] as > CREATE To > New Query Window / File:
- SQL Server Agent jobs (SQL Server Agent > Jobs): Inspect generated scripts to ensure job steps do not reference on-premises file paths or unsupported OS dependencies.
- Linked servers (Server Objects > Linked Servers): Follow the exact same scripting approach. Ensure you enable the cloudsql.linked_servers database flag on your Cloud SQL instance before deploying.
- Extended events (Management > Extended Events > Sessions): Follow the same scripting approach.
For TDE enabled instances, you will need to export the TDE certificate from source and import the TDE certificate to destination before.
Instance Configurations (such as database flags, Active Directory, Entra ID configurations etc): These must be manually configured on the destination instance to exactly match the source settings.
If using Polybase you will need to re-encrypt DMK for the database-scoped credentials to function correctly.

Post-cutover remediation: Resolving orphaned database users

Database users mapped to SQL Server logins that are either absent or misconfigured on a server instance are unable to authenticate. These are termed orphaned users. Because logins created on the destination instance will possess unique Security Identifiers (SIDs) that do not match the source SIDs preserved in your migrated databases, users will be orphaned by default. Addressing this requires manual re-mapping once the target database is brought online following the cutover.

Detecting & re-mapping orphaned users: While using sp_help_revlogin preserves SIDs during login migration, creating logins on the destination instance via alternative methods generates new server-level SIDs. Because these differ from the source SIDs retained in the DMS-restored user databases, database users become orphaned after cutover and require re-mapping.

1. Detect orphaned users: Connect to each destination user database and run the diagnostic query from Microsoft's official Troubleshoot Orphaned Users Guide:

code_block: <ListValue: [StructValue([('code', "SELECT dp.type_desc, dp.sid, dp.name AS user_name\r\nFROM sys.database_principals AS dp\r\nLEFT JOIN sys.server_principals AS sp ON dp.sid = sp.sid\r\nWHERE sp.sid IS NULL AND dp.authentication_type_desc = 'INSTANCE';"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f04480d0>)])]>

2. Fix orphaned users (ALTER USER): For each unlinked database user returned by the query, execute the following statement inside the context of that database to cleanly re-map the principal and its permissions to the new Cloud SQL server login:

code_block: <ListValue: [StructValue([('code', 'ALTER USER [user_name] WITH LOGIN = [login_name];'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f0448090>)])]>

Summary and next steps

DMS is more than a one-time bridge from on-premises environments into Google Cloud — it is a versatile tool for day-two operations in the Cloud SQL engineer's toolkit. When strict production SLAs prevent you from taking lengthy maintenance windows for a native disk shrink, DMS provides a reliable, managed replication pipeline to right-size your storage footprint and reduce infrastructure costs with minimal downtime.

To get started with configuring Cloud SQL for SQL Server as a DMS source, explore the official documentation. And if you're new to running SQL Server on Google Cloud, check out the Cloud SQL for SQL Server introduction.

Near-zero downtime upgrades: Announcing Cloud SQL for MySQL blue-green deployments

Wed, 28 Oct 2026 16:00:00 +0000

Keeping production databases on supported versions shouldn’t require giving up your weekend. To stay secure, compliant, and with the best performance and newest AI capabilities, organizations need to run on the latest database engine versions. Yet for many teams, major version upgrades mean weeks of cross-team coordination, negotiating narrow maintenance windows, and hours of application downtime. In an era where mission-critical applications and AI agents operate as 24/7 systems of action, even a brief maintenance outage can mean disrupted customer experiences and lost revenue.

To help, we are excited to announce the preview of fully managed blue-green deployments in Cloud SQL, starting with Cloud SQL for MySQL. You can now execute major version upgrades and configuration updates with minimal downtime via the Google Cloud console or automated CLI scripts. By continuously synchronizing a parallel environment, teams can safely test with production data before a seamless cutover — all while keeping the original database as a reliable safety net after the switchover.

Why the upgrade dilemma matters to your business

Putting off major version upgrades (like moving from MySQL 8.0 to 8.4) exposes you to security vulnerabilities and causes you to miss out on performance improvements and the latest AI capabilities. To test changes and avoid in-place upgrade downtime, teams often turn to a blue-green deployment — testing changes on a separate, upgraded staging copy (green) alongside the live database (blue) and switching traffic over once it’s ready. However, manual blue-green setups are incredibly complex to build, synchronize, and safely cut over.

Cloud SQL’s fully managed blue-green deployments can turn your database upgrades from a high-risk weekend project into a predictable, low-downtime operation.

The traditional upgrade	The managed blue-green way
Late-night maintenance: Scheduling hours of offline downtime at 2:00 AM on a Saturday.	Business-hour upgrades: Staging, sync, and testing happen in the background during normal work hours.
Limited testing options: Tests are limited to non-production environments, adding operational risks.	Pre-cutover verification: Safely testing actual applications against real-time production data copy before any traffic shifts.
High-stress rollbacks: If an issue is found, rebuilding or restoring from backups takes hours.	Built-in fallback: Cancel the upgrade and leave your blue instance as is if issues come up, or keep your “old blue” as a manual fallback after cutover.

The technical differentiators

With Cloud SQL, managed blue-green deployments automate the entire lifecycle across three stages:

Blue environment (production): Your current primary Cloud SQL instance, which continues serving live application traffic without interruption.
Green environment (staging): A temporary clone of your blue instance that Cloud SQL upgrades to your target database version and keeps in sync via continuous replication.
Switchover: When you're ready to cut over, Cloud SQL briefly sets the blue instance to read-only, finishes the final data synchronization, and promotes the green instance to your new primary database.

Three notable capabilities make this release a game-changer for enterprise workloads:

Near-zero downtime cutover: The transition from the old version to the new version is incredibly fast. Organizations using Cloud SQL Enterprise Plus experience sub-second downtime for maintenance.
No application changes: When you trigger the switchover, Cloud SQL automatically transfers your existing database IP address, instance name, and backups to the new database. Your developers do not need to rewrite connection strings, modify configuration files, or update client applications.
Safe, real-data testing: You can run real application traffic and queries against the upgraded green environment to verify performance before going live. If you spot an issue, simply delete the deployment before switchover without touching your live database.

Simple to orchestrate, built to scale

Upgrading your database environment can now be initiated in just a few clicks within the console, or via a simple CLI script.

code_block: <ListValue: [StructValue([('code', "# Create the synchronized, upgraded green deployment environment\r\ngcloud sql blue-green-deployments create my-upgrade \\\r\n --source-instance=prod-mysql-db \\\r\n --target-database-version=MYSQL_8_4\r\n\r\n# Check the status and get the test endpoint\r\ngcloud sql blue-green-deployments describe my-upgrade \\\r\n--region=us-central1\r\n\r\n# Connect and test the upgraded copy\r\ngcloud sql connect TARGET_INSTANCE_NAME --user=admin\r\n\r\n# Switch your live traffic to the upgraded database in seconds\r\ngcloud sql blue-green-deployments switchover my-upgrade\r\n\r\n# Clean up the old database when you're ready\r\ngcloud sql blue-green-deployments delete my-upgrade \\\r\n --region=us-central1 \\\r\n --delete-old-source"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f044af50>)])]>

With near-zero downtime switchovers, this feature takes the risk out of database upgrades. By shifting maintenance from a weekend chore to a routine daytime task, your team can stay ahead of security lifecycles, unlock the latest AI-ready engine features, and focus on building features that drive business growth, not managing infrastructure risk.

Try it today

Near-zero downtime database upgrades are available in preview for Cloud SQL for MySQL standalone primary instances, including High Availability configurations, with support for PostgreSQL coming soon.

Get started today by visiting the Cloud SQL console or reviewing the step-by-step documentation.

Introducing the AlloyDB Omni web UI for easier cluster observability

Wed, 28 Oct 2026 16:00:00 +0000

Managing PostgreSQL databases across on-premises, edge, and multi-cloud environments often forces administrators to rely on command-line tools, making it hard to see cluster health and quickly diagnose issues. To solve this, we are announcing the General Availability (GA) of the AlloyDB Omni Web UI for Red Hat and containers. This production-ready, read-only console gives you direct visibility into hardware telemetry, query performance, and configuration flags for AlloyDB Omni so you can troubleshoot workloads quickly and safely without leaving your local or enterprise environment.

What is AlloyDB Omni?

AlloyDB Omni is a downloadable, self-managed edition of AlloyDB that brings enterprise capabilities and full PostgreSQL compatibility to your own infrastructure. Built to run anywhere — across on-premises data centers, edge environments, and multi-cloud deployments — it includes built-in columnar acceleration and integrated AI tooling. With AlloyDB Omni, you can modernize demanding transactional and analytical workloads while retaining complete control over your operating environment.

Key Web UI capabilities at a glance

Cluster overview and profiles: Switch between target connection profiles quickly without using the command line.
System monitoring: View real-time performance telemetry for CPU, memory, and disk I/O to plan capacity and triage infrastructure issues.
Query performance: Identify resource-heavy SQL statements through a visual breakdown of database load, latency bottlenecks, and slow queries.
Columnar Engine diagnostics: Track the in-memory columnar store, caching, and coverage to verify analytical query acceleration.
Parameter and flag auditing: Inspect runtime database parameters centrally to identify configuration drift against your organizational baselines.
AlloyDB AI and ecosystem shortcuts: Inspect vector search and AI workflows, and access quick launchpad shortcuts for ecosystem tools like pgBackRest.

Production-safe diagnostics and profile management

When you triage complex database workloads across bare-metal VMs, containers, or Kubernetes, you need a clear, isolated view of instance health. The Web UI provides a dedicated visual environment for this.

Profile management: Store and toggle between saved connection profiles for local development, staging, or production environments.
Read-only operations: The interface operates strictly in read-only mode, ensuring that diagnostic sessions cannot execute write commands or modify your underlying database configurations.

Dual-layer observability: System and query monitoring

To isolate production performance spikes, you need immediate visibility into both hardware health and SQL statement execution.

System monitoring: The Web UI integrates with standard Prometheus telemetry to show you real-time CPU use, memory allocation, storage I/O, and active connection trends.
Query monitoring: The dashboard automatically ranks execution-heavy SQL statements by total latency and call frequency. You can instantly isolate unindexed or resource-intensive queries to resolve issues faster.

System Insights and Query Insights in AlloyDB Omni Web UI

Real-time Columnar Engine diagnostics

AlloyDB Omni includes an advanced Columnar Engine that organizes frequently accessed data in memory, accelerating analytical queries significantly compared to standard PostgreSQL. The Web UI helps you monitor this feature directly:

Memory footprint tracking: Check real-time columnar memory consumption against your hardware limits.
Columnarization and cache coverage: See exactly which tables and columns are actively cached in memory to confirm your workloads are running as fast as possible.

AlloyDB AI and Columnar Engine Insights in AlloyDB Omni Web UI

Database flag and parameter auditing

Keeping database parameters uniform across distributed environments keeps your operations stable. The Web UI gives you a central catalog to inspect active database flags and runtime settings. You can verify node settings against your organizational policies without connecting to servers via SSH or parsing configuration files manually. If configuration drift is detected, changes should be applied via the authoritative provisioning workflow (e.g., updating Kubernetes CRDs/GitOps manifests, running Ansible playbooks, or using alloydbctl).

Database flags in AlloyDB Omni Web UI

Built-in AlloyDB AI and tooling hub

As you build generative AI applications on top of PostgreSQL, the Web UI gives you direct visibility into vector search workflows and AI model integrations. You can also quickly navigate to documentation and external ecosystem resources, including backup tools like pgBackRest, directly from the console's tooling hub.

Operational scope and prerequisites

We designed the AlloyDB Omni Web UI with clear operational boundaries so you can deploy it easily within enterprise security boundaries and air-gapped environments.

Non-Intrusive Monitoring: The interface operates with minimal permissions out of the box, preserving your existing command-line tools and CI/CD pipelines as the authoritative controls for cluster provisioning and administrative workflows.
Standard monitoring prerequisites: To see complete system hardware metrics, you need standard Prometheus monitoring configured within your host environment.

Get started

Ready to modernize your PostgreSQL workloads and get dedicated observability for your AlloyDB Omni cluster ?

To request access to RPM-based enterprise distributions, advanced tooling, and the Web UI, complete the sign-up form for RPM based deployment of AlloyDB Omni.
To request access to container-based enterprise distributions, advanced tooling, and the Web UI, complete the sign-up form for container-based deployment.

Install the Web UI: Follow the step-by-step installation instructions in the AlloyDB Omni documentation to deploy the AlloyDB Omni Web UI

For deployment using Red Hat RPM: Deploy via native RPM Package
For deployment using Docker or Podman: Deploy via container image

How to upgrade embedding models in AlloyDB AI with zero downtime

Wed, 28 Oct 2026 16:00:00 +0000

Imagine you've just launched a new semantic search feature for product reviews to help customers quickly research products on your site. Users love it. But the next day, you see an announcement for a new vector embedding model promising higher accuracy, better nuance, and multilingual search out of the box. After testing it in your dev environment, you confirm it's a perfect fit. But now you're left with a common dilemma: How do you upgrade the model without disrupting your users?

Upgrading embedding models in production requires re-embedding your entire dataset, rebuilding indexes, and coordinating schema migrations without interrupting user queries. You may encounter several potential problems:

Embedding generation: All existing data must be re-embedded using the new embedding model, which typically requires updating complex pipelines and using dual write strategies while the new and old models run side-by-side for a period of time.
Mathematical incompatibility: Embeddings generated by different models inhabit distinct vector spaces and cannot be compared directly. Application query vectors and stored column vectors must switch models in unison to prevent severe search degradation.
Index rebuild overhead: Vector indexes must be built on the new vectors before similarity search can perform efficiently at scale.
Degraded user experience: Switching embedding models requires multiple DDL changes and other operations that acquire exclusive locks and can stall incoming read and write transactions if not carefully managed.

How AlloyDB AI simplifies vector lifecycle management

In this blog, we’ll highlight how the rich feature set in AlloyDB AI combines to enable zero-downtime embedding model upgrades, so that your users can benefit from the latest model advances without scheduled downtime maintenance. Model endpoint management gives you native in-database access to any embedding model (first party, third party, or even self-hosted), while auto embeddings rapidly generates new embeddings in the background (up to 3000/second) and keeps them up to date as source data changes. Using AlloyDB’s ScaNN index can also significantly reduce the wall clock time of your upgrade with index builds that are 10x faster than open source HNSW, and native PostgreSQL functionality like concurrent index creation, set-returning functions (SRF), and atomic column renames allow transparent blue/green and canary-style rollouts.

Step-by-step migration walkthrough

In this example, we demonstrate migrating a production table ecomm.product_reviews from text-embedding-005 (768-d) to gemini-embedding-001 (3072-d) while serving live traffic.

Step 0: Prerequisites and preflight checks

Verify that the google_ml_integration extension is updated to version 1.5.6 or later (tested on 1.6.1) and confirm model connectivity:

code_block: <ListValue: [StructValue([('code', "-- Check extension version\r\nSELECT extversion \r\nFROM pg_extension \r\nWHERE extname = 'google_ml_integration';\r\n\r\n-- Create extensions\r\nCREATE EXTENSION IF NOT EXISTS vector;\r\nCREATE EXTENSION IF NOT EXISTS alloydb_scann;\r\n\r\n-- Verify remote endpoint and output dimension (3072-d)\r\nSELECT vector_dims(ai.embedding('gemini-embedding-001', 'hello world')::vector);\r\n\r\n-- Inspect existing auto-embedding settings\r\nSELECT * FROM ai.embedding_settings_view;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f049e350>)])]>

Step 1: Add the new vector column

Adding a nullable column without default values updates catalog metadata without creating new versions of each row. This makes the operation nearly instant. Note that the size of your vector column (3072 in this example) must match the number of dimensions output by your embedding model. We also set the lock_timeout to 2 seconds to prevent the operation from blocking critical production traffic while it waits to acquire an exclusive lock. If the operation fails due to a longer wait for an exclusive lock, simply retry.

code_block: <ListValue: [StructValue([('code', "SET lock_timeout = '2s';\r\n\r\nALTER TABLE ecomm.product_reviews \r\nADD COLUMN new_embedding_col vector(3072);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f2ab7750>)])]>

Step 2: Initialize auto-embeddings and run backfill

Next, we initialize embeddings on our new embedding column using gemini-embedding-001.

code_block: <ListValue: [StructValue([('code', "CALL ai.initialize_embeddings(\r\n model_id => 'gemini-embedding-001',\r\n table_name => 'ecomm.product_reviews',\r\n content_column => 'review_text',\r\n embedding_column => 'new_embedding_col',\r\n batch_size => 50,\r\n incremental_refresh_mode => 'manual'\r\n);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f2ab6050>)])]>

Note that there are two modes available in auto embeddings to keep embeddings up to date as source data changes:

Transactional: When you set incremental_refresh_mode to transactional, embeddings are updated immediately when data changes. This keeps your embeddings transactionally consistent with your source data, but it also introduces additional write latency with an inline embedding model inference required for every INSERT/UPDATE operation. Use transactional mode when you want to keep your embedding data transactionally consistent with the source data and your rate of change is low.
Manual: When you set incremental_refresh_mode to manual, AlloyDB tracks which rows have changed but does not immediately update the embedding. This allows you to quickly regenerate embeddings for only the changed rows (not the full dataset) on a schedule that you define using ai.refresh_embeddings(), which uses efficient batching techniques just like the initial backfill operation. Use manual mode when you have a high rate of change on your source data and you can tolerate embedding drift for the period of time defined by your scheduled refresh interval.

We use manual mode in this example to keep writes on the base table performant, while still having full control of how quickly changes are reflected in the embeddings by periodically running ai.refresh_embeddings().

Managing concurrent writer wait times
During backfill, AlloyDB Auto Embeddings processes rows in parallel groups (batch_size × max_parallel_requests). While the process for each micro batch is usually fast, concurrent updates or deletes have to wait for the batch to commit.

To cap the maximum wait times for writers on busy transactional tables, you can lower max_parallel_requests during the backfill session:

code_block: <ListValue: [StructValue([('code', 'SET google_ml_integration.max_parallel_requests = 20;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f2ab5e90>)])]>

This change generally improves the commit time for each micro batch, while increasing the total wall clock time for the backfill operation. Our benchmark results on a 10,000-row table demonstrate the trade-off:

max_parallel_requests	Longest Concurrent Writer Wait	Total Backfill Duration
400 (Default)	5.1 seconds	5.2 seconds
20	0.39 seconds	7.6 seconds (+46%)

Monitoring auto embeddings progress

AlloyDB provides a view to track progress for auto embedding jobs, including state, rows processed, and estimated time remaining. To see how things are progressing, run the query below from a separate session:

code_block: <ListValue: [StructValue([('code', "SELECT status, percent_progress, rows_processed, error_count, estimated_time_remaining\r\nFROM ai.embedding_progress_view \r\nWHERE embedding_column = 'new_embedding_col';"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f2ab6590>)])]>

Step 3: Catch up updated rows

Once the initial backfill operation completes, we can optionally regenerate embeddings for rows that were modified while it was running. When using manual mode, Auto Embeddings automatically tracks which rows were changed so that they can be quickly updated without re-embedding data that is already current. To catch those rows up, simply execute a quick refresh using ai.refresh_embeddings(). You should also run this refresh command on a schedule using pg_cron or an external scheduler to ensure the latest changes continue to be reflected in your embeddings .

code_block: <ListValue: [StructValue([('code', "CALL ai.refresh_embeddings(\r\n table_name => 'ecomm.product_reviews', \r\n embedding_column => 'new_embedding_col', \r\n batch_size => 50\r\n);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f2ab56d0>)])]>

Step 4: Validate vector completeness and retrieval quality

Now that the embeddings are generated and caught up, we’ll ensure completeness and correctness prior to traffic cutover. Note that in a live product environment, you will likely have active INSERT / UPDATE / DELETE traffic that modifies rows between steps 3 and 4. To prevent the results below from being skewed, we added a filter on google_ml_track_stale_embedding = 'false' to include only rows that haven’t been changed since the last invocation of ai.refresh_embeddings(). Note that your table may use a different name for this column, such as google_ml_track_stale_embedding_2, so be sure to update the query accordingly.

code_block: <ListValue: [StructValue([('code', "-- 1. Ensure near-zero missing embeddings\r\nSELECT count(*) AS missing_count \r\nFROM ecomm.product_reviews\r\nWHERE new_embedding_col IS NULL\r\nAND google_ml_track_stale_embedding = 'false';\r\n\r\n-- 2. Spot-check distance against a fresh model invocation (should approach 0)\r\nSELECT max(new_embedding_col <=> ai.embedding('gemini-embedding-001', review_text)::vector) AS max_distance\r\nFROM (SELECT review_text, new_embedding_col FROM ecomm.product_reviews WHERE google_ml_track_stale_embedding = 'false' ORDER BY random() LIMIT 20) sample;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f2ab70d0>)])]>

At this point, you should also validate your application logic using the new embedding model. In internal benchmarks comparing text-embedding-005 and gemini-embedding-001, the top-10 nearest neighbor results overlapped by only 31%. This is a good thing because the shift reflects the higher semantic fidelity of the 3072-d model. However, the new results for recall@10 may impact assumptions in your application, so be sure to conduct offline testing against a set of sample golden queries before routing production traffic to the new embeddings.

Step 5: Build a new ScaNN vector index

Searching across a large number of vectors without an index is a computationally expensive and slow process. To avoid this, we’ll build a ScaNN index to enable industry leading filtered vector search performance, which is 10x faster than HNSW on open source PostgreSQL.

High dimensional models like gemini-embedding-001 require additional memory during index construction compared to lower dimensional models, so in this step we increase maintenance_work_mem to accommodate the additional overhead. We also build the ScaNN index using CONCURRENTLY to avoid blocking concurrent writes:

code_block: <ListValue: [StructValue([('code', "SET maintenance_work_mem = '1GB';\r\n\r\nCREATE INDEX CONCURRENTLY product_reviews_new_embedding_col_scann \r\nON ecomm.product_reviews\r\nUSING scann (new_embedding_col cosine);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f2ab6890>)])]>

Building a ScaNN index is also up to 10x faster than HNSW on open source PostgreSQL. In our tests on 100,000 rows, a 3072-d ScaNN index built in approximately 50 seconds, whereas an HNSW index using halfvec took 7 to 13 minutes.

Be sure to also tune scann.num_leaves_to_search to balance recall and latency. In our tests, increasing num_leaves_to_search from the default value to 25 improved Recall@10 from 0.35 to 0.95 at a search latency of 8.8 ms:

code_block: <ListValue: [StructValue([('code', 'SET scann.num_leaves_to_search = 25;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f2ab65d0>)])]>

Step 6: Cut over search traffic with zero downtime

At this stage, you have two primary options for your application cutover strategy:

Pattern A: Atomic column rename swap — This is a simple approach that supports blue/green deployments with zero application code changes. It does not, however, support canary-style releases or auto embeddings in transactional mode (supports manual mode only).
Pattern B: Custom GUC session variables + unified UNION ALL set-returning function (SRF) — This more advanced approach enables transparent query embedding model selection, live incremental traffic splitting, blue/green deployments, and canary A/B testing, while requiring only one application config change. This technique also supports auto embeddings in both transactional and manual modes.

Pattern A: Atomic column rename

First, we route application query embedding generation through a custom database function to abstract the model from the application. Application queries will use this function instead of directly using ai.embedding():

code_block: <ListValue: [StructValue([('code', "-- Define the query embedding function\r\nCREATE OR REPLACE FUNCTION ecomm.product_reviews_query_embedding(q text)\r\nRETURNS vector\r\nLANGUAGE sql\r\nSTABLE\r\nPARALLEL SAFE\r\nAS $$\r\n SELECT ai.embedding('text-embedding-005', q)::vector;\r\n$$;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f2ab6c10>)])]>

Then, when we’re ready to cut over, we simply execute a stored procedure like the example below to swap the column names and update the function to use the new model.

Create the procedure:

code_block: <ListValue: [StructValue([('code', "CREATE OR REPLACE PROCEDURE swap_embedding_columns(\r\n p_table regclass,\r\n p_live_column name,\r\n p_new_column name,\r\n p_retired_column name,\r\n p_query_schema name,\r\n p_query_function name,\r\n p_new_model_id text\r\n)\r\nLANGUAGE plpgsql AS $$\r\nBEGIN\r\n EXECUTE format('ALTER TABLE %s RENAME COLUMN %I TO %I', p_table, p_live_column, p_retired_column);\r\n EXECUTE format('ALTER TABLE %s RENAME COLUMN %I TO %I', p_table, p_new_column, p_live_column);\r\n EXECUTE format(\r\n 'CREATE OR REPLACE FUNCTION %I.%I(q text) RETURNS vector LANGUAGE sql STABLE AS $f$ SELECT ai.embedding(%L, q)::vector $f$',\r\n p_query_schema, p_query_function, p_new_model_id\r\n );\r\nEND;\r\n$$;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f2ab7c10>)])]>

Then execute the swap:

code_block: <ListValue: [StructValue([('code', "SET lock_timeout = '2s';\r\n\r\nCALL swap_embedding_columns(\r\n p_table => 'ecomm.product_reviews', \r\n p_live_column => 'embedding_col', \r\n p_new_column => 'new_embedding_col', \r\n p_retired_column => 'embedding_col_old', \r\n p_query_schema => 'ecomm',\r\n p_query_function => 'product_reviews_query_embedding', \r\n p_new_model_id => 'gemini-embedding-001'\r\n);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f2ab6150>)])]>

Because the column rename and model switch execute in the same transaction, queries never encounter a model mismatch. If you need to roll back, simply call the procedure with the column arguments reversed.

One advantage to this approach is that there are zero changes required on the application side. Your application code consistently uses the ecomm.product_reviews_query_embedding function to generate an embedding, and the column name used by the application query does not change.

code_block: <ListValue: [StructValue([('code', "WITH query_embedding AS (\r\n SELECT ecomm.product_reviews_query_embedding('product was defective') AS vec\r\n)\r\nSELECT id, review_text\r\nFROM ecomm.product_reviews\r\nORDER BY embedding_col <=> (SELECT vec FROM query_embedding)\r\nLIMIT 10;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f2ab4850>)])]>

While this column rename technique is simple, it has two limitations: First, it applies exclusively to manual mode for auto embeddings. In transactional mode, AlloyDB-managed triggers reference columns by hardcoded names, so renaming a column causes write operations to fail. Second, this pattern only supports blue/green style releases. You don’t have per-query control over which model you use, so you’re unable to use canary-style releases with, for example, 5% of traffic using the new model for testing while 95% of the traffic continues to use the existing model.

Pattern B: Custom GUC + set-returning function for canary A/B testing

To execute canary rollouts, traffic splitting, or instant rollbacks, you can use custom PostgreSQL Custom Grand Unified Configuration (GUC) variables paired with a set-returning function abstraction.

1. Define the table-valued function abstraction layer
Create a database SRF that inspects a session-level GUC variable (app.embedding_model) to dynamically evaluate search queries against either the legacy or the new embedding column.

code_block: <ListValue: [StructValue([('code', "CREATE OR REPLACE FUNCTION ecomm.search_product_reviews(\r\n p_query_vector vector,\r\n p_limit int DEFAULT 10\r\n)\r\nRETURNS TABLE (id bigint, review_text text)\r\nLANGUAGE sql\r\nSTABLE AS $$\r\n (\r\n SELECT id, review_text\r\n FROM ecomm.product_reviews\r\n WHERE current_setting('app.embedding_model', true) = 'text-embedding-005'\r\n ORDER BY embedding_col <=> p_query_vector\r\n LIMIT p_limit\r\n )\r\n UNION ALL\r\n (\r\n SELECT id, review_text\r\n FROM ecomm.product_reviews\r\n WHERE current_setting('app.embedding_model', true) = 'gemini-embedding-001'\r\n ORDER BY new_embedding_col <=> p_query_vector\r\n LIMIT p_limit\r\n );\r\n$$;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f2ab4450>)])]>

2. Standardized application queries
The application queries the unified SRF while generating query vectors via ai.embedding() using the active model parameter:

code_block: <ListValue: [StructValue([('code', "-- Application sets session model (for canary / A/B routing):\r\nSET app.embedding_model = 'gemini-embedding-001';\r\n\r\n-- The rest of the query remains the same:\r\nWITH query_embedding AS (\r\n SELECT ai.embedding(current_setting('app.embedding_model', true), 'product was defective')::vector AS vec\r\n)\r\nSELECT id, review_text \r\nFROM ecomm.search_product_reviews(\r\n p_query_vector => (SELECT vec FROM query_embedding), \r\n p_limit => 10);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f2ab4790>)])]>

3. Execute canary traffic splitting and A/B testing
Application connections or middleware can dynamically control which model processes a query by setting the session variable:

code_block: <ListValue: [StructValue([('code', "-- Route session to legacy model \r\nSET app.embedding_model = 'text-embedding-005';\r\n\r\n-- Route session to new model (Canary / A/B Test)\r\nSET app.embedding_model = 'gemini-embedding-001';"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f2ab5bd0>)])]>

Using this pattern, engineering teams can route a small percentage of application sessions to the new embedding model, validate accuracy and latency metrics, then gradually increase traffic to 100%. Or, they can revert instantly to the prior model by simply altering the session parameter.

A note on connection pooling: When using connection poolers such as PgBouncer in transaction-mode, a plain SET stays on the pooled server connection. This means it can leak to other clients, and it may not always apply to your own subsequent transactions. To mitigate this, you can use SET LOCAL inside an explicit transaction so the setting is cleared at COMMIT or ROLLBACK. You can also prevent this condition entirely by using PgBouncer in session mode, which clears session parameters when a connection is returned to the pool.

Step 7: Retire the legacy model and reclaim storage

Once you have completed the cutover to the new model and you are happy with its functionality and stability, you can decommission the legacy embedding column, index, and auto embedding configs to reclaim disk space.

Drop the auto embedding config and the old index:

code_block: <ListValue: [StructValue([('code', "-- 1. Drop auto embedding configuration on old column \r\n-- NOTE: 'embedding_col_old' assumes Pattern A was followed above. Use 'embedding_col' if you following Pattern B.\r\nCALL ai.drop_embedding_config(table_name => 'ecomm.product_reviews', embedding_column => 'embedding_col_old');\r\n\r\n-- 2. Drop old vector index concurrently (update to match the name of your previous index)\r\nDROP INDEX CONCURRENTLY ecomm.product_reviews_embedding_col_scann_old;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f2ab66d0>)])]>

Reclaiming disk space

Executing ALTER TABLE DROP COLUMN removes the column from PostgreSQL catalog tables, but the underlying TOAST space remains allocated on disk until data pages are rewritten. To make this disk space reusable for future operations, you can set the old vector column to NULL in batches before dropping it.

Note that this operation is a trade-off: It creates new row versions for every row in the table in order to make disk space reusable, which requires a VACUUM to clean up the old row versions. You should determine whether it is a priority for you to immediately free up disk space for reuse on the same table versus allowing the old row versions to naturally age out through the course of normal operations.

code_block: <ListValue: [StructValue([('code', "-- Helper procedure to nullify old vectors in small committed batches\r\nCREATE OR REPLACE PROCEDURE retire_embedding_column(\r\n p_table regclass, \r\n p_column name, \r\n p_batch_rows integer DEFAULT 1000\r\n)\r\nLANGUAGE plpgsql AS $$\r\nDECLARE\r\n n bigint;\r\nBEGIN\r\n LOOP\r\n EXECUTE format(\r\n 'UPDATE %1$s SET %2$I = NULL \r\n WHERE ctid = ANY (ARRAY(SELECT ctid FROM %1$s WHERE %2$I IS NOT NULL LIMIT %3$s))',\r\n p_table, p_column, p_batch_rows\r\n );\r\n GET DIAGNOSTICS n = ROW_COUNT;\r\n EXIT WHEN n = 0;\r\n COMMIT;\r\n END LOOP;\r\nEND;\r\n$$;\r\n\r\n-- Run the batched cleanup outside an explicit transaction block\r\nCALL retire_embedding_column('ecomm.product_reviews', 'embedding_col_old', 1000);\r\n\r\n-- Run VACUUM to place the freed TOAST pages on the table free space map (FSM)\r\nVACUUM (PROCESS_MAIN false) ecomm.product_reviews;\r\n\r\n-- Drop the column definition\r\nSET lock_timeout = '2s';\r\nALTER TABLE ecomm.product_reviews DROP COLUMN embedding_col_old;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f2ab5b10>)])]>

In our tests on 100,000 rows, zeroing out the vectors and vacuuming before dropping the column reclaimed 381 MB of TOAST space for reuse by subsequent model upgrades.

Transactional mode considerations for auto embeddings

If your application uses auto embeddings with incremental_refresh_mode => 'transactional', vectors are generated synchronously inside each write transaction. Upgrading in transactional mode follows the same sequence shown above, with three key differences:

Step 3 is skipped: You don’t need to run ai.refresh_embeddings(), because new writes update both columns automatically.
Column swap technique is not supported: You must use the GUC + SRF abstraction method, which does not impact the triggers managed by auto embeddings.
Write latency increases during migration: While both models are configured, each row write executes two remote embedding calls.

Because gemini-embedding-001 generates a more expressive 3072-dimensional embedding, its remote inference call takes longer. In our tests, we observed the following latency profiles for single-row UPDATE operations with transactional mode enabled for auto embeddings:

Migration Phase	Active Models	Median Single-Row Write Latency
Baseline	`text-embedding-005`	~37.5 ms
Active Migration	Both Models Active	~191.0 ms
Post-Cutover	`gemini-embedding-001`	~172.0 ms

Be sure to review your application's write latency budget before choosing transactional mode for auto embeddings, especially with higher-dimension models.

Disk space and memory resource planning

Because old and new vector columns coexist during migration, your table storage will grow based on peak usage. Based on our testing, we observed the following average storage per row for each embedding type:

Embedding Model	Output Dimensions	Estimated Storage Per Row
`text-embedding-005`	768	~4.5 KB / row
`gemini-embedding-001`	3072	~13.3 KB / row

Based on these estimates, you can use the formula below to estimate your storage space:

New vector column: Total Rows × ~13.3 KB (approx. 3x legacy storage).
New ScaNN index: Total Rows × ~0.6 KB.
Headroom Allocation: Add ~25% overhead for WAL generation and page updates during backfill and vacuum operations.

Example: A table with 10 million rows requires approximately 133 GB of disk space for the new vector column, 6 GB for the ScaNN index, plus headroom.

Summary

AlloyDB combines native PostgreSQL primitives with built-in AI capabilities to make upgrading vector embedding models in production faster and earlier, with no maintenance downtime or complex application-side changes. By combining AlloyDB AI auto embeddings and Model Endpoint Management with standard PostgreSQL operational controls, database administrators and developers can keep their semantic search models up to date with the latest embedding model capabilities as soon as they are released.

To get started, launch an AlloyDB trial cluster and consult the official AlloyDB AI documentation.

Powering the agentic enterprise with latest innovations in Oracle AI Database@Google Cloud

Tue, 27 Oct 2026 19:30:00 +0000

For decades, core enterprise operations — from global supply chains and financial ledgers to billing engines — have relied on the performance and transactional integrity of Oracle databases. Today at Oracle AI World in Las Vegas, Google Cloud and Oracle are expanding our partnership across the AI stack — spanning AI compute and infrastructure (including making Google TPUs available through Oracle Cloud Infrastructure), enterprise data, and agentic AI — by deepening the integration between Oracle AI Database@Google Cloud, Google’s Agentic Data Cloud, and Gemini.

By unifying Oracle’s mission-critical operational data and applications with Google Cloud’s data and AI ecosystem — including BigQuery and Gemini models — enterprises can move beyond static analytics and build autonomous, agentic workflows. This architecture breaks down traditional data silos, giving agents direct, secure, and real-time access to the foundational enterprise data required to drive real-world action.

Foundational building blocks for agentic AI

To unlock production-grade agentic AI, enterprises need more than raw model capability — they need strong database connectivity, unified data governance, and rich semantic context working together as a cohesive foundation. First announced at Google Cloud Next '26, the following core capabilities serve as the building blocks for building conversational intelligence, end-to-end governance, and autonomous agent environments on Oracle AI Database@Google Cloud:

Remote MCP Server with Data API: The Remote Model Context Protocol (MCP) Server for Oracle workloads — powered by the IAM-governed ExecuteSQL Data API — is now generally available. Together, they provide a unified, standardized interface that enables AI agents, Google Cloud services, and enterprise applications to securely run SQL and interact directly with Oracle databases. By resolving database credentials automatically through Secret Manager and enforcing Google Cloud IAM policies, this capability eliminates the need for custom natural-language-to-SQL middleware, bastion VMs, or plaintext passwords.
Knowledge Catalog integration: Google Cloud’s Knowledge Catalog integration, currently in preview, unifies enterprise metadata discovery and governance across Oracle databases and the broader Google Agentic Data Cloud. By automatically cataloging schemas and business metadata, organizations can enforce consistent access controls, track data provenance, and supply the rich semantic context that agents require to reason accurately over complex operational data. Click here to sign-up for this feature.

Gemini connector for Oracle AI Database@Google Cloud

Building directly on the foundational capabilities discussed above, we’re also announcing the preview of Gemini connector for Oracle AI Database@Google Cloud. With this native integration, business users and developers can interact directly with their Oracle databases on Oracle AI Database@Google Cloud using natural language queries right inside the Gemini app. Behind the scenes, Gemini automatically leverages Knowledge Catalog to discover, search, and ground responses strictly in the governed Oracle data each user is authorized to access — turning complex operational databases into an intuitive, conversational experience without compromising enterprise security. Click here to sign-up for this feature.

Oracle in Gemini and Gemini in Oracle

Beyond direct database queries, Google Cloud and Oracle are connecting agentic workflows across the broader application layer — bringing Oracle data into Gemini and embedding Gemini models natively inside Oracle applications:

Oracle in Gemini: Available today, Oracle in Gemini introduces conversational analytics, cross-cloud database connectors, and domain-aware agents — including the new Oracle AI Database Agent in Gemini. Anchored by over 135 pre-built operational and developer skills and standardized MCP connectivity with zero plaintext credentials, enterprises can orchestrate multi-agent workflows across JD Edwards, E-Business Suite, NetSuite, and Oracle Fusion without custom SQL or brittle middleware.
Gemini models in Oracle applications: Concurrently, Gemini frontier reasoning is embedded directly into Oracle Fusion AI Agent Studio, Fusion Claw, Oracle Fusion Applications, and NetSuite — combining Oracle database intelligence with Gemini’s agentic capabilities to automate execution across transactional systems of record.
Partner-ready agentic blueprints: To help organizations rapidly deploy these governed workflows, Google Cloud and Oracle are collaborating with leading partners and global systems integrators — including Argano, Automus, Calfus, HCL, IBM, Infosys, LTM, Pythian, TCS, and Tessell — who provide pre-built operational skills, custom MCP connectors, and industry blueprints for production-grade agentic architectures.

Empowering choice with lakehouse, streaming, and federation

Every enterprise has distinct architectural priorities, latency thresholds, and governance requirements when connecting operational Oracle data to BigQuery and AI workloads. Rather than prescribing a one-size-fits-all integration pattern, Google Cloud and Oracle offer a complete spectrum of data unification choices so customers can select — or combine — the exact approach that best fits their workload and use case:

Bi-directional Google Cloud borderless Lakehouse and Oracle AI Lakehouse

Modern enterprise data architectures are increasingly multi-cloud and grounded in open standards. Building on our strategic partnership, Google Cloud and Oracle delivered a key innovation earlier this year to support cross-cloud data interoperability powered by open Apache Iceberg standard: integrating Google’s borderless Lakehouse with Oracle AI Lakehouse.

This work enabled the borderless Lakehouse to read Apache Iceberg tables created by Oracle AI Lakehouse across any public cloud or on-premises environment, leveraging catalogs and storage supported by both platforms.

Today, we are advancing it further by making it fully bi-directional with Oracle Access to Google Cloud borderless Lakehouse (preview). Now, Oracle Autonomous AI Database and Google’s borderless Lakehouse (powered by BigQuery and Managed Spark) can directly query and execute write/DML operations on shared Apache Iceberg tables in Google Cloud Storage, completely eliminating the need for ETL or data replication.

By utilizing a Lakehouse runtime REST catalog integrated into Oracle via DBMS_CATALOG, this bi-directional integration delivers an open-standards storage framework while empowering analytical teams (using BigQuery and Gemini) and operational teams (using Oracle SQL) to collaborate over a single, unified source of truth. You can click here to sign-up for this feature.

Sub-second CDC streaming with OCI GoldenGate

For organizations that prefer to continuously reflect mission-critical operational data inside BigQuery for high-scale analytics and AI, real-time change data capture (CDC) provides an immediate competitive edge. Through native OCI GoldenGate integration, CDC events are extracted directly from Oracle redo logs without touching table blocks or adding overhead to transactional workloads. Transactions stream securely across Google Cloud private networks straight into BigQuery managed tables, delivering sub-second analytical freshness to power Gemini, feature stores, and live business intelligence dashboards, while also streamlining migrations from on-premises environments to Oracle AI Database@Google Cloud.

BigQuery query federation with Oracle

When workloads call for on-demand operational visibility or infrequent cross-estate joins without moving data, BigQuery query federation with Oracle, in preview, enables real-time, in-place analytics directly against live Oracle databases. Using the EXTERNAL_QUERY function, data teams can execute native Oracle SQL directly from BigQuery queries, dashboards, and pipelines to retrieve live transactional records and join them easily with petabyte-scale BigQuery datasets. This zero-copy approach simplifies operations and lowers Total Cost of Ownership (TCO) by eliminating duplicate storage and ETL maintenance for ad-hoc lookups, intraday risk reporting, and live fraud detection. Please reach out to your Google Cloud account team if you want to sign-up for this feature.

Architectural option	Mechanism	Data Freshness and Latency	Impact on Oracle OLTP	Primary Use Case
Bi-Directional Borderless Lakehouse	Shared Apache Iceberg tables via Lakehouse runtime REST catalog (DBMS_CATALOG) and Glue Federation	Real-time shared storage (zero copy)	Zero (shared analytical storage layer)	Multi-cloud open interoperability, unified BigQuery/Spark/Oracle SQL access without vendor lock-in
Real-Time CDC via OCI GoldenGate	Continuous log-based streaming into BigQuery via Storage Write API	Near real-time / Sub-second (< 1s)	Zero (non-invasive extraction from redo logs)	Mission-critical operational reflection in BigQuery, live AI feature stores, continuous ML and BI pipelines
BigQuery Query Federation (Preview)	In-place query execution using EXTERNAL_QUERY via BigQuery Connection API	On-demand / Live in-place	Variable (query-dependent)	Infrequent ad-hoc querying, zero-copy lookups, cross-estate joins, regulatory reporting

Global expansion and public sector compliance

We continue to expand the global physical footprint and compliance boundaries of Oracle AI Database@Google Cloud to meet customers wherever they operate, with new regions and certifications.

New regions launched

The europe-west12-a(Turin, Italy) and northamerica-south1-b(Queretaro, Mexico) regions are now live, joining global hubs including Milan, Iowa, São Paulo, Tokyo, Sydney, and Mumbai to deliver ultra-low-latency access and local data residency.

FedRAMP High certification

At Oracle AI World, Oracle announced plans to pursue FedRAMP High certification for Oracle AI Database@Google Cloud. This initiative will allow US government and public sector entities to run regulated workloads using Oracle Exadata Database Service on dedicated infrastructure within Google Cloud Assured Workloads.
Operating inside a FedRAMP High-authorized environment enables federal agencies to safeguard sensitive, unclassified data while co-locating Oracle workloads with existing applications and Gemini. Agencies can access live Oracle data with minimal latency, streamlined billing, and simplified procurement via Google Cloud Marketplace or certified partners — all within their compliance boundaries.

Building the agentic future

By combining Oracle’s proven transactional engine with Google’s Agentic Data Cloud, enterprise data transitions from passive systems of record into an active, intelligent operational engine. Whether unifying data through open Apache Iceberg lakehouses, streaming sub-second changes via GoldenGate, federating live queries from BigQuery, or conversing directly with governed Oracle data through Gemini, organizations now have the complete flexibility and security required to deploy autonomous AI agents at scale — driving predictive insights and automated actions across their most critical business processes.

Get started today

Oracle AI Database@Google Cloud is available directly through the Google Cloud Marketplace, allowing you to deploy using your existing Google Cloud commitments, unified billing, and Oracle Support Rewards.

To explore further, visit the Oracle AI Database@Google Cloud overview, dive into the official Oracle Database@Google Cloud Documentation, or reach out to your Google Cloud account team to schedule an architecture briefing.

Managed Apache Iceberg at scale: How Spanner powers Lakehouse runtime catalog

Tue, 06 Oct 2026 16:00:00 +0000

As customers modernize to lakehouse architectures, they are standardizing on open formats such as Apache Iceberg to create a shared data estate across compatible engines. This enables you to build AI-native lakehouses that turn data to semantic knowledge, enable proactive action, and operate at an agentic scale.

One of the key components of a lakehouse is the catalog, and in the Apache Iceberg environment, that usually means the Iceberg REST Catalog. An Apache Iceberg catalog is responsible for maintaining table pointers, handling atomic commits, and serving as the single source of truth for table locations. But as large enterprise organizations modernize to lakehouses, they have begun to realize that they need a highly scalable and available managed catalog as a part of their lakehouse architecture. This becomes even more important as querying scales with agents. To support a large amount of repeated small queries from agents, you will need to build on top of a managed catalog that provides atomicity, consistency, availability and concurrency at massive scale.

In this blog, we explore the challenges a managed catalog faces in modern cloud environments at agent scale, and show you how Google Cloud’s serverless Lakehouse runtime catalog can help address them. Powered by Spanner, Google Cloud’s always-on database with virtually unlimited scale, and built to meet the open Apache Iceberg REST catalog specification, the Lakehouse runtime catalog is the highly scalable and available foundation you need for the agentic era.

Challenges a managed catalog needs to solve in the Lakehouse

When speaking with data engineers and infrastructure leads running production analytics at scale in a Lakehouse, the following core pain points consistently emerge and need to be solved by a managed catalog:

Atomic commits and concurrency control: Iceberg guarantees ACID transactions via optimistic concurrency control (OCC). A managed catalog must implement a bulletproof atomic compare-and-swap (CAS) operation to swap the current metadata pointer.

High availability and operational maintenance: Because queries fail immediately if the catalog is down, a managed catalog becomes a critical Tier-1 service.

Scaling the database backing the catalog: Catalog architects typically face a difficult trade-off when choosing a backing database for table metadata and state. Traditional scale-up relational databases provide SQL and ACID transactions, but hit vertical CPU, memory, storage and connection limits under heavy concurrent read/write loads unless manually sharded which incurs a huge operational overhead; while scale-out database systems are either eventually consistent, hard to manage, not enterprise-ready, or all of the above.

Table maintenance coordination: A catalog alone does not optimize data; you must build and operate ancillary pipelines for compaction, snapshot expiration, manifest rewriting, and orphan file cleanup.

Governance and security: A catalog acts as the security gatekeeper. The catalog must implement and maintain:

Authentication protocols (e.g., OAuth2 token exchange, IAM federation)
Access control down to namespace and table levels
Vended storage credentials (e.g., generating short-lived tokens so query engines don't need broad, direct storage credentials)

Lakehouse runtime catalog

To solve these challenges, we built the Lakehouse runtime catalog (GA) with support for Iceberg Rest Catalog. The Lakehouse runtime catalog is a fully serverless, highly available, and unified metadata registry designed from the ground up to support modern open table formats like Apache Iceberg. By natively implementing the Apache Iceberg REST Catalog specification, the Lakehouse runtime catalog decouples metadata discovery from compute engines, helping ensure multiple Iceberg-compatible engines can access a shared data estate and enabling you to take your workloads to production sooner. We’ve helped many customers streamline the migration of their managed catalogs. For example, Etsy migrated its catalog to Lakehouse runtime catalog, joining data in place to accelerate pipeline queries by 60%.

This approach offers a number of architectural benefits:

Open APIs: Support for Iceberg Rest Catalog enables different teams to use their preferred analytics tools on a single, unified dataset.
Multi-engine interoperability: Once registered, tables are immediately discoverable and queryable across Google Cloud Managed Service for Apache Spark, BigQuery, and open-source engines via standard REST interfaces.
Read/write interoperability for Iceberg tables: Leverage Iceberg-compatible engines such as BigQuery, Managed Spark to write to Iceberg tables registered in the Lakehouse runtime catalog. Customers can also use Managed Spark to write to Iceberg tables in external catalogs.
Fully managed Iceberg storage with enterprise-grade features: Use Google's differentiated infrastructure to run analytics with performance on Iceberg tables. This gives you the benefits of open-source flexibility plus performance, scale, governance, and multimodal processing.
Zero data copy: Table definitions point directly to your existing data in the underlying object store. You do not move, rewrite, or duplicate your underlying data.
Bi-directional catalog federation across clouds: Access data from Databricks Unity, Snowflake Horizon and AWS Glue with support for vended credentials and OIDC token exchange. This lets you bring Google AI directly to your AWS and Azure data.
Secure access using credential vending: The catalog supports multiple authorization mechanisms, letting you choose between credential vending and end-user credentials. This means that you can access tables with modern mechanisms such as credential vending without needing direct access to the files in the underlying object store (Cloud Storage, AWS S3, Azure Blob Storage).
AI-powered context and governance: The Lakehouse runtime catalog integrates directly with Knowledge Catalog and Cloud IAM, allowing you to define trusted context for your agents, and apply table-level security consistently across all compute engines. Get out-of-the-box search, lineage, and insights for Iceberg tables in the catalog.
Atomic commits and concurrency control, high availability and scalability: Backed by Google’s planet-scale infrastructure and Spanner, you get the high availability, concurrency and scale you need for your metadata to scale with your data. Support for Cloud Storage dual-region and multi-region buckets enables failover use cases. It also provides reduced TCO due to serverless and no-ops environments, and scalability for any workload size.

What powers the Lakehouse runtime catalog?

The Lakehouse runtime catalog is a highly available, concurrent and scalable catalog with strong consistency guarantees because it is built on top of Spanner. Unlike traditional scale-up relational databases that hit vertical single-node ceilings, Spanner combines full relational SQL semantics and multi-table ACID transactions with the horizontal scale-out elasticity for both reads and writes of a NoSQL system.

Spanner makes the Lakehouse runtime catalog highly available through Spanner’s regional configurations with up to 99.99% availability. Spanner also delivers out-of-the-box scalability for Lakehouse runtime catalog: As a horizontally scalable database, Spanner does not require manual sharding and scales compute and storage independently and transparently. Spanner dynamically monitors data volume and query load, splitting and redistributing data ranges across nodes. Compute nodes scale dynamically based on CPU utilization and storage thresholds. Spanner automatically detects split-level overload and moves heavy splits away from overloaded nodes. Spanner also lets Lakehouse runtime catalog users eliminate the traditional trade-offs between relational consistency and distributed scalability, delivering Lakehouse runtime catalog’s industry-leading consistency guarantees. Lakehouse transactions are serializable — the order of transactions within the database is the same as the order in which clients observe the transactions to have been committed. This foundation allows Lakehouse users to operate at agent-scale.

Then, to further power agentic use cases, the Lakehouse runtime catalog integrates directly with Knowledge Catalog to easily discover lakehouse Iceberg tables and provide trusted context to agents. Knowledge Catalog leverages an efficient combination of full-text search and native vector search provided by Spanner; this approach enables better recall for search retrieval, pairing lexical keyword searches with semantic embeddings in a single query. Because both index types are built on the identical base dataset, they update with strict, transactional ACID consistency alongside base table DML operations. This removes operational overhead such as managing sync pipelines, and external-vector and full-text search systems. In short, the Lakehouse runtime catalog provides faster time-to-market for your agentic use cases.

Combine analytical and operational workloads

With Google Cloud’s borderless Lakehouse based on Apache Iceberg, you can combine your analytical data with your operational workloads. Use cases span combining data assets from your Lakehouse with OLTP data (from Spanner) for analytics, to low-latency serving applications where your Lakehouse assets are accessible in an operational database such as Spanner, to conversational analytics in first-party and third-party agents.

Below, in an example, you can see the Lakehouse runtime catalog with Spanner in action. Here, we combine analytical data for taxi trips in Manhattan (backed by Apache Iceberg) with operational data for taxi zones in Spanner to find the most congested traffic routes. The example also shows that you can also use a Conversational Analytics Agent to access the same data and get second-order insights.

Find out most congested traffic routes in Manhattan

Modernize to the borderless Lakehouse

Modernizing to Google Cloud’s Lakehouse minimizes data silos across analytics engines and agents, unifies multi-engine governance, provides trusted context to your agents and slashes operational TCO. Leveraging the Spanner-based Lakehouse runtime catalog helps prepare your modern cloud environments to operate at agent-scale. To learn more and get started with a free trial, visit the

Lakehouse web page and learn more about Spanner here.

AlloyDB: A unified database engine for hybrid search

Tue, 06 Oct 2026 07:00:00 +0000

For modern AI and RAG applications, achieving high search relevance requires that you combine at least two techniques: vector search for semantic context, and full-text search (FTS) for keyword precision. While AlloyDB for PostgreSQL supports both of these capabilities, managing them has traditionally required a more hands-on operational approach to ensure peak performance.

The challenge is not executing the searches, but the subsequent fusion of result sets. Merging results from the vector query (distance scores) and the FTS query (relevance scores) requires complex SQL queries or custom code in the application layer. This often means maintaining a separate system for fusion, score normalization, and re-ranking.

This article details how AlloyDB AI's hybrid search eliminates this complexity. We will explore how recent updates allow you to:

Simplify hybrid search: Consolidate complex SQL queries, or multi-step application workflows into a single, high-performance SQL function powered by Reciprocal Rank Fusion (RRF).
Optimize FTS performance: Use the new RUM extension to achieve low-latency relevance ranking and efficient phrase matching by storing word positions directly in the index.
Leverage industry-standard ranking: Utilize the new natively supported BM25 index for superior keyword-based scoring directly out of the box.
Expand search versatility: Execute queries against specialized external clusters, including Elasticsearch, OpenSearch, and Solr, using the new external search Foreign Data Wrapper (FDW) without leaving the AlloyDB environment.

The problem of the multi-step workflow

Before AlloyDB AI's native solution, achieving robust hybrid search was very demanding, especially for developers trying to keep this logic within the database using standard SQL. This approach required multi-step orchestration that was not only difficult to manage and maintain for two sources, but became virtually impossible to scale as additional sources were added. These steps included:

Executing vector search: Run a query using a vector column or vector index (for AlloyDB this can be a ScaNN index) to find top k results, generating vector scores.
Executing the FTS query: Run FTS query e.g. by using a generalized inverted index, or GIN, to find top k results.
Normalizing scores (the brittle step): Write complex SQL logic or custom application code to map both result sets onto a common scale. This logic is prone to breaking when data distributions change.
Performing cross-service joins and re-rankings: Use application memory to perform a complex FULL OUTER JOIN on document IDs, apply a weighted summation of the normalized scores, and finally sort the combined results in cases where the FTS search was done in a separate system than the one used for vector search.

This decentralized approach led to fragile score logic, increased latency, high operational load, and a dependency on application expertise for maintaining search quality.

Simplifying hybrid search architectures with AlloyDB

The key to simplifying this is to adopt RRF, which, because it is inherently rank-based, cleverly bypasses the brittle step of score normalization entirely. RRF consists of two parts:

1. The single source of truth

Instead of a multi-step fetch and join process, you simply call one SQL function, providing your search components as a declarative JSON array:

SQL

code_block: <ListValue: [StructValue([('code', 'SELECT id, score \r\nFROM ai.hybrid_search(\r\n search_inputs => ARRAY[\r\n -- Vector Component (Semantic) with dynamic embedding generation from natural language\r\n $json${\r\n "data_type": "vector",\r\n "limit": 10,\r\n "table_name": "documents",\r\n "key_column": "doc_id",\r\n "vec_column": "embedding",\r\n "distance_operator": "<=>",\r\n "query_vector": "ai.embedding(\'text-embedding-005\', \'alloydb search\')::vector"\r\n }$json$::jsonb,\r\n \r\n -- Text Component (Keyword)\r\n $json${\r\n "data_type": "text",\r\n "limit": 10,\r\n "table_name": "documents",\r\n "key_column": "doc_id",\r\n "text_column": "content",\r\n "query_text_input": "alloydb search"\r\n }$json$::jsonb\r\n ]\r\n);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f044b450>)])]>

(Note: The unified ai.hybrid_search API isn't limited to just two components. It also natively supports external search sources via the new FDW).

2. Database-native orchestration

The hybrid_search() function executes the entire workflow in a single query plan, minimizing overhead and helping ensure transaction consistency. It uses:

Dynamic CTE generation: The function constructs dynamic SQL, creating Common Table Expressions (CTEs) for each component. Each CTE is responsible for calculating the positional rank (ROW_NUMBER()) of its results.
Kernel-level fusion: All ranked component results are immediately combined using a FULL OUTER JOIN based on the document ID.
Final RRF score: The combined ranks are used to calculate the final, unified score using the RRF formula:

By taking this approach, brittle score calculations and application-side joins become unnecessary. While Reciprocal Rank Fusion (RRF) is our current ranking algorithm, we intend to introduce additional merging and ranking options down the road.

The result: Performance and operational simplicity

Moving from complex external workflows to native SQL functions provides immediate, measurable benefits:

Metric	Multi-step application workflow (simulated by manual SQL)	AlloyDB AI-native hybrid_search() UDF
Code complexity	Complex score normalization functions, service calls, and application joins.	Zero external logic; a single, declarative SQL call.
Maintenance	Constant tuning of normalization formulas.	No tuning required; RRF is rank-based and distribution-agnostic.

Enhancing FTS performance: The RUM extension

AlloyDB AI's hybrid_search() function is designed to deliver comprehensive search relevance by combining high-performance vector search with FTS. While AlloyDB's native FTS capabilities are powerful, reliance on the standard PostgreSQL GIN index for the text component can present a bottleneck for advanced operations.

The challenge with GIN indexes is that they do not store the positional information of words. This limitation forces costly table scans to re-analyze content for:

Relevance ranking: Calculating search result scores based on word proximity and frequency
Phrase searching: Finding exact word order, which requires positional information.

Introducing the RUM extension for low-latency FTS

The RUM extension is a powerful index access method based on GIN that directly resolves these performance issues.

RUM's core advantage: Unlike the GIN index (which maps word -> [docID]), the RUM index stores the positional information of each word directly within the index (e.g., word -> [(docID1, [pos])], ...).
Performance benefits: This allows RUM to perform complex operations like ranking and phrase matching predominantly within the index itself, avoiding expensive heap scans. RUM provides significantly faster relevance ranking and efficient phrase and proximity searches.
Great for hybrid search: RUM is a critical complement to vector search (like ScaNN) within a hybrid search framework, as GIN’s potential latency makes it less suitable for a real-time hybrid approach.

The RUM extension is an excellent choice for ranking-heavy or high-concurrency search applications. By storing word positions directly in the index, RUM cuts out the need to re-scan table pages during ranking, giving you fast and sorted results. However, it comes with some trade-offs: slower index builds and a larger disk footprint. If your workload prioritizes quick queries and precise ranking over write speeds and storage density, RUM is an investment that's well worth it.

Measurable gains and integration

Adopting RUM directly bolsters the performance of FTS in AlloyDB, showing considerable gains for raw FTS queries, as well as gains on hybrid search that includes FTS queries. Below are performance statistics showing the speedup between using RUM compared to GIN on various BEIR datasets.

(Performance based on tests using the BEIR Natural Questions benchmark.)

RUM integrates with the specialized <=> distance operator for search and ranking, which is supported for use in hybrid search SQL calls, delivering the best of both worlds. The following codelab shows how to configure and use the RUM index with the Hybrid Search UDF.

Introducing BM25 Index: The modern ranking standard

We recently introduced the BM25 index to CloudSQL and AlloyDB in preview, bringing the industry-standard relevance ranking via the pg_textsearch extension. BM25 uses TF-IDF, which accounts for term frequency saturation and document length normalization, delivering significantly higher precision and quality for keyword-based search queries. By leveraging native BM25 indexes within AlloyDB AI, you can achieve superior ranking accuracy out of the box, so that exact-term matches and critical keywords score well without having to implement external search engines or complex custom scoring logic.

code_block: <ListValue: [StructValue([('code', "-- Install pg_textsearch extension\r\nCREATE EXTENSION pg_textsearch;\r\n\r\n-- Create the native BM25 index on the content column\r\nCREATE INDEX idx_docs_bm25 \r\nON cymbal_products \r\nUSING bm25 (product_description) \r\nWITH (text_config='english');\r\n\r\n-- Full text search query\r\nSELECT product_name, product_description <@> 'cherry tree' AS bm25_score \r\nFROM cymbal_products\r\nORDER BY bm25_score \r\nLIMIT 5;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f07e3250>)])]>

External search: Extending versatility with foreign data wrappers

To further extend search capabilities, we also introduced the external search Foreign Data Wrapper (FDW) in AlloyDB AI. This lets you execute full-text searches against specialized external clusters, starting with Elasticsearch, OpenSearch, and Solr, and provides several key architectural advantages:

Optimized retrieval: Leverage ranking algorithms and a richer feature set from dedicated search backends without leaving the AlloyDB environment.
Unified SQL interface: Interact with external data, perform joins, and meld results using standard PostgreSQL SQL without losing the expressiveness of advanced FTS queries.
Strong portability: Maintain existing search infrastructure while benefiting from the simplified hybrid architecture offered by AlloyDB AI.

The following codelabs are end-to-end code guides walking through hybrid search with Elasticsearch and Solr integrations.

A unified architecture for AI-powered search

The true challenge of modern search is not technical in nature, but in achieving architectural simplicity and sustained operational stability. AlloyDB AI addresses this with a unified platform built on three core innovations:

Simplifying hybrid search: AlloyDB AI’s hybrid search function, powered by RRF, transforms a fragile, multi-step application workflow into a single, high-performance SQL call. This native implementation eliminates the need for complex score normalization and application-side joins, drastically lowering the operational and engineering cost while delivering consistently accurate and fast hybrid relevant results.
Optimizing FTS performance: To ensure the FTS component of Hybrid Search meets diverse application demands, AlloyDB AI offers distinct full-text search options. The RUM extension optimizes for low-latency performance by storing positional data directly in the index to enable faster relevance ranking and efficient phrase searches — important when you need fast query speeds. Alternatively, the BM25 index provides industry-standard relevance ranking, and is the preferred choice when prioritizing keyword scoring precision. You can select between these options depending on whether your focus is search latency or ranking precision.
Enhancing versatility through external search: The addition of the external search through FDW extends AlloyDB’s reach to specialized search backends like Elasticsearch. This lets you leverage the superior scale and advanced retrieval features of dedicated search clusters while maintaining a familiar PostgreSQL interface. By integrating these external results directly into the hybrid search framework, AlloyDB AI ensures that you can combine even the most massive text repositories with vector-based semantic insights.

By consolidating the complexity of scoring, joining, and re-ranking within the database kernel, and offering a dual-path approach to FTS — leveraging RUM for optimized, low-latency, internal FTS or external search for specialized, scalable backends — AlloyDB AI delivers a robust and self-contained search foundation. This coexistence is essential to the overall hybrid search story, providing the flexibility to choose the optimal path based on workload, data volume, and existing infrastructure. Now, you can focus on building intelligent application features, confident that your search architecture is both highly performant and easy to maintain.

Watch it in action

Watch how AlloyDB is the ultimate hybrid search engine.

Learn about BM25 support in AlloyDB.

Getting started

Ready to bring more speed and cost-efficiency to your AI workloads?

New to AlloyDB? Discover AlloyDB with a 30-day free trial.
Getting started with hybrid search: Set up a text-search index and select a vector search index. Once these are created, you can find some examples to perform various hybrid search queries.
External search: Create a foreign data wrapper and foreign table in AlloyDB to query external data from Elasticsearch, Solr, or OpenSearch.

Announcing Spanner queues: Transactional messaging for agentic workloads and beyond

Fri, 02 Oct 2026 16:00:00 +0000

AI agents don't just answer queries — they can autonomously issue refunds, manage inventory, execute multi-step handoffs, and orchestrate sub-agents, to name but a few complex agentic workflows. That frequently requires the agents to maintain internal state in an operational database while dispatching asynchronous actions through a separate messaging or event queue system. And unfortunately for the teams building these applications, managing two systems with disjointed commit points destroys transactional consistency in agentic systems.

Today, we are excited to announce the general availability of Spanner queues: native transactional messaging embedded directly within Spanner. Designed specifically for reliable agentic execution, with Spanner queues, creating a message is simply another write in your transaction. An agent's state change and intended downstream actions commit together atomically or fail completely.

Compare this to traditional approaches: When a database state update succeeds, but the action dispatch fails, your AI agent decides to act but doesn’t execute. If the message dispatch succeeds but the state transaction rolls back, your agent executes an action based on an invalid state. In asynchronous multi-agent coordination, retries, speculation, and race conditions amplify these failures, forcing developers to build complex outbox patterns, idempotency layers, and reconciliation workers — a heavy reliability tax on agentic architecture.

Key capabilities for agentic architectures

Spanner queues introduces several core capabilities to support these complex asynchronous workflows without introducing infrastructure overhead.

Atomic decide-and-act enqueue. Within a single Spanner read-write transaction, agents can update internal memory or state tables and enqueue tasks to peer agents simultaneously. Backed by Spanner's strict serializability and global external consistency, state changes and execution intent commit as a single atomic unit.

Scheduled execution and delays. Queue messages can be dispatched immediately upon commit or scheduled for future delivery. Agentic patterns such as delayed retries, scheduled agent check-ins, or SLA escalation timers can be enqueued transactionally alongside memory updates without requiring external cron schedulers or polling infrastructure.

Streaming SQL pull for agent workers. Autonomous agents consume tasks dynamically using streaming SQL reads. Agent runtimes stream incoming tasks, process them as capacity becomes available, and acknowledge task completion within a transaction to guarantee end-to-end task execution state reliability.

Episodic memory persistence and handoffs. Agent memory updates — including long-term episodic summaries, reflective state transitions, and context handoffs across sub-agents — can be persisted asynchronously and transactionally via queues. This guarantees that an agent's internal memory remains fully synchronized with its execution history without blocking real-time interactive turns.

Why Spanner queues are essential for autonomous agents

Transactional exactly-once agent execution: When an agent evaluates tool call results, state modification, reasoning persistence, and downstream tool invocation tasks are saved in one transaction. We guarantee at-least-once delivery and at-most-once ACK, enabling you to achieve exactly-once processing.

Robust multi-agent orchestration and handoffs: In multi-agent systems (A2A), handing off state from a primary agent to a specialist agent is represented as a durable, transactionally committed message. Specialist agent workers receive deliverable messages while maintaining a fully auditable lineage of agent interactions.

First-class timeouts and human-in-the-loop workflows: Agent workflows frequently require pausing for human approvals or scheduled follow-ups. Spanner queues handles timeout management natively: A single transaction records the pending approval state and schedules an automated escalation message, resolving whichever triggers first.

SQL-native observability for agent queues: Inspecting in-flight agent workloads, monitoring task backlogs, or auditing agent execution history can be accomplished with standard SQL queries over queue tables, avoiding opaque message-store black boxes.

Beyond agentic workflows

Beyond AI agents, Spanner queues serves as a flexible, multi-purpose messaging platform across a variety of traditional event-driven architectures. Whether powering real-time activity feeds in social applications, delivering live updates in news publishing, orchestrating order processing and inventory workflows in retail, or handling high-throughput asynchronous task processing and transaction notifications in financial services, Spanner queues provides a robust foundation for asynchronous message delivery and transactional event-driven workflows within your primary database.

Under the hood: Transactional mechanics of Spanner queues

Because Spanner queues are represented as first-class relational structures in Spanner, you define, inspect, and manage queues using familiar GoogleSQL.

1. Defining a queue and enqueuing atomically

When an agent decides to approve a customer refund, it updates the Orders table and dispatches an execution task to the OrderAgentTasks queue within a single ACID transaction:

code_block: <ListValue: [StructValue([('code', '-- Assume parent table:\r\n-- CREATE TABLE Orders (\r\n-- OrderId STRING(64) NOT NULL, ...\r\n-- )\r\n-- PRIMARY KEY (OrderId);\r\n\r\n-- Define the transactional queue table\r\nCREATE QUEUE OrderAgentTasks (\r\n OrderId STRING(64) NOT NULL,\r\n TaskId STRING(64) NOT NULL,\r\n TaskType STRING(64) NOT NULL,\r\n Payload JSON NOT NULL\r\n) PRIMARY KEY (OrderId, TaskId, TaskType), INTERLEAVE IN Orders;\r\n\r\n-- Inside a Read-Write Transaction:\r\n-- 1. Update business state atomically\r\n\r\n-- BEGIN TRANSACTION;\r\n\r\nUPDATE Orders\r\nSET Status = \'REFUND_APPROVED\',\r\n UpdatedAt = PENDING_COMMIT_TIMESTAMP()\r\nWHERE OrderId = @orderId;\r\n\r\n-- 2. Enqueue the asynchronous agent action in the same transaction\r\nINSERT INTO OrderAgentTasks (OrderId, TaskId, TaskType, Payload)\r\nVALUES (\r\n @orderId,\r\n @taskId,\r\n \'EXECUTE_REFUND\',\r\n JSON \'{"action": "execute_refund", "amount": 49.99}\'\r\n);\r\n\r\n-- COMMIT;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f00cc650>)])]>

This ensures that the EXECUTE_REFUND task exists if and only if the order status successfully transitioned to REFUND_APPROVED.

2. Temporal scheduling and atomic cancellation

For workflows that depend on time — such as waiting up to 72 hours for a manager's approval before escalating — agents populate the system DeliverTime column to defer message visibility:

code_block: <ListValue: [StructValue([('code', '-- Schedule an automated escalation check-in 72 hours in the future\r\nINSERT INTO OrderAgentTasks (OrderId, TaskId, TaskType, Payload, DeliverTime)\r\nVALUES (\r\n @orderId,\r\n @escalationTaskId,\r\n \'ESCALATE_UNAPPROVED_ORDER\',\r\n JSON \'{"action": "escalate_to_supervisor"}\',\r\n TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 72 HOUR)\r\n);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01ebe16810>)])]>

If the manager approves the request after four hours, your application doesn't have to deal with phantom escalation alerts firing days later. In a single transaction, you update the order status and cancel the pending escalation task using a standard SQL DELETE:

code_block: <ListValue: [StructValue([('code', "-- BEGIN TRANSACTION;\r\n\r\nUPDATE Orders\r\nSET Status = 'MANAGER_APPROVED',\r\n ApprovedBy = @managerId\r\nWHERE OrderId = @orderId;\r\n\r\n-- Atomically cancel the pending delayed escalation task. `ASSERT_ROWS_MODIFIED 1` will\r\n-- act as a safeguard and cause a statement level error.\r\n\r\n-- A statement level error can be captured and the transaction can be\r\n-- user-aborted/cancelled in case the queue entry was already deleted.\r\n-- Otherwise the transaction will complete successfully regardless if the \r\n-- queue entry still exists and you'll only know if a queue entry was deleted by\r\n-- checking the number of rows affected by the DELETE.\r\nDELETE FROM OrderAgentTasks\r\nWHERE OrderId = @orderId\r\n AND TaskId = @escalationTaskId\r\n AND TaskType = 'ESCALATE_UNAPPROVED_ORDER',\r\nASSERT_ROWS_MODIFIED 1;\r\n\r\n-- COMMIT;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01ebe16e10>)])]>

3. Streaming consumption, lease renewal, and atomic acknowledgment

Downstream agent workers consume tasks using the RECEIVE_<QueueName> table-valued function (TVF) over a streaming SQL connection (ExecuteStreamingSql). Spanner automatically manages message leases, returning a unique SpannerLeaseToken and expiration timestamp with each leased task:

code_block: <ListValue: [StructValue([('code', "-- Step 1: Consume tasks via a long-lived streaming SQL query\r\nSELECT\r\n OrderId,\r\n TaskId,\r\n TaskType,\r\n Payload,\r\n SpannerLeaseToken,\r\n SpannerLeaseExpirationTimestamp\r\nFROM RECEIVE_OrderAgentTasks(max_duration => '20m');"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01ebe17f10>)])]>

Because AI agent tasks often involve multi-turn LLM reasoning or external API calls that take longer than default lease windows, workers can actively extend their lease using the RENEWLEASE_<QueueName> function:

code_block: <ListValue: [StructValue([('code', '-- Step 2: Extend lease ownership during long-running agent execution\r\nSELECT *\r\nFROM RENEWLEASE_OrderAgentTasks(lease_tokens => [@leaseToken]);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01ebe14410>)])]>

When the agent finishes executing its external tool (passing TaskId as the external API's idempotency key), it opens a read-write transaction to record the final state and acknowledge the message by deleting it with ASSERT_ROWS_MODIFIED 1:

code_block: <ListValue: [StructValue([('code', "-- Step 3: Atomically checkpoint agent results and ACK the message\r\n-- BEGIN TRANSACTION;\r\n\r\nUPDATE Orders\r\nSET RefundTransactionId = @externalRefundId,\r\n Status = 'REFUND_COMPLETED'\r\nWHERE OrderId = @orderId;\r\n\r\nDELETE FROM OrderAgentTasks\r\nWHERE OrderId = @orderId\r\n AND TaskId = @taskId\r\n AND TaskType = @taskType\r\nASSERT_ROWS_MODIFIED 1;\r\n\r\n-- COMMIT;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01ebe15a50>)])]>

Using ASSERT_ROWS_MODIFIED 1 protects your system against lease-expiration races. If a worker stalled due to a network pause and its lease expired, another worker may have already processed and deleted the task. When the stalled worker resumes and attempts to execute the statement, ASSERT_ROWS_MODIFIED 1 detects that the queue row is already gone and throws a statement-level error. Catching that error and aborting that transaction prevents stale workers from overwriting newer database state.

Spanner change streams vs. Spanner queues

Spanner change streams capture database data changes (inserts, updates, and deletes) in near real-time for downstream integration and auditing. While both change streams and queues allow applications to react to data changes, Spanner change streams are designed for continuous change data capture (CDC) and data streaming to downstream analytics or storage. In contrast, Spanner queues are explicitly designed for transactional task orchestration, supporting native message leases, scheduled deliveries, SQL-based pulling, and atomic acknowledgments within read-write transactions.

Get started

Spanner provides a unified foundation for agentic data, combining relational, hybrid search, graph, and key-value capabilities under strict global consistency. Spanner queues completes the agentic loop by enabling agents to transition seamlessly from reasoning over data to executing transactional actions within a single unified platform. Spanner queues are now generally available.

Sign up for the Spanner 90-day free trial and read up our public documentation to start building resilient, exactly-once agentic workloads by creating a queue table in your Spanner database today.

How to implement long-term AI agent memory in AlloyDB and Memorystore for Valkey

Fri, 02 Oct 2026 07:00:00 +0000

Enterprise AI agents need persistent memory to execute complex, multi-day workflows and long-horizon tasks. In this blog, we examine how a 2-tier memory architecture using Memorystore for Valkey for short-term buffer memory and AlloyDB AI for long-term persistent memory can help reduce token spend by up to 70%, while maintaining critical data and enterprise guardrails.

Imagine building a personalized travel agent designed to help users book vacations. The user interacts with the agent many times over the course of several days, asking questions that range from brainstorming itineraries to actual purchase intent. To provide a truly seamless experience, this agent must remember flight preferences (e.g. “I only want non-stop flights”), hotel budgets, and dietary restrictions (e.g. “I need Gluten Free dining options”) established in previous sessions. More importantly, it has to hold onto these core facts even when the conversation gets deep into the weeds of sightseeing recommendations and itinerary planning. The agent must ensure that all the follow-up questions and exciting details about places to visit doesn’t cause it to forget or overwrite the user's fundamental requirements and decisions.

Stateful workflows inherently conflict with LLM’s stateless nature

AI agents are being used for multi-turn conversations and long-running workflows, but large language models (LLMs) remain stateless across sessions. When a user returns to an agent days later, the model starts with an empty context window. Unless your application is built to reconstruct past context using long-term agent memory, your users have to explain their goals and context all over again, resulting in a frustrating and fragmented experience.

With million-token context windows now the norm, a common shortcut to this problem is "context stuffing" - dumping everything you can fit into the prompt at every turn, from raw chat histories to tool execution logs. In fact, this was a common pattern in the early days of AI model usage. But this shortcut quickly creates issues at scale: token costs multiply with every message, response times can drag out past 30+ seconds for otherwise simple prompts, and the model starts suffering from "lost in the middle" degradation, overlooking critical instructions buried in mountains of prompt text.

Another common workaround is to use rolling summaries; asking an LLM to periodically compress older messages into a summary paragraph. While this trims prompt size, LLM-based summarization is inherently lossy. After a few rounds of compression, subtle but important details get filtered out as background noise. A few turns later, your agent quietly breaks the exact constraints you set earlier. Clearly, you need a more scalable approach for keeping a memory from previous conversations or multi-turn tasks, but without degrading the experience or creating new bottlenecks.

Implementing a 2-tier memory architecture

To build reliable, cost-effective enterprise agents that respect your guardrails and constraints, we recommend a 2-tier memory architecture:

Short-term session buffer: Caches active conversation turns in memory using a token-bounded sliding window, allowing the context window size to remain stable across multiple rounds, even across devices, while keeping the latest messages fresh. This tier requires sub-millisecond, high-throughput lookups on every turn, making Memorystore for Valkey well suited for maintaining active session state.
Long-term persistent memory: Stores important facts, user preferences, and episodic facts across sessions. This tier requires transactional integrity, data governance, and hybrid retrieval across relational data and vectors - capabilities provided natively by AlloyDB AI.

While it’s possible to store both long-term memory and active session buffers directly in a relational database, a two-tier architecture with an in-memory cache provides better performance and scalability. Active session buffers are highly ephemeral and require sub-millisecond, high-throughput updates on every single conversation turn. Handling these rapid-fire writes in an in-memory key-value cache prevents write amplification and table bloat in your relational database, which would otherwise require frequent row deletions and intensive vacuuming. This is analogous to adding a caching tier in front of your database to offload high-frequency lookups for hot rows. The division of labor keeps your primary database lean and responsive, allowing it to focus its resources on what it is designed for: transactional consistency, complex hybrid vector search, and long-term analytical query execution.

By pairing short-term caching in Memorystore for Valkey with native AlloyDB AI capabilities, you can run entity extraction, memory compaction, cross-session memory, and hybrid retrieval directly inside the database tier, keeping active prompts lean, fast, and cost-efficient.

aside_block: <ListValue: [StructValue([('title', 'Get started with a 30-day AlloyDB free trial instance'), ('body', <wagtail.rich_text.RichText object at 0x7f01ebe17090>), ('btn_text', ''), ('href', ''), ('image', None)])]>

Understanding the four memory types

To organize long-term agent state effectively, we further divide agent memory into four complementary types across the short-term and long-term storage tiers we described above:

Memory type	What it stores	Storage layer	Lifespan
Buffer (Short-term)	Recent raw conversation turns	Memorystore for Valkey	Active session
Summary memory	Compressed history of older turns, commonly referred to as “compaction”	Memorystore for Valkey	Multi-turn window
Episodic memory	Past actions, events, and tool outputs	AlloyDB for PostgreSQL (Hybrid retrieval with structured SQL + full-text search + vector)	Permanent
Entity & rule memory	User preferences, constraints, and vetoes	AlloyDB for PostgreSQL (Hybrid retrieval with structured SQL + full-text search + vector)	Permanent

By isolating short-term conversation context from structured long-term rules, your agent retrieves relevant context on demand without filling token windows with raw interaction logs.

The tiered memory architectural blueprint

The diagram below outlines the read and write paths connecting the application orchestration layer, the Memorystore for Valkey short-term buffer, and the AlloyDB AI long-term repository:

The system operates across two coordinated execution paths:

The read path: When a user asks a question, the application fetches the active sliding window from Memorystore for Valkey, runs in-database query normalization using ai.generate, and queries AlloyDB using ai.hybrid_search to retrieve scoped entity rules and relevant episodic facts.
The write path: After generating the response, the turn is immediately cached in Memorystore for Valkey. An asynchronous background queue worker extracts structured entities from the exchange and writes them directly into AlloyDB, where transactional auto-embeddings immediately compute and store vector representations in the database.

Business impact and ROI

In our benchmark testing across multi-turn development dialogues (45+ turns with heavy tool executions), separating short-term caching from long-term persistence delivered measurable cost and performance improvements compared to naive context stuffing:

Metric / dimension	Naive context stuffing	Tiered memory (AlloyDB + Valkey)	Net business impact
Active prompt size (turn 45)	747,033 tokens	83,262 tokens	88.9% smaller prompt
Turn 45 response latency	33.5 seconds	6.7 seconds	80.0% faster response
Per-turn response wait time	33.5 seconds	4.2s – 6.7s	36% to 80% reduction
Cumulative session tokens	17.9M tokens	4.09M tokens	72.0% token & cost savings
Rule & constraint recall	Degrades over turns	Does not degrade	ACID-preserved recall

Note: The metrics above reflect internal benchmark results from a simulated developer workload that mimics a real-world enterprise AI pair-programming assistant interacting with a developer over multiple sessions, projects, and context switches. Actual savings and latencies vary based on prompt structure, query frequency, and data volume.

These results demonstrate how tiered memory changes agent unit economics: instead of an escalating cost curve on every additional turn, prompt sizes remain bounded, reducing ongoing LLM API expenses while keeping response times fast.

Core AlloyDB AI technical advantages

AlloyDB AI reduces the operational overhead of implementing persistent agent memory by embedding core AI functions directly into the database engine:

Transactional in-database auto-embeddings (ai.initialize_embeddings): AlloyDB automatically generates vector embeddings for text columns using a native integration with Agent Platform (formerly Vertex AI), generating up to 3,000 embeddings per second. Using incremental_refresh_mode => 'transactional', AlloyDB keeps embeddings up to date as source data changes within the same transaction, removing the need for custom embedding pipelines, external schedulers, or complex retry logic.
In-database generative AI functions (ai.generate): AlloyDB allows you to execute foundation models, such as Gemini, directly from SQL queries. You can use this for in-database query decomposition - breaking compound user questions into single-aspect sub-queries and resolving relative time phrases (like "last session") into explicit identifiers - without making separate roundtrips from your application.
Native hybrid search with built-in Reciprocal Rank Fusion (ai.hybrid_search): AlloyDB provides a built-in SQL function that executes Reciprocal Rank Fusion (RRF) directly inside the engine. It combines vector cosine similarity (<=> over HNSW or ScaNN indexes) with PostgreSQL full-text search (using BM25, RUM, or GIN) in a single database call, blending semantic matching with exact keyword retrieval while supporting metadata filter pushdown (filter_condition) for improved performance and deterministic scope isolation.
Direct Agent Platform integration with IAM credentials: AlloyDB connects directly to Agent Platform foundation models over Google Cloud's private network using Google Cloud IAM service account roles and database authentication, avoiding the need to store, rotate, or pass API keys in application code.
Unified operational, vector, and governance engine: AlloyDB consolidates relational business data, vector embeddings, full-text indexes, and enterprise permissions in a single ACID-compliant PostgreSQL database, avoiding data drift and integration complexity across separate operational and vector databases.

Key implementation patterns

Below are the core database patterns used to configure the 2-tier memory architecture. For complete, runnable Python and SQL scripts, refer to the companion AlloyDB Agent Memory Codelab.

1. Setting up schema and auto-embeddings

In AlloyDB, install the necessary extensions and define the agent_entities table with structured metadata, a generated tsvector column for full-text search, and a vector embedding column.

code_block: <ListValue: [StructValue([('code', "-- 1. Enable AI extensions\r\nCREATE EXTENSION IF NOT EXISTS google_ml_integration CASCADE;\r\nCREATE EXTENSION IF NOT EXISTS vector CASCADE;\r\nCREATE EXTENSION IF NOT EXISTS rum CASCADE;\r\n\r\n-- 2. Create structured entity & preference table\r\nCREATE TABLE IF NOT EXISTS agent_entities (\r\n entity_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),\r\n user_id TEXT NOT NULL,\r\n entity_name TEXT NOT NULL,\r\n project_id TEXT DEFAULT 'global',\r\n session_id TEXT,\r\n scope TEXT NOT NULL DEFAULT 'session',\r\n summary TEXT NOT NULL,\r\n summary_embedding VECTOR(768),\r\n summary_tsv TSVECTOR GENERATED ALWAYS AS (\r\n to_tsvector('english', entity_name || ' ' || summary)\r\n ) STORED,\r\n updated_at TIMESTAMPTZ DEFAULT NOW(),\r\n UNIQUE (user_id, entity_name)\r\n);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f00e9d10>)])]>

Then, generate embeddings using ai.initialize_embeddings. Using transactional mode ensures the embeddings are kept up to date as source data changes.

code_block: <ListValue: [StructValue([('code', "-- Register transactional in-database auto-embedding\r\nCALL ai.initialize_embeddings(\r\n model_id => 'text-embedding-005',\r\n table_name => 'agent_entities',\r\n content_column => 'summary',\r\n embedding_column => 'summary_embedding',\r\n incremental_refresh_mode => 'transactional',\r\n batch_size => 10\r\n);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f00e8d50>)])]>

Finally, create the HNSW vector index and the RUM full-text search index to ensure your hybrid searches are fast and efficient.

code_block: <ListValue: [StructValue([('code', '-- Create vector and full-text indexes\r\nCREATE INDEX IF NOT EXISTS agent_entities_embedding_idx\r\n ON agent_entities USING hnsw (summary_embedding vector_cosine_ops);\r\n\r\nCREATE INDEX IF NOT EXISTS agent_entities_tsv_idx\r\n ON agent_entities USING rum (summary_tsv rum_tsvector_ops);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f0a9dd10>)])]>

2. Querying long-term memory with native hybrid search

On the read path, retrieve relevant long-term entities using ai.hybrid_search. This native SQL function executes Reciprocal Rank Fusion (RRF) directly in AlloyDB, seamlessly reranking and combining vector similarity search and full-text keyword search results in a single database query:

code_block: <ListValue: [StructValue([('code', 'def retrieve_hybrid_entities(\r\n conn,\r\n user_id: str,\r\n project_id: str,\r\n query_text: str,\r\n query_vector_literal: str,\r\n limit: int = 3\r\n) -> dict[str, Any]:\r\n """Queries long-term memory using AlloyDB native hybrid_search (RRF)."""\r\n filter_cond = f"user_id = \'{user_id}\' AND (project_id = \'{project_id}\' OR scope = \'global\')"\r\n \r\n search_inputs = [\r\n json.dumps({\r\n "data_type": "vector",\r\n "weight": 0.4,\r\n "table_name": "agent_entities",\r\n "key_column": "entity_name",\r\n "vec_column": "summary_embedding",\r\n "distance_operator": "<=>",\r\n "limit": 10,\r\n "query_vector": query_vector_literal,\r\n "filter_condition": filter_cond\r\n }),\r\n json.dumps({\r\n "data_type": "text",\r\n "weight": 0.6,\r\n "table_name": "agent_entities",\r\n "key_column": "entity_name",\r\n "text_column": "summary_tsv",\r\n "limit": 10,\r\n "ranking_function": "ts_rank",\r\n "query_text_input": query_text,\r\n "filter_condition": filter_cond\r\n })\r\n ]\r\n\r\n query_sql = """\r\n SELECT e.entity_name, e.summary, e.project_id, e.scope, e.updated_at\r\n FROM ai.hybrid_search(\r\n search_inputs => %s::JSONB[],\r\n include_json_output => false\r\n ) h\r\n JOIN agent_entities e ON e.entity_name = h.id\r\n WHERE e.user_id = %s\r\n LIMIT %s;\r\n """'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01eb00da50>)])]>

3. In-database memory compaction with AI functions

To manage long-term storage growth without writing custom pruning scripts, you can run automated extraction and compaction queries directly in AlloyDB to identify the important parts which can benefit from being stored. This pattern uses a SQL common table expression (CTE) with ai.generate to consolidate older episodic entries into a high-density summary:

code_block: <ListValue: [StructValue([('code', "-- Consolidate older episodic logs into a permanent summary\r\nWITH old_events AS (\r\n SELECT document AS content\r\n FROM langchain_pg_embedding\r\n WHERE cmetadata->>'user_id' = 'user_dev_42'\r\n AND created_at < NOW() - INTERVAL '30 days'\r\n ORDER BY created_at ASC\r\n LIMIT 50\r\n),\r\nconsolidated AS (\r\n SELECT ai.generate(\r\n 'Summarize these historical events into a dense memory paragraph:\\n' || string_agg(content, E'\\n'),\r\n model_id => 'gemini-3.5-flash'\r\n ) AS summary_text\r\n FROM old_events\r\n)\r\nINSERT INTO agent_entities (user_id, entity_name, summary, updated_at)\r\nSELECT 'user_dev_42', 'longterm_session_summary', summary_text, NOW()\r\nFROM consolidated\r\nWHERE summary_text IS NOT NULL\r\nON CONFLICT (user_id, entity_name)\r\nDO UPDATE SET summary = EXCLUDED.summary, updated_at = NOW();"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01eaf31150>)])]>

For example, a long interaction regarding the complexities of changing flights with kids can result in a summary of “I prefer non-stop flights”.

4. Connecting the 2-tier memory architecture to an ADK agent

With the core 2-tier memory architecture configured, you can now extend the default ADK Memory provider to use this 2-tier memory architecture as shown in the accompanying Codelab (e.g. ADKTieredMemoryProvider). To attach the long-term memory to an ADK agent, you simply provide it as a tool (e.g. longterm_memory_tool) like this:

code_block: <ListValue: [StructValue([('code', 'from adk_memory_provider import ADKTieredMemoryProvider\r\n\r\nagent = Agent(\r\n name="adk_memory_agent", model=GEMINI_MODEL, instruction="Initial instruction",\r\n tools=[long_term_memory_tool, run_command_tool],\r\n before_tool_callback=guardrail.create_before_tool_callback(USER_ID, "CloudRetail")\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f049ef90>)])]>

Enterprise governance and multi-tenant security

Running agent memory in enterprise production environments requires strict security boundaries and access controls. First of all, we need to maintain scope and multi-tenant isolation. By indexing user_id, project_id, and scope columns in agent_entities and enforcing PostgreSQL Row-Level Security (RLS), you can isolate memory stores across departments, teams, and individual users within the same database cluster. Parameterized Secure Views (PSV) offer another layer of deterministic application-level security, helping you protect against malicious prompts and overly-broad SQL queries.

In addition, we need to provide automated memory lifecycle management as shown in the implementation pattern above. Combining scheduled SQL compaction queries with time-based partition pruning helps maintain predictable database footprint and query latencies over time. See the accompanying Codelab for more details on this approach.

Summary and next steps

Decoupling active context windows from persistent storage is a practical approach to building production-ready AI agents. By pairing Memorystore for Valkey for sub-millisecond session caching with AlloyDB AI for transactional long-term storage, you can achieve substantial token cost savings and faster response times while maintaining strict business rules throughout long-horizon tasks and many-turn agentic experiences.

To get started:

Step through the complete hands-on tutorial in the companion AlloyDB Agent Memory Codelab to deploy the working 2-tier memory architecture.
Learn more about database-side machine learning features in the AlloyDB AI documentation.
Explore guides on generating auto vector embeddings and running hybrid vector search.

Spanner Omni, now GA: A distributed, multi-model database that you can deploy anywhere

Wed, 30 Sep 2026 16:00:00 +0000

Spanner Omni, the deploy-anywhere version of Spanner, is now generally available, ready to power your most demanding production workloads in your on-premises data center or on other clouds.

With Spanner, Google pioneered the distributed SQL market over a decade ago, combining the horizontal scalability of NoSQL with the ACID compliance and strong consistency of a traditional relational database. Since then, Spanner has evolved into an interoperable multi-model database that simplifies complex workloads and powers agentic AI, combining SQL, graph, key-value, full-text search, vector search, and analytical processing with a columnar engine, all in a single database.

When we debuted Spanner Omni at Google Cloud Next ’26, we untethered our distributed database from Google Cloud and brought it directly to your infrastructure. This generated incredible interest from both the enterprise customers and developer community. In fact, since its launch, Spanner Omni has over 2 million downloads.

Spanner Omni delivers the same core capabilities as the fully managed Spanner service, with the added freedom to deploy it wherever you need it. Whether you’re running virtual machines or Kubernetes in your private data centers, running a multi-cloud deployment that spans multiple clouds, or testing locally on a laptop, Spanner Omni brings Google-grade consistency, availability, scale, and interoperable multi-model capabilities directly to your next agentic AI applications.

Spanner Omni provides the freedom to deploy anywhere with the same core Spanner capabilities

Attio accelerates agentic application velocity with Spanner Omni

Spanner Omni enables teams to build once and deploy anywhere, providing application portability across Google Cloud, on-premises and other clouds. Attio, an AI-native CRM company based in London, has built its platform on Spanner. Attio migrated its agentic application environment to a full-featured and containerized Spanner Omni + managed Spanner.

"At Attio, we are building the world's most advanced agentic application environment on Spanner to deliver on our vision for an AI-native CRM platform. Spanner Omni has been a major win for us by delivering Spanner capabilities and performance across all our environments, allowing our agents to bring complex, mission-critical workflows such as the newly announced Spanner queues, to production. With Spanner Omni, we have been able to accelerate our production velocity at a truly enhanced level of scale and confidence that was not possible earlier." - Alexander Christie, Co-Founder & CTO, Attio

Built-in AI capabilities for any environment

Spanner Omni extends Spanner's converged multi-model foundation directly to your private infrastructure and third-party clouds, providing critical capabilities for AI workloads:

Vector search and Spanner Graph: Natively store and index vector embeddings alongside your relational tables. With support for KNN and ANN search, you can combine semantic similarity with structured SQL filters. Spanner Graph integrates property graphs directly into the engine, allowing you to trace complex entity relationships and connect graph traversals with vector search.
Model Context Protocol (MCP) support: Integrate directly with agentic systems using MCP Toolbox. This open standard allows autonomous agents to inspect schemas, retrieve relevant context, and use Spanner Omni as an operational memory layer across multi-cloud deployments.

What’s new with Spanner Omni

Since the preview, we’ve been working to expand Spanner Omni’s capabilities, and refine the pricing model for production-ready deployments.

Enterprise features for production deployments

The GA release unblocks the full suite of enterprise-grade capabilities required for mission-critical production workloads, including:

Robust security and governance: Support for advanced enterprise security, including TLS encryption, authentication and authorization, and audit logging
Data protection: High-performance backup and restore capabilities to safeguard your data against “fat-finger” deletion or data corruption
Worker nodes: Dedicated, stateless compute nodes unique to Spanner Omni that are designed to offload background and resource-intensive operations from primary Spanner Omni servers, so they can focus on handling core database workloads
Enterprise-grade support: Direct access to Google Cloud Customer Care to keep your critical workloads running smoothly

Flexible licensing and industry-standard pricing

To support you at every stage of development, Spanner Omni offers two distinct licensing tiers designed to fit your scale and budget:

Developer Edition (free): Tailored for development, testing, and prototyping in non-commercial, non-production environments and personal use, it includes all core Spanner features, allowing you to build and validate your applications before scaling to production. The Developer Edition includes a 90-day license and includes all features as the commercial edition, except backup features and worker nodes. When used in a single server deployment of 4 vCPUs or less, the Developer Edition license does not expire and all single server features including backup-restore aresupported. If you need to use the Developer Edition for more than 4vCPUs or beyond the 90 day limit, you can request a perpetual license by filling out this form.
Commercial Edition (paid): Designed for commercial, production workloads, this edition provides the full suite of Spanner Omni capabilities backed by enterprise support. It follows a highly industry-standard, predictable vCPU-based annual subscription model. We also offer a proof-of-concept license for pre-production evaluation at a discounted price.

Spanner Omni vs. fully managed Spanner on Google Cloud

Our goal with Spanner Omni is to provide parity with the fully managed Spanner service on Google Cloud. However, as self-managed software, deploying and operating Spanner Omni differs from fully managed Spanner on Google Cloud in several ways:

Self-managed operations: You are responsible for all day-to-day operations, including routine maintenance, version upgrades, and infrastructure monitoring.
Availability and SLAs: Because Spanner Omni runs on customer-managed infrastructure, Google does not provide availability SLAs. However, deploying according to our recommended reference architectures will help you achieve comparable high availability.
Integration with Google Cloud: To ensure it can run anywhere, Spanner Omni excludes capabilities available in managed Spanner that rely on Google Cloud-specific capabilities, such as native integrations with BigQuery, Knowledge Catalog, or Gemini Enterprise and other Google Cloud services.
Feature-parity roadmap: While some feature gaps exist today between Spanner Omni and fully managed Spanner, we are actively developing updates to close these gaps in future releases.

Get started today

Whether you’re modernizing on-prem legacy systems or building a resilient multi-cloud architecture, Spanner Omni is ready to help you scale.

For more information, visit the Spanner Omni website to explore documentation and use cases.
To use the Spanner Omni commercial edition with full features, please contact your Google Cloud account team or reach out to us at https://cloud.google.com/consulting/spanner-omni.
For developing and testing for non-commercial, non-production purposes, download the Spanner Omni developer edition.

Unlock 3x QPS and microsecond latency with Memorystore for Valkey 9.1

Fri, 25 Sep 2026 16:00:00 +0000

At Google Cloud, we are committed to delivering the best managed experience backed by open source software. Today, we’re announcing the general availability of Memorystore for Valkey 9.1, which achieves up to 3x queries per second (QPS) at microsecond latency compared to Memorystore for Redis Cluster.

Our support for Valkey dates back to 2024, when Redis Inc. shifted its licensing away from the permissive open-source BSD license to a dual-license model. In response, Google Cloud, alongside other technology leaders, backed the creation of Valkey, an open-source alternative governed by the Linux Foundation.

Valkey has come a remarkably long way since then, delivering major performance and feature updates that push boundaries far beyond the original fork. Valkey is particularly compelling for organizations scaling AI and microservices to handle millions of concurrent users. Here, backend developers and architects must deliver both massive throughput while also maintaining microsecond latency.

In this blog, let’s take a look at how Valkey 9.1 achieves its performance, new developer capabilities, how to get started, and how customers are using it.

Under the hood: Rethinking thread communication

In high-throughput, in-memory datastores, efficient I/O offloading is critical to keeping the main execution loop unblocked. Previously, Valkey assigned client sockets to I/O threads statically in a round-robin fashion, requiring the main thread to continuously poll lists of pending clients to detect completed work.

Valkey 9.1 replaces list-polling with a lock-free, multi-queue messaging architecture that eliminates cross-thread CPU waste and unlocks dynamic work balancing. It involves three complimentary queues:

Main thread to I/O thread queue: Dispatches read and write jobs to a single-producer multi-consumer (SPMC) queue. Free worker threads pull tasks on demand, enabling dynamic work-stealing that prevents thread starvation or hot-spotting.
I/O thread to main thread queue: Worker threads push completed tasks into a multi-producer single-consumer (MPSC) queue. The main thread pops completed work instantly, eliminating busy-wait list iteration.
I/O thread-specific queues: Dedicated single-producer single-consumer (SPSC) queues handle thread-affine memory cleanup and high-volume epoll offloading.

Valkey 9.1 also replaces static thread thresholds with a two-phase dynamic scaling engine:

CPU-driven "ignition": When main-thread CPU usage crosses 30%, the engine automatically activates the first background I/O thread to absorb incoming traffic before queue bottlenecks form.
Queue-depth auto-scaling: Once ignited, Valkey dynamically scales the number of active I/O worker threads up or down based on real-time SPMC queue backlog, ensuring extra cores are used only when needed and parked when idle.

New developer capabilities in Valkey 9.1

Beyond raw performance, Valkey 9.1 addresses key feature requests from engineering teams with powerful new commands and enhanced security controls. Here is a look at what you can do with these new capabilities:

1. Granular database-level access control (ACLs)

We recently launched support for access control lists on Memorystore for Valkey to provide more granular key-level and command-level authorization using IAM. This foundational security mechanism is offered at no additional cost and includes the following capabilities:

Centralized management: A 1:N mapping approach allows you to define a single ACL policy and attach it across multiple clusters.
Secure multi-tenancy: Organizations can easily enforce least privilege and secure multi-tenancy across their database fleets.
Enhanced observability: The feature includes versioned policy revisions and comprehensive audit logging.

Previously, ACL rules applied globally across an instance. Valkey 9.1 allows administrators to restrict user access at the specific numeric database level within the ACL framework.

Real-world example: You can configure a staging or service-specific user and isolate their access strictly to non-production databases:

production user: @all ~* db=0
staging user: @all ~* db=1
dev user: @all ~* db=2

Protect against unauthorized data access and guard against application bugs by leveraging database-level access control across multiple databases, all without needing to prefix your keys.

2. CLUSTERSCAN: Efficient cluster-wide key scanning

Previously, scanning keys across a large cluster required querying nodes individually. This approach was not cluster- or failover-aware. Consequently, scans could miss keys, return duplicates, or fail if slot migrations or node failovers occurred during the process.

The CLUSTERSCAN command addresses these limitations by introducing a topology-aware cursor. This cursor encodes the current slot, the fingerprint of the local hashtable, and the local cursor. With this additional encoded information, clients can scan keys across the entire cluster while gracefully handling topology changes and redirections.

CLUSTERSCAN supports two primary scanning strategies:

Use case 1: Sequential full cluster scan (single worker)

This strategy is suitable for simple scripts or background jobs that prioritize simplicity over speed. The client starts with cursor 0 and sequentially traverses all slots in the cluster:

code_block: <ListValue: [StructValue([('code', 'CLUSTERSCAN 0 MATCH "user:*" COUNT 10\r\n1) "0B3a21-{06S}-64"\r\n2) 1) "user:101"\r\n2) 2) ...'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01eaf31f10>)])]>

To continue the scan, pass the returned cursor to the next call. The cursor automatically transitions to the next slot when the current one is fully scanned.

code_block: <ListValue: [StructValue([('code', 'CLUSTERSCAN 0B3a21-{06S}-64 MATCH "user:*" COUNT 10\r\n1) "0B3a21-{07T}-0"\r\n2) 1) "user:102"\r\n2) 2) ...'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f052b650>)])]>

The scan is complete when the command returns a cursor of "0".

Use case 2: Parallelized cluster scan (multiple workers)

This strategy is suitable for high-throughput scans. Using the SLOT argument restricts the scan to a specific slot, allowing you to partition the 16,384 slots across multiple parallel workers.

Worker 1 (Scanning Slot 0):

code_block: <ListValue: [StructValue([('code', 'CLUSTERSCAN 0 SLOT 0 MATCH "user:*" COUNT 10\r\n1) "0B3a21-{06S}-64"\r\n2) 1) "user:101"\r\n2) 2) ...'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01eb11d0d0>)])]>

Worker 2 (Scanning slot 1000 in parallel):

code_block: <ListValue: [StructValue([('code', 'CLUSTERSCAN 0 SLOT 1000 MATCH "user:*" COUNT 10\r\n1) "0B3a21-{08X}-32\r\n2) 1) "user:999"\r\n2) 2) ...'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01eb11de10>)])]>

From here, Worker 1 continues to pass SLOT 0 and Worker 2 continues to pass SLOT 1000. Mismatching the slot and the cursor returns an error. Once all 16384 slots have been scanned, the cluster scan is considered complete.

3. More commands for atomicity and expirations

HGETDEL: Atomic fetch and delete
A frequent application pattern involves reading a hash field and deleting it immediately (such as consuming single-use authentication tokens or short-lived session states). Valkey 9.1 introduces HGETDEL, which retrieves the value of a hash field and deletes it atomically in a single network round-trip.

Real-world example:
HSET user:1001 temp_token "abcde"
(integer) 1
HGETDEL user:1001 FIELDS 1 temp_token
1. "abcde"
HGET user:1001 temp_token
(nil)

MSETEX: Shared expiration for multiple keys
To eliminate multi-command pipeline overhead, the new MSETEX command enables setting multiple keys simultaneously with a single, shared expiration time.

Real-world example: Setting up a temporary session state where multiple distinct keys must expire together in 300 seconds:
MSETEX 2 session:auth "ok" session:user_id "1001" EX 300
(integer) 1
TTL session:auth
(integer) 300

Enhanced HSETEX with conditional flags
HSETEX now supports the NX (only set if the field does not exist) and XX (only set if the field exists) conditional flags.

Real-world example: Initializing a rate-limit threshold field with a 1-hour TTL, ensuring you don't overwrite an existing active limit:
HSETEX config:123 NX EX 3600 FIELDS 1 "rate_limit" "100"
(integer) 1

Built on Memorystore for Valkey 9.0

The release of Valkey 9.1 builds upon the major updates we unveiled for Memorystore for Valkey at Google Cloud Next '26:

Built-in modules for AI & vector workloads: Native JSON support and Bloom filters enable fast document querying and membership checks.
Six new node sizes: To help you manage costs and scale, we added six new node sizes.

Small Size Nodes: Custom-Pico (1.25 GB), Custom-Micro (2.5 GB), and Custom-Mini (3.5 GB) for lightweight microservices and dev/test environments. These are only available for cluster mode disabled environments.
High CPU and Large SKUs: HighCPU-Medium (8 vCPU/13 GB) and Standard-Large (8 vCPU/26 GB) optimized for CPU-heavy applications.
XXL SKU: Highmem-XXLarge with 110 GB RAM and 16 vCPUs per node for massive cluster consolidation to power your most demanding workloads.

(Note: The figures above are based on open-source benchmarks; actual performance improvements will vary depending on your specific workloads.)

Migrating to fully managed Memorystore for Valkey

Having to self-manage your Redis OSS /Valkey caching layers drains valuable engineering bandwidth and creates operational friction during scaling. We are also excited to announce a new migration workflow to Memorystore for Valkey.

With this release, migrating your infrastructure is straightforward, fully managed, and requires a simple configuration change on your application to point to Memorystore for Valkey once your data is migrated. This workflow is generally available.To move off self-managed Redis or Valkey to fully managed Memorystore for Valkey, follow these four steps:

1. Provision the target instance: Deploy a Memorystore for Valkey instance configured with your required shard count, node sizing, and clustered database options.

2. Establish online replication: Initiate continuous, dual-sync online migration directly from your source database to Memorystore.

3. Validate data synchronization: Monitor replication metrics in real time to verify full dataset alignment and low-latency replication health.

4. Execute the cutover: Switch application connection endpoints over to Memorystore for Valkey to start using the new cache.

What Memorystore for Valkey customers are saying

Already, over 95% of the top 100 Google Cloud customers already rely on Google Cloud Memorystore to power demanding, high-throughput workloads, led by increasing numbers of Memorystore for Valkey users.

Consider the fast-paced world of live sports, where delivering a flawless digital experience is of utmost importance. When a game-changing play happens, millions of fans immediately reach for their devices to check real-time stats, watch highlights, and engage with interactive features. These massive, unpredictable traffic spikes require an underlying architecture capable of immense scale. For organizations like Major League Baseball (MLB) , a partner since Valkey’s early days, managing unpredictable traffic spikes without compromising performance is essential.

"We trust Memorystore for Valkey to power the massive scale of live baseball, delivering real-time stats and uninterrupted digital experiences to millions of fans. As we look ahead, we are incredibly excited about the Memorystore for Valkey 9.1 launch. The engine optimizations and latency enhancements will give us even more horsepower to handle the most unpredictable game-day traffic spikes, ensuring fans get the best technology-powered experience the game has to offer." - Rob Engel, SVP of Software Engineering, Major League Baseball

Beyond the stadium, the retail industry faces its own intense scaling challenges, particularly during major shopping holidays or flash sales. Modern e-commerce platforms rely on real-time personalization, dynamic pricing, and instant inventory updates to keep shoppers engaged. A lag of even a few milliseconds can disrupt the customer journey and impact the bottom line. To maintain a competitive edge, leading retailers such as Target require ultra-responsive caching layers to power their most crucial customer-facing platforms.

"By leveraging Google Cloud Memorystore for Valkey, Target delivers ultra-low-latency, resilient caching for personalization services. We look forward to leveraging the performance enhancements in Valkey 9.1 to make our personalization platform even faster, more scalable, and more resilient during periods of peak demand." - Scott Weide and Sumanth Huddar, Senior Engineering Managers, Target

The demand for these ultra-low-latency architectures extends far beyond sports and retail. Across the digital landscape, organizations in banking, AI-native development, digital streaming, and telecommunications all share a common mandate: the need for superfast, highly available caches. Whether it is processing high-frequency financial transactions, serving complex machine learning inferences in real time, delivering seamless global video streams, or routing immense volumes of telecom data, microsecond latency is the new baseline for success.

Make the move to Valkey

Stop letting cache bottlenecks slow down your most demanding applications. Experience the performance, dynamic scalability, and enhanced security of Memorystore for Valkey 9.1 today.

Start building: Create a Memorystore for Valkey 9.1 instance in the Google Cloud console.
Dive deeper: Read the technical documentation to view the full list of supported commands, ACL configurations, and detailed capabilities of Memorystore for Valkey.

AlloyDB delivers PostgreSQL for agents: Real-time data at agent scale, with full workload isolation

Thu, 24 Sep 2026 14:30:00 +0000

Enterprises rely on mission-critical operational databases where performance slowdowns simply aren’t an option. Yet when even a few agents execute dense reasoning loops, the unpredictable surge in queries can easily overwhelm traditional architectures.

Today, we’re announcing that AlloyDB delivers PostgreSQL for agents (in preview), enabling real-time data access without compromising your mission-critical systems. AlloyDB now scales to dynamic agent bursts by provisioning sandboxed database instances in seconds, enabling full workload isolation. You can cost-effectively run your agents at any scale — from a few agents to millions of agents — and the instances automatically spin down when agents finish. With this announcement:

AlloyDB now features an agentic database architecture engineered to scale PostgreSQL to thousands of serverless database instances that have up-to-the-second read-only access to production. These instances remain fully separated from the primary, standby, and read replica instances where production workloads run.
Each of these instances accesses real-time data in the database backed by a unified storage layer in Colossus, Google’s exabyte-scale distributed storage system. This helps agents achieve sub-millisecond I/O, and terabit-per-second aggregated scan throughput, supporting over 3 million queries per second.
These instances utilize the full AlloyDB PostgreSQL engine, providing access to every index, the full capability of SQL, and comprehensive vector, full-text, and spatial search. Agents can also leverage BigQuery and Spark to run lakehouse analytics without requiring complex ETL pipelines.
When agents complete their tasks, these instances scale right back to zero, thus reducing your cloud spend by billing only for active reasoning loops.

With AlloyDB for PostgreSQL, we pioneered the agentic enterprise relational database by integrating advanced vector operations, machine learning inference, and foundation model integrations directly within a 100% PostgreSQL-compatible engine. It protects your data through deep Google Cloud security integrations — replacing static passwords with IAM authentication, isolating traffic via VPC Service Controls, and providing customer-managed encryption and auditing. This functionality, combined with the scalability now provided by our agentic architecture, makes AlloyDB the premier enterprise-grade agentic PostgreSQL offering.

To get started, sign up here for the preview.

Why this matters

In today’s agentic era, we’re swiftly moving from single copilot agent interactions to networks of millions of agents collaborating simultaneously. When these agents query and operate all at once, sudden traffic spikes can overwhelm your core databases, competing with the systems that run your business.

Agents require both fast analytics and low-latency access to real-time production data, utilizing B-tree, vector, text, and spatial indexes to efficiently execute their workflows. Emerging architectures rely on page-caching layers that sit above object storage, but they suffer from scaling and cost challenges that can compromise the stability of production systems.

In addition, they face a challenging trade-off: To unlock production data for analytics, they create performance bottlenecks for operational access, putting mission-critical databases at risk the moment agents are unleashed in production. These approaches attempt to solve the problem using traditional object stores for database storage. While this enables analytical access that can help some agents, the underlying databases are too slow for production workloads, suffering from up to an order of magnitude higher I/O latency. Page caching layers are at best a patch; the caches themselves are often still not fast enough, and they pose a scalability bottleneck that is easily saturated by agentic workloads.

When active multi-agent systems execute dense reasoning cycles, they trigger highly concurrent, unpredictable bursts of queries that overwhelm these caching layers, leaving mission-critical production systems vulnerable to agent-induced outages. Consequently, an entire class of operational use cases is precluded from running on these architectures, locking businesses out of the transformative power of AI on live enterprise data.

AlloyDB’s unique agentic PostgreSQL architecture

We are taking a different approach. AlloyDB delivers an agentic database architecture purpose-built for the AI era, with four key differentiated capabilities:

Sub-millisecond I/O latency, without artificial choke points: Combining AlloyDB’s industry-leading transaction and query processing with low-latency object storage backed by Google’s planet-scale Colossus storage infrastructure, this architecture provides a large-scale, shared storage plane for agents. It achieves sub-millisecond I/O and over a terabit-per-second of aggregate scan bandwidth, allowing agents to execute intensive read queries and vector searches directly against fresh operational data.
Fully isolated from production workloads while scaling to meet demand: AlloyDB scales by dynamically provisioning sandboxed database instances in seconds against fresh production data. Unlike traditional architectures where agents compete for operational resources, agentic database compute remains completely isolated from the primary database clusters — allowing agents to execute dense, unpredictable reasoning loops without degrading performance in production. These robust safety guardrails, coupled with enterprise-grade governance and fine-grained access control, allow you to confidently unleash the full, unconstrained power of PostgreSQL on your production data — seamlessly mixing analytical queries, vector queries, and operational point lookups in active agentic loops.
Pay-as-you-go billing: Most agent activity is characterized by sharp spikes of concurrent queries followed by periods of inactivity. Provisioning dedicated read replicas to absorb these bursts forces you to maintain expensive infrastructure around the clock. To support massive groups of agents cost-effectively, and eliminate the idle compute overhead of provisioned systems, these agentic AlloyDB instances can rapidly scale to handle millions of queries per second, and automatically scale to zero with a flexible, pay-as-you-go pricing model.
Native lakehouse integration, without ETL: All production data is natively integrated with Google Cloud’s borderless Lakehouse. This allows agents to run federated queries across BigQuery and Lightning Engine for Apache Spark, joining massive lakehouse datasets with up-to-the-second transactional data in AlloyDB. This eliminates the need to build and maintain fragile batch ETL pipelines, giving autonomous agents instant access to both live operational state and historical lakehouse context.

“As supply chains become increasingly autonomous, our platform relies on real-time transactional intelligence to coordinate complex logistics workflows across thousands of facilities. AlloyDB's new PostgreSQL architecture for agents has been a game changer for us. We can now deploy networks of agents collaborating simultaneously to help us analyze inventory and order data with sub-second freshness, while ensuring our core transactional processing remains entirely untouched. It delivers the isolation, speed, and cost efficiency we need to power the next generation of enterprise supply chain AI.” - Sanjeev Siotia, Executive Vice President & Chief Technology Officer, Manhattan Associates

Availability

PostgreSQL for agents in AlloyDB is now available in preview.

To learn more, visit the documentation page, and sign up here to get started.

A new, no-compromises database architecture for the agentic era

Thu, 24 Sep 2026 14:30:00 +0000

Entire database engineering careers have been spent on a single question: How do you scale an OLTP workload without compromising the system of record that owns the data?

Exadata answered the question by offloading queries into a scale-out storage tier beneath the database, removing the network as the bottleneck. Azure SQL Hyperscale did it with shared block servers, scaling out to tens of read replicas. Aurora offloaded log application to distributed storage nodes, scaling reads across tens of PostgreSQL nodes. Meanwhile, emerging architectures persist data in traditional object storage with a provisioned cache tier in front, recovering latency for hot data but leaving a high-latency tail on every cache miss.

Each of these architectures is inherently constrained by at least one of these three properties: scale, latency, and isolation — and sometimes even two. For instance, architectures built on shared block servers compromise scalability, because I/O inevitably bottlenecks on the block server. They also sacrifice isolation, as production workloads get throttled whenever replica traffic spikes.

Some of these trade-offs were actually sound when they were developed; they met the requirements of enterprise database workloads for four decades. However, in the agentic era, these compromises are no longer acceptable. Agentic workloads are generated dynamically and cannot be vetted in advance, making it a business-continuity imperative to isolate them from mission-critical systems. Agentic workloads also require low latency that is only possible with the full power of the database engine and all its indexes, as well as a whole new level of elastic scale that has never been tried with a single database: a burst of agents that demand 1,000 compute nodes over a single database within seconds, and that may finish inside a minute.

Three tenets needed for a truly agentic database architecture

We believe the agentic era demands a new agentic database architecture defined by three fundamental tenets. An agentic database architecture must satisfy all three, or it isn’t really agentic.

Tenet: Isolation — isolation by design, but with real-time data access. Agents must read live production data with sub-second freshness over a data path that does not share database components with the primary cluster. Real-time means up-to-the-second, not a stale copy or branch. This is physical separation, not a quota — because shared allocations mean shared fate. The boundary extends straight through the storage layer, eliminating resource contention by design.
Tenet: Latency — sub-millisecond baseline I/O. Operational workloads demand sub-millisecond block I/O, and that bar does not drop for agents. While compute nodes leverage DRAM and local SSD for acceleration, cache misses that reach remote storage — whether application or agentic — must complete in under a millisecond. An architecture that degrades into an order-of-magnitude performance cliff is fundamentally unusable by agents.
Tenet: Scale — agent-scale compute and I/O. Agent scale is simultaneously instantaneous, volatile, and massive: Database compute nodes must spin up in seconds, scale to thousands, run for short bursts, and automatically spin down to zero when agents are done with them. No one has thus far ever dreamed of expecting a database to scale compute and I/O dynamically to thousands of nodes while leaving production untouched. Due to the dynamic nature of agents, pre-provisioning is a non-starter across the entire stack, whether it’s compute, storage I/O, or any caching tier in between.

Crucially, an agentic architecture must uphold all three tenets at once. And by doing so, the architecture allows agents to work directly against live operational data, i.e., enterprise truth, without compromising production stability. The outcome is transformative:

No correlated failures: Total decoupling between the engines running the business and the fleets of agents reasoning over it removes a path for agents to affect production.
No capacity guesswork: True elasticity that eliminates the friction of pre-provisioning for unforecastable agent scale.
No semantic compromises: Nothing is withheld from agents — they get access to the full power of relational SQL, hybrid search (vector, full-text, spatial), and indexes within every single reasoning step.

AlloyDB's agentic architecture

AlloyDB’s new agentic database architecture is the first system that satisfies all three tenets. We engineered this from the ground up across storage, network, compute and databases to deliver:

Isolation, avoiding shared fate by design: The transactional production cluster runs on dedicated, pre-provisioned infrastructure, completely isolated from agent workloads. Agents interface via the Model Context Protocol (MCP) to an independent, ephemeral pool of microVM-based AlloyDB nodes that read directly from dedicated Colossus storage segments, separate from those for production.
Predictable sub-millisecond storage I/O: Every storage read is served directly by Google’s Colossus storage system inheriting its baseline sub-millisecond latency, eliminating performance cliffs on cold cache misses.
True zero-to-thousands compute scaling: The agent pool scales rapidly from zero to thousands of nodes for bursty agentic activity, and scales back to zero the moment tasks complete.

Agents query production data with sub-second freshness, with the complete PostgreSQL engine — point lookups, index traversals, vector, full-text and spatial search, columnar scans, and federated queries across the lakehouse — at their disposal to power their reasoning loops.

Run agents against production data at any scale by joining the preview of AlloyDB PostgreSQL for agents. You can learn more about its full capabilities in the companion announcement blog.

Why existing architectures can’t satisfy all three tenets

Traditional and emerging operational databases attempt to scale using one of three architectural paradigms. When assessed against the demands of autonomous AI agents, each paradigm exhibits a fundamental structural compromise — none satisfies all three tenets simultaneously.

Independent replicas (shared-nothing storage)

Traditional relational architectures scale reads by streaming replication logs from a primary instance to dedicated replica databases, each with its own local or attached block storage. They meet Tenet: Isolation – replicas share no physical resources with the primary cluster, and continuous log replication maintains near-real-time currency. They meet Tenet: Latency – dedicated local storage guarantees predictable, sub-millisecond read latency. However, they fail Tenet: Scale – scaling requires provisioning a new replica and rehydrating hundreds of gigabytes or terabytes of storage. All this takes hours — an impossible mismatch for agent-reasoning bursts measured in seconds. Furthermore, statically provisioned compute and storage continue to incur idle costs long after the agent completes its run.

Disaggregated shared-storage servers

A second approach decouples stateless compute nodes from a shared, multi-tenant tier of custom storage servers that manage persistence, replication, and that may offload block writes. This approach meets the Tenet: Latency – reads hitting the optimized storage servers resolve with consistent, low operational latency. However, it fails the Tenet: Isolation – because every replica reads from the same servers as the primary, so agent I/O contends directly with production I/O, creating shared fate. It also fails the Tenet: Scale – stateless compute replicas spin up quickly because no data is copied, but total storage I/O bandwidth is fixed to the pre-provisioned storage tier. Adding compute nodes without scaling underlying I/O capacity simply accelerates storage saturation and throttling.

Object storage with shared-block servers

A third emerging approach keeps data durable in general-purpose object storage and serves block reads from a shared tier of block servers. Because a random read from object storage takes tens of milliseconds — an order of magnitude slower than traditional database storage, and slower than an enterprise disk array has been for at least 25 years — the block servers hold hot data in order to serve it at low latency. This approach meets the Tenet: Latency — with one caveat: A block server miss still falls through to object storage at unacceptably high latency. It fails the Tenet: Isolation — because replicas share the block servers with production: Agent I/O and production I/O draw on the same capacity, so when that capacity is exhausted or throttled, production is affected along with the agents. It also fails the Tenet: Scale, for the same reason as shared storage servers: Replicas start quickly, but the block servers do not scale their I/O with the burst.

Some architectures in this family also allow analytical engines like Apache Spark to read the underlying object storage directly, bypassing the database engine. For analytics workloads, that is a valuable and viable path. However, since agents need low-latency retrieval, stripping away indexes, point lookups, and vector search forces brute-force table scans, exploding latency, and therefore breaks the ability for agents to execute their retrieval-reasoning loops.

Evaluating existing architectures

We evaluated a commercially available service that uses the object storage architecture with shared block servers by running concurrent index lookups over a dataset larger than available DRAM, testing both scaling limits and production isolation. Starting with a single reader instance, we scaled the workload by adding up to eight read replicas.

In architectures with shared physical resources, scaling agents via read replicas quickly degrades both replica and primary performance. In our tests as seen in the chart below, adding replicas provided less than a 2x throughput increase, peaking at four replicas before dropping off as the shared block-server bandwidth saturated.

The impact on the primary database was immediate and severe: Primary throughput plummeted by more than 75% as replicas were added.

In short, neither traditional nor emerging architectures can meet the scale that agents demand, and certainly not without jeopardizing the stability of production systems.

Assessing against the Tenets

* Partially meets: Hot data is served at low latency from the block servers, but a block server miss falls through to object storage at tens of milliseconds.

In each case the gap is structural, not just a matter of tuning. Replication isolates by giving each replica its own storage, so it cannot add a replica faster than it can populate that storage. Shared storage servers add compute quickly by sharing storage, so they can neither isolate nor scale I/O. Block servers over object storage recover latency with a provisioned tier, so they can neither isolate nor burst, and every miss still reaches object storage. Each approach solves the problem at one layer and pays for it at another. Meeting all three tenets at once requires rethinking the database architecture across compute, network and storage together.

How we engineered AlloyDB across the stack

AlloyDB's agentic database architecture is vertically integrated across Google's data, AI and infrastructure stack: AI models, the database engine and analytical engines, but also storage, networking and compute infrastructure.

Storage: Colossus as the foundation

At the persistence layer, AlloyDB builds on Colossus, Google's exabyte-scale distributed storage system that underpins Google Search, YouTube, Gmail, Google Drive, Spanner, and Bigtable. A single Colossus cluster scales to exabytes of storage and tens of thousands of machines. With Spanner, we demonstrated that a transactional database engineered directly on Colossus can scale to thousands of nodes. The new AlloyDB architecture applies the same foundation to a new problem: agents.

Colossus has three properties that enable AlloyDB to satisfy the three tenets.

Direct, sub-millisecond I/O: Colossus is engineered to minimize read latency. A database node opening a Colossus stream receives a handle that describes where data physically resides. Authorization and metadata resolution happen once, when the stream is created; every subsequent read goes directly to the disks holding the data, over an optimized network protocol. The result is sub-millisecond latency across all of the database's data, with no intermediary to warm and no tier to miss.
Massive throughput: Colossus delivers up to 15 TB/s of aggregate throughput and 20 million queries per second to a single AlloyDB database without needing to provision bandwidth and with an unlimited number of concurrent hosts. At Colossus scale, a fleet of AlloyDB agent nodes is not a load the storage must be sized for; it is a fraction of the load the storage already serves!
Physical segment partitioning: AlloyDB serves agents from a separate set of Colossus segments, so agent I/O is deliberately spread away from the production data path rather than contending with it.

At no point along the data path — compute, network or storage — can an agent ever share a database component with production.

Network: Scalable bandwidth with Jupiter

Compute and storage are bound together by Jupiter, Google's high-capacity data center network. A single Jupiter fabric connects more than 100,000 servers with 13 petabits per second of bisection bandwidth — enough to carry a video call for every person on Earth.

Because Jupiter provides high bisection bandwidth with predictable low latency across the networking fabric, agent nodes can be scheduled flexibly anywhere in the cluster with consistent access to centralized storage. As the agent pool scales from zero to thousands, the underlying interconnect capacity absorbs the expanding traffic without creating placement bottlenecks.

Compute: Elastic and serverless PostgreSQL and analytics

At the compute layer, agents connect to AlloyDB's agent pool through MCP. The agent pool consists of AlloyDB agent nodes with read-only access to the up-to-the-second state of the database. This layer provides:

MicroVM isolation: Each agent node is a fully functional AlloyDB for PostgreSQL database engine running inside a lightweight, secure microVM. These instances are fully isolated from each other and from the dedicated primary cluster.
Rapid spin-up and scaling: Agent nodes are provisioned in response to requests from agents and stop automatically when the agents finish. In response to a burst, AlloyDB rapidly provisions thousands of agent nodes, serving millions of concurrent agents, and releases them as the agents finish. Because billing is per second of agent-node activity, a burst that uses a thousand nodes for tens of seconds will only be charged for the resources that the job consumed, and nothing more.

Meanwhile, the production cluster remains as it is today: pre-provisioned, on dedicated infrastructure, sized for the system of record. Agent nodes read from Colossus directly and see a consistent production state with sub-second freshness.

Beyond the agent pool, BigQuery and Spark can read AlloyDB data from Colossus with the same isolation from the production cluster, so agents can use lakehouse federation to join real-time operational data with large-scale lakehouse datasets.

By building on these Google-scale storage, network, and compute layers, AlloyDB’s new agentic database architecture achieves a remarkable goal: Share the data. Share nothing else.

Evaluating AlloyDB’s agentic database architecture

We tested AlloyDB by running concurrent index lookups over a dataset larger than available DRAM, testing scalability across the full stack. We ran the agentic workload starting with a single agent node — an independent database instance in the agent pool, rather than a traditional read replica — and scaled dynamically to thousands of nodes over a single database, measuring both the aggregate agentic throughput as well as any impact on production.

In this test, throughput scaled linearly from 3.9K to 41K QPS when expanding from one to 10 agent nodes. Scaling by two additional orders of magnitude yielded near-linear performance up to 1,000 nodes. We observed:

Zero primary degradation: Scaling from 1 to 1,000 agent nodes produced no measurable impact on primary cluster performance.
Massive throughput: Aggregate throughput dynamically scaled 773x to 3 million QPS, driving over 8 million IOPS in Colossus across 1,000 compute nodes.

In a similar benchmark running concurrent full table scans across 2,100 agent nodes, aggregate scan throughput exceeded 1 terabit per second.

Because the agent pool shares no physical infrastructure with the production cluster, teams can scale reasoning fleets to thousands of nodes without placing production systems at risk.

Engineering all three tenets by design

The table below shows how AlloyDB’s architecture satisfies each of the three tenets:

Every agentic database architecture will require these three foundational elements: storage with the properties of Colossus, a network that connects compute to that storage without constraint, and compute that can be provisioned and released at agent scale.

Google has spent more than two decades building exactly that, to run Google Search, YouTube, and Gmail. Now it underpins our agentic database architecture.

Give agents live data without impacting production

Every organization building with AI faces the same core dilemma: how to give agents full access to live operational data without putting the systems running the business at risk. Until now, architecture — not application needs — dictated that choice. Giving agents direct access to the database meant exposing mission-critical systems to unforecastable load, severe resource contention, and production outages.

An architecture built on these three tenets removes these compromises entirely. Agents reason over live production data with sub-second freshness. They have the complete engine at their disposal — every index, vector, full-text and spatial search, and the full capability of SQL — at sub-millisecond I/O. The architecture scales dynamically to thousands of isolated nodes when agents need it, then to zero when agents finish. Throughout, core transactional workloads remain untouched: no shared components, no shared quota, no correlated failures. Agents can deliver innovation without conflicting with business continuity.

The same property extends to every other reader of production data. Reporting, analytics and applications can freely read live data without putting production at risk, ending a constraint that has shaped operational databases for five decades.

The data in an enterprise's systems of record is its crown jewels. Built on this foundation, that data can finally be put to work in full.

Databases, unfettered.

To learn more, visit the documentation page, and sign up here to get started.

Announcing Native BM25 Ranking in AlloyDB and Cloud SQL

Fri, 18 Sep 2026 16:30:00 +0000

Vector search is a critical component of generative AI, retrieval-augmented generation (RAG), and data agent architectures, but sometimes vector search alone isn't enough. While vector embeddings are incredible at understanding conceptual meaning, they stumble on specific alphanumeric IDs and exact product SKU numbers. To build truly robust search and AI applications, you may need the combination of semantic vector search and traditional exact keyword full-text search — what we call hybrid search.

In search, Best Matching 25, or BM25, is a key algorithm used to estimate how relevant a document is to a given query. Until today, if you wanted BM25 ranking with AlloyDB or Cloud SQL, you needed to add an additional full-text search backend. This introduced data silos, sync lags, and operational complexity. Today, we are eliminating the friction of maintaining a separate full-text search backend altogether, with the preview of the native BM25 index in AlloyDB and Cloud SQL for PostgreSQL 17+, made possible through the open-source pg_textsearch extension created by Tiger Data.

Now, with a unified hybrid search backend, you no longer need to provision, manage, or pay for separate systems to get state-of-the-art full-text retrieval. It all happens directly inside your database, where your operational data lives, delivering:

Industry-standard keyword ranking: Powered by Tiger Data's pg_textsearch, bring lightning-fast, C-optimized BM25 scoring directly to your Postgres tables.
No complexity, total consistency: Eliminate the data duplication, ETL pipelines, and synchronization lag that you get when you maintain multiple backends for vector and full-text retrieval.
Supercharged semantic search (AlloyDB exclusive): Get up to 6x and 10x faster vector search queries (when compared to standard PostgreSQL) with ScaNN and HNSW index types.

Why pg_textsearch?

If you’ve used PostgreSQL's built-in ts_rank for full-text search at any meaningful scale, you already know its limitations. Ranking quality degrades as your corpus grows. There’s no support for inverse document frequency, so common words carry the same weight as rare ones. There’s no term-frequency saturation, so a document that mentions "database" 50 times outranks one that mentions it once.

BM25 is the information retrieval gold standard, providing inverse document frequency (rarer terms matter more), term frequency saturation (repetition doesn't dominate), and document length normalization. You can learn more in this blog post by Tiger Data about how they built a BM25 search engine on PostgreSQL pages.

Full-text search example

Here’s how to get started with BM25 full-text search on both AlloyDB and Cloud SQL. Consider a sample table, cymbal_products, that contains the unique identifier uniq_id, a product_name column, a product_description column containing a text description of each product, and a generated product_embedding column. cymbal_products contains information on various retail products, including indoor and outdoor plants.

Index creation

To use BM25, enable the pg_textsearch extension.

code_block: <ListValue: [StructValue([('code', '-- Install pg_textsearch extension\r\nCREATE EXTENSION pg_textsearch;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f0c117d0>)])]>

Create the index on the product_description column from the cymbal_products table.

code_block: <ListValue: [StructValue([('code', "-- Create the native BM25 index on the content column\r\nCREATE INDEX idx_docs_bm25 \r\nON cymbal_products \r\nUSING bm25 (product_description) \r\nWITH (text_config='english');"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f0c11910>)])]>

A BM25 full-text search query can be executed using the <@> special operator. In the snippet below, we search for ‘cherry tree’.

code_block: <ListValue: [StructValue([('code', "-- Full text search query\r\nSELECT product_name, product_description <@> 'cherry tree' AS bm25_score \r\nFROM cymbal_products\r\nORDER BY bm25_score \r\nLIMIT 5;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f0c11b50>)])]>

Sample output is shown below. A more negative score indicates a stronger relevance match.

AlloyDB hybrid search example

Setting up a hybrid search system in AlloyDB is simple. You can create both your vector and keyword indexes on the same table and merge the results seamlessly using the hybrid search user-defined function (UDF).

Vector index creation

Here is how to create a ScaNN vector search index:

code_block: <ListValue: [StructValue([('code', '-- Install vector extension\r\nCREATE EXTENSION vector;\r\n\r\n-- Install scann extension\r\nCREATE EXTENSION IF NOT EXISTS alloydb_scann;\r\n\r\n-- Create scann vector search index \r\nCREATE INDEX cymbal_products_embeddings_scann ON cymbal_products USING scann(product_embedding cosine);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f0c12610>)])]>

Hybrid search

AlloyDB provides an out-of-the-box hybrid search UDF that makes it very simple to run hybrid search queries. The UDF merges the ranked results from each search component into a single, unified list using the Reciprocal Rank Fusion (RRF) algorithm. This query utilizes the UDF to perform a vector search for ‘trees that grow taller than houses’ and a keyword search for ‘California’ in the product description.

code_block: <ListValue: [StructValue([('code', 'CREATE EXTENSION google_ml_integration;\r\n\r\nSELECT *\r\nFROM ai.hybrid_search(\r\n search_inputs => ARRAY[\r\n \'{\r\n "data_type": "vector",\r\n "weight": 0.5,\r\n "table_name": "cymbal_products",\r\n "key_column": "uniq_id",\r\n "vec_column": "product_embedding",\r\n "distance_operator": "public.<=>",\r\n "limit": 10,\r\n "query_vector": "ai.embedding(\'\'text-embedding-005\'\', \'\'trees that grow taller than houses\'\')::vector"\r\n }\'::JSONB,\r\n \'{\r\n "data_type": "text",\r\n "weight": 0.5,\r\n "table_name": "cymbal_products",\r\n "key_column": "uniq_id",\r\n "text_column": "product_description",\r\n "limit": 10,\r\n "ranking_function": "<@>",\r\n "query_text_input": "California"\r\n }\'::JSONB\r\n ],\r\n);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f0c12090>)])]>

As shown in the sample output below, results are ranked in descending order of their RRF scores.

Here, hybrid search bridges the gap between semantic intuition and exact keyword matching. While vector embeddings excel at grasping conceptual queries, like "trees that grow taller than houses", traditional full-text search provides the pinpoint precision needed for strict identifiers like "California." By fusing the two, AlloyDB helps ensure your application prioritizes highly specific, locally relevant results like ‘California Sycamore’ right at the top of the list.

Cloud SQL hybrid search example

In Cloud SQL, you can create both your vector and keyword indexes on the same table and merge the results seamlessly using Common Table Expressions (CTEs) and coalescing the RRF score, as shown below.

Vector index creation

Here is how to create an HNSW index in Cloud SQL.

code_block: <ListValue: [StructValue([('code', '-- Install vector extension\r\nCREATE EXTENSION vector;\r\n\r\n-- Create an HNSW index on the embedding column for fast approximate nearest neighbor search\r\nCREATE INDEX product_hnsw_idx ON cymbal_products USING hnsw(product_embedding vector_cosine_ops);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f0c13cd0>)])]>

Hybrid search

Here is the hybrid search query.

code_block: <ListValue: [StructValue([('code', "CREATE EXTENSION google_ml_integration;\r\n\r\n-- BM25 keyword results\r\nWITH keyword_results AS (\r\n SELECT uniq_id, product_name, \r\n ROW_NUMBER() OVER (ORDER BY product_description <@> 'California') AS rank_kw\r\n FROM cymbal_products\r\n ORDER BY product_description <@> 'California'\r\n LIMIT 10\r\n),\r\n-- Semantic vector results\r\nsemantic_results AS (\r\n SELECT uniq_id, product_name, \r\n ROW_NUMBER() OVER (ORDER BY product_embedding <=> google_ml.embedding('text-embedding-005', 'trees that grow taller than houses')::vector) AS rank_vec\r\n FROM cymbal_products\r\n ORDER BY product_embedding <=> google_ml.embedding('text-embedding-005', 'trees that grow taller than houses')::vector\r\n LIMIT 10\r\n)\r\n-- Reciprocal Rank Fusion (RRF) to merge and score both lists\r\nSELECT COALESCE(k.uniq_id, s.uniq_id) AS uniq_id,\r\n COALESCE(k.product_name, s.product_name) AS product_name,\r\n COALESCE(1.0 / (60 + k.rank_kw), 0) + COALESCE(1.0 / (60 + s.rank_vec), 0) AS rrf_score\r\nFROM keyword_results k\r\nFULL OUTER JOIN semantic_results s ON k.uniq_id = s.uniq_id\r\nORDER BY rrf_score DESC\r\nLIMIT 5;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f01f0c11710>)])]>

The resulting output is identical to the AlloyDB hybrid search results shown above.

Watch it in action

Watch how this all comes together in this demo video.

Relevant resources

We are incredibly excited to work with Tiger Data and cannot wait to see how you leverage native BM25 support to build faster, smarter, and simpler AI applications. Turn on the pg_textsearch extension today, and experience the ultimate hybrid search engine experience with AlloyDB and Cloud SQL.

Want to get started? Check out”

AlloyDB resources

New to AlloyDB? Discover AlloyDB with a 30-day free trial
Choose a vector index in AlloyDB AI
AlloyDB BM25 documentation
AlloyDB hybrid search UDF documentation

Cloud SQL resources

Cloud SQL BM25 documentation

Tiger Data pg_textsearch Release Page

How a solo founder runs a five-continent tender platform on AlloyDB and MCP

Thu, 17 Sep 2026 16:00:00 +0000

Editor's note: Lucius AI, a tender-intelligence startup covering markets across five continents, runs its entire data platform on AlloyDB for PostgreSQL with a single operator. By migrating semantic search to a ScaNN index and managing database operations through Model Context Protocol (MCP), query latency dropped by 47x while automating day-to-day administrative tasks via MCP.

Executive summary

Lucius AI runs a global tender platform spanning more than 210,000 tenders across the UK, EU, India, and Australia, requiring minimal operational overhead for a solo founder.
Lucius AI deployed AlloyDB for PostgreSQL to consolidate its relational catalog, audit logs, and vector embeddings into a single managed database engine.
Migrating semantic search to a ScaNN index lowered query latency from 1.14 seconds to 24 milliseconds — a 47x speedup on a representative production query.
Connecting an AI agent to AlloyDB using the Model Context Protocol (MCP) helps Lucius AI automate query analysis, data freshness checks, and incident forensics under strict least-privilege permissions.

Making tender intelligence work as a company of one

Lucius AI helps businesses bidding on public contracts evaluate opportunities across global markets. The platform ingests public procurement notices from the UK, the EU, the US and Canada, Australia and New Zealand, India and Singapore, alongside World Bank donor-funded notices across Africa and Asia. Lucius AI analyzes tender documents using Gemini to generate compliance matrices, bid recommendations, and draft responses citing original source pages. For small and mid-sized suppliers, this replaces days of manual document reviews and costly external consulting.

Running a platform of this scope requires extensive operational coordination:

Nightly ingestion from thirteen public procurement sources
A catalog of more than 210,000 tenders, including tens of thousands open for active bidding
Two production regions on Cloud Run: Europe, and an Australian deployment on its own AlloyDB cluster with customer-managed encryption keys (CMEK) for defense-adjacent customers
Ongoing analytics, performance tuning, data validation, and incident response

Managing these responsibilities without dedicated data engineering or database administration teams requires offloading operational maintenance. Lucius AI addressed this challenge on two fronts: using AlloyDB for PostgreSQL as the core system of record, and connecting an AI agent through the Model Context Protocol (MCP) to safely execute database operations.

Consolidating systems into AlloyDB

Rather than deploying separate relational databases, vector databases, and log stores, Lucius AI houses all core data in AlloyDB for PostgreSQL. The relational tender catalog, document metadata, audit logs, and vector embeddings reside in the same database engine. Storing vector embeddings alongside relational rows avoids managing separate vector stores, establishes a unified backup schedule, and centralizes identity management.

Authentication relies strictly on Cloud IAM. Services connect using dedicated Google Cloud service accounts mapped to database roles scoped to specific access requirements, without storing database passwords in application environments. Database reliability is managed natively by AlloyDB through automated backups and point-in-time recovery, avoiding custom disaster recovery procedures.

In production, this consolidated architecture supports:

More than 210,000 tenders in the catalog, with embeddings stored directly alongside them
Rebuilding the semantic index embedded 115,820 records in 10.6 minutes with the Gemini embedding model, for around three dollars in API spend; AlloyDB auto embeddings now keep those vectors current.
Retrieval reranking executed directly inside the database using the ai.rank function — with mean latency of 77-milliseconds - returning the most relevant results for search queries without requiring a standalone reranking microservice

Accelerating semantic search by 47x

Semantic search across the tender catalog initially relied on unindexed vector comparisons, where a representative query took 1.14 seconds. Migrating this workload to a ScaNN index in AlloyDB reduced query latency to 24 milliseconds — a 47x improvement.

The index recommendation originated from the AI agent during an automated performance audit, where it benchmarked the query plan before preparing the index migration.

Automating database operations with MCP

To delegate routine administrative tasks, Lucius AI configured the open-source MCP Toolbox for Databases using the prebuilt alloydb-postgres server.

Operational delegation requires strict access controls. The agent connects using a dedicated PostgreSQL role granted SELECT across the schema and UPDATE on a single operational table. Destructive commands (DROP, DELETE, TRUNCATE) are omitted, restricting agent actions to authorized operational boundaries.

Under this configuration, the AI agent performs regular database operations across four key areas:

On-demand analytics: Compiles retention cohorts, activation funnels, and catalog coverage by country via ad hoc SQL queries, removing the need to build and maintain manual dashboards or complex analytical pipelines.
Performance optimization: Performs query-plan inspections and index analysis, such as identifying the ScaNN indexing strategy.
Incident forensics: In response to an external security probe, the agent parsed audit logs to reconstruct the request timeline in minutes, verifying that tenant isolation remained intact.
Automated data-quality checks: Evaluates ingestion watermarks and freshness across all thirteen procurement sources every morning.

For teams adopting this architecture, establishing a progressive permission structure provides clear guardrails: start with read-only access, expand permissions as requirements dictate, and keep destructive operations restricted to human administrators.

Looking ahead

Lucius AI is planning three technical initiatives to further reduce operational overhead:

Automated vector embeddings in AlloyDB AI: After validating ai.initialize_embeddings across the full catalog, a weekly maintenance job uses ai.refresh_embeddings to update vectors.
Columnar engine acceleration: Having enabled AlloyDB’s columnar engine with auto-columnarization, the database identified and stored 40 frequently queried columns across four tables in memory within a day, accelerating reporting queries without a separate analytical store.
Managed Remote MCP Server: Transitioning from self-hosted Toolbox processes to Google Cloud's fully managed Remote MCP Server for AlloyDB will offload MCP server hosting and maintenance.

By anchoring core data in AlloyDB and managing routine operations through MCP, Lucius AI demonstrates how a single engineer can build and operate a resilient, multi-region procurement platform.

To explore Lucius AI, visit ailucius.com. To evaluate AlloyDB for PostgreSQL, deploy an AlloyDB cluster to test performance against your own workloads.

Databases

Bringing conversational analytics to Gemini with Agentic Data Cloud connectors

A new way to query enterprise data

How Data Cloud connectors work

Choosing between connectors and data agents

Built-in access controls and governance

How to create Data Cloud connectors

Why use Data Cloud connectors over custom MCP

Get started today

What’s new with Google Data Cloud

November 2 - November 6

October 26 - October 30

October 19 - October 23

October 12 - October 16

October 5 - October 9

September 7 - September 10

August 31 - September 4

July 6 - July 10

June 1 - June 5

May 11 - May 15

April 20 - April 24

April 13 - April 17

April 6 - April 10

March 23 - March 27

March 16 - March 20

February 23 - February 27

February 16 - February 20

February 2 - February 6

January 26 - January 30

January 19 - January 23

January 12 - January 16

What’s new with Google Data Cloud - 2025

New capabilities to power agents at scale with Google's Agentic Data Cloud

Activate all your data for agents with the borderless Lakehouse

Deliver accurate, trusted answers by grounding every agent in enterprise context

Transforming practitioner experiences with agents

Engineering a data infrastructure built for agent scale

Get started today

Unify operational and Lakehouse data with direct Iceberg access in AlloyDB for PostgreSQL

Decisions at the moment of action

Direct Iceberg access in AlloyDB (Preview)

AlloyDB sync for the lowest-latency reads (GA)

Lakehouse federation (GA)

Choose the right approach and get started

AlloyDB Express: Instant, scale-to-zero PostgreSQL for developers and coding agents

Accelerate development without compromising security

AlloyDB Express brings built-in AI capabilities

Start with the free tier and pay only for what you use

Get started today

Beyond Migration: Using Database Migration Service for disk shrinking

Why should I shrink my disk?

How DMS works under the hood: Replication between Cloud SQL for SQL Server instances

Disk shrinking with minimal downtime

Comparison: Native shrink vs. DMS migration

Pre and post-cutover essentials: Going beyond database replication

Pre-cutover preparation: Migrating server-level objects

Post-cutover remediation: Resolving orphaned database users

Summary and next steps

Near-zero downtime upgrades: Announcing Cloud SQL for MySQL blue-green deployments

Why the upgrade dilemma matters to your business

The technical differentiators

Simple to orchestrate, built to scale

Try it today

Introducing the AlloyDB Omni web UI for easier cluster observability

What is AlloyDB Omni?

Key Web UI capabilities at a glance

Production-safe diagnostics and profile management

Dual-layer observability: System and query monitoring

Real-time Columnar Engine diagnostics

Database flag and parameter auditing

Built-in AlloyDB AI and tooling hub

Operational scope and prerequisites

Get started

How to upgrade embedding models in AlloyDB AI with zero downtime

How AlloyDB AI simplifies vector lifecycle management

Step-by-step migration walkthrough

Step 0: Prerequisites and preflight checks

Step 1: Add the new vector column

Step 2: Initialize auto-embeddings and run backfill

Monitoring auto embeddings progress