Databases

What’s new with Google Data Cloud

Thu, 15 Oct 2026 16:00:00 +0000

October 12 - October 16

Looker Data Apps are now in Preview to build and host interactive, agent-generated applications natively inside Looker
Generated using natural language prompts, the Antigravity CLI, or Claude Code, these interactive web applications, natively hosted inside Looker, transform business reporting into operational systems of action. Users can test scenarios, adjust parameters, and trigger workflows in downstream platforms while inheriting LookML semantic modeling, row-level security, and access controls with no runtime LLM token costs. Read the announcement blog.
Build end-to-end data science pipelines with the Data Agent Kit
Following the general availability of the Data Agent Kit, a new end-to-end walkthrough demonstrates how data teams can build, test, and schedule machine learning pipelines directly from their preferred code editors. Explore the walkthrough and codelab.
Manage dashboard agents as code to bring CI/CD and version control to LookML dashboards
Data teams can now declare agent parameters, configurations, and prompt instructions alongside dashboard files in code. This integration enables standard developer workflows for conversational BI, including Git version control, peer review, branching, and automated CI/CD validation before agent modifications reach production end users. Read the technical guide on the Google Developer Community.
Simplify AI search with native AlloyDB hybrid search and Reciprocal Rank Fusion
AlloyDB for PostgreSQL simplifies hybrid search architectures by combining vector similarity search and keyword BM25 retrieval within a unified database engine. Using Reciprocal Rank Fusion (RRF), developers can merge lexical scoring and semantic embeddings into a single ranked result without writing a complex SQL query. This unified approach removes the operational overhead of running and synchronizing external search engines alongside operational databases for retrieval-augmented generation (RAG) and search applications. Explore hybrid search in AlloyDB.

October 5 - October 9

Data Agent Kit is now generally available to bring Google Data Cloud to any coding agent
Data Agent Kit provides a free collection of Model Context Protocol (MCP) tools and Google-authored agent skills that connect more than 15 Google Data Cloud services directly to coding agents across VS Code, Antigravity, Cursor, Claude Code, and Codex. This release expands support for developers and data practitioners to inspect schemas, author queries, and build end-to-end data pipelines in natural language directly from their IDE or CLI. Read the announcement blog.
Spanner Omni is now generally available for deploy-anywhere, multi-model workloads
Spanner Omni brings Spanner's distributed SQL, graph, vector, and key-value capabilities to any environment, including on-premises data centers, other public clouds via Kubernetes or VMs, and local developer laptops. Organizations can build and run multi-model applications with consistent APIs and query engines across hybrid and multi-cloud footprints while seamlessly scaling to managed Spanner on Google Cloud. Read the launch blog and explore how Spanner also removed cumulative mutation limits for DML transactions.
AlloyDB delivers PostgreSQL for agents and native BM25 hybrid search
We announced PostgreSQL for agents in AlloyDB, introducing an agentic database architecture engineered for real-time data access at agent scale with workload isolation. In addition, native BM25 ranking is now in Preview for AlloyDB and Cloud SQL, allowing developers to combine keyword search and vector embeddings directly in PostgreSQL without maintaining an external search engine. For hybrid deployments, the AlloyDB Omni RPM Orchestrator is now generally available, and you can dive deeper in our new Build AI apps and agents with AlloyDB eBook.
Lakehouse runtime catalog adds regional endpoints across 22 regions and GA support for flexible column names
Lakehouse runtime catalog regional endpoints are now live across 22 Google Cloud regions across the Americas, Europe, Asia-Pacific, and the Middle East, supporting both the Apache Iceberg REST catalog endpoint and the Apache Hive catalog endpoint to help customers meet strict data residency and sovereignty requirements. In addition, flexible column names for Apache Iceberg tables are now generally available by default across Iceberg external tables and managed tables, enabling seamless querying of columns containing spaces, hyphens, and Unicode characters. We also announced the Preview of cross-cloud caching and connections for the borderless Lakehouse.
Unlock predictive and agent-ready insights with BigQuery augmented analytics TVFs, TabFM, and identity columns
BigQuery introduced augmented analytics table-valued functions (TVFs): six built-in functions that automate metric diagnostics, trend analysis, and pattern discovery directly where your data lives and integrate as skills for AI agents. We also introduced TabFM in BigQuery to bring tabular foundation model capabilities to predictive analytics, alongside BigQuery identity columns to automatically generate sequential surrogate keys and simplify data warehouse pipelines.
Dataflow adds Pause/Resume and NVIDIA RTX PRO 6000 Blackwell GPU support for large-scale AI workloads
New Dataflow capabilities make it easier to run high-throughput streaming and batch AI pipelines at scale, including Pause/Resume job controls and support for NVIDIA RTX PRO 6000 Blackwell GPUs. And for teams running Apache Spark workloads, learn how to mitigate compute stockouts and maximize availability using flexible VMs.
Unlock up to 3x higher QPS and microsecond latency with Memorystore for Valkey 9.1
Memorystore for Valkey 9.1 is now available, delivering up to 3x higher queries per second (QPS) and microsecond-level caching latencies to accelerate high-throughput applications and real-time AI agent state management. Read the blog.
Customer momentum across Google Data Cloud — Yahoo, PayPal, Surescripts, Airwallex, and Lucius AI
Organizations across industries are scaling their data and AI foundations on Google Data Cloud. Yahoo uses flexible VMs in Managed Service for Apache Spark to reduce provisioning failures by 85%, while PayPal migrated mission-critical analytics workloads to Managed Service for Apache Spark to scale and reduce operational overhead. On the database front, Surescripts migrated 30.5 billion annual healthcare transactions to AlloyDB with 99.998% uptime, Airwallex manages more than 5,000 databases with 99.99% availability on Cloud SQL with a team of four DBAs, and Lucius AI runs a 210,000-tender global platform on AlloyDB and MCP with 47x faster ScaNN vector search.

September 7 - September 10

Pub/Sub SMTs can now AI Inference your Gemini Enterprise Agent Platform models!
Pub/Sub AI Inference SMTs allow you to apply inference on an incoming stream of events using models hosted in Gemini Enterprise Agent Platform. The model’s prediction is appended to your event, making it available for downstream processing in your data warehouse (like BigQuery) or operational database (like BigTable). This feature, now generally available, can dramatically simplify or enhance anomaly detection systems you are operating.
PostgreSQL Source Connector is now generally available in Managed Service for Apache Kafka!
Managed Service for Apache Kafka’s PostgreSQL connector allows customers to capture changes from their PostgreSQL database and ingest them into their Kafka infrastructure with low latency. This source connector is compatible with Cloud SQL for Postgres, AlloyDB, and self-managed PostgreSQL databases. Try this along with our entire portfolio of managed connectors, including MirrorMaker 2.0, BigQuery, Cloud Storage, and Pub/Sub! E-mail kafka-hotline@google.com if you have questions or feedback!

Pause-on-failure for Dataflow batch jobs is GA
Dataflow pause-on-failure enables you to preserve the state of a batch Dataflow job before it fails. By pausing your Dataflow job, you can address issues that are external to the pipeline and resume processing without losing completed work. This helps you better manage resource costs and improve job reliability when you face temporary outages or capacity constraints.

The insertAll API is now the BigQuery Storage Write API (REST)
The legacy insertAll streaming API is now rebranded as the BigQuery Storage Write API (REST). By dropping the "legacy" label, developers can confidently build long-term HTTP-based streaming workflows. This stateless JSON-over-HTTPS endpoint offers a lightweight alternative to heavy gRPC libraries—ideal for serverless web apps, IoT telemetry, and AI logging. The transition is seamless for existing users, requiring zero code changes and offering 100% backward compatibility. However, the Storage Write API (gRPC) version remains the recommended standard for high-throughput, continuous pipelines.

August 31 - September 4

Stateful processing is available in BigQuery continuous queries in Preview
Stateful operations significantly expand what’s possible with BigQuery continuous queries. This feature allows users to leverage functions like JOINs, aggregations, and windowing functions directly in their streaming queries. Now you can calculate metrics over time (for example, a 30-minute average) to power your downstream applications and AI agents with much richer, real-time signals.

Try out our feature here and share your feedback with bq-continuous-queries-feedback@google.com!
Synthetic data generator tool is available for Managed Service for Kafka
You’ve launched your first Kafka cluster. Now what? The next thing to do is to produce some data to the cluster, but that involves modifying a client application somewhere or spinning up a virtual machine. The synthetic data generator tool, now generally available, can start sending mock data to your cluster in 3 clicks, and will get data streaming into your cluster in less than two minutes. The perfect utility for those moments you just want to test your cluster and new features. Try our quickstart today!

Dataflow pipeline updates are faster & more flexible
Dataflow pipeline updates can now stop-and-replace pipelines, a major addition to the existing in-place-update feature. The new parallel pipeline option accelerates the migration between the old & new pipeline, resulting in reduced disruption to your business. You can also set a timeout on drains that prevents runaway costs for your pipeliness in the event of stuck processing. This feature is generally available. Try it here!

July 6 - July 10

New Lakehouse managed tables now in preview
Lakehouse tables for Apache Iceberg are now in preview and available in the console. By using Google-managed Apache Iceberg tables in Lakehouse, you can eliminate the overhead of maintaining duplicate data pipelines and complex synchronization logic between BigQuery and open-source engines. This unified table format delivers native, multi-engine read and write interoperability, allowing you to run concurrent DML/DDL operations across diverse analytics tools on a single, shared storage layer. Built-in automated table management handles painful background optimization tasks like compaction and partition tuning, freeing up your team to focus on building rather than managing storage maintenance.

June 1 - June 5

Beyond the Query: Powering AI Agents with Bigtable, Firestore & Memorystore
Discover the latest advancements in Google Cloud's NoSQL Database portfolio, including Bigtable, Firestore, and Memorystore. This series is designed for a broad audience: whether you are exploring these databases for the first time or are an existing user looking to leverage the new capabilities announced at Next '26.

Register here to secure your spot!

Cloud Engineer's AI Toolkit Workshops: Solve data-driven challenges with BigQuery, AlloyDB, Gemini and more. Hosted by Google Cloud Labs, this highly technical event is built specifically for Platform Engineers, SREs, and cloud infrastructure teams ready to bridge the gap between AI prototypes and production-grade deployments. Look out for more locations coming soon

Toronto - June 25 (Data Cloud) | RSVP Here
Chicago - June 30 (Data Cloud) | RSVP Here
Start a 10-day Bigtable free trial with a 1 node SSD cluster and up to 500GB of storage capacity. With no credit card required to start, you can easily ingest workloads and manage workloads that require low-latency, high-throughput, and predictable access. Plus, new Google Cloud customers get $300 in free credits on signup.

May 11 - May 15

Managed Service for Apache Airflow has launched a wave of new features, including the general availability of Airflow 3.1, AI-powered agentic troubleshooting, a new managed Airflow MCP Server for custom agent integration, and declarative YAML-based orchestration pipelines—discover all the details in the full blog post.

April 20 - April 24

Google-built ODBC Driver for BigQuery is now available in Preview
We are excited to announce the launch of the new, Google-built ODBC driver for BigQuery. This new open-source driver provides a direct, high-performance connection for applications to BigQuery and is developed entirely in-house by Google. Download a new driver and connect your application to BigQuery.

April 13 - April 17

We announced we are reintroducing Data Studio to play a significant role in the AI era, expanding from data visualizations and reports to host BigQuery conversational agents and data apps built in Colab notebooks.
We announced BigQuery Graph is now available in preview, offering an easy-to-use, highly scalable graph analytics solution, empowering data professionals to model, analyze and visualize massive-scale relationships in an entirely new way.

April 6 - April 10

We introduced Conversational Analytics for Looker Embedded environments, enabling users to add natural language experiences to their own custom data-driven applications, powered by Gemini.
We expanded Looker’s capabilities for faster ad-hoc analysis, with the introduction of self-service Explores, enabling you to bring your own data to Looker’s semantic layer and gain instant access to insights in a governed data environment.

March 23 - March 27

We showed you how you can scale your reads with Cloud SQL autoscaling read pools. This feature allows you to provision multiple read replicas that are accessible via a single read endpoint and to dynamically adjust your read capability based on real-time application needs.
Our customers are leveraging the full power of Conversational Analytics and Looker to drive major business and technical breakthroughs in the AI era. Companies like Telenor, Pet Circle, Fluent Commerce, Lighthouse Intelligence, Wego, and ROLLER are turning data into insights and actions, grounded by Looker’s semantic layer.

March 16 - March 20

We introduced an enhanced Gemini assistant in BigQuery Studio, transforming the agent from a code assistant into a fully context-aware analytics partner.

February 23 - February 27

We introduced managed and remote MCP support for Google Cloud databases, including AlloyDB, Spanner, Cloud SQL, Bigtable and Firestore, to power the next generation of agents. This announcement extends the ability for AI models to plan, build, and solve complex problems, connecting to the database tools our customers leverage daily as the backbone of their work environment.
We outlined how you can build a conversational agent in BigQuery using the Conversational Analytics API to help you build context-aware agents that can understand natural language, query your BigQuery data, and deliver answers in text, tables, and visual charts.

February 16 - February 20

Our customers are leveraging the full power of Looker to drive major business and technical breakthroughs. Companies like Arrive, Audika, Carousell, Framebridge, GumGum, Intel, Overdose Digital, Ocean Network Express, Subskribe and Promevo are leveraging Looker’s newest AI-driven capabilities, including Conversational Analytics, to transform data to insights and actions, and empower their entire organization with a single source of truth, powered by Looker’s semantic layer.

February 2 - February 6

Join us on March 4 for our webinar, Win Your AI Strategy with Cloud SQL Enterprise Plus, to learn how to power your generative AI workloads with 3x higher performance and 99.99% availability. Register today to discover how to build a scalable, enterprise-grade foundation for your most demanding AI applications.

January 26 - January 30

We introduced Conversational Analytics in BigQuery, which allows users to analyze data using natural language. Conversational Analytics in BigQuery is an intelligent agent that generates, executes and visualizes answers grounded in your business context directly in BigQuery Studio, making data insights for data professionals more conversational.
We outlined how data products have become the foundation for AI agents, providing the context needed to make autonomous agents reliable and trusted for real business use, backed by organized business logic and semantic understanding.
We highlighted how you can supercharge data analytics workflows, and outlined Google Cloud’s AI agent offerings for data engineering, data science, and development tools, so you can integrate agentic workflows in your applications, empower your teams and speed discovery.

January 19 - January 23

We have fundamentally reimagined Firestore with pipeline operations for Enterprise edition. Experience a powerful new engine featuring over a hundred new query features, index-less queries, new index types, and observability tooling to improve query performance. Seamlessly migrate using built-in tools and leverage Firestore’s existing differentiated serverless foundation, virtually unlimited scale, and industry-leading SLA. Join a community of 600K developers to craft expressive applications that maximize the benefits of rich queryability, real-time listen queries, robust offline caching, and cutting-edge AI-assistive coding integrations.
Introducing Google Cloud SQL on MSSQLTips: We are highlighting a new technical guide published on MSSQLTips titled "Introducing Google Cloud SQL." This article serves as an essential resource for SQL Server administrators and developers exploring Google Cloud's fully managed database service. It provides a detailed overview of Cloud SQL capabilities, including high availability, security integration, and the seamless transition of on-premises SQL Server workloads to the cloud, making it an ideal resource for those planning their migration strategy.
We are excited to announce the Public Preview of Microsoft Entra ID (formerly Azure Active Directory) integration with Cloud SQL for SQL Server. Designed to tackle the challenge of identity sprawl in multi-cloud environments, this integration allows organizations to govern database access using their existing Microsoft identity infrastructure. Key benefits include centralized identity management, enhanced security features like Multi-Factor Authentication (MFA), and simplified user administration through direct group mapping. This feature is available for SQL Server 2022 and supports both public and private IP configurations.

January 12 - January 16

Google-built JDBC Driver for BigQuery is now available in Preview
We are excited to announce the launch of the new, Google-built JDBC driver for BigQuery. This new open-source driver provides a direct, high-performance connection for Java applications to BigQuery and is developed entirely in-house by Google. Download a new driver and connect your Java application to BigQuery.
Troubleshoot Airflow tasks instantly with Gemini Cloud Assist investigations: Cloud Composer just got smarter. We are excited to announce that Gemini Cloud Assist investigations are now available directly within Cloud Composer 3. Instead of manually sifting through raw logs, you can now simply click "Investigate" on a failed Airflow task. Gemini analyzes logs and task metadata to identify failure patterns—such as resource exhaustion or timeouts—and provides actionable recommendations driven by Gemini Cloud Assist to resolve the issue. This integration shifts the debugging experience from manual toil to automated root cause analysis, significantly reducing the time required to restore your pipelines. Learn more about AI-assisted troubleshooting.

Managed Apache Iceberg at scale: How Spanner powers Lakehouse runtime catalog

Tue, 06 Oct 2026 16:00:00 +0000

As customers modernize to lakehouse architectures, they are standardizing on open formats such as Apache Iceberg to create a shared data estate across compatible engines. This enables you to build AI-native lakehouses that turn data to semantic knowledge, enable proactive action, and operate at an agentic scale.

One of the key components of a lakehouse is the catalog, and in the Apache Iceberg environment, that usually means the Iceberg REST Catalog. An Apache Iceberg catalog is responsible for maintaining table pointers, handling atomic commits, and serving as the single source of truth for table locations. But as large enterprise organizations modernize to lakehouses, they have begun to realize that they need a highly scalable and available managed catalog as a part of their lakehouse architecture. This becomes even more important as querying scales with agents. To support a large amount of repeated small queries from agents, you will need to build on top of a managed catalog that provides atomicity, consistency, availability and concurrency at massive scale.

In this blog, we explore the challenges a managed catalog faces in modern cloud environments at agent scale, and show you how Google Cloud’s serverless Lakehouse runtime catalog can help address them. Powered by Spanner, Google Cloud’s always-on database with virtually unlimited scale, and built to meet the open Apache Iceberg REST catalog specification, the Lakehouse runtime catalog is the highly scalable and available foundation you need for the agentic era.

Challenges a managed catalog needs to solve in the Lakehouse

When speaking with data engineers and infrastructure leads running production analytics at scale in a Lakehouse, the following core pain points consistently emerge and need to be solved by a managed catalog:

Atomic commits and concurrency control: Iceberg guarantees ACID transactions via optimistic concurrency control (OCC). A managed catalog must implement a bulletproof atomic compare-and-swap (CAS) operation to swap the current metadata pointer.

High availability and operational maintenance: Because queries fail immediately if the catalog is down, a managed catalog becomes a critical Tier-1 service.

Scaling the database backing the catalog: Catalog architects typically face a difficult trade-off when choosing a backing database for table metadata and state. Traditional scale-up relational databases provide SQL and ACID transactions, but hit vertical CPU, memory, storage and connection limits under heavy concurrent read/write loads unless manually sharded which incurs a huge operational overhead; while scale-out database systems are either eventually consistent, hard to manage, not enterprise-ready, or all of the above.

Table maintenance coordination: A catalog alone does not optimize data; you must build and operate ancillary pipelines for compaction, snapshot expiration, manifest rewriting, and orphan file cleanup.

Governance and security: A catalog acts as the security gatekeeper. The catalog must implement and maintain:

Authentication protocols (e.g., OAuth2 token exchange, IAM federation)
Access control down to namespace and table levels
Vended storage credentials (e.g., generating short-lived tokens so query engines don't need broad, direct storage credentials)

Lakehouse runtime catalog

To solve these challenges, we built the Lakehouse runtime catalog (GA) with support for Iceberg Rest Catalog. The Lakehouse runtime catalog is a fully serverless, highly available, and unified metadata registry designed from the ground up to support modern open table formats like Apache Iceberg. By natively implementing the Apache Iceberg REST Catalog specification, the Lakehouse runtime catalog decouples metadata discovery from compute engines, helping ensure multiple Iceberg-compatible engines can access a shared data estate and enabling you to take your workloads to production sooner. We’ve helped many customers streamline the migration of their managed catalogs. For example, Etsy migrated its catalog to Lakehouse runtime catalog, joining data in place to accelerate pipeline queries by 60%.

This approach offers a number of architectural benefits:

Open APIs: Support for Iceberg Rest Catalog enables different teams to use their preferred analytics tools on a single, unified dataset.
Multi-engine interoperability: Once registered, tables are immediately discoverable and queryable across Google Cloud Managed Service for Apache Spark, BigQuery, and open-source engines via standard REST interfaces.
Read/write interoperability for Iceberg tables: Leverage Iceberg-compatible engines such as BigQuery, Managed Spark to write to Iceberg tables registered in the Lakehouse runtime catalog. Customers can also use Managed Spark to write to Iceberg tables in external catalogs.
Fully managed Iceberg storage with enterprise-grade features: Use Google's differentiated infrastructure to run analytics with performance on Iceberg tables. This gives you the benefits of open-source flexibility plus performance, scale, governance, and multimodal processing.
Zero data copy: Table definitions point directly to your existing data in the underlying object store. You do not move, rewrite, or duplicate your underlying data.
Bi-directional catalog federation across clouds: Access data from Databricks Unity, Snowflake Horizon and AWS Glue with support for vended credentials and OIDC token exchange. This lets you bring Google AI directly to your AWS and Azure data.
Secure access using credential vending: The catalog supports multiple authorization mechanisms, letting you choose between credential vending and end-user credentials. This means that you can access tables with modern mechanisms such as credential vending without needing direct access to the files in the underlying object store (Cloud Storage, AWS S3, Azure Blob Storage).
AI-powered context and governance: The Lakehouse runtime catalog integrates directly with Knowledge Catalog and Cloud IAM, allowing you to define trusted context for your agents, and apply table-level security consistently across all compute engines. Get out-of-the-box search, lineage, and insights for Iceberg tables in the catalog.
Atomic commits and concurrency control, high availability and scalability: Backed by Google’s planet-scale infrastructure and Spanner, you get the high availability, concurrency and scale you need for your metadata to scale with your data. Support for Cloud Storage dual-region and multi-region buckets enables failover use cases. It also provides reduced TCO due to serverless and no-ops environments, and scalability for any workload size.

What powers the Lakehouse runtime catalog?

The Lakehouse runtime catalog is a highly available, concurrent and scalable catalog with strong consistency guarantees because it is built on top of Spanner. Unlike traditional scale-up relational databases that hit vertical single-node ceilings, Spanner combines full relational SQL semantics and multi-table ACID transactions with the horizontal scale-out elasticity for both reads and writes of a NoSQL system.

Spanner makes the Lakehouse runtime catalog highly available through Spanner’s regional configurations with up to 99.99% availability. Spanner also delivers out-of-the-box scalability for Lakehouse runtime catalog: As a horizontally scalable database, Spanner does not require manual sharding and scales compute and storage independently and transparently. Spanner dynamically monitors data volume and query load, splitting and redistributing data ranges across nodes. Compute nodes scale dynamically based on CPU utilization and storage thresholds. Spanner automatically detects split-level overload and moves heavy splits away from overloaded nodes. Spanner also lets Lakehouse runtime catalog users eliminate the traditional trade-offs between relational consistency and distributed scalability, delivering Lakehouse runtime catalog’s industry-leading consistency guarantees. Lakehouse transactions are serializable — the order of transactions within the database is the same as the order in which clients observe the transactions to have been committed. This foundation allows Lakehouse users to operate at agent-scale.

Then, to further power agentic use cases, the Lakehouse runtime catalog integrates directly with Knowledge Catalog to easily discover lakehouse Iceberg tables and provide trusted context to agents. Knowledge Catalog leverages an efficient combination of full-text search and native vector search provided by Spanner; this approach enables better recall for search retrieval, pairing lexical keyword searches with semantic embeddings in a single query. Because both index types are built on the identical base dataset, they update with strict, transactional ACID consistency alongside base table DML operations. This removes operational overhead such as managing sync pipelines, and external-vector and full-text search systems. In short, the Lakehouse runtime catalog provides faster time-to-market for your agentic use cases.

Combine analytical and operational workloads

With Google Cloud’s borderless Lakehouse based on Apache Iceberg, you can combine your analytical data with your operational workloads. Use cases span combining data assets from your Lakehouse with OLTP data (from Spanner) for analytics, to low-latency serving applications where your Lakehouse assets are accessible in an operational database such as Spanner, to conversational analytics in first-party and third-party agents.

Below, in an example, you can see the Lakehouse runtime catalog with Spanner in action. Here, we combine analytical data for taxi trips in Manhattan (backed by Apache Iceberg) with operational data for taxi zones in Spanner to find the most congested traffic routes. The example also shows that you can also use a Conversational Analytics Agent to access the same data and get second-order insights.

Find out most congested traffic routes in Manhattan

Modernize to the borderless Lakehouse

Modernizing to Google Cloud’s Lakehouse minimizes data silos across analytics engines and agents, unifies multi-engine governance, provides trusted context to your agents and slashes operational TCO. Leveraging the Spanner-based Lakehouse runtime catalog helps prepare your modern cloud environments to operate at agent-scale. To learn more and get started with a free trial, visit the

Lakehouse web page and learn more about Spanner here.

AlloyDB: A unified database engine for hybrid search

Tue, 06 Oct 2026 07:00:00 +0000

For modern AI and RAG applications, achieving high search relevance requires that you combine at least two techniques: vector search for semantic context, and full-text search (FTS) for keyword precision. While AlloyDB for PostgreSQL supports both of these capabilities, managing them has traditionally required a more hands-on operational approach to ensure peak performance.

The challenge is not executing the searches, but the subsequent fusion of result sets. Merging results from the vector query (distance scores) and the FTS query (relevance scores) requires complex SQL queries or custom code in the application layer. This often means maintaining a separate system for fusion, score normalization, and re-ranking.

This article details how AlloyDB AI's hybrid search eliminates this complexity. We will explore how recent updates allow you to:

Simplify hybrid search: Consolidate complex SQL queries, or multi-step application workflows into a single, high-performance SQL function powered by Reciprocal Rank Fusion (RRF).
Optimize FTS performance: Use the new RUM extension to achieve low-latency relevance ranking and efficient phrase matching by storing word positions directly in the index.
Leverage industry-standard ranking: Utilize the new natively supported BM25 index for superior keyword-based scoring directly out of the box.
Expand search versatility: Execute queries against specialized external clusters, including Elasticsearch, OpenSearch, and Solr, using the new external search Foreign Data Wrapper (FDW) without leaving the AlloyDB environment.

The problem of the multi-step workflow

Before AlloyDB AI's native solution, achieving robust hybrid search was very demanding, especially for developers trying to keep this logic within the database using standard SQL. This approach required multi-step orchestration that was not only difficult to manage and maintain for two sources, but became virtually impossible to scale as additional sources were added. These steps included:

Executing vector search: Run a query using a vector column or vector index (for AlloyDB this can be a ScaNN index) to find top k results, generating vector scores.
Executing the FTS query: Run FTS query e.g. by using a generalized inverted index, or GIN, to find top k results.
Normalizing scores (the brittle step): Write complex SQL logic or custom application code to map both result sets onto a common scale. This logic is prone to breaking when data distributions change.
Performing cross-service joins and re-rankings: Use application memory to perform a complex FULL OUTER JOIN on document IDs, apply a weighted summation of the normalized scores, and finally sort the combined results in cases where the FTS search was done in a separate system than the one used for vector search.

This decentralized approach led to fragile score logic, increased latency, high operational load, and a dependency on application expertise for maintaining search quality.

Simplifying hybrid search architectures with AlloyDB

The key to simplifying this is to adopt RRF, which, because it is inherently rank-based, cleverly bypasses the brittle step of score normalization entirely. RRF consists of two parts:

1. The single source of truth

Instead of a multi-step fetch and join process, you simply call one SQL function, providing your search components as a declarative JSON array:

SQL

code_block: <ListValue: [StructValue([('code', 'SELECT id, score \r\nFROM ai.hybrid_search(\r\n search_inputs => ARRAY[\r\n -- Vector Component (Semantic) with dynamic embedding generation from natural language\r\n $json${\r\n "data_type": "vector",\r\n "limit": 10,\r\n "table_name": "documents",\r\n "key_column": "doc_id",\r\n "vec_column": "embedding",\r\n "distance_operator": "<=>",\r\n "query_vector": "ai.embedding(\'text-embedding-005\', \'alloydb search\')::vector"\r\n }$json$::jsonb,\r\n \r\n -- Text Component (Keyword)\r\n $json${\r\n "data_type": "text",\r\n "limit": 10,\r\n "table_name": "documents",\r\n "key_column": "doc_id",\r\n "text_column": "content",\r\n "query_text_input": "alloydb search"\r\n }$json$::jsonb\r\n ]\r\n);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f156bc313d0>)])]>

(Note: The unified ai.hybrid_search API isn't limited to just two components. It also natively supports external search sources via the new FDW).

2. Database-native orchestration

The hybrid_search() function executes the entire workflow in a single query plan, minimizing overhead and helping ensure transaction consistency. It uses:

Dynamic CTE generation: The function constructs dynamic SQL, creating Common Table Expressions (CTEs) for each component. Each CTE is responsible for calculating the positional rank (ROW_NUMBER()) of its results.
Kernel-level fusion: All ranked component results are immediately combined using a FULL OUTER JOIN based on the document ID.
Final RRF score: The combined ranks are used to calculate the final, unified score using the RRF formula:

By taking this approach, brittle score calculations and application-side joins become unnecessary. While Reciprocal Rank Fusion (RRF) is our current ranking algorithm, we intend to introduce additional merging and ranking options down the road.

The result: Performance and operational simplicity

Moving from complex external workflows to native SQL functions provides immediate, measurable benefits:

Metric	Multi-step application workflow (simulated by manual SQL)	AlloyDB AI-native hybrid_search() UDF
Code complexity	Complex score normalization functions, service calls, and application joins.	Zero external logic; a single, declarative SQL call.
Maintenance	Constant tuning of normalization formulas.	No tuning required; RRF is rank-based and distribution-agnostic.

Enhancing FTS performance: The RUM extension

AlloyDB AI's hybrid_search() function is designed to deliver comprehensive search relevance by combining high-performance vector search with FTS. While AlloyDB's native FTS capabilities are powerful, reliance on the standard PostgreSQL GIN index for the text component can present a bottleneck for advanced operations.

The challenge with GIN indexes is that they do not store the positional information of words. This limitation forces costly table scans to re-analyze content for:

Relevance ranking: Calculating search result scores based on word proximity and frequency
Phrase searching: Finding exact word order, which requires positional information.

Introducing the RUM extension for low-latency FTS

The RUM extension is a powerful index access method based on GIN that directly resolves these performance issues.

RUM's core advantage: Unlike the GIN index (which maps word -> [docID]), the RUM index stores the positional information of each word directly within the index (e.g., word -> [(docID1, [pos])], ...).
Performance benefits: This allows RUM to perform complex operations like ranking and phrase matching predominantly within the index itself, avoiding expensive heap scans. RUM provides significantly faster relevance ranking and efficient phrase and proximity searches.
Great for hybrid search: RUM is a critical complement to vector search (like ScaNN) within a hybrid search framework, as GIN’s potential latency makes it less suitable for a real-time hybrid approach.

The RUM extension is an excellent choice for ranking-heavy or high-concurrency search applications. By storing word positions directly in the index, RUM cuts out the need to re-scan table pages during ranking, giving you fast and sorted results. However, it comes with some trade-offs: slower index builds and a larger disk footprint. If your workload prioritizes quick queries and precise ranking over write speeds and storage density, RUM is an investment that's well worth it.

Measurable gains and integration

Adopting RUM directly bolsters the performance of FTS in AlloyDB, showing considerable gains for raw FTS queries, as well as gains on hybrid search that includes FTS queries. Below are performance statistics showing the speedup between using RUM compared to GIN on various BEIR datasets.

(Performance based on tests using the BEIR Natural Questions benchmark.)

RUM integrates with the specialized <=> distance operator for search and ranking, which is supported for use in hybrid search SQL calls, delivering the best of both worlds. The following codelab shows how to configure and use the RUM index with the Hybrid Search UDF.

Introducing BM25 Index: The modern ranking standard

We recently introduced the BM25 index to CloudSQL and AlloyDB in preview, bringing the industry-standard relevance ranking via the pg_textsearch extension. BM25 uses TF-IDF, which accounts for term frequency saturation and document length normalization, delivering significantly higher precision and quality for keyword-based search queries. By leveraging native BM25 indexes within AlloyDB AI, you can achieve superior ranking accuracy out of the box, so that exact-term matches and critical keywords score well without having to implement external search engines or complex custom scoring logic.

code_block: <ListValue: [StructValue([('code', "-- Install pg_textsearch extension\r\nCREATE EXTENSION pg_textsearch;\r\n\r\n-- Create the native BM25 index on the content column\r\nCREATE INDEX idx_docs_bm25 \r\nON cymbal_products \r\nUSING bm25 (product_description) \r\nWITH (text_config='english');\r\n\r\n-- Full text search query\r\nSELECT product_name, product_description <@> 'cherry tree' AS bm25_score \r\nFROM cymbal_products\r\nORDER BY bm25_score \r\nLIMIT 5;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f1569fae710>)])]>

External search: Extending versatility with foreign data wrappers

To further extend search capabilities, we also introduced the external search Foreign Data Wrapper (FDW) in AlloyDB AI. This lets you execute full-text searches against specialized external clusters, starting with Elasticsearch, OpenSearch, and Solr, and provides several key architectural advantages:

Optimized retrieval: Leverage ranking algorithms and a richer feature set from dedicated search backends without leaving the AlloyDB environment.
Unified SQL interface: Interact with external data, perform joins, and meld results using standard PostgreSQL SQL without losing the expressiveness of advanced FTS queries.
Strong portability: Maintain existing search infrastructure while benefiting from the simplified hybrid architecture offered by AlloyDB AI.

The following codelabs are end-to-end code guides walking through hybrid search with Elasticsearch and Solr integrations.

A unified architecture for AI-powered search

The true challenge of modern search is not technical in nature, but in achieving architectural simplicity and sustained operational stability. AlloyDB AI addresses this with a unified platform built on three core innovations:

Simplifying hybrid search: AlloyDB AI’s hybrid search function, powered by RRF, transforms a fragile, multi-step application workflow into a single, high-performance SQL call. This native implementation eliminates the need for complex score normalization and application-side joins, drastically lowering the operational and engineering cost while delivering consistently accurate and fast hybrid relevant results.
Optimizing FTS performance: To ensure the FTS component of Hybrid Search meets diverse application demands, AlloyDB AI offers distinct full-text search options. The RUM extension optimizes for low-latency performance by storing positional data directly in the index to enable faster relevance ranking and efficient phrase searches — important when you need fast query speeds. Alternatively, the BM25 index provides industry-standard relevance ranking, and is the preferred choice when prioritizing keyword scoring precision. You can select between these options depending on whether your focus is search latency or ranking precision.
Enhancing versatility through external search: The addition of the external search through FDW extends AlloyDB’s reach to specialized search backends like Elasticsearch. This lets you leverage the superior scale and advanced retrieval features of dedicated search clusters while maintaining a familiar PostgreSQL interface. By integrating these external results directly into the hybrid search framework, AlloyDB AI ensures that you can combine even the most massive text repositories with vector-based semantic insights.

By consolidating the complexity of scoring, joining, and re-ranking within the database kernel, and offering a dual-path approach to FTS — leveraging RUM for optimized, low-latency, internal FTS or external search for specialized, scalable backends — AlloyDB AI delivers a robust and self-contained search foundation. This coexistence is essential to the overall hybrid search story, providing the flexibility to choose the optimal path based on workload, data volume, and existing infrastructure. Now, you can focus on building intelligent application features, confident that your search architecture is both highly performant and easy to maintain.

Watch it in action

Watch how AlloyDB is the ultimate hybrid search engine.

Learn about BM25 support in AlloyDB.

Getting started

Ready to bring more speed and cost-efficiency to your AI workloads?

New to AlloyDB? Discover AlloyDB with a 30-day free trial.
Getting started with hybrid search: Set up a text-search index and select a vector search index. Once these are created, you can find some examples to perform various hybrid search queries.
External search: Create a foreign data wrapper and foreign table in AlloyDB to query external data from Elasticsearch, Solr, or OpenSearch.

Announcing Spanner queues: Transactional messaging for agentic workloads and beyond

Fri, 02 Oct 2026 16:00:00 +0000

AI agents don't just answer queries — they can autonomously issue refunds, manage inventory, execute multi-step handoffs, and orchestrate sub-agents, to name but a few complex agentic workflows. That frequently requires the agents to maintain internal state in an operational database while dispatching asynchronous actions through a separate messaging or event queue system. And unfortunately for the teams building these applications, managing two systems with disjointed commit points destroys transactional consistency in agentic systems.

Today, we are excited to announce the general availability of Spanner queues: native transactional messaging embedded directly within Spanner. Designed specifically for reliable agentic execution, with Spanner queues, creating a message is simply another write in your transaction. An agent's state change and intended downstream actions commit together atomically or fail completely.

Compare this to traditional approaches: When a database state update succeeds, but the action dispatch fails, your AI agent decides to act but doesn’t execute. If the message dispatch succeeds but the state transaction rolls back, your agent executes an action based on an invalid state. In asynchronous multi-agent coordination, retries, speculation, and race conditions amplify these failures, forcing developers to build complex outbox patterns, idempotency layers, and reconciliation workers — a heavy reliability tax on agentic architecture.

Key capabilities for agentic architectures

Spanner queues introduces several core capabilities to support these complex asynchronous workflows without introducing infrastructure overhead.

Atomic decide-and-act enqueue. Within a single Spanner read-write transaction, agents can update internal memory or state tables and enqueue tasks to peer agents simultaneously. Backed by Spanner's strict serializability and global external consistency, state changes and execution intent commit as a single atomic unit.

Scheduled execution and delays. Queue messages can be dispatched immediately upon commit or scheduled for future delivery. Agentic patterns such as delayed retries, scheduled agent check-ins, or SLA escalation timers can be enqueued transactionally alongside memory updates without requiring external cron schedulers or polling infrastructure.

Streaming SQL pull for agent workers. Autonomous agents consume tasks dynamically using streaming SQL reads. Agent runtimes stream incoming tasks, process them as capacity becomes available, and acknowledge task completion within a transaction to guarantee end-to-end task execution state reliability.

Episodic memory persistence and handoffs. Agent memory updates — including long-term episodic summaries, reflective state transitions, and context handoffs across sub-agents — can be persisted asynchronously and transactionally via queues. This guarantees that an agent's internal memory remains fully synchronized with its execution history without blocking real-time interactive turns.

Why Spanner queues are essential for autonomous agents

Transactional exactly-once agent execution: When an agent evaluates tool call results, state modification, reasoning persistence, and downstream tool invocation tasks are saved in one transaction. We guarantee at-least-once delivery and at-most-once ACK, enabling you to achieve exactly-once processing.

Robust multi-agent orchestration and handoffs: In multi-agent systems (A2A), handing off state from a primary agent to a specialist agent is represented as a durable, transactionally committed message. Specialist agent workers receive deliverable messages while maintaining a fully auditable lineage of agent interactions.

First-class timeouts and human-in-the-loop workflows: Agent workflows frequently require pausing for human approvals or scheduled follow-ups. Spanner queues handles timeout management natively: A single transaction records the pending approval state and schedules an automated escalation message, resolving whichever triggers first.

SQL-native observability for agent queues: Inspecting in-flight agent workloads, monitoring task backlogs, or auditing agent execution history can be accomplished with standard SQL queries over queue tables, avoiding opaque message-store black boxes.

Beyond agentic workflows

Beyond AI agents, Spanner queues serves as a flexible, multi-purpose messaging platform across a variety of traditional event-driven architectures. Whether powering real-time activity feeds in social applications, delivering live updates in news publishing, orchestrating order processing and inventory workflows in retail, or handling high-throughput asynchronous task processing and transaction notifications in financial services, Spanner queues provides a robust foundation for asynchronous message delivery and transactional event-driven workflows within your primary database.

Under the hood: Transactional mechanics of Spanner queues

Because Spanner queues are represented as first-class relational structures in Spanner, you define, inspect, and manage queues using familiar GoogleSQL.

1. Defining a queue and enqueuing atomically

When an agent decides to approve a customer refund, it updates the Orders table and dispatches an execution task to the OrderAgentTasks queue within a single ACID transaction:

code_block: <ListValue: [StructValue([('code', '-- Assume parent table:\r\n-- CREATE TABLE Orders (\r\n-- OrderId STRING(64) NOT NULL, ...\r\n-- )\r\n-- PRIMARY KEY (OrderId);\r\n\r\n-- Define the transactional queue table\r\nCREATE QUEUE OrderAgentTasks (\r\n OrderId STRING(64) NOT NULL,\r\n TaskId STRING(64) NOT NULL,\r\n TaskType STRING(64) NOT NULL,\r\n Payload JSON NOT NULL\r\n) PRIMARY KEY (OrderId, TaskId, TaskType), INTERLEAVE IN Orders;\r\n\r\n-- Inside a Read-Write Transaction:\r\n-- 1. Update business state atomically\r\n\r\n-- BEGIN TRANSACTION;\r\n\r\nUPDATE Orders\r\nSET Status = \'REFUND_APPROVED\',\r\n UpdatedAt = PENDING_COMMIT_TIMESTAMP()\r\nWHERE OrderId = @orderId;\r\n\r\n-- 2. Enqueue the asynchronous agent action in the same transaction\r\nINSERT INTO OrderAgentTasks (OrderId, TaskId, TaskType, Payload)\r\nVALUES (\r\n @orderId,\r\n @taskId,\r\n \'EXECUTE_REFUND\',\r\n JSON \'{"action": "execute_refund", "amount": 49.99}\'\r\n);\r\n\r\n-- COMMIT;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f156b7d2b90>)])]>

This ensures that the EXECUTE_REFUND task exists if and only if the order status successfully transitioned to REFUND_APPROVED.

2. Temporal scheduling and atomic cancellation

For workflows that depend on time — such as waiting up to 72 hours for a manager's approval before escalating — agents populate the system DeliverTime column to defer message visibility:

code_block: <ListValue: [StructValue([('code', '-- Schedule an automated escalation check-in 72 hours in the future\r\nINSERT INTO OrderAgentTasks (OrderId, TaskId, TaskType, Payload, DeliverTime)\r\nVALUES (\r\n @orderId,\r\n @escalationTaskId,\r\n \'ESCALATE_UNAPPROVED_ORDER\',\r\n JSON \'{"action": "escalate_to_supervisor"}\',\r\n TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 72 HOUR)\r\n);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f156b7d0f10>)])]>

If the manager approves the request after four hours, your application doesn't have to deal with phantom escalation alerts firing days later. In a single transaction, you update the order status and cancel the pending escalation task using a standard SQL DELETE:

code_block: <ListValue: [StructValue([('code', "-- BEGIN TRANSACTION;\r\n\r\nUPDATE Orders\r\nSET Status = 'MANAGER_APPROVED',\r\n ApprovedBy = @managerId\r\nWHERE OrderId = @orderId;\r\n\r\n-- Atomically cancel the pending delayed escalation task. `ASSERT_ROWS_MODIFIED 1` will\r\n-- act as a safeguard and cause a statement level error.\r\n\r\n-- A statement level error can be captured and the transaction can be\r\n-- user-aborted/cancelled in case the queue entry was already deleted.\r\n-- Otherwise the transaction will complete successfully regardless if the \r\n-- queue entry still exists and you'll only know if a queue entry was deleted by\r\n-- checking the number of rows affected by the DELETE.\r\nDELETE FROM OrderAgentTasks\r\nWHERE OrderId = @orderId\r\n AND TaskId = @escalationTaskId\r\n AND TaskType = 'ESCALATE_UNAPPROVED_ORDER',\r\nASSERT_ROWS_MODIFIED 1;\r\n\r\n-- COMMIT;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f156b7d23d0>)])]>

3. Streaming consumption, lease renewal, and atomic acknowledgment

Downstream agent workers consume tasks using the RECEIVE_<QueueName> table-valued function (TVF) over a streaming SQL connection (ExecuteStreamingSql). Spanner automatically manages message leases, returning a unique SpannerLeaseToken and expiration timestamp with each leased task:

code_block: <ListValue: [StructValue([('code', "-- Step 1: Consume tasks via a long-lived streaming SQL query\r\nSELECT\r\n OrderId,\r\n TaskId,\r\n TaskType,\r\n Payload,\r\n SpannerLeaseToken,\r\n SpannerLeaseExpirationTimestamp\r\nFROM RECEIVE_OrderAgentTasks(max_duration => '20m');"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f1569393910>)])]>

Because AI agent tasks often involve multi-turn LLM reasoning or external API calls that take longer than default lease windows, workers can actively extend their lease using the RENEWLEASE_<QueueName> function:

code_block: <ListValue: [StructValue([('code', '-- Step 2: Extend lease ownership during long-running agent execution\r\nSELECT *\r\nFROM RENEWLEASE_OrderAgentTasks(lease_tokens => [@leaseToken]);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f1569393ad0>)])]>

When the agent finishes executing its external tool (passing TaskId as the external API's idempotency key), it opens a read-write transaction to record the final state and acknowledge the message by deleting it with ASSERT_ROWS_MODIFIED 1:

code_block: <ListValue: [StructValue([('code', "-- Step 3: Atomically checkpoint agent results and ACK the message\r\n-- BEGIN TRANSACTION;\r\n\r\nUPDATE Orders\r\nSET RefundTransactionId = @externalRefundId,\r\n Status = 'REFUND_COMPLETED'\r\nWHERE OrderId = @orderId;\r\n\r\nDELETE FROM OrderAgentTasks\r\nWHERE OrderId = @orderId\r\n AND TaskId = @taskId\r\n AND TaskType = @taskType\r\nASSERT_ROWS_MODIFIED 1;\r\n\r\n-- COMMIT;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f1569392910>)])]>

Using ASSERT_ROWS_MODIFIED 1 protects your system against lease-expiration races. If a worker stalled due to a network pause and its lease expired, another worker may have already processed and deleted the task. When the stalled worker resumes and attempts to execute the statement, ASSERT_ROWS_MODIFIED 1 detects that the queue row is already gone and throws a statement-level error. Catching that error and aborting that transaction prevents stale workers from overwriting newer database state.

Spanner change streams vs. Spanner queues

Spanner change streams capture database data changes (inserts, updates, and deletes) in near real-time for downstream integration and auditing. While both change streams and queues allow applications to react to data changes, Spanner change streams are designed for continuous change data capture (CDC) and data streaming to downstream analytics or storage. In contrast, Spanner queues are explicitly designed for transactional task orchestration, supporting native message leases, scheduled deliveries, SQL-based pulling, and atomic acknowledgments within read-write transactions.

Get started

Spanner provides a unified foundation for agentic data, combining relational, hybrid search, graph, and key-value capabilities under strict global consistency. Spanner queues completes the agentic loop by enabling agents to transition seamlessly from reasoning over data to executing transactional actions within a single unified platform. Spanner queues are now generally available.

Sign up for the Spanner 90-day free trial and read up our public documentation to start building resilient, exactly-once agentic workloads by creating a queue table in your Spanner database today.

How to implement long-term AI agent memory in AlloyDB and Memorystore for Valkey

Fri, 02 Oct 2026 07:00:00 +0000

Enterprise AI agents need persistent memory to execute complex, multi-day workflows and long-horizon tasks. In this blog, we examine how a 2-tier memory architecture using Memorystore for Valkey for short-term buffer memory and AlloyDB AI for long-term persistent memory can help reduce token spend by up to 70%, while maintaining critical data and enterprise guardrails.

Imagine building a personalized travel agent designed to help users book vacations. The user interacts with the agent many times over the course of several days, asking questions that range from brainstorming itineraries to actual purchase intent. To provide a truly seamless experience, this agent must remember flight preferences (e.g. “I only want non-stop flights”), hotel budgets, and dietary restrictions (e.g. “I need Gluten Free dining options”) established in previous sessions. More importantly, it has to hold onto these core facts even when the conversation gets deep into the weeds of sightseeing recommendations and itinerary planning. The agent must ensure that all the follow-up questions and exciting details about places to visit doesn’t cause it to forget or overwrite the user's fundamental requirements and decisions.

Stateful workflows inherently conflict with LLM’s stateless nature

AI agents are being used for multi-turn conversations and long-running workflows, but large language models (LLMs) remain stateless across sessions. When a user returns to an agent days later, the model starts with an empty context window. Unless your application is built to reconstruct past context using long-term agent memory, your users have to explain their goals and context all over again, resulting in a frustrating and fragmented experience.

With million-token context windows now the norm, a common shortcut to this problem is "context stuffing" - dumping everything you can fit into the prompt at every turn, from raw chat histories to tool execution logs. In fact, this was a common pattern in the early days of AI model usage. But this shortcut quickly creates issues at scale: token costs multiply with every message, response times can drag out past 30+ seconds for otherwise simple prompts, and the model starts suffering from "lost in the middle" degradation, overlooking critical instructions buried in mountains of prompt text.

Another common workaround is to use rolling summaries; asking an LLM to periodically compress older messages into a summary paragraph. While this trims prompt size, LLM-based summarization is inherently lossy. After a few rounds of compression, subtle but important details get filtered out as background noise. A few turns later, your agent quietly breaks the exact constraints you set earlier. Clearly, you need a more scalable approach for keeping a memory from previous conversations or multi-turn tasks, but without degrading the experience or creating new bottlenecks.

Implementing a 2-tier memory architecture

To build reliable, cost-effective enterprise agents that respect your guardrails and constraints, we recommend a 2-tier memory architecture:

Short-term session buffer: Caches active conversation turns in memory using a token-bounded sliding window, allowing the context window size to remain stable across multiple rounds, even across devices, while keeping the latest messages fresh. This tier requires sub-millisecond, high-throughput lookups on every turn, making Memorystore for Valkey well suited for maintaining active session state.
Long-term persistent memory: Stores important facts, user preferences, and episodic facts across sessions. This tier requires transactional integrity, data governance, and hybrid retrieval across relational data and vectors - capabilities provided natively by AlloyDB AI.

While it’s possible to store both long-term memory and active session buffers directly in a relational database, a two-tier architecture with an in-memory cache provides better performance and scalability. Active session buffers are highly ephemeral and require sub-millisecond, high-throughput updates on every single conversation turn. Handling these rapid-fire writes in an in-memory key-value cache prevents write amplification and table bloat in your relational database, which would otherwise require frequent row deletions and intensive vacuuming. This is analogous to adding a caching tier in front of your database to offload high-frequency lookups for hot rows. The division of labor keeps your primary database lean and responsive, allowing it to focus its resources on what it is designed for: transactional consistency, complex hybrid vector search, and long-term analytical query execution.

By pairing short-term caching in Memorystore for Valkey with native AlloyDB AI capabilities, you can run entity extraction, memory compaction, cross-session memory, and hybrid retrieval directly inside the database tier, keeping active prompts lean, fast, and cost-efficient.

aside_block: <ListValue: [StructValue([('title', 'Get started with a 30-day AlloyDB free trial instance'), ('body', <wagtail.rich_text.RichText object at 0x7f1569558490>), ('btn_text', ''), ('href', ''), ('image', None)])]>

Understanding the four memory types

To organize long-term agent state effectively, we further divide agent memory into four complementary types across the short-term and long-term storage tiers we described above:

Memory type	What it stores	Storage layer	Lifespan
Buffer (Short-term)	Recent raw conversation turns	Memorystore for Valkey	Active session
Summary memory	Compressed history of older turns, commonly referred to as “compaction”	Memorystore for Valkey	Multi-turn window
Episodic memory	Past actions, events, and tool outputs	AlloyDB for PostgreSQL (Hybrid retrieval with structured SQL + full-text search + vector)	Permanent
Entity & rule memory	User preferences, constraints, and vetoes	AlloyDB for PostgreSQL (Hybrid retrieval with structured SQL + full-text search + vector)	Permanent

By isolating short-term conversation context from structured long-term rules, your agent retrieves relevant context on demand without filling token windows with raw interaction logs.

The tiered memory architectural blueprint

The diagram below outlines the read and write paths connecting the application orchestration layer, the Memorystore for Valkey short-term buffer, and the AlloyDB AI long-term repository:

The system operates across two coordinated execution paths:

The read path: When a user asks a question, the application fetches the active sliding window from Memorystore for Valkey, runs in-database query normalization using ai.generate, and queries AlloyDB using ai.hybrid_search to retrieve scoped entity rules and relevant episodic facts.
The write path: After generating the response, the turn is immediately cached in Memorystore for Valkey. An asynchronous background queue worker extracts structured entities from the exchange and writes them directly into AlloyDB, where transactional auto-embeddings immediately compute and store vector representations in the database.

Business impact and ROI

In our benchmark testing across multi-turn development dialogues (45+ turns with heavy tool executions), separating short-term caching from long-term persistence delivered measurable cost and performance improvements compared to naive context stuffing:

Metric / dimension	Naive context stuffing	Tiered memory (AlloyDB + Valkey)	Net business impact
Active prompt size (turn 45)	747,033 tokens	83,262 tokens	88.9% smaller prompt
Turn 45 response latency	33.5 seconds	6.7 seconds	80.0% faster response
Per-turn response wait time	33.5 seconds	4.2s – 6.7s	36% to 80% reduction
Cumulative session tokens	17.9M tokens	4.09M tokens	72.0% token & cost savings
Rule & constraint recall	Degrades over turns	Does not degrade	ACID-preserved recall

Note: The metrics above reflect internal benchmark results from a simulated developer workload that mimics a real-world enterprise AI pair-programming assistant interacting with a developer over multiple sessions, projects, and context switches. Actual savings and latencies vary based on prompt structure, query frequency, and data volume.

These results demonstrate how tiered memory changes agent unit economics: instead of an escalating cost curve on every additional turn, prompt sizes remain bounded, reducing ongoing LLM API expenses while keeping response times fast.

Core AlloyDB AI technical advantages

AlloyDB AI reduces the operational overhead of implementing persistent agent memory by embedding core AI functions directly into the database engine:

Transactional in-database auto-embeddings (ai.initialize_embeddings): AlloyDB automatically generates vector embeddings for text columns using a native integration with Agent Platform (formerly Vertex AI), generating up to 3,000 embeddings per second. Using incremental_refresh_mode => 'transactional', AlloyDB keeps embeddings up to date as source data changes within the same transaction, removing the need for custom embedding pipelines, external schedulers, or complex retry logic.
In-database generative AI functions (ai.generate): AlloyDB allows you to execute foundation models, such as Gemini, directly from SQL queries. You can use this for in-database query decomposition - breaking compound user questions into single-aspect sub-queries and resolving relative time phrases (like "last session") into explicit identifiers - without making separate roundtrips from your application.
Native hybrid search with built-in Reciprocal Rank Fusion (ai.hybrid_search): AlloyDB provides a built-in SQL function that executes Reciprocal Rank Fusion (RRF) directly inside the engine. It combines vector cosine similarity (<=> over HNSW or ScaNN indexes) with PostgreSQL full-text search (using BM25, RUM, or GIN) in a single database call, blending semantic matching with exact keyword retrieval while supporting metadata filter pushdown (filter_condition) for improved performance and deterministic scope isolation.
Direct Agent Platform integration with IAM credentials: AlloyDB connects directly to Agent Platform foundation models over Google Cloud's private network using Google Cloud IAM service account roles and database authentication, avoiding the need to store, rotate, or pass API keys in application code.
Unified operational, vector, and governance engine: AlloyDB consolidates relational business data, vector embeddings, full-text indexes, and enterprise permissions in a single ACID-compliant PostgreSQL database, avoiding data drift and integration complexity across separate operational and vector databases.

Key implementation patterns

Below are the core database patterns used to configure the 2-tier memory architecture. For complete, runnable Python and SQL scripts, refer to the companion AlloyDB Agent Memory Codelab.

1. Setting up schema and auto-embeddings

In AlloyDB, install the necessary extensions and define the agent_entities table with structured metadata, a generated tsvector column for full-text search, and a vector embedding column.

code_block: <ListValue: [StructValue([('code', "-- 1. Enable AI extensions\r\nCREATE EXTENSION IF NOT EXISTS google_ml_integration CASCADE;\r\nCREATE EXTENSION IF NOT EXISTS vector CASCADE;\r\nCREATE EXTENSION IF NOT EXISTS rum CASCADE;\r\n\r\n-- 2. Create structured entity & preference table\r\nCREATE TABLE IF NOT EXISTS agent_entities (\r\n entity_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),\r\n user_id TEXT NOT NULL,\r\n entity_name TEXT NOT NULL,\r\n project_id TEXT DEFAULT 'global',\r\n session_id TEXT,\r\n scope TEXT NOT NULL DEFAULT 'session',\r\n summary TEXT NOT NULL,\r\n summary_embedding VECTOR(768),\r\n summary_tsv TSVECTOR GENERATED ALWAYS AS (\r\n to_tsvector('english', entity_name || ' ' || summary)\r\n ) STORED,\r\n updated_at TIMESTAMPTZ DEFAULT NOW(),\r\n UNIQUE (user_id, entity_name)\r\n);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f1569ff99d0>)])]>

Then, generate embeddings using ai.initialize_embeddings. Using transactional mode ensures the embeddings are kept up to date as source data changes.

code_block: <ListValue: [StructValue([('code', "-- Register transactional in-database auto-embedding\r\nCALL ai.initialize_embeddings(\r\n model_id => 'text-embedding-005',\r\n table_name => 'agent_entities',\r\n content_column => 'summary',\r\n embedding_column => 'summary_embedding',\r\n incremental_refresh_mode => 'transactional',\r\n batch_size => 10\r\n);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f15692594d0>)])]>

Finally, create the HNSW vector index and the RUM full-text search index to ensure your hybrid searches are fast and efficient.

code_block: <ListValue: [StructValue([('code', '-- Create vector and full-text indexes\r\nCREATE INDEX IF NOT EXISTS agent_entities_embedding_idx\r\n ON agent_entities USING hnsw (summary_embedding vector_cosine_ops);\r\n\r\nCREATE INDEX IF NOT EXISTS agent_entities_tsv_idx\r\n ON agent_entities USING rum (summary_tsv rum_tsvector_ops);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f156925b690>)])]>

2. Querying long-term memory with native hybrid search

On the read path, retrieve relevant long-term entities using ai.hybrid_search. This native SQL function executes Reciprocal Rank Fusion (RRF) directly in AlloyDB, seamlessly reranking and combining vector similarity search and full-text keyword search results in a single database query:

code_block: <ListValue: [StructValue([('code', 'def retrieve_hybrid_entities(\r\n conn,\r\n user_id: str,\r\n project_id: str,\r\n query_text: str,\r\n query_vector_literal: str,\r\n limit: int = 3\r\n) -> dict[str, Any]:\r\n """Queries long-term memory using AlloyDB native hybrid_search (RRF)."""\r\n filter_cond = f"user_id = \'{user_id}\' AND (project_id = \'{project_id}\' OR scope = \'global\')"\r\n \r\n search_inputs = [\r\n json.dumps({\r\n "data_type": "vector",\r\n "weight": 0.4,\r\n "table_name": "agent_entities",\r\n "key_column": "entity_name",\r\n "vec_column": "summary_embedding",\r\n "distance_operator": "<=>",\r\n "limit": 10,\r\n "query_vector": query_vector_literal,\r\n "filter_condition": filter_cond\r\n }),\r\n json.dumps({\r\n "data_type": "text",\r\n "weight": 0.6,\r\n "table_name": "agent_entities",\r\n "key_column": "entity_name",\r\n "text_column": "summary_tsv",\r\n "limit": 10,\r\n "ranking_function": "ts_rank",\r\n "query_text_input": query_text,\r\n "filter_condition": filter_cond\r\n })\r\n ]\r\n\r\n query_sql = """\r\n SELECT e.entity_name, e.summary, e.project_id, e.scope, e.updated_at\r\n FROM ai.hybrid_search(\r\n search_inputs => %s::JSONB[],\r\n include_json_output => false\r\n ) h\r\n JOIN agent_entities e ON e.entity_name = h.id\r\n WHERE e.user_id = %s\r\n LIMIT %s;\r\n """'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f15692596d0>)])]>

3. In-database memory compaction with AI functions

To manage long-term storage growth without writing custom pruning scripts, you can run automated extraction and compaction queries directly in AlloyDB to identify the important parts which can benefit from being stored. This pattern uses a SQL common table expression (CTE) with ai.generate to consolidate older episodic entries into a high-density summary:

code_block: <ListValue: [StructValue([('code', "-- Consolidate older episodic logs into a permanent summary\r\nWITH old_events AS (\r\n SELECT document AS content\r\n FROM langchain_pg_embedding\r\n WHERE cmetadata->>'user_id' = 'user_dev_42'\r\n AND created_at < NOW() - INTERVAL '30 days'\r\n ORDER BY created_at ASC\r\n LIMIT 50\r\n),\r\nconsolidated AS (\r\n SELECT ai.generate(\r\n 'Summarize these historical events into a dense memory paragraph:\\n' || string_agg(content, E'\\n'),\r\n model_id => 'gemini-3.5-flash'\r\n ) AS summary_text\r\n FROM old_events\r\n)\r\nINSERT INTO agent_entities (user_id, entity_name, summary, updated_at)\r\nSELECT 'user_dev_42', 'longterm_session_summary', summary_text, NOW()\r\nFROM consolidated\r\nWHERE summary_text IS NOT NULL\r\nON CONFLICT (user_id, entity_name)\r\nDO UPDATE SET summary = EXCLUDED.summary, updated_at = NOW();"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f1569259350>)])]>

For example, a long interaction regarding the complexities of changing flights with kids can result in a summary of “I prefer non-stop flights”.

4. Connecting the 2-tier memory architecture to an ADK agent

With the core 2-tier memory architecture configured, you can now extend the default ADK Memory provider to use this 2-tier memory architecture as shown in the accompanying Codelab (e.g. ADKTieredMemoryProvider). To attach the long-term memory to an ADK agent, you simply provide it as a tool (e.g. longterm_memory_tool) like this:

code_block: <ListValue: [StructValue([('code', 'from adk_memory_provider import ADKTieredMemoryProvider\r\n\r\nagent = Agent(\r\n name="adk_memory_agent", model=GEMINI_MODEL, instruction="Initial instruction",\r\n tools=[long_term_memory_tool, run_command_tool],\r\n before_tool_callback=guardrail.create_before_tool_callback(USER_ID, "CloudRetail")\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f156925bad0>)])]>

Enterprise governance and multi-tenant security

Running agent memory in enterprise production environments requires strict security boundaries and access controls. First of all, we need to maintain scope and multi-tenant isolation. By indexing user_id, project_id, and scope columns in agent_entities and enforcing PostgreSQL Row-Level Security (RLS), you can isolate memory stores across departments, teams, and individual users within the same database cluster. Parameterized Secure Views (PSV) offer another layer of deterministic application-level security, helping you protect against malicious prompts and overly-broad SQL queries.

In addition, we need to provide automated memory lifecycle management as shown in the implementation pattern above. Combining scheduled SQL compaction queries with time-based partition pruning helps maintain predictable database footprint and query latencies over time. See the accompanying Codelab for more details on this approach.

Summary and next steps

Decoupling active context windows from persistent storage is a practical approach to building production-ready AI agents. By pairing Memorystore for Valkey for sub-millisecond session caching with AlloyDB AI for transactional long-term storage, you can achieve substantial token cost savings and faster response times while maintaining strict business rules throughout long-horizon tasks and many-turn agentic experiences.

To get started:

Step through the complete hands-on tutorial in the companion AlloyDB Agent Memory Codelab to deploy the working 2-tier memory architecture.
Learn more about database-side machine learning features in the AlloyDB AI documentation.
Explore guides on generating auto vector embeddings and running hybrid vector search.

Spanner Omni, now GA: A distributed, multi-model database that you can deploy anywhere

Wed, 30 Sep 2026 16:00:00 +0000

Spanner Omni, the deploy-anywhere version of Spanner, is now generally available, ready to power your most demanding production workloads in your on-premises data center or on other clouds.

With Spanner, Google pioneered the distributed SQL market over a decade ago, combining the horizontal scalability of NoSQL with the ACID compliance and strong consistency of a traditional relational database. Since then, Spanner has evolved into an interoperable multi-model database that simplifies complex workloads and powers agentic AI, combining SQL, graph, key-value, full-text search, vector search, and analytical processing with a columnar engine, all in a single database.

When we debuted Spanner Omni at Google Cloud Next ’26, we untethered our distributed database from Google Cloud and brought it directly to your infrastructure. This generated incredible interest from both the enterprise customers and developer community. In fact, since its launch, Spanner Omni has over 2 million downloads.

Spanner Omni delivers the same core capabilities as the fully managed Spanner service, with the added freedom to deploy it wherever you need it. Whether you’re running virtual machines or Kubernetes in your private data centers, running a multi-cloud deployment that spans multiple clouds, or testing locally on a laptop, Spanner Omni brings Google-grade consistency, availability, scale, and interoperable multi-model capabilities directly to your next agentic AI applications.

Spanner Omni provides the freedom to deploy anywhere with the same core Spanner capabilities

Attio accelerates agentic application velocity with Spanner Omni

Spanner Omni enables teams to build once and deploy anywhere, providing application portability across Google Cloud, on-premises and other clouds. Attio, an AI-native CRM company based in London, has built its platform on Spanner. Attio migrated its agentic application environment to a full-featured and containerized Spanner Omni + managed Spanner.

"At Attio, we are building the world's most advanced agentic application environment on Spanner to deliver on our vision for an AI-native CRM platform. Spanner Omni has been a major win for us by delivering Spanner capabilities and performance across all our environments, allowing our agents to bring complex, mission-critical workflows such as the newly announced Spanner queues, to production. With Spanner Omni, we have been able to accelerate our production velocity at a truly enhanced level of scale and confidence that was not possible earlier." - Alexander Christie, Co-Founder & CTO, Attio

Built-in AI capabilities for any environment

Spanner Omni extends Spanner's converged multi-model foundation directly to your private infrastructure and third-party clouds, providing critical capabilities for AI workloads:

Vector search and Spanner Graph: Natively store and index vector embeddings alongside your relational tables. With support for KNN and ANN search, you can combine semantic similarity with structured SQL filters. Spanner Graph integrates property graphs directly into the engine, allowing you to trace complex entity relationships and connect graph traversals with vector search.
Model Context Protocol (MCP) support: Integrate directly with agentic systems using MCP Toolbox. This open standard allows autonomous agents to inspect schemas, retrieve relevant context, and use Spanner Omni as an operational memory layer across multi-cloud deployments.

What’s new with Spanner Omni

Since the preview, we’ve been working to expand Spanner Omni’s capabilities, and refine the pricing model for production-ready deployments.

Enterprise features for production deployments

The GA release unblocks the full suite of enterprise-grade capabilities required for mission-critical production workloads, including:

Robust security and governance: Support for advanced enterprise security, including TLS encryption, authentication and authorization, and audit logging
Data protection: High-performance backup and restore capabilities to safeguard your data against “fat-finger” deletion or data corruption
Worker nodes: Dedicated, stateless compute nodes unique to Spanner Omni that are designed to offload background and resource-intensive operations from primary Spanner Omni servers, so they can focus on handling core database workloads
Enterprise-grade support: Direct access to Google Cloud Customer Care to keep your critical workloads running smoothly

Flexible licensing and industry-standard pricing

To support you at every stage of development, Spanner Omni offers two distinct licensing tiers designed to fit your scale and budget:

Developer Edition (free): Tailored for development, testing, and prototyping in non-commercial, non-production environments and personal use, it includes all core Spanner features, allowing you to build and validate your applications before scaling to production. The Developer Edition includes a 90-day license and includes all features as the commercial edition, except backup features and worker nodes. When used in a single server deployment of 4 vCPUs or less, the Developer Edition license does not expire and all single server features including backup-restore aresupported. If you need to use the Developer Edition for more than 4vCPUs or beyond the 90 day limit, you can request a perpetual license by filling out this form.
Commercial Edition (paid): Designed for commercial, production workloads, this edition provides the full suite of Spanner Omni capabilities backed by enterprise support. It follows a highly industry-standard, predictable vCPU-based annual subscription model. We also offer a proof-of-concept license for pre-production evaluation at a discounted price.

Spanner Omni vs. fully managed Spanner on Google Cloud

Our goal with Spanner Omni is to provide parity with the fully managed Spanner service on Google Cloud. However, as self-managed software, deploying and operating Spanner Omni differs from fully managed Spanner on Google Cloud in several ways:

Self-managed operations: You are responsible for all day-to-day operations, including routine maintenance, version upgrades, and infrastructure monitoring.
Availability and SLAs: Because Spanner Omni runs on customer-managed infrastructure, Google does not provide availability SLAs. However, deploying according to our recommended reference architectures will help you achieve comparable high availability.
Integration with Google Cloud: To ensure it can run anywhere, Spanner Omni excludes capabilities available in managed Spanner that rely on Google Cloud-specific capabilities, such as native integrations with BigQuery, Knowledge Catalog, or Gemini Enterprise and other Google Cloud services.
Feature-parity roadmap: While some feature gaps exist today between Spanner Omni and fully managed Spanner, we are actively developing updates to close these gaps in future releases.

Get started today

Whether you’re modernizing on-prem legacy systems or building a resilient multi-cloud architecture, Spanner Omni is ready to help you scale.

For more information, visit the Spanner Omni website to explore documentation and use cases.
To use the Spanner Omni commercial edition with full features, please contact your Google Cloud account team or reach out to us at https://cloud.google.com/consulting/spanner-omni.
For developing and testing for non-commercial, non-production purposes, download the Spanner Omni developer edition.

Unlock 3x QPS and microsecond latency with Memorystore for Valkey 9.1

Fri, 25 Sep 2026 16:00:00 +0000

At Google Cloud, we are committed to delivering the best managed experience backed by open source software. Today, we’re announcing the general availability of Memorystore for Valkey 9.1, which achieves up to 3x queries per second (QPS) at microsecond latency compared to Memorystore for Redis Cluster.

Our support for Valkey dates back to 2024, when Redis Inc. shifted its licensing away from the permissive open-source BSD license to a dual-license model. In response, Google Cloud, alongside other technology leaders, backed the creation of Valkey, an open-source alternative governed by the Linux Foundation.

Valkey has come a remarkably long way since then, delivering major performance and feature updates that push boundaries far beyond the original fork. Valkey is particularly compelling for organizations scaling AI and microservices to handle millions of concurrent users. Here, backend developers and architects must deliver both massive throughput while also maintaining microsecond latency.

In this blog, let’s take a look at how Valkey 9.1 achieves its performance, new developer capabilities, how to get started, and how customers are using it.

Under the hood: Rethinking thread communication

In high-throughput, in-memory datastores, efficient I/O offloading is critical to keeping the main execution loop unblocked. Previously, Valkey assigned client sockets to I/O threads statically in a round-robin fashion, requiring the main thread to continuously poll lists of pending clients to detect completed work.

Valkey 9.1 replaces list-polling with a lock-free, multi-queue messaging architecture that eliminates cross-thread CPU waste and unlocks dynamic work balancing. It involves three complimentary queues:

Main thread to I/O thread queue: Dispatches read and write jobs to a single-producer multi-consumer (SPMC) queue. Free worker threads pull tasks on demand, enabling dynamic work-stealing that prevents thread starvation or hot-spotting.
I/O thread to main thread queue: Worker threads push completed tasks into a multi-producer single-consumer (MPSC) queue. The main thread pops completed work instantly, eliminating busy-wait list iteration.
I/O thread-specific queues: Dedicated single-producer single-consumer (SPSC) queues handle thread-affine memory cleanup and high-volume epoll offloading.

Valkey 9.1 also replaces static thread thresholds with a two-phase dynamic scaling engine:

CPU-driven "ignition": When main-thread CPU usage crosses 30%, the engine automatically activates the first background I/O thread to absorb incoming traffic before queue bottlenecks form.
Queue-depth auto-scaling: Once ignited, Valkey dynamically scales the number of active I/O worker threads up or down based on real-time SPMC queue backlog, ensuring extra cores are used only when needed and parked when idle.

New developer capabilities in Valkey 9.1

Beyond raw performance, Valkey 9.1 addresses key feature requests from engineering teams with powerful new commands and enhanced security controls. Here is a look at what you can do with these new capabilities:

1. Granular database-level access control (ACLs)

We recently launched support for access control lists on Memorystore for Valkey to provide more granular key-level and command-level authorization using IAM. This foundational security mechanism is offered at no additional cost and includes the following capabilities:

Centralized management: A 1:N mapping approach allows you to define a single ACL policy and attach it across multiple clusters.
Secure multi-tenancy: Organizations can easily enforce least privilege and secure multi-tenancy across their database fleets.
Enhanced observability: The feature includes versioned policy revisions and comprehensive audit logging.

Previously, ACL rules applied globally across an instance. Valkey 9.1 allows administrators to restrict user access at the specific numeric database level within the ACL framework.

Real-world example: You can configure a staging or service-specific user and isolate their access strictly to non-production databases:

production user: @all ~* db=0
staging user: @all ~* db=1
dev user: @all ~* db=2

Protect against unauthorized data access and guard against application bugs by leveraging database-level access control across multiple databases, all without needing to prefix your keys.

2. CLUSTERSCAN: Efficient cluster-wide key scanning

Previously, scanning keys across a large cluster required querying nodes individually. This approach was not cluster- or failover-aware. Consequently, scans could miss keys, return duplicates, or fail if slot migrations or node failovers occurred during the process.

The CLUSTERSCAN command addresses these limitations by introducing a topology-aware cursor. This cursor encodes the current slot, the fingerprint of the local hashtable, and the local cursor. With this additional encoded information, clients can scan keys across the entire cluster while gracefully handling topology changes and redirections.

CLUSTERSCAN supports two primary scanning strategies:

Use case 1: Sequential full cluster scan (single worker)

This strategy is suitable for simple scripts or background jobs that prioritize simplicity over speed. The client starts with cursor 0 and sequentially traverses all slots in the cluster:

code_block: <ListValue: [StructValue([('code', 'CLUSTERSCAN 0 MATCH "user:*" COUNT 10\r\n1) "0B3a21-{06S}-64"\r\n2) 1) "user:101"\r\n2) 2) ...'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f156bccdc50>)])]>

To continue the scan, pass the returned cursor to the next call. The cursor automatically transitions to the next slot when the current one is fully scanned.

code_block: <ListValue: [StructValue([('code', 'CLUSTERSCAN 0B3a21-{06S}-64 MATCH "user:*" COUNT 10\r\n1) "0B3a21-{07T}-0"\r\n2) 1) "user:102"\r\n2) 2) ...'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f156b7d34d0>)])]>

The scan is complete when the command returns a cursor of "0".

Use case 2: Parallelized cluster scan (multiple workers)

This strategy is suitable for high-throughput scans. Using the SLOT argument restricts the scan to a specific slot, allowing you to partition the 16,384 slots across multiple parallel workers.

Worker 1 (Scanning Slot 0):

code_block: <ListValue: [StructValue([('code', 'CLUSTERSCAN 0 SLOT 0 MATCH "user:*" COUNT 10\r\n1) "0B3a21-{06S}-64"\r\n2) 1) "user:101"\r\n2) 2) ...'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f156945fbd0>)])]>

Worker 2 (Scanning slot 1000 in parallel):

code_block: <ListValue: [StructValue([('code', 'CLUSTERSCAN 0 SLOT 1000 MATCH "user:*" COUNT 10\r\n1) "0B3a21-{08X}-32\r\n2) 1) "user:999"\r\n2) 2) ...'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f1588c8a190>)])]>

From here, Worker 1 continues to pass SLOT 0 and Worker 2 continues to pass SLOT 1000. Mismatching the slot and the cursor returns an error. Once all 16384 slots have been scanned, the cluster scan is considered complete.

3. More commands for atomicity and expirations

HGETDEL: Atomic fetch and delete
A frequent application pattern involves reading a hash field and deleting it immediately (such as consuming single-use authentication tokens or short-lived session states). Valkey 9.1 introduces HGETDEL, which retrieves the value of a hash field and deletes it atomically in a single network round-trip.

Real-world example:
HSET user:1001 temp_token "abcde"
(integer) 1
HGETDEL user:1001 FIELDS 1 temp_token
1. "abcde"
HGET user:1001 temp_token
(nil)

MSETEX: Shared expiration for multiple keys
To eliminate multi-command pipeline overhead, the new MSETEX command enables setting multiple keys simultaneously with a single, shared expiration time.

Real-world example: Setting up a temporary session state where multiple distinct keys must expire together in 300 seconds:
MSETEX 2 session:auth "ok" session:user_id "1001" EX 300
(integer) 1
TTL session:auth
(integer) 300

Enhanced HSETEX with conditional flags
HSETEX now supports the NX (only set if the field does not exist) and XX (only set if the field exists) conditional flags.

Real-world example: Initializing a rate-limit threshold field with a 1-hour TTL, ensuring you don't overwrite an existing active limit:
HSETEX config:123 NX EX 3600 FIELDS 1 "rate_limit" "100"
(integer) 1

Built on Memorystore for Valkey 9.0

The release of Valkey 9.1 builds upon the major updates we unveiled for Memorystore for Valkey at Google Cloud Next '26:

Built-in modules for AI & vector workloads: Native JSON support and Bloom filters enable fast document querying and membership checks.
Six new node sizes: To help you manage costs and scale, we added six new node sizes.

Small Size Nodes: Custom-Pico (1.25 GB), Custom-Micro (2.5 GB), and Custom-Mini (3.5 GB) for lightweight microservices and dev/test environments. These are only available for cluster mode disabled environments.
High CPU and Large SKUs: HighCPU-Medium (8 vCPU/13 GB) and Standard-Large (8 vCPU/26 GB) optimized for CPU-heavy applications.
XXL SKU: Highmem-XXLarge with 110 GB RAM and 16 vCPUs per node for massive cluster consolidation to power your most demanding workloads.

(Note: The figures above are based on open-source benchmarks; actual performance improvements will vary depending on your specific workloads.)

Migrating to fully managed Memorystore for Valkey

Having to self-manage your Redis OSS /Valkey caching layers drains valuable engineering bandwidth and creates operational friction during scaling. We are also excited to announce a new migration workflow to Memorystore for Valkey.

With this release, migrating your infrastructure is straightforward, fully managed, and requires a simple configuration change on your application to point to Memorystore for Valkey once your data is migrated. This workflow is generally available.To move off self-managed Redis or Valkey to fully managed Memorystore for Valkey, follow these four steps:

1. Provision the target instance: Deploy a Memorystore for Valkey instance configured with your required shard count, node sizing, and clustered database options.

2. Establish online replication: Initiate continuous, dual-sync online migration directly from your source database to Memorystore.

3. Validate data synchronization: Monitor replication metrics in real time to verify full dataset alignment and low-latency replication health.

4. Execute the cutover: Switch application connection endpoints over to Memorystore for Valkey to start using the new cache.

What Memorystore for Valkey customers are saying

Already, over 95% of the top 100 Google Cloud customers already rely on Google Cloud Memorystore to power demanding, high-throughput workloads, led by increasing numbers of Memorystore for Valkey users.

Consider the fast-paced world of live sports, where delivering a flawless digital experience is of utmost importance. When a game-changing play happens, millions of fans immediately reach for their devices to check real-time stats, watch highlights, and engage with interactive features. These massive, unpredictable traffic spikes require an underlying architecture capable of immense scale. For organizations like Major League Baseball (MLB) , a partner since Valkey’s early days, managing unpredictable traffic spikes without compromising performance is essential.

"We trust Memorystore for Valkey to power the massive scale of live baseball, delivering real-time stats and uninterrupted digital experiences to millions of fans. As we look ahead, we are incredibly excited about the Memorystore for Valkey 9.1 launch. The engine optimizations and latency enhancements will give us even more horsepower to handle the most unpredictable game-day traffic spikes, ensuring fans get the best technology-powered experience the game has to offer." - Rob Engel, SVP of Software Engineering, Major League Baseball

Beyond the stadium, the retail industry faces its own intense scaling challenges, particularly during major shopping holidays or flash sales. Modern e-commerce platforms rely on real-time personalization, dynamic pricing, and instant inventory updates to keep shoppers engaged. A lag of even a few milliseconds can disrupt the customer journey and impact the bottom line. To maintain a competitive edge, leading retailers such as Target require ultra-responsive caching layers to power their most crucial customer-facing platforms.

"By leveraging Google Cloud Memorystore for Valkey, Target delivers ultra-low-latency, resilient caching for personalization services. We look forward to leveraging the performance enhancements in Valkey 9.1 to make our personalization platform even faster, more scalable, and more resilient during periods of peak demand." - Scott Weide and Sumanth Huddar, Senior Engineering Managers, Target

The demand for these ultra-low-latency architectures extends far beyond sports and retail. Across the digital landscape, organizations in banking, AI-native development, digital streaming, and telecommunications all share a common mandate: the need for superfast, highly available caches. Whether it is processing high-frequency financial transactions, serving complex machine learning inferences in real time, delivering seamless global video streams, or routing immense volumes of telecom data, microsecond latency is the new baseline for success.

Make the move to Valkey

Stop letting cache bottlenecks slow down your most demanding applications. Experience the performance, dynamic scalability, and enhanced security of Memorystore for Valkey 9.1 today.

Start building: Create a Memorystore for Valkey 9.1 instance in the Google Cloud console.
Dive deeper: Read the technical documentation to view the full list of supported commands, ACL configurations, and detailed capabilities of Memorystore for Valkey.

AlloyDB delivers PostgreSQL for agents: Real-time data at agent scale, with full workload isolation

Thu, 24 Sep 2026 14:30:00 +0000

Enterprises rely on mission-critical operational databases where performance slowdowns simply aren’t an option. Yet when even a few agents execute dense reasoning loops, the unpredictable surge in queries can easily overwhelm traditional architectures.

Today, we’re announcing that AlloyDB delivers PostgreSQL for agents (in preview), enabling real-time data access without compromising your mission-critical systems. AlloyDB now scales to dynamic agent bursts by provisioning sandboxed database instances in seconds, enabling full workload isolation. You can cost-effectively run your agents at any scale — from a few agents to millions of agents — and the instances automatically spin down when agents finish. With this announcement:

AlloyDB now features an agentic database architecture engineered to scale PostgreSQL to thousands of serverless database instances that have up-to-the-second read-only access to production. These instances remain fully separated from the primary, standby, and read replica instances where production workloads run.
Each of these instances accesses real-time data in the database backed by a unified storage layer in Colossus, Google’s exabyte-scale distributed storage system. This helps agents achieve sub-millisecond I/O, and terabit-per-second aggregated scan throughput, supporting over 3 million queries per second.
These instances utilize the full AlloyDB PostgreSQL engine, providing access to every index, the full capability of SQL, and comprehensive vector, full-text, and spatial search. Agents can also leverage BigQuery and Spark to run lakehouse analytics without requiring complex ETL pipelines.
When agents complete their tasks, these instances scale right back to zero, thus reducing your cloud spend by billing only for active reasoning loops.

With AlloyDB for PostgreSQL, we pioneered the agentic enterprise relational database by integrating advanced vector operations, machine learning inference, and foundation model integrations directly within a 100% PostgreSQL-compatible engine. It protects your data through deep Google Cloud security integrations — replacing static passwords with IAM authentication, isolating traffic via VPC Service Controls, and providing customer-managed encryption and auditing. This functionality, combined with the scalability now provided by our agentic architecture, makes AlloyDB the premier enterprise-grade agentic PostgreSQL offering.

To get started, sign up here for the preview.

Why this matters

In today’s agentic era, we’re swiftly moving from single copilot agent interactions to networks of millions of agents collaborating simultaneously. When these agents query and operate all at once, sudden traffic spikes can overwhelm your core databases, competing with the systems that run your business.

Agents require both fast analytics and low-latency access to real-time production data, utilizing B-tree, vector, text, and spatial indexes to efficiently execute their workflows. Emerging architectures rely on page-caching layers that sit above object storage, but they suffer from scaling and cost challenges that can compromise the stability of production systems.

In addition, they face a challenging trade-off: To unlock production data for analytics, they create performance bottlenecks for operational access, putting mission-critical databases at risk the moment agents are unleashed in production. These approaches attempt to solve the problem using traditional object stores for database storage. While this enables analytical access that can help some agents, the underlying databases are too slow for production workloads, suffering from up to an order of magnitude higher I/O latency. Page caching layers are at best a patch; the caches themselves are often still not fast enough, and they pose a scalability bottleneck that is easily saturated by agentic workloads.

When active multi-agent systems execute dense reasoning cycles, they trigger highly concurrent, unpredictable bursts of queries that overwhelm these caching layers, leaving mission-critical production systems vulnerable to agent-induced outages. Consequently, an entire class of operational use cases is precluded from running on these architectures, locking businesses out of the transformative power of AI on live enterprise data.

AlloyDB’s unique agentic PostgreSQL architecture

We are taking a different approach. AlloyDB delivers an agentic database architecture purpose-built for the AI era, with four key differentiated capabilities:

Sub-millisecond I/O latency, without artificial choke points: Combining AlloyDB’s industry-leading transaction and query processing with low-latency object storage backed by Google’s planet-scale Colossus storage infrastructure, this architecture provides a large-scale, shared storage plane for agents. It achieves sub-millisecond I/O and over a terabit-per-second of aggregate scan bandwidth, allowing agents to execute intensive read queries and vector searches directly against fresh operational data.
Fully isolated from production workloads while scaling to meet demand: AlloyDB scales by dynamically provisioning sandboxed database instances in seconds against fresh production data. Unlike traditional architectures where agents compete for operational resources, agentic database compute remains completely isolated from the primary database clusters — allowing agents to execute dense, unpredictable reasoning loops without degrading performance in production. These robust safety guardrails, coupled with enterprise-grade governance and fine-grained access control, allow you to confidently unleash the full, unconstrained power of PostgreSQL on your production data — seamlessly mixing analytical queries, vector queries, and operational point lookups in active agentic loops.
Pay-as-you-go billing: Most agent activity is characterized by sharp spikes of concurrent queries followed by periods of inactivity. Provisioning dedicated read replicas to absorb these bursts forces you to maintain expensive infrastructure around the clock. To support massive groups of agents cost-effectively, and eliminate the idle compute overhead of provisioned systems, these agentic AlloyDB instances can rapidly scale to handle millions of queries per second, and automatically scale to zero with a flexible, pay-as-you-go pricing model.
Native lakehouse integration, without ETL: All production data is natively integrated with Google Cloud’s borderless Lakehouse. This allows agents to run federated queries across BigQuery and Lightning Engine for Apache Spark, joining massive lakehouse datasets with up-to-the-second transactional data in AlloyDB. This eliminates the need to build and maintain fragile batch ETL pipelines, giving autonomous agents instant access to both live operational state and historical lakehouse context.

“As supply chains become increasingly autonomous, our platform relies on real-time transactional intelligence to coordinate complex logistics workflows across thousands of facilities. AlloyDB's new PostgreSQL architecture for agents has been a game changer for us. We can now deploy networks of agents collaborating simultaneously to help us analyze inventory and order data with sub-second freshness, while ensuring our core transactional processing remains entirely untouched. It delivers the isolation, speed, and cost efficiency we need to power the next generation of enterprise supply chain AI.” - Sanjeev Siotia, Executive Vice President & Chief Technology Officer, Manhattan Associates

Availability

PostgreSQL for agents in AlloyDB is now available in preview.

To learn more, visit the documentation page, and sign up here to get started.

A new, no-compromises database architecture for the agentic era

Thu, 24 Sep 2026 14:30:00 +0000

Entire database engineering careers have been spent on a single question: How do you scale an OLTP workload without compromising the system of record that owns the data?

Exadata answered the question by offloading queries into a scale-out storage tier beneath the database, removing the network as the bottleneck. Azure SQL Hyperscale did it with shared block servers, scaling out to tens of read replicas. Aurora offloaded log application to distributed storage nodes, scaling reads across tens of PostgreSQL nodes. Meanwhile, emerging architectures persist data in traditional object storage with a provisioned cache tier in front, recovering latency for hot data but leaving a high-latency tail on every cache miss.

Each of these architectures is inherently constrained by at least one of these three properties: scale, latency, and isolation — and sometimes even two. For instance, architectures built on shared block servers compromise scalability, because I/O inevitably bottlenecks on the block server. They also sacrifice isolation, as production workloads get throttled whenever replica traffic spikes.

Some of these trade-offs were actually sound when they were developed; they met the requirements of enterprise database workloads for four decades. However, in the agentic era, these compromises are no longer acceptable. Agentic workloads are generated dynamically and cannot be vetted in advance, making it a business-continuity imperative to isolate them from mission-critical systems. Agentic workloads also require low latency that is only possible with the full power of the database engine and all its indexes, as well as a whole new level of elastic scale that has never been tried with a single database: a burst of agents that demand 1,000 compute nodes over a single database within seconds, and that may finish inside a minute.

Three tenets needed for a truly agentic database architecture

We believe the agentic era demands a new agentic database architecture defined by three fundamental tenets. An agentic database architecture must satisfy all three, or it isn’t really agentic.

Tenet: Isolation — isolation by design, but with real-time data access. Agents must read live production data with sub-second freshness over a data path that does not share database components with the primary cluster. Real-time means up-to-the-second, not a stale copy or branch. This is physical separation, not a quota — because shared allocations mean shared fate. The boundary extends straight through the storage layer, eliminating resource contention by design.
Tenet: Latency — sub-millisecond baseline I/O. Operational workloads demand sub-millisecond block I/O, and that bar does not drop for agents. While compute nodes leverage DRAM and local SSD for acceleration, cache misses that reach remote storage — whether application or agentic — must complete in under a millisecond. An architecture that degrades into an order-of-magnitude performance cliff is fundamentally unusable by agents.
Tenet: Scale — agent-scale compute and I/O. Agent scale is simultaneously instantaneous, volatile, and massive: Database compute nodes must spin up in seconds, scale to thousands, run for short bursts, and automatically spin down to zero when agents are done with them. No one has thus far ever dreamed of expecting a database to scale compute and I/O dynamically to thousands of nodes while leaving production untouched. Due to the dynamic nature of agents, pre-provisioning is a non-starter across the entire stack, whether it’s compute, storage I/O, or any caching tier in between.

Crucially, an agentic architecture must uphold all three tenets at once. And by doing so, the architecture allows agents to work directly against live operational data, i.e., enterprise truth, without compromising production stability. The outcome is transformative:

No correlated failures: Total decoupling between the engines running the business and the fleets of agents reasoning over it removes a path for agents to affect production.
No capacity guesswork: True elasticity that eliminates the friction of pre-provisioning for unforecastable agent scale.
No semantic compromises: Nothing is withheld from agents — they get access to the full power of relational SQL, hybrid search (vector, full-text, spatial), and indexes within every single reasoning step.

AlloyDB's agentic architecture

AlloyDB’s new agentic database architecture is the first system that satisfies all three tenets. We engineered this from the ground up across storage, network, compute and databases to deliver:

Isolation, avoiding shared fate by design: The transactional production cluster runs on dedicated, pre-provisioned infrastructure, completely isolated from agent workloads. Agents interface via the Model Context Protocol (MCP) to an independent, ephemeral pool of microVM-based AlloyDB nodes that read directly from dedicated Colossus storage segments, separate from those for production.
Predictable sub-millisecond storage I/O: Every storage read is served directly by Google’s Colossus storage system inheriting its baseline sub-millisecond latency, eliminating performance cliffs on cold cache misses.
True zero-to-thousands compute scaling: The agent pool scales rapidly from zero to thousands of nodes for bursty agentic activity, and scales back to zero the moment tasks complete.

Agents query production data with sub-second freshness, with the complete PostgreSQL engine — point lookups, index traversals, vector, full-text and spatial search, columnar scans, and federated queries across the lakehouse — at their disposal to power their reasoning loops.

Run agents against production data at any scale by joining the preview of AlloyDB PostgreSQL for agents. You can learn more about its full capabilities in the companion announcement blog.

Why existing architectures can’t satisfy all three tenets

Traditional and emerging operational databases attempt to scale using one of three architectural paradigms. When assessed against the demands of autonomous AI agents, each paradigm exhibits a fundamental structural compromise — none satisfies all three tenets simultaneously.

Independent replicas (shared-nothing storage)

Traditional relational architectures scale reads by streaming replication logs from a primary instance to dedicated replica databases, each with its own local or attached block storage. They meet Tenet: Isolation – replicas share no physical resources with the primary cluster, and continuous log replication maintains near-real-time currency. They meet Tenet: Latency – dedicated local storage guarantees predictable, sub-millisecond read latency. However, they fail Tenet: Scale – scaling requires provisioning a new replica and rehydrating hundreds of gigabytes or terabytes of storage. All this takes hours — an impossible mismatch for agent-reasoning bursts measured in seconds. Furthermore, statically provisioned compute and storage continue to incur idle costs long after the agent completes its run.

Disaggregated shared-storage servers

A second approach decouples stateless compute nodes from a shared, multi-tenant tier of custom storage servers that manage persistence, replication, and that may offload block writes. This approach meets the Tenet: Latency – reads hitting the optimized storage servers resolve with consistent, low operational latency. However, it fails the Tenet: Isolation – because every replica reads from the same servers as the primary, so agent I/O contends directly with production I/O, creating shared fate. It also fails the Tenet: Scale – stateless compute replicas spin up quickly because no data is copied, but total storage I/O bandwidth is fixed to the pre-provisioned storage tier. Adding compute nodes without scaling underlying I/O capacity simply accelerates storage saturation and throttling.

Object storage with shared-block servers

A third emerging approach keeps data durable in general-purpose object storage and serves block reads from a shared tier of block servers. Because a random read from object storage takes tens of milliseconds — an order of magnitude slower than traditional database storage, and slower than an enterprise disk array has been for at least 25 years — the block servers hold hot data in order to serve it at low latency. This approach meets the Tenet: Latency — with one caveat: A block server miss still falls through to object storage at unacceptably high latency. It fails the Tenet: Isolation — because replicas share the block servers with production: Agent I/O and production I/O draw on the same capacity, so when that capacity is exhausted or throttled, production is affected along with the agents. It also fails the Tenet: Scale, for the same reason as shared storage servers: Replicas start quickly, but the block servers do not scale their I/O with the burst.

Some architectures in this family also allow analytical engines like Apache Spark to read the underlying object storage directly, bypassing the database engine. For analytics workloads, that is a valuable and viable path. However, since agents need low-latency retrieval, stripping away indexes, point lookups, and vector search forces brute-force table scans, exploding latency, and therefore breaks the ability for agents to execute their retrieval-reasoning loops.

Evaluating existing architectures

We evaluated a commercially available service that uses the object storage architecture with shared block servers by running concurrent index lookups over a dataset larger than available DRAM, testing both scaling limits and production isolation. Starting with a single reader instance, we scaled the workload by adding up to eight read replicas.

In architectures with shared physical resources, scaling agents via read replicas quickly degrades both replica and primary performance. In our tests as seen in the chart below, adding replicas provided less than a 2x throughput increase, peaking at four replicas before dropping off as the shared block-server bandwidth saturated.

The impact on the primary database was immediate and severe: Primary throughput plummeted by more than 75% as replicas were added.

In short, neither traditional nor emerging architectures can meet the scale that agents demand, and certainly not without jeopardizing the stability of production systems.

Assessing against the Tenets

* Partially meets: Hot data is served at low latency from the block servers, but a block server miss falls through to object storage at tens of milliseconds.

In each case the gap is structural, not just a matter of tuning. Replication isolates by giving each replica its own storage, so it cannot add a replica faster than it can populate that storage. Shared storage servers add compute quickly by sharing storage, so they can neither isolate nor scale I/O. Block servers over object storage recover latency with a provisioned tier, so they can neither isolate nor burst, and every miss still reaches object storage. Each approach solves the problem at one layer and pays for it at another. Meeting all three tenets at once requires rethinking the database architecture across compute, network and storage together.

How we engineered AlloyDB across the stack

AlloyDB's agentic database architecture is vertically integrated across Google's data, AI and infrastructure stack: AI models, the database engine and analytical engines, but also storage, networking and compute infrastructure.

Storage: Colossus as the foundation

At the persistence layer, AlloyDB builds on Colossus, Google's exabyte-scale distributed storage system that underpins Google Search, YouTube, Gmail, Google Drive, Spanner, and Bigtable. A single Colossus cluster scales to exabytes of storage and tens of thousands of machines. With Spanner, we demonstrated that a transactional database engineered directly on Colossus can scale to thousands of nodes. The new AlloyDB architecture applies the same foundation to a new problem: agents.

Colossus has three properties that enable AlloyDB to satisfy the three tenets.

Direct, sub-millisecond I/O: Colossus is engineered to minimize read latency. A database node opening a Colossus stream receives a handle that describes where data physically resides. Authorization and metadata resolution happen once, when the stream is created; every subsequent read goes directly to the disks holding the data, over an optimized network protocol. The result is sub-millisecond latency across all of the database's data, with no intermediary to warm and no tier to miss.
Massive throughput: Colossus delivers up to 15 TB/s of aggregate throughput and 20 million queries per second to a single AlloyDB database without needing to provision bandwidth and with an unlimited number of concurrent hosts. At Colossus scale, a fleet of AlloyDB agent nodes is not a load the storage must be sized for; it is a fraction of the load the storage already serves!
Physical segment partitioning: AlloyDB serves agents from a separate set of Colossus segments, so agent I/O is deliberately spread away from the production data path rather than contending with it.

At no point along the data path — compute, network or storage — can an agent ever share a database component with production.

Network: Scalable bandwidth with Jupiter

Compute and storage are bound together by Jupiter, Google's high-capacity data center network. A single Jupiter fabric connects more than 100,000 servers with 13 petabits per second of bisection bandwidth — enough to carry a video call for every person on Earth.

Because Jupiter provides high bisection bandwidth with predictable low latency across the networking fabric, agent nodes can be scheduled flexibly anywhere in the cluster with consistent access to centralized storage. As the agent pool scales from zero to thousands, the underlying interconnect capacity absorbs the expanding traffic without creating placement bottlenecks.

Compute: Elastic and serverless PostgreSQL and analytics

At the compute layer, agents connect to AlloyDB's agent pool through MCP. The agent pool consists of AlloyDB agent nodes with read-only access to the up-to-the-second state of the database. This layer provides:

MicroVM isolation: Each agent node is a fully functional AlloyDB for PostgreSQL database engine running inside a lightweight, secure microVM. These instances are fully isolated from each other and from the dedicated primary cluster.
Rapid spin-up and scaling: Agent nodes are provisioned in response to requests from agents and stop automatically when the agents finish. In response to a burst, AlloyDB rapidly provisions thousands of agent nodes, serving millions of concurrent agents, and releases them as the agents finish. Because billing is per second of agent-node activity, a burst that uses a thousand nodes for tens of seconds will only be charged for the resources that the job consumed, and nothing more.

Meanwhile, the production cluster remains as it is today: pre-provisioned, on dedicated infrastructure, sized for the system of record. Agent nodes read from Colossus directly and see a consistent production state with sub-second freshness.

Beyond the agent pool, BigQuery and Spark can read AlloyDB data from Colossus with the same isolation from the production cluster, so agents can use lakehouse federation to join real-time operational data with large-scale lakehouse datasets.

By building on these Google-scale storage, network, and compute layers, AlloyDB’s new agentic database architecture achieves a remarkable goal: Share the data. Share nothing else.

Evaluating AlloyDB’s agentic database architecture

We tested AlloyDB by running concurrent index lookups over a dataset larger than available DRAM, testing scalability across the full stack. We ran the agentic workload starting with a single agent node — an independent database instance in the agent pool, rather than a traditional read replica — and scaled dynamically to thousands of nodes over a single database, measuring both the aggregate agentic throughput as well as any impact on production.

In this test, throughput scaled linearly from 3.9K to 41K QPS when expanding from one to 10 agent nodes. Scaling by two additional orders of magnitude yielded near-linear performance up to 1,000 nodes. We observed:

Zero primary degradation: Scaling from 1 to 1,000 agent nodes produced no measurable impact on primary cluster performance.
Massive throughput: Aggregate throughput dynamically scaled 773x to 3 million QPS, driving over 8 million IOPS in Colossus across 1,000 compute nodes.

In a similar benchmark running concurrent full table scans across 2,100 agent nodes, aggregate scan throughput exceeded 1 terabit per second.

Because the agent pool shares no physical infrastructure with the production cluster, teams can scale reasoning fleets to thousands of nodes without placing production systems at risk.

Engineering all three tenets by design

The table below shows how AlloyDB’s architecture satisfies each of the three tenets:

Every agentic database architecture will require these three foundational elements: storage with the properties of Colossus, a network that connects compute to that storage without constraint, and compute that can be provisioned and released at agent scale.

Google has spent more than two decades building exactly that, to run Google Search, YouTube, and Gmail. Now it underpins our agentic database architecture.

Give agents live data without impacting production

Every organization building with AI faces the same core dilemma: how to give agents full access to live operational data without putting the systems running the business at risk. Until now, architecture — not application needs — dictated that choice. Giving agents direct access to the database meant exposing mission-critical systems to unforecastable load, severe resource contention, and production outages.

An architecture built on these three tenets removes these compromises entirely. Agents reason over live production data with sub-second freshness. They have the complete engine at their disposal — every index, vector, full-text and spatial search, and the full capability of SQL — at sub-millisecond I/O. The architecture scales dynamically to thousands of isolated nodes when agents need it, then to zero when agents finish. Throughout, core transactional workloads remain untouched: no shared components, no shared quota, no correlated failures. Agents can deliver innovation without conflicting with business continuity.

The same property extends to every other reader of production data. Reporting, analytics and applications can freely read live data without putting production at risk, ending a constraint that has shaped operational databases for five decades.

The data in an enterprise's systems of record is its crown jewels. Built on this foundation, that data can finally be put to work in full.

Databases, unfettered.

To learn more, visit the documentation page, and sign up here to get started.

Announcing Native BM25 Ranking in AlloyDB and Cloud SQL

Fri, 18 Sep 2026 16:30:00 +0000

Vector search is a critical component of generative AI, retrieval-augmented generation (RAG), and data agent architectures, but sometimes vector search alone isn't enough. While vector embeddings are incredible at understanding conceptual meaning, they stumble on specific alphanumeric IDs and exact product SKU numbers. To build truly robust search and AI applications, you may need the combination of semantic vector search and traditional exact keyword full-text search — what we call hybrid search.

In search, Best Matching 25, or BM25, is a key algorithm used to estimate how relevant a document is to a given query. Until today, if you wanted BM25 ranking with AlloyDB or Cloud SQL, you needed to add an additional full-text search backend. This introduced data silos, sync lags, and operational complexity. Today, we are eliminating the friction of maintaining a separate full-text search backend altogether, with the preview of the native BM25 index in AlloyDB and Cloud SQL for PostgreSQL 17+, made possible through the open-source pg_textsearch extension created by Tiger Data.

Now, with a unified hybrid search backend, you no longer need to provision, manage, or pay for separate systems to get state-of-the-art full-text retrieval. It all happens directly inside your database, where your operational data lives, delivering:

Industry-standard keyword ranking: Powered by Tiger Data's pg_textsearch, bring lightning-fast, C-optimized BM25 scoring directly to your Postgres tables.
No complexity, total consistency: Eliminate the data duplication, ETL pipelines, and synchronization lag that you get when you maintain multiple backends for vector and full-text retrieval.
Supercharged semantic search (AlloyDB exclusive): Get up to 6x and 10x faster vector search queries (when compared to standard PostgreSQL) with ScaNN and HNSW index types.

Why pg_textsearch?

If you’ve used PostgreSQL's built-in ts_rank for full-text search at any meaningful scale, you already know its limitations. Ranking quality degrades as your corpus grows. There’s no support for inverse document frequency, so common words carry the same weight as rare ones. There’s no term-frequency saturation, so a document that mentions "database" 50 times outranks one that mentions it once.

BM25 is the information retrieval gold standard, providing inverse document frequency (rarer terms matter more), term frequency saturation (repetition doesn't dominate), and document length normalization. You can learn more in this blog post by Tiger Data about how they built a BM25 search engine on PostgreSQL pages.

Full-text search example

Here’s how to get started with BM25 full-text search on both AlloyDB and Cloud SQL. Consider a sample table, cymbal_products, that contains the unique identifier uniq_id, a product_name column, a product_description column containing a text description of each product, and a generated product_embedding column. cymbal_products contains information on various retail products, including indoor and outdoor plants.

Index creation

To use BM25, enable the pg_textsearch extension.

code_block: <ListValue: [StructValue([('code', '-- Install pg_textsearch extension\r\nCREATE EXTENSION pg_textsearch;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f1569617c10>)])]>

Create the index on the product_description column from the cymbal_products table.

code_block: <ListValue: [StructValue([('code', "-- Create the native BM25 index on the content column\r\nCREATE INDEX idx_docs_bm25 \r\nON cymbal_products \r\nUSING bm25 (product_description) \r\nWITH (text_config='english');"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f1569614090>)])]>

A BM25 full-text search query can be executed using the <@> special operator. In the snippet below, we search for ‘cherry tree’.

code_block: <ListValue: [StructValue([('code', "-- Full text search query\r\nSELECT product_name, product_description <@> 'cherry tree' AS bm25_score \r\nFROM cymbal_products\r\nORDER BY bm25_score \r\nLIMIT 5;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f1569615990>)])]>

Sample output is shown below. A more negative score indicates a stronger relevance match.

AlloyDB hybrid search example

Setting up a hybrid search system in AlloyDB is simple. You can create both your vector and keyword indexes on the same table and merge the results seamlessly using the hybrid search user-defined function (UDF).

Vector index creation

Here is how to create a ScaNN vector search index:

code_block: <ListValue: [StructValue([('code', '-- Install vector extension\r\nCREATE EXTENSION vector;\r\n\r\n-- Install scann extension\r\nCREATE EXTENSION IF NOT EXISTS alloydb_scann;\r\n\r\n-- Create scann vector search index \r\nCREATE INDEX cymbal_products_embeddings_scann ON cymbal_products USING scann(product_embedding cosine);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f15696144d0>)])]>

Hybrid search

AlloyDB provides an out-of-the-box hybrid search UDF that makes it very simple to run hybrid search queries. The UDF merges the ranked results from each search component into a single, unified list using the Reciprocal Rank Fusion (RRF) algorithm. This query utilizes the UDF to perform a vector search for ‘trees that grow taller than houses’ and a keyword search for ‘California’ in the product description.

code_block: <ListValue: [StructValue([('code', 'CREATE EXTENSION google_ml_integration;\r\n\r\nSELECT *\r\nFROM ai.hybrid_search(\r\n search_inputs => ARRAY[\r\n \'{\r\n "data_type": "vector",\r\n "weight": 0.5,\r\n "table_name": "cymbal_products",\r\n "key_column": "uniq_id",\r\n "vec_column": "product_embedding",\r\n "distance_operator": "public.<=>",\r\n "limit": 10,\r\n "query_vector": "ai.embedding(\'\'text-embedding-005\'\', \'\'trees that grow taller than houses\'\')::vector"\r\n }\'::JSONB,\r\n \'{\r\n "data_type": "text",\r\n "weight": 0.5,\r\n "table_name": "cymbal_products",\r\n "key_column": "uniq_id",\r\n "text_column": "product_description",\r\n "limit": 10,\r\n "ranking_function": "<@>",\r\n "query_text_input": "California"\r\n }\'::JSONB\r\n ],\r\n);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f1569615490>)])]>

As shown in the sample output below, results are ranked in descending order of their RRF scores.

Here, hybrid search bridges the gap between semantic intuition and exact keyword matching. While vector embeddings excel at grasping conceptual queries, like "trees that grow taller than houses", traditional full-text search provides the pinpoint precision needed for strict identifiers like "California." By fusing the two, AlloyDB helps ensure your application prioritizes highly specific, locally relevant results like ‘California Sycamore’ right at the top of the list.

Cloud SQL hybrid search example

In Cloud SQL, you can create both your vector and keyword indexes on the same table and merge the results seamlessly using Common Table Expressions (CTEs) and coalescing the RRF score, as shown below.

Vector index creation

Here is how to create an HNSW index in Cloud SQL.

code_block: <ListValue: [StructValue([('code', '-- Install vector extension\r\nCREATE EXTENSION vector;\r\n\r\n-- Create an HNSW index on the embedding column for fast approximate nearest neighbor search\r\nCREATE INDEX product_hnsw_idx ON cymbal_products USING hnsw(product_embedding vector_cosine_ops);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f1569614950>)])]>

Hybrid search

Here is the hybrid search query.

code_block: <ListValue: [StructValue([('code', "CREATE EXTENSION google_ml_integration;\r\n\r\n-- BM25 keyword results\r\nWITH keyword_results AS (\r\n SELECT uniq_id, product_name, \r\n ROW_NUMBER() OVER (ORDER BY product_description <@> 'California') AS rank_kw\r\n FROM cymbal_products\r\n ORDER BY product_description <@> 'California'\r\n LIMIT 10\r\n),\r\n-- Semantic vector results\r\nsemantic_results AS (\r\n SELECT uniq_id, product_name, \r\n ROW_NUMBER() OVER (ORDER BY product_embedding <=> google_ml.embedding('text-embedding-005', 'trees that grow taller than houses')::vector) AS rank_vec\r\n FROM cymbal_products\r\n ORDER BY product_embedding <=> google_ml.embedding('text-embedding-005', 'trees that grow taller than houses')::vector\r\n LIMIT 10\r\n)\r\n-- Reciprocal Rank Fusion (RRF) to merge and score both lists\r\nSELECT COALESCE(k.uniq_id, s.uniq_id) AS uniq_id,\r\n COALESCE(k.product_name, s.product_name) AS product_name,\r\n COALESCE(1.0 / (60 + k.rank_kw), 0) + COALESCE(1.0 / (60 + s.rank_vec), 0) AS rrf_score\r\nFROM keyword_results k\r\nFULL OUTER JOIN semantic_results s ON k.uniq_id = s.uniq_id\r\nORDER BY rrf_score DESC\r\nLIMIT 5;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f1569615410>)])]>

The resulting output is identical to the AlloyDB hybrid search results shown above.

Watch it in action

Watch how this all comes together in this demo video.

Relevant resources

We are incredibly excited to work with Tiger Data and cannot wait to see how you leverage native BM25 support to build faster, smarter, and simpler AI applications. Turn on the pg_textsearch extension today, and experience the ultimate hybrid search engine experience with AlloyDB and Cloud SQL.

Want to get started? Check out”

AlloyDB resources

New to AlloyDB? Discover AlloyDB with a 30-day free trial
Choose a vector index in AlloyDB AI
AlloyDB BM25 documentation
AlloyDB hybrid search UDF documentation

Cloud SQL resources

Cloud SQL BM25 documentation

Tiger Data pg_textsearch Release Page

How a solo founder runs a five-continent tender platform on AlloyDB and MCP

Thu, 17 Sep 2026 16:00:00 +0000

Editor's note: Lucius AI, a tender-intelligence startup covering markets across five continents, runs its entire data platform on AlloyDB for PostgreSQL with a single operator. By migrating semantic search to a ScaNN index and managing database operations through Model Context Protocol (MCP), query latency dropped by 47x while automating day-to-day administrative tasks via MCP.

Executive summary

Lucius AI runs a global tender platform spanning more than 210,000 tenders across the UK, EU, India, and Australia, requiring minimal operational overhead for a solo founder.
Lucius AI deployed AlloyDB for PostgreSQL to consolidate its relational catalog, audit logs, and vector embeddings into a single managed database engine.
Migrating semantic search to a ScaNN index lowered query latency from 1.14 seconds to 24 milliseconds — a 47x speedup on a representative production query.
Connecting an AI agent to AlloyDB using the Model Context Protocol (MCP) helps Lucius AI automate query analysis, data freshness checks, and incident forensics under strict least-privilege permissions.

Making tender intelligence work as a company of one

Lucius AI helps businesses bidding on public contracts evaluate opportunities across global markets. The platform ingests public procurement notices from the UK, the EU, the US and Canada, Australia and New Zealand, India and Singapore, alongside World Bank donor-funded notices across Africa and Asia. Lucius AI analyzes tender documents using Gemini to generate compliance matrices, bid recommendations, and draft responses citing original source pages. For small and mid-sized suppliers, this replaces days of manual document reviews and costly external consulting.

Running a platform of this scope requires extensive operational coordination:

Nightly ingestion from thirteen public procurement sources
A catalog of more than 210,000 tenders, including tens of thousands open for active bidding
Two production regions on Cloud Run: Europe, and an Australian deployment on its own AlloyDB cluster with customer-managed encryption keys (CMEK) for defense-adjacent customers
Ongoing analytics, performance tuning, data validation, and incident response

Managing these responsibilities without dedicated data engineering or database administration teams requires offloading operational maintenance. Lucius AI addressed this challenge on two fronts: using AlloyDB for PostgreSQL as the core system of record, and connecting an AI agent through the Model Context Protocol (MCP) to safely execute database operations.

Consolidating systems into AlloyDB

Rather than deploying separate relational databases, vector databases, and log stores, Lucius AI houses all core data in AlloyDB for PostgreSQL. The relational tender catalog, document metadata, audit logs, and vector embeddings reside in the same database engine. Storing vector embeddings alongside relational rows avoids managing separate vector stores, establishes a unified backup schedule, and centralizes identity management.

Authentication relies strictly on Cloud IAM. Services connect using dedicated Google Cloud service accounts mapped to database roles scoped to specific access requirements, without storing database passwords in application environments. Database reliability is managed natively by AlloyDB through automated backups and point-in-time recovery, avoiding custom disaster recovery procedures.

In production, this consolidated architecture supports:

More than 210,000 tenders in the catalog, with embeddings stored directly alongside them
Rebuilding the semantic index embedded 115,820 records in 10.6 minutes with the Gemini embedding model, for around three dollars in API spend; AlloyDB auto embeddings now keep those vectors current.
Retrieval reranking executed directly inside the database using the ai.rank function — with mean latency of 77-milliseconds - returning the most relevant results for search queries without requiring a standalone reranking microservice

Accelerating semantic search by 47x

Semantic search across the tender catalog initially relied on unindexed vector comparisons, where a representative query took 1.14 seconds. Migrating this workload to a ScaNN index in AlloyDB reduced query latency to 24 milliseconds — a 47x improvement.

The index recommendation originated from the AI agent during an automated performance audit, where it benchmarked the query plan before preparing the index migration.

Automating database operations with MCP

To delegate routine administrative tasks, Lucius AI configured the open-source MCP Toolbox for Databases using the prebuilt alloydb-postgres server.

Operational delegation requires strict access controls. The agent connects using a dedicated PostgreSQL role granted SELECT across the schema and UPDATE on a single operational table. Destructive commands (DROP, DELETE, TRUNCATE) are omitted, restricting agent actions to authorized operational boundaries.

Under this configuration, the AI agent performs regular database operations across four key areas:

On-demand analytics: Compiles retention cohorts, activation funnels, and catalog coverage by country via ad hoc SQL queries, removing the need to build and maintain manual dashboards or complex analytical pipelines.
Performance optimization: Performs query-plan inspections and index analysis, such as identifying the ScaNN indexing strategy.
Incident forensics: In response to an external security probe, the agent parsed audit logs to reconstruct the request timeline in minutes, verifying that tenant isolation remained intact.
Automated data-quality checks: Evaluates ingestion watermarks and freshness across all thirteen procurement sources every morning.

For teams adopting this architecture, establishing a progressive permission structure provides clear guardrails: start with read-only access, expand permissions as requirements dictate, and keep destructive operations restricted to human administrators.

Looking ahead

Lucius AI is planning three technical initiatives to further reduce operational overhead:

Automated vector embeddings in AlloyDB AI: After validating ai.initialize_embeddings across the full catalog, a weekly maintenance job uses ai.refresh_embeddings to update vectors.
Columnar engine acceleration: Having enabled AlloyDB’s columnar engine with auto-columnarization, the database identified and stored 40 frequently queried columns across four tables in memory within a day, accelerating reporting queries without a separate analytical store.
Managed Remote MCP Server: Transitioning from self-hosted Toolbox processes to Google Cloud's fully managed Remote MCP Server for AlloyDB will offload MCP server hosting and maintenance.

By anchoring core data in AlloyDB and managing routine operations through MCP, Lucius AI demonstrates how a single engineer can build and operate a resilient, multi-region procurement platform.

To explore Lucius AI, visit ailucius.com. To evaluate AlloyDB for PostgreSQL, deploy an AlloyDB cluster to test performance against your own workloads.

M4N VM family, now GA: Highest per-core IOPS and throughput for I/O and memory-bound workloads

Wed, 16 Sep 2026 16:00:00 +0000

As enterprise organizations scale mission-critical applications, storage I/O and memory access can become severe operational bottlenecks. Whether its Oracle databases, in-memory databases like SAP HANA, or high-throughput SQL Server clusters, EHR systems, and real-time big data analytics, memory-bound databases often force enterprises to over-provision compute cores (vCPUs) to get the RAM capacity and storage bandwidth they need, driving up costly third-party software licensing fees.

Today, we are thrilled to announce the general availability of the M4N machine series in Google Compute Engine, purpose-built for I/O intensive, high-memory workloads, the second offering in our network- and block-storage optimized VM family. Compared to similar offerings from other hyperscalers M4N provides the highest per-core IOPS and throughput for high-memory instances, and over 20% TCO reduction for Oracle databases.

M4N is also the industry’s first instance of network and block storage optimized with higher memory ratios (up to 26:1) and size (6TB). Powered by 5th Gen Intel® Xeon® Scalable processors and built on Google Cloud's custom Titanium offload architecture, M4N instances deliver up to 25,000 MiB/s (25 GiB/s) of aggregate host storage performance and up to 1 million IOPS when paired with Hyperdisk Extreme — doubling the block storage performance of current M4 instances.

M4N targets workloads that demand both extreme high-density RAM and uncompromising I/O performance, complementing our existing memory-optimized families (such as M1, M2, M3, M4, and X4) by solving specific storage and network bottlenecks for high-throughput enterprise applications.

Built for demanding workloads

Workload Category	Typical Applications	Why M4N Wins
Mission-critical enterprise DBs	Oracle, SAP HANA, SQL Server, IBM DB2, MySQL, PostgreSQL	Memory-to-core ratios (up to 26.57 GB/vCPU) paired with 25 GiB/s storage for rapid data ingestion, transaction logging, and zero-stall backup cycles.
Generative AI and RAG data layers	Milvus, Pinecone, Qdrant, Vespa, Redis, In-Memory Context Caching	Sub-millisecond similarity search across massive vector indexes in RAM, combined with 400 Gbps network bandwidth for distributed model retrieval.
Enterprise healthcare and ERP	Epic Systems (Operational Database), SAP ECC, SAP S/4HANA	Sustained I/O headroom that prevents query latency spikes during peak clinical/transactional hours.
Real-time analytics and EDA	Electronic Design Automation, Genomic Modeling, In-Memory OLAP	High memory capacity to load massive datasets entirely in RAM with maximum storage bandwidth for checkpoint dumps.

Optimizing Oracle licensing costs

Enterprise IT departments struggle with the rising cost of core-based software licensing. For workloads like Oracle database, licensing fees are typically calculated based on the number of vCPUs or physical cores assigned to the instance. Historically, this has forced a difficult trade-off: paying for more compute cores than necessary just to obtain the required amount of RAM and storage performance.

M4N changes this paradigm with its industry-leading high memory-to-vCPU ratio. By providing the highest per-core IOPS and throughput for high-memory instances of all the leading hyperscalers, M4N allows database administrators to:

Reduce TCO and licensing overhead: Stop over-provisioning of cores while meeting Oracle database performance density requirements, resulting in over 20% TCO reduction compared to similar offerings from leading hyperscalers.
Right-size infrastructure: Allocate the exact amount of compute power needed for the workload while still accessing massive memory pools.
Improve cache-hit ratios: With more memory available per core, larger portions of the database can reside in the system global area (SGA), reducing expensive I/O operations and further boosting efficiency.

What customers are saying

Early experiences with M4N show that workload-optimized infrastructure is the engine for transformation.

“Before M4N, meeting our demanding I/O requirements on Google Cloud often required over-provisioning our compute to achieve the necessary performance density. The new M4N instances solve this by delivering high throughput across the smaller to larger shapes.” - Sherri Trojan, Sr Principal Solution Architect, Sabre

"We are delighted to see Google Cloud introduce this next-generation high-performance infrastructure for mission-critical database workloads. The new compute platform demonstrates tremendous potential for enterprise Oracle deployments requiring scalability, resiliency, and performance. We are excited about what this innovation means for customers running Oracle workloads on Google Cloud.” - Bala Kuchibhotla, Co-Founder and CEO, Tessell

"With M4N, Google Cloud continues to push the boundaries of platform co-design. By combining 5th Gen Intel Xeon Scalable processors with Google's custom Titanium offload architecture, M4N delivers the extreme memory capacity, high memory bandwidth, and uncompromising I/O throughput required for the world’s most demanding mission-critical data environments." - Intel

What’s new: Scaling extreme data layers with M4N

M4N bridges two previously separate paradigms in cloud infrastructure: large memory footprints and extreme I/O performance. Engineered with custom Titanium offloads, M4N minimizes I/O bottlenecks without requiring infrastructure add-ons or compromises on memory density. Let’s take a look at how M4N fits into these environments.

1. Enabling high bandwidth data transfer

For workloads with large memory footprints, M4N provides:

Superior VM-to-VM bandwidth: Delivers up to 400 Gbps aggregate VM-to-VM network bandwidth and up to 50 Gbps single-flow bandwidth within the same VPC, unlocking non-blocking data exchange for distributed database clusters and real-time streaming data layers.
Enhanced internet and egress throughput: Enjoy up to 200 Gbps internet egress bandwidth and up to 48 MPPS packet processing performance.
High bandwidth out-of-the-box: Achieve full performance without needing to purchase or configure premium Tier_1 networking add-ons.

2. Dynamic storage performance with Hyperdisk

Paired with Google Cloud's next-generation storage portfolio, M4N with Hyperdisk lets you independently tune IOPS, throughput, and capacity:

Hyperdisk Extreme (HdX): Delivers up to 25 GiB/s aggregate block storage throughput and 1,000,000 IOPS—double the storage performance of standard M4. This is great for rapid database recovery, transactional checkpointing, and instant in-memory index reloads.
Hyperdisk Balanced (HdB): Scales up to 20 GiB/s throughput and 640,000 IOPS for cost-effective enterprise storage at scale.

M4N machine types and specifications

M4N instances are offered across three distinct memory-to-vCPU ratio tiers, scaling from 16 to 224 vCPUs and up to 5,952 GB of DDR5 RAM. M4N also offers predefined VM shapes across three distinct memory-to-vCPU ratios to match specific workload requirements, with support for Resource-based Committed Use Discounts (CUDs). Details here.

Get started today

The M4N instances are now available in select regions around the globe. To learn more about how the M4N family can enhance your memory- and I/O-bound applications and reduce your licensing costs, contact your account representative or explore the documentation.

Scaling Telco Autonomy: Leveraging GNNs with Distributed GraphFlow

Tue, 15 Sep 2026 16:00:00 +0000

The telecommunications industry is currently undergoing a paradigm shift, moving from traditional manual human-driven operations to fully Autonomous Network Operations. Modern networks have grown increasingly complex, heterogeneous, and large-scale, making handcrafted rules-based methods and traditional Machine Learning (ML) approaches alone insufficient to automate network operations. While ML methods can identify subtle patterns and make fine predictions from large amounts of structured data, they lack the ability to understand, reason about the data and the system it represents, and ultimately make the kind of decision a human operator would.

The growth of AI agents and their ability to reason is a promising solution to this shortcoming. However, in the same way a human operator is not capable of directly ingesting the statistical information spread across the billions of data points created in a large network, AI agents also lack the ability to operate at this scale. To address this challenge, telecommunications companies are adopting Graph Neural Networks (GNNs), a modern form of machine learning designed to operate natively on massive volumes of temporal and relational data. By integrating GNNs with AI agents, operators can combine advanced diagnostics such as root cause analysis, capacity planning, traffic forecasting, what-if simulations, and real-time anomaly detection with the reasoning power required to interpret these insights and execute justified actions. This powerful combination enables networks to safely move towards Level 5 Autonomy as defined by TM Forum, where the system operates autonomously.

In this post, we present the three components (Data, ML, and AI) that will power Google Cloud’s Autonomous Network Operations framework.

Google Autonomous Network Operations framework architecture

Foundation: Digital Twin on Spanner Graph

At the heart of Google Cloud’s Autonomous Network Operations framework is the network digital twin: a highly detailed, virtual replica that continuously mirrors its living telecommunications network in real time. Rather than being a static model, it is represented as a dynamic, temporal network graph that captures the evolving state and relations of its components over time. This architectural approach allows operators to "go back" in time to train and evaluate ML models on historical data, while providing AI agents with the foundational operational knowledge required to achieve Level 5 Autonomy. By simulating the impact of proposed network changes within this digital environment, the Digital Twin establishes a critical layer of trust, enabling AI agents to confidently design future states and automatically resolve network issues.

Google Cloud’s Spanner Graph is well suited to host this digital twin:

Scalability and Availability: Spanner Graph provides a no compromise foundation for modern applications, offering virtually unlimited scaling that grows as the network grows, along with 0-RPO/0-RTO and five 9s of availability.
Multi-Model Support: Supports multiple data models (Relational, Graph, Vector, and Full-Text Search) in a single platform allowing developers to build complex compositions such as graph transversals combined with nearest neighbor vector search.
Global Consistency: Spanner provides a globally consistent view of the network, simplifying system development.

The next figure illustrates a network topology with four node types: routers, interfaces (the physical ports), VPNs (L3VPN service instances), and flows (active traffic sessions). These are connected by directed edge types capturing the full network stack: physical containment (router-interface), physical links (interface-interface), control-plane peering (router-router via OSPF/iBGP), service membership (router-VPN), and traffic anchoring (flow-interface, flow-VPN).

High Level network topology

The ML layer: Distributed Graph Flow (DGF)

To predict how a network will behave and react, the digital twin leverages an ML layer powered by Distributed Graph Flow (DGF). By training on the vast volumes of structured historical data hosted within Spanner Graph, this layer uncovers critical predictive insights that enable human operators and AI agents to manage networks proactively rather than reactively.

DGF is a recently open-sourced Python library designed to manage the entire end-to-end lifecycle of GNN modeling. Developed by Google CoreML and Google Research, it brings a decade of internal Google-scale tools and expertise directly to Google Cloud enterprise clients. To accommodate different engineering needs, the library offers high-performance, composable, low-level primitives for advanced teams, alongside a simple API for rapid development that requires no prior GNN expertise.

For instance, training and evaluate a GNN model in GraphFlow with the high level API can be as simple as writing 5 lines of code:

code_block: <ListValue: [StructValue([('code', 'import dgf\r\n\r\n# Fetch the data from Spanner Graph\r\ngraph, schema = dgf.io.read_spanner_graph(...)\r\n\r\n# Train a node attribute prediction model\r\nmodel = dgf.learning.train_node_model(graph, schema, target_column="risk_score")\r\n\r\n# Evaluate the model\r\nmodel.evaluate()\r\n# Make predictions\r\nmodel.predict(graph, seed_node_idxs=[0, 1, 2])\r\n\r\n# Save the model for later\r\nmodel.save("/tmp/model")'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f156938cf50>)])]>

The DGF provides high-level concepts that map directly to Autonomous Network Operations requirements:

Use cases

By leveraging DGF and GNNs, telcos can move from reactive maintenance to proactive prevention through several advanced use cases:

Anomaly detection: GNNs generate node and edge embeddings that encapsulate historical patterns and current health. Any anomalous embeddings are flagged for review before they lead to service degradation.
Root cause analysis (RCA): DGF can output specific subgraphs containing only the relevant network instances related to an incident, such as "Attach Failures" in a specific ZIP code. This allows troubleshooting agents to perform high-speed analysis without scanning the entire global network.
Predictive maintenance: The system can predict the likelihood of device failures or edge breaks, such as "handover failures" for fast-moving equipment, enabling proactive load balancing or rerouting. Furthermore, by combining agents, remedial actions can be automated by adopting a ‘human-on-the-loop’/’human-in-the-loop’.
What-if analysis: GNNs enable Telcos to simulate scenarios like fiber cuts, or traffic surges or device configuration changes. By modeling topological dependencies, GNNs can predict how these local changes propagate across the entire network, allowing engineers to test resilience and evaluate mitigation strategies in a risk-free digital environment.

Scenario: Root cause analysis with GNNs and DGF

Once you have created a digital twin (example code), a straight-forward 5-step process can be used to implement Root Cause Analysis(RCA) detection using GNNs and DGF.

Connect to the Digital Twin: Use the DGF Spanner Graph connector (dgf.io.read_spanner_graph) to load the network topology directly from Spanner Graph's Digital Twin into the DGF environment.
Train a Supervised Node (or Edge) Prediction model: Depending on the training data and objective, you will train a supervised node prediction model to predict a target node feature or an edge prediction model to predict an edge between the root cause entity node and the affected entity node. For the given sample data you will use the high-level dgf.learning.train_node_model API to train a supervised node prediction model.
Use the node prediction model to predict root cause node: The node prediction model can be directly used to predict the impact score on the node with the anomaly. Entity nodes affected by the anomaly with highest predicted impact score will be the top candidates for root cause.
Deploy to Gemini Enterprise Agent Platform (formerly Vertex AI): Export the model and host it on a Gemini Enterprise endpoint to enable scalable, low-latency predictions.
Real-time Inference: Make prediction calls to the inference endpoint with the anomaly date as input. The endpoint will return the predicted root cause Entity nodes.

Get started today

The integration of GNN using Distributed Graph Flow into network operations is more than just a technical upgrade; it is a critical evolution for the telco industry. By moving towards a GNN-powered autonomous framework, operators can significantly shorten outage times, optimize capacity in real-time, and ultimately deliver a superior customer experience through improved operational efficiency.

To start building your own intelligent network applications, check out the Distributed GraphFlow (DGF) library, which provides the essential primitives for scalable GNN training and inference. For a hands-on experience, follow our step-by-step code sample. You can also explore our recent award-winning Moonshot project on Business-aware GNN-healing networks, and dive deeper into our approach on self-optimizing autonomous networks by reviewing this whitepaper.

Enterprise-grade PostgreSQL with AlloyDB Omni RPM Orchestrator is generally available

Wed, 09 Sep 2026 19:00:00 +0000

We are thrilled to announce the general availability of the AlloyDB Omni Red Hat RPM orchestrator, that brings production-ready security, resiliency, and low-downtime operations to PostgreSQL workloads in your enterprise environments. This GA milestone builds on the foundation laid during our preview release and launches alongside AlloyDB Omni version 18.3.0 to bring cloud-like database automation directly to your virtual machines and bare-metal servers with Google’s AI capabilities.

As of this GA release, AlloyDB Omni can be deployed in four modes to suit your requirements. Visit AlloyDB Omni documentation for more information.

Standalone container (Debian / UBI)
Container with Kubernetes operator for Highly Available enterprise deployment
Standalone RPM
With RPM orchestrator for Highly Available enterprise deployment

Why run a self-managed database?

For many use cases, a managed cloud database service is the simplest and most cost-effective option. However, there are scenarios where you may choose to run a PostgreSQL database yourself, on or off the cloud. The AlloyDB Omni RPM deployment is built for organizations that need the performance of the cloud with the control of local, non-containerized infrastructure, with use cases including:

Workload Modernization: AlloyDB Omni is more than 2X faster for transactional workloads and can deliver up to 100X faster analytical queries than standard PostgreSQL L , revitalizing existing infrastructure without a full migration.
Regulated Environments: For industries with strict data residency and security requirements, the RPM orchestrator provides the necessary tools like SELinux and local audit logging to stay compliant.
Edge and On-Premises Deployment: Deploying at the edge or on bare-metal servers allows for low-latency processing and disconnected operation.
AI-Ready Infrastructure: You can provision database clusters for AI integrations, using AlloyDB AI capabilities such as vector search for modern generative AI applications directly on-premises.

Flexible Reference Architectures

The AlloyDB Omni RPM orchestrator offers flexible deployment models tailored to your organization's specific operational requirements—whether your focus is maximizing performance, scaling read throughput, or ensuring robust high availability (HA). For more details, refer to the AlloyDB Omni availability reference architecture overview. The orchestrator simplifies cluster provisioning and lifecycle management by allowing you to define reference architecture specifications, customizable by adjusting instance parameters, node configurations, and networking options. Here is an example deployment view of scalable AlloyDB Omni HA reference architecture.

High availability reference architecture for AlloyDB Omni clusters with RPM Orchestrator

The diagram illustrates a highly available, distributed database reference architecture for AlloyDB Omni managed by the RPM Orchestrator. It shows that the client applications connect to a robust load-balancing tier and the load balancer routes read-write traffic directly to the active primary database node, while read-only traffic is routed to the replica nodes. The load balancer uses a Virtual IP (VIP) and is highly available itself, with Keepalived (for VIP failover), PgBouncer (for PostgreSQL connection pooling), and Haproxy (for routing and load balancing). The AlloyDB Omni Primary instance database nodes achieve high availability by replicating data synchronously across multiple zones. To scale out read-heavy workloads without impacting the primary HA cluster, separate Read Pool instances are deployed. These receive Async Replication (asynchronous) from the active node and can be scaled out.

It shows how an independent control plane manages the entire configuration and health of the clusters. The administrator interacts with the RPM Orchestrator, to oversee the lifecycle of the databases. The control plane includes redundant Cluster Managers and a 3-node etcd based Distributed Configuration Store to reliably maintain cluster state, and manage configurations. The controllers directly interface with the Node Manager running on each individual database node. For deploying only a single cluster, you may run control and data plane components on the same set of nodes.

High Availability and Read Scalability

Maintaining uptime and scaling reads for demanding workloads is simpler with the RPM orchestrator. It provides a resilient architecture capable of automatically handling failures across the stack, including the ability to handle failure of all Data / Control Path components, Readable Standby, as well as mitigating any network disruptions between the nodes.

The orchestrator now supports read pools for scaling out your read workloads and gives you the ability to create or add read pools to a cluster dynamically to meet the needs of analytical queries or similar workloads. The system also configures dedicated read endpoints for both standby nodes and readpools.

We are continuously expanding the capabilities of the AlloyDB Omni RPM orchestrator. Stay tuned for upcoming features, including advanced enterprise-grade capabilities to further strengthen business continuity and cross-region resiliency.

Data Protection and Security

Data security and recovery are at the core of the RPM Orchestrator. In this release, we have integrated automated backup and restore capabilities that enable you to configure a backup schedule and manage fully automated backup to GCS or S3-compatible storage automatically. The orchestrator allows you to execute backups to S3 or GCS buckets, or locally. Additionally, point-in-time recovery and fully automated in-place PIT restore are natively supported.

The RPM orchestrator supports SELinux enforcement at or after bootstrap to satisfy strict enterprise compliance and security standards and ensure mandatory access control and strong process isolation.

Database Operations

We’ve reduced the operational overhead associated with managing database fleets:

Zero-Hassle Low-Downtime Maintenance: Managing lifecycle updates and scaling operations is easier with the new, fully automated Low Downtime Maintenance (LDTM). Minor version upgrades as well as CPU and memory resource adjustments are executed with minimized downtime and automatic rollback support for maximum availability.
Dynamic Configuration: Database administrators can dynamically tune settings without hassle, including the ability to Modify GUCs/configs at or after bootstrap. You can also provision a cluster for AI integrations.
Simplified Cluster Maintenance: Managing your cluster footprint is straightforward, with native operational capabilities to add or remove database nodes as your workload demands shift.

Observability, AI, and Extensions

Monitoring and tuning your fleet requires deep visibility and the right set of tools:

Advanced Logging: The orchestrator simplifies auditability and debugging by providing Data and Control Path log direction to log disk.
Rich Observability: Custom metrics support allows you to fine-tune observability to suit your monitoring ecosystem. With custom metrics, you can track business-level events directly from the database, such as the number of new user registrations per minute, active sessions for a specific tenant, or the volume of orders processed, and export these to your company's central observability platform.
AI & Extensions: In addition to the list of extensions supported with AlloyDB Omni, the orchestrator includes support for all AlloyDB Omni's AI features such as query using natural language, AI-powered searches, AI functions, etc.

Get Started Today

The AlloyDB Omni Red Hat RPM orchestrator offers a new way to manage PostgreSQL-compatible workloads on bare metal or VM platforms, combining the high performance of AlloyDB, access to generative AI features and Gemini models to build AI agents and applications, and full automation.

Ready to elevate your on-premises database operations? Dive into our AlloyDB documentation to get started with the GA release today. Sign-up today !!

You can also try our new codelab to deploy a highly available AlloyDB Omni cluster using the RPM Orchestrator.

Beyond DMS: Accelerating Migrations SQL Server Logins and Users to Cloud SQL

Wed, 09 Sep 2026 16:30:00 +0000

So, you’ve planned your database modernization journey. You’ve set up Google Cloud’s Database Migration Service (DMS), configured replication, and successfully synchronized your application databases from your on-premises or cloud systems to a fully managed Cloud SQL for SQL Server instance.

The replication is complete, the data is up to date, and you’re ready for cutover. But when your application attempts to connect to the newly migrated database, you’re hit with a frustrating roadblock:

Msg 18456, Level 14, State 1, Line 1: Login failed for user 'app_user.

The culprit is simple: your SQL Server logins didn't migrate with your database. In this post, we’ll look at why this gap exists, why it actually protects your organization's security posture, and how easy it is to bridge using standard, time-tested SQL Server tools.

Why DMS doesn't migrate logins: Security and compliance

Database Migration Service (DMS) is highly efficient at replicating database-level schemas and transactional data. However, it purposefully doesn’t migrate instance-level objects, such as the system master database or server logins and permissions.

While this might feel like a missing feature, it is actually a deliberate design choice built around three core pillars:

Security Isolation and Privilege Boundaries: The source environment and the destination Cloud SQL environment operate under different security paradigms. Replicating the master system database directly could lead to unauthorized privilege escalation. For example, an on-premises login with sysadmin privileges shouldn’t have unrestricted sysadmin access to a fully managed Google Cloud database. When the cloud provider manages physical backups, patching, and security, it needs to limit underlying operating system access to ensure correct operation.
Compliance and Audit Governance: Automated migration of encrypted password hashes and server-level security credentials without explicit administrator oversight frequently violates enterprise compliance frameworks such as PCI-DSS or SOC 2. By keeping security object migration as a deliberate, administrator-driven step, organizations can guarantee that only approved identities are provisioned in the cloud landing zone.
The Need for Identity Modernization: Migrating to the cloud is the perfect opportunity to update and prune stale credentials. Frequently, on-premises instances carry legacy SQL logins that are no longer used. Replicating them blindly to a cloud-managed service is a security anti-pattern. Furthermore, moving to Cloud SQL is often the catalyst for shifting away from legacy SQL authentication toward modern, cloud-native identity solutions like Customer-Managed Active Directory (CMAD).

Understanding logins vs. users: The SID connection

To migrate logins successfully, let’s briefly revisit how SQL Server manages security. SQL Server separates identity into two distinct layers:

Logins (server-level): Stored in the master database. These authenticate a client connection to the SQL Server instance.
Users (database-level): Stored inside individual user databases. These authorize what actions a connection can perform within that specific database.

The bridge between a server login and a database user is a unique Security Identifier (SID).

When you backup and restore a database (or use DMS to replicate it), the database-level users (and their corresponding SIDs) are migrated inside the database files. However, if the corresponding server-level login does not exist in the destination master database—or exists but has a different SID—the mapping breaks. This results in "orphaned users" who have database access permissions but no way to authenticate at the server level.

Figure 1: How migrating databases without corresponding logins or with mismatched security identifiers (SIDs) results in orphaned users on the destination instance.

The recommended solution: Replicating logins using sp_help_revlogin

Instead of manually recreating every login and guessing password hashes, we can rely on a classic Microsoft-provided script: sp_help_revlogin.

This script generates a T-SQL query containing the CREATE LOGIN statement for every SQL Server authentication login on your source instance, complete with its original, encrypted password hash and its exact Security Identifier (SID).

Step 1: Create the helper procedures on your source instance

Connect to your source SQL Server instance using SQL Server Management Studio (SSMS). Copy and execute the official Microsoft script to create the two required stored procedures in your source master database: sp_hexadecimal and sp_help_revlogin.

Step 2: Generate the migration script

Once the procedures are created, run the following statement in your SSMS query window. Make sure to toggle your output settings to Results to Text (Ctrl + T) to copy the output cleanly:

code_block: <ListValue: [StructValue([('code', 'EXEC master.dbo.sp_help_revlogin;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f1569ffb090>)])]>

The output will contain auto-generated T-SQL statements that look similar to this:

code_block: <ListValue: [StructValue([('code', 'CREATE LOGIN [app_user] WITH PASSWORD = 0x01004F3D... HASHED, SID = 0x8D2F..., DEFAULT_DATABASE = [CustomerDB]'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f15694c2c50>)])]>

By scripting out the login with the HASHED password option and the original SID, SQL Server allows us to safely recreate the login with its original password and secure link intact.

Step 3: Apply the script to Cloud SQL

Copy the generated script, connect to your destination Cloud SQL for SQL Server instance, and execute the query. Your logins are instantly created in the cloud with their correct passwords.

By running the script generated by sp_help_revlogin, we replicate the logins onto the destination Cloud SQL instance with their exact security identifiers (SIDs) and password hashes intact. As shown below, this ensures that the database-level users automatically map to their server-level logins upon database migration, avoiding “orphaned users” entirely.

Figure 2: The unified migration process using the sp_help_revlogin script to preserve password hashes and original SIDs, resolving user mapping on Cloud SQL for SQL Server.

Note:
sp_help_revlogin is a stored procedure that was created and is maintained by Microsoft. Make sure to download the latest version and read the documentation.

Troubleshooting orphaned users

If you had created a login on the target Cloud SQL instance manually before running sp_help_revlogin, the SIDs might not match, causing the user to become "orphaned."

If you find an orphaned user (say, app_user), you can easily remap it to the newly created server login with a single command:

code_block: <ListValue: [StructValue([('code', 'ALTER USER [app_user] WITH LOGIN = [app_user];'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f15694c1f10>)])]>

With that command, the database user and the server login are immediately reunited via their SIDs, and application connectivity is fully restored.

Take your security a step further

While migrating SQL logins using sp_help_revlogin is the easiest path for a lift-and-shift migration, consider utilizing your cloud migration to modernize your authentication. Cloud SQL for SQL Server supports robust integrations with Customer-Managed Active Directory (CMAD). Integrating your destination instance with Active Directory allows you to deprecate legacy SQL logins in favor of centralized, enterprise-grade Kerberos authentication.

Wrap up

Database migration is more than just shifting rows of data—it’s about ensuring your applications remain secure, compliant, and operational from day one. While Google Cloud’s DMS handles the heavy lifting of data replication, migrating your logins is a straightforward, three-step process that guarantees a seamless cutover.

To learn more about optimizing your migration strategy, check out the Cloud SQL for SQL Server Migration Guide and explore how Database Migration Service can streamline your move to Google Cloud.

Spanner: Removing cumulative mutation limits for DML transactions

Wed, 09 Sep 2026 16:00:00 +0000

Spanner is Google Cloud’s no-compromise operational database that gives you the horizontal scale and always-on availability of a modern distributed system along with the rich feature set and familiar ecosystem of a relational database. Innovators in industries like banking, retail, media and entertainment, and AI infrastructure rely on Spanner today for their most critical workloads. We’re excited to announce a new, flexible way to handle larger, more complex transactions in Spanner, simplifying applications that need the highest levels of data consistency.

Operational workloads typically combine real-time decision making with granular updates: Think: identifying fraud as part of a multi-step checkout process in an ecommerce app. These changes must be transactional; either all of them succeed or none of them do and subsequent requests see the correct data. This update to Spanner’s ACID transactions allows applications to handle more data in an update without compromising on consistency, scalability, or availability using familiar DML.

Higher ceiling, more flexibility

Previously, Spanner capped the changes a query could perform in a transaction, for example using DML, at 80,000. That was roughly computed as the product of the number of rows and number of columns updated, plus any dependent indexes. Applications evolve over time to handle more data and provide new functionality. These changes increase the size of transactions, potentially causing previously small transactions to hit this limit.

This update shifts the 80,000 mutation mod limit from the transaction to individual DML statements. DML statements no longer contribute to an overall transaction-level mutation limit. A single transaction can now contain any number of DML statements, such as INSERT, UPDATE, or DELETE, provided that each individual statement generates fewer than 80,000 mutation mods.

Key benefits

Larger transactions: Group DML statements logically based on business requirements rather than artificially splitting them to comply with cumulative mutation limits.
Seamless transition: This change is compatible with all existing Spanner client libraries and requires no updates to application code.

Technical considerations

Locking and aborts

While you can now include more DML statements in a single transaction, be aware that larger and longer-running transactions hold locks for a greater duration. This may increase the likelihood of lock contention and transaction aborts. Keeping transactions concise helps maintain high performance and minimize resource contention.

DML vs. Mutation API

The application of limits depends on the method used to modify data:

DML Statements: Each statement (e.g., executeUpdate) is evaluated independently against the 80,000 mod limit.
Mutation API: When using client library methods like insert() or update(), mutations are provided during the Commit call. The 80,000 limit continues to apply to the entire set of mutations included in that single call.

Understanding mutation mods

Spanner counts "mods" based on the complexity of changes, including modified cells, primary keys, and secondary index updates. Please look at this blog for more details on how mutations are counted. You can monitor the total mods for a committed transaction via the mutation_count in the CommitStats. Note that the mutation_count will include all the mutations that are part of the transaction, across all DML statements and commit calls.

Java implementation example

The following example demonstrates how multiple DML statements can be executed within a single transaction under the new limit logic.

code_block: <ListValue: [StructValue([('code', 'import com.google.cloud.spanner.DatabaseClient;\r\nimport com.google.cloud.spanner.Statement;\r\nimport com.google.cloud.spanner.TransactionContext;\r\nimport com.google.cloud.spanner.TransactionRunner.Work;\r\n\r\n// Assuming dbClient is your initialized DatabaseClient\r\ndbClient\r\n .readWriteTransaction()\r\n .run(\r\n new Work<Void>() {\r\n @Override\r\n public Void doWork(TransactionContext transaction) throws Exception {\r\n // Each executeUpdate call is evaluated separately against the 80k mod limit.\r\n\r\n // Example 1: Updating specific products\r\n Statement stmt1 = Statement.newBuilder(\r\n "UPDATE Products SET InStock = FALSE WHERE ProductId = @productId")\r\n .bind("productId").to(1L)\r\n .build();\r\n transaction.executeUpdate(stmt1); // Verified against 80k limit\r\n\r\n Statement stmt2 = Statement.newBuilder(\r\n "UPDATE Products SET InStock = FALSE WHERE ProductId = @productId")\r\n .bind("productId").to(2L)\r\n .build();\r\n transaction.executeUpdate(stmt2); // Verified against 80k limit separately\r\n\r\n // Example 2: Inserting related order data\r\n Statement stmt3 = Statement.newBuilder(\r\n "INSERT INTO OrderItems (OrderId, ItemId, Quantity) VALUES (@orderId, @itemId, @qty)")\r\n .bind("orderId").to(100L)\r\n .bind("itemId").to(1L)\r\n .bind("qty").to(2)\r\n .build();\r\n transaction.executeUpdate(stmt3); \r\n\r\n Statement stmt4 = Statement.newBuilder(\r\n "UPDATE Orders SET LastUpdated = PENDING_COMMIT_TIMESTAMP() WHERE OrderId = @orderId")\r\n .bind("orderId").to(100L)\r\n .build();\r\n transaction.executeUpdate(stmt4); \r\n\r\n return null;\r\n }\r\n });'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f1569321390>)])]>

What has not changed

Individual statement limit: Any single DML statement that generates more than 80,000 mods on its own will still return the same error as we do today.
Other transaction limits: Other constraints such as the maximum transaction size in bytes remain in effect. They are documented here.

Best practices

Monitor CommitStats: Utilize the mutation_count returned in CommitStats to understand the load generated by your operations.
Optimize large operations: If a single statement (like a bulk update) exceeds the limit, consider using Partitioned DML or paginating through keys.

Spanner is the trusted choice for operational applications that need to scale without downtime. This increase to the mutation limit provides developers new flexibility to run larger transactions that leverage Spanner’s global consistency. Learn how Spanner can help your teams innovate faster with less risk, or try it on your own, with a free trial or production instances starting as low as $54/month.

External references

How AlloyDB ScaNN scales vector search to 10 billion vectors

Thu, 20 Aug 2026 16:00:00 +0000

To satisfy the demands of enterprise-grade agentic AI applications, underlying vector databases often struggle to scale effectively as modern use cases can scale to billions of vectors.

As a fully managed PostgreSQL-compatible database service, AlloyDB is engineered to handle demanding enterprise workloads. Combining Google's infrastructure with the reliability of commercial databases, it delivers high availability, scalability, and includes a cutting-edge analytical engine, optimal for agentic AI use cases. A key part of this is its ScaNN index, which now operates efficiently at a scale of 10 billion vectors. This was achieved through a major architectural enhancement: an innovative four-level tree (preview) paired with efficient memory usage.

The 10 billion vector scale challenge

Scaling to a 10 billion vector workload presents significant memory and computational challenges. Previous AlloyDB ScaNN tree-based index was limited to two- or three-level tree configurations, and attempting to scale those structures led to several bottlenecks:

Increased compute intensity: Larger tree structures demand significantly more operations for both index construction and query traversal.
Memory constraints: The sampling processes required for 10 billion vectors can easily exceed the system's available memory capacity.

Solution: Four-level architecture

The introduction of a four-level tree (preview) is the primary innovation in the recent AlloyDB ScaNN release. This architecture, illustrated in Figure 1, employs a top-down strategy to optimize the balance between accuracy and build efficiency. To maintain high performance and mitigate recall loss, the system integrates key enhancements such as Top-K branch, SOAR, centroid adjustment and balanced tree shape.

Figure 1. AlloyDB ScaNN four-level tree architecture

This design has two primary benefits:

1. Reduced compute intensity via hierarchical partitioning

The four-level architecture drastically reduces compute intensity by using hierarchical partitioning to restrict the volume of vectors scanned during a query. Instead of traversing a flat or poorly segmented space, the multi-layered hierarchy narrows down the search path exponentially. Figure 2 illustrates the search spaces across different tree levels, demonstrating how structural layering optimizes traversal efficiency:

Figure 2. Search space for two-, three- and four-level trees

Two-level: Utilizes coarse partitioning to guide queries, resulting in a basic search complexity of O(N^1/2).
Three-level: Introduces an intermediate layer to further subdivide clusters, narrowing exploration to O(N^1/3).
Four-level: Implements refined, highly granular partitions that optimize traversal efficiency down to O(N^1/4), sufficiently allowing for more than 10-billion vectors.

By dynamically expanding hierarchical layers as the dataset expands, AlloyDB ScaNN maintains ultra-low query latency and avoids computational scale walls from impacting performance.

2. Efficient memory usage

Achieving a 10 billion vector scale requires high memory efficiency. AlloyDB ScaNN uses these strategies to maximize memory management performance:

Balanced tree shape construction: The four-level tree utilizes a balanced configuration to circumvent memory limitations that restrict the size of training datasets. This balanced architecture effectively leverages reduced sampling sizes to construct high-fidelity tree partitions.
Sampling optimization: When the system encounters memory limitations, it generates a condensed sampling set that considers performance and accuracy.

Performance test results

By leveraging the innovative four-level tree architecture in our internal tests, we are able to achieve the following performance results:

AlloyDB can scale to over 10 billion vectors with its ScaNN index.
AlloyDB can deliver <= 51 ms p95 latency and 95% recall at 10 billion vectors with its ScaNN index.

Get started today

Experience AlloyDB ScaNN's four-level tree (preview) architecture today. You can deploy ScaNN for AlloyDB by following our quickstart guide to set up an instance. For optimized, high-speed vector search, refer to the official ScaNN documentation. New users can also explore AlloyDB through our 30-day free trial program. We can’t wait to hear about what you build!

Using BigQuery Graphs with measures for trusted agentic workloads

Thu, 13 Aug 2026 17:00:00 +0000

When enterprises transition from using simple chat assistants to autonomous, agentic workloads, they quickly run into a hard truth: Agents are prone to inaccurate insights when working with directly raw tables.

BigQuery Graph helps organizations move beyond flat, static tables to represent enterprises exactly how they exist in the physical world: as interconnected business entities with real-world dependencies. With the support of measures in BigQuery Graph (preview), we are unifying governed metrics with relationship mapping. This allows your agents to reason across complex dependencies captured in graphs with precision of measures.

Why relationships matter

Traditional data structures are blind to multi-hop business context, causing AI agents to make incorrect operational decisions:

The concrete problem: If a retailer has an agent who is asked why winter jacket sales dropped 12% in Seattle, it can query flat tables to report the what (the 12% dip). But it fails at the why because it cannot trace the relational path: Seattle orders ➔ distribution centers ➔ suppliers delayed by regional storms.
The risk of disjointed systems: Lacking relationship context, the agent suggests an irrelevant 15% markdown campaign, needlessly eroding margins. Furthermore, maintaining separate systems - where one team maps supplier relationships in a separate graph database while another maintains SQL metrics - forces your agent to stitch these stacks together at runtime. This process is slow, expensive, and leads to inconsistent KPI calculations.

Measures in BigQuery Graph solves this by letting you map existing tables to a property graph in-place with zero ETL. This unified setup enables a logical evolution of inquiry:

Metadata grounding establishes what data you have.
Business metrics (measures) calculate how your business performed.
Relationship mapping (graph) uncovers why it happened.

Under the hood

Historically, standard SQL joins during graph traversals duplicate rows, leading to incorrect aggregation calculations. BigQuery Graph solves this natively.

Data modelers define a MEASURE (like SUM or AVG) directly within the Property Graph DDL. Using standard SQL via the GRAPH_EXPAND function and the AGG aggregator, the engine resolves the structural graph paths before evaluating metrics. This ensures your agent is smart enough to know when it needs a calculator (SQL) and when it needs a map (graph).

Because public projects like bigquery-public-data are strictly read-only, you must map the logical property graph inside your own project using a placeholder variable (YOUR_PROJECT_ID), while directly referencing the read-only public tables as nodes and edges.

code_block: <ListValue: [StructValue([('code', '-- 1. Map the graph inside YOUR project \r\n\r\n\r\nCREATE OR REPLACE PROPERTY GRAPH `YOUR_PROJECT_ID.YOUR_DATASET.thelook_ecommerce_graph`\r\nNODE TABLES(\r\n `bigquery-public-data.thelook_ecommerce.users` AS User\r\n KEY(id)\r\n LABEL User PROPERTIES(id, city, country),\r\n `bigquery-public-data.thelook_ecommerce.orders` AS Order\r\n KEY(order_id)\r\n LABEL Order PROPERTIES(\r\n order_id, \r\n MEASURE(AVG(num_of_item)) AS avg_items_per_order,\r\n MEASURE(SUM(num_of_item)) AS total_items\r\n )\r\n)\r\nEDGE TABLES(\r\n `bigquery-public-data.thelook_ecommerce.orders` AS OrderedBy\r\n SOURCE KEY(order_id) REFERENCES Order(order_id)\r\n DESTINATION KEY(user_id) REFERENCES User(id)\r\n LABEL ORDERED_BY\r\n);\r\n\r\n-- 2. Query your new graph with standard SQL—using standard {Label}_{Property} column outputs\r\nSELECT\r\n User_city AS city,\r\n ROUND(AGG(Order_avg_items_per_order), 2) AS agg_avg_items,\r\n ROUND(AGG(Order_total_items), 2) AS agg_total_items\r\nFROM GRAPH_EXPAND("YOUR_PROJECT_ID.YOUR_DATASET.thelook_ecommerce_graph")\r\nGROUP BY User_city\r\nORDER BY agg_total_items DESC\r\nLIMIT 10;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f1569616390>)])]>

Democratizing graph intelligence in BigQuery Studio

To make managing and deploying these relationship networks frictionless for both developers and business users, we have built native, intuitive operational tools directly into BigQuery Studio:

Visual graph modeler: A no-code, drag-and-drop interface inside BigQuery Studio that lets you visually build, edit, and map property graphs, nodes, and edges without writing complex DDL scripts manually.

Conversational Analytics (CA) integration: Users can interact with the graph naturally. Instead of guessing table joins, Conversational Analytics agents navigate the deterministic, relationship-aware map of the graph, converting natural language questions into precise, boundary-constrained GoogleSQL or ISO GQL queries. This prevents model hallucinations and enforces semantic consistency.

Unified semantics: Native Looker integration

To avoid maintaining fragmented logic stacks, business metrics must live at the data layer. By integrating Looker (LookML) natively with BigQuery Graphs as in-database analytic models, you define logic once at the core:

Database-managed models (sql_analytic_model_name): Point Looker directly to your database-defined BigQuery Graph using sql_analytic_model_name to map standard LookML dimensions and measures directly to your graph properties.
Looker-managed models (derived_analytic_model): Define your BigQuery Graph schema directly inside your LookML view using derived_analytic_model. Looker will dynamically generate and execute the SQL DDL statements to maintain the graph inside BigQuery.
Enterprise DevOps workflows: Manage your graph's entire lifecycle using the Looker IDE, Git-based version control, and Continuous Integration (CI). Core KPIs (like Churn Rate) remain completely identical, verified, and trusted.

Accelerate PostgreSQL migrations using Gemini in Database Migration Service

Tue, 11 Aug 2026 16:00:00 +0000

Imagine this scenario: Your team decides to migrate a core application from an existing commercial database like Oracle or SQL Server to open source PostgreSQL or a fully managed service such as AlloyDB for PostgreSQL.

The initial phase goes smoothly. Schemas convert, tables populate, and data migration pipelines transfer terabytes of data in hours. The project looks ahead of schedule.

Then your team hits the bottleneck.

Buried inside the existing databases are hundreds of stored procedures, complex triggers, and custom functions written in proprietary SQL dialects like PL/SQL or T-SQL. These routines contain years of critical business logic handling transaction validation, order processing, and custom reporting.

Suddenly, your modernization project halts. Translating thousands of lines of procedural logic demands specialized dual-dialect expertise, months of manual rewriting, and high risk of conversion errors. This code translation represents the "last mile" bottleneck of database migration and is the most complex part of migrations.

Thankfully, recent advancements in AI provide a solution to the last mile problem. Database Migration Service (DMS) includes AI-assisted code conversion powered by Gemini. By bringing generative AI directly into your migration workflow, you can convert stored procedures, triggers, and custom functions into PostgreSQL PL/pgSQL code faster and with higher accuracy.

The stored procedure conversion challenge

Commercial database engines rely on vendor-specific syntax for stored procedures, user-defined functions, package bodies, and conditional logic. Converting this logic to PostgreSQL PL/pgSQL requires mapping variable definitions, exception handling blocks, cursor loops, and built-in functions.

When migrating complex enterprise schemas with hundreds of stored procedures, manual code conversion often demands months of engineering effort. Database teams must parse legacy logic line by line, re-implement conditional branches, and verify data type conversions between engines.

AI-assisted code conversion in DMS

Gemini in Database Migration Service accelerates this conversion work directly inside the Google Cloud console. DMS provides automated schema conversion alongside AI-generated code suggestions that explain structural differences between the source dialect and PostgreSQL.

The service presents converted PL/pgSQL code side-by-side with original source code, allowing database teams to review, edit, and validate suggestions in real time.

Figure 1: Database Migration Service interface displaying side-by-side code conversion and Gemini inline explanation.

Why integrated AI matters

Most AI apps and tools from major vendors have the ability to generate and convert code, including SQL code. However, converting enterprise databases demands far more than snippet translation offered by generic AI chat tools. Gemini in Database Migration Service offers several key advantages:

Full schema context: Rather than evaluating code snippets in isolation, Gemini in DMS analyzes your entire database context, including table relationships, data types, dependent views, and cross-procedure references across the whole migration project.
Enterprise security and privacy: Code conversion runs strictly within your Google Cloud project boundaries and IAM governance, protecting proprietary business logic and intellectual property.
Integrated execution workspace: DMS eliminates manual copy-pasting across hundreds of files. You can review side-by-side diffs, inspect inline AI explanations, edit code, and deploy validated PL/pgSQL routines directly to target databases within a single console.
Deterministic accuracy and AI compilation: DMS pairs deterministic compiler rules for 1:1 mappings (such as standard DDL transformations, scalar functions, and well-defined syntax conversions) with Gemini contextual synthesis for complex procedural blocks—guaranteeing exact, predictable translation without model drift.

Converting Oracle PL/SQL to PostgreSQL

Consider an Oracle PL/SQL stored procedure that calculates customer order totals and applies tier-based discounts using proprietary NVL and DECODE functions. In the original workflow, you must manually map NVL to COALESCE, rewrite DECODE statements as standard CASE expressions, and adjust exception blocks like WHEN NO_DATA_FOUND THEN.

When you run a migration assessment in DMS, Gemini analyzes the source procedure and produces native PostgreSQL PL/pgSQL code:

Source: Oracle PL/SQL

code_block: <ListValue: [StructValue([('code', 'CREATE OR REPLACE PROCEDURE calculate_discount (\r\n p_customer_id IN NUMBER,\r\n p_discount OUT NUMBER\r\n) AS\r\n v_total NUMBER := 0;\r\nBEGIN\r\n SELECT NVL(SUM(amount), 0) INTO v_total\r\n FROM orders WHERE customer_id = p_customer_id;\r\n \r\n p_discount := DECODE(TRUE, v_total > 10000, 0.15, v_total > 5000, 0.10, 0.05);\r\nEXCEPTION\r\n WHEN NO_DATA_FOUND THEN\r\n p_discount := 0;\r\nEND;\r\n/'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f158823f710>)])]>

Target: PostgreSQL PL/pgSQL (Converted by Gemini in DMS)

code_block: <ListValue: [StructValue([('code', 'CREATE OR REPLACE FUNCTION calculate_discount (\r\n p_customer_id NUMERIC,\r\n OUT p_discount NUMERIC\r\n) RETURNS NUMERIC AS $$\r\nDECLARE\r\n v_total NUMERIC := 0;\r\nBEGIN\r\n SELECT COALESCE(SUM(amount), 0) INTO v_total\r\n FROM orders WHERE customer_id = p_customer_id;\r\n\r\n p_discount := CASE\r\n WHEN v_total > 10000 THEN 0.15\r\n WHEN v_total > 5000 THEN 0.10\r\n ELSE 0.05\r\n END;\r\nEND;\r\n$$ LANGUAGE plpgsql;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f156b424690>)])]>

Along with the generated SQL, Gemini provides an inline explanation detailing why NVL was converted to COALESCE and how the Oracle DECODE function was converted into an explicit CASE block in PostgreSQL.

Figure 2: End-to-end database code conversion pipeline powered by Gemini in DMS.

Maintain full schema control and validation

Security, transparency, and code accuracy remain central to database modernization. Gemini in DMS operates strictly within your established Google Cloud security boundaries, keeping your code private to your project.

To ensure reliability, the conversion and validation process follows a structured workflow:

Automatic schema context pulling: When you set up a DMS conversion workspace, the service automatically parses your entire source database metadata—including table schemas, data types, foreign key constraints, and cross-procedure dependencies. Gemini references this project-wide context during code generation, eliminating the need to manually supply dependent object definitions.
Automated syntax and dependency validation: As code is generated, DMS runs a validation parser against target PostgreSQL syntax rules. Objects are assigned validation status indicators (e.g. Converted, Warning, or Action Required) to quickly highlight routines requiring manual review.
Interactive evaluation state: You maintain full control over every schema change. Within the conversion workspace, you can inspect side-by-side diffs, review inline AI explanations, and edit PL/pgSQL code directly before applying changes to your target database.
Staging deployment and verification: Once code passes workspace validation, you can apply the converted schema and functions to a target staging instance (e.g. Cloud SQL or AlloyDB) for functional execution and performance testing prior to production cutover.

Streamlining database modernization

AI-assisted code conversion in Database Migration Service helps database teams convert legacy database logic in days rather than months. Instead of spending precious time rewriting code from scratch, database administrators and application developers can shift their focus to adding new functionality, testing performance, and modernizing applications.

If you’d like some good examples of common Oracle and SQL Server conversion scenarios and how DMS converts them to PostgreSQL, check out our recent video series, Gemini taught me PostgreSQL.

Say goodbye to database migration headaches and let Gemini teach you how to seamlessly convert legacy database logic.

Get started with Database Migration Service

To start your database conversion, launch a migration assessment in the Database Migration Service console (https://console.cloud.google.com/dms) or read our heterogeneous migration guide (https://cloud.google.com/database-migration).

Introducing Database Operations Agents: The future of autonomous database management

Tue, 04 Aug 2026 16:00:00 +0000

As part of the Agentic Data Cloud launch at Google Cloud Next ‘26, we announced two AI-powered database agents to simplify database management. These include the Database Onboarding Agent for Day 0 operations — setup, configuration, and initial deployment — as well as the Database Observability Agent for Day 1 and 2 operations, including monitoring, troubleshooting, and ongoing maintenance.

These agents are always on, informed by Google’s years of experience, and integrated across Google surfaces such as Chat, CLI, the Google Cloud console, Managed Context Protocol (MCP) servers, and third-party tools — including your preferred integrated development environment (IDE), so you get help where and when you need it.

Traditionally, managing and creating databases has involved a combination of manual architecture planning, custom scripts, and distinct tools. Teams handle database provisioning, schema design, index configuration, and query tuning, alongside performance monitoring—often cycling through repeated testing and optimization cycles as application demands change. Although this method is functional, it demands substantial technical skill and continuous attention throughout the entire database lifecycle. For example, developers often fear making an update that may limit their ability to scale the system later. Similarly, when an application slows down, finding the exact query or resource constraint causing the issue can take hours of manual investigation and troubleshooting.

Intelligent AI-powered agents can simplify database lifecycle management by automating many of these tasks such as recommending the right database type for the workload, detecting anomalies, recommending the right configurations, optimizing queries, and providing actionable insights to improve operational efficiency. By embedding these capabilities directly into workflows where you need them, agents help organizations build, operate, and optimize databases more efficiently while reducing operational overhead.

Let’s take a closer look at these new database agents.

Database Observability Agent: From diagnosis to remediation

The Observability Agent empowers Site Reliability Engineers (SREs), DevOps pros, DBAs and developers to diagnose complex issues and remediate them using simple natural language prompts.

As your operations scale, identifying subtle issues like query hotspots or lock contention becomes an expensive burden. The database observability agent uses Google’s operational expertise and the reasoning capabilities of Gemini to solve these challenges. By automatically connecting telemetry across multiple sources including Database Insights, Cloud Monitoring, Cloud Logging, and Cloud Trace the agent provides a clear root cause analysis in minutes.

Beyond just identifying the "why," the agent suggests recommended actions to fix the issues found, and can execute validated actions with your approval. For example, if it detects a bottleneck, it might suggest you "Enable connection pooling for Cloud SQL instance," providing the rationale and expected impact before you commit to the change. Some capabilities include:

Fleet-level troubleshooting: The Observability Agent is integrated with Database Center so you can use Gemini Chat to ask complex fleet-wide questions like, "Which databases in my fleet consumed the most CPU in the last 7 days?" to receive a summarized analysis across your entire fleet.
In-product investigations: The agent correlates complex telemetry across Database telemetry, Cloud Monitoring, Cloud Logging, Cloud Trace, and multiple other data sources to pinpoint issues like latency spikes or lock contention. (In preview with select customers)
Validated remediations: Instead of just identifying problems, the agent provides crisp recommendations and can execute validated actions with your approval, such as adding indexes for a Cloud SQL instance. (In preview with select customers)
MCP tools: The Observability Agent derives insights with the help of tools such as system metrics, query metrics, fleet inventory, and issues, which are also available as MCP tools via the Database Insights MCP Server and Database Center MCP Server.

Integration that fits your workflow

You can access these Database Observability Agent capabilities directly within your existing database management processes. The agent powers several experiences, including:

Cloud Assist chat: Ask questions in natural language, for example, "What is the CPU utilization trend for my top Cloud SQL instances?" to get a summarized analysis complete with charts. Then, within the Chat window, you can start an investigation for any issues found,and get a root-cause analysis and remediations.

In-product investigations: Use Gemini Cloud Assist to investigate and remediate issues in-context on relevant database pages from the console.

Developer tools: Consume the agent’s capabilities through Antigravity or an IDE of your choice. This is augmented by the rich set of observability MCP tools that Google provides. All of these tools are available on Google Remote MCP servers. Combining them together is like giving developers a virtual DBA to optimize their databases, but all within their IDEs.

Supports multiple managed databases

You can use Observability Agent to get answers to your database queries, to access any database metric instantaneously, or to leverage AI-powered diagnosis to resolve complex problems. The agent covers a broad set of issues across a variety of Google Cloud databases, including:

Cloud SQL: Troubleshoot and optimize your database instance load, query performance or connectivity issues for all Cloud SQL database engines. For Cloud SQL for PostgreSQL, leverage the agent to troubleshoot common database issues.

Similarly, the agent helps you identify issues, find their root cause, and take remediation actions for other supported databases and issue types.

Spanner: Here, the most common troubleshooting scenario involves optimizing read and write latencies. The agent helps you do that in minutes, covering a broad set of scenarios ranging from hotspots to lock contentions.
AlloyDB: Troubleshoot and optimize your database instance load, query performance or replica lag issues.
Bigtable: Diagnose and optimize your read and write latencies, complete with crisp, actionable recommendations.

Database Onboarding Agent

The new Database Onboarding Agent is your active partner during the database selection process. Instead of spending hours reading documentation, you can describe your application requirements to the agent in natural language. The agent understands technical metrics like IOPS, latency limits, and replication lag, so it can provide a sound recommendation. You can access the Database Onboarding Agent’s capabilities directly within the Gemini chat interface. With the Database Onboarding Agent, you get:

Recommends database solutions: Analyzes user requirements regarding workload performance, scale, data type, and reliability to suggest optimal Google Cloud Managed Database services (e.g., Cloud SQL, Spanner, AlloyDB).
Smart recommendations: The agent reflects your requirements back to you, such as recommending AlloyDB for a high availability configuration, helping you have confidence in its selections.
Streamlined configuration: Once you choose a service, the agent generates the required commands. You can then use these commands to provision your database instance, configure the correct features, and deploy it.

Get started

The Database Observability and Onboarding Agent’s capabilities are available for a wide range of services, including AlloyDB, Bigtable, Cloud SQL (PostgreSQL, MySQL, SQL Server), Firestore, Memorystore, and Spanner. These agents are currently available via Gemini Cloud Assist. Explore AI assisted troubleshooting and Gemini Chat for AlloyDB, Cloud SQL, Spanner, and Visit Gemini Cloud Assist page to learn more.

Databases

What’s new with Google Data Cloud

October 12 - October 16

October 5 - October 9

September 7 - September 10

August 31 - September 4

July 6 - July 10

June 1 - June 5

May 11 - May 15

April 20 - April 24

April 13 - April 17

April 6 - April 10

March 23 - March 27

March 16 - March 20

February 23 - February 27

February 16 - February 20

February 2 - February 6

January 26 - January 30

January 19 - January 23

January 12 - January 16

What’s new with Google Data Cloud - 2025

Managed Apache Iceberg at scale: How Spanner powers Lakehouse runtime catalog

Challenges a managed catalog needs to solve in the Lakehouse

Lakehouse runtime catalog

What powers the Lakehouse runtime catalog?

Combine analytical and operational workloads

Modernize to the borderless Lakehouse

AlloyDB: A unified database engine for hybrid search

The problem of the multi-step workflow

Simplifying hybrid search architectures with AlloyDB

1. The single source of truth

2. Database-native orchestration

The result: Performance and operational simplicity

Enhancing FTS performance: The RUM extension

Introducing the RUM extension for low-latency FTS

Measurable gains and integration

Introducing BM25 Index: The modern ranking standard

External search: Extending versatility with foreign data wrappers

A unified architecture for AI-powered search

Watch it in action

Getting started

Announcing Spanner queues: Transactional messaging for agentic workloads and beyond

Key capabilities for agentic architectures

Why Spanner queues are essential for autonomous agents

Beyond agentic workflows

Under the hood: Transactional mechanics of Spanner queues

1. Defining a queue and enqueuing atomically

2. Temporal scheduling and atomic cancellation

3. Streaming consumption, lease renewal, and atomic acknowledgment

Spanner change streams vs. Spanner queues

Get started

How to implement long-term AI agent memory in AlloyDB and Memorystore for Valkey

Stateful workflows inherently conflict with LLM’s stateless nature

Implementing a 2-tier memory architecture

Understanding the four memory types

The tiered memory architectural blueprint

Business impact and ROI

Core AlloyDB AI technical advantages

Key implementation patterns

1. Setting up schema and auto-embeddings

2. Querying long-term memory with native hybrid search

3. In-database memory compaction with AI functions

4. Connecting the 2-tier memory architecture to an ADK agent

Enterprise governance and multi-tenant security

Summary and next steps

Spanner Omni, now GA: A distributed, multi-model database that you can deploy anywhere

Attio accelerates agentic application velocity with Spanner Omni

Built-in AI capabilities for any environment

What’s new with Spanner Omni

Enterprise features for production deployments

Flexible licensing and industry-standard pricing

Spanner Omni vs. fully managed Spanner on Google Cloud

Get started today

Unlock 3x QPS and microsecond latency with Memorystore for Valkey 9.1

Under the hood: Rethinking thread communication

New developer capabilities in Valkey 9.1

1. Granular database-level access control (ACLs)

2. CLUSTERSCAN: Efficient cluster-wide key scanning

Use case 1: Sequential full cluster scan (single worker)

Use case 2: Parallelized cluster scan (multiple workers)