Databases

A new, no-compromises database architecture for the agentic era

Thu, 24 Sep 2026 14:30:00 +0000

Entire database engineering careers have been spent on a single question: How do you scale an OLTP workload without compromising the system of record that owns the data?

Exadata answered the question by offloading queries into a scale-out storage tier beneath the database, removing the network as the bottleneck. Azure SQL Hyperscale did it with shared block servers, scaling out to tens of read replicas. Aurora offloaded log application to distributed storage nodes, scaling reads across tens of PostgreSQL nodes. Meanwhile, emerging architectures persist data in traditional object storage with a provisioned cache tier in front, recovering latency for hot data but leaving a high-latency tail on every cache miss.

Each of these architectures is inherently constrained by at least one of these three properties: scale, latency, and isolation — and sometimes even two. For instance, architectures built on shared block servers compromise scalability, because I/O inevitably bottlenecks on the block server. They also sacrifice isolation, as production workloads get throttled whenever replica traffic spikes.

Some of these trade-offs were actually sound at the time; they met the requirements of enterprise database workloads for four decades. However, in the agentic era, these compromises are no longer acceptable. Agentic workloads are generated dynamically and cannot be vetted in advance, making it a business-continuity imperative to isolate them from mission-critical systems. Agentic workloads also require low latency that is only possible with the full power of the database engine and all its indexes, as well as a a whole new level of elastic scale that has never been tried with a single database: a burst of agents that demand 1,000 compute nodes over a single database within seconds, and that may finish inside a minute.

Three tenets needed for a truly agentic database architecture

We believe the agentic era demands a new agentic database architecture defined by three fundamental tenets. An agentic database architecture must satisfy all three, or it isn’t really agentic.

Tenet: Isolation — isolation by design, but with real-time data access. Agents must read live production data with sub-second freshness over a data path that does not share database components with the primary cluster. Real-time means up-to-the-second, not a stale copy or branch. This is physical separation, not a quota — because shared allocations mean shared fate. The boundary extends straight through the storage layer, eliminating resource contention by design.
Tenet: Latency — sub-millisecond baseline I/O. Operational workloads demand sub-millisecond block I/O, and that bar does not drop for agents. While compute nodes leverage DRAM and local SSD for acceleration, cache misses that reach remote storage — whether application or agentic — must complete in under a millisecond. An architecture that degrades into an order-of-magnitude performance cliff is fundamentally unusable by agents.
Tenet: Scale — agent-scale compute and I/O. Agent scale is simultaneously instantaneous, volatile, and massive: Database compute nodes must spin up in seconds, scale to thousands, run for short bursts, and automatically spin down to zero when agents are done with them. No one has thus far ever dreamed of expecting a database to scale compute and I/O dynamically to thousands of nodes while leaving production untouched. Due to the dynamic nature of agents, pre-provisioning is a non-starter across the entire stack, whether it’s compute, storage I/O, or any caching tier in between.

Crucially, an agentic architecture must uphold all three tenets at once. And by doing so, the architecture allows agents to work directly against live operational data, i.e., enterprise truth, without compromising production stability. The outcome is transformative:

No correlated failures: Total decoupling between the engines running the business and the fleets of agents reasoning over it removes a path for agents to affect production.
No capacity guesswork: True elasticity that eliminates the friction of pre-provisioning for unforecastable agent scale.
No semantic compromises: Nothing is withheld from agents — they get access to the full power of relational SQL, hybrid search (vector, full-text, spatial), and indexes within every single reasoning step.

AlloyDB's agentic architecture

AlloyDB’s new agentic database architecture is the first system that satisfies all three tenets. We engineered this from the ground up across storage, network, compute and databases to deliver:

Isolation, avoiding shared fate by design: The transactional production cluster runs on dedicated, pre-provisioned infrastructure, completely isolated from agent workloads. Agents interface via the Model Context Protocol (MCP) to an independent, ephemeral pool of microVM-based AlloyDB nodes that read directly from dedicated Colossus storage segments, separate from those for production.
Predictable sub-millisecond storage I/O: Every storage read is served directly by Google’s Colossus storage system inheriting its baseline sub-millisecond latency, eliminating performance cliffs on cold cache misses.
True zero-to-thousands compute scaling: The agent pool scales rapidly from zero to thousands of nodes for bursty agentic activity, and scales back to zero the moment tasks complete.

Agents query production data with sub-second freshness, with the complete PostgreSQL engine — point lookups, index traversals, vector, full-text and spatial search, columnar scans, and federated queries across the lakehouse — at their disposal to power their reasoning loops.

Run agents against production data at any scale by joining the preview of AlloyDB PostgreSQL for agents. You can learn more about its full capabilities in the companion announcement blog.

Why existing architectures can’t satisfy all three tenets

Traditional and emerging operational databases attempt to scale using one of three architectural paradigms. When assessed against the demands of autonomous AI agents, each paradigm exhibits a fundamental structural compromise — none satisfies all three tenets simultaneously.

Independent replicas (shared-nothing storage)

Traditional relational architectures scale reads by streaming replication logs from a primary instance to dedicated replica databases, each with its own local or attached block storage. They meet Tenet: Isolation – replicas share no physical resources with the primary cluster, and continuous log replication maintains near-real-time currency. They meet Tenet: Latency – dedicated local storage guarantees predictable, sub-millisecond read latency. However, they fail Tenet: Scale – scaling requires provisioning a new replica and rehydrating hundreds of gigabytes or terabytes of storage. All this takes hours — an impossible mismatch for agent-reasoning bursts measured in seconds. Furthermore, statically provisioned compute and storage continue to incur idle costs long after the agent completes its run.

Disaggregated shared-storage servers

A second approach decouples stateless compute nodes from a shared, multi-tenant tier of custom storage servers that manage persistence, replication, and that may offload block writes. This approach meets the Tenet: Latency – reads hitting the optimized storage servers resolve with consistent, low operational latency. However, it fails the Tenet: Isolation – because every replica reads from the same servers as the primary, so agent I/O contends directly with production I/O, creating shared fate. It also fails the Tenet: Scale – stateless compute replicas spin up quickly because no data is copied, but total storage I/O bandwidth is fixed to the pre-provisioned storage tier. Adding compute nodes without scaling underlying I/O capacity simply accelerates storage saturation and throttling.

Object storage with shared-block servers

A third emerging approach keeps data durable in general-purpose object storage and serves block reads from a shared tier of block servers. Because a random read from object storage takes tens of milliseconds — an order of magnitude slower than traditional database storage, and slower than an enterprise disk array has been for at least 25 years — the block servers hold hot data in order to serve it at low latency. This approach meets the Tenet: Latency — with one caveat: A block server miss still falls through to object storage at unacceptably high latency. It fails the Tenet: Isolation — because replicas share the block servers with production: Agent I/O and production I/O draw on the same capacity, so when that capacity is exhausted or throttled, production is affected along with the agents. It also fails the Tenet: Scale, for the same reason as shared storage servers: Replicas start quickly, but the block servers do not scale their I/O with the burst.

Some architectures in this family also allow analytical engines like Apache Spark to read the underlying object storage directly, bypassing the database engine. For analytics workloads, that is a valuable and viable path. However, since agents need low-latency retrieval, stripping away indexes, point lookups, and vector search forces brute-force table scans, exploding latency, and therefore breaks the ability for agents to execute their retrieval-reasoning loops.

Evaluating existing architectures

We evaluated a commercially available service that uses the object storage architecture with shared block servers by running concurrent index lookups over a dataset larger than available DRAM, testing both scaling limits and production isolation. Starting with a single reader instance, we scaled the workload by adding up to eight read replicas.

In architectures with shared physical resources, scaling agents via read replicas quickly degrades both replica and primary performance. In our tests as seen in the chart below, adding replicas provided less than a 2x throughput increase, peaking at four replicas before dropping off as the shared block-server bandwidth saturated.

The impact on the primary database was immediate and severe: Primary throughput plummeted by more than 75% as replicas were added.

In short, neither traditional nor emerging architectures can meet the scale that agents demand, and certainly not without jeopardizing the stability of production systems.

Assessing against the Tenets

* Partially meets: Hot data is served at low latency from the block servers, but a block server miss falls through to object storage at tens of milliseconds.

In each case the gap is structural, not just a matter of tuning. Replication isolates by giving each replica its own storage, so it cannot add a replica faster than it can populate that storage. Shared storage servers add compute quickly by sharing storage, so they can neither isolate nor scale I/O. Block servers over object storage recover latency with a provisioned tier, so they can neither isolate nor burst, and every miss still reaches object storage. Each approach solves the problem at one layer and pays for it at another. Meeting all three tenets at once requires rethinking the database architecture across compute, network and storage together.

How we engineered AlloyDB across the stack

AlloyDB's agentic database architecture is vertically integrated across Google's data, AI and infrastructure stack: AI models, the database engine and analytical engines, but also storage, networking and compute infrastructure.

Storage: Colossus as the foundation

At the persistence layer, AlloyDB builds on Colossus, Google's exabyte-scale distributed storage system that underpins Google Search, YouTube, Gmail, Google Drive, Spanner, and Bigtable. A single Colossus cluster scales to exabytes of storage and tens of thousands of machines. With Spanner, we demonstrated that a transactional database engineered directly on Colossus can scale to thousands of nodes. The new AlloyDB architecture applies the same foundation to a new problem: agents.

Colossus has three properties that enable AlloyDB to satisfy the three tenets.

Direct, sub-millisecond I/O: Colossus is engineered to minimize read latency. A database node opening a Colossus stream receives a handle that describes where data physically resides. Authorization and metadata resolution happen once, when the stream is created; every subsequent read goes directly to the disks holding the data, over an optimized network protocol. The result is sub-millisecond latency across all of the database's data, with no intermediary to warm and no tier to miss.
Massive throughput: Colossus delivers up to 15 TB/s of aggregate throughput and 20 million queries per second to a single AlloyDB database without needing to provision bandwidth and with an unlimited number of concurrent hosts. At Colossus scale, a fleet of AlloyDB agent nodes is not a load the storage must be sized for; it is a fraction of the load the storage already serves!
Physical segment partitioning: AlloyDB serves agents from a separate set of Colossus segments, so agent I/O is deliberately spread away from the production data path rather than contending with it.

At no point along the data path — compute, network or storage — can an agent ever share a database component with production.

Network: Scalable bandwidth with Jupiter

Compute and storage are bound together by Jupiter, Google's high-capacity data center network. A single Jupiter fabric connects more than 100,000 servers with 13 petabits per second of bisection bandwidth — enough to carry a video call for every person on Earth.

Because Jupiter provides high bisection bandwidth with predictable low latency across the networking fabric, agent nodes can be scheduled flexibly anywhere in the cluster with consistent access to centralized storage. As the agent pool scales from zero to thousands, the underlying interconnect capacity absorbs the expanding traffic without creating placement bottlenecks

Compute: Elastic and serverless PostgreSQL and analytics

At the compute layer, agents connect to AlloyDB's agent pool through MCP. The agent pool consists of AlloyDB agent nodes with read-only access to the up-to-second state of the database. This layer provides:

MicroVM isolation: Each agent node is a fully functional AlloyDB for PostgreSQL database engine running inside a lightweight, secure microVM. These instances are fully isolated from each other and from the dedicated primary cluster.
Rapid spin-up and scaling: Agent nodes are provisioned in response to requests from agents and stop automatically when the agents finish. In response to a burst, AlloyDB rapidly provisions thousands of agent nodes, serving millions of concurrent agents, and releases them as the agents finish. Because billing is per second of agent-node activity, a burst that uses a thousand nodes for tens of seconds will only be charged for the resources that the job consumed, and nothing more.

Meanwhile, the production cluster remains as it is today: pre-provisioned, on dedicated infrastructure, sized for the system of record. Agent nodes read from Colossus directly and see a consistent production state with sub-second freshness.

Beyond the agent pool, BigQuery and Spark can read AlloyDB data from Colossus with the same isolation from the production cluster, so agents can use lakehouse federation to join real-time operational data with large-scale lakehouse datasets.

By building on these Google-scale storage, network, and compute layers, AlloyDB’s new agentic database architecture achieves a remarkable goal: Share the data. Share nothing else.

Evaluating AlloyDB’s agentic database architecture

We tested AlloyDB by running concurrent index lookups over a dataset larger than available DRAM, testing scalability across the full stack. We ran the agentic workload starting with a single agent node — an independent database instance in the agent pool, rather than a traditional read replica — and scaled dynamically to thousands of nodes over a single database, measuring both the aggregate agentic throughput as well as any impact on production.

In this test, throughput scaled linearly from 3.9K to 41K QPS when expanding from one to 10 agent nodes. Scaling by two additional orders of magnitude yielded near-linear performance up to 1,000 nodes. We observed:

Zero primary degradation: Scaling from 1 to 1,000 agent nodes produced no measurable impact on primary cluster performance.
Massive throughput: Aggregate throughput dynamically scaled 773x to 3 million QPS, driving over 8 million IOPS in Colossus across 1,000 compute nodes.

In a similar benchmark running concurrent full table scans across 2,100 agent nodes, aggregate scan throughput exceeded 1 terabit per second.

Because the agent pool shares no physical infrastructure with the production cluster, teams can scale reasoning fleets to thousands of nodes without placing production systems at risk.

Engineering all three tenets by design

The table below shows how AlloyDB’s architecture satisfies each of the three tenets:

Every agentic database architecture will require these three foundational elements: storage with the properties of Colossus, a network that connects compute to that storage without constraint, and compute that can be provisioned and released at agent scale.

Google has spent more than two decades building exactly that, to run Google Search, YouTube, and Gmail. Now it underpins our agentic database architecture.

Give agents live data without impacting production

Every organization building with AI faces the same core dilemma: how to give agents full access to live operational data without putting the systems running the business at risk. Until now, architecture — not application needs — dictated that choice. Giving agents direct access to the database meant exposing mission-critical systems to unforecastable load, severe resource contention, and production outages.

An architecture built on these three tenets removes these compromises entirely. Agents reason over live production data withsub-second freshness. They have the complete engine at their disposal — every index, vector, full-text and spatial search, and the full capability of SQL — at sub-millisecond I/O. The architecture scales dynamically to thousands of isolated nodes when agents need it, then to zero when agents finish. Throughout, core transactional workloads remain untouched: no shared components, no shared quota, no correlated failures. Agents can deliver innovation without conflicting with business continuity.

The same property extends to every other reader of production data. Reporting, analytics and applications can freely read live data without putting production at risk, ending a constraint that has shaped operational databases for five decades.

The data in an enterprise's systems of record is its crown jewels. Built on this foundation, that data can finally be put to work in full.

Databases, unfettered.

To learn more, visit the documentation page, and sign up here to get started.

AlloyDB delivers PostgreSQL for agents: Real-time data at agent scale, with full workload isolation

Thu, 24 Sep 2026 14:30:00 +0000

Enterprises rely on mission-critical operational databases where performance slowdowns simply aren’t an option. Yet when even a few agents execute dense reasoning loops, the unpredictable surge in queries can easily overwhelm traditional architectures.

Today, we’re announcing that AlloyDB delivers PostgreSQL for agents (in preview), enabling real-time data access without compromising your mission-critical systems. AlloyDB now scales to dynamic agent bursts by provisioning sandboxed database instances in seconds, enabling full workload isolation. You can cost-effectively run your agents at any scale — from a few agents to millions of agents — and the instances automatically spin down when agents finish. With this announcement:

AlloyDB now features an agentic database architecture engineered to scale PostgreSQL to thousands of serverless database instances that have up-to-the-second read-only access to production. These instances remain fully separated from the primary, standby, and read replica instances where production workloads run.
Each of these instances access real-time data in the database backed by a unified storage layer in Colossus, Google’s exabyte-scale distributed storage system. This helps agents achieve sub-millisecond I/O, and terabit-per-second aggregated scan throughput, supporting over 3 million queries per second.
These instances utilize the full AlloyDB PostgreSQL engine, providing access to every index, the full capability of SQL, and comprehensive vector, full-text, and spatial search. Agents can also leverage BigQuery and Spark to run lakehouse analytics without requiring complex ETL pipelines.
When agents complete their tasks, these instances scale right back to zero, thus reducing your cloud spend by billing only for active reasoning loops.

With AlloyDB for PostgreSQL, we pioneered the agentic enterprise relational database by integrating advanced vector operations, machine learning inference, and foundation model integrations directly within a 100% PostgreSQL-compatible engine. It protects your data through deep Google Cloud security integrations — replacing static passwords with IAM authentication, isolating traffic via VPC Service Controls, and providing customer-managed encryption and auditing. This functionality, combined with the scalability now provided by our agentic architecture for agents, makes AlloyDB the premier enterprise-grade agentic PostgreSQL offering.

Why this matters

In today’s agentic era, we’re swiftly moving from single copilot agent interactions to networks of millions of agents collaborating simultaneously. When these agents query and operate all at once, sudden traffic spikes can overwhelm your core databases, competing with the systems that run your business.

Agents require both fast analytics and low-latency access to real-time production data, utilizing B-tree, vector, text, and spatial indexes to efficiently execute their workflows. Emerging architectures rely on page-caching layers that sit above object storage, but they suffer from scaling and cost challenges that can compromise the stability of production systems.

In addition, they face a challenging trade-off: To unlock production data for analytics, they create performance bottlenecks for operational access, putting mission-critical databases at risk the moment agents are unleashed in production. These approaches attempt to solve the problem using traditional object stores for database storage. While this enables analytical access that can help some agents, the underlying databases are too slow for production workloads, suffering from up to an order of magnitude higher I/O latency. Page caching layers are at best a patch; the caches themselves are often still not fast enough, and they pose a scalability bottleneck that is easily saturated by agentic workloads.

When active multi-agent systems execute dense reasoning cycles, they trigger highly concurrent, unpredictable bursts of queries that overwhelm these caching layers, leaving mission-critical production systems vulnerable to agent-induced outages. Consequently, an entire class of operational use cases are precluded from running on these architectures, locking businesses out of the transformative power of AI on live enterprise data.

AlloyDB’s unique agentic PostgreSQL architecture

We are taking a different approach. AlloyDB delivers an agentic database architecture purpose-built for the AI era, with four key differentiated capabilities:

Sub-millisecond I/O latency, without artificial choke points: Combining AlloyDB’s industry-leading transaction and query processing with low-latency object storage backed by Google’s planet-scale Colossus storage infrastructure, this architecture provides a large-scale, shared storage plane for agents. It achieves sub-millisecond I/O and over a terabit-per-second of aggregate scan bandwidth, allowing agents to execute intensive read queries and vector searches directly against fresh operational data.
Fully isolated from production workloads while scaling to meet demand: AlloyDB scales by dynamically provisioning sandboxed database instances in seconds against fresh production data. Unlike traditional architectures where agents compete for operational resources, agentic database compute remains completely isolated from the primary database clusters — allowing agents to execute dense, unpredictable reasoning loops without degrading performance in production. These robust safety guardrails, coupled with enterprise-grade governance and fine-grained access control, allow you to confidently unleash the full, unconstrained power of PostgreSQL on your production data — seamlessly mixing analytical queries, vector queries, and operational point lookups in active agentic loops.
Pay-as-you-go billing: Most agent activity is characterized by sharp spikes of concurrent queries followed by periods of inactivity. Provisioning dedicated read replicas to absorb these bursts forces you to maintain expensive infrastructure around the clock. To support massive groups of agents cost-effectively, and eliminate the idle compute overhead of provisioned systems, these agentic AlloyDB instances can rapidly scale to handle millions of queries per second, and automatically scale to zero with a flexible, pay-as-you-go pricing model.
Native lakehouse integration, without ETL: All production data is natively integrated with Google Cloud’s borderless Lakehouse. This allows agents to run federated queries across BigQuery and Lightning Engine for Apache Spark, joining massive lakehouse datasets with up-to-the-second transactional data in AlloyDB. This eliminates the need to build and maintain fragile batch ETL pipelines, giving autonomous agents instant access to both live operational state and historical lakehouse context.

“As supply chains become increasingly autonomous, our platform relies on real-time transactional intelligence to coordinate complex logistics workflows across thousands of facilities. AlloyDB's new PostgreSQL architecture for agents has been a game changer for us. We can now deploy networks of agents collaborating simultaneously to help us analyze inventory and order data with sub-second freshness, while ensuring our core transactional processing remains entirely untouched. It delivers the isolation, speed, and cost efficiency we need to power the next generation of enterprise supply chain AI.” - Sanjeev Siotia, Executive Vice President & Chief Technology Officer, Manhattan Associates

Availability

PostgreSQL for agents in AlloyDB is now available in preview.

To learn more, visit the documentation page, and sign up here to get started.

Announcing Native BM25 Ranking in AlloyDB and Cloud SQL

Fri, 18 Sep 2026 16:30:00 +0000

Vector search is a critical component of generative AI, retrieval-augmented generation (RAG), and data agent architectures, but sometimes vector search alone isn't enough. While vector embeddings are incredible at understanding conceptual meaning, they stumble on specific alphanumeric IDs and exact product SKU numbers. To build truly robust search and AI applications, you may need the combination of semantic vector search and traditional exact keyword full-text search — what we call hybrid search.

In search, Best Matching 25, or BM25, is a key algorithm used to estimate how relevant a document is to a given query. Until today, if you wanted BM25 ranking with AlloyDB or Cloud SQL, you needed to add an additional full-text search backend. This introduced data silos, sync lags, and operational complexity. Today, we are eliminating the friction of maintaining a separate full-text search backend altogether, with the preview of the native BM25 index in AlloyDB and Cloud SQL for PostgreSQL 17+, made possible through the open-source pg_textsearch extension created by Tiger Data.

Now, with a unified hybrid search backend, you no longer need to provision, manage, or pay for separate systems to get state-of-the-art full-text retrieval. It all happens directly inside your database, where your operational data lives, delivering:

Industry-standard keyword ranking: Powered by Tiger Data's pg_textsearch, bring lightning-fast, C-optimized BM25 scoring directly to your Postgres tables.
No complexity, total consistency: Eliminate the data duplication, ETL pipelines, and synchronization lag that you get when you maintain multiple backends for vector and full-text retrieval.
Supercharged semantic search (AlloyDB exclusive): Get up to 6x and 10x faster vector search queries (when compared to standard PostgreSQL) with ScaNN and HNSW index types.

Why pg_textsearch?

If you’ve used PostgreSQL's built-in ts_rank for full-text search at any meaningful scale, you already know its limitations. Ranking quality degrades as your corpus grows. There’s no support for inverse document frequency, so common words carry the same weight as rare ones. There’s no term-frequency saturation, so a document that mentions "database" 50 times outranks one that mentions it once.

BM25 is the information retrieval gold standard, providing inverse document frequency (rarer terms matter more), term frequency saturation (repetition doesn't dominate), and document length normalization. You can learn more in this blog post by Tiger Data about how they built a BM25 search engine on PostgreSQL pages.

Full-text search example

Here’s how to get started with BM25 full-text search on both AlloyDB and Cloud SQL. Consider a sample table, cymbal_products, that contains the unique identifier uniq_id, a product_name column, a product_description column containing a text description of each product, and a generated product_embedding column. cymbal_products contains information on various retail products, including indoor and outdoor plants.

Index creation

To use BM25, enable the pg_textsearch extension.

code_block: <ListValue: [StructValue([('code', '-- Install pg_textsearch extension\r\nCREATE EXTENSION pg_textsearch;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f49c4b8d590>)])]>

Create the index on the product_description column from the cymbal_products table.

code_block: <ListValue: [StructValue([('code', "-- Create the native BM25 index on the content column\r\nCREATE INDEX idx_docs_bm25 \r\nON cymbal_products \r\nUSING bm25 (product_description) \r\nWITH (text_config='english');"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f49c4b8f750>)])]>

A BM25 full-text search query can be executed using the <@> special operator. In the snippet below, we search for ‘cherry tree’.

code_block: <ListValue: [StructValue([('code', "-- Full text search query\r\nSELECT product_name, product_description <@> 'cherry tree' AS bm25_score \r\nFROM cymbal_products\r\nORDER BY bm25_score \r\nLIMIT 5;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f49c55dc0d0>)])]>

Sample output is shown below. A more negative score indicates a stronger relevance match.

AlloyDB hybrid search example

Setting up a hybrid search system in AlloyDB is simple. You can create both your vector and keyword indexes on the same table and merge the results seamlessly using the hybrid search user-defined function (UDF).

Vector index creation

Here is how to create a ScaNN vector search index:

code_block: <ListValue: [StructValue([('code', '-- Install vector extension\r\nCREATE EXTENSION vector;\r\n\r\n-- Install scann extension\r\nCREATE EXTENSION IF NOT EXISTS alloydb_scann;\r\n\r\n-- Create scann vector search index \r\nCREATE INDEX cymbal_products_embeddings_scann ON cymbal_products USING scann(product_embedding cosine);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f49c55ddb10>)])]>

Hybrid search

AlloyDB provides an out-of-the-box hybrid search UDF that makes it very simple to run hybrid search queries. The UDF merges the ranked results from each search component into a single, unified list using the Reciprocal Rank Fusion (RRF) algorithm. This query utilizes the UDF to perform a vector search for ‘trees that grow taller than houses’ and a keyword search for ‘California’ in the product description.

code_block: <ListValue: [StructValue([('code', 'CREATE EXTENSION google_ml_integration;\r\n\r\nSELECT *\r\nFROM ai.hybrid_search(\r\n search_inputs => ARRAY[\r\n \'{\r\n "data_type": "vector",\r\n "weight": 0.5,\r\n "table_name": "cymbal_products",\r\n "key_column": "uniq_id",\r\n "vec_column": "product_embedding",\r\n "distance_operator": "public.<=>",\r\n "limit": 10,\r\n "query_vector": "ai.embedding(\'\'text-embedding-005\'\', \'\'trees that grow taller than houses\'\')::vector"\r\n }\'::JSONB,\r\n \'{\r\n "data_type": "text",\r\n "weight": 0.5,\r\n "table_name": "cymbal_products",\r\n "key_column": "uniq_id",\r\n "text_column": "product_description",\r\n "limit": 10,\r\n "ranking_function": "<@>",\r\n "query_text_input": "California"\r\n }\'::JSONB\r\n ],\r\n);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f49c55dc5d0>)])]>

As shown in the sample output below, results are ranked in descending order of their RRF scores.

Here, hybrid search bridges the gap between semantic intuition and exact keyword matching. While vector embeddings excel at grasping conceptual queries, like "trees that grow taller than houses", traditional full-text search provides the pinpoint precision needed for strict identifiers like "California." By fusing the two, AlloyDB helps ensure your application prioritizes highly specific, locally relevant results like ‘California Sycamore’ right at the top of the list.

Cloud SQL hybrid search example

In Cloud SQL, you can create both your vector and keyword indexes on the same table and merge the results seamlessly using Common Table Expressions (CTEs) and coalescing the RRF score, as shown below.

Vector index creation

Here is how to create an HNSW index in Cloud SQL.

code_block: <ListValue: [StructValue([('code', '-- Install vector extension\r\nCREATE EXTENSION vector;\r\n\r\n-- Create an HNSW index on the embedding column for fast approximate nearest neighbor search\r\nCREATE INDEX product_hnsw_idx ON cymbal_products USING hnsw(product_embedding vector_cosine_ops);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f49c55de090>)])]>

Hybrid search

Here is the hybrid search query.

code_block: <ListValue: [StructValue([('code', "CREATE EXTENSION google_ml_integration;\r\n\r\n-- BM25 keyword results\r\nWITH keyword_results AS (\r\n SELECT uniq_id, product_name, \r\n ROW_NUMBER() OVER (ORDER BY product_description <@> 'California') AS rank_kw\r\n FROM cymbal_products\r\n ORDER BY product_description <@> 'California'\r\n LIMIT 10\r\n),\r\n-- Semantic vector results\r\nsemantic_results AS (\r\n SELECT uniq_id, product_name, \r\n ROW_NUMBER() OVER (ORDER BY product_embedding <=> google_ml.embedding('text-embedding-005', 'trees that grow taller than houses')::vector) AS rank_vec\r\n FROM cymbal_products\r\n ORDER BY product_embedding <=> google_ml.embedding('text-embedding-005', 'trees that grow taller than houses')::vector\r\n LIMIT 10\r\n)\r\n-- Reciprocal Rank Fusion (RRF) to merge and score both lists\r\nSELECT COALESCE(k.uniq_id, s.uniq_id) AS uniq_id,\r\n COALESCE(k.product_name, s.product_name) AS product_name,\r\n COALESCE(1.0 / (60 + k.rank_kw), 0) + COALESCE(1.0 / (60 + s.rank_vec), 0) AS rrf_score\r\nFROM keyword_results k\r\nFULL OUTER JOIN semantic_results s ON k.uniq_id = s.uniq_id\r\nORDER BY rrf_score DESC\r\nLIMIT 5;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f49c55dc590>)])]>

The resulting output is identical to the AlloyDB hybrid search results shown above.

Watch it in action

Watch how this all comes together in this demo video.

Relevant resources

We are incredibly excited to work with Tiger Data and cannot wait to see how you leverage native BM25 support to build faster, smarter, and simpler AI applications. Turn on the pg_textsearch extension today, and experience the ultimate hybrid search engine experience with AlloyDB and Cloud SQL.

Want to get started? Check out”

AlloyDB resources

New to AlloyDB? Discover AlloyDB with a 30-day free trial
Choose a vector index in AlloyDB AI
AlloyDB BM25 documentation
AlloyDB hybrid search UDF documentation

Cloud SQL resources

Cloud SQL BM25 documentation

Tiger Data pg_textsearch Release Page

How a solo founder runs a five-continent tender platform on AlloyDB and MCP

Thu, 17 Sep 2026 16:00:00 +0000

Editor's note: Lucius AI, a tender-intelligence startup covering markets across five continents, runs its entire data platform on AlloyDB for PostgreSQL with a single operator. By migrating semantic search to a ScaNN index and managing database operations through Model Context Protocol (MCP), query latency dropped by 47x while automating day-to-day administrative tasks via MCP.

Executive summary

Lucius AI runs a global tender platform spanning more than 210,000 tenders across the UK, EU, India, and Australia, requiring minimal operational overhead for a solo founder.
Lucius AI deployed AlloyDB for PostgreSQL to consolidate its relational catalog, audit logs, and vector embeddings into a single managed database engine.
Migrating semantic search to a ScaNN index lowered query latency from 1.14 seconds to 24 milliseconds — a 47x speedup on a representative production query.
Connecting an AI agent to AlloyDB using the Model Context Protocol (MCP) helps Lucius AI automate query analysis, data freshness checks, and incident forensics under strict least-privilege permissions.

Making tender intelligence work as a company of one

Lucius AI helps businesses bidding on public contracts evaluate opportunities across global markets. The platform ingests public procurement notices from the UK, the EU, the US and Canada, Australia and New Zealand, India and Singapore, alongside World Bank donor-funded notices across Africa and Asia. Lucius AI analyzes tender documents using Gemini to generate compliance matrices, bid recommendations, and draft responses citing original source pages. For small and mid-sized suppliers, this replaces days of manual document reviews and costly external consulting.

Running a platform of this scope requires extensive operational coordination:

Nightly ingestion from thirteen public procurement sources
A catalog of more than 210,000 tenders, including tens of thousands open for active bidding
Two production regions on Cloud Run: Europe, and an Australian deployment on its own AlloyDB cluster with customer-managed encryption keys (CMEK) for defense-adjacent customers
Ongoing analytics, performance tuning, data validation, and incident response

Managing these responsibilities without dedicated data engineering or database administration teams requires offloading operational maintenance. Lucius AI addressed this challenge on two fronts: using AlloyDB for PostgreSQL as the core system of record, and connecting an AI agent through the Model Context Protocol (MCP) to safely execute database operations.

Consolidating systems into AlloyDB

Rather than deploying separate relational databases, vector databases, and log stores, Lucius AI houses all core data in AlloyDB for PostgreSQL. The relational tender catalog, document metadata, audit logs, and vector embeddings reside in the same database engine. Storing vector embeddings alongside relational rows avoids managing separate vector stores, establishes a unified backup schedule, and centralizes identity management.

Authentication relies strictly on Cloud IAM. Services connect using dedicated Google Cloud service accounts mapped to database roles scoped to specific access requirements, without storing database passwords in application environments. Database reliability is managed natively by AlloyDB through automated backups and point-in-time recovery, avoiding custom disaster recovery procedures.

In production, this consolidated architecture supports:

More than 210,000 tenders in the catalog, with embeddings stored directly alongside them
Rebuilding the semantic index embedded 115,820 records in 10.6 minutes with the Gemini embedding model, for around three dollars in API spend; AlloyDB auto embeddings now keep those vectors current.
Retrieval reranking executed directly inside the database using the ai.rank function — with mean latency of 77-milliseconds - returning the most relevant results for search queries without requiring a standalone reranking microservice

Accelerating semantic search by 47x

Semantic search across the tender catalog initially relied on unindexed vector comparisons, where a representative query took 1.14 seconds. Migrating this workload to a ScaNN index in AlloyDB reduced query latency to 24 milliseconds — a 47x improvement.

The index recommendation originated from the AI agent during an automated performance audit, where it benchmarked the query plan before preparing the index migration.

Automating database operations with MCP

To delegate routine administrative tasks, Lucius AI configured the open-source MCP Toolbox for Databases using the prebuilt alloydb-postgres server.

Operational delegation requires strict access controls. The agent connects using a dedicated PostgreSQL role granted SELECT across the schema and UPDATE on a single operational table. Destructive commands (DROP, DELETE, TRUNCATE) are omitted, restricting agent actions to authorized operational boundaries.

Under this configuration, the AI agent performs regular database operations across four key areas:

On-demand analytics: Compiles retention cohorts, activation funnels, and catalog coverage by country via ad hoc SQL queries, removing the need to build and maintain manual dashboards or complex analytical pipelines.
Performance optimization: Performs query-plan inspections and index analysis, such as identifying the ScaNN indexing strategy.
Incident forensics: In response to an external security probe, the agent parsed audit logs to reconstruct the request timeline in minutes, verifying that tenant isolation remained intact.
Automated data-quality checks: Evaluates ingestion watermarks and freshness across all thirteen procurement sources every morning.

For teams adopting this architecture, establishing a progressive permission structure provides clear guardrails: start with read-only access, expand permissions as requirements dictate, and keep destructive operations restricted to human administrators.

Looking ahead

Lucius AI is planning three technical initiatives to further reduce operational overhead:

Automated vector embeddings in AlloyDB AI: After validating ai.initialize_embeddings across the full catalog, a weekly maintenance job uses ai.refresh_embeddings to update vectors.
Columnar engine acceleration: Having enabled AlloyDB’s columnar engine with auto-columnarization, the database identified and stored 40 frequently queried columns across four tables in memory within a day, accelerating reporting queries without a separate analytical store.
Managed Remote MCP Server: Transitioning from self-hosted Toolbox processes to Google Cloud's fully managed Remote MCP Server for AlloyDB will offload MCP server hosting and maintenance.

By anchoring core data in AlloyDB and managing routine operations through MCP, Lucius AI demonstrates how a single engineer can build and operate a resilient, multi-region procurement platform.

To explore Lucius AI, visit ailucius.com. To evaluate AlloyDB for PostgreSQL, deploy an AlloyDB cluster to test performance against your own workloads.

M4N VM family, now GA: Highest per-core IOPS and throughput for I/O and memory-bound workloads

Wed, 16 Sep 2026 16:00:00 +0000

As enterprise organizations scale mission-critical applications, storage I/O and memory access can become severe operational bottlenecks. Whether its Oracle databases, in-memory databases like SAP HANA, or high-throughput SQL Server clusters, EHR systems, and real-time big data analytics, memory-bound databases often force enterprises to over-provision compute cores (vCPUs) to get the RAM capacity and storage bandwidth they need, driving up costly third-party software licensing fees.

Today, we are thrilled to announce the general availability of the M4N machine series in Google Compute Engine, purpose-built for I/O intensive, high-memory workloads, the second offering in our network- and block-storage optimized VM family. Compared to similar offerings from other hyperscalers M4N provides the highest per-core IOPS and throughput for high-memory instances, and over 20% TCO reduction for Oracle databases.

M4N is also the industry’s first instance of network and block storage optimized with higher memory ratios (up to 26:1) and size (6TB). Powered by 5th Gen Intel® Xeon® Scalable processors and built on Google Cloud's custom Titanium offload architecture, M4N instances deliver up to 25,000 MiB/s (25 GiB/s) of aggregate host storage performance and up to 1 million IOPS when paired with Hyperdisk Extreme — doubling the block storage performance of current M4 instances.

M4N targets workloads that demand both extreme high-density RAM and uncompromising I/O performance, complementing our existing memory-optimized families (such as M1, M2, M3, M4, and X4) by solving specific storage and network bottlenecks for high-throughput enterprise applications.

Built for demanding workloads

Workload Category	Typical Applications	Why M4N Wins
Mission-critical enterprise DBs	Oracle, SAP HANA, SQL Server, IBM DB2, MySQL, PostgreSQL	Memory-to-core ratios (up to 26.57 GB/vCPU) paired with 25 GiB/s storage for rapid data ingestion, transaction logging, and zero-stall backup cycles.
Generative AI and RAG data layers	Milvus, Pinecone, Qdrant, Vespa, Redis, In-Memory Context Caching	Sub-millisecond similarity search across massive vector indexes in RAM, combined with 400 Gbps network bandwidth for distributed model retrieval.
Enterprise healthcare and ERP	Epic Systems (Operational Database), SAP ECC, SAP S/4HANA	Sustained I/O headroom that prevents query latency spikes during peak clinical/transactional hours.
Real-time analytics and EDA	Electronic Design Automation, Genomic Modeling, In-Memory OLAP	High memory capacity to load massive datasets entirely in RAM with maximum storage bandwidth for checkpoint dumps.

Optimizing Oracle licensing costs

Enterprise IT departments struggle with the rising cost of core-based software licensing. For workloads like Oracle database, licensing fees are typically calculated based on the number of vCPUs or physical cores assigned to the instance. Historically, this has forced a difficult trade-off: paying for more compute cores than necessary just to obtain the required amount of RAM and storage performance.

M4N changes this paradigm with its industry-leading high memory-to-vCPU ratio. By providing the highest per-core IOPS and throughput for high-memory instances of all the leading hyperscalers, M4N allows database administrators to:

Reduce TCO and licensing overhead: Stop over-provisioning of cores while meeting Oracle database performance density requirements, resulting in over 20% TCO reduction compared to similar offerings from leading hyperscalers.
Right-size infrastructure: Allocate the exact amount of compute power needed for the workload while still accessing massive memory pools.
Improve cache-hit ratios: With more memory available per core, larger portions of the database can reside in the system global area (SGA), reducing expensive I/O operations and further boosting efficiency.

What customers are saying

Early experiences with M4N show that workload-optimized infrastructure is the engine for transformation.

“Before M4N, meeting our demanding I/O requirements on Google Cloud often required over-provisioning our compute to achieve the necessary performance density. The new M4N instances solve this by delivering high throughput across the smaller to larger shapes.” - Sherri Trojan, Sr Principal Solution Architect, Sabre

"We are delighted to see Google Cloud introduce this next-generation high-performance infrastructure for mission-critical database workloads. The new compute platform demonstrates tremendous potential for enterprise Oracle deployments requiring scalability, resiliency, and performance. We are excited about what this innovation means for customers running Oracle workloads on Google Cloud.” - Bala Kuchibhotla, Co-Founder and CEO, Tessell

"With M4N, Google Cloud continues to push the boundaries of platform co-design. By combining 5th Gen Intel Xeon Scalable processors with Google's custom Titanium offload architecture, M4N delivers the extreme memory capacity, high memory bandwidth, and uncompromising I/O throughput required for the world’s most demanding mission-critical data environments." - Intel

What’s new: Scaling extreme data layers with M4N

M4N bridges two previously separate paradigms in cloud infrastructure: large memory footprints and extreme I/O performance. Engineered with custom Titanium offloads, M4N minimizes I/O bottlenecks without requiring infrastructure add-ons or compromises on memory density. Let’s take a look at how M4N fits into these environments.

1. Enabling high bandwidth data transfer

For workloads with large memory footprints, M4N provides:

Superior VM-to-VM bandwidth: Delivers up to 400 Gbps aggregate VM-to-VM network bandwidth and up to 50 Gbps single-flow bandwidth within the same VPC, unlocking non-blocking data exchange for distributed database clusters and real-time streaming data layers.
Enhanced internet and egress throughput: Enjoy up to 200 Gbps internet egress bandwidth and up to 48 MPPS packet processing performance.
High bandwidth out-of-the-box: Achieve full performance without needing to purchase or configure premium Tier_1 networking add-ons.

2. Dynamic storage performance with Hyperdisk

Paired with Google Cloud's next-generation storage portfolio, M4N with Hyperdisk lets you independently tune IOPS, throughput, and capacity:

Hyperdisk Extreme (HdX): Delivers up to 25 GiB/s aggregate block storage throughput and 1,000,000 IOPS—double the storage performance of standard M4. This is great for rapid database recovery, transactional checkpointing, and instant in-memory index reloads.
Hyperdisk Balanced (HdB): Scales up to 20 GiB/s throughput and 640,000 IOPS for cost-effective enterprise storage at scale.

M4N machine types and specifications

M4N instances are offered across three distinct memory-to-vCPU ratio tiers, scaling from 16 to 224 vCPUs and up to 5,952 GB of DDR5 RAM. M4N also offers predefined VM shapes across three distinct memory-to-vCPU ratios to match specific workload requirements, with support for Resource-based Committed Use Discounts (CUDs). Details here.

Get started today

The M4N instances are now available in select regions around the globe. To learn more about how the M4N family can enhance your memory- and I/O-bound applications and reduce your licensing costs, contact your account representative or explore the documentation.

Scaling Telco Autonomy: Leveraging GNNs with Distributed GraphFlow

Tue, 15 Sep 2026 16:00:00 +0000

The telecommunications industry is currently undergoing a paradigm shift, moving from traditional manual human-driven operations to fully Autonomous Network Operations. Modern networks have grown increasingly complex, heterogeneous, and large-scale, making handcrafted rules-based methods and traditional Machine Learning (ML) approaches alone insufficient to automate network operations. While ML methods can identify subtle patterns and make fine predictions from large amounts of structured data, they lack the ability to understand, reason about the data and the system it represents, and ultimately make the kind of decision a human operator would.

The growth of AI agents and their ability to reason is a promising solution to this shortcoming. However, in the same way a human operator is not capable of directly ingesting the statistical information spread across the billions of data points created in a large network, AI agents also lack the ability to operate at this scale. To address this challenge, telecommunications companies are adopting Graph Neural Networks (GNNs), a modern form of machine learning designed to operate natively on massive volumes of temporal and relational data. By integrating GNNs with AI agents, operators can combine advanced diagnostics such as root cause analysis, capacity planning, traffic forecasting, what-if simulations, and real-time anomaly detection with the reasoning power required to interpret these insights and execute justified actions. This powerful combination enables networks to safely move towards Level 5 Autonomy as defined by TM Forum, where the system operates autonomously.

In this post, we present the three components (Data, ML, and AI) that will power Google Cloud’s Autonomous Network Operations framework.

Google Autonomous Network Operations framework architecture

Foundation: Digital Twin on Spanner Graph

At the heart of Google Cloud’s Autonomous Network Operations framework is the network digital twin: a highly detailed, virtual replica that continuously mirrors its living telecommunications network in real time. Rather than being a static model, it is represented as a dynamic, temporal network graph that captures the evolving state and relations of its components over time. This architectural approach allows operators to "go back" in time to train and evaluate ML models on historical data, while providing AI agents with the foundational operational knowledge required to achieve Level 5 Autonomy. By simulating the impact of proposed network changes within this digital environment, the Digital Twin establishes a critical layer of trust, enabling AI agents to confidently design future states and automatically resolve network issues.

Google Cloud’s Spanner Graph is well suited to host this digital twin:

Scalability and Availability: Spanner Graph provides a no compromise foundation for modern applications, offering virtually unlimited scaling that grows as the network grows, along with 0-RPO/0-RTO and five 9s of availability.
Multi-Model Support: Supports multiple data models (Relational, Graph, Vector, and Full-Text Search) in a single platform allowing developers to build complex compositions such as graph transversals combined with nearest neighbor vector search.
Global Consistency: Spanner provides a globally consistent view of the network, simplifying system development.

The next figure illustrates a network topology with four node types: routers, interfaces (the physical ports), VPNs (L3VPN service instances), and flows (active traffic sessions). These are connected by directed edge types capturing the full network stack: physical containment (router-interface), physical links (interface-interface), control-plane peering (router-router via OSPF/iBGP), service membership (router-VPN), and traffic anchoring (flow-interface, flow-VPN).

High Level network topology

The ML layer: Distributed Graph Flow (DGF)

To predict how a network will behave and react, the digital twin leverages an ML layer powered by Distributed Graph Flow (DGF). By training on the vast volumes of structured historical data hosted within Spanner Graph, this layer uncovers critical predictive insights that enable human operators and AI agents to manage networks proactively rather than reactively.

DGF is a recently open-sourced Python library designed to manage the entire end-to-end lifecycle of GNN modeling. Developed by Google CoreML and Google Research, it brings a decade of internal Google-scale tools and expertise directly to Google Cloud enterprise clients. To accommodate different engineering needs, the library offers high-performance, composable, low-level primitives for advanced teams, alongside a simple API for rapid development that requires no prior GNN expertise.

For instance, training and evaluate a GNN model in GraphFlow with the high level API can be as simple as writing 5 lines of code:

code_block: <ListValue: [StructValue([('code', 'import dgf\r\n\r\n# Fetch the data from Spanner Graph\r\ngraph, schema = dgf.io.read_spanner_graph(...)\r\n\r\n# Train a node attribute prediction model\r\nmodel = dgf.learning.train_node_model(graph, schema, target_column="risk_score")\r\n\r\n# Evaluate the model\r\nmodel.evaluate()\r\n# Make predictions\r\nmodel.predict(graph, seed_node_idxs=[0, 1, 2])\r\n\r\n# Save the model for later\r\nmodel.save("/tmp/model")'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f49c4bbcdd0>)])]>

The DGF provides high-level concepts that map directly to Autonomous Network Operations requirements:

Use cases

By leveraging DGF and GNNs, telcos can move from reactive maintenance to proactive prevention through several advanced use cases:

Anomaly detection: GNNs generate node and edge embeddings that encapsulate historical patterns and current health. Any anomalous embeddings are flagged for review before they lead to service degradation.
Root cause analysis (RCA): DGF can output specific subgraphs containing only the relevant network instances related to an incident, such as "Attach Failures" in a specific ZIP code. This allows troubleshooting agents to perform high-speed analysis without scanning the entire global network.
Predictive maintenance: The system can predict the likelihood of device failures or edge breaks, such as "handover failures" for fast-moving equipment, enabling proactive load balancing or rerouting. Furthermore, by combining agents, remedial actions can be automated by adopting a ‘human-on-the-loop’/’human-in-the-loop’.
What-if analysis: GNNs enable Telcos to simulate scenarios like fiber cuts, or traffic surges or device configuration changes. By modeling topological dependencies, GNNs can predict how these local changes propagate across the entire network, allowing engineers to test resilience and evaluate mitigation strategies in a risk-free digital environment.

Scenario: Root cause analysis with GNNs and DGF

Once you have created a digital twin (example code), a straight-forward 5-step process can be used to implement Root Cause Analysis(RCA) detection using GNNs and DGF.

Connect to the Digital Twin: Use the DGF Spanner Graph connector (dgf.io.read_spanner_graph) to load the network topology directly from Spanner Graph's Digital Twin into the DGF environment.
Train a Supervised Node (or Edge) Prediction model: Depending on the training data and objective, you will train a supervised node prediction model to predict a target node feature or an edge prediction model to predict an edge between the root cause entity node and the affected entity node. For the given sample data you will use the high-level dgf.learning.train_node_model API to train a supervised node prediction model.
Use the node prediction model to predict root cause node: The node prediction model can be directly used to predict the impact score on the node with the anomaly. Entity nodes affected by the anomaly with highest predicted impact score will be the top candidates for root cause.
Deploy to Gemini Enterprise Agent Platform (formerly Vertex AI): Export the model and host it on a Gemini Enterprise endpoint to enable scalable, low-latency predictions.
Real-time Inference: Make prediction calls to the inference endpoint with the anomaly date as input. The endpoint will return the predicted root cause Entity nodes.

Get started today

The integration of GNN using Distributed Graph Flow into network operations is more than just a technical upgrade; it is a critical evolution for the telco industry. By moving towards a GNN-powered autonomous framework, operators can significantly shorten outage times, optimize capacity in real-time, and ultimately deliver a superior customer experience through improved operational efficiency.

To start building your own intelligent network applications, check out the Distributed GraphFlow (DGF) library, which provides the essential primitives for scalable GNN training and inference. For a hands-on experience, follow our step-by-step code sample. You can also explore our recent award-winning Moonshot project on Business-aware GNN-healing networks, and dive deeper into our approach on self-optimizing autonomous networks by reviewing this whitepaper.

What’s new with Google Data Cloud

Thu, 10 Sep 2026 16:00:00 +0000

September 7 - September 10

Pub/Sub SMTs can now AI Inference your Gemini Enterprise Agent Platform models!
Pub/Sub AI Inference SMTs allow you to apply inference on an incoming stream of events using models hosted in Gemini Enterprise Agent Platform. The model’s prediction is appended to your event, making it available for downstream processing in your data warehouse (like BigQuery) or operational database (like BigTable). This feature, now generally available, can dramatically simplify or enhance anomaly detection systems you are operating.
PostgreSQL Source Connector is now generally available in Managed Service for Apache Kafka!
Managed Service for Apache Kafka’s PostgreSQL connector allows customers to capture changes from their PostgreSQL database and ingest them into their Kafka infrastructure with low latency. This source connector is compatible with Cloud SQL for Postgres, AlloyDB, and self-managed PostgreSQL databases. Try this along with our entire portfolio of managed connectors, including MirrorMaker 2.0, BigQuery, Cloud Storage, and Pub/Sub! E-mail kafka-hotline@google.com if you have questions or feedback!

Pause-on-failure for Dataflow batch jobs is GA
Dataflow pause-on-failure enables you to preserve the state of a batch Dataflow job before it fails. By pausing your Dataflow job, you can address issues that are external to the pipeline and resume processing without losing completed work. This helps you better manage resource costs and improve job reliability when you face temporary outages or capacity constraints.

The insertAll API is now the BigQuery Storage Write API (REST)
The legacy insertAll streaming API is now rebranded as the BigQuery Storage Write API (REST). By dropping the "legacy" label, developers can confidently build long-term HTTP-based streaming workflows. This stateless JSON-over-HTTPS endpoint offers a lightweight alternative to heavy gRPC libraries—ideal for serverless web apps, IoT telemetry, and AI logging. The transition is seamless for existing users, requiring zero code changes and offering 100% backward compatibility. However, the Storage Write API (gRPC) version remains the recommended standard for high-throughput, continuous pipelines.

August 31 - September 4

Stateful processing is available in BigQuery continuous queries in Preview
Stateful operations significantly expand what’s possible with BigQuery continuous queries. This feature allows users to leverage functions like JOINs, aggregations, and windowing functions directly in their streaming queries. Now you can calculate metrics over time (for example, a 30-minute average) to power your downstream applications and AI agents with much richer, real-time signals.

Try out our feature here and share your feedback with bq-continuous-queries-feedback@google.com!
Synthetic data generator tool is available for Managed Service for Kafka
You’ve launched your first Kafka cluster. Now what? The next thing to do is to produce some data to the cluster, but that involves modifying a client application somewhere or spinning up a virtual machine. The synthetic data generator tool, now generally available, can start sending mock data to your cluster in 3 clicks, and will get data streaming into your cluster in less than two minutes. The perfect utility for those moments you just want to test your cluster and new features. Try our quickstart today!

Dataflow pipeline updates are faster & more flexible
Dataflow pipeline updates can now stop-and-replace pipelines, a major addition to the existing in-place-update feature. The new parallel pipeline option accelerates the migration between the old & new pipeline, resulting in reduced disruption to your business. You can also set a timeout on drains that prevents runaway costs for your pipeliness in the event of stuck processing. This feature is generally available. Try it here!

July 6 - July 10

New Lakehouse managed tables now in preview
Lakehouse tables for Apache Iceberg are now in preview and available in the console. By using Google-managed Apache Iceberg tables in Lakehouse, you can eliminate the overhead of maintaining duplicate data pipelines and complex synchronization logic between BigQuery and open-source engines. This unified table format delivers native, multi-engine read and write interoperability, allowing you to run concurrent DML/DDL operations across diverse analytics tools on a single, shared storage layer. Built-in automated table management handles painful background optimization tasks like compaction and partition tuning, freeing up your team to focus on building rather than managing storage maintenance.

June 1 - June 5

Beyond the Query: Powering AI Agents with Bigtable, Firestore & Memorystore
Discover the latest advancements in Google Cloud's NoSQL Database portfolio, including Bigtable, Firestore, and Memorystore. This series is designed for a broad audience: whether you are exploring these databases for the first time or are an existing user looking to leverage the new capabilities announced at Next '26.

Register here to secure your spot!

Cloud Engineer's AI Toolkit Workshops: Solve data-driven challenges with BigQuery, AlloyDB, Gemini and more. Hosted by Google Cloud Labs, this highly technical event is built specifically for Platform Engineers, SREs, and cloud infrastructure teams ready to bridge the gap between AI prototypes and production-grade deployments. Look out for more locations coming soon

Toronto - June 25 (Data Cloud) | RSVP Here
Chicago - June 30 (Data Cloud) | RSVP Here
Start a 10-day Bigtable free trial with a 1 node SSD cluster and up to 500GB of storage capacity. With no credit card required to start, you can easily ingest workloads and manage workloads that require low-latency, high-throughput, and predictable access. Plus, new Google Cloud customers get $300 in free credits on signup.

May 11 - May 15

Managed Service for Apache Airflow has launched a wave of new features, including the general availability of Airflow 3.1, AI-powered agentic troubleshooting, a new managed Airflow MCP Server for custom agent integration, and declarative YAML-based orchestration pipelines—discover all the details in the full blog post.

April 20 - April 24

Google-built ODBC Driver for BigQuery is now available in Preview
We are excited to announce the launch of the new, Google-built ODBC driver for BigQuery. This new open-source driver provides a direct, high-performance connection for applications to BigQuery and is developed entirely in-house by Google. Download a new driver and connect your application to BigQuery.

April 13 - April 17

We announced we are reintroducing Data Studio to play a significant role in the AI era, expanding from data visualizations and reports to host BigQuery conversational agents and data apps built in Colab notebooks.
We announced BigQuery Graph is now available in preview, offering an easy-to-use, highly scalable graph analytics solution, empowering data professionals to model, analyze and visualize massive-scale relationships in an entirely new way.

April 6 - April 10

We introduced Conversational Analytics for Looker Embedded environments, enabling users to add natural language experiences to their own custom data-driven applications, powered by Gemini.
We expanded Looker’s capabilities for faster ad-hoc analysis, with the introduction of self-service Explores, enabling you to bring your own data to Looker’s semantic layer and gain instant access to insights in a governed data environment.

March 23 - March 27

We showed you how you can scale your reads with Cloud SQL autoscaling read pools. This feature allows you to provision multiple read replicas that are accessible via a single read endpoint and to dynamically adjust your read capability based on real-time application needs.
Our customers are leveraging the full power of Conversational Analytics and Looker to drive major business and technical breakthroughs in the AI era. Companies like Telenor, Pet Circle, Fluent Commerce, Lighthouse Intelligence, Wego, and ROLLER are turning data into insights and actions, grounded by Looker’s semantic layer.

March 16 - March 20

We introduced an enhanced Gemini assistant in BigQuery Studio, transforming the agent from a code assistant into a fully context-aware analytics partner.

February 23 - February 27

We introduced managed and remote MCP support for Google Cloud databases, including AlloyDB, Spanner, Cloud SQL, Bigtable and Firestore, to power the next generation of agents. This announcement extends the ability for AI models to plan, build, and solve complex problems, connecting to the database tools our customers leverage daily as the backbone of their work environment.
We outlined how you can build a conversational agent in BigQuery using the Conversational Analytics API to help you build context-aware agents that can understand natural language, query your BigQuery data, and deliver answers in text, tables, and visual charts.

February 16 - February 20

Our customers are leveraging the full power of Looker to drive major business and technical breakthroughs. Companies like Arrive, Audika, Carousell, Framebridge, GumGum, Intel, Overdose Digital, Ocean Network Express, Subskribe and Promevo are leveraging Looker’s newest AI-driven capabilities, including Conversational Analytics, to transform data to insights and actions, and empower their entire organization with a single source of truth, powered by Looker’s semantic layer.

February 2 - February 6

Join us on March 4 for our webinar, Win Your AI Strategy with Cloud SQL Enterprise Plus, to learn how to power your generative AI workloads with 3x higher performance and 99.99% availability. Register today to discover how to build a scalable, enterprise-grade foundation for your most demanding AI applications.

January 26 - January 30

We introduced Conversational Analytics in BigQuery, which allows users to analyze data using natural language. Conversational Analytics in BigQuery is an intelligent agent that generates, executes and visualizes answers grounded in your business context directly in BigQuery Studio, making data insights for data professionals more conversational.
We outlined how data products have become the foundation for AI agents, providing the context needed to make autonomous agents reliable and trusted for real business use, backed by organized business logic and semantic understanding.
We highlighted how you can supercharge data analytics workflows, and outlined Google Cloud’s AI agent offerings for data engineering, data science, and development tools, so you can integrate agentic workflows in your applications, empower your teams and speed discovery.

January 19 - January 23

We have fundamentally reimagined Firestore with pipeline operations for Enterprise edition. Experience a powerful new engine featuring over a hundred new query features, index-less queries, new index types, and observability tooling to improve query performance. Seamlessly migrate using built-in tools and leverage Firestore’s existing differentiated serverless foundation, virtually unlimited scale, and industry-leading SLA. Join a community of 600K developers to craft expressive applications that maximize the benefits of rich queryability, real-time listen queries, robust offline caching, and cutting-edge AI-assistive coding integrations.
Introducing Google Cloud SQL on MSSQLTips: We are highlighting a new technical guide published on MSSQLTips titled "Introducing Google Cloud SQL." This article serves as an essential resource for SQL Server administrators and developers exploring Google Cloud's fully managed database service. It provides a detailed overview of Cloud SQL capabilities, including high availability, security integration, and the seamless transition of on-premises SQL Server workloads to the cloud, making it an ideal resource for those planning their migration strategy.
We are excited to announce the Public Preview of Microsoft Entra ID (formerly Azure Active Directory) integration with Cloud SQL for SQL Server. Designed to tackle the challenge of identity sprawl in multi-cloud environments, this integration allows organizations to govern database access using their existing Microsoft identity infrastructure. Key benefits include centralized identity management, enhanced security features like Multi-Factor Authentication (MFA), and simplified user administration through direct group mapping. This feature is available for SQL Server 2022 and supports both public and private IP configurations.

January 12 - January 16

Google-built JDBC Driver for BigQuery is now available in Preview
We are excited to announce the launch of the new, Google-built JDBC driver for BigQuery. This new open-source driver provides a direct, high-performance connection for Java applications to BigQuery and is developed entirely in-house by Google. Download a new driver and connect your Java application to BigQuery.
Troubleshoot Airflow tasks instantly with Gemini Cloud Assist investigations: Cloud Composer just got smarter. We are excited to announce that Gemini Cloud Assist investigations are now available directly within Cloud Composer 3. Instead of manually sifting through raw logs, you can now simply click "Investigate" on a failed Airflow task. Gemini analyzes logs and task metadata to identify failure patterns—such as resource exhaustion or timeouts—and provides actionable recommendations driven by Gemini Cloud Assist to resolve the issue. This integration shifts the debugging experience from manual toil to automated root cause analysis, significantly reducing the time required to restore your pipelines. Learn more about AI-assisted troubleshooting.

Enterprise-grade PostgreSQL with AlloyDB Omni RPM Orchestrator is generally available

Wed, 09 Sep 2026 19:00:00 +0000

We are thrilled to announce the general availability of the AlloyDB Omni Red Hat RPM orchestrator, that brings production-ready security, resiliency, and low-downtime operations to PostgreSQL workloads in your enterprise environments. This GA milestone builds on the foundation laid during our preview release and launches alongside AlloyDB Omni version 18.3.0 to bring cloud-like database automation directly to your virtual machines and bare-metal servers with Google’s AI capabilities.

As of this GA release, AlloyDB Omni can be deployed in four modes to suit your requirements. Visit AlloyDB Omni documentation for more information.

Standalone container (Debian / UBI)
Container with Kubernetes operator for Highly Available enterprise deployment
Standalone RPM
With RPM orchestrator for Highly Available enterprise deployment

Why run a self-managed database?

For many use cases, a managed cloud database service is the simplest and most cost-effective option. However, there are scenarios where you may choose to run a PostgreSQL database yourself, on or off the cloud. The AlloyDB Omni RPM deployment is built for organizations that need the performance of the cloud with the control of local, non-containerized infrastructure, with use cases including:

Workload Modernization: AlloyDB Omni is more than 2X faster for transactional workloads and can deliver up to 100X faster analytical queries than standard PostgreSQL L , revitalizing existing infrastructure without a full migration.
Regulated Environments: For industries with strict data residency and security requirements, the RPM orchestrator provides the necessary tools like SELinux and local audit logging to stay compliant.
Edge and On-Premises Deployment: Deploying at the edge or on bare-metal servers allows for low-latency processing and disconnected operation.
AI-Ready Infrastructure: You can provision database clusters for AI integrations, using AlloyDB AI capabilities such as vector search for modern generative AI applications directly on-premises.

Flexible Reference Architectures

The AlloyDB Omni RPM orchestrator offers flexible deployment models tailored to your organization's specific operational requirements—whether your focus is maximizing performance, scaling read throughput, or ensuring robust high availability (HA). For more details, refer to the AlloyDB Omni availability reference architecture overview. The orchestrator simplifies cluster provisioning and lifecycle management by allowing you to define reference architecture specifications, customizable by adjusting instance parameters, node configurations, and networking options. Here is an example deployment view of scalable AlloyDB Omni HA reference architecture.

High availability reference architecture for AlloyDB Omni clusters with RPM Orchestrator

The diagram illustrates a highly available, distributed database reference architecture for AlloyDB Omni managed by the RPM Orchestrator. It shows that the client applications connect to a robust load-balancing tier and the load balancer routes read-write traffic directly to the active primary database node, while read-only traffic is routed to the replica nodes. The load balancer uses a Virtual IP (VIP) and is highly available itself, with Keepalived (for VIP failover), PgBouncer (for PostgreSQL connection pooling), and Haproxy (for routing and load balancing). The AlloyDB Omni Primary instance database nodes achieve high availability by replicating data synchronously across multiple zones. To scale out read-heavy workloads without impacting the primary HA cluster, separate Read Pool instances are deployed. These receive Async Replication (asynchronous) from the active node and can be scaled out.

It shows how an independent control plane manages the entire configuration and health of the clusters. The administrator interacts with the RPM Orchestrator, to oversee the lifecycle of the databases. The control plane includes redundant Cluster Managers and a 3-node etcd based Distributed Configuration Store to reliably maintain cluster state, and manage configurations. The controllers directly interface with the Node Manager running on each individual database node. For deploying only a single cluster, you may run control and data plane components on the same set of nodes.

High Availability and Read Scalability

Maintaining uptime and scaling reads for demanding workloads is simpler with the RPM orchestrator. It provides a resilient architecture capable of automatically handling failures across the stack, including the ability to handle failure of all Data / Control Path components, Readable Standby, as well as mitigating any network disruptions between the nodes.

The orchestrator now supports read pools for scaling out your read workloads and gives you the ability to create or add read pools to a cluster dynamically to meet the needs of analytical queries or similar workloads. The system also configures dedicated read endpoints for both standby nodes and readpools.

We are continuously expanding the capabilities of the AlloyDB Omni RPM orchestrator. Stay tuned for upcoming features, including advanced enterprise-grade capabilities to further strengthen business continuity and cross-region resiliency.

Data Protection and Security

Data security and recovery are at the core of the RPM Orchestrator. In this release, we have integrated automated backup and restore capabilities that enable you to configure a backup schedule and manage fully automated backup to GCS or S3-compatible storage automatically. The orchestrator allows you to execute backups to S3 or GCS buckets, or locally. Additionally, point-in-time recovery and fully automated in-place PIT restore are natively supported.

The RPM orchestrator supports SELinux enforcement at or after bootstrap to satisfy strict enterprise compliance and security standards and ensure mandatory access control and strong process isolation.

Database Operations

We’ve reduced the operational overhead associated with managing database fleets:

Zero-Hassle Low-Downtime Maintenance: Managing lifecycle updates and scaling operations is easier with the new, fully automated Low Downtime Maintenance (LDTM). Minor version upgrades as well as CPU and memory resource adjustments are executed with minimized downtime and automatic rollback support for maximum availability.
Dynamic Configuration: Database administrators can dynamically tune settings without hassle, including the ability to Modify GUCs/configs at or after bootstrap. You can also provision a cluster for AI integrations.
Simplified Cluster Maintenance: Managing your cluster footprint is straightforward, with native operational capabilities to add or remove database nodes as your workload demands shift.

Observability, AI, and Extensions

Monitoring and tuning your fleet requires deep visibility and the right set of tools:

Advanced Logging: The orchestrator simplifies auditability and debugging by providing Data and Control Path log direction to log disk.
Rich Observability: Custom metrics support allows you to fine-tune observability to suit your monitoring ecosystem. With custom metrics, you can track business-level events directly from the database, such as the number of new user registrations per minute, active sessions for a specific tenant, or the volume of orders processed, and export these to your company's central observability platform.
AI & Extensions: In addition to the list of extensions supported with AlloyDB Omni, the orchestrator includes support for all AlloyDB Omni's AI features such as query using natural language, AI-powered searches, AI functions, etc.

Get Started Today

The AlloyDB Omni Red Hat RPM orchestrator offers a new way to manage PostgreSQL-compatible workloads on bare metal or VM platforms, combining the high performance of AlloyDB, access to generative AI features and Gemini models to build AI agents and applications, and full automation.

Ready to elevate your on-premises database operations? Dive into our AlloyDB documentation to get started with the GA release today. Sign-up today !!

You can also try our new codelab to deploy a highly available AlloyDB Omni cluster using the RPM Orchestrator.

Beyond DMS: Accelerating Migrations SQL Server Logins and Users to Cloud SQL

Wed, 09 Sep 2026 16:30:00 +0000

So, you’ve planned your database modernization journey. You’ve set up Google Cloud’s Database Migration Service (DMS), configured replication, and successfully synchronized your application databases from your on-premises or cloud systems to a fully managed Cloud SQL for SQL Server instance.

The replication is complete, the data is up to date, and you’re ready for cutover. But when your application attempts to connect to the newly migrated database, you’re hit with a frustrating roadblock:

Msg 18456, Level 14, State 1, Line 1: Login failed for user 'app_user.

The culprit is simple: your SQL Server logins didn't migrate with your database. In this post, we’ll look at why this gap exists, why it actually protects your organization's security posture, and how easy it is to bridge using standard, time-tested SQL Server tools.

Why DMS doesn't migrate logins: Security and compliance

Database Migration Service (DMS) is highly efficient at replicating database-level schemas and transactional data. However, it purposefully doesn’t migrate instance-level objects, such as the system master database or server logins and permissions.

While this might feel like a missing feature, it is actually a deliberate design choice built around three core pillars:

Security Isolation and Privilege Boundaries: The source environment and the destination Cloud SQL environment operate under different security paradigms. Replicating the master system database directly could lead to unauthorized privilege escalation. For example, an on-premises login with sysadmin privileges shouldn’t have unrestricted sysadmin access to a fully managed Google Cloud database. When the cloud provider manages physical backups, patching, and security, it needs to limit underlying operating system access to ensure correct operation.
Compliance and Audit Governance: Automated migration of encrypted password hashes and server-level security credentials without explicit administrator oversight frequently violates enterprise compliance frameworks such as PCI-DSS or SOC 2. By keeping security object migration as a deliberate, administrator-driven step, organizations can guarantee that only approved identities are provisioned in the cloud landing zone.
The Need for Identity Modernization: Migrating to the cloud is the perfect opportunity to update and prune stale credentials. Frequently, on-premises instances carry legacy SQL logins that are no longer used. Replicating them blindly to a cloud-managed service is a security anti-pattern. Furthermore, moving to Cloud SQL is often the catalyst for shifting away from legacy SQL authentication toward modern, cloud-native identity solutions like Customer-Managed Active Directory (CMAD).

Understanding logins vs. users: The SID connection

To migrate logins successfully, let’s briefly revisit how SQL Server manages security. SQL Server separates identity into two distinct layers:

Logins (server-level): Stored in the master database. These authenticate a client connection to the SQL Server instance.
Users (database-level): Stored inside individual user databases. These authorize what actions a connection can perform within that specific database.

The bridge between a server login and a database user is a unique Security Identifier (SID).

When you backup and restore a database (or use DMS to replicate it), the database-level users (and their corresponding SIDs) are migrated inside the database files. However, if the corresponding server-level login does not exist in the destination master database—or exists but has a different SID—the mapping breaks. This results in "orphaned users" who have database access permissions but no way to authenticate at the server level.

Figure 1: How migrating databases without corresponding logins or with mismatched security identifiers (SIDs) results in orphaned users on the destination instance.

The recommended solution: Replicating logins using sp_help_revlogin

Instead of manually recreating every login and guessing password hashes, we can rely on a classic Microsoft-provided script: sp_help_revlogin.

This script generates a T-SQL query containing the CREATE LOGIN statement for every SQL Server authentication login on your source instance, complete with its original, encrypted password hash and its exact Security Identifier (SID).

Step 1: Create the helper procedures on your source instance

Connect to your source SQL Server instance using SQL Server Management Studio (SSMS). Copy and execute the official Microsoft script to create the two required stored procedures in your source master database: sp_hexadecimal and sp_help_revlogin.

Step 2: Generate the migration script

Once the procedures are created, run the following statement in your SSMS query window. Make sure to toggle your output settings to Results to Text (Ctrl + T) to copy the output cleanly:

code_block: <ListValue: [StructValue([('code', 'EXEC master.dbo.sp_help_revlogin;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f49c550ff90>)])]>

The output will contain auto-generated T-SQL statements that look similar to this:

code_block: <ListValue: [StructValue([('code', 'CREATE LOGIN [app_user] WITH PASSWORD = 0x01004F3D... HASHED, SID = 0x8D2F..., DEFAULT_DATABASE = [CustomerDB]'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f49c4c75910>)])]>

By scripting out the login with the HASHED password option and the original SID, SQL Server allows us to safely recreate the login with its original password and secure link intact.

Step 3: Apply the script to Cloud SQL

Copy the generated script, connect to your destination Cloud SQL for SQL Server instance, and execute the query. Your logins are instantly created in the cloud with their correct passwords.

By running the script generated by sp_help_revlogin, we replicate the logins onto the destination Cloud SQL instance with their exact security identifiers (SIDs) and password hashes intact. As shown below, this ensures that the database-level users automatically map to their server-level logins upon database migration, avoiding “orphaned users” entirely.

Figure 2: The unified migration process using the sp_help_revlogin script to preserve password hashes and original SIDs, resolving user mapping on Cloud SQL for SQL Server.

Note:
sp_help_revlogin is a stored procedure that was created and is maintained by Microsoft. Make sure to download the latest version and read the documentation.

Troubleshooting orphaned users

If you had created a login on the target Cloud SQL instance manually before running sp_help_revlogin, the SIDs might not match, causing the user to become "orphaned."

If you find an orphaned user (say, app_user), you can easily remap it to the newly created server login with a single command:

code_block: <ListValue: [StructValue([('code', 'ALTER USER [app_user] WITH LOGIN = [app_user];'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f49c4b8e5d0>)])]>

With that command, the database user and the server login are immediately reunited via their SIDs, and application connectivity is fully restored.

Take your security a step further

While migrating SQL logins using sp_help_revlogin is the easiest path for a lift-and-shift migration, consider utilizing your cloud migration to modernize your authentication. Cloud SQL for SQL Server supports robust integrations with Customer-Managed Active Directory (CMAD). Integrating your destination instance with Active Directory allows you to deprecate legacy SQL logins in favor of centralized, enterprise-grade Kerberos authentication.

Wrap up

Database migration is more than just shifting rows of data—it’s about ensuring your applications remain secure, compliant, and operational from day one. While Google Cloud’s DMS handles the heavy lifting of data replication, migrating your logins is a straightforward, three-step process that guarantees a seamless cutover.

To learn more about optimizing your migration strategy, check out the Cloud SQL for SQL Server Migration Guide and explore how Database Migration Service can streamline your move to Google Cloud.

Spanner: Removing cumulative mutation limits for DML transactions

Wed, 09 Sep 2026 16:00:00 +0000

Spanner is Google Cloud’s no-compromise operational database that gives you the horizontal scale and always-on availability of a modern distributed system along with the rich feature set and familiar ecosystem of a relational database. Innovators in industries like banking, retail, media and entertainment, and AI infrastructure rely on Spanner today for their most critical workloads. We’re excited to announce a new, flexible way to handle larger, more complex transactions in Spanner, simplifying applications that need the highest levels of data consistency.

Operational workloads typically combine real-time decision making with granular updates: Think: identifying fraud as part of a multi-step checkout process in an ecommerce app. These changes must be transactional; either all of them succeed or none of them do and subsequent requests see the correct data. This update to Spanner’s ACID transactions allows applications to handle more data in an update without compromising on consistency, scalability, or availability using familiar DML.

Higher ceiling, more flexibility

Previously, Spanner capped the changes a query could perform in a transaction, for example using DML, at 80,000. That was roughly computed as the product of the number of rows and number of columns updated, plus any dependent indexes. Applications evolve over time to handle more data and provide new functionality. These changes increase the size of transactions, potentially causing previously small transactions to hit this limit.

This update shifts the 80,000 mutation mod limit from the transaction to individual DML statements. DML statements no longer contribute to an overall transaction-level mutation limit. A single transaction can now contain any number of DML statements, such as INSERT, UPDATE, or DELETE, provided that each individual statement generates fewer than 80,000 mutation mods.

Key benefits

Larger transactions: Group DML statements logically based on business requirements rather than artificially splitting them to comply with cumulative mutation limits.
Seamless transition: This change is compatible with all existing Spanner client libraries and requires no updates to application code.

Technical considerations

Locking and aborts

While you can now include more DML statements in a single transaction, be aware that larger and longer-running transactions hold locks for a greater duration. This may increase the likelihood of lock contention and transaction aborts. Keeping transactions concise helps maintain high performance and minimize resource contention.

DML vs. Mutation API

The application of limits depends on the method used to modify data:

DML Statements: Each statement (e.g., executeUpdate) is evaluated independently against the 80,000 mod limit.
Mutation API: When using client library methods like insert() or update(), mutations are provided during the Commit call. The 80,000 limit continues to apply to the entire set of mutations included in that single call.

Understanding mutation mods

Spanner counts "mods" based on the complexity of changes, including modified cells, primary keys, and secondary index updates. Please look at this blog for more details on how mutations are counted. You can monitor the total mods for a committed transaction via the mutation_count in the CommitStats. Note that the mutation_count will include all the mutations that are part of the transaction, across all DML statements and commit calls.

Java implementation example

The following example demonstrates how multiple DML statements can be executed within a single transaction under the new limit logic.

code_block: <ListValue: [StructValue([('code', 'import com.google.cloud.spanner.DatabaseClient;\r\nimport com.google.cloud.spanner.Statement;\r\nimport com.google.cloud.spanner.TransactionContext;\r\nimport com.google.cloud.spanner.TransactionRunner.Work;\r\n\r\n// Assuming dbClient is your initialized DatabaseClient\r\ndbClient\r\n .readWriteTransaction()\r\n .run(\r\n new Work<Void>() {\r\n @Override\r\n public Void doWork(TransactionContext transaction) throws Exception {\r\n // Each executeUpdate call is evaluated separately against the 80k mod limit.\r\n\r\n // Example 1: Updating specific products\r\n Statement stmt1 = Statement.newBuilder(\r\n "UPDATE Products SET InStock = FALSE WHERE ProductId = @productId")\r\n .bind("productId").to(1L)\r\n .build();\r\n transaction.executeUpdate(stmt1); // Verified against 80k limit\r\n\r\n Statement stmt2 = Statement.newBuilder(\r\n "UPDATE Products SET InStock = FALSE WHERE ProductId = @productId")\r\n .bind("productId").to(2L)\r\n .build();\r\n transaction.executeUpdate(stmt2); // Verified against 80k limit separately\r\n\r\n // Example 2: Inserting related order data\r\n Statement stmt3 = Statement.newBuilder(\r\n "INSERT INTO OrderItems (OrderId, ItemId, Quantity) VALUES (@orderId, @itemId, @qty)")\r\n .bind("orderId").to(100L)\r\n .bind("itemId").to(1L)\r\n .bind("qty").to(2)\r\n .build();\r\n transaction.executeUpdate(stmt3); \r\n\r\n Statement stmt4 = Statement.newBuilder(\r\n "UPDATE Orders SET LastUpdated = PENDING_COMMIT_TIMESTAMP() WHERE OrderId = @orderId")\r\n .bind("orderId").to(100L)\r\n .build();\r\n transaction.executeUpdate(stmt4); \r\n\r\n return null;\r\n }\r\n });'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f49c4bcfbd0>)])]>

What has not changed

Individual statement limit: Any single DML statement that generates more than 80,000 mods on its own will still return the same error as we do today.
Other transaction limits: Other constraints such as the maximum transaction size in bytes remain in effect. They are documented here.

Best practices

Monitor CommitStats: Utilize the mutation_count returned in CommitStats to understand the load generated by your operations.
Optimize large operations: If a single statement (like a bulk update) exceeds the limit, consider using Partitioned DML or paginating through keys.

Spanner is the trusted choice for operational applications that need to scale without downtime. This increase to the mutation limit provides developers new flexibility to run larger transactions that leverage Spanner’s global consistency. Learn how Spanner can help your teams innovate faster with less risk, or try it on your own, with a free trial or production instances starting as low as $54/month.

External references

How AlloyDB ScaNN scales vector search to 10 billion vectors

Thu, 20 Aug 2026 16:00:00 +0000

To satisfy the demands of enterprise-grade agentic AI applications, underlying vector databases often struggle to scale effectively as modern use cases can scale to billions of vectors.

As a fully managed PostgreSQL-compatible database service, AlloyDB is engineered to handle demanding enterprise workloads. Combining Google's infrastructure with the reliability of commercial databases, it delivers high availability, scalability, and includes a cutting-edge analytical engine, optimal for agentic AI use cases. A key part of this is its ScaNN index, which now operates efficiently at a scale of 10 billion vectors. This was achieved through a major architectural enhancement: an innovative four-level tree (preview) paired with efficient memory usage.

The 10 billion vector scale challenge

Scaling to a 10 billion vector workload presents significant memory and computational challenges. Previous AlloyDB ScaNN tree-based index was limited to two- or three-level tree configurations, and attempting to scale those structures led to several bottlenecks:

Increased compute intensity: Larger tree structures demand significantly more operations for both index construction and query traversal.
Memory constraints: The sampling processes required for 10 billion vectors can easily exceed the system's available memory capacity.

Solution: Four-level architecture

The introduction of a four-level tree (preview) is the primary innovation in the recent AlloyDB ScaNN release. This architecture, illustrated in Figure 1, employs a top-down strategy to optimize the balance between accuracy and build efficiency. To maintain high performance and mitigate recall loss, the system integrates key enhancements such as Top-K branch, SOAR, centroid adjustment and balanced tree shape.

Figure 1. AlloyDB ScaNN four-level tree architecture

This design has two primary benefits:

1. Reduced compute intensity via hierarchical partitioning

The four-level architecture drastically reduces compute intensity by using hierarchical partitioning to restrict the volume of vectors scanned during a query. Instead of traversing a flat or poorly segmented space, the multi-layered hierarchy narrows down the search path exponentially. Figure 2 illustrates the search spaces across different tree levels, demonstrating how structural layering optimizes traversal efficiency:

Figure 2. Search space for two-, three- and four-level trees

Two-level: Utilizes coarse partitioning to guide queries, resulting in a basic search complexity of O(N^1/2).
Three-level: Introduces an intermediate layer to further subdivide clusters, narrowing exploration to O(N^1/3).
Four-level: Implements refined, highly granular partitions that optimize traversal efficiency down to O(N^1/4), sufficiently allowing for more than 10-billion vectors.

By dynamically expanding hierarchical layers as the dataset expands, AlloyDB ScaNN maintains ultra-low query latency and avoids computational scale walls from impacting performance.

2. Efficient memory usage

Achieving a 10 billion vector scale requires high memory efficiency. AlloyDB ScaNN uses these strategies to maximize memory management performance:

Balanced tree shape construction: The four-level tree utilizes a balanced configuration to circumvent memory limitations that restrict the size of training datasets. This balanced architecture effectively leverages reduced sampling sizes to construct high-fidelity tree partitions.
Sampling optimization: When the system encounters memory limitations, it generates a condensed sampling set that considers performance and accuracy.

Performance test results

By leveraging the innovative four-level tree architecture in our internal tests, we are able to achieve the following performance results:

AlloyDB can scale to over 10 billion vectors with its ScaNN index.
AlloyDB can deliver <= 51 ms p95 latency and 95% recall at 10 billion vectors with its ScaNN index.

Get started today

Experience AlloyDB ScaNN's four-level tree (preview) architecture today. You can deploy ScaNN for AlloyDB by following our quickstart guide to set up an instance. For optimized, high-speed vector search, refer to the official ScaNN documentation. New users can also explore AlloyDB through our 30-day free trial program. We can’t wait to hear about what you build!

Using BigQuery Graphs with measures for trusted agentic workloads

Thu, 13 Aug 2026 17:00:00 +0000

When enterprises transition from using simple chat assistants to autonomous, agentic workloads, they quickly run into a hard truth: Agents are prone to inaccurate insights when working with directly raw tables.

BigQuery Graph helps organizations move beyond flat, static tables to represent enterprises exactly how they exist in the physical world: as interconnected business entities with real-world dependencies. With the support of measures in BigQuery Graph (preview), we are unifying governed metrics with relationship mapping. This allows your agents to reason across complex dependencies captured in graphs with precision of measures.

Why relationships matter

Traditional data structures are blind to multi-hop business context, causing AI agents to make incorrect operational decisions:

The concrete problem: If a retailer has an agent who is asked why winter jacket sales dropped 12% in Seattle, it can query flat tables to report the what (the 12% dip). But it fails at the why because it cannot trace the relational path: Seattle orders ➔ distribution centers ➔ suppliers delayed by regional storms.
The risk of disjointed systems: Lacking relationship context, the agent suggests an irrelevant 15% markdown campaign, needlessly eroding margins. Furthermore, maintaining separate systems - where one team maps supplier relationships in a separate graph database while another maintains SQL metrics - forces your agent to stitch these stacks together at runtime. This process is slow, expensive, and leads to inconsistent KPI calculations.

Measures in BigQuery Graph solves this by letting you map existing tables to a property graph in-place with zero ETL. This unified setup enables a logical evolution of inquiry:

Metadata grounding establishes what data you have.
Business metrics (measures) calculate how your business performed.
Relationship mapping (graph) uncovers why it happened.

Under the hood

Historically, standard SQL joins during graph traversals duplicate rows, leading to incorrect aggregation calculations. BigQuery Graph solves this natively.

Data modelers define a MEASURE (like SUM or AVG) directly within the Property Graph DDL. Using standard SQL via the GRAPH_EXPAND function and the AGG aggregator, the engine resolves the structural graph paths before evaluating metrics. This ensures your agent is smart enough to know when it needs a calculator (SQL) and when it needs a map (graph).

Because public projects like bigquery-public-data are strictly read-only, you must map the logical property graph inside your own project using a placeholder variable (YOUR_PROJECT_ID), while directly referencing the read-only public tables as nodes and edges.

code_block: <ListValue: [StructValue([('code', '-- 1. Map the graph inside YOUR project \r\n\r\n\r\nCREATE OR REPLACE PROPERTY GRAPH `YOUR_PROJECT_ID.YOUR_DATASET.thelook_ecommerce_graph`\r\nNODE TABLES(\r\n `bigquery-public-data.thelook_ecommerce.users` AS User\r\n KEY(id)\r\n LABEL User PROPERTIES(id, city, country),\r\n `bigquery-public-data.thelook_ecommerce.orders` AS Order\r\n KEY(order_id)\r\n LABEL Order PROPERTIES(\r\n order_id, \r\n MEASURE(AVG(num_of_item)) AS avg_items_per_order,\r\n MEASURE(SUM(num_of_item)) AS total_items\r\n )\r\n)\r\nEDGE TABLES(\r\n `bigquery-public-data.thelook_ecommerce.orders` AS OrderedBy\r\n SOURCE KEY(order_id) REFERENCES Order(order_id)\r\n DESTINATION KEY(user_id) REFERENCES User(id)\r\n LABEL ORDERED_BY\r\n);\r\n\r\n-- 2. Query your new graph with standard SQL—using standard {Label}_{Property} column outputs\r\nSELECT\r\n User_city AS city,\r\n ROUND(AGG(Order_avg_items_per_order), 2) AS agg_avg_items,\r\n ROUND(AGG(Order_total_items), 2) AS agg_total_items\r\nFROM GRAPH_EXPAND("YOUR_PROJECT_ID.YOUR_DATASET.thelook_ecommerce_graph")\r\nGROUP BY User_city\r\nORDER BY agg_total_items DESC\r\nLIMIT 10;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f49c4b78350>)])]>

Democratizing graph intelligence in BigQuery Studio

To make managing and deploying these relationship networks frictionless for both developers and business users, we have built native, intuitive operational tools directly into BigQuery Studio:

Visual graph modeler: A no-code, drag-and-drop interface inside BigQuery Studio that lets you visually build, edit, and map property graphs, nodes, and edges without writing complex DDL scripts manually.

Conversational Analytics (CA) integration: Users can interact with the graph naturally. Instead of guessing table joins, Conversational Analytics agents navigate the deterministic, relationship-aware map of the graph, converting natural language questions into precise, boundary-constrained GoogleSQL or ISO GQL queries. This prevents model hallucinations and enforces semantic consistency.

Unified semantics: Native Looker integration

To avoid maintaining fragmented logic stacks, business metrics must live at the data layer. By integrating Looker (LookML) natively with BigQuery Graphs as in-database analytic models, you define logic once at the core:

Database-managed models (sql_analytic_model_name): Point Looker directly to your database-defined BigQuery Graph using sql_analytic_model_name to map standard LookML dimensions and measures directly to your graph properties.
Looker-managed models (derived_analytic_model): Define your BigQuery Graph schema directly inside your LookML view using derived_analytic_model. Looker will dynamically generate and execute the SQL DDL statements to maintain the graph inside BigQuery.
Enterprise DevOps workflows: Manage your graph's entire lifecycle using the Looker IDE, Git-based version control, and Continuous Integration (CI). Core KPIs (like Churn Rate) remain completely identical, verified, and trusted.

Accelerate PostgreSQL migrations using Gemini in Database Migration Service

Tue, 11 Aug 2026 16:00:00 +0000

Imagine this scenario: Your team decides to migrate a core application from an existing commercial database like Oracle or SQL Server to open source PostgreSQL or a fully managed service such as AlloyDB for PostgreSQL.

The initial phase goes smoothly. Schemas convert, tables populate, and data migration pipelines transfer terabytes of data in hours. The project looks ahead of schedule.

Then your team hits the bottleneck.

Buried inside the existing databases are hundreds of stored procedures, complex triggers, and custom functions written in proprietary SQL dialects like PL/SQL or T-SQL. These routines contain years of critical business logic handling transaction validation, order processing, and custom reporting.

Suddenly, your modernization project halts. Translating thousands of lines of procedural logic demands specialized dual-dialect expertise, months of manual rewriting, and high risk of conversion errors. This code translation represents the "last mile" bottleneck of database migration and is the most complex part of migrations.

Thankfully, recent advancements in AI provide a solution to the last mile problem. Database Migration Service (DMS) includes AI-assisted code conversion powered by Gemini. By bringing generative AI directly into your migration workflow, you can convert stored procedures, triggers, and custom functions into PostgreSQL PL/pgSQL code faster and with higher accuracy.

The stored procedure conversion challenge

Commercial database engines rely on vendor-specific syntax for stored procedures, user-defined functions, package bodies, and conditional logic. Converting this logic to PostgreSQL PL/pgSQL requires mapping variable definitions, exception handling blocks, cursor loops, and built-in functions.

When migrating complex enterprise schemas with hundreds of stored procedures, manual code conversion often demands months of engineering effort. Database teams must parse legacy logic line by line, re-implement conditional branches, and verify data type conversions between engines.

AI-assisted code conversion in DMS

Gemini in Database Migration Service accelerates this conversion work directly inside the Google Cloud console. DMS provides automated schema conversion alongside AI-generated code suggestions that explain structural differences between the source dialect and PostgreSQL.

The service presents converted PL/pgSQL code side-by-side with original source code, allowing database teams to review, edit, and validate suggestions in real time.

Figure 1: Database Migration Service interface displaying side-by-side code conversion and Gemini inline explanation.

Why integrated AI matters

Most AI apps and tools from major vendors have the ability to generate and convert code, including SQL code. However, converting enterprise databases demands far more than snippet translation offered by generic AI chat tools. Gemini in Database Migration Service offers several key advantages:

Full schema context: Rather than evaluating code snippets in isolation, Gemini in DMS analyzes your entire database context, including table relationships, data types, dependent views, and cross-procedure references across the whole migration project.
Enterprise security and privacy: Code conversion runs strictly within your Google Cloud project boundaries and IAM governance, protecting proprietary business logic and intellectual property.
Integrated execution workspace: DMS eliminates manual copy-pasting across hundreds of files. You can review side-by-side diffs, inspect inline AI explanations, edit code, and deploy validated PL/pgSQL routines directly to target databases within a single console.
Deterministic accuracy and AI compilation: DMS pairs deterministic compiler rules for 1:1 mappings (such as standard DDL transformations, scalar functions, and well-defined syntax conversions) with Gemini contextual synthesis for complex procedural blocks—guaranteeing exact, predictable translation without model drift.

Converting Oracle PL/SQL to PostgreSQL

Consider an Oracle PL/SQL stored procedure that calculates customer order totals and applies tier-based discounts using proprietary NVL and DECODE functions. In the original workflow, you must manually map NVL to COALESCE, rewrite DECODE statements as standard CASE expressions, and adjust exception blocks like WHEN NO_DATA_FOUND THEN.

When you run a migration assessment in DMS, Gemini analyzes the source procedure and produces native PostgreSQL PL/pgSQL code:

Source: Oracle PL/SQL

code_block: <ListValue: [StructValue([('code', 'CREATE OR REPLACE PROCEDURE calculate_discount (\r\n p_customer_id IN NUMBER,\r\n p_discount OUT NUMBER\r\n) AS\r\n v_total NUMBER := 0;\r\nBEGIN\r\n SELECT NVL(SUM(amount), 0) INTO v_total\r\n FROM orders WHERE customer_id = p_customer_id;\r\n \r\n p_discount := DECODE(TRUE, v_total > 10000, 0.15, v_total > 5000, 0.10, 0.05);\r\nEXCEPTION\r\n WHEN NO_DATA_FOUND THEN\r\n p_discount := 0;\r\nEND;\r\n/'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f49c55dcb50>)])]>

Target: PostgreSQL PL/pgSQL (Converted by Gemini in DMS)

code_block: <ListValue: [StructValue([('code', 'CREATE OR REPLACE FUNCTION calculate_discount (\r\n p_customer_id NUMERIC,\r\n OUT p_discount NUMERIC\r\n) RETURNS NUMERIC AS $$\r\nDECLARE\r\n v_total NUMERIC := 0;\r\nBEGIN\r\n SELECT COALESCE(SUM(amount), 0) INTO v_total\r\n FROM orders WHERE customer_id = p_customer_id;\r\n\r\n p_discount := CASE\r\n WHEN v_total > 10000 THEN 0.15\r\n WHEN v_total > 5000 THEN 0.10\r\n ELSE 0.05\r\n END;\r\nEND;\r\n$$ LANGUAGE plpgsql;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f49c5f34e90>)])]>

Along with the generated SQL, Gemini provides an inline explanation detailing why NVL was converted to COALESCE and how the Oracle DECODE function was converted into an explicit CASE block in PostgreSQL.

Figure 2: End-to-end database code conversion pipeline powered by Gemini in DMS.

Maintain full schema control and validation

Security, transparency, and code accuracy remain central to database modernization. Gemini in DMS operates strictly within your established Google Cloud security boundaries, keeping your code private to your project.

To ensure reliability, the conversion and validation process follows a structured workflow:

Automatic schema context pulling: When you set up a DMS conversion workspace, the service automatically parses your entire source database metadata—including table schemas, data types, foreign key constraints, and cross-procedure dependencies. Gemini references this project-wide context during code generation, eliminating the need to manually supply dependent object definitions.
Automated syntax and dependency validation: As code is generated, DMS runs a validation parser against target PostgreSQL syntax rules. Objects are assigned validation status indicators (e.g. Converted, Warning, or Action Required) to quickly highlight routines requiring manual review.
Interactive evaluation state: You maintain full control over every schema change. Within the conversion workspace, you can inspect side-by-side diffs, review inline AI explanations, and edit PL/pgSQL code directly before applying changes to your target database.
Staging deployment and verification: Once code passes workspace validation, you can apply the converted schema and functions to a target staging instance (e.g. Cloud SQL or AlloyDB) for functional execution and performance testing prior to production cutover.

Streamlining database modernization

AI-assisted code conversion in Database Migration Service helps database teams convert legacy database logic in days rather than months. Instead of spending precious time rewriting code from scratch, database administrators and application developers can shift their focus to adding new functionality, testing performance, and modernizing applications.

If you’d like some good examples of common Oracle and SQL Server conversion scenarios and how DMS converts them to PostgreSQL, check out our recent video series, Gemini taught me PostgreSQL.

Say goodbye to database migration headaches and let Gemini teach you how to seamlessly convert legacy database logic.

Get started with Database Migration Service

To start your database conversion, launch a migration assessment in the Database Migration Service console (https://console.cloud.google.com/dms) or read our heterogeneous migration guide (https://cloud.google.com/database-migration).

How Target is enhancing retail discovery and cutting database maintenance by 50% with Spanner Graph

Tue, 04 Aug 2026 16:00:00 +0000

In today’s retail environment, shoppers expect highly personalized product discovery experiences and conversational assistance that feels genuine, natural, and genuinely helpful. Today, successful product discovery is about understanding semantic meaning and the rich, connected relationships between products, categories, and guest intent. It is no longer just about keywords and basic browsing.

At Target, this work is handled by our Guest Product Confidence platform team. They are responsible for building the features that establish trust and guide purchasing decisions, such as ratings, reviews, and AI-driven digital shopping assistants. An exciting example of this is our Gift Finder chat agent, which we launched during the 2025 holiday season online and in the Target app to help shoppers discover the perfect items through friendly, conversational dialogue.

To deliver real-time personalization and context-rich semantic responses like these at global scale, we identified a critical architectural need to move away from a fragmented data ecosystem toward a unified data platform. We needed a solution capable of supporting high-throughput transactional workloads, highly connected graph relationships, vector similarity search, and full-text keyword search all at once.

In this post, we’ll explore how we achieved all four with Spanner.

Overcoming fragmented architecture

Previously, Target’s discovery data ecosystem relied on a combination of Elasticsearch clusters for search and inverted indexes, alongside separate NoSQL datastores for our transactional data. While functional, this fragmented architecture presented significant operational and technical challenges.

Disconnected context: Keeping separate search, vector, and transactional databases in perfect sync was a constant challenge. Siloed information led to missing context, disconnected attribute relationships, and inconsistent query results.
High operational overhead: Managing independent clusters, tuning search indexes, and handling complex, custom synchronization and aggregation logic required intensive manual intervention from our engineering teams.
Expansion bottlenecks: Expanding our retail data domains required adding new database collections, maintaining complex joins, and navigating weak transactional guarantees across our discovery and core transactional systems.
Siloed intelligence: We lacked the ability to query graph relationships, vector similarity, and keyword search indexes in a single transaction.

To build the next generation of AI-driven guest experiences, we needed to consolidate on one platform.

Building the enterprise ontology on Spanner Graph

We evaluated multiple specialized technologies, including standalone vector databases and niche graph databases. However, adding more single-purpose databases would have only worsened our operational complexity and data synchronization pipelines.

We ultimately chose Spanner Graph to build our enterprise ontology, which is a "graph-of-graphs" paradigm that allows us to construct a massive, generative AI-powered shopping graph.

By unifying our data, we bring semantic data, graph relationships, vector embeddings, and operational transactions under one roof. This establishes Spanner as our single authoritative source of truth for both transactional state and semantic intelligence.

Our high-level architecture now consists of three core pillars:

1. Enterprise augmentation
This layer captures our enterprise retail catalog, aggregates relevant metadata from multiple backend sources, and utilizes generative AI for agentic data enrichment to dramatically improve the quality and depth of the product data we ingest.

2. Unified graph, vector, and search store
Instead of shifting data across multiple databases, Spanner Graph stores our entity nodes, relationship edges, and vector embeddings in the same database engine. Spanner Graph natively supports multi-hop graph traversals, semantic vector similarity, and full-text keyword queries over our relational tables. Because this multi-model synergy is native, we get strict ACID transactions for absolute correctness across distributed workloads without the need for fragile external sync pipelines.

3. Orchestration and AI layer
This layer powers our conversational guest interfaces, utilizing rich, structured context fed directly from Spanner Graph to ground our LLMs. It extracts highly specific product relationships to power tools like the Gift Finder while governing responsible AI processes and evaluating generated outputs.

A smooth, zero-downtime incremental migration

Transitioning critical search and discovery infrastructure that millions of guests rely on required a cautious, zero-downtime approach. We executed this migration in four structured phases.

Schema and ontology mapping: We defined the specific retail entities, such as products, categories, brands, and guest preferences, and their corresponding relationships within the Spanner Graph schema.
Data integration and parallel replay: We built mutation-based data integrations in a parallel pipeline. This allowed us to continuously replay live transactional updates, apply schema transformations, generate embeddings, and write them directly into Spanner Graph in real-time.
Canary deployment: We gradually shifted live read traffic to the new Spanner Graph-backed platform, validating query performance, semantic accuracy, and database stability under real retail workloads.
Cutover and cleanup: Once performance was thoroughly verified, we fully transitioned all search and discovery traffic to Spanner and deprecated our legacy Elasticsearch stack, entirely removing the maintenance burden of those clusters.

Business impact

By building directly on Spanner Graph, we unlocked measurable technical and business outcomes:

The ultimate GraphRAG foundation: Traditional RAG relies on flat vector similarity, which often misses the structured associations between products, such as matching a toy with its compatible accessories or age-appropriateness. By combining deep graph traversals with semantic vector search in a unified GraphRAG architecture, we grounded our LLMs with highly precise context. This directly improved our recommendation relevancy, enhanced guest satisfaction, and boosted our Net Promoter Score.

Consolidated SQL + GQL interoperability: With Spanner Graph, our developers query structured relational catalog data and connected graph relationships in a single query using standard SQL and GQL (Graph Query Language). This eliminates the need for data duplication, latency, or complex ETL pipelines to bridge these paradigms.

Serverless scalability with zero growth ceiling: Spanner automatically handled massive, unpredictable traffic spikes during peak retail events like Black Friday and Cyber Monday. Spanner's built-in autoscaler dynamically adjusted computing capacity to handle burst traffic during high-intensity, limited-time promotional offers without sacrificing performance.

50% reduction in infrastructure maintenance: By consolidating our transactional NoSQL and search index databases into a single managed Google Cloud service, we eliminated the operational burden of maintaining separate database clusters. Our developers now spend 50% less time on database administration and infrastructure upkeep, allowing us to build and deploy new, customer-facing AI features much faster.

Migrating to Spanner Graph has accelerated our generative AI roadmap, serving as the ultimate proof of what is possible when you build on the right data foundation.

Want to supercharge your AI apps? It starts with databases with the right graph capabilities at virtually unlimited scale. Discover how Spanner Graph can turn data into action for your organization.

Multiple result sets: How Database Migration Service automates SQL server to PostgreSQL translation

Tue, 04 Aug 2026 16:00:00 +0000

In the Medium blog post, "From MARS to SETOF REFCURSOR: Migrating Multi-Result Stored Procedures to PostgreSQL," we explored the fundamental architectural differences between SQL Server and PostgreSQL regarding multiple result sets. We looked at how SQL Server natively streams multiple tabular streams from a single execution, whereas PostgreSQL requires a more deliberate strategy using explicit cursor manipulation.

If you’re facing a massive database migration with hundreds of these procedures, manually rewriting them is a non-starter. This is where automated tools come in. In this post, we’ll explore in detail how Google Cloud’s Database Migration Service (DMS) approaches this exact challenge, the conversion logic it applies under the hood, and how to actually run and test the generated code.

The Core Conversion Strategy of DMS

There are many reasons to migrate your databases to PostgreSQL, including enterprise performance and availability, a thriving developer and user community, and strong AI capabilities. But tricky queries, like those with multiple result sets, can slow down your migration project.

DMS looks at two specific things: How many result sets does the procedure return? and Does it use a scalar RETURN value?

The decision matrix for the translation looks like this:

Scenario	SQL Server Characteristic	PostgreSQL Target Object	Mechanism
Scenario A	1 Result Set OR a Scalar Return Value only	STORED PROCEDURE	Handled natively via an INOUT refcursor parameter or standard variable tracking.
Scenario B	Multiple Result Sets OR a combination of Result Sets + Scalar Return	FUNCTION	Converted to a RETURNS SETOF refcursor block. The scalar return value is appended as its own separate cursor dataset.

Automating Multi-Result Set Conversions: Inside DMS

SQL Server utilizes a tabular data stream protocol that allows multiple results to be transmitted over a single connection execution path without explicit declarations. PostgreSQL, by contrast, relies on a distinct execution protocol where multiple datasets are managed deterministically via cursors. To bridge this structural difference, DMS automates the translation logic.

Consider a baseline healthcare reporting scenario. We have a master procedure (sp_GetPatientSummary) that orchestrates data retrieval for a patient by conditionally calling two child procedures: one for lab results (sp_GetPatientLabResults) and one for clinical visits (sp_GetPatientDoctorVisits).

Depending on conditional logic and procedural execution paths, a single execution can return up to four distinct result sets plus a status integer indicating whether the patient was found.

SQL

code_block: <ListValue: [StructValue([('code', 'CREATE OR ALTER PROCEDURE sp_GetPatientSummary\r\n @PatientID INT\r\nAS\r\nBEGIN\r\n SET NOCOUNT ON;\r\n DECLARE @LabReturnValue INT;\r\n\r\n IF NOT EXISTS (SELECT 1 FROM Patient WHERE PatientID = @PatientID) \r\n RETURN 0; \r\n\r\n SELECT PatientID, FirstName, LastName, DateOfBirth FROM Patient WHERE PatientID = @PatientID;\r\n\r\n EXEC @LabReturnValue = sp_GetPatientLabResults @PatientID = @PatientID, @QueryType = 1;\r\n\r\n-- Conditional: If child returned 1, call it again for Result Set #3\r\n IF @LabReturnValue = 1\r\n EXEC sp_GetPatientLabResults @PatientID = @PatientID, @QueryType = 2;\r\n\r\n EXEC sp_GetPatientDoctorVisits @PatientID = @PatientID;\r\n \r\n RETURN 1; -- Success scalar status\r\nEND;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f49c55e3910>)])]>

The Target: PostgreSQL’s Deterministic Cursors

To align with PostgreSQL's execution model, DMS maps the original T-SQL behavior into a structural PL/pgSQL architecture using SETOF refcursor and explicit cursor management.

1. Simple Tracking: Stored Procedure Translation

For the child procedure sp_getpatientdoctorvisits, which yields exactly one result set, DMS creates a standard PostgreSQL PROCEDURE utilizing an explicit INOUT refcursor parameter to safely pass the pointer back to the caller.

SQL

code_block: <ListValue: [StructValue([('code', "CREATE PROCEDURE dbo.sp_getpatientdoctorvisits(\r\n _patientid INTEGER, \r\n INOUT result_set_refcursor refcursor\r\n)\r\nLANGUAGE plpgsql\r\nAS $$\r\nBEGIN\r\n sp_getpatientdoctorvisits.result_set_refcursor := NULL;\r\n OPEN result_set_refcursor FOR \r\n SELECT visitid, patientid, doctorname, visitdate, visitnotes\r\n FROM dbo.doctorvisits\r\n WHERE doctorvisits.patientid = sp_getpatientdoctorvisits._patientid\r\n AND doctorvisits.visitdate >= localtimestamp + interval '-6 MON';\r\nEND;\r\n$$;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f49c4b7b3d0>)])]>

2. Multi-Set Tracking: Function Translation

For the master routine and the complex lab child routine, an INOUT parameter isn't enough to capture the varying arrays of output. DMS transforms these into PL/pgSQL Functions returning a SETOF refcursor.

Take note of how the translated sp_getpatientlabresults builds its cursors sequentially and, at the very end, dynamically opens a distinct cursor explicitly named "return_value" to pass the scalar integer back to the execution stack:

SQL

code_block: <ListValue: [StructValue([('code', "CREATE OR REPLACE FUNCTION dbo.sp_getpatientlabresults(_patientid INTEGER, _querytype INTEGER)\r\nRETURNS SETOF refcursor\r\nLANGUAGE plpgsql\r\nAS $$\r\nDECLARE rc refcursor; _recordsfound INTEGER := 0;\r\nBEGIN\r\n IF _querytype IN (1, 3) THEN\r\n OPEN rc FOR \r\n SELECT resultid, patientid, testname, testdate, isexception\r\n FROM dbo.labresults\r\n WHERE patientid = _patientid \r\n AND testdate >= localtimestamp - interval '6 months';\r\n RETURN NEXT rc;\r\n\r\n IF EXISTS (\r\n SELECT 1 FROM dbo.labresults \r\n WHERE patientid = _patientid \r\n AND testdate >= localtimestamp - interval '6 months'\r\n ) THEN recordsfound := 1; \r\n END IF;\r\n END IF;\r\n\r\n IF _querytype IN (2, 3) THEN \r\n rc := NULL; -- Reset to generate a new unique cursor name\r\n OPEN rc FOR \r\n SELECT resultid, patientid, testname, testdate, isexception\r\n FROM dbo.labresults\r\n WHERE patientid = _patientid \r\n AND isexception = TRUE \r\n AND testdate >= localtimestamp - interval '2 years';\r\n RETURN NEXT rc;\r\n\r\n IF EXISTS (\r\n SELECT 1 FROM dbo.labresults \r\n WHERE patientid = _patientid \r\n AND isexception = TRUE \r\n AND testdate >= localtimestamp - interval '2 years'\r\n ) THEN \r\n _recordsfound := 1; \r\n END IF;\r\n END IF;\r\n\r\n rc := 'return_value';\r\n OPEN rc FOR SELECT _recordsfound AS return_value;\r\n RETURN NEXT rc;\r\nEND;\r\n$$;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f49c4b78850>)])]>

Parsing the Output Programmatically

When integrating these migrated routines back into your application data access layer, QA and application engineers need to adapt how they process execution results. Instead of reading standard tabular rows sequentially, the calling application or test harness receives an array of cursor references. To handle this programmatically, your application must fetch the data from each returned portal sequentially. DMS isolates the scalar return value by placing it inside its own dedicated, explicitly named "return_value" cursor dataset at the very end of the execution stack. Preparing your development teams for this structural mapping ensures that your application logic and validation scripts can accurately parse the multi-layered response arrays without disruption.

Executing and Testing the Migrated Code

Testing these migrated objects in PostgreSQL requires working within explicit transaction blocks. Because PostgreSQL cursors are bound to the transaction lifecycle, accessing the data from the memory portals requires encapsulating the execution and the data retrieval commands within a single BEGIN ... COMMIT block.

Here is how you execute and fetch the entire complex dataset for Patient 3 inside PostgreSQL:

SQL

code_block: <ListValue: [StructValue([('code', 'BEGIN;\r\n\r\n-- 1. Exec the function to initialize and stream back the cursors\r\nSELECT * FROM dbo.sp_getpatientsummary(3) AS summary_cursors; \r\n\r\n-- 2. Fetch data from the sequentially generated anonymous portals\r\nFETCH ALL FROM "<unnamed portal 1>"; -- Patient Demographics\r\nFETCH ALL FROM "<unnamed portal 2>"; -- Lab Results (Query Type 1)\r\nFETCH ALL FROM "<unnamed portal 3>"; -- Lab Results (Query Type 2)\r\nFETCH ALL FROM "<unnamed portal 4>"; -- Doctor Visits\r\n\r\n-- 3. Fetch the named scalar return value tracking cursor\r\nFETCH ALL FROM "return_value";\r\n\r\nCOMMIT;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f49c4b78dd0>)])]>

DMS Insight: The Mechanics of Result Set Counting

The DMS product employs a sophisticated pre-processing mechanism to understand the expected result set count and the possible existence of a return value. The ultimate goal is to map every SQL Server procedure into one of three distinct categories: no result sets, a single result set, or multiple/dynamic result set counts.

To achieve this accurately, DMS performs a deep structural analysis:

Direct Result Sets: First, the engine scans the procedure's body to count the direct result sets, which are the explicit SELECT statements executed directly within it.
Dynamic Considerations: Special consideration must be given when dealing with looped or conditional SELECT/EXEC statements. Because these constructs inherently mean the number of returned result sets can differ between executions, the count for that procedure is immediately marked as "dynamic".
Building the Call Network: When the engine encounters references to other stored procedures, it does not yet know how many result sets those child procedures expose. To solve this, DMS builds a comprehensive directed graph to model the entire call hierarchy between procedures.
DFS Propagation: Only once this directed network is fully built can the engine run a Depth First Search (DFS) algorithm. This DFS traversal systematically propagates the result set counts—whether fixed integers or dynamic flags—back up the call chain to the top-level procedures.

Consequently, the generated code is accurately modified to support both returning the outer datasets and allowing child routines to be invoked correctly within nested execution stacks. This graph-based approach guarantees accuracy and seamlessly supports highly complex inter-procedural interactions, including both direct and indirect recursions.

Summary

Google Cloud’s Database Migration Service takes the guesswork out of structural transformations by programmatically applying the FUNCTION vs. PROCEDURE decision tree based on result set counts. While it completely preserves your core business and conditional execution logic, it does change how application connection pools and QA engineers interact with execution results.

Understanding this automated architecture ensures you can effectively map out your validation scripts and configure your data access layers for seamless day-two operations in PostgreSQL.

Now that you learned how to convert multiple result sets, it's time to start a PostgreSQL database on Google Cloud. Please let us know how your journey was.

Get started with a Database Migration Service with the Google Cloud $300 free credits. Start building for free.

Introducing Database Operations Agents: The future of autonomous database management

Tue, 04 Aug 2026 16:00:00 +0000

As part of the Agentic Data Cloud launch at Google Cloud Next ‘26, we announced two AI-powered database agents to simplify database management. These include the Database Onboarding Agent for Day 0 operations — setup, configuration, and initial deployment — as well as the Database Observability Agent for Day 1 and 2 operations, including monitoring, troubleshooting, and ongoing maintenance.

These agents are always on, informed by Google’s years of experience, and integrated across Google surfaces such as Chat, CLI, the Google Cloud console, Managed Context Protocol (MCP) servers, and third-party tools — including your preferred integrated development environment (IDE), so you get help where and when you need it.

Traditionally, managing and creating databases has involved a combination of manual architecture planning, custom scripts, and distinct tools. Teams handle database provisioning, schema design, index configuration, and query tuning, alongside performance monitoring—often cycling through repeated testing and optimization cycles as application demands change. Although this method is functional, it demands substantial technical skill and continuous attention throughout the entire database lifecycle. For example, developers often fear making an update that may limit their ability to scale the system later. Similarly, when an application slows down, finding the exact query or resource constraint causing the issue can take hours of manual investigation and troubleshooting.

Intelligent AI-powered agents can simplify database lifecycle management by automating many of these tasks such as recommending the right database type for the workload, detecting anomalies, recommending the right configurations, optimizing queries, and providing actionable insights to improve operational efficiency. By embedding these capabilities directly into workflows where you need them, agents help organizations build, operate, and optimize databases more efficiently while reducing operational overhead.

Let’s take a closer look at these new database agents.

Database Observability Agent: From diagnosis to remediation

The Observability Agent empowers Site Reliability Engineers (SREs), DevOps pros, DBAs and developers to diagnose complex issues and remediate them using simple natural language prompts.

As your operations scale, identifying subtle issues like query hotspots or lock contention becomes an expensive burden. The database observability agent uses Google’s operational expertise and the reasoning capabilities of Gemini to solve these challenges. By automatically connecting telemetry across multiple sources including Database Insights, Cloud Monitoring, Cloud Logging, and Cloud Trace the agent provides a clear root cause analysis in minutes.

Beyond just identifying the "why," the agent suggests recommended actions to fix the issues found, and can execute validated actions with your approval. For example, if it detects a bottleneck, it might suggest you "Enable connection pooling for Cloud SQL instance," providing the rationale and expected impact before you commit to the change. Some capabilities include:

Fleet-level troubleshooting: The Observability Agent is integrated with Database Center so you can use Gemini Chat to ask complex fleet-wide questions like, "Which databases in my fleet consumed the most CPU in the last 7 days?" to receive a summarized analysis across your entire fleet.
In-product investigations: The agent correlates complex telemetry across Database telemetry, Cloud Monitoring, Cloud Logging, Cloud Trace, and multiple other data sources to pinpoint issues like latency spikes or lock contention. (In preview with select customers)
Validated remediations: Instead of just identifying problems, the agent provides crisp recommendations and can execute validated actions with your approval, such as adding indexes for a Cloud SQL instance. (In preview with select customers)
MCP tools: The Observability Agent derives insights with the help of tools such as system metrics, query metrics, fleet inventory, and issues, which are also available as MCP tools via the Database Insights MCP Server and Database Center MCP Server.

Integration that fits your workflow

You can access these Database Observability Agent capabilities directly within your existing database management processes. The agent powers several experiences, including:

Cloud Assist chat: Ask questions in natural language, for example, "What is the CPU utilization trend for my top Cloud SQL instances?" to get a summarized analysis complete with charts. Then, within the Chat window, you can start an investigation for any issues found,and get a root-cause analysis and remediations.

In-product investigations: Use Gemini Cloud Assist to investigate and remediate issues in-context on relevant database pages from the console.

Developer tools: Consume the agent’s capabilities through Antigravity or an IDE of your choice. This is augmented by the rich set of observability MCP tools that Google provides. All of these tools are available on Google Remote MCP servers. Combining them together is like giving developers a virtual DBA to optimize their databases, but all within their IDEs.

Supports multiple managed databases

You can use Observability Agent to get answers to your database queries, to access any database metric instantaneously, or to leverage AI-powered diagnosis to resolve complex problems. The agent covers a broad set of issues across a variety of Google Cloud databases, including:

Cloud SQL: Troubleshoot and optimize your database instance load, query performance or connectivity issues for all Cloud SQL database engines. For Cloud SQL for PostgreSQL, leverage the agent to troubleshoot common database issues.

Similarly, the agent helps you identify issues, find their root cause, and take remediation actions for other supported databases and issue types.

Spanner: Here, the most common troubleshooting scenario involves optimizing read and write latencies. The agent helps you do that in minutes, covering a broad set of scenarios ranging from hotspots to lock contentions.
AlloyDB: Troubleshoot and optimize your database instance load, query performance or replica lag issues.
Bigtable: Diagnose and optimize your read and write latencies, complete with crisp, actionable recommendations.

Database Onboarding Agent

The new Database Onboarding Agent is your active partner during the database selection process. Instead of spending hours reading documentation, you can describe your application requirements to the agent in natural language. The agent understands technical metrics like IOPS, latency limits, and replication lag, so it can provide a sound recommendation. You can access the Database Onboarding Agent’s capabilities directly within the Gemini chat interface. With the Database Onboarding Agent, you get:

Recommends database solutions: Analyzes user requirements regarding workload performance, scale, data type, and reliability to suggest optimal Google Cloud Managed Database services (e.g., Cloud SQL, Spanner, AlloyDB).
Smart recommendations: The agent reflects your requirements back to you, such as recommending AlloyDB for a high availability configuration, helping you have confidence in its selections.
Streamlined configuration: Once you choose a service, the agent generates the required commands. You can then use these commands to provision your database instance, configure the correct features, and deploy it.

Get started

The Database Observability and Onboarding Agent’s capabilities are available for a wide range of services, including AlloyDB, Bigtable, Cloud SQL (PostgreSQL, MySQL, SQL Server), Firestore, Memorystore, and Spanner. These agents are currently available via Gemini Cloud Assist. Explore AI assisted troubleshooting and Gemini Chat for AlloyDB, Cloud SQL, Spanner, and Visit Gemini Cloud Assist page to learn more.

Unifying public and private data: Scale knowledge graphs with Data Commons on Spanner

Mon, 03 Aug 2026 16:00:00 +0000

To make informed decisions, businesses often need to connect their internal data with public reference data, to create a knowledge graph that connects real-world things and their relationships. However, bridging data from public and private worlds has traditionally been complex. Today, we are streamlining these connections with the general availability of Data Commons on Spanner Graph and the preview of the new Data Commons Platform to unify your private knowledge with knowledge graphs from public datasets.

The overarching Data Commons project supports Google’s mission to organize the world's information and make it universally accessible and useful. Data Commons unifies fragmented public datasets from over 100 authoritative providers, including the United Nations, World Bank, US Census Bureau, Eurostat, WHO, and NOAA, with over 400 billion data points structured using standardized Schema.org definitions. Data Commons provides data exploration tools, MCP tools, and cloud-based APIs to access and integrate the clean datasets.

Data Commons integrates public information across multiple domains, including agriculture, demographics, economy, environment, and health. This standardized approach unlocks powerful use cases, for instance, letting you analyze national GDP trends, map regional smoke pollution levels, track local health equity, or demographic distributions over time, all using data that has already been preprocessed and normalized for you.

Data Commons knowledge graph dimensions

Dimension	Size	Technical description
Statistical observations	400+ billion	Individual metric data points
Graph edges	2.6+ billion	Relationships
Knowledge graph nodes	1.7+ billion	Standardized entities
Data sources	100+ providers	Authoritative institutions

^{Data Commons makes meaningful quantities of public administrative data available to users on readily consumable cloud-based infrastructure.}

A modern infrastructure powered by Spanner Graph

When we first built Data Commons, our goal was to aggregate massive, disparate public datasets using the tools available at the time. The platform relied on Bigtable as a caching layer, which was an effective strategy for handling large-scale lookups in the absence of native graph database technology.

Today, we have transitioned our architecture to a native graph model with Spanner Graph, which brings the convenience of a SQL-like interface and graph expressiveness to Spanner, with its high availability, horizontal scale-out, multi-region transactional consistency, and native ISO/IEC 39075 Graph Query Language (GQL) support.

By adopting a multi-entity Spanner Graph schema, we represent entities as nodes and their domain links as dynamic graph edges, allowing us to move away from pre-computed cache structures and perform complex relationship queries directly within the database using GQL.

This architecture also simplifies our pipelines by removing the need for complex, pre-computed indices that require costly in-memory rebuilds and multiple snapshots. Spanner Graph enables incremental updates to specific datasets without refreshing the entire database, while stale reads maintain consistent data snapshots during ingestion.

Key benefits by moving to Spanner Graph

Unified storage and incremental updates: By utilizing Spanner Graph’s multi-entity schema, the platform replaces complex caches with a model that supports incremental data imports, allowing for targeted updates to specific datasets.
Dynamic graph traversals via GraphRAG: The system executes multi-hop queries such as navigating hierarchies like continent → country → state → county → city on the fly. This removes reliance on static caches and enables GraphRAG workflows, where the database maps natural language queries directly to structured path-matching traversals.
Consistent data snapshots: Leveraging Spanner TrueTime and stale reads, the platform provides you with a version-consistent snapshot of data, maintaining integrity across distributed nodes following batch ingestion cycles.
Operational analytics at scale: Spanner’s columnar engine efficiently scans massive time-series datasets by reading only the necessary fields, while BigQuery federation that leverages Spanner’s Data Boost technology performs complex aggregations via EXTERNAL_QUERY in an isolated environment, helping isolate production traffic.

Bridging systems with SDMX 3.0 interoperability

To facilitate the use of complex statistical data, Data Commons adopts a lean implementation of Statistical Data and Metadata eXchange (SDMX) technical standard. As an ISO specification, SDMX provides a consistent approach for describing and exchanging statistical data along with descriptive statistical meta-information.

In this Data Commons Platform update we added support for the SDMX technical standard version 3.0, providing out-of-the-box integration with third-party tools like Tableau, Flourish, and Observable for multi-dimensional datasets. This is made possible using the API standard SDMX-JSON and SDMX-CSV 2.0 formats across two high-value endpoints:

The availability API: A programmatic discovery mechanism to identify existing dimensions, variables, and date ranges without reading raw values.
The data API: Retrieves actual observations and metadata, using named parameters to help prevent code from breaking when dimensions are added.

Transforming private instances of Data Commons Platform

For organizations that want to build private instances of the Data Commons Platform, this new modern architecture resolves legacy scaling limits and simplifies data schematization. Developers can instantiate a private instance of the Data Commons Platform leveraging the same scalable technology that powers Google’s Data Commons instance. As a private instance, users retain full control of their own data and have the ability to limit access, while enabling natural language queries to blend results from their private data with Google’s public data that is hosted on the Google Data Commons instance. By federating across our public knowledge graph and a private knowledge graph containing your own data, you can light up exciting new use cases, while maintaining data isolation and ensuring no data duplication.

For instance, a retail enterprise can combine public data such as national GDP trends, regional demographic breakdowns, and employment statistics, with their own enterprise data, including sales histories, store performance metrics, and supply chain logistics. This allows analysts to contrast public macroeconomic indicators against their own company transactions to optimize merchandise distribution and identify untapped markets.

Example of a natural language query combining statistical data from the Directorate General of Commercial Intelligence and Statistics (DGCIS) stored in a Data Commons Platform private instance with World Development Indicators from the World Bank stored in the Google Data Commons public instance.

A user is querying a Data Agent for average annual temperature trends in the country. The agent retrieves information from Data Commons, explaining that while historical data is available, it provides projected temperature changes, climate drivers, and CMIP6 climate model scenarios (SSPs), with options to export the generated report.

A user asks the Data Agent to compare the Worker Population Ratio (WPR) of rural versus urban males in a country. Fetching data from Data Commons, the agent defines WPR—the percentage of workers relative to the total population—and outlines the available demographic variables to analyze and compare both groups.

Get started today

Explore Data Commons: Visit datacommons.org to query global statistical knowledge.
Explore Spanner Graph's use cases and setup guide for your knowledge graphs.
Deploy Data Commons Platform: contact support@datacommons.org to request preview access and to review the developer tools.

AlloyDB adds group authentication to secure enterprise scale and AI agents

Thu, 30 Jul 2026 16:00:00 +0000

Database security traditionally relies on a fragile balance between the granular control developers need and the administrative overhead of managing thousands of individual database passwords. Between managing AI agent access, rotating static credentials, handling employee on-boarding and off-boarding, and auditing access logs, passwords remain an operational tax — and a potential security vulnerability.

At Google Cloud, our goal is to help make database access transparent, secure, and passwordless.

Today, we are taking an important step forward in that journey. We’re announcing Identity and Access Management (IAM) group authentication for AlloyDB, available in preview. This capability brings identity-driven access control to your enterprise workloads.

Cloud SQL customers have already adopted this authentication pattern with great success, and this launch unifies our security stance across both services.

The problem with individual scale

For years, both Cloud SQL and AlloyDB have mapped individual Google Cloud identities directly to database users using native IAM authentication. However, at enterprise scale, managing access on an individual basis can introduce significant complexity.

Without group-based management, scaling to hundreds of instances and thousands of users creates distinct challenges:

On-boarding bottlenecks: Every new team member requires individual database user provisioning.
Off-boarding risks: Ensuring an employee’s access is entirely removed across a distributed database environment can complicate auditing.
Policy drift: Maintaining identical permissions across development, staging, and production systems becomes highly error-prone.

Faced with these challenges, it is tempting to use a single, powerful user or service account to serve a whole application. However, oversimplifying access for such powerful application accounts comes at the cost of risk exposure and loss of granular auditing capabilities.

Securing the future of agentic AI

The scale challenge isn't just about human users anymore. As organizations deploy an increasing number of AI agents, managing database identity and access controls will become more complex.

If an AI agent connects to a database using a generic, shared account, or a broad service account, it risks acting as a confused deputy. When using credentials with overly-powerful permissions instead of carrying through the user’s identity, an agent could access or modify data beyond what the end user requesting the action is authorized to see. Crucially, it can hide individual accountability from audit logs because actions map to a generic service account.

Granular authentication can help mitigate this risk. Agents can pass the end user’s specific identity and authentication scope through to the database layer so that queries are run on behalf of the user, limiting data access to objects that an end user is allowed to.

Passing user group identity through an AI agent to AlloyDB allows the database to authorize access and record precise audit trails.

IAM group authentication simplifies this architecture. Instead of managing micro-permissions for every combination of agent and user, security teams can define up to 200 functional Google Groups (such as financial-agents@company.com or regional-analysts@company.com). Google Cloud’s managed database infrastructure validates the user's group context, helping ensure the database authorizes data access at the database or table level while audit logs capture exactly what data was accessed, modified, and on whose behalf.

Proven value for digital leaders

Enterprises are already improving their operational velocity by adopting centralized identity principles. Bilt, a leading platform rewards program, uses our unified approach to help enhance the security of its high-scale database environments:

"By combining AlloyDB’s group-based IAM with our automated group management and Privileged Access Manager (PAM) entitlements, we've eliminated the risk of shared credentials entirely. Database and role provisioning are now fully templated from day one, allowing our engineers to securely access only the data they need and exactly when they need it," said Kosta Krauth, CTO, Bilt.

A unified blueprint for passwordless access

With this launch, Google Cloud provides a unified approach for access control across both Cloud SQL and AlloyDB. Organizations can now enforce a standardized, defense-in-depth access strategy across their relational database portfolio.

By pairing IAM group authentication with features like VPC Service Controls, Organization Policies, IAM conditions, and Private Service Connect, security teams can help ensure that database access — whether by a human engineer or an autonomous AI agent — is bound to verified corporate identities and secure network perimeters.

Moving toward a Zero Trust database future

Security shouldn't force a trade-off between engineering velocity and compliance. By integrating AlloyDB with Cloud Identity and Workforce Identity Federation, we are removing the friction of database administration while helping you implement a more secure architecture.

You can explore how to set up group-based database roles by trying out the feature today. You can find more recommendations and best practices for security and compliance in our documentation.

Your AI agents are ready. Is your data?

Thu, 23 Jul 2026 16:00:00 +0000

What’s one of the biggest bottlenecks stopping organizations from scaling their AI initiatives? It isn’t the capabilities of today’s models — it’s their access to business context and semantic meaning.

In the agentic era, enterprises need to go beyond simply storing data to activating it with trusted context, moving from passive systems of record to proactive systems of action.

But AI agents operate with nonlinear speed; for example, a single prompt can trigger the agent to independently browse, query, and execute across multiple systems, placing stress on the underlying infrastructure. If the compute, networking, and storage layers aren't optimized for agentic AI, the data platform sitting on top of them will buckle.

It’s no wonder that, according to our State of infrastructure report, 83% of organizations believe they require infrastructure upgrades to support production-grade agentic AI systems.

To solve this problem, we introduced the Agentic Data Cloud at Google Cloud Next 2026; unifying your data, AI models, and operational databases into a single System of Action. To make an Agentic Data Cloud work, it must be AI-native from the chip to the model. The underlying infrastructure must be able to accommodate agentic load.

Google’s Agentic Data Cloud

Let’s explore how the right infrastructure foundation empowers an Agentic Data Cloud to solve the biggest data challenges organizations face today.

Overcoming a lack of context

To be effective, agentic systems require access to context that is often found in fragmented data systems and legacy architectures. This can make it hard for agents to get this context, leading to incomplete, inaccurate results. In fact, our report found that 43% of IT leaders cite “difficulty integrating with legacy APIs and data sources” as their biggest agentic AI infrastructure gap.

But organizations cannot simply move massive datasets and connect them to AI without increasing complexity and cost.

Our Agentic Data Cloud solves this by leveraging a borderless Lakehouse running on open, flexible infrastructure. By accessing powerful native engines like BigQuery and Spanner over open standards (Apache Spark, Apache Iceberg), agents can read, reason over, and activate data across environments as if it were local, bypassing the latency and costs of traditional setups.

Escaping unnecessary manual work

Scaling agents on a patchwork of disconnected systems can create significant bottlenecks. In our research, 81% of leaders called out operational complexity and engineering overhead as top unforeseen expenses when scaling AI, citing the time engineers spend doing manual work to patch together AI agents across disparate systems.

To move from thinking to doing, agents must be able to connect real-time data across both analytical and operational sources. This requires vertical integration. When an Agentic Data Cloud is built on an AI-native infrastructure where the models, data systems, and underlying accelerators are co-designed, there are fewer network hops and tooling is better integrated. This unified system allows an agent to reach an insight and trigger secure transactions without the typical engineering overhead.

Bringing trust and knowledge to the data

It’s not enough for agents to just discover and query data. To take safe, accurate actions, agents also need rich context and business logic. Yet, 36% of leaders cite a lack of specialized, high-throughput vector databases used for AI model grounding, as a key infrastructure gap, hindering their ability to give agents context.

In order to work to their full potential, agents need a foundation which is built to read and write data systems in real-time, including legacy ERPs and third-party CRMs. It also gives them the long-term memory to recall a user’s preference from, say, three weeks ago, while executing a complex task today. And without this real-time automation, agents have to re-process data for every single query.

To provide context for AI, organizations are using Knowledge Catalog to aggregate and enrich data in their data lakes, and enable agentic searches. By extracting meaning from unstructured data and automatically generating semantics, the catalog acts as an active reasoning layer. That catalog in turn, must be backed by high-throughput infrastructure, so that agents can retrieve the right context.

The path forward

To turn AI into a true competitive advantage, it’s time to build a connected, active data ecosystem. Giving your agents seamless access to all of your data is a must to move from pilots to production, and this must be supported by an infrastructure that can handle the demands of the agentic era. The winners in 2026 and beyond won’t necessarily be the ones with the smartest agents. They’ll be the ones who can feed those agents the right knowledge — securely, cost-effectively, and at scale. Is your data ready for the agentic era?

See how leaders are taking an AI-optimized approach to architecture in the State of infrastructure in the agentic AI era report.

Supercharging pgvector: 4x faster HNSW vector search with AlloyDB

Tue, 21 Jul 2026 16:00:00 +0000

AlloyDB is a fully managed, PostgreSQL-compatible database service built for your most demanding enterprise workloads. It combines the best of open source PostgreSQL with Google’s advanced technology, offering massive scalability, high availability, and native AI capabilities. It serves as a performant relational store, a unified backend for vector and full text search, and an analytics engine that is up to 100x faster than standard PostgreSQL.

Vector search is the foundation of modern AI and Retrieval Augmented Generation (RAG) applications. For developers using AlloyDB and other PostgreSQL databases, pgvector is a widely adopted extension for storing, indexing, and querying vector embeddings, and HNSW (Hierarchical Navigable Small World) is a highly efficient graph-based algorithm designed for approximate nearest neighbor search across multi-layered structures. With columnar engine accelerated HNSW in AlloyDB (now in preview), you can achieve up to 4x higher queries per second (QPS) for vector search compared to standard PostgreSQL HNSW.

Enterprise AI applications face a constant trade-off between speed and accuracy. When searching through millions or billions of vectors, maximizing Queries per Second (QPS) without sacrificing search quality (recall) is critical for scaling production workloads. The PostgreSQL pgvector extension offers HNSW as one of the indexes that can speed up Approximate Nearest Neighbor (ANN) searches. Let’s dive deep into how AlloyDB solves the speed vs. accuracy trade-off.

Note: While this post focuses on HNSW performance, it’s worth noting that HNSW is just one part of AlloyDB’s advanced vector toolkit. AlloyDB also features ScaNN—a cutting-edge index backed by over 14 years of Google Research—giving you the flexibility to choose the perfect index for your workload. Additionally, for use cases demanding absolute precision, standard k-nearest neighbor (KNN) search is always available for 100% recall. Check out our Choose a Vector Index Guide to see how they stack up.

aside_block: <ListValue: [StructValue([('title', 'Get started with a 30-day AlloyDB free trial instance'), ('body', <wagtail.rich_text.RichText object at 0x7f49c4b7afd0>), ('btn_text', 'Start building for free'), ('href', 'http://goo.gle/try_alloydb'), ('image', None)])]>

First, what is the AlloyDB columnar engine?

The AlloyDB columnar engine is a built-in, in-memory cache that automatically stores frequently queried data in a specialized, scan-optimized columnar format. It allows AlloyDB to handle heavy analytical queries up to 100x faster than standard PostgreSQL. Additionally, it accelerates ANN searches by storing the index in memory, using a vectorized memory layout for fast traversals, and bypassing standard PostgreSQL buffer manager overhead.

Performance visualization

To understand the real-world performance characteristics of columnar engine Accelerated HNSW, we plotted standard QPS vs Recall curves for the GloVe 100 Angular dataset by searching more than 1M records with a limit of 100.

Running this benchmark script yields the following visualization:

Note: These measurements were taken on an AlloyDB C4A 16vCPU machine. Due to the inherent randomness in HNSW graph building, results may slightly vary across runs.

The data reveals two transformative benefits:

Massive performance throughput gains: For any given target recall (e.g. 0.95), QPS is increased by approximately 4.2x to 4.9x. This allows you to handle significantly more concurrent vector searches on the same hardware.
Significant recall (accuracy) improvement: Conversely, at a fixed QPS level, columnar engine accelerated HNSW provides a substantial boost in recall. For example, we saw that at ~350 QPS (in the above chart), enabling the columnar engine improves recall from roughly 0.78 to over 0.94 – a 0.163 recall gain. This means your AI applications get much more accurate results without any latency impact.

It is important to note that the baseline (blue line) already represents the index being fully cached in the PostgreSQL shared buffer cache. The performance gains shown here are not the result of moving data from disk to RAM, but rather the result of a more efficient memory architecture.

How it works: Columnar engine Accelerated HNSW

In standard PostgreSQL architectures, index operations utilize the shared buffer cache. Even when data is fully in-memory, the database still incurs significant overhead from the buffer manager, which must handle operations such as page pinning and unpinning, lock acquisition, buffer table lookups, and Least Recently Used (LRU) management.

AlloyDB's columnar engine is a built-in, in-memory cache that stores data in a specialized, scan-optimized format.

With this release, AlloyDB can use columnar engine accelerated HNSW to:

Pin the index: The pgvector HNSW index is pinned (kept persistently in-memory to ensure fast access) directly into the columnar engine’s memory.
Vectorized access: It utilizes a memory layout specifically designed for the high-concurrency, pointer-heavy traversals required by HNSW graphs.
Bypass buffer overhead: By navigating the graph in a specialized memory space, AlloyDB avoids the standard buffer manager bottlenecks. This architectural shift is what enables the dramatic QPS and recall improvements shown above, even when comparing against a fully-cached standard index.

Why it Matters

For enterprise-scale applications, this isn't just about a faster database—it's about cost and quality:

Reduced infrastructure costs: Achieve the same performance with significantly lower compute resources.
Better AI accuracy: Reach higher recall and quality at speeds that were previously only possible for "draft" (high-speed, lower-accuracy results) quality search.
No application changes required: Because this is built into AlloyDB, you get these gains using the same standard pgvector SQL syntax.

Note that the columnar engine does utilize memory, but it is highly compressed and meticulously managed. Because the engine stores vector data in an efficient columnar format, the memory footprint is minimal compared to the massive performance gains—making it a highly favorable trade-off for enterprise workloads.

Quick Start Guide

To try out columnar engine accelerated HNSW in AlloyDB, follow these steps:

1. Enable the columnar engine and index caching

Ensure that both google_columnar_engine.enabled and google_columnar_engine.enable_index_caching flags are set to on for your AlloyDB instance.

2. Add the HNSW Index to columnar engine

Once your HNSW index is created via pgvector, execute the following SQL command to cache it in the columnar engine:

code_block: <ListValue: [StructValue([('code', "SELECT google_columnar_engine_add_index('<hnsw_index_name>');"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f49c55461d0>)])]>

3. Additional Resources

New to AlloyDB? Discover AlloyDB with a 30-day free trial.
Google Colab Notebook: An end-to-end Python script to ingest the GloVe dataset, create indexes, and plot Recall vs QPS curves.
Is HNSW the right vector index choice for your use case? Check our ‘Choose a vector index in AlloyDB AI’ guide.

Databases

A new, no-compromises database architecture for the agentic era

Three tenets needed for a truly agentic database architecture

AlloyDB's agentic architecture

Run agents against production data at any scale by joining the preview of AlloyDB PostgreSQL for agents. You can learn more about its full capabilities in the companion announcement blog.

Why existing architectures can’t satisfy all three tenets

Independent replicas (shared-nothing storage)

Disaggregated shared-storage servers

Object storage with shared-block servers

Evaluating existing architectures

Assessing against the Tenets

How we engineered AlloyDB across the stack

Storage: Colossus as the foundation

Network: Scalable bandwidth with Jupiter

Compute: Elastic and serverless PostgreSQL and analytics

Evaluating AlloyDB’s agentic database architecture

Engineering all three tenets by design

Give agents live data without impacting production

AlloyDB delivers PostgreSQL for agents: Real-time data at agent scale, with full workload isolation

Why this matters

AlloyDB’s unique agentic PostgreSQL architecture

Availability

Announcing Native BM25 Ranking in AlloyDB and Cloud SQL

Why pg_textsearch?

Full-text search example

Index creation

AlloyDB hybrid search example

Vector index creation

Hybrid search

Cloud SQL hybrid search example

Vector index creation

Hybrid search

Watch it in action

Relevant resources

How a solo founder runs a five-continent tender platform on AlloyDB and MCP

Executive summary

Making tender intelligence work as a company of one

Consolidating systems into AlloyDB

Accelerating semantic search by 47x

Automating database operations with MCP

Looking ahead

M4N VM family, now GA: Highest per-core IOPS and throughput for I/O and memory-bound workloads

Built for demanding workloads

Optimizing Oracle licensing costs

What customers are saying

What’s new: Scaling extreme data layers with M4N

1. Enabling high bandwidth data transfer

2. Dynamic storage performance with Hyperdisk

M4N machine types and specifications

Get started today

Scaling Telco Autonomy: Leveraging GNNs with Distributed GraphFlow

Foundation: Digital Twin on Spanner Graph

The ML layer: Distributed Graph Flow (DGF)

Use cases

Scenario: Root cause analysis with GNNs and DGF

Get started today

What’s new with Google Data Cloud

September 7 - September 10

August 31 - September 4

July 6 - July 10

June 1 - June 5

May 11 - May 15

April 20 - April 24

April 13 - April 17

April 6 - April 10

March 23 - March 27

March 16 - March 20

February 23 - February 27

February 16 - February 20

February 2 - February 6

January 26 - January 30

January 19 - January 23

January 12 - January 16

What’s new with Google Data Cloud - 2025

Enterprise-grade PostgreSQL with AlloyDB Omni RPM Orchestrator is generally available

Why run a self-managed database?

Flexible Reference Architectures

High Availability and Read Scalability

Data Protection and Security

Database Operations