Spanner

Serving data from Iceberg lakehouses fast and fresh with Spanner columnar engine

Thu, 26 Feb 2026 17:00:00 +0000

The divide between data in operational databases and analytical data lakehouses is disappearing fast. As businesses increasingly adopt zero ETL lakehouse architectures, the challenge shifts from simply storing data in an open data format such as Apache Iceberg to serving it with the low-latency performance and speed that modern applications and AI agents require. Whether it’s a cybersecurity provider like Palo Alto Networks requiring real-time threat detection insights, or a telecommunications giant like Vodafone looking to reimagine data workflows for better customer experiences, organizations need to serve precomputed insights and AI models at scale.

To help, today, we are excited to announce the preview of the Spanner columnar engine, which allows you to serve your Iceberg lakehouse data with the scale and low latency of Google’s Spanner.

Uniting OLTP and analytics: The Spanner columnar engine

Traditionally, organizations had to choose between the high-performance transactional capabilities of an OLTP database and the analytical power of a columnar warehouse. Spanner’s columnar engine ends this trade-off by uniting these two worlds in a single, horizontally scalable system.

The columnar engine uses a specialized storage mechanism designed to accelerate analytical queries by speeding up scans up to 200 times on live operational data. By storing data in a columnar format alongside traditional row-based storage, Spanner can execute complex queries automatically using vectorized execution — processing batches of data at once rather than row-by-row. Most importantly, this performance boost can be isolated from critical transaction workloads, so that customer-facing applications remain responsive while you gain real-time insights from your operational datastore.

New features

Since we first announced Spanner columnar engine, we’ve added several new capabilities to accelerate performance and enhance usability. These include:

Vectorized execution: The engine supports faster columnar scans and aggregations using vectorized execution to process data more efficiently.
Automatic query handling: Spanner automatically redirects large-scan analytical queries to the columnar representation, speeding up analytical queries without affecting concurrent transactional workloads, allowing for true hybrid processing.
On-demand columnar data conversion: In addition to automated columnar data conversion, a new major compaction API helps accelerate the conversion of existing non-columnar data into the columnar format.

Why Iceberg data needs a fast, low-latency serving platform

Apache Iceberg has become the standard for open lakehouse architectures, providing a robust way to manage massive open format datasets in cloud-based storage. However, while lakehouses are excellent for large-scale analytics, they aren’t usually designed for the sub-second, high-concurrency "point lookups" or aggregated serving that live applications require.

This is where Spanner provides a unique value proposition. By moving curated, processed data from your lakehouse into Spanner — a process known as reverse ETL — you transform "cold" analytical data into "hot" operational data. Spanner provides the global consistency and high availability that applications require, making your Iceberg data accessible via low-latency APIs for real-time decision-making and agentic AI features.

Benchmarking Spanner columnar engine

To demonstrate Spanner’s new serving capabilities, we used a leading industry benchmark for web analytics and real-time dashboards — the exact scenarios where low-latency serving is critical.

Our benchmark results with a single Spanner node showcase the power of the columnar engine:

Benchmark query	Spanner columnar engine speedup
Simple count of all records	46.3×
Basic aggregation with filtering	32.7×
High selectivity scan	46.7×
Global aggregation	58.6×

These results represent the acceleration of real world workloads on Spanner columnar engine and show that Spanner can take complex, scan-heavy queries and return results in milliseconds, making it a great choice for powering real-time dashboards and user-facing features. Spanner is now a high-performance engine capable of delivering complex analytical results at the speeds required by modern digital experiences.

Universal reverse ETL: Serving data from all lakehouses

Spanner is designed to be the serving layer for your entire data ecosystem. Whether your lakehouse lives in BigQuery, Snowflake, Databricks, or Oracle, Spanner offers an integrated pathway for high-speed serving.

Through our latest reverse ETL workflows, you can easily bridge the gap between your analytical and operational worlds:

BigQuery: The tight integration between Spanner and BigQuery provides a powerful, bidirectional bridge for managing Iceberg data across both operational and analytical environments. You can perform federated queries on BigLake Iceberg and Spanner tables using BigQuery external datasets for Spanner, allowing for real-time analysis without moving data. When you need to serve curated BigQuery insights at scale, reverse ETL workflows can push data from BigQuery and BigLake Iceberg tables directly into Spanner. Furthermore, you can capture live operational changes in Spanner and stream them into BigQuery and BigLake Iceberg tables using Datastream, ensuring your lakehouse remains synchronized with your transactional data from Spanner for agentic AI and real-time decision-making.
Databricks: Using Databricks' Universal Format (UniForm), you can generate Iceberg metadata for your Delta Lake tables automatically. This allows Spanner to ingest your processed Databricks data via BigQuery or Dataflow, so that your "curated" datasets are ready to power applications with minimal engineering overhead.
Snowflake: You can export Iceberg tables to Google Cloud Storage and use BigQuery BigLake as a zero-copy intermediary to push that data directly into Spanner via EXPORT DATA commands. Alternatively, for simpler migrations, you can export Snowflake data as CSVs and use Dataflow templates for high-throughput ingestion into Spanner.
Oracle Autonomous AI Lakehouse: Oracle Goldengate 26ai now allows you to replicate your Oracle Autonomous AI Lakehouse data into Spanner to serve insights generated from Oracle’s data ecosystem with Spanner’s scale and consistency.

Get started today

It’s time to stop waiting for your lakehouse queries to finish and start serving your data hot, fresh, and fast with Google Spanner powered by Spanner columnar engine. The Spanner columnar engine is now in preview. You can enable it on your existing Spanner tables today with a simple DDL change.

You can see the performance acceleration of Spanner columnar engine by running the analytics benchmark queries on Spanner available on Github for yourself.

To help you get started, here are codelabs to build reverse ETL pipelines to Spanner from

Databricks:

and

Snowflake:

Decommission your legacy Apache Cassandra stack and build for the future with Spanner

Thu, 19 Feb 2026 17:00:00 +0000

An increasing number of customers are migrating to Spanner from legacy NoSQL environments like Apache Cassandra. The strategic drivers are evident: a markedly lower total cost of ownership (TCO), elastic scalability, and near-zero maintenance overhead.

With the general availability of the native endpoint enabling Cassandra Query Language (CQL) APIs on Spanner, your existing Cassandra applications can now leverage Spanner’s enterprise foundation, featuring strong consistency, virtually limitless scale, and 99.999% availability — all while utilizing familiar CQL.

Better yet, migrating your application to Spanner with the CQL interface typically requires only a one-line code change, as your existing CQL statements remain valid. Combined with our integrated, high performance bulk and live migration tools, your transition from Cassandra to Spanner is simple.

Beyond NoSQL: Strategic solutions for Cassandra Users

While the CQL API facilitates the move, Spanner addresses the fundamental data integrity and operational constraints inherent in traditional Cassandra architectures:

Global ACID transactions: Minimize concerns regarding eventual consistency. Achieve comprehensive global ACID transactions to help ensure data integrity at any scale.
Powerful indexing: Strongly consistent secondary indexes support complex query patterns with built-in optimization and no integrity risks.
Rich SQL: Utilize a sophisticated SQL interface that supports joins and aggregations.
High reliability: Benefit from 99.99% availability in regional setups and 99.999% in multi-regional configurations.
Compliance and latency: Simplify data residency compliance with geo-partitioning, delivering low-latency local reads and writes to a global user base.
Built-in observability: Access a suite of performance metrics and charts directly in the Google Cloud console at no additional cost.

The native CQL endpoint provides a clear pathway to decouple your existing Cassandra applications and modernize them using the full power of Spanner. Let’s look at the next steps after migrating your data and applications from Cassandra to Spanner.

Tweak Spanner for your specific workload

Following your migration, here’s how to optimize your Spanner environment for your workload.

1. Optimize costs and operational efficiency

Workload Characteristics	Recommended Solution	Primary Benefit
Write-intensive traffic	Throughput-optimized writes	Up to a 6x increase in write throughput via request bundling (with minimal latency impact).
Variable or fluctuating traffic	Autoscaler	Automatically aligns capacity with demand, eliminating over-provisioning costs.
Steady-state, baseline capacity	Committed use discounts (CUDs)	Secure up to 40% savings on steady-state operational costs.
Storage-intensive workloads	Tiered storage (HDD)	Utilize cost-effective HDD storage for a significant reduction in long-term storage expenses.

2. Achieve low latencies
We continuously enhance Spanner's performance to support mission-critical, high-concurrency workloads.

Single-digit millisecond performance: Spanner consistently delivers under 5ms latency for both read and write operations.
Repeatable read isolation: This feature utilizes optimistic concurrency to reduce latency and transaction aborts in read-heavy, low-contention scenarios.
Read leases: Enable strongly consistent reads in multi-region instances without cross-region coordination, maximizing node efficiency and performance.

3. Prepare for peak traffic surges
For planned events like marketing launches or massive data ingestions, you can proactively manage capacity:

Manual split APIs: While Spanner handles data partitioning automatically, the pre-splitting capability allows you to exactly define how your database distributes data ahead of peak loads. This helps ensure immediate utilization of new capacity for stable performance.

4. Isolate operational and analytical pipelines
Prevent resource contention by isolating BI and ETL processes from core operations:

Dedicated resources: Leverage read-only replicas and directed reads to achieve workload isolation.
Advanced analytics: Spanner provides high-performance operational analytics through its columnar engine, and integrates with BigQuery via Data Boost, reverse ETL, and continuous queries.

Deconstruct your Cassandra ecosystem

Migrating from Apache Cassandra to Spanner is a strategic opportunity to decouple your architecture from a complex web of sidecar utilities. While the Cassandra-compatible API serves as the entry point, the true value lies in collapsing the operational "Cassandra tax" into a unified, managed multi-model ecosystem.

Here’s a quick guide to drastically lowering your TCO while significantly boosting performance.

1. Transition easily via connectors

Minimize initial friction by leveraging the Spanner Cassandra Adapter and connectors to port management and application layers with near-zero code modifications:

Orchestration: Maintain your existing Airflow DAGs by redirecting to the Spanner proxy.
Application frameworks: Transition from Spring Data Cassandra to Spring Data Spanner, preserving established repository patterns while unlocking a superior transaction model.
Data processing: Continue leveraging Apache Spark by connecting to Spanner’s native CQL endpoint or using the specialized Spanner Connector for Spark.

2. Retire sidecars via native integrations

Cassandra demands a "babysitting" layer that Spanner automates. Decommission legacy maintenance tools and unlock Spanner’s advanced capabilities:

Anti-entropy: Decommission Cassandra Reaper. Spanner’s Paxos-based replication manages consistency natively, removing the need for manual repair cycles.
Data protection: Replace Medusa with Spanner-native backups. Move away from fragile SSTable snapshots to reliable, Point-in-Time Recovery (PITR).
Observability: Swap complex JMX exporters for Cloud Monitoring. Focus on high-value metrics like Query Insights and Lock Statistics rather than maintenance debt.
Integrated search: Replace external Elasticsearch sidecars and complex ETL pipelines with Spanner full-text search to eliminate indices-synchronization issues.
Modern streaming: Swap legacy CDC for Spanner change streams, providing native integration with Dataflow and Kafka.
Graph analytics: Port JanusGraph to Spanner Graph to run openCypher queries directly on operational data without complex ETL processes.
Query federation: Replace Trino/Presto with BigQuery Federation via Data Boost. Join real-time transactional data with massive data lakes without impacting production I/O.

Build the future with Spanner’s multi-model advantage

Spanner is the definitive always-on database that integrates relational, key-value, graph, search, and vector search capabilities into a single, interoperable platform. By transitioning to Spanner, you eliminate the overhead of managing fragmented databases and unlock the ability to develop innovative applications on a unified data foundation. Build new applications or modernize your legacy applications to take full advantage of Spanner.

Get started today

Ready to refocus on building rather than managing? Initiate your migration today and experience the power of Spanner with your existing Cassandra Query Language.

Codelab: Gain practical experience with the native CQL endpoint.
Free trial: Explore Spanner for 90 days or start for as little as $65/month for a production-ready instance that scales without disruption.
Migration guide: Access deep technical documentation and comprehensive migration resources.

Powering the next generation of agents with Google Cloud databases

Wed, 18 Feb 2026 18:00:00 +0000

For developers building AI applications, including custom agents and chatbots, the open-source Model Context Protocol (MCP) standard enables your innovations to access data and tools consistently and securely. At the end of 2025, we introduced managed and remote MCP support for services like Google Maps and BigQuery, establishing a standard method for AI to connect with tools, and effectively creating a universal interface for applications. Today, we are expanding this offering to include PostgreSQL with AlloyDB, Spanner and Cloud SQL, as well as Firestore and Bigtable for high-performance NoSQL workloads, and introducing a new Developer Knowledge MCP server, which presents an API to connect IDEs to Google’s documentation. These servers run in Google Cloud, providing a secure interface for Gemini and other MCP-compliant clients to easily interact with data and infrastructure.

With the launch of Gemini 3, developers gained advanced reasoning capabilities to plan, build, and solve complex problems. But for an AI model to function as a useful "agent," it must reliably interact with its environment. Today’s announcement extends these capabilities more broadly to the database tools our customers leverage daily as the backbone of their work environment.

To connect your agents to these servers, you don’t need to deploy infrastructure. Just configure the MCP server endpoint in the agent configuration and immediately gain access to your operational data, backed by enterprise-grade auditing, observability and governance. With no infrastructure management, you can scale your agentic workloads without incurring operational overhead.

Bringing operational data to agents

These new managed servers enable agents to access specific capabilities across our portfolio:

AlloyDB for PostgreSQL: Agents can interact with PostgreSQL workloads, enabling tasks such as schema creation, diagnosing complex queries for slowness and performing vector similarity search.
Spanner: With unified multi-model capabilities in Spanner such as Spanner Graph, agents can model and query complex relationships directly alongside relational and semantic data using standard (SQL and GQL) queries. This allows agents to quickly uncover deep insights (like identifying fraud rings or generating product recommendations) using the MCP tools at its disposal.
Cloud SQL for PostgreSQL, MySQL and SQL Server: Developers and database administrators can use the Cloud SQL MCP Server across MySQL, PostgreSQL, and SQL Server fleets for natural language interactions with the database, AI-assisted app development, query performance optimization and database troubleshooting via agents.
Bigtable: Bigtable’s flexible schema and high-throughput ingestion capabilities are commonly used for building digital integration hubs and managing time series data. MCP simplifies automating operational workflows and developing agentic customer support, CRM, human resources, IT operations, supply chain and logistics applications with this data.
Firestore: Focused on mobile and web development, the Firestore MCP server enables agents to sync with live document collections. This supports dynamic interactions such as checking user session states or verifying order statuses via natural language prompts.

Managing applications and infrastructure

Beyond data retrieval, we are enabling agents to help build and manage applications. The Developer Knowledge MCP server connects IDEs to Google’s documentation, allowing agents to answer technical questions and troubleshoot code with relevant context.

Security and governance

Connecting an agent to a database requires robust security and governance. These servers are built on Google Cloud's standard identity and observability frameworks:

Identity-first security: Authentication is handled entirely through Identity and Access Management (IAM) rather than shared keys. This ensures agents can only access the specific tables or views explicitly authorized by the user.
Full observability: To track agent activity, every query and action taken via these MCP servers is logged in Cloud Audit Logs. This provides security teams with a record of every database interaction, maintaining visibility alongside ease of access.

Demo: From local code to managed data

Let’s see these new MCP servers in action.

Imagine an agent designed to automate the migration of a full-stack event management platform for fitness communities. Through a series of natural language instructions in the Gemini CLI, the agent utilizes the Cloud SQL remote MCP server to provision a managed PostgreSQL instance, apply the correct schema, and securely migrate your local data. You don't need to master complex gcloud commands or become a Cloud SQL expert; the agent handles the heavy lifting. This transition is architected in real-time by the Developer Knowledge MCP server, which references official documentation to guide the agent through best practices — easily upgrading your application's backbone from local storage to a fully managed enterprise database.

Support for third-party agents

Because these servers follow the open MCP standard, they also work with your favorite AI agents. You can easily connect clients like Anthropic’s Claude by adding a Custom Connector in the settings. Simply point it to your Google Cloud database MCP endpoint, and you are ready to start building — no complex configuration files required.

What’s next

We’ll continue to expand this ecosystem in the coming months with managed MCP support for Looker, Database Migration Service (DMS), BigQuery Migration Service, Memorystore, Database Center, Pub/Sub, Kafka and more.

To start building secure, data-driven agents, explore our guides for AlloyDB, Spanner, Cloud SQL, Bigtable, and Firestore. You can also check out these codelabs for Cloud SQL and Spanner, along with this demo video walking through the app migration to Google Cloud.

Google (Spanner) Ranks #1 for Lightweight Transactions Use Case in Gartner® Critical Capabilities Report

Wed, 18 Feb 2026 17:00:00 +0000

For the second consecutive year Google (Spanner) has been recognized by Gartner in the Critical Capabilities for Cloud Database Management Systems for Operational Use Cases report, where it ranked #1 in the Lightweight Transactions Use Case. In our opinion, this is a testament to its efficiency, low latency, and high-performance consistency. It was also ranked #2 in the OLTP Transactions Use Case in the latest rankings. We feel this shows that its "infinite" horizontal scale doesn't come at the cost of traditional transactional performance. To us, this back-to-back recognition underscores a theme of sustained excellence and continuous innovation for the world’s most demanding data environments

Check out the full Gartner report.

A look at operational performance

Looking at the Critical Capabilities Ratings, we believe Spanner’s performance is why it continues to be recognized across a variety of categories:

#1 in Lightweight Transactions Use Case (second year in a row): As microservices and event-driven architectures become the norm, the ability to handle high-frequency, low-latency transactions is paramount. We feel Spanner’s #1 ranking shows it is the most efficient engine for these modern workloads.
Distributed Transactions Support (Score: 5.0/5.0): Spanner is the only service to receive a 5.0 for the ability to support distributed transactions across multiple nodes, regions and geographies.
AI/Machine Learning and GenAI (Score: 4.6/5.0): We believe this score is the result of the massive wave of innovation we undertook in 2025 to integrate AI directly into Spanner.
Transactional Consistency (Score: 4.9/5.0): Spanner offers external consistency at planet-scale, helping ensure that every transaction is globally accurate, every time.
#2 in OLTP Use Case: Spanner is increasingly being used for traditional, high-volume transactional workloads, thanks to, in our view, its "infinite" horizontal scale, which makes it a top-tier choice for legacy migrations.

These positionings, to us, signal that Spanner has evolved from a specialized solution for global scale into the most technically capable database suitable for nearly any mission-critical application.

The foundation for the agentic enterprise

In 2025, Spanner became the "brain" of AI agents, driven by a number of new capabilities:

Spanner Graph (now GA): A unified graph and relational experience. With Spanner Graph, you can traverse complex relationships using ISO-standard GQL alongside SQL. This is critical for building knowledge graphs that ground AI agents in real-world facts.
Integrated hybrid search: Generally available since 2025, Spanner supports high-scale full text and vector search directly within its SQL and Graph modalities, allowing for integrated, low-latency retrieval-augmented generation (RAG) at scale.
Spanner columnar engine: This architectural breakthrough enables analytical queries to run up to 200× faster on live operational data served from Iceberg tables, providing agents with the real-time context they need without the "ETL tax."
Expanded interoperability: With new Cassandra-compatible APIs, we've made it easier than ever to bring scale-out workloads to Spanner with zero code changes, so that your legacy data can participate in the AI revolution.

A blueprint for tomorrow

In our opinion, the Gartner recognition validates our belief that the future of data is unified, open, and inseparable from AI. Every day, we see customers leveraging Spanner’s recent innovations:

Palo Alto Networks: As a global cybersecurity leader, Palo Alto Networks relies on Spanner to be the heart of its multi-tenant architecture. By leveraging Spanner’s 99.999% availability and Spanner Graph, they have unlocked crucial access-control use cases at planet-scale. This integrated approach allows them to secure AI at scale, stitching together a cohesive architecture that manages critical customer workflows without the need for specialized graph database silos.
Mercado Libre: The Latin American e-commerce and fintech giant uses Spanner for its foundational consistency and scale. By integrating Spanner with their internal "Fury" platform, they manage millions of daily financial transactions with near-perfect uptime. They leverage Spanner data boost for real-time analytics without impacting transactional performance, and they are now using Spanner as the reliable memory store for their next generation of AI-driven financial agents.
Unico: This leading IDTech company prevents billions in fraud by facilitating over 1.2 billion digital identity authentications, since 2021. Spanner’s integrated vector search allows Unico to perform complex 1:N facial biometric searches across billions of vectors with high precision and low latency. This eliminates the need for a separate vector database, so that their "Identity Network" remains fast, accurate, and maintenance-free.

Ready to see why Spanner was recognized?

The future of data is unified and built for AI. By ranking #1 in Lightweight Transactions and we believe Spanner is the premier choice for your mission-critical applications.

Read the full 2025 Gartner® Magic Quadrant for Cloud DBMS.
Explore the 2025 Gartner Critical Capabilities for Operational Cloud DBMS.
Try Spanner for free and start building your AI-ready applications today.

_{Critical Capabilities for Cloud Database Management Systems for Operational Use Cases, By Ramke Ramakrishnan, Masud Miraz, Xingyu Gu, Henry Cook, Aaron Rosenbaum, November 19, 2025.}

_{GARTNER and MAGIC QUADRANT are registered trademarks and service marks of Gartner, Inc. and/or its affiliates in the U.S. and internationally and are used herein with permission. All rights reserved.}

_{Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.}

_{This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from Google Cloud.}

Spanner in 2025: Innovations powering intelligent, multi-model AI applications

Wed, 28 Jan 2026 17:00:00 +0000

For over a decade now, Google has relied on Spanner, an always-on, virtually unlimited database to power global-scale applications like Gmail, YouTube, and Google Photos. Today, Spanner handles over 6 billion queries per second at peak and more than 17 exabytes of data, with five 9s of availability, plus global consistency.

2025 was a big year for Spanner on Google Cloud, too, where it powers demanding workloads for customers like Walmart, Goldman Sachs, Palo Alto Networks, Shopify, and MercadoLibre. At the same time, in an era defined by artificial intelligence, the role of databases is undergoing a profound transformation. The database is evolving beyond its traditional function as a passive data repository, emerging instead as an intelligent context hub. Spanner now powers key use cases like product search and recommendations, fraud detection, identity resolution, and autonomous network operations (ANO).

As we gear up for an exciting 2026, let’s take a minute to review some of last year’s highlights. In a nutshell, in 2025, we:

Made Spanner a first-class citizen for AI workloads
Bridged the gap between operational and analytical data
Made migrations simpler
Raised the bar on price performance, and
Enhanced enterprise safety and data protection

Let’s review!

Spanner AI with multi-model support

To build truly intelligent generative AI applications, you shouldn't have to glue together multiple point solutions. This is why in 2024, we integrated vectors, graphs, and relational data to provide context for your AI models. In 2025, we expanded these multi-model capabilities, adding new functionality across graph, vector, text search, as well as Vertex AI integrations.

AI capabilities: We introduced multiple capabilities to make Spanner an AI-first database. This includes built-in AI integrations such as ML.PREDICT for natural language queries, MCP Toolbox for Databases, Spanner extension for Gemini CLI, conversational data agents, and Agent Development Kit for Spanner, which together, empower developers to build sophisticated, agentic applications with Spanner. We also extended our multi-model platform to power advanced AI use cases, such as using graphs with vector search for GraphRAG use-cases. Finally, Vertex RAG Engine now lets you use Spanner as a RAG-managed database for data indexing and retrieval operations.
Graph enhancements: Spanner Graph added support for schemaless data that enables iterative development and frequent updates without requiring schema changes. You can also now build graphs on named schema objects and on SQL views, organizing and encapsulating important aspects of your app.
Full text search: The enhance_query option provides automatic synonym matching and spell correction for a number of functions, reducing the manual tuning needed to improve recall.
Approximate Nearest Neighbor (ANN): ANN with Google's ScaNN is a fast, scalable vector search technique optimized for efficiency in high-dimensional vector searches, crucial for AI recommendation systems, image search, and semantic search. In 2025, we announced the general availability of ANN search to enable developers to perform fast similarity searches on vector embeddings, powering generative AI applications where consistency at scale matters.

AI-powered operations

In 2025, we delivered multiple AI powered improvements that make it easier for developers to leverage Spanner for their applications:

Spanner index advisor: Index Advisor analyzes your query patterns and proactively suggests new indexes (or identifies unused ones) to optimize performance and cost.
Schema recommendations (Preview): An intelligent tool that scans your schema design for anti-patterns — like hotspot-prone primary keys — and suggests improvements before they become production issues.
Spanner CLI: The new command line interface, bundled directly with gcloud, allows you to run SQL, manage sessions, and automate scripts within existing developer workflows.

Unified analytics: Minimize silos and complexity

Last year, we bridged the gap between operational and analytic data with multiple new capabilities:

Spanner columnar engine (Preview): This engine automatically lets you analyze vast amounts of operational data in real time, while maintaining Spanner's global consistency, high availability, and strong transactional guarantees — without impacting transactional workloads. Verisoul.ai, a provider of an all-in-one platform to automate fake account detection used Spanner and Spanner columnar engine to support low-latency transactional writes and rich analytics to provide fast responses to their users.
The "Better with BigQuery" ecosystem: We strengthened the integrations between Spanner and BigQuery to bring operational and analytical engines together to provide real-time insights:

Apache Iceberg support: With this capability, data analysts can join live Spanner data with Apache Iceberg tables on BigQuery, as well as export Iceberg data into Spanner. This allows you to query and combine your real-time operational data (from Spanner) with your curated data lakehouse (in Iceberg).
Spanner external datasets materialized views: We announced the general availability of Spanner external datasets in BigQuery that enable BigQuery users to query live operational data in Spanner with zero ETL requirements. We further extended this capability to integrate with BigQuery materialized views for ultra-fast reporting using pre-computed query results.
Continuous queries with Reverse ETL (BigQuery to Spanner): You can now combine BigQuery’s continuous query capabilities with reverse ETL to Spanner to stream computed insights — such as fraud alerts or dynamic pricing signals — from BigQuery into Spanner in real-time, for low-latency serving of these insights. Fastweb+Vodafone was able to do just that by leveraging this feature to use Spanner as their real-time serving layer with BigQuery.

Simplified migration

Customers frequently migrate to Spanner to benefit from five nines of availability, virtually unlimited scale, low operational overhead, and global consistency. In 2025, we made it it easier for them to migrate their Cassandra, and MySQL workloads to Spanner:

Cassandra interface: Spanner’s Cassandra interface lets you take advantage of Spanner's fully managed, scalable, and highly available infrastructure using familiar Cassandra tools and syntax. Lift and shift existing CQL applications with virtually no changes.
MySQL interoperability: You can now install a library of 80 MySQL functions to help reduce the application changes required to migrate your MySQL workloads to Spanner.

Performance and cost improvements

Continuously raising the bar on price-performance is a north star for the Spanner team. In 2025, we introduced multiple capabilities to help you improve query performance and reduce costs:

Repeatable read isolation (Preview): This new isolation level reduces locking overhead to improve latency for workloads that have low read-write contention.
JSON indexing: Using the same foundation as Spanner’s full-text search, JSON indexing accelerates common queries over JSON, without having to predefine anything about the structure or data. This gives developers flexibility without having to compromise on performance.
Read leases: Read leases can improve read latency for strongly consistent data in multi-region configurations by trading off some write performance for common read-mostly workloads.
Tiered storage: Tiered storage allows you to manage data lifecycle costs efficiently by using SSD and HDD storage within the same instance, all via configuration and without having to modify the APIs you use to access the data. This functionality moves older data to lower cost HDD storage (~80% cheaper) while automatically keeping new data on high-performance SSDs.
Managed autoscaler: We announced the general availability of managed autoscaler and enhanced it to allow scaling read-only replicas independently from the read-write replicas, thereby improving read performance while optimizing costs based on traffic patterns.
Manual split points: While Spanner automates sharding to distribute work across nodes in an instance, sometimes you know your traffic patterns better than we do. The new split-point API allows you to “warm up” or pre-split your data to handle anticipated spikes, like a flash sale or game launch, to handle traffic immediately.
Query optimizer: The new default query optimizer (version 8, for those counting at home) brings a host of automated enhancements, optimizing join strategies and index usage to improve query performance and predictability.
Multiplexed sessions and lazy decoding: We enabled multiplexed sessions by default across major SDKs (Java, Go), significantly improving throughput and resource utilization. We also optimized our client libraries with lazy decoding for partitioned queries, allowing your applications to handle massive datasets with a significantly lower memory footprint. Plus, new built-in client metrics give you granular observability into your application's database performance.

Raising the bar on enterprise safety

For our customers, especially in industries like financial services, retail, and healthcare, reliability is non-negotiable. This year, we introduced additional "safety nets" that further protect your data from the most unpredictable variable in any system: human error.

Drop protection: Accidental deletions can be catastrophic. We introduced schema object drop protection as a safeguard for your production environments to prevent the accidental deletion of critical tables, indexes, and columns, acting as a line of defense against operational mishaps.
Zero-touch compliance with default backup schedules: Data protection shouldn't rely on manual checklists. We introduced default backup schedules to ensure a recovery baseline the moment a database is created. This minimizes the risk of forgotten backups and so that your workloads adhere to your organization's governance and compliance standards from day one.

Awards and recognition

While we continue to enhance Spanner based on customer needs and industry trends, it’s always great to get industry recognition for our work. In 2025,

Spanner was recognized as a Leader in the Gartner Magic Quadrant for Cloud DBMS and ranked #1 for the Lightweight Transactions Use Case (and Top 3 in all categories) in the Critical Capabilities for Operational Databases report for 2025, beating AWS Aurora to take the #2 rank in “OLTP Transactions”
Beyond technical performance, a March 2025 Forrester Total Economic Impact™ (TEI) study validated the significant business value Spanner delivers, finding that a representative composite organization realized a 132% ROI and $7.74M in total benefits over three years, with a rapid nine-month payback period.
We were also honored with the 2025 ACM SIGMOD Systems Award for "reimagining relational data management to enable serializability with external consistency at global scale."

Looking ahead

We are proud of what we have delivered for customers in 2025, and are excited to see the innovative solutions you are building on Spanner. Needless to say, we are just getting started and we have a lot more exciting capabilities lined up for 2026.

Want to learn more about what makes Spanner unique and how it’s being used today?

Try it yourself for free for 90-days or for as little as $65 USD/month for a production-ready instance that grows with your business without downtime or disruptive re-architecture.

Inside Mercado Libre's multi-faceted Spanner architecture

Mon, 03 Nov 2025 17:00:00 +0000

Mercado Libre, the e-commerce and fintech pioneer of Latin America, operates at a staggering scale, demanding an infrastructure that's not just resilient and scalable, but also a catalyst for rapid innovation. While our use of Spanner for foundational consistency and scale is known, a deeper dive reveals a sophisticated, multi-layered strategy. Spanner is not just a database here; it's a core engine powering our internal developer platform, diverse data models, advanced analytics loops, intelligent features, and even our roadmap for next-generation AI applications.

This blog explores the technical underpinnings of how Mercado Libre leverages Spanner in concert with our internal innovations like the Fury platform, achieving significant business impact and charting a course for an AI-driven future.

The dual challenge: internet-scale operations and developer velocity

Mercado Libre faces the classic challenges of internet-scale services: keeping millions of daily financial transactions safe, making it easy for developers to build apps, and maintaining near-perfect uptime. The solution required a database powerful enough for the core and an abstraction layer elegant enough for broad developer adoption.

Fury: Mercado Libre's developer gateway

At the heart of Mercado Libre's strategy is Fury, our in-house middleware platform. Fury is designed to abstract away the complexities of various backend technologies, providing developers with standardized, simplified interfaces to build applications.

Abstraction & Standardization: Fury allows development teams to focus on business logic rather than the nuances of distributed database management, schema design for specific engines, or optimal connection pooling.
Spanner as the Reliable Core: Spanner is an always-on, globally consistent, multi-model database with virtually unlimited scale. By designating Spanner as a choice within Fury, Mercado Libre ensures that applications built on the platform using Spanner inherit its best features – they stay consistent globally, scale without breaking, and rarely go down.

Fig. 1 - Fury’s core services

Spanner – the versatile backbone

Through Fury, Spanner empowers Mercado Libre's developers with remarkable versatility. Some apps need complex transactions, others need fast lookups. Spanner handles both, which means teams can use just one system:

Relational prowess for complex transactions: For sophisticated transactional workloads like order management, payments, and inventory systems, Spanner’s relational capabilities (SQL, ACID transactions, joins) remain critical.
High-performance key-value store: Many modern applications require fast point lookups and simple data structures. While Spanner isn't Mercado Libre's default backend for typical key-value workloads, there are specific applications running large scale non-relational, KV-style workloads on the Spanner.

Spanner’s foundational architecture — TrueTime for global consistency and automated sharding for effortless scaling — makes it an ideal candidate to reliably serve both these access patterns through the Fury platform.

Handling peak demand

Mercado Libre's Spanner instances demonstrate significant processing capacity, handling around 214K queries per second (QPS) and 30K transactions per second (TPS). To manage this substantial workload, the Spanner infrastructure dynamically scales to over 400 nodes (by 30%), highlighting the robust and elastic nature of the underlying system in accommodating high-demand scenarios. This level of throughput and scalability is critical for maintaining the performance and reliability of Mercado Libre's services during its busiest times.

Fig. 2 - Diagram of the solution built with Spanner, which uses current search data to predict and recommend products that a customer is most likely to purchase.

Turning data into action

Mercado Libre builds a dynamic data ecosystem around Spanner, leveraging advanced analytics to feed insights directly back into operational systems.

They achieve real-time analytics by combining Spanner Data Boost with BigQuery Federation. Data Boost isolates analytical queries, preventing them from impacting critical transactional performance. This allows for powerful, large-scale analytics to run directly on fresh Spanner data within BigQuery, integrating seamlessly with other data sources.

Insights from BigQuery, such as customer segmentations or fraud scores, are then actioned via Reverse ETL, feeding directly back into Spanner. This enriches operational data, enabling immediate action by frontline applications like serving personalized content or performing real-time risk assessments.

Furthermore, Spanner Change Streams coupled with Dataflow drive crucial service integrations. By capturing real-time data modifications from Spanner, they establish robust pipelines. These enable loading changes into BigQuery for analytics or streaming them to services like Fury Stream for real-time consumption, ensuring low-latency data propagation and enabling event-driven architectures across their systems.

The impact: cost savings, agility, and future-proofing

The strategic adoption of Spanner, amplified by internal platforms like Fury and sophisticated data workflows, has yielded significant benefits for Mercado Libre:

Significant cost savings & low total cost of ownership: The combination of Spanner's managed nature (reducing manual sharding, maintenance, and maintenance work), efficient resource utilization, and the abstraction provided by Fury has led to a lower Total Cost of Ownership and substantial cost savings.
Business impact & agility: Developers, freed from infrastructure complexities by Fury and empowered by Spanner's versatile capabilities, can deliver new features and applications faster. The reliability of Spanner underpins critical business operations, minimizing disruptions.
Low operational overhead: Automated scaling, sharding, and maintenance in Spanner significantly reduce the human effort required to manage large-scale database infrastructure.

Building for AI: Next-generation applications on Spanner

Looking ahead, Mercado Libre is exploring Spanner to support more AI workloads.

Spanner's characteristics make it an ideal foundation:

Consistent state management: Critical for AI systems that need to maintain and reliably update their state context.
Scalable memory/knowledge store: Ability to store and retrieve vast amounts of data for AI system memory, logs, and contextual information.
Transactional operations: Enabling AI systems to perform reliable actions that interact with other systems.
Integration with analytics & Machine Learning (ML): The existing data loops and ML.PREDICT capabilities can enrich AI systems with real-time insights and intelligence.

Spanner provides the transactional foundation these sophisticated, AI applications will require.

Conclusion: A Unified, Intelligent Data Foundation

Mercado Libre's adoption of Spanner demonstrates how to use a powerful, globally consistent database not just for its core capabilities, but as a strategic enabler for developer productivity, operational efficiency, advanced analytics, and future AI ambitions. Through their Fury platform, they've simplified access to Spanner's capabilities, allowing it to serve as a flexible foundation for both relational and non-relational needs. The integration with BigQuery via Data Boost demonstrates a comprehensive approach to building an intelligent, data-driven enterprise. As Mercado Libre builds AI applications, Spanner is set to continue its role as the consistent and scalable foundation for their next wave of innovation.

Learn more

Spanner columnar engine: Powering next-generation analytics on operational data

Wed, 06 Aug 2025 01:00:00 +0000

For years, organizations have struggled with the workload conflict between online transaction processing (OLTP) and analytical query processing. OLTP systems such as Spanner are optimized for high-volume, low-latency transactions, and use row-oriented storage that's efficient for individual record access. Analytical workloads, conversely, require rapid aggregations and scans across large datasets. These tasks are traditionally handled by separate data warehouses that employ columnar storage and incoming data pipelines from transaction systems. Separating OLTP and analytical workflows requires periodic data transfers, which often leads to stale data, complex ETL pipelines, and operational overhead.

Today, we're thrilled to announce Spanner columnar engine, which brings new analytical capabilities directly to Spanner databases. Just as AlloyDB's columnar engine enhanced PostgreSQL analytics, Spanner's new columnar engine lets you analyze vast amounts of operational data in real-time, all while maintaining Spanner's global consistency, high availability, and strong transactional guarantees — and without impacting transactional workloads.

The power of Spanner columnar engine helps organizations, such as Verisoul.ai, eliminate the problem of data silos typically found when combining high volume transaction systems with fast analytics. “Detecting fraud in real time is only half the story—showing customers the ‘why’ helps them act faster and turn trust into measurable ROI,” says Raine Scott and Niel Ketkar, founders of Verisoul.ai, a machine-learning platform that stops fake users and fraud. “Spanner’s new columnar engine allows high-velocity transactional writes and rich analytics in one place, eliminating data copies and replication lag so customers get instant answers."

Columnar storage meets vectorized execution

Figure: Spanner columnar engine architecture

The heart of the Spanner columnar engine is its innovative architecture, which combines columnar storage with vectorized query execution.

Columnar Storage in Spanner: Hybrid Architecture

Unlike traditional row-oriented storage, where an entire row is stored contiguously, columnar storage stores data column by column. This offers several advantages for analytical workloads:

Reduced I/O: Analytical queries often access only a few columns at a time. With columnar storage, only the relevant columns need to be read from disk, significantly reducing I/O operations.
Improved compression: Data within a single column is typically of the same data type and often exhibits similar storage patterns, leading to much higher compression ratios. This means more data can fit in memory and fewer bytes need to be read.
Efficient scans: When scanning a column, consecutive values can be processed together, for more efficient data processing.

Spanner columnar engine integrates a columnar format alongside its existing row-oriented storage. This unified transactional and analytical processing design allows Spanner to maintain its OLTP performance while accelerating analytical queries up to 200X on your live operational data.

Vectorized execution: turbocharging your queries

To complement columnar storage, the columnar engine makes use of Spanner’s vectorized execution capabilities. While traditional query engines process data tuple-by-tuple (row by row), a vectorized engine processes data in batches (vectors) of rows. This approach dramatically improves CPU utilization, with:

Reduced function call overhead: Instead of calling a function for each individual row, vectorized engines call functions once for an entire batch, significantly reducing overhead.
Optimized memory access: Vectorized processing often results in more cache-friendly memory access patterns, further boosting performance.

The combination of columnar storage and vectorized execution means that analytical queries on Spanner can run orders of magnitude faster, allowing for real-time insights on your global-scale data.

Better with BigQuery: Accelerating federated queries

The Spanner columnar engine takes its integration with Google’s Data Cloud ecosystem a step further, specifically enhancing integrations between Spanner and BigQuery. For enterprises that leverage BigQuery for data warehousing and analytics, federating queries directly to Spanner has always been a valuable capability. Now, with the Spanner columnar engine, this integration becomes even more potent, by delivering faster insights on operational data.

Data Boost, Spanner's fully managed, elastically scalable compute service for analytical workloads, is at the forefront of this acceleration. When BigQuery issues a federated query to Spanner, and that query can benefit from columnar scans and vectorized execution, Data Boost automatically leverages the Spanner columnar engine. This provides:

Faster analytical insights: Complex analytical queries initiated from BigQuery that target your Spanner data execute significantly faster, bringing near-real-time operational data into your broader analytical landscape.
Reduced impact on OLTP: Data Boost helps ensure that analytical workloads are offloaded from your primary Spanner compute resources, preventing impact on transactional operations.
Simplified data architecture: You can get the best of both worlds – Spanner's transactional consistency and BigQuery's analytical prowess – without the need for complex ETL pipelines to duplicate data.

This integration empowers data analysts and scientists to combine Spanner's live operational data with other datasets in BigQuery for richer, more timely insights and decision-making.

Columnar engine in action: Accelerating your analytical queries

Let's look at some sample queries that should see significant acceleration with the Spanner columnar engine. These types of queries, common in analytical and graph workloads, benefit from columnar scans and vectorized processing.

Scenario: Imagine a large e-commerce database; for demonstration purposes, we'll use the same schema as the TPC-H benchmark.

Query 1: Revenue from discounted shipments in a given Year

SQL

code_block: <ListValue: [StructValue([('code', '@{scan_method=columnar}\r\nSELECT\r\n sum(l.l_extendedprice * l.l_discount) AS revenue\r\nFROM\r\n lineitem l\r\nWHERE\r\n l.l_shipdate >= date "1994-01-01"\r\n AND l.l_shipdate < date_add(date "1994-01-01", INTERVAL 1 year)\r\n AND l.l_discount BETWEEN 0.08 - 0.01 AND 0.08 + 0.01\r\n AND l.l_quantity < 25;'), ('language', 'lang-sql'), ('caption', <wagtail.rich_text.RichText object at 0x7fa402872670>)])]>

Acceleration: This query heavily benefits from scanning only the l_shipdate, l_extendedprice, l_discount, and l_quantity columns from the lineitem table. Vectorized execution rapidly applies the date, discount, and quantity filters to identify qualifying rows.

Query 2: Total quantity of non-discounted items

SQL

code_block: <ListValue: [StructValue([('code', '@{scan_method=columnar}\r\nSELECT\r\n sum(l_quantity)\r\nFROM\r\n lineitem\r\nWHERE\r\n l_discount = 0;'), ('language', 'lang-sql'), ('caption', <wagtail.rich_text.RichText object at 0x7fa402872520>)])]>

Acceleration: This query heavily benefits from scanning only the l_discount and l_quantity columns from the lineitem table. Vectorized execution rapidly applies the equality filter (l_discount = 0) to identify matching rows.

Query 3: Item count and discount range for specific tax brackets..

SQL

code_block: <ListValue: [StructValue([('code', '@{scan_method=columnar}\r\nSELECT\r\n count(*),\r\n min(l_discount),\r\n max(l_discount)\r\nFROM\r\n lineitem\r\nWHERE\r\n l_tax IN (0.01, 0.02);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fa402872070>)])]>

Acceleration: This query benefits heavily from scanning only the l_tax and l_discount columns from the lineitem table. Vectorized execution rapidly applies the IN filter on the l_tax column to identify all matching rows.

Query 4: Scan friend relationships to find the N most connected people in the graph using Spanner Graph

GQL

code_block: <ListValue: [StructValue([('code', '@{scan_method=columnar}\r\nGRAPH social_graph\r\nMATCH (p:Person)-[k:Knows]->(:Person)\r\nRETURN COUNT(k) AS friend_count GROUP BY p ORDER BY friend_count DESC LIMIT 10;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fa402872250>)])]>

Acceleration:This query benefits heavily from scanning subgraphs and by loading only relevant columns from the graph.

Query 5: Perform a K-nearest neighbor vector similarity search to retrieve the top 10 most semantically similar embeddings with perfect recall

GQL

code_block: <ListValue: [StructValue([('code', '@{scan_method=columnar}\r\nSELECT e.Id as key, COSINE_DISTANCE(@vector_param, e.Embedding) as distance\r\nFROM Embeddings e\r\nORDER BY distance LIMIT 10;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fa402872940>)])]>

Acceleration:This query benefits heavily from scanning contiguously stored vector embeddings and by loading only relevant columns from the table.

Get started with Spanner columnar engine today!

The Spanner columnar engine is designed for businesses looking to unlock faster, deeper, real-time insights from their operational data without compromising Spanner's foundational strengths. We are incredibly excited about the possibilities this opens up for developers and data analysts alike. We invite you to be among the first to try the Spanner columnar engine. Request access to the Preview of Spanner columnar engine by signing up today. We look forward to seeing what you build!

Spanner's enduring impact: Celebrating the 2025 ACM SIGMOD Systems Award

Tue, 17 Jun 2025 16:00:00 +0000

Earlier this year, the Association for Computing Machinery's Special Interest Group on Management of Data (ACM SIGMOD) announced that Spanner, Google's globally distributed database, was awarded the 2025 SIGMOD Systems Award. The SIGMOD Systems Award specifically honors systems whose technical contributions have profoundly impacted the theory or practice of large-scale data management. On behalf of the entire Spanner team, especially the engineers who were there at the beginning of Spanner's journey, it is with deep humility and immense pride that we receive this recognition from such a distinguished community. We’re thrilled to be participating in the 2025 SIGMOD conference from June 22-27 in Berlin, Germany as a Platinum sponsor.

This honor feels particularly significant following Spanner's 2022 SIGOPS Hall of Fame Award, which highlighted the crucial role of technologies like TrueTime and our network infrastructure, reaffirming the lasting significance of the original vision laid out in the first Spanner paper.

Spanner’s core innovation: TrueTime and external consistency

For Spanner to be recognized in this way is a powerful affirmation of the vision we set out to achieve years ago and the new ways Spanner enables applications to be built. According to the award citation, Spanner is recognized “for reimagining relational data management to enable serializability with external consistency at global scale.”

Why “reimagining”? Before Spanner, databases offered a stark choice: you could have ACID transactions and SQL, or you could have scale and multi-datacenter reliability. Scale and availability required a distributed system, and that meant eventual consistency and other forms of best-effort synchronization. Spanner showed that this choice was not fundamental — that it was possible to build a database that offered the horizontal scalability of a distributed system with the power and ease of use of transactions and SQL. It enabled companies for whom scale is job #1 to regain developer velocity and agility. Spanner drastically simplifies the logic required in distributed applications. Developers can reason about the state of the database as if it were a single, consistent entity, even when it spans the globe.

The key enabler for Spanner's ability to deliver external consistency is TrueTime. Beyond just a synchronized global clock, TrueTime is an API that cleverly exposes clock uncertainty as a bounded interval, which allows higher-level algorithms to reason about the ordering of events. Google’s TrueTime implementation uses specialized hardware references like GPS receivers and atomic clocks to provide highly trustworthy and very tight time bounds. Spanner leverages this bounded uncertainty to achieve external consistency. When a transaction commits, Spanner assigns it a commit timestamp derived from TrueTime. Spanner then enforces a "commit wait" — which can be overlapped with making the transaction durable – to ensure that the commit timestamp is definitively in the past before making the transaction's effects visible. This ensures that the assigned commit timestamps definitively reflect the true global serialization order of transactions, even across data centers. The result is remarkable: external consistency with no performance cost.

Addressing the consistency-scale dilemma

To truly appreciate the journey, it helps to travel back to the early and mid-2000s at Google. The internet was exploding, and our biggest challenge was scaling our software infrastructure to keep pace. We needed databases that could store and process a copy of the internet using vast fleets of commodity servers. This spurred the development of internal systems that delivered incredible performance and scalability, but they came with trade-offs.

As we gained more familiarity with these systems, and started using them to build big interactive applications like Gmail, we consistently heard from internal developers about the challenge of working with eventual consistency and cross-shard synchronization, as well as the friction of modeling every problem (no matter how complex) as key-value pairs. It quickly became apparent that we needed to build a globally distributed database that offered the familiarity and guarantees of traditional relational databases — including ACID transactions, serializability, and external consistency — without giving up Google’s ever-growing need for bigger databases serving bigger audiences. Moreover, working closely with our customers, it became clear that actually, it was something we could build. The rest is history!

Spanner as a cloud service

As a cornerstone of Google's infrastructure, Spanner powers some of our most critical, planet-scale services, including Google Ads, Google Search indexing, Gmail, YouTube, Google Photos, metadata for Cloud Storage, and BigQuery, demonstrating its robustness and scalability under extreme load.

The next logical step was to make these capabilities available externally to customers through Google Cloud. The launch and subsequent evolution of Spanner aimed to democratize this technology, bringing the power of a globally consistent, scalable database to organizations of all sizes, from startups to global enterprises, simplifying their application development and operations.

Spanner’s core value proposition for customers stems directly from its unique architecture:

Global scale with strong consistency: Spanner delivers on the original promise: ACID transactions with external consistency across a database that can scale horizontally across regions and continents, automatically managing data distribution (sharding) as needed. This directly addresses the capability highlighted by the SIGMOD award.
Unmatched availability: Leveraging synchronous, Paxos-based replication across multiple zones or regions, Spanner offers an industry-leading 99.999% availability Service Level Agreement (SLA) for multi-region configurations. This helps provide extreme fault tolerance and minimizes downtime risk for mission-critical applications.
Simplified operations: As a fully managed service, Spanner automates complex operational tasks like sharding, replication management, backups, and maintenance. This frees development teams from significant operational burdens, allowing them to focus on building applications rather than managing database infrastructure. This contrasts sharply with the manual effort often required for traditional sharded databases or the complexity of implementing consistency logic at the application layer for NoSQL systems.
Developer productivity: Spanner offers familiar SQL query interfaces, supporting both Google SQL and PostgreSQL dialects, which significantly flattens the learning curve for developers. Furthermore, its strong consistency eliminates entire classes of complex problems related to data synchronization and reconciliation that often plague applications built on eventually consistent systems.

Empowering customers and industries

Of course, today’s data landscape has changed since we published the first Spanner paper in 2012. In the modern AI-first data world, we see customers more focused than ever on getting full value from their data, which is often spread out over many systems with different data models, varying scalability, and uneven reliability. We’re addressing these new challenges head on, introducing Spanner Graph, vector search for AI applications, and integrated full-text search. These allow you to bring together a wide range of data and iterate on it rapidly within a single, consistent, scalable platform. We've also helped you run even more cost-effectively as your data grows, with increased compute and storage density, tiered pricing through editions, and cost-optimizing features such as tiered storage. Finally, we’ve made it easier to bring scale-out workloads into Spanner through enhanced interoperability with tools and capabilities, such as Cassandra-compatible APIs.

While academic recognition such as the SIGMOD award is gratifying, we’ve always felt that the true measure of a system's impact lies in how it empowers users to solve real-world problems and build innovative applications. Spanner's unique combination of capabilities have proven transformative across various industries.

In financial services, where consistency, availability, and security are paramount, Spanner provides the foundation for many critical systems. Companies like Goldman Sachs use it to consolidate trade ledgers, while others like Arigato Bank rely on it to handle high volumes of financial transactions with perfect consistency, even during peak loads. Digital-native banks like Minna Bank have built their entire infrastructure on Spanner, leveraging its availability and consistency to meet stringent regulatory requirements and customer expectations.

The gaming industry is constantly pushing the boundaries of scale and real-time interaction. Spanner has helped game developers launch globally successful titles like Dragon Quest Walk by Colopl, handling millions of concurrent players from day one. Its ability to manage player profiles, in-game inventory, and leaderboards consistently across a global player base, while elastically scaling to handle unpredictable traffic spikes, has been crucial for delivering a seamless player experience

In retail and e-commerce, Spanner helps businesses manage the complexities of modern commerce. Walmart uses Spanner to modernize its inventory and payment management, providing a real-time, consistent view across online and physical stores. MercadoLibre, a global online marketplace and e-commerce provider, leverages Spanner to handle the global needs of their customers, including massive demand spikes during major product launches.

Leaders in transportation, such as Uber, rely on Spanner to handle millions of concurrent users and billions of trips per month across over 10,000 cities and billions of database transactions per day.

These examples illustrate that for many modern applications, Spanner's specific blend of global consistency, massive scalability, high availability and interoperable multi-model isn't just a technical advantage; it's a fundamental enabler. Business models built around real-time global inventory, instantaneously consistent financial records, or seamless worldwide multiplayer experiences become significantly simpler, less risky, and more feasible to implement with Spanner.

The future with Spanner

The SIGMOD award recognizes the contributions of over 30 individuals, and countless other current and prior Googlers who have played roles throughout its development and evolution. It’s been a privilege to witness the journey from the initial ambitious concepts to the globally impactful system it is today. The journey from the original OSDI paper published to this 2025 SIGMOD Systems Award highlights over a decade of sustained research, engineering, and investment by Google. This long-term commitment is rare and is a key factor behind Spanner's enduring success and impact.

Thanks to you, our customers, Spanner remains a living system that continues to evolve, benefiting from ongoing improvements, driven by both internal use and the needs of our cloud customers. Looking ahead, we remain committed to pushing the boundaries of what's possible with distributed databases. We're excited about enabling new kinds of applications, including those leveraging AI, and continuing our mission to simplify the complexities of data management for developers building the next generation of world-changing applications.

Experience the Spanner difference

The 2025 ACM SIGMOD Systems Award is a tremendous honor and a validation of the path we embarked on years ago. If you are at the 2025 SIGMOD conference, join us on Tuesday, June 24 at 4:30 pm to hear from and meet with the Google team.

If Spanner's capabilities for consistency, scale, and availability resonate with the challenges you face, we encourage you to learn more:

Explore Spanner to learn about supported features and use cases, and get started with a 90-day Spanner free trial instance.
Dive deeper with documentation
For the academically inclined, read the original and successor papers on Spanner

In closing, we believe Spanner unlocks new possibilities for building reliable, scalable, and globally consistent applications, and we are excited to see what you, our customers, will build with it.

Introducing Apache Cassandra®-compatible APIs in Spanner for zero-code-change migration

Thu, 15 May 2025 16:00:00 +0000

You heard at Next ‘25 that native support for the Apache Cassandra® Query Language (CQL) API in Spanner is available in preview, providing near-zero code-change application migration, along with a suite of complementary tools for zero-downtime data migration.

Spanner is known for its robust SQL support, ACID transactions, and change data capture (CDC), provided as a fully managed service. At the same time, Spanner excels at the basics of NoSQL: single-digit-millisecond read and write latencies, virtually unlimited horizontal scalability, and industry leading high availability. With the CQL addition to Spanner, migrating your application is typically a one-line code change — your application and CQL statements stay the same — while new data migration tooling simplifies live and bulk migration of data.

Why migrate from Cassandra to Spanner?

But first, let’s talk about why an organization might want to migrate from Cassandra to Spanner in the first place.

NoSQL databases like Apache Cassandra gained popularity for their horizontal scalability and high availability — attributes that are essential for modern, web-scale applications that manage large datasets and user traffic. But while Cassandra's masterless architecture and tunable consistency provides flexibility, but some key challenges persist:

Cassandra suffers from high total cost of ownership (TCO) via inelastic scaling and the substantial operational overhead it requires.
Queries that use secondary indexes may suffer from significant performance impact.
CQL lacks complex joins and sophisticated aggregation functionality.

In contrast, Spanner offers several key advantages for demanding NoSQL workloads:

Easy operations: Managing Cassandra requires dedicated expertise and effort for tasks such as hardware/VM provisioning, configuration tuning, scaling, and patching and upgrades. With Spanner, Google Cloud manages these operational tasks, dramatically reducing TCO and freeing engineers to focus on your application, not database administration.
Optimized elastic scale: Spanner offers truly elastic scalability via an autoscaler that automatically adjusts capacity in response to real-time load, helping avoid overprovisioning for peak workloads. Spanner handles data rebalancing in the background, so as not to impact availability or performance of production workloads.
Global ACID transactions: Cassandra's Lightweight Transactions (LWTs) offer limited compare-and-set functionality within a single partition and cannot guarantee atomicity for multi-partition or multi-step operations without complex, often brittle, application-level workarounds. Spanner supports fully ACID (atomic, consistent, isolated, durable) transactions, simplifying development and helping to ensure data integrity. In fact, Gartner ranks Spanner as #1 in the Lightweight Transactions Use Case and #3 in the OLTP Transactions Use Case in its Critical Capabilities for Cloud Database Management Systems for Operational Use Cases report.
Strongly consistent global secondary indexes: Cassandra secondary indexes are local indexes, stored as hidden tables on each node that contains the primary data to be indexed. This means that queries on a secondary index that accesses multiple nodes can greatly impact your application’s performance. Spanner offers global secondary indexes that can be used to efficiently query data across the entire database. In addition, Spanner’s secondary indexes are strongly consistent, so your different queries all return results that are consistent and up to date. Spanner also offers an index advisor that analyzes your queries to recommend new or altered indexes that can improve query performance.
Normalized data for data integrity: Data in Cassandra is often arranged as one query per table, and data is repeated amongst many tables, a process known as denormalization. Data denormalization has several drawbacks: increased data redundancy, inconsistencies between datasets, and increased storage and maintenance requirements as the number of tables grows. In contrast, data in Spanner is normalized, with full support for relational semantics, including enforced foreign keys and rich SQL query capabilities with efficient server-side aggregation.

Using Spanner’s new Cassandra-compatible APIs

The new APIs and the migration tools are meant to provide a seamless experience for your Cassandra to Spanner migrations. Here’s a quick guide:

Set up Spanner: Begin by creating a Spanner instance — regional or multi-regional. For each Cassandra keyspace you intend to migrate, create a corresponding database in your Spanner instance. Using the same name as the Cassandra keyspace is recommended to minimize application code modifications.
Convert your schema: Utilize the spanner-cassandra-schema-tool to automate schema conversion and Spanner tables creation. This tool processes your CQL table definitions, maps Cassandra data types to their Spanner equivalents (e.g., text to STRING(MAX), bigint to INT64, map to JSON), and automatically generates the transformed Spanner tables.
Migrate data: Migrate your data from Cassandra to Spanner and subsequently verify its integrity. We typically recommend a two-phased approach to minimize downtime:

Live migration (for zero downtime): To achieve a near zero-downtime migration, implement live migration for incoming data before initiating the bulk data transfer, via the included Docker file or Terraform template. The tool leverages the ZDM Proxy for dual-writes and the Cassandra-Spanner Proxy to transform CQL to Spanner API.
Bulk data migration: Use the SourceDB to Spanner Dataflow template for a highly parallelized transfer. This template reads data from Cassandra, performs the necessary transformations, and writes it to Spanner.
Data validation: After the bulk data migration is complete, it’s time to validate data accuracy and integrity. Common methods include comparing row counts, sampling row data, or performing an exhaustive comparison between the Cassandra and Spanner data.

Switch your application to use the native endpoint for Cassandra: We provide two options for doing this:

- Embed the Spanner Cassandra Client Library: For applications written in Java or Go, you can include the Spanner Cassandra client library as a dependency. The primary code modification involves changing how you build the CqlSession object to use the SpannerCqlSession.builder() and providing the Spanner database URI. Your existing data access logic using the CqlSession interface remains unchanged.

code_block: <ListValue: [StructValue([('code', 'CqlSession session =\r\n SpannerCqlSession.builder() \r\n .setDatabaseUri("projects/your_gcp_project/instances/your_spanner_instance/databases/your_spanner_database")\r\n .withKeyspace("your_spanner_database") // Corresponds to the Spanner database name\r\n .withConfigLoader(\r\n DriverConfigLoader.fromFile(new File("/path/to/config/file"))) // Optional: path to driver config\r\n .build();\r\n\r\n// Your existing code using the \'session\' object remains unchanged.'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fa402905dc0>)])]>

Use the Spanner Cassandra sidecar proxy: For applications where embedding the client library is not feasible or for connecting standard tools like cqlsh, you can deploy the Spanner Cassandra sidecar proxy. This proxy runs as a separate process, often containerized, alongside your application and handles the translation between the Cassandra wire protocol and Spanner's API.

At this point, your application and data should be successfully migrated to Spanner. We have more detailed migration instructions available in the documentation.

Spanner is cost-effective for your Cassandra workloads

Now that you’ve migrated your workloads to Spanner, you may wonder about costs. In fact, Spanner is a cost-effective alternative to Cassandra. Here’s why:

Fully managed service: This reduces your operational costs.
Enterprise license-Free, Usage-Based Billing: Spanner usage is billed hourly with no extra licensing fees, meaning no upfront costs, and you only pay for what you use.
Elastic scaling with Autoscaler: Spanner’s Autoscaler allows precise scaling according to workload requirements. Avoid over-provisioning for peak performance and pay only for the resources you consume.
Granular instances: Start small with a Spanner granular instance (100 processing units) for as little as $65/month.
Built-in observability dashboards: No need to pay extra to store and analyze your usage logs. The Spanner console has everything you need.
Tiered storage: For storage-heavy workloads, Spanner offers tiered storage options that can be up to 80% cheaper than solid-state drive (SSD) storage.
Incremental backups: Benefit from substantial cost savings on backups with Spanner's incremental backups.

A recent Total Economic Impact study by Forrester Consulting found that Spanner delivered a 132% ROI and $7.74 million in total benefits over three years for a composite organization representative of interviewed customers. These gains largely stemmed from retiring self-managed databases and leveraging Spanner’s elastic scalability and built-in, hands-free high availability operations.

Migrating to Spanner is just the starting point

Migrating your Cassandra application to Spanner is the first step to unlocking a world of new possibilities for your data. While your application continues interacting with Cassandra via CQL, you can build new microservices that access the same data. You can even consider moving your connecting ETL pipelines to Spanner leveraging Spanner's full multi-model capabilities. You can also implement sophisticated graph queries to analyze relationships between users, utilize full-text search across user profiles or product catalogs, or add vector embedding support for recommendation engines or similarity searches, combining transactional data with AI/ML workloads directly. Lastly, Spanner is tightly integrated with BigQuery, allowing you to unlock valuable insights and drive better decision-making by connecting operational and analytical workloads.

Get started today by testing our migration kit via the Spanner free trial. We believe this will empower next-gen application builders like you to modernize your Cassandra workloads and unlock the full potential of your data on Google Cloud with Spanner.

Google Cloud Spanner ranked #1 by Gartner® for Lightweight Transaction Use Case

Thu, 01 May 2025 18:00:00 +0000

We're thrilled to share that Google Cloud Spanner has been recognized by Gartner in the Critical Capabilities for Cloud Database Management Systems for Operational Use Cases report, where it was ranked #1 in the Lightweight Transactions Use Case and was ranked #3 in the OLTP Transactions Use Case and the Application State Management Use Case. This recognition showcases Spanner's strength and versatility to handle the most demanding workloads.

Check out the full Gartner report.

Beyond traditional transactions: Expanding capabilities

We believe the Gartner recognition isn't just about raw performance. We feel it's about Spanner's comprehensive feature set, which is designed to address the complex needs of modern enterprises. Beyond its renowned transactional consistency and global scalability, Spanner offers a powerful multi-model experience, seamlessly integrating the graph, full-text, and vector search functionality required by modern applications.

Graph database functionality: Spanner's ability to model and query relationships makes it a strong fit for applications requiring graph analysis, such as social networks, fraud detection, and recommendation engines.
Full-text search: Integrated full-text search capabilities enable efficient retrieval of unstructured data, powering features like product catalogs, content management systems, and knowledge bases.
Vector search: With the rise of AI and machine learning, Spanner's vector search capabilities facilitate similarity searches, enabling applications like image recognition, semantic search, and personalized recommendations.

This flexibility allows developers to build diverse applications on a single platform that provides dynamic elasticity combined with operational efficiency without the complexity of managing multiple specialized databases.

aside_block: <ListValue: [StructValue([('title', '$300 in free credit to try Google Cloud databases'), ('body', <wagtail.rich_text.RichText object at 0x7fa3f87c6e80>), ('btn_text', 'Start building for free'), ('href', 'http://console.cloud.google.com/freetrial?redirectPath=/products?#databases'), ('image', None)])]>

A truly global service: Transactions and analytics combined

Spanner's global footprint helps ensure low latency and high availability for transactional workloads, regardless of a user’s location. But its power extends beyond transactions. Spanner's deep integration with BigQuery allows for federated queries, enabling real-time analytics on transactional data without the need for complex ETL processes. This integration also supports reverse ETL from BigQuery, allowing you to push analytical insights back into Spanner for operational use.

Real-world impact: Customer success stories

The true testament to Spanner's capabilities is its impact on our customers. Here's a sampling of how it’s being used in the field:

Spanner's high availability, external consistency, and infinite horizontal scalability made it the ideal choice for Deutsche Bank’s business critical application for online banking.
By consolidating all user data with the exception of logs to a single database for development, COLOPL has eliminated the scalability constraints that occurred when using horizontally and vertically partitioned databases for large-scale services.
With Spanner's fully-managed relational database, Kroger has been able to build a true event-driven ledger, which enables the company to capture unique events to make better-informed decisions about how to direct associates to be more productive.

Looking ahead

We believe Spanner's recognition in the Gartner Critical Capabilities report reinforces Google’s position in the Cloud Database Management Systems market. We're committed to continuing to innovate and expand Spanner's capabilities, empowering our customers to build the next generation of mission-critical applications.

Whether you need a database for global transactions, multi-model applications or real-time analytics, Spanner is the solution you can rely on. Sign up for a free Spanner trial account and experience the power of multi-model Spanner today.

^{Gartner Critical Capabilities for Cloud Database Management Systems for Operational Use Cases, Ramke Ramakrishnan, Henry Cook, Xingyu Gu, Masud Miraz, Aaron Rosenbaum, 18 December, 2024.}

^{GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. in the U.S. and internationally and is used herein with permission. All rights reserved. This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from Google. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.}

Migrating your apps from MySQL to Spanner just got easier

Thu, 24 Apr 2025 16:00:00 +0000

Yesterday’s databases aren‘t sufficient for tomorrow’s applications, which need to deliver dynamic, AI-driven experiences at unpredictable scale and with zero downtime. To help, at Google Cloud Next 25, we announced new functionality, improved performance, and migration tooling to simplify modernizing database workloads from MySQL to Spanner, Google Cloud’s horizontally scalable, always-on operational database.

MySQL simply wasn't designed for today’s most pressing scaling and availability needs. Common fixes, like sharding or manual replication, are complex and risky—coming exactly at the time when the business can tolerate it least. Planning and executing scaling on self-managed databases typically require expensive after-market solutions, which can take months to architect and test, diverting development teams from more pressing user-facing features. And because of the overhead of scaling, organizations often provision for peak usage, even if that capacity remains unused most of the time.

Tomorrow’s applications also need to do more than just transaction processing. New experiences like semantic discovery, collaborative recommendations, real-time fraud detection, and dynamic pricing, require different ways of storing and querying data.

aside_block: <ListValue: [StructValue([('title', '$300 in free credit to try Google Cloud databases'), ('body', <wagtail.rich_text.RichText object at 0x7fa3f8353340>), ('btn_text', 'Start building for free'), ('href', 'http://console.cloud.google.com/freetrial?redirectPath=/products?#databases'), ('image', None)])]>

Simpler live migrations from MySQL to Spanner

To help organizations struggling to grow and modernize their apps, Spanner provides a well-defined migration path to safely and easily move production workloads from MySQL with virtually no downtime. Once there, they can take advantage of Spanner’s hands-free reliability and rich graph, full-text search, and integrated AI capabilities.

A key part of this is Spanner migration tool, which automates schema and data migration to support live cutovers, including consolidating petabyte-sized sharded MySQL databases in days, not months. Improved data movement templates provide increased throughput at significantly lower cost as well as new flexibility to transform data as it’s migrated, and updated built-in reverse replication synchronizes data back from Spanner to sharded MySQL instances to allow for near real-time failover in a disaster scenario. Finally, new Terraform configurations and CLI integration provide flexibility to customize implementations.

Spanner migration tool architecture

Improved latency with fewer code and query changes

To further reduce the cost and complexity of migrating application code and queries, we introduced a rich new set of relational capabilities in Spanner that map closely to MySQL.

Repeatable read is the default isolation level in MySQL, balancing performance and consistency. We’re excited to bring this flexibility to Spanner as well. New repeatable read isolation, now in preview, complements Spanner’s existing serializable isolation. It will be familiar to MySQL developers and gives them additional tools to significantly improve performance. In fact, most common workloads can see up to a 5x latency improvement compared to what was possible in Spanner previously. In addition, new auto_increment keys, SELECT…FOR UPDATE, and close to 80 new MySQL functions dramatically reduce the changes required to migrate an application to Spanner.

“As our calendar sharing service gained popularity, demand grew steadily. At 55 million users, we hit Aurora MySQL’s scalability limits for both data volume and active connections. But scalability wasn't the only issue. Our app teams spent too much time managing the database, leaving less for feature development. Fully managed Spanner solved this, significantly cutting costs and enabling future growth. Migrations are challenging, but with Google Cloud support and the Spanner migration tool, we completed it successfully with minimal downtime.” - Eiki Kanai, SRE Manager, TimeTree

A recent Total Economic Impact study from Forrester Consulting also found that Spanner provided a 132% ROI and $7.74M of total benefits over three years for a composite organization representative of interviewed customers. This comes largely from retiring self-managed databases and taking advantage of Spanner’s elastic scalability and built-in, hands-free, high availability operations. Forrester found that decreased disruptions from unplanned downtime and system maintenance with Spanner reduced time to onboard new apps and allowed development teams to address new opportunities without complex re-architecture projects or new capital expenditures.

Get started today

To learn more about how Spanner can take the stress out of your organization’s next growth milestone and set your development teams up for success, visit https://cloud.google.com/spanner. There, you’ll find reference architectures, examples of successful migrations, and a directory of qualified partners to help with a free assessment. Read up on how to run a successful migration from MySQL. Or try Spanner yourself today with a 90-day free trial and production instances starting around $65 USD/month.

Forrester study reveals significant benefits and cost savings with Spanner

Wed, 05 Mar 2025 17:00:00 +0000

Is your legacy database sticking you with rising costs, frustrating downtime, and scalability challenges? For organizations that strive for top performance and agility, legacy database systems can become significant roadblocks to innovation.

But there's good news. According to a new Forrester Total Economic Impact™ (TEI) study, organizations may realize significant benefits by deploying Spanner, Google Cloud’s always-on, globally consistent, multi-model database with virtually unlimited scale. What kind of benefits? We’re talking an ROI of 132% over three years, and multi-million-dollar benefits and cost savings for a representative composite organization.

Read on for more, then download the full study to see the results and learn how Spanner can help your organization increase cost savings and profit, as well as reliability and operational efficiencies.

The high cost of the status quo

Legacy, on-premises databases often come with a hefty price tag that goes far beyond initial hardware and software investments. According to the Forrester TEI study, these databases can be a burden to maintain, requiring dedicated IT staff and specialized expertise, as well as high capital expenditures and operational overhead. Outdated systems can also limit your ability to respond quickly to changing market demands and customer needs, such as demand spiking for a new game or a viral new product.

aside_block: <ListValue: [StructValue([('title', '$300 in free credit to try Google Cloud databases'), ('body', <wagtail.rich_text.RichText object at 0x7fa3d6f8c190>), ('btn_text', 'Start building for free'), ('href', 'http://console.cloud.google.com/freetrial?redirectPath=/products?#databases'), ('image', None)])]>

Spanner's economic impact: Forrester's key findings

To quantify the benefits that Spanner can bring to an organization, Forrester used its TEI methodology, conducting in-depth interviews with seven leading organizations across the globe who had adopted Spanner. These organizations came from a variety of industries such as retail, financial services, software and technology, gaming, and transportation. Based on its findings, Forrester created a representative composite organization: a business-to-consumer (B2C) organization with revenue of $1 billion per year, and modeled the potential financial impact of adopting Spanner.

In addition to a 132% return on investment (ROI) with a 9-month payback period, Forrester found that the composite organization also realized $7.74M in total benefits over the three years, from a variety of sources:

Cost savings from retiring on-prem legacy database: By retiring the on-prem legacy database and transitioning to Spanner, the composite organization can save $3.8 million over three years. Savings came from reduced infrastructure capital expenditure, maintenance costs, and system licensing expenses.

“The system before migration was more expensive. It was the cost of the entire system including the application, database, monitoring, and everything. We paid within the $5 million to $10 million range for a mainframe, and I expect that the cost of it would almost double within the next few years. Currently, we pay 90% less for Spanner.” - Senior Principal Architect at a software and technology organization

Profit retention and cost savings from reduced unplanned downtime: Prior to adopting Spanner, organizations suffered unplanned database downtime triggered by technical malfunctions, human errors, data integration issues, or natural disasters. With up to 99.999% availability, Spanner virtually eliminates unplanned downtime. Forrester calculates that the composite organization achieves $1.2 million in cost savings and profit retention due to reduced unplanned downtime.

“In the last seven years since we migrated to Spanner, the total number of failures caused by Spanner is zero. Prior to Spanner, some sort of problem would occur about once a month including a major problem once a year.” - Tech Lead, gaming organization

Cost savings from reduced overprovisioning for peak usage: With on-prem database systems, long infrastructure procurement cycles and large up-front expenditures mean that organizations typically provision for peak usage — even if that means they are over-provisioned most of the time. Spanner's elastic scalability allows organizations to start small and scale up and down effortlessly as usage changes. Databases can scale up for however long you need, and then down again, cutting costs and the need to predict usage. For the composite organization, this results in cost savings of $1 million over three years.

“The number of transactions we are able to achieve is one of the main reasons that we use Spanner. Additionally, Spanner is highly consistent, and we save on the number of engineers needed for managing our databases." - Head of SRE, DevOps, and Infrastructure, financial services organization

Efficiencies gained in onboarding new applications: Spanner accelerates development of new applications by eliminating the need to preplan resources. This resulted in 80% reduction in time to onboard new applications and $981,000 in cost savings for the composite organization.

Beyond the numbers

Beyond the quantifiable ROI, the Forrester TEI study highlights unquantified benefits that amplify Spanner's value. These include:

Improved budget predictability, as Spanner shifts expenditures from capex to opex, enabling more effective resource allocation and forecasting.
Greater testing and deployment flexibility, allowing software development engineers to rapidly scale development environments for testing, conduct thorough load tests, and quickly shut down resources.
Expert Google Cloud customer service, providing helpful guidance to maximize Spanner's benefits.

“The Spanner team are experts. They have a deep understanding of the product they’ve built with deep insights on how we’re using the product if we ask them.” - Head of Engineering, financial services organization

An innovation-friendly architecture, facilitating the design and implementation of new business capabilities and expansion, improving automation and customer satisfaction, all without incurring downtime.

Together, these strategic advantages contribute to organizational agility and long-term success.

Unlock the potential of your data with Spanner

We believe the Forrester TEI study clearly demonstrates that Spanner is more than just a database; it's a catalyst for business transformation. By eliminating the constraints of legacy systems, Spanner empowers organizations to achieve significant cost savings, improve operational efficiencies, and unlock new levels of innovation. Are you ready to transform your data infrastructure and unlock your organization's full potential?

Download the full Forrester Total Economic Impact study

Unico builds cutting-edge IDTech with Spanner Vector Search

Tue, 04 Mar 2025 17:00:00 +0000

Unico is a leading biometric verification and authentication company addressing the global challenges of identity management and fraud prevention.

With nearly two decades of experience in the Brazilian market, Unico has become a reliable supplier to over 800 companies, including four of the top five largest banks and leading retailers. Since 2021, Unico has facilitated more than 1.2 billion authentications through digital identity. It is estimated that Unico's solutions have thwarted $14 billion in fraud, by 2023. Valued at over $2.6 billion, the company stands as the second most valuable SaaS company in Latin America, backed by General Atlantics, SoftBank, and Goldman Sachs, and was recognized as the third most innovative company in Latin America by Fast Company in 2024.

Currently, working on its global expansion, Unico has an ambitious vision to become the main identity network in the world, thus moving beyond traditional ID verification and embracing a broader spectrum of identity-related technologies. In this article, we’ll explore how Google Cloud and Spanner — Google’s always-on, virtually unlimited-scale database — is helping Unico achieve this goal.

Why vector search shines in Unico's solutions

Unico is committed to delivering innovative, cutting-edge digital identity solutions. A cornerstone of this effort is the use of vector search technology, which enables powerful capabilities like 1:N search — the ability to search for a single face within a large set of many others. This technology drives Unico’s identity solutions by retrieving and ranking multiple relevant matches for a given query with high precision and speed.

However, developing 1:N searches poses a significant challenge: efficiently verifying facial matches within databases containing millions or billions of registered face vectors. Comparing an individual’s facial characteristics against each entry one by one is impractical. To address this, vector databases are often employed to perform Approximate Nearest Neighbor searches (ANN) and return the top-N most similar faces.

Unico found that Spanner supports vector search capabilities to solve these issues, providing:

Semantic retrieval: Leveraging vector embeddings, Unico’s solutions can retrieve results based on deeper semantic relationships rather than exact matches. This improves the quality of identity verification, such as identifying relevant facial matches even when minor variations exist between the source and target images.
Diversity and relevance: Using algorithms like ANN and exact K-Nearest Neighbors (KNN) vector search balances the need for diverse and relevant results, ensuring high reliability in fraud detection and identity verification.
Multimodal applications: Vector search supports embeddings from multiple data types, such as text, images, and audio, enabling its use in complex, multimodal identity scenarios.
Hybrid search: Modern vector search frameworks combine similarity search with metadata filters, allowing tailored results based on context, such as region or user preferences.

By integrating vector search, Unico provides customers with faster and smarter fraud detection tools. Leveraging high-precision algorithms, these tools can identify fraudulent faces with exceptional accuracy, effectively safeguarding businesses and individuals against identity theft and other threats. This innovation not only solidifies Unico’s position as a technology leader but also underscores its mission to build a safer, and more trusted world by creating a unified ecosystem to validate people's real identities.

aside_block: <ListValue: [StructValue([('title', '$300 in free credit to try Google Cloud databases'), ('body', <wagtail.rich_text.RichText object at 0x7fa3f82aa9a0>), ('btn_text', 'Start building for free'), ('href', 'http://console.cloud.google.com/freetrial?redirectPath=/products?#databases'), ('image', None)])]>

Some results

Operating at low latency while maintaining accuracy is crucial for Unico’s business, especially in applications that demand real-time performance, such as banking. Spanner was Unico’s first choice because its integrated support for vector search eliminates the need for separate, specialized vector database solutions.

Spanner provides transactional guarantees for operational data, delivers fresh and consistent vector search results, and offers horizontal scalability. Its features also include GRPC (Google Remote Procedure Call) support, geo partitioning, multi-region storage configuration, RAG and LLM integrations, high SLA levels (99.99%), and maintenance-free architecture. Spanner also currently supports KNN and ANN vector searches.

Unico currently operates 1:N services in Brazil and Mexico, storing more than 1 billion facial embeddings in Spanner to date. This setup enables Unico to achieve low latency at high percentiles, high throughput of 840 RPM, and a precision/recall of 96%. And it’s just the start — Unico processes around 35 million new faces every month and that number continues to grow.

Unico remains focused on growing its customer base, enhancing its existing products, and exploring new opportunities in international markets — with the aim of expanding the reach of its secure digital identity services beyond Brazil’s borders. With Spanner and the ability to tap into the full power of the Google Cloud’s ecosystem, Unico is confident that it can bring its ambitious vision to life and deliver innovative solutions that forge trust between people and companies.

Learn more:

Discover how Spanner can transform your business.
Get started on Spanner today with a 90 day free trial instance. Or try out Spanner Vector Search.

Learn how Spanner Vector Search, hybrid search, and Spanner Graph can supercharge AI apps at virtually unlimited scale.

8 steps to ensuring a smooth Spanner go-live

Fri, 21 Feb 2025 17:00:00 +0000

As a developer, there’s a lot to think about when you’re getting ready to launch an application. There’s the availability of the underlying database, of course, which stores application state, and determines how fast and you can recover if your application or web servers go down. Thankfully, if you’re running on Spanner, its 99.999% availability architecture means that application owners don’t have to worry about designing and operating active-active database deployments. While Spanner itself is designed to offer high availability, as with any other database, it’s important to have a checklist to ensure a successful launch of your workload as a whole and subsequent ongoing operation. We’ve captured a comprehensive list of items to keep in mind when launching workloads on Spanner in the Spanner Launch Checklist. In this blog post, we go over the key steps in the checklist, and provide additional color about each item. Once you’re done, you’ll be equipped to launch your Spanner-based application with confidence.

Let’s explore the key steps to focus on when gearing up for a Spanner launch.

1. Design, develop, test, optimize

Your Spanner schema and transactions are the blueprint for success. Designing for Spanner’s distributed architecture isn’t just a good idea — it’s the key to unlocking its full potential. Ensure your schema avoids bottlenecks and hotspots, and that transactions are tailored for minimal locking and maximum performance. But the real game-changer? Testing. Rigorous, real-world load testing, complete with traffic spikes and realistic concurrency, helps ensure your workload is ready for production scale and capable of handling unexpected spikes in traffic.

2. Choose your deployments well

Spanner deployments entail more than just provisioning nodes; they’re about aligning infrastructure with your business goals. Choose configurations that balance availability and performance while accounting for traffic peaks with features like auto-scaling. Geography matters as well — place leader regions near your users for low-latency performance. And don’t forget the small stuff, like setting up tags and labels for resource tracking or warming up your database for a smooth start.

3. Implement backup and disaster recovery

Downtime isn’t an option when critical data is on the line. A solid backup and recovery strategy helps your business weather unexpected failures. Define recovery objectives, automate backups, and test restore procedures regularly. Features like incremental backups and Point-in-Time Recovery (PITR) make Spanner a reliable ally in your data protection arsenal.

4. Stay secure

Protecting your data goes beyond locking it up. Spanner’s security features, like fine-grained access controls and least-privilege permissions, let you limit access to only what’s necessary and for the right principals. Add an extra layer of defense with deletion protection and carefully designed IAM policies. With these safeguards, your Spanner instance remains both robust and secure.

aside_block: <ListValue: [StructValue([('title', '$300 in free credit to try Google Cloud databases'), ('body', <wagtail.rich_text.RichText object at 0x7fa402164d00>), ('btn_text', 'Start building for free'), ('href', 'http://console.cloud.google.com/freetrial?redirectPath=/products?#databases'), ('image', None)])]>

5. Deploy logging, monitoring, and observability

To keep Spanner running smoothly, you need visibility. Audit logs, distributed tracing with OpenTelemetry, and real-time dashboards give you a clear view of what’s happening under the hood. Proactive alerts ensure you’re ready to act before small issues turn into big problems. We also invite you to check out Database Center, which provides a centralized view across your entire database fleet, including Spanner instances.

6. Optimize your configuration with the client library:

Spanner’s client library is more than a connection — it’s a tool for optimization. Add transaction tags for enhanced debugging, fine-tune session pooling to match workloads, and tweak retry policies to handle errors with grace. These configurations boost both performance and resilience.

7. Cut costs without compromising performance

Managing costs isn’t about cutting corners. Take advantage of committed use discounts for predictable savings and let auto-scaling handle spikes while minimizing idle resources. For non-production environments, scale down or shut off instances when they’re not needed.

8. Plan your Migration

If you’re moving to Spanner, a well-planned migration makes all the difference. From orchestrating a seamless cutover to preparing a reliable fallback strategy, every step counts. Communicate with stakeholders, validate data, and monitor the process to ensure success.

Ready, set, scale!

Launching a successful workload or application is the result of careful planning, rigorous testing, and proactive management. And it’s no different for workloads running on Spanner.

Deploying mission-critical workloads on Spanner is about building a solid foundation for scalability, resilience and peak performance. A well-structured launch checklist ensures you’ve covered all the bases, from design and deployment to backup strategies and cost management. You can review the Spanner Launch Checklist in the Spanner documentation.

Announcing the general availability of Spanner Graph

Thu, 30 Jan 2025 17:00:00 +0000

In today's complex digital world, building truly intelligent applications requires more than just raw data — you need to understand the intricate relationships within that data. Graph analysis helps reveal these hidden connections, and when combined with techniques like full-text search and vector search, enables you to deliver a new class of AI-enabled application experiences. However, traditional approaches based on niche tools result in data silos, operational overhead, and scalability challenges. That's why we introduced Spanner Graph, and today we’re excited to announce that it’s generally available.

In a previous post, we described how Spanner Graph reimagines graph data management with a unified database that integrates graph, relational, search, and gen AI capabilities with virtually unlimited scalability. With Spanner Graph, you gain access to an intuitive ISO Standard Graph Query Language (GQL) interface that simplifies pattern matching and relationship traversal. You also benefit from full interoperability between GQL and SQL, for tight integration between graph and tabular data. Powerful vector and full-text search enable fast data retrieval using semantic meaning and keywords. And you can rely on Spanner’s scalability, availability, and consistency to provide a solid data foundation. Finally, integration with Vertex AI gives you access to powerful AI models directly within Spanner Graph.

aside_block: <ListValue: [StructValue([('title', '$300 in free credit to try Google Cloud databases'), ('body', <wagtail.rich_text.RichText object at 0x7fa4024bf2b0>), ('btn_text', 'Start building for free'), ('href', 'http://console.cloud.google.com/freetrial?redirectPath=/products?#databases'), ('image', None)])]>

What’s new in Spanner Graph

Since the preview, we have added exciting new capabilities and partner integrations to make it easier for you to build with Spanner Graph. Let’s take a closer look.

1) Spanner Graph Notebook: Graph visualization is key to developing with graphs. The new open-source Spanner Graph Notebook tool provides an efficient way to query Spanner Graph visually. This tool is natively installed in Google Colab, meaning you can use it directly within that environment. You can also leverage it in notebook environments like Jupyter Notebook. With this tool, you can use magic commands with GQL to visualize query results and graph schemas with multiple layout options, inspect node and edge properties, and analyze neighbor relationships.

Open-source Spanner Graph Notebook.

2) GraphRAG with LangChain integration: Spanner Graph's integration with LangChain allows for quick prototyping of GraphRAG applications. Conventional RAG, while capable of grounding the LLM by providing relevant context from your data using vector search, cannot leverage the implicit relationships present in your data. GraphRAG overcomes this limitation by constructing a graph from your data that captures these complex relationships. At retrieval time, GraphRAG uses the combined power of graph queries with vector search to provide a richer context to the LLM, enabling it to generate more accurate and relevant answers.

3) Graph schema in Spanner Studio: The Spanner Studio Explorer panel now displays a list of defined graphs, their nodes and edges, and the associated labels and properties. You can explore and understand the structure of your graph data at a glance, making it easier to design, debug, and optimize your applications.

4) Graph query improvements: Spanner Graph now supports the path data type and functions, allowing you to retrieve and analyze the specific sequence of nodes and relationships that connect two nodes in your graph. For example, you can create a path variable in a path pattern, using the IS_ACYCLIC function to check if the path has repeating nodes, and return the entire path:

code_block: <ListValue: [StructValue([('code', 'GRAPH FinGraph\r\nMATCH p = (:Account)-[:Transfers]->{2,5}(:Account)\r\nRETURN IS_ACYCLIC(p) AS is_acyclic_path, TO_JSON(p) AS full_path;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fa4049335b0>)])]>

5) Graph visualization partner integrations: Spanner Graph is now integrated with leading graph visualization partners. For example, Spanner Graph customers can use GraphXR, Kineviz’s flagship product, which combines cutting-edge visualization technology with advanced analytics to help organizations make sense of complex, connected data.

"We are thrilled to partner with Google Cloud to bring graph analytics to big data. By integrating GraphXR with Spanner Graph, we're empowering businesses to visualize and interact with their data in ways that were previously unimaginable." - Weidong Yang, CEO, Kineviz

Transit Fraud Investigation in Kineviz GraphXR and Google Spanner Graph

Similarly, you can use Graphistry’s GPU-accelerated visual graph intelligence platform to extract meaningful insights from large complex data in Spanner Graph.

"Businesses can finally handle graph data with both speed and scale. By combining Graphistry’s GPU-accelerated graph visualization and AI with Spanner Graph’s global-scale querying, teams can now easily go all the way from raw data to graph-informed action. Whether detecting fraud, analyzing journeys, hunting hackers, or surfacing risks, this partnership is enabling teams to move with confidence." - Leo Meyerovich, Founder and CEO, Graphistry

Visual analytics capabilities in Graphistry: zooming, clustering, filtering, histograms, time bar filtering, node styling (colors), allowing point-and-click analysis to quickly understand the data, clusters, identify patterns, anomalies and other insights.

Furthermore, you can use G.V(), a quick-to-install graph database client, with Spanner Graph to perform day-to-day graph visualization and data analytics tasks with ease. Data professionals benefit from high-performance graph visualization, no-code data exploration, and highly customizable data visualization options.

“Graphs thrive on connections, which is why I’m so excited about this new partnership between G.V() and Google Cloud Spanner Graph. Spanner Graph turns big data into graphs, and G.V() effortlessly turns graphs into interactive data visualizations. I’m keen to see what data professionals build combining both solutions.” - Arthur Bigeard, Founder, gdotv Ltd.

Visually querying and exploring Spanner Graph with G.V().

What customers are saying

Through our road to GA, we have also been working with multiple customers to help them innovate with Spanner Graph:

“The Commercial Financial Network manages commercial credit data for more than 30 million U.S. businesses Managing the hierarchy of these businesses can be complex due to the volume of these hierarchies, as well as the dynamic nature driven by mergers and acquisitions, Equifax is committed to providing lenders with the accurate, reliable and timely information they need as they make financial decisions. Spanner Graph helps us manage these rapidly changing, dynamic business hierarchies easily at scale.” - Yuvaraj Sankaran, Chief Architect of Global Platforms, Equifax

“As we strive to enhance our fraud detection capabilities, having a robust, multi-model database like Google Spanner is crucial for our success. By integrating SQL for transactional data management with advanced graph data analysis, we can efficiently manage and analyze evaluated fraud data. Spanner's new capabilities significantly improve our ability to maintain data integrity and uncover complex fraud patterns, ensuring our systems are secure and reliable.” - Hai Sadon, Data Platform Group Manager, Transmit Security

"Spanner Graph has provided a novel and performant way for us to query this data, allowing us to deliver features faster and with greater peace of mind. Its flexible data modeling and high-performance querying have made it far easier to leverage the vast amount of data we have in our online applications." - Aaron Tang, Senior Principal Engineer, U-NEXT

Get started with Spanner Graph today

With Spanner Graph, we’re excited to offer graph data management alongside relational, search, and AI capabilities, all on a single unified, highly scalable database. Learn more about Spanner Graph benefits and use cases here. Use this quick setup guide to get started with Spanner Graph capabilities. You can also try out sample applications related to financial investing, fraud detection, and customer 360 and product recommendations.

Avoid global outages by partitioning cloud applications to reduce blast radius

Thu, 09 Jan 2025 17:00:00 +0000

Cloud applications like Google Workspace provide benefits such as collaboration, availability, security, and cost-efficiency. However, for cloud application developers, there’s a fundamental conflict between achieving high availability and the constant evolution of cloud applications. Changes to the application, such as new code, configuration updates, or infrastructure rearrangements, can introduce bugs and lead to outages. These risks pose a challenge for developers, who must balance stability and innovation while minimizing disruption to users.

Here on the Google Workspace Site Reliability Engineering team, we once moved a replica of Google Docs to a new data center because we needed extra capacity. But moving the associated data, which was vast, overloaded a key index in our database, restricting user ability to create new docs. Thankfully, we were able to identify the root cause and mitigate the problem quickly. Still, this experience convinced us of the need to reduce the risk of a global outage from a simple application change.

aside_block: <ListValue: [StructValue([('title', 'Try Google Cloud for free'), ('body', <wagtail.rich_text.RichText object at 0x7fa4049f1790>), ('btn_text', 'Get started for free'), ('href', 'https://console.cloud.google.com/freetrial?redirectPath=/welcome'), ('image', None)])]>

Limit the blast radius

Our approach to reducing the risk of global outages is to limit the “blast radius,” or extent, of an outage by vertically partitioning the serving stack. The basic idea is to run isolated instances (“partitions”) of application servers and storage (Figure 1). Each partition contains all the various servers necessary to service a user request from end to end. Each production partition also has a pseudo-random mix of users and workloads, so all the partitions have similar resource needs. When it comes time to make changes to the application code, we deploy new changes to one partition at a time. Bad changes may cause a partition-wide outage, but we are protected from a global application outage.

Compare this approach to using canarying alone, in which new features or code changes are released to a small group of users before rolling them out to the rest. While canarying deploys changes first to just a few servers, it doesn’t prevent problems from spreading. For example, we’ve had incidents where canaried changes corrupted data used by all the servers in the deployment. With partitioning, the effects of bad changes are isolated to a single partition, preventing such contagion. Of course, in practice, we combine both techniques: canarying new changes to a few servers within a single partition.

Benefits of partitioning

Broadly speaking, partitioning brings a lot of advantages:

Availability: Initially, the primary motivation for partitioning was to improve the availability of our services and avoid global outages. In a global outage, an entire service may be down (e.g., users cannot log into Gmail), or a critical user journey (e.g., users cannot create Calendar events) — obviously things to be avoided.

Still, the reliability benefits of partitioning can be hard to quantify; global outages are relatively infrequent, so if you don’t have one for a while, it may be due to partitioning, or may be due to luck. That said, we’ve had several outages that were confined to a single partition, and believe they would have expanded into global outages without it.
Flexibility: We evaluate many changes to our systems by experimenting with data. Many user-facing experiments, such as a change to a UI element, use discrete groups of users. For example, in Gmail we can choose an on-disk layout that stores the message bodies of emails inline with the message metadata, or a layout that separates them into different disk files. The right decision depends on subtle aspects of the workload. For example, separating message metadata and bodies may reduce latency for some user interactions, but requires more compute resources in our backend servers to perform joins between the body and metadata columns. With partitioning, we can easily evaluate the impact of these choices in contained, isolated environments.
Data location: Google Workspace lets enterprise customers specify that their data be stored in a specific jurisdiction. In our previous, non-partitioned architecture, such guarantees were difficult to provide, especially since services were designed to be globally replicated to reduce latency and take advantage of available capacity.

Challenges

Despite the benefits, there are some challenges to adopt partitioning. In some cases, these challenges make it hard or risky to move from a non-partitioned to a partitioned setup. In other cases, challenges persist even after partitioning. Here are the issues as we see them:

Not all data models are easy to partition: For example, Google Chat needs to assign both users and chat rooms to partitions. Ideally, a chat and its members would be in a single partition to avoid cross-partition traffic. However, in practice, this is difficult to accomplish. Chat rooms and users form a graph, with users in many chat rooms and chat rooms containing many users. In the worst case, this graph may have only a single connected component — the user. If we were to slice the graph into partitions, we could not guarantee that all users would be in the same partition as their chat rooms.
Partitioning a live service requires care: Most of our services pre-date partitioning. As a result, adopting partitioning means taking a live service and changing its routing and storage setup. Even if the end goal is higher reliability, making these kinds of changes in a live system is often the source of outages, and can be risky.
Partition misalignment between services: Our services often communicate with each other. For example, if a new person is added to a Calendar event, Calendar servers make an Remote Procedure Call (RPC) to Gmail delivery servers to send the new invitee an email notification. Similarly, Calendar events with video call links require Calendar to talk to Meet servers for a meeting id. Ideally, we would get the benefits of partitioning even across services. However, aligning partitions between services is difficult. The main reason is that different services tend to use different entity types when determining which partition to use. For example, Calendar partitions on the owner of the calendar while Meet partitions on meeting id. The result is that there is no clear mapping from partitions in one service to another.
Partitions are smaller than the service: A modern cloud application is served by hundreds or thousands of servers. We run servers at less than full utilization so that we can tolerate spikes in traffic, and because servers that are saturated with traffic generally perform poorly. If we have 500 servers, and target each at 60% CPU utilization, we effectively have 200 spare servers to absorb load spikes. Because we do not fail over between partitions, each partition has access to a much smaller amount of spare capacity. In a non-partitioned setup, a few server crashes may likely go unnoticed, since there is enough headroom to absorb the lost capacity. But in a smaller partition, these crashes may account for a non-trivial portion of the available server capacity, and the remaining servers may become overloaded.

Key takeaways

We can improve the availability of web applications by partitioning their serving stacks. These partitions are isolated, because we do not fail over between them. Users and entities are assigned to partitions in a sticky manner to allow us to roll out changes in order of risk tolerance. This approach allows us to roll out changes one partition at a time with confidence that bad changes will only affect a single partition, and ideally that partition contains only users from your organization.

In short, partitioning supports our efforts to provide stronger and more reliable services to our users, and it might apply to your service as well. For example, you can improve the availability of your application by using Spanner, which provides geo-partitioning out of the box. Read more about geo-partitioning best practices here.

References

Spanner in 2024: A year of innovation

Fri, 20 Dec 2024 17:00:00 +0000

Spanner is Google’s always-on, virtually unlimited database that powers planet-scale applications like Gmail, YouTube, and Google Photos. Outside of Google, Spanner powers demanding workloads for household brands like Yahoo!, The Home Depot, Wayfair, and Pokémon Go. Today, Spanner handles over 4 billion queries per second at peak and more than 15 exabytes of data, with five 9s of availability, plus global consistency.

Since we first discussed it in 2012, Spanner has evolved from a groundbreaking distributed SQL database into a versatile, intelligent innovation platform. 2024 was a big year for Spanner, with multiple launches that expanded its functional capabilities, pushed the envelope on price-performance, re-architected it for best-in-class reliability and security, and enhanced the developer experience. Here is a recap of Spanner’s biggest innovations of the year and how you can benefit from them.

aside_block: <ListValue: [StructValue([('title', '$300 in free credit to try Google Cloud databases'), ('body', <wagtail.rich_text.RichText object at 0x7fa401780580>), ('btn_text', 'Start building for free'), ('href', 'http://console.cloud.google.com/freetrial?redirectPath=/products?#databases'), ('image', None)])]>

1. Multi-model: one database, many possibilities

With the launch of Spanner Graph, full-text search and vector search, Spanner went from being a highly available, globally consistent and scalable database, to a multi-model database with intelligent, interoperable capabilities with which you can build AI-enabled applications. Unlike other multi-model databases on the market, Spanner offers a true multi-model experience that allows interoperability between different data models without downtime.

Spanner's multi-model support lets you consolidate databases, saving on costs and reducing operational overhead, governance, and security touchpoints, while its interoperability eliminates data movement for a "true ZeroETL" experience with consistent data across all models. This helps enable use cases like near-real-time fraud detection, supply chain optimization, or product recommendations.

Fig1: A SQL query on Spanner showing interleaved usage of graph, relational, and vector models and full-text search

2. Improving price-performance

Spanner’s price-performance lets you dream big, start small (for as little as $65/mo), and scale linearly with no cliffs. In 2022, we increased the storage per node from 2T to 4T, and in 2023 we built on this with a 50% increase in throughput and a 2.5X increase in storage at no additional cost.

This year, with the launch of new multi-model capabilities, we wanted to make it simple and cost effective for you to use these capabilities without charging incrementally for every new feature. The result was Spanner editions, an intuitive, tier-based pricing approach that offers different capabilities at various price points to fit you diverse needs and budgets, all while providing flexibility, cost transparency and additional cost saving opportunities.

3. A new home for your Cassandra workloads

The Cassandra NoSQL database is prized for its speed and scalability. It also has limitations, such as limited support for complex queries and difficulty modeling intricate relationships. Spanner combines the scalability and availability of NoSQL with the strong consistency and relational model of traditional databases, for the best of both worlds.

This year, we launched the Cassandra to Spanner Proxy Adapter, an open-source, plug-and-play tool that makes it easier than ever to move your Cassandra workload to Spanner with near-zero changes to your application logic. Customers like Yahoo! and Reltio are loving the ease of use of the Cassandra proxy adapter, and we’re excited to help customers be more successful with Cassandra on Spanner.

4. Generative AI and the Spanner ecosystem

Over the past year, we’ve witnessed a remarkable shift in how organizations leverage generative AI. But gen AI comes with risk of hallucinations. We believe that transactional and analytical databases can help reduce these, bridging the gap between foundation models and enterprise gen AI apps. Here’s how:

Vector support: With vector support for Spanner, developers can perform similarity searches on vector embeddings stored in the database. Spanner vector search supports both exact KNN and approximate ANN searches, providing flexibilit for different workloads that leverage Google's scalable nearest neighbor (ScaNN) algorithm, providing fast and accurate results, even on large datasets. Spanner now supports vector searches scaling to more than 10 billion vectors. Developers can combine vector searches with regular SQL and graph GQL queries to power use-cases like RAG applications.
BigQuery and Spanner better together: New, groundbreaking integrations between Spanner and BigQuery help businesses connect operational and analytical workloads, to unlock valuable insights and drive better decision-making. Spanner external datasets in BigQuery allows you to query transactional data residing in Spanner directly within BigQuery, without needing to move or duplicate data. Spanner now also supports reverse ETL from BigQuery to export data from BigQuery to Spanner, so you can operationalize the analytical insights that BigQuery enables.

5. Reliability, availability, security, and governance

Spanner customers expect the highest levels of reliability, availability, security, and governance controls for their mission-critical workloads. This year, we launched support for dual-region configurations and geo-partitioning to help you improve your availability SLAs, improve application performance for multi-region workloads, and meet governance requirements.

Dual-region support: Spanner dual-region configurations help meet local residency requirements while providing five 9s of availability and zero recovery-point objective (RPO) guarantees in geographies with only two regions.
Geo-partitioning: You can partition your table data at the row-level across the globe, to serve data closer to your users. With geo-partitioning, Spanner customers across industries like gaming, e-commerce, and financial services can provide their users reduced application latency, optimized costs, and data residency benefits such as storing sensitive user data within geographic jurisdictions.

Lastly, we introduced support for incremental and scheduled backups to make it easier and cheaper to back up your Spanner data.

6. The developer experience

At Google Cloud, we strive to make it ridiculously simple to build and manage applications built on our databases, including Spanner.

Protobuf improvements: Protocol Buffers, or protobuf, is a language-neutral way to encode and decode data structures for efficient transport and storage. You can now manage protobuf values in Spanner and access their fields using the dot operator in SQL, e.g., dimensions.size.width, without having to normalize into tables upfront. This dramatically simplifies writing queries that need to filter, group, or order by specific values within a protobuf.

Troubleshooting and Database Center support: Database Center is an AI-powered unified database management solution to monitor and manage diverse database services. This year, customers started to be able to use Database Center to manage their Spanner databases. We also added support for end-to-end tracing and client tracing to make it easier to troubleshoot performance issues.

We are proud of what we have delivered for customers in 2024, and are excited to see the innovative solutions you are building on Spanner. Needless to say, we are just getting started and we have a lot more exciting capabilities lined up for 2025.

Get started

Want to learn more about what makes Spanner unique and how it’s being used today? Try it yourself for free for 90-days or for as little as $65 USD/month for a production-ready instance that grows with your business without downtime or disruptive re-architecture.

New Cassandra to Spanner adapter simplifies Yahoo's migration journey

Mon, 18 Nov 2024 17:00:00 +0000

Cassandra, a key-value NoSQL database, is prized for its speed and scalability, and used broadly for applications that require rapid data retrieval and storage such as caching, session management, and real-time analytics. Its simple key-value pair structure helps ensure high performance and easy management, especially for large datasets.

But this simplicity also leads to limitations like poor support for complex queries, potential data redundancy, and difficulty in modeling intricate relationships. Spanner, Google Cloud’s always-on, globally consistent, and virtually unlimited-scale database, combines the scalability and availability of NoSQL with the strong consistency and relational model of traditional databases, positioning it for traditional Cassandra workloads. And today, it’s easier than ever to switch from Cassandra to Spanner, with the introduction of the Cassandra to Spanner Proxy Adapter, an open-source tool for plug-and-play migrations of Cassandra workloads to Spanner, without any changes to the application logic.

aside_block: <ListValue: [StructValue([('title', '$300 in free credit to try Google Cloud databases'), ('body', <wagtail.rich_text.RichText object at 0x7fa4025952b0>), ('btn_text', 'Start building for free'), ('href', 'http://console.cloud.google.com/freetrial?redirectPath=/products?#databases'), ('image', None)])]>

Spanner for NoSQL workloads

Spanner provides strong consistency, high availability, virtually unlimited scalability, and a familiar relational data model with support for SQL and ACID transactions for data integrity. As a fully managed service, it helps simplify operations, allowing teams to focus on application development rather than database administration. Furthermore, Spanner's high availability, even at a massive global scale, supports business continuity by minimizing database downtime.

We’re constantly evolving Spanner to meet the needs of modern businesses. Some of the latest Spanner capabilities include enhanced multi-model capabilities such as graph, full-text search, vector search, improved performance for analytical queries with Spanner Data Boost, and unique enterprise features such as geo-partitioning and dual-region configurations. For Cassandra users, these powerful features, along with Spanner’s compelling price-performance, unlock a world of new, exciting possibilities.

The Cassandra to Spanner adapter — battle-tested by Yahoo!

If you’re wondering, “Spanner sounds like a leap forward from Cassandra. How do I get started?” the proxy adapter provides a plug-n-play way to forward your client applications' Cassandra Query Language (CQL) traffic to Spanner. Under the hood, the adapter functions as a Cassandra client for the application but operates internally by interacting with Spanner for all data manipulation tasks. With the Cassandra to Spanner proxy adapter there is no migration for your application code needed — it just works!

Yahoo successfully migrated from Cassandra to Spanner, reaping the benefits of improved performance, scalability, consistency, and operational efficiency. And the proxy adapter made it easy to migrate.

“The Cassandra Adapter has provided a foundation for migrating the Yahoo Contacts workload from Cassandra to Spanner without changing any of our CQL queries. Our migration strategy has more flexibility, and we can focus on other engineering activities while utilizing the scale, redundancy, and support of Spanner without updating the codebase. Spanner is cost-effective for our specific needs, delivering the performance required for a business of our scale. This transition enables us to maintain operational continuity while optimizing cost and performance.” - Patrick JD Newnan, Principal Product Manager, Core Mail and Analytics, Yahoo

Another Google Cloud customer that successfully migrated from Cassandra to Spanner recently is Reltio. Reltio benefited from an effortless migration process to minimize downtime and disruption to their services while reaping the benefits of a fully managed, globally distributed, and strongly consistent database.

These success stories demonstrate that migrating from Cassandra to Spanner can be a transformative step for businesses seeking to modernize their data infrastructure, unlock new capabilities, and accelerate innovation.

How does the new proxy adapter simplify your migration? A typical database migration involves the following steps:

Some of these steps — migrate your application (step 4) and migrate the data (step 6) — are more complex than others. The proxy adapter vastly simplifies migrating a Cassandra-backed application to point to Spanner. Here's a high-level overview of the steps involved when using the new proxy adapter:

1. Assessment: Evaluate your Cassandra schema, data model, and query patterns which ones you can simplify after moving to Spanner.

2. Schema design: Spanner’s table declaration syntax and data types are similar to Cassandra’s; the documentation covers these similarities and differences in depth. With Spanner, you can also take advantage of relational capabilities and features like interleaved tables for optimal performance.

3. Data migration: There are several steps to migrate your data:

Bulk load: Export data from Cassandra and import it into Spanner using tools like the Spanner Dataflow connector or BigQuery reverse ETL.
Replicate incoming data: Replicate incoming updates to your Cassandra cluster to Spanner in real-time using Cassandra’s Change Data Capture (CDC).

Another possibility is to update your application logic to perform dual-writes to Cassandra and Spanner. We don’t recommend this approach if you’re trying to minimize changes to your application code.

4. Set up the proxy adapter and update your Cassandra configuration: Download and run the Cassandra to Spanner Proxy Adapter, which runs as a sidecar next to your application. By default, the proxy adapter runs on port 9042. In case you decide to use a different port, don’t forget to update your application code to point to the proxy adapter.

5. Testing: Thoroughly test your migrated application and data in a non-production environment to ensure everything works as expected.

6. Cutover: Once you're confident in the migration, switch your application traffic to Spanner. Monitor closely for any issues and fine-tune performance as needed.

What’s under the hood of the new proxy adapter?

The new proxy adapter presents itself as a Cassandra client to the application. From the application's perspective, the only noticeable change is the IP address or hostname of the Cassandra endpoint, which now points to the proxy adapter. This streamlines the Spanner migration, without requiring extensive modifications to application code.

We designed the proxy adapter to establish a one-to-one mapping between each Cassandra cluster and a corresponding Spanner database. The proxy instance employs a multi-listener architecture, with each listener bound to a distinct port. This facilitates concurrent handling of multiple client connections, where each listener manages a distinct connection with the specified Spanner database.

The proxy’s translation layer handles the intricacies of the Cassandra protocol. This layer performs message decoding and encoding, manages buffers and caches, and crucially, parses incoming CQL queries and translates them into Spanner-compatible equivalents.

The proxy adapter supports OpenTelemetry to collect and export traces to Cloud Trace.

For more details about different ways of setting up the adapter, limitations, mapping of CQL data types to Spanner, and more, refer to the proxy adapter documentation.

Addressing common concerns and challenges

Let's address a few concerns you may have with your migrations:

Cost: Have a look at Accenture’s benchmark result that demonstrates that Spanner ensures not only consistent latency and throughput but also cost efficiency. Furthermore, Spanner now offers a new tiered pricing model (Spanner editions) that delivers better cost transparency and cost savings opportunities to help you take advantage of all of Spanner’s capabilities.
Latency increases: To minimize an increase in query latencies, we recommend running the proxy adapter on the same host as the client application (as a side-car proxy) or running on the same Docker network when running the proxy adapter in a Docker container. We also recommend keeping the CPU utilization of the proxy adapter host to under 80%.
Schema flexibility: While Cassandra offers schema flexibility, Spanner's stricter relational schema provides advantages in terms of data integrity, query power, and consistency.
Learning curve: Spanner’s data types have some differences with Cassandra’s. Have a look at this comprehensive documentation that can ease the transition.

Get started today

The benefits of strong consistency, simplified operations, enhanced data integrity, and global scalability make Spanner a compelling option for businesses looking to leverage the cloud's full potential for NoSQL workloads. With the new Cassandra to Spanner proxy adapter, we are making it easier to plan and execute on your migration strategy, so you can unlock a new era of data-driven innovation for your organization.

Download the new Cassandra to Spanner proxy adapter, and try it out on a Spanner Free Trial instance at no cost today.

Unlocking the power of Spanner: 10 partners to revolutionize your data

Thu, 31 Oct 2024 16:00:00 +0000

In today's fast-paced digital landscape, businesses need a database solution that can keep up with their evolving demands. Spanner is the always-on, globally consistent, and virtually unlimited-scale database, powering a wide array of modern applications across different industries. Spanner powers a number of Google products with billions of users, including Gmail, and has become the foundation for many other innovative organizations in their data-driven transformations.

Spanner not only offers predictable single-digit millisecond latencies, virtually unlimited scalability, and a five 9s of availability SLA, but also strong consistency, familiar PostgreSQL-compatible syntax, and ACID transactions. This unique combination makes Spanner an excellent choice not just for relational data but also for read-heavy, key-value workloads. Furthermore, with Spanner granular instances you can start small and grow according to your business needs.

Spanner is constantly evolving to meet the needs of modern businesses. Some of the latest Spanner advancements include enhanced multi-model capabilities, including graph, full-text search, vector search, improved performance for analytical queries with Spanner Data Boost, and the addition of unique enterprise features, such as geo-partitioning and dual-region configurations. These new capabilities, along with compelling price-performance improvements, empower you to further optimize your database operations, gain deeper insights from your data, and build even more resilient and scalable applications.

To help our customers take full advantage of Spanner, we are ensuring it is also backed by a robust ecosystem of technology partners and system integrators ready to help you modernize your applications and migrate to the cloud.

aside_block: <ListValue: [StructValue([('title', '$300 in free credit to try Google Cloud databases'), ('body', <wagtail.rich_text.RichText object at 0x7fa4048edca0>), ('btn_text', 'Start building for free'), ('href', 'http://console.cloud.google.com/freetrial?redirectPath=/products?#databases'), ('image', None)])]>

Learn more about our Spanner partners

Our Spanner partners offer a wide range of solutions and services, including data integration, analytics, governance, and migration assessment to ensure a smooth transition and optimal use of Spanner. Google Cloud collaborates closely with these partners to guarantee the best possible experience and support as you embark on your cloud transformation journey.

66degrees offers assistance to customers across their full journey including migrations, application code updates, and infrastructure deployment.

“As a trusted partner in Spanner implementations, 66degrees brings a proven track record of successful database migrations and deep expertise in leveraging Spanner's capabilities. We have achieved exceptional results in Spanner migrations, including Storj, which we migrated from CockroachDB to Spanner. Our accelerators, such as Route66 and custom data flow templates, streamline the migration process and ensure efficient data transfer. We are committed to partnering with Google to empower organizations to harness the full potential of Spanner, enabling them to achieve scalability, performance, and reliability for their mission-critical applications.” - Daniel Zagales, VP of Data Platforms, 66degrees

Learn more about how 66degrees can help with your database migration journey to Spanner here.

Accenture’s 72,000 database professionals worldwide leverage deep industry and cloud expertise to empower clients to drive innovation and digital transformation.

“Google Cloud Spanner offers great performance with availability, scalability, and performance. Our benchmark result indicated Spanner provided predictable performance while maintaining consistent throughput with low cost. Partnering with Google Cloud on Spanner opportunities allows us to combine our deep industry expertise with cutting-edge database technology on Google Cloud, empowering our clients to unlock new levels of scalability, resilience, and real-time insights that drive their digital transformation journeys to the cloud." - Weidong Zhou, Principal Director, Accenture.

Learn more about how Accenture can help you unleash the full potential of human-centric design and Google’s innovative tech.

Amarello has been expanding their capabilities to migrate new customers to Spanner, enabled by a great team of application modernization and database migration experts working together and staying updated on the latest Google Cloud technology.

“Amarello is dedicated to reimagining core business solutions that demand elasticity, high availability, and high transactional processing for Google Cloud customers. Spanner has been proven to be our preferred core technology for delivering mission-critical solutions in any industry.” - Damián Contreras, Managing Partner, Amarello

Learn more about how Amarello can help you unleash the full potential of Spanner.

Ollion worked alongside Google Cloud to develop the critical migration tooling that helps customers move from Cassandra and DynamoDB to Spanner, building deep experience in the nuances and “gotchas” of switching apps over to Spanner.

“We're thrilled to partner with Google to help customers switch to the extraordinary Spanner platform. We’ve helped large-scale customers like ShareChat and Glance successfully move to Spanner from platforms like Amazon DynamoDB, Azure Cosmos DB and Apache Cassandra, with zero downtime and minimal code change. Spanner's new capabilities make it an incredibly flexible tool in any developer's arsenal, and we're excited to explore applying it to an even wider variety of complex use cases.” - Vishal Parpia, Head of Data & AI, Ollion

Learn more about how Ollion can help your business transform with Spanner.

Orijtech Inc. delivers high-quality engineering and computing solutions to hard problems in gritty areas, such as frameworks with terse and rigid flows. Orijtech also designs, builds, and delivers a plethora of integrations so that you can focus on business logic with nothing standing in the way of you using Cloud Spanner's powerful features in any environment.

“Partnering on Spanner is a very fulfilling experience that enables us to directly increase the reach and market share of Spanner, allowing the ecosystem and its customers to build more transformative experiences for example our django-spanner integration opened new avenues for Spanner to tap into the entire Django ecosystem (the world's most popular web framework) and being avid Spanner users ourselves, our solutions help solve problems for our subsidiary customers and big customers of Google Cloud alike. We build all sorts of integrations from frameworks to proxy-level adapters to even custom tools for analysis and automation plus graph and Artificial Intelligence experiences.” - Emmanuel Odeke, Founder and CEO, Orijtech Inc.

Learn more about Orijtech’s software and solutions for Spanner.

Pythian is a Google Cloud Premier Partner with nearly three decades of expertise in relational databases (Oracle, SQL Server, MySQL), including a track record of successful deployments on Spanner, helping Google Cloud customers maximize the benefits of Spanner.

“Enterprises want to take advantage of Spanner’s capabilities, but migrating databases can be a resource-intensive, complex task. Our team has deep expertise in both open source and traditional RDBMS databases, as well as Google Spanner, so we can manage these complexities and deliver low-risk, high-value migrations.” - Vanessa Simmons, SVP of business development, Pythian

Learn more about how Pythian can simplify your database migration to Spanner at every step.

Searce is proud to be one of the earliest engineering and consulting partners for Google Spanner to spearhead Spanner integration. Searce is a modern technology consulting firm that empowers clients to modernize their businesses, leveraging cloud, AI, and analytics.

“Google Spanner’s unique capabilities, such as horizontal scaling without compromising the benefits of relational databases, has allowed Searce to empower its clients, in the financial services, retail and travel, transportation, and logistics industries, to ‘futurify’ their technology solutions. Spanner has been super transformational in solving conflicting requirements of our clients around scalability, high availability and consistency. Spanner’s unique technical capabilities help us achieve our mission to continuously improve and find better ways of doing things for our clients.” - Vrinda Khurjekar, Head of AMER Business, Searce

Learn more about how Searce can help you with your database modernization journey to Spanner.

Tensure's unique strategy for migrating database workloads into Spanner manages complexity and risk while ensuring you get to take advantage of all that Spanner has to offer.

"Spanner is quickly becoming the go to cloud database of choice. It's a powerful tool that minimizes complexity of managing your database estate. Also spanners new AI features push the platform to a whole new level." - Dan Piening, Direct of Partnerships, Tensure

Learn more about how Tensure can help you unleash the full potential of Spanner.

Wipro's technology and IT consulting services enable enterprises to build innovative solutions for addressing the most complex digital transformation needs.

“Wipro is proud to be a partner with Google Cloud for Spanner. With the host of features available from transactional processing to graph capability and vector, Spanner will be a game changer in this space and an easy proposition to customers looking to implement a variety of business cases.” - Shiva Karra, Global Chief Architect, Wipro

Learn more about how Wipro can fuel your business growth with Spanner.

Zencore has helped many companies simplify their database operations and increase scale by migrating their most critical databases to Spanner. These solutions have been tailored to specific customer industry use cases, including financial services, gaming, and advertising. Zencore provides a comprehensive Spanner migration methodology and associated tooling that drastically simplifies and accelerates the Spanner adoption journey.

"Google Spanner has been a game-changing technology since it came to market. We have had the privilege to partner with Google Cloud on this product for multiple years and support many of our customers in benefitting from its unparalleled scaling abilities." - Guilhem Tesseyre, CTO & Co-Founder, Zencore

Learn more about how Zencore can help you run a successful POC, and migrate to Spanner.

Get started with Spanner

Modernization of your database is the critical first step to delivering a truly world class user experience. We hope we have helped you discover a partner that can help you in your transformation strategy and implementation to onboard Spanner.

In the meantime, take advantage of Spanner's free trial instances and put this globally distributed, virtually unlimited scalable database to the test today! Explore Spanner’s features, experiment with different configurations, and see firsthand how Spanner can revolutionize your data management.

Spanner and PostgreSQL at Prefab: Flexible, reliable, and cost-effective at any size

Wed, 23 Oct 2024 16:00:00 +0000

TL;DR: We use Spanner’s PostgreSQL interface at Prefab, and we’ve had a good time. It’s easy to set up, easy to use, and — surprisingly — less expensive than other databases we’ve tried for workloads that need the option to scale. We’re already impressed by the performance to date, and we’re nowhere close to its limits yet.

PostgreSQL is akin to a Swiss Army knife in the world of databases — a fabulous OLTP database, capable of fulfilling the same roles as Redis for real-time access, MongoDB for schema flexibility, and Elastic for data that doesn’t neatly fit into tables or SQL. Its transaction integrity is also impeccable, and whether you need it for analytics queries or JSON storage, PostgreSQL handles it all with grace.

On the opposite end of the database spectrum are NoSQL databases like HBase, Cassandra, and DynamoDB. Unlike PostgreSQL’s versatility, these databases are notoriously challenging to set up, understand, and work with. At the same time, their inflexibility is offset by their ability to scale infinitely. NoSQL databases can support vast volumes of data and high read/write throughput, making them the titans of web-scale databases.

But is there a database that can deliver versatility and incredible scale?

Our encounter with Spanner suggests that perhaps we can have our cake and eat it too.

aside_block: <ListValue: [StructValue([('title', '$300 in free credit to try Google Cloud databases'), ('body', <wagtail.rich_text.RichText object at 0x7fa40489f0a0>), ('btn_text', 'Start building for free'), ('href', 'http://console.cloud.google.com/freetrial?redirectPath=/products?#databases'), ('image', None)])]>

Why Spanner’s PostgreSQL interface?

At Prefab, we help developers ship apps faster with feature flags, dynamic logging, and secrets management. We use Spanner as a datastore for our customers’ configurations, feature flags, and generated client telemetry, which lets us build critical features, such as evaluation charts, that help us operate, scale, and make our product better.

Here are some of the key capabilities that drew us to Spanner:

99.99% uptime by default (multi-availability zone) or up to 99.999% uptime if you run in multi-region
Strong ACID transactions
Horizontal scaling — even for writes
PostgreSQL schemas, queries, and clients

In other words, Spanner promises the resilience and uptime of a Google-scale, massively-replicated database with the simplicity and portability that makes PostgreSQL so appealing.

How Prefab uses Spanner

Prefab’s architecture is split into two parts, so it was very reasonable for us to have a different database for each side, allowing us to choose the best tool for the job. Here are the two sides of our architecture:

Our core Prefab APIs that developers can use to serve their own customers via our software development kits (SDKs).
A web application, which our customers use to manage their configurations and monitor their apps.

Our feature flag services need to be able to scale to meet the demands of the downstream customers of the developers while also delivering extremely low latency. Java and the Java virtual machine (JVM), backed by Spanner, are the right choice for this high throughput, low latency, and high scalability domain. Our application’s user interface (UI) also needs to let us move quickly to ship features to our customers, but it has significantly lower throughput. For this part of our architecture, we use Ruby on Rails, React, and PostgreSQL.

Spanner in action

One feature that uses Spanner today is the backend for the volume tracking of our dynamic logging. Our SDK sees log requests in our customer applications and sends the volume for each log level and logger to Spanner. We then use this data to help users figure out how many log statements will output to their log aggregator if they turn on logging at various levels using the Prefab UI.

To enable this capture, we have a table of the following shape:

code_block: <ListValue: [StructValue([('code', 'CREATE TABLE logger_rollup\r\n(\r\n id varchar(36) NOT NULL,\r\n start_at timestamptz NOT NULL,\r\n end_at timestamptz NOT NULL,\r\n project_id bigint NOT NULL,\r\n project_env_id bigint NOT NULL,\r\n logger_name text NOT NULL,\r\n trace_count bigint NOT NULL,\r\n debug_count bigint NOT NULL,\r\n info_count bigint NOT NULL,\r\n warn_count bigint NOT NULL,\r\n error_count bigint NOT NULL,\r\n fatal_count bigint NOT NULL,\r\n created_at spanner.commit_timestamp,\r\n client_id bigint,\r\n api_key_id bigint,\r\n PRIMARY KEY (project_env_id, logger_name, id)\r\n);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fa404c66940>)])]>

This table scales pretty quickly and unpredictably as clients report back the telemetry for our dynamic logging. Yes, you could also do this in a time series database or some smart things with windowing and removing old data. For the purposes of this post, though, this is an easy way to demonstrate how Spanner helps manage performance for a table that has a bunch of data.

So does it scale?

This example table is not a perfect benchmark here, but it does give you an idea of what easy vertical scaling looks like. I ran the query select project_env_id, count(id) from logger_rollup group by 1 over the table above, which had 37M rows and was around 11GB at the time.

I did this for a varying amount of Processing Units (PUs), giving Spanner about 30 seconds or so to reflect the changes.

The real key here is that scaling this was extremely easy; you can change the number of PUs, from 100 to 500 or even 4000, and Spanner handles the rest — zero downtime and virtually instantaneous at this scale.

100X more storage and no maintenance downtime for ⅓ of the cost

In production, we need to replicate Prefab’s database across multiple zones. Reliability is paramount for feature flags and dynamic configuration systems as they are single points of failure due to their very nature.

We take a belt and suspenders approach here, but multi-availability zone replication and Spanner’s uptime SLA ensures our “belt” is very strong. To achieve this with PostgreSQL on its own, you would need to triple the bill of a single instance. However, Spanner pricing includes replication and automatic failover out of the box. You also get a ton of storage capacity with each node — up to 10TB with Spanner’s recent updates — and you only pay for the bytes you actually use. For us, this makes the comparison look like this:

At a small scale, the best practice of having a database instance per environment can start to be annoyingly expensive. When I first investigated Spanner a few years ago, this was a sticking point because the smallest instance size was one node, or 1,000 PUs. Since then, Spanner’s scale has evolved to scale down to less than a full node, making an easy choice an even easier decision for us. It also gives us the option to scale up whenever we need it, without downtime or expensive rearchitecture in our applications.

Recent improvements to Spanner and the Google Cloud ecosystem

We initially had some bumps in the road using the PostgreSQL interface for Spanner. However, Google Cloud is constantly innovating and improving its products and services, so we’ve been super excited that most of the original things we encountered have been addressed.

Some of our favorite updates include:

Query editor: It’s darn convenient to have a query editor right in the Google Cloud console, which allows us to investigate and tune any queries that exhibit poor performance.

Key Visualizer: If you’re looking at big volume NoSQL databases with HBase, row keys become pretty important to understand. The Key Visualizer allows us to analyze Spanner data access patterns over time and diagnose common issues that cause hotspots. I’m excited to see that there are nice tools included to help understand key distribution.

Looking forward, we plan to explore how we can use Ruby on Rails directly on Spanner. In particular, we believe Spanner could be an intriguing option for new projects, especially those with tables likely to scale significantly — like audit logs or histories — Spanner could be an intriguing option.

Summary

We have a lot of previous experience using HBase and PostgreSQL, but we’re pretty excited about our decision to adopt Spanner as the horizontally scalable operational database of choice for Prefab. We’ve found it to be easy to use for our needs, delivering all the same scaling properties as HBase, without the headaches of doing it ourselves. Fewer points of potential failure and fewer things to manage save us time and money.

If you have some big tables that scare you, but you haven’t looked beyond PostgreSQL, you might consider expanding your horizons. Spanner’s PostgreSQL interface gives you the portability and ease-of-use of PostgreSQL with the proven reliability and scale of Spanner and Google Cloud.

Plus atomic clocks! How cool is that?

Get Started

You can get started with Spanner today for free for 90 days or as little as $65/month after that. We’d also be happy to connect with you and love it if you’d explore the product we’ve built on top of Spanner by learning more about our Feature Flags, Dynamic Logging and Secret Management.