Gaming

The AI-powered shift to “living games:” Meet the customers and partners leveling up the ways we play

Mon, 18 Aug 2025 07:00:00 +0000

The games industry is on a powerful ride, surging forward with innovation and a sharp focus on the player experience. For years, the industry’s evolution was defined by familiar IPs getting better graphics and gameplay. At Google Cloud, we believe we’re on the cusp of something far more radical — a shift on the scale of the transition from cartridges to CD-ROMs, or 2D to 3D graphics. This new era is defined by the rise of "living games," a new form of dynamic, ever-evolving experiences powered by AI that captivate players for years.

With the global market for games surpassing $180 billion in 2024, this fundamental shift in how games are developed, played, and experienced creates an entirely new opportunity for the industry. A big part of what’s driving this shift is the transformative power of cloud computing and AI, and many cutting-edge developers and startups are already taking advantage of these advances.

Cloud platforms are now the core of a $12.9 billion ecosystem within games, with AI adoption emerging as a central growth driver. In fact, Google Cloud’s new survey, conducted by The Harris Poll, reveals just how deeply integrated AI has become: 97% of game developers agree that AI is reshaping the games industry. They already see this evolving technology as fundamentally changing how they create games and what players expect. This new technology is turning the weeks-long live-operations cycle into an instantaneous, AI-driven feedback loop that creates a game world that feels truly alive.

The vision of truly living games is no longer a distant dream; it's a reality unfolding today. Google Cloud is helping drive this forward through the powerful combination of Google’s deep live service expertise and cutting-edge cloud and generative AI technologies.

It’s this kind of innovation that’s driving new and expanded collaborations with incredible games customers and partners, including Atlas, Embody, Ludeo, Nacon, and Nitrado. These pioneers are pushing the boundaries of what's possible in games, from creating more immersive player experiences to accelerating game development and scaling their operations.

Atlas: AI for creating vast 3D game worlds

Atlas is an agentic 3D-content creation platform designed for professional game studios, enabling them to generate game-ready assets, environments, tools, and workflows. It focuses on production-scale workflows rather than one-off asset generation, acting as a creative assistant through its multi-agent AI system. Developers can co-create with intelligent AI agents using natural language prompts, ensuring the output is tailored to their specific technical and aesthetic goals. Atlas integrates with industry-standard pipelines like Unreal Engine, Unity, and Houdini, making it ideal for AA+ teams building complex games at scale.

“We believe AI-native games will define the next chapter in interactive entertainment,” said Ben James, chief executive officer, Atlas. “These experiences will be dynamic, personalized, and constantly evolving — and they’ll require a new creative infrastructure. Partnering with Google Cloud gives us the compute foundation and orchestration support to bring that vision to life.”

Atlas is collaborating with Google Cloud to supercharge its multi-agent AI infrastructure and accelerate the development of AI-native games. The platform is built entirely on Google Cloud's infrastructure and uses our model orchestration tools, including Vertex AI. This provides Atlas with the robust compute foundation and orchestration support necessary to bring its vision to life, enabling a new era of dynamic, evolving interactive entertainment.

"Atlas's ability to seamlessly integrate with our highly customized workflows has been a game changer,” said Joseph Burnette, technical director of the Innovation Technology Division at SQUARE ENIX. “By deeply understanding the nuances of our pipeline, they've become an invaluable partner, enabling us to deliver high-quality, performance-optimized solutions with impressive agility."

Footage shown is from Parallel’s upcoming game, Colony, using Atlas technology.

Embody: Personalized spatial audio for unrivaled immersion

Embody is an AI technology company revolutionizing sound for games, music, and XR experiences through its Immerse AI Engine. This engine uses machine learning, 3D neural networks, and computer vision to deliver personalized spatial audio on any headphones. Simply by analyzing a short smartphone video of a player's head and ears, Embody creates a unique sound profile for a hyper-realistic and deeply immersive experience. AI-native head tracking and adaptive EQ further enhance the audio, ensuring a consistent and top-tier sound experience across all devices.

To power these complex, real-time calculations and scale to millions of gamers, Embody relies on Google Cloud's infrastructure. Access to massive, cost-effective GPU compute power is crucial for generating personalized audio profiles in seconds and for their continuous innovation in spatial sound. Our collaboration also allows Embody's R&D team to rapidly prototype new ideas and ensure their technology is economically viable and globally scalable. Embody's Immerse is already enhancing AAA titles like Call of Duty: Black Ops 6, War Zone, Final Fantasy XIV, Cyberpunk 2077, and The Witcher 3: Wild Hunt — and just announced it’s launching with Sea of Thieves.

“Sound is the emotional core of every game, and we believe it should be personal,” said Kapil Jain, chief executive officer, Embody. “With Google Cloud, we’re scaling our AI-powered sound personalization engine to meet the demand of millions of gamers around the world.”

Ludeo: Redefining game discovery with playable moments

Ludeo is the world’s first playable media platform, enabling users to instantly play game highlight clips. Unlike gameplay content consumed today — which turns the experience of playing games into passive videos — Ludeo works directly with studios and publishers to create “playable moments,” called Ludeos, that users can instantly jump into and experience themselves. They can do so whether they own the game or not, without any downloads or lengthy installs.

These Ludeo moments can be shared with a link anywhere, from social media platforms to messaging apps. Passive viewers become active participants in seconds. This helps game studios attract new players, re-engage existing ones by showcasing new content, and even lets players "try before they buy" in-game items, boosting interest and conversion.

To power this vision, Ludeo will bolster its core infrastructure with Google Cloud, using Google Kubernetes Engine (GKE) and GPUs to create a highly optimized, low-latency infrastructure that’s required for their platform. Ludeo will also aim to build the "playable YouTube," fundamentally changing how players discover and socially engage with games, from popular AAA titles to AAs and indies.

“Google Cloud’s infrastructure strengthens the capabilities and scale of the Ludeo platform,” said Uri Levanon, vice president of business development and partnerships at Ludeo. “This powerful combination will give players the magic of instantly playing game highlights instead of just watching them, in addition to unlocking new growth opportunities for game studios.”

NACON: Accelerating game production with AI transformation

NACON stands as a prominent AA video games company and a leader in high-end games hardware, known for popular titles like RoboCop: Rogue City, Ravenswatch and Test Drive Unlimited Solar Crown. With 15 game studios under its belt, NACON is making a bold strategic pivot, embedding AI at the core of its operations, from game development to marketing.

NACON’s goal is to increase annual game launches, a move heavily reliant on streamlining processes and boosting creativity with AI. This vision encompasses everything from crafting captivating trailers and in-game cinematics to optimizing game maps for racing titles, all designed to enhance player experience and developer efficiency.

Google Cloud is NACON’s partner in this AI-driven transformation, helping NACON innovate faster and deliver unforgettable games experiences. NACON has selected Google Cloud as its preferred partner for game servers, ensuring scalable and reliable infrastructure for their diverse portfolio. They are also using Google’s Veo 3 model as a complementary tool to help produce cinematic trailers and Google’s Gemini model to support localization efforts, enabling NACON to reach new global markets more efficiently. Additionally, NACON will use Looker for deep insights into in-game analytics and player behavior, and Google Threat Intelligence to help their ability to proactively secure their operations against industry threats.

“Partnering with Google Cloud marks a pivotal moment in NACON’s journey to transform game development with AI at its core,” said Alain FALC, president and chief executive officer, NACON. “Google Cloud’s cutting-edge tools empower our teams to innovate faster, streamline production, and deliver richer, more immersive experiences to gamers worldwide."

Nitrado: Hybrid cloud scaling for flawless multiplayer gaming

Nitrado, a global leader in game server hosting, is making multiplayer game creation even easier for studios with a new capability for their orchestration solution, GameFabric. This platform acts as a unified orchestration layer, allowing game developers to seamlessly combine Nitrado’s high-performance bare metal infrastructure with the elasticity of the cloud, all managed through GameFabric.

This means studios can automatically use Google Cloud to support traffic spikes, like during a big game launch or a busy weekend. Furthermore, studios can bring games closer to their players by instantly deploying servers in new regions, using Google’s planet-scale network to ensure low-latency performance for a global audience. GameFabric can scale up or down automatically, allowing developers to focus on the player experience, not the infrastructure, and studios to keep their games running smoothly and cost-efficiently, no matter how many players jump online or where they are.

With Google Cloud as GameFabric's preferred cloud provider, studios benefit from Google Cloud's low-latency global network for a flawless player experience and elastic infrastructure for unlimited scalability. This partnership is built on operational tools like GKE and Agones, which are trusted for managing game servers efficiently and reliably. Plus, Google Cloud's built-in security and reliability protect game and player data around the clock.

“GameFabric brings together bare metal and cloud in a unified orchestration layer, so studios can scale up, stay fast, and keep costs predictable,” said Raphael Stange, chief executive officer, Nitrado. “This partnership strengthens the hybrid model we’ve built to serve multiplayer studio needs.”

Bring your living games to life

The future of living games isn't just a concept — it's being built right now, powered by the dynamic combination of cloud and AI. We’re committed to being the foundational partner for game developers and studios of all sizes, offering the scalable infrastructure, powerful AI tools, and deep expertise needed to bring truly dynamic, immersive, and successful games to players worldwide. We're excited to see what new experiences emerge as our customers and partners continue to push the boundaries of creativity and technology.

Build your AI-powered POC with Google Cloud for games. Visit cloud.google.com/solutions/games.

Enhancing multiplayer games with gen AI and cloud gaming: a technical deep dive

Mon, 17 Mar 2025 16:00:00 +0000

Generative AI is poised to revolutionize the gaming landscape. It will offer — and in many cases, already is offering — personalized NPCs, dynamic narratives, and tailored environments, thereby reshaping player experiences. It’s a concept increasingly known as living games.

The recent unveiling of Genie 2: a large scale foundation world model created by Google DeepMind exemplifies this potential. Living cloud gaming, underpinned by Kubernetes, Agones, and Open Match, and integrated with generative AI, can extend these advancements to a vast audience without sacrificing performance or user experience.

In this blog post, we'll take you on a journey of integrating cutting-edge generative AI models — like Google’s Gemini or open models such as Gemma — into scalable multiplayer game servers powered by Google Kubernetes Engine (GKE) and Agones. This blog will showcase how a sample game can be enhanced with generative AI, enabling developers to enrich their games with AI-driven features such as interactive user guidance, real-time chat capabilities, and personalized summaries of user progress, all while maintaining scalability and reliability.

Cloud gaming with Agones: the foundation for scalable multiplayer experiences

Agones is an open-source, Kubernetes-based game-server-hosting solution that enables developers to efficiently manage large numbers of dedicated game servers. While Agones works on any Kubernetes deployment, running it on GKE provides Google Cloud customers with official support.

Agones also integrates with Open Match, an open-source matchmaking framework that simplifies the creation of multiplayer matches based on player criteria such as skill level or location.

This combination of Agones and Open Match provides a powerful foundation for scalable cloud games.

aside_block: <ListValue: [StructValue([('title', 'Try Google Cloud for free'), ('body', <wagtail.rich_text.RichText object at 0x7fe389f8cc70>), ('btn_text', 'Get started for free'), ('href', 'https://console.cloud.google.com/freetrial?redirectPath=/welcome'), ('image', None)])]>

Generative AI's Potential in Cloud Gaming

By utilizing generative AI in real time, living games can go beyond traditional live services, creating a dynamic and engaging player experience without constant developer input. Living games can foster a deep connection between players and the game world, ensuring that each playthrough is unique, through features like dynamic storyline adaptation, responsive NPC interactions, and personalized in-game content.

Building a language learning game with gen AI: A technical walkthrough

To illustrate a simple example of Agones and generative AI integration, we created a language learning game called Lexitrail. Lexitrail helps players memorize Chinese characters in an interactive way while also competing against other players in multiplayer mode.

Lexitrail utilizes a React frontend, a Python Flask backend, and a MySQL database. All of them are deployed in-cluster on Google Kubernetes Engine (GKE). A key component is the game session management layer, deployed as Agones game servers for seamless game session handling.

Within Lexitrail, we leverage the power of Gemini and Imagen models. These models generate subtle, ambiguous hint images that aid players in memorizing and recalling words. To further personalize the experience, we provide additional user-specific context alongside the LLM prompts.

High-Level Architecture of Lexitrail

The diagram below illustrates the fundamental architecture and components that underpin the Lexitrail system.

This architecture is designed to provide key features that enhance the gaming experience:

Seamless game sessions: Utilizing Agones for game session management ensures uninterrupted gameplay with quick state transitions in both single-player and multiplayer modes.
Integration of generative AI: The generative AI integration layer's adaptability enables developers to create endpoints for both closed models (e.g. Gemini, Imagen) and open models (e.g., Llama, Stable Diffusion, Gemma). These endpoints can then be integrated with other components in the game application, allowing for the creation of detailed and suggestive hint images specifically tailored to the individual game's requirements
Skill-based matchmaking: In multiplayer environments, Open Match creates balanced and engaging matches by pairing players based on their skill levels.
Scalability and reliability: The foundation of this architecture is GKE, providing the Kubernetes infrastructure necessary for managing and autoscaling game servers. Running Agones and Open Match on GKE guarantees that the language learning game can effortlessly support a global player base, from small study groups to massive tournaments.

Here's a breakdown of the Lexitrail game application's architecture, designed for a smooth and scalable multiplayer experience:

Lexitrail frontend: This is the initial interface for players. It manages authentication through Google login. The user interface component and the business logic governing its interactions are both managed within the frontend. Various game modes are provided. Players can practice solo or compete against others in multiplayer matches. Additionally, players can select wordsets based on different HSK (Hanyu Shuiping Kaoshi) levels for each game mode. When a player opts for multiplayer mode, the game's frontend triggers the matchmaking process. This process involves sending matchmaking requests to the Open Match component.
Open Match: Lexitrail employs Open Match which is deployed as a GKE service for its matchmaking system. The current setup is straightforward: players who select the same wordset are paired together. However, Open Match's flexibility allows developers to implement more intricate matchmaking algorithms tailored to their specific game requirements. Once a match is determined, Open Match collaborates with the game server orchestration system to designate a dedicated game server for that match. Then the matchmaker passes back to the game clients (through the frontend) the address and port each client will need to connect directly to, since the game state is an in-memory simulation, and then, they play Lexitrail!
Game server: In Lexitrail, the game server is responsible for managing game sessions for pairs of players. It oversees gameplay logic, tracks intermediate word recall results, and maintains the final scoreboard. Additionally, the game server enhances performance by acting as a buffer layer. It batches time-consuming tasks, such as database updates and asynchronously flushes them to the backend, ensuring that these operations don't negatively impact the responsiveness of the frontend.
Agones: Lexitrail employs Agones for game server orchestration, leveraging its allocator service to facilitate matchmaker’s requests to get the dedicated game server for each match. Agones manages these game servers to ensure availability so the player experiences aren’t interrupted while players are playing on them. Additionally, Lexitrail utilizes Agones' fleet autoscaler to dynamically adjust the number of game servers based on demand.
Game backend: Lexitrail's game backend, deployed as a GKE service, manages the API interface and business logic for database interactions. It handles requests from both the frontend and game servers, retrieving and updating player data like wordsets, words, and recall history. Additionally, the backend integrates with the generative AI endpoints to generate hint texts and images that aid players in memorizing and recalling words.
Database: Lexitrail's backend database is powered by MySQL, deployed within the GKE environment. This database houses all essential game data which includes user profiles, word sets, individual word entries, and player recall history. The database schema can be found here.
Generative AI integration: In the game's user interface, players are presented with words to memorize during gameplay, accompanied by visual hints to aid in recall. These image hints are generated by integrating with generative AI endpoints, which are deployed within the GKE environment. The game backend queries the image generation model endpoint to create the hint images for each word. The flexibility of kubernetes based architecture allows developers to easily select and utilize different underlying LLMs for image generation.

Take the Next Step

The potential for gen AI in gaming is enormous, and the tools are now more accessible than ever. We invite you to explore the repo as a starting point.

Explore gen AI integration: Start integrating gen AI models into your own applications using Agones and GKE.
Leverage Vertex AI and GKE models: Experiment and compare integration between Vertex AI or open models on GKE to see how AI can transform your gaming experience.
Join the community: Engage with the developer community for Agones, Open Match, and broader GKE AI projects — share your insights, learn from others, and help shape the future of AI-powered gaming.

Stay at the forefront of gaming technology by embracing scalable AI-driven features that deliver new levels of personalization and engagement to players, bringing living games to reality.

Kubernetes and GKE with Agones and Open Match provide a scalable, production-proven enterprise-grade solution for cloud games that leverage state-of-the-art components. The journey of scalable, AI-powered gaming is just beginning — and we can’t wait to see what you build!

Co-op mode: New partners driving the future of gaming with AI

Mon, 17 Mar 2025 13:00:00 +0000

Leaders in the games industry are using Google Cloud’s AI to drive unprecedented advancements in game development, including smarter, faster, and more immersive gaming experiences. And just like any successful game studio is the work of a broad and diverse team, our partner ecosystem is ready to help level up any project, including with game-industry-specific AI tools. Together, we can minimize engineering effort, maximize production efficiency, and help gaming companies grow their businesses.

At this year’s Games Developers Conference, we're announcing new partnerships with nunu.ai, ElevenLabs and Common Sense Machines. These innovative startups are using Google Cloud’s AI platform, including Vertex AI, Google Kubernetes Engine (GKE), and Agentspace, to enhance creativity, automate workflows, and elevate player immersion.

nunu.ai: Improving game testing with AI agents

Game testing and quality assurance, traditionally time-consuming and costly initiatives, are being transformed by AI. Google Cloud is partnering with nunu.ai to bring innovative, AI-powered game-testing solutions to developers worldwide. As games grow increasingly complex, nunu.ai's multimodal AI agents—powered by Gemini models and running on Google Cloud infrastructure—are transforming how game testing is conducted.

Nunu.ai's agents navigate game environments just like human players, identifying bugs traditional testing might miss. These AI agents observe the game through rendered frames and interact using keyboard and mouse inputs, providing authentic testing experiences.

Our partnership with nunu.ai delivers:

Human-like testing: AI agents play games just like humans do, catching bugs that other testing solutions miss.
Cost efficiency: Game studios working with nunu.ai have reduced manual QA costs by up to 50% by automating repetitive testing tasks.
24/7 availability at scale: Unlike human testers, nunu.ai's AI agents work around the clock with consistent performance and scale to meet growing demands.
Accelerated time-to-market: By automating and streamlining the testing process, nunu.ai helps developers identify and fix issues faster, significantly reducing development cycles and enabling quicker game releases without sacrificing quality.
Easy integration: Developers can implement nunu.ai's lightweight SDK quickly or use black-box testing for compatible games with minimal maintenance.
Multiplayer capabilities: Easily spawn multiple AI agents to simulate real player interactions, solving the persistent challenge of multiplayer testing.
Cross-platform support: Currently supporting PC and mobile games (iOS and Android), with console support coming soon.

These advanced AI agents automatically test your game throughout your development pipeline, providing continuous quality assurance at every stage. By automating repetitive testing tasks, nunu.ai ensures developers can focus on what matters most: creating exceptional gaming experiences for players.

aside_block: <ListValue: [StructValue([('title', 'Try Google Cloud for free'), ('body', <wagtail.rich_text.RichText object at 0x7fe3885aa2b0>), ('btn_text', 'Get started for free'), ('href', 'https://console.cloud.google.com/freetrial?redirectPath=/welcome'), ('image', None)])]>

ElevenLabs: AI-powered voice synthesis for immersive experiences

Beyond gameplay mechanics and quality assurance, AI is enhancing another fundamental aspect of player immersion: voice and sound. ElevenLabs is at the forefront of this transformation.

ElevenLabs provides AI-powered voice technology for immersive, dynamic, and scalable experiences. By combining Google Cloud’s infrastructure with ElevenLabs’ voice AI, studios can create new kinds of real-time interactions, and localize and produce the games simply and quickly.

ElevenLabs AI voice solutions enable:

Real-time conversational character voices: AI-powered interactions for immersive NPCs.
Custom voice and sound design: Instant, unique voice creation and tailored sound effects.
Faster dialogue production: AI text-to-speech for dialogue and voiceovers.
Easier global localization: Automated, high-quality voice translation.
Synthetic voiceovers for marketing and trailers: Generate compelling AI-driven voiceovers for game promotions and storytelling.
AI-Generated narration and audiobooks: Streamline long-form voice content production for in-game lore, cutscenes, and interactive storytelling.
Multimodal AI integration: Combine voice AI with LLMs and other generative AI tools for intelligent, context-aware conversations.
AI-Driven contact centers and support: Enhance customer interactions with natural, responsive AI voice agents.

With Google Cloud’s scalable AI infrastructure and ElevenLabs’ industry-leading voice synthesis, studios can enhance player immersion, reduce production costs, and bring their worlds to life.

Common Sense Machines (CSM.ai): Best-in-class AI models for 3D content creation

Common Sense Machines (CSM) is redefining AI-native game development by building best-in-class AI models for 3D content generation. They were recently ranked as the No. 1 3D/AI model globally, on the world's largest AI model benchmarking site, Hugging Face, in the 3D Arena category.

CSM’s models set a new standard for high-quality AI-generated assets, enabling artists to enhance their creativity, build immersive environments, and reduce time spent on tedious tasks like retopology and UV remapping.

CSM is already making a real impact through strategic partnerships. They’re working with Cosmic Lounge, a studio building games with their proprietary AI-powered puzzle games engine, to generate 3D assets for marketing and LiveOps content—the dynamic, continuously updated elements that enhance games after their initial launch (ie: new character skins, seasonal items, and environmental assets). By focusing on use cases that drive user acquisition and revenue, studios can take advantage of AI to drive growth, rather than just improving efficiencies and resolving production bottlenecks.

CSM is passionate about assisting artists, not replacing them. With AI-hybrid workflows evolving quickly, CSM expects human modelers will always play a crucial role in the creative process.

The AI-driven future of gaming

"We see a powerful shift in gaming, where AI empowers creators and developers to achieve unprecedented levels of immersion and efficiency," said Macy Mills, partner at A16Z Games, a unit of venture firm Andreessen Horowitz. "Nunu.ai, ElevenLabs, and CSM.ai are pioneering tools that we believe will reshape game development, allowing studios to focus on crafting truly compelling worlds and experiences."

AI is transforming how games are created, tested, and experienced. By partnering with innovative startups like nunu.ai, ElevenLabs and CSM, Google Cloud is helping game developers push the boundaries of what’s possible—delivering smarter, faster, and more immersive gaming experiences.

^{Opening image created with Imagen 3 running on Vertex AI, using the prompt: An overhead shot in a realistic video-game style, wide angle, of a group of characters seen from behind, from different game genres, like fantasy, shooter/military, 8-bit, etc., standing on a mountain top of cliffside, looking out to an exciting world and the horizon. This should all represent the concept "the future of gaming.” ... Make it a little more 8-bit.}

Powering Web3-enabled gaming with GameShift by Solana Labs

Mon, 23 Sep 2024 16:00:00 +0000

As we join the vibrant community at Solana Breakpoint this week, we're excited to share how Google Cloud is helping shape the future of gaming through its partnership with Solana Labs. The Gamer Village at Breakpoint is a testament to the innovative spirit driving Web3 gaming forward, and we're proud to be at the forefront of this revolution.

At the nexus of our partnership is GameShift, which provides the full slate of Web3 primitives and actions — like wallets, tokenized assets, and on-chain markets — that games require; as of today, Gameshift’s offerings are now available through the Google Cloud Marketplace.

We want to share with the Google Cloud community the potential for Web3 in games, how GameShift can help developers realize this potential, the safety of using GameShift for Web3, and how GameShift and Google Cloud now work together seamlessly. We welcome anyone interested to reach out to their Google Cloud account manager or to GameShift.

Google Cloud has long recognized the potential of Web3 technologies. Its support for this ecosystem includes its Blockchain Node Engine product, Web3 BigQuery datasets, and the Web3 for Startups program.

As a leading blockchain, Solana brings top transaction throughput, 99.95% uptime, and billions in transactions daily. This integration connects games to millions of users in a unified ecosystem of wallets, assets, and activity. Solana is consistently a top-3 performer in terms of NFT, decentralized exchange trading, and stablecoin volume, as well as transactions-per-second in production, and active wallets.

These results make Solana a great option for games seeking to leverage Web3 to create new player experiences and new revenue channels.

The Web3 evolution of gaming

Video games are constantly evolving. Just like other forms of media, games need to constantly reinvent themselves to offer novel experiences to their users. This could be in the form of new creative assets, gen AI-generated digital assets, story lines, watermarking, provenance or IP licensing. For the same reason, games developers have been rapid adopters of new technology to not only enable new gameplay mechanics, but also to find their way into new parts of our everyday lives, including crossover points into seemingly mundane activities like loyalty programs.

Web3 provides a new avenue for games to reach players with an evolved value proposition. With Web3, games can easily convert their players from customers into partners and co-creators. This is important because players are increasingly sophisticated in their understanding of how they provide value to a game, and how they get monetized.

In short, players are expecting more from their experiences.

Blockchains provide the building blocks that power complete game asset economies, including user-generated content, global game item trading, gamer identity, anti-cheat, and cross-game portability. They also facilitate nuanced monetization schemes aligned with how players interact with the game. Games can implement cost-efficient microtransactions, generate secondary market revenue on first-party and user-generated assets, and monetize their in-game currencies. To players, these revenue strategies feel cooperative rather than extractive.

By simplifying interactions between on-chain programs from different providers — also known as “composability” — blockchains enable novel experiences and developer efficiencies. In the payments domain, composability dramatically reduces the cost and complexity of offering real-money gaming services, earning interest from stored value balances, and providing player choice in payment methods and pay-outs. Composability also opens the door to exciting features like guild management, open modding, and global player reputation systems, all of which are on the forefront of Web3 game development.

GameShift simplifies Web3 for games

“Game studios have recently been considering Web3 for new approaches to growth and development, but are often discouraged by the seeming complexity of the technology and the Web3 community” said Jack Buser, director for Games at Google Cloud. “Game studios are already overburdened, and need solutions like GameShift that provide simplified technical and cultural interfaces to Web3.”

GameShift’s API-first philosophy meets game developers where they’re at, without requiring any blockchain experience to get started. “Our goal in building GameShift,” said Davis Hart, general manager of GameShift, “is to provide a development and player experience as similar to existing services as possible while leveraging the unique capabilities of Web3.”

Earth From Another Sun, a cinematic-quality RPG, is using GameShift to power their entire blockchain integration, said Freeman Fan, CEO of the game’s developer. “By leveraging GameShift's proven toolset and innovative features, we can implement best-in-class Web3 gaming experiences while reducing development time and costs by 90%,” Fan said.

Key Advantages

GameShift’s features cover the full spectrum of Web3 gaming functionality in a developer-friendly REST API. The team at Solana Labs offers GameShift customers deep connectivity to the Solana ecosystem, support designing Web3 gaming strategies, and growth support. This means that with GameShift, studios don’t need multiple providers or specialized blockchain developers, and they can rest assured that GameShift will grow with them over time.

Today, GameShift offers five core features:

Seamless wallet integration: Create embedded wallets in your game or support players with existing Solana wallets. GameShift wallets abstract key storage and provide a clean and friendly transaction signing flow, providing a smooth, secure experience for players without requiring them to leave your game environment. The fact that our wallets are non-custodial means that games don’t need to manage pools of user assets.
Game asset management: Easily create and manage in-game assets on the Solana blockchain, with custom asset attributes, rarity, and in-game functionality. It also handles asset transfers, making it simple to implement features like trading or gifting within your game economy.
Pay-in and pay-out systems: Implement secure, low-cost transactions for in-game purchases, subscriptions, or rewards. This API supports both fiat and cryptocurrency payments, handling currency conversion and transaction processing behind the scenes. It also facilitates easy withdrawals, allowing players to transfer their game-related assets back to their personal wallets or bank accounts.
Marketplace functionality: Quickly set up and manage in-game marketplaces where players can buy, sell, or trade their assets. This API handles listings, bids, transactions, and transfers, ensuring a secure and efficient marketplace. It also provides tools for implementing royalties, allowing you to earn a percentage of secondary sales, even on user-generated content.
Rewards programs: Design and implement sophisticated reward systems using blockchain technology. This API allows you to create token-based loyalty programs or achievement-based rewards that provide players with tangible value. It handles program enrollment, tracking, and rewards distribution with custom logic, enabling you to incentivize player engagement and retention in innovative ways.

Reducing the risks of Web3 in gaming

“Developers are rightly concerned about player safety given the history of hacks and scams in Web3,” noted Hart. “But, with the combination of the right security practices and game design, these concerns can be brought to the level that developers routinely address in their off-chain systems.”

At the core of GameShift’s strategy is its invisible non-custodial wallet system. This feature provides individual user wallets, each backed by its own cryptographic keys that segregate assets between users, reducing the risk that an attacker could access the assets of all users simultaneously. Key management is abstracted away from o the user, eliminating the risks associated with user-managed keys and seed phrases.

Players using GameShift wallets are further protected from scams because the game is in control of when and what types of transactions are presented to the user. Transaction Transaction approval flows simulate the effect of any on-chain action and alert the user of any balance changes. GameShift uses only on-chain services from vetted and trusted providers. These strategies prevent a third party from defrauding users of their assets through phishing attacks or hacks.

“Our approach allows us to provide an approachable player experience that avoids many of the risks game studios are worried about,” Hart said.

Leveraging Google Cloud with Web3

With GameShift now deployed on the Google Cloud Marketplace, developers can quickly enable Web3 within their existing Cloud game backend services.

This reference architecture hints at what’s possible with GameShift on Google Cloud, including:

Supplementing existing player data in BigQuery with GameShift transactions, creating a richer view of player activity.
Deploying BigQuery and Vertex AI against the Solana public BigQuery dataset to understand how players interact on-chain.
Using Cloud Storage to easily host user-generated content globally.
Applying generative AI for gaming assets as well as user-generated content to ensure compliance with content policies, content moderation, toxicity, anti-cheat and player privacy, safety, and security.
Using Pub/Sub to rapidly ingest live blockchain event data and trigger in-game activity.

Looking Ahead

The future of Web3 gaming is bright, and we're excited about upcoming developments:

Solana Mobile: Announced at Breakpoint 2024, the next generation web3 mobile device will offer a new platform for mobile Web3 gaming.

Solana Permissioned Environments: Combined with GameShift on Google Cloud, Solana Permissioned Environments will allow studios to benefit from the Solana ecosystem while maintaining control over their chain access.

Tournament Management: GameShift has just announced a real-money gaming tournament management product, simplifying the integration of fiat and Web3 payments for game tournaments.

Google Cloud and GameShift are committed to empowering game developers with the tools and infrastructure they need to create the next generation of engaging, secure, and innovative gaming experiences. Through its collaboration with Solana Labs and the power of GameShift, Google Cloud is helping turn the promise of Web3 gaming into reality.

Next Steps

We encourage anyone interested in GameShift and Google Cloud to contact their Google Cloud account rep, contact GameShift, and review the GameShift documentation.

Square Enix’s journey of building an AI-driven Customer Data Platform

Mon, 24 Jun 2024 16:00:00 +0000

Editor’s note: In this post, we’ll learn from Japanese video game and entertainment company Square Enix about its journey building a customer data platform with Google Cloud to enhance its first-party data strategy by collecting and connecting its marketing, player, and game data.

In today’s data-driven world, understanding who your customers are is more critical than ever. For game developers and publishers, in particular, gaining a complete picture of who your players can quite literally be a game changer — transforming games that never quite find the right audience into standout titles. Data can unlock a trove of valuable insights that help gaming companies gain a deeper understanding of their product discovery journey and player base, unlocking insights into playing style, playtime metrics and more to create engaging, personalized communications.

Unfortunately, traditional data analysis is often siloed with player information scattered across multiple systems and platforms, making it cumbersome and time-consuming to achieve a consolidated view of a player’s interactions and habits. That’s where Customer Data Platforms (CDPs) come in, acting as a central hub that can bring all your player data together and deliver a unified player profile that empowers game developers to truly get to know their players and create games that resonate. Developing a CDP for games is especially complex. This is due to the sheer number of players a game can attract and the massive volume of data each player generates.

When set up and configured correctly, CDPs are an integral part in driving player-centric experiences and game design across your business. Typically, CDPs comprise four key areas:

Player data aggregation: Collecting, integrating, and storing all forms of player data, including platform preferences, in-game achievements, playstyle behavior, and social interactions.
Cross-platform player identification: Unifying player data across different gaming platforms, devices (PC, console, mobile), in-game IDs, and email addresses into a comprehensive player profile.
Game analytics and AI insights: Utilizing advanced AI and machine learning models to deliver out-of-the-box insights that power predictive player behavior, forecasting churn, in-game and outer-game engagement, and optimal communication strategies.
Player activation: Utilizing preference based segmentation to engage players with targeted messaging across different channels including in-game notifications, emails and social media. Offering personalized catalog title recommendations to players can also help foster lifelong relationships.

While organizations are able to purchase many of these technologies off the shelf, there are many benefits to building your own CDP and creating a more tailored platform to meet your specific needs and use cases. In this post, we’ll share our own journey of how we built our own CDP with Google Cloud at Square Enix.

Building an AI-driven CDP with Google Cloud

As an early adopter of data collection, analysis and machine learning, the Square Enix Analytics and Insights department has been at the center of bringing business value to our organization through data-driven game innovations, such as using telemetry data to discover new markets for games and inform patching decisions about areas where players get stuck.

Our European division — Square Enix West — faced a significant shift in data strategy in 2017, driven by the General Data Protection Regulation (GDPR) due to come into force the following year. Recognizing the changing landscape and policy around marketing data, our team, along with technology leadership, saw the need to focus on collecting and connecting our first-party data across multiple sources — thus began the journey to aggregate marketing, player, and game data under a single roof.

Given that gaming is often a lifelong passion, the ability to analyze fan’s engagement level with Square Enix’s services over periods of five to ten years and across multiple data sources is crucial. Our vision was to craft a solution that would allow the team to conduct end-to-end analysis of gamers’ behavior, including touchpoints, purchases, gameplay and retention. We knew that the right CDP tool would be essential to service players at every step along their discovery and buying journey.

Using Google Cloud services, particularly serverless, we built a proprietary system called Single Gamer View (SGV), which delivers both CDP and customer relationship management features. The new solution unifies all of our databases, provides marketing automation, and feeds data back into marketing platforms to drive fan engagement.

The system architecture powering the Square Enix Single Gamer View.

We built pipelines leveraging PubSub and Dataflow to collect data into BigQuery for data processing. This data comes from sources across Square Enix, including game data, website data from Google Analytics 360, aggregated sales data, digital advertising data, and even metrics from emails and surveys. We run AI and machine learning models daily, leveraging Vertex AI for training and managing models for larger-scale tasks.

In addition to building SGV, the alignment of all team members — from marketing to analytics to engineering to machine learning experts — is key. Our projects begin with data analysts, who draft proposals and estimate the value of the project. Community managers and brand teams then review these proposals to ensure alignment with player needs. Data scientists, following an agile approach, start with rule-based segmentation, and later transition to more advanced machine learning methods once early strategies prove successful. Finally, our data engineering and data protection teams provide oversight throughout, ensuring data security and effective deployment.

How SGV delivers AI-driven marketing

Today, the significance of SGV is set to grow even more, especially as we move toward the phasing-out of third-party cookies in 2024. This shift will highlight the importance of having robust internal systems for gathering and effectively using data to nurture relationships with fans.

The value of having an integrated CDP is often poorly understood until it is tied into marketing activations, fan retention, and marketing activity modeling. Consequently, here are three core initiatives we championed from the beginning and continue to support.

1. Marketing automation and feedback loop
Having all our marketing and game data in one place allowed us to develop and train more advanced ML-powered personalization systems. For instance, we designed a marketing automation framework with a feedback loop to test out different marketing actions. The process uses a contextual bandit model pipeline built with AutoML Tables and helps drive our marketing email campaigns, encouraging better retention, demo trials, and survey responses.

SGV also helped us build a self-optimizing recommendation system, which can detect when a player has reached a milestone in the game, compares them to similar players, and sends recommendations tailored to the individual. The system, which leverages BigQuery ML to make recommendations, receives feedback on whether the demo recommended was played and if so, how long the player played.

Self-optimizing demo recommendation system

2. Fan onboarding and retention
With SGV, we have improved Square Enix’s ability to understand and nurture life-long relationships with fans. Game teams can create more personalized customer experiences, which better service players’ needs by leveraging ML-based segmentations powered by BigQuery ML to identify users by play style, class or character preferences, or content they find interesting. For example, we can now provide players who enjoy player versus player (PvP) conflict with specific news regarding PvP content updates. This personalization not only improves the user experience but also helps to deepen their relationship and retention with the game. In fact, our teams saw a significant increase in players returning to specific Square Enix games after implementing this new data-driven approach.

3. Marketing ROI challenges
With evolving digital advertising standards and increased privacy regulations, particularly the phase-out of third-party cookies, digital tracking methodologies have become less viable. This has led to the resurgence of Marketing Mix Modeling (MMM). MMM is a statistical analysis method used to estimate the impact of various marketing tactics on sales and determine the effectiveness of each promotion.

Marketing mix models evaluate the impact of promotional activities on sales

Organizations without a centralized data repository would have to start an MMM project by first defining the necessary data, locating it, acquiring it, and then cleaning it. Using traditional data analysis methods, just preparing the data for modeling could take several months. However, as we have major data points inside SGV, data scientists can now quickly access data and develop proof-of-concept models. Additionally, leveraging tools like Supermetrics and Google Trends through Google Cloud, we can now pull more granular marketing data from our marketing sources and gain more valuable insights.

Bringing gaming fan engagement to the next level with Google Cloud

Overall, SGV has been quite literally a game changer for Square Enix, helping us to increase player engagement and enhance marketing ROI measurement. By integrating Google Cloud tools, advanced analytics, and proprietary data, we have consistently delivered impressive results. Our overall email marketing performance has increased by 20%, with some campaigns seeing as much as a 150% increase particularly when our recommendation engine effectively addresses long-tail needs. We have also seen a 10% increase in game retention and completion rates.

In addition, building a custom CDP tailored to our specific requirements has increased our work efficiency. Our teams benefit from an automated optimization system, allowing us to save time managing tasks like promoting catalog titles, sending retention emails, and collecting feedback surveys.

The next step is to continue enhancing the management and scalability of our machine learning processes. We believe that effectively using Vertex AI will be crucial during this next phase, especially as we tackle the challenge of moving from experimentation to production. For example, we are already considering how to use Google Cloud’s new generative AI features in BigQuery to enrich and help automate our existing data processes. We are also keen to see how new data scientist-friendly features, such as managed notebooks and BigQuery DataFrames, can help us accelerate the development and initial adaptation of notebook code. While notebooks may not fully meet these standards on their own, these tools can simplify experimentation and transition to production without needing specialized expertise.

While the full impact of AI in the gaming industry remains to be seen, we are looking forward to seeing (and also shaping) a future where we can finally bridge the “last mile” and deliver new innovative experiences to our players.

Interested in starting your own customer data platform journey? Learn more about how Google Cloud can help you prepare for a privacy-centric future.

Game-changing assets: Making concept art with Google Cloud's generative AI

Thu, 16 May 2024 16:00:00 +0000

Developing games is unique in that it requires a large variety of media assets such as 2D images, 3D models, audio, and video to come together in a development environment. However, in small game teams, such as those just getting started or “indie” teams, it’s unlikely that there are enough people to create such a wide variety and amount of assets. The lack of assets can create a bottleneck, throttling the entire game development team.

In this blog, we demonstrate how easy it is for gaming developers to deploy generative AI services on Google Cloud, showcase the available tooling of Model Garden on Vertex AI (including partner integrations like Hugging Face and Civitai), and highlight their potential for scaling game-asset creation.

Solution

Google Cloud offers a diverse range of generative AI models, accessible to users for various use cases. This solution focuses on how game development teams can harness the capabilities of Model Garden on Vertex AI, which incorporates partner integrations such as Hugging Face and Civitai.

Many artists run these models on their local machine, e.g., Stable Diffusion on a local instance of Automatic 1111. However, considering the cost of high-end GPUs, not all people have access to hardware required to do so. Therefore, running these models in the cloud is a way to access the compute needed while mitigating the need to invest in high-end hardware upfront.

Our primary objective is to explore how these tools can streamline and scale game-asset creation.

Concept or pre-production assets

Assets are the visual and audio elements that make up a game's world. They have a significant impact on the player's experience, contributing to the creation of a realistic and immersive environment. There are many different types of game assets, including:

2D and 3D models
Textures
Animations
Sounds and music

Here's a typical life journey of a typical 3D game asset, such as a character:

Concept art: Initial design of the asset
3D modeling: Creation of a three-dimensional model of the asset
Texturing: Adding color and detail to the model in alignment with the game's style
Animation: Bringing movement to the asset (if applicable)
Sound effects: Adding audio elements to enhance the asset
Import to game engine: Integration of the asset into the game engine that powers the gameplay

Generative AI can streamline the asset-creation process by generating initial designs, 3D models, and high-quality textures tailored to the game's style. In this way, game artists can quickly provide assets that unlock the rest of the game team in the short term, while allowing them to focus on long term goals like art direction and finalized assets.

Read on to learn how to accomplish the first step of game asset creation – generating concept art – on Google Cloud using Vertex AI and Model Garden with Stable Diffusion. We'll cover how to access and download popular LoRA (Low-Rank Adaptation) adapters from Hugging Face or Civitai, and serve them alongside the stabilityai/stable-diffusion-xl-base-1.0 model (from Model Garden) on Vertex AI for online prediction. The resulting concept art images will be stored in a Google Cloud Storage bucket for easy access and further refinement by artists.

Infrastructure setup

1. Prerequisites:

Google Cloud Project: Select or create a project. When you first create an account, you get a $300 free credit towards your compute/storage costs.
Billing enabled: Verify that billing is enabled for your project.
APIs enabled: Enable both the Vertex AI and Compute Engine APIs.

2. Storage and authentication:

Cloud Storage Bucket: Create a bucket to store downloaded LORAs and experiment outputs.
Service Account: Create a service account with the following roles:
- Vertex AI User
- Storage Object Admin

We'll use this service account with our Python notebook for model creation and storage management.

3. Colab Enterprise setup:

Runtime remplate: Create a runtime template in Colab Enterprise following the instructions at https://cloud.google.com/vertex-ai/docs/colab/create-runtime-template
Runtime instance: Create a runtime instance based on your Runtime template created above. Follow instructions at https://cloud.google.com/vertex-ai/docs/colab/create-runtime.
Upload notebooks: Download these three notebooks from git and upload them to the Colab Enterprise.

4. Running your notebooks:

Connecting notebooks: Once you've uploaded the notebooks, ensure they are connected to the runtime you created in step 3 above. This ensures your notebooks have access to the necessary resources for execution.
Cloud NAT: If your runtime environment requires internet access to download packages, you can create a Cloud NAT following these instructions.

This completes the infrastructure setup. You're ready to run your Jupyter notebooks to deploy a LoRA model with stabilityai/stable-diffusion-xl-base-1.0 on a Vertex AI prediction endpoint.

ExecutionUpon successful execution of all the above steps, you should see three Jupyter notebook files in Colab Enterprise as follows:

1. Create_mg_pytorch_sdxl_lora.ipynb

This notebook contains steps to download popular LoRA (Low-Rank Adaptation) adapters from either huggingface.co or civitai.com. It then serves the adapter alongside the stabilityai/stable-diffusion-xl-base-1.0 model on Vertex AI for online prediction.
In this notebook, set the following variables to begin:
- HUGGINGFACE_MODE: If enabled, the LoRA will be downloaded from Hugging Face. Otherwise, it will be downloaded from Civitai.

Upon successful execution, this notebook will print "Model ID" and "Endpoint ID." Save these values for use in the following notebooks.
If HUGGINGFACE_MODE is unchecked or disabled, ensure you update the Civitai variables within the notebook.

2. GenerateGameAssets.ipynb

This notebook contains code to convert text to images. Set the following variables to begin:
- ENDPOINT_ID: Obtained from successful execution of "1.Create_mg_pytorch_sdxl_lora.ipynb".

Update the prompts in the notebook as needed.

Upon successful execution, you should see the following results:

Concept art images will be uploaded to your configured GCS storage bucket.
Images will be displayed for reference.

3. CleanupCloudResources.ipynb

Execute this notebook to clean up resources, including the endpoint and model.
Before executing, set the following variables:
- MODEL_ID and ENDPOINT_ID: Obtained from successful execution of "1.Create_mg_pytorch_sdxl_lora.ipynb".

Congratulations! You've successfully deployed the stabilityai/stable-diffusion-xl-base-1.0 model from Model Garden on Vertex AI, generated concept art for your games, and responsibly deleted models and endpoints to manage costs.

Final thoughts

Integrating Stable Diffusion-generated images into a game requires careful planning:

Legal rights: Ensure you have the necessary permissions to use generated images. Always consult a legal professional if you have any questions about image usage rights.
Customization: Edit and refine the images to match your game's style and technical needs.
Optimization: Optimize images for in-game performance and smooth integration into your game engine.
Testing: Thoroughly test for quality and performance after incorporating the assets.
Ethics and compliance: Prioritize ethical considerations and legal compliance throughout the entire process.
Documentation and feedback: Maintain detailed records, backups, and be responsive to player feedback after your game's release.

References

Explore AI models in Model Garden https://cloud.google.com/vertex-ai/docs/start/explore-models
Your guide to generative AI support in Vertex AI https://cloud.google.com/blog/products/ai-machine-learning/vertex-ai-model-garden-and-generative-ai-studio

Tencent: Building analytics culture for better game development

Fri, 10 May 2024 16:00:00 +0000

Tencent is a leading internet and technology company headquartered in Shenzhen, China. Our mission is “value for users, tech for good.” We are also the company behind world-famous games like Level Infinite, PubG Mobile, Honor of Kings, GTFO, and Assassin’s Creed Jade. In February 2020, we acquired Funcom, a gaming company that recently celebrated its 30th anniversary, with a critically acclaimed portfolio that includes games like The Longest Journey, Anarchy Online, and Metal Hellsinger.

The acquisition of Funcom brought together the best of tech and the gaming industry, and with that, we decided to build an analytics culture at Funcom, supported by Google Cloud. We’ll share the challenges we faced as well as the solutions that were implemented in the process.

To demonstrate, we’ll use the online multiplayer survivor game, Conan Exiles, as an example. Developed by Funcom and released in 2017, this game constantly updates and releases new content, and was transitioned into a live service game model in 2022. As such, we needed data to support our business decisions.

An architecture for scalability and growth

Funcom’s architecture was developed to support the internal development team with live operations and to monitor the game servers’ health. The entire architecture was made of on-premises virtual machines and open-source frameworks, which limited our use cases and scalability. The legacy technology stack was not built with a data-driven approach in mind from a live service game model standpoint.

Based on interest from both developers and executives at Funcom, we decided to collaborate with Google Cloud to develop a new architecture. With only a few months to go before the release of the first season of Conan Exiles, the Google Cloud team provided us with a fully operational data warehouse that could be used to build dashboards and provide insights to key stakeholders, including executives, marketing, and live operations. The diagram below illustrates the architecture that we used:

We built our new technology stack according to a few key criteria: ease of integration, diverse use case coverage, and optimizing the total cost of ownership (TCO).

Building this data platform was like putting together puzzle pieces. We replaced our legacy data infrastructure using key products like Cloud Storage and BigQuery, which acted as a data lake and query engine. As a result, we were able to build a robust data pipeline and a well-established data platform foundation in less than two months, enabling access to a host of new game data that was previously unseen, such as in-game player activity playtests. This includes marketing data, such as social listening and community responses, performance data about CPU, graphic or memory usage, and even crash monitoring data.

A new foundation to connect gaming and marketing datasets

With the new architecture set up, we decided to explore other ways of using data to optimize cost performance and get better control of the data stack to connect marketing and sales datasets. For example, game leads need to understand how data can support in-game development, while our marketing teams should have easy access to in-game data to support marketing efforts.

To help, we automated the entire pipeline with always up-to-date KPI reports to monitor our marketing performance within Google Cloud. Additionally, the data team can provide recommendations based on in-depth insight analytics by connecting the data across player behavior, community, and marketing.

Moving forward with a revenue-generating tech stack

With Google Cloud, we’ve been able to redesign the raw data pipeline and data lake architecture without impacting our gaming data pipeline, day-to-day decision-making systems, or requiring additional engineering overhead. As a result, we are able to process twice as much game data on a daily basis, compared to our previous architecture. In addition, we have also decreased our overall monthly costs by 70% using BigQuery and Cloud Composer.

Moving forward, we’re looking to further expand this architecture with near real-time pipelines built with Pub/Sub. We are also planning to improve data quality monitoring and alerting, and standardizing our data structure, so that we can deploy newly developed features directly to beta and accelerate our time to market.

The evolution of play: From live to living games

Mon, 18 Mar 2024 12:00:00 +0000

Over the last year, we have seen the live game model explode in popularity, with live service modalities now incorporated into just about every top game release. Player engagement is at an all-time high, and a significant portion of that comes from live games. A recent survey paints a clear picture: a staggering 95% of studios are developing some form of a live game experience. In fact, Electronic Arts (EA) reported that live services comprised more than 73% of its business in 2023, even while EA continues to release traditional retail games. This is a testament to the power of this evolving model and signifies a crucial shift – developers understand the need for games that evolve alongside their passionate player bases.

Now, with generative AI (gen AI), there is an opportunity for developers to take live games to the next level. Enter: living games. Living games will transform the games landscape, empowering creators — designers, developers, artists, and marketers — to reimagine their workflows and craft experiences in ways that were previously unimaginable.

Live games: a new era, new needs

For players, live games offer exciting features like constant updates, new content, and long-term support. For developers to keep pace they turn to Google Cloud, allowing them access to the infrastructure needed to build these seamless virtual experiences. But it's not just about game servers — databases are equally crucial. Games live and breathe on player data. For every ‘prestige’ and ‘platinum’ earned within a game, Google Cloud ensures that they’re all meticulously remembered. We also equip developers with insights that allow them to refine player engagement strategies. With live games, developers can:

Launch and operate planet scale games: Of course, before anything else, a game simply has to work when a player wants to game! Google Kubernetes Engine (GKE) provides a scalable foundation for managing game servers, effortlessly accommodating player fluctuations. Cloud Spanner offers a globally distributed, high-performance database ideal for the always-on demands of live games. Store player profiles, in-game economies, and more, ensuring smooth gameplay for a massive, worldwide player base.
Make data-driven decisions: Analytics are paramount. Understanding player behavior through robust data analysis using BigQuery and Looker allows developers to tailor content and features for maximum engagement. Imagine a game that dynamically adjusts difficulty based on a player's performance tracked in BigQuery and visualized in Looker.

By providing a powerful, adaptable, and secure platform for live games, developers can focus on what they do best: crafting exceptional games and building thriving player communities.

Live games transform into living games with generative AI

As we look ahead, living games will disrupt the live game model. Now, the game itself will be an active participant. Gone are the days of static experiences. Using Google Cloud products like GKE and Vertex AI, developers can create personalized games that dynamically adapt based on player choices, generating unique quests and challenges, and continuously evolving their narratives. This creates a deeper level of player immersion and engagement, leading to a more enriching experience for everyone involved.

NCSOFT, a global game developer and publisher of popular titles including Guild Wars 2 and Blade & Soul, is using Google Cloud's AI infrastructure to power its in-house large language model (LLM) set, VARCO LLM. NCSOFT's VARCO LLM is specialized for generating high-quality content required for game development. Using VARCO LLM, NCSOFT has developed a suite of AI power tools ranging from text creation, managing "digital" human characters, player-to-virtual-character conversation, and dynamic storyline generation based on player actions. The result is a more engaging and dynamic gaming experience for players.

Building the future of play through a collaborative ecosystem

The future of games is also collaborative. By fostering a thriving ecosystem of game developers, technology providers, and players, we are building truly living worlds that push the boundaries of storytelling and engagement.

Google Cloud goes beyond just offering its own technology and infrastructure. We act as a bridge, connecting you to the most powerful and innovative game-development tools from industry leaders.

For example, Replica Studios, an AI voice technology company, is partnering with Google Cloud, and GlobalLogic to enhance game production, gameplay, and distribution. Replica Studio’s Voice Lab technology breathes new life into non-player characters (NPCs) and stories with advanced, cost-effective gen AI. Replica’s Voice Lab takes its cues from a prompt to generate a unique voice, which when combined with Google’s Gemini language models, can utilize a game’s assets, environment, and lore to create a cast of thousands of diverse characters with unique voices, personalities, motivations, and behaviors. Replica Studios, Globallogic and Google Cloud are ushering in a new era of living games that will define the next generation of interactive entertainment.

Google Cloud’s partner ecosystem provides access to the latest engines, online back-end solutions, Web3 platforms, and gen AI tools, empowering game developers to focus on what matters most: crafting experiences that resonate with players on a deeper level, forging lasting connections, and a shared sense of community within living games.

Google Cloud stands ready to empower the next generation of game experiences. Transformative technologies like gen AI allow developers to unleash their creativity and bring to life games that are more immersive, dynamic, and deeply engaging than ever before. The future of play is bright, and with Google Cloud as a partner, developers have the tools and support to make their most ambitious visions a reality.

To learn more about Google Cloud for Games, visit goo.gle/cloudforgames.

Running machine learning in the cloud for live service games

Thu, 22 Feb 2024 17:00:00 +0000

Generative AI has become the number one technology of interest across many industries over the past year. Here at Google Cloud for Games, we think that online game use cases have some of the highest potential for generative AI, giving creators the power to build more dynamic games, monetize their games better, and get to market faster. As part of this, we’ve explored ways that games companies can train, deploy, and maintain GenAI utilizing Google Cloud. We'd like to walk you through what we’ve been working on, and how you can start using it in your game today. While we’ll focus on gen AI applications, the framework we’ll be discussing has been developed with all machine learning in mind, not just the generative varieties.

Long term, the possibilities of gen AI in Games are endless, but in the near term, we believe the following are the most realistic and valuable to the industry over the next 1-2 years.

Game production
Adaptive gameplay
In-game advertising

Each of these helps with a core part of the game development and publishing process. Generative AI in game production, mainly in the development of 2D textures, 3D assets, and code, can help decrease the effort to create a new game, decrease the time to market, and help make game developers more effective overall. Thinking towards sustaining player engagement and monetizing existing titles, ideas like adaptive dialogue and gameplay can keep players engaged, and custom in-game objects can keep them enticed. In-game advertising opens a new realm of monetization, and allows us not only the ability to hyper-personalize ads to views, but to personalize their placement and integration into the game, creating seamless ad experiences that optimize views and engagement. If you think about the time to produce a small game, never mind a AAA blockbuster, development of individual game assets consumes an immense amount of time. If generative models can help reduce developer toil and increase the productivity of studio development teams by even a fraction, it could represent a faster time to market and better games for us all.

As part of this post, we introduce our Generative AI Framework for Games, which provides templates for running gen AI for games on Google Cloud, as well as a framework for data ingest and storage to support these live models. We walk you through a demo of this framework below, where we specifically show two cases around image generation and code generation in a sample game environment.

But before we jump into what we’re doing here at Google Cloud, let’s first tackle a common misconception about machine learning in games.

Cloud-based ML plus live services games are a go

It’s a common refrain that running machine learning in the cloud for live game services is either cost prohibitive or prohibitive in terms of the induced latency that the end user experiences. Live games have always run on a client-server paradigm, and it’s often preferable that compute-intensive processes that don’t need to be authoritative run on the client. While this is a great deployment pattern for some models and processes, it’s not the only one. Cloud-based gen AI, or really any form of AI/ML, is not only possible, but can result in significantly decreased toil for developers, and reduced maintenance costs for publishers, all while supporting the latencies needed for today’s live games. It’s also safer — cloud-based AI safeguards your models from attacks, manipulation, and fraud.

Depending on your studio’s setup, Google Cloud can support complete in-cloud or hybrid deployments of generative models for adaptive game worlds. Generally, we recommend two approaches depending on your technology stack and needs;

If starting from scratch, we recommend utilizing Vertex AI’s Private Endpoints for low latency serving, which can work whether you are looking for a low ops solution, or are running a service that does not interact with a live game environment.
If running game servers on Google Cloud, especially if they are on Google Kubernetes Engine (GKE), and are looking to utilize that environment for ultra-low latency serving, we recommend deploying your models on GKE alongside your game server.

Let’s start with Vertex AI. Vertex AI supports both public and private endpoints, although for games, we generally recommend utilizing Private Endpoints to achieve the appropriate latencies. Vertex AI models utilize what we call an adaptor layer, which has two advantages: you don’t need to call the entire model when making a prediction, and any fine tuning conducted by you, the developer, is contained in your tenant. Compared to running a model yourself, whether in the cloud or on prem, this negates the need to handle enormous base models and the relevant serving and storage infrastructure to support them. As mentioned, we’ll show both of these in the demo below.

If you’re already running game servers on GKE, you can gain a lot of benefit from running both proprietary and open-source machine learning models on GKE as well as taking advantage of GKE’s native networking. With GKE Autopilot, our tests indicate that you can achieve prediction performance in the sub-ms range when deployed alongside your game servers. Over the public internet, we’ve achieved low millisecond latencies that are consistent, if not better, with what we have seen in classic client side deployments. If you’re afraid of the potential cost implications of running on GKE, think again — the vast majority of gaming customers see cost savings from deploying on GKE, alongside a roughly 30% increase in developer productivity. If you manage both your machine learning deployments and your game servers with GKE Autopilot, there’s also a significant reduction in operational burden. In our testing, we’ve found that whether you are deploying models on Vertex or GKE, the cost is roughly comparable.

Unified data platforms enable real-time AI

AI/ML driven personalization thrives on large amounts of data regarding player preferences, gameplay, and the game’s world and lore. As part of our efforts in gen AI in games, we’ve developed a data pipeline and database template that utilizes the best of Google Cloud to ensure consistency and availability.

Live games require strong consistency, and models,whether generative or not, require the most up-to-date information about a player and their habits. Periodic retraining is necessary to keep models fresh and safe, and globally available databases like Spanner and BigQuery ensure that the data being fed into models, generative or otherwise, is kept fresh and secure. In many current games, users are fragmented by maps/realms, with hard lines between them, keeping experiences bounded by firm decisions and actions. As games move towards models where users inhabit singular realms, these games will require a single, globally available data store. In-game personalization also requires the live status of player activity. A strong data pipeline and data footprint is just as important for running machine learning models in a liveops environment as the models themselves. Considering the complexity of frequent model updates across a self-managed data center footprint, we maintain it’s a lighter lift to manage the training, deployment, and overall maintenance of models in the cloud.

By combining a real-time data pipeline with generative models, we can also inform model prompts about player preferences, or combine them with other models that track where, when, and why to personalize the game state. In terms of what is available today, this could be anything from pre-generated 3D meshes that are relevant to the user, retexturing meshes to different colors, patterns or lighting to match player preferences or mood, or even the giving the player the ability to fully customize the game environment based natural language. All of this is in service of keeping our players happy and engaged with the game.

Demoing capabilities

Let’s jump into the framework. For the demo, we’ll be focusing on how Google Cloud’s data, AI, and compute technology can come together to provide real-time personalization of the game state.

The framework includes:

Unity for the client and server

Open source:

Terraform
Agones

Google Cloud:

GKE
Vertex AI
Pub/Sub
Dataflow
Spanner
BigQuery

As part of this framework, we created an open-world demo game in Unity that uses assets from the Unity store. We designed this to be an open world game — one where the player needs to interact with NPCs and is guided through dynamic billboards that assist the player in achieving the game objective.. This game is running on GKE with Agones, and is designed to support multiple players. For simplicity, we focus on one player and their actions.

Now, back to the framework. Our back-end Spanner database contains information on the player and their past actions. We also have data on their purchasing habits across this make-believe game universe, with a connection to the Google Marketing Platform. This allows us in our demo game to start collecting universal player data across platforms. Spanner is our transactional database, and BigQuery is our analytical database, and data flows freely between them.

As part of this framework, we trained recommendation models in Vertex AI utilizing everything we know about the player, so that we can personalize in-game offers and advertising. For the sake of this demo, we’ll forget about those models for a moment, and focus on two generative AI use cases: image generation, NPC chat, and code generation for our adaptive gameplay use case. To show you both deployment patterns that we recommend for games, deploying on GKE alongside the game server, and utilizing Vertex AI. For image generation, we host an open-source Stable Diffusion model on GKE, and for code generation and NPC chat we’re using the gemini-pro model within Vertex AI. In cases where textures need to be modified or game objects are repositioned, we are using the Gemini LLM to generate code that can render, position, and configure prefabs within the game environment.

As the character walks through the game, we adaptively show images to suggest potential next moves and paths for the player. In practice, these could be game-themed images or even advertisements. In our case, we display images that suggest what the player should be looking for to progress game play.

In the example above, the player is shown a man surrounded by books, which provides a hint to the player that maybe they need to find a library as their next objective. That hint also aligns with the riddle that the NPC shared earlier in the game. If a player interacts with one of these billboards, which may mean moving closer to it or even viewing the billboard for a preset time, then the storyline of our game adapts to that context.

We can also load and configure prefabs on the fly with code generation. Below, you’ll see our environment as is, and we ask the NPC to change the bus color to yellow, which dynamically updates the bus color and texture.

Once we make the request, either by text or speech, Google Cloud GenAI models generate the exact code needed to update the prefab in the environment, and then renders it live in the game.

While this example shows how code generation can be used in-game, game developers can also use a similar process to place and configure game objects within their game environment to speed up game development.

If you would like to take the next step and check out the technology, then we encourage you to explore the Github link and resources below.

Additionally, we understand that not everyone will be interested in every facet of the framework. That's why we've made it flexible – whether you want to dive into the entire project or just work with specific parts of the code to understand how we implemented a certain feature, the choice is yours.

If you're looking to deepen your understanding of Google Cloud generative AI, check out this curated set of resources that can help:

Last but not least, if you’re interested in working with the project or would like to contribute to it, feel free to explore the code on Github, which focuses on the GenAI services used as part of this demo:

Generative AI Quickstart for Gaming

Game on: Aiven for Apache Kafka and BigQuery - your ultimate gaming cheat code

Tue, 30 Jan 2024 17:00:00 +0000

The games industry is one of the most data-driven industries in the world. Games generate massive amounts of data every second, from player behavior and in-game transactions to social media engagement and customer support tickets. This data can be used to improve games, make better business decisions, and create new and innovative experiences for players.

However, the games industry also faces a unique challenge: how to analyze this data in real-time or near-real-time. This is because games are now constantly changing and evolving, and players expect a personalized and highly responsive experience. Imagine, you and your friends are all set to join your favorite game’s new version launch! You take the day off, get your snacks ready and POOF! Server unresponsive… Reload… Still nothing.

Massively Multiplayer Online Role-Playing Games (MMORPGs), for instance, need to be able to handle a large number of concurrent players while simulating a virtual world in real time. This can put a strain on game server infrastructure, and it can be difficult to scale the infrastructure to meet the needs of a growing player base. Here is where real-time analytics plays a role in auto-scaling infrastructure in response to these events.

Automating infrastructure scaling in real-time

Aiven for Apache Kafka provides time-value to player-volume-based data, allowing automation of infrastructure scaling, based on traffic patterns and load. In addition, with Aiven for InfluxDB and Aiven for Grafana, data infrastructure teams have insights into the health of gaming services — as the gameplay is happening. Once certain thresholds are detected, automation scripts employing the Kubernetes Operator or Terraform Provider can spin up new game services to answer demand.

As one of the most demanding industries for computing power, online or mobile games need to be able to handle a large number of concurrent players. This can put a strain on game server infrastructure, and it can be difficult to scale the infrastructure to meet the needs of a growing player base.

The Aiven Platform offers multiple features, across all managed services, that make it well-suited for automated game service scaling, including:

Scalability: Aiven services can be scaled to handle any amount of data, making it ideal for high player volume gaming scenarios, such as a highly anticipated version launch.
High reliability and availability: Via the management plane — or Aiven Console — highly reliable services are designed to be always available. (Typically, even smaller plans provided by Aiven for Apache Kafka include three High-Available Nodes, out of the box.)
Security: The Aiven Console offers a number of security features and compliance by default to protect your data, including encryption and authentication.

Some of the benefits gained when using the Aiven Stack, shown in our reference architecture, to scale game servers are:

Improved performance: By automatically scaling the number of game servers up or down as needed, you can ensure that your game servers are always operating at optimal capacity.
Cost optimization: You can save money on your cloud computing costs by only running the number of game servers that you need.
Improved scalability: Aiven for Apache Kafka, coupled with Aiven observability services (Aiven for InfluxDB and Aiven for Grafana), can be used to scale your game server infrastructure to meet the needs of your growing player base.

Future-proofing automated scaling

Google Pub/Sub capabilities within the BigQuery suite can be used together to perform longer-term analytics for the games industry. Pub/Sub is a near-real-time/longer-term messaging service that can be used to collect data from game servers, in our case, messages from Aiven for Apache Kafka.

There are a variety of use cases beyond the scope of auto-scaling infrastructure that can be leveraged when using Pub/Sub and BigQuery for longer-term analytics of player telemetry data:

Player behavior analysis: By tracking player behavior over time, game companies can identify trends and patterns in how players are playing their games. This information can be used to improve the player experience, develop new content, and balance in-game economies.
Game performance analysis: By tracking game performance over time, game companies can identify areas where technical performance is struggling. This information can be used to fix bugs, optimize performance, and improve the overall quality of the game experience.
Business intelligence: By analyzing data on player engagement, revenue, and other metrics, companies can make better business decisions. For example, a gaming company could use this data to identify their most popular and profitable titles.

Several benefits games industry customers will see by using Pub/Sub and BigQuery for longer-term analytics in the games industry are:

Scalability and reliability: Pub/Sub and BigQuery are both highly scalable and highly available services that can handle any amount of data.
Security: Pub/Sub and BigQuery offer a number of security features to help protect your data, including encryption and authentication.
Cost-optimization: By analyzing longer term data points, Pub/Sub and BigQuery can help forecast future player workloads and enable adjustments to auto-scaling behavior.

Aiven + Google Cloud = better together

By partnering with Google Cloud and Aiven on Google Cloud, the games industry can prepare for a worry-free launch while having the data to understand players and keep them coming back for more. Service reliability is key — a hassle-free experience dictates the success of the game! — but cost should always be considered. By lowering the total cost of operations, and right-sizing in real-time, you can achieve greater game-play usability while minimizing unnecessary overscaling.

Predictive analytics in BigQuery allows you to tweak the scaling parameters based on past data, enabling greater control of future volumes that would otherwise be lost. The combination of managed services from Aiven and Google Cloud add time value to data — and increased revenue from a successful launch. Game on!

Conclusion and next steps

As the games industry continues to evolve and embrace data-driven decision-making, the combination of Aiven for Kafka and Pub/Sub capabilities within the BigQuery suite will become increasingly essential for success. By harnessing the power of real-time data, games companies can unlock new opportunities, enhance player experiences, and drive sustainable growth. If you’re ready to learn more, check out the following links below:

For further reading of connecting Aiven with Google Native services: Shorten the path to insights with Aiven for Apache Kafka and BigQuery.
How Google Cloud empowers the games industry to achieve success: Game on and on and on: Google Cloud's strategy for live service games
Ready to give it a try? Click here to check out Aiven’s listing on Google Cloud Marketplace, and let us know what you think.

The generative AI revolution in the games industry: A path to boundless creativity

Fri, 06 Oct 2023 16:00:00 +0000

The games industry has continually pushed the boundaries of technology, harnessing it to unleash the human imagination. Game developers have embraced cutting-edge advancements in computing, graphics, networking, social media, and data to craft experiences that today captivate over 3 billion players worldwide.

By being on the forefront of technology, the games industry serves as a beacon of innovation for other businesses, showcasing new methods to inform, engage, and connect people at scale. For example, while consumers were beginning to adopt mobile devices en masse in the mid 2000’s, games were early pioneers, adopting the trend into their business models. Today, nearly half the planet plays games; the majority through their mobile device.

The industry’s early learnings in mobile has led to “live service games” which is today the dominant business model in the industry across nearly all game platforms. Live service games constantly change and evolve, relying on a cycle of creation, engagement, and learning between developer and player.

Today, we stand at the forefront of a new era. It is paramount to explore how the games industry can leverage the transformative potential of generative artificial intelligence (gen AI) to further push the boundaries of game experiences. With generative AI, those that create games — designers, developers, artists, marketers, and more — can revolutionize how they work, and expand the range of experiences they can deliver.

Understanding the generative AI difference for games

So, what sets generative AI apart from its predecessors? On one level, it's evolutionary. To think back, games have long used the term “AI,” but our definition was a constrained set of rules, decision trees, and behaviors. Players engaged with “bots,” but compared to a human, bots often weren’t as engaging. More recently, game companies have been using true AI and ML frameworks (built on data and analytics) to understand players, manage churn, and increase monetization. Some have even integrated their frameworks to advanced AI algorithms like large language models (LLMs) which brings us to generative AI.

Generative AI goes well beyond an evolutionary step forward, however. It’s a whole new dimension for innovation, and, I posit, the biggest change to the industry of games since the introduction of real-time 3D graphics. Here are two fundamental ways generative AI will transform games:

1. Scalable Development with generative AI: Creating content is one of - if not the largest - expenses that games can incur. According to the UK’s CMA, blockbuster games can have development budgets well over $100 million. Even with these massive investments, game teams can struggle to keep up with player demand for new content, especially as these audiences grow to planet-scale. Generative AI can help accelerate game production across the board - code, art, dialogue, and more, enabling these teams to better serve their players at scale. It can help existing teams create better content at a faster pace, with improved collaboration, ideation, and personalization. What’s more, enterprise-grade generative AI allows developers to leverage this new technology in a way which is respectful of intellectual property, while protecting one's own proprietary data.

2. Real-Time generative AI: As development teams integrate generative AI into their production process, some have begun integrating it into the game itself. This has the potential of revolutionizing player experiences, with generative AI running in real-time as the game is played. This is generative AI that is responsive and dynamic, distilling huge LLMs to react and interact with a player in real-time. Games themselves will be able to generate content based on the explicit or even implicit actions of players: from instantly generated new items and levels to in-game characters that can have lifelike discussions. Imagine games that can naturally respond to a player’s voice, or generate entirely novel content in response to player behavior. With these examples, one can imagine a new paradigm of player personalization and game interaction which is wholly unlike anything the world has seen before.

Getting started on the path of innovation

The future I've described may sound visionary, but it is closer than we might think. In fact, the foundation for this future is already being laid today, as individuals and companies explore the fundamental models that will shape the games industry's transformation. I believe that as the games industry embraces generative AI, the business will go through another tectonic shift. Just as the preeminent business model evolved from boxed software games to live service games, we will evolve again; this time into “living games”. In such a model, the relationship cycle between the player and developer expands to the game itself, with all three interacting to enrich the player experience along with business outcomes.

Successfully incorporating generative AI in games will take more than desire and drive. Developers will need to harness their data and use it in entirely new ways; training models on reliable datasets capable of generating consistent and compelling outcomes. Game developers will also need to implement AI responsibly and securely, safeguarding intellectual property while vigilantly respecting the player.

At Google Cloud, we are dedicated to providing a comprehensive ecosystem for living games, encompassing powerful game servers, robust data assets, fast networks, and the most powerful and responsible AI capabilities. Today, Game developers should pay particular attention to Vertex AI, which empowers generative AI on proprietary data in a secure and powerful environment. Within Vertex AI, our model garden offers a diverse range of proprietary and open foundation models for teams to learn from and create with, all in a secure and robust enterprise environment, perfectly suited to game development. Over the past few months, we have garnered invaluable feedback and developed roadmaps based on what game companies aspire to see next with generative AI from Google.

As I mentioned earlier, the games industry has always thrived on human invention, creativity, and the drive to adopt state-of-the-art technologies. While generative AI may accelerate workflows, create novel player experiences, and open new avenues for audience engagement and monetization, it will not alter this fundamental truth. Like the revolutionary technologies that preceded it, game developers will use generative AI to amplify rather than replace their gifts, as it’s their creativity that's guaranteed to keep this industry so exciting.

For more details on jump starting your generative AI journey, download our eBook, The Executive’s Guide to Generative AI.

aside_block: <ListValue: [StructValue([('title', 'What\'s your prompt? Introducing "The executive\'s guide to generative AI"'), ('body', <wagtail.rich_text.RichText object at 0x7fe38924bb80>), ('btn_text', 'Read more'), ('href', 'https://cloud.google.com/blog/transform/introducing-executives-guide-to-generative-ai'), ('image', <GAEImage: introducing-executives-guide-to-generative-ai-midjourney-hero.png>)])]>

How DeNA developed a live streaming app in just 2 months with Live Stream API

Mon, 07 Aug 2023 16:00:00 +0000

As a provider of several mobile portal and e-commerce platforms in Japan, DeNA Co., Ltd (DeNA) has continuously sought to expand its portfolio of services to create value for society and innovation in the entertainment business by leveraging internet and AI technologies. That is why it launched Play-by-Play LIVE, an application that allows professional or amateur play-by-play commentators to stream live sporting events and engage with the audience as they do so. This is done by setting up multiple commentary channels for every match, so fans can engage with their favourite sports teams and commentators in real-time.

There are as many ways to enjoy sports as there are commentators. The concept of Play-by-Play LIVE is for the audience to enjoy sports in a new way that is not limited to just watching live sports streaming. This was done through the effort of our two business areas, from sports business and live streaming business.

One of the biggest hurdles DeNA faced was the short development time frame. With only two months from the start of the project to producing a proof of concept, building a video streaming server from scratch was a sizable challenge. The company decided to turn to Live Stream API by Google Cloud to deploy the application, entrusting the fully managed service to build the infrastructure so that we can focus more on the development of user experience. We chose Live Stream API for its ease of usability, cost and operations.

Putting the app together with Google Cloud

One of the company’s first steps in development was to use Flutter, an open-source software development kit (SDK) by Google, to create the application’s user interface, while the backend system was built with Google Cloud and Firebase. We also built an authentication feature that is aligned with our own user identification system, BAYSTARS ID, that is required to be able to use the service provided by Yokohama DeNA Baystars, a platform for the fans of the baseball team.

Here is how Play-by-Play LIVE was developed with Google Cloud:

Live streams of baseball games are received via Real Time Messaging Protocol (RTMP), and converted to HTTP Live Streaming (HLS) with Live Stream API. Information about the game, such as scores, pitching charts, and the name of players will be overlaid on screen, with this functionality being built on Flutter and Firebase.
The progress of live matches are updated in real time on the Cloud Firestore database, as a third party vendor is asked to input these information on a management console that’s developed with Firebase.
The information will then be displayed as a subtitle or caption. To fully utilize the application, users will need to log in with a guest or Baystars account.
Videos of these live sporting events are captured by OBS Studio, an open source software for video recording and live streaming, then combined with the footage of commentators on OBS Studio.The final videos are then exported to the Live Stream API via RTMP, converted to HLS and delivered to the audience on their mobile app.
Additional features such as quizzes and questions are built with Firestore. For functions that cannot be processed through Firestore and mobile apps, Cloud Functions is used to process them server side, through a serverless environment.
Flutter is also used to develop a console application for monitoring the comment and quiz functions, with this hosted on Firebase as a web application.

Broadcasting major sporting events confidently

With Google Cloud, DeNA was able to deliver a proof of concept for Play-by-Play LIVE within the short timeline of two months. On top of managing the hefty workload of live video streaming with Google Cloud, these tools have provided its team the means to build and deploy simple functions swiftly, such as with open source function-as-a-service frameworks in Cloud Functions. This has helped DeNA eliminate the complexity of constructing the non-essential components of the application’s backend system, while accelerating development.

Another benefit of using Google Cloud was to leverage the function of Firestore. When there was a specification change in the midst of development, we were able to reduce the amount of workload for the change since Firestore’s structure was simple.

Play-by-Play LIVE was eventually used to broadcast the 3x3 football championship that was held on March 26, 2023. The braodcast’s success has given DeNA the confidence to livestream other major sporting events in Japan through the application in the future.

DevOps Awards winner Improbable on “unleashing the full power of the cloud”

Fri, 21 Jul 2023 16:00:00 +0000

Improbable Worlds Limited — commonly known as Improbable — is a metaverse technology company that’s been at the forefront of building virtual worlds since 2012. With a world-class team, Improbable creates immersive gaming and event experiences using its Morpheus Technology, allowing over 15,000 users to interact as if they were in the same place at the same time. In this blog post, we’re highlighting Improbable for the DevOps achievements that earned the company the ‘Unleashing the Full Power of the Cloud’ award in the 2022 DevOps Awards. If you want to learn more about the winners and how they used DORA metrics and practices to grow their businesses, start here.

Video game builds traditionally require choosing between high infrastructure costs or longer wait times for developers and other downstream processes, but neither is tenable when you are trying to bring together tens of thousands of users in one single virtual environment. Rapid prototyping and QA are essential to the games industry or companies building virtual worlds, so developers must get working builds and deployments out as soon as possible to be able to validate, gather feedback, iterate, and try again. A single build failure can block the work and testing of hundreds of individuals — waiting hours on a fix is not an option — so the systems we provide must be fast and reliable.

Beyond just speed, scalability and stability were becoming major issues as Improbable’s original static and inflexible system had to adapt to a rapid expansion with more aggressive deadlines and an intense growth in daily build requirements. With an old infrastructure relying on tightly integrated systems, even upgrades and new features could lead to a failure in a single, small system, which could lead to an outage of the company’s entire service.

Meeting customer needs

To meet the growing demands, our organization saw that we needed to address both technological and process-based challenges. To meet customers’ needs, we needed a purpose-built infrastructure for Windows Metaverse (Game) development that was fast, cheap, highly reliable, and highly scalable. With this infrastructure also came the need to provide top-class support to keep developers unblocked.

The key to this project's success was adopting CI/CD as a service. This meant that we would provide guidance to developers on CI/CD development, as well as providing:

Infrastructure
Scripting
Source control
Automated merge tools
Automated release tools

Solution

A complicated problem like this needed a more technically elegant and complex solution to optimize build times rather than just throwing compute at the problem. In addition, Windows VMs can become difficult to manage without containerization. Costs can also spiral quickly, with diminishing returns on investment. Our teams found all of the technical solutions they needed in the cloud.

Using Google Cloud, we were able to develop a more stable, scalable, and sustainable approach to development that integrates a number of Google Cloud tools and services right from the beginning. When a job request comes in, Cloud Run scalers respond immediately to get the process going as quickly as possible — including a webhook scaler for instant response and speed, as well as a polling scaler for backup in case there are any webhook or external service-related issues. The scalers being Cloud Run, also auto-scale themselves to match demand and are highly reliable.

Rather than simply building directly on a VM, we utilized Compute Engine’s Windows Server for Containers images and launch a secondary Windows instance as a container on the host VM. Here we can isolate source code, assets, and build output to Virtual Hard Drives (VHDs) running on the host VM. During a build, changes made and build output can either be cached or reset at the end of the run, as we delete the container and reset the VHD back to a known state. This gives us absolute build isolation and reproducibility, as well as quickly allowing the build agent to return to the pool for its next job.

Our development process also introduces the use of “golden images” — Google Cloud images of a VM that has just run a full suite of all known possible build combinations for a specific project. For game design with Unreal, this includes builds for all platforms across debug, development, test, and shipping build configurations. All of the source, assets and build data will be cached to VHDs on the image. This cached data present on a golden image allows the next job to be iterative and thus significantly faster, while maintaining a known state to reset to.

These and other technological tools and improvements have reduced many of our biggest pain points, but even with these technical solutions, Improbable’s digital transformation would only be half-complete without also fostering a DevOps-first and engineering culture. Some of the most notable successes in this cultural transformation included:

Tracking metrics as soon as possible to identify and correct key time-wasting areas
Empowering smoother outage mitigation with reliable backup systems and clear workflows and guidelines on how best to deal with specific issues
Checking in on health checks with system redundancy to ensure key systems are working as expected
Implementing data sharing across teams for metrics around build times, reliability, and costs to keep teams honest and encourage cooperative problem solving
Staying proactive in finding, reporting, and addressing problems rather than remaining passive and reactive

To help developers do their best, our organization began to prioritize developer time over infrastructure costs. By reducing variance in lead time, we reduce developer frustration and allow developers to plan their time accordingly based on reliable delivery time averages. We also introduced practices to reduce outages, including rapid-state reporting, reducing complexity in shared systems and codebases.

The power of cloud

With the power of the cloud and focused DevOps practices, our organization has seen notable improvements in both cost savings and development efficiency. The number of build jobs performed daily went from 500 to 3000+, now with eight preflight validations for source changes whereas before there were only two. Costs dropped dramatically — from $1.7 per job to $0.5 — and projects that would have cost $900k per year in the old system can now be accomplished for $120k with the same build output.

Through the adoption of all five capabilities of cloud computing, we saw improvements to our software delivery and organizational performance. This includes:

Resource pooling: By running build requests in parallel on hundreds of VMs — including thousands of vCPUs working at maximum capacity — we optimize processing power by spreading the resource workloads, meaning that the more projects and customers involved, the lower the price is for each build job.
Rapid elasticity: With a double scaler tech stack that keeps our VM pool at optimum capacity all the time, build requests are serviced within 10-160 seconds. The VM pool is dynamically resized to match demand — including adding more VMs as needed and killing idle ones.
Measured service: We have visibility on whether we are on track with SLOs and feature budget spend by tracking our performance using bots made with Cloud Run that post easy-to-digest reports and updates, as well as track everything from build times to pass/fail rates using DataDog tracing and metrics stacks.
On demand self-service: Our developers can run experiments and gather test data without blockers or bureaucratic processes with tools that can automatically spin up a VM for their specific build request in an environment that is completely isolated from the production environment.
Broad network access: By using Cloud Identity-Aware Proxy (IAP) for access control and resource management with a zero-trust security model, our developers can remotely use our cloud resources at will without the limitations of office IP or broad, catch-all firewalls.

With these applications of the full power of the cloud, we’ve seen measurable improvements in the development process, including:

Deployment frequency: Build system rollouts went from weekly to after every merge, with projects, products, and customers now being able to deploy their metaverses hundreds of times per day.
Lead time for changes: Using the exotic cloud stack reduced lead times by at least 300% — with primary CI build average times dropping from 60 minutes to 15 and metaverse deployment builds going from 90 minutes to 25 — enabling Improbable to move faster and rapidly iterate and test new features.
Change failure rate: Overall failure rates for servicing build jobs went from 96% success to 99.99% with master build job success rates going from 80% to 99%, saving money and time — especially in the reduction of support calls.
Time to restore service: Between the dual scaler system, the container setup, and other cloud-based solutions, there have only been three occasions where we had to completely wipe the pool of VMs and restart the scalers due to technical issues, and these outages only lasted 10 minutes before all builds were back online and serviced.

Stay tuned for the rest of the series highlighting the DevOps Award Winners and read the 2022 State of DevOps report to dive deeper into the DORA research.

Accelerate your Multiplayer Game Development with the Global Scale Multiplayer Demo

Fri, 14 Jul 2023 16:00:00 +0000

Building the backend for a fast-paced competitive multiplayer game is a complex task. There are many moving parts to coordinate, and the product and solution landscape is practically endless. There are a myriad of managed products, supporting open source solutions and more that need to be evaluated, experimented with and implemented, and the path to using all of them in conjunction with one another is often not clear.

To solve this problem, and kickstart your journey on building your own multiplayer game, a multidisciplinary team of Google Cloud for Games experts banded together to create a global scale, integrated demo of a fast paced multiplayer game. The aim of this demo is to show the technologies and techniques we at Google Cloud have seen work in production to host, scale and orchestrate game backends for multiplayer games - from services and data storage all the way through to dedicated game servers.

This includes:

Game Engine

Unreal Engine 5 for the game client and dedicated server code.
A custom Go game launcher for client side authentication.

Open Source

Terraform, for infrastructure as code.
Agones for hosting and scaling dedicated game servers.
Open Match for match making our global player base.

Google Cloud

Cloud Build and Cloud Deploy for Continuous Integration and Deployment.
GKE Autopilot for hosting the backend microservices.
Globally distributed GKE Autopilot and Standard clusters running for hosting Agones orchestrated Unreal dedicated game servers
Anthos Service Mesh for cross cluster service discovery and communication.
Cloud Spanner for storing the player in-game data.

We’ll also share some tips and tricks the team of people working on this project learnt along the way so you can turbocharge your development journey even more!

Droidshooter

“Droidshooter” the global scale, multiplayer demo being played

We deliberately built a very simple multiplayer game called “Droidshooter” to highlight and make easily discoverable the server side technologies and techniques we took advantage of during the development of this demo.

In this game you fly a small spaceship against another two players, while attempting to get their health to zero by firing bursts of energy against the other ships. For each successful destruction of a rival spaceship, you receive a point. The player with the most number of points at the end of thirty seconds wins the round!

Architecture overview

Before we jump into the details of each component, let’s look at an architecture overview:

Architecture Overview of “Droid Shooter” (Click to Enlarge)

Since this is a game that is targeted at a global audience, we also have a global infrastructure footprint.

Our general backend services for our game are hosted in us-central1, since they aren’t latency sensitive, and consolidating them into a single region and cluster makes managing them much easier. Since player connection to the multiplayer, dedicated game servers are latency sensitive, we have them on multiple Kubernetes clusters in multiple regions around the world to ensure that they are close to our players locations.

Game state flow

Launching the game client

In the above architecture, each player will start an instance of the Game Launcher. This provides authentication via Google login, as well as options for launching the game.

Game State Sequence Diagram

At log in, a profile record is created if it doesn’t exist in the database. From there, the profile information is retrieved from the profile service and displayed here in the launcher (That’s one of our names above!) as well as in the game client.

Clicking “Open Droidshooter” will start the game client, which will request a game through the frontend service, providing an authenticated endpoint for almost all communications to hosted backend services for the game client, as well as the previously mentioned launcher.

Match making

From here, the client will need to request the global set of “ping locations” in each of the regions that our game servers will be hosted in, since our game requires low latency for an optimal, real time experience. Therefore we need to actively measure ping latencies from each region from the player’s location, so that they can be matched to players of similar latency from where they are situated, and ignore regions where latency is too high. Unfortunately, you can’t just send players to only the closest region to them, as that splits your player base, and results in an inability to match players that would otherwise be able to play a game together.

Once the game client has utilized each of the latency measurement endpoints in each region and recorded the player’s latency to each region, it sends this data to the matchmaking system (through the authenticated frontend) requesting to be grouped with other players so they can play a game! In Droidshooter, we have a very simple matchmaking system, where our matchmaker looks to match three players together that have the closest latency to each other.

Allocating a game server

Once a group is formed to play a game, the matchmaker requests a dedicated game server from the game server orchestration system in the region in which it has determined the best place to place these players. Since all these players need to have the same experience, the dedicated game server's job is to simulate the game world state, and communicate this information to all connected players in real time, so they can all enjoy the fun experience of playing Droidshooter.

Therefore the matchmaker passes back to the game clients (through the frontend) the address and port each client will need to connect directly to, since the game state is an in-memory simulation, and then, they play Droidshooter!

Final cleanup

Once the game is complete, the game server shuts itself down, and thanks to our game server fleet management system, a new one is spun up right away for new players to play on.

Ecosystem breakdown

Let’s look at each of the products and open source projects we used to implement the above flow, and why we chose each of them:

Terraform

When getting started with this project, the first thing we did was decide that we wanted to have an infrastructure as code set up for automating the creation of the infrastructure we would be running the backend for our game on. We decided on Terraform, an open-source infrastructure as code software tool that enables you to safely and predictably create, change, and improve infrastructure through a declarative configuration language.

We chose this tool, as it is widely adopted and meant that each of the people developing on the project would have the same infrastructure setup as each other, our final end users could easily spin up the same infrastructure themselves using tools they were already likely familiar with.

Tips and Tricks

Make liberal use of Terraform variables throughout your code! Thanks to our set of configurable variables, end users could choose to spin up a full global configuration of the project, or aim for a much smaller, and cheaper to run, configuration, possibly for development purposes.

GKE Autopilot and Standard

GKE Autopilot is a fully managed Kubernetes experience where you only pay for the Pods you run, and don’t have to worry about the underlying infrastructure that hosts it. It was a perfect fit for all our services, such as custom ones like our profile and front end service as well as the Open Match platform services (more on that later). The truly delightful thing about developing on Autopilot was that as a team we never had to worry about new services that were being built and if there was capacity on the GKE cluster that they were going to run on - we could just deploy knowing that Autopilot would take care of the infrastructure.

We actually host our game server containers on both GKE Standard and Autopilot in this demo. While you probably would want to settle on a single platform in production, we wanted to highlight that if the ease of use of Autopilot fits your requirements for your game server workloads then it works perfectly with this entire stack, but if your game server requires very specific tuning of the underlying GKE standard nodes - then GKE standard is also a great solution.

Tips and tricks

Start with Autopilot and only move over to GKE standard if you need to. If you don’t have a reason to move to GKE standard, the ease of use of Autopilot can save you a lot of time. Since it’s all Kubernetes underneath, migration between platforms is very straightforward.

Agones

Agones is an open-source game server orchestration platform originally co-created by Ubisoft and Google Cloud, but now with over 200 contributors from a variety of game studios. Built recognizing that Kubernetes needed game-specific optimizations, it teaches Kubernetes about the lifecycle of game server workloads, and ensures that player experiences aren’t interrupted while players are playing on them. Therefore our decision to use Agones was pretty straightforward as this meant we didn’t have to worry about game server lifecycle management or infrastructure usage optimisation - Agones would take care of all that for us.

Agones also provides us with a set of APIs and SDKs that make it easy to integrate our demo with the platform. This includes features like game state management and player management, which significantly decreases the time needed to implement these features from scratch.

Tips and tricks

Take advantage of the Terraform modules Agones comes with for you to use to create both GKE standard and autopilot clusters. You could create your own from the cluster creation instructions - but why go to all that effort?

Open Match

Open Match is another open-source tool for multiplayer games that was originally developed in collaboration between Google Cloud and Unity Technologies. It is a matchmaking framework that helps game developers build scalable and extensible matchmaking systems. It provides a set of tools and APIs that game developers can use to create their own matchmaking logic, while Open Match handles the underlying infrastructure to support a global player base.

We primarily chose this platform because it allowed us to have full customization over our match making logic, but also provided us with the ability to scale to whatever traffic we want to drive this multiplayer game demo to handle.

Tips and Tricks

While Open Match bundles a version of Redis hosted within the same Kubernetes cluster it is deployed on, take advantage of MemoryStore for Redis for a managed Redis offering on Google Cloud that includes high availability, failover, patching, and monitoring.
We recommend hosting Open Match on GKE Autopilot, since you only pay for the Pods you are running, and no longer have to concern yourself with node infrastructure - making Open Match much easier to manage and maintain.

Unreal Engine 5

Unreal Engine is a great choice for building multiplayer games, as it has a long history of use within the industry, provides fantastic networking tools and also is used by many developers in production for their multiplayer games! So since we were looking to match production workloads with this sample, it seemed like an easy decision.

One of the many benefits of using Unreal Engine 5 is its seamless integration with Agones, which can be achieved both via C++ and Blueprints. With the Agones SDK integrated into Unreal Engine 5, developers can easily manage the game server’s lifecycle, configuration, metadata and player tracking. You can download the source and build the plugin directly from here: https://github.com/googleforgames/agones/tree/release-1.30.0/sdks/unreal/Agones

Tips and tricks

To query the latency times to each region we are hosting game servers in, our Unreal Engine 5 based client also needs to send some UDP packets to Agones’ Latency Testing Services.. This can be a little tricky to find the documentation for, but to achieve that, we can use Unreal's FUDPPing API, to send an ICMP Ping packet over UDP, like so: FUDPPing::UDPEcho()

The Unreal Engine container ecosystem for interacting with the engine in the cloud - particularly building container images of dedicated game servers, is really well implemented and has a very smooth developer experience - strongly recommend taking it for a spin if you haven’t!

Cloud Build and Cloud Deploy

Cloud Build and Cloud Deploy are two managed Google Cloud Platform products that can be used together to automate the Continuous Integration and Continuous Deployment of applications that also have strong integration with GKE standard and Autopilot. For the Global Scale Game Demo, we are using them for both creating the artifacts and images for the backend services as well for the Unreal Engine 5 dedicated game servers, using the aforementioned Unreal Engine container system.

Cloud Deploy was also particularly chosen because of its ability to arbitrarily group multiple clusters together as deployment targets. This means we can group our clusters per region, and do regional rollouts of our clusters.

Each Regional Target contains Multiple GKE Clusters Behind it

This means we can do test deployments in smaller population areas, such as Australia, and assuming all things go to plan, move out to large and large population sets.

Tips and Tricks

If the generated YAML that is being applied to your cluster through Cloud Deploy results in more than 262144 bytes, use the --server-side kubectl flag, and you may also want to combine it with --force-conflicts flag to handle any Kubernetes server side apply ownership issues.
If you have a Deployment that may fail several times before coming to a healthy state, take advantage of the tolerateFailuresUntilDeadline feature of Cloud Deploy, which allows the system to tolerate failures up until the specified deadline. This is particularly useful for Istio and/or Anthos Service Mesh, which is covered in later sections.

Artifact Registry

The artifacts that are built and deployed by Cloud Build and Cloud Deploy are stored in an Artifact Registry container repository, which gives us the added benefit of faster spin up times for our backend service and dedicated game server images via image streaming, as well as vulnerability scanning and more.

Anthos Service Mesh

To build a full global scale game we needed a way to communicate between our set of GKE clusters that are distributed around the world. This included our backend services clusters as well as our Agones clusters. This is particularly important to our Open Match based matchmaker, as it needs to route requests to allocate game servers from Agones powered GKE clusters in specific regions around the world.

Thankfully, Google Cloud comes with Anthos Service Mesh, a managed Service Mesh offering based on the open source project Istio. This provides us both with DNS based service discovery within clusters as well as the ability to facilitate multi-cluster communications - which made it a perfect fit for our needs.

The ability to use Istio’s VirtualService was key, as it allowed us to have a single DNS entry for all Agones Allocation REST endpoints in our backend service cluster, and provide routing to specific regions using a custom region HTTP header in the request, while also managing access control and security through Workload Identity. This solved a lot of problems for us in one go!

Tips and Tricks

You may need to use Cloud Build to do some Anthos Fleet Management via gcloud, since some operations are not yet supported via Terraform.
If the expected multicluster Service DNS entry is not found within the cluster that needs it, make sure to ensure the namespace and matching Service are also created in that cluster as well. It doesn't need the implementation, but it does need both the matching namespace and Service to create the DNS entry.

Cloud Spanner

The classic problem with game launches is that you are never sure how much traffic you are going to get. At the same time launches are critically important because if they go badly, it’s extremely hard to recover your player base from the peak you would receive at launch.

To solve those very problems, we chose Spanner for our data storage platform. Since Spanner has both high availability, with a 99.999% uptime SLA for Multi-Region instances, horizontally scalability to match any demand a launch can throw at it, and also strong consistency globally, so you always know that players are seeing accurate in-game information.

Tips and Tricks

It is important to place the GKE clusters as geographically close to Spanner as defined by the instance configuration to reduce request latencies for writes and consistent reads. In this example, we put Spanner in us-central1 because the backend services are also in us-central1.
Liquibase is a great tool for managing your Spanner schema changes and updates, through a yaml configuration file.
Configuring Workload Identity to enable your application connectivity to Spanner has multiple steps, and can be tricky to get right. To solve this we generated the Cloud Deploy yaml files from the Terraform created IAM configuration to ensure both our infrastructure setup and our continuous deployment configuration aligned without manual intervention.

Conclusion

Choosing a set of products to build a multiplayer game out of the myriad of open source tools and products can be overwhelming and time consuming. The aim of the Global Scale Multiplayer Demo is to provide some guidelines, architecture patterns and building blocks that you can use to build your next global scale, production multiplayer game!

We’re excited for the future of the Global Scale Multiplayer Demo. We have big plans for the future to continue to grow and extend this project to show you all the ways you can use Google Cloud to host and scale your game.

If you want to get involved or just have a look at the project:

How Glance is collaborating with Google to build a next-level recommendation engine for its gaming platform, Nostra

Mon, 15 May 2023 16:00:00 +0000

Mobile gaming has exploded in popularity in recent years, with billions of people around the world playing games on their smartphones and tablets. Glance, a subsidiary of InMobi, is one of the world’s largest lock screen platforms with over 220 million active users. Redefining the way people use their mobile phone lock screen, over 400 million smartphones now come enabled with Glance’s next-generation internet experience. Glance aims to increase user adoption of Nostra, their mobile gaming platform, by providing personalized game recommendations to users. Nostra currently has a few hundred games, with over 75 million monthly active users and a rapidly-growing user base.

Glance and Nostra had an urgent need to recommend and personalize the games displayed on the user's lock screen, in order to improve the overall user experience. They turned to Google Cloud to build recommendation systems to personalize their gaming content according to the user's preferences.

Let’s take a deep dive into the recommendation system and the various Google AI technologies that were implemented to build the solution.

The historical dataset

Historical interactions on Glance that users had voluntarily provided, compliant with privacy guidelines, produced the following datasets:

User metadata: Information on the user, such as the unique user ID, the handset details, and the manufacturer details.

Game metadata: Information regarding the games, such as their name, version, description, and unique game ID. The dataset also contained secondary information, such as which regions the game was released in and the game’s category and subcategory.

User-Game interaction data: Interaction data, i.e. data capturing the unique user ID, the unique game ID, the interaction start time, the duration of interaction, and the game version.

These were the primary data sources used further for analysis, experimentation, and ultimately modeling. These datasets were partitioned by time, and in the spirit of experimentation, we considered a data chunk of 30 days for the solution approach.

Identifying data trends

We observed the plots of session duration vs. the number of events. From those, we decided to remove user interaction events that were less than two seconds long, as these events were likely unintentional or accidental.

Another interesting trend we observed was that the majority of the sessions occurred at night, followed by events that occurred after midnight.

We also observed that some games had a relatively higher median session interaction duration compared to other games, which was expected.

Dealing with the outliers

For the given interaction dataset, the observed value of the session duration varied across the board. Consequently, we observed outliers on both ends of the session duration spectrum.

As mentioned earlier, the minimum possible session duration was capped at two seconds, and any lower values were discarded. Through mathematical analysis, we also affixed a certain maximum session duration to be the representative value for a single session interaction; any value higher than this was considered spurious.

Splitting the data

We used 30 days' worth of data, but to fairly and accurately access the recommendation system, we needed to fix the data split methodology.

We considered several approaches, including the following:

Random split in 80/10/10 fashion. This might lead the model to preemptively learn trends from the future and ultimately cause data leakage.
Split stratified based on unique games/users. This would lead to the cold-start problem with a collaborative filtering approach.
Split based on a “last one out” strategy, which would imply we consider only (N-1) games with which the user interacted, keeping the last game for the test set. This would cause data leakage across the temporal dimension.
Data split stratified over N weeks, as we did explore some weekly trends. Hence, the first N weeks can be considered as a training set, and subsequent sets can be for testing and validation. This split type is also known as global-temporal-split.

Since the number of games was less than the number of users, we considered all the games across sets, i.e., all N games should be considered that have at least *one* user interaction per each set train/test/validation (or at least three user interactions in total).

We chose the fourth approach, global temporal split, where we first ordered the data chronologically, then defined the split interval and created the train, test, and evaluation splits. This way, we could avoid temporal data leakage and ensure that the trends were being captured properly across all users.

Preprocessing the data

The next logical step was to scale the values. However, due to the non-normal nature of the distribution, we could not perform any normalization, so we thus went with the approach of median centering over the aggregated duration.

The aggregated duration was calculated by summing the total interaction time between a unique user-game pair across all sessions, per split set, so each set had train, test, and evaluation.

In this approach, we selected the median sum of the duration of the training distribution and used it within a formula to scale the values down. Scaling the values ensured that the model we trained would be able to converge swiftly; it also ensured that all values that were lesser than the median were transformed as negative numbers and that all values above the median were transformed as positive numbers.

Experimentation and modeling approaches

We experimented deep and wide to find the best-fit solution for this use case. The experiments that we conducted included methods such as matrix factorization, two tower modeling through Vertex AI and TFRS implementation, and reinforcement learning.

Matrix factorization through BQML - Matrix factorization is a class of collaborative filtering algorithms used in recommender systems. Matrix factorization algorithms work by decomposing the user-item interaction matrix into the product of two lower-dimensionality rectangular matrices. BQML offers matrix factorization out of the box. This model only requires us to write SQL scripts to create a model and start training the same. BQML implements the WALS algorithm to decompose the user-item interaction matrix.

This was the baseline model that was created and trained, and the feature we considered was the aggregate session duration for each user-game interaction.

This model performed quite well, and after hyper-parameter tuning, it was the model with the best scores for test and evaluation sets. The following diagram describes the flow of data in the BigQuery pipeline:

Two Tower Embedding Algorithm with Vertex AI matching engine - This is an embedding-based approach to build the recommendation system. The two-tower approach is a ranking model that uses a query tower and a candidate tower to generate embeddings of the query and its related candidate passed on to their respective towers. The generated embeddings are then represented in a vector embedding space.

Embedding-based models use the user and the item context to create a well-informed embedding in a shared vector space. In our experimentation approach, we considered numerous combinations of user features and game features to construct the embeddings. The best results were observed when the game embeddings included the category and the user embeddings included the temporal features, such as the day of the week and the time of the day.

After these embeddings were created, we created a Vertex AI Matching Engine Index. Vertex AI Matching Engine provides the industry's leading high-scale low latency vector database (a.k.a, vector similarity-matching or approximate nearest neighbor service). More specifically, given the query item, Matching Engine finds the most semantically similar items from a large corpus of candidate items.

Utilizing this, we created indexes for Bruteforce search and Approximate Nearest Neighbour Search. Then, we performed a series of experimentations with various distance metrics, including dot product, cosine similarity, etc. The results from these experimentations concluded that we may require additional modeling and additional contextual data to make the embedding model perform better.

Custom modeling with TFRS - TensorFlow Recommenders (TFRS) is a library for building recommender system models. It is built on top of TensorFlow and streamlines the process of building, training, and deploying recommender systems.

The TFRS retrieval model built for this approach was composed of two sub-models viz, a query model (UserModel) computing the query representation using query features, and a candidate model (GameModel) computing the candidate representation using the candidate features. The outputs of the two models were used to give a query-candidate affinity score, with higher scores expressing a better match between the candidate and the query. Similar to the VertexAI Two Tower approach, we conducted multiple experiments defining various user features and game features to create the embeddings, and created Indexes on Vertex AI Matching Engine.

Best results and generalization was observed when only the user features were considered and the no game feature was considered, so only the unique game ID was used as a reference to embed the unique games in the vector space. The index used for the analysis was a Brute Force one to recommend top K games. We used Brute Force as the search space was limited.

Reinforcement learning through contextual bandits - We implemented a multi-armed contextual bandit algorithm implemented through tf-agents. Multi-Armed Bandit (MAB) is a machine learning framework in which an agent has to select actions (arms) to maximize its cumulative reward in the long term. In each round, the agent receives some information about the current state (context), and then it chooses an action based on this information and the experience gathered in previous rounds. At the end of each round, the agent receives the reward associated with the chosen action.

In this approach, we utilize the Multi-Armed Contextual Bandit (MACB) modeling, with per-arm features, i.e. the learning agent has the option to choose games from one of the "arms" having their respective features. This helps in generalization as the user and the game context are embedded through a neural network.

In the methodology, the context was derived from the BQML matrix factorization model. We used an epsilon Greedy agent with a prespecified exploration factor, which ensured that the model is capable of learning trends as well as exploring beyond.

This model gave surprisingly good results, considering that it was only trained on a subset of 1 million unique users. The model gained exceptional generalizability from the limited training samples and was able to understand and map out the context space for a total of 19 million unique users.

This model was one of the top performers throughout this engagement, wherein it aimed at optimizing model metrics like the Sub-Optimal Arms Metrics and the Regret Metric. For experimentation of MACB we followed the following approach:

Experimentation metrics

In all of these experimentations we calculated four metrics:

Mean Average Precision at K (MAP)
Normalized Discounted Cumulative Gain at K (NDCG)
Average Rank
Mean Squared Error

The "K" here represents the number of results to be recommended. We experimented with values of K being 5, 10 and All.

These metrics were used to rank models across different modeling methodologies. However, out of these four metrics, we mainly focused on comparing the MAP and NDCG values, as they do a better job of representing the problem statement at hand.

Results and observations

With our best model, we observed a significant performance improvement compared to the baseline model used previously. The BQML Matrix Factorization model was the best performer, followed by the Contextual Bandits model, VertexAI Two Tower, and finally the TFRS approach. The embedding-based models suffered because of the lack of certain user demographic features in the given dataset. We also observed that as the K value increased, the model performance also increased, which is expected behavior.

Explaining the productionization and scaling approach of these models, the Nostra team initially deployed new models on a small percentage of traffic, comparing engagement metrics with control traffic in an A/B testing fashion. This deployment was tuned to ensure low latency output and to gain a deeper understanding of the key drivers behind user engagement. Once the model showed good performance, Nostra was positioned for further cost optimisations if required, and to scale the model to a larger percentage of traffic.

After this engagement, the Nostra team decided to implement the BQML Matrix Factorization model in production, after which they observed a preliminary increase in the time spent by the users interacting with the games on their platform.

Fast track end-to-end deployment with Google Cloud AI Services (AIS)

The partnership between Google Cloud and Glance is just one of the latest examples of how we’re providing AI-powered solutions to solve complex problems to help organizations drive the desired outcomes. To learn more about Google Cloud’s AI services, visit our AI & ML Products page.

Paul Duff, Consultant - Data Sciences with Nostra emphasized how happy he is with Cloud AIS and out-of-the-box products like BQML that helped get to productionable models in just a few weeks, stating, “The Google team was able to exceed our agreed objective to develop an offline recommendation system for Nostra, completing all milestones on or before time and delivering four fully documented working models. We were able to deploy the first of these in production (beyond the agreed project scope) and saw an uplift in our key metrics before the project end date. The organization of the team was exemplary with excellent communication throughout.”

^{We’d like to give special thanks to Naveen Poosarla and Charu Shelar for their support and guidance throughout the project. We are grateful to the Nostra team (Gaurav Konar, Rohit Anand and Rohith Kalyan) and our partner team from Wipro and Capgemini (Tulluru Durga Pravallika and Viral Gorecha) who partnered with us in delivering this successful project. And a special shout out to our super awesome leader Nitin Aggarwal.}

^Disclaimer:^{The information presented in this blog is intended to provide insights into a specific subset of data related to product usage metrics. It is important to note that the findings presented here may not necessarily reflect the overall product usage metrics.}

Building multiplayer Google Doodle games with GKE, Open Match and Agones

Mon, 03 Apr 2023 18:00:00 +0000

Introduction

As anyone who visits the Google Search homepage may realise, there is often a surprising, creative and delightful change in our logo that sits above the Search bar. They’re called “Google Doodles.”

In recent years, we’ve seen them not only be interactive, but also multiplayer! Letting anyone who may be browsing the Google homepage play a game with one or more other people also browsing the same page. But the question begets itself -- what technologies are they using behind the scenes to power these games that are serving this global audience of Google users? And are those technologies available to the reader to build similar experiences?

We’re joined by Jacob Howcroft, who has been a software engineer on the Google Doodle team for the past seven years, and has been leading efforts into the journey the Google Doodle team has been going on into multiplayer Google Doodle experiences.

What is a Google Doodle?

Jacob: A Google “Doodle” is a spontaneous change to the Google logo that celebrates diverse holidays, people, places, and things that have impacted and shaped local and global culture. The original Doodles were static illustrations, but now we also create games and other interactive experiences.

What is a multiplayer Google Doodle?

Jacob: It’s an interactive experience that connects players together, so that they can play with their friends or other random users online. Anyone who visits google.com in a browser or mobile device sees the Google logo transformed with a “play” button, and clicking it takes them to a multiplayer game.

Google Multiplayer Doodle for Halloween 2022

What multiplayer Google Doodles did you recently launch? How did they go?

Jacob: Our first multiplayer Doodle was The Great Ghoul Duel, for Halloween 2018. Since it was one of our most popular Doodles ever, we launched The Great Ghoul Duel 2 on Halloween in 2022. We also launched Celebrating Pétanque, a multiplayer game celebrating the French sport, in July 2022.

The Great Ghoul Duel 2 (2022)

Celebrating Pétanque

Millions of people played the Petanque and Halloween Doodles, and we also saw that a lot of them played multiple rounds, especially for Halloween. "The Great Ghoul Duel" has a fan community on Twitter and Discord that has remained active since 2018, and of course they were really happy to see the sequel.

What are the challenges of launching a multiplayer game for the Google Homepage?

Jacob: The biggest challenge is getting ready for scale. Every day, there are billions of searches on Google, and we’re making a game available to those users, so we have to handle a huge number of players right at launch. Additionally, these games launch all over the world in a matter of hours, so we have to scale up really fast. Since the Doodle is a surprise, we don’t have an opportunity to test in a beta release. This means we need to perform very rigorous QA, load testing, and internal playtesting.

How many players do you get on a multiplayer Google Doodle?

Jacob: When a Doodle is live on our homepage, our users play many millions of times. After the Doodle’s run on the homepage, we regularly see thousands of games played each day on our google.com/doodles archive.

How does a browser based game differ from other types of multiplayer games?

Jacob: The main difference for browser-based games, written in JavaScript or TypeScript, is that they use the WebSocket protocol, whereas mobile, console, or PC games would typically use UDP. The challenge this creates is the requirement for each GameServer to have a certificate for HTTPS / Web Socket Secure (WSS). Since the game runs on an https page, this means that all the WebSocket connections need to be secure as well. Doing this also requires DNS addresses for the game server, as opposed to connecting directly to IP and port, as they would for a more traditional UDP multiplayer game connection.

Can you tell us how you built these multiplayer Doodle games?

We used a bunch of Google Cloud products and took advantage of freely available game infrastructure and distributed systems open source projects. For the game itself, we use many TypeScript tools and libraries that we’ve created over the years for Doodles, as well as some open source engines like PIXI.

What Google Cloud Products did you use?

Jacob: Google Kubernetes Engine (GKE) runs most of everything.

We utilise Ingress for External HTTP(S) Load Balancing to handle global load balancing for the matchmaker and for the game server clusters.

Redis Memorystore is used to configure routing for game servers, and to store matchmaking tickets for our matchmaker.

Google Managed Certificates allows for the automatic creation and rotation of SSL certificates for the matchmakers and game servers. This was really helpful, since it turned certificate handling into zero work.

Cloud Logging, Cloud Monitoring and Logs-based Metrics allowed us to have a set of dashboards to monitor production, looking at metrics like game server status, number of matchmaking tickets, etc.

Cloud Run and Cloud Endpoints power the multicluster game server selection system, which chooses appropriate game server instances for our players from a set of GKE clusters and marks those game servers as “Allocated” - basically that they have players on them, and therefore shouldn’t be interrupted. We can go deeper into this later, but the multicluster allocator system helped a lot for scaling.

Why did you choose to use open source projects for the backend?

Jacob: We are mostly a frontend team - we build content for the web, and the frontend servers. When it comes to deeper backend tech like Google Cloud and Kubernetes, it isn’t our expertise. But the open source systems available bridge this skills gap, and let us work outside our comfort zone to create these more complex experiences.

Which open source projects did you take advantage of?

Jacob: Our first multiplayer Doodle back in 2018 took advantage of Open Match to matchmake our players together to play a game, and it worked fantastically, so we have continued to use that in our newer multiplayer game stack.

Agones is installed on our GKE clusters to host and scale our game servers and make sure they run uninterrupted while players are playing the game. Agones provides us with the GameServer and Fleet custom resource definitions we can use to declare the types and number of game server processes we are running in our GKE clusters. It also gives us the ability that we just talked about to select and allocate game servers, and mark them ready for player connection. We run multiple GKE and Agones clusters around the world to ensure that we have game servers near all our potential players (which is all of the users of Google.com!).

Also, not only were these terrific solutions, but since Google Cloud founded both Open Match and Agones and continues to maintain these projects with help from their respective communities, we knew we were in good hands and had people we could ask for help (and we did!).

We also use Traefik proxy to coordinate the websocket connections for matchmade players to specific Agones GameServer Pods hosted on our GKE clusters, so that players can play their games together.

For development and operational tools, we use Terraform for provisioning our infrastructure, and Helm and Helmfile for coordinating deployments of the open source tooling and our custom components into all of our GKE clusters.

How much custom code did you end up writing?

Jacob: Not as much as you would think!

Open Match requires you to write several custom components for match functions and integration with external systems like Agones, but thanks to the interoperability of gRPC, we could write all of them in Node.js even though Open Match is written in Go. We wrote our websocket based game servers in Node.js as well. This had the advantage of allowing us to share code between the client and server and speed up our development time.

We also wrote a custom Kubernetes controller to configure Traefik Routes. Traefik can be set up to detect configurations from Redis by writing to certain key formats. Our controller watched for Agones GameServer events, and wrote the name, pod IP, and port to Memorystore when one occurred. Traefik dynamically updates its routing so that we can access the GameServer using wss://<project subdomain>.cloud.doodles.goog/<game server name>.

Can you step us through your architecture for matchmaking and assigning a game server to a group of players?

High level architecture of a multiplayer Google Doodle

Jacob: First, players connect to a global Multi Cluster Ingress HTTP(S) load balancer, which routes players to their closest match maker frontend hosted in globally distributed GKE clusters. We usually have a handful of these clusters, with one or two per continent.

Through the matchmaker frontend, a Ticket is created in Open Match, which is matched with other players.

When the Agones Fleet starts new GameServers, our custom Agones-Traefik controller writes their Pod IPs to a Memorystore Redis instance. Traefik Proxy reads from this instance to set up routes in the cluster as we mentioned before - this way, players will connect to GameServer “foo” by url, i.e. “wss://cluster.game.cloud.doodles.goog/foo”.

The Open Match Director component, which we wrote in NodeJS, receives a Match and uses the Cloud Run-powered Agones multicluster Allocation service to allocate a GameServer. Each Matchmaker cluster allocates from one or more GameServer clusters.

When doing our load tests we noticed that the optimal GameServer Fleet (a grouping of GameServers) size is mostly limited by the Kubernetes control plane. So the allocation rate, amount of game server activity, plus your control plane size determines the size you can handle. Our game sessions are usually pretty short, and all that turnover creates a lot of load on the Kubernetes API.

We originally had an instance of our Open Match matchmaker and Agones GameServer Fleet in each cluster. However, we found that there was an optimal size for our GameServer clusters, around 6,000 Agones GameServers, so a one-to-many architecture for Matchmaker and GameServer clusters allowed us to scale higher, since more clusters equaled more overall throughput.

Once a GameServer allocation succeeds, the matchmaker replies to the players with the url of the GameServer. Players connect to it through a separate Ingress, and Traefik Proxy routes the request to the appropriate GameServer.

At the end of the game session, we don’t shutdown the game server, instead, we reset the game server and move it back to the “Ready” state through the Agones SDK, so it can be allocated for a new game. This saves us some load on the Kubernetes API and the Traefik proxy, by reducing the GameServer turnover.

We usually advocate for a single instance of Open Match to support a global player base - why have an instance in each region?

Jacob: While we know that Open Match can handle a global player base, we used multiple Open Match instances in different regions, so that we could rely on the Load Balancing to route traffic to the nearest Matchmaker. Since we had such a massive pool of players, we didn’t need to worry about sharding our player base - there were plenty for each Matchmaker.

Since we didn’t have the option of beta testing before launch, having more instances of Open Match around the world reduced launch day risks and allowed us to horizontally scale Open Match instances and the Agones clusters behind them if we had to.

Can you go into more detail on the websocket based communications? That sounds quite tricky.

Websocket Connections to Agones GameServer Pods

Jacob: WebSocket games require Web Socket Secure (WSS), which means that every GameServer needs to be addressed by URL and have an SSL certificate. Managing this for the dynamic Fleets of thousands of GameServers required a novel solution.

For the Halloween 2018 Doodle, we had statically generated thousands of DNS entries routing to GameServers by subdomain. However, this approach stopped us from taking advantage of all the dynamic, scaleable features of Agones and GKE.

So for our new setup, we found Traefik Proxy. We chose this one because routes can be configured dynamically with various providers, one of them being Redis. So, we wrote a Kubernetes controller that watches GameServers, and writes the configurations to Redis. We were concerned about how a proxy would affect players’ ping, but with some load testing, we found that it didn’t have a noticeable impact. This setup allowed us to dynamically scale the Fleet and have the new GameServers available for connection as soon as they could spin up without having to manipulate DNS entries.

We also wrote a WebSocket library, using Google Closure WebSockets on the client and ws on the servers. For each game, we define an API using protocol buffers - a client-to-server message and a server-to-client message. Usually, this message contains a oneof field for various sub message types for different actions within the game. The library uses TypeScript generics to make sure that the game code sends and receives messages using this API.

Ensuring a game is fun is one the most important aspects of any game - how did you test for that?

Jacob: We test for fun in a few ways. The first one is a weekly team playtest; the whole Doodle team gathers together to play the latest demos and share feedback and ideas. The second way is internal user tests, or “Cafe tests”. We find some random Googlers, or set up a stand in a micro-kitchen, and get their feedback in real time. Googlers are usually pretty honest, so we get some good results this way. Finally, we will set up playtests with external users from a variety of backgrounds and tech familiarities, which provides very valuable and diverse perspectives that inform the final product.

Given you are on the front page of Google, how do you load test for that kind of traffic?

Jacob: The first thing, from an early stage, is to make a headless game client. This means that the game can be played by a script, without interacting with any UI. Then, we wrote a Node-based load testing client that simulated a user.

Using Kubernetes Jobs, we would deploy thousands of these fake users to GKE, and point them at a cluster. We start with a small number of users, and if the cluster handles them properly, then we step it up, and repeat this process until something breaks down.

Then, we would debug and fix the issue, and start over. Using this process, we were able to find the optimal Fleet size for our Agones clusters, which was about 6,000 GameServers. Note that this number depends on your Kubernetes provider, the size of your matches, the duration of your games, and other factors, so it isn’t a one-size-fits-all.

How did you estimate the amount of traffic to load test for?

Jacob: We looked at metrics from our past games, and used that to estimate the potential traffic for this one. However, it can vary a ton depending on a lot of different factors, so we have to aim as high as we can. Even still, we ended up deploying additional clusters during launch because the peak traffic was so high.

When you were ready to deploy to production, how did you do this?

Once we had determined the approximate scale via load testing and analysing past Doodle performance, we would define the entire Fleet in Terraform and Helmfile, and deploy it. The Doodle begins rolling out based on local time around the world, so we plan for traffic to increase in large steps over about 24 hours.

Did you autoscale any of your resources, or just keep a static infrastructure?

Jacob: We enabled node autoscaling on our GKE node pools, except for some cases where we wanted to avoid scale-down, to make operational maintenance easier. Since we had long-running WebSocket connections, we had to reduce the amount of pods being moved around as nodes turned over. We generally didn’t use autoscaling for our Kubernetes Deployments, because we knew we would be receiving a lot of traffic in a short time; we preferred to pre-scale to high levels and then manually adjust throughout the short launch window. We also didn’t use Agones Fleet autoscaling, as we ran at 100% capacity for the entire game launch, however, we took advantage of being able to easily resize the Agones Fleets by running: kubectl scale fleet halloween --replicas=N, and since we use node autoscaling, the cluster would automatically resize to fit the Fleet.

How many GKE clusters did you have, and how did you manage them?

Jacob: For the most recent game (Halloween 2022), we had 30 clusters. On average, we had 3-5 GameServer clusters per matchmaker cluster. Using Terraform and Helm and taking advantage of their loop constructs was crucial to manage all these clusters. It made configs that could have been hundreds of lines into a few dozen.

You mentioned using Cloud Monitoring, what did you actively monitor?

Jacob: We monitored a variety of metrics across the system, but to highlight some of the most important ones:

We had dashboards for Agones Fleet status showing how many GameServers were in each state, especially Ready vs Allocated. This let us know when a Fleet was reaching its capacity. Since the launch period is only about 48 hours, we didn’t set up alerts and just stayed on call throughout. However, we set up some alerts based on error rates for when the game runs on the google.com/doodles archive.

We also monitored game server and matchmaker connections. Especially for the matchmaker, we monitored the rate of successful matches made. This was important as it was our number one metric of the user experience - failing to find a match was the least fun thing that could happen for the user. Another important metric is how long users spend waiting for enough players to join a match. Luckily we didn’t have to worry about this one, because we had so much traffic.

We tracked our own custom “Game Over” events. Looking at the running total of games completed was really exciting! We did this by creating a chart summing our custom log-based metrics for things such as matchmaker connections, successes and failures in the open match components we wrote, game server connections, etc.

And finally, we utilised all the built-in metrics from Agones and Open Match to give us an overview of the entire platform health.

6 Google Doodle games.jpg

Cloud Monitoring Chart: Matchmaker Connections
7 Google Doodle games.jpg

Cloud Monitoring Chart: Game Server Connections
8 Google Doodle games.jpg

Cloud Monitoring Chart: Total Game Servers

What would you have done differently if you had to do it again?

Jacob: One lesson we learned was that we should have obtained more than double the amount of quota to handle our expected traffic. The system was designed so that it was really easy to bring new clusters online during launch, which we made use of and was really helpful in scaling up at the last minute. However, we didn’t have enough quota in some regions to do a full blue-green deployment (running an entire deployment of a new version before switching over from an older version). That meant we had to incur some downtime when making infrastructure changes during launch. This can be avoided by having plenty of headroom in your quotas. So our recommendation is to work with your Google Cloud customer account representatives to ensure that you have a full launch review with quota requirements well ahead of your launch date.

Another thing that was difficult was getting the entire matchmaking and game server orchestration systems to spillover gracefully to another region if they had no more Agones GameServers available in their region. This meant that during launch, we had to make sure each Open Match instance was backed by enough GameServers that we wouldn’t run out, given the peak load we expected and encountered. We avoided the cascading failures during launch, but the tradeoff was having to manually scale up the busiest clusters.

To explain, users connect to the matchmaking frontend and request to join a match. They wait until there’s enough players, and then an allocation request is sent to Agones. But if all Agones clusters are full, it will reply with an error. At this point, there might be thousands of players attempting to join a match already, and then all of those matchmaking requests will fail, since there is nowhere for them to go (remember, we run at 100% capacity). Additionally, new users are still connecting to the matchmaking system, and all those requests will fail as well, leading to cascading failure for the matchmaking cluster. Ideally, the matchmaking system would stop accepting new match making requests early if it knows that its Fleet is above a certain Allocated threshold.

To better handle this in the future, we need the frontend to have up-to-date data on the Fleets behind each Open Match instance, so it can start telling clients to use a different instance before it becomes overloaded. If you wait until the Allocator Service reports that the Fleet is exhausted, you may already have a problem.

We are also considering switching to Multi Cluster Gateways, which would allow us to configure rate limits. Then we could just compute our max QPS for a cluster based on its maximum capacity, set the rate limit accordingly, and let GCP handle the spillover. It would be a less dynamic solution, but it would be very simple to implement.

Conclusion

Thank you to Jacob for such a detailed description of how the Google Doodle team develops, hosts, scales and orchestrates their multiplayer Google Doodle games.

The combination of Google Cloud - particularly Google Kubernetes Engine - and the open source Google Cloud Gaming solutions allowed the Google Doodle team to successfully run successive multiplayer Doodles at Google Search scale - a truly impressive feat!

It’s also pretty amazing to see that such a comprehensive multiplayer game platform could be built with relatively minor custom code, when utilising Open Source solutions for multiplayer games from Google Cloud.

If any of this sounds interesting and you want to learn more about any of the solutions listed above, you can go to:

And finally, if you want to play/see any of the over 5000 Google Doodles we’ve launched around the globe throughout the decades, you can do so at google.com/doodles!

Run your game infrastructure on GKE Autopilot to focus on player experience

Tue, 21 Mar 2023 16:00:00 +0000

Earlier this month, we shared Google Cloud's strategy for live service games which outlines our focus on the three critical pillars of (1) serving players everywhere, (2) collecting and organizing game data, and (3) unlocking player and game insights. Starting today, you can run your game workloads on Google Kubernetes Engine (GKE) Autopilot — a truly automated experience in running global multiplayer game servers. This blog explores how GKE Autopilot can help you serve players globally.

For players, games just have to work. Game developers must have underlying cloud infrastructure that is simple to operate and seamlessly scales with player traffic, so that game developers can focus on what truly matters: building great games. At Google Cloud, we are fixated on making game launches boring by making GKE Autopilot the platform-of-choice for running game workloads for scalability, reliability, and automation.

One of the most critical components of a multiplayer game is the dedicated game server — an authoritative source of game state to which players' clients connect to interact with the game world. These dedicated game servers have to stay up during a game session, handle many connections, and scale based on metrics such as concurrent players.

Example architecture of a global scale game

Now you can run dedicated game servers on GKE Autopilot using Agones, an open source game server orchestrator that we have been developing with the games industry. When running game servers on GKE Autopilot, Google Cloud manages the clusters for you, and you only pay for the CPU, memory, and storage that your game servers request while running. You don’t pay for unallocated nodes or system overhead, regardless of how your game servers are scheduled.

"We use Google Kubernetes Engine to run our dedicated games servers and we are sensitive to the operating costs of clusters. We are excited to run our game servers using GKE Autopilot with Agones because it eliminates the need to manage Kubernetes node pools helping us scale our game servers and our team without scaling our costs."—Hiroki Awata, Software Engineer, DeNA

Power-up: Game servers on GKE Autopilot

To get started with GKE Autopilot for games, first create a GKE Autopilot cluster, then create a firewall rule to allow UDP player traffic, and install Agones using Helm.

code_block: <ListValue: [StructValue([('code', 'gcloud container clusters create-auto agones-autopilot \\\r\n --region=us-central1 \\\r\n --autoprovisioning-network-tags=game-server'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe3889de190>)])]>

Since GKE Autopilot is Kubernetes, we can create a game server fleet with the same fleet configuration yaml that we apply to any Kubernetes cluster.

code_block: <ListValue: [StructValue([('code', 'apiVersion: "agones.dev/v1"\r\nkind: Fleet\r\nmetadata:\r\n name: simple-game-server\r\nspec:\r\n replicas: 5\r\n template:\r\n spec:\r\n ports:\r\n - name: default\r\n containerPort: 7777\r\n template:\r\n spec:\r\n containers:\r\n - name: simple-game-server\r\n image: us-docker.pkg.dev/agones-images/examples/simple-game-server:0.14'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe3889de9a0>)])]>

Notice how we didn’t have to think about the number of nodes, node pools, or cluster autoscaler configuration we would need to run our game servers: we simply defined the number of game servers we wanted to run. Wasn’t that easy?

For the win: Hands-off infrastructure

GKE Autopilot is perfect for running your dedicated game servers because it is a hands-off, fully-managed Kubernetes platform that manages your cluster’s underlying infrastructure without you needing to configure or monitor - a workload level SLA backed by Google SRE. By using Agones FleetAutoscaler with GKE Autopilot, your global game server fleets automatically and dynamically scale as players flow in and out of your game - you can set it and forget it!

code_block: <ListValue: [StructValue([('code', 'apiVersion: "autoscaling.agones.dev/v1"\r\nkind: FleetAutoscaler\r\nmetadata:\r\n name: simple-game-server-autoscaler\r\nspec:\r\n fleetName: simple-game-server\r\n policy:\r\n type: Buffer\r\n buffer:\r\n bufferSize: 2\r\n minReplicas: 5\r\n maxReplicas: 100'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe3889de6a0>)])]>

With traditional Kubernetes, this scaling requires resources and time for planning, right-sizing, and bin-packing. You might overprovision node pools much earlier in anticipation of scaling up and keep those node pools running longer before scaling down. All this costs money.

Scaling game servers with GKE Autopilot makes sense because game servers are pods and Autopilot speaks in units of pods. Autopilot ensures you pay only for your running game server pods and you don’t pay for system components, operating system overhead, or unallocated capacity. According to a Forrester study, using GKE Autopilot results in up to 85% reduction in infrastructure costs and 45% improvement in developer efficiency.

noclip On: Full control, full freedom

Fully managed doesn’t have to mean vendor lock-in. Some game companies on Google Cloud run their games on cluster fleets that span across regions, on-premises, or other cloud vendors. Since Agones is open source and GKE Autopilot is based on open source Kubernetes, when serving your games with GKE Autopilot you don’t have to sacrifice portability or flexibility. Better yet, Google Cloud continues to be the largest contributor to both the Agones and Kubernetes projects allowing you to leverage our expertise and help define the roadmap to serve the next generation of games.

Pro-move: Leverage the integrations with the best of Google Cloud

The technology needed for successfully running global multiplayer games extends beyond dedicated game servers and includes matchmaking services, custom network proxies, databases, and analytics. When running game servers on GKE, these integrations are easy. For example, you can use Open Match on GKE Autopilot to scale the matchmaker with your dedicated game servers or manage player data with Cloud Spanner.

Secret Mission: Continuous delivery with Google Cloud Deploy multi-target parallel deployment

Let’s say that you want to launch your game server with new features for an upcoming global game event, region-by-region. Cloud Deploy’s new multi-target deployment capability enables you to do just that. Create GKE clusters in the regions your players are close to, group them (as targets) into multi-targets, and deploy your Agones game servers in waves. Need to first validate in a staging environment? No problem, create a staging target at the start of your progression sequence. Need an approval pause button before deployment to production? Cloud Deploy supports that too, along with multi-target rollback.

A Cloud Deploy delivery pipeline progression sequence using multi-targets

Game on

We understand that every resource that you have to spend on managing and scaling infrastructure is one less resource spent on making great games. To get started with running game servers on GKE Autopilot, learn more.

Build a multiplayer game with Cloud Spanner

Tue, 21 Mar 2023 16:00:00 +0000

Fantasy sports games are live service games built on real-world data. The data is collected from various sources such as real-world sports leagues, statistics websites, and news outlets. A virtual world is built where players can draft teams and compete against each other at a global scale in real time.

These games are characterized by low latency requirements, huge fluctuations in workload, and the need for a zero downtime architecture. A successful game requires a robust database that is highly reliable, globally available and quick to respond to varying workloads.

Niveus is a Google Cloud partner that specializes in helping enterprises digitally transform workloads to both enhance their customer experience and differentiate themselves in the marketplace. The Niveus team recently evaluated Cloud Spanner and we built a demo fantasy sports game; read on to see how Spanner was used for this workload.

Why consider Cloud Spanner for real time multiplayer games?

Globally distributed, and scalable database: Spanner is a globally distributed database that can span across multiple regions. The Spanner architecture decouples compute from storage, which allows it to scale processing resources separately from storage. This allows for increased flexibility, reduced cost, improved performance, and increased reliability. The distributed nature makes it an ideal solution for global unpredictable workloads, like online games. When a game goes viral, Spanner's infinite scalability ensures that the game doesn't become a victim of its own success.
Transactional and strong consistency: Spanner supports ACID transactions so that millions of player actions are secure and consistent, resulting in a single source of truth for the game state. By default, Spanner guarantees "strong" reads that observe the effects of all transactions before the start of the operation, regardless of distribution or node failures. This is critical for games for storing game state and in-game transactions.
High availability: Spanner's 99.999% uptime with zero maintenance windows is a great match for gamers' high expectations around responsiveness and uptime.

Multiple games customers, like Niantic, Bandai Namco and Embark Games use Spanner as their backend games database because of Spanner’s differentiated capabilities. More details can be found on the Databases for Games page.

Fantasy Sports League (FSL) Game Implementation

Each matchup in the sports league starts with creating a fantasy team. Each gamer selects players from two competing teams in an upcoming cricket match. Points are awarded for different game activities. A live leaderboard is maintained throughout the game and the fantasy team with the most points at the end of the match is the winner.

A multiplayer game requires read/write operations for each activity a gamer performs along with storage of that data (id, username, password, players on the gamer’s fantasy team, their stats, etc.). In addition to basic gamer details, the system must also store backend game state and derived data like leaderboard snapshots. A viral multi-player game whose user count shoots up would see an explosion of compute and storage requirements; satisfying these requirements with a database like MySQL would require a developer to create, deploy, and maintain a sharded environment. This is tricky, complex, and expensive. In contrast, a Spanner database automatically shards data and distributes based on compute load, while also managing replication to provide the "database is never down" experience that assures a great experience to gamers.

The game uses Node JS for the application layer, Spanner as the backend game database, and Cloud Memorystore as a caching layer for storing leaderboard data.

While our code is focused on running the game on Spanner, one could extend this architecture to include the analytics piece that is critical to any game deployment. This will include Dataflow, BigQuery, and Looker for analytics on the game data collected. In this architecture, we use Spanner change streams to pull data into BigQuery. BigQuery is a scalable, highly reliable, and secure cloud-based data warehouse for storing and analyzing large datasets and Looker is a powerful business intelligence platform that helps you explore, share, and visualize your data.

High Level Architecture

aside_block: <ListValue: [StructValue([('title', 'Check out the source code for the game!'), ('body', <wagtail.rich_text.RichText object at 0x7fe38926ae80>), ('btn_text', 'See code'), ('href', 'https://github.com/google/fsl-gaming'), ('image', None)])]>

As you browse through the code you can see the microservices that we developed:

FSL-Backend-Common - Contains several methods and utilities used by other microservices.
FSL-MS-Resource-Management - Handles CRUD operation endpoints for tables like Team Details, Match Details, Player Details, Fantasy Team Details, etc.
FSL-MS-Simulator - Simulates the match.
FSL-MS-Update-Leaderboard - To calculate the score for each fantasy team and update the leaderboard in Memorystore.
FSL-MS-Display-Leaderboard - Displaying the leaderboard data for a contest.

Match Simulation

FSL-MS-Simulator microservices include APIs to simulate individual balls and matches. /simulateBall API is used to simulate a ball. Similarly, an entire match gets simulated using a simulate match API by calling /simulateMatch.

SimulateBall pushes a message to Pub/Sub topic simulatorListener and updateScore. UpdateScoreListener receives the data packet from Pub/Sub and calls the /updateScore API, which makes batches of a fixed length and pushes to updateRedis Pub/Sub topic to execute a huge amount of data in parallel.

Updating Scores

Score calculations are based on incrementing score for each team in Memorystore as the game progresses. Once the score is updated in Memorystore, it is updated in Spanner. To scale the writes, the data is pushed to Pub/Sub where all the subscribers listen to publish events and start writing to Spanner.

Storing Leaderboard data in Spanner

After the final ball, we update the leaderboard snapshots in Spanner, enabling anyone who visits the site after the match completion to see the final leaderboard for the match. Machine Learning models can be built on the stored data in the future for predicting user scores and potentially winners.

Leaderboard

Fantasy teams are listed on the leaderboards and are ranked based on the scores. The score gets calculated in Memorystore for each team. During a live match, all the writes happen to the cache and a live leaderboard is retrieved from Memorystore.

Post Simulation Results

Post-simulation, we store the top ten fantasy teams in the contestDetails and MatchResults tables in Spanner.

Conclusion

Spanner with its high availability, strong consistency, and distributed nature became a goto database for our multiplayer game development. Many gaming studios and game developers trust Cloud Spanner to provide a rock-solid backend that lets them focus on what they do best: building great games.

aside_block: <ListValue: [StructValue([('title', 'Review the ER diagrams for the game'), ('body', <wagtail.rich_text.RichText object at 0x7fe38926a160>), ('btn_text', 'Review diagrams'), ('href', 'https://github.com/google/fsl-gaming/blob/main/images/README.md'), ('image', None)])]>

Learn more about the Spanner free trial instances available at no cost to both existing and new Google Cloud customers.

If you want to learn more about uses of Spanner, and specifically how it helps in games:

Cloud Spanner as gaming server backend
Best practices for using Cloud Spanner as a gaming database

^{We would like to thank Aalok Muley, Group Product Manager, Databases (Google Cloud) for his help on this blog.}

Game on and on and on: Google Cloud's strategy for live service games

Wed, 08 Mar 2023 17:00:00 +0000

It's undeniable: Live service games (“live games”) are the future of the games industry. In 2022, the top 10 most played games were all live, continuously delighting players with new content, features, and gameplay. Today, the live service model is prevalent across all console, PC and mobile platforms and genres. And with a projected 3.6 billion players by 2024, the opportunity for live games has never been larger.

Google is no stranger to this model, operating some of the world’s largest live services like Google Search and YouTube to billions of users around the world. Likewise, Google Cloud is committed to bringing the best of Google to game developers who are delivering live game experiences to players everywhere.

We already work with most of the top-grossing game companies. In our journey with them, we’ve learned we need to help developers unite performance and understanding via three critical undertakings:

Serve players everywhere.
Collect and organize game data.
Unlock player and game insights.

Here’s how.

Serve players everywhere

Live games just have to work. Google Cloud helps developers connect players with game servers built on the world's largest, most secure network.

Take, for example, game launches. It's a hard problem because it requires developers to effectively bet on how successful their game will be before launch. "Will I attract a thousand players? A million? More? At the same time?" Ironically, the nightmare scenario is often extraordinary success: being unable to serve players who would have played your game had they been able.

Success shouldn't worry a game developer. To that end, one of the most important capabilities Google Cloud can provide is scale. Kubernetes is recognized as one of the best ways to handle computing at massive scale and no one knows it better than us — we founded the project and continue to be its most prolific contributor.

With our deep knowledge of Kubernetes, we built Google Kubernetes Engine (GKE), which gives game developers the capability and confidence to serve even the largest player audiences. It’s the most scalable and automated managed Kubernetes service, capable of running 15,000+ node clusters, outscaling other cloud providers by as much as 10x.

While GKE works for a range of compute tasks, we know workloads for live games are unique. That's why we partnered with Ubisoft to create Agones, an open-source game server orchestrator for Kubernetes. Using Agones with GKE enables developers to focus on player experience instead of infrastructure.

At Ubisoft we needed a centralized, standard solution for game servers. We partnered with Google Cloud to develop Agones as a result, maximizing players’ engagement and trust over time. Pierre Romagné
VP Engineering and Platform, Ubisoft

Collect and organize game data

Live games can't forget player progress. Google Cloud helps developers secure player experiences without having to worry about scale or performance.

Live service games are persistent. Players earn levels, add friends, buy skins. Even the smallest bits of data can be prized by players, down to every shot they made (“How accurate am I getting?”). Multiply this desire with the potential for a live game to have millions of concurrent players all over the world. The challenge seems daunting.

Leveraging our vast experience with our own live services, we created Cloud Spanner to fully address data scalability. Spanner provides a single, logical, horizontally scalable database that can process 2+ billion requests per second. This means game developers don’t have to compromise between performance, manageability, and scale (no more sharding!). They can have a single database that scales to handle the largest player audiences.

At any given time, we have about 5,000 Spanner nodes handling traffic… All of them work together to support millions of players playing all across the world at a given moment. James Prompanya
Sr. Engineering Manager, Pokémon GO, Niantic

Beyond scale, Spanner can also reduce costs (up to 50%) while improving player experience through latency reduction (up to 25%). That way, for example, when a player unlocks a new skin, they can equip it immediately.

Unlock player and game insights

Live games need to continuously evolve in the right direction. Google Cloud helps developers understand what's working to find and retain players with Google's analytics and AI.

There are over 3 billion players in the world, and they come in all shapes and sizes. To delight them, developers need to understand what works and segment players accordingly. Take, for example, Square Enix, a global games publisher. They’re using data and analytics to build a Single Gamer View in order to understand and engage their players, resulting in increased engagement and customer satisfaction.

BigQuery is a catalyst for generating insights and is a killer app among large game developers. It’s used by nearly all of Google Cloud’s top game companies. When it comes to data analytics BQ is OP, helping developers analyze and unite huge amounts of player and game data. From predicting the success of a new feature to personalizing offers based on player behavior, BigQuery helps game companies find answers to questions they may not even have thought to ask. Or thought who to ask. BigQuery is designed for approachability, so employees across the business can generate insights using data previously locked in silos. BigQuery goes further with built-in ML and Vertex AI to unlock insights across the organization.

Powering an everlasting love for the game

At Google Cloud, we believe that a live game’s success should depend on the strength of the game’s enjoyment, not limited to the strength of its infrastructure. That’s why we’re building an ecosystem for live games that unifies understanding and performance.

Google Cloud for Games is an ecosystem that includes not just Google Cloud, but also solutions from enablers like Ubitus, Incredibuild, Heroic Labs, and Unity.

We are excited to work with Google to offer developers our suite of Unity Gaming Services which provides the most robust tools and infrastructure to build, launch, and grow games seamlessly at a global scale Jeff Collins
SVP & GM of Unity Gaming Services, Unity

Solutions across this ecosystem work best when they work together as a cohesive whole. While there are standouts, the winningest MVP is backed by a well oiled team; just like a squad, clan, or raid group.

To learn more about Google Cloud for Games, visit goo.gle/cloudforgames

GLHF.

Power Matchmaking with Redis Enterprise via Google Cloud Marketplace

Fri, 16 Dec 2022 15:00:00 +0000

Have you ever played a multiplayer game where the players could have been better? This scenario is common and is likely not an ideal experience for anyone.

Game developers try to ensure that when you play, your opponents are at least as good as you are. Accurately matching players is a requirement for multiplayer games because creating compatible matches means more retention and revenue. Most of all, it increases players' loyalty to the game and boosts turnover.

While it's essential, developing effective matchmaking functionality is a challenging task. Evaluating skill level and many other attributes instantly is a requirement. Matchmaking in gaming needs to be blazing fast, as no player will wait long to get into a game session.

Challenges

The advancement of both hardware and software enables large-scale games where hundreds of players are getting matched to start a game or an experience. Concurrent players in popular games are on a steady rise. Nowadays, popular games' concurrent players range anywhere from 3 million to 10 million¹ depending on the game, day, and time of the year.

While the player pools are expanding, the latency expectations are shrinking. These two forces drive the need for horizontally scalable and blazing-fast matchmaking backends. Because of this, Google teamed up with Unity and created a framework called Open Match. Open Match runs on Kubernetes, and Redis is the data layer. The goal was to create an auto-scalable matchmaking framework to prevent reinventing the wheel and allow developers to focus on the logic to connect players for games. Previously, we looked into Memorystore with Open Match so now we will look into Redis Enterprise in Google Cloud Marketplace.

Redis Enterprise in Google Cloud Marketplace

As the advancement of both hardware and software pushes more concurrent players and large numbers of players per game, scalability has become the top priority for Open Match. Since Open Match runs on Kubernetes, the compute layer can auto-scale with technologies like GKE's Cluster autoscaler and is usually not a big concern for DevOps engineers. However, achieving scalability and high availability at the data layer is a mundane task for Open Match DevOps teams.

As the first step towards the scalability and high availability challenges, we stood up the Open Match framework with Redis Enterprise Cloud in place of OSS Redis. Redis Enterprise scales linearly and scales with a click of a button without requiring downtime, which massively reduces the scalability burden. On top, high availability is just a slider on the admin console.

Scalability Settings (left) and High Availability Settings (right)

Redis Enterprise Cloud is a fully managed service with 24/7x365 support and 99.999% SLA and offers simplified billing through a single Google Cloud Account.

VPC peering

You can deploy Redis Enterprise Cloud through the Google Cloud Marketplace or through the Terraform provider in a matter of minutes. Once the database is created, by providing the Google Cloud Project and Network name, you can peer the VPC the GKE cluster belongs to with Redis’s managed VPC. After the peering is completed, the GKE cluster could connect to the Redis Enterprise database through the database's private endpoint.

Open Match with Redis Enterprise Cloud

Assuming your newly created Redis Enterprise Cloud database has been provisioned, it is ready for use. You can run the following helm command to install Open Match on your GKE cluster pointing to the Redis Enterprise Cloud instance:

code_block: <ListValue: [StructValue([('code', 'helm install open-match open-match/open-match \\\r\n--set open-match-customize.enabled=true \\\r\n--set open-match-customize.evaluator.enabled=true \\\r\n--set open-match-override.enabled=true \\\r\n--set open-match-core.redis.enabled=false \\\r\n--set open-match-core.redis.hostname="Default:<password>@<Redis-Enterprise-Database-Private-Endpoint>" \\\r\n--set open-match-core.redis.port=<Redis-Enterprise-Database-Port-Number> \\\r\n-nopen-match'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe3892930a0>)])]>

For example,

code_block: <ListValue: [StructValue([('code', 'helm install open-match open-match/open-match \\\r\n--set open-match-customize.enabled=true \\\r\n--set open-match-customize.evaluator.enabled=true \\\r\n--set open-match-override.enabled=true \\\r\n--set open-match-core.redis.enabled=false \\\r\n--set open-match-core.redis.hostname="Default:ZDtglM5R@redis-16264.internal.c21576.us-west1-mz.gcp.cloud.rlrcp.com" \\\r\n--set open-match-core.redis.port=16264 \\\r\n-nopen-match'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe3892a92b0>)])]>

You can follow this tutorial to deploy and run the Open Match demo to generate matches and see Redis Enterprise in action.

Summary

Redis Enterprise on Google Cloud provides linear scalability, high availability and five-nines (99.999% SLA) of availability through Active-Active Geo Distribution feature spanning over 20 Google Cloud regions to support global game launches. User spikes are real and significant opportunities for the success of the game. Game companies need to be ready for these moments and do everything possible to retain the players on the platform. Adopt Open Match with Redis Enterprise for accurate, compatible, and ultra-fast matchmaking functionality at scale.

Next Steps

Footnotes:

^{1. Fortnite Player Count - How Many People Are Playing Now?}

^{A special 'thank you' to Gilbert Lau, Cloud Partner Solution Architect, Redis for his contribution to this article.}

aside_block: <ListValue: [StructValue([('title', 'Redis Enterprise on Google Cloud | Redis Cloud Partner'), ('body', <wagtail.rich_text.RichText object at 0x7fe3889e9820>), ('btn_text', 'Learn More'), ('href', 'https://redis.com/cloud-partners/google/'), ('image', <GAEImage: The Turbo Charged Partnership of Redis Enterprise and Google Cloud>)])]>

Gaming

The AI-powered shift to “living games:” Meet the customers and partners leveling up the ways we play

Atlas: AI for creating vast 3D game worlds

Footage shown is from Parallel’s upcoming game, Colony, using Atlas technology.

Embody: Personalized spatial audio for unrivaled immersion

Ludeo: Redefining game discovery with playable moments

NACON: Accelerating game production with AI transformation

Nitrado: Hybrid cloud scaling for flawless multiplayer gaming

Bring your living games to life

Enhancing multiplayer games with gen AI and cloud gaming: a technical deep dive

Cloud gaming with Agones: the foundation for scalable multiplayer experiences

Generative AI's Potential in Cloud Gaming

Building a language learning game with gen AI: A technical walkthrough

High-Level Architecture of Lexitrail

Take the Next Step

Co-op mode: New partners driving the future of gaming with AI

nunu.ai: Improving game testing with AI agents

ElevenLabs: AI-powered voice synthesis for immersive experiences

Common Sense Machines (CSM.ai): Best-in-class AI models for 3D content creation

The AI-driven future of gaming

Powering Web3-enabled gaming with GameShift by Solana Labs

The Web3 evolution of gaming

GameShift simplifies Web3 for games

Key Advantages

Reducing the risks of Web3 in gaming

Leveraging Google Cloud with Web3

Looking Ahead

Next Steps

Square Enix’s journey of building an AI-driven Customer Data Platform

Building an AI-driven CDP with Google Cloud

How SGV delivers AI-driven marketing

Bringing gaming fan engagement to the next level with Google Cloud

Game-changing assets: Making concept art with Google Cloud's generative AI

Solution

Concept or pre-production assets

Infrastructure setup

Final thoughts

References

Your guide to Generative AI support in Vertex AI

Tencent: Building analytics culture for better game development

An architecture for scalability and growth

A new foundation to connect gaming and marketing datasets

Moving forward with a revenue-generating tech stack

The evolution of play: From live to living games

Live games: a new era, new needs

Live games transform into living games with generative AI

Building the future of play through a collaborative ecosystem

Running machine learning in the cloud for live service games

Cloud-based ML plus live services games are a go

Unified data platforms enable real-time AI

Demoing capabilities

Game on: Aiven for Apache Kafka and BigQuery - your ultimate gaming cheat code

Automating infrastructure scaling in real-time

Future-proofing automated scaling

Aiven + Google Cloud = better together

Conclusion and next steps

The generative AI revolution in the games industry: A path to boundless creativity

Understanding the generative AI difference for games

Getting started on the path of innovation

How DeNA developed a live streaming app in just 2 months with Live Stream API

Putting the app together with Google Cloud

Broadcasting major sporting events confidently

DevOps Awards winner Improbable on “unleashing the full power of the cloud”

Meeting customer needs

Solution

The power of cloud

Accelerate your Multiplayer Game Development with the Global Scale Multiplayer Demo

Droidshooter

Architecture overview

Game state flow

Launching the game client

Match making

Allocating a game server

Final cleanup

Ecosystem breakdown

Terraform

GKE Autopilot and Standard

Agones

Open Match

Unreal Engine 5