Cloud First

Energy Drive "Actualises Sustainability" - dynamically monitoring and analyzing system performance with Google Cloud

Tue, 29 Nov 2022 17:00:00 +0000

Electric motors in everything from pool pumps and gate motors to the large industrial fans and pumps consume approximately 50% of the world’s generated electricity. Optimizing old equipment with more efficient engineering has the potential to deliver significant energy savings. But a big challenge remains: how do you increase the energy efficiency of large industrial equipment, especially when it’s dozens of years old?

This challenge and the opportunities to reduce electricital consumption, inspired us to found Energy Drive. We wanted to help large energy users actualise their energy related sustainability goals and overcome the barriers to efficiency in industrial settings, especially within the so-called triple M segment: mining, metals and minerals.

Our approach is very focused. We retrofit technology, called a variable speed drive (VSDs), onto industrial equipment such as fan and pump motors. These VSDs convert a fixed frequency electrical supply into a variable frequency output. The drive (VSD) controls how the motor operates: A low frequency for a slow speed, and a higher frequency for faster speeds. Put simply, you can make relatively small adjustments in motor speed in return for a large reduction in power consumed.

Energy Drive team members

Reducing energy consumption, minimizing downtime

Every system is monitored for energy consumption and system performance using the latest metering technology and industrial computers. This data is relayed to the client via a web application that shows quickly how plant process optimization is performing and the resulting cost and energy savings that are being generated.

Energy Drive also uses its own dedicated systems to monitor client installations. “We can create early warnings for impending equipment failure and increase visibility of issues to reduce unplanned downtime.”

As we’ve grown, so has the amount of data that we store and make accessible to clients. When we chose a database environment that matched our goals for growth, Google Cloud stood out. Without sacrificing functionality, Google Cloud is more scalable and flexible and a lot easier to manage than some of the other cloud environments we looked at.

We use Google Compute Engine to run virtual machines on Google Cloud infrastructure and Google App Engine to run our custom-built web tool which clients use to view performance data. In addition, we use Google Cloud Storage for storing and accessing data in our environment.

BigQuery is especially critical to our business. When we looked at other cloud-based data warehouses, they didn’t have a pricing model that matched our requirements. With BigQuery, we only pay for data that we extract and interact with. In fact, we managed to reduce data usage by 80% when switching from a SQL environment.

With BigQuery our clients can access current or historical data in seconds. That’s a big improvement on the past when many relied on manual systems where you had to sift through the network and thousands of data points to extract the information you were seeking.

750 gigawatt hours (GWh) of electricity saved

We can also overlay this data with insights that make a real difference to a client’s bottom line. Now when a client asks, "What did that system do this time last year?”, we can pull those details quickly and present them in an easy-to-understand format. At the same time, our installations are based on a zero-capex performance model, which simplifies our clients’ project delivery, saves money, and eliminates the short-term shock of large capital outlays which must compete for capital with other internal projects.

Above all, Google Cloud is the powerhouse behind our efforts to reduce energy consumption. In the lifetime of the business, we’ve saved 750 GWh of electricity, the equivalent of half the annual consumption of an average sized steel mill. Another way of looking at it is that we take the consumption equivalent of about 16,500 households off the grid every day.

We’re also proud to be recognized for our achievements in the energy saving industry. In 2019, we were awarded the Energy Efficiency Company of the Year by the South African Energy Efficiency Confederation. In 2021, we were awarded Industrial Energy Project of the Year for our work supporting a South African mining company towards their 2040 carbon neutrality and 2050 net zero target.

We’re also committed to developing engineering talent in South Africa. Our program for graduates offers young people experience in many fields including electrical, mechanical, and process engineering. They can choose a career path with the confidence that comes from first-hand knowledge of the latest developments in engineering and analytics.

This links back to our determination to embed best-in-class skills and technology in our business, including the latest tools and enhancements in Google Cloud. Right now, we are excited about powerful machine learning capabilities in Google Cloud and the potential to apply this to our data and provide further energy efficiencies to our clients. Knowing that Google will continue to innovate gives us the confidence to grow the business and help our clients achieve their sustainability goals now and in the future.

If you want to learn more about how Google Cloud can help your startup, visit our page here to get more information about our program, and sign up for our communications to get a look at our community activities, digital events, special offers, and more.

Upskill for in-demand cloud roles with no-cost training on Coursera

Tue, 08 Nov 2022 17:00:00 +0000

Cloud technology has experienced accelerated adoption in recent years, with continued growth expected into 2023.¹ This means that the need for organizations to attract and retain professionals with cloud skills continues to grow in parallel.²

Keep your cloud career growing, at pace with digital transformation

In partnership with Google Cloud, Coursera is offering no-cost access to some of our most popular cloud training to help you hone your skills and stand out in the job-market. Whether you’re looking to enhance your technical competencies, advance your career, acquire more hands-on experience, or earn learning credentials to validate your knowledge, we have resources available to support your journey.

Future-proof your career with select no-cost training and earn certificates

Claim one choice from a variety of popular Google Cloud Projects, Professional Certificates, Specializations and courses, available to claim until December 31st, 2022.

The Google Cloud training included in this promotion spans a variety of roles, like machine learning engineering; data engineering; and cloud engineering, architecture and security. Training content is available for both technical and non-technical roles, from foundational to advanced knowledge and experience levels. The training descriptions include any prerequisite knowledge you should have before getting started.

The time requirements for completion also vary, so we’ve summarized it below to help you make your choice, and pick the level of commitment that is right for you. When you finish the training on Coursera, you will earn a certificate that you can share with your network on social media and your resume.

Types of Google Cloud training available on Coursera

Here is a rundown of the different types of training available on Coursera included in this offer, in order of time required to complete it:

Projects: Approximately 30-90 minute time commitment to complete

Learn new skills in an interactive environment by using software and tools in a cloud workspace with no download required.

Courses: Approximately 4-19 hour time commitment to complete

Courses typically include a series of introductory lessons, step-by-step hands-on exercises, Google knowledge resources, and knowledge checks.

Specializations: Approximately 2-6 months time commitment to complete

Specializations are a series of courses that help you master a skill, and include a hands-on project.

Professional Certificates: Approximately 1-9 months

Professional Certificates include hands-on projects and courses, and upon completion you will earn a Professional Certificate. These can help you prepare for the relevant Google Cloud certification exam.

Here is a look at some of our most popular training for in-demand cloud roles

Work through training at your own pace, and upskill for the role you’re in, or the one you’re looking to grow into. Popular training for in-demand roles include:

For those in non-technical roles, working closely with cloud technology

Professional Certificate - Cloud Digital Leader
This is a foundational level series of four courses designed to give you knowledge about cloud technology and data, and digital transformation. It helps increase confidence in contributing to cloud-related business initiatives and discussions. If you’re in a tech-adjacent role such as sales, HR or operations, you will benefit from this training.

For Application Developers

Specialization - Developing applications with Google Cloud
This Specialization is built for application developers who want to learn how to design, develop, and deploy applications that seamlessly integrate managed services from Google Cloud. It includes a variety of learning formats, including labs, presentations and demos. Labs can be completed in your preferred language: Node.js, Java, or Python. You’ll learn practical skills that are ready for immediate use in real IT environments.

For experienced ML and AI Engineers

Professional Certificate - ML Engineer
Prepare for Google Cloud Certification with the Machine Learning Engineer Professional Certificate. This is an intermediate-level training recommended for participants who have data engineering or programming experience, and who want to learn how to apply machine learning in practice and to be successful in a machine learning role. There are 9 courses in this Professional Certificate, and completion time is about 7 months at the suggested pace of 5 hours per week.

For beginners with Google Cloud in technical roles

Course - Google Cloud Fundamentals for AWS Professionals
This course introduces key concepts and terminology through a combination of videos and hands-on labs that can be completed in approximately 9 hours. You’ll learn about the components of the Google network infrastructure and differences between infrastructure as a service and platform as a service; how to organize projects and interact with Google Cloud; and jump into Google Cloud Compute Engine with a focus on virtual networking.

For beginners in Data Engineering

Project - Introduction to SQL for BigQuery and Cloud SQL
This is a self-paced lab that takes place in the Google Cloud console, giving you interactive practice running structured queries on BigQuery and Cloud SQL. This is a beginner level project that takes about an hour to complete.

As the year comes to a close, it’s a great time to prioritize growing your cloud skills. Check out our no-cost Google Cloud training offers on Coursera, available until December 31, 2022.

^{1. According to Forbes: The Top 5 Cloud Computing Trends in 2023
2. According to Forbes: From Crisis to Opportunity: Tackling the U.S. Cloud Skills Gap}

Access modeled data from Looker Studio, now in public preview

Mon, 07 Nov 2022 13:00:00 +0000

In April, we announced the private preview of our integration between Looker and Looker Studio (previously known as Data Studio). At Next in October, to further unify our business intelligence under the Looker umbrella, we announced that Data Studio has been renamed Looker Studio. The products are now both part of the Looker family with Looker Studio remaining free of charge. At Next we also announce that the integration between these two products is now available in public preview with additional functionality.

How does the integration work?

Customers using the Looker connector will have access to governed data from Looker within Looker Studio. The Looker connector for Looker Studio makes both self-serve and governed BI available to users in the same tool/environment. When connecting to Looker, Looker Studio customers are able to leverage its semantic data model which enables complex data to be simplified for end users with a curated catalog of business data, pre-defined business metrics, and built-in transformations. This helps users make calculations and business logic consistent within a central model and promotes a single source of truth for their organization.

Access to Looker-modeled datawithin Looker Studio reports allows people to use the same tool to create reports that rely on both ad-hoc and governed data. They can use LookML to create Looker data models by centrally defining and managing business rules and definitions in one Git, version-controlled data model..

Users can analyze and rapidly prototype ungoverned data (from spreadsheets, csv files, or other cloud sources) within Looker Studio and blend governed data from Looker with data available from over 800 data sources in Looker Studio to rapidly generate new insights. They can turn their Looker-governed data into informative, highly customizable dashboards and reports in Looker Studio and collaborate in real-time to build dashboards with teammates or people outside the company.

What’s new in the public preview version?

We are excited that we are now able to offer this preview to a broader reach of customers, many of whom have already asked for access to the Looker connector for Looker Studio. Additionally, with this Public Preview, additional capabilities have been added to more fully represent the Looker model in Looker Studio:

We are providing support for field hierarchies in the Looker Studio data panel, to keep fields organized when working with large Explores. The data panel will now show a folder structure, and you will be able to see your fields organized in the usual ways - for Views, Group Labels, and Dimension Groups.
We are providing greater visibility by exposing field descriptions in new ways to enable users to quickly check the description information specified in the Looker model. Field descriptions will be available within the data panel and within tables in the report.
Users will also see an option to "Open in Looker Studio" from Explores in Looker, enabling them to quickly create a Looker Studio report with a data source pointing back to that Explore.
And to ensure users are getting the most current data from the underlying data source, refreshing data in Looker Studio now also refreshes the data in the Looker cache.

Specifically, for this public preview, we’ve implemented enhanced restrictions on Looker data sources in Looker Studio, so admins can rest easy about testing out the functionality:

We’ve disabled owner’s credentials for Looker data sources in Looker Studio, so each and every viewer needs to supply their own credentials including for shared reports.
We’re also currently disabling data download and email scheduling for these data sources in Looker Studio. We’re planning to integrate with these permissions in Looker in the near future.
Calculated fields are disabled, so end users cannot define their own custom metrics and dimensions in Looker Studio, and need to rely on the fields defined in the Looker Explore.

How do I access the preview?

This integration encompasses the connector along with changes made to both Looker Studio and Looker to represent the Looker model and extend Looker governance in Looker Studio. There is much more to come as we continue our efforts to bring together a complete, unified platform balancing self-service and governed BI. We’re planning to continue adding functionality in Looker Studio to fully represent the Looker model, and want to ensure Looker admins have insight into API activity coming from Looker Studio - similar to the way they might use System Activity in Looker today. In extending governance, we want to expand the circle of trust from Looker to Looker Studio, and we’ll be looking for customers to help us plan the best way forward.

This integration is compatible with Google Cloud hosted instances with Looker version 22.16 or higher. To get access, an admin of a Looker instance can submit the sign-up form providing an instance URL and specifying which organizational domain to enable. For more information on how to get started go to the Looker Studio Help Center.

For more information and demo, watch the Next ‘22 session ANA202: Bringing together a complete, unified BI platform with Looker and Data Studio and Keynote: ANA100: What’s new in Looker and Data Studio.

Introducing Cloud Workstations: Managed and Secure Development environments in the cloud

Tue, 01 Nov 2022 15:00:00 +0000

With the unprecedented increase in remote collaboration over the last two years, development teams have had to find new ways to collaborate, driving increased demand for tools to address the productivity challenges of this new reality. This distributed way of working also introduces new security risks, such as data exfiltration — information leaving the company's boundaries. For development teams, this means protecting the source code and data that serves as intellectual property for many companies.

At Google Cloud Next, we introduced the public Preview of Cloud Workstations, which provides fully managed and integrated development environments on Google Cloud. Cloud Workstations is a solution focused on accelerating developer onboarding and increasing the productivity of developers’ daily workflows in a secure manner, and you can start using it today simply by visiting the Google Cloud console and configuring your first workstation.

Cloud Workstations: Just the facts

Cloud Workstations provides managed development environments with built-in security, developer flexibility, and support for many popular developer tools. Cloud Workstations addressing the needs of enterprise technology teams.

Developers can quickly access secure, fast, and customizable development environments anywhere, via a browser or from their local IDE. With Cloud Workstations, you can enforce consistent environment configurations, greatly reducing developer ramp-up time and addressing “works on my machine” problems.
Administrators can easily provision, scale, manage, and secure development environments for their developers, providing them access to services and resources that are private, self-hosted, on-prem, or even running in other clouds. Cloud Workstations makes it easy to scale development environments, and helps automate everyday tasks, enabling greater efficiency and security.

Cloud Workstations focuses on three core areas:

Fast developer onboarding via consistent environments
Customizable development environments
Security controls and policy support

Fast developer onboarding via consistent environments

Getting developers started on a new project can take days or weeks, with much of that time spent setting up the development environment. The traditional model of local setup may also lead to configuration drift over time, resulting in “works on my machine” issues that erode developer productivity and stifle collaboration.

To address this, Cloud Workstations provides a fully managed solution for creating and managing development environments. Administrators or team leads can set up one or more workstation configurations as their teams’ environment templates. Updating or patching the environments of hundreds or thousands of developers is as simple as updating their workstation configuration and letting Cloud Workstations handle the updates.

Developers can create their own workstations by simply selecting among the configurations to which they were granted access, making it easy to ensure consistency. When developers start writing code, they can be certain that they are using the right version of their tools.

Customizable development environments

Developers use a variety of tools and processes optimized to their needs. We designed Cloud Workstations to be flexible when it comes to tool choice, enabling developers to use the tools they’re the most productive with, while enjoying the benefits of remote development. Here are some of the capabilities that enable this flexibility:

Multi-IDE support: Developers use different IDEs for different tasks, and often customize them for their maximum efficiency. Cloud Workstations supports multiple managed IDEs such as IntelliJ IDEA Ultimate, PyCharm Professional, GoLand, WebStorm, Rider, Code-OSS, and many more. We’ve also partnered with JetBrains so that you can bring your existing licenses to Cloud Workstations. These IDEs are provided via optimized browser-based or local-client interfaces, avoiding the latency and challenges of general-purpose remote desktop tools such as latency and limited customization.

Container-based customization: Beyond IDEs, development environments also comprise libraries, IDE extensions, code samples, and even test databases and servers. To help ensure your developers are getting the tools they need quickly, you can extend the Cloud Workstations container images with the tools of your choice.
Support for third-party DevOps tools: Every organization has its own tried and tested tools — Google Cloud services such as Cloud Build, but also third-party tools such as GitLab, TeamCity, or Jenkins. By running Cloud Workstations inside your Virtual Private Cloud (VPC), you can connect to tools self-hosted in Google Cloud, on-prem, or even in other clouds.

Security controls and policy support

With Cloud Workstations, you can extend the same security policies and mechanisms you use for your production services in the cloud to your developer workstations. Here are some of the ways that Cloud Workstations helps to ensure the security of your development environments:

No source code or data is transferred or stored on local machines.
Each workstation runs on a single dedicated virtual machine, for increased isolation between development environments.
Identity and Access Management (IAM) policies are automatically applied, and follow the principle of least privilege, helping to limit workstation access to a single developer.
Workstations can be created directly inside your project and VPC, allowing you to help enforce policies like firewall rules or scheduled disk backups.
VPC Service Controls can be used to define a security perimeter around your workstations, constraining access to sensitive resources, and helping prevent data exfiltration.
Environments can be automatically updated after a session reaches a time limit, so that developers automatically get any updates in a timely manner.
Fully private ingress/egress is also supported, so that only users inside your private network can access your workstations.

What customers and partners are saying

"We have hundreds of developers all around the world that need to be able to be connected anytime, from any device. Cloud Workstations enabled us to replace our custom solution with a more secure, controlled and globally managed solution." — Sebastien Morand, Head of Data Engineering, L'Oréal

“With traditional full VDI solutions, you have to take care of the operating system and other factors which are separate from the developer experience. We are looking for a solution that solves problems without introducing new ones.” — Christian Gorke, Head of Cyber Center of Excellence, Commerzbank

“We are incredibly excited to tightly partner with Google Cloud around their Cloud Workstations initiative, that will make remote development with JetBrains IDEs available to Google Cloud users worldwide. We look forward to working together on making developers more productive with remote development while improving security and saving computation resources.” — Max Shafirov, CEO, JetBrains

Get started today

Try Cloud Workstations today by visiting your console, or learn more on our webpage, in our documentation or by watching this Cloud Next session. Cloud Workstations is a key part of our end-to-end Software Delivery Shield offering. To learn more about Software Delivery Shield, visit this webpage.

Tell us about your Cloud Architecture dreams

Thu, 06 Oct 2022 16:00:00 +0000

Whether you are new to Google Cloud or an existing customer running thousands of workloads, one of the most common questions we get is, “How would Google architect and design my environment?” The answer to this question would truly be a dream come true, right? Just to set expectations, we won’t answer that here. This short blog is intended to give you a hint of how we’re thinking about this question, before taking a deeper dive into the subject in a future whitepaper. More importantly, we hope it gets you thinking about how you’d like to see us answer that question in the future. We’d love to hear from you, so if you are itching to tell us your cloud architecture dreams, you’ll find ways to let us know at the end of this blog.

Can We Talk About Architecture?

If you were to look up the term “architecture”, it’s likely there would be references to designing and constructing buildings or perhaps more broadly, designing and constructing “something”. When we think about cloud architecture specifically, we have certain requirements in mind and expectations for what we’ve built. Whatever it is we build needs to provide some utility to us. Technical architectures typically must meet certain non-functional requirements like resilience, performance, security, and more. One method of design takes functional and non-functional elements of an architecture and aligns them with a specific set of design principles. Design principles act as a compass for what you are trying to create. While they don’t define exactly how to get to your destination, they keep you pointed in the right direction. Using design principles is certainly not a new concept. However, a design principle driven approach to architecture allows us to come up with a slightly better definition of what architecture is. For purposes of the discussion here, let’s think of architecture as a purpose built service that conforms to a set of prioritized design principles. In other words, the thing that you’ve built is providing some value or utility to the organization and at the same time must have certain characteristics to stay on course with the “path” your design intended.

What is a Picture Worth?

It’s been said that cloud architects sometimes create a bunch of pretty documents and pictures but at the end of the day, those assets get shoved into a drawer; never to be read again. Assuming what we’ve designed has value to us, there are likely certain design principles that drive the architecture. We expect what we’ve built to not only provide some utility, but also meet the design principles we had in mind; secure, resilient, high performance, and so on. How would you ensure that your new system continues to meet the standards of design you’ve established? There are configuration management solutions like Chef, Puppet, or cloud based services like Deployment Manager… or Terraform. Those are all great at managing configuration, but are they managing architecture? Do any of these tools know that a particular managed configuration will affect a design principle like performance, security or resiliency? Are any of these tools guiding you on what should even be managed or configured to begin with? What do you do with these tools when there are new architectural requirements? We realize that these are more questions than answers. The point is that maintaining the integrity of your environment’s architecture is much different from maintaining the integrity of its configuration. In fact, if the design principles on which you based your architecture provide value to your organization, then one would think that maintaining the integrity of the architecture is actually the most important thing.

What’s Your Dream Cloud Architecture?

What would your dream scenario be with regard to your cloud architecture? If we had to guess, we think you’d want (at least in part) something that answered the first question posed; “How would Google architect and design my environment?”. What does this really mean to you though? Companies are very different and Google is no exception to that. What works for Google might not work for you. Would we ask you what you need to do and then based on your response generate an architecture? Would it mean you telling us what you prioritize, reviewing something you’ve built, and then providing specific recommendations? What if your environment changed? What if our cloud services changed? We are starting to think about how to approach cloud architecture differently and will be discussing more thoughts on the subject in future blog posts. Until then, we would love to hear what is important to you about your cloud architecture. If you’d like to learn more, take a look at the latest version of our Architecture Framework. If you’d like to connect with us, come talk to us in our community or share your cloud architecture dreams with us at cloud-architecture-dreams@google.com.

How to get the most from your Intel-based instances on Google Cloud

Thu, 29 Sep 2022 16:00:00 +0000

Deploying mission-critical applications on Google Cloud can yield immediate benefits in terms of performance and total cost of ownership (TCO). That’s why Google Cloud is partnering with Intel to help our mutual customers optimize their most demanding workloads on Intel-based instances. The Intel Software Center of Excellence (CoE) for Google Cloud launched as a pilot in North America last year, and the results were dramatic — with an increase in Tensorflow inference performance for ad ranking algorithms, gain in throughput and reduction in latency for Redis under heavy loads and faster transcoding of videos to 1080p.

Now, Intel and Google Cloud are expanding the program globally by opening it up to select high-growth enterprise accounts.

The Intel Software Center of Excellence is a white-glove, high-touch program for customers looking to reduce latency and improve workload efficiency. This program is designed to enhance the value customers get from their Intel Xeon Scalable processors running in Google Cloud, offering them benchmarking and performance optimization techniques. It provides:

Direct access to Intel engineers providing white-glove service
Guidance for improving the price performance and the operational performance of Intel-based Google Cloud instances
Code-level recommendations on Intel processors so customers can experience the most benefit possible from their Google Cloud investments

Optimizing the performance of your most demanding workloads

Joining the CoE program is an opportunity to work directly with Intel engineers to maximize the performance of your workloads on Intel instances in Google Cloud. Here are just a few examples of workloads that CoE participants are able to fine-tune and better manage performance through the program:

Databases: Learn to use a wide variety of relational and nonrelational databases to address latency issues at peak loads or under complex conditions, such as Redis “latency knee” for e-commerce applications.
Analytics: Get guidance on using Apache Spark.
AI inference, including Recommendation, natural language processing, and vision recognition: Take advantage of Intel DL Boost in N2/C2 instances and Intel Math Kernel Library, and Tensorflow optimization.
Secure web transactions: Built-in Intel crypto instructions can speed security processing for applications such as NGINX and Wordpress.
Language runtime libraries, including Golang Crypto, Java, and Python.
Media, including video transcode, encode, and decode, as well as image processing, such as AVX-512 optimization and library optimizations like multithreading.

"The collaboration between Intel, Google and Equifax utilizing the Intel Software CoE successfully optimized Equifax's environment by producing nearly 2x throughput and a 50% drop in our critical metric for latency,” says Bryson Koehler, Chief Product, Data, Analytics & Technology Officer at Equifax. “The CoE met our objectives around cost optimization while improving performance SLAs for our end-customers.”

How it works

The Intel Software Center of Excellence engagement takes place in three phases:

Phase 1: Discovery. The Google Cloud and Intel teams collaborate with you to review your performance objectives and identify key projects and long-term goals that may influence compute trends.
Phase 2: Performance Review. Intel engineers leverage your metrics and Intel internal resources to review resource utilization across your service footprint.
Phase 3: Performance report. The engagement concludes with the Intel team delivering a Performance Report, which includes detailed optimization recommendations, an action plan, and an implementation plan, and recommendations for potential support from Google Cloud Professional Services Organization (PSO), which can give operational guidance on getting the most value from your Google Cloud products.

Get in touch to participate

The Intel Software CoE is open by application. Once your application is reviewed and accepted, there is no charge for the service. To participate, please complete the online application.

Introducing support for GPU workloads and even larger Pods in GKE Autopilot

Thu, 29 Sep 2022 16:00:00 +0000

Autopilot is a fully managed mode of operation for Google Kubernetes Engine (GKE). But being fully managed doesn’t mean that you need to be limited in what you can do with it! Our goal for Autopilot is to support all user workloads (those other than administrative workloads which require privileged access to nodes) so they can be run across the full GKE product.

Many workloads, especially those running AI/ML training and inference require GPU hardware. To enable such workloads on Autopilot, we are launching support in Preview for the NVIDIA T4 and A100 GPUs in Autopilot. Now you can run ML training, inference, video encoding and all other workloads that need a GPU, with the convenience of Autopilot’s fully-managed operational environment.

The great thing about running GPU workloads on Autopilot is that all you need to do is specify your GPU requirements in your Pod configuration, and we take care of the rest. No need to install drivers separately, or worry about non-GPU pods running on your valuable GPU nodes, because Autopilot takes care of GPU configuration and Pod placement automatically.

You also don’t have to worry about a GPU node costing you money without any currently running workloads, since with Autopilot you are just billed for Pod running time. Once the GPU Pod terminates, so do any associated charges—and you’re not charged for the setup or tear down time of the underlying resource either.

Some of our customers and partners have already been trying it out. Our customer CrowdRiff had the following to say:

"CrowdRiff is an AI-powered visual content marketing platform that provides user-generated content discovery, digital asset management, and seamless content delivery for the travel and tourism industry. As users of Google Kubernetes Engine (GKE) and its support for running GPU-accelerated workloads, we were excited to learn about GKE Autopilot's upcoming support for GPUs. Through our initial testing of the feature we found that we were able to easily take advantage of GPUs for our services without having to manage the underlying infrastructure to support this. Utilizing this functionality we expect to see reduced costs versus using standard GKE clusters and lower operational complexity for our engineers." — Steven Pall, Reliability Engineer, CrowdRiff

And our partner SADA comments:

“Our recommendation to customers is to leverage Autopilot whenever possible because of its ease of use and the reduction of operational burden. The whole GKE node layer is offloaded to Google, and GPU pods for Autopilot enable an entirely new workload type to run using Autopilot. The Autopilot mode is an exciting enhancement for our customers to run their AI/ML jobs.” — Christopher Hendrich, Associate CTO, SADA

Using GPUs with Autopilot

You can request T4 and A100 GPUs in several predefined GPU quantities. You can accept the defaults for CPU and Memory, or specify those resources as well, within certain ranges. Here’s an example Pod that requests multiple T4 GPUs.

code_block: <ListValue: [StructValue([('code', 'apiVersion: apps/v1\r\nkind: Deployment\r\nmetadata:\r\n labels:\r\n app: tensorflow\r\n name: tensorflow-t4\r\nspec:\r\n replicas: 1\r\n selector:\r\n matchLabels:\r\n app: tensorflow\r\n template:\r\n metadata:\r\n labels:\r\n app: tensorflow\r\n spec:\r\n nodeSelector:\r\n cloud.google.com/gke-accelerator: nvidia-tesla-t4\r\n containers:\r\n - image: tensorflow/tensorflow:latest-gpu-jupyter\r\n name: tensorflow-t4\r\n resources:\r\n limits:\r\n nvidia.com/gpu: "4"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fa47c139040>)])]>

^{Listing 1: Simply specify your nvidia-tesla-t4 node selector and your pod will run on GPU}

Those few lines in your Kubernetes configuration is all you need to do! Just specify your GPU requirements in the PodSpec, and create the object via kubectl. Autopilot takes care of tainting GPU nodes to prevent non-GPU Pods running on them, and tolerating these taints in your workload specifications - all automatically. We will automatically provision a GPU-enabled node matching your requirements, including any required Nvidia driver setup.

If for some reason your GPU Pod doesn’t become ready, check what’s going on with kubectl get events -w, and double-check that your resource values are within the supported ranges.

Run Large Pods on Autopilot with the Balanced Compute Class

And GPU isn’t the only thing we’re adding today! Autopilot already supports a leading 28 vCPU maximum Pod size with the default compute, and up to 54 vCPU with the Scale-Out compute class, but we wanted to push the limits even higher for those workloads that need a bit extra.

For those times when you need computing resources on the larger end of the spectrum, we’re excited to also introduce the Balanced compute class supporting Pod resource sizes up to 222vCPU and 851GiB! Balanced joins the existing Scale-Out compute class (which has a focus on high single-threaded CPU performance), and our generic compute platform (designed for everyday workloads).

To get started with Balanced, simply add a node selector to your pods. Listing 2 is an example pod definition. Be sure to adjust the resource requirements to what you actually need though! Refer to this page for the pricing information of Balanced Pods.

code_block: <ListValue: [StructValue([('code', 'apiVersion: apps/v1\r\nkind: Deployment\r\nmetadata:\r\n name: nginx-arm64\r\nspec:\r\n selector:\r\n matchLabels:\r\n app: nginx\r\n template:\r\n metadata:\r\n labels:\r\n app: nginx\r\n spec:\r\n nodeSelector:\r\n cloud.google.com/compute-class: Balanced\r\n containers:\r\n - name: nginx\r\n image: nginx:latest\r\n resources:\r\n requests:\r\n cpu: 222\r\n memory: 851Gi'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fa47c1392b0>)])]>

^{Listing 2: Run large pods using the Balanced compute class}

As with GPU Pods, Autopilot automatically handles the placement of Balanced compute class Pods for you, so that you will only be charged the Balanced compute class prices for Pods that directly specify it. By default, Pods without the compute class nodeSelector will run on Autopilot’s original compute platform (where they can request up to 28 vCPUs).

We can’t wait to see what you do with these new capabilities of GKE Autopilot.

View our docs to read more about Autopilot GPU, and the new Balanced Compute Class.

Google Cloud Deploy adds Cloud Run and deployment verification support

Tue, 27 Sep 2022 16:00:00 +0000

Editors note: This blog post was updated to reflect the general availability status of these features as of February 27, 2023.

Google Cloud customers want to be able to easily deploy their applications to the full breadth of platforms that we offer, including Cloud Run. And when they push out code to production, they want confirmation that the deployment was successful.

Today, we’re pleased to announce the general availability of Cloud Run targets and deployment verification for Google Cloud Deploy.

Deploy to Cloud Run

Support for Cloud Run, our managed serverless container runtime, has been a top feature request for Google Cloud Deploy. It’s not hard to understand why: Adding a Cloud Run target to Google Cloud Deploy makes it easier to develop and deliver your enterprise applications.

Generally available, delivery pipelines can now specify and deploy to Cloud Run targets, enabling continuous delivery of Cloud Run services.

All the continuous delivery capabilities that Google Cloud Deploy provides for other targets — rollback, approval, audit, and delivery metrics, to name just a few — are also available for Cloud Run targets. This consistency and feature parity allows platform operators and application developers to manage and reason about their application delivery pipelines in the same way, regardless of the runtime target.

This consistency is enabled by Skaffold, an open-source cloud-native tool developed by Google that’s the foundation of Cloud Deploy. With the recent 2.0 beta 2 release, Skaffold users can now develop and deploy Cloud Run services just as they already do for Google Kubernetes Engine and Anthos clusters, making Skaffold workflows a consistent point of adoption and extension for Google Cloud Deploy.

Continuous delivery pipeline with two Cloud Run targets

Verify your deployment

The success or failure of a deployment frequently involves more than just rolling out an artifact to a target platform — it also involves testing to further confirm the deployment, often in the form of automated integration and canary testing.

Customers told us they wanted formal support for deployment verification within Google Cloud Deploy. And when a deployment succeeds but a post-deployment verification test fails, the rollout should be identified as a failure, too.

Within Google Cloud Deploy, you can now specify one or more (testing) containers to execute immediately when an application is successfully deployed. This deployment verification support within Google Cloud Deploy is based on Skaffold 2.0's recently introduced verify command. You can use any process that runs in a container to verify the state of the application. An example could be as simple as issuing a curl command, or more complex, like validating all of the links via a third-party tool, or even gathering performance metrics. Verifying a deployment is as easy as configuring Skaffold to test the deployment ("command"), then specifying a ‘verify: true’ in the Cloud Deploy delivery pipeline’s progression sequence.

As with render and deploy operations, deployment verification in Google Cloud Deploy is performed in its own execution environment. This allows for custom verification configurations using a specified worker pool or service account, and storing results in a preferred Cloud Storage location. Verification results are factored in when determining whether the rollout was a success or failure. When a deployment verification failure occurs, it’s easy to inspect the logs and, if necessary, rerun the deployment verification without having to re-deploy.

Deployment verification is available for all target types, including Cloud Run. You can learn more about deployment verification in this detailed blog post.

Deployment verification status and results in rollout details

The future

Comprehensive, easy-to-use, and cost-effective DevOps tools are key to building an efficient software delivery capability, and it’s our hope that Google Cloud Deploy will help you implement complete CI/CD pipelines. And we’re just getting started. Stay tuned as we introduce exciting new capabilities and features to Google Cloud Deploy in the months to come.

In the meantime, check out the product page, documentation, quickstarts, and tutorials. Finally, If you have feedback on Google Cloud Deploy, you can join the conversation. We look forward to hearing from you.

Prepare for Google Cloud certification with top tips and no-cost learning

Mon, 11 Jul 2022 16:30:00 +0000

Becoming Google Cloud certified has proven to improve individuals’ visibility within the job market, and demonstrate ability to drive meaningful change and transformation within organizations.

1 in 4 Google Cloud certified individuals take on more responsibility or leadership roles at work, and 87% of Google Cloud certified users feel more confident in their cloud skills¹.
75% of IT decision-makers are in need of technologically-skilled personnel to meet their organizational goals and close skill gaps².
94% of those decision-makers agree that certified employees provide added value above and beyond the cost of certification³.

Prepare for certification with a no-cost learning opportunity

That's powerful stuff, right? That’s why we've teamed up with Coursera to support your journey to becoming Google Cloud certified.

As a new learner, get one month of no-cost access to your selected Google Cloud Professional Certificate on Coursera to help you prepare for the relevant Google Cloud certification exam. Choose from Professional Certificates in data engineering, cloud engineering, cloud architecture, security, networking, machine learning, DevOps and for business professionals, the Cloud Digital Leader.

Become Google Cloud certified

To help you on your way to becoming Google Cloud certified, you can earn a discount voucher on the cost of the Google Cloud certification exam by completing the Professional Certificate on Coursera by August 31, 2022

Simply visit our page on Coursera and start your one month no-cost learning journey today.

Top tips to prepare for your Google Cloud certification exam

Get hands-on with Google Cloud
For those of you in a technical job role, we recommend leveraging the Google Cloud projects to build your hands-on experience with the Google Cloud console. With 500+ Google Cloud projects now available on Coursera, you can gain hands-on experience working in the real Google Cloud console, with no download or configuration required.
Review the exam guide
Exam guides provide the blueprint for developing exam questions and offer guidance to candidates studying for the exam. We´d encourage you to be prepared to answer questions on any topic in the exam guide, but it's not guaranteed that every topic within an exam guide will be assessed.
Explore the sample questions
Taking a look at the sample questions on each certification page will help to familiarize you with the format of exam questions and example content that may be covered.

Start your certification preparation journey today with a one month no-cost learning opportunity on Coursera.

Want to know more about the value of Google Cloud Certification? Find out why IT leaders choose Google Cloud Certification for their teams.

^{1. Google Cloud, Google Cloud certification impact report, 2020
2. Skillsoft Global Knowledge, IT skills and Salary report, 2021
3. Skillsoft Global Knowledge, IT skills and Salary report, 2021}

Incorporating quota regression detection into your release pipeline

Tue, 28 Jun 2022 16:00:00 +0000

On Google Cloud, one of the ways an organization may want to enforce fairness in how much of a resource can be consumed is through the use of quotas. Limiting resource consumption on services is one way that companies can better manage their cloud costs. Oftentimes, people associate quotas with APIs to access that said resource. Although an endpoint may be able to handle a high number of Queries Per Second (QPS), the quota gives them a means to ensure that no one user or customer has monopoly over the available capacity. This is where fairness comes into play. It allows people to put limits that can be scoped per user or per customer and allows them to increase or lower those limits.

Although quota limits address the issue of fairness from a resource providers’ point of view — in this case, Google Cloud — you still need a way as the resource consumer to ensure that those limits are adhered to and, just as importantly, ensure that you don’t inadvertently violate those limits. This is especially important in a continuous integration and continuous delivery (CI/CD) environment, where there is so much automation going on. CI/CD is heavily based on automating product releases and you want to ensure that the products released are always stable. This brings us to the issue of quota regression.

What is quota regression and how can it occur?

Quota regression refers to the unplanned change in an allocated quota that oftentimes results in a reduced capacity for resource consumption.

Let's take for example an accountant firm. I have many friends in this sector and they can never hang out with me during their busy season between January and April. At least, that’s the excuse. During the busy season, they have an extraordinarily high caseload, and a low caseload the rest of the year. Let’s assume that these caseloads actually have an immediate impact on your resource costs on Google Cloud. Since this high caseload only occurs at a particular point throughout the year, it may not be necessary to maintain a high quota at all times. It’s not financially prudent since resources are paid on a “per-usage” model.

If the accountant firm has an in-house engineering team that has built load-tests to ensure the system is functioning as intended, you would expect the load capacity to increase before the busy season. If the load test is being done in an environment separate from the serving one (which it should be due to reasons such as security and avoiding unnecessary access grants to data), this is where you might start to see a quota regression. An example of this is load testing in your non-prod Google Cloud project (e.g. your-project-name-nonprod) and promoting images to your serving project (e.g. your-project-name-prod).

In order for the load tests to pass, there must be a sufficient quota allocated to the load testing environment. However, there exists a possibility that that quota has not been granted in the serving environment. It could be due to simply an oversight in the process where the admin needed to request the additional quota in the serving environment, or it could be because that quota was reverted after a busy season and thus went unnoticed. Whatever the reason, it still depends on human intervention to assert that the quotas are consistent across environments. If this is missed, the firm can go into a busy season with passing load tests and still have a system outage due to lack of quota in the serving environment.

Why not just use traditional monitoring?

This brings to mind the argument of “Security Monitor vs Security Guard.” Even with monitoring to detect such inconsistencies, alerts can be ignored and alerts can be late. Alerts work if there is no automation tied to the behavior. In the example above, alerts may just suffice. However, in the context of CI/CD, it’s likely for a deployment that introduces a higher QPS on dependencies to be promoted from a lower environment to the serving environment, because the load tests pass if the lower environment has sufficient quota. The problem here is that now that deployment is automatically pushed to production with alerts probably occurring with the outage.

The best way to handle these scenarios is to incorporate an automated way of not just monitoring and alerting, but a means for preventing promotion of that regressive behavior to the serving environment. The last thing you want is new logic that requires a higher resource quota than what is granted being automatically promoted to prod.

Why not use existing checks in tests? The software engineering discipline offers several types of tests (unit, integration, performance, load, smoke, etc…), none of which address something as complex as cross-environment consistency. Most of them focus on the user and expected behaviors. The only test that really focuses on infrastructure is the load test, but a quota regression is not necessarily part of the load test. It's not something you're going to detect since a load test occurs in its own environment and is agnostic of where it's actually running.

In other words, a quota regression test needs to be aware of the environments — it needs an expected baseline environment where the load test occurs and an actual serving environment where the product will be deployed. What I am proposing is an environment aware test to be included in the suite of many other tests.

Quota regression testing on Google Cloud

Google Cloud already provides services that you can use to easily incorporate this feature. This is more of a systems architecture practice that you can exercise.

The Service Consumer Management API provides the tools you need to create your own quota regression test. Take for example the ConsumerQuotaLimit Resource that’s returned via the list api. For the remainder of this discussion, let’s assume an environment setup such as this:

Diagram demonstrating an extremely simple deployment pipeline for a resource provider.

In the diagram above, we have a simplified deployment pipeline:

Developers submit code to some repository
The Cloud Build build and deployment trigger gets fired

Tests are run
Deployment images are pushed if the prerequisite steps succeed

Images are pushed to their respective environments (in this case build to dev, and previous dev to prod)
Quotas are defined for the endpoints on deployment
Cloud Load Balancer makes the endpoints available to end users

Quota limits

With this mental model, let’s hone in on the role quotas play in the big picture. Let’s assume we have the following service definition for an endpoint called “FooService”. The service name, metric label and quota limit value are what we care about for this example.

gRPC Cloud Endpoint Yaml Example

code_block: <ListValue: [StructValue([('code', 'type: google.api.Service\r\nconfig_version: 3\r\nname: fooservice.endpoints.my-project-id.cloud.goog\r\ntitle: Foo Service gRPC Cloud Endpoints\r\napis:\r\n - name: com.foos.demo.proto.v1.FooService\r\nusage:\r\n rules:\r\n # ListFoos methods can be called without an API Key.\r\n - selector: com.foos.demo.proto.v1.FooService.ListFoos\r\n allow_unregistered_calls: true\r\n # GetFoo methods can be called without an API Key.\r\n - selector: com.foos.demo.proto.v1.FooService.GetFoo\r\n allow_unregistered_calls: true\r\n # UpdateFoo methods can be called without an API Key.\r\n - selector: com.foos.demo.proto.v1.FooService.UpdateFoo\r\n allow_unregistered_calls: true\r\nmetrics:\r\n - name: library.googleapis.com/read_calls\r\n display_name: "Read Quota"\r\n value_type: INT64\r\n metric_kind: DELTA\r\n - name: library.googleapis.com/write_calls\r\n display_name: "Write Quota"\r\n value_type: INT64\r\n metric_kind: DELTA\r\nquota:\r\n limits:\r\n - name: "apiReadQpmPerProject"\r\n metric: library.googleapis.com/read_calls\r\n unit: "1/min/{project}"\r\n values:\r\n STANDARD: 1\r\n - name: "apiWriteQpmPerProject"\r\n metric: library.googleapis.com/write_calls\r\n unit: "1/min/{project}"\r\n values:\r\n STANDARD: 1\r\n # By default, all calls are measured with a cost of 1:1 for QPM.\r\n # See https://github.com/googleapis/googleapis/blob/master/google/api/quota.proto\r\n metric_rules:\r\n - selector: "*"\r\n metric_costs:\r\n library.googleapis.com/read_calls: 1\r\n - selector: com.foos.demo.proto.v1.FooService.UpdateFoo\r\n metric_costs:\r\n library.googleapis.com/write_calls: 2'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fa47f7a6df0>)])]>

In our definition we’ve established:

Service Name: fooservice.endpoints.my-project-id.cloud.goog
Metric Label: library.googleapis.com/read_calls
Quota Limit: 1

With these elements defined, we’ve now restricted read calls to exactly one per minute for the service. Given a project number, (e.g., 123456789) we can now issue a call to the Consumer Quota Metrics Service to display the service quota.

Example commands and output.

code_block: <ListValue: [StructValue([('code', '$ alias gcurl=\'curl -H "Authorization: Bearer $(gcloud auth print-access-token)" -H "Content-Type: application/json"\'\r\n$ gcurl https://serviceconsumermanagement.googleapis.com/v1beta1/services/fooservice.endpoints.my-project-id.cloud.goog/projects/my-project-id/consumerQuotaMetrics'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fa47f7a60a0>)])]>

Response example (truncated)

code_block: <ListValue: [StructValue([('code', '{\r\n "metrics": [\r\n {\r\n "name": "services/fooservice.endpoints.my-project-id.cloud.goog/projects/123456789/consumerQuotaMetrics/library.googleapis.com%2Fread_calls",\r\n "displayName": "Read Quota",\r\n "consumerQuotaLimits": [\r\n {\r\n "name": "services/fooservice.endpoints.my-project-id.cloud.goog/projects/123456789/consumerQuotaMetrics/library.googleapis.com%2Fread_calls/limits/%2Fmin%2Fproject",\r\n "unit": "1/min/{project}",\r\n "metric": "library.googleapis.com/read_calls",\r\n "quotaBuckets": [\r\n {\r\n "effectiveLimit": "1",\r\n "defaultLimit": "1"\r\n }\r\n ]\r\n }\r\n ],\r\n "metric": "library.googleapis.com/read_calls"\r\n }\r\n …'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fa47f7a60d0>)])]>

In the above response, the most important thing to note is the effective limit for a given service’s metric. The effective limit is the limit being applied to a resource consumer when enforcing customer fairness as discussed earlier.

Now that we’ve established how to get the effectiveLimit for a quota definition on a resource per project, we can define the assertion of quota consistency as:

Load Test Environment Quota Effective Limit <= Serving Environment Quota Effective Limit

Having a test like this, you can then integrate that with something like Cloud Build to block the promotion of your image from the lower environment to your serving environment if that test fails to pass. That saves you from introducing regressive behavior from the new image into the serving environment that would otherwise result in an outage.

The importance of early detection

It’s not enough to alert on a detected quota regression and block the image promotion to prod. It’s better to raise alarms as soon as possible. If resources are lacking when it’s time to promote to production, you’re now faced with the problem of wrangling enough resources in time. This may not always be possible in the desired timeline; it’s possible that the resource provider needs to scale up its resources to handle the increase in quota. This is not always something that can just be done in a day. For example, is the service hosted on Google Kubernetes Engine (GKE)? Even with autoscale, what if the ip pool is exhausted? Cloud infrastructure changes, although elastic, are not instant. Part of production planning needs to account for the time needed to scale.

In summary, quota regression testing is a key component that should be added to the entire concept of handling overload and dealing with load balancing in any cloud service — not just Google Cloud. It is important for product stability with the dips and spikes in demands, which will inevitably show up as a problem in many spaces. If you continue to rely on human intervention to ensure consistency of your quota across your configurations, you will only guarantee that eventually, you will have an outage when that consistency is not met. For more on working with quotas, check out the documentation.

Earn Google Cloud swag when you complete the #LearnToEarn challenge

Mon, 27 Jun 2022 16:00:00 +0000

The MLOps market is expected to grow to around $700m by 2025¹. With the Google Cloud Professional Data Engineer certification topping the list of highest paying IT certifications in 2021², there has never been a better time to grow your data and ML skills with Google Cloud.

Introducing the Google Cloud #LearnToEarn challenge

Starting today, you’re invited to join the data and ML #LearnToEarn challenge - a high-intensity workout for your brain. Get the ML, data, and AI skills you need to drive speedy transformation in your current and future roles with no-cost access to over 50 hands-on labs on Google Cloud Skills Boost. Race the clock with players around the world, collect badges, and earn special swag!

How to complete the #LearnToEarn challenge?

The challenge will begin with a core data analyst learning track. Then each week you’ll get new tracks designed to help you explore a variety of career paths and skill sets. Keep an eye out for trivia and flash challenges too!

As you progress through the challenge and collect badges, you’ll qualify for rewards at each step of your journey. But time and supplies are limited - so join today and complete by July 19!

What’s involved in the challenge?

Labs range from introductory to expert level. You’ll get hands-on experience with cutting edge tech like Vertex AI and Looker, plus data differentiators like BigQuery, Tensorflow, integrations with Workspace, and AutoML Vision. The challenge starts with the basics, then gets gradually more complex as you reach each milestone. One lab takes anywhere from ten minutes to about an hour to complete. You do not have to finish all the labs at once - but do keep an eye on start and end dates.

Ready to take on the challenge?

Join the #LearnToEarn challenge today!

^{1. IDC, Market Analysis Perspective: Worldwide AI Life-Cycle Software, September 2021
2. Skillsoft Global Knowledge, 15 top-paying IT certifications list 2021, August 2021}

Introducing managed zone permissions for Cloud DNS

Tue, 14 Jun 2022 17:00:00 +0000

The Cloud DNS team is pleased to announce the Preview launch of managed zone permissions. Cloud DNS is integrated with the Identity and Access Management (IAM) service, which gives you the ability to control access to Cloud DNS resources and prevent unwanted access. This launch enables enterprises with distributed DevOps teams to delegate Cloud DNS managed zone administration to their individual application teams.

Up until now, Cloud DNS supported resource permissions at the project level only. This allowed for centralized management of IAM permissions at a higher level of granularity. Any user with a specific permission in a project would have that permission apply to all the resources under it. If the project contained multiple managed zones for instance, a single user with project access could make changes to the DNS records in any of the managed zones in that project. This created two challenges for some customers:

It resulted in the reliance on a centralized team to manage all the DNS zones and record creations. This is usually fine when the deployment sizes are small. However for customers with a large number of managed DNS zones, administration becomes a toil usually borne by the central team.
A single user could modify the DNS records in multiple managed zones, resulting in either broken DNS records or causing security issues.

With this launch, Cloud DNS will allow access controls at a finer granularity at a managed zone level. This allows admins to delegate responsibilities for managed zone operations to individual application teams and prevents one application team from accidentally changing the DNS records of another application. It also allows for a better security posture because only authorized users will be able to modify managed zones and better supports the principle of least privilege. The launch does not change the IAM roles and permissions; we’ve merely added additional granularity to use when needed. For more details on IAM roles for Cloud DNS, please see Access control with IAM.

This capability also helps when customers are using a Shared VPC environment. A typical Shared VPC setup has service projects that take ownership of a virtual machine (VM) application or services, while the host project takes ownership of the VPC network and network infrastructure. Often, a DNS namespace is created from the VPC network's namespace to match the service project's resources. For such a setup, it can be easier to delegate the administration of each service project's DNS namespace to the administrators of each service project (which are often different departments or businesses). Cross-project binding lets you separate the ownership of the DNS namespace of the service project from the ownership of the DNS namespace of the entire VPC network. This, coupled with managed zone permissions, ensures that the managed zone within the service project can only be administered by service project owners while allowing for the host project (and other service projects) to access the namespace from their respective projects.

For more details on how to configure and use managed zone permissions, please look at our documentation.

How Google Cloud is helping more startups build, grow, and scale their businesses

Thu, 02 Jun 2022 16:00:00 +0000

Today is our second-annual Google Cloud Startup Summit, and we couldn’t be more excited to connect with leaders, founders, and innovators from across the startup ecosystem to learn how they’re innovating and solving some of the world’s biggest challenges.

Startups come to Google Cloud to build their companies and products on our trusted, developer-friendly infrastructure, and to take advantage of our leading capabilities in areas like artificial intelligence (AI), machine learning (ML), and analytics. In January, we expanded our support for early-stage startups to make it even easier for them to get up-and-running on Google Cloud by covering their cloud costs up to $100,000 USD for two years, providing dedicated Startup Success Managers to assist along their journeys, and much more.

We’re excited today to announce several new offers, resources, and programs that will help even more startups build and grow their businesses.

Expanding early-stage financial support to include bootstrapped startups

The first of these new offers is that self-funded startups who apply to our Google for Startups Cloud Program can now receive $2,000 USD in Google Cloud credits to use over two years to help fund the development of proof-of-concepts and showcase their products to investors, talent, and customers.

In addition, bootstrapped startups may also qualify for discounts on Google Domains and Google Workspace, giving them the tools they need to help build their companies and products, and collaborate with their teams.

We'll update our Google for Startups Cloud Program website in the coming weeks with more information and details about this offer.

Setting founders up for success with new Startup School workshops

We also know that an innovative product or vision can help startups stand out, particularly in the early days of development. That’s why in 2020, Google for Startups created Startup School — a comprehensive training program designed to equip early-stage startups and founders with the tools, products, and knowledge needed to support their growing company. The curriculum provides education on growth, technologies, and products, and covers topics like getting started with Google Cloud, making technical hires, preparing a data pipeline, and setting business objectives and key results (OKRs).

As an extension of this work, this fall we’ll be offering new, live workshops on a monthly basis that will be focused on topics like data analytics and application development, which will help startups build faster with tools and technologies tailored to their needs. These live Startup School sessions will also give founders and teams the opportunity to ask Google experts questions directly and discuss common challenges.

Supporting diverse founders and inclusivity across the globe

We remain committed to supporting a diverse ecosystem of startups, so we’re thrilled to host, in partnership with Inicio Ventures, 30 leading LatinX investors, founders, and ecosystem leaders in Miami next week during the first-ever Google for Startups LatinX Leaders Summit. At the summit, members of the LatinX startup community will have the opportunity to connect with key figures in the space and work together to develop more resources and ways to support startups in the LatinX community so they can overcome barriers and succeed, in the cloud and beyond.

We also recently announced that Google for Startups has teamed up with Visible Hands to run a 20-week fellowship program to support the next wave of early-stage LatinX founders across the United States. In addition to hands-on support from industry and Google experts, the program will provide $10,000 in cash for every participant to help kickstart their ideas. Participants in the program will also have the opportunity to receive additional investments from Visible Hands, up to $150,000. Click here to learn more about this program and apply.

And to help startups prioritize equity and inclusion in their business from day one, we’re proud to have Google DEI and Product Inclusion leads Alexandra Garcia and Steph Boudreau lead a session during the Startup Summit today, outlining how founders can build inclusive products and teams from the start.

Celebrating startups across the ecosystem

No matter the industry, solution, or geography, we’re proud of how startups are using Google Cloud technologies to create impactful customer experiences.

Take web3 startup IoTeX, whose blockchain-based platform handles millions of transactions for machines, devices, and people — with 99.9% platform reliability — by using Google Kubernetes Engine (GKE) to absorb load during surges, so the company can continue growing at scale. Or 3co, which uses Google Cloud technologies like Compute Engine to scale compute power quickly, and Tensorflow to provide 3D technology experiences for global e-commerce companies.

In Singapore, retailTech startup Palexy is using Google Cloud AI/ML products like Vertex AI alongside GKE to quickly build, test, deploy, and manage their applications, which in turn helps helps Palexy deliver solutions that help retailers use data and AI to sustainably grow their business and improve the in-store shopping experience for customers.

And for France-based social media startup BeReal, creating an impactful customer experience means giving people a platform to become closer to the friends and family they care most about. Using Google Firebase as the foundation to build the prototype, the BeReal platform now uses Cloud Functions for Firebase and GKE, so Alexis and the BeReal team can build and deploy new platform functionalities with speed as the business scales.

The flexibility made possible by the cloud—and greater access than ever before to bleeding edge capabilities in AI, ML, and analytics—are opening new doors for startups to innovate and grow, and we're committing to helping them do so on Google Cloud.

We’re grateful to be on this journey with our startup customers, and we look forward to continuing to provide resources and facilitate discussions to better serve the specific needs in this space. Keep an eye out for our “Ask TK” blog series, launching later this June, in which Google Cloud CEO Thomas Kurian and other top thought leaders will address common questions and challenges faced by startups. You can also catch a few top questions, answered by Thomas, in today’s event.

To learn more about the Google for Startups Cloud Program, or to apply to get more support for your startup, please visit our program application page.

Snap Inc. adopts Google Cloud TPU for deep learning recommendation models

Tue, 31 May 2022 16:00:00 +0000

While many people still think of academic research when it comes to deep learning, Snap Inc. has been applying deep learning models to improve its recommendation engines on a daily basis. Using Google's Cloud Tensor Processing Units (TPUs), Snap has accelerated its pace of innovation and model improvement to enhance the user experience.

Snap’s blog Training Large-Scale Recommendation Models with TPUs tells the story of how the Snap ad ranking team leveraged Google’s leading-edge TPUs to train deep learning models quickly and efficiently. But there’s a lot more to the story than the how, and that’s what we’re sharing here.

Faster leads to better

Snap’s ad ranking team is charged with training the models that make sure the right ad is served to the right Snapchatter at the right time. With 300+ million users daily and millions of ads to rank, training models quickly and efficiently is a large part of a Snap ML engineer’s daily workload. It’s simple, really: the more models Snap’s engineers can train, the more likely they are to find the models that perform better—and the less it costs to do so. Better ad recommendation models translate to more relevant ads for users, driving greater engagement and improving conversion rates for advertisers.

Over the past decade, there has been tremendous evolution in the hardware accelerators used to train large ML models like those Snap uses for ad ranking, from general-purpose multicore central processing units (CPUs) to graphics processing units (GPUs) to TPUs.

TPUs are Google’s custom-developed application specific integrated circuits (ASICs) used to accelerate ML workloads. TPUs are designed from the ground up to minimize time to accuracy when training large models. Models that previously took weeks to train on other hardware platforms can now be trained in hours on TPUs—a product of Google’s leadership and experience in machine learning (dig into the technology in Snap’s blog).

Benchmarking success

Snap wanted to understand for itself what kind of improvements in training speed it might see using TPUs. So, the Snap team benchmarked model training using TPUs versus both GPUs and CPUs, and the results were impressive. GPUs underperformed TPUs in terms of both throughput and cost, with a reduction in throughput of 67 percent and an increase in costs of 52 percent when using GPUs. Similarly, TPU-based training drastically outperformed CPU-based training for Snap’s most common models. For example, when looking at their standard ad recommendation model, TPUs slashed processing costs by as much as 74 percent while increasing throughput by as much as 250 percent—all with the same level of accuracy.

Because TPU embedding API is a native and optimized solution for embedding-based operations, it performs embedding-based computations and lookups more efficiently. This is particularly valuable to recommenders, which have additional requirements such as fast embedding lookups and high memory bandwidth.

Benefits across the board

For Snap’s ad ranking team, those improvements translate into tangible workflow advantages. It’s not unusual for Snap to have a month’s worth of data that includes all the logs of users who were shown particular ads and a record of whether they interacted with an ad or not. That means it has millions of data points to process, and Snap wants to model them as quickly as possible so it can make better recommendations going forward. It’s an iterative process, and the faster Snap can get the results from one experiment, the faster its engineers can spin up another with even better results—and they’d much prefer to do that in hours rather than days.

Increased efficiency and velocity benefit Snapchatters, too. The better the models are, the more likely they are to correctly predict the likelihood that a given user will interact with a particular ad, improving the user experience and boosting engagement. Improved engagement leads to higher conversion rates and greater advertiser value—and given the volumes of ads and users Snap deals with, even a one percent improvement has real monetary impact.

Working at the leading edge

Snap is working hard to improve its recommendation quality with the goal of delivering greater value to advertisers and a better experience for Snapchatters. That includes going all-in on leading-edge solutions like Google TPUs that allow its talented ML engineers to shine.

Now that you know the whole story, see how Snap got there with the help of Google: Training Large-Scale Recommendation Models with TPUs.

Build your cloud skills with no-cost access to Google Cloud training on Coursera

Thu, 28 Apr 2022 16:00:00 +0000

Attracting talented individuals with cloud skills is critical to success, as organizations continue to adopt and optimize cloud technology. The lack of cloud expertise and experience is a top and growing challenge for businesses as they expand their cloud footprint and search for skilled talent.

To help meet this need, we are now offering access to over 500 Google Cloud self-paced labs made available on Coursera. A selected collection of the most popular self-paced labs, known as projects, are available at no cost for one month from April 28 - May 29, 2022. Learners can choose their preferred format to claim one month free access to either a top Google Cloud Project, course, Specialization or Professional Certificate.

What is a lab?

A lab is a learning experience where you complete a scenario based use case by following a set of instructions in a specified amount of time in an interactive hands-on environment. Labs are completed in the real Google Cloud Console and other Google Cloud products using temporary credentials, as opposed to a simulation or demo environment and take 30 - 90 minutes to complete (depending on difficulty level). Our goal is to enable you to apply your new skills and be effective immediately in real-world cloud technology settings.

Many of these labs, known in Coursera as projects, include a variety of tasks and activities for you to choose from to best fit your needs. Combine bite-size individual labs to create a personalized set of learning and upskilling with clear application in a sandbox environment.

Labs are available for all skill levels, and cover a wide range of topics:

Cloud essentials
Cloud engineering and architecture
Machine learning
Data analytics and engineering
DevOps

Here is a roundup of some popular and trending labs right now:

Getting Started with Cloud Shell and gcloud
Kubernetes Engine: Qwik Start
Introduction to SQL for BigQuery and Cloud SQL
Migrating a Monolithic Website to Microservices on Google Kubernetes Engine

Get a feel for the lab experience

Creating a Virtual Machine is one of our most popular labs, taking place directly in Google Cloud Console. In this beginner level project, you will learn how to create a Google Compute Engine virtual machine and understand zones, regions and machine types. It takes 40 minutes to complete and you’ll earn a shareable certificate.

As an example of more advanced content, Predict Baby Weight with TensorFlow on AI Platform requires experience to train, evaluate and deploy a machine learning model to predict a baby’s weight. The lab activities are completed in a real cloud environment, not in a simulation or demo environment. It takes 90 minutes to complete and you will earn a shareable certificate.

Kick off your no-cost learning journey today

For direct access to self-paced labs, we recommend that you get started by taking a look at Coursera’s Collection Page, where you can browse labs/projects by our most popular topics, or explore the full catalog to find the cloud projects that are right for your career goals by browsing Google Cloud ‘projects’ on Coursera.

The month of free Google Cloud learning on Coursera is available from April 28 - May 29, 2022, so join us to evolve your skill set and cloud knowledge.

Ready to start your learning Google Cloud at no-cost for 30 days? Sign up here.

What’s new in cloud-native apps?

Fri, 15 Apr 2022 20:00:00 +0000

Developers and IT operations pros of all stripes come to Google Cloud to build modern, cloud-first and cloud-native applications. Here’s the latest from Google Cloud on everything app dev, containers, Kubernetes, DevOps, serverless and open source, all in one place.

Week of Apr 11 - Apr 15, 2022

Listen to a Prodcast
Google’s SRE team has launched a “Prodcast” focusing on concepts from its SRE book. Available from wherever you get your podcasts.

Run Apache Spark on a modern container base
Dataproc, our managed version of Apache Spark, is now generally available on Google Kubernetes Engine (GKE), allowing you to create a Dataproc cluster and submit Spark jobs on a self-managed GKE cluster. Read all about it.

Loads of new runtimes in App Engine and Cloud Functions
Java, Ruby, Python and PHP developers, rejoice! You can now update or develop new App Engine apps and Cloud Functions using Java 17, Ruby 3, Python 3.10 and PHP 8.1.

BeReal shows you how modern app development is done
Social media company BeReal discusses how it uses Google Cloud services including Firebase, Cloud Functions and GKE to build its app.

Build fast without breaking things
In this three-part series, learn about the Supply-chain Levels for Software Artifacts (SLSA) framework designed to improve the integrity of your software packages and infrastructure. Start with, How to SLSA Part 1 - The Basics, then move on to part 2 and part 3.

Week of Apr 4 - Apr 8, 2022

How to migrate a container from a VM to Cloud Run
With Cloud Run, you can migrate a legacy VM to a container and save money – even if you don’t know Kubernetes. This video shows you how.

Receive Error Reporting notifications through Slack and Webhooks
Error Reporting can analyze, aggregate, and notify DevOps teams about crashes that happened in their cloud services, right to their preferred channels. Learn more in this blog.

Cloud-native architecture is in the cards at NCR
Earlier this year, NCR Authentic Cards talked about how it built a transaction processing platform on Google Cloud. NCR and its consulting partner Opus Systems are back for part two of the migration story, taking a detailed look at all the components that went into the cloud-based architecture.

How to easily share a service with Cloud Run
Have you ever written a script that you wanted to make available to others? Cloud Run makes it easy to deploy a processing service quickly and easily. In this blog post, Developer Advocate Laurent Picard creates an image processing service that generates coloring pages, then makes it available to others — all in under 200 lines of Python and JavaScript. Follow along in this tutorial.

Week of Mar 28 - Apr 1, 2022

Another cool thing you can do with Cloud Functions
Got data you want to ingest from Cloud Storage to BigQuery? Cloud Functions can help with that. This tutorial shows you how.

Add custom severity levels to Cloud Monitoring alert policies
Not all alerts are created equal. In this blog post, learn how to add static and dynamic severity levels to a Cloud Monitoring alert policy, with enhanced notification channels including email, webhooks, Cloud Pub/Sub and PagerDuty.

Learn how to use CPU allocation controls in Cloud Run
Last fall, we added “always-on CPU” capabilities to Cloud Run, making it a better fit for running background- and other asynchronous-processing tasks. In this post, Developer Advocate Wesley Chun uses a weather alerting app to demonstrate how to use the feature, and along the way, reduces the app’s average user response latency by over 80%.

Week of Mar 21 - Mar 25, 2022

Get Going with latest Go 1.18 release
With the release of version 1.18, the Go programming language now includes support for generic code using parameterized types, integrated fuzz testing, and a new Go workspace mode that makes it simple to work with multiple modules. Learn more here.

Week of Mar 14 - Mar 18, 2022

Create EventArc triggers with Terraform
In addition to the Google Cloud Console or gcloud, you can also use a Terraform resource to create an Eventarc trigger. Mete Atamel shows you how.

Scaling to new markets with Cloud Run
French publisher Les Echos Le Parisien Annonces switched from dedicated on-prem infrastructure to Cloud Run to supplement its main news site with regional variations. Les Echos shares its website architecture here.

The serverless way to celebrate Pi Day
In honor of Pi Day, Google Cloud Developer Advocate Emma Haruka Iwao shows you how to use the new Cloud Functions (2nd gen) to calculate π — serverlessly.

Week of Mar 07 - Mar 11, 2022

Rhode Island moves to Google Cloud-based job board
When the pandemic hit, the State of Rhode Island moved its workforce development operations entirely online on a foundation of Google Workspace and Google Cloud resources, including Firestore, Cloud Functions, and Kubernetes, among others. Check out how they did it.

Containerized microservices at Lowe’s
Lowe’s already told us how they use SRE. They’re at it again, describing how they built an e-commerce website using a containerized microservices architecture and Kubernetes, with Istio for service mesh and Cloud Operations for good measure.

Cruise AVs hit the road with Google Cloud services
Autonomous Vehicle (AV) startup Cruise detailed how it’s using data analytics and machine learning on a foundation of Google Kubernetes Engine (GKE) and other services to develop and test its self-driving cars. Read the guest post.

L’Oréal’s data analytics gets a makeover with serverless
We’re hurtling toward a programmable cloud — a world where developers use cloud-native serverless tools like Cloud Functions to quickly prototype and build powerful, data-driven business insights. L’Oréal is a great example.

Better telemetry for your Anthos clusters
Anthos Service Mesh Dashboard is now available (public preview) on the Anthos clusters on Bare Metal and Anthos clusters on VMware. Now, you can get out-of-the-box telemetry dashboards to see a services-first view of your application on the Cloud Console.

Instrument your Java apps
With the new version of the Google Cloud Logging Java library, you can wire your application logs with more information — without adding a single line of code.

Visualize metrics from Cloud Spanner
Building an app on top of Cloud Spanner but can’t assess how well it’s operating? The new OpenTelemetery receiver for Cloud Spanner provides an easy way for you to process and visualize metrics from Cloud Spanner System tables, and export these to the APM tool of your choice. Read more here.

Week of Feb 28 - Mar 4, 2022

Introducing Cloud SDK
The rebranded Cloud SDK is a collection of all the libraries and tools (including Google Cloud CLI) you need to interact with Google Cloud products and services. Learn more here.

Cloud CLI, meet Terraform
Google Cloud CLI’s new Declarative Export for Terraform allows you to export the current state of your Google Cloud infrastructure into a descriptive file compatible with Terraform (HCL) or Google’s KRM declarative tooling, and is now available in preview.

Knative graduates to incubating project
Congratulations to Knative, which has been accepted by the Cloud Native Computing Foundation, or CNCF, as an incubating project, enabling the next phase of serverless architecture.

We manage Prometheus so you don’t have to
Google Cloud Managed Service for Prometheus is now generally available! Get all the benefits of open source-compatible monitoring with the ease of use of Google-scale managed services. Learn more here.

Free your mainframe data with data-first digitization

Mon, 14 Feb 2022 17:00:00 +0000

For many enterprises, the venerable mainframe is home to decades’ worth of data about the company’s customers, processes and operations. And it goes without saying that the business would like access to that mainframe data — to report on it, to analyze it with big data analysis tools, or to use it as the basis of new machine learning and artificial intelligence initiatives.

At Google Cloud, we are eager to work with organizations to help them transform their mainframe assets for the cloud era. Of course, we can help them modernize their mainframe applications by migrating them to the cloud. At the same time, working with partners and customers, we’ve developed another, more lightweight approach that can help them start to leverage the cloud for their mainframe assets much more quickly than performing a full-fledged migration. We call this approach data-first digitization.

In this rapidly evolving digital ecosystem, it’s imperative to understand the difference between ‘modernization’ and ‘digitization.’ With modernization you start with the current state and look forward, and rely on mainframe application migration approaches such as rehosting (emulation), refactoring (automated code transformation), reengineering — or simply replacing a custom application with a commercial package. With digitization, you start with the future state that you want to achieve, and work back to what is required to get there.

Click to enlarge

This data-first digitization approach includes a mainframe data-first integration framework comprising in-house and partner products and tools to migrate heterogeneous data sources from the mainframe to Google Cloud Storage. Once mainframe data has been copied to Cloud Storage, it can then be integrated and leveraged by Google Cloud tools such as BigQuery, AI and machine learning prodcuts and Smart and Stream analytics platforms. The integration framework covers both bulk batch data transfers and real-time data replication (change data capture).

Click to enlarge

Data-first digitization is based on the tenet that ‘applications are transient, data is permanent.’ By bringing data first to Google Cloud instead of traditional ways of modernizing applications (for example, with Gartner’s 7 options to Modernize), this allows organizations to leapfrog to new business models, use cases and innovative ways to serve end customers. For example:

Making decisions with smart and stream analytics platforms and AI/ML engines. These tools need data to make decisions. Google is a pioneer in extracting information and value from the raw structured and unstructured data, and this approach opens up mainframe data for use by BigQuery and AI/ML models.

Building new reporting applications. With access to mainframe data, you can use Google cloud products like Looker and Appsheet to build net-new reporting applications, expediting the process of retiring mainframe reporting applications, and accelerating your overall transformation.

In our experience, taking a data-first digitization approach to your mainframe offers a number of benefits:

Faster time-to-business: Because data-first modernization is built on existing products, the implementation cycle is much shorter.
Less capital investment: You spend your time integrating products, not developing applications.
Minimized risk: Data-first integrates with existing, proven and reliable Google Cloud products.
Faster overall mainframe transformation: When you shift your modernization center of gravity from the application to the data, you look at mainframe applications from a business perspective instead of just “keeping the lights on.” As a result, only the most business-critical applications are modernized and many support applications can be decommissioned, accelerating your transformation journey.

Taking a data-first approach to digitization is still relatively new, but we’re heartened by customers’ early successes. Watch this space for additional insights, reference architectures and technical white papers around data-first. And if you think this approach may be right for you, reach out to mainframe@google.com.

Learn more:

Getting Started with Google Cloud Logging Python v3.0.0

Mon, 07 Feb 2022 18:00:00 +0000

We’re excited to announce the release of a major update to the Google Cloud Python logging library.

v3.0.0 makes it even easier for Python developers to send and read logs from Google Cloud, providing real-time insights into what is happening in your application. If you’re a Python developer working with Google Cloud, now is a great time to try out Cloud Logging!

If you're unfamiliar with the `google-cloud-logging` library, getting started is simple. First, download the library using pip:

code_block: <ListValue: [StructValue([('code', '$ pip install "google-cloud-logging>=3.0.0"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fa47f34f4c0>)])]>

Now, you can set up the client library to work with Python's built-in `logging` library. Doing this will make it so that all your standard Python log statements will start sending data to Google Cloud:

code_block: <ListValue: [StructValue([('code', '# set up the Google Cloud Logging python client library\r\nimport google.cloud.logging\r\nclient = google.cloud.logging.Client()\r\nclient.setup_logging()\r\n\r\n# use Python\'s standard logging library to send logs to GCP\r\n\r\nimport logging\r\nlogging.warning("Hello World")'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fa47fb2eaf0>)])]>

We recommend using the standard Python `logging` interface for log creation, as demonstrated above. However, if you need access to other Google Cloud Logging features (reading logs, managing log sinks, etc), you can use `google.cloud.logging` directly:

code_block: <ListValue: [StructValue([('code', 'import google.cloud.logging\r\n\r\nclient = google.cloud.logging.Client()\r\nlogger = client.logger(name="log_id")\r\n\r\nclient.list_entries(max_size=5) # read logs from GCP\r\nlogger.log("hello world", resource={"type":"global", "labels":{}}) # write log to GCP'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fa47fb2e130>)])]>

Here are some of the main features of the new release:

Support More Cloud Environments

Previous versions of google-cloud-logging supported only App Engine and Kubernetes Engine. Users reported that the library would occasionally drop logs on serverless environments like Cloud Run and Cloud Functions. This was because the library would send logs in batches over the network. When a serverless environment would spin down, unsent batches could be lost.

v3.0.0 fixes this issue by making use of GCP’s built in structured JSON logging functionality on supported environments (GKE, Cloud Run, or Cloud Functions). If the library detects it is running on an environment that supports structured logging, it will automatically make use of the new StructuredLogHandler, which writes logs as JSON strings printed to standard out. Google Cloud’s built-in agents will then parse the logs and deliver them to Cloud Logging, even if the code that produced the logs has spun down.

Structured Logging is more reliable on serverless environments, and it allows us to support all major GCP compute environments in v3.0.0. Still, if you would prefer to send logs over the network as before, you can manually set up the library with a CloudLoggingHandler instance:

code_block: <ListValue: [StructValue([('code', 'from google.cloud.logging.handlers import CloudLoggingHandler\r\nfrom google.cloud.logging_v2.handlers import setup_logging\r\n\r\n# explicitly set up a CloudLoggingHandler to send logs over the network\r\nhandler = CloudLoggingHandler(client)\r\nsetup_logging(handler)\r\n\r\nimport logging\r\nlogging.warning(“Hello World”)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fa46c6b9be0>)])]>

Metadata Autodetection

When you troubleshoot your application, it can be useful to have as much information about the environment as possible captured in your application logs. `google-cloud-logging` attempts to help in this process by detecting and attaching metadata about your environment to each log message. The following fields are currently supported:

`resource`: The Google Cloud resource the log originated from

for example, Functions, GKE, or Cloud Run

`httpRequest`: Information about an HTTP request in the log’s context

Flask and Django are currently supported

`sourceLocation` : File, line, and function names
trace, spanId, and traceSampled: Cloud Trace metadata

Supports X-Cloud-Trace-Context and w3c transparent trace formats

The library will make an attempt to populate this data whenever possible, but any of these fields can also be explicitly set by developers using the library.

code_block: <ListValue: [StructValue([('code', 'logging.info("hello", extra={\r\n "labels": {"foo": "bar"},\r\n "http_request": {"requestUrl": "localhost"},\r\n "trace": "01234"\r\n})'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fa46c6b9160>)])]>

JSON Support in Standard Library Integration

Google Cloud Logging supports both string and JSON payloads for LogEntries, but up until now, the Python standard library integration could only send logs with string payloads.

In `google-cloud-logging` v3, you can log JSON data in two ways:

1. Log a JSON-parsable string:

code_block: <ListValue: [StructValue([('code', 'import logging\r\nimport json\r\n\r\ndata_dict = {"hello": "world"}\r\nlogging.info(json.dumps(data_dict))'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fa46c6b9d00>)])]>

2. Pass a `json_fields` dictionary using Python logging's `extra` argument:

code_block: <ListValue: [StructValue([('code', 'import logging\r\n\r\ndata_dict = {"hello": "world"}\r\nlogging.info("message field", extra={"json_fields": data_dict})'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fa46c6b9a90>)])]>

Next Steps

With version v3.0.0, the Google Cloud Logging Python library now supports more compute environments, detects more helpful metadata, and provides more thorough support for JSON logs. Along with these major features, there are also user-experience improvements like a new log method and more permissive argument parsing.

If you want to learn more about the latest release, these changes and others are described in more detail in the v3.0.0 Migration Guide. If you’re new to the library, check out the google-cloud-logging user guide. If you want to learn more about observability on GCP in general, you can spin up test environments using Cloud Ops Sandbox.

Finally, if you have any feedback about the latest release, have new feature requests, or would like to make any contributions, feel free to open issues on our GitHub repo. The Google Cloud Logging libraries are open source software, and we welcome new contributors!

5 key metrics to measure Cloud FinOps impact in your organization in 2022 and beyond

Wed, 08 Dec 2021 17:00:00 +0000

Value of Establishing a Baseline for Metrics

As organizations continue to leverage cloud investments to drive their business growth and top line revenue, business, finance, and technology executives need to become increasingly connected in their efforts to deliver strong business outcomes. More than ever before, executives need to quantify the value of their investments in business and technology capabilities. As such, business and IT leaders need a set of value metrics that cover both operational and strategic outcomes, as well as risks and opportunities. Nevertheless, operational IT metrics are often disconnected from business outcomes, and executives need to establish the connection between technology and business outcomes to facilitate a meaningful dialogue between IT and business leaders.

Like many aspects of IT operations, metrics and KPIs are commonly a journey. Organizations typically start this journey with unit metrics focusing on cloud costs and eventually progress toward a set of clearly defined business value metrics.

As we define the set of metrics across the five key building blocks of Cloud FinOps, which include Accountability & Enablement, Measurement & Realization, Cost Optimization, Planning & Forecasting, and Tools & Accelerators, we ensure that these metrics are easily measurable and commonly attainable across the organizations that are on the journey of digital transformation.

Accountability and Enablement Metric

The Accountability and Enablement pillar is foundational to building a culture of cost and value awareness and charts the course for both the process and cultural transformation journey in cloud FinOps. The primary goal is to help drive financial accountability and accelerate business value realization by streamlining IT financial processes and enabling frictionless cloud governance. Enablement empowers IT, finance, and business teams with training to better understand cloud resources and strategies to efficiently deploy and manage them. Driving accountability and enablement starts with a charter and core governance policies, and then guides the transformation of processes that link finance, IT, and business owners.

We recommend adopting Cloud Enablement % as the standard metric for the accountability and enablement pillar, measured by the # of business leaders trained and certified / total # of business leaders in the organization.

This is an important metric as many organizations fail to adopt Cloud FinOps because of lack of awareness and training. This cloud enablement metric will help business leaders better understand the value of cloud and how it can be an enabler to drive sustainable business outcomes.

The cloud enablement metric can easily be implemented through a set goal based on the number of identified business leaders across the organization. With that said, it is important to utilize the Pareto principle of 80/20 rule here and identifying the key business leaders who are extensively consuming services on the cloud should be the primary focus. Google Cloud recently published a new Cloud Digital Leader certification that is aimed for business leaders and executives. By obtaining the Cloud Digital Leader certification, it ensures the individual is well-versed in basic cloud concepts and can demonstrate a broad application of cloud computing knowledge in a variety of applications and how Google Cloud services can help achieve desired business goals. In addition, the FinOps Foundation also provides training and certification to practitioners in a large variety of cloud, finance and technology roles to validate their FinOps knowledge and enhance their professional credibility.

Ultimately, we see that a target goal of over 70% of business leaders achieving the Cloud Digital Leader certification can significantly drive alignment and adoption of Cloud FinOps across the organization and leverage cloud technologies as an enabler to create sustainable business outcomes.

Measurement and Realization Metric

Foundational to any good process is accurate data and effective metrics, which starts with the notion of cloud costs visibility and traceability. This is driven by proper resource hierarchy and project structure standards and supported by a labeling and tagging data architecture behind your organization’s use of cloud resources. While many common tags include IT-driven designators such as application, environment, and project, it is important to design a direct connection to your P&L into your labeling and tagging architecture, by including cost centers or the chart of accounts as tags. Furthermore, automation of tagging ensures that all taggable resources are deployed with consistent and accurate labels and feed FinOps metrics with reliable data.

Establishing consistent and detailed tagging is essential to attributing cloud resources not only to specific products and projects, but also to detailed cost centers aligned with lines of business and associated P&Ls. In order to establish a full chargeback of typical cloud services, customers will need to attribute costs associated with 3 types of cloud resources. The first and most straightforward will be attributing taggable resources (compute instances, databases, and storage buckets) that are aligned to a specific P&L, such as where a given application is solely consumed by one line of business.

The second situation is where taggable resources are shared across multiple lines of business. Many customers will resort to using traditional P&L allocation models, such as using business revenue or headcount of the associated business units to divy up the costs. In order to more accurately allocate shared application costs, leading-edge customers use elements in their cloud microservices architecture, such as API calls, to specifically measure the relative consumption of shared applications.

The third type of cloud resources are those that cannot be tagged. Common examples include support, networking costs, and third party Marketplace costs. Here, traditional P&L allocation models as described above (using headcount or revenue) are commonly used. Some customers will use the relative distribution of their taggable resource allocations to appropriate non-taggable costs to their business units, while some types of costs, such as networking, are allocated based on API calls.

To measure the effectiveness of the Measurement & Realization pillar of cloud FinOps across these three types of cloud resources, we recommend adopting Cloud Allocation % as the lead metric. This metric is measured as the percentage of total cloud costs (taggable resources consumed by individual business units, taggable resources shared across multiple business units, and non-taggable resources) allocated to responsible business owners.

This metric can be used to support both Showback (cloud costs held in a central IT P&L but reported to business units) and Chargeback models (cloud costs fully charged to business unit P&Ls), and reflects the underlying effectiveness and accuracy of resource tagging and cost attribution to business units. Cloud Allocation % can be implemented in two ways. The basic implementation would qualify costs apportioned by any P&L metric (either by consumption or by traditional P&L allocation such as by revenue or headcount). The more advanced implementation of this metric would only qualify those resources (both specific and shared) that use either tagging or API calls to measure consumption and attribute associated costs to business units.

Customers evolving from a Crawl to a Walk stage of implementation will seek to allocate 70% or more of their total cloud costs, while those moving to a Run state will achieve 90% or greater cost attribution based on direct consumption measures.

Cost Optimization Metric

Cloud cost optimization is not just about cutting costs—it's about knowing where to spend your money to maximize the business value. It is an iterative and continuous process that provides a consistent methodology to visualize and manage cloud consumption in a most cost effective way. Success in cost optimization can result not only in significant reductions of cloud spend, but sometimes also in improved application performance to manage higher traffic (user requests per seconds or transaction processed) within the same cost envelope.

It is important for an organization to automate reports generated by ingesting billing usage and cost data as well as recommendations generated for optimizations. These optimizations reflect the potential savings (also known as unrealized savings) which allows the team to prioritize implementations to realize the cost savings.

Typically potential savings contains adoption of:

Pricing optimizations like Committed Use Discounts (resource-based and spend-based), BigQuery reservations, etc.
Resource optimizations of wasteful resources (including aged snapshots, idle instances, and over-sized databases) that don't provide any business value.

Capturing this metric is important as it allows the organization to keep a pulse on inefficiencies that exist in the organization and allows businesses to focus on achieving cost savings thereby capturing true value of running their workloads in the cloud.

The cost optimization metric can be implemented by integrating Recommendation Hub in your FinOps workflows. Recommendations Hub is part of Active Assist that contains a portfolio of intelligent tools and capabilities to help you optimize your workloads with minimal effort. It surfaces a summary of all recommendations across your projects along with potential cost savings ($) so you can prioritize your cost optimization effort. We have seen customers realize savings by taking action on recommendations generated by idle VM recommender, Committed Use Discount recommender, VM machine type recommender and many more.

Ultimately we see customers achieving realized savings of over 90% on total cloud service optimizable. We have seen customers reinvest these savings into creating differentiated products and offerings and improving their customer experience, thus accelerating business value realization from the cloud.

Planning and Forecasting Metric

Financial planning is a foundational capability within finance organizations that will directly influence each company’s capabilities of cloud computing forecast accuracy. Financial planning focuses on accurately forecasting financial metrics that are set on an annual basis to guide the company’s financial objectives. The annual plans are measured on a quarterly basis and adjusted based on performance throughout the year; the forecast performance is monitored on a monthly basis to help influence operational results.

Planning and forecasting cloud computing costs is typically the responsibility of the team responsible for cloud operations. Operational forecast planning is based on consumption workload plans, historical trajectory, seasonality and leading indicators. Transformational projects also create material risks to forecast accuracy.

Establishing accurate financial forecasting in the cloud spend requires rethinking traditional approaches to asset depreciation run-outs and trend-based forecasting of maintenance and licensing costs. Using workload-specific forecasting models that leverage a combination of trend-based models for steady-state workloads, driver-based models for scaling applications, as well as monthly variance analysis can greatly improve the accuracy of dynamic cloud needs.

Capturing and measuring forecast accuracy enables companies to understand if they do what they plan. Companies get what they measure and so by measuring and discussing variances to forecast accuracy it enables better control of cloud spend allocations.

Cloud computing forecast accuracy should be included as a topic that Finance and Cloud operations teams discuss at least monthly. The cloud operations team should monitor forecast trajectory during the month and evaluate adjustments when they identify unexpected shifts.

An effective forecast accuracy is one that avoids surprises to company executives and investors. Cloud computing often has more variability and seasonality than depreciation of capex from on prem environments. Coordinating project and sprint agile management can help avoid surprises. If a development change creates an unexpected jump in spending then change management processes should be reviewed to avoid future surprises.

Tools and Accelerators Metric

Employing proper tools and accelerators are important to fully benefiting from FinOps practices. In earlier stages, companies may have limited their ability to report detailed analysis of cloud spend. As practices mature and improve, labeling and tagging of resources proves valuable to understanding costs for specific projects/teams and for building unit cost metrics.

These capabilities can become even more powerful through automated monitoring of resources that offers insights on spend, value, compliance and recommendations.

Therefore the recommended measure of Tools & Accelerators maturity is to evaluate the # of automated recommendations that have been implemented as a % of total list of automated recommendations generated that results in cost savings

This is an important metric because as the organization onboards newer workloads to the Cloud environment, lack of robust actionable recommendations and monitoring can lead to increased cloud waste. This has been a key component prohibiting organizations from realizing the total value of their cloud investment.

Customers starting on their tool maturity journey can leverage Google’s out of the box recommendations Hub to get started. The Recommendation Hub is a place in the Google Cloud Console where you can view, prioritize, and apply these recommendations. Some examples include VM right sizing recommendations, BQ slot optimizations, Committed use Discount etc, Idle resource recommendations. This can further be integrated into any existing enterprise tooling using the recommendations API. As organizations mature, they can leverage Cloud Monitoring to create advanced recommendations based on custom business logic.

Ultimately, we see that a target goal of over 50% of automated recommendations implemented as the tooling for surfacing recommendations matures and this will ensure that the organization can minimize and eliminate cloud waste to maximize value from cloud investment.

Bringing this together with a Cloud FinOps Dashboard

As technology and business goals continue to evolve over time, it is essential to establish a process where the Cloud FinOps metrics are continuously reviewed whenever the goals change. Furthermore, it is important to note that not all organizations need to achieve the “Run” state of the identified metrics target. The metrics are means to achieve the business outcomes based on the organization’s priorities. By collaborating with cross-functional teams to quantify and measure the impact of the Cloud FinOps metrics, executive leaders can quickly obtain buy-in, highlight common-shared goals, and move fast.

At Google Cloud, we have developed solutions to help our customers build a Cloud FinOps Dashboard to capture these metrics to drive a culture of change and equip the transformation and business leaders with the tools to share and track the results of the key metrics. Successful adoption of the Cloud FinOps metrics enable organizations to focus on the business outcomes and the dashboard provides a meaningful feedback loop to report on the impact and drive visibility across the organization.

So, where are you now in your FinOps journey, and how do you move beyond the challenges ahead? Google can help you start the conversation and accelerate your path to maximizing business value with the cloud.

No matter where you are on the cloud transformation journey, through an interactive session with Google, we can bring executives across the organization together to work toward a shared vision and a plan to accelerate and realize business value in the cloud. If you are interested in more information, please contact us.

^{Special thanks to Daniel Pettibone, Amitai Rottem, Bruce Warner, Jon Naseath, and Nihar Jhawar for co-authoring and contributing to this blog post and the members of the FinOps Foundation including J.R. Storment, Vas Markanastasakis, Anders Hagman, John McLoughlin, Mike Bradbury, and Rich Hoyer for providing their domain expertise and continuous support to this important cloud FinOps topic.}

Decoding Cloud FinOps to accelerate digital transformation

Wed, 27 Oct 2021 16:00:00 +0000

Accelerating digital transformation

Digital transformation is what propels businesses and industries forward. Organizations of all sizes—from startups to global enterprises—focus on digital transformation not only to make scaled improvements, but also to drive significant change and fully embrace the digital age. The pandemic has jump started and pushed many organizations into full gear to digitize their business models and transform with increased business agility, resiliency, and velocity, while driving new innovation and business values for the customers.

However, according to the Boston Consulting Group, only about 30% of companies navigate a digital transformation successfully¹. Many large scale digital transformation programs failed because of lack of clear business priorities, top-down executive sponsorship, or dedicated resources and commitment to see it through.

Laying the foundation for digital transformation success

Digital transformation drives foundational change in how an organization operates, optimizes internal resources, and delivers value to customers; however, this doesn’t just happen overnight. Digital transformation requires a programmatic approach through an incremental yet agile, cost-effective, value-driven, and sustainable strategy to drive successful transformation across the organization.

One of the critical factors foundational to success is Cloud FinOps (Cloud Financial Operations). Cloud FinOps is an operational framework and cultural shift that brings technology, finance, and business together to drive financial accountability and accelerate business value realization through cloud transformation. In the context of Digital Transformation, it requires new ways of working and operating models to drive behaviors and cultural change that foster cross-functional collaboration, drive accountability, provide greater cost transparency, and promote a blameless culture.

Most importantly, Cloud FinOps serves as an enabling function to drive successful digital transformation programs and enable business agility by breaking down the boundaries between technology, finance, and business teams. Through this cross-functional team collaboration, technology leaders partner with finance and business leaders to better understand the technology investments to create sustainable business outcomes. By doing so, business priorities become more clear and the focus shifts to value creation, customer-centricity, and innovation.

As such, companies are reinventing their business models to fund value streams and connect cloud technology investments to strategic business outcomes. With the increased visibility of the cloud costs, finance teams are also gaining greater accuracy in tracking cloud spend against budgets. Organizations can align the TCO of the technology services to the value metrics to make better informed future investment decisions and forecast demand.

Cloud FinOps to accelerate business value realization

The successful deployment and implementation of Cloud FinOps building blocks will enable organizations to accelerate digital transformation beyond cost savings, including the ability to:

Accelerate business value realization and innovation
Drive financial accountability and visibility
Optimize cloud usage and cost efficiency
Enable cross organizational trust and collaboration
Prevent cloud spend sprawl
Break down of departmental silos

Organizations that are successful in digital transformation most often have established processes to measure and track business value. One of the key building blocks of Cloud FinOps is “Measurement & Realization.” By establishing a robust value measurement approach to track and monitor the business value metrics toward business goals, we are bringing technology, finance, and business leaders together through the discipline of Cloud FinOps to show how digital transformation is enabling the organization to create new innovative capabilities and generate top-line revenue.

Business value metrics fall across several factors: cost efficiency, resiliency, velocity, innovation, and sustainability. We suggest assigning KPIs to the following metric categories:

Cost efficiency: Measure cost efficiency through infrastructure savings, migration, and support costs. Customers will commonly start with metrics such as cost of compute and storage per day-week-month, and evolve to unit metrics such as cost per customer served or cost per transaction, where the cost of an application stack is aligned to customer drivers.

Resiliency: Enhance operational resiliency with improvement in service quality and security risk posture. Traditional measures such as system service level and the frequency and duration of critical downtime events are effective measures of IT durability. Customers can also augment these metrics by associating a cost per minute of downtime events, reflecting not only the direct impact of these events but opportunity costs as well.

Velocity: Decrease time to market by accelerating fluidity in product and service delivery. By moving to a cloud-based microservices architecture, customers commonly achieve benefits of increasing software release frequency, as well as being able to run many more test scenarios prior to release, resulting in higher quality code. As an example, our recent Google’s State of DevOps Report 2021, shows that elite performers have 973x more frequent code deployment and release frequency than the low performers.

Innovation: Enable a culture of rapid experimentation to drive innovation and cloud transformation. With cloud technology, companies can avoid the financial constraints of fixed cost investments and lengthy procurement lead times. As a result, the marginal cost of experimentation and time from ideation to experimentation can drop significantly while the number of experiments per unit of time can grow dramatically.

Sustainability: Embed true environmental and social sustainability metrics across the organization by adopting a circular economy strategy and building sustainability into everything we do - from running applications on zero net emissions virtual machines to reducing carbon footprint with enhanced productivity and collaboration services. According to Accenture, companies with average on-premise to cloud migrations can drive 65% energy reduction and carbon emission reduction of 84%¹.

Getting started

The Cloud FinOps journey starts with defining or updating your metrics. Since business goals and strategic imperatives will likely change over time, it is important to review the Cloud FinOps metrics whenever the goals change. The review of metrics should include the changes in business goals when there are changes in the internal priorities of the team. Executive leaders need to identify dependency relationships between technology and business outcomes to improve the impact of metrics on decision making and to better prioritize and invest in evolving business and technology capabilities. Defining good metrics is not just about aligning to business goals and demonstrating value. It is also important to help prioritize strategic initiatives, guide effective resource allocation, and generate awareness across the organization to drive a shift in mindset with the new way of operating in the cloud.

The pandemic has accelerated the need for companies to modernize their digital capabilities. With technology-driven disruptions across all industries, it has never been more important for organizations to transform themselves, embrace an agile mindset, and make bold investments in cloud technology and capabilities to achieve sustainable business outcomes.

So, where are you now in your cloud FinOps journey, and how do you move beyond the challenges ahead? Google can help you start the conversation and accelerate your path to maximizing business value with the cloud.

Special thanks to Pathik Sharma, Bruce Warner, Jon Naseath, and Nihar Jhawar for their contributions and sharing their domain expertise to this important Cloud FinOps topic.

Cloud First

Energy Drive "Actualises Sustainability" - dynamically monitoring and analyzing system performance with Google Cloud

Reducing energy consumption, minimizing downtime

750 gigawatt hours (GWh) of electricity saved

Upskill for in-demand cloud roles with no-cost training on Coursera

Keep your cloud career growing, at pace with digital transformation

Future-proof your career with select no-cost training and earn certificates

Types of Google Cloud training available on Coursera

Here is a look at some of our most popular training for in-demand cloud roles

Build your cloud skills with no-cost access to Google Cloud training on Coursera

Access modeled data from Looker Studio, now in public preview

How does the integration work?

What’s new in the public preview version?

How do I access the preview?

Introducing Cloud Workstations: Managed and Secure Development environments in the cloud

Cloud Workstations: Just the facts

Fast developer onboarding via consistent environments

Customizable development environments

Security controls and policy support

What customers and partners are saying

Get started today

Tell us about your Cloud Architecture dreams

Can We Talk About Architecture?

What is a Picture Worth?

What’s Your Dream Cloud Architecture?

How to get the most from your Intel-based instances on Google Cloud

Optimizing the performance of your most demanding workloads

How it works

Get in touch to participate

Introducing support for GPU workloads and even larger Pods in GKE Autopilot

Using GPUs with Autopilot

Run Large Pods on Autopilot with the Balanced Compute Class

Google Cloud Deploy adds Cloud Run and deployment verification support

Deploy to Cloud Run

Verify your deployment

The future

Prepare for Google Cloud certification with top tips and no-cost learning

Prepare for certification with a no-cost learning opportunity

Become Google Cloud certified

Top tips to prepare for your Google Cloud certification exam

Why IT leaders choose Google Cloud certification for their teams

Incorporating quota regression detection into your release pipeline

What is quota regression and how can it occur?

Why not just use traditional monitoring?

Quota regression testing on Google Cloud

The importance of early detection

5 principles for cloud-native architecture—what it is and how to master it

Earn Google Cloud swag when you complete the #LearnToEarn challenge

Introducing the Google Cloud #LearnToEarn challenge

How to complete the #LearnToEarn challenge?

What’s involved in the challenge?

Ready to take on the challenge?

Introducing managed zone permissions for Cloud DNS

Announcing private network solutions on Google Distributed Cloud Edge

How Google Cloud is helping more startups build, grow, and scale their businesses

Expanding early-stage financial support to include bootstrapped startups

Setting founders up for success with new Startup School workshops

Supporting diverse founders and inclusivity across the globe

Celebrating startups across the ecosystem

The Retirement Tracker simplifies and socializes early retirement on Google Cloud

Snap Inc. adopts Google Cloud TPU for deep learning recommendation models

Faster leads to better

Benchmarking success

Benefits across the board

Working at the leading edge

Cloud TPU VMs are generally available

Build your cloud skills with no-cost access to Google Cloud training on Coursera

What is a lab?

Get a feel for the lab experience

Kick off your no-cost learning journey today

Training more than 40 million new people on Google Cloud skills

What’s new in cloud-native apps?

Week of Apr 11 - Apr 15, 2022

Picture this: How the U.S. Forest Service uses Google Cloud tools to analyze a changing planet

Week of Apr 4 - Apr 8, 2022

Introducing Topaz — the first subsea cable to connect Canada and Asia

Week of Mar 28 - Apr 1, 2022

Go 1.18 and Google Cloud: Go now with Google Cloud

Week of Mar 21 - Mar 25, 2022

Week of Mar 14 - Mar 18, 2022