Storage Classes for Google Cloud Storage to Simplify Storage and Management of Hot and Cold Data

This article was posted on October 16, 2016 by Kirill Tropin, product manager, Google, Inc.:

Announcing new storage classes for Google Cloud Storage: simplifying the storage and management of hot and cold data

Businesses seek the best price and performance to suit the storage needs of workloads ranging from multimedia serving, to data analytics and machine learning, to backup/archiving, all of which drive demand for a variety of storage options.

At Google, we aim to build a powerful cloud platform that can meet the needs of the most demanding customer workloads. Google Cloud Storage is a key part of that platform and offers developers and IT organizations durable and highly available object storage, with consistent APIs for ease of application integration, all at a low cost. Today, we’re excited to announce a major refresh of Google Cloud Storage. We’re introducing new storage classes, data lifecycle management tools, improved availability and lower prices-all to make it easy for our customers to store their data with the right type of storage. Whether a business needs to store and stream multimedia to their users, store data for machine learning and analytics or restore a critical archive without waiting for hours or days, Cloud Storage offers a broad range of storage options to meet those needs. We’re also excited to announce the continued expansion of our Google Cloud Platform (GCP) partner ecosystem, with partners already using the new Cloud Storage capabilities for use cases including content delivery, hybrid storage, archival, backup and DR.

Storage classes for Google Cloud Storage
We’re announcing the availability of four storage classes for Cloud Storage. These offer customers a consistent API and data access performance for all of their hot and cold data, with simple-to-understand and highly competitive pricing.

Google Storage Cloud Classes

Cloud Storage Coldline: Low-latency storage class for long-term archiving
Coldline is a Cloud Storage class designed for long-term archival and DR. Coldline is perfect for the archival needs of big data or multimedia content, allowing businesses to archive years of data. It provides fast and instant (millisecond) access to data and changes the way that companies think about storing and accessing their cold data. At GCP, we believe that archival data should be as accessible as any other data. Coldline’s API and low latency data access are consistent with other storage classes. This means existing systems can now store and access Coldline data without any updates to the application, and can serve that data directly to end users in milliseconds. Priced at $0.007 per gigabyte per month plus a simple and predictable access fee of $0.05 per gigabyte retrieved, Coldline is an economical storage class for data that’s accessed less than once per year.

Coldline also works well with Nearline to provide tiered storage for data as it cools. Our recent work on Nearline latency and throughput ensures comparable performance across all storage classes.

To help you migrate your data to Coldline, and other Cloud Storage classes, we offer an easy-to-use Google Cloud Storage Transfer Service and have extended our Switch and Save program to include Coldline. Depending on the amount of data you’re bringing to Coldline, you can receive several months of free storage, for up to 100PB of data.

Google Cloud Storage Multi-Regional and Regional
GCP customers use Cloud Storage for a variety of demanding use cases. Some use cases require highly available storage close to the Google Compute Engine instances. Others need higher levels of availability and geo-redundancy. We’re updating our storage classes to address those needs:

Google Cloud Storage Multi-Regional is a highly available and geo-redundant storage class. It’s the best storage class for BC, or for serving multimedia content to geographically distributed users. In the case of a regional outage, Cloud Storage transparently routes requests to another available region, ensuring that applications continue to function without disruption. Multi-Regional storage is priced at $0.026 per gigabyte per month, including storage of all replicas, replication over the Google network and connection rerouting. It’s currently available in three locations: US, EU and Asia. All existing Standard storage buckets in a multi-regional location have been converted to Multi-Regional storage class. Vimeo, a media hosting, sharing and streaming service, leverages Google Cloud Storage Multi-Regional to ensure HA, and low-latency access to data. Cloud Storage Nearline is used to minimize overall storage costs. To deliver the best possible experience, Vimeo leverages integration between Google Cloud Storage and Fastly, a real-time CDN service. With Fastly, Vimeo can deliver content from Google Cloud Storage to users’ instantly-performing at sub-150 millisecond response times.

“We use Google Cloud Platform, including Google Cloud Storage and Compute Engine along with Fastly, for storing and delivering all popular and infrequently accessed content and to handle our peak transcode loads.” Said Naren Venkataraman, senior director, engineering, Vimeo LLC.

“Fastly customers need low-latency, high-throughput storage and fast, flexible, secure content delivery at the edge. The combined power of Google Cloud Storage and Fastly’s Cloud Accelerator allows customers like Vimeo to fully optimize content storage and delivery, controlling costs and improving global performance.” Said Lee Chen, head, strategic partnerships, Fastly, Inc.

Google Cloud Storage Regional is a available storage class redundant within a single region. It’s ideal for pairing storage and compute resources within a region, to deliver low end-to-end latency and high throughput for workloads such as data transcoding or big data analytics workloads running on Google Compute Engine, Google Cloud DataProc, Google Cloud Machine Learning or BigQuery for example.

Regional storage class is priced at $0.02 per gigabyte per month. All existing Standard storage buckets in a regional location have been converted to the Regional storage class. This is equivalent to a 23% price reduction, and the pricing change for converted buckets takes effect November 1st.

One of our customers who’s already using Regional storage class is Spotify AB. Spotify streams music for over 100 million users with GCP and uses Cloud Storage, along with Compute Engine to scale while controlling costs in a reliable and highly durable environment. Cloud Storage stores more than 30 million songs.

“Spotify uses Google Cloud Storage for storing and serving music. Using Regional storage class allowed us to run audio transcoding in Google Compute Engine close to production storage. Google also offers great networking with open and explicit peering setup, as well as interconnect and partnerships with all of our CDN providers.” Said Jyrki Pulliainen, software engineer, Spotify.

Effective November 1st we’re also introducing lower API operations pricing for both Multi-Regional and Regional storage classes. Class A operations will cost $0.005 per 1,000 operations (50% price reduction), and Class B will cost $0.004 per 10,000 operations (60% price reduction).

With the addition of Coldline and the refresh of our storage classes with Multi-Regional and Regional, GCP customers will continue to enjoy the same API and consistent data access performance for all of their hot and cold data. With Coldline, no application changes are needed to leverage archived data and there’s no compromise on access time for that data, while Multi-Regional makes it simple to ensure that your data is highly-available and geo-redundant. Plus, we’re delivering all of this with simple to understand and highly competitive pricing:

Data management lifecycle capabilities
Many of our customers use multiple storage classes for their different workloads. Having a single API and consistent data access performance ensures applications can seamlessly leverage multiple storage classes. It should be easy for customers to also manage the appropriate storage tier for their data.

We’re introducing the beta of data lifecycle management capabilities to make it easier to manage data placement. Any Google Cloud Storage bucket can now hold data in different storage classes, and the lifecycle policy feature can automatically transition objects in-place to the appropriate colder storage class based on the age of the objects.

Expanding Cloud Storage partner ecosystem
Many customers already use multiple Cloud Storage classes and will benefit from these storage updates, both directly through us and through our partners, a number of whom have already integrated the new Coldline storage class. Starting today, these partners are available to help you use our new storage classes in your own environment:

Fastly: Fastly is a content delivery network that lets businesses control how they serve content, provides real-time performance analytics and caches frequently changing content at the edge. Fastly enables customers to configure Google Cloud Storage as the origin, and Fastly’s Origin Shield designates a single point-of-presence (POP) to handle cache-misses across our entire network.
Veritas: Building on existing support for GCP, Veritas Technologies LLC, is committed to supporting Cloud Storage Coldline. The combination of Veritas Information Map, Veritas NetBackup and the GCP ensures customers can gain greater controls on data visibility as they move to the Google Cloud at global enterprise scale. Veritas’ collaboration with Google further demonstrates the shared commitment to helping organizations around the world manage information.
Cloudian: The Cloudian, Inc.‘s HyperStore smart storage platform seamlessly integrates with Google Cloud Storage (including Coldline) to provide anywhere from terabytes to hundreds of petabytes of on-premises storage. Policy-based data migration lets you move data to Coldline based on rules such as data type, age and frequency of access.
Cloudberry Lab: CloudBerry Lab, Inc.‘s Backup is a cloud backup solution that leverages Coldline. In addition to offering real-time and/or scheduled regular backups, encryption, local disk image or bare metal restore, CloudBerry employs block level backup for maximum efficiency and provides alerting features to track each backup and restore plan remotely.
Komprise: Komprise, Inc.’s data management software enables businesses to manage the lifecycle of data and cut costs by over 70% by leveraging all the tiers of Cloud Storage transparently with existing on-premises storage. In under 15 minutes, customers can get a free assessment of how much data can move to Cloud Storage and the projected ROI with a free Komprise trial.
StorReduce: StorReduce, Inc.‘s inline deduplication software enables you to move terabytes to petabytes of data into Coldline (or other Cloud Storage tiers) and then use cloud services such as search on that data.
Cohesity: The Cohesity, Inc.‘s hyper-converged secondary storage system for enterprise data consolidates fragmented, inefficient islands of secondary storage into a virtually limitless storage platform. Coldline can be used for any data protection workload via. Cohesity’s policy-based administration capabilities.
Sureline: Sureline Systems, Inc.‘s application mobility software delivers migration and recovery of virtual, cloud, physical or containerized applications and servers. It allows enterprises to use Coldline as the DR target for occasionally accessed DR images with SUREedge DR.