Cloud Filestore Service Brings Google Cloud Platform File Storage Capabilities

This article, published on June 26, 2018 on a blog of Google Inc., was written by Tad Hunt, product manager and Chris Talbott, product marketing, in the company.

New Cloud Filestore service
brings GCP users high-performance file storage

As we celebrate the upcoming Los Angeles region for Google Cloud Platform (GCP) in one of the creative centers of the world, we’re really excited about helping you bring your creative visions to life. At Google, we want to empower artist collaboration and creation with high-performance cloud technology. We know folks need to create, read and write large files with low latency. We also know that film studios and production shops are always looking to render movies and create CGI images faster and more efficiently. So alongside our LA region launch, we’re pleased to enable these creative projects by bringing file storage capabilities to GCP for the first time with Cloud Filestore.

Cloud Filestore^beta is managed file storage for applications that require a file system interface and a shared file system. It gives users a simple, integrated, native experience for standing up fully managed NAS with their Google Compute Engine and Kubernetes Engine instances.

We’re pleased to add Cloud Filestore to the GCP storage portfolio because it enables native platform support for a broad range of enterprise applications that depend on a shared file system.

Cloud Filestore will be available as a storage option in the GCP console
We’re especially excited about the high performance that Cloud Filestore offers to applications that require high throughput, low latency and high IO/s. Applications such as content management systems, website hosting, render farms and virtual workstations for artists typically require low-latency file operations, high-performance random I/O, and high throughput and performance for metadata-intensive operations. We’ve heard from some of our early users that they’ve saved time serving up websites with Cloud Filestore, cut down on hardware needs and sped up the compute-intensive process of rendering a movie.

Putting Cloud Filestore into practice
For organizations with lots of rich unstructured content, Cloud Filestore is a good place to keep it. For example, graphic design, video and image editing, and other media workflows use files as an input and files as the output. Filestore also helps creators access shared storage to manipulate and produce these types of large files. If you’re a web developer creating websites and blogs that serve file content to your audience, you’ll find it easy to integrate Cloud Filestore with web software like WordPress. That’s what Jellyfish did.

Jellyfish Group is a boutique marketing agency focused on delivering high-performance marketing services to their global clients. A major part of that service is delivering a modern and flexible digital web presence.

“WordPress hosts 30% of the world’s websites, so delivering a highly available and high performance WordPress solution for our clients is critical to our business. Cloud Filestore enabled us to simply and natively integrate WordPress on Kubernetes Engine, and take advantage of the flexibility that will provide our team,” said Ashley Maloney, lead DevOps engineer, Jellyfish Online Marketing.

Cloud Filestore also provides the reliability and consistency that latency-sensitive workloads need. One example is fuzzing, the process of running millions of permutations to identify security vulnerabilities in code. At Google, ClusterFuzz is the distributed fuzzing infrastructure behind Chrome and OSS-Fuzz that’s built for fuzzing at scale. The ClusterFuzz team needed a shared storage platform to store the millions of files that are used as input for fuzzing mutations.

“We focus on simplicity that helps us scale. Having grown from a hundred VMs to tens of thousands of VMs, we appreciate technology that is efficient, reliable, requires little to no configuration and scales seamlessly without management. It took one premium Filestore instance to support a workload that previously required 16 powerful servers. That frees us to focus on making Chrome and OSS safer and more reliable,” said Abhishek Arya, information security engineer, Google Chrome.

WORM is another type of workload where consistency and reliability are critical. At ever.ai, they’re training an advanced facial recognition platform on 12 billion photos and videos for tens of millions of users in 95 countries. The team constantly needs to share large amounts of data between many servers that will be written once but read a bunch. They faced a challenge in writing this data to a non-POSIX object storage, reading from which required custom code or to download the data. So they turned to Cloud Filestore.

“Cloud Filestore was easy to provision and mount, and reliable for the kind of workload we have. Having a POSIX file system that we can mount and use directly helps us speed-read our files, especially on new machines. We can also use the normal I/O features of any language and don’t have to use a specific SDK to use an object store,” said Charlie Rice, CTO, ever.ai.

Cloud Filestore is also particularly helpful with rendering requirements. Rendering is the process by which media production companies create computer-generated images by running specialized imaging software to create one or more frames of a movie. We’ve just announced our newest GCP region in Los Angeles, where we expect there are more than a few of you visual effects artists and designers who can use Cloud Filestore. Let’s take a closer look at an example rendering workflow so you can see how Cloud Filestore can read and write data for this specialized purpose without tying up on-site hardware.

Using Cloud Filestore for rendering
When you render a movie, the rendering job typically runs across fleets (‘render farms’) of compute machines, all of which mount a shared file system. Chances are you’re doing this with on-premises machines and on-premises files, but with Cloud Filestore you now have a cloud option.

To get started, create a Cloud Filestore instance, and seed it with the 3D models and raw footage for the render. Set up your Compute Engine instance templates to mount the Cloud Filestore instance. Once that’s set, spin up your render farm with however many nodes you need, and kick off your rendering job. The render nodes all concurrently read the same source data set from the NFS share, perform the rendering computations and write the output artifacts back to the share. Finally, your reassembly process reads the artifacts from Cloud Filestore and assembles it and writes into the final form.

Cloud Filestore Price and Performance
We offer two price-for-performance tiers. The high-performance Premium tier is $0.30 per gigabyte per month, and the mid-range performance Standard tier is $0.20 per gigabyte per month in US East, US Centra1, and US West (other regions vary). To keep your bill simple and predictable, we charge for provisioned capacity. You can resize on demand without downtime to a maximum of 64TB*. We do not charge per-operation fees. Networking is free in the same zone, and cross zone standard egress networking charges apply.

Cloud Filestore Premium instance throughput is designed to provide up to 700MB/s and 30,000 IO/s for read, regardless of the Filestore instance capacity. Standard instances are lower priced and performance scales with capacity, hitting peak performance at 10TB and above. A simple performance model makes it easier to predict costs and optimize configurations. High performance means your applications run faster. As you can see in the image below, the Cloud Filestore Premium tier outperforms the design goal with the specified benchmarks, based on performance testing we completed in-house.

Trying Cloud Filestore for yourself
Cloud Filestore will release into beta next month. To sign up to be notified about the beta release, complete this request form. Visit our Filestore page to learn more.

In addition to our new Cloud Filestore offering, we partner with many file storage providers to meet all of your file needs. We recently announced NetApp Cloud Volumes for GCP and you can find other partner solutions in our launcher.

Click to enlarge