Google Cloud: What’s New with Filestore, Enhancing Stateful Workloads on Google Kubernetes Engine

By Barak Shein, senior product manager, Google, and Harry Sadoyan, software engineer, Google Cloud

Many customers use Filestore for stateful applications running on Google Kubernetes Engine (GKE). For background on the benefits of using the 2 together, see this blog.

Filestore, Google Cloud’s managed file storage service, is a multi-reader, multi-writer solution that is decoupled from compute VMs, making it resilient to VM changes/failures.

Filestore’s Enterprise and Zonal (formerly known as High Scale) tiers are built on Filestore’s modern platform that utilizes scale-out architecture, designed for performance, availability and durability. Customer data is stored twice to ensure high durability within a zone, and Zonal offers a 99.9% availability SLA. Enterprise, a regional service, replicates the data synchronously across 3 zones within a region, offering 99.99% availability. Better yet, in October, we reduced Filestore prices by up to 25% on Enterprise, Zonal, and Basic HDD tiers (Basic SSD pricing remains unchanged). The new lower prices are applied automatically with no action required.

Filestore is a fully managed service integrated into GKE’s CSI driver, and is continuously evolving with new features, functionality, and GKE integrations.

Read on to learn about 3 new features, which have all recently become generally available:

1. CSI Driver support for Filestore Zonal High-capacity (100TB)
The new CSI driver integration of our high-capacity, Zonal offering with GKE starts at 10TB and scales capacity and performance linearly to meet your high-capacity and high performance needs up to 100TB/instance. The high throughput of Filestore Zonal (up to 26GB/s) is useful for large-scale AI/ML training frameworks like PyTorch/Tensorflow that expect a file interface. Additionally, it features non-disruptive upgrades, and 1,000 NFS connections per 10TB. That’s up to 10,000 concurrent NFS connections supporting large GKE deployments and demanding multi-writer AI/ML workloads.

The addition of the CSI driver for Filestore Zonal complements our previously available offerings of Filestore Enterprise and Filestore Basic. This now enables you to choose the appropriate Filestore tier based on your performance, capacity and data protection (e.g., snapshot and backup) needs. Note that Filestore Enterprise supports non-disruptive upgrades while Filestore Basic tier does not.

2. Backups
We announce the CSI-integrated backup for Filestore Enterprise.

Customers were already able to protect their data with the Volume Snapshots API on Filestore Basic volumes, and now they can protect Filestore Enterprise volumes the same way. It’s important to note the naming of the VolumeSnapshot API is a bit misleading as it’s actually a method to backup the data and is not a local file system snapshot as the name implies. The process of using the API to invoke a backup for Filestore Basic and Enterprise is identical.

You can now take backups of your existing Enterprise volumes, and create new volumes with the snapshot as a data source. The new feature enables you to backup and restore your instances directly from the CSI driver. This capability is available on the Enterprise tier in single-share configurations. The Volume Snapshots API is not supported with Enterprise Multishare configurations at this time.

Creating and restoring backups directly from the CSI driver

3. Smaller capacity Filestore Enterprise multi-shares
GKE and Filestore Enterprise customers enjoy the efficiency of the multishare instances launched last year, enabling them to subdivide a 1TB instance into multiple 100GB persistent volumes to improve storage utilization.

Today, we’re announcing the next stage of our multi-shares feature, exclusively available on the Enterprise tier through the GKE CSI driver. With multishares, you can now subdivide your Enterprise instance into as many as 80 shares (up from 10 shares previously), and can start from a minimum share (persistent volume) capacity of 10GB (down from 100GB previously).

With Enterprise Multishares, persistent storage for your GKE stateful workloads is not only highly durable (99.99% SLA) and high-performance, it’s also highly efficient, offering comprehensive instance utilization of up to 80 shares per instance. Learn more about how you can increase your GKE storage efficacy with the improved, multishares feature here.

Stateful workloads, Filestore way
Filestore is a fully managed, GKE integrated NFS storage service. With the addition of the Filestore Zonal CSI integration, Volume Snapshot API availability for Filestore Enterprise, and the more granular Filestore Enterprise Multishare persistent volume support, we continue to strive to support large-scale, demanding GKE stateful workloads such as AI/ML training.

Get started here to use Filestore and the CSI driver with GKE.