Availability of IBM Spectrum Scale in Google Cloud

By Barak Epstein, product management, storage, Google Cloud, and

Scott Fadden, senior HPC architect, Sycomp, Inc.

Google Cloud and Sycomp, Inc. are introducing Sycomp Storage fueled by IBM Spectrum Scale.

This launch is highly useful for customers looking to run HPC, AI/ML, and big data workloads on a proven file system, in a fully-persistent deployment, and with Google’s Cloud Storage integration for cost optimization.

Sycomp Storage Fueled by Spectrum Scale optimizes performance for the cloud, deploys in minutes, supports auto-tiering to reduce costs, and allows customers to concurrently access data from thousands of VMs, with data access via NFS and the native Spectrum Scale client. The solution is available in the Google Cloud Marketplace, as a VM-based offering that deploys within a customer’s project, and is supported and maintained by Sycomp.

With Sycomp Storage Fueled by Spectrum Scale, you can:

• Take advantage of auto-tiering technology: Automatically manage tiers of storage to optimize performance and cost to deliver the speed and reliability you need to succeed. 
• Deploy in minutes: Deploy Spectrum Scale in minutes via Google Cloud Marketplace, and enjoy access to your data via NFS and the native Spectrum Scale client. 
• Maximize performance: With up to 320GB/s of storage performance capability, you can select the VM and disk combination that best meets your requirements. 

About Spectrum Scale
It is a parallel file system that securely manages large volumes of data. The high performance, feature-rich filesystem is well-suited for HPC, AI/ML, big data and other applications requiring a POSIX compliant shared filesystem. With adaptable storage capacity and performance scaling it can support demanding workloads.

You can deploy an instance of Spectrum Scale using Terraform provided by Sycomp through the Marketplace.

When deploying a cluster, you choose:

• How you access the data. You can use the Spectrum Scale clients deployed by Sycomp or access the data from your existing application VMs.
• The amount and type of storage
• Instance type and quantity of NSD Servers, NFS Servers, and Spectrum Scale Clients

Deployment models:

Core considerations
Spectrum Scale offers multiple deployment models based on your performance and capacity needs. Customers can choose instance type, the number of nodes and select from Google’s Persistent Disk and Local SSD options.

Storage type
Persistent Disk offers higher density and increased data protection. Customers can choose between standard, balanced, or SSD to optimize $/GB or $/Perf.

Local SSD (LSSD) offers reduced latency and high IO/s optimizing for $/Perf. With LSSD, Spectrum Scale leverages data replication across VMs to protect vs. data loss in the event of a VM failure, migration, or accidental deletion.

Protocols:
Options include:

• NFS for protocol compliance. Deploy with converged (NFS and NSD Server in the same VM) or a separate NFS server VM 
• NSD native client for best per-client throughput

Spectrum Scale supports multiple protocols and storage media
Click to enlarge

Hybrid deployment
Spectrum Scale allows to integrate your on-premises systems with Google Cloud with bi-directional data flow, leveraging Spectrum Scale Active File Management (AFM).

Click to enlarge

Your on-premises application can use AFM cloud object storage (COS) through Google’s Cloud Storage or AFM directly to create a hybrid cloud environment with Spectrum Scale deployment in Google Cloud.

You can use AFM to connect:

• Directly through 2 Spectrum Scale clusters
• Through cloud storage to share data between your on-premises cluster and your Sycomp Storage deployment in Google Cloud.

AFM provides a way to move data between your on-premises and Google Cloud-based clusters, and are deployed in a single Google Cloud project.

Key features:

File system encryption
A deployment can be enhanced by enabling data encryption with an HA pair of VMs running IBM Security Guardium Key Lifecycle Manager (GKLM) to manage and protect your encryption keys. Tools provided by Sycomp automate the deployment of an HA pair of GKLM server VMs.

Integration with cloud storage
Once the cluster is deployed, you can enable Spectrum Scale AFM to cloud object storage (COS) to automatically tier your application data to cloud storage or dynamically hydrate the Sycomp storage cluster from existing data in cloud storage

Spectrum Scale storage cluster with AFM to COS configured to communicate with cloud storage

Cluster expansion
You can add storage, NFS servers and Spectrum Scale client VMs to an existing Sycomp storage cluster. Adding storage capacity is achieved by adding NSD servers or disks to existing NSD servers. When you add NSD servers to the cluster, the storage solution implements best practices, as each new server inherits the disk configuration of the existing NSD servers. Storage is added in this manner to maintain balance across NSD servers. You can add NFS servers to increase capacity. Adding clients in groups allows you to customize the names and machine types of each set of clients giving you more control over your deployment.

Clusters have been expanded to a petabyte and have been tested to demonstrate linear performance scaling to 250TB.

Special thanks to Billy Kettler, partner engineer and Sean Derrington, product manager, for contributing to this post.

Resources:
Deeper dive: Sycomp’s architectural overview     
Sycomp Storage Fueled by IBM Spectrum Scale on Google Cloud        
Sycomp Storage Fueled by IBM Spectrum Scale In Marketplace