Lab-Specific HPC Storage Pricing Model

From University of California San Francisco, CA

University Of California

Summary

All user accounts come with a quota of 500GB of storage, which is free of charge
Additional Wynton HPC storage can be purchased at $160/TB (one-time fee)
After 5 years from purchase, when the warranty runs out, a small maintenance fee might be introduced
Contact to purchase more storage

Plan
We are currently offering the ability for research labs to purchase additional storage at a one-time charge of $160/TB for RAID-6 equivalent storage (or $320/TB for mirrored RAID-6). In contrast to /wynton/home, purchased storage will be dedicated to the lab that bought it – no oversubscription will occur. The purchased storage will have similar performance to the rest of the BeeGFS infrastructure (/wynton/scratch and /wynton/home).

Given prices of HDDs, the stated rate might seem high, but there are 3 mitigating factors:

First, we have enabled ZFS compression, so the actual available space might be more.
Second, the price includes the cost of the networking, metadata servers, storage server, maintenance, and administration.
Third, we have proven that the performance of our BeeGFS infrastructure is much higher than the typical NFS server (in some respects, the performance is more than an order of magnitude faster). In the future, if absolutely necessary, we may also charge a ‘maintenance fee’ for storage after the initial 5-year hardware warranty expires, but nothing has been decided as of yet. Similarly, any future storage purchases may be priced differently than that described here, to reflect the situation present at that time.

There are some additional parameters:

Unlike nodes, the storage cannot be extracted from the system once it’s been brought online. Once a lab or a group has ‘bought in’, they will not be able to retrieve their portion of storage hardware if they choose to leave Wynton/HPC (unlike compute nodes/shares).
Storage is not available for mounting outside of the cluster.
At this point, the storage may not be used for PHI data (this may change in the future).
Storage is not backed up.

Background:
We have an immediate need to provide for Wynton/HPC storage expansion to meet the demands of various research groups on campus. Given the difficulty of predicting longer-term costs and issues, the current pricing is considered short-term and may change as we understand the evolving needs and operational realities of Wynton/HPC. Thus, this pricing model will be re-evaluated at the beginning of the next fiscal year (July 1, 2019).

This model is based on some assumptions important to understand:

Various components of the shared storage environment are considered ‘infrastructure’ and are currently funded from the ongoing support provided by the campus. These components include the networking infrastructure, and management and metadata servers that are part of the overall storage infrastructure. We don’t know with certainty that these components will continue to be funded by the campus and this introduces additional uncertainty as to the future pricing for storage, beyond the current offering described here.
The $160/TB price is for non-mirrored, potentially non-geographically redundant storage. While we hope to always purchase storage servers in pairs, providing failover between servers, there is no guarantee that we will always be able to do that. If you wish to protect your data beyond the level of RAID-6 (allows for two failed disks), we suggest you consider purchasing mirrored storage which completely duplicates data using a separate set of (RAID-6) disk drives.
Any new storage will be added to our existing BeeGFS installation and will not use separate instances of BeeGFS (which would increase the potential costs, if not in hardware, certainly in terms of personnel effort).
We assume that our last purchase represents a reasonable scalable unit of storage. This purchase provided 1,200TB (raw) storage and 2 storage servers. At ~$112,500, this results in a cost of $93/TB (raw) or $136/TB after accounting for RAID-Z2 and BeeGFS filesystem costs. We also need to add 2 additional metadata servers at a cost of $20,000. Taken together, this results in the $160/TB price.
The current storage hardware increment is ~$160,000, which may result in some delay between the first contributions and an actual purchase, although there is already pent-up demand and hence we are trying to proceed with the purchase as quickly as possible.

Storage hardware:

4 nodes of 4U 36-bay servers
2 metadata+storage nodes
2 storage only nodes

Metadata+Storage node:

CPU: Xeon E5-2630 V4 x 2
Motherboard: Supermicro X10DRL-i
HBA: LSI SAS 9300-8I
System disk drive: 480GB SSDx2 + 240GB SSDx2
Storage disk drive: 10TB HDDx34
RAM: 128GB
Network port: Four 10GbE

Storage-only node:

CPU: Xeon E5-2630 V4x2
Motherboard: Supermicro X10DRL-i
HBA: LSI SAS 9300-8I
System disk drive: 240GB SS x2
Storage disk drive: 10TB HDDx36
RAM: 64GB
Network port: Four 10GbE

Key solution values:

Total raw capacity: 1.4PB
Throughput: read 6.8GB/s, write 10GB/s (asynchronous)
Automated self-monitoring and self-healing
Flexible data protection, xd (n+m)
If using 16+4, allows four failed disks and 80% usable capacity of 1,120TB
Directory-based data mirroring, no operation interruption even with one failed node
Any new storage will be added to existing LeoFS installation with option to build separate instances of LeoFS
Potentially geographically redundant storage (LeoSync)
Hyper-convergence option, running applications directly on storage nodes
Integrated virtualization technology based on storage, users can create VMs to run applications
Provide interfaces such as CIFS, NFS, FTP, http, iSCSI, openstack and hadoop
LeoSAN and LeoFS private block and file interfaces
Fine-grained access rights can be defined according to user requirements.
Simple scalability, minutes to add petabyte capacities, automatic data load-balance
Performance linearly increase with additional capacity
Data read, write, and recovery are done cross-nodes, creating performance and speed