SC23: Google Supports DAOS Foundation for Next-Gen HPC and AI Storage

By Barak Epstein, product management, storage, and

Dean Hildebrand, office of CTO, Google Cloud

Google announced that Google Cloud is a founding member of the DAOS (Distributed Asynchronous Object Storage) Foundation, launched by the Linux Foundation.

Created in collaboration with partners, the DAOS Foundation will accelerate investment and development of the world-record-setting open-source DAOS storage stack, which supports next-gen HPC and AI/ML workloads.

Intel Corp. initiated investment in DAOS in 2012 to drive a new paradigm in storage for HPC. The combination of fully-distributed metadata management, a key-value store architecture, and the lack of dependency on kernel modifications has led to notable breakthroughs, such as:

• gains in throughput, I/O, and metadata operations performance

• easier container/Kubernetes integration

• increased architectural flexibility and decreased infrastructure overhead (eliminating the need for metadata servers)

The upcoming deployment at Argonne National Laboratories, which is targeted to deliver >25TB/s of throughput for HPC, demonstrates the impact DAOS has had on the world of scientific computing.

Further, Google recently launched Parallelstore, a scratch storage service based on DAOS that delivers up to 6.3x read-throughput performance compared to competitive Lustre scratch offerings. Both large language models (LLMs) and HPC have demanding IO/s, throughput, MDOPS, and low latency requirements. Customer demand for AI/ML training based on large data sets with small files that require low-latency operations aligns well with the performance characteristics and architecture of DAOS.

“Google is delighted to be a founding member of the DAOS Foundation,” said Sameet Agarwal, VP and GM, Google Storage. “This industry-leading architecture underpins our new storage service, Parallelstore, and we are excited to accelerate the growth of HPC and AI/ML workloads in the cloud.”

The DAOS Foundation will be established as a Linux Foundation project, which provides a structure for shared governance and continued investment in the open source community. The collaboration across companies has been strong and Google Cloud is excited to continue this investment.