French Atomic Energy Group Expands HPC Storage to 11PB

The Military Applications Department of the French Alternative Energies and Atomic Energy Commission (CEA/DAM) doubled the capacity of its largest HPC production file system.

With over 11 Petabytes of usable storage space, this Lustre v2 file system is one of the largest single namespace system ever deployed.

Starting in September 2010, the storage capacity has been gradually increased, to reach today 75% of TERA computing center’s global Lustre storage space and bandwidth (200GB/s), a step forward to its final configuration of 15PB planned this year.

With 768 16TB Object Storage Targets (OST), TERA global file system consists in 70 bullx S6030 Lustre servers, a Voltaire InfiniBand QDR storage fabric and 89 NetApp (formerly LSI) E5400-60 disk arrays. Tera 100, Europe’s first supercomputer to break the Petaflops barrier, and other ancillary systems such as post-processing clusters, mount this file system through more than 50 Lustre routers.

System administration
To optimize deployment time, availability and to simplify daily administration tasks, CEA engineers use a common set of open libraries and tools accross CEA’s computing centers: TERA, TGCC and CCRT. Shine, an open-source Lustre administration tool developed at CEA, manages all Lustre components, allowing configuration of servers, routers and thousands of clients.

Data workflow
With such a large amount of data, traditional tools usually used to scan and maintain a file system are notoriously inefficient. CEA developed Robinhood, an open-source policy engine designed to address this specific issue. It uses Lustre MDT changelogs (a Lustre v2 feature) to update an internal database reflecting the file system state. CEA also contributes to Lustre development, with features such as OST pools, and is the main developer for the forthcoming Lustre-HSM binding feature (Lustre 2.x).

CEA also develops NFS-Ganesha, a user-space NFS server capable of using the Lustre API to export very large file system to non Lustre clients, such as smaller systems or individual workstations.