Toshiba HDDs Help CERN Keep Track of Huge Generated Data

Introduction
CERN’s Large Hadron Collider (LHC) is at the forefront of physics research. The data output from its “Run 1″ and “Run 2″ phases have already been used to demonstrate the existence of a previously undetected subatomic particle and extend its understanding of the universe and how it formed. Notably in 2012 it confirmed the existence of the Higgs boson.

The scale of CERN is astounding. From the size of the large hadron collider – a circular particle accelerator with a radius of 4.3km – to the rate of particle collisions – up to 1 billion particle collisions can take place every second inside the LHC experiment’s detectors.

But it is the data that is most impressive, with the collisions generating 1PB of data per second. Even after filtering only the interesting events, the facility requires approximately 10PB of new data to be stored for analysis each month.

This data is stored in the CERN data centre and is shared with a network of about 170 data centres for analysis, thanks to the Worldwide LHC Computing Grid (WLCG). The current storage setup at CERN consists of HDD buffers with 3,200 JBODs carrying 100,000 HDDs providing a total of 350PB.

LHC Runs are set to continue, and with each new “Run“, storage increases significantly. Following upgrades, CERN’s “Run 3” is scheduled for 2021.

Toshiba Electronics Europe GmbH’s HDDs are used by CERN to manage huge volumes of data since 2014, with 3 generations of HDD technology giving it the capacity increases it requires. But, can this continue when, as CERN’s manager of the facility planning and procurement section at the IT department, Eric Bonfillou, puts it: “The planned upgrades of the LHC machine will require scaling of compute and storage resources beyond what today’s technology can offer.“

Timeline of how CERN has scaled its storage with Toshiba

Click to enlarge

2014 – CERN purchases Toshiba HDDs
Since 2014, CERN uses Toshiba’s HDDs. At the heart of its needs are server-grade HDDs with performance, reliability and optimised for highest capacities. During 2013-14, CERN held a scheduled shutdown phase to prepare for “Run 2″. During this period, it upgraded its storage systems, adding arrays of 24-bay 4U JBODs populated with Toshiba’s MG03SCA400 HDDs.

With 4TB of capacity per HDD, each spinning at 7,200rpm and receiving data via a 6 GB/s interface, CERN gained 96TB per JBOD expansion unit. The HDD’s mean time to failure (MTTF) was 1.2 million hours, which translates to a predicted annualised failure rate (AFR) of 0.72%.

2015/2016 – CERN begins “Run 2”: Storage needs to be increased significantly
With the start of the “Run 2” in 2015, the need for data storage increased immensely. CERN adds new storage capacity using Toshiba’s new 6TB SATA model, the MG04ACA600E. Again, installed in a 24-bay 4U front loader JBOD, the total raw capacity was increased to 144TB per unit. The MTTF for this model improved to 1.4 million hours, giving a predicted AFR of 0.62%

2018 – Toshiba launches MG07 helium-filled model to extend capacity to 14TB per HDD
After 2016, Toshiba accelerated the development and introduction of new high capacity enterprise HDD models to serve the need for cloud storage – launching 2 new generations – the MG05 and MG06. The MG06 series became available with capacities up to 10TB per drive. And although the construction was still air-filled, the manufacturer was able to increase the MTTF to 2.5 million hours, which translates into an AFR of 0.35%, the lowest currently on the market for air-filled HDDs.

For Conventional Magnetic Recording (CMR) technology in a 3.5” form-factor, air-filled HDD technology reaches a limit at 7 platters (and therefore 10TB) of capacity.

And while it is theoretically possible to increase this using thinner platters the air would be too heavy a gas, resulting in severe flattering. Alternatively, Shingled Magnetic Recording (SMR) technology can be used, but this requires special handling to avoid serious performance issues, and is limited by the need for special file systems tailored to the technology’s intricacies.

CERN’s 2018 storage upgrade coincided with the launch of Toshiba’s MG07 range, the first server HDDs with up to 14TB. With the purchase of the 12TB variant, CERN doubled its capacity per JBOD to 288TB.

The MG07 increases the capacity without changing the form factor by using helium rather than air. This allows the use of thinner platters without the associated flattering, with up to nine per drive.

By using CMR, the MG07 is suited to any workload without the loss of performance associated with SMR. Additionally, helium creates less friction than air, reducing the energy required to rotate the platter stack. With precise optimization of the spindle motor, the operating power has reduced by a third (from c.11W to <7W) for the helium-based model of the MG07 series.The MG07’s MTTF extended to 2.5 million hours.

The increased data capacity demands from “Run 3“
In 2019, the LHC machine shut down again to install further upgrades before restarting for “Run 3” in 2021. A significant increase in storage demands is expected, with the acceleration in data generated.

Bonfillou states: “Toshiba’s products and support have met CERN’s stringent requirements. Our IT infrastructure, in terms of computing power and storage capacity, has scaled well with the scientific computing needs, making maximum use of Toshiba’s high capacity and reliable HDD.”

Toshiba’s planned launches for CMR and SMR based drives that use the same 3.5” form factor will give CERN access to 16TB and 18TB drives adding 432TB of new capacity per JBOD.

“Toshiba products are well suited for large scale datacenter storage and the successful deployment and operation of three generations of Toshiba Enterprise HDDs in CERN’s challenging IT environment is a perfect reference case,” says Larry Martinez-Palomo, GM of the HDD business unit, Toshiba Electronics Europe.

As for longer term R&D activity, Toshiba is developing a next-gen magnetic recording technology that will further extend capacities to exceed 20TB per HDD, while still maintaining the 3.5” form factor.

Martinez-Palomo adds: “We are confident that our HDD next generation technologies will contribute to solve CERNs future scaling challenges in terms of storage capacity, investment budget, power consumption and reliability.“

Video