Kyoto University Lost 77TB of Research Data Due to “Accident” During Backup

A HPC system in Kyoto University lost 77TB of research data due to an “accident” during a backup.

Below is the statement posted by Kyoto University IT department’s Supercomputing Division, translated to English:

Dear Supercomputing Service Users

Today, a bug in the backup program of the storage system caused an accident in which some files in / LARGE0 were lost. We have stopped processing the problem, but we may have lost nearly 100TB of files, and we are investigating the extent of the impact.

We will contact those affected individually. We apologize for the inconvenience caused to all users.

Regarding this matter, from 17:50 to 19:00 on December 16, 2021, we contacted the applicants of the target group by e-mail. The extent of the impact of the disappearance accident that occurred this time turned out to be as follows.

– Target file system: / LARGE0
– File deletion period: December 14, 2021 17:32 – December 16, 2021 12:43
– Disappearance target file: December 3, 2021 17:32 or later, files that were not updated
– Lost file capacity: Approximately 77TB
– Number of lost files: Approximately 34 million files
– Number of affected groups: 14 groups (of which 4 groups cannot be restored by backup)

Kyoto University’s HPCs come from HPE Cray and their storage system is from DDN, a Californian company specialising in supercomputing storage solutions.