104,130 HDDs Tested by Backblaze

This article was written on April 30, 2019 by Andy Klein , director of product marketing, Backblaze, Inc.

Backblaze HDD Stats 1Q19

As of March 31, 2019, Backblaze had 106,238 spinning HDDs in its cloud storage ecosystem spread across three data centers. Of that number, there were 1,913 boot drives and 104,325 data drives. This review looks at the 1Q19 and lifetime HDD failure rates of the data drive models currently in operation in our data centers and provides a handful of insights and observations along the way. In addition, we have a few questions for you to ponder near the end of the post.

HDD Failure Stats for 1Q19
At the end of 1Q19, Backblaze was using 104,325 HDDs to store data. For this evaluation we remove from consideration those drives that were used for testing purposes and those drive models for which we did not have at least 45 drives (see why below). This leaves us with 104,130 HDDs. The table below covers what happened in 1Q19.

Notes and Observations
If a drive model has a failure rate of 0%, it means there were no drive failures of that model during 1Q19. The two drives listed with zero failures in Q1 were the 4TB and 5TB Toshiba models. Neither has a large enough number of drive days to be statistically significant, but in the case of the 5TB model, you have to go back to 2Q16 to find the last drive failure we had of that model.

There were 195 drives (104,325 minus 104,130) that were not included in the list above because they were used as testing drives or we did not have at least 45 of a given drive model. We use 45 drives of the same model as the minimum number when we report quarterly, yearly, and lifetime drive statistics. The use of 45 drives is historical in nature as that was the number of drives in our original Storage Pods. Beginning next quarter that threshold will change; we’ll get to that shortly.

The Annualized Failure Rate (AFR) for 1Q19 is 1.56%. That’s as high as the quarterly rate has been since 4Q17 and its part of an overall upward trend we’ve seen in the quarterly failure rates over the last few quarters. Let’s take a closer look.

Quarterly Trends
We noted in previous reports that using the quarterly reports is useful in spotting trends about a particular drive or even a manufacturer. Still, you need to have enough data (drive count and drive days) in each observed period (quarter) to make any analysis valid. To that end the chart below uses quarterly data from Seagate and HGST drives while leaving out Toshiba and WDC drives as we don’t have enough drives from those manufacturers over the course of the last three years.

Over the last three years, the trend for both Seagate and HGST/WD AFR had improved, i.e. gone down. While Seagate has reduced their failure rate over 50% during that time, the upward trend over the last three quarters requires some consideration. We’ll take a look at this and let you know if we find anything interesting in a future post.

Changing the Qualification Threshold
As reported over the last several quarters, we’ve been migrating from lower density drives, 2, 3, and 4TB drives, to larger 10, 12, and 14TB HDDs. At the same time, we have been replacing our stand-alone 45-drive Storage Pods with 60-drive Storage Pods arranged into the Backblaze Vault configuration of 20 Storage Pods per vault. In 1Q19, the last stand-alone 45-drive Storage Pod was retired. Therefore, using 45 drives as the threshold for qualification to our quarterly report seems antiquated. This is a good time to switch to using Drive Days as the qualification criteria. In reviewing our data, we have decided to use 5,000 Drive Days as the threshold going forward. The exception, any current drives we are reporting, such as the Toshiba 5TB model with about 4,000 hours each quarter, will continue to be included in our HDD Stats reports.

Fewer Drives = More Data
Those of you who follow our quarterly reports might have observed that the total number of HDDs in service decreased in 1Q19 by 648 drives compared to 4Q18, yet we added nearly 60PB of storage. You can see what changed in the chart below.

Lifetime HDD Stats
The table below shows the lifetime failure rates for the HDDs we had in service as of March 31, 2019. This is over the period beginning in April 2013 and ending March 31, 2019.

Predictions for the Rest of 2019
As 2019 unfolds, here are a few guesses as to what might happen over the course of the year. Let’s see what you think.

By the end of 2019, which, if any, of the following things will happen? Let us know in the comments.
• Backblaze will continue to migrate out 4TB drives and will have fewer than 15,000 by the end of 2019; we currently have about 35,000.
• We will have installed at least twenty 20TB drives for testing purposes.
• Backblaze will go over 1EB of available cloud storage. We are currently at about petabyte of available storage.
• We will have installed, for testing purposes, at least one HAMR based drive from Seagate and/or one MAMR drive from Western Digital.

If you just want the summarized data used to create the tables and charts in this blog post you can download the ZIP file containing the MS Excel spreadsheet.