What are you looking for ?
Infinidat
Articles_top

Backblaze Updates Reliability Report, on 68,813 HDDs

Following acquisition of new Seagate 8TB drives looking good

Backblaze,kleinThis study was published on the blog of Backblaze, Inc. on August 2, 2016, and written by Andy Klein.

 

 

HDD Stats for Q2 2016

Q2 2016 saw Backblaze: introduce 8TB drives into our drive mix, kickoff a pod-to-vault migration of over 6.5PB of data, cross over 250PB of data stored, and deploy another 7,290 drives into the data center for a total of 68,813 spinning HDDs under management.

With all the ins-and-outs let’s take a look at how our HDDs fared in Q2 2016.

Backblaze HDD reliability for Q2 2016
Below is the HDD failure data for Q2 2016. This chart is just for the period of Q2 2016. The HDD models listed below are data drives (not boot drives), and we only list models which have 45 or more drives of that model deployed.

Backblaze f1

A couple of observations on the chart:

  • The models that have an annualized failure rate of 0.00% had zero HDD failures in Q2 2016.
  • The annualized failure rate is computed as follows: ((Failures)/(Drive Days/365)) * 100. Therefore consider the number of “Failures” and “Drive Days” before reaching any conclusions about the failure rate.

Later in this post we’ll review the cumulative statistics for all of our drives over time, but first let’s take a look at the new drives on the block.

8TB HDDs have arrived
For the last year or so we kept saying we were going to deploy 8TB drives in quantity. We did deploy 45 8TB HGST drives, but deploying these drives en masse did not make economic sense for us. Over the past quarter, 8TB drives from Seagate became available at a reasonable price, so we purchased and deployed over 2,700 in Q2 with more to come in Q3. All of these drives were deployed in Backblaze Vaults with each vault using 900 drives, that’s 45 drives in each of the 20 Storage Pods that form a Backblaze Vault.

Yes, we said 45 drives in each storage pod, so what happened to our 60 drive Storage Pods? In short, we wanted to use the remaining stock of 45 drive Storage Pods before we started using the 60 drive pods. We have built two Backblaze Vaults using the 60 drive pods, but we filled them with 4TB and 6TB drives. The first 60 drive Storage Pod filled with 8TB drives (total 480TB) will be deployed shortly.

HDD Migration – 85 Pods to 1 Vault
One of the reasons that we made the move to 8TB drives was to optimize storage density. We’ve done data migrations before, for example, from 1TB pods to 3TB and 4TB pods. These migrations were done one or two Storage Pods at a time. It was time to ‘up our game.’ We decided to migrate from individual Storage Pods filled with HGST 2TB drives, average age 64 months, to a Backblaze Vault filled with 900 8TB drives.

We identified and tagged 85 individual Storage Pods to migrate from. Yes, 85. The total amount of data to be migrated was about 6.5PB. It was a bit sad to see the 2TB HGST drives go as they have been really good over the years, but getting 4 times as much data into the same space was just too hard to resist.

The first step is to stop all data writes on the donor HGST 2TB Storage Pods. We then kicked off the migration by starting with 10 Storage Pods. We then added 10 to 20 donor pods to the migration every few hours until we got to 85 pods. The migration process is purposely slow as we want to ensure that we can still quickly read files from the 85 donor pods so that data restores are not impacted.

The process is to copy a given RAID from a Storage Pod to a specific ‘Tome’ in a Backblaze Vault. Once all the data in a given RAID-array has been copied to a Tome, we move on to the next RAID-array awaiting migration and continue the process. This happens in parallel across the 45 Tomes in a Backblaze Vault.

We’re about 50% of the way through the migration with little trouble. We did have a Storage Pod in the Backblaze Vault go down. That didn’t stop the migration, as vaults are designed to continue to operate under such conditions, but more on that in another post.

250PB of data stored
Recently we took a look at the growth of data and the future of cloud storage. Given the explosive growth in data as a whole it’s not surprising that Backblaze added another 50PB of customer data over the last two quarters and that by mid-June we had passed the 250PB mark in total data stored.

You can see our storage growth below:

backblaze F2

Back in December 2015, we crossed over the 200PB mark and at that time predicted we would cross 250PB in early Q3 2016. So we’re a few weeks early. We also predicted we would cross 300PB in late 2016. Given how much data we are adding with B2, it will probably be sooner, we’ll see.

Cumulative HDD failure rates by model
In the table below we’ve computed the annualized drive failure rate for each drive model. This is based on data from April 2013 through June 2016.

bakblaze f3
Some people question the usefulness of the cumulative Annualized Failure Rate. This is usually based on the idea that drives entering or leaving during the cumulative period skew the results because they are not there for the entire period. This is one of the reasons we compute the Annualized Failure Rate using ‘Drive Days’.

A Drive Day is only recorded if the drive is present in the system. For example, if a drive is installed on July 1st and fails on August 31st, it adds 62 drive days and 1 drive failure to the overall results. A drive can be removed from the system because it fails or perhaps it is removed from service after a migration like the 2TB HGST drives we’ve covered earlier. In either case, the drive stops adding Drive Days to the total, allowing us to compute an Annualized Failure Rate over the cumulative period based on what each of the drives contributed during that period.

As always, we’ve published the Q2 2016 data we used to compute these drive stats.

Which HDDs do we use?
We’ve written previously about our difficulties in getting drives from Toshiba and Western Digital. Whether it’s poor availability or an unexplained desire not to sell us drives, we don’t have many drives from either manufacturer. So we use a lot of Seagate drives and they are doing the job very nicely. The table below shows the distribution of the HDDs we are currently using in our data center.

backblaze f4

Recap
The Seagate 8TB drives are here and are looking good. Sadly we’ll be saying goodbye to the HGST 2TB drives, but we need the space. We’ll miss those drives, they were rock stars for us. The 4TB Seagate drives are our workhorse drives today and their 2.8% annualized failure rate is more than acceptable for us. Their low failure rate roughly translates to an average of one drive failure per Storage Pod per year. Over the next few months expect more on our migrations, a look at the day in the life of a data center tech, and an update of the ‘bathtub’ curve, i.e. HDD failure over time.

Read also:
HDD Reliability Based on 114,155 Years of 61,590 Spinning Devices – Backblaze
HGST on top, beating Toshiba, WD and Seagate
2016.05.18 | Press Release

Articles_bottom
AIC
ATTO
OPEN-E