Backblaze Storage Pod Enhanced

This article was written on October 5, 2022 by Andy Klein, principal storage cloud evangelist, Backblaze Inc.

The Storage Pod Story: Innovation to Commodity

It has been over 6 years since we released Storage Pod 6.0. Yes, we have improved that system since then, several times. We’ve added more memory, upgraded the CPU, and of course deployed larger disks. I suppose we could have written blog posts about those improvements, a Storage Pod 6.X post or 2 or 3, but somehow that felt a bit hollow.

About 18 months ago, we talked about The Next Backblaze Storage Pod. We had started using Dell servers in our Amsterdam data center, although we were still building and deploying the version 6.X storage pods in our US data centers. That changed about 6 months ago and we haven’t built or deployed a Storage Pod since that time. Here’s what we’ve done instead.

A Backblaze-Worthy Storage Server
In September of 2019, we wrote a blog post to celebrate the 10-year anniversary of open sourcing our Storage Pod design. In that post we mused about the build/buy decision and stated the criteria we needed to consider if we were going to buy storage servers from someone else: cost, ease of maintenance, the use of commodity parts, ability to scale production, and so on. Also in that post, we compiled a list of storage servers on the market at the time which were similar to our Storage Pod design.

We then proceeded to test several different storage servers from the list and elsewhere. The testing was done over a period of about a year using the criteria noted earlier. The process progressed and one server, a 60-drive Supermicro server, was selected to move on to the next stage, production performance testing.

Here we would observe the server’s performance and test its compatibility with our operational architecture. We built a vault of 20 Supermicro servers and placed it into production, and at the same time we placed a standard Storage Pod vault into production. The 2 vaults ran the same software and we would track each vault’s performance throughout.

When a Backblaze Vault enters production, 60 tomes of storage come online at the same time joining thousands of other tomes ready to receive data. Each vault has the same opportunity to load data, but this will vary depending on the performance of the vault to process the requests received. In general, the more performant the vault, the more data it can upload each day.

The comparison of how much data each vault uploaded each day is shown below. Vault 1084 is composed of 20 Supermicro servers and Vault 1085 is composed of 20 Storage Pods.

The Supermicro vault (1084) started with a limit of 2,500 simultaneous connections allowed for the first 7 days. Once that limit was lifted and both vaults were set to 5,000 simultaneous connections, the Supermicro vault generally outperformed the Backblaze vault over the remainder of the observation period.

What happened to the data once the test was over? It stayed in the Supermicro vault and that vault became a permanent part of our production environment. It is still in operation today, joined by over 1,100 additional Supermicro servers. Safe to say, we moved ahead with using these servers in our environment in place of building new Storage Pods.

Server Model We Use
The Supermicro model we order is the PIO-5049P-E1CR60L (PIO-5049). That model is not sold via the Supermicro website. That said: model SSG-6049P-E1CR60L (SSG-6049) is similar and is widely available. Both models have 60 drives, but the chassis is slightly different, and the motherboards are different with the PIO-5049 model having a single CPU slot, and the SSG-6049 model having 2 CPU slots. Let’s compare the basics of the two models below.

In practice, the SSG-6049 model supports newer components such as the latest CPUs and allows more memory vs. the PIO-5049 model, but the latter is more than capable of supporting our needs.

Can You Build It?
A little over 13 years ago, we wrote the Petabytes on a Budget blog post introducing Storage Pods to the world and open sourcing the design. Since then, many individuals, organizations and businesses have taken the various storage pod designs we published over the years and built their own storage servers. That’s awesome.

We know building a Storage Pod was not easy. Oh, the assembly was simple enough, but getting all the parts you needed was a challenge: searching endlessly for 5-port backplanes (minimum order quantity 1,000-ouch, sorry) or having to build your own power supply cables. While many of you enjoyed the challenge; many didn’t.

For the Supermicro system, let’s work with the SSG-6049 model as it is available to everyone and see what it would take for you to acquire/assemble/build a single Supermicro storage server.

Option One: Go Standard
The easiest thing to do is to order a pre-configured SSG-6049 model or you can try one of their online reseller sites such as Canada Computers & Electronics or ComputerLink, which offer the same “complete system”. In these cases, the ability to customize the server is minimal and requires direct contact with the vendor for most changes. If that works for you, then you’re all set.

Option Two: Configure
If you want to design your own system you can try Supermicro resellers such as IT Creations (US) and Server Simply (EU) which have configurators that allow you to select your CPU, motherboard, network cards, memory, and various other components. This is a great option but given the number of different options and the possibility of incompatibilities between components, you need to be careful here. Don’t rely on the configurator to catch a component mismatch.

Option Three: Create
Here you might buy the most stripped-down server you can find and replace nearly everything inside – motherboard, CPU, fans, switches, cables and so on. You’ll probably void any warranty you had on the system, but we suspect you knew that already. Regardless, you can take the base system and stuff it full of smoking-fast everything so that your copy of “Ferris Buellers Day Off” downloads in picoseconds. That’s the fun part of building your own storage server, when you are done it is uniquely yours.

Which option you choose is, of course, your choice, and while ordering a standard system from Supermicro may not be as satisfying as soldering heat sinks to the motherboard or neatly tying off your SATA cable runs, it will give you more time to watch Ferris, so there’s that.

FYI, Supermicro has an extensive network of resellers around the world. While the options above fall neatly into 3 categories, each reseller has their own way of working with their clients. If you are going to buy or build your own Supermicro storage server or have already done so, share your experience with your colleagues in the comments below or on your favorite forum or community site.

What About Pricing?
Supermicro does not publish prices and we are not going to out them here, but we wanted to see if we could determine the street price for the SSG-6049 system by surveying reseller websites. It was not pretty. In our research, we saw prices for the SSG-6049 model range from $6,000 to 40,000 on different reseller sites. On the website with the $6,000 price they started with a fictitious base system that you could not order, and then listed the various components you were required to add, such as CPU, memory, HDDs, etc. At the $40,000 website the reseller didn’t bother to list any of the components; it just had the model and the price-no specs or technical information. Classic buyer beware scenarios in both cases.

The other variable that made the street price hard to determine was that resellers often bundled other services into the price of the system such as installation, annual maintenance, and even shipping. All are reasonable services for a reseller to offer, but they cloud the picture when trying to determine the actual cost of the product you are trying to buy. At best, we can say that the street price is somewhere between $20,000 and $30,000, but we are not very confident with that range.

Storage Server Pricing Over Time
Since 2009 we have tracked the cost per gigabyte of each Storage Pod version we have produced. We’ve updated the chart below to add both Storage Pod version 6.X, our most current Storage Pod configuration, and the Supermicro storage server we are buying, model PIO-5049.

The cost per gigabyte is computed by taking the total hardware cost of a storage server, including the HDDs, and dividing by the total storage in the server at the time. When Storage Pod 1.0 was released in September 2009, the system cost was about $0.12/GB, and as you can see that has decreased over time to $0.02/GB in the Supermicro systems.

One point to note is that both the Storage Pod 6.X ($0.028/GB) and Supermicro ($0.020/GB) servers use the same 16TB HDD models. We believe the difference between the cost per gigabyte of the 2 cohorts ($0.008) is primarily based on the operational efficiency obtained by Supermicro in making and selling tens of thousands of units a month vs. Backblaze assembling a hundred 6.X Storage Pods on our own. In other words, Supermicro’s scale of production has enabled us to get performant systems for less than if we continued to build them ourselves.

What’s Next for Storage Pods
No one here at Backblaze is ready to write the obituary for our red Storage Pods. Afterall, the innovation that was the Storage Pod created the opportunity for Backblaze to change the dynamics of the storage market. Now that the Storage Pod hardware has been commoditized, our cloud storage software platform is what enables us to continue to deliver value to businesses and individuals alike.

All that means is that our next Storage Pod probably won’t be an incremental change, but instead something completely new, at least for us. It may not even be a Storage Pod – who knows? That said: we will continue to upgrade our existing Storage Pods with new CPUs, memory, and such, and they’ll be around for years to come. At which point we may give them away or crush them (again). In the meantime, we’ll probably do another blog post or two so we can post a few pictures and tell a few stories. Or maybe we’ll just move on. Hard to say right now.