Coho Data About New Pure Storage FlashBlade

This article was written on the blog of Coho Data, Inc. by Andy Warfield, co-founder and CTO of the company

Musings about enterprise IT
Marketing your way to scale-out

Last week, Pure Storage announced their FlashBlade: a blade-based flash storage array that will be available for purchase a little over three quarters from now.

Here’s a weird fact: in early 2013, the Coho Data engineering team took the decision not to build a remarkably similar platform. While most of our engineering team was developing the initial version of Coho’s storage software, we had a small effort to build the densest, network integrated hardware platform that we could get that software to run on. This is the story of why, in the end, we decided not to do it.

Density, compute and connectivity
FlashBlade addresses a problem that I have written about extensively over the past few years: the performance density of PCIe flash demands completely fresh architectures for storage systems. PCIe flash needs more compute and it needs more connectivity, while its falling prices demand the ability to purchase storage incrementally. The performance density of these devices argues strongly for the disaggregation of compute and storage resources into separate, elastic services.

Back in 2012, we saw these problems very clearly as we first started to work with PCIe flash: the cards were so fast that a single device could saturate a 10Gb NIC. You needed a lot of CPU to drive them that fast, which was what led Coho to build systems that could scale compute, storage, and connectivity together. The question we were left with was one of density: these flash devices were incredible, but they needed to be paired with compute and network to really scale well. So how could we pack these three things together as tightly as possible into a single physical form factor?

Working with Intel and a fantastic boutique design firm, we started to explore the idea of taking Intel’s Seacliff Trail 10Gb top of rack switch architecture that had been released to the Open Compute Project, and to use it as the basis for a backplane of a blade-based storage server. We called the prototype system the Coho ‘SwitchStore.’  The idea was to build a completely converged top of rack switch and storage system: half of the ports would be exposed as an Ethernet switch, while the other half would be made available internally for storage blades. Each blade, in our design, would be a single modular PCIe flash device connected to a server card and then into the switch’s 10Gb backplane. To this day, I love the aesthetic of this design and the technical enthusiast in me can’t help drooling over the glossy 3D-rendered images that accompany Pure’s much more recent announcement, and the incredible amount of hardware design work that has gone into it.

Scale-out is more than this.
So why did we decide not to build the SwitchStore? The further along that we got with the design, and at the same time continued to work with an all-software prototype with some very early Coho customers, we were struck by a set of practical realities that made us completely put the brakes on building our own hardware. Today, I think these lessons apply equally, if not more, to Pure’s array.

First of all, we decided that it’s a really bad idea to bet against commodity hardware. The rate of innovation on commodity hardware today is remarkable, and the cost savings associated with large-volume commodity systems are profound. Building a system based on proprietary hardware involves longer cycle times, higher costs, bigger challenges in getting validation and QA right: it’s easy to take for granted the breadth of test and quality improvement that commodity hardware platforms get just as a result of sheer volume deployment. At the end of the day, the decision to build your own hardware means you end up being slower to market, on a less reliable platform that doesn’t turn corners well as other interesting new technologies arrive to compete with you. The design process around Coho’s SwitchStore demonstrated this: over the year that we investigated it, the density of a typical commodity multi-server chassis doubled, and whitebox 40Gb switches started to emerge. The idea of being stuck with the delivery lifecycle of proprietary hardware scared the crap out of us. Remarkably, while Pure appears to be trying to pay homage to the proprietary hardware bent of systems like EMC’s Clariion, even NetApp has realized that working commodity gear allows modern storage systems the opportunity to really innovate in software over breakneck changes in storage hardware capabilities. Personally, I couldn’t agree with this more.

Second, scale-out is also a network problem. The issue of performance density in flash isn’t limited to single devices: the specific problem with building a hyper-dense chassis of high performance flash is that you are effectively building a Disneyland-caliber theme park at the end of a one-lane, winding gravel road. It’s hard to keep the thing busy. The FlashBlade looks like it will suffer from exactly this problem. On one hand, the apparent 160Gb/s (4×40) of external connectivity represents a significant restriction of what a fully-loaded chassis should be capable of delivering internally. But on the other hand, even at 160Gb/s, the array itself represents a consolidation of enormous amounts of storage performance into a single rack. To keep that loaded chassis busy, you likely need to involve compute from multiple other racks, and as a result need to draw high-rate storage traffic across the core of the network. In contrast, Coho has recently been working with large enterprise deployments that desire to scale storage nodes across multiple racks – effectively providing ‘top of rack storage’ – in order to ensure that the majority of storage requests to be served in a rack-local manner. As a result, the same technical argument that suggested that you build blades in the first place comes back to suggest that you not build blades at all: if you are going to scale-out storage performance, you should do it in a way that is flexible enough to address the broader topology and network efficiency of your datacenter.

Finally, scale-out solves a business problem. Large enterprise environments are really sick of two things: they are sick of paying through the teeth for storage hardware, and they are sick of wasting time coaxing that hardware into something that is actually useful to the business. Not only are enterprise storage companies today competing with each other, they are competing with the fact that the dated notion of storage as an operationally complex hardware device is being held up and compared to cloud-based storage systems that offer elastic scale, consumption pricing, and a rolling cadence of new features over time. When I talk to our customers about scale-out, it is exactly these properties that we discuss: true scale-out is about flexibility, the flexibility to adapt to emerging hardware, to buy hardware as you need it, to maximize the efficiency of how that hardware works in your environment, and to largely ignore the ‘hardware’ aspects of that hardware altogether. Scale-out is an all-out attack on the complexity of traditional enterprise storage systems, and an effort to demonstrate that enterprises can realize the same benefits that are offered by scalable public cloud-based storage systems in a form factor that resides on premises.

Proprietary hardware flies in the face of almost all of these goals. Customers pay more for a bespoke, lower-volume device that is slower to evolve and locked into a backplane and topology that is decided at the initial time of purchase. That isn’t scale-out, it’s a traditional EMC-style array with a partially-populated shelf.

Pure is very aggressively trying to position the FlashBlade as a scale-out storage product. Unfortunately, the FlashBlade is not a scale-out storage product. Certainly not in the way that real scale-out companies are using that term to describe the benefits of scale-out today. Instead, the FlashBlade is an expensive, proprietary, blade-chassis-based array. Most of the real effort to make the FlashBlade scale-out appears to have been assigned to their marketing department, who made sure to use the term ‘scale-out’ in pretty much every box they could fit it into on Pure’sarchitecture diagram.

The FlashBlade is a really interesting, impressively complicated, proprietary piece of enterprise storage hardware. I am impressed with its ambitious (albeit 1990’s era) hardware co-design, and I really appreciate the chops of a software team that elects to take on the absolutely masochistic challenge of writing stable enterprise code for three different processor architectures (Xeon, ARM, and FPGA) at the same time.

Please though Pure marketing team: let’s not co-opt more terms in an already unclear storage vocabulary. Scale-out, you are not.