Weka Dominates SPECstorage Benchmark, Shattering Records

Blog post written by Colin Gallagher, VP product marketing, WekaIO

Update
WekaIO is a company of principles. Following those principles means sometimes doing things that are uncomfortable, such as expressing regret when mistakes are made. In a prior version of this blog, we went a step beyond what we should have and made statements that were overreaching on a technical and business level and should be ignored. For SPECstorage 2020, the only items to have a conversation on are the load point value and ORT. Anything else, such as applying latency as an extrapolation, is a speculative estimation and should not be used. SPEC specifically calls this out at https://www.spec.org/fairuse.html In addition to this, we addressed several of our competitors in a manner that benefited no one.

As such, we are apologizing: To SPEC, to our competitors, and to the readers of this blog. You expect better, and WekaIO has the principles to stand up and acknowledge the work done by SPEC in making a fair and transparent set of benchmarks that have rigorous rules for how to talk to benchmark results. We also want to acknowledge the work done by our industry peers and competitors before us. That work has paved the way with many technological innovations that are well regarded and have raised the bar. Your competition is what drives everyone including WekaIO to be better, and the company will be more principled in how we talk about you. With our sincere apologies to SPEC, our competitors, and our readers, we have updated the blog below on the recent firm’s results.

We announce a new set of SPECStorage Solution 2020 benchmarks showcasing the WekaIO Data Platform’s performance capabilities in hyperscale cloud environments.

As noted in a previous blog, firm’s philosophy is to be as transparent as possible. We use highly structured and audited benchmarks whenever possible and include detailed configuration information when providing internal testing results so customers can expect repeatable results if they deploy the same configurations. SPEC benchmarks provide the details to allow this repeatability.

Record-Breaking Benchmark Results

First up, a new SPEC_ai_image result focused on value. A recent vendor submission in this category using a cloud hyperscaler showed a result with a load point of 400 jobs with an Overall Response Time (ORT) of 1.22ms. The vendor produced 700 jobs at an ORT of .85ms.

Using the pricing calculators for Amazon Web Services and Azure, we also discovered that company’s infrastructure cost was less than that of the benchmarked competitor.

Benchmark	Infrastructure	Results
SPEC_ai_image	Azure	WekaIO running in Microsoft Azure did 175% the number of AI_jobs at 64% of the infrastructure cost, resulting in a 2.7x lower cost per job based on infrastructure costs alone

While cost efficiency is a valuable measure, sometimes outright speed is needed. WekaIO then deployed a larger cluster with zero tuning changes between the different benchmarks to showcase lightning performance capabilities across all IO profiles. The results speak for themselves.

Benchmark	Infrastructure	Results
SPEC_ai_image	AWS	#1 position WekaIO delivers 6x higher load count (2,400) than another cloud based benchmark at only 3/4 the cost per job. Net: Big or small, in multiple clouds, WekaIO is faster and has a better cost-per-job.
SPEC_vda	AWS	#1 position Company still holds the #1 spot from 2 years ago with a compact on-prem system (8,000 streams). We beat it with 12,000 streams in the cloud. Net: On-prem or in the cloud, the firm is the highest performing video benchmark
SPEC_genomics	AWS	#1 position Took the #1 spot from a small, specialist vendor in this benchmark with 2,200 jobs achieved.
SPEC_eda_blended	AWS	#1 position The vendor delivered 6,310 jobs at 0.87ms ORT. It delivered a 60% lower ORT in the cloud vs. an on-prem NVMe based all-flash system.
SPEC_sw_build	AWS	#2 position The company achieved 3,500 builds with an ORT of 0.74ms compared to an on-prem NVMe based all-flash system that achieved 6,120 builds at 1.58ms ORT. The results cannot be correlated between the two other than where the build counts overlap. We invite the reader to view the graphs and data on the SPEC website to compare for themselves.

What makes these results so interesting isn’t just that the company is either the #1 or #2 result in all the SPECstorage 2020 benchmarks – it’s the impact of being able to handle any IO profile with zero tuning changes between benchmarks. Read, write, metadata-intensive, IOP-driven and throughput-driven IO are all handled equally efficiently. For customers, this advantage in handling any IO profile shows up as real wall-clock time reductions in time-to-completion of data pipelines. It enables AI workloads to checkpoint often without stalling model training and increases utilization rates of GPUs attached to WEKA.

Conclusions
The firm is performant whether in any cloud or on-prem. With the hyperscalers continually improving the capabilities of their back-end infrastructure, the public cloud can now provide comparable results to on-prem for storage. This gives customers the confidence to deploy their applications and workflows in the infrastructure that best meets their needs without compromising performance.

Everyone wins when the Data Platform’s performance breaks records. It inspires companies to challenge themselves to try and ‘up their game’ while removing legacy constraints that have bottlenecked the most performant AI workloads. Technologies get upgraded, organizations have more options, and big problems have the potential to get solved faster than they ever have before. But we’re not just talking about infrastructure solutions. We’re committed to eliminating the compromises of the past by redefining modern storage performance in the cloud and AI era. Watch this space as we keep breaking records while building the AI-native data platform of the future. Here’s to breaking down barriers and shattering records for many years.

Click to enlarge