WekaFS in AWS Under Stac-M3 Benchmarks

STAC (Securities Technology Analysis Center LLC) recently performed STAC-M3 benchmarks on a stack involving kdb+, the WekaFS parallel filesystem, and an AWS EC2 cluster.

STAC-M3 is the set of industry standard enterprise tick-analytics benchmarks for database software/hardware stacks that manage large time series of market data (‘tick data’). In this project, it ran both the baseline benchmark suite (code named Antuco) and the optional scaling suite (code named Kanaga).

The stack under test (SUT) was Kx’s kdb+ 4.0 database system distributed across 15 Amazon EC2 C5n.9xlarge instances, using the WekaIO, Inc. (Weka) WekaFS parallel file system to access data stored in a cluster of 40 Amazon EC2 i3en.6xlarge instances that provided a total of 546TB of SSD capacity.

Weka chose to highlight the following from the results:

In terms of mean-response time benchmarks, kdb+ 4.0 solution in AWS with 15 database server nodes and 40 storage nodes:

• Outperformed all publicly disclosed results in 3 of 24 Kanaga benchmarks

• vs. a kdb+ 4.0 solution running on a 10-node cluster with 60TB of persistent memory (KDB200603):

  • • was faster in 16 of 24 Kanaga and 9 of 17 Antuco benchmarks

• vs. a kdb+ 3.6 solution on a parallel file system with 15 database servers accessing all-flash storage appliances (KDB200915):

  • • was faster in 20 of 24 Kanaga benchmarks and 4 of 17 Antuco benchmarks

• vs. a kdb+ 3.6 solution involving a previous version of Weka with 14 on-prem database nodes and 18 storage nodes (KDB200401):

  • • was faster in 3 of 24 Kanaga and 4 of 17 Antuco benchmarks

• vs. a kdb+ 3.6 solution involving 9 database servers accessing networked flash storage (KDB200914):

  • • was faster in 15 of 17 Antuco benchmarks

In terms of throughput benchmarks (MB/s), solution:

• Outperformed all publicly disclosed results in 3 of 5 Kanaga benchmarks
  (STAC-M3.β1.1T.{3,4,5}YRHIBID.MB/S)

Results are available (registration required). Premium subscribers have access to the code used in this project as well as the micro-detailed configuration information for the solution.