StorPool With Record of 13.8 Million IO/s

StorPool Ltd. holds a world record: 13.8 million IO/s.

The company unveils performance test report with NVMe-class storage performance in VMs.

In this performance test the firm measured 13.8 million IO/s on a 12-node cluster built on standard off-the-shelf Intel servers.

This is a high storage performance reached by an HCI. The company also shows that even a small the firm’s storage system, with just 48 NVMe drives, can provide millions of IO/s at latencies of a fraction of a millisecond. For real-world applications, microsecond latencies translate to remarkable application responsiveness and performance.

It delivers on the SDS promise – fast and highly available shared-storage system, built with standard servers, with rich data services (snapshots/clones, thin provisioning, end-to-end data integrity, multi-site functionality). The results show that with the solution, a private or public cloud can deliver high-available VMs with the performance of local NVMe SSDs.

The firm configured and tested a 12-server KVM hyper-converged system on hardware provided by the Intel Corp.^(*)Datacenter Builders program. The tests show this 12-node hyper-converged environment delivers 13.8 million random read IO/s at latency of 404 microseconds. R/W latencies under low load are 137 and 95 microseconds respectively. These are results impossible with the vast majority of storage solutions.

Test results

IO/s and MB/s numbers in this table were rounded down to three significant digits for clarity of presentation.

^(*) backend IO/s and backend GB/s calculation estimates load on storage servers and underlying NVMe drives. Write operations are counted three times for 3x replication. Read operations are counted as one backend operation each. For example 20,800MB/s writes by FIO translate to 63,400MB/s ‘backend’ writes on underlying drives.

Here is some context to aid understand the results.

Understanding results in context
In this 12-node StorPool-powered hyper-converged set-up we run 96 VMs, each executing a single FIO job. From just 48 NVMe drives, StorPool delivers 143,000 IO/s per VM simultaneously to 96 VMs. This performance is well above typical requirements, so a cloud using the firm’s solution can safely focus on delivering the best service to their customers, instead of constantly battling with high latencies under load and ‘noisy neighbors’ problems, which are typical for many storage solutions.

In the sequential tests, 64.6GB/s is equal to 86% of the theoretical network throughput (75GB/s).

In the random R/W tests, 13.8 million IO/s (*) 4 KiB is equal to 75% of theoretical network throughput. The relatively small number of NVMe drives (48 drives total) is the limiting factor for this benchmark.

There is a performance test by Microsoft Corp. on a similar 12-node hardware set-up, titled “The new HCI industry record: 13.7 million IO/s with Windows Server 2019 and Intel Optane DC persistent memory“. In the Microsoft S2D/Hyper-V/Optane test, because of the very small active set (3.0TB) and caching, almost all storage operations are served from RAM or Optane memory, not from the underlying NVMe storage drives. Storage experts like Chris M Evans call this ‘cheating’, rightfully so in our opinion. The company’s system is not only faster, with IO being processed by the actual NVMe drives, but also uses less storage hardware.

The 95 microseconds write latency includes three network transfers and three NVMe drives committing the write. It is ten times faster than the ‘sub millisecond’ latency advertised by traditional AFAs. For the first time one can get a shared storage system, as fast as locally attached drives (DAS).

In all tests here, the company performs end-to-end data integrity checking (generating, storing and verifying checksums). End-to-end data integrity is crucial to protect the data against hardware and firmware issues, but most storage systems don’t have this functionality.

Description of the environment
The setup consists of twelve servers with identical hardware configuration, provided by the Intel Datacenter Builders Program. Each server is connected using dual 25GbE to an Ethernet switch. In this setup there is a single switch for simplicity. Production-grade environments typically use two switches for redundancy.

Following diagram illustrates physical topology:

Each of twelve servers has the following hardware configuration:

56 cores/112 threads (two 2nd Gen Xeon Scalable CPUs)
384GB RAM (twelve 32GB RDIMMs)
32TB NVMe (four Intel P4510 8TB NVMe drives)
Intel XXV710 dual-port 25G NIC

Software installed on each server is:

OS – CentOS 7.6, Linux kernel 3.10.0-957
Hypervisor – KVM, libvirt and qemu-kvm-ev from CentOS Virt-SIG
SDS – StorPool v19
Storage benchmarking software – FIO

The following diagram illustrates how the hardware and software components inter-operate with each other. For simplicity it shows just one of the twelve servers.

Each host runs eight VMs, each VM has four vCPUs, 1GB RAM and a 500GB virtual disk on StorPool. VMs are used only for running the benchmark tools.

In this test scenario the company uses approximately 11% of the total CPU resources. The remaining 89% of the CPU cores are free for running VMs.

To process more than 1 million IO/s per node the solution uses six CPU cores (incl. StorPool server, StorPool client).

Memory usage:

25GB per node (6.5%) for StorPool (client and server combined) and
359GB per node (93.5%) remain available for the host OS and VMs

The memory usage for the firm’s solution is small – less than 1GB RAM per 1TB raw storage, and it’s mostly used for metadata.

Testing methodology
The firm use the FIO client-server functionality to perform the benchmark. A FIO server is installed in each of the 96 VMs. A FIO client is configured to connect to the 96 FIO servers. The FIO servers perform IO operations using the test profile specified by the FIO client. The FIO client collects and summarizes the benchmark results.

The firm performed tests designed to measure the performance envelope of the system under test. The performance envelope covers the extremes of the capabilities of a storage system and is sufficient for general characterization of performance.

Maximum IO/s
- - random read
  - random R/W
  - random write
Maximum MB/s
- - sequential read
  - sequential write
Minimum latency
- - random read with queue depth 1
  - random write with queue depth 1

Parameters used for tests are summarized in following table:

In all tests FIO was configured as follows:

ioengine=libaio – Use Linux Native Async IO;
direct=1 – Bypass the guest’s buffer cache;
randrepeat=0 – Produce a unique random pattern on every run instead of repeating previous random patterns, to avoid cache effects;
numjobs=1 (default) – Use a single thread for execution.

Quality controls for test results

No data locality: Data for each VM was specifically placed on non-local drives to avoid any positive skew caused by the data being locally stored.
No caching: There is no data caching anywhere in the stack. Each read or write operation is processed by the NVMe drives. The number of operations processed by the underlying drives was compared to the number of operations counted by the benchmark tool. The numbers were identical.
Active set size: The active set was sufficiently large to exclude effects from caching, even if it was used. The 48,000 GiB active set is equal to 44% of system capacity.
No write buffering: All write operations were submitted by the test tools with a ‘sync’ flag, instructing the storage sytem to persist them to power-loss protcted devices (NVMe drives) before completing the write operations.
Repeatability and steady state: Tests were run for sufficient duration, such that running them for longer does not influence the result.
End-to-end measurement: IO/s, MB/s and latency are measured by the test tool (FIO) inside the VMs, so the presented results are end-to-end (including all overheads across the storage and virtualization stacks)

Why these numbers matter
Storage has been one of the main limiting factors in modern IT. There has always been a trade-off between speed and HA. Here we prove that with a best-of-breed software-defined storage solution, you can eliminate this trade-off altogether. Having a shared storage system (highly available) with the performance of local NVMe drives is now possible.

By using this technology any public and private cloud builder can deliver unmatched performance for their applications, VMs and containers. If you aim to build a powerful public or private cloud, this solution can permanently solve all your storage performance issues.

Additional resources:
Public cloud performance measurement report

Read also:
Comparison of 7 Block Storage Offerings of Popular Public Clouds by StorPool
AWS, Digital Ocean, eApps, OVH, DreamHost, StorPool, ToggleBox
April 26, 2019 | Press Release