Industry Validation of DDN Infinite Memory Engine

Continuing to build on the company’s HPC innovation, at this year’s SC14, DataDirect Networks, Inc. (DDN) is revealing broad customer adoption and latest performance testing of its Infinite Memory Engine (IMETM) burst buffer and application acceleration technology.

For a range of performance-hungry HPC applications, such as real-time analytic processing and modeling of big data across life sciences, oil and gas, research and financial services industries, where support for high bandwidth and high IO/s workloads is essential, DDN is placing previews of IME technology into HPC facilities around the world and currently has placed test beds in over a dozen Top100 sites.

Planning for exascale, accelerating time to discovery and extracting results from massive data sets requires organizations to continually seek faster and more efficient solutions to provision I/O and accelerate applications. 

IME is an application-aware, non-volatile memory enabled acceleration engine and buffer cache. It resides between the high-performance compute and parallel file system in an HPC environment, and maximizes compute utilization without crippling the storage back-end. By virtualizing SSDs across the entire environment into a single fast tier of storage, users now can take advantage of the declining price of commodity hardware and deliver near in-memory performance to accelerate I/O, reduce latency and provide operational and economic efficiency.

IME delivers scale-out data protection with distributed erasure coding, integration with both Lustre and GPFS environments, intelligent adaptive data placement, Posix compatibility and standards-based HPC application interfaces.

It is designed for HPC users looking to benefit from technology that delivers improvements in checkpointing and I/O-intensive HPC application performance.

Heralding a step forward in DDN’s software defined storage strategy, the IME technology demonstrates DDN’s focus on designing HPC and big data solutions, which also include WOS Object Storage technologies as well as DDN’s optimized SFA storage appliances.

IME Decouples Physical Storage from Compute Resources
to Redefine the I/O Provisioning Paradigm,
Driving Increased I/O and Application Performance, Flexibility
and Lower Total Cost of Storage

•  IME enables organizations to separate the provisioning of peak and sustained performance requirements with up to 70% greater operational efficiency and cost savings than utilizing exclusively disk-based parallel file systems.
• Inserting IME between the compute and parallel file system removes Posix semantics that bring cluster performance to its knees due to the proliferation of small and mal-aligned I/O that can often make up 90% of the HPC workload. By shielding parallel file systems from fragmented I/O in cache, with IME users now can run jobs at or near line rate with more jobs in parallel resulting in faster time to insight.
• Unlike simple burst buffer technologies that have been designed for write-intensive, large file sequential operations such as checkpoint-restart, IME enables acceleration across an array of HPC applications and workloads with no application modifications required.
• In addition, IME enables greater flexibility and choice for end users due to its non-vendor-captive software-based approach. DDN’s open, no vendor lock-in solution provides flexibility in how users can architect their environments and a selection of specialty and commodity hardware platforms and components.

Molly Rector, CMO, EVP product management and WW marketing, DDN, said: “DDN’s IME delivers a wide range of advances that will enable the next generation of HPC capabilities. IME is a complete game changer in application acceleration, performance optimization, SSD utilization and high-performance storage. Years of development and investment from some of the greatest minds in HPC have delivered IME as a transparent solution to any data-intensive HPC application. Building on DDN’s unparalleled leadership in delivering extreme performance, and with more than a dozen active test beds in use today, this announcement delivers once again on DDN’s leadership in HPC.”

Gilles Civario, senior software architect, ICHEC, said: “DDN’s IME can transform the I/O-bound problem back into a compute-bound problem. Using IME, ICHEC is seeing a 3x acceleration of the entire Tullow Oil production application – not just the I/O portion. Clearly there is great potential for this technology in this and other sectors.”

Tommy Minyard, Phd, director of advanced computing systems,Texas Advanced Computing Center, The University of Texas at Austin, said: “TACC is very excited to participate with DDN to perform large scale testing of DDN’s IME on Stampede, currently one of the world’s fastest supercomputers. This type of joint partnership testing enables TACC to prove powerful advanced computing technologies for the open science community.”

IME is available in limited release as part of DDN’s IME testbed program.