Storage I/O Performance Number One Issue in HPC for Big Data

HPC end users are elevating the discussion about storage and recognizing the importance of storage I/O – not just compute – in an effort to hit the exascale threshold by the end of the decade according to the results of a DataDirect Networks, Inc.‘s HPC trends survey.

The survey polled a cross-section of 60 DDN HPC end users representing hundreds of petabytes of storage investment, which included individuals responsible for HPC allocations encompassing HA storage and file systems across a variety of industries. Of those surveyed, 60% manage or use at least 1PB of storage, and over 20% manage or use 10PB or more.

According to the survey results, more than two-thirds (68%) of those polled agree that data and storage has become the most strategic part of the HPC datacenter. Moreover, by a margin of two to one respondents, the survey also reveals that today’s storage technologies will need to undergo massive change to hit exascale proportions.

In addressing this challenge, the overwhelming majority of respondents (78%) agree that hybrid storage is the evolutionary next step for HPC storage, combining both the performance and cost efficiency benefits required of storage at exascale.

Future Advances in Storage Performance to Deliver Massive I/O Gains
and Reduction in Storage Costs for HPC, Analytics and Cloud Computing Environments

• With a nod to future technology requirements, the survey also shines light on the biggest storage and/or big data challenges for HPC users today, with nearly half of respondents citing mixed I/O performance the primary issue (47%), followed by managing rapid data growth (22%), performance (13%) and multi-site replication or collaboration (5%). 13% of those polled name ‘other’ concerns, including the need to manage the overall total cost of storage without compromising storage scalability and performance.
• With storage performance a critical requirement especially at today’s petascale levels, of those respondents with an opinion, 85% agree that site-wide file systems like those at at Oakridge National Laboratory (ORNL), National Energy Research Scientific Computing Center (NERSC) and Texas Advanced Computing Center (TACC) are the next infrastructure trend in HPC environments.
• With HPC no longer limited to data centers at research institutions of the Top500 Supercomputing sites, the survey also examines end users opinions on analytics and integrating computing workloads into large scale HPC cloud environments. More than two-thirds (70%) of those polled point to storage I/O as the biggest bottleneck in analytics workflows, suggesting that compute is no longer the main challenge for analytics platforms. While multi-site HPC collaboration may be on the rise, security and data sharing complexity remain the biggest impediments according to 75% of those surveyed.
• Finally, HPC organizations continue to find themselves under increasing funding scrutiny and according to nearly three quarters of the survey respondents, it has become more difficult for HPC sites to obtain funding for compute infrastructure without demonstrating ROI.

Jeff Denworth, VP marketing, said: “The results of DDN’s annual HPC Trends Survey reinforce what we hear from customers on a daily basis. Users are identifying storage I/O as the primary challenge in computing and the industry conversation needs to shift to reflect that. As the evolutionary gap between compute and storage media grows, DDN is focused on conquering the I/O performance problem and providing new approaches and innovative solutions to help our customers reach any level of computing scale faster, more efficiently and most cost effectively.”