Argonne National Laboratory and DDN Researchers Moved 65TB in Under 100 Minutes From Louisiana to Ottawa

A team of researchers from Argonne National Laboratory and DataDirect Networks, Inc. (DDN) moved 65TB in under 100 minutes at a recent supercomputing conference.

Typically, two days are needed to move this volume of data between sites with a 10Gb/s connection.

With help from Ciena Corporation, Brocade Communication Systems, Inc., and International Center for Advanced Research (ICAIR), the team sustained data transfer rates in excess of 85Gb/s with peaks at over 9 Gb/s between storage systems in Ottawa, Canada, and New Orleans, LA, over a 100Gb/s WAN connection.

The demonstration took place on November 19, 2014 at SC14, the international conference for HPC, networking, storage and analysis.

This achievement required combining the embedded file system and VM capabilities of the DDN storage controller, the high-speed wide-area data transfer capabilities of the Globus GridFTP server, and an advanced 100G WAN.

“Embedding the GridFTP servers in VMs on DDN’s storage controller eliminates the need for external data transfer nodes and network adapters,” explained Raj Kettimuthu, principal software development specialist, Argonne National Laboratory. “We sustained a data transfer rate of 85Gb/s for over 60 minutes-and occasionally for as long as 90 minutes-several times during the SC14 conference.“

Achieving 90+Gb/s for memory-to-memory transfers using a benchmarking tool like iperf is straightforward and has been demonstrated several times in the past. Achieving similar rates for disk-to-disk transfers however, presents a number of challenges, including choosing the appropriate block size that works well for both disk I/O and network I/O, and selecting the appropriate combination of parallel storage I/O threads and parallel TCP streams for optimal performance.

Network experts often claim that storage is the bottleneck in the transfers on high-speed networks, while storage experts claim that the network is often the bottleneck on transfers between sites with parallel file systems.

“This demonstration was aimed at bringing together the experts and latest developments in all aspects concerning disk-to-disk WAN data movement, including network, storage, and data movement tools,” said Kettimuthu.

The team expects that the approach can be used to achieve 100+Gb/s wide-area transfer rates between storage systems using multiple WAN paths and additional storage resources in the end systems.

Team members were Kevin Harms, Eun-Sung Jung, Raj Kettimuthu, Linda Winkler from Argonne National Laboratory and the University of Chicago and Mark Adams from DDN, with help from Jim Chen and Joe Mambretti from ICAIR, Doug Hogg and Marc Lyonnais from Ciena, Wilbur Smith from Brocade, Jon Dugan and Brian Tierney from ESnet, Ian Foster and Mike Link from Argonne National Laboratory and the University of Chicago, and Clayton Walker, Laura Shepard, Susan Presley, and Bob Vassar from DDN.

Argonne National Laboratory seeks solutions to pressing national problems in science and technology. The nation’s first national laboratory, Argonne conducts basic and applied scientific research in virtually every scientific discipline. Argonne researchers work closely with researchers from hundreds of companies, universities, and federal, state and municipal agencies to help them solve their specific problems, advance America’s scientific leadership and prepare the nation for a better future. With employees from more than 60 nations, Argonne is managed by UChicago Argonne, LLC for the US Department of Energy‘s Office of Science.

Read also:
Argonne National Lab Selects DDN’s Storage
35PB for 10 petaflop/s supercomputer