ESnet Software Help SLAC Researchers in Record-Setting Transfer of 1PB of Data in 29 Hours

From Esnet (Energy Sciences Network)

Using a 5,000-mile network loop operated by ESnet, researchers at the Department of Energy’s SLAC National Accelerator Laboratory (SLAC) and Zettar Inc. recently transferred 1PB in 29 hours, with encryption and checksumming, beating last year’s record by five hours, an almost 15% improvement.

This screen shot from MyESnet shows the southern links
of the network dark red, indicating data transfers exceeding 50Gb/s
during the 5,000-mile transfer conducted by SLAC and Zettar.
Click to enlarge

The project is aimed at achieving the high data transfer rates needed to accommodate the amount of data to be generated by the Linac Coherent Light Source II (LCLS II), which is expected to come online in 2020.

The LCLS is the world’s first hard X-ray free-electron laser (XFEL) and its strobe-like pulses are just a few millionths of a billionth of a second long, and a billion times brighter than previous X-ray sources. LCLS II will provide a major jump in capability – moving from 120 pulses per second to one million pulses per second.

Scientists use LCLS to take crisp pictures of atomic motions, watch chemical reactions unfold, probe the properties of materials and explore fundamental processes in living things.

The increased capability is expected to generate data transfers of multiple terabits per second – as the experimental results are sent from SLAC to Department of Energy‘s (DOE) supercomputing facilities for analysis. As the DOE’s dedicated network user facility for scientific research, ESnet carries data between universities and DOE’s national laboratories and national user facilities along a multi-100Gb/s backbone network.

According to Les Cottrell, Stanford emeritus, the current demonstration of high speed data transport is part of a program to develop and evaluate tools to address the upcoming decade’s critical data transfer requirements for the next-generation XFELs at SLAC.

Zettar, a National Science Foundation-funded software firm in Palo Alto, CA, develops hyperscale data distribution software solution capable of multi-100+Gb/s transfer rates and collaborates with ESnet and DOE national labs. For the trial, the team used ESnet’s On-Demand Secure Circuits and Advance Reservation System (OSCARS) to reserve the bandwidth for the run.

“The innovative and versatile nature of ESnet’s OSCARS enabled us to design a unique 5000-mile 100Gb/s loop, which our ESnet colleagues quickly helped establish in 2015,” said Chin Fang, founder and CEO, Zettar. “The design of ESnet enables the use of a separate dedicated channel for the testing without impact to or interference from production traffic. We were thus able to have the data storage clusters and the data transfer nodes co-located in the same rack although they were at both ends of the loop. Physically they were separated by inches, but digitally they are separated by 5000 miles. Further the clusters themselves have a self-imposed cap of 80Gb/s.“

This graphs shows that the petabyte transfer trial accounted
for one-third of the total network traffic on ESnet
during the 29-hour run.
Click to enlarge

Fang said the five-hour improvement in transferring the petabyte of data was accomplished thanks to improvements in both software and hardware. This year, Zettar significantly improved the parallelism of its software to speed up data transfers for both encrypted and non-encrypted data. Intel Corp. also provided the team with faster processors and solid-state memory products for moving data in and out of main storage systems faster. In this case, the project used simulated datasets with file size distributions provided by the LCLS Data Management Group.

“The flexible traffic engineering design of ESnet’s network ensures that TCP-based transfer protocols can be used effectively for high-throughput data movement,” said Inder Monga, director, ESnet . “ESnet’s guiding strategy is to anticipate and stay ahead of the data transfer requirements of our scientific users. Ongoing trials like the ones we are supporting for SLAC are important for us in that we see the directions that the research teams are heading and we can enhance our support and infrastructure to be ready when facilities like LCLS II come online.“

ESnet is a DOE Office of Science User Facility. SLAC and Lawrence Berkeley national laboratories are supported by the Office of Science of the U.S. Department of Energy. The Office of Science is the single largest supporter of basic research in the physical sciences in the United States, and is working to address some of the most pressing challenges of our time.

Read also:
Using AIC Servers, Zettar Transferred 1PB in 29 Hours With Encryption
With data integrity checksum unconditionally enabled, over distance of 5,000 miles
2018.10.12 | Press Release