Globus Reached 1EB Transfer Barrier

Globus reached a milestone by breaking the exabyte barrier.

Click to enlarge

While it took over 2,000 days for the service to transfer the first 200PB of data, the last 200PB were moved in 247 days. This rapidly accelerating growth is reflected by the more than 150,000 registered users who have now transferred over 120 billion files using Globus.

IDC predicts that 59ZB will be created, captured, copied and consumed in 2020. Over the past decade many new technological developments have led to this data explosion. AI, IoT, the proliferation of HPC systems with faster processors and better networking, and new scientific instruments such as genome sequencers are all contributors. Just a few years ago a genome sequencer generated 15GB of data. Now, scientists leveraging next gen sequencers can generate over 6TB of raw data per run — and even more if you factor in the necessary downstream analysis.

“It is such an exciting time for the research community. The massive data volumes being collected, analyzed, and shared today will enable us to address and solve some of the world’s most challenging problems, from discovering new vaccines to tackling climate change and uncovering some of the mysteries of the Universe“, states Ian Foster, co-founder.

So exactly how big is 1EB? It is a billion billion bytes. That is 10¹⁷, or a one with 18 zeros. It would take a person on a continuous video call 237,823 years to transfer 1EB, or a Netflix subscriber 3.5 billion years of watching movies at their current rate to surpass the 1EB mark.

This growth in data creates opportunities in modern scientific research. But progress requires that the data be effectively managed. Last year for example, a team led by Argonne National Laboratory ran 3 of the largest cosmological simulations known to date, and generated a total of 8PB of data. And at the Advanced Photon Source, ultra-bright, high-energy x-ray beams routinely generate more than 30TB per week as scientists collect data in unprecedented detail and in short time frames. Securely and reliably moving and processing tens of terabytes on a daily basis is the ‘new normal’ for many research computing facilities in this exascale era.

“Our mission at Globus is to make the management of research data as frictionless as possible so researchers can get on with their important work“, said Rachana Ananthakrishnan, executive director. “We are gratified to see the service becoming a critical part of cyberinfrastructure at thousands of leading research institutions around the world.“

The solution can be used by non-profit research institutions to move data at no cost; more advanced features are available via a paid subscription. All that’s required is a standard Internet connection to initiate data sharing or transfers of any size, from anywhere to anywhere, using any web browser. It provides instant access to tens of thousands of storage endpoints, including all XSEDE systems as well as systems at most national laboratories and hundreds of leading research universities.

Many commercial organizations also use Globus to address their data management needs and facilitate collaboration with their partners–to inquire about a 90-day free trial.

Read also:
SC19: Availability of Globus Connector For Google Cloud
Solution for accessing and managing data stored in Google Cloud object storage
November 25, 2019 | Press Release
Argonne Team Breaks Record For Globus Data Movement for 2.9PB File Transfer
On Oak Ridge Summit HPC largest ever for Globus; involves 3 of the largest known cosmological simulations.
July 15, 2019 | Press Release