Open Source Commitment for Seagate

Seagate Technology plc announced that as part of its continued commitment to open source communities, the contribution of an Apache Hadoop on Lustre Connector. 

It improves workflow efficiency by eliminating the need to copy data to the Hadoop Distributed File System (HDFS) prior to running Apache Hadoop jobs. The Hadoop on Lustre Connector also provides an alternative to Hadoop’s reliance on the HDFS file system and enables Hadoop ecosystem tools such as Mahout, Hive and Pig to take advantage of the Lustre file system. 

In addition, Seagate is releasing source code for a patch to Hadoop that allows Map and Reduce processes to share files and enables the use of ‘diskless’ Hadoop compute clusters, allowing Hadoop to function with HPC architectures that use Lustre for storage. 

HPC customers in the life science and energy fields are increasingly using Hadoop and Lustre together as part of their data analysis workflows. The Hadoop on Lustre Connector helps HPC customers streamline their Hadoop workflows and accelerate time to results.

Also announced is an agreement to transfer assets relating to Lustre.org to Open Scalable File Systems, Inc. (OpenSFS) and European Open Filesystem SCE (EOFS). They are stewards of the Lustre distributed file software community and will jointly manage Lustre.org. 

Seagate continues to demonstrate a commitment to Lustre through financial contributions to OpenSFS at the ‘Promoter’ level and as a board member. The company has involvement with OpenSFS and EOFS on all levels and is one of the largest code contributors to the Lustre code tree.

“Seagate believes that direct involvement enabling core capabilities as well as fostering the addition of new application environments is critical to open source community vitality, especially for Lustre which is a foundation for much of the success of HPC among science, government and business community leaders. Our wok with OpenStack Swift, the Open Compute Project, OpenSFS, EOFS and now Hadoop is just the beginning,” said Ken Claffey, VP of ClusterStor, Seagate cloud systems and solutions. “We are committed to driving open source innovation and partnering with open source communities as they develop cutting-edge enabling technologies that are foundational for the entire industry.”

This news follows Seagate’s recent announcement to make its Ethernet Drive interface specification and T-Card development adapter available to the Open Compute Project in January of this year.

Seagate was exhibiting at SC14, November 16-21 in New Orleans, LA, during which it demonstrated the Hadoop on Lustre Connector.