R&D: Improving Access to HDFS Using NVMeoF

IEEE has published, in 2019 IEEE International Conference on Cluster Computing (CLUSTER) proceedings, an article written by Daegyu Han, and Beomseok Nam, Sungkyunkwan University.

Abstract: “In this extended abstract, we discuss how to improve access to HDFS by employing NVMe over Fabrics (NVMeoF), which has emerged as a new communication protocol between a host and a storage system. To address the legacy shortcomings of HDFS, we explore the opportunity of enabling all-to-all connections between NVMe SSDs and DataNodes rather than dedicating each NVMe to a single DataNode. Our experimental study shows that we can achieve up to 2.55 times higher I/O throughput than legacy HDFS by making HDFS leverage NVMeoF for remote data access.“