R&D: Facilitate External Sorting for Large-Scale Storage on SMR HDDs

Springer Nature Switzerland AG has published, in Advances in Information and Communication Proceedings, an article written by Yu-Pei Liang, Min-Hong Shen, Department of Computer Science, National Tsing Hua University, Hsinchu, Taiwan, Yi-Han Lien, Department of Electronic Engineering, National Taipei University of Technology,Taipei, Taiwan, and Wei-Kuan Shih, Department of Computer Science, National Tsing Hua University, Hsinchu, Taiwan .

Abstract: “In the era of big data and cloud computing, both external data process techniques and new storage mediums are proposed to process and accommodate the sheer amount of information with data-intensive applications. For instance, external sorting algorithms perform sorting operations directly on the storage devices to lower the data transfer latency and increase system performance. On the other hand, Shingled Magnetic Recording (SMR) is proposed to increase the areal density by overlapping tracks. However, the overlapping technique also introduces the random-write restriction because writing a track also destroys the valid data on overlapped tracks. This constraint could induce signification write amplification issue when performing external sorting on SMR drives. To mitigate the write amplification issue, this paper proposes a sort-friendly SMR drive design to lower the write amount of external sorting algorithms on SMR drives. The experimental results show that the proposed design could lower the external sorting latency by 61.99% when compared with the external merge sort algorithm.“