Fujitsu Laboratories Developed High-speed Access Technology for Magnetic Tape

Fujitsu Laboratories Ltd. announced the development of an high-speed access technology for magnetic tape storage, which is attracting renewed interest as a low-cost, large-capacity storage solution alternative to HDDs.

Traditionally, magnetic tape storage has been used mainly for backup purposes, but because of its high capacity and low cost, as well as the acceleration of transfer speeds and the spread of the LTFS in recent years, it is expected to find increasing use for archival purposes.

Fujitsu Ltd. has expanded the functionality of LTFS by innovating a file system that virtually integrates multiple tape cartridges. By improving random read performance from tape through data management and access order control in accordance with tape characteristics, this technology achieves speeds 4.1 times faster than conventional methods.

This technology will accelerate the adoption of magnetic tape storage technology as an archival medium in anticipation of the growth of data in the future. The firm will help companies to achieve their digital transformation goals through storage solutions that offer superior performance at a reasonable cost. The company’s laboratories is currently conducting a verification trial assuming the application of this technology to its operations, with plans to commercialize the technology by the end of FY22.

Details of this technology was presented at the 12th Forum on Data Engineering and Information Management (DEIM 2020).

Newly developed technology
While magnetic tape storage is suited to reading from and writing to sequential areas of tape, its ability to randomly access discontinuous locations remains limited. The technology’s relative inability to deliver random access reading presents a roadblock to broadening its use into high volume data archiving applications.

In general, to manage a large amount of data with magnetic tape storage, data is held under a different directory for each tape cartridge on LTFS, which uses multiple tape cartridges and can access data on a file-by-file basis in the same way as data on HDDs, USB memory, etc.

The company developed a new file system on LTFS that virtually integrates multiple tape cartridges. This virtually integrated file system consolidates multiple tape cartridges into one, allowing users to access the data they need without thinking about each individual tape cartridge. In addition, the following newly developed technologies have been applied to this file system to achieve high-speed magnetic tape access performance.

Fig.1 Configuration image of the developed system

Access order control with physical location
On a magnetic tape, data is divided along its length in units called wraps, and each wrap wraps around and is written in a write-once fashion. Therefore, the distance between the logical address and the physical address is very different. The virtual consolidated file system accepts multiple random read requests and processes them starting with the closest physical location on the tape, not the logical address.

When writing to magnetic tape, write and error checking are performed in parallel, and when an error occurs, only the part of the error that occurred after the end of writing is automatically rewritten. Therefore, it is difficult to predict the physical location where a rewrite has finished from the change in file size. The physical location of each file is estimated by periodically measuring the head position after writing the file.

Also, when accessing the magnetic tape, it takes time to align the head to the start position. Therefore, two read requests that are close to each other on the same wrap will not read two files at once, but will read all the files in between and discard unnecessary files.

Fig.2 Rewrite image on write error

Fig.3 Image of access order control with physical location

Multiple file aggregation function
LTFS maintains an index of each file on magnetic tape, and the impact increases exponentially as the number of files increases. When using tapes for archival purposes, users write and access files of various sizes, but writing large numbers of small files can degrade read performance.

Therefore, the company has developed a mechanism to keep small files smaller than a specified file size together as large files on LTFS so that users can access them without worrying about the location of the files. In addition, by managing the metadata of user files in the virtual integrated file system, it is possible to quickly display a list in a way other than data reading, add extended attributes, or delete files without accessing the magnetic tape.

Fig.4 Multiple file aggregation image

Outcomes
The firm has constructed a hierarchical storage system for HDDs and magnetic tapes by using Ceph ⁽¹⁾, an open source distributed storage software, and evaluated the access performance of the system. As a result, the time required to read 100 files randomly from a total of 50,000 individual 100MB files stored on magnetic tape was 5,400s with the conventional method. By using this new technology, it was possible to confirm a read in 1,300s, which is 4.1 times faster than the conventional method. In addition, while the conventional method required 2.5s to move 256 individual 1MB files on the HDD onto magnetic tape, this technology enabled us to confirm data movement in 1.3s, which is 1.9 times faster than conventional method.

This technology enables high-speed tape access performance, such as random reads and writes of various sizes occurring in archive applications, and is expected to rovide a cost-effective data archiving infrastructure for long-term archiving of large volumes of data.

^[1] Ceph: Open source distributed storage software developed at the University of California and managed by the Ceph community since 2004.