iXsystems and Klara Donation of Fast Dedup Feature

iXsystems, Inc. reasserted their joint commitment to open source with the contribution of the Fast Dedup feature to the OpenZFS and TrueNAS communities.

In a move sponsored and supported by engineers from the company and the development efforts of Klara, Inc., the contribution of Fast Dedup enhances sustained de-dupe performance for ZFS and TrueNAS storage systems, marking a new milestone in storage efficiency and performance.

De-dupe performance has been limited in ZFS and OpenZFS since the inception of the file system due to its metadata handling. Fast Dedup ameliorates this with the introduction of a re-engineered metadata structure dynamically sized to operate in RAM or on dedicated flash devices for expedited processing. These advancements collectively enable up to 20x faster performance and 5x more usable capacity for certain workloads while improving reliability for systems with de-dupe enabled. The technology outperforms legacy de-dupe by employing algorithms and strategies that scan, identify, and eliminate duplicates, while keeping metadata in RAM and SSDs.

“The OpenZFS Community is very excited to receive this major contribution of Fast Dedup software,” said Matt Ahrens, co-creator of ZFS and OpenZFS Project. “Fast Dedup has been a major user desire for many years and we are pleased to see that Klara and iXsystems have developed the feature in-line with our general requirements. We look forward to testing and integrating the new software.”

Key Fast Dedup advancements include:

• Metadata size is now dynamically sized to fit on either RAM or dedicated flash devices to avoid hitting the performance penalty wall.
  
• Metadata structure completely re-engineered to enable efficient updates using a log append process, greatly improving performance for large updates such as deletions.
  
• De-dupe table now favors de-dupable data and prune blocks that show no de-dupe potential.
  
• Metadata improvements combined with properly configured storage, including dedicated metadata flash devices designed to improve sustained de-dupe performance by an order of magnitude or greater for larger systems.

“Klara is proud of the speed and functional quality of the Fast Dedup development,” said Allan Jude, FreeBSD engineering manager, Klara “We look forward to completing the integration and testing work needed for this to be a standard OpenZFS capability, with no extra cost.”

“Fast Dedup has been a longstanding desired feature of ZFS and TrueNAS, and can deliver 5X the usable capacity and 20X the performance,” said Kris Moore, SVP, iXsystems engineering. “These attributes will significantly improve the economics of OpenZFS storage relative to cloud storage and proprietary storage, and our team could not be more ecstatic to see this come to fruition.”

The donation of Fast Dedup technology is a testament to the shared vision and collaboration between the 2 partners, underscoring their dedication to the advancement of open-source storage solutions. Expected in 2Q24 for TrueNAS SCALE, Fast Dedup will be available as test software, not to be used for production. This will allow validation of functionality and performance prior to a standard product release later in 2024.

Resources:
Fast Dedup is a Valentines Gift to the OpenZFS and TrueNAS Communities
Introducing Fast Dedup
Video: Introducing Fast Dedup, by Allan Jude, CTO, Klara
Blog: iXsystems and Klara Celebrate Valentine’s Day with a Heartfelt Donation of Fast Dedup to OpenZFS and TrueNAS