Next-Gen De-Dupe and Compression: Object-Based De-Dupe

This article has been written by John Everett, storage business manager, Dell EMEA.

The Next-Generation of Deduplication and Compression

With data growing at an unprecedented rate, organisations of all sizes
are looking to maximise the efficiency of how they store and manage data
throughout its entire lifecycle. This ongoing challenge has led to the
proliferation of technologies such as thin provisioning, automated
tiering and scale-out storage, which can deliver both Capex and Opex
savings through smart resource management for better utilisation rates,
increased energy efficiency and simplified administration.

Now, advances in deduplication and compression technologies are allowing
organisations to push utilisation rates even higher through what Dell
calls ‘content-aware storage optimisation’ – also known as object-based
deduplication – shrinking meaningful amounts of data for significant
cost and management savings.

At a basic level, deduplication is the process of eliminating duplicate
copies of data and replacing them with pointers to a single copy. Its
function helps organisations reach two primary goals: to reduce the
amount of storage capacity needed to store a myriad of data, and to
decrease the amount of data in flight during backup or replication
processes. As it stands, the dominant use case for deduplication is
backup storage, because of the amount of static data that organisations
have to backup. Nevertheless, deduplication technology has developed
into other data centre storage platforms such as NAS.

Some deduplication processes examine files in their entirety to
determine whether they are duplicates, which is referred to as
file-level deduplication, or ‘Single Instance Storage,’ while others
break the data into blocks and try to find duplicates among the blocks,
which is referred to as block-level deduplication. Block-level
deduplication typically provides more granularity and a greater
reduction in the amount of utilised storage capacity compared with
file-level deduplication. This is particularly appealing from a bac-up
perspective. Both types of deduplication are commonly used and offered
today; however, there is a growing appreciation that these approaches
may not be sufficient to handle the growth of big data in verticals such
as oil and gas, life sciences, media and entertainment.

A more intelligent form of deduplication has emerged in the form of
object-based deduplication. Now, organisations can take advantage of
next-generation technology that is tailored to their particular
vertical. This can be achieved with a solution that bridges the gap
between applications and native storage platforms to optimise the way
data is stored. This optimisation technology identifies how a given
file is structured, breaking it down to component sub files and then
selecting which is most effective from a library of more than 100
different compression algorithms for the targeted file. Even if the
file has never before been identified, and there is no content-specific
compressor, the technology will infer information about the structure
and nature of the contents to select the most effective data-reduction
algorithm. By understanding the layout of specific application files –
like an email programme or a digital image – IT can make intelligent
decisions about how to de-dupe and compress that data for optimal
storage.

The central components of Dell’s data-processing system include two
types of content-aware algorithms and a neural net framework for testing
and selecting different compressors for best run-time efficiency. The
two types of content-aware algorithms are de-layering algorithms, which
dissect files to identify the contiguous sub-objects, and data-shrinking
algorithms, which include deduplication and compression. These custom
compressors are more capable of shrinking meaningful amounts of data
that plague specific verticals.

To further reap the benefits of deduplication, this technology should be
able to be seamlessly applied across the entire IT infrastructure. To
this end, Dell is rolling out storage optimisation technology across a
variety of solutions for primary storage, archive, and backup.
Deduplication and compression will be integrated in the Dell Scalable
File System and Dell Object storage; once data is deduplicated, it can
move in a deduplicated state from one storage system to another. For
example, data that is deduplicated on Dell primary storage solutions can
be backed up without rehydration to Dell backup storage, which can then
be replicated in a deduped state over a LAN/WAN to a Dell backup
storage replica. It is this end-to-end optimisation of data from the
server to storage to the cloud that brings the most value to an end user
organisation in a data heavy world.

Even though dededuplication and compression technology has been around
for a few years, it is here to stay and is evolving rapidly. To be truly
effective in today’s business world as well as tomorrow’s,
organisations should look to a solution that adheres to three main
tenants:

• to be transparent to the end user and applications, meaning that there should not be any performance delays upon retrieval;
• to be customised to specific verticals with more and better algorithms and logic; and
• to be utilised end-to-end across the entire workflow to ensure optimisation of the overall IT environment.