Understanding Progressive De-Duplication

By WD Guest Blog

Storage managers dealing with network backup solutions constantly seek to minimize the time required for backups and restores, ever mindful of costs associated with maintaining multiple restore points (i.e. materialized snapshots), and under pressure to do more with less available budget and staffing resources.

As tape backup has given way to disk backup, and file system source data is edged out by volume-image backups, optimum data protection solutions have evolved significantly in the last ten years. The growth impact of virtualized environments and the increasing prevalence of cloud storage also present new challenges.

Data deduplication

Local compression strategies target the compression of individual files and use familiar algorithms including DEFLATE (i.e. zip), JPEG, or MPEG. Distinct from local compression, data deduplication is a specialized technique, often referred to as ‘global compression’ that looks for similar files across various data blocks and eliminates creation of duplicate copies and repeating data.

Network data deduplication reduces the number of bytes transferred between endpoints making it especially relevant to backup for two reasons: multiple computers within an enterprise often contain similar documents, operating system files and databases; and successive backups of the same computer often differ only slightly from previous versions. By having to store content only once, multiple files can be dramatically compressed to reduce storage space requirements and the amount of bandwidth required can also be reduced.

Data deduplication has been used to reduce data volumes (both for storage and transmission) since the 1990s. Traditional deduplication methods involve fixed- and variable-block technologies. Fixed-block dedupe is beneficial when applied to large files expanded by data appended to existing files. Variable-block dedupe requires more processing but offers flexibility since it can identify file changes that are modified, appended, and inserted within files.

Progressive deduplication

Superior in both speed and compression is the alternative technology of Progressive Deduplication from WD Arkeia which combines the best features of fixed- and variable-block deduplication. Like variable-block dedupe, Progressive Deduplication delivers high compression because it can tolerate ‘data inserts’ into files, as well as ‘data appends’ and ‘data modifies;’ plus, block boundaries can be present anywhere in a file. Different from variable-block, with Progressive Deduplication known content is processed quickly like fixed-block dedupe.

Benefits to MSPs

Managed service providers (MPSs) can benefit from utilizing deduplication to better manage budgets and reduce costs. Using Progressive Deduplication backup sets can be replicated in deduplicated form to both private and public cloud storage, offering a cost-effective alternative to tape for off-site storage. MSPs and value-added resellers (VARs) can take advantage of WD Arkeia’s Multi-tenant Replication to deliver backup-to-cloud Services.

MSPs can harness the power of superior deduplication using Progressive Deduplication technology to offer increased backup speed and reduced costs of storage and network infrastructure. As with most technologies, there are some tradeoffs between time (less computation) and space (better compression), but Progressive Deduplication successfully pushes out the curve by applying new and better algorithms.

Some storage experts have decried the tangible benefits of deduplication and even compression as being only temporary remedies but in today’s economically-challenged environment, any technology that saves companies time and storage capacity is a welcome practice, even when applied during short-term interims while other methods are evaluated.

Guest blogs such as this one are published monthly , and are part of MSPmentor’s annual platinum sponsorships.