Opendedup

Global Deduplication

  • Increase font size
  • Default font size
  • Decrease font size

What is Deduplication and Why does it Matter?

E-mail Print PDF

What Deduplication is

According to wikipedia, "Data deduplication is a specific form of compression where redundant data is eliminated, typically to improve storage utilization. In the deduplication process, duplicate data is deleted, leaving only one copy of the data to be stored. However, indexing of all data is still retained should that data ever be required. Deduplication is able to reduce the required storage capacity since only the unique data is stored. For example, a typical email system might contain 100 instances of the same one megabyte (MB) file attachment. If the email platform is backed up or archived, all 100 instances are saved, requiring 100 MB storage space. With data deduplication, only one instance of the attachment is actually stored; each subsequent instance is just referenced back to the one saved copy. In this example, a 100 MB storage demand could be reduced to only 1 MB. Different applications have different levels of data redundancy. Backup applications generally benefit the most from de-duplication due to the nature of repeated full backups of an existing file system."[1]

SDFS leverages data deduplication for primary storage. It acts as a normal file system that can be used for typical IO operations similiar to EXT3, NTFS ... etc. The difference is SDFS hashes blocks of data as they are written to the file system and only writes those that are unique to disk. Blocks that are not unique just refernce the data that is already on disk.

Why it does matter

Using deduplication has two big advantages over a normal file system.

  • Reduced Storage Allocation - Deduplication can reduce storage needs by up to 90%-95% for files such VMDKs and backups. Basically situations where you are storing a lot of redundant data can see huge benefits.
  • Efficient Volume Replication - Since only unique data is written disk, only those blocks need to be replicated. This can reduce traffic for replicating data by 90%-95% depending on the application.

Using SDFS adds a couple additional advantages based on its achitecture.

  • Scalability : SDSF can store huge amounts of data (over a Petabyte) and can deduplicated at block sizes as small as 4k.
  • IO Performance : SDFS can be setup in a Redundant Array of Inexpensive Nodes (RAIN) configuration. In a RAIN configuration SDFS can stream reads and writes, in parallel, across multiple nodes allowing for huge IO performance increases.

Great Applications for SDFS and Deduplication

  • Backups
  • Virtual Machines
  • Network shares for unstructured data such as office documents and PSTs
  • Any application with a large amount of deduplicated data

Applications that are not a good fit for Deplication

  • Anything that has totally unique data
  • Pictures
  • Music Files
  • Movies/Videos
  • Encrypted Data

References

  1. Wikipedia - Data Deduplication


Last Updated on Friday, 07 May 2010 22:41  

SDFS Info

Latest News

SDFS 2.0 RC2 was released with fixes for Replication, Redhat, and Variable Block deduplication.
 
SDFS 2.0 RC1 Released with fixed and some minor enhancements. Check out the changes or download.