Deduplication - impacts of data the is already compressed, encrypted or large numbers of tiny files.
Is this still a significant problem? Had a quick look at the doco on deduplication and there is no mention of if. In the past vendors including Veritas/Symantec warned that data that is already compressed and/or encrypted would result in poor deduplication performance, the same with large numbers of tiny files files. Compressed data includes stuff likeaudio files (mp3, wma, ogg etc), video files (avi, mkv, mp4 etc), image files (png, bmp, jpeg etc), PDFs, zipped files. A while back I tested the rumour that of the gzip & pigz using the --rsyncable option is deupe friendly friendly. I found it had significant improvment in dedupe performance on commvault, and it there was only a very small size difference between the compressed files with/without the --rsyncable option. The amount data can be deduped can have a big impact on what can be logically stored.... example - click to see full view
