Gary's Dedupe experiences Update with stats

[macpiano]

I have decided to start a blog for my experiences with dedupe.

My setup right now is 2 Dedupe servers in 2 buildings with a 100MB wan. One building has a file server with 2.2 TB of data and several other smaller servers. Both dedupe servers are housing the dedupe folder in both buildings with about 4 TB available. 

I have a tape drive attached to both but what I want is mainly in the second building.

My finding(s) at least to me.

1. Live Update. That was the first hurdle. It was necessary to go to the control panel, live update then manually add the proxy exception, using IE settings did not work. I also changed the cache size to 100 MB just to be sure.

2. Remote Agent has to be manually removed from each server then the new agent installed. I was going from 2010 to R2 so your milage may vary.

3. I am using Backup to dedupe, duplicate that dedupe job to the other dedupe folder in the other building, then backup to tape in the second building. You end up with 4 copies including the original actual files. You get 3 backups of that and they will all have identical times etc. You have to look at its properties to see where it backed up to. SUSH... had a great reply to this at https://www-secure.symantec.com/connect/forums/how-do-i-duplicate-tape-after-dedup-duplicate-another...

4. You CANNOT do #3 unless you setup all the jobs on the MMS that actually has the tape drive attached to it. NOTE: Update is I am still having trouble getting the last part to work.

5. Do not do verify along with the job. Some say don't do at all. At some point I will implement that.

6. Initially I was getting slow speed on client side dedupe but after a full backup to the dedupe folder I get 2500 MB per minute on either media or client side deupe.

7. If you need to remove the dudupe folder to start over as I did just removing it from the devices will not do it. I deinstalled BE itself as I am still in testing mode. Some say you can just remove the deduplication option from BE then you can manually delete the dedupe folder itself.

8. Windows firewall can be problematic. It stopped my dedupe folder from working. There is a workaround for it.

9. Sometimes the dedupe folder will be in red on the devices so if I just click on a server in the list that is supposed to have it and go to properties and close it it will then be active.

10. WAN consumption. I have a 100MB WAN. On a duplicate backup job from one dedupe folder to the other dedupe folder in the othe building it will consume 97% of the WAN connection-the first time to get the data there. On subsequent ones it does just the changes. I will check this and correct if I am wrong. This is also true going from backed up server in Building A directly to tape in the other building all the time (no dedupe involved). Backing up a server (not the dedeup one) in Building A to dedupe folder in Building B it will consume 70% of the WAN. Note that our traffic is about 10-20MB per building during the day the dedupe job just scales back and the users done't even notice.

11.Mulitple jobs and NICS. I have run mulitple jobs to the local server. 2 different servers always get 1000 MB per minute by themselves so I upped the dedupe folder to 6 connections. I ran both jobs and they both still got 1000MB at the same time on the one nic. I added a 3rd one to the file server and it used a 2nd nic and got its usual 2500MB per minute. I checked the let the media server manage the nics and it did.

12. If I try to run a duplicate job from one dedupe folder to the other dedupe folder that duplicate job will appear to just hang there because of the next item in 13..

13. A duplicate job will not give you a running total of the byte count, just the total byes that it is going to do. The job will start out very high in MBs per minute but keeps going down as the job completes.

14. My first full duplicate of 2.2 TB from one server's dedupe folder to another one's dedupe folder in another building (100MB WAN connection) took 40 hours. The 2nd one this week took 3 hours and 14 minutes, only jammed the router for part of the time, and did it at almost 17,000 MB per minute. The dedupe ratio was 50:1 Fantastic and finally worked as I had hoped.