Client Side Dedupe 5-10x slower then Media-Server based?

[Speeder]

Setup-

Client - W2k8R2 file server with lots of small files (about 7 million files, 3tb data). VM, on a san with 10k FC disks. 

BackupExec 2010 R3 with all hotfixes. Latest RAWS installed on Client.

Netbackup 5020 devices configured as OST, local to backupexec server.

Wan - 50mb with Riverbeds.

When media server backup is selected (Full - archive, or full -modified time, open file enabled/disabled) backup maxes out WAN (going at about 500mb/min)

When client-side dedupe is enabled, backup runs at 50-150mb/min. Client is not high in CPU or bandwidth. WAN is barely being used.

any ideas?

[pkh]

The max'ing out of the WAN can be easily explained.  When you do server-side dedup, all your 3TB is passed to the media server for dedup.  When you do client-side dedup, only the changed blocks are sent to the media server.

Does your client have enough memory for 3TB of data?  Also, there are some other discussions that suggest that VM's does not give as good a performance as physical machines.

[Simon_B_]

I guess that the problem is the latency which comes with the WAN connection. When you perform media side dedup it can simply create one connection to push all the files to the media server.

When using clientside dedup for every single block of data the Agent has to ask the media server if a chunk with this hash is already present, if not, transfer it. I assume that these checks get a lot slower due to typical lan latency (20ms+) which then leads to the decreased overall speed.

[Riyaj_S]

Is the verify option for Backup job enabled ?

http://www.symantec.com/docs/TECH141528

Hope this helps.

[Speeder]

thanks for the responses.

Latency is <15ms. Riverbed also optimizes against high latency issues.

Separate verify policy is not configured.

Memory was not being particularly used on the client, it also has 12gb ram.

Maybe riverbed is the culprit....

[Simon_B_]

I don't know about these riverbed appliances but maybe they perform some form of caching etc which slows down the dedup process. If possible I'd suggest to test the backup with these turned off.

[teiva-boy]

You seem to think that WAN accelerators are a fix all to WAN links.  They are not.  They take an often slower link, and optimize it as best as it can through packet re-ordering, QoS, shaping, caching, and more. But the physics still apply, it's a slower link.  There is only so much it can do.

At times, you need to disable any shaping features in an accelerator for particular apps like when doing client-side dedupe or the like.  This is a testing phase you need to embark on.  There is no clear-cut answer.  

Other times, you need to disable encryption from the client for the accelerator to do it's thing.  Not an issue with BackupExec as that is disabled by default.  

To get the most out of the WAN, you need to send more concurrent streams/threads over the wire to make use of the bandwidth.  No single stream is enough to do it unless it's UDP.  TCP just can't alone.

[Speeder]

I don't think Riverbeds fix all the issue. Also, we're talking about a 50mb <15ms wan link here. 5 Years ago, this would be half lan speed. Also, the fact that the backup saturates the pipe with media side dedupe, but client side crawls (which should isntead be faster).

I'm just trying to figure out the issue here.

[Simon_B_]

I agree with teiva-boy that you at least temporarily eliminate the riverbeds from the equation just to be sure.

Also important to point out: the 15ms you are talking about are raw ICMP latency time. This does not have anything to do with the time which is needed to establish a TCP session for example, as this will require multiple packets between client and server. So even if you've got 15ms latency this will not closely be comparable to the link quality of a LAN with <1ms. TCP can certainly scale up to 50MBit/s over WAN (look at HTTP downloads), but not if there are many small connections involved which have to be opened/closed. Unfortunately idk how client side dedup uses the TCP sessions (eg if only one is established or one session for each "do you have this chunk of data"-check).

If the accelerator performes compression that wont work for client side dedup as most of the redundancy within the data should already be eliminated. Also with R3 by default traffic between client and server is encrypted, so compression will not be possible.