02-21-2013 02:54 AM
Hopefully someone can help me - I'm running out of ideas:
Problem:
We're using DeDuplication for the firststage-backup, afterwards the data is copied to external media - this time USB-disks (USB3 with USB3-controller)
The first stage is running fine, but the copy to usb is awfully slow - and gets slower every day.
It started with about 1400 MB/min, now it's down to 150-250 Mb/min.
To have a comparison to a "normal" duplicate to usb-job, I've created a b2d-device on the same storage as the dedup-foleder, made a backup and duplicated that to usb-disk - everthing identical, except the b2d-device used as a source instead of the dedup-device.
The result is about 6.000-7.500 MB/min - so the diskchannel and the usb-disk-channel is fine.
I don't have a problem with the slower speed from the dedup-device (due to the rehydrating) - as long it stays above 1000 MB/min, but it mustn't drop that far.
Environment:
BE2012 with actual updates. The hardware is a HP DL180G6, CPU: E5606, 12 GB RAM. 5 SAS 1 TB Harddisks attached to an HP SmartArray P410 with 256 MB Cache and battery. CPU usage below 50%, RAM-usage ~ 50%.
Any idea ?
Thx
Olaf
Solved! Go to Solution.
06-13-2013 04:44 AM
Hi folks!
I just wanted to update and close this case.
To make a very long story short:
According to the 1st, 2nd and 3rd level supportengineers from Symantec we do not have a problem - the dedup works like designed and they can NOT do anything to speed up the process of dplicating FROM the dedup to any external non-dedup-device. :-((
Due to this sad result and the amount of time we've invested in this case we've decided to discard dedup as a primary backup target at our and all customer site's.
From my point of view the product can't keep the expectations from the customers and promisses made by symantec.
CU
Olaf
02-21-2013 03:16 AM
Unless you're duplicating to another dedupe folder, you should actually be rehydrating your data...this is 1 reason for slower speeds. Add in a USB drive and you would have the right environment for slow duplication rates.
The server hardware itself looks to be fine, with no changes to be made!
Thanks!
02-21-2013 03:47 AM
The absolut speed of the (beginning) dedup rate isn't the problem.
THe quesion is - why is it getting slower every day?
Without any visible bottleneck regarding disk, cpu or ram ?
Imho that's not the way it should be.
Olaf
02-21-2013 03:50 AM
Have you used Perfmon.exe to see whether or not there is a bottleneck on that USB drive?
Thanks!
02-21-2013 04:24 AM
I'm pretty sure, that the usb-disk is not the botteneck due to 2 facts:
- Copy from B2D to USB works with ~ 7.000 MB/min
- Copy from DeDup to USB worked in the past with more than 1.500 MB/min.
I haven't used perfmon but the ressourcemonitor in the w2k8 task manager and couldn't see any problems.
Regards
Olaf
02-21-2013 06:22 AM
Is that a GRT / non GRT enabled backup you are duplicating?
02-21-2013 06:25 AM
...please edit your previous post if you post within minutes of the last one!
Thanks!
02-21-2013 06:26 AM
As a test what happens when you do a duplicate from your dedupe back to the dedupe folder?
Is that a GRT / non GRT enabled backup you are duplicating?
02-21-2013 06:27 AM
You seem to read my mind!
02-21-2013 10:34 AM
Its a duplicate of a backup from a w2k8-server - raws based.
It includes Exchange-Data/mailboxes - so the answer is yes and no.
Does that make any difference ?
02-21-2013 01:17 PM
When a backupset is duplicated from a dedupe device to a non dedupe device, data is rehydrated i.e reconstructed. The metadata(catalogs) also has to be rebuilt. GRT backups are catalog intensive than a non GRT backup. To isolate the issue, I would suggest you to create a backup without GRT and test a duplicate backup. I assume there are no other operations running on the dedupe folder during the duplicate backup.
02-22-2013 01:54 AM
Hi all!
a few answers / comments to the suggestions made by you:
1. Yesterday (out of the blue sky) the dup-job ran with unbelievable 1700 MB/min - completly unexpected, because it was the same job / usb-disk / data that was duplicated.
The one and only difference was:
I started the duplicate-job manually with "run now" from the serverconsole itself - not via rdp.
So imho - there is no connection betwenn grt/non-grt data and the dup-speed.
2. Today - same dup job starts automatically at 9 am - and it's running with only 223 MB/min :(
Changes since then:
I've installed last night the patch 201596 and restarted the server and today it's a different (identical type) usb-disc.
I'm gonna check if there is a difference when:
- The job is started manualy directly on the console without rdp (it shouldn't...)
- The disc from yesterday is used
02-22-2013 03:42 AM
New facts:
1. Starting the job manually from the serverconsole directly didn't change a thing - still veeeeeeeeery slow.
2. Using a brandnew empty freshly formatted usb-disc afterwords - even slower - 180 MB/min
3. Restarting the be-services before restarting the same job - juhu - speed is up to 1126 MB/min - still accelerating.
Some conclusions:
1. there seem's to be no connection between the usb-disc itself and the speed of the backup
2. the "BackupExec-setup" itself (hardware, software, configuration) seems to be ok, as the system is able to perform the duplicate-job with high-speed
Do you agree with ?
But why change a restart of the be-services speed up the duplicate job ??
Waiting for some ideas..
Olaf
02-22-2013 05:41 AM
...restarting the BE services is obviously releasing "something".
My suggestions:
1. Check that no AV is actively scanning the dedupe folder, and if so, put in an exclusion for that.
2. Check to see if there is a Known Issue in that section around this:
https://www-secure.symantec.com/connect/backup-and-recovery/issues
Thanks!
02-22-2013 08:57 AM
Craig,
I've got the same impression - it's pretty obvious.
The AV check exclusion is already done - and not modified between the diferrent tests / results, so that couldn't be the reason for different behaviour.
I couldn't find a similar problem / error in the list.
Regards
Olaf
02-27-2013 12:39 AM
I've opened a case with symantec.
I'll keep this post updated.
Olaf
02-27-2013 12:46 AM
Hi Olaf,
In connection with my post on the link below, if you updated BE 2012 which caused the issues you now face, see if you can uninstall those patches and make sure you can operate as-per-normal. Once done, install the patches one-by-one to test...
https://www-secure.symantec.com/connect/forums/problems-deduplication-2010-r3?page=0#comment-8417931
Thanks!
03-15-2013 12:58 PM
Statusupdate:
Case is opened with Symantec TechSupp - they are working on it - already escalated to the next level.
04-09-2013 04:49 AM
Just a little update:
The case is opened for several weeks now, but Symantec is totally ignoring the facts and sticks to the statement "rehydrating makes duplicate from dedup-devices slow".
But the case is not closed yet.
06-13-2013 04:44 AM
Hi folks!
I just wanted to update and close this case.
To make a very long story short:
According to the 1st, 2nd and 3rd level supportengineers from Symantec we do not have a problem - the dedup works like designed and they can NOT do anything to speed up the process of dplicating FROM the dedup to any external non-dedup-device. :-((
Due to this sad result and the amount of time we've invested in this case we've decided to discard dedup as a primary backup target at our and all customer site's.
From my point of view the product can't keep the expectations from the customers and promisses made by symantec.
CU
Olaf