Very slow Exchange GRT backup to B2D

[robnicholson]

There are lots of posts on here with exactly the same problem but many of them are locked because it appears no feedback or solution was ever found. So here's ours on BE 2010 latest version:

• Passive backup of ~40GB Exchange 2010 mailbox without GRT to B2D: a perfectly acceptable 2,237MB/min
• Backup of same with GRT enabled: a rather unacceptable 196MB/min

That's 10 times slower which as Alex said in this idea post, is borderline useless as a practical solution:

https://www-secure.symantec.com/connect/ideas/improving-b2d-performance-grt-enabled-exchange-backup

So is in an inherent problem in BE?

Cheers, Rob.

[robnicholson]

Duplicated the same job but changed media to tape and it runs at >2,000MB/min which makes me suspect that GRT backups to tape are doing it very differently to GRT backups to disk. Anyone know more about this?

Cheers, Rob.

[Sush---]

Is the drop in the Job rate observed at the end of the backup job when it is actually creating catalog for GRT restore? OR is the low job rate observed right from the start ?

Also have you check the Fragmentation level of the disk where you are taking the backups?? De-fragmenting it (if its  highly fragmented) would surely help.

Thanks,

-Sush...

[burtos]

Is your disk backup folder location located on the mail server or a different server, as i have the same issue as well takes about 45 minutes via a tape back up and 12 hrs when doing it to disk.

[robnicholson]

The rate is at the very start but will let the same backup complete now.

But I'd like to throw in some more speed tests which are just perplexing.

B2D device #1 - RAID-0 array using 2 x 2TB SATA-II drives connected via PCI-E HP Smart Array P411 SAS/SATA disk controller

1. Non-GRT backup: 2689MB/min
2. GRT backup: 199MB/min

B2D device #2 - external SATA-II hard drive connected via USB-2

1. Non-GRT backup: 1947MB/min
2. GRT backup: 921MB/min

This just doesn't make any sense. How come a lowly USB-2 external hard drive can out perform a RAID-0 array connected via a mid-range disk controller?

USB-2 SATA-II should max out at around 30MB/s (1800MB/min) so those second speed settings for device #2 are in the right ballpark.

The first device has a much faster interface and should be able to run at the close to the native speed of the SATA-II interface which I think is around 80MB/s (4800MB/min) so the non-GRT should be faster for starters.

The problem here appears to be BE using this P411 based array as a backup device. I'm going to do some speed tests of copying large files to get the raw speed of that array.

Cheers, Rob.

[robnicholson]

The B2D folder is local to the Backup Exec media server - a RAID-0 2 x 2TB SATA-II array connected via HP Smart Array P411.

Cheers, Rob.

[robnicholson]

Hmm, my beady eye is falling upon the P411 disk controller as something is not stacking up. Ignoring BE completely, copy a 20GB test file from the internal SATA-II drive (which can sustain ~75MB/s according to ATTO disk benchmark) to the B2D device only copies at 42MB/s whereas ATTO gives it's raw speed as XYZMB/s.

I've just created another single 1TB SATA-3 array on the controller and ATTO disk benchmark gives that a pathetic 7MB/s write speed which is backed up by using Teracopy to copy the same 20GB test file.

The P411 disk controller is not running as fast as it should so later... as they say.

Cheers, Rob.

[robnicholson]

Hmm, I knew this already but we're on an old firmware for this card and the number of "CRITICAL" advisory from HP about problems with corruption, performance etc doesn't fill one with confidence. Unfortunately, when I tried to upgrade the firmware to the latest version, it ended up with a non-bootable media server. So might be time to dump this card...

Cheers, Rob.

[robnicholson]

I keep having a bash at resolving this same problem, i.e. incredibly slow GRT backup to B2D and deduplication.

At the moment, I'm focusing on B2D as you can't have multiple dedupe storage folders on the same media server.

I've just added 4 x 2TB SATA-2 (3Gbit/s) 7500rpm 64k cache Hitatchi drives locally via HP Smart Array P411 with 256MB cache configured in RAID-0 (just for test!).

You would normally expect pretty reasonable performance from such an array and indeed, a non-GRT backup of a 41GB Exchange mailbox runs at 2,147MB/min backup and 9,143MB/min verify (testament to RAID-0 stripe read performance).

Change one setting to use GRT and the speed drops to an unusable 288MB/min backup but interesting still excellent 8,241MB/min verify speed. That is seven times slower during backup!

Just what exactly is BE doing to run so slowly??? My guess is millions of little writes compared to a continuous read & write of a 41GB Exchange EDB file.

I also guess that tape GRT backup works differently to B2D as it isn't really capable of random access so it much be streaming one big file.

Cheers, Rob.

PS. The above drives are actually SATA-3 (6Gbit/s) but the card is only SATA-2. I've ordered a SATA-3 speed compatible controller as an experiment.

[robnicholson]

Okay, so over the weekend I added a new LSI Logic 9280-8e storage controller connected to the same 4 x 2TB SATA-3 7200RPM 64MB cache disks in RAID-0 configuration with 512MB write-through cache. This is now (IMO) a pretty speedy set-up.

The non-GRT backup speed of the 42GB Exchange mailbox database is similar as the same disks connected to the SATA-2 P411 controller - about 2,347MB/min or 40MB/s. This shows that the backup speed here is constrained by the device being backed up across the network and 40MB/s is a reasonable speed considering the disk system being read from (RAID-10 array based on 15.5k SAS disks but in a shared SAN environment).

The verify speed is off the scale at 22,451MB/min which shows that a four disk RAID-0 array using SATA-3 disks/controller really can speed along.

So with anticipation I ran the same job with GRT enabled and *thud*... still the same unacceptable slow 300MB/min which is x8 slower.

Not sure where to take this now. I'll left the GRT backup finish to get the final backup & verify speed (and the later will be very fast I expect). I'll them rebuild the array as RAID-5 for a laugh - it will show whether the bottleneck is at the media server or the Exchange server.

Rob.

[Mike_Stone]

I'm having this very same issue.

Resource being backed up is an SBS 2011 box (Exchange 2010).

Media Server is a 2008 R2 box.

All backups run at about 2GB/min, EXCEPT the exchange stores, they drop to about 160MB/min.  If I turn GRT off--everything is fast.

The media server is doing B2D to a local RAID-5 volume.  All hardware is brand new HP Proliant gear.  vmware guest storage is all off SAN.  Performance is magnificent in every other respect.  As soon as GRT is enabled--forget it... My 100GB mail store takes 12 hours to backup.

I have this exact issue happening at two sites--nearly the exact same config--something is deffinately wrong.