Local B2D is very slow

[ScottL]

I have an isolated department with a single server that backs itself up to disk and then to tape. It's server 2003 r2 sp2 with Backup exec 12.5 sp3. They have 2tb of local disk storage (raid5, in addition to a 200g local OS raid 1) and a 7TB DAS (raid 10 over 14 1tb drives) with 1.5TB of data on it currently. 

I run a full backup of this 1.5tb from the DAS to the local storage (D) and it takes 180 hours. It works out to be in the 100mb/minute range by the time it finishes. Basically, it starts off with a good rate and will finish the first 90-100GB in just over 30 minutes (rate of 2.5gb/min) and will continually slow down for the rest of the backup. CPU starts out very low and as the backup starts to slow the CPU will start ramping up from 0-10% up towards 30-40%. By the end of the 1.5T its going incredibly slow.

For some tests i did this backup from the DAS straight to tape and got a rate of over 5GB/Min. It's obviously not a read problem from the DAS. 
Then i restored all of the data from the tape back onto the local backup volume. This rate was around 4.3 GB/min, so just writing to that volume isn't a problem. 
So individually both storage systems seem to be fine but using them simultaneously is causing some major issue. Any ideas?

[teiva-boy]

 RAID 10 is always going to be faster than RAID5 for writes.  Add to that, you have more spindles for your DAS than you do for local storage.
What you are seeing is expected behavior.  

Your local storage and DAS could go even faster if you do a "diskpart align=32" command on the partitions.  You should see a 10% boost immediately.  

The other thing to think about is your DAS and local storage and the RAID card in question...  Are you exceeding the throughput of the CPU on the RAID card?  Often, to get max throughput, you want your local drives on some on-board RAID card, and your external DAS on a dedicated RAID card on it's own PCI-e (or better) slot.  

Also D2T is always going to be faster than tape directly.  Backing up a single large contiguous file is always faster to tape than millions of little files to tape.  Thus B2D2T is the preferred route to go!!! <-I cant emphasize that enough.

That said, separating the RAID functions would help.  the align command in diskpart can too (it's free why not?), even possibly playing with various sector and block sizes too on the volumes to see which is the better performing combo.

[ScottL]

 The local disk is on perc 5/i integrated raid controller and the DAS is on a perc 6/e SAS controller in a pci-e slot. My problem is that writing to the local storage can be fast (when coming from tape it was over 4GB/min), but is very slow only when reading the data from the DAS (slows down eventually to ~100MB/min). I'm never writing to the RAID 10, only reading from.

[teiva-boy]

 So read from DAS (RAID 10) is slow?  

Defrag.  Backup to disk in BE fragments the disk heavily.  it's pretty absurd actually.  Either defrag regularly or allocate all your space for the B2D space now rather than letting it create new BKF files.

Side note, you should go to BE2010 with the dedupe option, it'll save you a ton of space, and reduce your backup windows to disk.  

[ScottL]

 The read from the DAS raid10 isn't slow...when writing to tape. This is what i was trying to explain in my first post. 

Read from DAS + write to tape = 5+GB/min
Read from Tape + write to local = 4+GB/min

Read from DAS + Write to local = 100MB/min

so independently we know we can read from the DAS at a fast speed, and write to the local backup volume at a fast speed, but doing both at the same time is causing some major bottleneck.

[teiva-boy]

Writing to a RAID 5 array is typically slower.  
Flip it, write to the DAS from local storage, whats the performance like?  More than likely it'll write to DAS faster.

Also use a benchmarking program like HDTune, HDTach or Intel's I/O Meter.  Those show the true performance of a volume both peak, sustained, and over time.


And finally a pet peeve of BackupExec for the last decade..  MB/min.  Why cant it be measured MB/sec like any other benchmark tool  Why cant it be real time rather than an average over time.  Show both would be preferred.

[CraigV]

Just be aware that when aligning disks, it is a destructive task and will destroy all your data.