Forum Discussion

Level 3

14 years ago

Backup to Disk - Extremely slow verify (BE 2010 R2)

Hello everyone,

I encounter extremely slow verify rate when doing backup-to-disk.

Media server:

-Hardware: HP ProLiant DL360 G5 (2 x Xeon 5140, 18 GB RAM, Smart Array P400i with Battery Back Write Cache and brand-new battery, 4 x HP 600GB SAS-2 6Gbps 10,000 rpm drives in a RAID 5 array, 512MB strip size => 1.5 MB stripe, 2 logical drives, HP SCSI HBA which is ATTO ExpressPCI UL5D LP)

-OS: Windows Server 2008 R2 Standard, all updates up to and including February 2011Backup to Disk target:

-Tape drive: Tandberg 820, SCSI

-Local logical drive on media server mapped to drive letter D, 1.5 TB, NTFS file system, 64K allocation unit size

Backup Exec version: 2010 R2 (13.0 Rev. 4164), with Service Pack 1 and Hotfix 150096 installed

Backup to Disk Folder settings:

-Maximum size for backup-to-disk files: 4 GB (also tested with 1 GB)

-Allocate maximum size for backup-to-disk files: YES

-Maximum number of backup sets per backup-to-disk file: 100 (application default)

-Concurrent: 1

Results of running B2D Compatibility Test Tool (B2DTest_64.exe) on B2D target: ALL PASSED (I examined the log in C:\ProgramData\Symantec\BackupExec\Logs\B2DTest.log and saw that all 16 tests passed.)

Media sets:

-Weekly: 13-day overwrite protection, 1 hour append (just a formality, since there's no option to set it lower)

-Daily: 6-day overwrite protection, 1 hour append (same as above)

Remote server being backed up:

-Hardware: HP ProLiant DL380 G7 (1 x Xeon 5620, 36 GB RAM, Smart Array P410i with 512MB Flash Back Write Cache, 8 x HP 300GB SAS-2 6Gbps 10,000 rpm drives in 2 arrays - RAID 1 for OS, RAID 5 for data, 2 logical drives)

-OS: Windows Server 2008 R2 Standard, all updates up to and including February 2011

-RAWS (x64) which was push-installed from BE media server

Network: Both the media server and the server being backed up have Gigabit NIC, both are connected to the same HP ProCurve 2910al switch (Layer 2/3 managed switch), both NICs and ports on the switch are forced to 1000, no auto-negotiation.

I originally set up a policy to have Full backup (Copy files) running on each Friday evening which will back up the entire remote server described above with verify, then a duplicate job linked to this Full backup job will run immediately afterwards to duplicate the sets to tape (LTO Ultrium 3). During the week Monday through Thursday a recurring job will run to back up Working Set (changed today).

When I was test-running the Working Set job last week (to disk), it took some 3 hours to back up/verify 24 GB. I examined the job history and found that the verify speed was the culprit, as slow as 282 MB/min. The rate for various sets varied between 261 MB/min - 1152 MB/min.

In terms of backup speed, I thought the network was the bottleneck, so I performed a test by copying a set of 11 files (a total of 6.42 GB, which consists of the DVD ISO for BE 2010 R2 which is 3.3 GB, plus those 1of4, 2of4, etc. splitted files) from a Windows file share on the server being backed up to the media server (stored them in the same folder as the BKF files). Speed was ~6600 MB/min. Then I created a test backup job in Backup Exec with AOFO (VSS) and no compression, and that job ran at 3988 MB/min backup rate (and decreasing), but a mere 378 MB/min for verify (yes, I know verify is performed locally, still the 6600 MB/min copying using SMB v1 (for testing I disabled SMB v2) vs. 3988 MB/min backup rate shows that something is wrong, probably with BE.

Now, in terms of verify speed, I know that BE attempts to read the sets from the BKF files and calculate checksums to compare with stored checksums. I know that RAID 5 is slow, but it can't be that slow. Something is seriously wrong with BE if the verify speed is 378 MB/min average.

During verify operations, I checked the CPU and memory usage on the media server, and there's no spike whatsoever, CPU barely even hit 1%.

I have run both ATTO Disk Benchmark Tool and IOMeter to benchmark the RAID and can't see anything wrong from the results, so it can't be the disk subsystem.

At this point, I'm so frustrated that I don't even know what else to do, short of NOT verifying. I had just rebuilt the media server from scratch and tested again, pretty much same results.

If anyone could chime in and point me to the right direction, it would be much appreciated.

2010

Backup and Recovery

Backup Exec

Backup Exec (media server only)

Remote Agent for Windows Servers

18 Replies

pkh
Moderator
14 years ago
Have you tried defragmenting the drive where your B2D folder resides?
Mary_Land
Level 3
14 years ago
I forgot to mention that there is no dedupe involved here, since we never purchased the deduplication option for BE.

Just straight B2D.
JoaoMatos
Level 6
14 years ago
Hi,

what is the performance when you copy data to the B2D drive?

Try disable antivirus checks is not active for writing.
Mary_Land
Level 3
14 years ago
Hi pkh,

I checked for fragmentation, and there is no fragmentation (since that entire logical drive is dedicated to the B2D folder, and all BKF files were created with maximum size pre-allocated).

Thanks.
Mary_Land
Level 3
14 years ago
Hi JoaoMatos,

No anti-virus is currently installed on either the media server or the (remote) server being backed up.

I am running Microsoft SQLIO Disk Benchmarking Tool on the B2D drive now (sequential read, sequential write, random read, random write). I will post the results shortly.
Mary_Land
Level 3
14 years ago

The SQLIO disk benchmarking results on the drive (RAID 5) where the B2D folder resides are posted below. I know that SQLIO and other benchmark tools are synthetic, but still the results (throughput, latency, etc.) look acceptable.

I still don't know why Verify takes so long if what it does primarily involves reading the data back from a drive which is local. Had it involved deduplication, that would have made sense because in that case a full rehydration of the files had to be performed (at least that's how I understand it) which would resulted in a significant slow-down. In my case, as I have mentioned, there's no dedupe at all.

----------------------------------------------------------------------------------------

SEQUENTIAL READ 64K (64 KB I/O request size, 2 threads, multiple I/Os per thread enabled with 64 outstanding requests per thread, running continuously for 10 minutes)

sqlio -kR -s600 -fsequential -o64 -b64 -LS -Fparam.txt

CUMULATIVE DATA:

throughput metrics:

IOs/sec: 12647.58

MBs/sec: 790.47

latency metrics:

Min_Latency(ms): 1

Avg_Latency(ms): 9

Max_Latency(ms): 27

---------------------------------------------------------------------------------------

SEQUENTIAL WRITE 64K (64 KB I/O request size, 2 threads, multiple I/Os per thread enabled with 64 outstanding requests per thread, running continuously for 10 minutes)

sqlio -kW -s600 -fsequential -o64 -b64 -LS -Fparam.txt

CUMULATIVE DATA:

throughput metrics:

IOs/sec: 7349.04

MBs/sec: 459.31

latency metrics:

Min_Latency(ms): 1

Avg_Latency(ms): 17

Max_Latency(ms): 503

---------------------------------------------------------------------------------------

SEQUENTIAL READ 256K (256KB I/O request size, 2 threads, multiple I/Os per thread enabled with 64 outstanding requests per thread, running continuously for 10 minutes)

sqlio -kR -s600 -fsequential -o64 -b256 -LS -Fparam.txt

CUMULATIVE DATA:

throughput metrics:

IOs/sec: 3217.58

MBs/sec: 804.39

latency metrics:

Min_Latency(ms): 5

Avg_Latency(ms): 39

Max_Latency(ms): 67

---------------------------------------------------------------------------------------

SEQUENTIAL WRITE 256K (256KB I/O request size, 2 threads, multiple I/Os per thread enabled with 64 outstanding requests per thread, running continuously for 10 minutes)

sqlio -kW -s600 -fsequential -o64 -b256 -LS -Fparam.txt

CUMULATIVE DATA:

throughput metrics:

IOs/sec: 1860.14

MBs/sec: 465.03

latency metrics:

Min_Latency(ms): 13

Avg_Latency(ms): 68

Max_Latency(ms): 105

---------------------------------------------------------------------------------------

RANDOM READ 64K (64KB I/O request size, 2 threads, multiple I/Os per thread enabled with 64 outstanding requests per thread, running continuously for 10 minutes)

sqlio -kR -s600 -frandom -o64 -b64 -LS -Fparam.txt

CUMULATIVE DATA:

throughput metrics:

IOs/sec: 2928.54

MBs/sec: 183.03

latency metrics:

Min_Latency(ms): 0

Avg_Latency(ms): 43

Max_Latency(ms): 642

---------------------------------------------------------------------------------------

RANDOM WRITE 64K (64KB I/O request size, 2 threads, multiple I/Os per thread enabled with 64 outstanding requests per thread, running continuously for 10 minutes)

sqlio -kW -s600 -frandom -o64 -b64 -LS -Fparam.txt

CUMULATIVE DATA:

throughput metrics:

IOs/sec: 7340.28

MBs/sec: 458.76

latency metrics:

Min_Latency(ms): 0

Avg_Latency(ms): 17

Max_Latency(ms): 2089

---------------------------------------------------------------------------------------
Mary_Land
Level 3
14 years ago

With respect to the drive where the B2D folder lives, what I will do this weekend is to use HP Array Configuration Utility to delete the RAID 5 drive and use the same space (exact same space on same physical disks) to create a RAID 1+0. In other words, the B2D folder will live on a RAID 1+0 drive, while everything else is kept the same. I will manually force the same jobs to run (with Verify) to see whether it would make a difference.

I will also use SQLIO with the same parameters to benchmark that newly created RAID 1+0 B2D drive and will post the results here.

To be quite honest, if Verify is read-intensive, I don't expect much of a difference in read performance when changing the B2D drive from RAID 5 to RAID 10, because from what I understand, RAID 5 only takes a performance hit when writing (because of the parity calculation), whereas the only thing RAID 5 does when reading is some sort of block redirection (due to striping) and no calculation is involved.
Mary_Land
Level 3
14 years ago
When reading contents back from those BKF files (in order to calculate/compare checksums), what type of read (Sequential or Random) does a Verify operation perform on that B2D disk?

Could someone confirm? Thanks.
Dariel_Cruz
Level 3
14 years ago
am running into the same issue, seems to be related to reading from the B2D, Writing seems fine. I have a fibre channel disk array and tape drive combination, the backups to the disk are fine, the backups to the tape directly run fine, but from B2D to tape are bad. I have tried everything I could think of including updated drivers and flash updates for hardware, nothing seems to work, I ran out of ideas, I see the disks usage at 100% when backup exec reads from it yet the speed sucks.
TadSend
Level 0
14 years ago
I upgraded from BE12.5 to BE 2010. The DLT tape drive used to do 500 MB/min. Now it's down to 115 MB/min!

I installed a removeable HD. This will backup at 1016 MB/min but degrades down to 115 MB/min or so after some hours.

Tape Backup used to take 5 to 6 hours for BU and verify on the DLT tape drive. Now it take at least 20 hours unless I give up and cancel first.

This is a major problem for me.

I have tried to change the Quantum Tape drive driver from Symantec to Quantum. No apparent change.

Do anyone have any suggestions please?

Tad Sendzimir