BE 2010 R3 Incremenal Verify taking hours to start

[naslanidis]

Backup Exec 2010 R3 SP2 fully updated

VMware 4.1 U2 VMware Agent GRT backup

Backup to HP 2024 LTO-5 SAS drive

We have a daily incremental job that backs up a number of virtual and physical servers. Every day the incremental job has kept around the same size but the verify is taking a very long time to get started. For example the verify begins at 3am, however it doesn't actually start verifying until around 7am. Once it begins it's very fast and the job rate is fine.  I'm at a loss to figure out what it's doing in the 3 hours. That period where it is apparently sitting idle is increasing every day. A couple of days ago it took around an hour to start. Yesterday it took around 2 hours to start the verify. The most recent one took 3 hours and the below shows this.

I've looked at the vcenter logs and there is no indication that anything is happening there (snapshot creation etc.) during that period.

Below is an excert from the latest BEX file in the BE Data folder. I've changed the server names but that's all.

Any ideas would be appreciated.

<?xml version="1.0" encoding="UTF-16"?>
<joblog><job_log_version version="2.0"/><header><filler>
        </filler><server>Job server: CXXXXXX1
</server><name>Job name:  Master Selection List-Weekly backup Cycle-Verify Daily Incremental
</name><start_time>Job started: Friday, 23 March 2012 at 3:04:13 AM
</start_time><type>Job type: Verify
</type><log_name>Job Log: BEX_CXXXXXX1_00085.xml
</log_name><filler>
        </filler></header><verify><filler>
        </filler><title>Job Operation - Verify
</title><filler>
        </filler><machine><machine_name>CXXXXXX1</machine_name><media_mount_date>
Drive and media mount requested: 23/03/2012 3:04:17 AM
</media_mount_date><media_drive_and_media_info><media_mount_date>
Drive and media information from media mount: 23/03/2012 3:04:17 AM
</media_mount_date><robotic_library_name>Robotic Library Name: HP MSL2024
</robotic_library_name><drive_name>Drive Name: HP Ultrium 5
</drive_name><slot>Slot: 5
</slot><media_label>Media Label: Daily
</media_label><media_guid>Media GUID: {7bd51a5f-a576-426a-b1f5-e9bfe337d6d2}
</media_guid><media_overwrite_date>Overwrite Protected Until: 12/07/2012 5:14:45 AM
</media_overwrite_date><media_append_date>Appendable Until: 31/12/9999 11:00:00 AM
</media_append_date></media_drive_and_media_info><set><set_resource_name>VMVCB::\\CXXXXXX1\VCGuestVm\(DC)Datacenter(DC)\vm\ Virtual Machines\CXXXXXX4 VMVCB::\\CXXXXXX1\VCGuestVm\(DC)Datacenter(DC)\vm\Virtual Machines\CXXXXXX4</set_resource_name><description>
Verify of "VMVCB::\\CXXXXXX1\VCGuestVm\(DC)Datacenter(DC)\vm\Virtual Machines\CXXXXXXX4 VMVCB::\\CXXXXXX1\VCGuestVm\(DC) Datacenter(DC)\vm\Virtual Machines\CXXXXXX4"
Backup set #266 on storage media #1
Backup set description: ""
</description><start_time>Verify started on 23/03/2012 at 6:26:18 AM.
</start_time><virtual_machine_object>Virtual machine
</virtual_machine_object><virtual_machine_object>Virtual machine CXXXXXX4
</virtual_machine_object><directory><directory_name>Directory \
</directory_name><directory><directory><directory_name>Directory \C

[naslanidis]

Update:

I've done some other test backups and it would appear that in the period before it finally starts the verify the status is "Loading Media" and if I go and look at the web console for the tape library the drive status is "seeking". The length of time it takes before it begins the verify seems to have increased as more backup jobs have been appended to the same tape. I still don't understand why It could possibly take so long to begin the verify though, especially when if I append a new backup the system has no problem seeking to the next appendable point and starts writing data almost immediately. 

Any help with this problem would be appreciated.

[Ken_Putnam]

Are you using the latest Symantece device drivers for your tape drive?

Just for Grins, have you tried the mfg drivers?

[naslanidis]

Hey,

Yes we are using the latest Symantec device drivers.

By mfg drivers do you mean the HP drivers for the tape library?

[Ken_Putnam]

By mfg drivers do you mean the HP drivers for the tape library?

I mean the manufacturer drivers for the tape device

[naslanidis]

OK thanks yes I have tried that.

This problem seems to be veruy consistnet. If a backup is towards the start of a tape the time before it starts actually verifying data is quite short. If the backup is at the end of of a tape it can take several hours to seek to the part of the tape it needs to then begin the verify.

We're getting about 3.4:1 compression on our LTO-5 tapes so there is a lot of data on there. Is it at all possible that what we're seeing is normal?

[Ken_Putnam]

If a backup is towards the start of a tape the time before it starts actually verifying data is quite short. If the backup is at the end of of a tape it can take several hours to seek to the part of the tape it needs to then begin the verify.

If that is the case, it could be the drive itself going bad

Have you run a few passes of the manufacturer's diagnostics against the tape drive??

[naslanidis]

yes we've used the HP tools to do drive assessment tests and media tests and there is no problem. The drive and library is about 3 weeks old.

[naslanidis · Accepted solution]

Just in case anyone else might come across this problem:

Changing from a separate verify job in my policy to just ticking the box to verify the backup in the backup job itself seems to have resolved this problem. It doesn't spend ages on "loading media" when I just tick the verify option within my bacup job. 

So it would seem that choosing that option allows BE and or the tape library to keep it's place on the tape when it is looking to start the verify after it completes the backup. When the verify job is seperate in the policy it seems to have to go through the tape start to finish to find the recently backed up data.

At least that's how it looks to me.