Dedupe verify backup job hangs/how do you stop a job that's crashed?

[robnicholson]

I suspect we may have to raise a support call (is there a link for this?). New Windows 2008 R2 server, new Backup Exec 2010 R3 media server installation, backing it via de-duplication. The verify of a full backup of a relatively small drive (20GB C: drive of  Windows 2008 R2 file server) fails each time it's run on the same file:

http://www.picpaste.com/verify_hang-7B1hAK4x.png

Interestingly, it's a Backup Exec file: C:\Program Files\Symantec\Backup Exec\RAWS\VMware\VixDiskLib\bin\vmomi.dll - assuming the hang is occuring on the last file displayed and not the next one it's trying to do. Equally interesting, vserver003 is indeed a virtual server hosted on ESX Server 3.5 but as far as I know, we're not trying to do any kind of virtual machine backup here (how would I check if we were?).

Problem #1: why is verify hanging on this file and any idea how to fix it?

Problem #2: the job won't cancel, it sticks at the "cancel pending" forever (last time we left it 48 hours). So how do you manually stop a job that's crashed like this without restarting the services? We've got another important Exchange job that is still running and we need that to complete as the logs are growing and growing.

With this crashed job and Exchange running, no other backups can run (no idle devices). I tried increasing the number of concurrent jobs on the de-duplication folder from 2 to 3 but that requires the services to restart anyway. Service restart from changing one small parameter????

Cheers, Rob.

[pkh]

Could your problem be related to this?

https://www-secure.symantec.com/connect/issues/verify-job-fails-0xe0000602-query-media-which-data-se...

[robnicholson]

The symptoms are certainly similar to that post but the job hasn't actually failed so there is no error message. It's been in the "cancel pending" state for nearly 24 hours. I'll certainly get in touch though.

I'm loathe to touch the backup system as we're trying to get a complete Exchange backup (just one would be nice) but the backup of our 100GB Exchange system has taken 47 hours so far. It's still doing something (byte count is slowly creeping up) with the logs so we're going to leave it until after the weekend. It's a separate issue why this Exchange backup is so slow.

Cheers, Rob.

[pkh]

Have you tried backing up your Exchange without dedup?  By doing so, would isolate what is the problem.

[robnicholson]

The job that's crashed isn't the Exchange job - the crashed job was a test de-dupe backup of the C: drive (OS mainly) of our file server. The Exchange job just happened to start afterwards and as we've currently got de-dupe configured for 2 jobs at once, the crashed job plus Exchange are holding up the queue.

But I agree that having Exchange running at the same time could be a contributing factor. When (if!) the Exchange backup finisheds (47 hours so far), we can carry out some further tests but right now, getting a backup of our mail system is priority #1.

I'll do the same backup to a B2D folder as a test. It's only a small backup and doesn't take long.

Cheers, Rob.

[robnicholson]

Later, B2D of same selection works fine. Will re-test de-dupe job later when server is free.

One perplexing thing, the C: drive of VSERVER003 is this:

Fri 15/07/11 09:46:35>free \\vserver003\c$

    32,119,975,936 bytes total disk space
    20,582,866,944 bytes used
    11,537,108,992 bytes free
              64.1 % in use

But the backup job is only backing up 2.5GB as shown in the report below.

Is that because system state is not enabled therefore some folders are excluded from normal backups? I'm guessing the 4GB swap file can be taken off that 20G figure above for starters.

Cheers, Rob.

Family Name: "Media created 14/07/2011 13:34:48"
Backup of "\\VSERVER003.company.local\C:"
Backup set #5 on storage media #37
Backup set description: "friday copy"
Backup Method: Copy - Back Up Files

Backup started on 15/07/2011 at 09:36:15.
Backup completed on 15/07/2011 at 09:43:01.

Backup Set Summary
Backed up 6508 files in 2984 directories. Processed 2,508,720,395 bytes in 6 minutes and 46 seconds. Throughput rate: 354 MB/min Compression Type: Software Software compression ratio: 1.6:1

[robnicholson]

Just tried this - B2D works fine (backup & verify no problems). Same selection list, de-dupe policy - crashes at the same point every time.

Cheers, Rob.

[pkh]

Quite a bit of files, like the swap file, hibernation file, system state, shadow copy components are automatically excluded from "normal" backups.  Also, AFE will exclude database files.

[pkh]

If you are in test mode, try re-creating the dedup folder.