Backup fails once in a while with "Corrupt data encountered"

[cpegel]

Hello,

I am running Backup Exec 12.5 on Windows Server 2003 x64. The Server is backing up some of it's own data on the harddrives and is using Microsofts Shadow Copy Service.

Every now and then a full backup fails with the error 0xe000fe36 - Corrupt data encountered.

I don't think there is really corrupt data, because the data claimed to be corrupt is not always the same and the same backup job sometimes runs just fine when I rerun it.

Any clues on how to fix this issue? It's really annoying to repeat backups again and again, not knowing why they fail.

Thanks in advance,
Christoph

[Dev_T]

Hello Chris,

This KB might help:

http://support.veritas.com/docs/193940
How to prevent Backup Exec from failing a backup operation when encountering corrupt files

[cpegel]

Thanks, but this KB does not seem to refer to 0xe000fe36, which is the final error code in my case.
Also this workaround would just flag the backup as succesfull even it wasn't (since it backed up corrupt data).

This might help getting the backup to finnish, but it's not a satisfying solution in my opinion.

[CraigV]

Hi,

Stupid question, but are you using AOFO at all in your backups?

[cpegel]

Yes, I'm using AOFO set to Microsoft Volume Shadow Copy Service with Snapshot Provider "System".

I had a deeper look in the related logs and found this exception:

V-79-57344-34108 - An unexpected error occurred when cleaning up snapshot volumes. Confirm that all snapped volumes ... AOFO: Started for resource: "Shadow?Copy?Components". Advanced Open File Option used: Microsoft Volume Shadow Copy Service (VSS). The following volumes are dependent on resource: "C:" . The snapshot provider used by VSS for volume C: - Microsoft Software Shadow Copy provider 1.0 (Version 1.0.0.7).

So it's a problem with the shadow copies?

Edit: It came into my mind that this also might be related to a running antivirus software. I'm running Kaspersky antivirus on the server.

Edit: I added the Backup Exec binaries (beserver.exe, beagent.exe, etc.) to the exclusion list of Kaspersky, so it wont prevent files to be opened, still no success. After I did the changes I ran a full backup, successfully. Then I waited for the next full backup in the regular schedule,. and it failed.

[Dev_T]

Hello,

Exclude the folder  C:\ OR D:\Backup Exec AOFO Store (HIDDEN FOLDER) form the Virus Scan. This is the temp location for storing the Snapshots and is deleted after the backup is over. The Antivirus is preventing it from deletion thats the reason the following error:

V-79-57344-3410 An unexpected error occurred when cleaning up snapshot volumes. Confirm that all snapped volumes are correctly resynchronized with the original volumes.

Hope this helps...

[cpegel]

Thanks, I added both C:\Backup Exec AOFO Store\ and D:\Backup Exec AOFO Store\ to the trusted zone of Kaspersky.

I'll check the logs next week to see if it helped, will report back then. Thanks!

[cpegel]

The daily differential backups went all fine, but yesterday the full backup failed again.

The Job completion status is

Job ended: Sonntag, 7. März 2010 at 04:02:02
Completed status: Failed
Final error: 0xe000fe36 - Corrupt data encountered.
See the job log for details. Final error category: Resource Errors For additional information regarding this error refer to link V-79-57344-65078

A lot of errors occur because Backup Exec can't find directories, but those directories are in fact existent and I can open them in the Explorer very well.
E.g:

Backup- D: 
Directory Ressourcen\Mitarbeiter\...\Download\...\ was not found, or could not be accessed. None of the files or subdirectories contained within will be backed up. V-79-57344-65033 - Directory not found. Cannot backup directory \Ressourcen\Mitarbeiter\...\Download\...\... and its subdirectories.

The Exceptions say:

Backup- D: 
WARNING: "D:\Ressourcen\Mitarbeiter\...\Download\...\....iso" is a corrupt file. This file cannot verify. The item D:\Ressourcen\Mitarbeiter\...\Download\...\setup.exe in use - skipped. V-79-57344-34108 - An unexpected error occurred when cleaning up snapshot volumes. Confirm that all snapped volumes are correctly resynchronized with the original volumes.

Backup- Shadow?Copy?Components 
The .DR file that is required for Intelligent Disaster Recovery is not updated.

Backup- System?State 
The .DR file that is required for Intelligent Disaster Recovery is not updated.

Verify- D: Storage

WARNING: "....iso" is a corrupt file.

This file cannot verify.
 

I replaced some paths with dots to protect our users privacy.

Any further ideas what is going wrong?

[CraigV]

OK...time to look into creating a new selection list.
My suggestion is to do that, and then try a manual backup using that selection list. I have seen this happen before. This resolved it for me, so give it a bash. No need to delete the current selection list though, just in case it doesn't work!

[cpegel]

Thanks for your help. I created a new selection list, set the daily differential and weekly full backup jobs to use this new list.

Right now I am running a full backup manually, let's see if that works out. I will report back if that works and will mark your post as solution if the next few scheduled full backups succeed (which I hope to be true).

[cpegel]

The manual backup with the new selection list also failed with 'Corrup data encountered'. :(

Edit: I had a closer look in the knowledge base and found that Dell OpenManage Server Administrator might prevent BE from updating the DR-files. I deinstalled the OpenManager Server Administrator, restarted the BE services and will run a manual backup again, let's see..

[CraigV]

Hi there

Did you come rght with disabling the Dell software?

[cpegel]

Unfortunately this didn't help. Full backups are still failing with corrupt data encountered. I'm running out of ideas.

[CraigV]

OK...I would then suggest keeping AOFO selected, but allow BEWS to select the snapshot provider to use. Set this to Automatic in Tools --> Options...

Post the results here...

[cpegel]

Okay, I set all jobs to automatically select the snapshot provider, will run a full backup later this day and report back. Thanks

[cpegel]

Now I get

Final error: 0xe000fe7d - Access is denied.
To back up or restore System State, administrator privileges are required.
Final error category: Security Errors

I'm using the domains Administrator account for the backup and checking the credentials in the job setup runs fine.

Edit: On https://www-secure.symantec.com/connect/de/forums/access-denied-errors-and-faliure-messages i found that uninstalling HF 334837 might fix this issue, did that, rerunning the backup now.

Edit: And again: Corrupt data encountered.

[CraigV]

Check out this link below...it has to do with your VSS error in that log you posted...

http://seer.entsupport.symantec.com/docs/276343.htm

[cpegel]

Thanks,  but I already checked everything that is suggested in this link. I added the AOFO Store directories to the trusted zone of my virus scanner (Kaspersky) and uninstalled the Dell OpenManage Server Administrator.

[CraigV]

...Do you get the same error with the AV disabled?

[SystemsAdminAS]

Christoph,

Are you specifically using the VSS shadow copy service or leaving BE to automatically choose the provider?  Also, do you have all BE updates / hotfixes applied?

Also, since you're running x64, this page might have a clue...

http://msdn.microsoft.com/en-us/library/bb968832(VS.85).aspx

I would think backing up the system state would be possible, but if you're using a x64 system, and since BE is generally a x86 application, I wonder if BE is using a 32-bit VSS requestor under WOW64, which according to this article is not supported for system state...

Craig, any thoughts on this?