Strange "corrupt" file behavior in Exchange backup...

[Darren_Berkey]

Greetings to all...

I am a junior network admin at a small office with about 60 active mailboxes on an Exchange 2003 server.  Recently, the Backup Exec job history has been reporting corrupt email messages like this...

WARNING: "\\EXCH\Microsoft Exchange Mailboxes\Username [unique ID]Top of Information StoreInbox 〰...
This file cannot verify.

Now, I've done some research on this, and I know that there are several support pages that address the issue, but let me elaborate on this situation a little bit.

For the past month or so, every few days, the job history would show 1 or 2 of those corrupt/cannot verify errors.  In each case, the sender was always the same person and the messages always went to the same recipient here, but only when the message included an attachment. They were also usually recently sent... within a day or two of being reported as corrupt.  Typically, I would just delete the offending messages from the server.  (After making sure the recipient had a local copy, of course.)

Well, today was very different.

Today, the job history showed 74 corrupt messages like the one above that could not verify!!  This time, they were from several different senders and affected more than a dozen recipients.  Even more strange, the messages showed original send dates varying anywhere from 3 days to 8 months ago.  (Yeah, messages that have been sitting on the server for 8 months, doing nothing, suddenly show up as "corrupt" today.  Huh???)

I'm at a complete loss to explain it.  Naturally, I thought of a virus, but our AVG Pro is active, updated and doesn't have anything significant in it's virus vault.

The only two things that these stored "corrupt" messages have in common is they all originated from our overseas (China/Taiwan) offices, and they all have attachments.  (.eml, .doc, .pdf, .jpg... the usual stuff.)  I've checked most of the messages that are supposedly corrupt and I've had no trouble opening the attachments, or reading the messages themselves.  I don't know if the asian characters have anything to do with it, but we deal with them in our email all the time, and they've never been a problem.

My main concern is the overnight transition from only one or two messages every so often that were sent very recently, to this sudden reporting of a large number of messages that have been stored on the server for several months.

I'd be very grateful if anyone has any clues on what I should look for.  Granted, I may discover that this is not necessarily a Backup Exec problem, but I need to start somewhere...

Thanks!

 - Darren B. -

[Darren_Berkey]

Minor addition...

Stephen mentioned that in his case only messages with other attached messages were being affected, and at the time, I thought I had a mix of emails-attached-to-emails along with emails with only files (jpg, pdf, doc, etc...) as attachments.

I looked more closely at most of the messages that are flagged as corrupt, and indeed, each one that I looked at had another email message attached to the first.  (Some of them went 2 or 3 attachments deep.)

In BE's options sheet, under Microsoft Exchange, there is the option "Enable single instance backup for message attachments".  Do any of you have that option checked, or have you used it in the past?  I read an article while doing research on this problem that suggested trying this option.  I may give it a shot.

EDIT:  I went ahead and enabled it... I guess I'll see what happens on Monday. 

 - D -

Message Edited by Darren Berkey on 05-18-200703:00 PM

[Jon_Ziminsky]

I have been running my backups with the "Enable single instance backup for message attachments" since before this broke.

Still no word back from Symantec. Isn't Symantec a certified Windows partner? If so, shouldn't they be responsible for patching their software, as i can only assume Microsoft is going to tell them that their patch is working as intended.


JZ

[Darren_Berkey]

Yeah, this is getting kinda silly...

The option for single instances of attached messages made no difference on my backup either.  I still had the same 73 corrupt messages listed on the backup log this morning.  *sigh*

I really don't want to spend another 3 hours on the phone with MS Tech Support...

 - D -

[MikenTech]

We just did updates to one of our clients servers and they are receiving the same errors - we are removed KB931768 and KB931832 to resolve the issue.

Hopefully that will help convince Symantec and Microsoft that there is a problem.

[Jon_Ziminsky]

Response i got from tech support.

Our developers are currently working on a resolution for the issues caused by http://www.microsoft.com/technet/security/bulletin/ms07-026.mspx. We cannot provide an estimate on when or what form of resolution will be provided at this time.

There is no need to remove the patches. The backups you are taking are good backups, with the exception of the messages that are being flagged. I have tested, and am able to restore messages from the backups. If the messages that are being flagged are crucial, back them up to a .pst and back the .pst up.

Removing patches is NOT! a viable workaround, or solution.

[DCanfield]

I am having the same problem.  The failures started on 5/16/07 the same day I applied KB931832.  I hope Symantec releases a hotfix soon.

[SS_2]

I am also experiencing the same problem. I logged a case with tech support on 5/14/07 and was told to run ISINTEG and call them back. As we all know it did not fix the problem and I too am still waiting for a solution...

[Darren_Berkey]

Thanks for participating in this thread, guys!  I've also been able to verify that the backups are perfectly good, despite the "corrupt" status being reported.

JZ, it's encouraging to see that Symantec is working on the issue, even if it might be awhile before they figure anything out.  A little progress is better than none at all. 

Have a great Memorial Day weekend!

- D -

[BackupExec_Vete]

I have also experienced the same problem. I have my nightly backups configured to backup the information stores and not individual mailboxes. I would get the corrup message but it would not identify which mailbox or item was corrupted.

By running a backup of all mailboxes you will be able to identify the user and e-mail object that is corrupt. Once identified, the item is removed from their mailboxes and the backup of the information store completes now with no problem.

[INeedHelp]

I have been experiencing the same problems associated with KB931832 since May 14th.  I am actually a little suprised that we don't have a fix for this yet as it is obviously affecting multiple Symantec patrons.  I did some research on this last week and came across the article that would have you apply the registry fix (to allow the backup to complete with exceptions versus failing) as though this was a desirable solution.  I got quite a chuckle on that one since it does not even come close to fixing the problem, just masking it. This may only be a cosmetic error but it's an error that should not be happening.  You would think that since Backup Exec is such a widely used program that Microsoft would have done some testing with KB931832 and Backup Exec before going live with it! 

I was completely in the dark until I came across this piece.  Thanks for all of the information.

[Shelby]

The Microsoft patch is affecting you in others ways as well.  Check you mail queue, you will find mail getting stuck (not all mail, just some).  The e-mails that Veritas says is corrupt, are basically corrupt from the patch, if you go to the bad e-mail, you will not be able to open the attachment.

Microsoft has put out a bad patch that needs to be corrected soon.  (there is a hotfix out there for the mail queue, use the Google and you will find it)

I have a case open with Symantec, and the Tech has been very helpful.  Microsoft has put us all in a jam with this one, but at least the backups do appear to be good (minus all the corrupt e-mail).

Happy patching.

[Jon_Ziminsky]

I just checked my mail queues, and i am not experiencing any issues.

The emails that are being marked as corrupt, and their associated attachments are all readable and openable. They are not corrupt in any sense, except in BackupExec's mind.

[BackupExec_Vete]

Same here. I checked my queues and there are no pending messages. When I ran a backup of all mailboxes not the mailbox stores, BE listed the user and e-mail that was corrupt. We were able to open the attachments and the messages are not corrupt. Once removed from the mailbox I am able to successfully backup the mailbox store.

[Kruizer]

I just wanted to note that we are here experiencing the same behavior as well as a failure to read the log files.  We are receiving the corrupt file messages that are only on forwarded emails stored in the information store.  We are also receiving this error:

Illegal xml character.

261: C:\Program Files\VERITAS\Backup Exec\NT\Data\BEX_SERVER_02352.xml

Which started happening around the same time.

Again, just trying to keep the thread current, and add to the list for Symantec to repair the issue.

Thanks,

[Jon_Ziminsky]

I receive the same error as well when trying to view the logs from the "problem" jobs.

JZ

[INeedHelp]

Hello Guys,

I hope everyone had an excellent weekend.  I just wanted to add a few things regarding the past few posts.  I am not experiencing any further problems other than the backups being "completed with exceptions".  My exchange server has no stuck emails and all of the "corrupt" emails are in fact not corrupted and readable. 

When viewing the logs, are we talking about viewing them through BE or actually viewing the file itself?  Through BE I can view them without any problems but I haven't tried opening the log file itself just yet. 

I hope we aren't experiencing two or maybe even three problems here with a few similar symptoms!  That might cause the fix to be alittle delayed.

I hope we all get this problem fixed soon.

Does anyone know if the MS patch in question is causing these problems internationaly or if they are localized to just the North American market?

Y'all have a good one.

Message Edited by INeedHelp on 06-04-200704:41 PM

[gdelorey_2]

I am also experiencing the same issue as everyone else. My clients backup jobs have been failing since this Exchange patch rolled out a few weeks ago. Does anyone know if Symanted is currently working on a patch to resolve this issue?

Thanks,

: Greg

[Raymond_Ho]

I have the exact problem since having the MS Ex patches installed. 

I went as far as uninstalling BE 10.0 and now upgraded to 10D on my Ex 2003 and the same issues continued.  Symantec should by now issue some official statement so users like us don't have to search the net for answers.  Some of the errors found since then also included "Top of Information StoreSync Issues" wonder if anyone has any idea.

[Veritas_user_JJ]

My backups of mail server are "failing". I got this error message:

Illegal xml character.

126: D:\Ohjelmat\VERITAS\Backup Exec\NT\Data\BEX_DC2_01371.xml

In fact BE could not backup one message, but log doesn't say witch one?

WARNING: "\\MAIL\Microsoft Exchange Mailboxes\ 䅁36CBB109FD706C101993E93CE6BDCA84C8D8731A41379AB0" is a corrupt file.
This file cannot verify.
Backup completed on 9.6.2007 at 1:13:41.
Backed up 204949 mail messages in 4635 folders in 155 mailbox(es)
1 corrupt file was backed up
Processed 12 318 961 570 bytes in  1 hour,  58 minutes, and  22 seconds.
Throughput rate: 99.3 MB/min

How can i track down that one corrupt file, witch seems to cause backup job to fail?

At first there was another corrupt file, but that one i managed to find and delete, because log said in witch folder corrupted message were.

This all started in May after i updated our mailserver from Windows update like i do every month after MS releases their security patches.

[Dean_Bliss]

First - Let me provide some information about when Backup Exec marks a resource as corrupt:

"The error message is generated when Backup Exec (tm) attempts to back up a resource that is in use by another application or user. Backup Exec may back up partial data from such a resource, but when it is not able to continue the backup of the resource, Backup Exec marks the resource on tape as corrupt. The resource, as it exists on the server, is INTACT. The error message will fail the backup job."

So you see, just because Backup Exec is unable to backup the item doesn't mean the item is corrupted on the actual production server.

So, what causes Backup Exec from being able to backup a resource?

Well, this can can be due to anything in the environment. In this post with Exchange mailboxes it could be caused by Email Virus Scanners, MS Exchange Patches, or Exchange configuration changes.

I would recommend that you disable email scanning during the time of the backup - (please refer to your scanning software).

Verify any changes that the Microsoft patches apply to your system - if a particular Exchange setting is changed - revert the changes to the original or default setting. I.E. - mailbox security.