Strange "corrupt" file behavior in Exchange backup...

[Darren_Berkey]

Greetings to all...

I am a junior network admin at a small office with about 60 active mailboxes on an Exchange 2003 server.  Recently, the Backup Exec job history has been reporting corrupt email messages like this...

WARNING: "\\EXCH\Microsoft Exchange Mailboxes\Username [unique ID]Top of Information StoreInbox 〰...
This file cannot verify.

Now, I've done some research on this, and I know that there are several support pages that address the issue, but let me elaborate on this situation a little bit.

For the past month or so, every few days, the job history would show 1 or 2 of those corrupt/cannot verify errors.  In each case, the sender was always the same person and the messages always went to the same recipient here, but only when the message included an attachment. They were also usually recently sent... within a day or two of being reported as corrupt.  Typically, I would just delete the offending messages from the server.  (After making sure the recipient had a local copy, of course.)

Well, today was very different.

Today, the job history showed 74 corrupt messages like the one above that could not verify!!  This time, they were from several different senders and affected more than a dozen recipients.  Even more strange, the messages showed original send dates varying anywhere from 3 days to 8 months ago.  (Yeah, messages that have been sitting on the server for 8 months, doing nothing, suddenly show up as "corrupt" today.  Huh???)

I'm at a complete loss to explain it.  Naturally, I thought of a virus, but our AVG Pro is active, updated and doesn't have anything significant in it's virus vault.

The only two things that these stored "corrupt" messages have in common is they all originated from our overseas (China/Taiwan) offices, and they all have attachments.  (.eml, .doc, .pdf, .jpg... the usual stuff.)  I've checked most of the messages that are supposedly corrupt and I've had no trouble opening the attachments, or reading the messages themselves.  I don't know if the asian characters have anything to do with it, but we deal with them in our email all the time, and they've never been a problem.

My main concern is the overnight transition from only one or two messages every so often that were sent very recently, to this sudden reporting of a large number of messages that have been stored on the server for several months.

I'd be very grateful if anyone has any clues on what I should look for.  Granted, I may discover that this is not necessarily a Backup Exec problem, but I need to start somewhere...

Thanks!

 - Darren B. -

[SBell]

Hi Darren,

I'm also getting this error.  The only messages affect are ones that have e-mails as attachments.  I only started getting this error since installing the latest MS patches and I'm assuming that it is related to KB931832 (security update for Exchange 2003).  Did you install any patches prior to getting the backup errors?

Thanks,

Stephen.

[Jon_Ziminsky]

I have been experiencing the same situation.

Everything went downhill after MS released their patches.

I sure hope Symantec will release a hotfix to address this issue.

JZ

[Darren_Berkey]

Hi guys,

Alot of the "corrupt" messages do have other messages attached to them, but there are also plenty of them that just have the typical file attachements.  (Usually .pdf and .xls).

As for patches, we run a WSUS server, so most of our machines receive automatic updates that way.  On the Exchange server, there was in fact a security patch added on Monday (5-14), but it was KB-931768.  Although it's different from the one you mention, it is curious to see that all my "corrupt" messages showed up in the job history the day after the security patch was applied.  Might be worth investigating though... thanks for the tip!

- Darren -

Message Edited by Darren Berkey on 05-16-200706:29 AM

[Jon_Ziminsky]

I am also receiving the "corrupt" messages. They are in addition to the ones i have mentioned in the other thread.

I am on hold for Tech Support at Symantec. I will post the response.

JZ

[Jon_Ziminsky]

I just got off the phone with tech support....

They tell me that since the issue is not affecting all of my mailboxes, it is my problem. They were very hesitant and non accepting of the patches breaking their software theory.

I told them that was NOT an acceptable resolution, and i want a call from a senior technician, or a tier 2 rep. I am currently waiting for their "Senior Tech" to call me back... I will post the results.

[Darren_Berkey]

I'm monitoring this thread so I'll be very interested in reading about your findings, JZ.

IMO, the fact that not all email boxes are affected is inconclusive and doesn't necessarily mean that Backup Exec is not at fault.  Bottom line is, I just want to know what's causing it and how to fix it, whether it's Backup Exec or not.  I'm still looking into the security updates and any possible connection there, but I haven't found anything worth noting yet.  I'm also kinda new to all of this, so I'm learning as I go.

 - Darren -

[SBell]

The more I look at this the more I think it is related to the MS patch(es).

• The messgaes that are being reported as corrupt have been on the mail server for a while and have been backed up successfully in the past. 
• The messages and their attachments are not corrupt and can be opened and read by the users.

The following MS updates where applied to the Exchange server:

• KB890830
• KB931768
• KB931832
• KB925876
• KB934268

The following updates where applied to the backup server:

• KB890830
• KB931768
• KB925876
• KB934268

If I get a chance I will check Microsofts website to see if any problems with these patches have been reported there.

[Jon_Ziminsky]

I agree. It is too coincidental for everything to be working and happy before patches, and it is not working after patches.

I just had the "Senior Tech" call me. He tended to agree and requested me to send him begather.exe data. I just sent that to him, and am awaiting a response.

I will keep you all updated.

JZ

[Aaron_O_Conner]

I didnt see this post before I posted my own, but I have the same problem just recently.  I dont notice that this on BE 11, just 10.

Interesed to know what caused this in the end.

Aaron

[Darren_Berkey]

I did some more checking on the updates and compared my installed patches with the list that Stephen provided from his servers, and only the following matches were made;

Updates on my Exchange server...

  KB931832

  KB931768  -  installed on 5/14

Updates on my backup server...

  KB931768  -  installed on 5/9

I'm tempted to remove KB931768 and restart the server... but it's in the middle of the business day, so it's not a good idea right now.

 - D -

[SBell]

I got the same errors for the same e-mails in last nights backup after installing SP3 for Backup Exec on the media server and pushing it out to the remote agents.

I have removed the KB931768 & KB931832 from the Exchange server.  I intended to only remove KB931832 but was advised that this may affect the performance of KB931768 so I removed them both.  I have just completed a backup of the Exchange mailboxes and it completed sucessfully.

[Darren_Berkey]

Ahh, so it seems that the Microsoft patches are the culprit, huh?  Somehow, I'm not surprised.  I just recently spent some time troubleshooting a problem where our client PC's were spontaneously rebooting when trying to send a fax through Reporting Services, which turned out to be caused by a recent security update.  (KB925902)

Anyway, I'll also remove those same updates from our Exhange server tonight and see if I get the same positive result on tomorrow's Backup Exec logs.  I'll post my results here.

Thanks guys!! 

 - D -

[Jon_Ziminsky]

I am glad that we have found out what is casuing the failures.... But is removing OS Security patches a viable solution? I for one am not comfortable unpatching my systems, especially security patches on my Exchange server.

I hope Symantec steps up to the plate and fixes this.

JZ

[Darren_Berkey]

From a network administrator viewpoint, no, we shouldn't have to remove OS patches to solve problems with applications.

Personally, my real-world experience has shown me that the vast majority of so-called "vulnerabilities" in Windows are over-hyped to begin with and for every hole they patch, there are 10 more waiting to be found anyway.

I'm going to remove the suspect updates to verify that they are causing problems, but I don't know if I'll leave them excluded indefinately.  There is also the other option of modifying the registry so that Backup Exec will not report job failure if it encounters corrupt files...

 - D -

[Jon_Ziminsky]

The "over-hype" is what concerns me. The ten more vulnerabilities that exist but haven't been found aren't "published" yet either, so the script kiddies aren't activley exploiting them. The ones that are released are fair game for them. It is the script kiddies i am trying to thwart, as the real hackers are going to get in whether you are patched or not. Not to mention we are talking about patches on a server that most likely has open ports facing outside. I would be alot less concerned if it was an internal server.

I am confident that the backups are actually backing data up, minus the messages that are being flagged as corrupt/access denied. I would much rather explain why i can't restore a handful of messages, rather then explain why i left known vulnerabilties open to correct a problem with a third party application.

I already flipped the "fail if corrupt flag" and my jobs are still failing.

[SBell]

I agree that removing the patches is not a fix for this problem.  I am going to re-apply the patches one at a time to see which one is causing the errors (although I am fairly sure it is going to be KB931832).  I am not going to be able to complete this unitl next week.

The onus is on Symantec to advise us on a fix for this.

I have also made the registry change, and my jobs are being reported as "Complete (with exceptions)"

[Brian7M3]

I am getting this exact same problem and it started right after I applied KB931832.  I have a case open with tech support and they told me I had a corrupt Exchange database and to run the eseutil.  I did this and it passed the integrity check but still failed the backup.  I followed up with an email to Symantec with a link to this thread.

Hopefully they will have a solution soon.

Brian.

[Jon_Ziminsky]

The more people that open issues regarding this problem, the more attention it will get. Hopefully if we get enough people with the same problem Symantec will patch their software. I spoke with someone at tech support yesterday that was taking the issue to the development people that specialize in Exchange, to try and get something going. He is supposed to call me back today. I have this feeling that they are going to tell me to contact Microsoft.

If you are experiencing this problem, please open an issue with Symantec.

JZ

[Darren_Berkey]

After some careful second thoughts, I decided not to remove the patches from our Exchange server for the reasons that JZ pointed out, plus the fact that I'd just end up putting them back on anyway.  For now, I'm just going through job log each morning to see just what's in it and make sure that it doesn't contain anything more dubious than the "corrupt" messages.

For the record, the same 74 messages that showed up in the log a couple days ago are still being reported as corrupt.  I guess the upside is that the number isn't increasing (knock on wood).

Thanks for sharing your info here, guys.  I'm going to try and open a case with Symantec support next week to see if we can indeed bring more attention to this problem, although I too believe that they will point us towards Microsoft for the final solution.

 - D -

Message Edited by Darren Berkey on 05-18-200702:08 PM