Strange "corrupt" file behavior in Exchange backup...

[Darren_Berkey]

Greetings to all...

I am a junior network admin at a small office with about 60 active mailboxes on an Exchange 2003 server.  Recently, the Backup Exec job history has been reporting corrupt email messages like this...

WARNING: "\\EXCH\Microsoft Exchange Mailboxes\Username [unique ID]Top of Information StoreInbox 〰...
This file cannot verify.

Now, I've done some research on this, and I know that there are several support pages that address the issue, but let me elaborate on this situation a little bit.

For the past month or so, every few days, the job history would show 1 or 2 of those corrupt/cannot verify errors.  In each case, the sender was always the same person and the messages always went to the same recipient here, but only when the message included an attachment. They were also usually recently sent... within a day or two of being reported as corrupt.  Typically, I would just delete the offending messages from the server.  (After making sure the recipient had a local copy, of course.)

Well, today was very different.

Today, the job history showed 74 corrupt messages like the one above that could not verify!!  This time, they were from several different senders and affected more than a dozen recipients.  Even more strange, the messages showed original send dates varying anywhere from 3 days to 8 months ago.  (Yeah, messages that have been sitting on the server for 8 months, doing nothing, suddenly show up as "corrupt" today.  Huh???)

I'm at a complete loss to explain it.  Naturally, I thought of a virus, but our AVG Pro is active, updated and doesn't have anything significant in it's virus vault.

The only two things that these stored "corrupt" messages have in common is they all originated from our overseas (China/Taiwan) offices, and they all have attachments.  (.eml, .doc, .pdf, .jpg... the usual stuff.)  I've checked most of the messages that are supposedly corrupt and I've had no trouble opening the attachments, or reading the messages themselves.  I don't know if the asian characters have anything to do with it, but we deal with them in our email all the time, and they've never been a problem.

My main concern is the overnight transition from only one or two messages every so often that were sent very recently, to this sudden reporting of a large number of messages that have been stored on the server for several months.

I'd be very grateful if anyone has any clues on what I should look for.  Granted, I may discover that this is not necessarily a Backup Exec problem, but I need to start somewhere...

Thanks!

 - Darren B. -

[Jon_Ziminsky]

Dean, thanks for the reply. I must ask though, did you read this entire thread?

We are all quite aware of how messages are marked as corrupt, as well as AV or any other application accessing data while being backed up.

This is an issue with Veritas and Microsoft. The patches were not fully tested, and are causing isues.

Here is the latest repsonse from Veritas:

Good morning Jon,

We are still researching this issue with Microsoft, and would like some additional information from you, if possible. The previous log files showed 49 emails as "access denied". Are those same messages still showing the same error? If so, can you check those specific messages, and see if there is any commonality between them, such as them all having attachments, or .html links, etc.?

Thanks,

[Jim_Schueckler]

We have seen the same problem here starting about a month ago.

Two apparently related problems:

1) The "corrupt file can't be backed up" message.

2) The XML log file will not display as XML unless I use notepad to modifiy the filename of the corrupt file within the XML log file.

The problem (for us) appears to be the result of a new spam technique that started about the same time. The spammers make the subject line of an email note in unicode. The unicode header defines the string as some language other than English but the string contains characters that *look* like English. A human reader visually sees the subject line as readable, but computer anaylsis of the string does not cause a match with the typical spam subject lines because the computer comparison uses the unicode language identifier.

It looks like attachments to spam notes had file names also in the scrambled unicode. So badly scrambled that the attachment filename, when converted to English, has some characters that are considered valid by some processes (e.g. Exchange) but *invalid* as filename characters for other processes (e.g. BackupExec).

There were a couple of cases where BackupExec reported which user's mailbox contained the corrupt file. In those cases, I was able to go to that person and have her/him delete the email note. The notes were always spam. That evening's backup no longer had a "corrupt file" error message.

However, for about three weeks now there is one such corrupt file at the Exchange system level, so I can't just delete it from a specific user's mail folder. I can't find the file to delete it.

Soooo. I still need help to find and delete the ugly file so BackupExec stops reporting it.

Thanks,

Jim

[GaryDarby]

HI Jim,

I am experiencing the exact issues you are on both problems listed and have not been able to locate the corrupt file. 

Have you had in luck in the last couple of days resolving the problem?

Thanks,

Gary

[BackupExec_Vete]

By running a backup of all mailboxes you will be able to identify the user and e-mail object that is corrupt. Once identified, the item is removed from their mailboxes and the backup of the information store completes now with no problem.

[Ken_Worley_2]

I maintain about 80 small Business servers and I have had this issue all the way back to version 9.0 to version 11D

I have been dealing with this issue for a while.. This is how I have handled the issue so far.

I create the IFSdrive ( Drive M Mapping ofr exchange you normally see in exchange 2000 )

I have created a user called MailMan.. He is God as far as exchange is concerned.

With it in Drive M mapping I can then search all the mail boxes for the e-mail concerned. I then delete them.. this allows me to get a good backup.

Not all of the e-mails have been spam on the servers we monitor.  I would say 95 % arte spam.

The e-mails are all viewable. I have even exported them to a pst file . then scaned the PST file for errors.

I have run all the tools on exchange to test for curruption and have found none.

I have used the built in Backup and it has no problem backing up the messages.

I have also tried ARCserv and it has no problem.. ( The interface sucks for ARCserv)

I have this occur on servers using Trend Client Server SMB  and Norton Corporate  10,  I have even removed the antivirus thinking it might be the issue.

[Matthew10011]

Hi,

Just want to chime in that we've been having the same two problems:

1) Corrupt files that cannot verify, and

2) Illegal XML characters when trying to view log.

At first, I was able to track down all the files that it was calling corrupt, and delete them.  Their mailbox was provided in the log file.  But now there's just one file:

WARNING: "\\SAI-SVR\Microsoft Exchange Mailboxes\ 䅁3F48F2FB20887C1045AD7C674F479B24DB8D78491D2DA9DD" is a corrupt file.
This file cannot verify.

How can I locate this file??

We do not have time/$ for me to sit on the phone with tech support, so I deeply thank those of you who have already done so, and I hope a resolution is posted very soon!!

Matthew

[SS_2]

One of the other replies suggested-

"Running a backup of all mailboxes you will be able to identify the user and e-mail object that is corrupt. Once identified, the item is removed from their mailboxes and the backup of the information store completes now with no problem."

I did this and was able to find the corrupt messages and remove them.

[Matthew10011]

Thanks.  We already do run backups of all mailboxes.   At first, there were a few corrupt files that were identifiable because of the mailbox they were in.  I was able to successfully deal with (delete) those.  But the one problem file that is left does not have a mailbox.  Here's the error:

WARNING: "\\SAI-SVR\Microsoft Exchange Mailboxes\ 3F48F2FB20887C1045AD7C674F479B24DB8D78491D2DA9DD" is a corrupt file.
This file cannot verify.

Also, I still get the "Illegal XML character" error when trying to view the Job Log.

*I just tried posting this, and was told that: "Your message has been changed because an invalid or malformed HTML tag or attribute was found. The invalid HTML has been removed..."  I'm guessing that the characters in the file name which are causing problems with BE are also causing problems with Symantec's forum.  In my log, it looks like two little boxes at the beginning of the filename (before the 3F48F...).

Thanks again,

Matthew

[Old_Timer]

Since 19 May 07 when KB931768 and KB931832 were installed, the following is reported every night:

Backup - \\Server Name\Microsoft Exchange Mailboxes 
WARNING: "\\Server Name\Microsoft Exchange Mailboxes\User Name [user.name].Top of Information Store.Inbox. Œ〰〰〰〰㜷攳散敦户㐳攰㤴搹㘷ㅣ㜳㜶〴昳㜰㜰〰㈳㜲㠱㜹攸㡤㜴ㄴぢㄱ㈹愸戶摤㤴晥〰〰〰㐰挴攴〰〰攸攰挵ち㜵摥㌴㤴㝡摢㙡扡㡢㜲搶㐹〰〰〰昲っ㉣〰〰Rock'n'Reel/Festival Issue (3)" is a corrupt file.

This file cannot verify.

This email does not exist anywhere in the Information Store that is visible.

It looks like an example of what others have been reporting here for a long time.

Is there a fix on the way?

My attempt to open an “email case”, as I used to do with Veritas, resulted in:

"E-mail Support and Support Forums

Please note that since January 13, 2005 Symantec Enterprise Technical Support has migrated select support service requests from e-mail support to our moderated Support Forums.

We believe that this will provide a much richer experience, allowing access to the knowledge and experience of other customers as well as our Support staff.

VERITAS Backup Exec for Windows Servers version 11d (11.0) no longer offers e-mail support. For all electronic support for this product visit our moderated Support Forums."

[Joho]

We have the same problem, which started on May 25 after the following Microsoft Updates were installed.

KB931784
KB929123
KB932168
KB933566
KB935839
KB935840
KB935966
KB927891
KB925902
KB930178

Can someone tell me which update is causing the problem, so we can remove it, or if there is a Symantec update to fix this?

Allan

[Joho]

Sorry, I meant to say June 25 was when we did the Microsoft Updates.

[cypherx]

Last week our company was also experiencing difficulty saving exchange mailboxes.  I would get random Access Denied and corrupt file --cannot verify errors on full mailbox backups.  Were using Backup Exec 10d and Exchange 2003 SP2 for e-mail.  This occurred Tuesday night onward.  What did I do during the day on Tuesday?  Windows updates.  After doing some reading in this thread I decided to uninstall Microsoft KB931832 update for exchange SP2 on Friday.  Since then all backups this weekend have been successful.

I've tried everything prior.  Access denied errors were really misleading.  So much that I created a new backup operator account, pushed the exchange backup agent to the mail server all over again, restarted servers, etc.  I was at wits end.  I cannot believe that a Microsoft security patch caused all of this frustration.  E-mail backups are very important to us.  Believe me they've come in handy if a user accidentally deletes something important in their inbox.  I think the solution, uninstalling a security patch is not necessarily a great idea... but everything is working fine without this one patch.  Luckily I write down the Microsoft KB #'s of the updates I download before I install them. 

Anyone still having issues, go in your add/remove programs and make sure show updates is checked.  Under Microsoft Exchange look and see if you have KB931832 in there.  Uninstall it.  You won't be required to reboot the Exchange server but the uninstall process will take down the services for 5-10 minutes while it performs its job.  During that time users will not be able to access their e-mail.  Once it is complete all services will start automatically and the brief outage will be over.

I hope Symantec and Microsoft can work out a fix for this issue.  I can tell you first hand in my trials that this problem was without a doubt caused by MS KB931832.  Details to this MS KB can be found here: http://support.microsoft.com/kb/931832

Hope this helps users out there with this same issue.  I also hope this helps Symantec isolate the cause of this issue for future tech support calls.  Please work with Microsoft on a resolution to this.

Sincerely,
Keith
Systems Administrator

[BackupExec_Vete]

This is just a quick question.

What is the total size of the mailbox stores for your location?
How many users?
How long does it take to perform nightly backups?

[cypherx]

---

@Backupexec Veteran wrote:

This is just a quick question.

What is the total size of the mailbox stores for your location?
How many users?
How long does it take to perform nightly backups?

---

Straight from last nights successful backup:

Backed up 152973 mail messages in 2911 folders in 82 mailbox(es)
Processed 23,195,225,440 bytes in  2 hours,  23 minutes, and  23 seconds.
Throughput rate: 154 MB/min

[BackupExec_Vete]

This will probably shock everyone. I changed jobs last year and the size of mailbox stores at current employer is 170 GB for 443 users. Backups start at 5:30 pm and do not complete until around 9:00 am each day.

Just curious for all who respond. What is the total size of all of your databases?

[cypherx]

Do you mean this?

[BackupExec_Vete]

No, I meant the size of your Exchange Mailbox Stores.

[cypherx]

How about this information?

First Public Store
Backed up 2 Exchange Server stores
Backed up 1 Exchange Server logs
Processed 15,767,610,158 bytes in  24 minutes and  58 seconds.
Throughput rate: 602 MB/min

Exchange Public Folders
Backed up 1059 mail messages in 5 folders in 0 mailbox(es)
Processed 10,830,581 bytes in  23 seconds.
Throughput rate: 26.9 MB/min

Microsoft Exchange Mailboxes
Backed up 152973 mail messages in 2911 folders in 82 mailbox(es)
Processed 23,195,225,440 bytes in  2 hours,  23 minutes, and  23 seconds.
Throughput rate: 154 MB/min

[wombatclov]

Thank you for this post, it is interesting.   I have not yet tried this- has anyone else tried it and got the same result?  I did start the un-install, but it warns me about other programs that may not function.  Did you just ignore that?  Any ramificatons?  Thanks.

[J_O_A_T_]

I have had the same problems as many of the users in this thread.  It started with several messages traceable to a specific message in a specific mailbox that I could remove or tell the software not to backup. After several weeks this error began appearing:

"\\MicrosoftExchangeMailboxes\about50randomcharactersthatarenotfoundbysearchingthemailboxes" is a corrupt file.

This file cannot verify.

(I removed spaces so the above message would display on the forum better.)

The problem started at the beginning of June.  I only started to suspect it was a bigger problem about 2 weeks ago when the above error began appearing.  I verified that the backups were complete except for the one "corrupt file" that I couldn't find and postponed contacting support until I had more time.

Is there any news on a fix from either Microsoft or Symantec?  It's hard to believe that this has been a problem since the middle of May and there has been no solution provided by either company.

David Simon