Strange "corrupt" file behavior in Exchange backup...

[Darren_Berkey]

Greetings to all...

I am a junior network admin at a small office with about 60 active mailboxes on an Exchange 2003 server.  Recently, the Backup Exec job history has been reporting corrupt email messages like this...

WARNING: "\\EXCH\Microsoft Exchange Mailboxes\Username [unique ID]Top of Information StoreInbox 〰...
This file cannot verify.

Now, I've done some research on this, and I know that there are several support pages that address the issue, but let me elaborate on this situation a little bit.

For the past month or so, every few days, the job history would show 1 or 2 of those corrupt/cannot verify errors.  In each case, the sender was always the same person and the messages always went to the same recipient here, but only when the message included an attachment. They were also usually recently sent... within a day or two of being reported as corrupt.  Typically, I would just delete the offending messages from the server.  (After making sure the recipient had a local copy, of course.)

Well, today was very different.

Today, the job history showed 74 corrupt messages like the one above that could not verify!!  This time, they were from several different senders and affected more than a dozen recipients.  Even more strange, the messages showed original send dates varying anywhere from 3 days to 8 months ago.  (Yeah, messages that have been sitting on the server for 8 months, doing nothing, suddenly show up as "corrupt" today.  Huh???)

I'm at a complete loss to explain it.  Naturally, I thought of a virus, but our AVG Pro is active, updated and doesn't have anything significant in it's virus vault.

The only two things that these stored "corrupt" messages have in common is they all originated from our overseas (China/Taiwan) offices, and they all have attachments.  (.eml, .doc, .pdf, .jpg... the usual stuff.)  I've checked most of the messages that are supposedly corrupt and I've had no trouble opening the attachments, or reading the messages themselves.  I don't know if the asian characters have anything to do with it, but we deal with them in our email all the time, and they've never been a problem.

My main concern is the overnight transition from only one or two messages every so often that were sent very recently, to this sudden reporting of a large number of messages that have been stored on the server for several months.

I'd be very grateful if anyone has any clues on what I should look for.  Granted, I may discover that this is not necessarily a Backup Exec problem, but I need to start somewhere...

Thanks!

 - Darren B. -

[OptimusPrime]

Moved Per User's Company Policy

Message Edited by John_B on 09-13-2007 12:43 PM

[J_O_A_T_]

Hi All,

I disabled the "Single Instance Backup" option and this weekend's full backups worked with no problem.  Now that I think about it the problems started about a month after I enabled them.

Other than this problem I haven't had any problems with a properly setup Backup Exec installation.  Don't use "Single Instance Backup!"

David

[Ben_L_]

Hey guys,

I was talking with some of our engineers today about KB931832 and what they told me was it is something that they are aware of and looking at.  But they could not say if patch or hotfix was scheduled to be released to address the issue.

I wish I could give you guys more information on it, but that is all the information I could get.

Regards,

[cypherx]

Hi Ben. 

It's about time a Symantec employee shows up on this forum.

Please give a go at my C++ errors that I get randomly.  It's like no one looked over the C++ code syntax before shipping out Backup Exec.  Thread is here:
https://forums.symantec.com/syment/board/message?board.id=103&message.id=57009&jump=true#M57009

Needless to say I am very frustrated and would definitely go with another backup solution in the future.  As long as it doesn't have Symantec's name on it, we should be fine.  That is unless you Symantec employees can prove me wrong...  Here's your chance to keep me as a lifetime customer.

[Matthew10011]

Hi All,

Just posting back, I followed the lead of.. Enrique Rodriguez.  I disabled "single instance backup" for exchange.  The very next backup log showed me exactly whose mailboxes had the corrupt emails (previous logs were not specifying this).  I deleted those emails, the backup succeeded (first success since before June).  For tonight's backup, I'm re-enabling single instance backup.

Matthew

[Andrew_Lomakin]

Seen this problem multiple times now.
Freaks out customers as hell.
I wonder why so few people have complained so far.
The scope ot the bug is just immense....
Anyways there's now a symantec KB out there:
http://seer.entsupport.symantec.com/docs/255047.htm
It describes exactly what we're all seing here,
but the advices they give are more fit for a joke website,
because customers with databases of 100+GB
(wink to the admin that has 60 GB of exchange emails)
don't want to hear about `manually deleting emails`.
I'm going to give symantec a call and see if we can put a little bit pressure on them.

Message Edited by Andrew Lomakin on 08-31-2007 05:04 AM

[Thorben]

Did you get a reply for your call? On customer side i have the next one with that problem.

I really don´t want to search for each mail to delete them manually.

[WebCrunchy]

I finally had the chance this past weekend to apply SP3 and Hotfix48 to our Backup Exec 10.1d. All went smoothly!

In addition, I made a new backup job with reference to our normal Daily Full Backup. The only real main exception was that I chose NOT to perform brick-level backups of ALL our mailboxes as the Exchange's Information Stores were already being backed up. However, I did choose brick-level backups of our top 3 key personnel. The result was a 100% successful backup of quite a considerable GB's of data!

So far, all tests have been 100% good.

My next test will run tonight: I will be allowing the old normal Daily Full Backup to run which will also brick-level backup almost all of our mailboxes... i.e. using the same backup that caused the errors as previously discussed in this forum and which got me involved in this forum.

I'll post my results in the next few days, and I hope that it will be of some benefit to those who are still having problems.

ABR

[WebCrunchy]

Well, my findings are thus:

As previously reported, performing a full backup of data, etc., and on the Exchange 2003 to ONLY backup the Information Stores and with brick-level backups of only our top 3 key personnel results in a 100% successful backup.

However, when returning to the the old normal Daily Full Backup job, and as suspected, it failed!

In fact, and as before, there were so many errors that the XML could not load and the job was viewable only in text!!!

Seeing that my new backup job was successful, I then took the next step: I added to include in the new backup job an additional 7 personnel's individual mailboxes. Of the total 10, 4 reported the same-old "access denied" errors.

However, because of the "small" amount of errors (that would have otherwise been rather large had I chosen all the individual mailboxes), the job is viewable with XML...

I have been researching page after page on the web and I still find no solution: everything appears in order as per the "solutions" thus so far found, and yet the tests I have subsequently been performing during the day have resulted in failure after failure (I have created a separate "Test" backup job and selected to brick-level backup only 2 of the 4 failed mailboxes to perform these tests).

So I am now looking rather favourably at backing up only the Information Stores (as previously mentioned by someone else in this thread).

However, I would be most interested to know how to restore a user's individual email from the Information Store.

Any ideas anyone?

ABR

[Cameron_Watt]

Essentially I wanted to add a "+1" or "me too" to this thread. We have been experiencing the problem where the nightly backup job is failing due to the following error:

{{{
WARNING: "\\STASBS-SERVER\Microsoft Exchange Mailboxes\user name [user.logon]Top of Information StoreInbox 〰〰〰〰扡扣㌰㤱ㄲ攳㐶ㄴ㑡㜹ぢ㙤〵㝥㐸㡡㜰〰搴㑤晢㙥㈹㕦摤〴㐹㐳敥㉣㔳㑣摡ぢ〰〰〰〰戴㔹〰〰搴㑤晢㙥㈹㕦摤〴㐹㐳敥㉣㔳㑣摡ぢ〰〰〰搰搵搸〰〰Weekly report." is a corrupt file.

This file cannot verify.
}}}

 I am shocked at the lack of action from within Symantec on this matter and the current workaround which seems to be: To successfully backup the dataset first delete the data.

Pull your finger out Symantec!!! Does the problem exist in the latest version of Backup Exec? If not, just offer an upgrade to affected users if you don't want to resource a proper solution.

[bboerst]

Just adding another "me too"  to this thread.  Our nightly incremental and weekly full Exchange backups are failing and are littered with lots of these:

WARNING: "\\XXXXX\Microsoft Exchange Mailboxes\XXXXXXXX [35342ecb-d2cb5643-8234523c-4fbd49]Top of Information StoreDeleted Items 〰〰〰〰收㈴ㄷㄳ搷〱㤴〴㥢ㄳ㜸捦攴㡣㍢㔰㜰〰㐶㙥㡤〴敤愳㙢戴㌸〱㍦㌱㠶㉤㘷㝡〰〰〰㐵㡦敡〰〰〸攰っ㝦ㄳ戸㈷㤴㍢㘳攲〰㘲㝣攸晣〰〰〰㔰㈶㈸〰〰Delivery Status Notification (Failure)" is a corrupt file.
This file cannot verify.
WARNING: "\\XXXXX\Microsoft Exchange Mailboxes\XXXXXXXXX [XXXXX]Top of Information StoreDeleted Items 〰〰〰〰〸〲晦㍤㜴挰敦搴㝢ぢ扤挹昱敡ち㈵㜰〰晡㔷㌱㡦㡥晢挵搴㙡㑡㕢〳㜶昸晥㐷〰〰〰ㄱ㤷㡢〰〰㜶〵搱㥥昲戳捣攴㕢㝤㌰晡㐹㝢㌲㘲〰〰〰搰㘱㕤〰〰Cancelled: Sue's - Staff Meeting" is a corrupt file.
This file cannot verify.

 
We've got a lot of mail and to individually go in and delete each of these items just isn't a viable solution...

Message Edited by bboerst on 09-24-2007 01:55 PM

[Ken_Putnam]

However, I would be most interested to know how to restore a user's individual email from the Information Store.  

You have two choices. 

1)  Setup a standby server and restore the entire IS to that, then EXMerge the mailbox/item

2)  Setup a Recovery storage Group to receive the restore on your production EMail server, then EXMerge from that

[farrel]

I just upgraded to 11D and also have the same issues.

1. Illigal XML Charactors

2. Log file went from 37k to 280MB. I can't open it because it is so filled with corrupt item errors

Is there a fix yet?

Message Edited by farrel on 11-14-2007 01:00 PM

[vacavfan]

This appears to be a temporary solution that allows the file to be backed up and the corruption to be ignored by Verify. I hope this helps

http://seer.support.veritas.com/docs/193940.htm

[Mark_Matthews]

Hi Vacavfan,

I think most of know about this 'fix' and a lot of us have tried it and it doesnt work, but notice that at the top it says "Excluding MS Exchange files"!

Just another "Me too" to add to this thread.

I look after many customers backups and have seen this issue a lot over the last couple of years over different versions of BE.

I have been lucky enough to only have one or two messages reported which I have been able to find and delete.

But yes it does freak the customers out a bit!

Hopefully there will be a proper fix along soon....

[dproshek]

Eight months later...and still no solution.  Unbelievable.

[sluke]

I'm getting the same error.

Completed status: Failed
Final error: 0xe0008488 - Access is denied.
Final error category: Security Errors

Backup Exec 10d with SP4.

Any help would be greatly appreciated.

[cwdbob]

I am also having this problem but it just recently started, I do find that if I remove all prior backup files and start new it works fine for a while then this little gem of a problem comes back.

Has anyone solved this yet?

[Nico_Van_de_Ven]

All these posts and still no solution... well try this one;

get in to your registry and

1. explorer to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\EXIFS\Parameters
2. Then add a Dword with "DriveLetter" and Value "M"
3. restart the store in which the corrupt files are located
4. Go to explorer
5. op the M drive
6. brouwse to the corrupt files
7. Shift+delete them

Next day - backup succesfull...

 I'm not responcible for any consequenses of these actions, I'm mearly stating a work around.

Business reveniew could be affected by this action !!!

[Nobby123]

Anyone come up with a supported fix yet please?