Enterprise Vault and Exchange 2007/2010

[ESI_EVAULT]

Hello,

I have a customer in the process of migrating from Exchange 2007 to Exchange 2010.  I have all mailbox servers set up as targets on EV.  In addition, I have 2 EV servers. 

Currently, the Exchange 2010 servers are being archived successfully.  However, I'm not so sure about the Exchange 2007 server.  The MSMQs related to the archiving task for the Exchange 2007 server have messages piling up, and they're not moving; ie they appear to be stuck.  They've been in the same state since about 7 pm last night (sunday, 4/7/2013).  The queues that have messages backing up are:  a1, a2, a3, a5, a6, a7, and the admin queue.  Since the a3 and a5 queues have messages backed up, I'm reluctant to just clear the queues and try again.  All mailboxes on the Exchange 2007 server have been provisioned and have successfully archived prior to the addition of Exchange 2010.

A little more background: 

On one EV server, I have the the archiving task for the Exchange 2010 servers, along with Outlook 2010 for MAPI compatibiltiy, and an EV system account

On the other EV server, I have the archving task for the Exchange 2007 server, along with Outlook 2003 for MAPI compatibility and a different EV system account.  This task was previously on the same server as the Outlook 2010 mailbox archiving tasks, but it wasn't working.  I separated it out to another EV server for a more granular approach plus I was concerned about MAPI compatibility for Exchange 2007/2010 with EV, so with this configuration I can have the 2 different versions of Outlook. 

The bottom line, though, is how do I get the previously mentioned queues to empty, and ensure EV is processing properly for my Exchange 2007 mailbox server?

Thanks all

[TonySterling]

So first off, what events do you see in the EV application log for the Exchange 2007 server?

If you restart the EV tasks it should generate some warnings or errors. 

Also, make sure the task are not in report mode.

[ESI_EVAULT]

Tony,

Thanks for your quick response. 

The archiving tasks are definitely not in report mode; they're running live. 

Unfortunatley, I didn't record the events in the event viewer (I knew I forgot something) and I won't be able to access the EV servers until later today/this evening.  Once I do, I will restart the task, start another archiving run, and record the events in the EV event log.

[JesusWept3]

Uninstall outlook 2010 and install 2007 because its not supported specifically due to performance reasons

[Old_Man_Jensen]

One thing to note here... the A3 and A5 queues are not that much of a concern.  Those queues are used to keep track of the mailboxes that are remaining to be archived - when one mailbox is processed, the queue drops by one - so purging those queues doesn't result in any data loss.  You simply wind up starting from the top again the next time a scheduled or one-time run is kicked off.  A7 is similar to this for synch requests, so again, no danger in purging.

Not that I'm recommending purging, mind you... Tony's correct in that you need to figure out what the event logs are saying before proceeding.  If there's issues in the queues, however, it's typically the A1, A2, or maybe even the A6 queues that hold the issue (in the case of corruption clogging up a queue).  More than likely, however, the logs will come up with something like storage problems, or rights issues, and once those are fixed you'll see the A1 and A2 queues immediately drain down to nothing.

It'll be interesting to see what the EV and App Event Logs turn up.

[TonySterling]

Duh!  I totally read past the fact OL 2010 was installed on the EV Server.

[Old_Man_Jensen]

Unfortunately, the server with Outlook 2010 on it isn't the one having issues.

While JW is correct that you should replace Outlook 2010 with 2007 on that server archiving Exchange 2010, the one you're having issues with (which is running Outlook 2003) should be okay as far as Outlook versions go.

One thing you might want to be sure of, however... with Outlook 2003 (Assuming SP3), you need the following hotfix:

 Requires Microsoft hotfix http://support.microsoft.com/kb/948073. To use this hotfix, you need to set the
registry entries mentioned in the Microsoft Knowledge Base article http://support.microsoft.com/kb/948074.
The recommended registry value is 0xFFFFFFFF.

That has something similar for Outlook 2007 as well.  As per this one:

Set the registry entries described inthe Microsoft Knowledge Base article
http://support.microsoft.com/kb/952295. The recommended registry value is 0xFFFFFFFF.

Good luck!

[JesusWept3]

Whole thjng doesn't make much sense, install outlook 2007 on the entire environment, use the same evadmin for each, set exchangeGCOverride where appropriate

[ESI_EVAULT]

Thanks for the info on Outlook 2007/2010; I will take the steps to get Outlook 2007 configured on both EV servers.  Exchange 2007 is in the process of being migrated over to Exchange 2010.  The Exchange 2010 servers appear to be archiving properly, although there are a limited number of users on that version. 

I checked in on the archving task and event log messages for exch01 and found the following:

event ID 8390 on the archive task
The EnterpriseVault.DirectoryConnection object reported an error.
 
An attempt was made to move an ArchiveFolder beneath itself (hierarchy inversion)

event ID 8391 on the directory service

The EnterpriseVault.DirectoryService object reported an error.
 
An attempt was made to move an ArchiveFolder beneath itself (hierarchy inversion)

these events occur on ev01 which is archiving the exchange 2007 server.  Also please note that this is the EV server with Outlook 2003 installed. 

The good news is that the MSMQs clear out (as expected) on the ev01 server.

However, I still have a build up of MSMQ messages on the ev02 server which is where this task originally ran.  (I moved it to the other EV server Sunday).  These messages are in the same queues mentioned in my original post.  Given that the task has moved to another server, can I just clear out those MSMQs?  Or, if I just leave them as they are, is it hurting anything?

The bottom line, I have the 2 error messages listed above that are repeated listed in the event viewer, and I have to figure out the best way to handle the buildup of MSMQ messages on the EV server that previous hosted this one archiving task (even if the best thing to do is to do nothing, I need to make sure0

Appreciate the help and comments. 

[TonySterling]

So the MSMQ on EV02 is for a task that is no longer on that EV server?  If yes, those will never clear as the task is not going to process them if it is on EV01.

The errors you are seeing:

event ID 8390 on the archive task
The EnterpriseVault.DirectoryConnection object reported an error.
 
An attempt was made to move an ArchiveFolder beneath itself (hierarchy inversion)

event ID 8391 on the directory service

The EnterpriseVault.DirectoryService object reported an error.

An attempt was made to move an ArchiveFolder beneath itself (hierarchy inversion)

You could check this TN:

Event 8390 - hierarchy inversion - 0cX0041BF0

Article:TECH162185

|

Created: 2011-06-13

|

Updated: 2013-03-14

|

Article URL http://www.symantec.com/docs/TECH162185

[Old_Man_Jensen]

Tony's right;  those queues should've been cleared before you moved the task to another server.  At this point, those messages aren't going anywhere.

The good news is that it's doubtful you have any data loss.  It's far more likely that there's some user mailboxes that have items now stuck in pending mode, since they never got the message back from the A1 queue to convert the items to full shortcuts.  That, and the order of moving archived items in the folders got all skewed.

I think the best steps you can take now would be the following:

• In the appropriate Mailbox Archiving policy, go to the advanced properties and set the 'Pending shortcut timeout' to '0'.  Run the MBA task you moved to the other EV server in Report mode (Run Once).  This'll clear out items stuck in pending, so be sure to do this *after* your EV backups to be sure that only the items stuck in pending are actually pending.
• Perform the 'Zap Mailbox' method listed in the article Tony referenced.  Using the PST export method would be a total last resort, as that one will flatten the archive and get rid of all folder references.
• Once you no longer have errors and are sure everything is working as it should, then you can go ahead and delete the queues on EV02 that relate to the original task.  The messages in those queues are orphaned, and EV will never use those queues again, so may as well just get rid of them rather than simply purging.  Just make absolutely sure that you're *only* deleting/purging the orphaned queues!

Best of luck!

[ESI_EVAULT]

Thanks everyone for the directions.  I'm dedicated elsewhere during the day, but I'll work those instructions tonight and post the results or any difficulties/questions I may have. 

[ESI_EVAULT]

I'm still having issues.  In the DTrace log, I have a number of lines that say:

***** Trace buffer overflow. Trace entries lost: 2583906 to 2584407 *****

Does that mean I'm not capturing some events?  Because when I look in Article:TECH162185 and search for the referenced information, it is not in m DTrace log, even though the DTrace was running during the archive task.  I have it set for verbose and to capture events.   However, when I search on the events (Event ID 8390 and 8391 in the Event Log) they're not found.  And, as previously mentioned, when I search on either

CFolderHelper::UAFR() - Error updating record <folderID>, error = 0xC0041BF0|Internal:CFolderAgent::UpdateArchiveFolderRecord() ..\AgentsCommon\FolderHelper.cpp [lines {2371,2402}], built Feb  9 09:23:12 2011

or

CFolderHelper::GetFolderSettings - HRXEX fn trace : Error 0xc0041bf0, ..\AgentsCommon\FolderHelper.cpp [lines (15){...,437,440,779,780,793,794,795,796,797,798,799,897}], built Feb  9 09:23:12 2011

nothing is found.  I perform my search only using the first part (ie CFolderHelper::UAFR or CFolderHelper::GetFolderSettings). 

I have installed Outlook 2003 to this EV server, and applied the previously mentioned hotfix and registry modifications.  The issues occur when I attempt an archive on an Exchange 2007 server in a mixed Exchange 2007/2010 environment. 

Yet the events still show up in the event viewer.  I'm assuming once I'm able to capture everything, I'll be able to see the specific mailboxes that are having the issue and work with those  Is that correct?

What am I doing wrong with this?  Should I not capture everything or not have it in verbose mode?  Is the Trace Buffer Overflow causing me to miss these events?

Thanks all

[ESI_EVAULT]

Also, I was capturing events for the archiving task only; (as the referenced document said). The 8391 errors are referring to the directory service. Should I be capturing that service also?

[Rob_Wilcox1 · Accepted solution]

I would personally recommend contacting Symantec Support, if this is still an ongoing issue.  If it's a lab environment, then I'd suggest starting over.