Unable to restore archived email items from EV8

[gparker]

My customer is running EV8 with Exchange 2007 for a couple of years now without a problem. But a couple of days ago the users started complaining that they were unable to restore archived items via the restore feature on the Outlook add-in. They the shortcut icon changes to a pending icon and that's all that happens - the item doesn't get restored. Double-clicking shortcutted items retrieves the item ok, but restore doesn't want to work.

They've re-booted the EV server as well as the SQL server hosting the EV database without luck. The EV Console shows no issues. The Windows event logs seem clear.

The problem is system wide, it's not just affecting a select number of users/mailboxes. I went to the properties of the archives and lokked for any index issues, but it reports the indexes are ok.

The customer swears that they haven't made any changes on their network or upgraded their Exchange server.

The issue is also happenning to their OWA users, not just to their internal Outlook users.

Any help?

[Mohawk_Marvin]

I would:

Run the deployment scanner, you never know your luck it might be something obvious that you overlooked

Check the MSMQ's

Check Event log warnings as well as errors

Change the system mailbox to a new mailbox (for testing purposes)

Failiing all the above, run dtraces 

[Nick_White]

Can you check to see if the Vault Store(s) are in backup mode? It might be that the restore process is still in offline mode which would definitely stop restores from working

[gparker]

Hi Nick,

You might have hit the nail on the head. I recall the customer telling me that the issue started hapenning soon after he performed a backup. Perhaps his backup software set backup mode but didn't clear backup mode afterwards. I will check now.

Regards,

George.

[gparker]

Hi,

Backup Mode on the vault store was already clear, but I got the customer to "set backup mode" and "clear backup mode", but this didn't solve the problem.

I looked at the MSMQ's and in the Private Queue section where all the EV queues are located, there are 2 "EV exchange mailbox task" queues with messages that seem to be stuck. One of the queues has 3960 messages and the other one which is directly beneath it has 505 messages. All other queues have 0 messages. I bounced the MSMQ service (which also restarted the ev task and storage services) but the numbers haven't changed in either of these queues.

In the EV Event log, there are a number of "event 2216, Message dispenser will suspend processing for 5 minutes due to a recoverable error. Task: Exchange Mailbox Archiving Task for <ev hostname> Retrieval, Task Category: Retrieval Task". There's also a number of "event 6190, Error allocating memory at Ref: .\SavesetMessage.cpp Line 445, Task Category: Retrieval Task".

Any clues to whether these events are related to the problem?

I haven't yet run the Deployment Scanner as yet, nor have I changed the system mailbox.

[gparker]

Hi,

Backup Mode on the vault store was already clear, but I got the customer to "set backup mode" and "clear backup mode", but this didn't solve the problem.

I looked at the MSMQ's and in the Private Queue section where all the EV queues are located, there are 2 "EV exchange mailbox task" queues with messages that seem to be stuck. One of the queues has 3960 messages and the other one which is directly beneath it has 505 messages. All other queues have 0 messages. I bounced the MSMQ service (which also restarted the ev task and storage services) but the numbers haven't changed in either of these queues.

In the EV Event log, there are a number of "event 2216, Message dispenser will suspend processing for 5 minutes due to a recoverable error. Task: Exchange Mailbox Archiving Task for <ev hostname> Retrieval, Task Category: Retrieval Task". There's also a number of "event 6190, Error allocating memory at Ref: .\SavesetMessage.cpp Line 445, Task Category: Retrieval Task".

Any clues to whether these events are related to the problem?

I haven't yet run the Deployment Scanner as yet, nor have I changed the system mailbox.

[gparker · Accepted solution]

Hi again,

The 2 MSMQ private queues that contain stuck messages are "R1" and "R2" queues, which are the retrieve queues.

I ran Deployment Scanner, it reports everything is AOK. The EV system mailbox is intact, it hasn't been disabled and it hasn't been moved/relocated.

I found technote http://www.symantec.com/docs/TECH77506 which is almost identical to the problem I'm facing. It says to destroy and re-create/re-install  the MSMQ. But it says to make sure that the private queues are empty, which in my case they're not. What are the ramifications of re-installing MSMQ and re-creating fresh queues? Will users still be able to search their archives? Will the indexes need rebuilding?

[Nick_White]

If you purge the MSMQ queues then you won't lose any data. The worst case scenario is that any items which were in the archiving queues may remain in the pending state in the mailbox depending on whether we can reset them back to their original state or not. If not, there is a policy setting which will force pending items to be returned to their original state after a configurable number of days

[JesusWept3]

have actually seen this a number of times when users get moved from one exchange server to another and haven't been provisioned/synced in EV
So they are on Exchange A, they get moved to Exchange B, but EV still has records for Exchange A.
User then attempts to restore something and it goes to Exchange A Archiving Task MSMQ
EV then attempts to connect to the mailbox on Exchange A, doesn't see it, puts the task to sleep and continually loops over the mailbox attempting to connect and putting the task to sleep.

[gparker]

Thanks for your responses.

JesusWept2 (love your handle BTW!), in my customer's case they have not moved from one Exchange server to another. AFAIK, the Exchange server has been the same for the past 2 years and they swear to me that the EV system mailbox hasn't moved from its mailbox store location.

Nick, I found a Symantec Tech article describing how to purge the MSMQ queues, but it states in bold writing -  "NOTE: The following queues should never be purged without consulting with technical support: A1, A6, J1, P1, R1, Storage Archive, and Storage Spool". http://www.symantec.com/docs/TECH48896 
 

As you can see, it mentions R1 in that note which makes me nervous...Should I be nervous?

George.

[JesusWept3]

Nick pointed out what the down side is and that is you will have an item left in restore pending, meaning those people who are waiting for their items to restore will continue to wait with a shortcut set to pending You can either set the shortcut pending timeout in your policy or give users the cancel button via the policy and any pending items they can manually select and press cancel item and restore it again The other alternative is to use msmq explorer and you can selectively choose which items to delete from the queues rather than the all or northing that msmq natively provides

[JesusWept3]

http://www.cogin.com/mq/

[gparker]

Problem solved :)

The messages became stuck in the R1 and R2 queues due to the fact that a user who had permissions to another users mailbox - who has left the company - tried restoring an archived item from the other users mailbox, but the user didn't have sufficient EV permissions on the mailbox in question. This resulted in the "R queues" becoming stuck!

We purged the MSMQ private queues and bounced the EV services and it's all working again. We didn't even need to worry about the downside stuff as it seems as though everyone's pending icon's have magically changed back to the shortcut state.

Regards,

George.