Queue R2 is clogged up and restore jobs are pending

[ricp]

Hi,

We are running EV 8 SP3 on a Centera setup and have been getting calls today regarding restore items being stuck in rthe pending state in Outlook. I've checked the queues and the R2 restore queue has 165 messages stuck in it. We are planning a reboot tonight to try and resolve the issue, but is there any way to get this queue processed without a restart? Apparently, EV services have all been restarted but has had no effect.

Thanks,

Ric

[JesusWept3 · Accepted solution]

just as a matter of interest, how big is the MSMQ storage location?
Personally i've seen this when a user has been moved from one exchange server to another but not synchronized.

So lets say i'm on ExchangeServerA and all my requests go to the ExchangeServerA Task, I then get moved to ExchangeServerB, but for whatever reason, my mailbox hasn't been synchronized.

I then do a restore, and rather than the request going to the exchangeServerB Restore queue, it goes to ExchangeServerA..... the task then picks up that restore request, and then says hey the mailbox doesn't exist, but rather than discard it, it keeps retrying the same message over and over and over

So expand out the queue, have a look at the top couple of restore requests, and see if that user belongs to that exchange server or not.

If they have moved, you are going to either have purge the queue, or download something like QueueExplorer and delete the users individual R2 messages.

I'm  also assuming the vault store is not in backup mode or doesnt have the EnableRestore registry key set to 0

[ricp]

Funny you should mention that. The queue points to the exchange cluster name and that cluster failed over last Friday to the secondary exchange server, which may have caused the R2 queue to jam up.

Can I just purge this R2 queue then and tell end users to resubmit their restore mail jobs?

[ricp]

MSMQ location has 376GB free on a 1.37TB partition. 

[JesusWept3]

you can purge it, its probably the quickest way and restart the task.
However anyone doing a restore will have to do a Cancel on the item and then attempt to do a restore again.

[ricp]

In the default exchange policy there is a pending short cut timeout option, but this seems just for archiving items. Is this the case? Is there an option for restore pending timeout?

Thanks for the above by the way

[AndrewB]

you also might want to check what your msmq size limit is set to. the default used to be 1gb which is not enough. now it's increased to 8gb as the standard

[ricp]

purged the queue and it is now empty. Tested restoring email and it remains in restore pending. Hopefully a reboot will sort this out. Anybody any further ideas? Appreciate everything so far. The MSMQ is set to 8GB.

[JesusWept3]

get a dtrace of the the RetrievalTask and StorageRestore if you can?