cancel
Showing results for 
Search instead for 
Did you mean: 

Build up of items in Enterprise Vault Storage Archive Queue

Dead-Data
Level 5
Partner Accredited Certified

Problem on EV 9.0.3 Journal Server

Since weekend partition rollover suffering building up of pending items in mailbox and the storage MSMQ queue.

Only other non-empty queue is J3 with single item.

Minimal CPU activity

No clues in Event logs as to what's happening or not happing.

Not in backup or report mode.

Have tried restarting services and server.

Currently 20,000+ items in journal mailbox, half in pending state.  6,500 items in storage queue.

Every four-six hours there's some movement, but not much.

Suggestions as to where to start diagnosing this?

 

1 ACCEPTED SOLUTION

Accepted Solutions

LCT
Level 6
Accredited Certified

Things that I would do are the following:

1. Increase the MSMQ memory if you can, try the best practice at 8GB or can just untick the limit (if you can), get the MSMQ going then you can set the limit back again after. How much RAM is on the EV server, is it VM or physical?

2. Cancel the pending icons in the journal mailnox manually (this will be painful as Outlook will keep saying not responding) and then restart the journal task and then they will get picked up again.

3. start dtracing the journal task processes after the above for as long as you can see if there's anything showing any time delay or slowness.

View solution in original post

6 REPLIES 6

GertjanA
Moderator
Moderator
Partner    VIP    Accredited Certified

Hi,

On the juste closed old partition, how is the diskspace?

On the new opened partition, how is the diskspace?

You say items are moving (from MSMQ) I assume?

Any large items in the mbx? What is the size of MSMQ? What is the free space on the storage for MSMQ? Is there a limit on MSMQ?

Can you define another Journal Mailbox, create a task etc, and have the old task work it's way through the existing mbx?

Can you in the task sync the journal mailbox? did you try restarting the storage-service?

 

Regards. Gertjan

Dead-Data
Level 5
Partner Accredited Certified

Hi Gertjan,

Yes have tried rolling back pending items using report mode, resyncing mailbox and rearchiving.

Partition storage is on NetApp CIFS so 'Infinite' diskspace. (OK 8TB free currently)

MSMQ is on direct attached storage for Journal server. currently at 800MB, so maybe close to the 1GB limit. 

Yes, creating further Journal tasks and Journal mailboxes is possible, although not ideal. 

 

 

 

 

LCT
Level 6
Accredited Certified

Things that I would do are the following:

1. Increase the MSMQ memory if you can, try the best practice at 8GB or can just untick the limit (if you can), get the MSMQ going then you can set the limit back again after. How much RAM is on the EV server, is it VM or physical?

2. Cancel the pending icons in the journal mailnox manually (this will be painful as Outlook will keep saying not responding) and then restart the journal task and then they will get picked up again.

3. start dtracing the journal task processes after the above for as long as you can see if there's anything showing any time delay or slowness.

GertjanA
Moderator
Moderator
Partner    VIP    Accredited Certified

I'd go with the suggestions LCT does.

Increasing MSMQ memory to 8GB is advisable. It might run over the limit during archiving, which might cause this issue. As for my advise to create a new task and journalmailbox, I know from experience that these kind of issues take a while to fix. This in turn leads to journalmailboxes piling up, and you might find yourself fighting other fires because of this. Creating a new journalmailbox and task enables you to focus on the archivng for this one mailbox, without having new data added to it.

But, it is ofcourse your call.

To cancel the icons in the journalmailbox, I have used DocMEssageClass (which is as slow, but automatic), but as far as I am concerned, it is a last resort. the main thing is to find out why it is so slow. I assume ou have checked AntiVirus exclusions for the EV locations? Any logs on the NetApp indicating issues?

Regards. Gertjan

Dead-Data
Level 5
Partner Accredited Certified

Update on this issue.

There was a limit of 800MB on the MSMQ queues on all EV servers.  Once this was removed the queues have started moving at a decent pace (25,000 per hour per server).  Strange that this only became a problem recently. 

Other related issues were; 1/2 million ExchPerfLog.Dat files in temp folder.  Registry keys to clean up temp files now set. Workaround of changing temp folder locations for KVSEV service account  and system.

Additional Journal task and mailbox setup to spread the workload to other EV servers on this site.

Index locations were also heavily fragmented. Plus SQL Indexes on Vault store tables heavily fragmented.  Scheduled defrag tasks and SQL maintenance plans now in place. 

Also for good measure, doubled the amount of memory and CPU allocated to each EV server and the SQL server.

AV scanning not present on NetApp locations or EV index locations.  NetApp CIFS reasonable at 10,000 IOPS.

 

 

GertjanA
Moderator
Moderator
Partner    VIP    Accredited Certified

Hello DEJ,

Sounds like you have it under control now.

If you are satisfied, can you mark one of the answers as solution. That way, other users having this can find the solution more easily.

Thanks!

Regards. Gertjan