Journaling Mailbox keeps Increasing

[jhanice]

Hello,

I would like to ask ideas about our current issues with the Enterprise Vault. The journaling mailbox suddenly grows up to 7.5 GB and keeps increasing. We already tried to reboot the server but it still the same. We saw some errors from logs about Event ID 3310 "There was a problem accessing a network service or resource.  The dispenser will re-queue the current item and sleep for 10 minute(s). " and we saw solution from here

http://www.symantec.com/business/support/index?page=content&id=TECH48862. But my worry is, it really alright to purge the queues what will happen to them?

The current queue's size are as follows and didn't change. Do you have any idea if it's alright to purge these?

Enterprise Vault Storage Archive Queue - 3949
Enterprise Vault Exchange Mailbox Task for XXXXXXX A5 - 1680
Enterprise Vault Exchange Journaling Task for XXXXXXX J1 - 1595
Enterprise Vault Exchange Journaling Task for XXXXXXX J2 - 73
Enterprise Vault Storage Restore Queue - 10
Enterprise Vault Exchange Journaling Task for XXXXXXX J3 - 1

Anyone's help will be gladly appreciated.

[GertjanA]

Can you verify that the storage queue goes down when you restart the storage service?

do not empty queues yet. no events in the eventlog?

[Percy_Vere]

I would suggest that you perform the dtrace as detailed in the technote and ensure you have the same error message as 3310 errors can have multiple sources that cause the error and it may be that the issue in that technote is not your problem.

[NaturesRevenge]

Yeah, don't purge those queues. You'll lose the content.

Are you in danger of running out of storage space where your Journaling mailbox exists, either the database and/or logs disks? If you've got room, let the mailbox continue to queue. When EV journaling is functioning the way it should, it'll scrape that mailbox in no time.

We get Event 3310 all the time. The journaling task should pause for 10 minutes but should start back up afterwards. If it continues to fail and pause, we reboot both our journaling EV server AND the Exchange server that has journaling mailbox. Our journaling mailbox is isolated so rebooting does not affect any users.

[JesusWept3]

check that you haven't breached the 1GB MSMQ quota as well

[GertjanA]

create a 2nd journal mailbox, a second task to that mailbox, re-configure journaling, and let the old mailbox be emptied by the already existing task..

[Chris_Warren]

You may be seeing a couple of issues here.  First you are receiving the 3310's, which usually means EV was unable to stay connected to Exchange and the task will pause for 10 mins to see if it gets better.  Next, you have the current queues. 

For the 3310's, I'd suggest looking into what else is going on during the time.  Since it is commonly network related, it could be backups or exchange maintanence interfering with archiving.  Usually, I look at this way... a single 3310 event isn't so bad, could have been a 'hiccup' on the network...  now a repetitive 3310, every 10 minutes... that says somethings going on between EV and Exchange during that time causing a network connection issue for awhile. 

Commonly, you want your Exchange backups to occur while EV is being backed up... if possible.

Next the MSMQs :

- As JW2 referenced, it is common to see the MSMQ at the default of 1 GB.  The usual suggestion here is that if you are running a 32 bit OS, the maximum size shouldn't be set to more than 4 GB.  The larger this gets on 32 bit, the more unstable it can get.  If you have a 64 bit OS, this is more stable and can be set much larger, but usually the recommendation is still a max of 8 GB.  With the numbers you are showing in only those few queues, this could be a potential concern.

- Next, we should look at the individual queues.  Storage Archive is is over 3900.  Once this queue goes over 1500, EV will throttle down the Journal Task, slowing ingesting to allow Storage to catch up.  The fact that this queue is so high makes me want to look at the storage device it's going to.  If it's a remote storage, network can come into play.  Then there is the concerns of storage function and if there is a bad drive, etc.  Dealing with other APIs can be trickier with storage, for example storing to Centera, since there is also the need to wait for replication.  I'd recommend Dtracing JournalTask, StorageArchive and StorageFileWatch and look "time taken" references to see how long individual 'high level' processes took (aka Process item, Storing Item, Post Process item).  This can possibly point you in the direction.  As a base line, with Journaling processing, individual items should commonly take a second or less from Process to Post Process.

Oh... one final piece of info.  All journal and mailbox archiving Tasks on the same EV Server will share the single Storage Archive and Storage Resotre queues, so it may be possible that there are items in the Storage Archive queue that came from your Mailbox Task.  You can open the Message Queues, Right click on the Storage Archive queue and select 'new window from here' and check what items are in the Storage Archive 'queued messages'.

Hope this helps.

[jhanice]

Hello All,

Thank you very much for the inputs.

I already tried to reboot the EV server 2x and nothings change journaling mailbox still increasing with 9 GB now. I've requested for the reboot of the exchange servers also but i need to wait for the weekends before to proceed and see what will happen.

Just want to update that the queues have changed.

Enterprise Vault Storage Archive Queue - 3949 - now(6229)

Enterprise Vault Exchange Mailbox Task for XXXXXXX A5 - 1680 now(2192)

Enterprise Vault Exchange Journaling Task for XXXXXXX J1 - 1595 (same)

Enterprise Vault Exchange Journaling Task for XXXXXXX J2 - 73 now(1)

Enterprise Vault Storage Restore Queue - 10 now(12)

Enterprise Vault Exchange Journaling Task for XXXXXXX J3 - 1 (same)

Enterprise Vault Exchange Mailbox Task for XXXXXXX A5 – 1

@Chris Warren - I tried to run dtrace using the  GUI of EV Console and I was able to pull data related to Journal Task and Storage Archive and look for "time taken" but i wasn't able to find any entries but I found below lines with word error.  

117246  00:22:37.648       [9688]  (JournalTask)     <4856>EV:M     CExchangeArchivableItem::BuildFromMAPIMessageEx - Entered Routine|

117247  00:22:37.648       [9688]  (JournalTask)     <4856>EV:M     (BlacklistedDLs) 'BlacklistedDLs' registry key not found. No DLs will be blacklisted.

117248  00:22:37.648       [9688]  (JournalTask)     <4856>EV:M     CArchivingAgent::PI_BuildObjectsFromMessage - GetMessageClass() returned error 0x00000000

117249  00:22:37.648       [9688]  (JournalTask)     <4856>EV:M     EPC::GP - Returning Policy : [Default Exchange Journaling Policy][159CEBA3FCCBDED41BF15EEBB5851BCF81012700evserver.domain.com]

117250  00:22:37.648       [9688]  (JournalTask)     <4856>EV:M     CAA::GUP() - /O=DOMAIN/OU=NSL/CN=RECIPIENTS/CN=EVJOURNAL is using Policy : Default Exchange

Does anyone know what is this for? 

@GertjanA

- If i will create a new Journal Mailbox and Task is it sure the the 1st Journal Mailbox with 9GB will lower down and no impact on the emails to be archive?

@JesusWept2

- the size of the folder "C:\WINDOWS\system32\msmq\storage" 434 MB so it doesn't reach yet the 1GB threshold.

Again, your help  will be gladly appreciated.

-

[Chris_Warren]

What version of EV are you running with?  Regarding that error, it could be a couple of things.  Could be a corrupt message or something revovling around encryption, if they are encrypted items and EV cannot read them.  There still seems like there is something going on with communication to Storage, based on the Storage Archive queue growing.  Is EV in a Backup mode?  Perhaps you can toggle it in and out of backup mode.

Also, if you open MSMQ and expand the queue itself, you can see when these items were placed in the queue.  This can give you an idea of how long this has been going on, what the oldest items in the Storage Archive Queue are and, over time, if these change... aka, are any items being removed from the Storage Archive queue at all?

Another thing you can try is to enable 'journaling' on the Storage Archive Queue TEMPORARILY.  As items pass through (added and removed) from the queue, a journal record is added to the MSMQ Storage Archive Journal Queue (Not to be confused with Journal Task queues or Journal Archving. This is strictly for MSMQ.)  This can show if any items are being sent to Storage and clearing from the Storage Archive. Again, this is a temporary troubleshooting step and should be disabled when you are done watching.

For now, I would recommend focusing on why the Storage Archive queue is so backlogged.  Once that is under control, Journaling may very well start catching up.  Can you get a Dtrace of StorageArchive and StorageFileWatch and possibly attach it here?

[jhanice]

Here's an interesting part. When i tried to delete an old mailbox archive of a certain user it says "Could not delete the archive 'account' because the Storage Service on 'server.domain.com' is not running. Start the service and try again".Actually the EV Storage Service is running. I tried to restart it and it's still the same.I've reviewed the logs and there are some informational alert about Storage Archive.

6973 - Storage Crawler is not enabled. 

6608 - Storage Expiry is not enabled 

6631 - Storage Archive is not enabled 

6634 - Storage Restore is not enabled 

6632 - StorageFileWatch is not enabled 

Not sure though if this is normal. 

Enterprise Vault 2007 V7.5

I did find the date and time via the MSMQ when messages started to queue up, it is Jan.21 and there's a patch for the OS installed on that day also. Not sure though it this affect the EV but i'll take a look also. 

I'll try Dtrace again for StorageArchive and StoraFileWatch and let you know.

[Rob_Wilcox1]

Have a look at the registry keys mentioned in this technote :

http://www.symantec.com/business/support/index?page=content&id=TECH35932

Maybe that will shed some light on things?

[Chris_Warren]

Were you able to fix this?  Was it in Backup mode?

[Prone2Typos · Accepted solution]

Those events are related to the services being in backup mode and are usually indicitive of a backup job starting but not finishing

This is what Rob W was suggesting and I would suggest that you manually change them but monitor for recurrence. . .which would be indicitive of the bakcup not running.

I Agree with Chris W . . .the 3110 are likely not directly realted to the storage issues. . This happens for basically any reason that  Exchagne stops communicating for any reason. If you are used to Outlook reconnecting on your client you will see these regularly . It is not a serious concern until you see them happening consecutively . . .but could be indicitive of another problem.

You may also wanna look to your Exchange server to see if you are getting any errors on it that oculd be related to client connectibity issues.

If this is resolved please mark it as such to give the person who helped you credit for doing so and to prevent people like me from commenting unnecessarily.