Forum Discussion

tolinrome1's avatar
tolinrome1
Level 5
15 years ago

eVault task controller service and storage service stopping

This has been ing on for a couple of weeks now and I still cant find a solution. THe task controller and storage service are always in a stopped state. If I start them it may last for a few seconds to a couple of hours but they always end up stopping and clients cant view any archived emails. I tried many suggested solutions from Symantecs kb articles and called tech support and still no go.

Windows 2003 SP2 , eVault 7.5 SP3
I did all this below:

1. Added the registry keys for the memory management - http://seer.entsupport.symantec.com/docs/292506.htm
2. increased page file size - http://seer.entsupport.symantec.com/docs/346629.htm
3. In creased the MSMQ from 1 GB to 8GB - http://seer.entsupport.symantec.com/docs/301437.htm

I eventually called a tech and we found over 300,000 messages in the MSMQ, We purged them all and it seemed to run great but then a few hours later...same problem. Below are the technicians notes of what we performed.

-Configured MSMQ-related registry changes in accordance with Enterprise Vault best practices
-Examined MSMQs. Several queues large numbers of unprocessed items.
     -r1 has 37k
     -storage restore has 300k
     -storage spool has a few
     -r1's Admin has 400
-Purged storage restore, then r1, then storage spool, then admin, testing in between each to see if the services would stay up.
-After storage spool, the services did not automatically stop themselves. However, they still stopped when we actually ran an EV Provisioning Task.
-After purging the Admin (for r1) as well, we were able to run the Provisioning Task successfully
-Verified that we were able to archive and retrieve items from the vault.

Any suggestion? I'm out of ideas. Thanks.
Enterprise Vault
Error messages
Evaluating
Information Governance
  • JesusWept3's avatar
    JesusWept3
    14 years ago

    Yeah i noted above that it most likely breached its Quota.
    Also for clarities sake, when you purge an item out of an MSMQ, by design they go to the Admin Queues with in MSMQ for EV, and then EV will report that messages have been purged.

    What you should normally do is, stop the EV Services, purge the queue, then note that all the items now populate under the Admin queues, after that you purge that Admin queue.


    I do believe there are a few technotes around this describing this behavior for some time, some though get pulled because purging queues is not encouraged as some can lead to some bad issues (i.e purging the A1 queue is bad).

    Also the attachmentmax and recipientmax won't affect items in the queue, it will just force EV to give a corrupt data error message (which is actually gotten from outlook).

    The Transactional Dead Letter Queues are only used when MSMQ receives a Negative Acknowldegement (NACK) from the queue its attempting to reach or its timed out trying to get where it needs to go, usually this is seen when you have Outgoing queues that push messages to another EV Server etc
     

11 Replies

Sort By
  • Michael_Bilsbor's avatar
    Michael_Bilsbor
    Level 6
    15 years ago
    Hi,
    Provisioning task shouldn't cause EV shutdown. Are you sure about that.
    The tasks you menton will shutdown if MSMQ logs an error about insufficient resources (because at that point EV won't work).  The root cause to a certain degree depends upon the queues with problems (in your case r1 etc.)  Sounds like perhaps there are restore issues and requests are building up and not being processed. 
    Is someone doing *A LOT* of resotres and why?

    I suggest you get dtrace of retrievaltask as it sounds like perhaps it's that which is getting stuck and so causing a buiild up of requests.

    Mike
  • tolinrome1's avatar
    tolinrome1
    Level 5
    15 years ago

    No, I dont think this is related to the provisioning task, although at times I have to restart the evault services to get it running.
    I think this problem is like you mentioned; related to the queue, probably the MSMQ.
    I just did a dtrace on retrievaltask and got this only:

    DTrace trace log
    [Opened: 2010/06/04 07:53:05]

    Enabled Trace options
    Id    Name                             Detail     Include Events
    73  - RetrievalTask                    Verbose    Yes        
    -----

    Display Filter:
      Include strings:
        EV-
        EV:
        EV~
      Exclude strings:
        CMMStream
        DBFramework
        DirectoryConnection
    -----

    Running in Terminal Services session...
      Monitoring debug output from processes running in the console session
      Monitoring debug output from processes running in the current Terminal Services session



    Any suggestions? Thanks a lot!
  • tolinrome1's avatar
    tolinrome1
    Level 5
    15 years ago

    I had to manually purge again tasks in the queue. Is there a way this can be solved? I'm doing the logging on the retrieval task but theres only a few lines, nothing important so far.

  • Michael_Bilsbor's avatar
    Michael_Bilsbor
    Level 6
    15 years ago
    Hi,

    When the problem occurs are you seeing high CPU usage?  I know we recently fixed a bug wher a certain type of restore would cause high CPU in retrieval task and stop further restores.

    Mike
  • tolinrome1's avatar
    tolinrome1
    Level 5
    15 years ago
    I dont think so since cpu usage is usually between 1-10%. - but I do get alot of errors in the event viewer regarding low resources such as CPU time, although the hardware is sufficient enough by far.
    What's strange is that some user's can still get archived emails while others cant, yet the two services are stopped.
  • tolinrome1's avatar
    tolinrome1
    Level 5
    15 years ago
    anyway, I'd like to know what certain type of restore caussed the problem and what the fix is - this may help.
  • jpriddy's avatar
    jpriddy
    Level 4
    15 years ago
    I'm having the same problem on one of my servers... did you ever get this figured out?
  • Peter_Oldenburg's avatar
    Peter_Oldenburg
    Level 2
    14 years ago

    Can anyone tell what the status is from this post? I am having the same problem. Even the services in the VAC are not running, some people can open archived messages and some people can't.

    For starting the services I created a call at professional support. If i have the answer for that problem I will let you know.

    After moving a evserver with a new name you must add a dns alias with the old server name and point it to the new server name. Services are starting normally after doing this 

  • JesusWept3's avatar
    JesusWept3
    Level 6
    14 years ago

    Most likely the issue revolved around a user that was no longer on that exchange server attempting to restore items, it blocks the R1, causes the Archive task to throw 3310's and then any subsequent restores will also fail.

    For example

     - User A is on Exchange Server A and is enabled
     - They restore an item and it goes to the R1 queue for Exchange Server A
     - User A then gets moved to Exchange Server B
     - Provisioning and synchronization doesn't run or doesn't pick up the changes
     - User A then tries to restore an item
     - The item goes to the R1 queue for Exchange Server A (NOT Exchange Server B)
     - Enterprise Vault then tries to connect to Exchange Server A to find User A
     - An error gets thrown in the background saying User A no longer exists on Exchange Server A
     - The Archiving Task then goes to sleep for 10 minutes, throwing a 3310 error
     - The Archiving Task then wakes up and tries the same process, user can't be found, sleeps for 10 mins

    This would continue until you manually purge the queue, make sure the user is provisioned and then synced correctly on Exchange Server B, and the problem goes away

    This has been resolved in EV9 SP1.

    On top of that if you have things stuck in your Storage Spool, Storage Archive or Storage Restore Queues, remember these are the actual messages, not just small 1k/2k pointers to where the message is, so if someone restores a 10mb item, you will have about 5 messages in the Storage queues as it will be 10mb split in to chunks.

    So it is possible that with 300,000 messages in a storage queue that the 1GB quota limit on MSMQ has been breached, in Windows 2000 and early versions of 2003, the quota was 8GB, then by at least SP2 it was bought down to 1GB

    The combination of large items in the Storage queues, anything going to the dead letter queues, large backlogs in the Outgoing queues and if someone had turned on journaling on the MSMQ, that quota can be breached quite easily, and it is just like an Exchange quota, once it has been breached, you have to clear items out before it becomes usable again.


    In future cases what i would suggest is if you have a large backlog in your R2's/R1's and storage queues, firstly make sure you haven't breached your Quota on the MSMQ.
    Secondly, actually look at the messages that are "stuck", it will give the DN's of the user it is trying to store or restore to, make sure that user is on that queue.

    Plus if its the same user trying to store or restore a huge amount of items, maybe its best contacting them, find out why they are trying to restore that much email, and see if there is any alternative method you can use.

    For instance lets just say they're tired of the shortcuts and using search or explorer, maybe offer them virtual vault and vault cache, then they can have their email local on their machine, looks like a PST file, they don't end up restoring email every single time it gets archived etc etc etc

  • tolinrome1's avatar
    tolinrome1
    Level 5
    14 years ago

    This is what finally resolved it in my case:

     

    In the MSMQ (Microsoft Message Queue) the storage limit was exceeded due to the transactional-dead letter queue not clearing, this stopped the two services we were having problems with.  Once the messages were purged from the queue the problem resolved. So my question was, “Why are there messages in the transactional-dead letter queue and why aren’t they clearing?

     

    Well, messages can be stuck in the queue for a number of reasons, message time outs or corrupted messages and according to the Symantec tech we needed to also increase registry keys on the system for RecipientMax and AttachmentMax so this wouldn’t effect messages being retrieved from the vault.

    Also, if the messages are purged from the queue while the eVault services are running the messages can seem to be purged but they can be picked up again by the MSMQ for further processing which may clog the queue up again, so I had to stop all the services then purge the queue, then restart the services. This is something that’s not available for public knowledge yet…as of June 2010 when I wrote this above, it maybe public knowledge now.