cancel
Showing results for 
Search instead for 
Did you mean: 

Task Controller service stops

fasoy
Level 4
3 EV servers 7.5 SP 5
2 of them have this problem when we start Archiving task after few minutes it logs this event and task controller stops.

Event Type: Error
Event Source: Enterprise Vault
Event Category: Archive Task
Event ID: 6197
Date: 6/16/2009
Time: 6:33:40 PM
User: N/A
Computer: Removed
Description:
Error occurred when sending a message
There are insufficient resources to perform this operation. [0xc00e0027]

These servers are only doing archiving no storage and Indexing.

Any help would be appreciated.



1 ACCEPTED SOLUTION

Accepted Solutions

MichelZ
Level 6
Partner Accredited Certified
That's definitely a size issue.
Have you tried to restart the server? Maybe there is something stuck.

You could also try to increase the MSMQ size slightly, you do this on the properties of the main "Message Queuing" node in Computer Manager.

Cheers



cloudficient - EV Migration, creators of EVComplete.

View solution in original post

17 REPLIES 17

TonySterling
Moderator
Moderator
Partner    VIP    Accredited Certified
Do you also see Event ID 3249?

If you have had to purge MSMQ you may need to purge the MSMQ>System Queue>ransactional dead-letter queue as they will build up here and MSMQ will be over it's size limit.

If you don't see that error then check the following technote.
http://seer.entsupport.symantec.com/docs/292506.htm


fasoy
Level 4

Tony,

I am not getting 3249 and I dont have anything in system queue. I have already added the registy keys and rebooted the server. It worked for little while but the moment I started Archiving the task controller stopped.

Thanks,

TonySterling
Moderator
Moderator
Partner    VIP    Accredited Certified
Hmm, can you verify the size of MSMQ just to be safe.  Typically I have seen 0xc00e0027 in relation to it.

Wayne_Humphrey
Level 6
Partner Accredited Certified
Please also see this article i wrote ;)

Microsoft Message Queuing (MSMQ) issues?

--wayne

fasoy
Level 4
Hi Wayne,

Unable to accces the article..

I noticed that there are 25000+ messages in Outgoing queue for Storage Archive (going to server which has storage) and state is showing as "waiting to connect" they are not processing looks like they are stuck.

Any way of forcing these messages?

MichelZ
Level 6
Partner Accredited Certified
That's definitely a size issue.
Have you tried to restart the server? Maybe there is something stuck.

You could also try to increase the MSMQ size slightly, you do this on the properties of the main "Message Queuing" node in Computer Manager.

Cheers



cloudficient - EV Migration, creators of EVComplete.

TonySterling
Moderator
Moderator
Partner    VIP    Accredited Certified
  I would do this 1. Stop all your ev services 2. Restart msmq on your server that has the storage service and then restart ev 3. Restart msmq on the other EV servers 4. Restart the ev services on those servers

fasoy
Level 4
I have increased the size and after that Task controller service is running. I have stopped Archiving task so it would not process any more messages but these messages are stuck in outgoing queue.

I am going to try what Tony suggested...

fasoy
Level 4
Tony,

I tried what you mentioned in your last post but still those messages are not going anywhere

Outgoing: Direct=osxxxxxxxxxxprivate$\Enterprise Vault Storage Archive...... "Waiting to connect"

MichelZ
Level 6
Partner Accredited Certified
Yes, I'd definitely recommend the same.
Do not start archiving again until your storage queue is empty.

Anything more in the eventlogs?

Cheers

cloudficient - EV Migration, creators of EVComplete.

MichelZ
Level 6
Partner Accredited Certified
Have a look here, maybe it has something to do with your problem:
http://kbalertz.com/904331/Messages-arrive-destination-computer-Message-Queuing.aspx

cloudficient - EV Migration, creators of EVComplete.

MichelZ
Level 6
Partner Accredited Certified
Or this:
http://blogs.msdn.com/johnbreakwell/archive/2007/06/28/why-does-msmq-lock-your-outgoing-queue.aspx

cloudficient - EV Migration, creators of EVComplete.

Paul_Grimshaw
Level 6
Employee Accredited Certified
If it is the outgoing queue then this is really out of the realm of EV and is within the realm of MSMQ. So what you must have is archives that are attached to a storage service on another ev server so we are sending the request over and for whatever MSMQ reason they are not going.

TonySterling
Moderator
Moderator
Partner    VIP    Accredited Certified
another thing you can look at is this
http://support.microsoft.com/default.aspx?scid=kb;EN-US;329492


Wayne_Humphrey
Level 6
Partner Accredited Certified
Seeing that my article has not been approved yet here is a copy and past from it:

I always get asked about MSMQ, here is a short note on things to look at if you have MSMQ issue's, even if you don't have issues you can always benefit from tweaking MSMQ and enhance Enterprise Vault’s performance.

Microsoft Message Queuing (MSMQ)

The root cause I believe lies in the way MSMQ stores messages. In the Storage directory (normally under windows\system32\msmq) will be a number of 4MB files that will contain the messages. Each file can contain as many messages as will fit which works quite well to start with. If a message cannot fit into one storage file, though, because it is too big for the remaining free space then a fresh file is created. There can only be as many storage files as would fit within the storage quota and when this limit is reached then no more storage files can be created. For example from windows 2003 SP2, a 2GB quota could accommodate 500 4MB storage files. In older versions Microsoft used to have an 8GB quota which would accommodate 2,000 4MB storage files.

Note: MSMQ storage works differently from file storage - there is no linked list stitching together parts of a message. An MSMQ storage file has to have inside it a contiguous block of free space that is large enough to accommodate the whole message. MSMQ messages cannot span storage files either - if the message is greater than 4MB then MSMQ cannot store it.

This problem is made worse because MSMQ uses a pipeline for sending messages. If there is a large message at the front of the queue then it will block all the (possibly smaller) messages behind it until delivery becomes possible.

What can be done about this? Here are a few suggestions:

1. Wait it out - storage space should become available eventually. After the House Keeping has happened six hours by default. (Restarting Tasks however full up the cache locations)
2. Set the House Keeping to run every hour (I would recommend this firstly)
3. Set the size of the Queue higher (I would suggest using 4GB to start off with) See bellow for instructions on MessageCleanupInterval
4. Set a Time To Reach Queue on the messages - the large message will expire and be removed from the outgoing queue, allowing the smaller messages through. (This has been added to Enterprise Vault, as of 2007 SP5)
5. Purge the outgoing queue and resend the smaller messages first (I don’t suggest this)

Set Message Cleanup Interval

The following fix installs a periodic cleanup of empty Message Queuing message files. The default interval for cleanup is six hours. This interval can be modified by a new registry key. To change the default interval, you must add MessageCleanupInterval to the registry.

1. Start Registry Editor (REGEDT32.EXE).
2. Locate and select the following registry subkey:
3. HKEY_LOCAL_MACHINE\Software\Microsoft\MSMQ\Parameters
4. From the Edit menu, click ADD VALUE.
5. In the Value Name Field, type MessageCleanupInterval.
6. In the Data Type Field, select REG_DWORD.
7. For the data value, enter the desired interval in milliseconds (for example, 3600000 (decimal) for one hour).
8. Exit Registry Editor to save the changes.

The next time you restart the Message Queuing service, the new interval will be used to cleanup empty message files.

Microsoft SQL Query to find out archiving rate:

This is a Microsoft SQL script to get a list of archiving rates for the past twenty four hours.

-----------------------------------------------------------------
-- Author: Wayne Humphrey
-- Date: 26th March 2009
------------- Notes -------------
-- Work out the hourly archiving rate, for the past 24 hours
-- You need to define all the Vault Stores in this query
-----------------------------------------------------------------

USE EnterpriseVaultDirectory

declare @VSName varchar(4000)
declare @sql varchar(4000)
declare TCursor CURSOR READ_ONLY for
select DatabaseDSN from dbo.VaultStoreEntry

OPEN TCursor

FETCH NEXT from TCursor INTO @VSName

WHILE @@fetch_status = 0
begin

select @sql = 'select VSName=''' + @VSName + ''' ,' +
' min(archiveddate) as ArchivedDate, ' +
' count(*) as HourlyRate, ' +
' sum(itemsize)/count(*) as AvSize from ' +
@VSName + '.dbo.saveset s' +
' where archiveddate > dateadd(hh, -24, getutcdate()) ' +
' group by left(convert(varchar, s.archiveddate,20),14) ' +
' order by left(convert(varchar, s.archiveddate,20),14) desc'
exec (@sql)

fetch next from TCursor into @VSName
end

close TCursor
deallocate TCursor

fasoy
Level 4
Thanks everyone for all the help, Tony, Michelz, and Wayne (nice explanation).

I had to change the schedule on all Archive Tasks to never archive.

Restart the Storage EV server where all these messages were going from other EV server.
Then restarted all other EV Servers which then starting processing 100,000+ messages slowly


There were lot of 4 MB messages in those as well.

MichelZ
Level 6
Partner Accredited Certified
Glad to help!

Have fun with EV :)

Cheers

cloudficient - EV Migration, creators of EVComplete.