Task Controller service stops

3 EV servers 7.5 SP 5
2 of them have this problem when we start Archiving task after few minutes it logs this event and task controller stops.

Event Type: Error
Event Source: Enterprise Vault
Event Category: Archive Task
Event ID: 6197
Date: 6/16/2009
Time: 6:33:40 PM
User: N/A
Computer: Removed
Description:
Error occurred when sending a message
There are insufficient resources to perform this operation. [0xc00e0027]

These servers are only doing archiving no storage and Indexing.

Any help would be appreciated.



1 Solution

Accepted Solutions
Accepted Solution!

That's definitely a size

That's definitely a size issue.
Have you tried to restart the server? Maybe there is something stuck.

You could also try to increase the MSMQ size slightly, you do this on the properties of the main "Message Queuing" node in Computer Manager.

Cheers


View solution in original post

17 Replies

Do you also see Event

Do you also see Event ID 3249?

If you have had to purge MSMQ you may need to purge the MSMQ>System Queue>ransactional dead-letter queue as they will build up here and MSMQ will be over it's size limit.

If you don't see that error then check the following technote.
http://seer.entsupport.symantec.com/docs/292506.htm


No 3249

Tony,

I am not getting 3249 and I dont have anything in system queue. I have already added the registy keys and rebooted the server. It worked for little while but the moment I started Archiving the task controller stopped.

Thanks,

Hmm, can you verify the size

Hmm, can you verify the size of MSMQ just to be safe.  Typically I have seen 0xc00e0027 in relation to it.

Please also see this article

Please also see this article i wrote Smiley Wink

Microsoft Message Queuing (MSMQ) issues?

--wayne

25000+ messages in Outgoing Queue

Hi Wayne,

Unable to accces the article..

I noticed that there are 25000+ messages in Outgoing queue for Storage Archive (going to server which has storage) and state is showing as "waiting to connect" they are not processing looks like they are stuck.

Any way of forcing these messages?
Accepted Solution!

That's definitely a size

That's definitely a size issue.
Have you tried to restart the server? Maybe there is something stuck.

You could also try to increase the MSMQ size slightly, you do this on the properties of the main "Message Queuing" node in Computer Manager.

Cheers


 

  I would do this 1. Stop all your ev services 2. Restart msmq on your server that has the storage service and then restart ev 3. Restart msmq on the other EV servers 4. Restart the ev services on those servers

I have increased the size and

I have increased the size and after that Task controller service is running. I have stopped Archiving task so it would not process any more messages but these messages are stuck in outgoing queue.

I am going to try what Tony suggested...

Still 25000+ items

Tony,

I tried what you mentioned in your last post but still those messages are not going anywhere

Outgoing: Direct=osxxxxxxxxxxprivate$\Enterprise Vault Storage Archive...... "Waiting to connect"

Yes, I'd definitely recommend

Yes, I'd definitely recommend the same.
Do not start archiving again until your storage queue is empty.

Anything more in the eventlogs?

Cheers

Have a look here, maybe it

Have a look here, maybe it has something to do with your problem:
http://kbalertz.com/904331/Messages-arrive-destination-computer-Message-Queuing.aspx

Or

Or this:
http://blogs.msdn.com/johnbreakwell/archive/2007/06/28/why-does-msmq-lock-your-outgoing-queue.aspx

If it is the outgoing queue

If it is the outgoing queue then this is really out of the realm of EV and is within the realm of MSMQ. So what you must have is archives that are attached to a storage service on another ev server so we are sending the request over and for whatever MSMQ reason they are not going.

another thing you can look at

another thing you can look at is this
http://support.microsoft.com/default.aspx?scid=kb;EN-US;329492


Seeing that my article has

Seeing that my article has not been approved yet here is a copy and past from it:

I always get asked about MSMQ, here is a short note on things to look at if you have MSMQ issue's, even if you don't have issues you can always benefit from tweaking MSMQ and enhance Enterprise Vault’s performance.

Microsoft Message Queuing (MSMQ)

The root cause I believe lies in the way MSMQ stores messages. In the Storage directory (normally under windows\system32\msmq) will be a number of 4MB files that will contain the messages. Each file can contain as many messages as will fit which works quite well to start with. If a message cannot fit into one storage file, though, because it is too big for the remaining free space then a fresh file is created. There can only be as many storage files as would fit within the storage quota and when this limit is reached then no more storage files can be created. For example from windows 2003 SP2, a 2GB quota could accommodate 500 4MB storage files. In older versions Microsoft used to have an 8GB quota which would accommodate 2,000 4MB storage files.

Note: MSMQ storage works differently from file storage - there is no linked list stitching together parts of a message. An MSMQ storage file has to have inside it a contiguous block of free space that is large enough to accommodate the whole message. MSMQ messages cannot span storage files either - if the message is greater than 4MB then MSMQ cannot store it.

This problem is made worse because MSMQ uses a pipeline for sending messages. If there is a large message at the front of the queue then it will block all the (possibly smaller) messages behind it until delivery becomes possible.

What can be done about this? Here are a few suggestions:

1. Wait it out - storage space should become available eventually. After the House Keeping has happened six hours by default. (Restarting Tasks however full up the cache locations)
2. Set the House Keeping to run every hour (I would recommend this firstly)
3. Set the size of the Queue higher (I would suggest using 4GB to start off with) See bellow for instructions on MessageCleanupInterval
4. Set a Time To Reach Queue on the messages - the large message will expire and be removed from the outgoing queue, allowing the smaller messages through. (This has been added to Enterprise Vault, as of 2007 SP5)
5. Purge the outgoing queue and resend the smaller messages first (I don’t suggest this)

Set Message Cleanup Interval

The following fix installs a periodic cleanup of empty Message Queuing message files. The default interval for cleanup is six hours. This interval can be modified by a new registry key. To change the default interval, you must add MessageCleanupInterval to the registry.

1. Start Registry Editor (REGEDT32.EXE).
2. Locate and select the following registry subkey:
3. HKEY_LOCAL_MACHINE\Software\Microsoft\MSMQ\Parameters
4. From the Edit menu, click ADD VALUE.
5. In the Value Name Field, type MessageCleanupInterval.
6. In the Data Type Field, select REG_DWORD.
7. For the data value, enter the desired interval in milliseconds (for example, 3600000 (decimal) for one hour).
8. Exit Registry Editor to save the changes.

The next time you restart the Message Queuing service, the new interval will be used to cleanup empty message files.

Microsoft SQL Query to find out archiving rate:

This is a Microsoft SQL script to get a list of archiving rates for the past twenty four hours.

-----------------------------------------------------------------
-- Author: Wayne Humphrey
-- Date: 26th March 2009
------------- Notes -------------
-- Work out the hourly archiving rate, for the past 24 hours
-- You need to define all the Vault Stores in this query
-----------------------------------------------------------------

USE EnterpriseVaultDirectory

declare @VSName varchar(4000)
declare @sql varchar(4000)
declare TCursor CURSOR READ_ONLY for
select DatabaseDSN from dbo.VaultStoreEntry

OPEN TCursor

FETCH NEXT from TCursor INTO @VSName

WHILE @@fetch_status = 0
begin

select @sql = 'select VSName=''' + @VSName + ''' ,' +
' min(archiveddate) as ArchivedDate, ' +
' count(*) as HourlyRate, ' +
' sum(itemsize)/count(*) as AvSize from ' +
@VSName + '.dbo.saveset s' +
' where archiveddate > dateadd(hh, -24, getutcdate()) ' +
' group by left(convert(varchar, s.archiveddate,20),14) ' +
' order by left(convert(varchar, s.archiveddate,20),14) desc'
exec (@sql)

fetch next from TCursor into @VSName
end

close TCursor
deallocate TCursor

Restart of all EV servers

Thanks everyone for all the help, Tony, Michelz, and Wayne (nice explanation).

I had to change the schedule on all Archive Tasks to never archive.

Restart the Storage EV server where all these messages were going from other EV server.
Then restarted all other EV Servers which then starting processing 100,000+ messages slowly


There were lot of 4 MB messages in those as well.

Glad to help! Have fun with

Glad to help!

Have fun with EV Smiley Happy

Cheers