Enterprise Vault Getting Retrieval Task error

[Suraj_K]

Can some one give some thouhts on below error.

Due to this email not retiving from EV 8.0 SP2.

Event Type: Error
Event Source: Enterprise Vault
Event Category: Retrieval Task
Event ID: 6190
Date:  11/15/2010
Time:  12:16:33 PM
User:  N/A
Computer: Server Name
Description:
Error allocating memory at
Ref: .\SavesetMessage.cpp Line 445

For more information, see Help and Support Center at http://evevent.symantec.com/rosetta/showevent.asp

[Maverik]

Hi,

In my view going to be pretty hard to resolve just based on this event unfortunatley. I would recommend the following.

1: Perhaps look to upgrade to Ev8SP5.

2: Does the issue always occur for every user and verry message?

3: If you cycle the EV services and try again does the issue still occur?

4: If you reboot the EV server does the issue still occur?

5: Have you run Deployment Scanner and checked all optimization settings?

6: Any other errors occurring in System or application event log?

7: Any other notibale memory problems? When issue occurs can you spot anything that may be of concern via Task Manager memory monitor?

[Rob_Wilcox1]

It might be worth giving an indication of how often this happens.. and on what version of Enterprise Vault.

[Suraj_K]

Thanks for replies :)

Its happens on all the mails. If any user retrieve archive mails, EV genrate mentioned error.

We are using EV in Microsoft cluster. I have restarted both servers but facing same issue.

What you think, Is it require to escalate to Symantec.

[Rob_Wilcox1]

What version of EV?

[Suraj_K]

EV 8.0 SP2

[Suraj_K]

Sorry its SP4 :)

[Rob_Wilcox1]

Have you tried increasing ThrottleLowerThreshold, and ThrottleUpperThreshold, per the performance guide :

http://www.symantec.com/business/support/index?page=content&id=TECH64450

[Suraj_K]

Thanks Rob.

As per performance guide all setting are intact.

Also no need to change ThrottleLowerThreshold, and ThrottleUpperThreshold as per performance guide.  As nothing is in MSMQ queue. But after server restart mentioned error was not genrated. Still Archive mails not retrives & showing pending.

Any thing other we can do?

Thanks,

Suraj 

[JesusWept3]

Really need more detail to be honest, Slow retrievals can be for so many reasons...
Slow Storage or communication issues with the storage (Such as a centera or items on NetBackup)
Could be that the SQL Server has a query thats blocking other queries or making them run slower or the SQL Servers CPU is pegged at 100% and queries for the items are taking a really long time to complete.
It could be that the EV Server itself is 100% because of the EVConverterSandBox.exe converting items from their Word doc formats etc in to Text or HTML.

To be honest though, to me it sounds like its an issue with MSMQ

I think what you need to do is just dtrace some processes and retrieve a few items and record how fast they go and look at the dtrace to see what is really taking so long

[Rob_Wilcox1]

I suggest ..

1.  Stop the EV services (stop the admin service, and it will stop all the others)

2.  Open DTRACE, and set it up for verbose logging of RetrievalTak.

3.  Start all services.

4.  Try opening one item.

5.  Wait a few minutes, and then stop the DTRACE..  and attach it here (if it isn't too big).

You may want to do that 2-3 times on different messages.

Also indicate if you get the event logged again, during the tests.

Hope that helps,

[Rob_Wilcox1]

Your DTRACE contains no trace lines, which means either :

a) It wasn't captured at the right time

b) It didn't get as far as the RetrievalTask

Given that the top of the trace says that nothing is being traced then can you confirm that you did correctly do :

DTRACE

set retrievaltask v

log c:\dtrace.txt

Then reproduce the problem

Then back in the DTRACE window, type "log", then "exit" to come out of DTRACE

This is what it should look like :

Id    Name                             Detail     Include Events

74  - RetrievalTask                    Verbose    Yes  

Please can you repeat the process, and in addition do a "set w3wp v", and "set agenticlientbroker v" in your DTRACE window before you start logging to a file.  Lastly check the IIS log on the EV server, it should contain 2 entries for clientaction.asp for the time that you try to restore an item.  The first will be a 401 meaning you need to authenticate, that'll happen automagically in the background, and the second entry immediately after the 401 should then be a 200 for the same request.

Your client log looks good.  It shows the items being marked, and the request going off to the EV server :

16/11/2010 07:44:10.586[11596]: Sending HTTP request: http://ARLEVCLS.india.eclerx.com/EnterpriseVault/clientaction.asp?act=4&fdrenc=*v9JAG4AYgBvAHgA&dn=/o%3dINDIAECLERX/ou%3dFirst%20Administrative%20Group/cn%3dRecipients/cn%3dSuraj.Karande&svr=ARLEXCHCLS1&sid=1F89092F6EA6DA24C8B6F90D508CFA01B1d10000ARLEVCLS&tsp=2010-11-16T07:44:10&pdl=AAAAAAAADCNMOAEKIMKJPPEPJCLOMPMJLKLJBKPHABAAFHLDCFHIGELDCFEKKMGKJFHLGNGCLBKMAAAABLFGFOKEAAAA

[Suraj_K]

Hi Rob,

Please find new Dtrace log.

Also I found below entry in IIS log not sure is it correct entry.

EnterpriseVaultOutlookExt-V8.0.3.1845 200 0 0 424 3048 390

Thanks.

[Nick_White]

Hi,

looking at the log file you are getting a lot of access denied errors. From the properties of the Directory in the Enterprise Vault Administration console can you re-enter then Vault Service Account password and, when it has finished (assuming no errors), restart the services and try again with Dtrace running. You might also want to check that nothing has changed for the Service Account (added to/removed from AD groups for example)

[Rob_Wilcox1]

In the IIS logs, the entry you are looking for will contain the string clientaction.asp, as that's what the page is called that the client is requesting to do your archived item restore.

It will look a bit like this :

2010-11-16 09:05:47 192.168.1.2 GET /EnterpriseVault/clientaction.asp act=4&fdrenc=*v9JAG4AYgBvAHgA&dn=/o%3dEV%20Training/ou%3dExchange%20Administrative%20Group%20(FYDIBOHF23SPDLT)/cn%3dRecipients/cn%3dex2k7testuser1&svr=SERVER1&sid=18ADD8B942CD441489E2BEA133EBBEF481d10000evsite&tsp=2010-11-16T09:05:47&pdl=AAAAAAAANAHAELDMCGAGIAENKAOKDCDHCAMJCBEIABAAEOMBPBNBHMFMPKEGKFJCMOLDMMEBFDBPAAKNJJBHLCIOAAAA 80 - 192.168.1.199 EnterpriseVaultOutlookExt-V8.0.4.2019 401 2 2148074254

2010-11-16 09:05:47 192.168.1.2 GET /EnterpriseVault/clientaction.asp act=4&fdrenc=*v9JAG4AYgBvAHgA&dn=/o%3dEV%20Training/ou%3dExchange%20Administrative%20Group%20(FYDIBOHF23SPDLT)/cn%3dRecipients/cn%3dex2k7testuser1&svr=SERVER1&sid=18ADD8B942CD441489E2BEA133EBBEF481d10000evsite&tsp=2010-11-16T09:05:47&pdl=AAAAAAAANAHAELDMCGAGIAENKAOKDCDHCAMJCBEIABAAEOMBPBNBHMFMPKEGKFJCMOLDMMEBFDBPAAKNJJBHLCIOAAAA 80 EV\ex2k7testuser1 192.168.1.199 EnterpriseVaultOutlookExt-V8.0.4.2019 200 0 0

The first one being the 401, for authentication, and the second being the success.

With regards to your Dtrace, you're gettings lots of access denied...  so see Nick's response for that.

[Nick_White]

That has managed to get us a step further. Now the log shows that the account that is running your Exchange tasks is a member of the Domain Admins group. This is a bad thing as the Exchange Server permissions for Domain Admins includes an explicit deny for the send as permission which we require to be able to process mailboxes. You need to either remove the Vault Service Account from the Domain Admins group and correctly apply the permissions for the account or alternatively allow the send as permission for the Domain Admins group on your Exchange server(s)

[Suraj_K]

Hi Rob,

I have genrated RetrievalTak DTrace as per your steps. Please find attached.

No event genrated but still archive mails Retrieval is in pending state.

Also find EV client log.

Thanks.

[Suraj_K]

I have re-enter then Vault Service Account password and, restarted the services in properties of the Directory in the Enterprise Vault Administration console.

All the IIS log entries are similar as Robs.

Still same issue.

Please find new Dtrace log.

[Nick_White]

Getting there one step at a time :)

The log includes the following which suggests a problem with MSMQ:

154	20:20:51.903	 [5816]	(RetrievalTask)	<8516>	EV:H	An unexpected timeout occurred while attempting to access a queued message.| |This may have occurred because a message was not on a queue as expected or because the system is heavily loaded, thus causing excessive MSMQ delays.| |If you suspect the problem is caused by MSMQ delays, try increasing the Enterprise Vault "QueueTimeout" registry value, as described in the "Modifying Registry Settings" section in the Troubleshooting section of the Administrator help. |

Can you go through the troubleshooting steps as suggested to see if that kicks things into life

[Suraj_K]

We have solve permission issue but still same perssist.

PFB new DTrace log.