EV Storage Crawler Errors and event ID 1000

Question

Hi All,

We recently deployed 2x new EV servers, one file and one mail.

Both are VM's with the following specs:

OS: Server 2012 R2 (Latest patches as of 07/07/2014)

Ram: 16GB

CPU: 8x vProcs

EV Version: 10.0.4.1354

VMware Tools: 9.4.5, build 1734305

Our EV file archiving server has been generating errors around the Storage Crawler process for quite some time now. Symantec support are recommending a re-install of the entire OS, and also EV itself. Before we go down that path, I wanted to get the advice of some of the community members on here first. There must be a better way of resolving this then just blasting the thing and starting all over again. Yes it's a virtual machine, and yes all the config is back on SQL, but that's not really the point.

Below are the errors from the application log and specific Symantec Enterprise Vault Log. When it throws error 28998 for storage crawler, it also appears to throw event id 1000 in application logs. Both are listed below. I have also attached dtrace log when event occured. Dtrace was only against the storage crawler process, happy to re-run dtrace if people require.

Log Name: &nbsp; &nbsp; &nbsp;Symantec Enterprise Vault

Source: &nbsp; &nbsp; &nbsp; &nbsp;Enterprise Vault&nbsp;
	Date: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;3/09/2014 2:38:48 PM
	Event ID: &nbsp; &nbsp; &nbsp;28998
	Task Category: Storage Crawler
	Level: &nbsp; &nbsp; &nbsp; &nbsp; Error
	Keywords: &nbsp; &nbsp; &nbsp;Classic
	User: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;N/A
	Computer: &nbsp; &nbsp; &nbsp;BNE-EV02.corp.detir.qld.gov.au
	Description:
	Retrieval of a saveset failed.&nbsp;
	&nbsp;
	Reason: The data is invalid. &nbsp;(0x8007000d)&nbsp;
	Saveset Id: 201408319519198~201106200139000000~Z~E01A8113B44E1DA1F250EA9485D42421&nbsp;
	Vault Id: 16518AE13DD9FF64F8B266CB373D69EB31110000SSA-EVSITE01&nbsp;
	Index Sequence Number: 66594&nbsp;
	Internal Reference: Fetching Saveset for indexing&nbsp;

For more information, see Help and Support Center at http://entced.symantec.com/entt?product=ev&amp;language=english&amp;version=10.0.4.0&amp;build=10.0.4.1189&amp;error=V-437-28998
	Event Xml:
	&lt;Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"&gt;
	&nbsp; &lt;System&gt;
	&nbsp; &nbsp; &lt;Provider Name="Enterprise Vault " /&gt;
	&nbsp; &nbsp; &lt;EventID Qualifiers="49156"&gt;28998&lt;/EventID&gt;
	&nbsp; &nbsp; &lt;Level&gt;2&lt;/Level&gt;
	&nbsp; &nbsp; &lt;Task&gt;113&lt;/Task&gt;
	&nbsp; &nbsp; &lt;Keywords&gt;0x80000000000000&lt;/Keywords&gt;
	&nbsp; &nbsp; &lt;TimeCreated SystemTime="2014-09-03T04:38:48.000000000Z" /&gt;
	&nbsp; &nbsp; &lt;EventRecordID&gt;41305&lt;/EventRecordID&gt;
	&nbsp; &nbsp; &lt;Channel&gt;Symantec Enterprise Vault&lt;/Channel&gt;
	&nbsp; &nbsp; &lt;Computer&gt;BNE-EV02.corp.detir.qld.gov.au&lt;/Computer&gt;
	&nbsp; &nbsp; &lt;Security /&gt;
	&nbsp; &lt;/System&gt;
	&nbsp; &lt;EventData&gt;
	&nbsp; &nbsp; &lt;Data&gt;The data is invalid. &nbsp;(0x8007000d)&lt;/Data&gt;
	&nbsp; &nbsp; &lt;Data&gt;201408319519198~201106200139000000~Z~E01A8113B44E1DA1F250EA9485D42421&lt;/Data&gt;
	&nbsp; &nbsp; &lt;Data&gt;16518AE13DD9FF64F8B266CB373D69EB31110000SSA-EVSITE01&lt;/Data&gt;
	&nbsp; &nbsp; &lt;Data&gt;66594&lt;/Data&gt;
	&nbsp; &nbsp; &lt;Data&gt;Fetching Saveset for indexing&lt;/Data&gt;
	&nbsp; &lt;/EventData&gt;
	&lt;/Event&gt;

Log Name: &nbsp; &nbsp; &nbsp;Application

Source: &nbsp; &nbsp; &nbsp; &nbsp;Application Error
	Date: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;3/09/2014 4:21:34 PM
	Event ID: &nbsp; &nbsp; &nbsp;1000
	Task Category: (100)
	Level: &nbsp; &nbsp; &nbsp; &nbsp; Error
	Keywords: &nbsp; &nbsp; &nbsp;Classic
	User: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;N/A
	Computer: &nbsp; &nbsp; &nbsp;BNE-EV02.corp.detir.qld.gov.au
	Description:
	Faulting application name: StorageCrawler.exe, version: 10.0.4.1189, time stamp: 0x51dd972f
	Faulting module name: MSVCR90.dll, version: 9.0.30729.8387, time stamp: 0x51ea24a5
	Exception code: 0xc0000417
	Fault offset: 0x00059e63
	Faulting process id: 0x3cc
	Faulting application start time: 0x01cfc73f4e985040
	Faulting application path: D:\Program Files (x86)\Enterprise Vault\StorageCrawler.exe
	Faulting module path: C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.8387_none_5094ca96bcb6b2bb\MSVCR90.dll
	Report Id: 8d48bc51-3332-11e4-80d4-005056a9664b
	Faulting package full name:&nbsp;
	Faulting package-relative application ID:&nbsp;
	Event Xml:
	&lt;Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"&gt;
	&nbsp; &lt;System&gt;
	&nbsp; &nbsp; &lt;Provider Name="Application Error" /&gt;
	&nbsp; &nbsp; &lt;EventID Qualifiers="0"&gt;1000&lt;/EventID&gt;
	&nbsp; &nbsp; &lt;Level&gt;2&lt;/Level&gt;
	&nbsp; &nbsp; &lt;Task&gt;100&lt;/Task&gt;
	&nbsp; &nbsp; &lt;Keywords&gt;0x80000000000000&lt;/Keywords&gt;
	&nbsp; &nbsp; &lt;TimeCreated SystemTime="2014-09-03T06:21:34.000000000Z" /&gt;
	&nbsp; &nbsp; &lt;EventRecordID&gt;191934&lt;/EventRecordID&gt;
	&nbsp; &nbsp; &lt;Channel&gt;Application&lt;/Channel&gt;
	&nbsp; &nbsp; &lt;Computer&gt;BNE-EV02.corp.detir.qld.gov.au&lt;/Computer&gt;
	&nbsp; &nbsp; &lt;Security /&gt;
	&nbsp; &lt;/System&gt;
	&nbsp; &lt;EventData&gt;
	&nbsp; &nbsp; &lt;Data&gt;StorageCrawler.exe&lt;/Data&gt;
	&nbsp; &nbsp; &lt;Data&gt;10.0.4.1189&lt;/Data&gt;
	&nbsp; &nbsp; &lt;Data&gt;51dd972f&lt;/Data&gt;
	&nbsp; &nbsp; &lt;Data&gt;MSVCR90.dll&lt;/Data&gt;
	&nbsp; &nbsp; &lt;Data&gt;9.0.30729.8387&lt;/Data&gt;
	&nbsp; &nbsp; &lt;Data&gt;51ea24a5&lt;/Data&gt;
	&nbsp; &nbsp; &lt;Data&gt;c0000417&lt;/Data&gt;
	&nbsp; &nbsp; &lt;Data&gt;00059e63&lt;/Data&gt;
	&nbsp; &nbsp; &lt;Data&gt;3cc&lt;/Data&gt;
	&nbsp; &nbsp; &lt;Data&gt;01cfc73f4e985040&lt;/Data&gt;
	&nbsp; &nbsp; &lt;Data&gt;D:\Program Files (x86)\Enterprise Vault\StorageCrawler.exe&lt;/Data&gt;
	&nbsp; &nbsp; &lt;Data&gt;C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.8387_none_5094ca96bcb6b2bb\MSVCR90.dll&lt;/Data&gt;
	&nbsp; &nbsp; &lt;Data&gt;8d48bc51-3332-11e4-80d4-005056a9664b&lt;/Data&gt;
	&nbsp; &nbsp; &lt;Data&gt;
	&nbsp; &nbsp; &lt;/Data&gt;
	&nbsp; &nbsp; &lt;Data&gt;
	&nbsp; &nbsp; &lt;/Data&gt;
	&nbsp; &lt;/EventData&gt;
	&lt;/Event&gt;

&nbsp;

massimovagli · Accepted Answer

Hello James,&nbsp;

I think the best action would be for you to open a support case, this because of the complexity of the troubleshooting and the kind of data analysis.

Support would collect and analyze a memory dump of the StorageCrawler.exe together with the dtrace of the StorageCrawler.exe; eventually also the DVSs being re-indexed at the time of the crash.

The analysis of the above would definitely help in the root cause understanding.

Massimo

tonysterling · Answer

Are you doing collections?&nbsp; I have typically seen that with corrupt .cab files.

Does AV scan your vault store partitions?

james14 · Answer

Yes we are doing collections. We previously had bad cab files, but have just recently restored raw DVS files and got it back on track. I wrote a script that uses the test functionality of 7zip to run through and test every CAB file in the environment, so they should all now be 100%. Sophos exclusions has the entire drive listed not to scan. Currently utilising mount points for storage.

james14 · Answer

I have disabled sophos scanning, and will see how that goes today. Can you recommend and tasks I can run on EV that might trigger this event again whilst sohpos is disabled?

tonysterling · Answer

The event is being triggered by the index process. You can check your indexes to make sure they are all healthy and no missing items for this Archive, 16518AE13DD9FF64F8B266CB373D69EB31110000SSA-EVSITE01

best,

&nbsp;

james14 · Answer

Thanks for your help Tony. I will check that Index now and get back to you.

Forum Discussion

EV Storage Crawler Errors and event ID 1000

12 Replies

Related Content

Storage Crawler Errors

Storage Crawler consuming high CPU

Lots of Storage Crawler Event ID: 6882 and Index Volumes Processor Error 41329

LTO-5 storage error

Storage Error

Recent Discussions

EV Client Reset for MAC book

Attachments in Mails - EnterprsieVault

Enterprise Vault MPIP encrypted email decryption is not working

Defender 365 detecting malware in a CAB/DVSSP file

Broken link