Highlighted

EV Storage Crawler Errors and event ID 1000

Hi All,

We recently deployed 2x new EV servers, one file and one mail.

Both are VM's with the following specs:

OS: Server 2012 R2 (Latest patches as of 07/07/2014)

Ram: 16GB

CPU: 8x vProcs

EV Version: 10.0.4.1354

VMware Tools: 9.4.5, build 1734305

Our EV file archiving server has been generating errors around the Storage Crawler process for quite some time now. Symantec support are recommending a re-install of the entire OS, and also EV itself. Before we go down that path, I wanted to get the advice of some of the community members on here first. There must be a better way of resolving this then just blasting the thing and starting all over again. Yes it's a virtual machine, and yes all the config is back on SQL, but that's not really the point.

Below are the errors from the application log and specific Symantec Enterprise Vault Log. When it throws error 28998 for storage crawler, it also appears to throw event id 1000 in application logs. Both are listed below. I have also attached dtrace log when event occured. Dtrace was only against the storage crawler process, happy to re-run dtrace if people require.

Log Name:      Symantec Enterprise Vault

Source:        Enterprise Vault 
Date:          3/09/2014 2:38:48 PM
Event ID:      28998
Task Category: Storage Crawler
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      BNE-EV02.corp.detir.qld.gov.au
Description:
Retrieval of a saveset failed. 
 
Reason: The data is invalid.  (0x8007000d) 
Saveset Id: 201408319519198~201106200139000000~Z~E01A8113B44E1DA1F250EA9485D42421 
Vault Id: 16518AE13DD9FF64F8B266CB373D69EB31110000SSA-EVSITE01 
Index Sequence Number: 66594 
Internal Reference: Fetching Saveset for indexing 

For more information, see Help and Support Center at http://entced.symantec.com/entt?product=ev&language=english&version=10.0.4.0&build=10.0.4.1189&error=V-437-28998
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Enterprise Vault " />
    <EventID Qualifiers="49156">28998</EventID>
    <Level>2</Level>
    <Task>113</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2014-09-03T04:38:48.000000000Z" />
    <EventRecordID>41305</EventRecordID>
    <Channel>Symantec Enterprise Vault</Channel>
    <Computer>BNE-EV02.corp.detir.qld.gov.au</Computer>
    <Security />
  </System>
  <EventData>
    <Data>The data is invalid.  (0x8007000d)</Data>
    <Data>201408319519198~201106200139000000~Z~E01A8113B44E1DA1F250EA9485D42421</Data>
    <Data>16518AE13DD9FF64F8B266CB373D69EB31110000SSA-EVSITE01</Data>
    <Data>66594</Data>
    <Data>Fetching Saveset for indexing</Data>
  </EventData>
</Event>

Log Name:      Application

Source:        Application Error
Date:          3/09/2014 4:21:34 PM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      BNE-EV02.corp.detir.qld.gov.au
Description:
Faulting application name: StorageCrawler.exe, version: 10.0.4.1189, time stamp: 0x51dd972f
Faulting module name: MSVCR90.dll, version: 9.0.30729.8387, time stamp: 0x51ea24a5
Exception code: 0xc0000417
Fault offset: 0x00059e63
Faulting process id: 0x3cc
Faulting application start time: 0x01cfc73f4e985040
Faulting application path: D:\Program Files (x86)\Enterprise Vault\StorageCrawler.exe
Faulting module path: C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.8387_none_5094ca96bcb6b2bb\MSVCR90.dll
Report Id: 8d48bc51-3332-11e4-80d4-005056a9664b
Faulting package full name: 
Faulting package-relative application ID: 
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2014-09-03T06:21:34.000000000Z" />
    <EventRecordID>191934</EventRecordID>
    <Channel>Application</Channel>
    <Computer>BNE-EV02.corp.detir.qld.gov.au</Computer>
    <Security />
  </System>
  <EventData>
    <Data>StorageCrawler.exe</Data>
    <Data>10.0.4.1189</Data>
    <Data>51dd972f</Data>
    <Data>MSVCR90.dll</Data>
    <Data>9.0.30729.8387</Data>
    <Data>51ea24a5</Data>
    <Data>c0000417</Data>
    <Data>00059e63</Data>
    <Data>3cc</Data>
    <Data>01cfc73f4e985040</Data>
    <Data>D:\Program Files (x86)\Enterprise Vault\StorageCrawler.exe</Data>
    <Data>C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.8387_none_5094ca96bcb6b2bb\MSVCR90.dll</Data>
    <Data>8d48bc51-3332-11e4-80d4-005056a9664b</Data>
    <Data>
    </Data>
    <Data>
    </Data>
  </EventData>
</Event>

 

 

1 Solution

Accepted Solutions
Accepted Solution!

Hello James,  I think the

Hello James, 

I think the best action would be for you to open a support case, this because of the complexity of the troubleshooting and the kind of data analysis.

Support would collect and analyze a memory dump of the StorageCrawler.exe together with the dtrace of the StorageCrawler.exe; eventually also the DVSs being re-indexed at the time of the crash.

The analysis of the above would definitely help in the root cause understanding.

Massimo

View solution in original post

12 Replies

Are you doing collections?  I

Are you doing collections?  I have typically seen that with corrupt .cab files.

Does AV scan your vault store partitions?

For the best O365 Reporting Solution click here Radar-Reporting

Yes we are doing collections.

Yes we are doing collections. We previously had bad cab files, but have just recently restored raw DVS files and got it back on track. I wrote a script that uses the test functionality of 7zip to run through and test every CAB file in the environment, so they should all now be 100%. Sophos exclusions has the entire drive listed not to scan. Currently utilising mount points for storage.

I have disabled sophos

I have disabled sophos scanning, and will see how that goes today. Can you recommend and tasks I can run on EV that might trigger this event again whilst sohpos is disabled?

The event is being triggered

The event is being triggered by the index process.  You can check your indexes to make sure they are all healthy and no missing items for this Archive, 16518AE13DD9FF64F8B266CB373D69EB31110000SSA-EVSITE01 

best,

 

For the best O365 Reporting Solution click here Radar-Reporting

Thanks for your help Tony. I

Thanks for your help Tony. I will check that Index now and get back to you.

Also getting this error Tony

Also getting this error Tony when we re-run all checks.

That can be related.  You 

That can be related.  You  will need to verify your indexes as you won't be able to find items via search until they are indexed.

For the best O365 Reporting Solution click here Radar-Reporting

Hi Tony, I did a quick verify

Hi Tony,

I did a quick verify for index volumes and they all came back fine?

I have attached another

I have attached another dtrace log file. Can someone please take a look?

I have managed to get rid of all the errors under symantec logs by doing plain txt conversion instead of html which appears to have helped. Our users never use explorer for file previewing so no point doing html conversion when it's faster and we save space doing plain txt.

So now I am not getting any more sandbox errors under symantec logs, but still getting error 1000 application logs listed. I was thinking about re-install the C++ librarys?

Thoughts?

Faulting application name: StorageCrawler.exe, version: 10.0.4.1189, time stamp: 0x51dd972f

Faulting module name: MSVCR90.dll, version: 9.0.30729.8387, time stamp: 0x51ea24a5
Exception code: 0xc0000417
Fault offset: 0x00059e63
Faulting process id: 0x443c
Faulting application start time: 0x01cfcd87fa804b1e
Faulting application path: D:\Program Files (x86)\Enterprise Vault\StorageCrawler.exe
Faulting module path: C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.8387_none_5094ca96bcb6b2bb\MSVCR90.dll
Report Id: 38e20949-397b-11e4-80d5-005056a9664b
Faulting package full name: 
Faulting package-relative application ID: 

Accepted Solution!

Hello James,  I think the

Hello James, 

I think the best action would be for you to open a support case, this because of the complexity of the troubleshooting and the kind of data analysis.

Support would collect and analyze a memory dump of the StorageCrawler.exe together with the dtrace of the StorageCrawler.exe; eventually also the DVSs being re-indexed at the time of the crash.

The analysis of the above would definitely help in the root cause understanding.

Massimo

View solution in original post

Re: Hello James, I think the

Hi,

Getting below error.

Faulting application name: StorageCrawler.exe, version: 12.3.0.1365, time stamp: 0x5aae7b2b
Faulting module name: MSVCR120.dll, version: 12.0.21005.1, time stamp: 0x524f7ce6
Exception code: 0xc0000409
Fault offset: 0x000a46a9
Faulting process id: 0x36a4
Faulting application start time: 0x01d46625a34c14fd
Faulting application path: E:\Program Files (x86)\Enterprise Vault\StorageCrawler.exe
Faulting module path: C:\Windows\SYSTEM32\MSVCR120.dll
Report Id: e1a44f73-d218-11e8-80d8-901b0ef536cf
Faulting package full name:
Faulting package-relative application ID:

 

Re: Hello James, I think the

Hello Biswadev,

Please open a new thread. Include your EV version, OS, and more relevant info. People do not look at old closed discussions.

Regards. Gertjan