Forum Discussion

james14's avatar
james14
Level 3
11 years ago

EV Storage Crawler Errors and event ID 1000

Hi All,

We recently deployed 2x new EV servers, one file and one mail.

Both are VM's with the following specs:

OS: Server 2012 R2 (Latest patches as of 07/07/2014)

Ram: 16GB

CPU: 8x vProcs

EV Version: 10.0.4.1354

VMware Tools: 9.4.5, build 1734305

Our EV file archiving server has been generating errors around the Storage Crawler process for quite some time now. Symantec support are recommending a re-install of the entire OS, and also EV itself. Before we go down that path, I wanted to get the advice of some of the community members on here first. There must be a better way of resolving this then just blasting the thing and starting all over again. Yes it's a virtual machine, and yes all the config is back on SQL, but that's not really the point.

Below are the errors from the application log and specific Symantec Enterprise Vault Log. When it throws error 28998 for storage crawler, it also appears to throw event id 1000 in application logs. Both are listed below. I have also attached dtrace log when event occured. Dtrace was only against the storage crawler process, happy to re-run dtrace if people require.

Log Name:      Symantec Enterprise Vault

Source:        Enterprise Vault 
Date:          3/09/2014 2:38:48 PM
Event ID:      28998
Task Category: Storage Crawler
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      BNE-EV02.corp.detir.qld.gov.au
Description:
Retrieval of a saveset failed. 
 
Reason: The data is invalid.  (0x8007000d) 
Saveset Id: 201408319519198~201106200139000000~Z~E01A8113B44E1DA1F250EA9485D42421 
Vault Id: 16518AE13DD9FF64F8B266CB373D69EB31110000SSA-EVSITE01 
Index Sequence Number: 66594 
Internal Reference: Fetching Saveset for indexing 

For more information, see Help and Support Center at http://entced.symantec.com/entt?product=ev&language=english&version=10.0.4.0&build=10.0.4.1189&error=V-437-28998
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Enterprise Vault " />
    <EventID Qualifiers="49156">28998</EventID>
    <Level>2</Level>
    <Task>113</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2014-09-03T04:38:48.000000000Z" />
    <EventRecordID>41305</EventRecordID>
    <Channel>Symantec Enterprise Vault</Channel>
    <Computer>BNE-EV02.corp.detir.qld.gov.au</Computer>
    <Security />
  </System>
  <EventData>
    <Data>The data is invalid.  (0x8007000d)</Data>
    <Data>201408319519198~201106200139000000~Z~E01A8113B44E1DA1F250EA9485D42421</Data>
    <Data>16518AE13DD9FF64F8B266CB373D69EB31110000SSA-EVSITE01</Data>
    <Data>66594</Data>
    <Data>Fetching Saveset for indexing</Data>
  </EventData>
</Event>

Log Name:      Application

Source:        Application Error
Date:          3/09/2014 4:21:34 PM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      BNE-EV02.corp.detir.qld.gov.au
Description:
Faulting application name: StorageCrawler.exe, version: 10.0.4.1189, time stamp: 0x51dd972f
Faulting module name: MSVCR90.dll, version: 9.0.30729.8387, time stamp: 0x51ea24a5
Exception code: 0xc0000417
Fault offset: 0x00059e63
Faulting process id: 0x3cc
Faulting application start time: 0x01cfc73f4e985040
Faulting application path: D:\Program Files (x86)\Enterprise Vault\StorageCrawler.exe
Faulting module path: C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.8387_none_5094ca96bcb6b2bb\MSVCR90.dll
Report Id: 8d48bc51-3332-11e4-80d4-005056a9664b
Faulting package full name: 
Faulting package-relative application ID: 
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2014-09-03T06:21:34.000000000Z" />
    <EventRecordID>191934</EventRecordID>
    <Channel>Application</Channel>
    <Computer>BNE-EV02.corp.detir.qld.gov.au</Computer>
    <Security />
  </System>
  <EventData>
    <Data>StorageCrawler.exe</Data>
    <Data>10.0.4.1189</Data>
    <Data>51dd972f</Data>
    <Data>MSVCR90.dll</Data>
    <Data>9.0.30729.8387</Data>
    <Data>51ea24a5</Data>
    <Data>c0000417</Data>
    <Data>00059e63</Data>
    <Data>3cc</Data>
    <Data>01cfc73f4e985040</Data>
    <Data>D:\Program Files (x86)\Enterprise Vault\StorageCrawler.exe</Data>
    <Data>C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.8387_none_5094ca96bcb6b2bb\MSVCR90.dll</Data>
    <Data>8d48bc51-3332-11e4-80d4-005056a9664b</Data>
    <Data>
    </Data>
    <Data>
    </Data>
  </EventData>
</Event>

 

 

Enterprise Vault
Information Governance
Windows Server (2003-2008)
  • MassimoVagli's avatar
    MassimoVagli
    11 years ago

    Hello James, 

    I think the best action would be for you to open a support case, this because of the complexity of the troubleshooting and the kind of data analysis.

    Support would collect and analyze a memory dump of the StorageCrawler.exe together with the dtrace of the StorageCrawler.exe; eventually also the DVSs being re-indexed at the time of the crash.

    The analysis of the above would definitely help in the root cause understanding.

    Massimo

  • TonySterling's avatar
    TonySterling
    Moderator
    11 years ago

    Are you doing collections?  I have typically seen that with corrupt .cab files.

    Does AV scan your vault store partitions?

  • james14's avatar
    james14
    Level 3
    11 years ago

    Yes we are doing collections. We previously had bad cab files, but have just recently restored raw DVS files and got it back on track. I wrote a script that uses the test functionality of 7zip to run through and test every CAB file in the environment, so they should all now be 100%. Sophos exclusions has the entire drive listed not to scan. Currently utilising mount points for storage.

  • james14's avatar
    james14
    Level 3
    11 years ago

    I have disabled sophos scanning, and will see how that goes today. Can you recommend and tasks I can run on EV that might trigger this event again whilst sohpos is disabled?

  • TonySterling's avatar
    TonySterling
    Moderator
    11 years ago

    The event is being triggered by the index process.  You can check your indexes to make sure they are all healthy and no missing items for this Archive, 16518AE13DD9FF64F8B266CB373D69EB31110000SSA-EVSITE01 

    best,

     

  • james14's avatar
    james14
    Level 3
    11 years ago

    Thanks for your help Tony. I will check that Index now and get back to you.

  • james14's avatar
    james14
    Level 3
    11 years ago

    Also getting this error Tony when we re-run all checks.

  • TonySterling's avatar
    TonySterling
    Moderator
    11 years ago

    That can be related.  You  will need to verify your indexes as you won't be able to find items via search until they are indexed.

  • james14's avatar
    james14
    Level 3
    11 years ago

    Hi Tony,

    I did a quick verify for index volumes and they all came back fine?

  • james14's avatar
    james14
    Level 3
    11 years ago

    I have attached another dtrace log file. Can someone please take a look?

    I have managed to get rid of all the errors under symantec logs by doing plain txt conversion instead of html which appears to have helped. Our users never use explorer for file previewing so no point doing html conversion when it's faster and we save space doing plain txt.

    So now I am not getting any more sandbox errors under symantec logs, but still getting error 1000 application logs listed. I was thinking about re-install the C++ librarys?

    Thoughts?

    Faulting application name: StorageCrawler.exe, version: 10.0.4.1189, time stamp: 0x51dd972f

    Faulting module name: MSVCR90.dll, version: 9.0.30729.8387, time stamp: 0x51ea24a5
    Exception code: 0xc0000417
    Fault offset: 0x00059e63
    Faulting process id: 0x443c
    Faulting application start time: 0x01cfcd87fa804b1e
    Faulting application path: D:\Program Files (x86)\Enterprise Vault\StorageCrawler.exe
    Faulting module path: C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.8387_none_5094ca96bcb6b2bb\MSVCR90.dll
    Report Id: 38e20949-397b-11e4-80d5-005056a9664b
    Faulting package full name: 
    Faulting package-relative application ID: 

  • MassimoVagli's avatar
    MassimoVagli
    Level 4
    11 years ago

    Hello James, 

    I think the best action would be for you to open a support case, this because of the complexity of the troubleshooting and the kind of data analysis.

    Support would collect and analyze a memory dump of the StorageCrawler.exe together with the dtrace of the StorageCrawler.exe; eventually also the DVSs being re-indexed at the time of the crash.

    The analysis of the above would definitely help in the root cause understanding.

    Massimo

    • Biswadev's avatar
      Biswadev
      Level 1
      7 years ago

      Hi,

      Getting below error.

      Faulting application name: StorageCrawler.exe, version: 12.3.0.1365, time stamp: 0x5aae7b2b
      Faulting module name: MSVCR120.dll, version: 12.0.21005.1, time stamp: 0x524f7ce6
      Exception code: 0xc0000409
      Fault offset: 0x000a46a9
      Faulting process id: 0x36a4
      Faulting application start time: 0x01d46625a34c14fd
      Faulting application path: E:\Program Files (x86)\Enterprise Vault\StorageCrawler.exe
      Faulting module path: C:\Windows\SYSTEM32\MSVCR120.dll
      Report Id: e1a44f73-d218-11e8-80d8-901b0ef536cf
      Faulting package full name:
      Faulting package-relative application ID:

       

      • GertjanA's avatar
        GertjanA
        Moderator
        7 years ago

        Hello Biswadev,

        Please open a new thread. Include your EV version, OS, and more relevant info. People do not look at old closed discussions.