Forum Discussion

Jakob's avatar
Jakob
Level 5
11 years ago
Solved

Indexing is using to much ressources - EV Services shut down

Hi together,

 

we´re using one single EV Server 10.0.0.4 CH3 for Exchange Mailbox Archiving for ca. 2500 Mailboxes. We´re also migrate Contents from another EV site into the new Archives on the actual affected Server. In general Archiving the Mailboxes is running fine but we have big problems with the indexing. It seems that the indexing services were stucking somewhere. Meanwhile we´ve had about 4,5 Million Elements which were not indexed. In this time the Index Server is using as much memory that the admin service is shutting down because of running out of memory. The EV is an virtual Windows Server 2008 R2 in VMWare. The Server has 32GM RAM dedicated. The Page File is set to 65536MB on Partition D. The SQL Database is installed on a dedicated Server.

We tried to adjust the values for "Maximum concurrent indexing capacity" to 5 and "Indexing memory throttling threshold" to 0.2 but the indexing service was still using to much memory.

Than we changed the value for "Indexing memory throttling threshold" to 0,2 (COMMA ). After that the server didn´t used all resources but the indexing was very slow. We also set it to 1,5 but the the service was shuttong down again. We have installed a english OS but have set the regional settings to german, that´s why I have tried the different seperator. In generall i could say that the Indexing Service was running more stable with a comma seperated value.

The index "queue" was not shrinking at all so i´ve tried to reset the search components as descibed in http://www.symantec.com/business/support/index?page=content&id=TECH201600. After that for about 24 hours the indexing was working fine (with comma seperated ) and shrinked the amount to ~1,5 Million Elements. Than without any known reason it stopped again to work.

Thats it... I´ve made diffent dtrace´s but i didn´t find any conspicous errors in the logs. But - to be honest -  i also don´t know for what i should search in this archiving traces.

In the application logs i´ve found many events of this type:

EventID 41315

A search failed with error "Search engine fault: Unrecognised".

...

 

combinded with event 7182

Index Search failed: Index Search failed: Index: Internal reference: (0xc0041c0e)

...

 

Futhermore i can find the error very often in the dtrace or application a message like:

(EVIndexAdminService) <17056> EV-L {ErrorInfoCollection.LoadMessages} ErrorInfoCollection::LoadMessages() - Message loaded: INCLUDE - LIBSQLITE_CRITICAL MESSAGE DISABLED True  False Overrides:[LIBSQLITE_CRITICAL WARNING LOG True The indexing log database is corrupt. There is no data loss. Enterprise Vault will recover from this problem. True \[.+\\reporting\\system-reporting\\.+\]: \[database disk image is malformed\]]

 

But we don´t have a virus scanner installed which can be responsible for this errors.

Any ideas what i can do next to find the error?

Could there be a problem with the regional settings and the seperator for the Indexing Values?

 

  • Hi,

    our Vendor leaded me to the solution of this issue. There have been single corrupted indexes.

    In the Reporting Files (..\indexmetadata\reporting\system-reporting\...) they found many errors like this one:

    Could not start the [live] [indexer] ([...)\Enterprise Vault\EVIndexing\bin\indexer-service]) in [...\index7\119DD0076B5A49D48997FCEE29D820AF1_1542\live] for collection [119DD0076B5A49D48997FCEE29D820AF1_1542]:  <log ><log ><error time="193130" date="1415746863" id="SERVICE_STARTUP_RECEIVE" >Failed to receive startup confirmation from the service <string name="command" >...\Enterprise Vault\EVIndexing\bin\indexer-service</string> in directory <string name="path" >...\index7\119DD0076B5A49D48997FCEE29D820AF1_1542\live</string>: <string name="error" >An existing connection was forcibly closed by the remote host.

    We´ve renamed the mentioned index folders and the repository.xml and rebuilded the affected user archiv indexes. After this i´ve re-adjusted the values for "Maximum concurrent indexing capacity"and "Indexing memory throttling threshold" to the default values.

    So - a similar error to http://www.symantec.com/business/support/index?page=content&id=TECH213907 but with other effects.

     

  • may  be worth reinstalling the index binaries and reapplying CHF3 to be honest, as for the indexing its quite resource intensive anyway, so when you say too much resources, is it EV that is throwing an event or an alert, or are you just worried that its using the memory you've made available to it ?

  • Hi JesusWept3,

     

    i´ve already reinstalled the index binaries...

    Every time when the ressources getting low because of the indexing processes the following event is comming up:

    EventID 4140

    Source Admin Service

    Insufficient Virtual Memory is available for the Enterprise Vault to run
    Enterprise Vault will now be shut down; to correct this problem, free memory by closing any unwanted applications and restart Enterprise Vault

    For more information, see Help and Support Center at http://entced.symantec.com/entt?product=ev&language=english&version=10.0.4.0&build=10.0.4.1189&error=V-437-4140

    Regards

    Jakob

     

  • Hi,

    no further Ideas?  I assume that there are some Elements in the Indexing "Queue" which lead to this behavior. Can you give me informations how to search in the DTrace for Elements which could stuck in the Indexing Queue?

    Regards

    Jakob

  • Hi,

    our Vendor leaded me to the solution of this issue. There have been single corrupted indexes.

    In the Reporting Files (..\indexmetadata\reporting\system-reporting\...) they found many errors like this one:

    Could not start the [live] [indexer] ([...)\Enterprise Vault\EVIndexing\bin\indexer-service]) in [...\index7\119DD0076B5A49D48997FCEE29D820AF1_1542\live] for collection [119DD0076B5A49D48997FCEE29D820AF1_1542]:  <log ><log ><error time="193130" date="1415746863" id="SERVICE_STARTUP_RECEIVE" >Failed to receive startup confirmation from the service <string name="command" >...\Enterprise Vault\EVIndexing\bin\indexer-service</string> in directory <string name="path" >...\index7\119DD0076B5A49D48997FCEE29D820AF1_1542\live</string>: <string name="error" >An existing connection was forcibly closed by the remote host.

    We´ve renamed the mentioned index folders and the repository.xml and rebuilded the affected user archiv indexes. After this i´ve re-adjusted the values for "Maximum concurrent indexing capacity"and "Indexing memory throttling threshold" to the default values.

    So - a similar error to http://www.symantec.com/business/support/index?page=content&id=TECH213907 but with other effects.