cancel
Showing results for 
Search instead for 
Did you mean: 

Indexing is using to much ressources - EV Services shut down

Jakob
Level 5
Partner

Hi together,

 

we´re using one single EV Server 10.0.0.4 CH3 for Exchange Mailbox Archiving for ca. 2500 Mailboxes. We´re also migrate Contents from another EV site into the new Archives on the actual affected Server. In general Archiving the Mailboxes is running fine but we have big problems with the indexing. It seems that the indexing services were stucking somewhere. Meanwhile we´ve had about 4,5 Million Elements which were not indexed. In this time the Index Server is using as much memory that the admin service is shutting down because of running out of memory. The EV is an virtual Windows Server 2008 R2 in VMWare. The Server has 32GM RAM dedicated. The Page File is set to 65536MB on Partition D. The SQL Database is installed on a dedicated Server.

We tried to adjust the values for "Maximum concurrent indexing capacity" to 5 and "Indexing memory throttling threshold" to 0.2 but the indexing service was still using to much memory.

Than we changed the value for "Indexing memory throttling threshold" to 0,2 (COMMA ). After that the server didn´t used all resources but the indexing was very slow. We also set it to 1,5 but the the service was shuttong down again. We have installed a english OS but have set the regional settings to german, that´s why I have tried the different seperator. In generall i could say that the Indexing Service was running more stable with a comma seperated value.

The index "queue" was not shrinking at all so i´ve tried to reset the search components as descibed in http://www.symantec.com/business/support/index?page=content&id=TECH201600. After that for about 24 hours the indexing was working fine (with comma seperated ) and shrinked the amount to ~1,5 Million Elements. Than without any known reason it stopped again to work.

Thats it... I´ve made diffent dtrace´s but i didn´t find any conspicous errors in the logs. But - to be honest -  i also don´t know for what i should search in this archiving traces.

In the application logs i´ve found many events of this type:

EventID 41315

A search failed with error "Search engine fault: Unrecognised".

...

 

combinded with event 7182

Index Search failed: Index Search failed: Index: Internal reference: (0xc0041c0e)

...

 

Futhermore i can find the error very often in the dtrace or application a message like:

(EVIndexAdminService) <17056> EV-L {ErrorInfoCollection.LoadMessages} ErrorInfoCollection::LoadMessages() - Message loaded: INCLUDE - LIBSQLITE_CRITICAL MESSAGE DISABLED True  False Overrides:[LIBSQLITE_CRITICAL WARNING LOG True The indexing log database is corrupt. There is no data loss. Enterprise Vault will recover from this problem. True \[.+\\reporting\\system-reporting\\.+\]: \[database disk image is malformed\]]

 

But we don´t have a virus scanner installed which can be responsible for this errors.

Any ideas what i can do next to find the error?

Could there be a problem with the regional settings and the seperator for the Indexing Values?

 

1 ACCEPTED SOLUTION

Accepted Solutions

Jakob
Level 5
Partner

Hi,

our Vendor leaded me to the solution of this issue. There have been single corrupted indexes.

In the Reporting Files (..\indexmetadata\reporting\system-reporting\...) they found many errors like this one:

Could not start the [live] [indexer] ([...)\Enterprise Vault\EVIndexing\bin\indexer-service]) in [...\index7\119DD0076B5A49D48997FCEE29D820AF1_1542\live] for collection [119DD0076B5A49D48997FCEE29D820AF1_1542]:  <log ><log ><error time="193130" date="1415746863" id="SERVICE_STARTUP_RECEIVE" >Failed to receive startup confirmation from the service <string name="command" >...\Enterprise Vault\EVIndexing\bin\indexer-service</string> in directory <string name="path" >...\index7\119DD0076B5A49D48997FCEE29D820AF1_1542\live</string>: <string name="error" >An existing connection was forcibly closed by the remote host.

We´ve renamed the mentioned index folders and the repository.xml and rebuilded the affected user archiv indexes. After this i´ve re-adjusted the values for "Maximum concurrent indexing capacity"and "Indexing memory throttling threshold" to the default values.

So - a similar error to http://www.symantec.com/business/support/index?page=content&id=TECH213907 but with other effects.

 

View solution in original post

5 REPLIES 5

JesusWept3
Level 6
Partner Accredited Certified

may  be worth reinstalling the index binaries and reapplying CHF3 to be honest, as for the indexing its quite resource intensive anyway, so when you say too much resources, is it EV that is throwing an event or an alert, or are you just worried that its using the memory you've made available to it ?

https://www.linkedin.com/in/alex-allen-turl-07370146

Jakob
Level 5
Partner

Hi JesusWept3,

 

i´ve already reinstalled the index binaries...

Every time when the ressources getting low because of the indexing processes the following event is comming up:

EventID 4140

Source Admin Service

Insufficient Virtual Memory is available for the Enterprise Vault to run
Enterprise Vault will now be shut down; to correct this problem, free memory by closing any unwanted applications and restart Enterprise Vault

For more information, see Help and Support Center at http://entced.symantec.com/entt?product=ev&language=english&version=10.0.4.0&build=10.0.4.1189&error=V-437-4140

Regards

Jakob

 

Jakob
Level 5
Partner

sorry - double post

Jakob
Level 5
Partner

Hi,

no further Ideas?  I assume that there are some Elements in the Indexing "Queue" which lead to this behavior. Can you give me informations how to search in the DTrace for Elements which could stuck in the Indexing Queue?

Regards

Jakob

Jakob
Level 5
Partner

Hi,

our Vendor leaded me to the solution of this issue. There have been single corrupted indexes.

In the Reporting Files (..\indexmetadata\reporting\system-reporting\...) they found many errors like this one:

Could not start the [live] [indexer] ([...)\Enterprise Vault\EVIndexing\bin\indexer-service]) in [...\index7\119DD0076B5A49D48997FCEE29D820AF1_1542\live] for collection [119DD0076B5A49D48997FCEE29D820AF1_1542]:  <log ><log ><error time="193130" date="1415746863" id="SERVICE_STARTUP_RECEIVE" >Failed to receive startup confirmation from the service <string name="command" >...\Enterprise Vault\EVIndexing\bin\indexer-service</string> in directory <string name="path" >...\index7\119DD0076B5A49D48997FCEE29D820AF1_1542\live</string>: <string name="error" >An existing connection was forcibly closed by the remote host.

We´ve renamed the mentioned index folders and the repository.xml and rebuilded the affected user archiv indexes. After this i´ve re-adjusted the values for "Maximum concurrent indexing capacity"and "Indexing memory throttling threshold" to the default values.

So - a similar error to http://www.symantec.com/business/support/index?page=content&id=TECH213907 but with other effects.