Event 41312

[rld]

Hello,

I'm working on a freshly installed EV server. Mailbox archiving was working a few days ago and today I noticed the Indexing service was stopped. When I try to start the service I get the error 41312 in the event log. I tried to reboot the server and to move the EV_Cache to another folder but I'm still not able to start the Indexing service.

Enteprise Vault 10.0.1.1171

Windows Server 2008 R2

Exchange 2010 SP2 UR3

Index service were not able to start and it throws an event 41312 error in the event viewer

Log Name: Symantec Enterprise Vault

Source: Enterprise Vault

Event ID: 41312

Level: Error

Task Category: Index Admin Service

Description:

The Indexing Service could not complete all the required startup routines: Could not obtain the service entry identifier for this service. Service will be stopped.

Any clue?

[LCT]

Are you able to dtrace the process when you start the indexing service? Perhaps the dtrace will show more details to why it is failing.

[LCT]

Check the dtrace and look in the dtrace for errors and failures:

You already have seen this:

http://www.symantec.com/business/support/index?page=content&id=TECH171108

But one or more of the following could be causing your issue:

http://www.symantec.com/business/support/index?page=content&id=TECH169002

http://www.symantec.com/business/support/index?page=content&id=TECH168789

http://www.symantec.com/business/support/index?page=content&id=TECH184636

Hope that helps.

[rld]

Hello,

Thanks for your reply.

Here the dtrace log on Index Admin Service:

132 16:48:25.046  [6908] (EVIndexAdminService) <6716> EV-M {StateManager.AddVSAToIIS_IUSRSGroup} User s-ev01 already exists in group IIS_IUSRS
133 16:48:25.047  [6908] (EVIndexAdminService) <6716> EV-L {StateManager.IsStartedFromSCMAndIsCluster} Checking if service is started from SCM in cluster environment.
134 16:48:25.049  [6908] (EVIndexAdminService) <6716> EV-L {StateManager.IsDBAvailable} Synchronising with Directory Service.
135 16:48:25.049  [6908] (EVIndexAdminService) <6716> EV-L {StateManager.IsDBAvailable} Waiting for response from Directory Service.
136 16:48:25.100  [6908] (EVIndexAdminService) <6716> EV-L {DirectorySync.IsDirectoryAvailable} IsDirectoryAvailable()
137 16:48:25.100  [6908] (EVIndexAdminService) <6716> EV-L {DirectorySync.IsDirectoryAvailable} Try to create Directory Service locally first.
138 16:48:25.102  [6908] (EVIndexAdminService) <6716> EV-L {DirectorySync.IsDirectoryAvailable} Local creation for DirectoryService succeeded.
139 16:48:25.102  [6908] (EVIndexAdminService) <6716> EV-M {DirectorySync.IsDirectoryAvailable} IsDirectoryAvailable() - Check if config wizard has being run
140 16:48:25.116  [6908] (EVIndexAdminService) <6716> EV-M {DirectorySync.IsDirectoryAvailable} IsDirectoryAvailable() - Releasing Directory Public Service interface
141 16:48:25.116  [6908] (EVIndexAdminService) <6716> EV-M {DirectorySync.IsDirectoryAvailable} IsDirectoryAvailable() - Directory Service is available
142 16:48:25.117  [6908] (EVIndexAdminService) <6716> EV-L {StateManager.IsDBAvailable} Synchronising with Directory Service succeeded.
145 16:48:25.216  [6908] (EVIndexAdminService) <6716> EV-M {StateManager.GetExtendedSetting} Could not read server extended setting. Using default value for MaximumChildProcessShutdownTime.
146 16:48:25.217  [6908] (EVIndexAdminService) <6716> EV-M {StateManager.GetExtendedSetting} Could not read server extended setting. Using default value for MaximumIndexingEngineShutdownTime.
147 16:48:25.218  [6908] (EVIndexAdminService) <6716> EV-M {StateManager.GetExtendedSetting} Could not read server extended setting. Using default value for MaximumChildProcessStartupTime.
148 16:48:25.218  [6908] (EVIndexAdminService) <6716> EV-M {StateManager.GetExtendedSetting} Could not read server extended setting. Using default value for MaximumChildProcessShutdownTime.
149 16:48:25.219  [6908] (EVIndexAdminService) <6716> EV-M {StateManager.GetExtendedSetting} Could not read server extended setting. Using default value for MaximumIndexingEngineStartupTime.
150 16:48:25.220  [6908] (EVIndexAdminService) <6716> EV-M {StateManager.GetExtendedSetting} Could not read server extended setting. Using default value for MaximumIndexingEngineShutdownTime.
151 16:48:25.221  [6908] (EVIndexAdminService) <6716> EV-M {StateManager.GetExtendedSetting} Could not read server extended setting. Using default value for MaximumIndexingApplicationPoolStartTime.
152 16:48:25.236  [6908] (EVIndexAdminService) <6716> EV-H {EVIndexAdminUtils.GetServiceEntryIdFromComputerName} Exception: Retrieving the COM class factory for component with CLSID {4EC6FF78-C97A-11D1-90E0-0000F879BE6A} failed due to the following error: 8007007e. Info: Diag: Type:System.IO.FileNotFoundException ST:   at KVS.EnterpriseVault.Common.Computer.ReadComputerInformation()|   at Symantec.EnterpriseVault.Indexing.Admin.EVIndexAdminUtils.GetServiceEntryIdFromComputerName() Inner:None
153 16:48:25.236  [6908] (EVIndexAdminService) <6716> EV-H {IndexAdminService.OnPerformStartup} The Indexing Service could not complete all the required startup routines: Could not obtain the service entry identifier for this service. Service will be stopped.
154 16:48:25.261  [6908] (EVIndexAdminService) <6716> EV~E |Event ID: 41312 The Indexing Service could not complete all the required startup routines: Could not obtain the service entry identifier for this service. Service will be stopped.
155 16:48:25.262  [6908] (EVIndexAdminService) <6716> EV-H {ServiceBaseEx.Run} Exception of type 'KVS.EnterpriseVault.Common.ServiceBaseExException' was thrown.
156 16:48:25.262  [6908] (EVIndexAdminService) <6716> EV-L {ServiceBaseEx.Run} Executing OnPerformShutdown()
157 16:48:25.264  [6908] (EVIndexAdminService) <6716> EV-L {IndexAdminService.OnPerformShutdown} Starting Index Admin Service shutdown routines in a background thread
158 16:48:25.264  [6908] (EVIndexAdminService) <6716> EV-L {IndexAdminService.OnPerformShutdown} Miscellaneous indexing service shutdown tasks started
159 16:48:25.266  [6908] (EVIndexAdminService) <6716> EV-H {EVIndexAdminUtils.InvokeHelper} Calling Void <OnPerformShutdown>b__16(). Reference: Stopping StorageRevisionDiscovery
160 16:48:25.266  [6908] (EVIndexAdminService) <6716> EV-H {StoragePendingWorkDiscovery.Stop} StoragePendingWorkDiscovery stopping
161 16:48:25.266  [6908] (EVIndexAdminService) <6716> EV-L {IndexAdminService.OnPerformShutdown} Miscellaneous indexing service shutdown tasks completed (1.9534 ms)
162 16:48:25.266  [6908] (EVIndexAdminService) <6716> EV-L {IndexAdminService.OnPerformShutdown} Stop child processes and unregister indexing service endpoints started
163 16:48:25.267  [6908] (EVIndexAdminService) <6716> EV-L {StateManager.UnregisterForDCOMClient} Inside UnregisterForDCOMClient
164 16:48:25.269  [6908] (EVIndexAdminService) <6716> EV-L {StateManager.StopChildProcesses} Stopping child processes of Index Admin Service.
167 16:48:25.271  [6908] (EVIndexAdminService) <6716> EV-L {StateManager.StopChildProcesses} Stopping child processes: {EVIndexVolumesProcessor,EVIndexQueryServer}
168 16:48:25.273  [6908] (EVIndexAdminService) <6716> EV-L {ChildProcessManager.StopChildProcesses} Stopping child processes of Indexing Service [Max shutdown time: 900000 ms].
169 16:48:25.273  [6908] (EVIndexAdminService) <6716> EV-L {ChildProcessManager.StopChildProcesses} Stopping child process 'EVIndexVolumesProcessor'
170 16:48:25.273  [6908] (EVIndexAdminService) <6716> EV-L {ChildProcessManager.StopChildProcesses} Stopping child process 'EVIndexQueryServer'
171 16:48:25.273  [6908] (EVIndexAdminService) <6716> EV-L {ChildProcessManager.StopChildProcesses} Waiting for all child process '{EVIndexVolumesProcessor,EVIndexQueryServer}' to stop.
172 16:48:25.275  [6908] (EVIndexAdminService) <5400> EV-L {ChildProcessManager.AsyncStopChildProcess} Inside AsyncStopChildProcess to stop child process EVIndexQueryServer of Indexing Service [Max shutdown time: 900000 ms].
173 16:48:25.275  [6908] (EVIndexAdminService) <5236> EV-L {ChildProcessManager.AsyncStopChildProcess} Inside AsyncStopChildProcess to stop child process EVIndexVolumesProcessor of Indexing Service [Max shutdown time: 900000 ms].
174 16:48:25.277  [6908] (EVIndexAdminService) <5400> EV-L {ChildProcessManager.StopChildProcess} Stopping child process EVIndexQueryServer of Indexing Service [Max shutdown time: 900000 ms].
175 16:48:25.277  [6908] (EVIndexAdminService) <5236> EV-L {ChildProcessManager.StopChildProcess} Stopping child process EVIndexVolumesProcessor of Indexing Service [Max shutdown time: 900000 ms].
176 16:48:25.278  [6908] (EVIndexAdminService) <5400> EV-L {ChildProcessManager.StopChildProcess} Child process EVIndexQueryServer already not running or not reachable. So Stop not actioned.
177 16:48:25.278  [6908] (EVIndexAdminService) <5236> EV-L {ChildProcessManager.StopChildProcess} Child process EVIndexVolumesProcessor already not running or not reachable. So Stop not actioned.
178 16:48:25.279  [6908] (EVIndexAdminService) <6716> EV-L {IndexAdminService.OnPerformShutdown} Stop child processes and unregister indexing service endpoints completed (12.6971 ms)
179 16:48:25.279  [6908] (EVIndexAdminService) <6716> EV-L {IndexAdminService.OnPerformShutdown} Stop indexing engine started
180 16:48:25.279  [6908] (EVIndexAdminService) <6716> EV-H {EVIndexAdminUtils.InvokeHelper} Calling Void <OnPerformShutdown>b__17(). Reference: Stopping VelocityProcessPriorityMonitor
181 16:48:25.280  [6908] (EVIndexAdminService) <6716> EV-M {VelocityProcessHealthMonitor.Stop} VelocityProcessPriorityMonitor stopping
182 16:48:25.280  [6908] (EVIndexAdminService) <6716> EV-M {VelocityProcessHealthMonitor.Stop} VelocityHealthMonitor stopping
183 16:48:25.280  [6908] (EVIndexAdminService) <6716> EV-L {IndexAdminService.OnPerformShutdown} Stop indexing engine completed (0.9767 ms)
184 16:48:25.280  [6908] (EVIndexAdminService) <6716> EV-L {IndexAdminService.OnPerformShutdown} Completed Index Admin Service shutdown routines in a background thread
185 16:48:25.284  [6908] (EVIndexAdminService) <6716> EV-L {ServiceBaseEx.OnStop} Stopping Indexing Service ...
186 16:48:25.284  [6908] (EVIndexAdminService) <6716> EV-H {ServiceBaseEx.OnStop} Indexing Service OnStop called from ServiceThread as a result of error handling.
187 16:48:25.285  [6908] (EVIndexAdminService) <6716> EV~I |Event ID: 41298 The Indexing Service has stopped (10.0.1.1171).

[LCT]

152 16:48:25.236  [6908] (EVIndexAdminService) <6716> EV-H {EVIndexAdminUtils.GetServiceEntryIdFromComputerName} Exception: Retrieving the COM class factory for component with CLSID {4EC6FF78-C97A-11D1-90E0-0000F879BE6A} failed due to the following error: 8007007e. Info: Diag: Type:System.IO.FileNotFoundException ST:   at KVS.EnterpriseVault.Common.Computer.ReadComputerInformation()|   at Symantec.EnterpriseVault.Indexing.Admin.EVIndexAdminUtils.GetServiceEntryIdFromComputerName() Inner:None
 

Looks like you have a DCOM issue on your EV server. Have you got any errors on your Event logs on the Ev server, check Applications and Systems etc.

Have you got Windows Firewall enabled on your EV server? If you do, turn it off.

[rld]

Firewall is off but I do have some DistributedCOM errors in the System log, Event ID: 10016:

The application-specific permission settings do not grant Local Launch permission for the COM Server application with CLSID

{F4D3EB5B-C7C5-11D1-90DB-0000F879BE6A}

and APPID

{95DCB63F-C982-11D1-90E0-0000F879BE6A}

to the user <DOMAIN>\s-ev01 SID (S-1-5-21-1206409372-228415166-929701000-5424) from address LocalHost (Using LRPC). This security permission can be modified using the Component Services administrative tool.

But these errors were already there when the server was working properly and I found in the forums that these errors can be ignored if they appear when the Admin service starts.

[LCT]

I would log this issue with Tech Support as it's definitely causing the issue that you are seeing. The thread number is the same [6908] (EVIndexAdminService) <6716> all the way through and the DCOM error occurs just before the error that you see regarding the Indexing Service failed to start.

[Rob_Wilcox1]

Looking at that snippet of trace, I'd suggest that there might be an issue with WMI...

Have a look for some really simple WMI scripts which will give you things like the computer name of the host you run the script on.

Do they work on this server?  Or do they fail, but work on other servers?

[rld]

I did several WMI queries and it works.

[Rob_Wilcox1]

ok - can you elaborate?  Share them with us?  I can try them here, or maybe adapt them, suggest some changes... or you'll need to go off to SYMC support with the errors you've got already.

[rld]

I've added permissions for the EV service account on the DCOM object called "Enterprise Vault Task Controller Service" and DCOM errors have disappeared.$

But I'm still not able to start the Indexing service.

I did a new Dtrace afterwards (4EC6FF78-C97A-11D1-90E0-0000F879BE6A = Enterprise Vault Directory Connection)

43 10:12:09.112  [6228] (EVIndexAdminService) <688> EV-L {StateManager.IsDBAvailable} Synchronising with Directory Service succeeded.
44 10:12:09.303  [6228] (EVIndexAdminService) <688> EV-M {StateManager.GetExtendedSetting} Could not read server extended setting. Using default value for MaximumChildProcessShutdownTime.
45 10:12:09.304  [6228] (EVIndexAdminService) <688> EV-M {StateManager.GetExtendedSetting} Could not read server extended setting. Using default value for MaximumIndexingEngineShutdownTime.
46 10:12:09.305  [6228] (EVIndexAdminService) <688> EV-M {StateManager.GetExtendedSetting} Could not read server extended setting. Using default value for MaximumChildProcessStartupTime.
47 10:12:09.306  [6228] (EVIndexAdminService) <688> EV-M {StateManager.GetExtendedSetting} Could not read server extended setting. Using default value for MaximumChildProcessShutdownTime.
48 10:12:09.308  [6228] (EVIndexAdminService) <688> EV-M {StateManager.GetExtendedSetting} Could not read server extended setting. Using default value for MaximumIndexingEngineStartupTime.
49 10:12:09.309  [6228] (EVIndexAdminService) <688> EV-M {StateManager.GetExtendedSetting} Could not read server extended setting. Using default value for MaximumIndexingEngineShutdownTime.
50 10:12:09.311  [6228] (EVIndexAdminService) <688> EV-M {StateManager.GetExtendedSetting} Could not read server extended setting. Using default value for MaximumIndexingApplicationPoolStartTime.
51 10:12:09.333  [6228] (EVIndexAdminService) <688> EV-H {EVIndexAdminUtils.GetServiceEntryIdFromComputerName} Exception: Retrieving the COM class factory for component with CLSID {4EC6FF78-C97A-11D1-90E0-0000F879BE6A} failed due to the following error: 8007007e. Info: Diag: Type:System.IO.FileNotFoundException ST:   at KVS.EnterpriseVault.Common.Computer.ReadComputerInformation()|   at Symantec.EnterpriseVault.Indexing.Admin.EVIndexAdminUtils.GetServiceEntryIdFromComputerName() Inner:None
52 10:12:09.333  [6228] (EVIndexAdminService) <688> EV-H {IndexAdminService.OnPerformStartup} The Indexing Service could not complete all the required startup routines: Could not obtain the service entry identifier for this service. Service will be stopped.
53 10:12:09.380  [6228] (EVIndexAdminService) <688> EV~E |Event ID: 41312 The Indexing Service could not complete all the required startup routines: Could not obtain the service entry identifier for this service. Service will be stopped.
54 10:12:09.381  [6228] (EVIndexAdminService) <688> EV-H {ServiceBaseEx.Run} Exception of type 'KVS.EnterpriseVault.Common.ServiceBaseExException' was thrown.
55 10:12:09.381  [6228] (EVIndexAdminService) <688> EV-L {ServiceBaseEx.Run} Executing OnPerformShutdown()
56 10:12:09.385  [6228] (EVIndexAdminService) <688> EV-L {IndexAdminService.OnPerformShutdown} Starting Index Admin Service shutdown routines in a background thread
57 10:12:09.385  [6228] (EVIndexAdminService) <688> EV-L {IndexAdminService.OnPerformShutdown} Miscellaneous indexing service shutdown tasks started
58 10:12:09.388  [6228] (EVIndexAdminService) <688> EV-H {EVIndexAdminUtils.InvokeHelper} Calling Void <OnPerformShutdown>b__16(). Reference: Stopping StorageRevisionDiscovery
59 10:12:09.388  [6228] (EVIndexAdminService) <688> EV-H {StoragePendingWorkDiscovery.Stop} StoragePendingWorkDiscovery stopping
60 10:12:09.389  [6228] (EVIndexAdminService) <688> EV-L {IndexAdminService.OnPerformShutdown} Miscellaneous indexing service shutdown tasks completed (2.9298 ms)
61 10:12:09.389  [6228] (EVIndexAdminService) <688> EV-L {IndexAdminService.OnPerformShutdown} Stop child processes and unregister indexing service endpoints started
62 10:12:09.390  [6228] (EVIndexAdminService) <688> EV-L {StateManager.UnregisterForDCOMClient} Inside UnregisterForDCOMClient
63 10:12:09.394  [6228] (EVIndexAdminService) <688> EV-L {StateManager.StopChildProcesses} Stopping child processes of Index Admin Service.
64 10:12:09.397  [6228] (EVIndexAdminService) <688> EV-L {StateManager.StopChildProcesses} Stopping child processes: {EVIndexVolumesProcessor,EVIndexQueryServer}
65 10:12:09.400  [6228] (EVIndexAdminService) <688> EV-L {ChildProcessManager.StopChildProcesses} Stopping child processes of Indexing Service [Max shutdown time: 900000 ms].
66 10:12:09.401  [6228] (EVIndexAdminService) <688> EV-L {ChildProcessManager.StopChildProcesses} Stopping child process 'EVIndexVolumesProcessor'
67 10:12:09.401  [6228] (EVIndexAdminService) <688> EV-L {ChildProcessManager.StopChildProcesses} Stopping child process 'EVIndexQueryServer'
68 10:12:09.401  [6228] (EVIndexAdminService) <688> EV-L {ChildProcessManager.StopChildProcesses} Waiting for all child process '{EVIndexVolumesProcessor,EVIndexQueryServer}' to stop.
69 10:12:09.402  [6228] (EVIndexAdminService) <7028> EV-L {ChildProcessManager.AsyncStopChildProcess} Inside AsyncStopChildProcess to stop child process EVIndexVolumesProcessor of Indexing Service [Max shutdown time: 900000 ms].
70 10:12:09.402  [6228] (EVIndexAdminService) <5764> EV-L {ChildProcessManager.AsyncStopChildProcess} Inside AsyncStopChildProcess to stop child process EVIndexQueryServer of Indexing Service [Max shutdown time: 900000 ms].
71 10:12:09.405  [6228] (EVIndexAdminService) <7028> EV-L {ChildProcessManager.StopChildProcess} Stopping child process EVIndexVolumesProcessor of Indexing Service [Max shutdown time: 900000 ms].
72 10:12:09.405  [6228] (EVIndexAdminService) <5764> EV-L {ChildProcessManager.StopChildProcess} Stopping child process EVIndexQueryServer of Indexing Service [Max shutdown time: 900000 ms].
73 10:12:09.408  [6228] (EVIndexAdminService) <7028> EV-L {ChildProcessManager.StopChildProcess} Child process EVIndexVolumesProcessor already not running or not reachable. So Stop not actioned.
74 10:12:09.408  [6228] (EVIndexAdminService) <5764> EV-L {ChildProcessManager.StopChildProcess} Child process EVIndexQueryServer already not running or not reachable. So Stop not actioned.
75 10:12:09.410  [6228] (EVIndexAdminService) <688> EV-L {IndexAdminService.OnPerformShutdown} Stop child processes and unregister indexing service endpoints completed (20.5086 ms)
76 10:12:09.410  [6228] (EVIndexAdminService) <688> EV-L {IndexAdminService.OnPerformShutdown} Stop indexing engine started
77 10:12:09.410  [6228] (EVIndexAdminService) <688> EV-H {EVIndexAdminUtils.InvokeHelper} Calling Void <OnPerformShutdown>b__17(). Reference: Stopping VelocityProcessPriorityMonitor
78 10:12:09.411  [6228] (EVIndexAdminService) <688> EV-M {VelocityProcessHealthMonitor.Stop} VelocityProcessPriorityMonitor stopping
79 10:12:09.411  [6228] (EVIndexAdminService) <688> EV-M {VelocityProcessHealthMonitor.Stop} VelocityHealthMonitor stopping
80 10:12:09.411  [6228] (EVIndexAdminService) <688> EV-L {IndexAdminService.OnPerformShutdown} Stop indexing engine completed (0.9766 ms)
81 10:12:09.411  [6228] (EVIndexAdminService) <688> EV-L {IndexAdminService.OnPerformShutdown} Completed Index Admin Service shutdown routines in a background thread
82 10:12:09.418  [6228] (EVIndexAdminService) <688> EV-L {ServiceBaseEx.OnStop} Stopping Indexing Service ...
83 10:12:09.418  [6228] (EVIndexAdminService) <688> EV-H {ServiceBaseEx.OnStop} Indexing Service OnStop called from ServiceThread as a result of error handling.
84 10:12:09.420  [6228] (EVIndexAdminService) <688> EV~I |Event ID: 41298 The Indexing Service has stopped (10.0.1.1171).

[rld]

PS C:\Users\s-ev01> gwmi win32_computersystem
Domain              : <domain>
Manufacturer        : VMware, Inc.
Model               : VMware Virtual Platform
Name                : MVEV01
PrimaryOwnerName    : Windows User
TotalPhysicalMemory : 12884434944

[Ksimir]

Hi RLD!

I think in your case, you should have an indexing engine corruption.

There's a KB to reinstall Indexing service: http://www.symantec.com/business/support/index?page=content&id=HOWTO59060

Hope it will solve your issue.

Samir

[rld]

Hi Ksimir,

Sorry it does not solve my issue.

[Ksimir · Accepted solution]

Ok so maybe you should try to reinstall the EV server itself and by keeping the same DB like you migrate on the same computer...

[rld]

Reinstalling is not a real solution but it worked. I'm now able to start the Indexing service.

Thanks all for your support.

[Rob_Wilcox1]

Glad you got it sorted.  I guess something didn't register correctly the first time through the install..  (just a guess)

[ZeRoC00L]

Same issue appeared here, reinstall of the EV binaries also solved this. Strange but true.