DCOM Event 10028 flooding System eventlog - EVMonitoring.exe

[Marcde]

Hello together,

this behavior is showing up on a system since a few years now. System eventlog is showing massiv 10028 events with the following message:

DCOM was unable to communicate with the computer "EVSERVER01&EVHOST=HTTP://EVSERVER01.DOMAIN.NET/EVANON" using any of the configured protocols; requested by PID 1577 (D:\Program Files (x86)\Enterprise Vault\EVMonitoring.exe).

Noticed the strange hostname? Well, we analyzed the EVMonitoring.exe by Dtracing it and it seems that these errors are getting generated by clients which connect to EV through the Netscaler requesting build jobs. We found the following information in the dtrace:

1395 10:33:14.452 [3476] (EVMonitoring) <6692> EV-H {ContentCacheRequestManager.StartBuild} CCRM: Incoming Build request from: Domain\User
1396 10:33:14.452 [3476] (EVMonitoring) <6692> EV-H {ContentCacheRequestManager.GenerateJob} CCRM: No free ContentCacheAssemblers
1397 10:33:14.452 [3476] (EVMonitoring) <6692> EV-H {ContentCacheRequestManager.StartBuild} CCRM: Turning away client Domain\User. (The CCRM is running at full capacity)
1398 10:33:14.452 [3476] (EVMonitoring) <6692> EV-H {ContentCacheRequestManager.RemoveEntry} CCRM: Removing Job: 13E46DE226CBCAC4481309BF584DB02FE1p10000evserver01&evhost=http://evserver01.domain.net/EVAnon
1400 10:33:14.452 [3476] (EVMonitoring) <6692> EV:L {CBaseDirectoryServiceWrapper::CreateDirectoryService} Directory Name [Hostname], Try Local Service [False]
1401 10:33:14.452 [3476] (EVMonitoring) <6692> EV:L {VaultCreateTrustedInstanceRequest::CreateTrustedRemoteInstance} Connected to 'Hostname'. Authorization service [Kerberos] Server Principal Name [dcom/Hostname.domain.net]
1402 10:33:14.452 [3476] (EVMonitoring) <6692> EV:L {VaultCreateInstanceRequest::CreateInstance} CLSID [{F4D3EB5B-C7C5-11D1-90DB-0000F879BE6A} (EnterpriseVault.DirectoryService.1)] Server Name [Hostname] Used Server Name [Hostname] Num of attempts [1] Total elapsed [0.002s] Result [Success (0)]
...
1418 10:33:14.531 [3476] (EVMonitoring) <6692> EV:L {CBaseDirectoryServiceWrapper::CreateDirectoryService} Directory Name [EVSERVER01&EVHOST=HTTP://EVSERVER01.DOMAIN.NET/EVANON], Try Local Service [False]
1419 10:33:14.531 [3476] (EVMonitoring) <6692> EV:L {VaultCreateInstanceRequest::CreateInstance} Attempt [1] to create COM object failed. CLSID [{F4D3EB5B-C7C5-11D1-90DB-0000F879BE6A} (EnterpriseVault.DirectoryService.1)] Server Name [EVSERVER01&EVHOST=HTTP://EVSERVER01.DOMAIN.NET/EVANON] Elapsed [0.002s] Result [No such host is known. (0x80072af9)]

First, why Is the job ID 13E46DE226CBCAC4481309BF584DB02FE1p10000evserver01&evhost=http://evserver01.domain.net/EVAnon and not something like simply: 13E46DE226CBCAC4481309BF584DB02FE1p10000evserver01 ?

As I mentioned, only client requests coming over the Netscaler showing these strange IDs. 

When tracing w3wp in addition I can see the following:

1 10:22:55.332 [35424] (w3wp) <8860> EV-L {GetIncrSlotWithServer.Page_Load} GetIncrSlotWithServer called
2 10:22:55.332 [35424] (w3wp) <8860> EV-H {Proxy.GetRedirectedURL} GetRedirectedURL - Redirecting to EV Server URL http://evserver01.domain.net/EVAnon/GetIncrSlotWithServer.aspx?x=evoutlookext
3 10:22:55.332 [35424] (w3wp) <8860> EV-L {Proxy.IsEvHostRemote} Checking with Application cache whether host is local or not
4 10:22:55.332 [35424] (w3wp) <8860> EV-L {ApplicationCache.IsHostRemote} Already determined that the local machine is hosting the archive
5 10:22:55.332 [35424] (w3wp) <8860> EV-H {Proxy.DoRemoteRequest} Not redirecting as evhost header is same as host

Why is EV doing this? I did not find any configuration in EV which is pointing to the EVANON VD. Is that some kind of a standard behavior when clients (Outlook) connecting from external? (RPC over HTTP)

Vault Cache health in general is ok and the error is occuring for different users. There is a second EV server in this environment which is published by TMG and which is having these strange IDs as well but without the system eventlog getting flooded (None of the Events 10028 on that system). Currently using EV 12.1.1. We did not find a misconfiguration on Netscaler / TMG.

Anyone seen this before and able to answer some of my questions? Are we heading in the right direction? 

Thanks and kind Regards

Marc

[Marcde]

Veitas technical support assisted on that issue.

Following the solution:

On a 64-bit OS: HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\KVS\Enterprise Vault

1. Create a new DWORD called UseLocalDirectory and set the value to 1.

2. Restart the Enterprise Vault Admin Service to make the setting take effect.

The hint was hidden in the following line:

1418 10:33:14.531 [3476] (EVMonitoring) <6692> EV:L {CBaseDirectoryServiceWrapper::CreateDirectoryService} Directory Name [EVSERVER01&EVHOST=HTTP://EVSERVER01.DOMAIN.NET/EVANON], Try Local Service [False]