cancel
Showing results for 
Search instead for 
Did you mean: 

EV 11.0.1 - DCOM errors since May2020 security patches

Ampersound
Level 3

Hi All,

I've got an issue since this weekend when our 16x EV 11.0.1 servers were patched with the May 2020 security patches then rebooted. Now every server is now complaining with Event ID: 29014 issues in the Eventlogs and our users now cannot access their archives via the ArchiveExplorer (website). We migrated our mail to O365 so archive explorer is now our users only method to access their legacy archives.

Log Name: Symantec Enterprise Vault
Source: Enterprise Vault
Date: 26/05/2020 08:40:45
Event ID: 29014
Task Category: Web Application (WP)
Level: Error
Keywords: Classic
User: N/A
Computer: SERVERNAME.CHANGEDTXT.COM
Description:
Storage DCOM error.
Reason: Access is denied. (0x80070005)
Reference: Get storage object: Computer name [SERVERNAME.CHANGEDTXT.COM ]

For more information, see Help and Support Center at http://entced.symantec.com/entt?product=ev&language=english&version=11.0.1.0&build=11.0.1.3706&error...

On one server I uninstalled KB4556852 & KB4558640 rebooted in the hope that it would resolve it but no joy.

I've run a DTRACE (attached) and here is an excerpt of that...

29 13:38:53.549 [6804] (w3wp) <7608> EV-L {ListArchives.Page_Load} ListArchives.Page_Load
30 13:38:53.550 [6804] (w3wp) <7608> EV:L {CAutoJournalAccessor::GetSyncSlot} (Entry)
31 13:38:53.553 [6804] (w3wp) <7608> EV:H {CAutoJournalAccessor::GetSyncSlot:#56} _com_error exception: [Access is denied. (0x80070005)]
32 13:38:53.554 [6804] (w3wp) <7608> EV:H {CAutoJournalAccessor::GetSyncSlot} (Exit) Status: [Access is denied. (0x80070005)]
33 13:38:53.560 [6804] (w3wp) <7608> EV-H {ListArchives.Page_Load} Exception: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)) Info: Diag: Type:System.UnauthorizedAccessException ST: at KVS.EnterpriseVault.Interop.AutoJournalAccessorClass.GetSyncSlot(String auth, String archiveID, UInt32& timeOut)| at DesktopClientCacheWeb.ListArchives.GetSyncSlot(IAutoJournalAccessor ja)| at DesktopClientCacheWeb.ListArchives.List()| at DesktopClientCacheWeb.ListArchives.Page_Load(Object sender, EventArgs args) Inner:None
34 13:38:53.611 [6804] (w3wp) <7120> EV-L {ClientDiagnostics.Page_Load} ClientDiagnostics called
35 13:38:53.611 [6804] (w3wp) <7120> EV-H {ClientDiagnostics.Page_Load} AUTH_USER string: .
36 13:38:53.612 [6804] (w3wp) <7120> EV:L {CClientAuthenticate::GenAuthString:#361} Generating auth string...
39 13:38:53.621 [6804] (w3wp) <7120> EV:M ClientAuthHelperImpl::GenAuthString Authentication Type: Currently impersonated user Client:(null) ==> AuthToken:SERVERNAME.CHANGEDTXT.COM 3Q1K*****
40 13:38:53.627 [6804] (w3wp) <7120> EV-L {ClientDiagnostics.PostDiagnosticValues} Exception: Failed to connect to an IPC Port: The system cannot find the file specified.| Info:ClientDiagnostics PostDiagnosticValues failed Diag: Type:System.Runtime.Remoting.RemotingException ST:|Server stack trace: | at System.Runtime.Remoting.Channels.Ipc.IpcPort.Connect(String portName, Boolean secure, TokenImpersonationLevel impersonationLevel, Int32 timeout)| at System.Runtime.Remoting.Channels.Ipc.ConnectionCache.GetConnection(String portName, Boolean secure, TokenImpersonationLevel level, Int32 timeout)| at System.Runtime.Remoting.Channels.Ipc.IpcClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)| at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)|Exception rethrown at [0]: | at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)| at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)| at KVS.EnterpriseVault.ClientDiagnosticsHandler.ClientDiagnosticsHandler.PostClientDiagnosticValues(NameValueCollection diagnosticValues, String authStr)| at DesktopWeb.ClientDiagnostics.PostDiagnosticValues(NameValueCollection queryStringParams) Inner:None
45 13:39:08.270 [6804] (w3wp) <7608> EV-L {Slot.Page_Load} Slot.Page_Load
46 13:39:08.270 [6804] (w3wp) <7608> EV-L {Slot.GetSyncSlot} Slot.GetSyncSlot - VEID:1D67ECF7CDE3DD3418AF7F977957226191110000evault2, TimeOut:0
47 13:39:08.271 [6804] (w3wp) <7608> EV:L {CAutoJournalAccessor::GetSyncSlot} (Entry)
49 13:39:08.272 [6804] (w3wp) <7608> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 40]
50 13:39:08.274 [6804] (w3wp) <7608> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Successfully communicated with an EV Directory Service on the local machine
51 13:39:08.322 [6804] (w3wp) <7608> EV:L {VaultCoCreateInstanceEx} CLSID [{4EC6FF76-C97A-11D1-90E0-0000F879BE6A}] Server Name [(null)] Used Server Name [(null)] Num of attempts [1] Total elapsed [0.000s] Result [Success (0)]
52 13:39:08.323 [6804] (w3wp) <7608> EV:L {GetStorageObject:#46} Calling VaultCoCreateInstanceEx
53 13:39:08.323 [6804] (w3wp) <7608> EV:L {CStorageOnlineOpnsInstanceHelper::GetLoadBalancedStorageOnlineOpnsCLSID:#61} Non-StorageOnlineOpns CLSID. Returning Input.
54 13:39:08.332 [6804] (w3wp) <7608> EV:L {VaultCoCreateInstanceEx} Attempt [1] to create COM object failed. CLSID [{957FF4B4-162B-4708-843A-0134868699B4}] Server Name [EXC7.CHANGEDTXT.COM] Elapsed [0.009s] Result [Access is denied. (0x80070005)]

I'm at a loss where to go now.

Any help would be appreciated please as I've been trying to keep these servers going whilst waiting for a project to migrate our Databases over to a SQL Server 2012 (or higher) so I can get our environment upgraded to V12.

Thanks,

Andy

1 ACCEPTED SOLUTION

Accepted Solutions

Morning All,

I thought I'd update this post and close it off.

So the issue lasted an entire week until the scheduled reboots for the EV servers early yesterday morning. The DCOM errors stopped immediately once the servers came up. I hadn't made any changes other changes as per listed above.

The only thing I can think of is that the SQL Server farm we utilise for EV performed its monthly reboot on Sunday and after the EV servers reconnected post reboot, it seems everything is working again.

Due to the DCOM issues, I think they sent me on a wild goose chase looking at IIS and EV.. not the SQL Servers.

I'll have to see what I can find out with the SQL servers (as they're looked after by a different team).

At least its fully operational again now.

Thanks,

Andy

View solution in original post

9 REPLIES 9

plaudone1
Level 6
Employee
Hi Andy,
Did the updates make any changes to IIS as there are exceptions on vault cache operations and search operations which go through IIS. Does the VSA work on search or AE page?

If you search support page for “v-437-29014” in quotes it will return articles with that event ID.

Regards,
Patrick

Hi Patrick,

Thanks for your reply.

The KB articles state that it does update IIS :-

"Security updates to the Microsoft Scripting Engine, Windows App Platform and Frameworks, Windows Input and Composition, Windows Media, Windows Kernel, Windows Core Networking, Windows Peripherals, Internet Information Services, Windows Network Security and Containers, Windows Active Directory, the Microsoft JET Database Engine, Internet Explorer, and Windows Storage and Filesystems."

My access doesn't seem to work in pulling items from EV although as per the screenshot, IIS webpage seems to display the folders of my archive just nothing from within them.

The VSA displays the same error message "failed to perform the search request" error.

I did try re-registering ASP.NET 2 solution found here (on the server after uninstalling the patches) but no joy - https://www.veritas.com/content/support/en_US/article.100009900

Not seens any RPC issues either as in some of the other articles.

I've not checked if any tweaks were made to IIS, I've checked some of the files for any changes and nothing popped out that had changed in years.

I definitely think its more IIS related though.

Regards,

Andy

GertjanA
Moderator
Moderator
Partner    VIP    Accredited Certified

I believe there is an article somewhere on what permissions need to be where, but I could not locate it. Have a look at this one to get you started. https://www.veritas.com/support/en_US/article.100030535

Additionally, verify permissions on the .Net folders. I've seen issues where the full access for the VSA was missing on the 'temporary ASP.NET Files' folders. 

If all else fails, I suggest to first run Deployment Scanner (to find if prereqs are all green). If that is ok, you might want to consider a re-install of the binaries. That would set permissions etc. again on OS level.

Regards. Gertjan

Thanks Gertjan.

First I've tried adding permissions to the temporary ASP.NET Files like you mentioned, it already had Modify access via IIS-USR which I increased to Full. I stopped the Index service and performed an IISReset /restart.

I then accessed AE via IE11 and unlike yesterday when I opened my inbox in the archive, I can see archived items. So that is an improvement. I can search for a specific user and it looks like the search component is now working.

Trouble now is when I try to click on an item to view it, a new tab opens and I get

"Symantec Enterprise Vault - Error
The Enterprise Vault service is not available. "

I've tried the following 

https://www.veritas.com/support/en_US/article.100009900

Ran the poweshell script to check for blocked files. Nothing found. 

used aspnet_regiis -iru to re-register into IIS.. no joy.

https://www.veritas.com/support/en_US/article.100011786

Removing  HTTP Activation and Non-HTTP Activation from .Net3.5 and adding back again - no joy

https://www.veritas.com/support/en_US/article.100030535

All those scenarios checked and look fine - I've reapplied the permissions on the Webapp folder too - no change 

Also I ran DS to see what that found and the only thing suggested was a minor issue pointing to this KB

https://support.microsoft.com/en-gb/help/2803161/net-tcp-does-not-have-an-implementation-of-hostedtr...

Looking at our %windir%\Microsoft.NET\Framework64\v2.0.50727\CONFIG\web.config method 1 in the KB article doesn't exist in there. Not changed it yet as the items in my archive can be seen but I cannot open them now.

Still getting the exact same DCOM errors in the event logs but the W3WP Dtrace looks better initially than yesterdays before it hits the DCOM error...

9 12:15:11.539 [5688] (w3wp) <6644> EV-L {Slot.Page_Load} Slot.Page_Load
10 12:15:11.539 [5688] (w3wp) <6644> EV-L {Slot.GetSyncSlot} Slot.GetSyncSlot - VEID:1D50062C880FD5E4594D5C1FD912DDD2F1110000evault2, TimeOut:0
11 12:15:11.539 [5688] (w3wp) <6644> EV:L {CAutoJournalAccessor::GetSyncSlot} (Entry)
13 12:15:11.550 [5688] (w3wp) <6644> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 40]
14 12:15:11.552 [5688] (w3wp) <6644> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Successfully communicated with an EV Directory Service on the local machine
17 12:15:11.583 [5688] (w3wp) <6644> EV:L {VaultCoCreateInstanceEx} CLSID [{4EC6FF76-C97A-11D1-90E0-0000F879BE6A}] Server Name [(null)] Used Server Name [(null)] Num of attempts [1] Total elapsed [0.000s] Result [Success (0)]
18 12:15:11.584 [5688] (w3wp) <6644> EV:L {GetStorageObject:#46} Calling VaultCoCreateInstanceEx
19 12:15:11.584 [5688] (w3wp) <6644> EV:L {CStorageOnlineOpnsInstanceHelper::GetLoadBalancedStorageOnlineOpnsCLSID:#61} Non-StorageOnlineOpns CLSID. Returning Input.
20 12:15:11.694 [5688] (w3wp) <6644> EV:L {VaultCoCreateInstanceEx} Attempt [1] to create COM object failed. CLSID [{957FF4B4-162B-4708-843A-0134868699B4}] Server Name [hostnamet] Elapsed [0.110s] Result [Access is denied. (0x80070005)]
21 12:15:11.899 [5688] (w3wp) <6644> EV:L CHostNameHelper: IsLocalMachineExtendedCheck returned [True] for server [hostname].

This component - {957FF4B4-162B-4708-843A-0134868699B4} seems to be the StorageOnlineOpns.JournalAccessor looking through the Registry but cannot find any similar issues on the Support portal.

Your final idea of re-installing the binaries scares the hell out of me. Not sure I would know where to start with that..just install 11.0.1 over the top then upgrade to CHF5 or just run the upgrade again? I've got a funny feeling some of these servers might have been upgraded from V10 to 11 by our previous Evault person before he left.

 

GertjanA
Moderator
Moderator
Partner    VIP    Accredited Certified

Hello again,

I need to do some digging. If you are uncomfortable with reinstalling, then don't. Although it is straight forward, if you're not sure what will happen, it can make things worse instead of assisting. 

In general, 1 EV Site (= one directory database, plus all underneath) can/must run an equal version of EV. You cannot mix 10 with 11 (or even 11.01 with 11.01.05). I'm working with EV version 12.3, coming from EV9. version. No worries there..

In regards to the ASP settings, also check the 3.5 and 4.0 folders, just to be sure.It might even be you need to reregister asp to IIS. See https://www.veritas.com/support/en_US/article.100009900 for W2008. Search google for W2012.

Also look at this one: https://www.veritas.com/support/en_US/article.100006408

Step 1 and 5 might be worth a try.

 

Regards. Gertjan

Prone2Typos
Moderator
Moderator
Partner    VIP    Accredited Certified

you seem capible IMHO and I was thinking Reinstall too. Just because I have chased this deamon too many times. All that said, I also agree that if you are not cozy with the idea  you should not do it.

If you are thinking permission issue you shoudl be able to find it by using Process Monitor and Filtering on Access denied while DTracing (for good measure) and reproducing the issue. These are agressive logs... so start it and stop it then save it and filter it to see what you want.

To me chasing .net permissions on a file level can be challenging or if tinkered with inappropriately may also introduce vulenerability.

 

I have to say ... great work at researching and implementing and providing the results. .... thanks for all the effort..

GertjanA
Moderator
Moderator
Partner    VIP    Accredited Certified

So, re-install. See https://www.veritas.com/support/en_US/article.100025472

That describes reinstall for 'main version', i.e. 11.0.1

In your case as you seem to be on 11.0.1 CHF5, you need to install 11.0.1, do not reboot. Then install CHF5. Then reboot

 

 

Regards. Gertjan

Thanks for all the advice.

Morning All,

I thought I'd update this post and close it off.

So the issue lasted an entire week until the scheduled reboots for the EV servers early yesterday morning. The DCOM errors stopped immediately once the servers came up. I hadn't made any changes other changes as per listed above.

The only thing I can think of is that the SQL Server farm we utilise for EV performed its monthly reboot on Sunday and after the EV servers reconnected post reboot, it seems everything is working again.

Due to the DCOM issues, I think they sent me on a wild goose chase looking at IIS and EV.. not the SQL Servers.

I'll have to see what I can find out with the SQL servers (as they're looked after by a different team).

At least its fully operational again now.

Thanks,

Andy