Forum Discussion

Ampersound's avatar
Ampersound
Level 3
5 years ago

EV 11.0.1 - DCOM errors since May2020 security patches

Hi All,

I've got an issue since this weekend when our 16x EV 11.0.1 servers were patched with the May 2020 security patches then rebooted. Now every server is now complaining with Event ID: 29014 issues in the Eventlogs and our users now cannot access their archives via the ArchiveExplorer (website). We migrated our mail to O365 so archive explorer is now our users only method to access their legacy archives.

Log Name: Symantec Enterprise Vault
Source: Enterprise Vault
Date: 26/05/2020 08:40:45
Event ID: 29014
Task Category: Web Application (WP)
Level: Error
Keywords: Classic
User: N/A
Computer: SERVERNAME.CHANGEDTXT.COM
Description:
Storage DCOM error.
Reason: Access is denied. (0x80070005)
Reference: Get storage object: Computer name [SERVERNAME.CHANGEDTXT.COM ]

For more information, see Help and Support Center at http://entced.symantec.com/entt?product=ev&language=english&version=11.0.1.0&build=11.0.1.3706&error=V-437-29014

On one server I uninstalled KB4556852 & KB4558640 rebooted in the hope that it would resolve it but no joy.

I've run a DTRACE (attached) and here is an excerpt of that...

29 13:38:53.549 [6804] (w3wp) <7608> EV-L {ListArchives.Page_Load} ListArchives.Page_Load
30 13:38:53.550 [6804] (w3wp) <7608> EV:L {CAutoJournalAccessor::GetSyncSlot} (Entry)
31 13:38:53.553 [6804] (w3wp) <7608> EV:H {CAutoJournalAccessor::GetSyncSlot:#56} _com_error exception: [Access is denied. (0x80070005)]
32 13:38:53.554 [6804] (w3wp) <7608> EV:H {CAutoJournalAccessor::GetSyncSlot} (Exit) Status: [Access is denied. (0x80070005)]
33 13:38:53.560 [6804] (w3wp) <7608> EV-H {ListArchives.Page_Load} Exception: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)) Info: Diag: Type:System.UnauthorizedAccessException ST: at KVS.EnterpriseVault.Interop.AutoJournalAccessorClass.GetSyncSlot(String auth, String archiveID, UInt32& timeOut)| at DesktopClientCacheWeb.ListArchives.GetSyncSlot(IAutoJournalAccessor ja)| at DesktopClientCacheWeb.ListArchives.List()| at DesktopClientCacheWeb.ListArchives.Page_Load(Object sender, EventArgs args) Inner:None
34 13:38:53.611 [6804] (w3wp) <7120> EV-L {ClientDiagnostics.Page_Load} ClientDiagnostics called
35 13:38:53.611 [6804] (w3wp) <7120> EV-H {ClientDiagnostics.Page_Load} AUTH_USER string: .
36 13:38:53.612 [6804] (w3wp) <7120> EV:L {CClientAuthenticate::GenAuthString:#361} Generating auth string...
39 13:38:53.621 [6804] (w3wp) <7120> EV:M ClientAuthHelperImpl::GenAuthString Authentication Type: Currently impersonated user Client:(null) ==> AuthToken:SERVERNAME.CHANGEDTXT.COM 3Q1K*****
40 13:38:53.627 [6804] (w3wp) <7120> EV-L {ClientDiagnostics.PostDiagnosticValues} Exception: Failed to connect to an IPC Port: The system cannot find the file specified.| Info:ClientDiagnostics PostDiagnosticValues failed Diag: Type:System.Runtime.Remoting.RemotingException ST:|Server stack trace: | at System.Runtime.Remoting.Channels.Ipc.IpcPort.Connect(String portName, Boolean secure, TokenImpersonationLevel impersonationLevel, Int32 timeout)| at System.Runtime.Remoting.Channels.Ipc.ConnectionCache.GetConnection(String portName, Boolean secure, TokenImpersonationLevel level, Int32 timeout)| at System.Runtime.Remoting.Channels.Ipc.IpcClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)| at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)|Exception rethrown at [0]: | at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)| at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)| at KVS.EnterpriseVault.ClientDiagnosticsHandler.ClientDiagnosticsHandler.PostClientDiagnosticValues(NameValueCollection diagnosticValues, String authStr)| at DesktopWeb.ClientDiagnostics.PostDiagnosticValues(NameValueCollection queryStringParams) Inner:None
45 13:39:08.270 [6804] (w3wp) <7608> EV-L {Slot.Page_Load} Slot.Page_Load
46 13:39:08.270 [6804] (w3wp) <7608> EV-L {Slot.GetSyncSlot} Slot.GetSyncSlot - VEID:1D67ECF7CDE3DD3418AF7F977957226191110000evault2, TimeOut:0
47 13:39:08.271 [6804] (w3wp) <7608> EV:L {CAutoJournalAccessor::GetSyncSlot} (Entry)
49 13:39:08.272 [6804] (w3wp) <7608> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 40]
50 13:39:08.274 [6804] (w3wp) <7608> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Successfully communicated with an EV Directory Service on the local machine
51 13:39:08.322 [6804] (w3wp) <7608> EV:L {VaultCoCreateInstanceEx} CLSID [{4EC6FF76-C97A-11D1-90E0-0000F879BE6A}] Server Name [(null)] Used Server Name [(null)] Num of attempts [1] Total elapsed [0.000s] Result [Success (0)]
52 13:39:08.323 [6804] (w3wp) <7608> EV:L {GetStorageObject:#46} Calling VaultCoCreateInstanceEx
53 13:39:08.323 [6804] (w3wp) <7608> EV:L {CStorageOnlineOpnsInstanceHelper::GetLoadBalancedStorageOnlineOpnsCLSID:#61} Non-StorageOnlineOpns CLSID. Returning Input.
54 13:39:08.332 [6804] (w3wp) <7608> EV:L {VaultCoCreateInstanceEx} Attempt [1] to create COM object failed. CLSID [{957FF4B4-162B-4708-843A-0134868699B4}] Server Name [EXC7.CHANGEDTXT.COM] Elapsed [0.009s] Result [Access is denied. (0x80070005)]

I'm at a loss where to go now.

Any help would be appreciated please as I've been trying to keep these servers going whilst waiting for a project to migrate our Databases over to a SQL Server 2012 (or higher) so I can get our environment upgraded to V12.

Thanks,

Andy

  • Ampersound's avatar
    Ampersound
    5 years ago

    Morning All,

    I thought I'd update this post and close it off.

    So the issue lasted an entire week until the scheduled reboots for the EV servers early yesterday morning. The DCOM errors stopped immediately once the servers came up. I hadn't made any changes other changes as per listed above.

    The only thing I can think of is that the SQL Server farm we utilise for EV performed its monthly reboot on Sunday and after the EV servers reconnected post reboot, it seems everything is working again.

    Due to the DCOM issues, I think they sent me on a wild goose chase looking at IIS and EV.. not the SQL Servers.

    I'll have to see what I can find out with the SQL servers (as they're looked after by a different team).

    At least its fully operational again now.

    Thanks,

    Andy

  • plaudone1's avatar
    plaudone1
    Level 6
    5 years ago
    Hi Andy,
    Did the updates make any changes to IIS as there are exceptions on vault cache operations and search operations which go through IIS. Does the VSA work on search or AE page?

    If you search support page for “v-437-29014” in quotes it will return articles with that event ID.

    Regards,
    Patrick
    • Ampersound's avatar
      Ampersound
      Level 3
      5 years ago

      Hi Patrick,

      Thanks for your reply.

      The KB articles state that it does update IIS :-

      "Security updates to the Microsoft Scripting Engine, Windows App Platform and Frameworks, Windows Input and Composition, Windows Media, Windows Kernel, Windows Core Networking, Windows Peripherals, Internet Information Services, Windows Network Security and Containers, Windows Active Directory, the Microsoft JET Database Engine, Internet Explorer, and Windows Storage and Filesystems."

      My access doesn't seem to work in pulling items from EV although as per the screenshot, IIS webpage seems to display the folders of my archive just nothing from within them.

      The VSA displays the same error message "failed to perform the search request" error.

      I did try re-registering ASP.NET 2 solution found here (on the server after uninstalling the patches) but no joy - https://www.veritas.com/content/support/en_US/article.100009900

      Not seens any RPC issues either as in some of the other articles.

      I've not checked if any tweaks were made to IIS, I've checked some of the files for any changes and nothing popped out that had changed in years.

      I definitely think its more IIS related though.

      Regards,

      Andy

      • GertjanA's avatar
        GertjanA
        Moderator
        5 years ago

        I believe there is an article somewhere on what permissions need to be where, but I could not locate it. Have a look at this one to get you started. https://www.veritas.com/support/en_US/article.100030535

        Additionally, verify permissions on the .Net folders. I've seen issues where the full access for the VSA was missing on the 'temporary ASP.NET Files' folders. 

        If all else fails, I suggest to first run Deployment Scanner (to find if prereqs are all green). If that is ok, you might want to consider a re-install of the binaries. That would set permissions etc. again on OS level.