cancel
Showing results for 
Search instead for 
Did you mean: 

DCOM Permission Errors - Event ID 10016 or 10015 - When restarting EV Admin service

fsalas
Level 3

Hello Folks,

I am new to this forum, but not new to Enterprise Vault.  I am not sure if this is the correct place to place this question/request.  If it isn’t, I apologize in advance.

 

My goal is to get the assistance from the EV Product Team.  I would like to see this escalated so it gets resolved.  I have already gone through support and it appears they can’t do anymore as only EV Engineers can fix it.

 

Problem Definition

===============

Symptom:  DCOM 10016 or 10015 Errors observed in Windows Event log.

Symptom:  All errors point to the EV service account or EV processes not having sufficient DCOM permissions assigned.

Fact:         Enterprise Vault Admin service resets DCOM permissions every time it is restarted.

Fact:         Errors do not appear to “break” overall EV functionality.

Fact:         Feedback from Tech Support is that this is “known issue” by the EV Engineering Group.

Fact:         Due to perceived “low impact” (i.e. server not down) issue has not gained attention to become an escalation.

Fact:         Customer’s confidence in the product going into a production environment is affected.

Fact:         There are many customers out there having this issue (as per Support’s feedback & what I see in the Discussion groups).

Action:      Replicated issue on three different environments with various EV versions.

Action:      Used “SkipChecks” (EV 7) and “DoNotChangeAnonymousPermissions” Registry Keys (EV 5 & 6) but NO effect. 

Action:      Researched DCOM permission changes made by Microsoft in Windows 2003 SP1 and higher.

Action:      Month-old Symantec Technical Support case open with no results or resolution (since December 11th, 2007)

Action:      Followed all Symantec’s technotes outlining steps/suggestions to correct DCOM permission.   For example:

                 http://seer.entsupport.symantec.com/docs/279588.htm

                 http://seer.entsupport.symantec.com/docs/284386.htm

 

Environment:

==========

Windows 2003 Enterprise SP1 and SP2

Enterprise Vault 7.0 SP1,

Enterprise Vault 2007 SP1, 2007 SP2

 

Errors in Event log:

==============

Event Type: Error Event Source: DCOM Event Category: None Event ID: 10016 Date: 12/15/2007 Time: 9:45:00 AM User: DOMAIN\EVSA_ACCOUNT Computer: ENTERPRISE_VAULT_SERVER Description: The application-specific permission settings do not grant Remote Launch permission

for the COM Server application with CLSID {562FF725-C308-11D1-93DA-0000F87A3C75}

to the user DOMAIN\ EVSA_ACCOUNT  SID (S-1-5-21-2076772084-1888234209-63777028-948613).

This security permission can be modified using the Component Services administrative tool.

 

To replicate the problem:

===================

1.       Follow any of the Symantec technotes and assign suitable DCOM permissions to the EV account and EV processes as required.

2.       Restart EV admin service

3.       Check DCOM permissions that were just set.  They will be gone!

 

Current Status:

============

My colleagues and I have been troubleshooting and replicating this issue for the past month and followed Symantec technotes and/or suggestions provided by EV Technical Support.  The technotes or suggestions are really useless because the Enterprise Vault Admin service wipes and resets the DCOM permissions when it is restarted.  Due to the low severity of the support case (server is not down) and the fact that the issue has not been considered an actual problem to resolve; there has been no action or feedback indicating issue will be addressed.  It is unknown when this will get corrected.

 

Comments:

=========

Our understanding is that Microsoft did in fact tighten DCOM access by removing the Anonymous account from the “Everyone” group, so as to move away from allowing Anonymous Logon access for security reasons:

http://technet2.microsoft.com/windowsserver/en/library/4c9a2873-2010-4dbb-b9dd-6a7d1e275f0f1033.mspx...

 

Microsoft’s suggestions to correct any of these DCOM permission errors indicate that you can:

- Add the account used by that program to the new group called “Distributed DCOM users” …or…

- Give the account or process that is identified by the error the necessary DCOM permissions.

*** However, due to the EV admin service resetting them, this would seem to be impossible to make stick for any length of time. 

 

In our research to understand the problem, we saw that these errors are a common problem for a lot of other applications that rely on DCOM.  In all the resolutions, they advise to set the DCOM permission manually for the account/process involved and the problem disappears.

 

I am confused why Symantec appears to be relying on still giving the Anonymous account the same DCOM permissions when Microsoft has moved away from this for security reasons.  Our customers wonder why the Anonymous account is needed when a special dedicated account (EVSA) is being used to run the services and processes for EV to function. 

 

Questions:

========

o        Shouldn’t the EVSA account be given the appropriate DCOM permissions instead? 

o        Why is DCOM permissions hard coded into the service? 

o        Should we not have a choice between running a script (once!) and then do DCOM permission individually without losing them?

o        Why are the registry keys specifically created to bypass the “checking/setting” of the DCOM permissions (“SkipChecks” for EV 7 & DoNotChangeAnonymousPermissions” for EV 5 & 6) NOT working?

o        Why is the focus on the Anonymous account, when these errors are being flagged as attributed to the EVSA account and EV processes?

o        Shouldn’t the DCOM permissions be set up correctly for the EVSA at minimum?

 

 

Please help!  Thanks in advance!

 

 

Francisco.

29 REPLIES 29

A_Z_rcher
Level 5
Partner Accredited
Dear Francisco
 
Thank you for your complete note!
I was analyzing the same problem the last view days (event 10015). I hoped that SP2 would correct this but as I can see in your post it won't... so I don't need to waste time on this
 
sorry, I cannot help you, but I can confim your results.
My biggest fear is that we do not know how this impacts EV....... Can you escalate this?
 
 

MarkB_2
Level 4
The CLSID in the event log is, I think, the IndexBroker. Normally when the Admin Service starts it then launches the other services and we have seen approx. 6 DCOM errors being logged at this point. The only time you need to really worry about DCOM issues is when you see error being logged that are not at startup.
 
Have you looked at these technotes?
 
 
 
Also, have you looked at your group policies and the local security policy. It migh also be worth checking the local security settings on the EV server.
Under Local Policies - Security Policies, check
DCOM: Machine Access restrictions in security descriptor language (SDDL) Syntax
DCOM: Machine Launch restrictions in security descriptor language (SDDL) Syntax
under the properties edit the security and select OK to populate the security descriptor.
 
Open a command prompt and enter
 
gpupdate /Force
 
This should stop any changes being made via GP etc.
 
 
This has been a known issue for a very long time and as the system is not affected that is probably why it is not being pushed as an escalation.
 
Can I ask a question? For every warning or error being logged in the event logs from other applications, including Microsoft, have you logged calls with the software provider to get these resolved even if they are not affecting the operation of the system?????
 
 
Mark
 
 
 

jimbo2
Level 6
Partner
Verify that you have set DisableLoopBackCheck and DisableStrictNameChecking MS keys to (1).
 

fsalas
Level 3
Thanks for the feedback.  We have asked for escalation of this issue via Support yesterday.  Still waiting to hear back what and when it will happen.  I will keep this post updated.

fsalas
Level 3
Mark,
 
Thanks for your answer.  To further clarify my post:
 
You are correct, the various CLSID's seen correspond to any of the EV tasks when the EV admin service restarts them.
 
With respect to the Technotes:
 
** I have not seen this error yet.  The one we have seen is 10005 and 10006.  Re-entering the EVSA again is one of the suggestions made in several of the other technotes we followed.  The result is that it does not prevent the EV Admin service from wiping the DCOM permissions that have been set.
** This technote we already tried without success.  The answer is accurate in that the issue can be resolved by changing the DCOM permissions as per steps 1 & 2.  The underlying problem though is that EV Admin service appears to wipe the DCOM permissions that have been set on steps 1 & 2, effectively ending up right back where you were before.
 
To rule out the possibility of a Group Policy making changes, we have used the "gpupdate /Force" command right after the DCOM Permissions have been set correctly.  The moment you recycle the EV Admin service you will see that they are wiped away ending up right back where you were before, please see my steps to replicate the problem in my post.  This is what I feel needs to be corrected and Symantec Technical Support has not thrown the issue back at me to say that it is on our side of the fence.  The EV documentation clearly states that the EV admin service sets DCOM permissions, but if they were correct you wouldn't see these specific errors telling you that they are not. 
 
With respect to the local policies, I am not sure how they would affect or prevent the EV admin service from setting the DCOM permissions correctly.  If you have an angle here, please elaborate.
 
With respect to your question "For every warning or error being logged in the event logs from other applications, including Microsoft, have you logged calls with the software provider to get these resolved even if they are not affecting the operation of the system?????".  My answer is no.  I don't need to log calls with the application provider as simply making the DCOM permission changes (for example as per:  http://support.veritas.com/docs/279588) gets rid of the problem because the DCOM permissions actually stick and don't vanish after you recycle the EV admin service.  If you leave the EV service untouched you will not see these errors until it's bounced again (or server is rebooted) - hence my logic to seek the assistance of the software provider as I would assume would know how their code accomplishes this.  No one from Symantec has asked me to prove anything yet as they are aware there is an issue, it's just that it appears not to be of great concern or priority at this point.
 
I have asked the Symantec Technical Support team if I might be overlooking something in this scenario but the answers that I have received are that they "are aware that there is an issue" and that there a lot of customers with "same problem" as described in my post.  My understanding is that the support folks can replicate the same issue as I described.
 
Also, the fact that Symantec has created Registry entries to "not check" or "not set" DCOM permissions tells me that they are making changes and somehow they don't address the needs of the majority of folks out there.  My expectations are that if I change the DCOM permissions that they are not wiped away by bouncing their own service.  If the Registry keys actually worked in preventing this behaviour then I would be happy - problem solved, no more resetting permissions over and over again.  As you can see here I have outlined all my actions and logic with lots of details so they can see I am not rushing to conclusions and hope that they look at it on their own (and then tell me if I am out to lunch if applicable).  Maybe they are doing things correctly but they overlooked this scenario.  I have heard lots of folks say "well it's only a nuisance error...nothing is really affected".  I am not so sure of that and to someone who likes a clean install the natural reaction is to see it solved.  Also I have seen other technotes that outline functionality issues and that are cross-reference by Symantec Technical Support back to the same technotes that I stated earlier.
 
My intention is not to cause trouble, or point fingers.  I simply want to move it forward as I don't feel it should be just ignored because it "appears not to hurt anything".
 

fsalas
Level 3
With respect to the suggestion by Jimbo2 to make sure "DisableLoopBackCheck" and "DisableStrictNameChecking" were on.  The answer is Yes.  I ran the Deployment Scanner utility initially where it does a Registry Check and these reported being set ok and I ran it again just now to confirm (looks good).

fsalas
Level 3
An update: 
=======
I received word from Symantec Technical Support this week indicating that they will not be opening an escalation with Engineering as this issue is not considered a critical problem since things "appear to work ok".  The case is being closed/archived.
 
They will be modifying the technotes so that they don't make mention of the by-pass switch that doesnt work.  They have pointed me back to the Symantec Sales organization or suggested that it be taken up with EV Product Management.  I will try that avenue once again.  No progress to report unfortunately.
 
Does EV Product Management read these blogs?  How does one get EV Product Management to look at these?

Michael_Bilsbor
Level 6
Accredited
Hi,
Can you provide me with the case reference number please.

fsalas
Level 3
Technical Support Case:  290-903-613

Michael_Bilsbor
Level 6
Accredited
Hi,
 
By the way, when you get the DCOM error, can you check are you receiving requests in the IIS log as well?

Popolou
Level 4
Misery certainly enjoys company! Smiley Very Happy

Joking aside, i spent the good portion of two days trying to resolve this problem with several KB and forum searches that didn't really turn up anything meaningful, that is until i stumbled upon this thread.

First off, thank you for a thoroughly details post on the issue fsalas. You've summed up the problem and my feelings in a nutshell. As to the issue, yes i too would like to see a resolution to this as the current state of affiars is rather dissapointing.

I will be keeping an eye on this thread for any further updates.

Regards

Pop

fsalas
Level 3
In response to "The Dodo's" question (sorry just saw that update today) there are no related entries in the IIS logs. 
 
Update on this issue since the last time I updated.  I did not get any further via Support and/or the Sales folks.  We were requested to quantify the impact and the customer simply gave up was too busy to spend any cycles on this.  I still keep seeing these DCOM permission errors on other (new) installations.  The reason that they are easily missed is that they show on the System Event logs and that the majority of the Symantec folks (in Support and Sales) attribute or associate it as a "Microsoft problem" - which is not accurate at all.  I am a bit disappointed as I provided very detailed information to them and it went nowhere since it wasn't hot enough to escalate.

Kopfjager
Level 5
Employee Certified
Just a note so you know that we do read these posts and react to them when we can.
 
The behavior that occurs when starting the Admin service before the Directory service has started completely has been defected and I will post any updates when they become available.

fsalas
Level 3
Paul, thanks for the response.  I am looking forward to your updates.

Robert_Primozic
Level 4
Partner Accredited Certified
I will join to this discussion...
Also related to DCOM errors, Role based administration does not work as expected... When you install EV management console to users's workstation and assign him a role (no matter what), this user mustalso be a member of local admin group on EV server. It is not neccesary to be a local admin on own workstation.
I also opened (and closed too) a support case # 320-071-058 related with this issue. Delivered sulution was that this will be fixed in next SP at the end of May 2008.

Michael_Bilsbor
Level 6
Accredited
Hi,
 
The fix (as I understand it) is that you won't be required be a member of local admins to perform pst migration.  That your expectation?

chileno
Level 3
Partner Accredited Certified
My userid has changed since I created this post.  Just wondering if there was any news on the initial DCOM permission errors?

Michael_Bilsbor
Level 6
Accredited
Hi,
 
Yes, this problem is planned to be fixed in V7.5 sp4. 

Robert_Primozic
Level 4
Partner Accredited Certified
As the matter of fact, this problem was solved with SP3 for EV2007. Tested already.
R.