The Enterprise Vault service is not available

[SHI-CRO]

EV 9.0 sp1 running on Windows 2003 R2 Enterprise.  We have 8 archiving servers and this issue is only happening on one of them.

When opening up the /enterprisevault/search.asp page, we get the message "The Enterprise Vault service is not available".  Also, on the same server, when opening up Archive Explorer, we get a pop up box saying "The XML returned for the tree view is not as expected. (500)".  I assume these errors are caused by the same problem, but I'm not sure what it might be.

In the event log we get:

Event Type:    Error
Event Source:    Enterprise Vault
Event Category:    Web Application (WP)
Event ID:    8390
Date:        1/19/2011
Time:        2:29:08 PM
User:        N/A
Computer:    ComputerName
Description:
The EnterpriseVault.DirectoryConnection object reported an error.
 
Unspecified error

Internal references:
Error 0x80004005
CDirectoryConnectionObject::GetNextLevelArchiveXML .\DirectoryConnectionObject.cpp [lines {3307,3316}], built Nov 24 00:04:09 2010

For more information, see Help and Support Center at http://evevent.symantec.com/rosetta/showevent.asp

A dtrace of w3wp gives this info when trying to open the web pages:

4    14:21:22.985     [9060]    (w3wp)    <7724>    EV:L    CIndexSearch2::Reset
5    14:21:22.985     [9060]    (w3wp)    <7724>    EV:L    CIndexSearch2::FinalRelease
6    14:21:22.985     [9060]    (w3wp)    <7724>    EV:L    CAuthHelper::Reset Cancel registration? true CancelId: 0
7    14:21:22.985     [9060]    (w3wp)    <7724>    EV:H    Calling GenAuthString
8    14:21:23.000     [9060]    (w3wp)    <7724>    EV:H    ClientAuthHelperImpl::GenAuthString Unable to register by impersonated user. hr=Either the application has not called WSAStartup, or WSAStartup failed.  [0x8007276d]
9    14:21:23.000     [9060]    (w3wp)    <7724>    EV:H    CClientAuthenticate::GenAuthString exception raised!
11    14:21:23.000     [9060]    (w3wp)    <7724>    EV:L    CAuthHelper::Reset Cancel registration? true CancelId: 0

I've found this technote: http://www.symantec.com/business/support/index?page=content&id=TECH50333&actp=search&viewlocale=en_US&searchid=1295465931138, but there aren't any problems with the rights of the vault service account.

Anyone have any ideas?

Thanks in advance.

[Paul_Grimshaw]

If you just go to /EnterpriseVault do you get the same error?

This generally indicates an IIS or ASP issue. I would do the following:-

1. Reboot the server and see if that fixes it

2. Re-register ASP and see if that fixes it

3. Reinstall IIS

[RahulG]

Update the Crendentials fro the VSA from the VAC  right clik on "Directory on server name " go to properties --Click Service account and update the Credentails .

Restarting the Ev admin service shud take care of it .

[SHI-CRO]

Still no luck.  I've rebooted the server, uninstalled and reinstalled IIS, re-registered ASP.NET and re-installed EV using the setup program and it's still not working.

Going to http://evserver/enterprisevault gives the same result.

The vault store reporter page (usage.asp) works.

The Symantec support tech has concluded that the only resolution is to wipe the server and do a clean install.  I'm hoping that someone has seen and fixed this is the past before I re-install the server.

[JesusWept3]

woah wait, WHAT?!

just as a matter of interest, if you go to Search.asp and use the evadmin credentials or someone who's a local admin on that machine, i don't spose that works does it

[JesusWept3]

ok dude, what i would do if i were you , get some winhttp traces going, and ask symc to create a TSANet case with microsoft, reinstalling an OS is unacceptable, from everything i've read it looks like its down to some mismatched DLL's

You may also want to grab some debugdiags also as this will help them too

[SHI-CRO]

Nope I'm afraid not.  I've attached screenshots of what I'm seeing.  These are all using the VSA.

[JesusWept3 · Accepted solution]

ok so firstly tell your TSE that reinstalling the server is a terrible idea, i mean honestly it is horrifyingly bad.... ask them to escalate the case and get a TSA Net with MS and Symantec

Microsoft will request some trace logging through IIS and possibly some debugdiag outputs
You can either try and get these proactively or you can wait for what Microsoft suggests

WinHTTPTrace Config details can be found here:
http://msdn.microsoft.com/en-us/library/aa384119(v=vs.85).aspx

DebugDiag can be found here:
http://www.microsoft.com/downloads/details.aspx?FamilyID=28bd5941-c458-46f1-b24d-f60151d875a3

Make sure you download the x86 version regardless of whether you're on x64 or not, because the 64bit version of DebugDiag is diagnostic only, though some engineers may prefer you use ADPlus however debugdiag is more IIS Centric and a lot friendlier to use

So what i would do is the following
 

Download and install Debug Diag and the windows resource Kits
1. Download and install DebugDiag to your EV Server
2. Download and install the Windows Server Resource Kit

Configure DebugDiag
3. Open up DebugDiag
4. From the wizard choose "Crash" and press Next
5. Choose "A specific process" and press Next
6. From Selected Processes type "w3wp.exe" and press Next
7. From Action Type , choose "Log Stack Trace" and then put something like 10 instances
8. Press the "Exceptions" button and then press "Add Exception"
9. In Exception Code put in "8007276d"
10. For Action Type put "Full User Dump"
11. For Action Limit put in a number such as 10
12. Press OK and then press Save & Close
13. Press Next and then choose a suitable place for the memory dumps to be saved to
14. Press Next and then press "Do Not Activate The Rule At This " and press Finish


Configure WinHTTPTrace
15. Open a command prompt
16. CD to your Resource kit directory
17. type the following

Winhttptracecfg -e 1 -l E:\YourLogFileLocation\yourserver -d 0 -s 1 -t 1

18. -e 1 is to enable logging (0 = off, 1 = one)
19. -l is to specify the log location and the prefix, put this where your dumps are set to go
20. -d 0 is to write it to a file as opposed to a debugger
21. -s 1 is to show the calls in text as opposed to hex (though MS may change this)
22. -t 1 is to show top level functions that are called, more verbose the better at this point
23. Minimize the command prompt

Configure DTrace
24. Open up another Command Prompt
25. CD to your \Program Files\Enterprise Vault directory
26. Type DTrace and press enter
27. Type "set w3wp v" and press Enter
28. Type "log <yourDebugDiagLocation>" and press enter

Now go back to DebugDiag where you have the rule set to not activated, right click the rule and then click "Activate Now"

Now replicate the behavior several times going to http://yourEVServer/EnterpriseVault/
Ensure that the Winhttp log file is being written to, ensure that debugdiag is dumping out full memory dumps

After this zip up the contents, and upload it to symantec.
If you're curious you can open up the debugdiag outputs and it will show you its best guess as to what is causing the issue to occur, though because this is not really an unhandled exception it may just show you the stack trace of what was occuring at that time

[JesusWept3]

oh sorry forgot, after you've got the memory dumps, the dtrace and winhttp logs, go back to the winhttp configuration and just put winhttptracecfg -e 0

and then in debugdiag right click the rule and press DeActivate (if it says Complete then you needn't worry)

[SHI-CRO]

Thanks tons JW for the write-up; I'm definitely going to keep that for future use.

I told the TSE that we'd rather try anything to keep from rebuilding the server and he told me he'd escalate it and I'd get a call back in 2 hours.  Of course that didn't happen.

Since the case was over a week old, and user complaints were adding up, I thought it would be quicker and easier to do the rebuild than to wait on a resolution through support.  That is one of the great things about EV: once the OS was built and storage presented, it only took about 15 minutes to get the server back and everything working great again.

Seems that support has been pretty overwhelmed the last little while.  It would be nice if a server rebuild wasn't the quickest way to a fix, but for an issue like this, it seems to be.  I only wish we would get a more truthful estimate of a resonse; it really gets me that they will still promise a 2 hour call back when they know there's no chance of it happening.  Although I think there may be a chance that I'm blacklisted in there somewhere; I've called for some pretty boneheaded things in the past .

[JesusWept3]

everyones in the same boat, its difficult though, EV9 SP1 has been a difficult release, everyone is now back from their christmas and new years holidays and have set the agenda for whats going to happen, people upgrading because now they have the time and the budget that its new year...etc etc

It's not been easy but really though i commend you for reinstalling the OS, i used to work in EV Support myself and have had a couple of customers not want to troubleshoot and just reinstall, I guess the desire to troubleshoot is outweighed by the desire to get people to stop calling them.

But the way i think about it is this, imagine how many users were calling you saying things were broken, things were down, some easy questions, some hard questions, somethings you already knew about and then you call your symc rep and tell them whats happening etc, now thats what symc has just multiplied by thousands of customers :)

But yeah, if it happens again, get the logs before you reboot and then you might be able to resolve it after the fact and then if it happens again you'll know your resolution

(Oh and i still think rebuilding the server was a terrible idea, i know you had to do it because of requirements and pressure, but for a TSE to suggest that? wow, you are the exception when it comes to following that suggestion :)