No EV Toolbar in Exchange 2007 OWA

[cmarkovits]

Hello all and thanks for looking.

/// Environment ///
- All Exchange servers are Exchange 2007 and run on physical W2K3 Ent. Ed Sp2 systems.
- All Exchange 2007 systems are patched with SP3 RollUp 2
- All Exchange 2007 Mailbox servers are dedicated (no multi-roles) and are clustered (N+1)
- All Exchange 2007 CAS Servers are also dedicated, and sit behind an F5 Load Balancer
- All EV Servers are running version 9.0.1 and are deployed on Windows Server 2008 SP2 Std. Ed.

/// Issue ///
It appears that after a recent Exchange Cluster failover, those customers who connect with OWA are not able to see their "EV Toolbar", nor can they see the "Search Vaults" icon or the "Archive Explorer" icon.  EV icons for each message do appear properly, however when a customer opens an archived message there is a banner at the top that reads:

The archived item is currently unavailable.
If you choose reply or forward, only the content shown below will be included.
Click here to preview the original item. 

Clicking on the banner brings up a "Page cannot be found" error.

/// What we have done so far ///

1. We know that this issue is limited to just the customers on the Exchange 2007 Server that experienced a cluster failover about a week ago.  IE, only customers on Mailbox Server 1 (Which is serviced by EV Mbx Archive Server 1) experience this issue.  Customers on other Mailbox Servers do not experience this issue.  This led us to focus on just the EV Mailbox Archive Server, and Exchange Virtual Server.
2. We enabled OWA Client Logging, and we found that customers on the EVS that failed over get the following error.

 [WebDAVRequest::Send] Exception sending WebDAV request: System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
   at System.Net.HttpWebRequest.GetResponse()
   at Symantec.EnterpriseVault.Owa.ExchangeStoreAccess.WebDAVRequest.Send() 

1. We examined the "Exchange" IIS Virtual Directory on the physical cluster node that hosts the Exchange Virtual Server.  We compared the configuration of that virtual directory to the "Exchange" virtual directory on other cluster nodes and don't see any difference.
2. We referenced Symantec TechNote TECH45814 (http://www.symantec.com/business/support/index?page=content&id=TECH45814&actp=search&viewlocale=en_U...) and in the process did find that we had a configuration problem with the shopping service.  The EV Account used for OWA did not have full permissions on the shopping directory, so we granted it full permissions on that folder and all subfolders and files.  After that we ran the "shoppingservice.exe/regserver" command just be sure and recycled all of the EV services on that EV server.  That did not help.
3. In reviewing the IIS Logs on the Exchange Server, we don't see anything that would looks unusual.

Any thoughts / suggestions would be greatly appreciated and thank you in advance!

[cmarkovits]

Sorry, forgot to include one other error that shows up in the EV OWA Client log.  It is...

 EVContext::LoadHiddenSettings]     System.NullReferenceException: Object reference not set to an instance of an object.
   at Symantec.EnterpriseVault.Owa.ExchangeStoreAccess.WebDAVHelpers.GetEVHiddenMessageId(Log oLog, EVContext oEVContext)
   at Symantec.EnterpriseVault.Owa.Core.EVContext.LoadHiddenSettings(Log oLog, Boolean brefresh) 

[links10]

Anytime I have issues with EV and OWA I either restart the CAS server and/or reinstall the OWA extensions for EV.

[cmarkovits · Accepted solution]

All,

    OK, so for the benefit of the community, I am posting what the final resolution to this issue was - hopefully it will benefit others.

So the root issue was that there was a problem with the “Exchange” Virtual Directory within IIS on node that our Exchange Virtual Server was running on.  Everything was working as expected up until we had an unexpected Exchange Cluster failover.  After the failover, EV clients could not load the EV Toolbar and other icons when connecting with OWA.  The problem was specifically with the “Exchange” Virtual Directory on the node that we failed over to.  For some reason, requests sent to that node were responding with the “http://<Exchange_Virtual_Server_Name>/Exchange”, instead of http://localhost/exchange. We actually determined this by comparing NetMon captures on the CAS servers between a good set of Exchange / EV servers, and the problematic set of Exchange / EV servers.  The solution was to…

1. Remove the Exchange Virtual Directory using the Remove-OwaVirtualDirectory “Exchange  (Default Web Site)” cmdlet on Node B.
2. Recreate the Exchange Virtual Directory using the New-OwaVirtualDirectory -OwaVersion “Exchange2003or2000″ -Name “Exchange (Default Web Site)”
3. Fail the Exchange Virtual Server off of Node B to another open node.
4. Reboot Node B.
5. Move the Exchange Virtual Server back to Node B and confirm that EV functions properly.

Coincidentally, for those of you who also support some of the older Entourage clients, you will notice that any Exchange Server that is experiencing this problem with its “Exchange” Virtual Directory will not let your Entourage clients connect as Entourage is dependent on this Virtual Directory.  It is also probably worth mentioning that Microsoft For those that are not aware, the “Exchange” Virtual Directory is depreciated (IE it is only intended to provide backwards functionality), so it would probably benefit Symantec (and us) if they would start using the “OWA” Virtual Directory and dropped “Exchange”.

Corey