Issues binding to GC

[_Josh_]

Hi guys,

We currently run two Enterprise Vault Servers and pretty well only use the Email archiving copmponent.  In the last week or so, we've found that on one of the hosts, EV2, all Archive tasks fail to start.  I've cleared all the message queues but the problem persisted.  Moving any/all of the Archive tasks from EV2 to EV1 start and run fine and as such have left them there while I investigate the cause.

Running a dtrace shows issues with EV2 binding to the GC.  It searches and finds one of the 5 GC's fine, it's just the binding that fails.  See logs below.

Snippet from log (I've renamed the FQDN stuff for privacy)

270 08:21:54.641  [6288] (ArchiveTask) <7668> EV:L {GetExchangeServerValues:#4269} Using GCOverride from [organization] entry [GC://SERVER.FQDN]
271 08:21:54.641  [6288] (ArchiveTask) <7668> EV:M {FindExchServerDn:#723} Getting search base, DN and version for Exchange server [XCH1]. Forcing LDAP: [True].
272 08:21:54.641  [6288] (ArchiveTask) <7668> EV:L {FindExchServerDn:#776} Forcing LDAP. Using Configuration Naming Context as the search base.
273 08:21:54.641  [6288] (ArchiveTask) <7668> EV:M GetDomainNamingContexts - Binding using string GC://SERVER.FQDN/RootDSE
274 08:21:54.657  [6288] (ArchiveTask) <7668> EV:L {FindExchServerDn:#790} Binding to search base [LDAP://SERVER.FQDN/CN=Configuration,DC=SERVER.FQDN].
275 08:21:54.672  [6288] (ArchiveTask) <7668> EV:H {FindExchServerDn:#851} Error: [0x80072020]
276 08:21:54.688  [6288] (ArchiveTask) <7668> EV:H {GetExchangeServerValues:#4320} Error [0x80072020]
277 08:21:54.688  [6288] (ArchiveTask) <7668> EV:H {CAgentTask::Initialise} HRXEX fn trace : Error [0x80072020], [.\AgentTask.cpp, lines {1450,1456,1463,1475,1479}, built Jun 29 21:09:35 2012].
278 08:21:54.688  [6288] (ArchiveTask) <7668> EV:H {CAgentTask::Initialise:#1623} Error [0x80072020]
279 08:21:54.688  [6288] (ArchiveTask) <7668> EV:H {CAgentTask::Initialise} (Exit) HRXEX fn trace : Unexpected fn exit path, returning error [0x80072020], [.\AgentTask.cpp, lines {1450,1456,1463,1475,1479}, built Jun 29 21:09:35 2012].
280 08:21:54.688  [6288] (ArchiveTask) <7668> EV:H {CArchiveAgentTask::Initialise} HRXEX fn trace : Error [0x80072020], [..\AgentsCommon\ArchiveAgentTask.cpp, lines {358,364}, built Jun 29 21:09:35 2012].
281 08:21:54.688  [6288] (ArchiveTask) <7668> EV:M CArchiveAgentTask::Initialise - Com Result [0x80072020]
283 08:21:54.688  [6288] (ArchiveTask) <7668> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 1]
284 08:21:54.703  [6288] (ArchiveTask) <7668> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Successfully communicated with an EV Directory Service on the local machine
285 08:21:54.703  [6288] (ArchiveTask) <7668> EV:M CDirectoryVaultObject::GetAttributeListFromType Attribute list for type Task : (null)
286 08:21:54.703  [6288] (ArchiveTask) <7668> EV:L CDirectoryVaultObject::GetAttributeListFromType Unknown type - Task
287 08:21:54.703  [6288] (ArchiveTask) <7668> EV~E Event ID: 2256 The Archiving Task 'Exchange Mailbox Archiving Task for XCH1' could not be started due to startup errors. |
288 08:21:54.703  [6288] (ArchiveTask) <7668> EV:L {CAgentTaskLoader::DeInitialiseTaskLoader} (Entry)
289 08:21:54.703  [6288] (ArchiveTask) <7668> EV:L {CAgentTaskLoader::DeInitialiseTaskLoader} (Exit) Status: [Success]
290 08:21:54.703  [6288] (ArchiveTask) <7668> EV:H {CAgentTaskLoader::TaskMain} (Exit) Status: [An operations error occurred.  (0x80072020)]

I did try forcing EV to look at a different GC for testing purposes but the issue persistsed.  EV1 does not experience this issue.

I realise we're running behind on the current CU level which may help the issues but I'd like to get it stable before I look at product upgrades. 

Any help would be much appreciated.

Product versions : Exchange 2003 SP2 / Enterprise Vault 9.0.4.1097

Josh

[Arjun_Shelke]

Do you see any Kerberos errors on that EV Server?

[Arjun_Shelke]

WireShark or Netmon tools can give you insight of these bindings?

[_Josh_]

Cheers for the replies, no Kerberos errors.  Will indeed WireShark it up, was hoping for a "sure, it's just this..."

Keep you posted.

[JesusWept3 · Accepted solution]

Could be the other server has closest GC or DS server key Could also be a binding order issue on the NICS, but I would try DS Server pointing to a CAS server like GC://casserver.mydomain.com

[_Josh_]

Remembering we're on Exchange 2003 though  so no CAS role as such, just frontend or backend.

The odd part of it all is it was working fine then stopped suddenly.

[JesusWept3]

Apologies, then try using a regular GC via DS Server setting

[Arjun_Shelke]

Any luck on wireshark?

[VirgilDobos]

Hi Josh,

Can you please confirm the Closest GS key is set up on the affected EV server?

HKEY_CURRENT_USER
\Software
 \Microsoft
  \Exchange
   \Exchange Provider

Looking forward for your reply.

Cheers,

Virgil

[_Josh_]

Hi guys,

Sorry for the delay.  Indeed the Closer GC key is set (see below).

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\Microsoft\Exchange\Exchange Provider]
"Closest GC"=dword:00000001

As for Wireshark, I'll be running that today to see what it shows.  Keep you posted.