MSMQ Outgoing Queues referencing failed EV server - using Building Blocks for High Availability

[arnoldmathias]

Hi Everyone

We make use of Building Blocks to build High Availability in our EV environment. We have come across a peculiar problem and wanted to know if anyone here has noted a similar issue.

Scenario:

We are in the archiving schedule and there are (say) 5 EV Mailbox Archiving Servers archiving 5 Exchange Servers. There are cross-references across EV and Exchange Servers i.e. the Storage Service for an archive is on an EV server other than the one which is running the Archival Task.

Problem:

If an EV Mailbox Archiving Servers fails, we use Building Blocks to quickly bring up services on a standby server by running Update Service Locations. However, we notice that items are still in archive pending state in user mailboxes being archived by other EV Servers. New items are also going into archive pending state across all user mailboxes.

Analysis:

We have already run a flushdns (or Clear-DNSClientCache in Powershell) across all EV servers. In the MSMQ Outgoing Queues, we noticed that there are queues which still resolve to the old failed servers and are in Failed to Connect or Waiting to Connect state, since MSMQ does not resolve to the EV aliases but the EV server hostnames.

Solution:

To resolve the issue, we need to add a host file entry resolving the hostname of the failed server to the IP address of the now active server and restart the storage/taskcontroller service.

Has anyone of you come across a similar situation? We wish to avoid making host file entries and make the failover process seamless.

[GertjanA]

Interesting. MSMQ being a Microsoft app, you might need to talk to them too. The easiest (but definitely not the fastest) method would be to make sure the archives are alligned to the storage service.

I once got a query somewhere which finds archives which use the wrong storage service, see attached txt. run this against your directory database.

[CConsult]

Hi,

and yes I have had a similar problem some years ago. We used host file entries to solve it.

[arnoldmathias]

True @CConsult. This somewhat hinders the effectiveness of building blocks in an Enterprise Vault environment. However, whenever we do a switchover, we update host files on all EV servers via Powershell and restart the Storage Service on all servers. Its a rudimentary method, but it works.

[arnoldmathias]

Thank you the script, @GertjanA

Yes we are running a similar script to find the cross references between Exchange and EV. In a large environment, it is very difficult to align the mailbox with it's corresponding archive as the databases are moved acrosss the DAG servers for maintenance or other reasons. If only the Move Archive feature was improved (ofcourse we can spend $$$ on a 3rd party utility) to quickly move the archives, we could have achieved alignment with Exchange fairly quickly.

Technically, I believe Microsoft would not be able to help with fixing the server references in the Outgoing Queues as MSMQ is unaware of the EV switchover. If MSMQ references the EV Alias instead of EV server hostname, that should resolve the problem.