EV The learning Curve
I have written this article as it covers all the points I wish were made available to me when I started to support our Enterprise Vault environment. Maybe a lot of these topics are covered in the basic EV training; however in these times with IT budgets stretched more and more…In our situation EV training was a nice to have not an essential. So I learnt as I went along thinking on my feet.
Windows 2003 SP1 & MSMQ
I have had bitter experience with SP1 MSMQ on Windows 2003. It seems there is a serious problem with this version of MSMQ whereby the outgoing queues are never processed.
In our environment we have six Exchange active/passive clusters and an EV box serving archiving for each cluster. In the situation where the Exchange mailbox and the EV archive are not on the same cluster / EV server pair messages would be sent to the outgoing queue telling another EV server to archive a particular piece of email. As the outgoing queues never got processed the archiving did not complete. So the result is endless users are reporting their emails are not fully archiving, and we as the email support team had to move endless mailboxes onto the Exchange server paired with the correct EV server.
Put in short, if any of your EV servers are currently running Windows 2003 SP1 upgrade them to SP2 as soon as practical. The sooner you do this, the less pain you will feel as an email admin.
MSMQ & Journaling
The size of the MSMQ folder (In our environment d:\msmq) has caught us out a couple of times now.
As most support guides will tell you journaling should only be enabled on production systems for limit periods of time. Fair enough, it’s easy as long as you know this. I have found in the past once the size of the MSMQ folder gets past the limit set on the MSMQ properties, general tab the mailbox archiving task will fail.
Building blocks failovers
This was not obvious to me or my work colleague when we first failed over an EV server to a warm standby.
When a warm standby server is used in a disaster recover scenario and becomes a production server. You will notice in the administration console the services will say hosted on either the original server or the standby server now in use (I cannot remember which off the top of my head). This essentially means the EV system at some point expects you to fail back to the original server once the cause of the problem has been resolved.
From what I understand further failovers of the same service to a third server are not possible until the service has been restored to the original server.
Opening the admin console and not seeing all EV servers
This is more than likely EV training 101; however when I first started out only when I logged in with the EV service account could I see all EV servers. This is a simple one (when you know how :) ).
In the admin console logged on as your EV service account:
• Right click server concerned and click properties.
• Go to the admin permissions and add yourself into the list.
You will now be able to open the admin console as yourself and see your servers.
Archiving Trigger Bit & KVS Storage keys
If your storage device (SAN, Netapp etc) does not support removal of archive bits from files as they are backed up to tape then EV can be setup to use the archive trigger bit in order to work out which email items can be removed from the Exchange mailboxes.
There are probably multiple configurations for how this can work in different environments however below is the screenshot of the registry for one of our EV servers.
Suffice to say if any one of these keys is not set to 1 outside of the backup window then archiving does not function to varying degrees.
In our environment there is a trigger bit file. This file is recreated with a new timestamp every night after the backup runs by way of the following command:
echo off > "%StoragePath%\IgnoreArchiveBitTrigger.txt"
The timestamp on this file relates to the time and date which items are archived up to.
DocMessageClass.exe – Handy utility for checking archiving status in a mailbox
This is a standalone free utility for checking / changing the message class of messages in a mailbox. See http://www.publicshareware.com/public-share-outlook-utilities.php for further info and a download link.
This tool has saved me plenty of time when it comes to searching through mailboxes looking for messages which have not been correctly archived. The EV message classes are easily identifiable, and logical.
Watch File and Journal Archive Counts
Depending on your configuration you may use one of these tables or both. I only know we operate off of the watch file counts.
I have found this invaluable information for working out if the backups for EV have been successful and / or if there is a back log of messages which have not been archived / backed up.
For my environment the values for each table do not tend to exceed 5000. This is for a user population of around 6500 – 7000. Once I get to around 8000 or more I start to ask my backup guys questions.
Keeping an eye on these values on a day to day basis has definitely helped keep our EV environment in tip top shape.
SQL query examples:
Use EVMailboxDB01
Select count(*) AS D002EVMBX01 from WatchFile
Use EVMailboxDB01
Select count(*) AS D002EVMBX01 from JournalArchive
Ignore Desktop Settings registry key
In our environment we have various policy settings which are applied to the users EV experience. For example the delete from archive has been disabled.
The following registry key can be set on your management servers or admins desktops to enable all buttons for EV within Outlook.
HKEY_CURRENT_USER\Software\KVS\Enterprise Vault\Client
Create a dword value with 1 as the value.
"IgnoreDesktopSettingsUI"=dword:00000001
Queue Explorer
Have a read of http://www.cogin.com/mq/. This application gives you much more power and flexibility when working with MSMQ. Although I am not recommending a particular software solution I wanted to mention the options and controls built into MSMQ are frankly, missing.
With a utility like this you are able to remove a single message within a queue. Within MSMQ your only option is to purge the entire queue.
There are a lot of other feature available, and similar there are probably other products available if you do not like the particular one mentioned above.
I have written this article as it covers all the points I wish were made available to me when I started to support our Enterprise Vault environment. Maybe a lot of these topics are covered in the basic EV training; however in these times with IT budgets stretched more and more…In our situation EV training was a nice to have not an essential. So I learnt as I went along thinking on my feet.
Windows 2003 SP1 & MSMQ
I have had bitter experience with SP1 MSMQ on Windows 2003. It seems there is a serious problem with this version of MSMQ whereby the outgoing queues are never processed.
In our environment we have six Exchange active/passive clusters and an EV box serving archiving for each cluster. In the situation where the Exchange mailbox and the EV archive are not on the same cluster / EV server pair messages would be sent to the outgoing queue telling another EV server to archive a particular piece of email. As the outgoing queues never got processed the archiving did not complete. So the result is endless users are reporting their emails are not fully archiving, and we as the email support team had to move endless mailboxes onto the Exchange server paired with the correct EV server.
Put in short, if any of your EV servers are currently running Windows 2003 SP1 upgrade them to SP2 as soon as practical. The sooner you do this, the less pain you will feel as an email admin.
MSMQ & Journaling
The size of the MSMQ folder (In our environment d:\msmq) has caught us out a couple of times now.
As most support guides will tell you journaling should only be enabled on production systems for limit periods of time. Fair enough, it’s easy as long as you know this. I have found in the past once the size of the MSMQ folder gets past the limit set on the MSMQ properties, general tab the mailbox archiving task will fail.
Building blocks failovers
This was not obvious to me or my work colleague when we first failed over an EV server to a warm standby.
When a warm standby server is used in a disaster recover scenario and becomes a production server. You will notice in the administration console the services will say hosted on either the original server or the standby server now in use (I cannot remember which off the top of my head). This essentially means the EV system at some point expects you to fail back to the original server once the cause of the problem has been resolved.
From what I understand further failovers of the same service to a third server are not possible until the service has been restored to the original server.
Opening the admin console and not seeing all EV servers
This is more than likely EV training 101; however when I first started out only when I logged in with the EV service account could I see all EV servers. This is a simple one (when you know how :) ).
In the admin console logged on as your EV service account:
• Right click server concerned and click properties.
• Go to the admin permissions and add yourself into the list.
You will now be able to open the admin console as yourself and see your servers.
Archiving Trigger Bit & KVS Storage keys
If your storage device (SAN, Netapp etc) does not support removal of archive bits from files as they are backed up to tape then EV can be setup to use the archive trigger bit in order to work out which email items can be removed from the Exchange mailboxes.
There are probably multiple configurations for how this can work in different environments however below is the screenshot of the registry for one of our EV servers.
Suffice to say if any one of these keys is not set to 1 outside of the backup window then archiving does not function to varying degrees.
In our environment there is a trigger bit file. This file is recreated with a new timestamp every night after the backup runs by way of the following command:
echo off > "%StoragePath%\IgnoreArchiveBitTrigger.txt"
The timestamp on this file relates to the time and date which items are archived up to.
DocMessageClass.exe – Handy utility for checking archiving status in a mailbox
This is a standalone free utility for checking / changing the message class of messages in a mailbox. See http://www.publicshareware.com/public-share-outlook-utilities.php for further info and a download link.
This tool has saved me plenty of time when it comes to searching through mailboxes looking for messages which have not been correctly archived. The EV message classes are easily identifiable, and logical.
Watch File and Journal Archive Counts
Depending on your configuration you may use one of these tables or both. I only know we operate off of the watch file counts.
I have found this invaluable information for working out if the backups for EV have been successful and / or if there is a back log of messages which have not been archived / backed up.
For my environment the values for each table do not tend to exceed 5000. This is for a user population of around 6500 – 7000. Once I get to around 8000 or more I start to ask my backup guys questions.
Keeping an eye on these values on a day to day basis has definitely helped keep our EV environment in tip top shape.
SQL query examples:
Use EVMailboxDB01
Select count(*) AS D002EVMBX01 from WatchFile
Use EVMailboxDB01
Select count(*) AS D002EVMBX01 from JournalArchive
Ignore Desktop Settings registry key
In our environment we have various policy settings which are applied to the users EV experience. For example the delete from archive has been disabled.
The following registry key can be set on your management servers or admins desktops to enable all buttons for EV within Outlook.
HKEY_CURRENT_USER\Software\KVS\Enterprise Vault\Client
Create a dword value with 1 as the value.
"IgnoreDesktopSettingsUI"=dword:00000001
Queue Explorer
Have a read of http://www.cogin.com/mq/. This application gives you much more power and flexibility when working with MSMQ. Although I am not recommending a particular software solution I wanted to mention the options and controls built into MSMQ are frankly, missing.
With a utility like this you are able to remove a single message within a queue. Within MSMQ your only option is to purge the entire queue.
There are a lot of other feature available, and similar there are probably other products available if you do not like the particular one mentioned above.
