Upgrade path to EV 9.03

[goof717]

hello all.  thanks for taking the time to read and help...

i am a noob to EV and the first task ive been given is to get EV ready for Exchange 2010...thus i will need to upgrade;  we are currently running on 8.0.  I have been reading the upgrade guide and we have multiple servers.  1 for Exchange vault mail.  1 for discovery accelerator. 1 for File share and sharepoint.  In the guide it doesnt say what goes first ..if any.  And from looking at the upgrade path for DA it first needs to be on SP5.  Do all the servers need SP5 or just the DA?  And for the 9.03 upgrade what server goes first or do I have to run them at the same time?  here is my breakdown from the upgrade guide.  Please provide feedback and recommendations.  Thank you again for the time and effort.

Steps to upgrade:

-EV server preparation (in order):

     -Backing up the system:

          -backup all EV data (store partitions)

          -watchfile table should be reduced in size (report for size: http://evserver/enterprisevault/usage.asp

          -Awaiting Backup column

          -backup changed language files if any changes have been made

     -Run EV Deployment Scanner

          -must be local admin..

          -vault service account

     -setting database permissions

          -vault service account must be owner of all EV databases; enalbes db schema and updates

     -Allowing the MSMQ queues to empty

          -?how?

     -Checking the archiving and expiry schedules

          -disable archiving and expiry before upgrading...leaves time to examine a good/checked upgrade

          -first archiving run will be longer

     -Multiple servers: upgrading EV software

          -install the EV 9.0.3 software:

               -log on as vault service account

               -computer management > services and applications

               -Services > stop EV admin service (stops admin service, directory service and others)

               -stop all EV services or apps (admin console and accelerator manager)

               -close any applications that are running

               -load media

               -go to \symantec enterprise vault 9.0.3\server

               -setup.exe

               -go through the components

               -restart and log back in as service account

               -do not start any services

          -upgrading the Directory database:

               -on one EV server...start the EV Directory service

               -service start kickoffs the upgrade...upgrades schema and syncs policy settings

               - 2 events to check for: 8785 - start directory update; 8576 - completed (could be hours)

          -starting Storage service and upgrading storage databases:

               -start EV storage service

               -storage databases usually require a database schema upgrade

                    -if required...storage service upgrades each fingerprint database and each vault store db

                    -event 7035: upgrade of db has started - fingerprint

                    -event 7036: upgrade of db has completed - fingerprint

                    -event 6958: upgrade of store db has started

                    -event 6959 - upgrade of store db has completed

               -wait for event 6221 - storage service started...then continue

          -backing up the upgraded EV databases

               -stop any running EV services

               -backup Directory database

               -backup each vault store database and fingerprint database (if they were upgraded)

          -extra config tasks

               -install scom management pack

               -scom operations console>import wizard> enterprisevault.mp

[JesusWept3]

pretty much on the money there to be hoest.
My suggestion is don't wait to be in backup mode or backup data to run the Deployment scanner, just install the newer version, run it, and then fix whatever issues are discovered, maybe a missing hotfix or unset registry key or something such as that.

Then once you are happy with the deployment scanner, go ahead and do the upgrade.
The MSMQ's being clear isn't such a big deal, just make sure that the A1 and Storage queues are 0, anything else such as A5 is a waste of time trying to wait that out.

If you do have items in your A1 or Storage queues, you can't be in backup mode for them to clear out, as that will just put EV in to read only mode.

There is a big issue regarding the amount of items in the WatchFile tables and JournalArchive tables as if its in the hundreds of millions territory then you are looking at a very slow and painful upgrade, but as long as you have your backups running correctly or using the trigger files correctly, these tables should be good to.

Typically the way that I do it is the following.

1. Log on to each EV Server as the EV Admin
2. Start the setup.exe on each server and go through the screens until it tells you it will stop these services
3. Make a backup of the databases right there and then so if you need to fall back you can
4. Make sure you close all the SQL Management studios etc and anything else that might stop the databases going in to single user mode for the upgrade
5. Go through each machine you RDP'd to and hit next so that it stops the services and begins the installation
6. Wait for the machines to say they need to be rebooted and reboot each one as it prompts you
7. On the primary or first server where the alias points to, log on to the machine after a reboot

Note: Logging on for the first time after an upgrade can seem like its stuck, it takes a couple of minutes for the screen to say that the installation is complete, so don't think the server is hung, just use that time to go through the other machines and log on

8. Back on the first server, start the Admin and Directory Service.
9. Await the event on that server saying the directory database has been upgraded
10. Then start the Storage Service and wait for the event to say the storage database has been upgraded
11. Go through each other machine and start the Admin, Directory and Storage service and wait for the even tot say the storage has been upgraded
12. On the last server to finish the upgrade of the storage database, that server will then perform the upgrade of the fingerprint database, it will also post an event to say its complete

After that you can start all the other services (shopping, Indexing, Task Controller and what not) and perform your tests.

In previous upgrades i've done using those steps I can upgrade about 20 servers in less than an hour

[AndrewB]

yes, DA is the only one that needs to be at SP5 first. you can do the rest directly from 8.x

[goof717]

Thank you both for the quick responses... so just to confirm a few more things to calm my nerves...

-Where do I find the WatchFile tables and JournalArchive tables?

-stopping SQL management studios on the EV servers itself? correct?

-so it doesnt matter which server I start firt as long as the Discovery server is up to SP5 ...anyone of them could be first.

-SP5 upgrade...anything special about that?

-lastly any changes for end users? new client? or agents that need to be installed on Exchange for OWA? or SharePoint?

Appreciate your time and efforts.

[goof717]

Sorry had another question... 

-if I upgrade the DA to SP5 but the other EV servers are on 8.0.  will DA still function?  the reason being we might want to do SP5 and then a few weeks later to 9.03. smart? worth it?

to quote the upgrade guide:

"If the major version of DA is the same version of EV, the minor version (Service Pack) of DA must be the same as, or later than, the minor version of EV."

so this says it should work...

[TonySterling]

Yes, that will be fine, it just won't work the other way. 

[JesusWept3 · Accepted solution]

-Where do I find the WatchFile tables and JournalArchive tables?

They're in the Vault Store Databases (such as EVVSMyStore)
you would just do a simple
SELECT COUNT(*) FROM EVVSMyStore.dbo.WatchFile
and SELECT COUNT(*) FROM EVVSMyStore.dbo.JournalArchive WHERE BackupComplete = 0

-stopping SQL management studios on the EV servers itself? correct?

Just closing the SQL Management Studio should be good enough, but what can happen is if you do a query against a database and don't close the studio, then you may find that the upgrade will error out, because it needs to put the database in to single user mode, which it can't do with open connections running against it

-so it doesnt matter which server I start first as long as the Discovery server is up to SP5 ...anyone of them could be first.

To be honest I would do the upgrade at the point where legal teams know that the upgrades are going to mean the Discovery Accelerator services will not be available to use, and get them all upgraded in a straight shot... concentrate on getting EV server upgraded and functional first, and then upgrade your DA to 8 SP5 then to 9 SP3, and make sure you install the EV9 SP3 binaries on that DA server also.

-SP5 upgrade...anything special about that?

None that i'm aware of

-lastly any changes for end users? new client? or agents that need to be installed on Exchange for OWA? or SharePoint?

Clients are recommended to be upgraded but you can take your time distributing the new client, its just that the new clients fix a lot of known issues and what not, if i were you I would be tempted to have everyone on the unified 10 SP1 client rather than the other clients in EV9.

But if the client stays on EV8 SP3 and you're on EV9 SP3 on the server, nothing is going to fail or break necessarily

OWA and Sharepoint will definitely need to have the extensions and webparts upgraded

[goof717]

 

-Where do I find the WatchFile tables and JournalArchive tables?

They're in the Vault Store Databases (such as EVVSMyStore)
you would just do a simple
SELECT COUNT(*) FROM EVVSMyStore.dbo.WatchFile
and SELECT COUNT(*) FROM EVVSMyStore.dbo.JournalArchive WHERE BackupComplete = 0

We did the JournalArchive count and saw it was nearing 100million.  And saw an event in the logs saying that:

There are 99439763 savesets that are waiting to be indexed and/or waiting to be backed up or replicated in Vault Store 'MailVaultStore'.

Check that your Indexing and Storage services are running.

Review your procedures and make any changes needed to ensure that backups happen in a timely fashion.

Our setting for removing safety copies is Immediately after archive.  I did some research and read the following article: https://www-secure.symantec.com/connect/articles/changes-backup-procedures-enterprise-vault-8

where if I am interpreting it correctly...the JournalArchive is not scanned and it is sets the BackupComplete to 1.  so it never resets to 0.

we ran this command: 

SELECT COUNT(*) FROM JournalArchive WHERE BackupComplete = '0'  

result was 99439763.

so having it set to immediately archive after archive is causing this since it is not being passed through the watchfile table.  

my question then is what should we do?  I know you can manual delete the attribute.. using:

http://esd.element5.com/publisher/50364/proxy.cgi/62438/business/support/index?page=content&id=TECH6...

attrib -a /s *.*
 

but will this result in data loss? or stub shortcut loss for users? 

I didnt understand this part: 

In order to start post processing please follow the steps below:
 

Set the Vault Store in backup mode
Set the Vault Store to clear backup mode

will i have run the attrib -a/s *.* command first and then set the vault store to backup mode and then wait for what? and then set the vault store to clear backup mode? 

thanks again for your time and efforts.

[JesusWept3]

I wrote that article :)

So basically the old behavior before EV8 was that the only thing that went to WatchFile were items that were set to "After Backup" and BackupComplete = 0, until the backup program removed the archive attribute and then storage file watch scanned and removed the watchfile entry and also set BackupComplete to 0

However now in EV8+ everything goes to JournalArchive and WatchFile and needs to be backed up, even if its set to Immediately After Archive.

So will running the attrib command produce dataloss?
Not really, the thing is, because you're in an Immediately After Archive situation it will have set all Pending items to shortcutted items anyway. What it does mean though is you have to be confident in your backups that it really has gotten all those items.

If you're doing snapshots for the backup then the archive bits wont be removed, if you're using something like a netapp, the archivebits doesn't matter because windows cant query for it properly.

If you are confident about the backups, my suggestion would be to use the IgnoreArchiveBitTrigger.txt file instead of the attrib command, which will be far quicker for you.

The IgnoreArchiveBitTrigger.txt is usually placed on a file system that can't support the Archive attribute, and after the backup run has completed, the txt file gets created, and EV Reads the text file and then scans the system.... Anything On or before the create date of the file will be treated as backed up.

so this in essence is just the one scan.

However if you do the attrib command, windows will take forever to go through every single file and folder and remove the attribute, and then EV has to scan the same files, so if it took 2 hours to remove the archive bit, its going to take 2 hours to scan etc.

Going in and out of backup mode is just a way to force EV to look at the files again, otherwise you may have to wait the normal 2 hours for the StorageFileWatch to run its scan again.

[goof717]

thanks for the article... it helps. and for the quick responses.

we use netbackup 7.0 for backing up the partitions.  we do full every weekend and incrementals during the weekday. does it only clear the savesets after a full? or both?

the IgnoreArchiveBitTrigger.txt is should be placed by the netbackup software. so I am guessing not much admin intervention here.  thus i will try this out.

the 99million has been a builtup from 2009 ... i am fairly confident we have a good backup by now. :)

also do savesets cause any problems? performance? having that many sounds ridiculous.  

[JesusWept3]

you will have more than 100 million savesets :)
the 100 million is 100 million email, however an email with a single large attachment could create 3 items, and depending on your sharing levels and how much is truly unique, you may have more than 100 million items etc.

just means you archive a lot, thats all!
And you don't have to wait for netbackup to do it, i mean you should configure it that way, but just create the txt file yourself now and get the process kicked off.

And i know a full backup will remove the backup , but the others are a little murky for me, i think its incremental that removes the archive bit?  really not sure