Upgrade from EV 8.0.5 to 9.0.1 is taking a huge time

[Cdee]

Hi All,

I have upgraded my Symantec enterprise vault from 8.0.5 to EV 9.0.1.

The setup went smoothly without error.

Then i followed the steps from symantec site for starting the services after upgrade.

1) Started the EV admin service.

2) Started the Directory service and got the event of confirmation that the directory has been upgraded.

3) Then started the EV storage service and i saw the event 6958 that the upgrade process of the store database has started

but its now more than 48 hours that the ugprade process is running and i do not see event 6959 which will say that the DB is upgraded,  My vault store size on SQL server is around 103 GB.

Just want to know how can i monitor the progress of this upgrade as i do not see any errors in the event viwever.

Any inputs will be highly appreciated.

[TonySterling]

How many ev servers do you have?

How many vault stores?

Does the SQL transaction log have plenty of space?

[JesusWept3]

It shouldn't take that long to be honest How many ev servers do you have? What you might be finding is that it can't put the vault store dbs in to single user mode Do you do mirroring at all? If the database is mirrored you have to disable that first then try again due to the fact that mirroring stops the databases from going in to single user mode My advice would be 1. Stop all enterprise vault services 2. Restart the sql services R. Start the ev admin, directory and storage service and then see if it completes The reason being is that its more likely than not something has a persistent connection to the database such as discovery accelerator or even SQL Management studio, restarting the sql services will kill these open. Connections

[Cdee]

I did the following steps as mentioned above by you. But still it wont help me.

I have restarted the SQL server and the server now is in recovery mode and the database is getting recoverd.

Recovery of database 'EVDb(11) is 43%% complete (approximately 5253 seconds remain). Phase 3 of 3. This is an informational message only. No user action is required.

Symantec suggested.

1. Stop all the upgrade procedure. (including Restart SQL server)

 2. Restore EV Directory DB.

 3. Start the directory service again and wait for the event log.

 Does this make sense to you guys???

[Cdee]

1 SQL server 1 Journaling server (In SQL DB) 1 CA and 1 DA server

All the DB is on 1 SQL server. The drive space is more then 3 times of the DB size.

[Paul_Grimshaw]

So does it stay in recovery mode at 43%? If not then I would just wait for SQL to do its thing. Alternatively scan the internet for advice on the same scenario as there are lots of articles out there. Or as has been suggested restore a good know copy from before the upgrade

[Cdee]

No the recovery is completed to 100%

Now i have started the storage service and the update is going on and its more than 12 hrs

In SQL process viewer i can see that the update command coming from EV server. but cant see it is updating any database on the SQL server.

I am totaly lost on this any inputs will be highly appreciated.

[GertjanA]

I strongly suggest to call support, and log a prio 1 (site down) to get assistance.

My experience is that it might take a little while to upgrade the databases, but not 2 days.

It might be best to:

Revert back to the 8.05 environment: Stop upgrade, restore 8.05 databases, re-install 8.05 binaries, start EV, point to existing db.

Then, take a backup of the ev databases, put them in a testlab. Use (when possible) same name so you do not need to change rows in database, then perform upgrade there. It might be something is wrong with your databases, and then you need to work with support to figure out what.

[Cdee]

I have already opened up a critical case with symantec.

And they have ifnormed me that as we are not getting any errors there is no issue in upgrade so we have to wait and watch, they say this upgrade latency is due to SQL performance issue which is causing upgrade process to go slow. Also when we check the Activity on the sql server the update command shows an increase in IO, and looking at that my SQl team says that the upgrade is still in progress and the query is not in hung or block state.

So keeping my fingers cross to see what happens next as there is no error on ev servers or sql server in app logs...

[GertjanA]

I'll keep my fingers crossed too for you!

You DO have good backups I hope?

[Cdee]

Yes i have taken the backup of the journal DB from SQL and also had pulled one of the HDD from server before upgrade process to be in safer side.

I hope that stage does not come up.

[Paul_Grimshaw]

I must admit I have never heard of a Vault Store upgrade as part of the whole upgrade process taking that length of time

[Cdee · Accepted solution]

Phew!!!!!! Finally!!!!!!! today it got completed and i was able to see the lovely event id's

6959,7036 and 6221.

also the upgrade to CA and DA also complted within few mins and things are back to normal.

Thanks to all for their inputs. :)

[Jeff_Shotton_2]

This sounds very much like a case I was helping one of the frontline TSE's with.

Basically, you need to check how many records exist in your journalarchive table. There is an update that takes place to the journalarchive table, and the length of that update is proportional to the size of this table. Considering the length of time it took, you may have as many records in there as the saveset table. All of which indicates a problem.

In essence it means either:

1) You haven't been backing things up properly

2) You haven't been indexing things properly

3) EV hasn't been post-processing the records properly.

If im right, then over half your vault store database consists of data which needs to be processed and removed.

Run the following in SQL:

Select count(*) from saveset

Select count(*) from journalarchive

And let us know the results....though if the two come back with similar numbers, I'd get back on the phone to support.

Regards,

Jeff

[Cdee]

Yes Jeff that was my case you were helping Rahul.

As he infomred me that there were more then 80 million saveset and 80 million watchfile it was taking a lot time..

[Jeff_Shotton_2]

...which you need to resolve. You shouldn't have 80 million records in the journalarchive table/watchfile table. The table isn't designed to take that sort of load - it's temporary transactional data, and unless you sort it out the system is going to come to a grinding halt....for example  - when you do an upgrade :)

Normally the buildup is due to not realising you still need to back up the savesets in EV8 when the storage option 'immediately after archive' is set. This is different from EV2007 where you could get away with not backing up the EV store. In EV8 it is mandatory. As a slight variation on the theme, sometimes it is due to using backup software which doesn't clear the archive bit.

If you don't sort this out now it's just going to get worse, and my guess is if you implement vault cache you can kiss the SQL server goodbye.

You will likely need to work with support to do this as the worst case I've seen previously was 20 million records and we ended up purging the table (after running checks to make sure the data could actually be safely purged) because SQL simply could not cope with clearing it down programmatically.

Regards,

Jeff

[GertjanA]

Hi Jeff,

That 2 queries you mention, I assume you run these against the VaultStore databases? Especially the JournalVaultstore?

Thanks

(we'll be upgrading to 9 somewhere in the future, and I would like to check these counts to be sure!)

[Jeff_Shotton_2]

The table name journalarchive is actually slightly misleading. It is in fact more important for mailbox archiving than journal archiving.

The table historically retained a record of items which needed to be backed up and/or indexed in EV2007. In EV8+ we use it to retain tranasction history for updates to vault cache etc (hence my comment about EV_Learner likely to have SQL issues when enabling vault cache).

Thus you will see a records in the journalarchive table due to normal activity. If you see the same number of rows as in the saveset table, and you have been running EV for a while, you potentially have an issue as the records should not persist longer than the transaction history retention period (set at the site level)

If there are legitimately millions of rows in the journalarchive table, then expect the upgrade to take a while.

By the way, EV9 sp2 is out now.

Regards,

Jeff

[Rob_dos_Ramos]

HI All

Having the same issue. Busy watching the event logs with bated breath.

Ran the Select count(*) from saveset script against one of the journal databases and got this result, 108314238  and the other DB 77218895

When I tried the other part of the script

Select count(*) from journalarchive I got some error. Is there anyway of checking if the upgrade on the SQL server is going through correctly? I am not the best at SQL so any ideas would be great.

thanks

Rob

[Cdee]

Hi Rob,

I truely Understand your situation as i was also in the same when i was upgrading it.
The only reason you can see the upgrade progress is due to fact that update command is adding enteries to the table and hence it goes in lock mode so there is no query which can help you to find where the process has reached.
Keep you fingures cross and wait and watch.