EV items can no longer be recalled following EV8 to EV10 upgrade

[KeirL]

Hi

I’m after some thoughts on an issue that’s developed during an EV upgrade. In summary all the users who’s archives reside on a particular EV server are now no longer able to store or recall archived items.

I’ll explain the background:

The environment consists of two sites each with a single EV 8.0.1 server each of which does Exchange mailbox archiving only – the Exchange environment is Exchange 2003 SP2 and there is a mailbox server on each site. Each Exchange server is archived into its own Vault Store Group which has just a single Mailbox Vault Store within it. There are about 150 users per site and there is no journal archives on either site

Also each EV server has a local copy of SQL 2000 installed which host the EV databases. The EV server on site 1 (EVServer01) hosts the Directory Database, Monitoring Database its local VSG fingerprint database and local Vault store database. The SQL2000 server residing on the EV server on site 2 (EVServer02) hosts just it’s local fingerprint and Vault Store databases.

The task is to upgrade this environment to EV10.0.4 and move the databases to local dedicated SQL 2012 servers.

We elected on installing a dedicated SQL2008 SP2 server at site 2 and move all the databases to this server as SQL2008 SP2 is supported on EV8, EV9 and EV10 and would allow us to upgrade EV without the need to keeping SQL server in step. First we had to go to patch to EV8.0.5 as 8.0.1 isn’t supported on SQL2008.

So EV8.0.1 -> EV8.0.5 completed            upgrade was fine with no errors

Then move all databases from both sites to SQL2008 SP2 server on site B – again completed with no errors (old SQL2000 DB’s were backed up first)

Next EV8.0.5 -> EV9.0.5 upgrade completed         upgrade was fine no errors (but I must admit the testing here could have been more thorough – I can’t definitely say we tested BOTH EVservers would access their archives…. We tested the server on site 2 and my colleague (thinks) he tested several users on Site 1…. Certainly no users complained they couldn’t get to their archives.

Next we built 2 x Wn2K8 R2 servers to host the EV10 environment and used the migration wizard to upgrade from EV9.0.5 to EV10.0.4. We followed the migration wizard technote to the letter, preventing user access, creating the migration package and copied the Archives and Indexes to the same volume drive letters and paths on the new servers and changed the DNS alias.

Again the upgrade went without a hitch and the migration import was green ticks all the way.

So then we noticed the problem – users at Site 1 could not recall archived items whereas users on site 2 were all fine. The outlook add-ins had mostly all been upgraded to EV9 prior to any of this work taking place.

The error manifested itself in a number of ways depending on how the item was trying to be recalled. From an outlook perspective the user would be asked for credentials when they double clicked an item and when the DOMAIN\username was supplied the credentials were asked for again and again. From EV Search it was possible to search  no problem but when an item was opened again the credentials were requested constantly. From Archive Explorer a browse would return all the expected results but when an item was tried to be restored an error saying the item could not be retrieved from the storage service appeared.

Also attempts to ‘Store’ just result in the email icon changing to pending and then returning to the IPM.Note state.

I’ve worked through many technotes including http://www.symantec.com/business/support/index?page=content&id=TECH56220

about checking IE settings, IIS authentication, folder permissions for WebApp and Enterprise Vault directories, SPN’s and nothing has worked. Our main troubleshooting approach has been to compare the two EV servers as one is perfect and the other can’t access any archives. Both servers are identical in configuration and credentials/permissions.

Below is a link that almost nails the exact problem

http://www.symantec.com/business/support/index?page=content&id=TECH216779

I get the same Event Viewer error ID but the reason I get is “unspecified Error” however I did run a DTrace on EVSVR for a saveset dump but have yet to fully review it. It did start me thinking about the moving of the databases I did from Site 1 to Site 2. The EV server that is no longer working is the site that has had the DBs moved offsite to Site 2…… could there be some comm’s issue that’s causing this – Deployment Scanner had no issue and all the upgrades went well, plus it can connect to the Directory DB fine……

So I’m hoping for a few thoughts….. :o)

Is there a way I can confirm if I have a corrupt fingerprint database for the affected VSG? Perhaps a SQL query I can run or something with EVSVR? I’m really just looking to localise the issue to either that EVServer, the SQL server or perhaps IIS……

Any pointers appreciated.

More than happy to supply any more details as requested

Many thanks to those who managed to read the whole post :o)

[Merv]

Hi Kier, You say that you get the exact event id 6882? So storageonlineopns is the one throwing up the errors? Any 8390 or 13360 errors at the same time? 1. Upload your dumpsaveset log when it errors. 2. Do a restart of storage service and dtrace storage server , storageonlineopns and check for errors? Also check for errors in the system event logs ?dcom errors? 3. Upload a dtrace log of the dumpsaveset and when you doubleclick a shortcut or link to a saveset. 4. If you can do a check of that saveset id in the vaultstore db saveset table. The dumpsaveset log should tell me if it found the saveset record on the vault store Am doing some research but if i were you i would have a sev1 case open asap with support

[KeirL]

Hi

Many thanks - I will open a support call when I'm back on site in the morning. And will try the things you've suggested.

Yes - I get the 6882 (Unable to complete retrieval request) each time an item is attempted to be recalled

I don't get any 8390 or 13360 errors but I do get an error from WP (might be error 6287) about not being able to fetch the item

The link below again is close to what I get with the exception that these aren't shortcuts migrated from Domino - however, the symptons and much of the error is the same

http://www.symantec.com/business/support/index?page=content&id=TECH173781

I did a client trace bit all it gives is  IDispatch::GetIDsOfNames("MessageClass") failed, error 80020006 as the only error

I think I'll remove and resinstall the Outlook client on the problem EV server - it's definitely on there but perhaps corrupted....

I'll be able to get a lot more info and logs over the next 24Hrs

I need to go through the logs on the old EV8\9 servers to see if this issue occured prior to the platform migration and was also planning to move the SQL databases back to the original site to get back to a situation closer to when things were last working well :o)

Thanks for the help

I'll update this thread with more info soon

[Merv · Accepted solution]

Hi Keir,

- If you did not receive any 8390 or 13360 the problem is unlikely SQL however if could still be missing sisparts

- The way I see things - It's got absolutely nothing to do with Outlook on the EV server as you can't even access items directly via the Web through Search of AE and as searching works I doubt it is an IIS issue too

- We absolutely need the dumpsaveset log and dtrace of dumpsaveset. I suspect issues with the Storage Service and specifically storageonlineopns - so dtraces of that when it starts up and when you try to retrieve or use dumpsaveset. Just in case dtrace w3wp too.

- Remember to check for DCOM errors in the system logs too.

- A quick way to reproduce those  retreival errors is to construct a quick download.asp URL from the IIS logs or from those Event ID: 6287 errors. You just need an archive id and a saveset id in the format

locally on ev server 

http://localhost/enterprisevault/download.asp?VaultID=???l&SaveSetID=???

or remotely

http://evserver/enterprisevault/download.asp?VaultID=???l&SaveSetID=???

[KeirL]

Hi Merv

I've been working with Symantec support this morning and all is now well. I was very impressed with the support - I know they sometimes get a hard time on these forums so it's worth praising them when it's due.

They reviewed lots of logs and effectively determined it was an issue with EV trying to piece together all the OSIS parts before retieving the item. Re-ran the install to re-register all the .dlls and now all works great.

Thanks for you're input - you were definitely on the right lines and stopped me from heading down blind alley's

cheers

Keir

[Merv]

Good to know you are up and running. So solution was a reinstall - yes sometimes the dlls either don't get registered or regkeys are not permissioned properly. In those cases a process monitor log is good filtering for access denied or errors and the time of those errors which can match up with dtrace or event log errors.

Did you try the dll register batch? Fileregister.bat or something along those lines? 

Yes support is improving! Give them a good survey 10/10 they'll love you and remember you for that! Just curious which support  region did you get? 

[KeirL]

Thanks - I let the support guy do all of it so not sure if he tried the fileregister.bat. If he did I missed him doing it.

I think I got the UK as I logged the call about 9am UK time and went straight through to a support engineer call Manish.

thanks

[GertjanA]

Manish is really good. Helped me out a few times too.