12-03-2012 06:38 AM
I'm trying to perform a full disaster recovery test of my Exchange / EV environment. Running Exchange 2010 and EV 10. The Exchange recovery works without issue. The integration with EV appears to function - retrieving items is working, the journal archive appears to be working. But there's something not right with the journal indexing. I'm not able to search for new journaled items and the archive vault stops at the date of the backup.
The backup is done with Dell Appassure Reply which takes a snapshot. The SQL database and the Evault server are backed up at the same time while the stores are in backup mode.
Solved! Go to Solution.
03-26-2013 05:39 AM
I finally got a good restore with both SQL and the EV data from the same point in time while everything was in backup mode. The current journal index is still showing failed, but I can search up to the point of the backup.
The server is 10.0.1.1171. It was upgraded many times from the old KVS versions through current. I believe it was on 2007 at one point.
All the partition paths are present and in the correct place. The restore was a full snapshot - everything is identical.
Running EVSVR verify on the index comes up with many errors like:
2013-03-22 21:50:42 Event Output: Unable to complete retrieval request Reason: The system cannot find the path specified. (0x80070003) Vault: 12 13 2012 to Vault Id: 10F1CB019EF2ECE48A220E7F980872F251110000archive.domain.com Item Id: 201303213854289~201303210821230000~Z~50DB6BC227898B02E2138DD7F731FAD1 Reference: Vault/RI Extended Item Info: Supplementary Info: (0xc0041ae2)
I remember when I was on 9, I was told by Symantec there was a way to recover the index if it was completely lost. Is that an option? Since this discussion was begun I cut a new journal index so that it would be smaller and easier to recover.
12-03-2012 06:42 AM
Silly questions maybe, but ...
After your test recovery.. are items being picked up from the journal mailbox? Is the indexing service running? Are the indexes and journal vault stores both back out of backup-mode?
12-03-2012 06:47 AM
You may also want to run a synchronization against the journal indexes anyway which is good practice after a DR scenario as well
But agreed with rob, sounds like its just in a read only state
12-03-2012 06:56 AM
It makes sense that it would still be in backup mode after the restore. I just confirmed that it is out of backup mode. How long should it take for the new items to show up considering there has been no email at all since the backup. How is the synchronization performed?
12-03-2012 07:05 AM
If there has been no email since the backup, then nothing will be in the journal mailbox, and therefore nothing new will be added to the journal archive.
Or am I missing something there?
EV10 index synchronisation is done from the 'Manage Indexes' tool, which you get to by right clicking 'Indexes' in the VAC.
12-03-2012 07:10 AM
Sorry, I wasn't clear. There was no mail since the backup except my test emails today.
12-03-2012 07:17 AM
I just did a synchronize and it took about a minute to run. Still don't see the new items and the date still ends at the backup time.
12-03-2012 07:28 AM
Where have you done the restore/recovery to?
Is journaling on the Exchange-side of things configured in that environment?
If it isn't then sending/receiving a few test emails won't add items to the journal archive.. since there is nothing being journaled by Exchange.
If journaling is enabled on Exchange, then have a look at the journal mailbox. Are there any items in it?
12-03-2012 07:29 AM
In the log I see:
Index volume is marked as failed.
Archive Name: Journal Archive
Index Volume ID: 1956A8B1D0F8AC24EAE14D8BDF7AE99471110000archive.domain.com_475
Reason: None
Error Type: FrenzyErrorDetected
12-03-2012 07:31 AM
It's a full recovery environment. All of Exchange and Evault is present. Items hit the journal mailbox, disappear but never show up in the search.
12-03-2012 07:38 AM
Looking in Google-land, the frenzyerrordetected coule be solved by:
http://www.symantec.com/business/support/index?page=content&id=TECH196901
However I would expect your production journal archive would be suffering this same problem - simply doing a restore of the environment isn't going to cause the index to be rebuilt.... unless you issued a rebuild request?
Have a look at the Journal Archive index.. is it marked as failed?
The event you referenced above, when was it logged in relation to:
* The backup
* The restore
* The synchronise
* Ingesting new items (your test emails)
12-03-2012 07:46 AM
The journal archive index is marked as failed. The event occurred during the synchronize. But the synchronize said it was successful.
I think it must be some sort of corruption. The restore process must be flawed. Restoring 4TB seems to be a challenge. I notice that when it reboots, it wants to run chkdsk. I let it run the last time.
Is a rebuild worth trying?
12-03-2012 08:08 AM
Don't rebuild, you'll just run in to the same errors
The frenzy being detected means that it had 10,000 errors in 7200 seconds, so you need to find out what those errors are and resolve them.
Its most likely something like path not found for the source DVS files or something like that, once you determine what the errors are and correct them, then resynchronize..but again a rebuild will just give you the same issue
12-03-2012 08:12 AM
Any way to narrow it down? The example errors I see specify a CAB file. Mine doesn't. Mine is also showing errors 1-25.
12-03-2012 08:17 AM
Verifying the index volume comes back successful. I'm not able to verify the index location - it's greyed out.
12-03-2012 08:18 AM
OK so the questions would be:
What exact version of EV 10 are you running? (Base? SP1? SP2?)
Have you ever been on EV2007 or earlier or did you start with EV8 or above?
I think we'd need to see some actual example errors from your logs to see what is actually going on.
You can test a full dump through EVSVR to see if that helps retrieve the items or narrow the issue down further, my best guess is that its most likely pointing to an invalid path somewhere
So like if you do
USE EnterpriseVaultDirectory
SELECT * FROM PartitionEntry
The partition paths specified there, are they correct and do they exist?
Or did drive letters get changed or were they restored to a slightly different place etc?
12-05-2012 07:17 AM
Rob, URL mentioned in your post says
Solution
This issue will be fixed in Enterprise Vault 10.0.3 **not yet released**
http://www.symantec.com/business/support/index?page=content&id=TECH196901
12-05-2012 07:21 AM
I think the restore was not done properly. It appears that the SQL database was newer than the EV storage partitions. I need to get a restore from the same point in time for both.
12-05-2012 08:03 AM
Yep - that's right.
03-26-2013 05:39 AM
I finally got a good restore with both SQL and the EV data from the same point in time while everything was in backup mode. The current journal index is still showing failed, but I can search up to the point of the backup.
The server is 10.0.1.1171. It was upgraded many times from the old KVS versions through current. I believe it was on 2007 at one point.
All the partition paths are present and in the correct place. The restore was a full snapshot - everything is identical.
Running EVSVR verify on the index comes up with many errors like:
2013-03-22 21:50:42 Event Output: Unable to complete retrieval request Reason: The system cannot find the path specified. (0x80070003) Vault: 12 13 2012 to Vault Id: 10F1CB019EF2ECE48A220E7F980872F251110000archive.domain.com Item Id: 201303213854289~201303210821230000~Z~50DB6BC227898B02E2138DD7F731FAD1 Reference: Vault/RI Extended Item Info: Supplementary Info: (0xc0041ae2)
I remember when I was on 9, I was told by Symantec there was a way to recover the index if it was completely lost. Is that an option? Since this discussion was begun I cut a new journal index so that it would be smaller and easier to recover.