cancel
Showing results for 
Search instead for 
Did you mean: 

Full server recovery EV 10

MikeM-2468
Level 4

I'm trying to perform a full disaster recovery test of my Exchange / EV environment.  Running Exchange 2010 and EV 10.  The Exchange recovery works without issue.  The integration with EV appears to function - retrieving items is working, the journal archive appears to be working.  But there's something not right with the journal indexing.  I'm not able to search for new journaled items and the archive vault stops at the date of the backup. 

The backup is done with Dell Appassure Reply which takes a snapshot.  The SQL database and the Evault server are backed up at the same time while the stores are in backup mode.

1 ACCEPTED SOLUTION

Accepted Solutions

MikeM-2468
Level 4

I finally got a good restore with both SQL and the EV data from the same point in time while everything was in backup mode.  The current journal index is still showing failed, but I can search up to the point of the backup.

The server is 10.0.1.1171.  It was upgraded many times from the old KVS versions through current.  I believe it was on 2007 at one point. 

All the partition paths are present and in the correct place.  The restore was a full snapshot - everything is identical.

Running EVSVR verify on the index comes up with many errors like:
2013-03-22 21:50:42                      Event Output: Unable to complete retrieval request      Reason:           The system cannot find the path specified.  (0x80070003)   Vault:              12 13 2012 to   Vault Id:           10F1CB019EF2ECE48A220E7F980872F251110000archive.domain.com   Item Id:            201303213854289~201303210821230000~Z~50DB6BC227898B02E2138DD7F731FAD1   Reference:          Vault/RI   Extended Item Info:    Supplementary Info:                                (0xc0041ae2)

I remember when I was on 9, I was told by Symantec there was a way to recover the index if it was completely lost.  Is that an option?  Since this discussion was begun I cut a new journal index so that it would be smaller and easier to recover.

View solution in original post

19 REPLIES 19

Rob_Wilcox1
Level 6
Partner

Silly questions maybe, but ...

 

After your test recovery..  are items being picked up from the journal mailbox?  Is the indexing service running?  Are the indexes and journal vault stores both back out of backup-mode?

Working for cloudficient.com

JesusWept3
Level 6
Partner Accredited Certified

You may also want to run a synchronization against the journal indexes anyway which is good practice after a DR scenario as well

But agreed with rob, sounds like its just in a read only state

https://www.linkedin.com/in/alex-allen-turl-07370146

MikeM-2468
Level 4

It makes sense that it would still be in backup mode after the restore.  I just confirmed that it is out of backup mode.  How long should it take for the new items to show up considering there has been no email at all since the backup.  How is the synchronization performed? 

Rob_Wilcox1
Level 6
Partner

If there has been no email since the backup, then nothing will be in the journal mailbox, and therefore nothing new will be added to the journal archive.

 

Or am I missing something there?

 

EV10 index synchronisation is done from the 'Manage Indexes' tool, which you get to by right clicking 'Indexes' in the VAC.

Working for cloudficient.com

MikeM-2468
Level 4

Sorry, I wasn't clear.  There was no mail since the backup except my test emails today.

MikeM-2468
Level 4

I just did a synchronize and it took about a minute to run.  Still don't see the new items and the date still ends at the backup time. 

Rob_Wilcox1
Level 6
Partner

Where have you done the restore/recovery to?

Is journaling on the Exchange-side of things configured in that environment?

If it isn't then sending/receiving a few test emails won't add items to the journal archive.. since there is nothing being journaled by Exchange.

If journaling is enabled on Exchange, then have a look at the journal mailbox.  Are there any items in it?

Working for cloudficient.com

MikeM-2468
Level 4

In the log I see:

Index volume is marked as failed.

Archive Name: Journal Archive
Index Volume ID: 1956A8B1D0F8AC24EAE14D8BDF7AE99471110000archive.domain.com_475
Reason: None
Error Type: FrenzyErrorDetected

MikeM-2468
Level 4

It's a full recovery environment.  All of Exchange and Evault is present.  Items hit the journal mailbox, disappear but never show up in the search. 

Rob_Wilcox1
Level 6
Partner

Looking in Google-land, the frenzyerrordetected coule be solved by:

 

http://www.symantec.com/business/support/index?page=content&id=TECH196901

 

However I would expect your production journal archive would be suffering this same problem - simply doing a restore of the environment isn't going to cause the index to be rebuilt.... unless you issued a rebuild request?

 

Have a look at the Journal Archive index..  is it marked as failed?

 

The event you referenced above, when was it logged in relation to:

* The backup

* The restore

* The synchronise

* Ingesting new items (your test emails)

Working for cloudficient.com

MikeM-2468
Level 4

The journal archive index is marked as failed.  The event occurred during the synchronize.  But the synchronize said it was successful.

I think it must be some sort of corruption.  The restore process must be flawed.  Restoring 4TB seems to be a challenge.  I notice that when it reboots, it wants to run chkdsk.  I let it run the last time. 

Is a rebuild worth trying?

JesusWept3
Level 6
Partner Accredited Certified

Don't rebuild, you'll just run in to the same errors
The frenzy being detected means that it had 10,000 errors in 7200 seconds, so you need to find out what those errors are and resolve them.

Its most likely something like path not found for the source DVS files or something like that, once you determine what the errors are and correct them, then resynchronize..but again a rebuild will just give you the same issue

https://www.linkedin.com/in/alex-allen-turl-07370146

MikeM-2468
Level 4

Any way to narrow it down?   The example errors I see specify a CAB file.  Mine doesn't.  Mine is also showing errors 1-25.

MikeM-2468
Level 4

Verifying the index volume comes back successful.  I'm not able to verify the index location - it's greyed out.

JesusWept3
Level 6
Partner Accredited Certified

OK so the questions would be:
What exact version of EV 10 are you running? (Base? SP1? SP2?)
Have you ever been on EV2007 or earlier or did you start with EV8 or above?

I think we'd need to see some actual example errors from your logs to see what is actually going on.
You can test a full dump through EVSVR to see if that helps retrieve the items or narrow the issue down further, my best guess is that its most likely pointing to an invalid path somewhere

So like if you do
USE EnterpriseVaultDirectory
SELECT * FROM PartitionEntry

The partition paths specified there, are they correct and do they exist?
Or did drive letters get changed or were they restored to a slightly different place etc?

https://www.linkedin.com/in/alex-allen-turl-07370146

ia01
Level 6
Partner Accredited

Rob, URL mentioned in your post says wink

Solution

This issue will be fixed in Enterprise Vault 10.0.3 **not yet released**

http://www.symantec.com/business/support/index?page=content&id=TECH196901

MikeM-2468
Level 4

I think the restore was not done properly.  It appears that the SQL database was newer than the EV storage partitions.  I need to get a restore from the same point in time for both.

Rob_Wilcox1
Level 6
Partner

Yep - that's right.

Working for cloudficient.com

MikeM-2468
Level 4

I finally got a good restore with both SQL and the EV data from the same point in time while everything was in backup mode.  The current journal index is still showing failed, but I can search up to the point of the backup.

The server is 10.0.1.1171.  It was upgraded many times from the old KVS versions through current.  I believe it was on 2007 at one point. 

All the partition paths are present and in the correct place.  The restore was a full snapshot - everything is identical.

Running EVSVR verify on the index comes up with many errors like:
2013-03-22 21:50:42                      Event Output: Unable to complete retrieval request      Reason:           The system cannot find the path specified.  (0x80070003)   Vault:              12 13 2012 to   Vault Id:           10F1CB019EF2ECE48A220E7F980872F251110000archive.domain.com   Item Id:            201303213854289~201303210821230000~Z~50DB6BC227898B02E2138DD7F731FAD1   Reference:          Vault/RI   Extended Item Info:    Supplementary Info:                                (0xc0041ae2)

I remember when I was on 9, I was told by Symantec there was a way to recover the index if it was completely lost.  Is that an option?  Since this discussion was begun I cut a new journal index so that it would be smaller and easier to recover.