cancel
Showing results for 
Search instead for 
Did you mean: 

Corrupted Data. Had to do a restore.

rbollinger1212
Level 4

Hey Guys, 

(EV 11 CHF 5) 

One of our CLOSED Evault partitions became corrupted and we had to do a restore from CommVault. The backup of this partition was from about 10 days ago. We also restore an incremental as well after the full backup was restored. 

What steps do we need to take now that we have complete the restore? The EV services have all started and EV is working as expected. 

Thanks, 

Robert 

12 REPLIES 12

GertjanA
Level 6
Partner    VIP    Accredited Certified

Hello Robert,

You might be able to use EVSVR, use Verify to verify data is where SQL thinks it is. I am not sure exactly what to use, the utilities guide should have guidance on that. You can select the specific partition to be verified. I believe the verify can be run regardless of the backup state of the partition, but you'll get a notification in the EVSVR logfile if needed.

Regards. Gertjan

Thanks for the response. I have a ticket open with Veritas support but they abosltely hate people who date to still run EVault 11. I hope they call us back and help us out here. 

I found some documentation on EVSR but its really confusing and i just need to know which commands run to verify that the data restored is correct. 

Thanks, 

Robert 

Robert,

You'll want to run the EVSVR like so:

1. Open your EV installation directory and double-click evsvr.exe.

2. At the EVSVR prompt, type "edit" and hit Enter.

3. Configure the following options:

     Operation: Verify
     Option: ArchiveObjects
     Level: SavesetValid

     Uncheck the "Process All Vault Store Groups," "Process All Vault Stores," and "Process All Partitions" checkboxes at the top and select the affected partition from the Partition: dropdown.

     Everything else you can leave default or set to taste.

4. Click the Save button and save the settings to an XML file.

5. Click OK and the EVSVR prompt should load the settings you selected.

6. Type "start" and hit Enter.

7. Wait a while. This step will take some time, as every saveset in the partition will be retrieved to verify that it is intact. There is no progress indicator, but you can safely go do other things while it runs. Just don't close the EVSVR window or reboot the box.

8. When it is complete, a message in the EVSVR prompt will tell you it is finished.

9. Review the report file and see if any items could not be retrieved. Depending on what the report says, there may be additional steps necessary, but those should get you started. Your Veritas Technical Support Engineer can help you interpret the report, as we do not, in fact, hate you. Smiley Happy

 

--Chris

Thanks for the response. I was able to get veritas to help out even though we are running version 11. the process you outlined is running now. It has been for sometime. here is the tail end of the log file: 

2017-10-31 14:17:47 Finding CAB Collection records with Collection Identity mismatches and RefCount errors
2017-10-31 14:17:47 --------------------------------------------------------------------------------------
2017-10-31 14:17:47
2017-10-31 14:17:48 CAB Collection records inspected: 1370
2017-10-31 14:17:48 CAB Collection records with a Collection Identity mismatch: 0
2017-10-31 14:17:48 CAB Collection records with RefCount > TotalCount: 0
2017-10-31 14:17:48
2017-10-31 14:17:48
2017-10-31 14:17:48 Finding Savesets with cross partition Collection Identities
2017-10-31 14:17:48 -----------------------------------------------------------
2017-10-31 14:17:48
2017-10-31 14:18:23 Savesets in this partition that incorrectly reference another partition's Collection records: 0
2017-10-31 14:18:23 Savesets in other partitions that incorrectly reference this partition's Collection records: 0
2017-10-31 14:18:23
2017-10-31 14:18:23
2017-10-31 14:18:23 Step 1 - Verify that Saveset SISPart entries exist in the Fingerprint Database
2017-10-31 14:18:23 ------------------------------------------------------------------------------
2017-10-31 14:18:23
2017-10-31 14:18:23
2017-10-31 14:18:23 Saveset records: 6432963

It has not updated the report file since 2:17 PM PST. However it still shows this: 

object state: RUNNING

application state: ACTIVE

When checking the status of the evsvr.exe window. i am hoping it will be done tomorrow. 

Robert 

 

It is normal for it to run for some time, as it is checking every item on the partition. The process only writes failures to the log file, so it is normal to see long periods of inactivity in the log, and then sporadic writing periods when it comes across items that are invalid. As long as you see "Running" as the state, it's still checking items.

If you want to watch all of its activity, even the successful checks, you can dtrace the "evsvr" process.

 

--Chris

Thanks for the response. The process is still running. Though in EVs defense it looks like EV was in backup mode the entire night. Hopefully now that its out of backup mode things will go faster. 

Robert 

No, Backup Mode won't interfere with this process. The operation you ran is a Verify, which is a read-only operation that hums happily along whether in Backup Mode or not. If you were to run a Repair operation, which makes modifications, then you'd need to avoid Backup Mode for the duration of that.

 

--Chris

Ok thanks for the response. Should it be taking this long? Were talkming about 18 hours +. The total size of hte disk is 1 TB but each of the partitions (on the same disk) is 300GB. 

Robert 

GertjanA
Level 6
Partner    VIP    Accredited Certified

I've seen days... You seem to have 6432963 items, so it might take a while. EV needs to unpack the cabs for evsvr to be able to verify properly. As chris says, as long as the status is running, it is fine. As it is a closed partition, just leave it running.

Regards. Gertjan

OK then i will just be patient. 

 

Robert 

Just for reference, we usually see rates of about 50,000-100,000 items/hour for this sort of operation, with some pretty high variability dependent on the storage technology and the database health. Since your log states you are checking 6.4 million items, you're probably looking at anywhere from 2.5 to 6 days.

One technique I often use to estimate the likely duration of these operations is to start the operation, let it run for five or ten minutes, then type "stop" which stops the operation and writes the wrapup statistics to the report. This includes the average processing rate. So long as all the partitions in the job are similarly situated (e.g., not on different storage types or in different geographic locations), then it's usually reasonable to extrapolate that rate across your total item set and you can make a decent guess at how long the whole thing will take.

It's too late to be worth doing this now for your job, but I thought I would post it for your future reference and in case it can help others.

 

--Chris

Thats pretty smart Chris. That frickin process is still running. we're on day 3 now, but based on what you said, it may take through the weekend to complete. 

Thanks for you help. 

 

Robert