SP Consistency Check Error

[Speeder]

I have a 5020 appliance running 1.3.0.2 (planning to upgrate to 1.4.1.1 at some point), but there are 2 SP consistency jobs that keep constantly failing (after running for a while), week after week, on 2 specific datasets (across 2 different servers). Other datasets on those servers pass SP consistency checks without errors. The *Error message* generates a critical alert. I was able to restore from the datasets before (not in its entirety, but a couple random files). It seems that both jobs fail on some DOs on "Verifying the security data", and that the content is not affected.

Here's an example of the log file

Verifying level is 'Exists'.
Verifying the content.
.Move the failed DOs from the check list to the black list.
.POs processed: 199902, POs moved: 0
.Move the failed DOs from the black list to the error list.
.POs processed: 0, POs moved: 0
.If we found problems report them to the spa.
Verifying the meta data.
.Move the failed DOs from the check list to the black list.
.POs processed: 4177, POs moved: 0
.Move the failed DOs from the black list to the error list.
.POs processed: 0, POs moved: 0
.If we found problems report them to the spa.
Verifying the security data.
.Move the failed DOs from the check list to the black list.
Error: 2 : 57cb1abc5d9d31a4782b2a1510685c0b: verification failed. DO does not exist.
Error: 2 : ac5b7ec647cac5abe7427fd0f371200c: verification failed. DO does not exist.
Error: 2 : e45232a916e9c54ec57c197fdec8a0fe: verification failed. DO does not exist.
Error: 2 : fc61c7bfd1778a10c46d135b2d1d7917: verification failed. DO does not exist.
.POs processed: 912, POs moved: 4
.Move the failed DOs from the black list to the error list.
Error: 2 : 57cb1abc5d9d31a4782b2a1510685c0b: verification failed. DO does not exist.
Error: 2 : ac5b7ec647cac5abe7427fd0f371200c: verification failed. DO does not exist.
Error: 2 : e45232a916e9c54ec57c197fdec8a0fe: verification failed. DO does not exist.
Error: 2 : fc61c7bfd1778a10c46d135b2d1d7917: verification failed. DO does not exist.
.POs processed: 4, POs moved: 4
.If we found problems report them to the spa.
*** Error Message ***
mbe.spconsistency.1011
severity: 6
server: 1656000000
source: MBE-CLI_Component
description:
 Errors found in the DO list. Failed DO list: /tmp/XUUw5o.
*** End ***
Agent Jobstep analysis: exitcode 1, status 3, progress 0.

 *** Supportability Summary ***
jobid                 = 4754
jobstepid             = 19133
agentid               = 1656000000
hostname              = *EDITED*
starttimejobstep      = December 26, 2011, 1:41 pm
endtimejobstep        = December 26, 2011, 2:30 pm
workflowstepname      = SPChk
status                = ERROR

Execute WFAction: Mark Error

 *** Supportability Summary ***
jobid                 = 4754
jobstepid             = 19709
agentid               = 1656000000
hostname              = *EDITED*
starttimejobstep      = December 26, 2011, 2:30 pm
endtimejobstep        = December 26, 2011, 2:30 pm
workflowstepname      = markerror
status                = SUCCESS

Execute WFAction: Exit
 Job exited with 1 errors, 0 warnings, 2 successes

 *** Supportability Summary ***
jobid                 = 4754
jobstepid             = 19710
agentid               = 1656000000
hostname              = *EDITED*
starttimejobstep      = December 26, 2011, 2:30 pm
endtimejobstep        = December 26, 2011, 2:30 pm
workflowstepname      = exit
status                = SUCCESS

[S_Williamson]

Hi

You will need to log a case with Symantec to get a script called RecoverCR. This scans your databases and works out what objects are missing and does its best to restore them or tell you which clients will need a new FULL run to restore the missing objects.

This happens whe the data removal jobs seem to clean up some of the DO's that its not meant to. I think if some DO's are shared by multiple server and you delete an object from one server and its not correctly marked as also belonging to another it gets deleted and here you are.

You can trall through the DO log files if you wish but its better to let a support person do the running around :)

Simon

[S_Williamson]

Its definately a good idea to run this check around once a month to get on top of any errors before they get too far out of hand though.