Agent Replication Failed

[Fr_d_ric_RIGAL]

Hi,

We have set replication between 2 PureDisk SPA (6.6.1.2) for several agent and all is working fine except for one agent. We have the same error each time for him and backups are fine :

 *** Start: Replication Prepare ***
The is Classic Replication or PDDO Indirect Replication, link to replicant dsid on target will be made.
The remote dataselection mirror for source dataselection 24 is: 15
 *** Stop: Replication Prepare ***
Agent Jobstep analysis: exitcode 0, status 2, progress 100.

 *** Supportability Summary ***
jobid                 = 2781
jobstepid             = 12679
agentid               = 1000000
hostname              = padeobsbkpa03.corp.leroymerlin.com
starttimejobstep      = October 23, 2011, 10:00 am
endtimejobstep        = October 23, 2011, 10:00 am
workflowstepname      = Prepare Replication
status                = SUCCESS

r is Version 6.6.1.47576, Protocol Version 6.6.1
[2011-Oct-23 10:24:51 CEST][2781][stream3] Error: 2 : CRReplicateReceiveDO: DO download failed do fingerprint 369140a6569410bf36e28462d9682f94
[2011-Oct-23 10:24:51 CEST][2781][stream3] Error: 2 : CRReplicate: Could not receive DO 369140a6569410bf36e28462d9682f94 to replicate: no such object
[2011-Oct-23 10:24:51 CEST][2781][stream3]
[2011-Oct-23 10:24:51 CEST][2781][stream3] Fatal error: zif_cr_replicate: could not process the replication batch: no such object in /opt/pdmbe/mgmtclass/ReplicationStream.php on line 212
[2011-Oct-23 10:24:51 CEST][2781] Stream 3 completed with exit value 255
[2011-Oct-23 10:24:52 CEST][2781] Replication will retry sending data for attempt number: 8 after sleeping 60 second(s).
[2011-Oct-23 10:25:52 CEST][2781] Starting multi-stream replication.
[2011-Oct-23 10:25:59 CEST][2781] Starting multi-stream replication with 4 stream(s)
[2011-Oct-23 10:25:59 CEST][2781] Successfully started stream 0
[2011-Oct-23 10:25:59 CEST][2781] Successfully started stream 1
[2011-Oct-23 10:25:59 CEST][2781] Successfully started stream 2
[2011-Oct-23 10:25:59 CEST][2781] Successfully started stream 3
[2011-Oct-23 10:26:15 CEST][2781][stream0] Compression flag retrieved from agent.cfg: 1
[2011-Oct-23 10:26:15 CEST][2781][stream0] Forwarding data (NUMBER OF FINGERPRINTS in this batch:20559)
[2011-Oct-23 10:26:15 CEST][2781][stream0] Info: Server is Version 6.6.1.46349, Protocol Version 6.6.1
[2011-Oct-23 10:26:15 CEST][2781][stream0] Info: Server is Version 6.6.1.47576, Protocol Version 6.6.1
[2011-Oct-23 10:26:15 CEST][2781][stream0] Info: Server is Version 6.6.1.47576, Protocol Version 6.6.1
[2011-Oct-23 10:26:15 CEST][2781][stream0] Error: 2 : CRReplicateReceiveDO: DO download failed do fingerprint b87643875c6c9147f95364fb70a645ea
[2011-Oct-23 10:26:15 CEST][2781][stream0] Error: 2 : CRReplicate: Could not receive DO b87643875c6c9147f95364fb70a645ea to replicate: no such object
[2011-Oct-23 10:26:15 CEST][2781][stream0]
[2011-Oct-23 10:26:15 CEST][2781][stream0] Fatal error: zif_cr_replicate: could not process the replication batch: no such object in /opt/pdmbe/mgmtclass/ReplicationStream.php on line 212
[2011-Oct-23 10:26:15 CEST][2781] Stream 0 completed with exit value 255
[2011-Oct-23 10:26:15 CEST][2781] Replication will retry sending data for attempt number: 9 after sleeping 60 second(s).
[2011-Oct-23 10:27:15 CEST][2781] Starting multi-stream replication.
[2011-Oct-23 10:27:21 CEST][2781] Starting multi-stream replication with 4 stream(s)
[2011-Oct-23 10:27:21 CEST][2781] Successfully started stream 0
[2011-Oct-23 10:27:21 CEST][2781] Successfully started stream 1
[2011-Oct-23 10:27:21 CEST][2781] Successfully started stream 2
[2011-Oct-23 10:27:21 CEST][2781] Successfully started stream 3
[2011-Oct-23 10:27:32 CEST][2781][stream3] Compression flag retrieved from agent.cfg: 1
[2011-Oct-23 10:27:32 CEST][2781][stream3] Forwarding data (NUMBER OF FINGERPRINTS in this batch:20597)
[2011-Oct-23 10:27:32 CEST][2781][stream3] Info: Server is Version 6.6.1.46349, Protocol Version 6.6.1
[2011-Oct-23 10:27:32 CEST][2781][stream3] Info: Server is Version 6.6.1.47576, Protocol Version 6.6.1
[2011-Oct-23 10:27:32 CEST][2781][stream3] Info: Server is Version 6.6.1.47576, Protocol Version 6.6.1
[2011-Oct-23 10:27:32 CEST][2781][stream3] Error: 2 : CRReplicateReceiveDO: DO download failed do fingerprint 369140a6569410bf36e28462d9682f94
[2011-Oct-23 10:27:32 CEST][2781][stream3] Error: 2 : CRReplicate: Could not receive DO 369140a6569410bf36e28462d9682f94 to replicate: no such object
[2011-Oct-23 10:27:32 CEST][2781][stream3]
[2011-Oct-23 10:27:32 CEST][2781][stream3] Fatal error: zif_cr_replicate: could not process the replication batch: no such object in /opt/pdmbe/mgmtclass/ReplicationStream.php on line 212
[2011-Oct-23 10:27:32 CEST][2781] Stream 3 completed with exit value 255
[2011-Oct-23 10:27:32 CEST][2781] Replication will retry sending data for attempt number: 10 after sleeping 60 second(s).
[2011-Oct-23 10:28:32 CEST][2781] Replication has tried 10 time(s) to replicate data, but was not successful.
[2011-Oct-23 10:28:32 CEST][2781] Checking the execution status of each remote MBImport Job.

[2011-Oct-23 10:28:32 CEST][2781] The batchnumber could not be increased, Failing the replication Job.
[2011-Oct-23 10:28:32 CEST][2781] Save point failure. Skipping batchnr update
[2011-Oct-23 10:28:32 CEST]Stop forwarding actual content.
[2011-Oct-23 10:28:32 CEST]Start finalizing Replication.
[2011-Oct-23 10:28:32 CEST]Cleaning up temporary routing tables.
[2011-Oct-23 10:28:32 CEST]Stop finalizing Replication.
[2011-Oct-23 10:28:32 CEST]Stopping Replication.
Agent Jobstep analysis: exitcode 1, status 3, progress 0.

 *** Supportability Summary ***
jobid                 = 2781
jobstepid             = 12691
agentid               = 1000000
hostname              = padeobsbkpa03.corp.leroymerlin.com
starttimejobstep      = October 23, 2011, 10:16 am
endtimejobstep        = October 23, 2011, 10:28 am
workflowstepname      = Forward Data
status                = ERROR

Execute WFAction: Mark Error

 *** Supportability Summary ***
jobid                 = 2781
jobstepid             = 12717
agentid               = 1000000
hostname              = padeobsbkpa03.corp.leroymerlin.com
starttimejobstep      = October 23, 2011, 10:28 am
endtimejobstep        = October 23, 2011, 10:28 am
workflowstepname      = Error
status                = SUCCESS

Execute WFAction: Exit
 Job exited with 1 errors, 0 warnings, 3 successes

 *** Supportability Summary ***
jobid                 = 2781
jobstepid             = 12718
agentid               = 1000000
hostname              = padeobsbkpa03.corp.leroymerlin.com
starttimejobstep      = October 23, 2011, 10:28 am
endtimejobstep        = October 23, 2011, 10:28 am
workflowstepname      = Exit
status                = SUCCESS

If someone has an idea, it will be fine.

Thank you all

[S_Williamson]

Looks like your server is running into an issue where two DO's should exist but doesnt (Probably got deleted by another removal job where it wasnt meant to. If you have the RecoverCR script you will need to run this. If not you need to start a support call with Symantec.

[2011-Oct-23 10:26:15 CEST][2781][stream0] Error: 2 : CRReplicateReceiveDO: DO download failed do fingerprint b87643875c6c9147f95364fb70a645ea
[2011-Oct-23 10:26:15 CEST][2781][stream0] Error: 2 : CRReplicate: Could not receive DO b87643875c6c9147f95364fb70a645ea to replicate: no such object

[2011-Oct-23 10:27:32 CEST][2781][stream3] Error: 2 : CRReplicateReceiveDO: DO download failed do fingerprint 369140a6569410bf36e28462d9682f94
[2011-Oct-23 10:27:32 CEST][2781][stream3] Error: 2 : CRReplicate: Could not receive DO 369140a6569410bf36e28462d9682f94 to replicate: no such object

You can confirm that they existed and were deleted by grepping through the /Storage/history/dataobjects folder for the I'ds

Also try run these commands which will no doubt confirm they are missing

/opt/pdcr/bin/dbutil -d b87643875c6c9147f95364fb70a645ea

/opt/pdcr/bin/dbutil b87643875c6c9147f95364fb70a645ea

/opt/pdcr/bin/dbutil -d 369140a6569410bf36e28462d9682f94

/opt/pdcr/bin/dbutil 369140a6569410bf36e28462d9682f94

Simon

[Fr_d_ric_RIGAL]

Thank you Simon,

a case is already open in parallel of my post

[f25]

Hi,

Such issues usually end-up with a running a CR-recovery script and a manual backup of the data selection that is related with this missing agent. recoverCR is meant to be run only with Symantec Support's assistance.

Never the less some data has been lost because the file did not manage to replicate before it was deleted from the agent-source.

Good luck with this one.