Highlighted

error in duplication

Hi!
We've more than 80 virtual servers on VMWare 6.5 platform. Daily backups of VM run fine. We duplicate the data every weekend to a DR site via SLP process. Only 2 windows servers fail to duplicate.
the activity log show:

17:37:19 08/06/2018 - requesting resource  LCM_B2D-DR-stu
17:37:19 08/06/2018 - granted resource  LCM_B2D-DR-stu
17:37:20 08/06/2018 - started process RUNCMD (pid=233188)
17:37:20 08/06/2018 - ended process 0 (pid=233188)
17:37:21 08/06/2018 - begin Duplicate
17:37:22 08/06/2018 - requesting resource  B2D-DR-stu
17:37:22 08/06/2018 - reserving resource @aaaab
17:37:23 08/06/2018 - resource @aaaab reserved
17:37:23 08/06/2018 - granted resource  MediaID=@aaaaf;DiskVolume=PureDiskVolume;DiskPool=B2D-DR;Path=PureDiskVolume;StorageServer=bkp-dr-media12.aro.local;MediaServer=bkp-dr-media12.aro.local
17:37:23 08/06/2018 - granted resource  B2D-DR-stu
17:37:24 08/06/2018 - Info Duplicate (pid=233188) Initiating optimized duplication from @aaaab to @aaaaf
17:37:24 08/06/2018 - requesting resource  @aaaab
17:37:24 08/06/2018 - granted resource  MediaID=@aaaab;DiskVolume=PureDiskVolume;DiskPool=PureDIsk01;Path=PureDiskVolume;StorageServer=bkp-media12r2.aro.local;MediaServer=bkp-dr-media12.aro.local
17:37:25 08/06/2018 - Info bpduplicate (pid=233188) Suspend window close behavior is not supported for optimized duplications
17:37:25 08/06/2018 - Info bpduplicate (pid=233188) window close behavior: Continue processing the current image
17:37:25 08/06/2018 - Info bpdm (pid=38504) started
17:37:25 08/06/2018 - started process bpdm (pid=38504)
17:37:29 08/06/2018 - Info bpdm (pid=38504) requesting nbjm for media
17:37:30 08/06/2018 - Critical bpdm (pid=38504) image open failed: error 2060001: one or more invalid arguments
17:37:32 08/06/2018 - Error bpduplicate (pid=233188) host bkp-media12r2.aro.local backup id AV_ePO_DB_1525994014 optimized duplication failed, media open error (83).
17:37:32 08/06/2018 - Error bpduplicate (pid=233188) Duplicate of backupid AV_ePO_DB_1525994014 failed, media open error (83).
17:37:32 08/06/2018 - Error bpduplicate (pid=233188) Status = no images were successfully processed.
17:37:32 08/06/2018 - end Duplicate; elapsed time 0:00:11
no images were successfully processed  (191)

bpdm.log show:
17:36:34.821 [24692.57960] <2> send_begin_repl_marker: AV_ePO_DB_1525994014_C2_OPT_DUP_BEGIN image created
17:36:34.823 [24692.57960] <2> bp_sts_open_image: STH set to STS_SA_IMAGE
17:36:34.835 [24692.57960] <16> 162589:bptm:24692:bkp-media12r2.aro.local: [ERROR] PDSTS: impl_image_handle: PdvfsOpen(/bkp-media12r2.aro.local#1/2/AV_ePO_DB/VMware-ITVC-Policy/AV_ePO_DB_1525994014_C1_F1.img) failed (9 Bad file descriptor) for image
17:36:34.835 [24692.57960] <16> 162589:bptm:24692:bkp-media12r2.aro.local: [ERROR] PDSTS: impl_open_image: impl_image_handle() failed (2060001:one or more invalid arguments)
17:36:34.835 [24692.57960] <16> 162589:bptm:24692:bkp-media12r2.aro.local: [ERROR] PDSTS: pi_open_image_v10: impl_open_image() failed (2060001:one or more invalid arguments)
17:36:34.835 [24692.57960] <32> bp_sts_open_image: sts_open_image failed: error 2060001

Of course, NBU tries to duplicate the images all the weekend unsuccesfully.
I can't explain why this failure occurs only on those servers.
I'll appreciate your help.
Thanks,
Meir
Tags (2)
6 Replies
Highlighted

Re: error in duplication

If you are on 7.6, this tech note may apply

https://www.veritas.com/support/en_US/article.100014409

If MSDP is running on Linux, check the OS has the right kernel setting applied

https://www.veritas.com/support/en_US/article.TECH167095

Highlighted

Re: error in duplication

Hi Nicolai!

my version is 8.1 .

Thanks

Tags (2)
Highlighted

Re: error in duplication

I can't explain why this failure occurs only on those servers.

Interesting. It's the same repositories for all VM? Do all VM use the same SLP?

Your error sounds as if the SLP is trying to duplicate images that have already been expired.
Can you try to verify the first copies of the backups? Or try to restore them?

Highlighted

Re: error in duplication

Hi

I am facinig similar issue with replication (well import at target domain) with 2 backups out of 5. 3 are being imported these remaining two do fail. I am replicating from 8.1 to 8.0. I have a case opened with VRTS and we did following:

  • bpverify at source domain - no issus
  • fileget at source
  • dcscan
  • catdbutil

back line CE is about to send further steps.

Highlighted

Re: error in duplication

VRTS is opening a Etrack for the same as looks like there is some bigger issue...

I was asked also to run msdpcheck.pl --fullcheck (I was provided with this perl script) - but it will last 1h per each 1TB and during that period storage server will be in down state... So since this will take long time for now I can not run it now... 

Highlighted

Re: error in duplication

Do we got any resolution on it ?