Strange issue. Duplicaiton of a single backup image is failing to MSDP. Consistent failure mode. Strange thing is - it is a *single* VMware backup image... no issues with any other VMware images or any other types of images.
Environment:
- Solaris 10 master v. 7.6.1
- Symantec 5220 Appliance media servers v. 2.6.1
- Backup to Advanced Disk -> duplication to MSDP and tape
Job details:
04/06/2015 11:30:06 - started process bpdm (pid=13428)
04/06/2015 11:30:06 - Info bptm (pid=32082) backup child process is pid 32333
04/06/2015 11:30:06 - Info bpdm (pid=13428) reading backup image
04/06/2015 11:30:06 - Info bpdm (pid=13428) using 512 data buffers
04/06/2015 11:30:06 - Info bpdm (pid=13428) spawning a child process
04/06/2015 11:30:06 - Info bpbrm (pid=13428) child pid: 13429
04/06/2015 11:30:06 - Info bpdm (pid=13428) requesting nbjm for media
04/06/2015 11:30:07 - begin reading
04/06/2015 11:30:08 - Critical bptm (pid=32082) image write failed: error 2060022: software error
04/06/2015 11:30:13 - Error bptm (pid=32082) cannot write image to disk, Invalid argument
04/06/2015 11:30:13 - Error bpdm (pid=13429) cannot write data to socket, Connection reset by peer
04/06/2015 11:30:13 - Info bptm (pid=32082) EXITING with status 84 <----------
04/06/2015 11:30:13 - Info densyma02p (pid=32082) StorageServer=PureDisk:densyma02p; Report=PDDO Stats (multi-threaded stream used) for (densyma02p): scanned: 53251 KB, CR sent: 23 KB, CR sent over FC: 0 KB, dedup: 100.0%, cache disabled
04/06/2015 11:30:15 - Error bpdm (pid=13428) media manager terminated by parent process
04/06/2015 11:30:15 - Error bpduplicate (pid=5447) host densyma01p backup id dennagxip2_1428276467 read failed, media manager killed by signal (82).
04/06/2015 11:30:15 - Error bpduplicate (pid=5447) host densyma02p backupid dennagxip2_1428276467 write failed, media write error (84).
04/06/2015 11:30:15 - Error bpduplicate (pid=5447) Duplicate of backupid dennagxip2_1428276467 failed, media write error (84).
04/06/2015 11:30:15 - Error bpduplicate (pid=5447) Status = no images were successfully processed.
04/06/2015 11:30:15 - end Duplicate; elapsed time 0:00:22
no images were successfully processed (191)
bptm log from target:
11:30:08.038 [32082] <16> 8756459:bptm:32082:densyma02p: [ERROR] PDSTS: fbu_sort_extents: no imag
e extents
11:30:08.068 [32082] <2> send_media_kbytes_written_establish_threshold: CINDEX 0, RB Kbytes for m
onitoring = 3000000
11:30:08.928 [32082] <16> 8756459:bptm:32082:densyma02p: [ERROR] PDSTS: fbu_fill_bitmap: (3437) s
tream_offset(999507456) + length(2560) > (3438) stream_offset(999507456)
11:30:08.928 [32082] <16> 8756459:bptm:32082:densyma02p: [ERROR] PDSTS: fbu_scan_buf: fbu_fill_bi
tmap() failed (2060022:software error)
11:30:08.928 [32082] <16> 8756459:bptm:32082:densyma02p: [ERROR] PDSTS: write_special_buf: fbu_sc
an_buf failed: (2060022:software error)
11:30:08.928 [32082] <16> 8756459:bptm:32082:densyma02p: [ERROR] PDSTS: write_unknown_len_buf: wr
ite_special_buf() failed, byteswritten:<0>, status:<0x1> (2060022:software error)
11:30:08.928 [32082] <16> 8756459:bptm:32082:densyma02p: [ERROR] PDSTS: impl_write_image: write_u
nknown_len_buf() failed (2060022:software error)
11:30:08.928 [32082] <16> 8756459:bptm:32082:densyma02p: [ERROR] PDSTS: impl_write_image: STAT co
ntents:
11:30:08.928 [32082] <16> 8756459:bptm:32082:densyma02p: [ERROR] PDSTS: impl_write_image: exit (0
0x1 2060022:software error)
11:30:08.928 [32082] <16> 8756459:bptm:32082:densyma02p: [ERROR] PDSTS: pi_write_image_v7: impl_w
rite_image() failed (2060022:software error)
11:30:08.928 [32082] <32> do_write_image: sts_write_image failed: byteswritten = 0, len = 201216
11:30:08.928 [32082] <32> do_write_image: sts_write_image failed: error 2060022 (54525952 201216
0)
11:30:08.929 [32082] <2> ConnectionCache::connectAndCache: Acquiring new connection for host oxfo
rdc, query type 1
11:30:08.932 [32082] <2> vnet_pbxConnect: pbxConnectEx Succeeded
11:30:08.932 [32082] <2> logconnections: BPDBM CONNECT FROM 169.143.74.89.44039 TO 169.143.74.127
.1556 fd = 0
11:30:08.944 [32082] <2> db_end: Need to collect reply
11:30:08.955 [32082] <32> write_data: image write failed: error 2060022:
11:30:09.500 [32082] <2> delete_image_disk_sts_impl: Deleting disk header for dennagxip2_14282764
67_C2_HDR_R18
11:30:09.505 [32082] <2> ConnectionCache::connectAndCache: Acquiring new connection for host oxfo
rdc, query type 161
11:30:09.509 [32082] <2> vnet_pbxConnect: pbxConnectEx Succeeded
11:30:09.509 [32082] <2> logconnections: BPDBM CONNECT FROM 169.143.74.89.57919 TO 169.143.74.127
.1556 fd = 30
11:30:09.520 [32082] <2> db_end: Need to collect reply
11:30:13.252 [32082] <2> ConnectionCache::connectAndCache: Acquiring new connection for host oxfo
rdc, query type 1
11:30:13.255 [32082] <2> vnet_pbxConnect: pbxConnectEx Succeeded
11:30:13.255 [32082] <2> logconnections: BPDBM CONNECT FROM 169.143.74.89.59894 TO 169.143.74.127
.1556 fd = 30
11:30:13.266 [32082] <2> db_end: Need to collect reply
11:30:13.278 [32082] <16> write_data: cannot write image to disk, Invalid argument
11:30:13.278 [32082] <4> write_backup: Calling close_all_ft_pipes
11:30:13.278 [32082] <2> KILL_MM_CHILD: Sending SIGUSR2 (kill) to child 32333 (bptm.c:19651)
11:30:13.313 [32082] <2> bptm: Calling tpunmount for media @aaaal
11:30:13.313 [32082] <2> send_MDS_msg: MEDIA_DONE 0 8756459 0 @aaaal 0 180 {89BFC592-DC82-11E4-8B
AF-001B21890135}
11:30:13.313 [32082] <2> packageBptmResourceDoneMsg: msg (MEDIA_DONE 0 8756459 0 @aaaal 0 180 {89
BFC592-DC82-11E4-8BAF-001B21890135})
11:30:13.313 [32082] <2> packageBptmResourceDoneMsg: keyword MEDIA_DONE version 0 jobid 8756459 c
opyNum 0 mediaId @aaaal mediaKey 0 unloadDelay 180 allocId {89BFC592-DC82-11E4-8BAF-001B21890135}
11:30:13.313 [32082] <2> packageBptmResourceDoneMsg: returns 0
11:30:13.315 [32082] <2> JobInst::sendIrmMsg: returning
11:30:13.645 [32082] <2> main: Got bpduplicate acknowledgement status: 0, err: 0
11:30:13.645 [32082] <2> bptm: EXITING with status 84 <----------
