Very slow AIR job between two media server using AIR
OS : Red Hat Enterprise Linux Server release 6.6 (Santiago)
Netbackup Version :
HARDWARE LINUX_RH_X86
VERSION NetBackup 7.7.3
RELEASEDATE Thu May 19 16:24:17 CDT 2016
BUILDNUMBER 0044
Issue: Replication of backup image whose size in hundred of gigs are running slow while replication for backup image whose size is less than 30 GB are running fine
Job Details:
-- Deatiled job status for replication job running for 90 hours
12/22/2016 16:04:01 - Info nbreplicate (pid=5881) Suspend window close behavior is not supported for nbreplicate
12/22/2016 16:04:01 - Info nbreplicate (pid=5881) window close behavior: Continue processing the current image
12/22/2016 16:04:01 - requesting resource LCM_*Remote*Master*
12/22/2016 16:04:01 - granted resource LCM_*Remote*Master*
12/22/2016 16:04:01 - started process RUNCMD (pid=5881)
12/22/2016 16:04:02 - Info bpdm (pid=5886) started
12/22/2016 16:04:02 - started process bpdm (pid=5886)
12/22/2016 16:04:02 - requesting resource @aaaab
12/22/2016 16:04:02 - reserving resource @aaaab
12/22/2016 16:04:02 - resource @aaaab reserved
12/22/2016 16:04:02 - granted resource MediaID=@aaaab;DiskVolume=PureDiskVolume;DiskPool=source-pool;Path=PureDiskVolume;StorageServer=source.domain.com;MediaServer=source_master.domain.com
12/22/2016 16:04:03 - Info source_master.domain.com (pid=5886) Using OpenStorage to replicate backup id client2356.gen_1482371231, media id @aaaab, storage server source.domain.com, disk volume PureDiskVolume
12/22/2016 16:04:03 - Info source_master.domain.com (pid=5886) Replicating images to target storage server destination.domain.com, disk volume PureDiskVolume
12/26/2016 10:24:12 - Info source_master.domain.com (pid=5886) StorageServer=PureDisk:source.domain.com; Report=PDDO Stats for (source.domain.com): scanned: 281088088 KB, CR sent: 32333351 KB, CR sent over FC: 0 KB, dedup: 88.5%, cache disabled
12/26/2016 10:24:13 - Info bpdm (pid=5886) EXITING with status 0
12/26/2016 10:24:13 - Replicated backup id client2356.gen_1482371231 successfully
the requested operation was successfully completed (0)
-- Checked another detailed job status for replication job which finished quickly compare to above one:
12/25/2016 02:29:28 - requesting resource LCM_*Remote*Master*
12/25/2016 02:29:29 - granted resource LCM_*Remote*Master*
12/25/2016 02:29:29 - Info nbreplicate (pid=57805) Suspend window close behavior is not supported for nbreplicate
12/25/2016 02:29:29 - Info nbreplicate (pid=57805) window close behavior: Continue processing the current image
12/25/2016 02:29:29 - Info bpdm (pid=57814) started
12/25/2016 02:29:29 - started process bpdm (pid=57814)
12/25/2016 02:29:29 - started process RUNCMD (pid=57805)
12/25/2016 02:29:29 - requesting resource @aaaab
12/25/2016 02:29:29 - reserving resource @aaaab
12/25/2016 02:29:29 - resource @aaaab reserved
12/25/2016 02:29:29 - granted resource MediaID=@aaaab;DiskVolume=PureDiskVolume;DiskPool=source-pool;Path=PureDiskVolume;StorageServer=source.domain.com;MediaServer=source_master.domain.com
12/25/2016 02:29:30 - Info source_master.domain.com (pid=57814) Using OpenStorage to replicate backup id client2920.gen_1482624099, media id @aaaab, storage server source.domain.com, disk volume PureDiskVolume
12/25/2016 02:29:30 - Info source_master.domain.com (pid=57814) Replicating images to target storage server destination.domain.com, disk volume PureDiskVolume
12/25/2016 02:34:30 - Info source_master.domain.com (pid=57814) StorageServer=PureDisk:source.domain.com; Report=PDDO Stats for (source.domain.com): scanned: 25272765 KB, CR sent: 27906 KB, CR sent over FC: 0 KB, dedup: 99.9%, cache disabled
12/25/2016 02:34:30 - Info bpdm (pid=57814) EXITING with status 0
12/25/2016 02:34:31 - Replicated backup id client2920.gen_1482624099 successfully
12/25/2016 02:34:31 - Info bpdm (pid=1329) started
12/25/2016 02:34:31 - started process bpdm (pid=1329)
12/25/2016 02:34:31 - requesting resource @aaaab
12/25/2016 02:34:31 - Info source_master.domain.com (pid=1329) Using OpenStorage to replicate backup id client1517.gen_1482624156, media id @aaaab, storage server source.domain.com, disk volume PureDiskVolume
12/25/2016 02:34:31 - Info source_master.domain.com (pid=1329) Replicating images to target storage server destination.domain.com, disk volume PureDiskVolume
12/25/2016 02:34:31 - granted resource MediaID=@aaaab;DiskVolume=PureDiskVolume;DiskPool=source-pool;Path=PureDiskVolume;StorageServer=source.domain.com;MediaServer=source_master.domain.com
12/25/2016 02:45:02 - Info source_master.domain.com (pid=1329) StorageServer=PureDisk:source.domain.com; Report=PDDO Stats for (source.domain.com): scanned: 24590077 KB, CR sent: 60134 KB, CR sent over FC: 0 KB, dedup: 99.8%, cache disabled
12/25/2016 02:45:02 - Info bpdm (pid=1329) EXITING with status 0
12/25/2016 02:45:02 - Replicated backup id client1517.gen_1482624156 successfully
12/25/2016 02:45:02 - Info bpdm (pid=20370) started
12/25/2016 02:45:02 - started process bpdm (pid=20370)
12/25/2016 02:45:02 - requesting resource @aaaab
12/25/2016 02:45:02 - granted resource MediaID=@aaaab;DiskVolume=PureDiskVolume;DiskPool=source-pool;Path=PureDiskVolume;StorageServer=source.domain.com;MediaServer=source_master.domain.com
12/25/2016 02:45:03 - Info source_master.domain.com (pid=20370) Using OpenStorage to replicate backup id client1008.gen_1482624408, media id @aaaab, storage server source.domain.com, disk volume PureDiskVolume
12/25/2016 02:45:03 - Info source_master.domain.com (pid=20370) Replicating images to target storage server destination.domain.com, disk volume PureDiskVolume
12/25/2016 04:16:34 - Info source_master.domain.com (pid=20370) StorageServer=PureDisk:source.domain.com; Report=PDDO Stats for (source.domain.com): scanned: 16192032 KB, CR sent: 545449 KB, CR sent over FC: 0 KB, dedup: 96.6%, cache disabled
12/25/2016 04:16:34 - Info bpdm (pid=20370) EXITING with status 0
12/25/2016 04:16:34 - Replicated backup id client1008.gen_1482624408 successfully
12/25/2016 04:16:34 - Info bpdm (pid=54749) started
12/25/2016 04:16:34 - started process bpdm (pid=54749)
12/25/2016 04:16:34 - requesting resource @aaaab
12/25/2016 04:16:34 - granted resource MediaID=@aaaab;DiskVolume=PureDiskVolume;DiskPool=source-pool;Path=PureDiskVolume;StorageServer=source.domain.com;MediaServer=source_master.domain.com
12/25/2016 04:16:35 - Info source_master.domain.com (pid=54749) Using OpenStorage to replicate backup id client3026.gen_1482626438, media id @aaaab, storage server source.domain.com, disk volume PureDiskVolume
12/25/2016 04:16:35 - Info source_master.domain.com (pid=54749) Replicating images to target storage server destination.domain.com, disk volume PureDiskVolume
12/25/2016 04:21:05 - Info source_master.domain.com (pid=54749) StorageServer=PureDisk:source.domain.com; Report=PDDO Stats for (source.domain.com): scanned: 22523628 KB, CR sent: 22310 KB, CR sent over FC: 0 KB, dedup: 99.9%, cache disabled
12/25/2016 04:21:05 - Info bpdm (pid=54749) EXITING with status 0
12/25/2016 04:21:05 - Replicated backup id client3026.gen_1482626438 successfully
12/25/2016 04:21:05 - Info bpdm (pid=63464) started
12/25/2016 04:21:05 - started process bpdm (pid=63464)
12/25/2016 04:21:06 - Info source_master.domain.com (pid=63464) Using OpenStorage to replicate backup id client2563.gen_1482629188, media id @aaaab, storage server source.domain.com, disk volume PureDiskVolume
12/25/2016 04:21:06 - Info source_master.domain.com (pid=63464) Replicating images to target storage server destination.domain.com, disk volume PureDiskVolume
12/25/2016 04:21:06 - requesting resource @aaaab
12/25/2016 04:21:06 - granted resource MediaID=@aaaab;DiskVolume=PureDiskVolume;DiskPool=source-pool;Path=PureDiskVolume;StorageServer=source.domain.com;MediaServer=source_master.domain.com
12/25/2016 04:21:37 - Info source_master.domain.com (pid=63464) StorageServer=PureDisk:source.domain.com; Report=PDDO Stats for (source.domain.com): scanned: 38 KB, CR sent: 10 KB, CR sent over FC: 0 KB, dedup: 73.7%, cache disabled
12/25/2016 04:21:37 - Info bpdm (pid=63464) EXITING with status 0
12/25/2016 04:21:37 - Replicated backup id client2563.gen_1482629188 successfully
the requested operation was successfully completed (0)