cancel
Showing results forΒ 
Search instead forΒ 
Did you mean:Β 

Need help on configuring Optimized Duplication between 2 Quantum DXi

hariharan24
Level 4
Certified

Need help on configuring Optimized Duplication between 2 Quantum DXi

I am having solaris master server running in 7.7.3 version, we are having 3 remote sites which are having local backups happening to there own local Dxi boxes (3 Dxi boxes) - each of 5 TB capcity - model Dxi4700

we have one main site where we has around 180 TB quantum Dxi box - model DXi6802

Our requirement is to do a optimized duplciation from the remote site DXi box to main site DXi box.

I had established the connectivity and firewall rules were added for the respective ports and DXi replication configuraitons were done to enabled the opt dup.

SLPs are also created with source backup --> remote (source) site DXi STU and Duplication --> targe site Dxi STU

Now when i run backup and monitored the Duplication job it was giving below errors:

errors are bolded below:

Can you please help me on this to fix this issue? Also it would be great if you share me the steps to configure opt dup from one Dxi to another Dxi.

Detailed status from the activity monitor:


04/03/2017 21:32:26 - requesting resource LCM_stu_ds03ss01_remote_dup
04/03/2017 21:33:23 - granted resource  LCM_stu_ds03ss01_remote_dup
04/03/2017 21:33:23 - started process RUNCMD (pid=2321)
04/03/2017 21:33:24 - begin Duplicate
04/03/2017 21:33:24 - ended process 0 (pid=2321)
04/03/2017 21:33:24 - requesting resource stu_ds03ss01_remote_dup
04/03/2017 21:33:24 - reserving resource @aaaa_
04/03/2017 21:34:25 - resource @aaaa_ reserved
04/03/2017 21:34:25 - granted resource  MediaID=@aaaaK;DiskVolume=_PhysicalLSU;DiskPool=dp_xrxsenbuds03ss01;Path=_PhysicalLSU;StorageServer=xrxsenbuds03ss01_10.173.133.25;MediaServer=xrxsenbumed04
04/03/2017 21:34:25 - granted resource  stu_ds03ss01_remote_dup
04/03/2017 21:34:25 - requesting resource @aaaa_
04/03/2017 21:34:26 - Info Duplicate (pid=2321) Initiating optimized duplication from @aaaa_ to @aaaaK
04/03/2017 21:34:26 - Info bpduplicate (pid=2321) Suspend window close behavior is not supported for optimized duplications
04/03/2017 21:34:26 - Info bpduplicate (pid=2321) window close behavior: Continue processing the current image
04/03/2017 21:34:26 - granted resource  MediaID=@aaaa_;DiskVolume=_PhysicalLSU;DiskPool=dp_disk_Secaucus;Path=_PhysicalLSU;StorageServer=SS-Secaucus-DXi_10.159.72.90;MediaServer=xrxsenbumed04
04/03/2017 21:34:27 - Info bpdm (pid=62088) started
04/03/2017 21:34:27 - started process bpdm (pid=62088)
04/03/2017 21:34:28 - Info bpdm (pid=62088) requesting nbjm for media
04/03/2017 21:34:31 - begin writing
04/03/2017 21:34:31 - end writing; write time: 0:00:00
04/03/2017 21:50:35 - Critical bpdm (pid=62088) sts_close_handle failed: 2060022 software error
04/03/2017 21:50:45 - Critical bpdm (pid=62088) sts_close_handle failed: 2060014 operation aborted
04/03/2017 22:06:43 - Critical bpdm (pid=62088) sts_close_handle failed: 2060022 software error
04/03/2017 22:06:43 - Warning bpdm (pid=62088) source image close failed: error 2060022: software error
04/03/2017 22:06:43 - Critical bpdm (pid=62088) sts_close_handle failed: 2060014 operation aborted
04/03/2017 22:06:43 - Critical bpdm (pid=62088) destination image close failed: error 2060014: operation aborted
04/03/2017 22:07:32 - Error bpduplicate (pid=2321) host xrxsenbumed04 backup id servus-newjers3.nam.gad.schneider-electric.com_1491260523 optimized duplication failed, media write error (84).
04/03/2017 22:07:32 - Error bpduplicate (pid=2321) Duplicate of backupid servus-newjers3.nam.gad.schneider-electric.com_1491260523 failed, media write error (84).
04/03/2017 22:07:32 - requesting resource @aaaa_
04/03/2017 22:07:34 - Info bpdm (pid=73803) started
04/03/2017 22:07:34 - started process bpdm (pid=73803)
04/03/2017 22:07:34 - granted resource  MediaID=@aaaa_;DiskVolume=_PhysicalLSU;DiskPool=dp_disk_Secaucus;Path=_PhysicalLSU;StorageServer=SS-Secaucus-DXi_10.159.72.90;MediaServer=xrxsenbumed04
04/03/2017 22:07:36 - Info bpdm (pid=73803) requesting nbjm for media
04/03/2017 22:07:40 - begin writing
04/03/2017 22:07:40 - end writing; write time: 0:00:00
04/03/2017 22:23:31 - Critical bpdm (pid=73803) sts_close_handle failed: 2060022 software error
04/03/2017 22:23:41 - Critical bpdm (pid=73803) sts_close_handle failed: 2060014 operation aborted
04/03/2017 22:23:43 - begin writing
04/03/2017 22:23:43 - end writing; write time: 0:00:00
04/03/2017 22:39:32 - Critical bpdm (pid=73803) sts_close_handle failed: 2060022 software error
04/03/2017 22:39:42 - Critical bpdm (pid=73803) sts_close_handle failed: 2060014 operation aborted
04/03/2017 22:55:33 - Critical bpdm (pid=73803) sts_close_handle failed: 2060022 software error
04/03/2017 22:55:33 - Warning bpdm (pid=73803) source image close failed: error 2060022: software error
04/03/2017 22:55:33 - Critical bpdm (pid=73803) sts_close_handle failed: 2060014 operation aborted
04/03/2017 22:55:33 - Critical bpdm (pid=73803) destination image close failed: error 2060014: operation aborted
04/03/2017 22:55:34 - Error bpduplicate (pid=2321) host xrxsenbumed04 backup id servus-newjers3.nam.gad.schneider-electric.com_1491260571 optimized duplication failed, media write error (84).
04/03/2017 22:55:34 - Error bpduplicate (pid=2321) Duplicate of backupid servus-newjers3.nam.gad.schneider-electric.com_1491260571 failed, media write error (84).
04/03/2017 22:55:34 - requesting resource @aaaa_
04/03/2017 22:55:35 - Info bpdm (pid=92140) started
04/03/2017 22:55:35 - started process bpdm (pid=92140)
04/03/2017 22:55:35 - granted resource  MediaID=@aaaa_;DiskVolume=_PhysicalLSU;DiskPool=dp_disk_Secaucus;Path=_PhysicalLSU;StorageServer=SS-Secaucus-DXi_10.159.72.90;MediaServer=xrxsenbumed04
04/03/2017 22:55:36 - Info bpdm (pid=92140) requesting nbjm for media
04/03/2017 22:55:39 - begin writing
04/03/2017 22:55:39 - end writing; write time: 0:00:00
04/03/2017 23:11:41 - Critical bpdm (pid=92140) sts_close_handle failed: 2060022 software error
04/03/2017 23:11:52 - Critical bpdm (pid=92140) sts_close_handle failed: 2060014 operation aborted
04/03/2017 23:28:24 - Critical bpdm (pid=92140) sts_close_handle failed: 2060022 software error
04/03/2017 23:28:24 - Warning bpdm (pid=92140) source image close failed: error 2060022: software error
04/03/2017 23:28:24 - Critical bpdm (pid=92140) sts_close_handle failed: 2060014 operation aborted
04/03/2017 23:28:24 - Critical bpdm (pid=92140) destination image close failed: error 2060014: operation aborted
04/03/2017 23:29:04 - Error bpduplicate (pid=2321) host xrxsenbumed04 backup id servus-newjers3.nam.gad.schneider-electric.com_1491260572 optimized duplication failed, media write error (84).
04/03/2017 23:29:04 - Error bpduplicate (pid=2321) Duplicate of backupid servus-newjers3.nam.gad.schneider-electric.com_1491260572 failed, media write error (84).
04/03/2017 23:29:04 - Error bpduplicate (pid=2321) Status = no images were successfully processed.
04/03/2017 23:29:04 - Error bpduplicate (pid=2321) Continuing with normal duplication.
04/03/2017 23:29:04 - requesting resource stu_ds03ss01_remote_dup
04/03/2017 23:29:04 - requesting resource @aaaa_
04/03/2017 23:29:04 - reserving resource @aaaa_
04/03/2017 23:30:16 - Info bpduplicate (pid=2321) window close behavior: Suspend
04/03/2017 23:30:16 - resource @aaaa_ reserved
04/03/2017 23:30:16 - granted resource  MediaID=@aaaaK;DiskVolume=_PhysicalLSU;DiskPool=dp_xrxsenbuds03ss01;Path=_PhysicalLSU;StorageServer=xrxsenbuds03ss01_10.173.133.25;MediaServer=xrxsenbumed04
04/03/2017 23:30:16 - granted resource  stu_ds03ss01_remote_dup
04/03/2017 23:30:16 - granted resource  MediaID=@aaaa_;DiskVolume=_PhysicalLSU;DiskPool=dp_disk_Secaucus;Path=_PhysicalLSU;StorageServer=SS-Secaucus-DXi_10.159.72.90;MediaServer=xrxsenbumstr01
04/03/2017 23:30:17 - Info bptm (pid=104851) start
04/03/2017 23:30:17 - started process bptm (pid=104851)
04/03/2017 23:30:17 - Info bptm (pid=104851) using 131072 data buffer size
04/03/2017 23:30:17 - Info bptm (pid=104851) setting receive network buffer to 131072 bytes
04/03/2017 23:30:17 - Info bptm (pid=104851) using 128 data buffers
04/03/2017 23:30:18 - Info bptm (pid=104851) start backup
04/03/2017 23:30:20 - Info bpdm (pid=15926) started
04/03/2017 23:30:20 - started process bpdm (pid=15926)
04/03/2017 23:30:20 - Info bpdm (pid=15926) reading backup image
04/03/2017 23:30:20 - Info bptm (pid=104851) setting receive network buffer to 131072 bytes
04/03/2017 23:30:20 - Info bptm (pid=104851) backup child process is pid 104871
04/03/2017 23:30:21 - Info bpdm (pid=15926) using 128 data buffers
04/03/2017 23:30:21 - Info bpdm (pid=15926) spawning a child process
04/03/2017 23:30:21 - Info bpbrm (pid=15926) child pid: 15943
04/03/2017 23:30:21 - Info bpdm (pid=15926) requesting nbjm for media
04/03/2017 23:30:24 - begin reading
04/03/2017 23:30:38 - Info bptm (pid=104851) waited for full buffer 498 times, delayed 1065 times
04/03/2017 23:30:38 - end reading; read time: 0:00:14
04/03/2017 23:30:38 - Info bpdm (pid=15926) completed reading backup image
04/03/2017 23:30:52 - Info bptm (pid=104851) setting receive network buffer to 131072 bytes
04/03/2017 23:30:52 - Info bptm (pid=104851) backup child process is pid 104959
04/03/2017 23:30:53 - Info bpdm (pid=15926) using 128 data buffers
04/03/2017 23:30:53 - Info bpdm (pid=15926) spawning a child process
04/03/2017 23:30:53 - Info bpbrm (pid=15926) child pid: 16337
04/03/2017 23:30:53 - begin reading
04/03/2017 23:30:54 - end reading; read time: 0:00:01
04/03/2017 23:30:54 - begin reading
04/03/2017 23:30:55 - Info bptm (pid=104851) waited for full buffer 25 times, delayed 182 times
04/03/2017 23:30:55 - end reading; read time: 0:00:01
04/03/2017 23:30:55 - Info bpdm (pid=15926) completed reading backup image
04/03/2017 23:31:08 - Info bptm (pid=104851) setting receive network buffer to 131072 bytes
04/03/2017 23:31:08 - Info bptm (pid=104851) backup child process is pid 105280
04/03/2017 23:31:08 - Info bpdm (pid=15926) using 128 data buffers
04/03/2017 23:31:08 - Info bpdm (pid=15926) spawning a child process
04/03/2017 23:31:08 - Info bpbrm (pid=15926) child pid: 16528
04/03/2017 23:31:08 - begin reading
04/03/2017 23:31:58 - Info bptm (pid=104851) waited for full buffer 1120 times, delayed 3055 times
04/03/2017 23:31:58 - end reading; read time: 0:00:50
04/03/2017 23:31:58 - Info bpdm (pid=15926) completed reading backup image
04/03/2017 23:31:58 - Info bpdm (pid=15926) EXITING with status 0
04/03/2017 23:32:17 - Info bptm (pid=104851) EXITING with status 0 <----------
04/03/2017 23:32:17 - end Duplicate; elapsed time 1:58:53
The requested operation was partially successful  (1)

 

14 REPLIES 14

D_Flood
Level 6

Just to confirm a few things:

1) The SLP in use has two steps:

Step 1 - Backup to the local storage

Step 2 - Duplicate to the remote storage

2) When you check Media and Device Management/Credentials, the source storage has the destination storage checked so that they've internally exchanged passwords (and vice-versa with the destination storage).

3) Both the source and destination are listed in NetBackup as PureDisk type?

4) If you go into Catalog, pick Duplicate, pull up any image there, and rightclick/duplicate and pick the other destination, does your duplication job get the same type errors?

Thanks for your response.

It is not PureDisk storage,

It is Quantum Deduplication DXi storage boxes (6802 and 4700 boxes)

This line in the logs says that NetBackup is trying Puredisk to Puredisk copying:

04/03/2017 21:34:26 - Info Duplicate (pid=2321) Initiating optimized duplication from @aaaa_ to @aaaaK

So something about the way those disk units are set up makes NetBackup think they're either a Media Server Dedup Pool (MSDP) or an Appliance...

 

Both are Quantum Disk STU's - refer the below media from nbdevquery output and the disk type flag is also pointing to Quantum only.

Disk Pool Name      : dp_disk_Secaucus
Disk Type           : Quantum
Disk Volume Name    : _PhysicalLSU
Disk Media ID       : @aaaa_
Total Capacity (GB) : 4656.61
Free Space (GB)     : 3100.56
Use%                : 33
Status              : UP
Flag                : ReadOnWrite
Flag                : AdminUp
Flag                : InternalUp
Num Read Mounts     : 0
Num Write Mounts    : 1
Cur Read Streams    : 1
Cur Write Streams   : 1
Num Repl Sources    : 0
Num Repl Targets    : 0




Disk Pool Name      : dp_xrxsenbuds03ss01
Disk Type           : Quantum
Disk Volume Name    : _PhysicalLSU
Disk Media ID       : @aaaaK
Total Capacity (GB) : 144383.88
Free Space (GB)     : 103626.67
Use%                : 28
Status              : UP
Flag                : ReadOnWrite
Flag                : AdminUp
Flag                : InternalUp
Num Read Mounts     : 0
Num Write Mounts    : 1
Cur Read Streams    : 5
Cur Write Streams   : 8
Num Repl Sources    : 0
Num Repl Targets    : 0

I'm afraid I've hit the limit of what I can help with then since I don't have Quantums so don't know exactly how they're supposed to look like in NetBackup.  But that one log line explains why there's a lot of follow-on PureDisk type errors.

 

quebek
Moderator
Moderator
   VIP    Certified

Hey

Please check if your OST plugins are up to date... bpstsinfo -pi

make sure you are having supported configuration by looking into HCL https://www.veritas.com/support/en_US/article.000025228

 

RiaanBadenhorst
Moderator
Moderator
Partner    VIP    Accredited Certified

hi D_Flood, that message doesn't indicate anything relating to PureDisk. Any OST enabled diskpool to diskpool operation would say that.

RiaanBadenhorst
Moderator
Moderator
Partner    VIP    Accredited Certified

Please set BPDM logging to 5 and rerun wait for the duplication to run again. Then post the BPDM log.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

In addition to Riaan's request - it seems that bpdm on the source media server is failing.
So, that is where we need the bpdm log.

Please ensure that both bptm and bpdm log folders exists on both media servers (Source and target).

I agree with checking software versions at both ends - on the DXi's as well as plugin versions on the media servers.

mph999
Level 6
Employee Accredited

04/03/2017 21:34:26 - Info Duplicate (pid=2321) Initiating optimized duplication from @aaaa_ to @aaaaK

There is nothing in this line suggesting msdp.

MediaIDs such as @aaaab are used for both MSDP, advanced disk and various OST type disks such as Quantum Dxi

please find the plugin version and firmware version from my DXi box and media server:

please check and confirm me the compatibility for this,  also I am gathering bpdm log with verbose 5, i will keep you updated once it is generated.

Source Site – Dxi4700

DXi code version - 3.0.3_47 (11966-55002 Build 14)

Product Model          = DXi4700

Hardware Configuration = 4701

System Version         = 3.0.3_47

Base OS Version        = 3.0.3_47-11966

Operating System       = CentOS release 6.5 (Final)

Application Version    = 3.0.3_47-55002

SCM Build Version      = Build14

Kernel Version         = 2.6.32-504.1.3.el6.Q_4.x86_64

Installation Date      = Thu Jan 22 11:27:03 EST 2015

 

Target Site – Dxi6802

DXi code version - 2.3.2.1 (12459-55813 Build6)

Product Model          = DXi6802

Hardware Configuration = 6802

System Version         = 2.3.2.1

Base OS Version        = 2.3.2.1-12459

Application Version    = 2.3.2.1-55813

SCM Build Version      = Build6

Kernel Version         = 2.6.18-164.15.1.qtm.4

 

OST plugin on Source media server – windows 2012 R2 media server – 773 version

      Plugin Name: libstspiQuantum.dll

       Prefix: Quantum

       Label: Quantum OST plugin

       Build Version: 11

       Build Version Minor: 1

       Operating Version: 11

       Vendor Version: 3.0.1.2841 Quantum plugin (Wind

 

OST plugin on target media server – NBU Appliance 5220 model – 273 version

        Plugin Name: libstspiQuantumMT.so

        Prefix: Quantum

        Label: Quantum OST plugin

        Build Version: 11

        Build Version Minor: 1

        Operating Version: 11

        Vendor Version: 3.1.1.2936 Quantum plugin (Redh

 

 

Attaching the bpdm error log:

Any help on this? still I am facing the same issue :(

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Have you confirmed that these different software versions are supported for duplication/replication?

The source site has a higher version DXi code - 3.0.3_47 
but a lower OST plugin version: 3.0.1.2841

The target site has a lower DXi version - 2.3.2.1
but a higher OST plugin version: 3.1.1.2936

IMHO - this is probably causing the issue.