Forum Discussion

KBfinder506's avatar
KBfinder506
Level 4
6 months ago

NBU 10.3 tape to tape duplicate optimization.

Hi people:

I'm reading the tunning guide and other documentation but feeling a little bit lost. 

We have a 4 LTO9 drives library (Spec of Library).  It's connected via FC to our master NBU (on windows 2022) (sorry I'm bit older to catch new terms). The master is using 1Gpbs ethernet interface to the vms and rest of the infrastructure. 

Today I'm duplicating a 14TB bkp tape-to-tape.  But feel the need some optimization since have this messages:

Jul 28, 2025 11:50:14 AM - begin Duplicate
Jul 28, 2025 11:50:17 AM - requesting resource  mynbu01-hcart3-robot-tld-0
Jul 28, 2025 11:50:17 AM - requesting resource  ADR025
Jul 28, 2025 11:50:17 AM - reserving resource ADR025
Jul 28, 2025 11:50:21 AM - Info bpduplicate (pid=6972) The data-in-transit encryption (DTE) is enabled, as the DTE mode of the backup image is set to 'On'
Jul 28, 2025 11:50:21 AM - Info bptm (pid=14752) start
Jul 28, 2025 11:50:21 AM - Info bptm (pid=14752) Encrypting data-in-transit
Jul 28, 2025 11:50:21 AM - started process bptm (pid=14752)
Jul 28, 2025 11:50:21 AM - resource ADR025 reserved
Jul 28, 2025 11:50:21 AM - granted resource  000025
Jul 28, 2025 11:50:21 AM - granted resource  HPE.ULTRIUM9-SCSI.001
Jul 28, 2025 11:50:21 AM - granted resource  mynbu01-hcart3-robot-tld-0
Jul 28, 2025 11:50:21 AM - granted resource  ADR025
Jul 28, 2025 11:50:21 AM - granted resource  HPE.ULTRIUM9-SCSI.002
Jul 28, 2025 11:50:22 AM - Info bptm (pid=14752) start backup
Jul 28, 2025 11:50:22 AM - Info bptm (pid=19164) start
Jul 28, 2025 11:50:22 AM - Info bptm (pid=19164) Encrypting data-in-transit
Jul 28, 2025 11:50:22 AM - started process bptm (pid=19164)
Jul 28, 2025 11:50:22 AM - Info bptm (pid=19164) reading backup image
Jul 28, 2025 11:50:22 AM - Info bptm (pid=19164) using 30 data buffers
Jul 28, 2025 11:50:22 AM - Info bptm (pid=14752) Waiting for mount of media id 000025 (copy 1) on server mynbu01.mydomain.local.
Jul 28, 2025 11:50:22 AM - started process bptm (pid=14752)
Jul 28, 2025 11:50:22 AM - mounting 000025
Jul 28, 2025 11:50:22 AM - Info bptm (pid=19164) Waiting for mount of media id ADR025 (copy 2) on server mynbu01.mydomain.local.
Jul 28, 2025 11:50:22 AM - started process bptm (pid=19164)
Jul 28, 2025 11:50:22 AM - mounting ADR025
Jul 28, 2025 11:50:22 AM - Info bptm (pid=19164) INF - Waiting for mount of media id ADR025 on server mynbu01.mydomain.local for reading.
Jul 28, 2025 11:50:22 AM - Info bptm (pid=14752) INF - Waiting for mount of media id 000025 on server mynbu01.mydomain.local for writing.
Jul 28, 2025 11:51:07 AM - mounted ADR025; mount time: 0:00:45
Jul 28, 2025 11:51:07 AM - Info bptm (pid=19164) ADR025
Jul 28, 2025 11:51:07 AM - Info bptm (pid=19164) INF - Waiting for positioning of media id ADR025 on server mynbu01.mydomain.local for reading.
Jul 28, 2025 11:51:07 AM - positioning ADR025 to file 11
Jul 28, 2025 11:51:36 AM - Info bptm (pid=14752) media id 000025 mounted on drive index 1, drivepath {1,0,2,0}, drivename HPE.ULTRIUM9-SCSI.001, copy 1
Jul 28, 2025 11:51:53 AM - positioned ADR025; position time: 0:00:46
Jul 28, 2025 11:51:53 AM - begin reading
Jul 28, 2025 12:40:12 PM - Info bptm (pid=19164) waited for empty buffer 55084 times, delayed 161109 times
Jul 28, 2025 12:40:12 PM - end reading; read time: 0:48:19
Jul 28, 2025 12:40:12 PM - positioning ADR025 to file 12
Jul 28, 2025 12:40:12 PM - positioned ADR025; position time: 0:00:00
Jul 28, 2025 12:40:12 PM - begin reading
Jul 28, 2025 1:23:37 PM - Info bptm (pid=19164) waited for empty buffer 52280 times, delayed 136968 times
Jul 28, 2025 1:23:37 PM - end reading; read time: 0:43:25
Jul 28, 2025 1:23:37 PM - positioning ADR025 to file 13
Jul 28, 2025 1:23:37 PM - positioned ADR025; position time: 0:00:00
Jul 28, 2025 1:23:37 PM - begin reading
Jul 28, 2025 2:17:07 PM - Info bptm (pid=19164) waited for empty buffer 51567 times, delayed 166587 times
Jul 28, 2025 2:17:07 PM - end reading; read time: 0:53:30
Jul 28, 2025 2:17:07 PM - positioning ADR025 to file 14
Jul 28, 2025 2:17:07 PM - positioned ADR025; position time: 0:00:00
Jul 28, 2025 2:17:07 PM - begin reading
Jul 28, 2025 3:21:39 PM - Info bptm (pid=19164) waited for empty buffer 56165 times, delayed 190912 times
Jul 28, 2025 3:21:39 PM - end reading; read time: 1:04:32
Jul 28, 2025 3:21:39 PM - positioning ADR025 to file 15
Jul 28, 2025 3:21:39 PM - positioned ADR025; position time: 0:00:00
Jul 28, 2025 3:21:39 PM - begin reading
Jul 28, 2025 4:21:36 PM - Info bptm (pid=19164) waited for empty buffer 57278 times, delayed 142774 times
Jul 28, 2025 4:21:36 PM - end reading; read time: 0:59:57
Jul 28, 2025 4:21:36 PM - positioning ADR025 to file 16
Jul 28, 2025 4:21:37 PM - positioned ADR025; position time: 0:00:01
Jul 28, 2025 4:21:38 PM - begin reading
Jul 28, 2025 4:56:32 PM - Info bptm (pid=19164) waited for empty buffer 41788 times, delayed 110933 times
Jul 28, 2025 4:56:33 PM - end reading; read time: 0:34:55
Jul 28, 2025 4:56:33 PM - positioning ADR025 to file 17
Jul 28, 2025 4:56:33 PM - positioned ADR025; position time: 0:00:00
Jul 28, 2025 4:56:33 PM - begin reading
Jul 28, 2025 5:39:12 PM - Info bptm (pid=19164) waited for empty buffer 48429 times, delayed 133305 times
Jul 28, 2025 5:39:12 PM - end reading; read time: 0:42:39
Jul 28, 2025 5:39:12 PM - positioning ADR025 to file 18
Jul 28, 2025 5:39:12 PM - positioned ADR025; position time: 0:00:00
Jul 28, 2025 5:39:12 PM - begin reading
Jul 28, 2025 6:34:23 PM - Info bptm (pid=19164) waited for empty buffer 64518 times, delayed 172092 times
Jul 28, 2025 6:34:23 PM - end reading; read time: 0:55:11
Jul 28, 2025 6:34:23 PM - positioning ADR025 to file 19
Jul 28, 2025 6:34:25 PM - positioned ADR025; position time: 0:00:02
Jul 28, 2025 6:34:25 PM - begin reading
Jul 28, 2025 8:12:14 PM - Info bptm (pid=19164) waited for empty buffer 101831 times, delayed 305001 times
Jul 28, 2025 8:12:14 PM - end reading; read time: 1:37:49
Jul 28, 2025 8:12:14 PM - positioning ADR025 to file 20
Jul 28, 2025 8:12:14 PM - positioned ADR025; position time: 0:00:00
Jul 28, 2025 8:12:14 PM - begin reading
Jul 28, 2025 9:44:53 PM - Info bptm (pid=19164) waited for empty buffer 99616 times, delayed 245508 times
Jul 28, 2025 9:44:53 PM - end reading; read time: 1:32:39
Jul 28, 2025 9:44:53 PM - positioning ADR025 to file 21
Jul 28, 2025 9:44:53 PM - positioned ADR025; position time: 0:00:00
Jul 28, 2025 9:44:53 PM - begin reading
Jul 28, 2025 11:50:55 PM - Info bptm (pid=19164) waited for empty buffer 128929 times, delayed 350979 times
Jul 28, 2025 11:50:55 PM - end reading; read time: 2:06:02
Jul 28, 2025 11:50:55 PM - positioning ADR025 to file 22
Jul 28, 2025 11:50:55 PM - positioned ADR025; position time: 0:00:00
Jul 28, 2025 11:50:55 PM - begin reading
Jul 29, 2025 1:22:57 AM - Info bptm (pid=19164) waited for empty buffer 127122 times, delayed 284730 times
Jul 29, 2025 1:22:57 AM - end reading; read time: 1:32:02
Jul 29, 2025 1:22:57 AM - positioning ADR025 to file 23
Jul 29, 2025 1:22:57 AM - positioned ADR025; position time: 0:00:00
Jul 29, 2025 1:22:57 AM - begin reading
Jul 29, 2025 2:16:30 AM - Info bptm (pid=19164) waited for empty buffer 73439 times, delayed 171201 times
Jul 29, 2025 2:16:31 AM - end reading; read time: 0:53:34
Jul 29, 2025 2:16:31 AM - positioning ADR025 to file 24
Jul 29, 2025 2:16:31 AM - positioned ADR025; position time: 0:00:00
Jul 29, 2025 2:16:31 AM - begin reading
Jul 29, 2025 3:58:59 AM - Info bptm (pid=19164) waited for empty buffer 137768 times, delayed 323641 times
Jul 29, 2025 3:58:59 AM - end reading; read time: 1:42:28
Jul 29, 2025 3:58:59 AM - positioning ADR025 to file 25
Jul 29, 2025 3:58:59 AM - positioned ADR025; position time: 0:00:00
Jul 29, 2025 3:58:59 AM - begin reading
Jul 29, 2025 5:42:05 AM - Info bptm (pid=19164) waited for empty buffer 131438 times, delayed 329688 times
Jul 29, 2025 5:42:05 AM - end reading; read time: 1:43:06
Jul 29, 2025 5:42:05 AM - positioning ADR025 to file 26
Jul 29, 2025 5:42:05 AM - positioned ADR025; position time: 0:00:00
Jul 29, 2025 5:42:05 AM - begin reading
Jul 29, 2025 6:52:23 AM - Info bptm (pid=19164) waited for empty buffer 88174 times, delayed 222527 times
Jul 29, 2025 6:52:23 AM - end reading; read time: 1:10:18
Jul 29, 2025 6:52:23 AM - positioning ADR025 to file 27
Jul 29, 2025 6:52:23 AM - positioned ADR025; position time: 0:00:00
Jul 29, 2025 6:52:23 AM - begin reading
Jul 29, 2025 8:09:06 AM - Info bptm (pid=19164) waited for empty buffer 110740 times, delayed 257401 times
Jul 29, 2025 8:09:06 AM - end reading; read time: 1:16:43
Jul 29, 2025 8:09:06 AM - positioning ADR025 to file 28
Jul 29, 2025 8:09:06 AM - positioned ADR025; position time: 0:00:00
Jul 29, 2025 8:09:06 AM - begin reading
Jul 29, 2025 9:57:42 AM - Info bptm (pid=19164) waited for empty buffer 129522 times, delayed 344650 times
Jul 29, 2025 9:57:42 AM - end reading; read time: 1:48:36
Jul 29, 2025 9:57:42 AM - positioning ADR025 to file 29
Jul 29, 2025 9:57:42 AM - positioned ADR025; position time: 0:00:00
Jul 29, 2025 9:57:42 AM - begin reading
Jul 29, 2025 11:34:56 AM - Info bptm (pid=19164) waited for empty buffer 119928 times, delayed 314205 times
Jul 29, 2025 11:34:56 AM - end reading; read time: 1:37:14
Jul 29, 2025 11:34:56 AM - positioning ADR025 to file 30
Jul 29, 2025 11:34:56 AM - positioned ADR025; position time: 0:00:00
Jul 29, 2025 11:34:56 AM - begin reading
Jul 29, 2025 1:57:12 PM - Info bptm (pid=19164) waited for empty buffer 146636 times, delayed 462434 times
Jul 29, 2025 1:57:12 PM - end reading; read time: 2:22:16
Jul 29, 2025 1:57:12 PM - positioning ADR025 to file 31
Jul 29, 2025 1:57:12 PM - positioned ADR025; position time: 0:00:00
Jul 29, 2025 1:57:12 PM - begin reading
Jul 29, 2025 4:06:34 PM - Info bptm (pid=19164) waited for empty buffer 134277 times, delayed 423117 times
Jul 29, 2025 4:06:35 PM - end reading; read time: 2:09:23
Jul 29, 2025 4:06:35 PM - positioning ADR025 to file 32
Jul 29, 2025 4:06:35 PM - positioned ADR025; position time: 0:00:00
Jul 29, 2025 4:06:35 PM - begin reading
Jul 29, 2025 5:58:56 PM - Info bptm (pid=19164) waited for empty buffer 134166 times, delayed 353150 times
Jul 29, 2025 5:58:56 PM - end reading; read time: 1:52:21
Jul 29, 2025 5:58:56 PM - positioning ADR025 to file 33
Jul 29, 2025 5:58:56 PM - positioned ADR025; position time: 0:00:00
Jul 29, 2025 5:58:56 PM - begin reading
Jul 29, 2025 7:43:59 PM - Info bptm (pid=19164) waited for empty buffer 149262 times, delayed 318682 times
Jul 29, 2025 7:43:59 PM - end reading; read time: 1:45:03
Jul 29, 2025 7:43:59 PM - positioning ADR025 to file 34
Jul 29, 2025 7:44:01 PM - positioned ADR025; position time: 0:00:02
Jul 29, 2025 7:44:01 PM - begin reading
Jul 29, 2025 9:02:57 PM - Info bptm (pid=19164) waited for empty buffer 135835 times, delayed 272807 times
Jul 29, 2025 9:02:57 PM - end reading; read time: 1:18:56
Jul 29, 2025 9:02:57 PM - positioning ADR025 to file 35
Jul 29, 2025 9:02:57 PM - positioned ADR025; position time: 0:00:00
Jul 29, 2025 9:02:57 PM - begin reading
Jul 29, 2025 10:26:39 PM - Info bptm (pid=19164) waited for empty buffer 132874 times, delayed 267257 times
Jul 29, 2025 10:26:39 PM - end reading; read time: 1:23:42
Jul 29, 2025 10:26:39 PM - positioning ADR025 to file 36
Jul 29, 2025 10:26:39 PM - positioned ADR025; position time: 0:00:00
Jul 29, 2025 10:26:39 PM - begin reading
Jul 29, 2025 11:43:23 PM - Info bptm (pid=19164) waited for empty buffer 130568 times, delayed 262648 times
Jul 29, 2025 11:43:23 PM - end reading; read time: 1:16:44
Jul 29, 2025 11:43:23 PM - positioning ADR025 to file 37
Jul 29, 2025 11:43:23 PM - positioned ADR025; position time: 0:00:00
Jul 29, 2025 11:43:23 PM - begin reading
Jul 30, 2025 1:04:25 AM - Info bptm (pid=19164) waited for empty buffer 141535 times, delayed 283765 times
Jul 30, 2025 1:04:25 AM - end reading; read time: 1:21:02
Jul 30, 2025 1:04:25 AM - positioning ADR025 to file 38
Jul 30, 2025 1:04:25 AM - positioned ADR025; position time: 0:00:00
Jul 30, 2025 1:04:25 AM - begin reading
Jul 30, 2025 2:24:22 AM - Info bptm (pid=19164) waited for empty buffer 137087 times, delayed 275441 times
Jul 30, 2025 2:24:22 AM - end reading; read time: 1:19:57
Jul 30, 2025 2:24:22 AM - positioning ADR025 to file 39
Jul 30, 2025 2:24:22 AM - positioned ADR025; position time: 0:00:00
Jul 30, 2025 2:24:22 AM - begin reading
Jul 30, 2025 3:40:34 AM - Info bptm (pid=19164) waited for empty buffer 128606 times, delayed 258613 times
Jul 30, 2025 3:40:34 AM - end reading; read time: 1:16:12
Jul 30, 2025 3:40:34 AM - positioning ADR025 to file 40
Jul 30, 2025 3:40:34 AM - positioned ADR025; position time: 0:00:00
Jul 30, 2025 3:40:34 AM - begin reading
Jul 30, 2025 4:54:11 AM - Info bptm (pid=19164) waited for empty buffer 118125 times, delayed 238708 times
Jul 30, 2025 4:54:11 AM - end reading; read time: 1:13:37
Jul 30, 2025 4:54:11 AM - positioning ADR025 to file 41
Jul 30, 2025 4:54:11 AM - positioned ADR025; position time: 0:00:00
Jul 30, 2025 4:54:11 AM - begin reading
Jul 30, 2025 6:16:58 AM - Info bptm (pid=19164) waited for empty buffer 132649 times, delayed 266171 times

With your expertise can we squish some optimization of the hardware playing with the size and/or numbers of buffers or the NBU is having the max of the drives? The server has 128GB of RAM and  two Processor    Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHz, 2095 Mhz, 12 Core(s), 24 Logical Processor(s).

The original bkp took almost 96 hours, but since LTO9 have speeds around 400MB/s I was hoping at least a 24 hours tape-to-tape copy but at this time 47% completed at 54:40 hours:minutes. Seems that this copy will take a bit more time. 

Sorry about the length of the question but willing to hear your thoughts from you!

 

optimization
speed
tape-2-tape

6 Replies

  • KBfinder506's avatar
    KBfinder506
    Level 4
    6 months ago

    Hi people, I just want to thank everyone who read and those who post their opinions on the subject. 

    For my lucky sake, the tape to tape had several shortcomings that allow me to test the increase of buffers from 30 to 3000. After every increase I ran perfmon on diag and performance.  Every change had little or no impact on used/free RAM.  My last setting let 109GB free RAM and reduced the duplication time from 100 hours to just 24:16 hours. A huge difference on tape to tape copy/duplication. 

     

  • tcsimerson's avatar
    tcsimerson
    Level 3
    6 months ago

    You can do some buffer tuning that should improve tape throughput speeds.  Toward the beginning of your output there is a line that states "using 30 data buffers".  You can create a setting file that contains an integer value that will increase the number of data buffers used for tape drives.  That file is install_path\VERITAS\NetBackup\db\config\NUMBER_DATA_BUFFERS.  Create that file and put in the value of 60 as a starting value.  You should not need to restart NBU as this file is looked for every time a tape job is started.  Tuning the number of buffers should help both your backup speeds and your duplication speeds.

    One important note:  over allocating buffers can quickly suck up all your system memory if you don't calculate the sizes correctly.  At the default value of 30 buffers, each active tape drive will allocate 1920KB of memory buffer space.  When all 4 of your drives are active, you'll be allocating 7680KB (7.5MB) of memory.  By doubling the buffer count to 60, this would allocate 15MB of memory.  So, do your tuning in increments closely monitoring your overall system memory usage and performance.  I would suggest doing your tuning in increments of 30, i.e., initially set to 60, then 90, 120, etc.

    Please understand that you are severely limited by your network speeds at this time.  A 1Gbit Ethernet link could transfer a maximum of 10TB per day.  With the proper tuning, I suspect you should be able to get your backups down to 36 - 40 hours (14TB / 10TB per day = 1.4 days or about 34 hours).  This is assuming that there are no other bottlenecks in the environment and that I've done my math correctly.

    • KBfinder506's avatar
      KBfinder506
      Level 4
      6 months ago

      I'm agree about the network bottleneck, we expect to change the port switch from 1Gbps to 10Gbps before 2026 began. 

  • StefanosM's avatar
    StefanosM
    Level 6
    6 months ago

    Let me check if I understand correctly.

    You are backing up 14TB of VMs directly to tape using a 1Gbit network interface, and then duplicating from tape to tape. Your server is an "all-in-one" server (both master and media).

    Could you tell me the speed of your Fibre Channel (FC) connection and how many FC connections you are using to connect to the drives?

    • KBfinder506's avatar
      KBfinder506
      Level 4
      6 months ago

      Hi StefanosM, sorry about the confusion. I made a bkp of 14TB of data (several files) from a vms.

      We always for sensitive data make a dual tape bkp from 1 session. One tape is VolPoolOne and the other tape is on VolPoolTwo. The policy writes on both through [X] Multiple copies on the schedule tab. 

      Since one tape went bad that's for another discussion; but the 2nd tape finished successfully the backup.  We are duplicating the good bkp tape to another tape. 

      We choose tape-to-tape since in the past was better than load the network redoing the backup and in general terms; don't have to deal with changes on the backed data. 

      The Master is also the MEdia (I didn't tell it?). Sadly, is on Windows and with two FC hbas to the fabric 32Gbps each one.. but will confirm.

      • KBfinder506's avatar
        KBfinder506
        Level 4
        6 months ago

        Just to clarify, the Master and Media server is using a dualport  32Gbps HPE SN1610E 32Gb hba.