Forum Discussion

Yet's avatar
Yet
Level 5
6 years ago

Restore is very slow

Backup of 50GB took only 1hr20mins but restore is 10hrs.

Backup host (host1) is different than restore host (host2), but they have same disk (SAN flash disk) and network (10Gb) type.

Media server is Appliance 5240 @ ver 3.1.1

Master server is Linux @ 8.1.2

Client is Linux @ 8.1.1

I've attached job details and restoretrace output.

What's interesting is below snippet, too much delays on emptying bufffer. Not sure if this issue is from client side (reading from 10Gb network and writing to its disks), or within media server (emptying buffer to its 10G network). 

======

Jan 17, 2019 1:07:34 PM - started process bptm (pid=324936)
Jan 17, 2019 1:07:34 PM - Info bptm (pid=324936) reading backup image
Jan 17, 2019 1:07:34 PM - Info bptm (pid=324936) using 40 data buffers
Jan 17, 2019 1:07:34 PM - Info bptm (pid=324936) spawning a child process
Jan 17, 2019 1:07:34 PM - Info bptm (pid=324936) child pid: 324938
Jan 17, 2019 1:07:36 PM - begin reading
Jan 17, 2019 10:48:21 PM - Info bptm (pid=324936) waited for empty buffer 42196 times, delayed 2300024 times
Jan 17, 2019 10:48:21 PM - end reading; read time: 9:40:45
Jan 17, 2019 10:57:46 PM - restored from image host1.abc.com_1547686800; restore time: 9:52:43
Jan 17, 2019 10:57:47 PM - end Restore; elapsed time 9:58:56

======

I tried working on NUMBER_DATA_BUFFERS and NET_BUFFER_SZ but no significant results. 

Any idea?

8.1.2
Linux
  • Marianne's avatar
    Marianne
    6 years ago

    Please try to test network transfer / connection outside of NBU (e.g. ftp or even continuous ping) between media server and destination client. 

    We see quite a number of warnings like these:

    put_length_bytes_nonblocking: could only write 18 of 87 bytes to network:  Resource temporarily unavailable (11)

  • Marianne's avatar
    Marianne
    Level 6
    6 years ago

    Please try to test network transfer / connection outside of NBU (e.g. ftp or even continuous ping) between media server and destination client. 

    We see quite a number of warnings like these:

    put_length_bytes_nonblocking: could only write 18 of 87 bytes to network:  Resource temporarily unavailable (11)

    • Nicolai's avatar
      Nicolai
      Moderator
      6 years ago

      Remove NET_BUFFER_SZ, setting is no longer in use, but it is still referenced in manny of Veritas tech notes.

      See Best practices for NET_BUFFER_SZ and Buffer_size, why can't NetBackup change the TCP send/receive space

      https://www.veritas.com/support/en_US/article.TECH28339

      Also I highly recommend you set NUMBER_DATA_BUFFERS & SIZE_DATA_BUFFERS. Tape performance will go up substantial. You are using the 64K block size. I recommend 262144 as block size and setting SIZE_DATA_BUFFERS to 256

      Windows :
      https://www.veritas.com/support/en_US/article.100030830.html

      UNIX/Linux:
      https://www.veritas.com/support/en_US/article.TECH1724

      • mph999's avatar
        mph999
        Level 6
        6 years ago

        I could be wrong, but I thought you had to specfically set NET_BUFFER_SZ to 0 in order to 'disable' it.

    • Yet's avatar
      Yet
      Level 5
      6 years ago

      I tested network transfer rate (nbperfchk) beetween media server & client, and found network to be the culprit, response ranges from 0.0 - 0.2MB/s. Later on I found out that NIC is not 10G but 1G! So much for taking their word. 

      Re "Resource temporarily unavailable", this caught my attention but noticed that it's from BPRD log of master server. 

      Many thanks for all repsonses.

      • sdo's avatar
        sdo
        Moderator
        6 years ago

        further info:

        Do NOT be tempted to use a client side "buffer size" of zero.

        I was burned by this.

        .

        This TN:                https://www.veritas.com/support/en_US/article.TECH28339

        …says that it is good practice to use a zero for Windows NetBackup Client “buffer_size”.

         .

        I would recommend that we do not use:   buffer_size = 0

         .

        …because you’ll get terrible backup performance for plain client backups when this is true:

                                        buffer_size = 0

                        AND       NetBackup Client is v7.7.2            (but possibly also affects v7.7 and v7.7.1)

                        AND       ( OS = Any Of( Windows 2008, 2008R2SP1, 2012, 2012R2 ) )

        .

        Something must have changed in at least v7.7.2 NetBackup Client for Windows…

        …because a “buffer_size = 0” value is fine for Windows NetBackup Client v7.6.1.2.