cancel
Showing results for 
Search instead for 
Did you mean: 

Restore is very slow

Yet
Level 5
Partner Accredited

Backup of 50GB took only 1hr20mins but restore is 10hrs.

Backup host (host1) is different than restore host (host2), but they have same disk (SAN flash disk) and network (10Gb) type.

Media server is Appliance 5240 @ ver 3.1.1

Master server is Linux @ 8.1.2

Client is Linux @ 8.1.1

I've attached job details and restoretrace output.

What's interesting is below snippet, too much delays on emptying bufffer. Not sure if this issue is from client side (reading from 10Gb network and writing to its disks), or within media server (emptying buffer to its 10G network). 

======

Jan 17, 2019 1:07:34 PM - started process bptm (pid=324936)
Jan 17, 2019 1:07:34 PM - Info bptm (pid=324936) reading backup image
Jan 17, 2019 1:07:34 PM - Info bptm (pid=324936) using 40 data buffers
Jan 17, 2019 1:07:34 PM - Info bptm (pid=324936) spawning a child process
Jan 17, 2019 1:07:34 PM - Info bptm (pid=324936) child pid: 324938
Jan 17, 2019 1:07:36 PM - begin reading
Jan 17, 2019 10:48:21 PM - Info bptm (pid=324936) waited for empty buffer 42196 times, delayed 2300024 times
Jan 17, 2019 10:48:21 PM - end reading; read time: 9:40:45
Jan 17, 2019 10:57:46 PM - restored from image host1.abc.com_1547686800; restore time: 9:52:43
Jan 17, 2019 10:57:47 PM - end Restore; elapsed time 9:58:56

======

I tried working on NUMBER_DATA_BUFFERS and NET_BUFFER_SZ but no significant results. 

Any idea?

1 ACCEPTED SOLUTION

Accepted Solutions

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Please try to test network transfer / connection outside of NBU (e.g. ftp or even continuous ping) between media server and destination client. 

We see quite a number of warnings like these:

put_length_bytes_nonblocking: could only write 18 of 87 bytes to network:  Resource temporarily unavailable (11)

View solution in original post

8 REPLIES 8

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Please try to test network transfer / connection outside of NBU (e.g. ftp or even continuous ping) between media server and destination client. 

We see quite a number of warnings like these:

put_length_bytes_nonblocking: could only write 18 of 87 bytes to network:  Resource temporarily unavailable (11)

Nicolai
Moderator
Moderator
Partner    VIP   

Remove NET_BUFFER_SZ, setting is no longer in use, but it is still referenced in manny of Veritas tech notes.

See Best practices for NET_BUFFER_SZ and Buffer_size, why can't NetBackup change the TCP send/receive space

https://www.veritas.com/support/en_US/article.TECH28339

Also I highly recommend you set NUMBER_DATA_BUFFERS & SIZE_DATA_BUFFERS. Tape performance will go up substantial. You are using the 64K block size. I recommend 262144 as block size and setting SIZE_DATA_BUFFERS to 256

Windows :
https://www.veritas.com/support/en_US/article.100030830.html

UNIX/Linux:
https://www.veritas.com/support/en_US/article.TECH1724

mph999
Level 6
Employee Accredited

I could be wrong, but I thought you had to specfically set NET_BUFFER_SZ to 0 in order to 'disable' it.

Nicolai
Moderator
Moderator
Partner    VIP   

You are correct @mph999

From tech note :

 Accordingly the best NetBackup configuration is to disable tuning by placing a zero (0) into the NET_BUFFER_SZ file on media servers and UNIX/Linux clients.   Simply deleting the file is not equivalent because some NetBackup process have default setsockopt API calls configured to overcome past external problems with various platforms and drivers.

So either a 0 in NET_BUFFER_SZ or remove the file to get to a fuctional setup.

Yet
Level 5
Partner Accredited

I tested network transfer rate (nbperfchk) beetween media server & client, and found network to be the culprit, response ranges from 0.0 - 0.2MB/s. Later on I found out that NIC is not 10G but 1G! So much for taking their word. 

Re "Resource temporarily unavailable", this caught my attention but noticed that it's from BPRD log of master server. 

Many thanks for all repsonses.

sdo
Moderator
Moderator
Partner    VIP    Certified

further info:

Do NOT be tempted to use a client side "buffer size" of zero.

I was burned by this.

.

This TN:                https://www.veritas.com/support/en_US/article.TECH28339

…says that it is good practice to use a zero for Windows NetBackup Client “buffer_size”.

 .

I would recommend that we do not use:   buffer_size = 0

 .

…because you’ll get terrible backup performance for plain client backups when this is true:

                                buffer_size = 0

                AND       NetBackup Client is v7.7.2            (but possibly also affects v7.7 and v7.7.1)

                AND       ( OS = Any Of( Windows 2008, 2008R2SP1, 2012, 2012R2 ) )

.

Something must have changed in at least v7.7.2 NetBackup Client for Windows…

…because a “buffer_size = 0” value is fine for Windows NetBackup Client v7.6.1.2.

mph999
Level 6
Employee Accredited

Really ... ohhhh ....

What values have you found to be good ?

I'll see if I can speak with my colleague who wrote that TN.

mph999
Level 6
Employee Accredited

I 'spoke' to my collegue Craig who wrote the TN, he very helpfully also provided an explanation of what he thinks is the issue.

In summary - with the introduction of 'nbtar' process, some changes were made that inadvertidly upset NET_BUFFER_SZ.

An EEB was created to 'fix' this, ET 3925723 and EEBs were created for :

7.7.3, 8.0 and 8.1

It may well be that this is the issue SDO reports on 7.7.2.  It seems no EEB has been craeted for that version, but this could simply be because nobody request one.

The issue shows as 'fixed' in NBU 8.1.1 / 3.1.1

Hope this help,

Martin