Windows 2008 R2, Netbackup 7.1, NET_BUFFER_SZ

[SYMAJ]

I am seeing 'unexpected' results when setting the NET_BUFFER_SZ and Communications buffers on the Windows platform with NBU 7.1 (update 7101) and Windows 2008R2.

Whatever I set the values to, the job log reports that it is using 262144 data buffer size and the the receive network buffer size is being set to 1049600.  I have varied the values in the communications buffer on te client from 16, 32, 64, 129, 256 - and the NET_BUFFER_SZ on the media server to the corresponding values.

Are these values recognised in 7.1 ?

My issue is that I have seen a significant decrease in throughput from the network clients since a move of hardware and implemented the server touchfiles NET_BUFFER_SZ.  Previously the site had got all clients set to 129 communications buffer, but had the NET_BUFFER_SZ touchfile in the wrong location.  I have now created it in the correct location (install_directory\netbackup) but the values appear to be being ignored.

Any help appreciated.

AJ.

[Yasuhisa_Ishika · Accepted solution]

Check page 116 on this guide. NET_BUFFER_SZ is a parameter for media server, and must be specified by bytes. Value of 256(or less) is too small. Your current value must be brought by a value of SIZE_DATA_BUFFERS(256kb x 4 + 1024).

http://www.symantec.com/docs/DOC4483

Setting the network buffer size for the NetBackup media server
The NetBackup media server has a tunable parameter that you can use to adjust
the size of the network buffer space. The operating system uses this buffer space
for the connection between the bptm child process and the client process. This
buffer space caches either received data from the network (a backup) or written
data to the network (a restore). The parameter sets the value for NetBackup to
use for the network buffer space, but the operating system may not allow the
change.
   :
Windows
The default value for this parameter is derived from the NetBackup data buffer
size using the following formula:
For backup jobs: (<data_buffer_size> * 4) + 1024
For restore jobs: (<data_buffer_size> * 2) + 1024
For tape:
Because the default value for the NetBackup data buffer size is 65536 bytes, the
formula results in the following: a default NetBackup network buffer size of 263168
bytes for backups and 132096 bytes for restores.
For disk:
Because the default value for the NetBackup data buffer size is 262144 bytes, the
formula results in the following: a default NetBackup network buffer size of
1049600 bytes for backups and 525312 bytes for restores.
Tuning the NetBackup data transfer path 117
NetBackup network performance in the data transfer path
To set the network buffer size
1 Create the following files:
UNIX
/usr/openv/netbackup/NET_BUFFER_SZ
/usr/openv/netbackup/NET_BUFFER_SZ_REST
Windows
install_path\NetBackup\NET_BUFFER_SZ
install_path\NetBackup\NET_BUFFER_SZ_REST

Your performance degration may be caused by "a move of hardware" itself (change of bandwidth or latency, incorrect setting of NIC negotiation, ...).

[SYMAJ]

Thanks for that.

The hardware move was from older servers to HP DL360G7 servers - much 'meatier' servers and should provide betther throughput.

Tape drives are the same (LTO4 - fibre attached), O/S was changed from W2K3 to W2K8R2, all drivers for fibre cards / tape drives etc are current (64 bit).  NBU was at 7101 on the old servers so no changes there.  NIC's are set to 1000/FULL - so no auto-negotiate issues.

We did have a master/media and a media server previously - now we have a dedicated master server (not doing any tape work) and three media servers.  Clients are all at 7101 level, as they were before the h/w move so no changes there.

All clients have communications buffer set to 129 in the client GUI.  Master Server / Media Servers have NO NET_BUFFER_SZ file at present, Master / Media Servers have their communications buffer set to 16 in the client GUI.  Whilst these values don't make any sense to me these are what gives us the best performance at present.  As per previously - when I change the communications buffer setting on the client or the NET_BUFFER_SZ file on the media server the job log still reports that it is using 262144 data buffer size and the the receive network buffer size is being set to 1049600.  Why does this not change as I change the values ?

It is puzzling to see why the performance has decreased.

Any more ideas ?

AJ.

[SYMAJ]

I have raised a support call regarding this issue, and am following up also with local Symantec support.  I will update with any findings.

AJ

[Philip_Drew]

Hi,

I'm having the exact same issue - did you get yours resolved ??

[Philip_Drew]

I have noticed that even while the network buffer size has been changed from the default the default continues to show in the job log.  Could possibly be a bug.

[Michael_G_Ander]

As it is Windows you might have better luck with setting the net buffer in the registry, know we use registry for our CLIENT_READ_TIMEOUT & CLIENT_CONNECT_TIMEOUT settings

[fbmw]

I have tried the things you have all tried.  I also have an open issue with Symantec Support.  I also moved to Windows Server 2008 r2 64-bit with 1 master and 3 media servers. 

Support has had me looking at network issues.  Not sure if that is the answer.  I will keep you posted if I get any news.

MW

[SYMAJ]

Did you get any resolution to your issue ??  I am still in the same situation - with Symantec looking towards the network / NICs.  It only seems to be an issue when the system is under load, I hve run test jobs when the system is idle and can get the performance expected / required (I even re-introduced a W2K3 media server from the old environment).  In all cases once I 'load-up' the system the performance degrades significantly.  At presentwe are looking into the area of open-sockets / socket timeouts on the W2K8R2 servers, as this area seems to be the only common factor (Network issues on W2K8R2 when under load - as even with a W2K3 client through a W2K3 media server - I still have a W2K8R2 master server in the picture.

In summary, I have gone back to a situation whereby I am using a W2K3 client (exact config as per previous environment - same physical server and O/S build untouched), a W2K3 media server (exact config as perprevious environment - same physical server and O/S build untouched) - the only new thing in the environment is a W2K8R2 server.  Whether I run backups through the W2K8R2 media servers or theW2K3 media server - performance is good when running a single job but once I run the regular nightly schedule it all goes pear shaped !!

AJ.

[jmanini]

I have been over and over these settings for months now and cannot get any consist ant faster speeds. What kind of speeds are folks seeing out there? My fastest speeds are from Linux clients which can run as high as 50ish MB's per where windows clients may get half or less of that. All of my backups are directly to tape no disk.  My environment is similar to the other folks. Things should be flying...

NetBackup 7.1.0

Master\Media Windows 2008R2 Dual 6 core Dell r710's with 98GB of Ram

4 gb Ethernet nics teamed

1 Dual port Qlogic2562 going to redundant switches

Media Server (Same as above)

10 HP Ultrium LTO5 drives using shared storage (Load Balanced) 

I've tried all different setting and I'm so far not able to find the right mix...(maybe a secret) lol

C:\Program Files\Veritas\NetBackup

NET_BUFFER_SZ = 1050624

NET_BUFFER_SZ_REST = 1050624

C:\Program Files\Veritas\NetBackup\db\config

NUMBER_DATA_BUFFERS = 64

SIZE_DATA_BUFFERS = 262144

SIZE_DATA_BUFFERS_DISK = 1048576

NUMBER_DATA_BUFFERS_RESTORE = 32

NUMBER_DATA_BUFFERS_DISK = 32

I have the windows client buffers set to 256

[jmanini]

HI AJ, when you say performance is good with no load what speeds are you seeing? MB per second? or KB per second

do you also have the below buffers configered? Not neccessarily the same numbers of course..

C:\Program Files\Veritas\NetBackup\db\config

NUMBER_DATA_BUFFERS = 64

SIZE_DATA_BUFFERS = 262144

SIZE_DATA_BUFFERS_DISK = 1048576

NUMBER_DATA_BUFFERS_RESTORE = 32

NUMBER_DATA_BUFFERS_DISK = 32

[Omar_Villa]

Buffer settings look good I will increase the NUMBER_DATA_BUFFERS to 128, will not harm, but need to understand that overall the OS's out there Windows is the worst one on I/O a Media Server should never be in a Windows box because normaly windows devices drivers like NIC's and HBA's colapse when you raise the block size to more than 64KB, what this means upgrade your NIC and HBA drivers, check that your Cards support those block sizes, there are some HBA's like Emulex L10000's that split every block bigger than 64KB in multiple blocks like a 256KB block, will be split in 4 X 64KB blocks, upgrading the driver fix this, who know what else under windows fails like this, but worth to check.

Regards.