Netbackup perfomance troubleshooting question

[Sergei_Khloupno]

I'm trying to get reasonable speed using LTO-3 drive and GigE network link between two hosts.

- iperf measured network throughput is 110 MB/sec
- client disk reads 85 MB/sec
- scp transfer rate from client to server is about 27 MB/sec
- netbackup over GigE from client to the server disk 32 MB/sec (server's disk is not fast - no raid or anything)

- netbackup from the client to server's LTO-3 drive is 12 MB/sec
- I couldn't get bpdm_dev_null test working (from Veritas's "Backup  Planning and Perfomance" pdf)

I've read Veritas perfomance document and various web pages about tuning. Have all buffers at 256k. Server is Netbackup 6 Linux Centos 3.8 with 2x2 core Xeon 3.06 Ghz, 2 GB memory with LSI 20320 HBA.

I don't see what could be a bottleneck here. Even backup to the server disk gets about 30 MB/sec. Why writing to the seemingly FAST LTO-3 is so slow ? Anything else I can try to determine what slows me down ?

thanks

Message Edited by Sergei Khloupnov on 04-05-200704:46 PM

[Rakesh_Khandelw]

Are your tape and GigE sharing the same PCI bus on server?

Have you tried NET_BUFFER_SZ? what's your NUMBER_DATA_BUFFERS value?

[Sergei_Khloupno]

Good idea. I'll have to check whether its on the same PCI bus and what kind of server will let me put it on different buses.

echo "262144" > /usr/openv/netbackup/NET_BUFFER_SZ
echo "262144" > /usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS
echo "16" > /usr/openv/netbackup/db/config/NUMBER_DATA_BUFFERS

[Rakesh_Khandelw]

NET_BUFFER_SZ should be set to same value on media server and client.

TCP/IP Network Buffer Size.

/usr/openv/netbackup/NET_BUFFER_SZ is a file containing a number indicating the TCP/IP socket buffer size that should be used for data transfers between the NetBackup media server and its clients. If the file does not exist, the default value used is 32032 bytes.

The default value can be changed by creating the file /usr/openv/netbackup/NET_BUFFER_SZ on the NetBackup media server and client systems.

The /usr/openv/netbackup/NET_BUFFER_SZ file is a text file containing a single line specifying the value of the Network Buffer Size in bytes in decimal format.

For example, a Network Buffer Size of 64KB could be configured and checked like this:

# echo "65536" > /usr/openv/netbackup/NET_BUFFER_SZ
# cat /usr/openv/netbackup/NET_BUFFER_SZ
65536
#

Specifying a larger value in the NET_BUFFER_SZ file may improve the performance of backups and restores. It is recommended that if NET_BUFFER_SZ is used, the same value should be set on all the NetBackup media servers and clients.

[Stumpr2]

Sergei Khloupnov wrote:

I'm trying to get reasonable speed using LTO-3 drive and GigE network link between two hosts.

- scp transfer rate from client to server is about 27 MB/sec

You should at least be able to match scp. I didn't see the platform you are using, however even this technote says "AIX" it has some good stuff that applies to other platforms

Slow backup performance of VERITAS NetBackup (tm) 4.5 AIX clients running AIX 4.3.3, 5.1, and 5.2

http://support.veritas.com/docs/265527

GENERAL ERROR: VERITAS NetBackup (tm) is unable to set the send and/or receive network buffer size to more than 65535 bytes, regardless of the value set in the NET_BUFFER_SZ file.

http://support.veritas.com/docs/262759

[Sergei_Khloupno]

It x86 Linux (Xeon, 2 CPUs, 2 cores each, recommended U320 PCI HBA). One of those papers says that tar is good enough to test how fast localhost can write to the tape. So there seem to be a problem. I'm getting about 12 MB/sec if I tar local 500 MB file to /dev/st0. Same tar to the disk file is about 40 MB/sec.

Guess I need more details as to what that server is made of (other than the CPU speed). An alternative would be try it on a Sun hardware but fastest we have here is some dual 440 mhz E420. Not sure if its enough for this.

[Sergei_Khloupno]

I was able to get tar write to the tape at 32 MB/sec by setting st driver block size to 256kb and disabling "domain verification" in mptscsih HBA driver. Still have to figure out a way to automate this. So far /etc/modules.conf was not doing what I want. Doing it manually with

/sbin/rmmod mptscsih
/sbin/insmod mptscsih mpt_dv=0

/sbin/rmmod st
/sbin/insmod st buffer_kbs=512

Just finished Netbackup over GigE : 4323360 Kbytes at 55763.704 Kbytes/sec Wow. Thanks all

[Stumpr2]

Great news! But please remember that any backup is only as good as its restore. Make sure you can get a good restore.