cancel
Showing results for 
Search instead for 
Did you mean: 

job through put lower than expected

Mark_Henrie_2
Level 4
It seems we can only get 16mb/sec on any of our jobs. I have check all our fiber connections and it seems to be a netbackup bottleneck.

We are running NBU 6.0 MP3

Windows master server
Unix Media Server
Unix Clients

Are there any settings In NBU that I should be looking for?

Sorry for being so vague I am just starting at a new job and am not yet familiar with all the systems.

Thanks in advance
28 REPLIES 28

h_m
Level 6
2 streams of SQL dumps, through fibre, running @ 45MB/sec each, through a W2K3 media ... equating to 90MB/sec, to one LTO3. So it can be done. Yet to try with 3 streams...

Mark_Henrie_2
Level 4
We are even seeing 12/gb with no multiplexing. But if we multiplex we will see 12/mb on each drive. So its not the EMC.

We have 2 drives off each fiber connection.

sdo
Moderator
Moderator
Partner    VIP    Certified
Hi Mark,

Can you confirm whether the following are defined on the media server?
SIZE_DATA_BUFFERS
NUMBER_DATA_BUFFERS
NET_BUFFER_SZ
...and if they are, then what the values are.

If they are not defined, then enable level 5 logging for bptm on the media server, and then run a test backup and then search the bptm logs for:
"data buffer size" tells you what your SIZE_DATA_BUFFERS is...
"data buffers" and "io_init:" tells you what your NUMBER_DATA_BUFFERS is...
"receive network buffer is" tells you what your NET_BUFFER_SZ is...

I know that LTO1 and LTO2 drives have a hardware 256KB buffer. NetBackup defaults to sending data to the tape drive in 64KB chunks to maintain default compatibility with older DLT tapes and older SCSI HBAs. I strongly suggest that you look at increasing your SIZE_DATA_BUFFERS to 256KB, i.e. 262144.

For modern installations I would also set the client communication send buffer size to 256KB, and also the media server receive buffer size (i.e. NET_BUFFER_SZ). This way, you would have 256KB buffers all the way from the client through the media server through to the tape drive.

The whole toipic is rather too much to repeat here, and many web pages etc refer to such things. I really recommend that you look for an article by George Winter, re NetBackup Performance Tuning:
http://eval.veritas.com/downloads/van/4019.pdf

As for my points above, they may be enough to help you. Other people/pages/blogs report a quadruple performance jump from 16MB/s (same as your speed) to 60MB/s+ when using the above.

You'll have to research what the best setting for NUMBER_DATA_BUFFERS is, as this eats memory. The memory consumption is calculated as:
mem used = (buffer_size * num_buffers) * drives * MPX
i.e:
MEM_USAGE = SIZE_DATA_BUFFERS * NUMBER_DATA_BUFFERS * DRIVES * MPX

The only time that you won't want to make any changes to buffering is if your DR site/machine will be using direct attached SCSI tape drives. All modern FC HBAs, switches and LTO tape drives work best with 256KB buffers - BUT you will *NOT* be able to read the tapes on some low-end SCSI HBAs on Windows (and maybe Unix too). (Actually you can, even the Adaptec 29160 can be made to read/write 256KB with a simple and supported registry change), but, as ever, backward compatibility demands that NetBackup ships with a default 64KB, and that this is actually a definite performance hindrance for mdern systems.

If your DR server is FC too, then you have no excuses for not using 256KB data buffers.

Here's an HP article talking about LTO3 drives and NetBackup...
http://h71028.www7.hp.com/ERC/downloads/5982-9971EN.pdf
Check from page 37 to 41.

As ever, always test backup and retore after making these changes. And test that you can restore from backups before the change - and always have a backout plan.

One last thing, make sure you haven't enabled compression on the policy. This is only useful in some DSU scenarios, and should never ever be enabled for writing to LTO tape drives, as all LTO drives implement hardware compression by default.

Regards,
Dave.

zippy
Level 6
Nice library, I want one :)

Mark_Henrie_2
Level 4
Very GOOD information there...

I have tested my buffer sizes and still all i can get is a tops of 20MB

The drivers that are used in our media servers are a part of the sles9 kernel. They should be all up to date.

VNR_VNR
Level 3
The (Unauthorized) Lan Throughput Netbackup Decaloge:( joke mode on )

1. Until the Departament of backup and Departament of restore will not be the same, don't use Multiplexing, you'll be the fast backup-er but the fired restorer.

2. Be very good friend of your Network Administrator, if not, you'll have mounts of problems and mounts of delays solving these problems. Firewall CPU Bottlenecks, Ports & card modes, and speed negotiation are the most common problems.

3. Check the antivirus and exclude the directories of netbackup soft everywhere ( Masters, Medias, Clients, Agents )

4. VSP, VSS, OSD and all the stuff are speed-eaters, use them only for online SO aplications, agents do the rest. Open file window local user apps always run on night backup windows.

5. Assign calculated Buffer sizes to your Buffer parameters ( NET_BUFFER_SZ, ... ), there's a lot of information docs at symantec web.

6. Disks RPMs are important, like Storage SAN solutions. Use LAN for UNIX SO local drives, and SAN for aplication drives.

7. Encrypted mode absorbs infinite CPU's and murders your client links, CPU graphs seem like an Urano-one-step-stair.

8. Use an only-backup-purposed LAN, configure correctly routes and gateways ( beware with the Service Lanes )

9. You have to know the backup arquitecture like the way home.

10. Trace, study your statistics, tunn your debug levels, and have a good time with it.

sdo
Moderator
Moderator
Partner    VIP    Certified
Hi Mark,
Did you get anywhere with analyzing the parent and child delay numbers? Did this indicate the source of any bottleneck, i.e. either with incoming (from client to media server) or outgoing (from media server to SAN)?
Regards,
Dave.

The_Director
Level 5
Certified
Well, here is the latest....

We are still only seeing 16-20MB/sec throughput. After a week with a symantec consultant, the only thing we could contribute the performance issues to is not enough ram.

Initially the media servers were ordered with 16GB ram and 4 - 3.0 dual core procs. Well dell screwed up and no one caught it so we ended up with 2 dual core 3.0 procs. and only 4GB ram. We are now waiting for 80,000 worth of procs and ram to get us up to 4 procs and 48GB ram. Hopefully this will fix the problem. This was the solution the symantec consultant gave us. If anything when we get this hardware in we can cross that out and see what else it could be.

All our settings in NetBackup seem to be optimized.

I'll let you know what happens. We should be up and running in a week.

BTW, its nice to see so many links to this thread I hope it helps everyone throttle their performance.

Later.

Chuck_Stevens
Level 6
Hmm. A part of me says that's not right. You shouldn't need that much. It'll be interesting to see if it actually affects the problem.