Slow Filesystem Backup Performance after Upgrading to 7.1

This statement surely excludes 7.1 as the cause:

"The same poor filesystem backup performance can be observed when writing to a  7.1 media server as well as to a 6.5.5 media server"

We assume that 6.5 media servers are only backing up 6.5 clients?

Hello Marianne,

You are right, it is probably not the NetBackup 7.1 media servers causing the behavior as writing to a 6.5 media server storag unit is also slow.

And yes we tested with a 6.5 client when writing to a 6.5 media server.

Could it be the 7.1 master? I suspected it could be something about writing the meta data into the image catalog. But then all other backup types should be slow as well, shouldn't they?

Thomas

Backup performance is determined by data transfer between bpbkar on client and bptm on media server. Check bptm log at the end of the backup for 'waited for full/empty buffers'. bpbrm on media server sends meta-data to bpdbm on master. Verbose logging of bpbrm should tell you how often metadata is sent.

I have found that a lot of users look at their backups much closer after an upgrade and only notice performance issues then. If you search this forum or Google, you will find performance complaints after EACH new NetBackup release...

Another thought - can we safely assume that filesystem backups are reading from different filesystems as database backups? Please do yourself a favour and perform all the 'usual' performance testing - use bpbkar on client to backup to /dev/null, and if read performance is acceptable, ftp a fairly large file on the same filesystem to the media server.

I have looked for the empty/full data buffers waits and delays and can influence those by tuning NET_BUFFER_SZ. However this doesn't seem to have any impact on the backup performance in my case. So I assume that something else is throttling the filesystem backups.

You are right about the filesystems. Oracle backups are reading data from SAN LUNs while all the regular filesystem backups get their data from internal disks.

One more thing to notice: performance drops even more, if more than one client is doing a standard filesystem backup. Backup rates drop to ridiculous 6K/sec.

And yes, I've checked the backup performance of standard policies that have run in the past. As I mentioned we have never ever seen rates below 10M/sec. It is not just a vague feeling. :)

We are now upgrading the remaining media servers to 7.1. I heard about similar cases opened with Symantec support with no definite solution. But it seems that the problems vanished into thin air after all the media servers had been upgraded.

I will keep you updated.

Cheers,

Thomas

What are you seeing in bptm? Waiting for full or waiting for empty buffers? Tweaking buffer sizes will only help if bptm is waiting for empty buffers (i.e. data is received fast enough but cannot be written away quick enough). If bptm is waiting for full buffers, the data is simply not getting to bptm fast enough.

Please do the bpbkar test to backup a filesystem to /dev/null. See this TN for Windows test procedure: http://www.symantec.com/docs/TECH17541 or else Planning and Performance Guide  http://www.symantec.com/docs/TECH62317  and if acceptable read speed is seen, ftp to media server.

Hi,

I think this is not a problem of NetBackup. Performance tunning parameters may not be of any help in this. Problem may be with file system you are backing up.

I have seen File System backup with more than 50 MBPS in NBU 65...

Check the file system, how many files are there? what is the file size (average)?

If you are backing up millions of small files, then you will never get performance.

Checkout for fragmentation on file system, get it defraged, this can also help increasing the backup speed.

Checkout for compression/encryption is enabled on file system....

Check out other methods of backing up file system like raw disk backup - snapshot client - flash backup may be a good option.

Deepak

Please keep in mind that we never had filesystem performance problems before upgrading to 7.1.

We are seeing performce problems for all(!) clients and filesystems when doing standard backups.

For testing purposes we were backing up "/usr" which I think is rather not a candidate for flashbackup.

Marianne, I have not tested bpbkar throughput locally (/dev/null) as all the clients are showing the same weak performance. If the problem still persists after upgrading all media servers I will give it a try though.

Best regards,

Thomas

HI,

I upgraded today my 3 media server and my master server from 7.0.1 to 7.1, they are all running W2K8R2 X64.

I have the exact same problem that thran his having, all my backup are now running at around 150 KB/Sec (from 20K to 50K)

I made no other modification on my systems or my networks, it is affecting my 3 sites (in 3 different continent) and they are all using different LTO drives (LTO 2-4 and 5) with different interface (SCSI, SAS, FC)

Something his going on here with this update... I will call technical support first thing tomorrow morning and I will follow-up if they have any idea what is going on.

Hello therockyb,

if you open up a case with Symantec would you mind passing me the case-id? I would give it to the case owner working on my incident and try to get those two cases connected. This might speed up the progress on this case.

If you'd rather send me the case-id by PM, please send it to trahn[at]@anykey[dot]de.

BTW: Upgrading all Media Servers to 7.1 didn't help.

Cheers,

Thomas

Thomas,

I have opened a case more than 10 hours ago and I still can't get technical support to call me back... apparently 7.1 generated a storm of call for support.

I will send you my case number in PM.