03-22-2011 05:07 PM
Solved! Go to Solution.
03-25-2011 07:34 AM
In the NB7.1 release notes on page 74:
"Do not use a LIMIT_BANDWIDTH (IPv4) configuration or a
THROTTLE_BANDWIDTH (IPv6) configuration in the initial release of
NetBackup 7.1. If you use either of these configuration options, you can
encounter a sever performance degradation.
An EEB will be available to fix the issues."
Damn.
03-22-2011 09:48 PM
This statement surely excludes 7.1 as the cause:
"The same poor filesystem backup performance can be observed when writing to a 7.1 media server as well as to a 6.5.5 media server"
We assume that 6.5 media servers are only backing up 6.5 clients?
03-22-2011 11:56 PM
Hello Marianne,
You are right, it is probably not the NetBackup 7.1 media servers causing the behavior as writing to a 6.5 media server storag unit is also slow.
And yes we tested with a 6.5 client when writing to a 6.5 media server.
Could it be the 7.1 master? I suspected it could be something about writing the meta data into the image catalog. But then all other backup types should be slow as well, shouldn't they?
Thomas
03-23-2011 01:11 AM
Backup performance is determined by data transfer between bpbkar on client and bptm on media server. Check bptm log at the end of the backup for 'waited for full/empty buffers'. bpbrm on media server sends meta-data to bpdbm on master. Verbose logging of bpbrm should tell you how often metadata is sent.
I have found that a lot of users look at their backups much closer after an upgrade and only notice performance issues then. If you search this forum or Google, you will find performance complaints after EACH new NetBackup release...
Another thought - can we safely assume that filesystem backups are reading from different filesystems as database backups? Please do yourself a favour and perform all the 'usual' performance testing - use bpbkar on client to backup to /dev/null, and if read performance is acceptable, ftp a fairly large file on the same filesystem to the media server.
03-23-2011 03:11 AM
I have looked for the empty/full data buffers waits and delays and can influence those by tuning NET_BUFFER_SZ. However this doesn't seem to have any impact on the backup performance in my case. So I assume that something else is throttling the filesystem backups.
You are right about the filesystems. Oracle backups are reading data from SAN LUNs while all the regular filesystem backups get their data from internal disks.
One more thing to notice: performance drops even more, if more than one client is doing a standard filesystem backup. Backup rates drop to ridiculous 6K/sec.
And yes, I've checked the backup performance of standard policies that have run in the past. As I mentioned we have never ever seen rates below 10M/sec. It is not just a vague feeling. :)
We are now upgrading the remaining media servers to 7.1. I heard about similar cases opened with Symantec support with no definite solution. But it seems that the problems vanished into thin air after all the media servers had been upgraded.
I will keep you updated.
Cheers,
Thomas
03-23-2011 03:36 AM
What are you seeing in bptm? Waiting for full or waiting for empty buffers? Tweaking buffer sizes will only help if bptm is waiting for empty buffers (i.e. data is received fast enough but cannot be written away quick enough). If bptm is waiting for full buffers, the data is simply not getting to bptm fast enough.
Please do the bpbkar test to backup a filesystem to /dev/null. See this TN for Windows test procedure: http://www.symantec.com/docs/TECH17541 or else Planning and Performance Guide http://www.symantec.com/docs/TECH62317 and if acceptable read speed is seen, ftp to media server.
03-23-2011 03:40 AM
Hi,
I think this is not a problem of NetBackup. Performance tunning parameters may not be of any help in this. Problem may be with file system you are backing up.
I have seen File System backup with more than 50 MBPS in NBU 65...
Check the file system, how many files are there? what is the file size (average)?
If you are backing up millions of small files, then you will never get performance.
Checkout for fragmentation on file system, get it defraged, this can also help increasing the backup speed.
Checkout for compression/encryption is enabled on file system....
Check out other methods of backing up file system like raw disk backup - snapshot client - flash backup may be a good option.
Deepak
03-23-2011 04:15 AM
Please keep in mind that we never had filesystem performance problems before upgrading to 7.1.
We are seeing performce problems for all(!) clients and filesystems when doing standard backups.
For testing purposes we were backing up "/usr" which I think is rather not a candidate for flashbackup.
Marianne, I have not tested bpbkar throughput locally (/dev/null) as all the clients are showing the same weak performance. If the problem still persists after upgrading all media servers I will give it a try though.
Best regards,
Thomas
03-23-2011 09:40 PM
HI,
I upgraded today my 3 media server and my master server from 7.0.1 to 7.1, they are all running W2K8R2 X64.
I have the exact same problem that thran his having, all my backup are now running at around 150 KB/Sec (from 20K to 50K)
I made no other modification on my systems or my networks, it is affecting my 3 sites (in 3 different continent) and they are all using different LTO drives (LTO 2-4 and 5) with different interface (SCSI, SAS, FC)
Something his going on here with this update... I will call technical support first thing tomorrow morning and I will follow-up if they have any idea what is going on.
03-24-2011 02:07 AM
Hello therockyb,
if you open up a case with Symantec would you mind passing me the case-id? I would give it to the case owner working on my incident and try to get those two cases connected. This might speed up the progress on this case.
If you'd rather send me the case-id by PM, please send it to trahn[at]@anykey[dot]de.
BTW: Upgrading all Media Servers to 7.1 didn't help.
Cheers,
Thomas
03-24-2011 08:31 AM
Thomas,
I have opened a case more than 10 hours ago and I still can't get technical support to call me back... apparently 7.1 generated a storm of call for support.
I will send you my case number in PM.
03-24-2011 09:32 AM
It seems that I've found the cause, probably a bug. I need to check a little bit further before I post it here. Still possible that I am mistaken. I will keep you updated. Regards, Thomas
03-24-2011 01:35 PM
Hi Thomas,
Any development in your investiguation?
Today I tried a different mix of client, Windows, Linux 6.5, 7.0, 7.0.1, 7.1
I have the same problem with every client version.
03-24-2011 04:06 PM
I'm having performance problem with 7.0 version using FlashBackup Windows of VMWare 4.1.
I getting 30 to 40 MB/sec using FC and this error in bpbkar and bptm .. I'm using a LTO 4 of HP FC (MSL8048 )
BPTM
11:48:56.282 [6004.4904] <2> write_data: waited for full buffer 174 times, delayed 10959 times
BPBKAR
11:48:56.266 AM: [5884.5472] <4> BufferManagerLegacySharedMemory::~BufferManagerLegacySharedMemory(): INF - bpbkar waited 1324 times for empty buffer, delayed 1559 times.
It is a problem or a tuning?
03-25-2011 05:01 AM
Hi therockyb,
please check if you've got "Bandwidth" or "Throttle Bandwidth" selections set in "Host Properties -> Master Server".
These are two separated entries in the master server properties and result in two different options in bp.conf or the Windows registry.
1.) LIMIT_BANDWIDTH (for IP ranges)
2.) THROTTLE_BANDWIDTH (for hosts and networks)
Not sure if THROTTLE_BANDWIDHT existed prior to 7.1.
If LIMIT_BANDWIDTH is used to throttle bandwidth for a single host
e.g.
LIMIT_BANDWIDTH = 192.168.235.107 192.168.235.107 100
try to use
THROTTLE_BANDWIDTH = 192.168.235.107 100
Either use the NetBackup Admin Console or the Windows registry. I was testing on Linux though.
Do a
bprdreq -rereadconfig
and rerun a policy to check wether the perfomance problems persist.
Please let me know if this helped.
Regards,
Thomas
03-25-2011 07:34 AM
In the NB7.1 release notes on page 74:
"Do not use a LIMIT_BANDWIDTH (IPv4) configuration or a
THROTTLE_BANDWIDTH (IPv6) configuration in the initial release of
NetBackup 7.1. If you use either of these configuration options, you can
encounter a sever performance degradation.
An EEB will be available to fix the issues."
Damn.
03-25-2011 08:11 AM
Thomas,
This is awesome that you found the problem… thank you so much !
You were right, I had two entries in my LIMIT_BANDWITH configuration, I removed them and everything is back to normal.
You have no idea all the problems you just saved me… I really appreciate your help !!!
Yan
03-25-2011 08:51 AM
Yan,
glad it worked for you, too.
I just got the reply from support that an eeb is not yet available.
Regards,
Thomas
03-25-2011 01:39 PM
I have marked your post as Solution - thanks for the feedback!