07-16-2014 08:55 AM
Hello
I've been reading up on a few similar cases like https://www-secure.symantec.com/connect/forums/slow-backups-after-upgrading-7503 and https://www-secure.symantec.com/connect/forums/netbackup-data-consumer-waiting-full-buffer-delayed-2186355-times and many others but there doesn't seem to be a solid solution from Symantec.
For us, it started happening shortly after installing new Media appliances (Symantec 5230)
We're running:
Symantec Netbackup 7.5.0.7 on Windows Server 2008 R2 x64
Symantec Appliance 5230 2.5.4 with 4GBIT LAN and LTO5 drivers connected via fiber
Backup jobs are not THAT affected:
Info bptm(pid=14572) waited for full buffer 318 times, delayed 505 times
Info bptm(pid=5678) waited for full buffer 2388 times, delayed 3075 times
It's really when you get into duplication to tape that it gets incredibly slow:
Info bptm(pid=16544) waited for full buffer 17723 times, delayed 51251 times
Info bptm(pid=710) waited for full buffer 18813 times, delayed 57167 times
Info bptm(pid=18885) waited for full buffer 41759 times, delayed 139268 times
Our buffer settings:
7/16/2014 2:00:29 AM - Info bptm(pid=32588) using 262144 data buffer size
7/16/2014 2:00:29 AM - Info bptm(pid=32588) using 128 data buffers
They used to be at 30 buffers, now with 128 I still get similar results... Here's a shot from the appliance settings:
Any other ideas that might help? I thought getting the appliances was a good idea but now... not so much!
Thanks
07-16-2014 10:41 AM
Hello,
bptm waiting for full buffer. It delays till it gets enough data to be full and then emptys. You had 30 buffers which could not be filled fast enough and now incresed it to 128, so you have even more buffers now to be filled. What you will want to look at is slow read speed from the disk, or network issues between the servers.
Please review the NetBackup Backup Planning and Performance Tuning Guide Release 7.5 and Release 7.6: DOC7449
07-16-2014 11:02 AM
Thanks, the disks are running fine (also the appliance is very new) and backups done to disk are good, even with the delays.
The problem is duplication to tape (to drives that are connected directly to the appliance via 8gbit fiber) - so not network-related really...
I've gone through the tuning process with no luck, that's why I came here ... to see if any other people have had better luck with the 5230 appliance before we change it!
Thanks
07-16-2014 11:30 AM
Hi,
Please tell us more about the setup, how many appliances and what do the SLP's look like? Is the master still connected to the tape as well?
Have you tried backing up something large (test file not backup images) located on the appliance to tape? Try that on multiple appliances. If the master has direct access to tape try with the same files from the master too.
Are you sure you're not traversing the lan?
07-17-2014 06:07 AM
hi Riaan
The setup is 1x Master (Gen7 from HP), 1x Media (Symantec 5230). 1x DR Media (Symantec 5230), 1x IBM TS3310 Library.
I have 2/3rds of the drives connected to the Media server, and 1/3rd connected to the master for immediate straight-to-tape backups/restores
We do pretty big backups just about every night, so I have plenty of examples that all seem to have:
7/17/2014 5:00:27 AM - Info bptm(pid=10445) using 262144 data buffer size
7/17/2014 5:00:27 AM - Info bptm(pid=10445) using 128 data buffers
7/17/2014 5:04:40 AM - Info bptm(pid=10445) waited for full buffer 186 times, delayed 13460 times
Delayed 13460 times... the job itself was 5min 2sec long in total. This backup job was over the LAN.
For duplication to tape, jobs used to run between 160 and 240 GB/hour and now run between 60 and 80.
We still get:
7/17/2014 8:56:31 AM - Info bptm(pid=1086) waited for full buffer 26521 times, delayed 33578 times
7/17/2014 9:06:03 AM - Info bptm(pid=11887) waited for full buffer 24583 times, delayed 32333 times
etc...
07-17-2014 10:53 PM
Hi Georges,
A delay is only 30 micro seconds. If you like you can determine where the issue is by taking a backup of some of you large files to
Then duplicate from adv to tape.
Just to get some figures. I had a client who's duplication were slow and it ended up being a fibre cable issue.
Have you rebooted the appliance recently?
07-18-2014 06:57 AM
Yikes
So I ran a full file-server backup and I got this end result:
Info bptm(pid=1599) waited for full buffer 16173 times, delayed 3103985 times
I'm waiting for a few things to finish to reboot the media appliance...
I double-checked everything - databuffers set to 128 buffers and 256k all around (except restore @ 256 buffers)
07-18-2014 07:02 AM
Yowser, that must be a record. Try the reboot and the tests i suggested if you like (dont have to backup the entire appliance, just put a large file on its / somewhere.
07-18-2014 08:01 AM
Have the tapes been used by previosu media servers or are they new?
In the jobs do they actually say they are using that buffer size?
In the All Log Entries what throughput rate do they say they are running at?
If possible go to 7.6.0.2 .. you get much better de-dupe performance and can also schedule your duplications so that they do not clash with backups which improves performance further
07-31-2014 08:58 AM
Thanks guys, I've rebooted the whole environment and done some more tweaking, I still get similar results. I get the feeling it might be the tapes - I'll be renewing many of them to see if that makes a difference.
It does get a little ridiculous at times...
7/30/2014 5:14:15 AM - Info bptm(pid=20011) waited for full buffer 323527 times, delayed 500136 times
I just found something odd: it says the buffer used was 64k even though it's 128k everywhere (double checked the appliance, master server, everything is 128k)
7/31/2014 11:53:05 AM - Info bptm(pid=10123) using 65536 data buffer size
7/31/2014 11:53:05 AM - Info bptm(pid=10123) using 128 data buffers
Is this because it's an old tape?
08-04-2014 04:56 AM
I am confused...
Your opening post shows:
Our buffer settings:
7/16/2014 2:00:29 AM - Info bptm(pid=32588) using 262144 data buffer size
7/16/2014 2:00:29 AM - Info bptm(pid=32588) using 128 data buffers
Where are you seeing the 65536 data buffer size ? For which media server?
Please bear in mind that buffer settings is used on the particular media server/appliance doing the backup/duplication.
We were told that duplication from dedupe to tape was dramatically improved in NBU 7.6.x and Appliance 2.6.x.
I am battling to find documentation at the moment... will post here when I find it...