cancel
Showing results for 
Search instead for 
Did you mean: 

Huge buffer delays

Georges_N
Level 4

Hello

 

I've been reading up on a few similar cases like https://www-secure.symantec.com/connect/forums/slow-backups-after-upgrading-7503 and https://www-secure.symantec.com/connect/forums/netbackup-data-consumer-waiting-full-buffer-delayed-2186355-times and many others but there doesn't seem to be a solid solution from Symantec.

For us, it started happening shortly after installing new Media appliances (Symantec 5230)

We're running:

Symantec Netbackup 7.5.0.7 on Windows Server 2008 R2 x64

Symantec Appliance 5230 2.5.4 with 4GBIT LAN and LTO5 drivers connected via fiber

 

Backup jobs are not THAT affected:

 Info bptm(pid=14572) waited for full buffer 318 times, delayed 505 times  

Info bptm(pid=5678) waited for full buffer 2388 times, delayed 3075 times

 

It's really when you get into duplication to tape that it gets incredibly slow:

Info bptm(pid=16544) waited for full buffer 17723 times, delayed 51251 times 

Info bptm(pid=710) waited for full buffer 18813 times, delayed 57167 times

Info bptm(pid=18885) waited for full buffer 41759 times, delayed 139268 times 

 

Our buffer settings:

7/16/2014 2:00:29 AM - Info bptm(pid=32588) using 262144 data buffer size        
7/16/2014 2:00:29 AM - Info bptm(pid=32588) using 128 data buffers        

 

They used to be at 30 buffers, now with 128 I still get similar results... Here's a shot from the appliance settings:

16-07-14 11-47-07 AM.png

16-07-14 11-47-58 AM.png

 

Any other ideas that might help? I thought getting the appliances was a good idea but now... not so much!

 

Thanks

10 REPLIES 10

SymTerry
Level 6
Employee Accredited

Hello,

bptm waiting for full buffer. It delays till it gets enough data to be full and then emptys. You had 30 buffers which could not be filled fast enough and now incresed it to 128, so you have even more buffers now to be filled. What you will want to look at is slow read speed from the disk, or network issues between the servers.

Please review the NetBackup Backup Planning and Performance Tuning Guide Release 7.5 and Release 7.6: DOC7449

Georges_N
Level 4

Thanks, the disks are running fine (also the appliance is very new) and backups done to disk are good, even with the delays.

 

The problem is duplication to tape (to drives that are connected directly to the appliance via 8gbit fiber) - so not network-related really...

I've gone through the tuning process with no luck, that's why I came here ... to see if any other people have had better luck with the 5230 appliance before we change it!

 

Thanks

RiaanBadenhorst
Moderator
Moderator
Partner    VIP    Accredited Certified

Hi,

 

Please tell us more about the setup, how many appliances and what do the SLP's look like? Is the master still connected to the tape as well?

 

Have you tried backing up something large (test file not backup images) located on the appliance to tape? Try that on multiple appliances. If the master has direct access to tape try with the same files from the master too.

 

Are you sure you're not traversing the lan?

Georges_N
Level 4

hi Riaan

The setup is 1x Master (Gen7 from HP), 1x Media (Symantec 5230). 1x DR Media (Symantec 5230), 1x IBM TS3310 Library.

I have 2/3rds of the drives connected to the Media server, and 1/3rd connected to the master for immediate straight-to-tape backups/restores

We do pretty big backups just about every night, so I have plenty of examples that all seem to have:

7/17/2014 5:00:27 AM - Info bptm(pid=10445) using 262144 data buffer size        
    
7/17/2014 5:00:27 AM - Info bptm(pid=10445) using 128 data buffers 

7/17/2014 5:04:40 AM - Info bptm(pid=10445) waited for full buffer 186 times, delayed 13460 times 

 

Delayed 13460 times... the job itself was 5min 2sec long in total. This backup job was over the LAN.

For duplication to tape, jobs used to run between 160 and 240 GB/hour and now run between 60 and 80.

We still get:

7/17/2014 8:56:31 AM - Info bptm(pid=1086) waited for full buffer 26521 times, delayed 33578 times

7/17/2014 9:06:03 AM - Info bptm(pid=11887) waited for full buffer 24583 times, delayed 32333 times

etc...

 

RiaanBadenhorst
Moderator
Moderator
Partner    VIP    Accredited Certified

Hi Georges,

 

A delay is only 30 micro seconds. If you like you can determine where the issue is by taking a backup of some of you large files to

  1. Dedupe disk
  2. Adv disk
  3. Direct to tape

Then duplicate from adv to tape.

Just to get some figures. I had a client who's duplication were slow and it ended up being a fibre cable issue.

Have you rebooted the appliance recently?

Georges_N
Level 4

Yikes

So I ran a full file-server backup and I got this end result:

Info bptm(pid=1599) waited for full buffer 16173 times, delayed 3103985 times  

I'm waiting for a few things to finish to reboot the media appliance...

I double-checked everything - databuffers set to 128 buffers and 256k all around (except restore @ 256 buffers)

 

RiaanBadenhorst
Moderator
Moderator
Partner    VIP    Accredited Certified

Yowser, that must be a record. Try the reboot and the tests i suggested if you like (dont have to backup the entire appliance, just put a large file on its / somewhere.

Mark_Solutions
Level 6
Partner Accredited Certified

Have the tapes been used by previosu media servers or are they new?

In the jobs do they actually say they are using that buffer size?

In the All Log Entries what throughput rate do they say they are running at?

If possible go to 7.6.0.2 .. you get much better de-dupe performance and can also schedule your duplications so that they do not clash with backups which improves performance further

Georges_N
Level 4

Thanks guys, I've rebooted the whole environment and done some more tweaking, I still get similar results. I get the feeling it might be the tapes - I'll be renewing many of them to see if that makes a difference.

It does get a little ridiculous at times...

7/30/2014 5:14:15 AM - Info bptm(pid=20011) waited for full buffer 323527 times, delayed 500136 times

I just found something odd: it says the buffer used was 64k even though it's 128k everywhere (double checked the appliance, master server, everything is 128k)

 

7/31/2014 11:53:05 AM - Info bptm(pid=10123) using 65536 data buffer size       
7/31/2014 11:53:05 AM - Info bptm(pid=10123) using 128 data buffers       

Is this because it's an old tape?

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

I am confused...

Your opening post shows:

Our buffer settings:

7/16/2014 2:00:29 AM - Info bptm(pid=32588) using 262144 data buffer size        
7/16/2014 2:00:29 AM - Info bptm(pid=32588) using 128 data buffers        

Where are you seeing the 65536 data buffer size ? For which media server?

Please bear in mind that buffer settings is used on the particular media server/appliance doing the backup/duplication.

We were told that duplication from dedupe to tape was dramatically improved in NBU 7.6.x and Appliance 2.6.x.
I am battling to find documentation at the moment... will post here when I find it...