My weekend back up is running very very slow

[kakaluru]

Hello Marianne

After resolving catalog issues etc.I don't know the reasons my weekend back up is running very very slow.

Let  me know which log files you need to check this.

**** Moved to new discussion from Catlogback up Delay ****

[#2]

Sounds like a separate issue to me.  Re-post as a new question perhaps?

[Marianne]

Can we assume that your entire backup infrastructure is as old and unsupported as your NBU 5.1 software?

When last did you power-cycle the tape library and the NBU master/media server?
Have you spoken with network admins to find out if there was an issue over the weekend?
Were backups slow for all clients or just some?

bptm log for the date of slow backups will be a good start.

[kakaluru]

I understood but we never faced this issue earlier.

I asked network team there was no issue

We are managing one client. Server and client are in same network. I tried to run manually but its running on b/w 800 to 3000kbs.

Media IDs are 0245L3,0031L3,0174L3

[Marianne · Accepted solution]

I understood but we never faced this issue earlier.

Well, there is no way that NetBackup can be at fault since the SAME binaries have been in place for way too many years!

This is Windows... when last did you reboot it?

What is the Windows version and patch level?
Any chance that Windows updates were installed recently? 

On  which date did the slow backups start?
I do not have the time to go through all the logs...

PS:
The files are quite small. In future, PLEASE copy files to .txt (e.g. bptm042415.txt) and upload as File attachments.
.txt files can be viewed online. zipfiles need to be downloaded and extracted. .log files cannot be viewed on a mobile device.

[Marianne]

I have just had a look at 2 of the logs.
The only issue in these logs is status 150.

There is nothing to look at, nothing to troubleshoot if you kill backups.

Logging level is at 0.

You need logging to be at least at level 3 to contain anything useful....

[kakaluru]

Last Friday onwards it was listening very very slow. After that i have rebooted and checked but no use.

ws server 2003 R2 Enterprise Edition service pack 2. What are the patches you want me to tell.

Recently we haven't done any updates

Find log files for 000028 id which was successfully completed but it was earlier listening time 20mts .Recently it took 5.30 hrs

[kakaluru]

one more attachment
 

[Marianne · Accepted solution]

What are the patches you want me to tell. 

Nothing.

I was asking you to look for any changes outside of NBU. We know that the ONE THING that has NOT changed is NBU. 

Looking at the log file, we see the following:

04:19:55.586 [2796.884] <2> write_data: waited for full buffer 36317 times, delayed 619034 times

This means that NBU (bptm buffer) was waiting for data. 
The problem is therefore outside of NBU. 
You need to look at the other components in the data path:
Client disk -> Client NIC -> network -> Media server NIC -> bptm buffer -> tape drive

We have in the past seen that Windows updates has enabled TCP Chimney, which resulted in socket errors or just very slow throughput.

Check TCP Chimney settings on master/media server and client(s):  http://www.symantec.com/docs/TECH60844 

[kakaluru]

Hello marianne

where to check this path

Client disk -> Client NIC -> network -> Media server NIC -> bptm buffer -> tape drive

Check TCP Chimney settings what exactly it is

[jim_dalton]

Marianne is saying that theres a bottleneck and that you need to check that data path to determine whereabout the bottleneck is. Its something you and your colleagues need to investigate, jim

[sdo]

TCP Chimney disable usually applies only to W2003.

Symantec recommends to leave it enabled for W2008:

See http://www.symantec.com/docs/TECH145791

Technotes, such as 294308 and 29008 it was recommend to disable TCP Chimney Offload. For Windows 2008 R2 platforms, performance gains have been realized when TCP Chimney Offload is enabled.

...however, I have seen sites where the Windows and Network teams decided to 'disable' on al of their Windows 2008 builds too.

.

How to disable:

netsh interface tcp show global
netsh interface tcp set global chimney=disabled
netsh interface tcp show global

[Marianne]

From previous posts we know this user is still using NBU 5.1.
So, it can only be Windows 2003!

[Marianne · Accepted solution]

Speak to your OS and network administrators - they should be able to explain to you the data flow that is involved in a backup.

As far as TCP Chimney is concerned - please read the TN. 
Or ask the OS Admin to help.

More info about general troubleshooting for slow backups:

Overview of NetBackup performance testing. 
http://www.symantec.com/docs/TECH147296 
(This TN contains a diagram that explains the data path)

This doc is for newer NBU versions, but the principals remain the same: 

Symantec NetBackup™ 7.0 - 7.1 Backup Planning and Performance Tuning Guide 
http://www.symantec.com/docs/DOC4483 
(Click on the 'Attachments' link.)

[kakaluru]

Hello all

After observing 1 week and  restarting of client server my back up is running as usual writing speed.

Let me know the reasons please and which log files you are required to troubleshott?

[Marianne · Accepted solution]

Seeing that the problem is fixed by restarting the client, you need to have a look at Windows logs on the client.

Start with Windows Event Viewer System and Application logs.

Since nothing at NBU level has changed, the components that are refreshed by a reboot can be confirmed with Windows sysadmin.

Just some of the components that comes to mind:

Memory & cpu
Path from OS to disk
Disk drivers
Network card
etc...

There is nothing at NBU level to troubleshoot.
bpbkar logs on the client before and after reboot will simply confirm that read speed was slow before the reboot and faster after the reboot.
NBU can only report the transfer rate, not what is causing slow or fast transfer.
 

[kakaluru]

It might be client side hardware issue. Thank you for all your support and patience.

Can i close to this by mark as solution or what ?

[Marianne]

You can close the discussion by marking a helpful post as Solution.

If more than one post has helped you, you can click on 'Request split solution' and select all helpful posts.

[kakaluru]

marianne, I could not find option to marking this as a helpful post as solution.

[Marianne]

I don't understand what you are saying. There is a 'Mark as solution ' and 'Request split solution ' link at the bottom of each post. You then look at all the posts and attempts to help you, decide which post has helped you most, then mark that post as Solution. If more than one post has helped you, select 'Request split solution ' and then click on all posts that have helped you.