Forum Discussion

kakaluru's avatar
kakaluru
Level 4
10 years ago

My weekend back up is running very very slow

Hello Marianne

After resolving catalog issues etc.I don't know the reasons my weekend back up is running very very slow.

Let  me know which log files you need to check this.

 

**** Moved to new discussion from Catlogback up Delay ****

7.1.x and Earlier
Backup and Recovery
NetBackup
Windows Server (2003-2008)
  • Marianne's avatar
    Marianne
    10 years ago

    I understood but we never faced this issue earlier.

    Well, there is no way that NetBackup can be at fault since the SAME binaries have been in place for way too many years!

    This is Windows... when last did you reboot it?

    What is the Windows version and patch level?
    Any chance that Windows updates were installed recently? 

    On  which date did the slow backups start?
    I do not have the time to go through all the logs...

    PS:
    The files are quite small. In future, PLEASE copy files to .txt (e.g. bptm042415.txt) and upload as File attachments.
    .txt files can be viewed online. zipfiles need to be downloaded and extracted. .log files cannot be viewed on a mobile device.

  • Marianne's avatar
    Marianne
    10 years ago

    What are the patches you want me to tell. 

     

    Nothing.

    I was asking you to look for any changes outside of NBU. We know that the ONE THING that has NOT changed is NBU. 

    Looking at the log file, we see the following:

    04:19:55.586 [2796.884] <2> write_data: waited for full buffer 36317 times, delayed 619034 times

    This means that NBU (bptm buffer) was waiting for data. 
    The problem is therefore outside of NBU. 
    You need to look at the other components in the data path:
    Client disk -> Client NIC -> network -> Media server NIC -> bptm buffer -> tape drive

    We have in the past seen that Windows updates has enabled TCP Chimney, which resulted in socket errors or just very slow throughput.

    Check TCP Chimney settings on master/media server and client(s):  http://www.symantec.com/docs/TECH60844 

  • Marianne's avatar
    Marianne
    10 years ago

    Speak to your OS and network administrators - they should be able to explain to you the data flow that is involved in a backup.

    As far as TCP Chimney is concerned - please read the TN. 
    Or ask the OS Admin to help.

    More info about general troubleshooting for slow backups:

    Overview of NetBackup performance testing. 
    http://www.symantec.com/docs/TECH147296 
    (This TN contains a diagram that explains the data path)

    This doc is for newer NBU versions, but the principals remain the same: 

    Symantec NetBackup™ 7.0 - 7.1 Backup Planning and Performance Tuning Guide 
    http://www.symantec.com/docs/DOC4483 
    (Click on the 'Attachments' link.)

  • Marianne's avatar
    Marianne
    10 years ago

    Seeing that the problem is fixed by restarting the client, you need to have a look at Windows logs on the client.

    Start with Windows Event Viewer System and Application logs.

    Since nothing at NBU level has changed, the components that are refreshed by a reboot can be confirmed with Windows sysadmin.

    Just some of the components that comes to mind:

    Memory & cpu
    Path from OS to disk
    Disk drivers
    Network card
    etc...

    There is nothing at NBU level to troubleshoot.
    bpbkar logs on the client before and after reboot will simply confirm that read speed was slow before the reboot and faster after the reboot.
    NBU can only report the transfer rate, not what is causing slow or fast transfer.
     

19 Replies

Sort By
  • Anonymous's avatar
    Anonymous
    10 years ago

    Sounds like a separate issue to me.  Re-post as a new question perhaps?

  • Marianne's avatar
    Marianne
    Level 6
    10 years ago

    Can we assume that your entire backup infrastructure is as old and unsupported as your NBU 5.1 software?

    When last did you power-cycle the tape library and the NBU master/media server?
    Have you spoken with network admins to find out if there was an issue over the weekend?
    Were backups slow for all clients or just some?

    bptm log for the date of slow backups will be a good start.

  • kakaluru's avatar
    kakaluru
    Level 4
    10 years ago

    I understood but we never faced this issue earlier.

    I asked network team there was no issue

    We are managing one client. Server and client are in same network. I tried to run manually but its running on b/w 800 to 3000kbs.

    Media IDs are 0245L3,0031L3,0174L3

  • Marianne's avatar
    Marianne
    Level 6
    10 years ago

    I understood but we never faced this issue earlier.

    Well, there is no way that NetBackup can be at fault since the SAME binaries have been in place for way too many years!

    This is Windows... when last did you reboot it?

    What is the Windows version and patch level?
    Any chance that Windows updates were installed recently? 

    On  which date did the slow backups start?
    I do not have the time to go through all the logs...

    PS:
    The files are quite small. In future, PLEASE copy files to .txt (e.g. bptm042415.txt) and upload as File attachments.
    .txt files can be viewed online. zipfiles need to be downloaded and extracted. .log files cannot be viewed on a mobile device.

  • Marianne's avatar
    Marianne
    Level 6
    10 years ago

    I have just had a look at 2 of the logs.
    The only issue in these logs is status 150.

    There is nothing to look at, nothing to troubleshoot if you kill backups.

    Logging level is at 0.

    You need logging to be at least at level 3 to contain anything useful....

     

  • kakaluru's avatar
    kakaluru
    Level 4
    10 years ago

    Last Friday onwards it was listening very very slow. After that i have rebooted and checked but no use.

    ws server 2003 R2 Enterprise Edition service pack 2. What are the patches you want me to tell.

    Recently we haven't done any updates

    Find log files for 000028 id which was successfully completed but it was earlier listening time 20mts .Recently it took 5.30 hrs

  • Marianne's avatar
    Marianne
    Level 6
    10 years ago

    What are the patches you want me to tell. 

     

    Nothing.

    I was asking you to look for any changes outside of NBU. We know that the ONE THING that has NOT changed is NBU. 

    Looking at the log file, we see the following:

    04:19:55.586 [2796.884] <2> write_data: waited for full buffer 36317 times, delayed 619034 times

    This means that NBU (bptm buffer) was waiting for data. 
    The problem is therefore outside of NBU. 
    You need to look at the other components in the data path:
    Client disk -> Client NIC -> network -> Media server NIC -> bptm buffer -> tape drive

    We have in the past seen that Windows updates has enabled TCP Chimney, which resulted in socket errors or just very slow throughput.

    Check TCP Chimney settings on master/media server and client(s):  http://www.symantec.com/docs/TECH60844 

  • kakaluru's avatar
    kakaluru
    Level 4
    10 years ago

    Hello marianne

    where to check this path

    Client disk -> Client NIC -> network -> Media server NIC -> bptm buffer -> tape drive

    Check TCP Chimney settings what exactly it is

  • jim_dalton's avatar
    jim_dalton
    Level 6
    10 years ago

    Marianne is saying that theres a bottleneck and that you need to check that data path to determine whereabout the bottleneck is. Its something you and your colleagues need to investigate, jim