Forum Discussion

aidil_1's avatar
aidil_1
Level 3
7 years ago

full backup failed with error code 26

Hi all

Thanks in advance for reading this post. any tips\ideas\advices are highly appreciated

environment setting
- NBU 7.7.2 on Windows 2012 R2
- 1x master server which also act as media server
- all clients are VMware, communicated via Virtual Center, snapshot backup using LAN @ NBD
- all backups goes to disk
- daily @ incremental backup from Monday to Friday
- weekly\monthly\yearly @ full on Saturday
- no scheduled backup on Sunday

issue
- no issue with daily backup
- 50% of full backup will failed with EXIT STATUS 26 (client/server handshaking failed)
- restart of the jobs in stages will eventually completed

based on reading, error 26 related to network or communication error. when the issue occured, i would test the network (bpclntcmd) and it turned out ok.
later on, restart of the jobs also turn out ok. if too many failure, i have to restart it in stages, else it will resulted the same; error code 26.
it suggest congestion of bottleneck somewhere but i have no clear path to look to
in worst case, jobs will stuck & bpup/bpdown has to be done.

a case logged to Symantec & he suspected issue with network sockets which to me not very clear or have a good understanding on it. hopefully good folks out there can help me with it.
Symantec advice to clear the network socket connections which i have to request from NW team to do it but no logs/eviden shared it was done. totally dependent on NW team.
And reboot the server which i confirmed rebooted based on server uptime.
And still the issue keep on occuring and ruined my lovely weekend.

Thanks again in advance for reading this post. any tips\ideas\advices are highly appreciated

7.7
exit status 26
NBD
VMware
Windows
windows server 2012

12 Replies

Sort By
  • Marianne's avatar
    Marianne
    Level 6
    6 years ago

    aidil_1 

    I was hoping that one of 'resident network experts' would pick up on this.

    I am surprised that you say that you ' logged a case to Symantec' when the name changed to Veritas about 3 years ago... 

    Anyway - did the Support engineer ask you for bpbrm and bpcd logs ?
    I would be curious to see logs for a successful connection attempt and for a failed attempt.

    Just out of curiousity - if master is also media server, then it means that this is also the backup host (NBU client) for the VMware backups, right?
    So, which hostnames do you use for bpclntcmd testing? 
    bpclntcmd only tests forward and reverse name lookup, it does not test port connection.
    bptestbpcd on master/media server will test name lookup as well as port connection, but since the one server is master, media and client, all initial comms are internal (nothing going out yet to VCenter or ESX server).

    To trace ports that are being used on the NBU server, you can check 'netstat -a' output when backups fail. Save output to a file, and compare with same output when backups are good. 

    Your network team should also be able to trace/monitor network comms between vCenter and ESX servers during backup window.

    • aidil_1's avatar
      aidil_1
      Level 3
      6 years ago

      Thanks for the advice.

      the last time i worked with NBU is early 2016 before switching to other software and now i'm picking it up again.
      maybe that time it still Symantec? :)

      yes, my master is also the backup host for the vmware backup.
      when a backup for client A failed, i perform bpclntcmd for client A

      i'll get the logs and see if can task schedule the "netstat -a" on hourly basis.
      thanks again, will keep posting

  • Krutons's avatar
    Krutons
    Moderator
    6 years ago

    Has this been an issue since the beginning or something new that you have seen slowly start to occur more often?

     

     

    • aidil_1's avatar
      aidil_1
      Level 3
      6 years ago

      hi,

      can't really tell as i'm taking over as it is from someone. it's puzzle me a lot as it mostly impact full (weekend\monthly\yearly) backup only. any thought or idea?

      • Krutons's avatar
        Krutons
        Moderator
        6 years ago

        How many full backups are running at a time?

        During those full backups, I would try to netstat -t a few times and copy the output so you can post it here.

        Could you post the output of our TCP parameters on the master server?
        netsh int tcp show global