Backup failing for client server(Windows) in Netbackup 5.1

[NBFREAK]

Hi Folks,

Backup is failing for one of the client server with error codes 58 and 54. kindly help me to fix the issue. details are as below

Master server: Unix - we are using Java console to monitor the jobs.
Client: Windows 2003 server.

Master: Netbackup version : 5.1 MP4 
Client :Netbackup version : 5.1 MP7

Logs:

58:
1/01/10 - Error bprd (pid=1234) Unable to write progress log </C/ProgramA Files/VERITAS/NetBackup/Logs/user_ops/dbext/logs10132.1262353370> on client XXXXXXXXXXXXXXXXX Policy =<Policy name> Sched=NONE
1/01/10 - Error bprd (pid=1234) CLIENT XXXXXXXXXXXXX POLICY <policy Name> SCHED NONE EXIT STATUS58(can't connect to client)

54: Timed out connecting to client..


The backup failing for this cliennt alsone

FYI 100 clients are configured under the same master server rest all backups are running fine.

Please help

Thanks in Advance.... Happy New year all

[Marianne]

As a start: a higher NBU version on the client than on Master/Media server is not supported. All sorts of problems can be expected.

Next, confirm forward and reverse lookup between server and client:

On server:
bpclntcmd -hn <client_name>
bpclntcmd -ip <client_IP>

On client:
bpclntcmd -hn <server_name>
bpclntcmd -ip <server_IP>

[NBFREAK]

Hi Marianne ,

i am successfull in reverse lookup between server and client

On server:Unix

<Master Server>:/usr/openv/netbackup/bin#->bpclntcmd -hn <Client_name>

host Client_name: Client Name at XX.XXX.XX.XX (0xad11565)

checkhname: aliases:

Master_Name:/usr/openv/netbackup/bin#->

Master_Name:/usr/openv/netbackup/bin#->bpclntcmd -ip XX.XXX.XX.XXX

checkhaddr: host : Clinet: Client at XX.XXX.XX.XXX (0xad11565)

checkhaddr: aliases:

Client:

C:\Program Files\VERITAS\NetBackup\bin>bpclntcmd -hn Master_Name

host Master: Master_Name at XX.XXX.XX.XXX (0xaf10d10a)

checkhname: aliases:

C:\Program Files\VERITAS\NetBackup\bin>bpclntcmd -ip XX.XXX.XX.XXX

checkhaddr: host   : Mater: Master_Name at XX.XXX.XX.XXX (0xaf10d1

0a)

checkhaddr: aliases:

in other clients also MP7 is installed but they are backingup with no issue ..

is there thing i can check on ports ?
if so please let me know i can check on that based on ur advice.

Thanks for your advice.
Ram

[SteveYu]

1. What type of backup is this? Type Windows-nt?
2. Do you have the bpcd log from the client?
3. Are you able to pull the client host properties from the administration console?
4. Has this ever worked before?

-Sy

[Marianne]

Have you tested a restore for the other MP7 clients? There have been posts on this forum with similar scenario where backups have been successful but restores failed.

You have asked for port info - is this client behind a firewall? If so, have a look at this TechNote (it's valid for NBU 4.5 and 5.x):

How to set up a client that is behind a firewall, filtering by destination port using VNETD

Please also post contents of bpcd log file on client.

[NBFREAK]

Hi Steve,

Good day!!

1: This is Windows -nt backup .
2: Yes i have the BPCD logs but they are not creating daily ...
3: I am unable to pull the client properties for this from master server admin console.
4: yes it was working till last month.

This client is Windows 2003 X64 version.
is there any specific procedure to install client for x64?

Please advice.

[NBFREAK]

Hi Marianne,

I have not tried in restoring the files for MP7 clients.

and also the server is not behind firewall. This client is x64 version.

BPCD log file are creating but not daily .. they are getting created on its own way as there is a dily scheduled backup configiured for this client.

Thanks all

[NBFREAK]

Hi All,

Sorry ..Correction here,

Java Console is with 5.1 MP4
Netbackup version in the Unix server(Master ) is 5.1 MP7

Sorry again

[Marianne]

Great stuff! Now we can concentrate on the errors...
About the x64 client - 5.1 support for x64 started with MP4.
NB_51_4_M.winnt.x64.exe is the BASE INSTALL NetBackup software for x64 clients. It can be downloaded from the link below.
After installing this, the latest patch for x64 can be installed.
http://seer.support.veritas.com/docs/280950.htm

Is the master server also the media server for this client? If not, you will also have to test lookup between media server and client using bpclntcmd.
The entries in bpcd logs on client will be as a result of connection attempts. We'd like to see what's in there...

Even if the client is not behind a firewall, it might still be a good idea to configure it for vnetd connect-back - that will make troubleshooting a lot easier.
With vnetd configured for connect-back, you simply need to test a telnet to bpcd on the client (from master and media server) and telnet from client to master/media server on vnetd. Without vnetd configuration, the connect-back is on a range from 512 - 1023.

Any chance you will be upgrading to 6.5.x any time soon? Troubleshooting is a lot easier with bptestbpcd in 6.x and port usage between servers and clients is on vnetd only.

[takaweb]

1st of all, Best wishes for 2010 to everybody !

1st an strange question to Marianne, do you happen to be the wife, sister, anyhow related to  "Mario van den
Berg",  who happens to be the "Backup Manager", responsable for the NetBackup Enterprise systems at an DataCentre/Payroll company at brouwersvliet, Antwerp,  Belgium? Or is this an "alias" of Mario himself ?  ;)
in that case: "Hello, ex-collegue/ex-boss" or "wife of/related to.."  :)
Otherwise (if none of the above) the world is getting really small, and is this an really funny coïncidence...

I used to work below mario (01/2006 - 07/2007) as ext. consultant, which was the Netbackup 5.1 MP1/MP2
period, MP4 was released (or at least implemented) somewhere in 2007, they also migrated (end 2006)
from an Windows only NB environment (Master/Media servers) to mixed Unix/Windows environment.. Windows Masterserver was migrated to Sun Solaris 10 Masterserver..
in the end, they were running 11 NetBackup servers, of which 8 Unix servers and 3 Windows servers..


it was quite an complex environment, as Netbackup was backing up 564 Windows/Citrix serverclients
(of which 415 unique), which were spread over 3 forests and 29 domains. In total, 66 different policies
(of which 61 effective and 4 test policies) were used in 7 different levels of criticalness..
Backup priorities and backup schemes:
File - CRI- x - type pol: 28,   Database - CRI - x - type pol: 20,  Dump - CRI - x - type pol: 17,   test pol: 4
also, an seperate Master server Catalog database backup policy
they used an firewalled external network, accessible for customers, so there was an secondary NIC
build in (not firewalled) per server, specific for NetBackup purposes...
---------------------------------------------------------------------------------------------------------------------------------------------------
* Status-code 58 and 54 issue :

  - perform an ping -a on the troubled client, & verify if IP-address is correct.

  - perform an NSLOOKUP to verify if DNS record of this server, is pointing to the correct IP-address

  - ALSO (!) Verify if there are no double/multi and/or obsolete entry's in DNS regarding this server,
    it happens that a server has 2 different records in DNS, each pointing to an different IP-address
    -> when the order in DNS = correct, you'll have no issue's, but if records/IP-addresses are swapped
        in other direction (example: deleted good DNS record & then re-added the record) this usually
        results in the status-code's 58, 54 as described above (DNS record now points to another server
        (duplicate) hostname with correct IP-address OR with correct hostname, but wrong IP-address
        of an different server (much more difficult to spot in long lists of records, category "blind spot")

  - ALSO (!) Exactly the same goes for your HOSTS files (if used anyhow) on one of following servers:
    -> check your HOSTS file on your troubled client, the backup initiating Netbackup server, all Media
        servers & also Master server, [C:\WINDOWS\system32\drivers\etc\HOSTS] to verify if they are
        all pointing to the correct server, with correct HOSTNAME and according correct IP-address...
        [on Unix server it was on root/etc/inet directory/file i believe, allthough not quite sure anymore..]
        if you're able to backup the client correctly on one media-server and it allways fails on the other
        one (Media/Master server) it's allmost guaranteed either an DNS record or HOSTS file issue)

REMARK: due to this inter-forest/inter-domain/NIC1 firewalled & external accessible/NIC2 internal use
                   non-firewalled NetBackup only, we've then started heavely to depend & rely on the HOSTS
                   files, as they preceed all other records/pointers, and started to centrally manage these
                   3 HOSTS files (1 Master HOSTS file pér forest) from an single-point-of-origin (Netbackup
                   management/java console) & afterwards after adjustments, auto-updated all related servers
                   in all forests, via an automated CMD batchscript to push the latest HOSTS file across..
                   => but afcourse, with +500 clients, copy/paste is frequently used, and on this way, you'dd
                         copy an HOSTS file with wrong IP-address/hostname/double entry for either hostname
                         and/or IP-address immediatly onto all, so check carefully...
NOTE: This works for about all servers & types, except for SQL Servers and ? (forgotten),
              both heavely relies on:  hostname,  DNS  &  WINS  registration, so keep this in mind...
---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---    - Also, if not 100% sure regarding firewall, check availability of your NetBackup BPCD port 13782
     (client deamon), BPRD port 13720 (request deamon), VNETD port 13724 (network deamon)
     {optional NDMP port 10000 if used} (correct me if i'm wrong !) i would put an exclusion-rule in the
    centralized/local firewall by default anyhow.. (see ports System Admin guide VOL 1, page 504 of 566
---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---
- 1 final option that I can think of: IF you are working in an mixed environment and/or coming from windows
   only environment, but now also using Unix -> Unix is Case-Sensitive, unlike Windows, which is NOT
case-sensitive -> on all the above points: DNS, HOSTS, hostname.. if you make an error on this issue,
then the job, clients hostname, DNS record.... even in the HOSTS file, your client will not be your client...
therefor, he will not be able to connect to the machine, as this is an different client to Unix, resulting in
an unknown and/or unreachable client.. resulting in status code's 58, 54....
---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     --- 
as far as I can remember, to my knowledge, status code 58, 54, was allways Network Related, you have
an "communication issue", either incorrect DNS, HOSTS, IP-address settings, also unavailable NB ports,
or blocking firewalls, possibly even Case sensitivity might cause this issue (Unix), but it shouldn't be client
(server) related, NetBackup softw. install related or client hardware related, but don't shoot me if i'm wrong!
Mea Culpa when i am mistaken/or putting you on the wrong foot by this !
---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---     ---

---------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------
* Multiplexing indeed enhances you're performance considerably, so if optional possible, do use the
  multiplexing feature, especially when you get into time-need.. When performing an full backup of
  1,2Tbyte to 1,5Tbyte on friday (the 2 largest dataservers then) the weekend is not even long enough
  when having issue's.. Also, the daily incremental/differential backup will not be completed in the
  morning when customers have the possibility to remotely login/work on the server until 23h in the
  evening (realtime webbased payroll applications), without the multiplexing feature...
  indeed, you're restores will take longer to complete using this feature, but without an good backup
  there (might) be nothing to restore, so.. Also, to shorten restore times, you can reserve specific tape
  units/drive's/libraries for restore tasks during the day, so you'll also restore multiplexed streams or
  restore from different tapes/units concurrently (if possible/correctly configured), so shouldn't be issue
  (depending environment) HOWEVER, avoid multiple concurrent streams to/from an single device..
  => see also System Admin guide, VOL 1 for more information on page 138 of 566...

---------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------

* Bare-Metal-Restore's: an bare-metal-restore on an 32-bit x86 client didn't caused any problems
                                  whatsoever (runned almost fluently) HOWEVER, on an 64-bit x64 system,
                                  restore was initiated & completed perfectly, but after reboot, an non-bootable
                                  DOA Server 2003 resulted ('NTLDR is missing' i believe, or NTDEDECT.COM)
  --> due to the fact that this was an business-critical server, there was no time available for indept
       troubleshooting, but following steps resolved the issue:
       -  Re-install an clean Server 2003 OS from MS bootable CD
       -  configure NIC's & rename the host to the original required hostname + reboot
       -  install & configure NetBackup 5.1 MP4
       -  perform bare-metal-restore on top of the freshly installed Server 2003 & reboot

=> problem resolved..


Damned !  Urgently need to go, hope this helps you & good luck !
Kind Regards,

[nbu2k4]

Hi Marianne,
I am friend to the initial requestor of this thread and would like to add some info / reply on his behalf some queries:

As informed by him both master and client have same nbu versions. NBU 5.1 MP7.
we have downloaded and reinstalled the client version from http://seer.entsupport.symantec.com/docs/302688.htm

Master server also acts as media server.

telnet from master to client:
telnet itsapcondp2p01p.ap.jnj.com vnetd
Trying...
Connected to itsapcondp2p01p.ap.jnj.com.
Escape character is '^]'.
Connection closed by foreign host.
note: this is same result for bpcd also.

telnet from client to master:
it gets established.
note: same for bpcd too.

I am unable to  enable vnetd callback option as mentioned in
http://seer.entsupport.symantec.com/docs/260010.htm

I didnt find the client info under  /usr/openv/netbackup/db/client.

trying to configure vnetd call back also failed.
/usr/openv/netbackup/bin/admincmd#->bpclient -client <clientname> -L
bpclient: no entity was found (227)

/usr/openv/netbackup/bin/admincmd#->/usr/openv/netbackup/bin/admincmd/bpclient -update -client <clientname> -no_callback 1
db_CLIENT() failed: no entity was found (227)bpclient: no entity was found (227)


Apart from analysing vnetd configuations, we have few more things to tell:

1. this server has total 4 nics. 2 are teamed for network. 1 for backup and the rest is disabled.
2. teaming ip (k.k.k.k) is resolved in dns as <clientname>, while backup ip (z.z.z.z) in dns is resolved as <clientnamep>
3. all backup policies of the server has <clientnamep>.
4. Whenever server is rebooted, for the next 2-3 days backups are successful. some times we would be lucky for more than 2 weeks.


BPCD log:

02:23:51.146 [3588.3188] <2> bpcd main: offset to GMT -28800
02:23:51.146 [3588.3188] <2> bpcd main: Got socket for input 428
02:23:51.146 [3588.3188] <2> logconnections: BPCD ACCEPT FROM x.x.x.x.845 TO z.z.z.z.13782
02:23:51.146 [3588.3188] <2> bpcd main: setup_sockopts complete
02:23:51.162 [3588.3188] <2> bpcd peer_hostname: Connection from host <master> (x.x.x.x) port 845
02:23:51.162 [3588.3188] <2> bpcd valid_server: comparing <master> and <master>
02:23:51.162 [3588.3188] <4> bpcd valid_server: hostname comparison succeeded
02:23:51.443 [3588.3188] <2> bpcd main: output socket port number = 569
02:23:51.443 [3588.3188] <2> bpcd peer_hostname: Connection from host <master> (x.x.x.x) port 845
02:23:51.443 [3588.3188] <2> bpcd main: Peer hostname is <master>
02:23:51.443 [3588.3188] <2> bpcd main: Got socket for output 832, lport = 1014
02:23:51.443 [3588.3188] <2> bpcd main: Connected on output socket
02:23:51.443 [3588.3188] <2> bpcd main: Duplicated socket on stderr
02:23:51.443 [3588.3188] <2> bpcd main: <---- NetBackup 5.1 0 ------------initiated
02:23:51.443 [3588.3188] <2> bpcd main: Not using VxSS authentication with ncsapsm2.ncsau.jnj.com

Interface: k.k.k.k --- 0x10003
  Internet Address      Physical Address      Type
  a.b.c.d           00-00-0c-07-ac-aa     dynamic   
  e.f.g.h         00-1c-c4-bd-4c-ca     dynamic  

Interface: z.z.z.z--- 0x10004
  Internet Address      Physical Address      Type
  x.x.x.x         00-00-0c-07-ac-b4     dynamic  
02:23:51.475 [3588.3188] <2> bpcd main:
02:23:51.521 [3588.3188] <2> bpcd main: BPCD_GET_VERSION_RQST
02:23:51.615 [3588.3188] <2> bpcd main: BPCD_GET_STDIN_SOCKET_RQST
02:23:51.615 [3588.3188] <2> bpcd main: socket port number = 894
02:23:51.615 [3588.3188] <2> bpcd peer_hostname: Connection from host <master> (x.x.x.x) port 845
02:23:51.615 [3588.3188] <2> bpcd main: Socket peer hostname is <master>
02:23:51.615 [3588.3188] <2> nb_getsockconnected: Connect to <master> on port 607
02:23:51.615 [3588.3188] <2> logconnections: bprd CONNECT FROM z.z.z.z.607 TO x.x.x.x.894
02:23:51.615 [3588.3188] <2> bpcd main: Connected on output socket
02:23:51.662 [3588.3188] <2> bpcd main: BPCD_FORK_CMD_RQST
02:23:51.771 [3588.3188] <2> bpcd main: fork cmd = /usr/openv/netbackup/bin/bpmount bpmount -i -streams
02:23:51.771 [3588.3188] <2> bpcd main: filter type = -1
02:23:51.771 [3588.3188] <2> bpcd main: Convert args to CreateProcess format
02:23:51.771 [3588.3188] <2> bpcd main: Done converting args to CreateProcess format
02:23:51.771 [3588.3188] <2> bpcd main: new fork cmd = C:\Program Files\VERITAS\NetBackup\bin\bpmount.exe -i -streams
02:23:51.771 [3588.3188] <2> bpcd main: Before CreateProcess
02:23:51.771 [3588.3188] <2> bpcd main: StdOutput assigned the value STDOUTSOCK
02:23:51.771 [3588.3188] <2> bpcd main: After CreateProcess, pid = 10152
02:23:51.771 [3588.3188] <2> bpcd exit_bpcd: exit status 0  ----------->exiting
 

[Venkat_2]

Reboot the client.. as a work around..! it will surely work

[Marianne · Accepted solution]

The fact that you re-installed NBU software and still have the same problem says to me that the problem is unlikely to be with NBU.
Also the following statement:
"Whenever server is rebooted, for the next 2-3 days backups are successful. some times we would be lucky for more than 2 weeks."  confirms my belief that the problem is not with NBU.
The bpcd log that you've posted is obviously from a successful backup.
When you telnet from master to client on bpcd and the connection is dropped immediately, does bpcd log get updated?
What does netstat -a on the client reflect at this point w.r.t bpcd?
Have you checked if the latest drivers for backup network card is installed?
Any software on the client that might prevent port setup/connections?
DNS reliable at all times? As a test, add hosts entries for client name (clientnamep) and master on both client and master.

Have you added clientnamep as Required Interface in Client's Host Properties?
What does 'bpclntcmd -self' on client reflect? If Required Interface is set, it should only display clientnamep and it's IP address.

As far as bpclient on master is concerned, the TechNote unfortunately fails to say that client name must first be added with 'bpclient  -client <client_name> -add' before it can be updated with 'bpclient  .... -update ...'

Hopefully there are lists members with more networking knowledge than me who can provide more useful info...

[nbu2k4]

Hi Marianne/takaweb

I have tried your above mentioned steps.

thanks a lot for your suggestions.

At last I again did the basic ground work ; logging in server when the backup starts :) and there popped a disk C full alert. Checked and found .vsp file getting created when backup started.
I disabled the windows open file backup configuration.

Now its working cool.

Thanks for your time. I appreciate it.

[Omar_Villa]

Just one question, is the master server also the media server where the client is trying to connect? if not errors 58 are because is the media server the one that cannot connect to the client. just want to be sure.

Best Regards