backups failing with error 58

[Sathisp83]

We have rebooted our Master server one of our master server for tape library not detecting, after rebooting the master many policies are getting failed with error 58. Remaining jobs are running fine. The policies which are failing now was not having any issues before we reboot the master server. Logs below. Please advice.

11/10/2013 00:30:08 - Info nbjm (pid=3088) starting backup job (jobid=527777) for client 10.20.30.85, policy NOSXPRRSNP05-STAGE5-DUMP-C, schedule DAILY-FULL
11/10/2013 00:30:08 - Info nbjm (pid=3088) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=527777, request id:{29939E0C-49C9-11E3-8510-00144F81D726})
11/10/2013 00:30:08 - requesting resource orlsxbk01-grp
11/10/2013 00:30:08 - requesting resource orlsxbk01.NBU_CLIENT.MAXJOBS.10.20.30.85
11/10/2013 00:30:08 - requesting resource orlsxbk01.NBU_POLICY.MAXJOBS.NOSXPRRSNP05-STAGE5-DUMP-C
11/10/2013 00:30:16 - granted resource  orlsxbk01.NBU_CLIENT.MAXJOBS.10.20.30.85
11/10/2013 00:30:16 - granted resource  orlsxbk01.NBU_POLICY.MAXJOBS.NOSXPRRSNP05-STAGE5-DUMP-C
11/10/2013 00:30:16 - granted resource  NO0618
11/10/2013 00:30:16 - granted resource  HP.ULTRIUM4-SCSI.000
11/10/2013 00:30:16 - granted resource  orlsxbk01-hcart-robot-tld-0
11/10/2013 00:30:57 - estimated 245385752 kbytes needed
11/10/2013 00:30:57 - Info nbjm (pid=3088) started backup job for client 10.20.30.85, policy NOSXPRRSNP05-STAGE5-DUMP-C, schedule DAILY-FULL on storage unit orlsxbk01-hcart-robot-tld-0
11/10/2013 00:40:40 - Error bpbrm (pid=6912) cannot connect to 10.20.30.85, Operation now in progress (150)
11/10/2013 00:40:40 - end writing
11/10/2013 00:50:40 - Info nbjm (pid=3088) starting backup job (jobid=527777) for client 10.20.30.85, policy NOSXPRRSNP05-STAGE5-DUMP-C, schedule DAILY-FULL
can't connect to client  (58)
 

[Marianne]

client 10.20.30.85...

Why are you using IP addresses instead of hostnames? NBU prefers hostnames.

Status 58 is because one of two reasons:

1. Firewall - either company firewall or OS firewall on server or client.
2. Reverse hosrname lookup failure on client.

Please verify that firewall ports 1556 and 13724 are open in both directions between media server orlsxbk01 and client and that no OS firewall is enabled on server or client.

Verify that client can resolve media server IP address to a valid hostname that is in the client's SERVER list. (bp.conf on Unix client, registry entry on Windows client)

Please do the following to troubleshoot:

Create bpcd log folder on client under ...netbackup/logs.

Run this command on media server orlsxbk01:

bptestbpcd -client <client> -debug -verbose
(command is in /usr/openv/netbackup/bin/admincmd folder)

Check log in bpcd folder on the client. If no log is created, there is a firewall problem.
If log is created, check if client is able to resolve IP to hostname.

Please post output of bptestbpcd command as well as client's bpcd log.

[Sathisp83]

bptestbpcd command logs

---------------------------------------

[control@orlsxbk01]#bptestbpcd -client ork3fsocnp03-priv -debug -verbose
03:26:54.097 [29444] <2> bptestbpcd: VERBOSE = 0
03:26:54.099 [29444] <2> read_client: dname=., offline=0, online_at=0
03:26:54.099 [29444] <2> read_client: dname=.., offline=0, online_at=0
03:26:54.099 [29444] <2> read_client: dname=host_info, offline=0, online_at=0
03:26:54.099 [29444] <2> read_client: dname=CO_0, offline=0, online_at=0
03:26:54.099 [29444] <2> read_client: dname=OA_0, offline=0, online_at=0
03:26:54.116 [29444] <2> db_freeEXDB_INFO: ?
03:30:05.929 [29444] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 148 0x00000094
03:30:15.139 [29444] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 148 0x00000094
03:30:15.139 [29444] <2> connect_to_service: ../../libvlibs/vnet_connect.c.382: 0: vnet_async_connect() failed: 18 0x00000012
03:30:15.139 [29444] <2> vnet_connect_to_bpcd: ../../libvlibs/vnet_connect.c.237: 0: connect_to_service() failed: 18 0x00000012
03:30:15.139 [29444] <2> local_bpcr_connect: Can't connect to client ork3fsocnp03-priv
03:30:15.139 [29444] <2> ConnectToBPCD: bpcd_connect_and_verify(ork3fsocnp03-priv, ork3fsocnp03-priv) failed: 25
<16>bptestbpcd main: Function ConnectToBPCD(ork3fsocnp03-priv) failed: 25
03:30:15.140 [29444] <16> bptestbpcd main: Function ConnectToBPCD(ork3fsocnp03-priv) failed: 25
<2>bptestbpcd: cannot connect on socket
03:30:15.141 [29444] <2> bptestbpcd: cannot connect on socket
<2>bptestbpcd: EXIT status = 25
03:30:15.141 [29444] <2> bptestbpcd: EXIT status = 25
cannot connect on socket
 

Client's bpcd log

---------------------

03:32:48.382 [23888.1012] <2> bpcd main: offset to GMT 18000
03:32:48.382 [23888.1012] <2> bpcd main: Got socket for input 624
03:32:48.382 [23888.1012] <2> logconnections: BPCD ACCEPT FROM 172.24.129.204.64456 TO 10.20.30.66.13782
03:32:48.382 [23888.1012] <2> bpcd main: setup_sockopts complete
03:32:49.898 [23888.1012] <2> bpcd peer_hostname: Connection from host NAV8SOCVANP03 (172.24.129.204) port 64456
03:32:49.898 [23888.1012] <2> bpcd valid_server: comparing orlsxbk01 and NAV8SOCVANP03
03:32:49.898 [23888.1012] <2> bpcd valid_server: comparing orlsxbk01-priv and NAV8SOCVANP03
03:32:49.898 [23888.1012] <2> bpcd valid_server: comparing nolxbkocnp01-priv and NAV8SOCVANP03
03:32:49.898 [23888.1012] <4> bpcd valid_server: NAV8SOCVANP03 is not a master server
03:32:49.898 [23888.1012] <2> bpcd valid_server: comparing nolxbkocnp01 and NAV8SOCVANP03
03:32:49.898 [23888.1012] <16> bpcd valid_server: NAV8SOCVANP03 is not a media server either
03:32:52.867 [23888.1012] <16> bpcd main: read failed: The operation completed successfully.

[RamNagalla]

bpcd log is not matching with the bptestbpcd outputs.. both are from Different times..

what is this server NAV8SOCVANP03? is it media or master server?

show us the output of below commands

from client:-

bpclntcmd -self

bpclntcmd -pn

bpcltncmd -hn <client name>>

bpclntcmd -ip <client IP>

From Master :- 

bpcltncmd -hn <client name>>

bpclntcmd -ip <client IP>

[Sathisp83]

sorry. I just saw the bpcd log date it's as of yesterday. Today no logs are generated.

I done access to client and from Master server it cannot connect 58 error.

[RiaanBadenhorst]

Hi,

Please run the commands Nagalla asked for.

You're using ork3fsocnp03-priv as the client name. Does the "-priv" indicate it has more than one NIC/IP address configured?

It could be communicating back to the master using a "default" interface, in other words not the IP address that is linked to ork3fsocnp03-priv.

[Marianne]

Have you tested 'normal' network connectivity/lookup between master and/or media server to client?

Can you 'ping ork3fsocnp03-priv' ? and 'traceroute ork3fsocnp03-priv'?

It is possible that route to client was manually added on cmd but never added to OS startup scripts.