Backup failing with status 58

[Shrikumar]

Hi All,

One of my clinet's backup is failing with status 58,please see some of my findings and suggest what to be done to resolve the issue.

Master Server Details - (OS - Solaris 5.10,Netbackup Version -6.5.3.1)
Media Server Details - (OS - Solaris 5.10,Netbackup Version -6.5.3.1)
Client Server Deails - ( OS _ Solaris 5.8,Netbackup Version

==================================================================
Connectivity checked from Master Server :-
-------------------------------------------------------------
sn013c@asprd111: bpclntcmd -hn nbxdb-ebr
host nbxdb-ebr: nbxdb-ebr at 135.40.227.56 (0x8728e338)
aliases:
sn013c@asprd111:  bpclntcmd -ip 135.40.227.56
checkhaddr: host   : nbxdb-ebr: nbxdb-ebr at 135.40.227.56 (0x8728e338)
checkhaddr: aliases:
sn013c@asprd111:
-----------------------------------------------------------------------------
Connectivity checked from Media Server :-
sn013c@rsprd179: bpclntcmd -hn nbxdb-ebr
host nbxdb-ebr: nbxdb-ebr at 135.40.227.56 (0x8728e338)
aliases:
sn013c@rsprd179: bpclntcmd -ip 135.40.227.56
checkhaddr: host   : nbxdb-ebr: nbxdb-ebr at 135.40.227.56 (0x8728e338)
checkhaddr: aliases:
sn013c@rsprd179:
==================================================================
Bptestbpcd output from Master Server :-
----------------------------------------------------
sn013c@asprd111: bptestbpcd -verbose -debug -client nbxdb-ebr
15:33:55.141 [11904] <2> bptestbpcd: VERBOSE = 1
15:33:55.143 [11904] <2> read_host_info: CONNECT_NON_RES_PORT 2
15:33:55.143 [11904] <2> read_host_info: DYNAMIC_ADDRESS 0
15:33:55.143 [11904] <2> read_host_info: FREE_BROWSE 0
15:33:55.143 [11904] <2> read_host_info: LIST_RESTORE 0
15:33:55.143 [11904] <2> read_host_info: MAX_JOBS_THIS_CLIENT 0
15:33:55.143 [11904] <2> read_host_info: NO_CALLBACK 2
15:33:55.143 [11904] <2> read_host_info: WOFB_ENABLED 1
15:33:55.143 [11904] <2> read_host_info: WOFB_FIM 0
15:33:55.143 [11904] <2> read_host_info: WOFB_USAGE 0
15:33:55.143 [11904] <2> read_host_info: WOFB_ERROR 0
15:33:55.143 [11904] <2> read_host_info: DAEMON_PORT_TYPE 3
15:33:55.147 [11904] <2> init_cache: vnet_hosts.c.990: host_cache_size: 200 0x000000c8
15:33:55.147 [11904] <2> init_cache: vnet_hosts.c.991: cache_time: 3600 0x00000e10
15:33:55.147 [11904] <2> init_cache: vnet_hosts.c.1003: host_failed_cache_size: 40 0x00000028
15:33:55.148 [11904] <2> init_cache: vnet_hosts.c.1004: cache_time: 3600 0x00000e10
15:33:55.148 [11904] <2> init_cache: vnet_hosts.c.990: host_cache_size: 200 0x000000c8
15:33:55.148 [11904] <2> init_cache: vnet_hosts.c.991: cache_time: 3600 0x00000e10
15:33:55.148 [11904] <2> init_cache: vnet_hosts.c.1003: host_failed_cache_size: 40 0x00000028
15:33:55.148 [11904] <2> init_cache: vnet_hosts.c.1004: cache_time: 3600 0x00000e10
15:33:55.316 [11904] <2> version_connect: vnet_vnetd.c.1830: Function failed: 24 0x00000018
15:33:55.317 [11904] <2> do_vnetd_service: vnet_vnetd.c.4393: version_connect failed: 24 0x00000018
15:33:55.317 [11904] <2> vnet_async_connect: vnet_vnetd.c.4193: do_vnetd_service failed: 24 0x00000018
15:33:55.386 [11904] <2> logconnections: BPCD CONNECT FROM 135.53.28.54.858 TO 135.40.227.56.13782
15:34:36.466 [11904] <2> vnet_pop_byte: vnet.c.184: errno: 2 0x00000002
15:34:36.467 [11904] <2> vnet_pop_byte: vnet.c.186: Function failed: 9 0x00000009
15:34:36.467 [11904] <2> vnet_pop_string: vnet.c.266: Function failed: 9 0x00000009
15:34:36.467 [11904] <2> vnet_pop_signed: vnet.c.310: Function failed: 9 0x00000009
15:34:36.467 [11904] <2> vnet_pop_status: vnet.c.388: Function failed: 9 0x00000009
15:34:36.467 [11904] <2> vnet_begin_connect_back: vnet_vnetd.c.720: status: 9 0x00000009
15:34:36.467 [11904] <2> ConnectToBPCD: bpcd_connect_and_verify(nbxdb-ebr, nbxdb-ebr) failed: 0
<16>bptestbpcd main: Function ConnectToBPCD(nbxdb-ebr) failed: 25
15:34:36.468 [11904] <16> bptestbpcd main: Function ConnectToBPCD(nbxdb-ebr) failed: 25
<2>bptestbpcd: cannot connect on socket
15:34:36.468 [11904] <2> bptestbpcd: cannot connect on socket
<2>bptestbpcd: EXIT status = 25
15:34:36.468 [11904] <2> bptestbpcd: EXIT status = 25
cannot connect on socket
---------------------------------------------------------------------------------------------------------------
Bptestbpcd output from Media Server :-
----------------------------------------------------
sn013c@rsprd179: bptestbpcd -verbose -debug -client nbxdb-ebr
13:35:26.163 [7890] <2> bptestbpcd: VERBOSE = 1
13:35:26.165 [7890] <2> init_cache: vnet_hosts.c.990: host_cache_size: 200 0x000000c8
13:35:26.165 [7890] <2> init_cache: vnet_hosts.c.991: cache_time: 3600 0x00000e10
13:35:26.165 [7890] <2> init_cache: vnet_hosts.c.1003: host_failed_cache_size: 40 0x00000028
13:35:26.165 [7890] <2> init_cache: vnet_hosts.c.1004: cache_time: 3600 0x00000e10
13:35:26.166 [7890] <2> init_cache: vnet_hosts.c.990: host_cache_size: 200 0x000000c8
13:35:26.166 [7890] <2> init_cache: vnet_hosts.c.991: cache_time: 3600 0x00000e10
13:35:26.166 [7890] <2> init_cache: vnet_hosts.c.1003: host_failed_cache_size: 40 0x00000028
13:35:26.167 [7890] <2> init_cache: vnet_hosts.c.1004: cache_time: 3600 0x00000e10
13:35:26.359 [7890] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2043: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
13:35:26.359 [7890] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2057: service: bpdbm
13:35:26.501 [7890] <2> logconnections: BPDBM CONNECT FROM 135.33.138.27.44811 TO 135.53.28.53.13724
13:35:27.367 [7890] <2> version_connect: vnet_vnetd.c.1830: Function failed: 24 0x00000018
13:35:27.367 [7890] <2> do_vnetd_service: vnet_vnetd.c.4393: version_connect failed: 24 0x00000018
13:35:27.367 [7890] <2> vnet_async_connect: vnet_vnetd.c.4193: do_vnetd_service failed: 24 0x00000018
13:35:27.581 [7890] <2> logconnections: BPCD CONNECT FROM 135.33.138.27.554 TO 135.40.227.56.13782
13:36:08.723 [7890] <2> vnet_pop_byte: vnet.c.184: errno: 2 0x00000002
13:36:08.723 [7890] <2> vnet_pop_byte: vnet.c.186: Function failed: 9 0x00000009
13:36:08.723 [7890] <2> vnet_pop_string: vnet.c.266: Function failed: 9 0x00000009
13:36:08.723 [7890] <2> vnet_pop_signed: vnet.c.310: Function failed: 9 0x00000009
13:36:08.723 [7890] <2> vnet_pop_status: vnet.c.388: Function failed: 9 0x00000009
13:36:08.723 [7890] <2> vnet_begin_connect_back: vnet_vnetd.c.720: status: 9 0x00000009
13:36:08.724 [7890] <2> ConnectToBPCD: bpcd_connect_and_verify(nbxdb-ebr, nbxdb-ebr) failed: 0
<16>bptestbpcd main: Function ConnectToBPCD(nbxdb-ebr) failed: 25
13:36:08.724 [7890] <16> bptestbpcd main: Function ConnectToBPCD(nbxdb-ebr) failed: 25
<2>bptestbpcd: cannot connect on socket
13:36:08.725 [7890] <2> bptestbpcd: cannot connect on socket
<2>bptestbpcd: EXIT status = 25
13:36:08.725 [7890] <2> bptestbpcd: EXIT status = 25
cannot connect on socket
================================================================================================

Please suggest what needs to be done.

[Marianne · Accepted solution]

Also do the same bpclntcmd tests from client to master and media server.

Please also confirm client's NBU version.
Have you checked the client's bpcd log file?  Connection to client is different if client is pre-6.x - connection goes straight to bpcd instead of via vnetd. Connection back will also be attempted on the default of 512 - 1023 (unless vnetd is specified in Host Properties).  We need bpcd log to see what's happening on the client.

STATUS CODE 58: Can't connect to client. Troubleshooting procedures for Status 58 errors.
http://seer.entsupport.symantec.com/docs/321172.htm

[Andy_Welburn]

That's the third so far!

[Shrikumar]

Netbackup Client Version  is 5.1
-----------------------------------------------
Bpclntcmd to Master Server from Client :-
----------------------------------------------------
/usr/openv/netbackup/bin/bpclntcmd -hn asprd111.aldc.att.com
host asprd111.aldc.att.com: asprd111.aldc.att.comat 135.53.28.53 (0x87351c35)
host asprd111.aldc.att.com: asprd111.aldc.att.comat 135.53.28.54 (0x87351c36)
checkhname: aliases:
/usr/openv/netbackup/bin/bpclntcmd -ip 135.53.28.53
checkhaddr: host : asprd111: asprd111.aldc.att.comat 135.53.28.53 (0x87351c35)
checkhaddr: aliases:
--------------------------------------------------------------------------
Bpclntcmd to Media Server from Client :-
----------------------------------------------------------
/usr/openv/netbackup/bin/bpclntcmd -hn rsprd179.ims.att.com
host rsprd179.ims.att.com: rsprd179.ims.att.comat 135.33.138.26 (0x87218a1a)
host rsprd179.ims.att.com: rsprd179.ims.att.comat 135.33.138.27 (0x87218a1b)
==================================================================
Please let me know what needs to be done.

[Marianne]

Why 2 IP addresses for one hostname?

As explained in my previous post - you need bpcd log directory on the client to see how the client resolves the server IP address and which port is chosen for connect-back.

[Shrikumar]

As you have mentioned in your previos post that connection to client is diffetent in 5.x & 6.x

Can you please explain a little bit.

What is the vnetd,bpcd when both are communication purpose only.

Thanks in advance.

Shrikumar

[Yasuhisa_Ishika]

Serch into NetBackup 5.1 SYstrem Adminsitrator's Guide Volime I with keyword Firewall, vnetd, and connect-back.

In older version, NetBackup uses only bpcd for communication. Client(not means NetBackup client, means communication initiator) connect to server's(communitcation target) bpcd port, server forks a process for requested task, then server's new process connect to client's port(connect-back). Port for connect-back is selected from 512-1023 port. This by-direct communication methos method requries broaded range of port exposure.

In newer NetBackup version, NetBackup uses vnetd for uni-direct connection. The initiator connect to target's vnetd port, target forks a process thru bpcd , target tells named pipe that new process listend on. Next, initiator connect to target' vnetd process again and request vnetd to redirect the connection to named pipe.

In short, you need to enabled debug logging for both bpcd and vnetd. Refer Troubleshooting Guide for debug logging.

Well, I can not remind somethings needed to use of 5.x client and 6.x server. Something have to be done.

[Yasuhisa_Ishika]

See pege 15 of this doc and try it !

http://www.symantec.com/docs/TECH4607

[Redders75]

'Ello all,

My Backup Administrator came to me with the exact same issue on a new client.  The 'bptestbpcd' command returned the same output as what Srikumar had.

/etc/nsswitch.conf was fine, hostname resolution was fine, the bpclntcmd output was fine wherever I ran it (on client and NBU server).

The fix for me was /etc/hosts.allow and /etc/hosts.deny.  Our Solaris 10 Jumpstart build uses JASS security hardening and it was this which had copied the original, blank files to a new location and hardened them up with new entries.

With no /etc/hosts.allow or /etc/hosts.deny in place, it all worked fine.