cancel
Showing results forΒ 
Search instead forΒ 
Did you mean:Β 

after upgrade Master/Media servers to 8.1, a couple 7.7.2 clients can't connect

drooks
Level 2

After upgrading the master and media servers to 8.1, a couple of 7.7.2 clients can't connect. 

Has anyone run into this?  The backups worked prior to upgrading the master and media servers, they were on 7.7.2 also.  The only odd thing is that these couple of servers run on the production network and not our dedicated backup network.  No secondary nics on the clients.  I've opened a case and so far there hasn't been any luck with the tech.  I tried everything she did:

bpclntcmd, ping, checking bp.conf. nslookup, etc...   telnet is not enabled on the servers for sox/pci requirements.

Master and media servers are RHEl 6. 

A couple clients are windows, which I don't have access to.

one is RHEL 6 also.

 

Below is some output from running against the RHEL client:

 

=====================================================================
Job Details


Jul 10, 2018 10:46:39 AM - Error bpbrm (pid=1194) cannot connect to RHEL_CLIENT02, Transport endpoint is not connected (107)
Jul 10, 2018 10:46:39 AM - Info bpbkar (pid=0) done. status: 58: can't connect to client
Jul 10, 2018 10:46:39 AM - end writing
can't connect to client  (58)
=====================================================================

 


MASTER_SERVER admincmd]$ bptestbpcd -client RHEL_CLIENT02 -verbose -debug

15:39:56.540 [3266] <2> bptestbpcd: VERBOSE = 0
15:39:56.540 [3266] <2> read_client: dname=., offline=0, online_at=0 offline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
15:39:56.540 [3266] <2> read_client: dname=.., offline=0, online_at=0 offline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
15:39:56.540 [3266] <2> read_client: dname=host_info, offline=0, online_at=0 offline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
15:39:56.540 [3266] <2> read_client: dname=CO_0, offline=0, online_at=0 offline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
15:39:56.540 [3266] <2> read_client: dname=OA_0, offline=0, online_at=0 offline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
15:39:56.540 [3266] <2> read_client: dname=COA_0, offline=0, online_at=0 offline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
15:39:56.540 [3266] <2> read_client: dname=OAA_0, offline=0, online_at=0 offline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
15:39:56.559 [3266] <2> vnet_pbxConnect_ex: pbxConnectExEx Succeeded
15:39:56.642 [3266] <2> logconnections: BPCD CONNECT FROM 10.250.....35639 TO 10.250.......1556 fd = 4
15:39:56.643 [3266] <2> vnet_connect_to_vnetd_bpcd: js_bpcd_info: 0xdae0f0
15:39:56.644 [3266] <2> vnet_pbxConnect_ex: pbxConnectExEx Succeeded
15:39:56.650 [3266] <8> do_pbx_service: [vnet_connect.c:2579] via PBX VNETD CONNECT FROM 10.250.......64637 TO 10.250.......1556 fd = 5
15:39:56.650 [3266] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:458] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
15:39:56.692 [3266] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:483] ipc_string /usr/openv/var/tmp/vnet-35365530560396687548000002075-tz1OUz
15:41:56.834 [3266] <2> bpcr_put_vnetd_forward_socket: bpcd returned failure: 21
15:41:56.834 [3266] <2> local_bpcr_connect: bpcr_put_vnetd_forward_socket failed: 21
15:41:56.834 [3266] <2> ConnectToBPCD: bpcd_connect_and_verify(RHEL_CLIENT02, RHEL_CLIENT02) failed: 21
15:41:56.834 [3266] <16> bptestbpcd main: JSON proxy message = {"allow_large_status": {"timestamp": 1530560396, "who": "vnet_tss_init", "line_number": 32, "comment": "allow vnet status > 255", "data": true}, "direct_connect": {"timestamp": 1530560396, "who": "connect_to_service", "line_number": 834, "comment": "connect parameters", "data": {"who": "vnet_connect_to_vnetd_bpcd", "host": "RHEL_CLIENT02", "service": "vnetd", "override_required_interface": null, "extra_tries_on_connect": 0, "getsock_disable_to": 0, "overide_connect_timeout": 0, "connect_options": null}}, "proxy_request": {"timestamp": 1530560396, "who": "vnet_proxy_build_request", "line_number": 3283, "comment": "request to local proxy", "data": {"bpcd_info": {"remote_client": "RHEL_CLIENT02", "remote_host": "RHEL_CLIENT02", "remote_probably_dhcp": false, "proxy_flag": 32, "proxy_flag_hex": "0x00000020", "cmd_sock": 4}, "i_am_daemon": false, "domain_constraints": {}, "service": "vnetd", "auth_only": false, "pid": 3266, "ssa": false, "local_user": "root", "host": "RHEL_CLIENT02", "peer_host": "RHEL_CLIENT02", "peer_client": "RHEL_CLIENT02", "local_client": "MASTER_SERVER", "local_dhcp_interval": 0, "peer_dhcp_info": null, "local_hostname": "MASTER_SERVER", "secure_peer": false, "resilient": false, "nbrntd_peer_addr": null, "nbrntd_sock_addr": null, "hand_back": false, "request_timeout": 310}}, "proxy_response": {"timestamp": 1530560396, "who": "vnet_proxy_parse_response", "line_number": 3455, "comment": "response from local proxy", "data": {"status": 0, "local_proxy_info": {}, "domain_constraints_set": {"process_hint": "4b354b21-7de6-4e91-9f85-bfcbc9369a88", "process_hint_reason": "there was no inherited hint, the connected peer was not a master server, so defaulted to primary master", "process_hint_server_name": "MASTER_SERVER"}, "statusmsgxml": "<?xml version=\"1.0\" encoding=\"utf-8\"?><StatusMsgDoc><StatusMsg NBUStatus=\"0\" Severity=\"0\"></StatusMsg></StatusMsgDoc>"}}, "status": {"timestamp": 1530560396, "who": "connect_to_service", "line_number": 981, "comment": "vnet status", "data": 0}}
<16>bptestbpcd main: Function ConnectToBPCD(RHEL_CLIENT02) failed: 21
15:41:56.834 [3266] <16> bptestbpcd main: Function ConnectToBPCD(RHEL_CLIENT02) failed: 21
<16>bptestbpcd main: socket open failed
15:41:56.839 [3266] <16> bptestbpcd main: socket open failed
<2>bptestbpcd: socket open failed
15:41:56.839 [3266] <2> bptestbpcd: socket open failed
<2>bptestbpcd: EXIT status = 21
15:41:56.839 [3266] <2> bptestbpcd: EXIT status = 21
socket open failed


===================================================================================

8 REPLIES 8

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
Do you have bpcd log on the RHEL client?

Mouse
Moderator
Moderator
Partner    VIP    Accredited Certified

Sounds like a DNS issue to me. I would check reverse lookup zone on the server and on the client

Here is the bpcd log from the client after trying to connect this morning.

 

09:07:39.017 [9732] <2> vnet_pbxAcceptSocket: Accepted sock[12] from 10.250....:40597
09:07:39.018 [9732] <2> bpcd main: accept sock = 12
09:07:39.019 [8158] <2> vnet_check_resilient_socket: [vnet_nbrntd.c:804] the socket is 0 0x0
09:07:39.019 [8158] <2> vnet_check_resilient_socket: [vnet_nbrntd.c:804] the socket is 1 0x1
09:07:39.019 [8158] <2> vnet_check_resilient_socket: [vnet_nbrntd.c:804] the socket is 2 0x2
09:07:39.019 [8158] <2> ReadKeyfile: keyfile /usr/openv/netbackup/keyfile does not exist
09:07:39.019 [8158] <2> process_requests: offset to GMT 18000
09:07:39.019 [8158] <2> logconnections: BPCD ACCEPT FROM 10.250.....40597 TO 10.250......1556 fd = 0
09:07:39.020 [8158] <2> process_requests: setup_sockopts complete
09:07:39.039 [8158] <2> vnet_pcache_init_table: [vnet_private.c:214] starting cache size 200 0xc8
09:07:39.048 [8158] <2> vnet_cached_getnameinfo: [vnet_addrinfo.c:2023] found via getnameinfo OUR_HOST=MASTER_SERVER IPSTR=10.250....
09:07:39.073 [8158] <2> print_arp_info: MASTER_SERVER (10.250....) -- no entry
09:07:39.073 [8158] <2> bpcd peer_hostname: Connection from host MASTER_SERVER (10.250....) port 40597
09:07:39.073 [8158] <2> bpcd valid_server: comparing MASTER_SERVER and MASTER_SERVER
09:07:39.073 [8158] <4> bpcd valid_server: hostname comparison succeeded
09:07:39.115 [8158] <2> process_requests: output socket port number = 1
09:09:39.281 [8158] <8> vnet_pop_byte: [vnet.c:199] errno 104 0x68
09:09:39.281 [8158] <2> vnet_pop_byte: vnet.c.201: 0: Function failed: 43 0x0000002b
09:09:39.281 [8158] <2> vnet_pop_string: vnet.c.281: 0: Function failed: 43 0x0000002b
09:09:39.281 [8158] <8> vnet_receive_network_socket: [vnet_vnetd.c:1077] status 43 0x2b
09:09:39.281 [8158] <2> get_vnetd_forward_socket: vnet_receive_network_socket failed: 43
09:09:39.281 [8158] <16> process_requests: get_vnetd_forward_socket failed: 21
09:09:39.296 [9732] <2> vnet_pbxAcceptSocket: Accepted sock[12] from 10.250....:63650
09:09:39.296 [9732] <2> bpcd main: accept sock = 12
09:09:39.298 [8408] <2> vnet_check_resilient_socket: [vnet_nbrntd.c:804] the socket is 0 0x0
09:09:39.298 [8408] <2> vnet_check_resilient_socket: [vnet_nbrntd.c:804] the socket is 1 0x1
09:09:39.298 [8408] <2> vnet_check_resilient_socket: [vnet_nbrntd.c:804] the socket is 2 0x2
09:09:39.298 [8408] <2> ReadKeyfile: keyfile /usr/openv/netbackup/keyfile does not exist
09:09:39.298 [8408] <2> process_requests: offset to GMT 18000
09:09:39.298 [8408] <2> logconnections: BPCD ACCEPT FROM 10.250.....63650 TO 10.250......1556 fd = 0
09:09:39.299 [8408] <2> process_requests: setup_sockopts complete
09:09:39.299 [8408] <2> vnet_pcache_init_table: [vnet_private.c:214] starting cache size 200 0xc8
09:09:39.310 [8408] <2> vnet_cached_getnameinfo: [vnet_addrinfo.c:2023] found via getnameinfo OUR_HOST=MASTER_SERVER IPSTR=10.250....
09:09:39.331 [8408] <2> print_arp_info: MASTER_SERVER (10.250....) -- no entry
09:09:39.332 [8408] <2> bpcd peer_hostname: Connection from host MASTER_SERVER (10.250....) port 63650
09:09:39.332 [8408] <2> bpcd valid_server: comparing MASTER_SERVER and MASTER_SERVER
09:09:39.332 [8408] <4> bpcd valid_server: hostname comparison succeeded
09:09:39.332 [8408] <2> process_requests: output socket port number = 1
09:09:39.447 [9732] <2> daemon_check_for_zombies: waited for 1 child processes including: 8158
09:11:39.474 [8408] <8> vnet_pop_byte: [vnet.c:199] errno 104 0x68
09:11:39.474 [8408] <2> vnet_pop_byte: vnet.c.201: 0: Function failed: 43 0x0000002b
09:11:39.474 [8408] <2> vnet_pop_string: vnet.c.281: 0: Function failed: 43 0x0000002b
09:11:39.474 [8408] <8> vnet_receive_network_socket: [vnet_vnetd.c:1077] status 43 0x2b
09:11:39.474 [8408] <2> get_vnetd_forward_socket: vnet_receive_network_socket failed: 43
09:11:39.474 [8408] <16> process_requests: get_vnetd_forward_socket failed: 21
09:11:39.481 [9732] <2> daemon_check_for_zombies: waited for 1 child processes including: 8408
09:12:39.541 [9732] <16> resync_host_cache: bpclntcmd fork pid = 10092
09:12:39.553 [10094] <2> unix_daemonize: 0 = close(4) errno = 2 mode = 0x000021b6
09:12:39.554 [10094] <2> unix_daemonize: 0 = close(5) errno = 2 mode = 0x00008180
09:12:39.561 [10094] <2> unix_daemonize: 0 = close(6) errno = 2 mode = 0x0000c1ff
09:12:39.561 [10094] <2> unix_daemonize: 0 = close(7) errno = 2 mode = 0x0000c1ff
09:12:39.561 [10094] <2> unix_daemonize: 0 = close(8) errno = 2 mode = 0x0000c1ff
09:12:39.561 [10094] <2> unix_daemonize: 0 = close(9) errno = 2 mode = 0x0000c1ff
09:12:39.561 [10094] <2> unix_daemonize: 0 = close(10) errno = 2 mode = 0x0000c1ff
09:12:39.561 [10094] <2> unix_daemonize: 0 = close(11) errno = 2 mode = 0x0000c1ff
09:12:39.562 [10094] <16> resync_host_cache: about to execv(/usr/openv/netbackup/bin/bpclntcmd, ...)
09:13:39.621 [9732] <2> daemon_check_for_zombies: waited for 1 child processes including: 10092

dns is not the issue.  nslookup works forward and reverse.  The clients worked fine before upgrading the master and media server from 7.7.2 to 8.1

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

We see status 43 just before status 21:

09:09:39.281 [8158] <2> get_vnetd_forward_socket: vnet_receive_network_socket failed: 43
09:09:39.281 [8158] <16> process_requests: get_vnetd_forward_socket failed: 21

Status 43:
Message: unexpected message received
Explanation: The client and the server handshake were not correct.
Recommended Action: Do the following, as appropriate:
β–  Verify that the correct version of software is running on the client and the server.
β–  To enable detailed debug logging, do the following:
β–  On the server, create a bpbrm debug log directory.
β–  On clients, create a bpcd debug log directory.
β–  Increase the amount of debug information to include in the logs.

I suggest that you log a Support call with Veritas.
Be sure to have level 5 bpbrm (on media server) and bpcd (on client) when logging the call. 

PS: 
The secure comms in NBU 8.x is still a mystery to me, but you may want to check if 'insecure comms' is enabled on your master server.

See https://www.veritas.com/support/en_US/doc/125240132-127350446-0/v127792335-127350446

About certificate requirements for NetBackup 8.1

Starting with NetBackup 8.1, Veritas requires all communications among 8.1 systems be secure. A certificate is required for communication among 8.1 systems. By default, insecure communication with back-level media servers, Auto Image Replication master servers, and clients is enabled. Once all computers in your environment are upgraded to 8.1, Veritas suggests you disable further insecure communication. See the section on enabling insecure communication in the Veritas NetBackup Security and Encryption Guide.

NetBackup Security and Encryption Guide 

The client is at 7.7.2

I have a support call open and had a webex session.  So far the tech is stumped as well.  By all accounts, it should be working. 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Please double-check that 'insecure communication ' is enabled on your master server.

Insecure communication is enabled, that was the first thing I checked