backup is failing with error (112) no files specified in the file list.

[S_Surya]

We run monthly backup for a client, which was running OK till last backup.. now it is failing with error (112) no files specified in the file list. We have not made any changes to the configuration.

Master & Media server is Linux and the client is Windows. 

Under host properties > clients > Client says " cannot connect to the socket" 

I am able to ping and telnet from both ports 1556 and 13724

bptestbpcd Logs:

[root@nbmasterrk1 admincmd]# ./bptestbpcd -client rkxaapp2 -debug
03:46:59.102 [24364] <2> bptestbpcd: VERBOSE = 5
03:46:59.102 [24364] <2> read_client: dname=CO_0, offline=0, online_at=0 offline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
03:46:59.102 [24364] <2> read_client: dname=OA_0, offline=0, online_at=0 offline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
03:46:59.102 [24364] <2> read_client: dname=.., offline=0, online_at=0 offline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
03:46:59.102 [24364] <2> read_client: dname=., offline=0, online_at=0 offline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
03:46:59.102 [24364] <2> read_client: dname=COA_0, offline=0, online_at=0 offline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
03:46:59.102 [24364] <2> read_client: dname=host_info, offline=0, online_at=0 offline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
03:46:59.102 [24364] <2> read_client: dname=OAA_0, offline=0, online_at=0 offline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
03:46:59.119 [24364] <2> vnet_pbxConnect: ../../libvlibs/vnet_pbx.c.654: pbxSetAddrEx/pbxConnectEx return error 104:Connection reset by peer
03:46:59.119 [24364] <8> do_pbx_service: [vnet_connect.c:2231] vnet_pbxConnect() failed, status=18, errno=104, use_vnetd=0, cr->vcr_service=bpcd
03:46:59.119 [24364] <8> async_connect: [vnet_connect.c:1823] do_service failed 18 0x12
03:46:59.481 [24364] <8> vnet_vnetd_pbx_c_supported: [vnet_vnetd.c:3510] VN_REQUEST_PBX_C_SUPPORTED 13 0xd
03:46:59.725 [24364] <8> do_vnetd_service: [vnet_connect.c:2152] remote host supports PBX, but PBX is not running 0 0x0
03:46:59.726 [24364] <8> vnet_vnetd_service_socket: [vnet_vnetd.c:2032] VN_REQUEST_SERVICE_SOCKET 6 0x6
03:46:59.726 [24364] <8> vnet_vnetd_service_socket: [vnet_vnetd.c:2046] service bpcd
03:46:59.942 [24364] <2> logconnections: BPCD CONNECT FROM 192.168.44.80.56224 TO 192.168.72.142.13724 fd = 3
03:46:59.948 [24364] <2> vnet_pbxConnect: pbxConnectEx Succeeded
03:47:00.397 [24364] <8> do_pbx_service: [vnet_connect.c:2302] via PBX VNETD CONNECT FROM 192.168.44.80.58537 TO 192.168.72.142.1556 fd = 4
03:47:00.397 [24364] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:455] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
03:47:00.630 [24364] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:480] ipc_string 65471
03:47:00.956 [24364] <2> get_long_base: (1) cannot read (byte 1) from network: Connection reset by peer (104)
03:47:00.956 [24364] <2> local_bpcr_connect: bpcd protocol error - failed to read connected message.
03:47:00.956 [24364] <2> ConnectToBPCD: bpcd_connect_and_verify(rkxaapp2, rkxaapp2) failed: 0
<16>bptestbpcd main: Function ConnectToBPCD(rkxaapp2) failed: 25
03:47:00.957 [24364] <16> bptestbpcd main: Function ConnectToBPCD(rkxaapp2) failed: 25
<16>bptestbpcd main: cannot connect on socket
03:47:00.965 [24364] <16> bptestbpcd main: cannot connect on socket
<2>bptestbpcd: cannot connect on socket
03:47:00.965 [24364] <2> bptestbpcd: cannot connect on socket
<2>bptestbpcd: EXIT status = 25
03:47:00.965 [24364] <2> bptestbpcd: EXIT status = 25
cannot connect on socket

[Marianne]

Can you really telnet on master to 1556 on the client?

This message says that PBX is not running on the client:

... remote host supports PBX, but PBX is not running

Please check on the client if PBX and NBU Client Services are running.

[S_Surya]

Yes, I am able to telnet and that is what is confusing

nc -v rkxaapp2 1556
Connection to rkxaapp2 1556 port [tcp/veritas_pbx] succeeded!

[Marianne]

Do you have bpcd log folder on the client?
Is anything written to bpcd log file during connect attempt?

[Anshu_Pathak]

If telnet is working, then there is some problem with the client. Couple of things that you can try before getting into logs.

1. Stop-Check(bpps)-Start NetBackup services on client - Normally issue gets resolved by this if services were in not responding state.

2. Reboot client - Some third part (non Veritas) software might be cauing issue during vnetd to bpcd port forward.

3. If it is NetBackup 7.x you can try connect options and check if daemon port connection works. If it works then it will confirm issue mentioned in point# 2

Example: On master server - bptestbpcd -connect_options 1 1 2 -host rkxaapp2

If these steps do not work then you can get into logs on client (all logs from client).

PBX (to check if master/media connection ever reached to PBX) - already enabled - process it using "C:\Program Files (x86)\VERITAS\VxPBX\bin\vxlogview.exe" -p 50936 -i 103 -d all  (Search for master/media server IP address in the output)

7/24/2018 22:58:08.746 [Info] PBX_Client_Proxy::parse_line, line = extension=EMM From 192.168.2.43, Port 61639
7/24/2018 22:58:08.746 [Info] hand_off looking for proxy for = EMM

bpcd & vnetd logs -

C:\Program Files\Veritas\NetBackup\logs\bpcd

C:\Program Files\Veritas\NetBackup\logs\vnetd

From my experience, status 25 issue is mostly due to firewall (hardware / software), third party software or OS issue.

[Marianne]

I agree with @Anshu_Pathak

We also see this in the log - inital connection succeeded: 

03:47:00.397 [24364] <8> do_pbx_service: [vnet_connect.c:2302] via PBX VNETD CONNECT FROM 192.168.44.80.58537 TO 192.168.72.142.1556 fd = 4
....
03:47:00.956 [24364] <2> get_long_base: (1) cannot read (byte 1) from network: Connection reset by peer (104)
03:47:00.956 [24364] <2> local_bpcr_connect: bpcd protocol error - failed to read connected message.
03:47:00.956 [24364] <2> ConnectToBPCD: bpcd_connect_and_verify(rkxaapp2, rkxaapp2) failed: 0

So, it seems that 'something' on the client dropped the connection.
Could be security software on the client (some anti-virus software do more than just virus checking).