Backup failure with error code 24 & 58

[BalajiP]

Hi,

We have a new customer onboarded who has the Master Server NBU version 7.0.1 on Windows 2003 R2 Std. And the client server NBU version 6.5.4 on Windows 2003
R2 Std. I know, both are EOSL(End Of Support Life). So, I have suggested the customer to upgrade the environment which will be delayed due to
critical legacy applications' compatibility. But, I need to fix this backup failure issue on high priority.

Coming to the issue, the client was getting backed up till 16th July, 2017. All of a sudden, the backup failed with error 58. We checked and found
that the server went to hung state, due to which we are unable to login. We had then fixed login issue with the help of Wintel team, but the bakcup
failure still persists.

Below are errors and checks I have followed, as per Technotes.

1. Able to ping and nslookup bidirectionally
2. Firewall is disabled on the client server.
3. Able to telnet the 13782, 1556 bidirectionally between Master and Client server.
4. Added host entries in Master and Client server's hosts file
5. Created bpcd folder on client and ran a backup and 'bptestbpcd' command. But, it did not create any log file.
6. Restarted NetBackup client service and PBX service on the client server
7. No error on excuting the commands 'bpclntcmd -self', 'bpclntcmd -hn <hostname>' and 'bpclntcmd -ip 'ipaddress' on both media server and client server
8. Cleared cache on media server and unable to clear the cache in client as no option on 6.5.4
9. Error details: There is change in teh output, so provided twice for more clarification

C:\Program Files\Veritas\NetBackup\bin\admincmd>bptestbpcd -client bbbbb -verbose -debug
15:11:10.774 [29496.46924] <2> bptestbpcd: VERBOSE = 0
15:11:10.837 [29496.46924] <2> vnet_pbxConnect: ../../libvlibs/vnet_pbx.c.656: pbxSetAddrEx/pbxConnectEx return error 10054:A
n existing connection was forcibly closed by the remote host.
15:11:10.837 [29496.46924] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1658: 0: vnet_pbxConnect() failed: 18 0x00000012

15:11:10.837 [29496.46924] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1659: 0: save_errno: 10054 0x00002746
15:11:10.837 [29496.46924] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1660: 0: use_vnetd: 0 0x00000000
15:11:10.837 [29496.46924] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1661: 0: cr->vcr_service: bpcd
15:11:10.837 [29496.46924] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1293: 0: do_service failed: 18 0x00000012
15:11:10.853 [29496.46924] <2> vnet_pop_byte: ../../libvlibs/vnet.c.187: 0: errno: 10054 0x00002746
15:11:10.853 [29496.46924] <2> vnet_pop_byte: ../../libvlibs/vnet.c.189: 0: Function failed: 43 0x0000002b
15:11:10.853 [29496.46924] <2> vnet_pop_string: ../../libvlibs/vnet.c.269: 0: Function failed: 43 0x0000002b
15:11:10.853 [29496.46924] <2> vnet_pop_signed: ../../libvlibs/vnet.c.313: 0: Function failed: 43 0x0000002b
15:11:10.853 [29496.46924] <2> vnet_version_connect: ../../libvlibs/vnet_vnetd.c.1823: 0: Function failed: 43 0x0000002b
15:11:10.853 [29496.46924] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1550: 0: vnet_version_connect failed: 43 0x000
0002b
15:11:10.853 [29496.46924] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1293: 0: do_service failed: 43 0x0000002b
15:11:10.868 [29496.46924] <2> logconnections: BPCD CONNECT FROM 0.0.0.0.985 TO 10.21.1.35.13782 fd = 1628
15:11:10.868 [29496.46924] <2> bpcr_authenticate_connection: bpcr protocol error - couldn't send bpcd magic number
15:11:10.868 [29496.46924] <2> local_bpcr_connect: bpcr_authenticate_connection() failed: socket write failed
15:11:10.868 [29496.46924] <2> ConnectToBPCD: bpcd_connect_and_verify(bbbbb, bbbbbb) failed: 24
<16>bptestbpcd main: Function ConnectToBPCD(bbbbbbb) failed: 24
15:11:10.868 [29496.46924] <16> bptestbpcd main: Function ConnectToBPCD(bbbbbb) failed: 24
<2>bptestbpcd: socket write failed
15:11:10.868 [29496.46924] <2> bptestbpcd: socket write failed
<2>bptestbpcd: EXIT status = 24
15:11:10.868 [29496.46924] <2> bptestbpcd: EXIT status = 24
socket write failed
**************************************************************************************************
C:\Program Files\Veritas\NetBackup\bin\admincmd>bptestbpcd -client bbbbbb -verbose -debug
15:13:05.539 [46384.44488] <2> bptestbpcd: VERBOSE = 0
15:13:05.571 [46384.44488] <2> vnet_pbxConnect: ../../libvlibs/vnet_pbx.c.656: pbxSetAddrEx/pbxConnectEx return error 10054:A
n existing connection was forcibly closed by the remote host.

[Marianne · Accepted solution]

Can we assume that it was the client that was in a hung state?

Something may have been corrupted on the client - either with NetBackup or the network stack.

Please try to remove the 6.5 Client software and at least install NBU 7.0.1.
This way we know that NBU Comms works exactly the same on client and server (port 1556 used since 7.0.1) and will rule out NBU being at fault if the issue persists.

The problem with unsupported OS is that patches and updates are no longer available. 
So, no chance of reinstalling NIC drivers, etc...

[BalajiP]

Hi Marianne, Thank you very much for your reply.

Yes, it was the same client which went hung state and had started failing since then.

Sure, i will upgrade the client to the masters server version 7.0.1 and will let you the status. As i need to proceed with approvals, there may be delay in providing update on upgrade.

Yes, agree with you and the server is not being patched since long time and however, i will check with team if we can update any drivers or patches if required.

[BalajiP]

Hi Marianne,

I believe, the output of bptestbpcd command was not properly uploaded. As i said, there was a changes in the output, uploaded the both.

C:\Program Files\Veritas\NetBackup\bin\admincmd>bptestbpcd -client bbbbb -verbose -debug
15:11:10.774 [29496.46924] <2> bptestbpcd: VERBOSE = 0
15:11:10.837 [29496.46924] <2> vnet_pbxConnect: ../../libvlibs/vnet_pbx.c.656: pbxSetAddrEx/pbxConnectEx return error 10054:A
n existing connection was forcibly closed by the remote host.
15:11:10.837 [29496.46924] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1658: 0: vnet_pbxConnect() failed: 18 0x00000012

15:11:10.837 [29496.46924] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1659: 0: save_errno: 10054 0x00002746
15:11:10.837 [29496.46924] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1660: 0: use_vnetd: 0 0x00000000
15:11:10.837 [29496.46924] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1661: 0: cr->vcr_service: bpcd
15:11:10.837 [29496.46924] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1293: 0: do_service failed: 18 0x00000012
15:11:10.853 [29496.46924] <2> vnet_pop_byte: ../../libvlibs/vnet.c.187: 0: errno: 10054 0x00002746
15:11:10.853 [29496.46924] <2> vnet_pop_byte: ../../libvlibs/vnet.c.189: 0: Function failed: 43 0x0000002b
15:11:10.853 [29496.46924] <2> vnet_pop_string: ../../libvlibs/vnet.c.269: 0: Function failed: 43 0x0000002b
15:11:10.853 [29496.46924] <2> vnet_pop_signed: ../../libvlibs/vnet.c.313: 0: Function failed: 43 0x0000002b
15:11:10.853 [29496.46924] <2> vnet_version_connect: ../../libvlibs/vnet_vnetd.c.1823: 0: Function failed: 43 0x0000002b
15:11:10.853 [29496.46924] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1550: 0: vnet_version_connect failed: 43 0x000
0002b
15:11:10.853 [29496.46924] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1293: 0: do_service failed: 43 0x0000002b
15:11:10.868 [29496.46924] <2> logconnections: BPCD CONNECT FROM 0.0.0.0.985 TO 10.21.1.35.13782 fd = 1628
15:11:10.868 [29496.46924] <2> bpcr_authenticate_connection: bpcr protocol error - couldn't send bpcd magic number
15:11:10.868 [29496.46924] <2> local_bpcr_connect: bpcr_authenticate_connection() failed: socket write failed
15:11:10.868 [29496.46924] <2> ConnectToBPCD: bpcd_connect_and_verify(bbbbb, bbbbbb) failed: 24
<16>bptestbpcd main: Function ConnectToBPCD(bbbbbbb) failed: 24
15:11:10.868 [29496.46924] <16> bptestbpcd main: Function ConnectToBPCD(bbbbbb) failed: 24
<2>bptestbpcd: socket write failed
15:11:10.868 [29496.46924] <2> bptestbpcd: socket write failed
<2>bptestbpcd: EXIT status = 24
15:11:10.868 [29496.46924] <2> bptestbpcd: EXIT status = 24
socket write failed
**************************************************************************************************
C:\Program Files\Veritas\NetBackup\bin\admincmd>bptestbpcd -client bbbbbb -verbose -debug
15:13:05.539 [46384.44488] <2> bptestbpcd: VERBOSE = 0
15:13:05.571 [46384.44488] <2> vnet_pbxConnect: ../../libvlibs/vnet_pbx.c.656: pbxSetAddrEx/pbxConnectEx return error 10054:A
n existing connection was forcibly closed by the remote host.
15:13:05.571 [46384.44488] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1658: 0: vnet_pbxConnect() failed: 18 0x00000012

15:13:05.571 [46384.44488] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1659: 0: save_errno: 10054 0x00002746
15:13:05.571 [46384.44488] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1660: 0: use_vnetd: 0 0x00000000
15:13:05.571 [46384.44488] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1661: 0: cr->vcr_service: bpcd
15:13:05.571 [46384.44488] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1293: 0: do_service failed: 18 0x00000012
15:13:05.571 [46384.44488] <2> vnet_pop_byte: ../../libvlibs/vnet.c.187: 0: errno: 10054 0x00002746
15:13:05.571 [46384.44488] <2> vnet_pop_byte: ../../libvlibs/vnet.c.189: 0: Function failed: 43 0x0000002b
15:13:05.571 [46384.44488] <2> vnet_pop_string: ../../libvlibs/vnet.c.269: 0: Function failed: 43 0x0000002b
15:13:05.571 [46384.44488] <2> vnet_pop_signed: ../../libvlibs/vnet.c.313: 0: Function failed: 43 0x0000002b
15:13:05.571 [46384.44488] <2> vnet_version_connect: ../../libvlibs/vnet_vnetd.c.1823: 0: Function failed: 43 0x0000002b
15:13:05.571 [46384.44488] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1550: 0: vnet_version_connect failed: 43 0x000
0002b
15:13:05.571 [46384.44488] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1293: 0: do_service failed: 43 0x0000002b
15:13:05.571 [46384.44488] <2> logconnections: BPCD CONNECT FROM 10.21.1.39.612 TO 10.21.1.35.13782 fd = 1656
15:13:05.586 [46384.44488] <2> vnet_pop_byte: ../../libvlibs/vnet.c.187: 0: errno: 10054 0x00002746
15:13:05.586 [46384.44488] <2> vnet_pop_byte: ../../libvlibs/vnet.c.189: 0: Function failed: 43 0x0000002b
15:13:05.586 [46384.44488] <2> vnet_pop_string: ../../libvlibs/vnet.c.269: 0: Function failed: 43 0x0000002b
15:13:05.586 [46384.44488] <2> vnet_pop_signed: ../../libvlibs/vnet.c.313: 0: Function failed: 43 0x0000002b
15:13:05.586 [46384.44488] <2> vnet_pop_status: ../../libvlibs/vnet.c.391: 0: Function failed: 43 0x0000002b
15:13:05.586 [46384.44488] <2> vnet_begin_connect_back: ../../libvlibs/vnet_vnetd.c.728: 0: status: 43 0x0000002b
15:13:05.586 [46384.44488] <2> ConnectToBPCD: bpcd_connect_and_verify(bbbbb, bbbbbb) failed: 25
<16>bptestbpcd main: Function ConnectToBPCD(bbbbbb) failed: 25
15:13:05.586 [46384.44488] <16> bptestbpcd main: Function ConnectToBPCD(bbbbbb) failed: 25
<2>bptestbpcd: cannot connect on socket
15:13:05.586 [46384.44488] <2> bptestbpcd: cannot connect on socket
<2>bptestbpcd: EXIT status = 25
15:13:05.586 [46384.44488] <2> bptestbpcd: EXIT status = 25
cannot connect on socket

[Thiago_Ribeiro]

Hi,

The first error message from bptestbpcd commands was error 10054:An existing connection was forcibly closed by the remote host, that is associeted with status code 13. This kind of error points to some problem on server side and not NBU side. For this error take a look on these TNS.

Status 24 > https://www.veritas.com/support/en_US/article.000011538

Status 13 - https://www.veritas.com/support/en_US/article.TECH37372

Regards,

Thiago

C:\Program Files\Veritas\NetBackup\bin\admincmd>bptestbpcd -client bbbbb -verbose -debug
15:11:10.774 [29496.46924] <2> bptestbpcd: VERBOSE = 0
15:11:10.837 [29496.46924] <2> vnet_pbxConnect: ../../libvlibs/vnet_pbx.c.656: pbxSetAddrEx/pbxConnectEx return error 10054:An existing connection was forcibly closed by the remote host.
15:11:10.837 [29496.46924] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1658: 0: vnet_pbxConnect() failed: 18 0x00000012

[BalajiP]

Hi All,

Finally i got an approval to upgrade the client server from 6.5.4 to 7.0.1 and it is scheduled on 3rd Aug. As the customer does not have the binaries of 7.0.1, please anyone suggest me the site or link to download the binaries of 7.0.1 as i do not want to download the binaries from untrusted site. Thanks.

[andrew_mcc1]

Well the 7.0.1 binaries are still available from Veritas at https://www.veritas.com/support/en_US/article.000008727 _BUT_ first you need the 7.0 Base Install Media which is no longer available. Unless you or the customer has these binaries you probably out of luck (not to say very unsupported...)

Sorry, Andrew 

[Thiago_Ribeiro]

Hi

So as this version no more supported, is difficult to find binaries..As your S.O is Windows 2003 R2 Standard, why dont try to update for at least NBU 7.6 version?

Of the most recent versions of the NBU, this is the last one that supports Windows 2003. Take a look

Regards,

Thiago Ribeiro

[Marianne]

7.6 client will not work with 7.0.1 master.

Your client should have base installation media.

Your only other option is to reach out to your local Veritas office and Veritas reseller to see if one of them perhaps have 7.0 software in an 'archive'.

[Michael_G_Ander]

Things i have seen cause 10054 on Windows are

1. Not properly configured network 

2. Security Patches on the OS (Can some times be "fixed" by repairing netbackup)

3. Anti-virus

4. VSS error

5. Firewalls (Did see that the client one was disabled)

Edit just saw that the client had hung, which could be indicative of bad hardware, which certainly also can cause problems with the backups.

Have there been run chkdsk after the hang ? Bad folder/files can also causes issue like this

[BalajiP · Accepted solution]

Thanks Marianne. The issue has been resolved after removing the 6.5.4 client software and installing 7.0.1. Also rebooted the server prior installation. Now the backup is running fine.

Thanks everyone for your great inputs.