Exit Status 58 - New message what I've never seen

[Nithin_S1]

Offlate I'm seeing many clients failing with exit statys '58' though configurations looks good for me. Primary error message from the master server is

status: FAILED, (11) TIMEOUT; system: (10036) A blocking operation is cu

rrently executing. ; FROM 0.0.0.0 TO 10.186.242.56 10.186.242.56 vnetd VIA pbx

        status: FAILED, (11) TIMEOUT; system: (10036) A blocking operation is cu

rrently executing. ; FROM 0.0.0.0 TO 10.186.242.56 10.186.242.56 vnetd VIA vnetd

status: FAILED, (11) TIMEOUT; system: (10036) A blocking operation is cu

rrently executing. ; FROM 0.0.0.0 TO 10.186.242.56 10.186.242.56 vnetd VIA pbx

        status: FAILED, (11) TIMEOUT; system: (10036) A blocking operation is cu

rrently executing. ; FROM 0.0.0.0 TO 10.186.242.56 10.186.242.56 vnetd VIA vnetd

Verified the logs and it mostly looks for me a client level firewall issue but I disabled all the firewall with no luck and same error. Any help is much appreciated.

Here are the test bpcd logs from master and corresponding logs from the client.

TEST-BPCD from master results:

D:\Veritas\NetBackup\bin\admincmd>bptestbpcd -client lcspepas56rnb -verbose -deb

ug

11:30:28.281 [4868.9624] <2> bptestbpcd: VERBOSE = 0

11:30:28.294 [4868.9624] <2> vnet_pbxConnect: pbxConnectEx Succeeded

11:30:28.296 [4868.9624] <2> logconnections: BPCD CONNECT FROM 10.186.240.20.586

37 TO 10.186.242.56.1556 fd = 608

11:30:28.309 [4868.9624] <2> vnet_pbxConnect: pbxConnectEx Succeeded

11:35:28.323 [4868.9624] <8> vnet_sock_ready: [vnet.c:1485] max_time=300 sock=63

11:35:28.340 [4868.9624] <16> connect_to_service: connect failed STATUS (18) CON

NECT_FAILED

        status: FAILED, (11) TIMEOUT; system: (10036) A blocking operation is cu

rrently executing. ; FROM 0.0.0.0 TO 10.186.242.56 10.186.242.56 vnetd VIA pbx

        status: FAILED, (11) TIMEOUT; system: (10036) A blocking operation is cu

rrently executing. ; FROM 0.0.0.0 TO 10.186.242.56 10.186.242.56 vnetd VIA vnetd

11:35:28.341 [4868.9624] <8> vnet_connect_to_vnetd_ipi: [vnet_connect.c:258] con

nect_to_service() failed 18 0x12

11:35:28.341 [4868.9624] <16> bpcr_vnetd_connect_forward_socket_begin: vnet_conn

ect_to_vnetd_ipi(10.186.242.56) failed: 18

11:35:28.341 [4868.9624] <2> local_bpcr_connect: bpcr_vnetd_connect_forward_sock

et_begin failed: 25

11:35:28.342 [4868.9624] <2> ConnectToBPCD: bpcd_connect_and_verify(lcspepas56rn

b, lcspepas56rnb) failed: 25

<16>bptestbpcd main: Function ConnectToBPCD(lcspepas56rnb) failed: 25

11:35:28.343 [4868.9624] <16> bptestbpcd main: Function ConnectToBPCD(lcspepas56

rnb) failed: 25

<2>bptestbpcd: cannot connect on socket

11:35:28.343 [4868.9624] <2> bptestbpcd: cannot connect on socket

<2>bptestbpcd: EXIT status = 25

11:35:28.343 [4868.9624] <2> bptestbpcd: EXIT status = 25

cannot connect on socket

Client BPCD logs for the above test:

14:10:12.282 [2452.7104] <2> logparams: C:\Program Files\Veritas\NetBackup\bin\bpcd.exe -standalone

14:10:12.282 [2452.7104] <2> CrankUpNTPrivs: About to EnablePrivilege(SeBackupPrivilege)

14:10:12.282 [2452.7104] <2> CrankUpNTPrivs: About to EnablePrivilege(SeDebugPrivilege)

14:10:12.282 [2452.7104] <2> CrankUpNTPrivs: About to EnablePrivilege(SeRestorePrivilege)

14:10:12.282 [2452.7104] <2> CrankUpNTPrivs: About to EnablePrivilege(SeSecurityPrivilege)

14:10:12.282 [2452.7104] <2> CrankUpNTPrivs: About to EnablePrivilege(SeTakeOwnershipPrivilege)

14:10:12.282 [2452.7104] <2> CrankUpNTPrivs: About to EnablePrivilege(SeTcbPrivilege)

14:10:12.282 [2452.7104] <2> setup_debug_log: switching debug log file for bpcd

14:10:12.282 [2452.7104] <2> setup_debug_log: switched debug log file for bpcd

14:10:12.298 [2452.7104] <2> daemon_startup_listeners: 4 listening for legacy service bpcd

14:10:12.298 [2452.7104] <2> daemon_startup_listeners: listening for vnetd service bpcd

14:10:12.313 [2452.7104] <2> vnet_registerPBXServer: ../../libvlibs/vnet_pbx.c.132: pbxRegisterEx successful at 10.186.242.56:5126/bpcd, returns with 1 alt_addrs

14:10:12.313 [2452.7104] <2> vnet_registerPBXServer: ../../libvlibs/vnet_pbx.c.144: alt_addr: 10.186.163.91:5126

14:15:31.581 [2452.7104] <2> vnet_pbxAcceptSocket: invalid socket passed to getpeername 10022:An invalid argument was supplied.  )

14:30:34.868 [2452.7104] <2> vnet_pbxAcceptSocket: invalid socket passed to getpeername 10022:An invalid argument was supplied.  )

14:32:59.291 [2452.7104] <2> vnet_pbxAcceptSocket: invalid socket passed to getpeername 10022:An invalid argument was supplied.  )

14:35:35.057 [2452.7104] <2> vnet_pbxAcceptSocket: invalid socket passed to getpeername 10022:An invalid argument was supplied.  )

Client VNETD logs for above test:

12:58:33.645 [984.852] <2> main: VERBOSE = 0

12:58:33.645 [984.852] <2> logparams: C:\Program Files\Veritas\NetBackup\bin\vnetd.exe -standalone

12:58:33.661 [984.852] <8> ProcessRequests: [.\vnetd.c:635] msg VNETD ACCEPT FROM 10.186.240.20.59478 TO 10.186.242.56.13724 fd = 520

12:58:33.661 [984.852] <8> vnet_pop_byte: [vnet.c:1168] errno 0 0x0

12:58:33.661 [984.852] <2> vnet_pop_byte: vnet.c.1170: 0: Function failed: 9 0x00000009

12:58:33.661 [984.852] <2> vnet_pop_string: vnet.c.1250: 0: Function failed: 9 0x00000009

12:58:33.661 [984.852] <2> vnet_pop_signed: vnet.c.1293: 0: Function failed: 9 0x00000009

12:58:33.661 [984.852] <2> vnet_version_accept: .\vnetd.c.1089: 0: Function failed: 9 0x00000009

12:58:33.661 [984.852] <8> ProcessRequests: [.\vnetd.c:640] version_accept failed 9 0x9

12:58:33.661 [984.852] <8> main: [.\vnetd.c:405] ProcessRequests failed 9 0x9

12:58:33.661 [984.852] <8> main: [.\vnetd.c:406] ProcessRequests returned 9 0x9

13:06:33.804 [5624.684] <2> setup_debug_log: switched debug log file for vnetd

13:06:33.804 [5624.684] <2> main: VERBOSE = 0

[RamNagalla]

Does your windows filrewall is turned off.. 

its more likely fire wall issue?

[Marianne]

Couple of questions:

Has this worked previously?
Or is this a new installation that has never worked before?

Have you confirmed that Windows Firewall is disabled on this machine?

Have you checked for third-party software that could be blocking or preventing port connection and/or socket setup?
Some Anti-Virus software is doing this. We have even seen that Download Manager sofware was preventing socket setup.
Some other ideas as to what third-part software to check for:
http://www.symantec.com/docs/TECH88628

We don't see TCP/IP 10038 error in your bpcd log, but worth a look.

We do see TCP/IP 10022 error in your log. I have Googled the error which points to misconfigured sofware or even 'improperly-formatted IPv4 or IPv6 Network Address'.

Perhaps multiple NICs on this client?

If so, does each NIC/IP address have a different hostname?

Have you tried to remove and reinstall NBU software?

[Nithin_S1]

Nagalla - Windows firewall was already disabled and also disabled the IP Sec service and rebooted the machine with no luck.

Marianne - This was all working since 6-7 days back. Windows Firewall and IP sec disabled. I also tried disabling McAfee for few moment. As you see, I can telnet to all ports 1556, 13724, 13782 and also ping the client from the master. Client also able to resolve the master/media back.

I can again check on the McAfee part.

There are 10038 but that's for a different client (I've many failing:-)). For this client only 10022 in the logs.

We do have couple of NICs and one for production and other dedicated for backup and each NIC having a different DNS name. We do DNS resolution and resolution seems to be good. I can uninstall and try to reinstall again, yes.

[Mark_Solutions]

If you have renamed the McAfee forewall driver you will need to reboot the client for that to take effect

Also check the McAfee Access Protection as that can also block it.

See this tech note for everything you need to do:

http://www.symantec.com/docs/TECH56658

I have seen some recent updates seem to re-set the McAfee settings so they need to be re-applied

Hope this helps

[Nithin_S1]

I tried to play around with McAfee but no luck. If McAfee is blocking, it should appear in McAfee access protection logs and I don't see anything related to NBU there.

Last time when McAfee created trouble, telnet from master to client over 1556/13782 or 13724 never worked but presently, that's all working without any issues. I've opened an SYMC case anyway as more client appear to have same issue and it's happening only for Windows 2003 machines.

[Nithin_S1]

After working with Symantec support,    Under Master server properties ==> Firewall

Added the host and set as below

BPCD Connect back = VNET port
Ports = Reserved port
Daemon connection port = Daemon port only

Looks like "Daemon port only" did the trick but I'm still not sure why it's not working with VNETD port and we have the 1556 port opened bidirectional

Windows 2008 client on same subnet works fine without above settings but only Windows 2003 is having issues. Not sure if Windows 2003 is having a compatibility issue with BPX, does any of you know?

[Marianne]

Which NBU version (and patch level) on W2003 clients?

Check if Clients can telnet to vnetd and/or PBX ports on the master and media servers.

Please share W2003 client's bpcd log as File attachment.