sc=58 error.

[Brais]

Hello,

       I have a client which is failing with sc=58 (it has not ever worked before). Versions are as folow: client 7.0.1 and Master server 7.1

       Bp.conf seems ok.

       Regarding ports, 1556 is open both ways

       Bpcoverage throws sc=58 error, while bptestbpcd shows:

       Please, could you tell me where the problem reside?

12:37:56.951 [45416612] <2> bptestbpcd: VERBOSE = 0
12:37:56.954 [45416612] <2> vnet_pbxConnect: pbxConnectEx Succeeded
12:37:56.954 [45416612] <2> logconnections: BPCD CONNECT FROM X.X.X.X.41232 TO X.X.X.X.1556 fd = 3
12:37:56.956 [45416612] <2> vnet_pbxConnect: ../../libvlibs/vnet_pbx.c.666: pbxSetAddrEx/pbxConnectEx return error 73:Connection reset by peer
12:37:56.956 [45416612] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1718: 0: vnet_pbxConnect() failed: 18 0x00000012
12:37:56.956 [45416612] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1719: 0: save_errno: 73 0x00000049
12:37:56.956 [45416612] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1720: 0: use_vnetd: 1 0x00000001
12:37:56.956 [45416612] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1721: 0: cr->vcr_service: vnetd
12:37:56.957 [45416612] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1353: 0: do_service failed: 18 0x00000012
12:37:56.957 [45416612] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 79 0x0000004f
12:37:57.957 [45416612] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 79 0x0000004f
12:37:59.957 [45416612] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 79 0x0000004f
12:38:03.958 [45416612] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 79 0x0000004f
12:38:11.958 [45416612] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 79 0x0000004f
12:38:11.958 [45416612] <2> connect_to_service: ../../libvlibs/vnet_connect.c.382: 0: vnet_async_connect() failed: 18 0x00000012
12:38:11.958 [45416612] <2> vnet_connect_to_vnetd_ipi: ../../libvlibs/vnet_connect.c.216: 0: connect_to_service() failed: 18 0x00000012
12:38:11.959 [45416612] <16> bpcr_vnetd_connect_forward_socket_begin: vnet_connect_to_vnetd_ipi(X.X.X.X) failed: 18
12:38:11.959 [45416612] <2> local_bpcr_connect: bpcr_vnetd_connect_forward_socket_begin failed: 25
12:38:11.959 [45416612] <2> ConnectToBPCD: bpcd_connect_and_verify(client01, client01) failed: 25
<16>bptestbpcd main: Function ConnectToBPCD(client01) failed: 25
12:38:11.959 [45416612] <16> bptestbpcd main: Function ConnectToBPCD(client01) failed: 25
<2>bptestbpcd: cannot connect on socket
12:38:11.959 [45416612] <2> bptestbpcd: cannot connect on socket
<2>bptestbpcd: EXIT status = 25
12:38:11.959 [45416612] <2> bptestbpcd: EXIT status = 25
cannot connect on socket
      

Thanks for your help.

[mph999]

OK, from the master / media server try this :

telnet <client name> 1556

telnet <client name> 13724

telnet <client name> 13782

Lets see if we can establish a connection to either bpcd/ vnetd or pbx without using NBU.

This will help narrow down where the issue is, that is, is it NetBackup, or Network.

Thanks,

Martin

[Marianne]

This is most of the time caused by client that is unable to resolve server's IP address to a hostname that appears in its SERVER list.

Please check that client can resolve IP to hostname via DNS or via local hosts entry.

Please also enable bpcd log on the client for troubleshooting.
After bptestbpcd, look in client's bpcd log for server's incoming IP address.
See in log if IP is resolved to hostname and if hostname is found in SERVER list.

After above name resolution, bpcd will also show if client can connect back on 1556 or 13724.

[revarooo]

Brais,

mph999 has made an excellent recommendation above. Carry that suggestion out.

Also what is the Operating System of the client? Does it have a firewall? If so, ensure the ports 1556, 13724 and 13782 are not blocked or temporarily turn off the firewall (if it is a software firewall on the client) and test again.

[Brais]

Hello,

       @mph999 From master to client, telnet works on all the above ports but 13724. However, I understood that, in version 7.X, PBX port should be enough.

      @Marianne it is not a name resolution issue. In the log I posted above "12:37:56.954 [45416612] <2> logconnections: BPCD CONNECT FROM X.X.X.X.41232 TO X.X.X.X.1556 fd = 3" names were resolved, but I changed them by X.X.X.X to avoid disclosure of information.

    @revaroo   client's OS is AIX. Yes port 13724 is not being reached. But, shouldn't pbx be doing the work just with port 1556?
 

Regards.

[mph999]

OK, thank you for taking the time to provide the information.

Lets run one more command :

bptestbpcd -client <client name> -connect options 1 0 2

Just post up the output - many thx ...

M

[revarooo]

Brais, yes it should be PBX in your case. Have you analyzed the the PBX log to see if the connection is received?

Run the bptestbpcd command again (use the options mph999 suggested in the post above)  from the server, then on the client run:

vxlogview -p 50936 -o 103 -d all -t 00:10:00 

(this will log activity for the last 10 minutes)

[Marianne]

"BPCD CONNECT FROM X.X.X.X.41232 TO X.X.X.X.1556 fd = 3"

The X's seem to me like IP addresses NOT hostnames.
NBU needs to resolve connection from IP address to hostname.

It also shows bptestbpcd output, not the Client's bpcd log file that tells us how server's X.X.X.X IP address is resolved to a hostname and if this hostname can be found in Client's SERVER list.

Since 7.0.1, server will first try to connect to 1556. Failure to connect will fail back to 13724 (vnetd).

You can test name lookup on client as follows:

bpclntcmd -ip <Server-IP>

PS: Please change IP addresses and hostnames to something a bit more meaningful, eg.
10.10.10.1  master
10.10.10.2  media1
10.10.10.3  client1
 

[Brais]

Thank you for assisting, btw you forgot the underscore between connect and options.

Here's the result.

 bptestbpcd -client  client01 -connect_options 1 0 2
1 0 2
MasterIP:53771 -> ClientIP:13782
MasterIP:3333 <- ClientIP:63813
MasterIP:1968 <- ClientIP:63814
 

Regards.

[Brais]

Yes X.X.X.X also seems to me like an IP addresses. Since I run bptestbpcd against client name, and the output shows an IP, name resolution has been correctly done.

In addition I tested  both name resolutions with ping and nslookup, which worked ok.

Thanks for clarifying about PBX. So, since my environment is version 7+ it should be working only with 1556 no need for testing the other ports.

Regards.

[mph999]

" no need for testing the other ports."

Kind of .. if it doesn't work on PBX, but we can get it to work for example on vnetd / bpcd - then although th e problem is not resolved, we have :

1.  Narrowed down the problem

2. Found a tmp solution to allow at least a few backups to run ...

This seems to connect ...

bptestbpcd -client  client01 -connect_options 1 0 2
1 0 2
MasterIP:53771 -> ClientIP:13782
MasterIP:3333 <- ClientIP:63813
MasterIP:1968 <- ClientIP:63814

Now let us try

bptestbpcd -client  client01 (sorry, should have asked you to run this previouslyI)

I suspect this will give us a status 25.

Thanks,

Martin

[Marianne]

bptestbpcd will ONLY show us the forward lookup, i.e. hostname -> IP address and then display IP address:port combination in the resulting output...

You need to check on the client if server's IP address can be resolved to hostname.

The client's bpcd log as well as 'bpclntcmd -ip <server-ip>' will tell us if the client can do reverse lookup of server's IP address.

[Brais]

@mph99: You are right it returns sc=25

 bptestbpcd -client client01
<16>bptestbpcd main: Function ConnectToBPCD(client01) failed: 25
cannot connect on socket
 

@Marianne: I don't think it is a name resolution issue.

As I stated above, I tested name resolution with ping and reverse resolution with nslookup. Both worked as expected.

Thanks guys.

[Marianne]

Please do us this one favour: Post client's bpcd log file after you have done bptestbpcd. If log is more than a couple of lines, post log as attachment.

[mph999]

OK - can you increase the VERBOSE level on the client ...

VERBOSE = 5 into bp.conf

Then re-run the command bptestbpcd -client  client01

In fact do it this way 

bptestbpcd -client  client01 -verbose -debug

Lets see what we get in the log (previous log was at VERBOSE = 0 )

Thanks,

Martin

[Marianne]

@Brais - you have gone very quiet...

Have you been able to resolve this?

If so, please share how it was resolved.

If not, please post client's bpcd log.

[Brais · Accepted solution]

Hello,

Sorry for the late reply, I forgot I left this thread open.

It was solved quite time ago.

If I recall well. The issue was due to backup going through the wrong NIC.

Thanks for your help