03-27-2017 10:48 AM
This issue just started yesterday after apply the Monthly Microst Patches for Mar 2017 to the client server (2012 Std) and the Master\Media Server (2008 R2 SP1) then rebooting the servers. These servers were warm rebooted and not shut down then started.
- From the Veritas Master\Media server (version 7.5.0.7) I can successfully telnet to the client by hostname and IP Address on port 13782.
- From the client I can successfully telnet to the Master\Media server by hostname and IP Address on port 13720.
- I have restarted the NetBacup services on the client via the services.msc and utilizing bpdown and bpup commands.
-Both the Master\Media server can ping each other by hostname and IP Address. bpclntcmd -pn & bpclntcmd -ip both appear to return the appropriate responses.
- On the Master\Media server I have removed the client from the policies then recreated the client for the policies to preclude a possible bad profile as the cause of the issue.
I have attached diagnostics that I've run from the servers to this posting. This is a production server and the IP Addresses and servernames in the attachment have been modified to have place holders in the 1st 3 octets for privacy concerns.
I am not sure what to try next as so far everything that I've read and tried has not resolved the issue. I am not ruling out the need to perform a cold reboot of the client server since the Master\Media server does not have any issues with the other servers who received the same changes as the client server experiencing the issue.
Solved! Go to Solution.
03-27-2017 12:43 PM
I found the solution to the issue and the HTML link is given below. I have modified the text slightly to put a (-) sign in front of the client in the bpclient commands as it would not work without it. Even thought NBU Admin console showed the client hostname I had to ADD the client hostname with the communication parameters as shown below then to be sure I did the UPDATE command. Once they were done I performed the
https://www.veritas.com/support/en_US/article.000108081
Solution
<Install_Path>\VERITAS\NetBackup\bin\admincmd\bptestbpcd -client <CLIENTNAME> -connect_options 1 0 2
Note: If the bptestbpcd fails to connect, review the bptestbpcd output using verbose switch to troubleshoot further.
<Install_Path>\VERITAS\NetBackup\bin\admincmd\
a. bpclient -update -client <CLIENTNAME> -connect_options 1 0 2
b. If the client is not listed in the client list need to add it with
bpclient -add -client <CLIENTNAME> -connect_options 1 0 2
Test the connection again using the following command :
bptestbpcd -client <CLIENTNAME>
Actual process utilized
C:\Program Files\Veritas\NetBackup\bin\admincmd>bpclient -add -client fxxxxxxxx4 -connect_options 1 0 2
C:\Program Files\Veritas\NetBackup\bin\admincmd>bptestbpcd -client fxxxxxxxx4
1 0 2
X.X.X.71:61813 -> X.X.X.78:13782
X.X.X.71:1276 <- X.X.X.78:50150
X.X.X.71:2467 <- X.X.X.78:50151
C:\Program Files\Veritas\NetBackup\bin\admincmd>bpclient -update -client fxxxxxxxx4 -connect_options 1 0 2
C:\Program Files\Veritas\NetBackup\bin\admincmd>bptestbpcd -client fxxxxxxxx4
1 0 2
X.X.X.71:62228 -> X.X.X.78:13782
X.X.X.71:1283 <- X.X.X.78:50487
X.X.X.71:2859 <- X.X.X.78:50488
C:\Program Files\Veritas\NetBackup\bin>bptestbpcd -verbose -debug -client fxxxxxxxx4
14:07:06.990 [11992.17684] <2> bptestbpcd: VERBOSE = 0
14:07:06.990 [11992.17684] <2> read_client: dname=., offline=0, online_at=0 offline_at=0
14:07:06.990 [11992.17684] <2> read_client: dname=.., offline=0, online_at=0 offline_at=0
14:07:07.005 [11992.17684] <2> read_client: dname=CO_0, offline=0, online_at=0 offline_at=0
14:07:07.005 [11992.17684] <2> read_client: dname=host_info, offline=0, online_at=0 offline_at=0
14:07:07.005 [11992.17684] <2> read_client: dname=OA_0, offline=0, online_at=0 offline_at=0
14:07:07.005 [11992.17684] <2> db_freeEXDB_INFO: ?
14:07:07.021 [11992.17684] <2> logconnections: BPCD CONNECT FROM X.X.X.71.62810 TO X.X.X.78.13782 fd = 940
1 0 2
X.X.X.71:62810 -> X.X.X.78:13782
X.X.X.71:3045 <- X.X.X.78:50627
14:07:07.286 [11992.17684] <2> bpcr_get_peername_rqst: Server peername length = 8
14:07:07.505 [11992.17684] <2> bpcr_get_hostname_rqst: Server hostname length = 10
14:07:07.723 [11992.17684] <2> bpcr_get_clientname_rqst: Server clientname length = 10
14:07:07.926 [11992.17684] <2> bpcr_get_version_rqst: bpcd version: 07500007
14:07:08.144 [11992.17684] <2> bpcr_get_platform_rqst: Server platform length = 7
14:07:08.363 [11992.17684] <2> bpcr_get_version_rqst: bpcd version: 07500007
14:07:08.628 [11992.17684] <2> bpcr_patch_version_rqst: theRest == > <
14:07:08.628 [11992.17684] <2> bpcr_get_version_rqst: bpcd version: 07500007
14:07:08.877 [11992.17684] <2> bpcr_patch_version_rqst: theRest == > <
14:07:08.877 [11992.17684] <2> bpcr_get_version_rqst: bpcd version: 07500007
PEER_NAME = gxxxxxx2
HOST_NAME = fxxxxxxxx4
CLIENT_NAME = fxxxxxxxx4
VERSION = 0x07500007
PLATFORM = win_x64
PATCH_VERSION = 7.5.0.7
SERVER_PATCH_VERSION = 7.5.0.7
MASTER_SERVER = gxxxxxx2
EMM_SERVER = gxxxxxx2
NB_MACHINE_TYPE = CLIENT
X.X.X.71:4868 <- X.X.X.78:50630
<2>bptestbpcd: EXIT status = 0
14:07:09.205 [11992.17684] <2> bptestbpcd: EXIT status = 0
C:\Program Files\Veritas\NetBackup\bin>
03-27-2017 11:04 AM
Please review your registery setting on the clients, it should have the name of the Master and media servers.
HKLM\SOFTWARE\Veritas\NetBackup\CurrentVersion\Config\Server
03-27-2017 11:15 AM
What is this? It looks like you've done 0 testing between master/media/client and whatever the Y.Y.Y.52 is
09:19:53.608 [10172.10280] <2> logconnections: BPCD ACCEPT FROM Y.Y.Y.52.52737 TO Y.Y.Y.51.13782 fd = 584
09:19:53.608 [10172.10280] <2> process_requests: setup_sockopts complete
09:19:53.652 [9260.9908] <2> logconnections: BPCD ACCEPT FROM Y.Y.Y.52.52738 TO Y.Y.Y.51.13782 fd = 584
09:19:58.271 [10172.10280] <8> retry_getnameinfo: [vnet_addrinfo.c:1151] getnameinfo() failed 11001 0x2af9
09:19:58.271 [10172.10280] <8> vnet_cached_getnameinfo: [vnet_addrinfo.c:1900] retry_getnameinfo() failed RV=11001 IPSTR=Y.Y.Y.52 PORT=52737
09:19:58.271 [10172.10280] <2> bpcd peer_hostname: vnet_cached_getnameinfo() failed: 6 11001
09:19:58.271 [10172.10280] <2> bpcd peer_hostname: Connection from host Y.Y.Y.52 (Y.Y.Y.52) port 52737
09:19:58.271 [10172.10280] <2> bpcd valid_server: comparing gxxxx102 and Y.Y.Y.52
09:19:58.272 [10172.10280] <2> bpcd valid_server: comparing X.X.X.71 and Y.Y.Y.52
09:19:58.272 [10172.10280] <4> bpcd valid_server: Y.Y.Y.52 is not a master server
09:19:58.272 [10172.10280] <16> bpcd valid_server: Y.Y.Y.52 is not a media server either
09:19:58.571 [10172.10280] <16> process_requests: read failed: An existing connection was forcibly closed by the remote host.
You do testing from the client which is X.X.X.78 and your master server which is X.X.X.71
03-27-2017 11:22 AM
I checked the registry setting,HKLM\SOFTWARE\Veritas\NetBackup\CurrentVersion\Config\Server, on the client server per your suggestion and it shows the Master\Media server hostname in the registry.
The environment has been up and operating for years and the client has been successfully backing up to the Master\Media server through the early morning hours on 26 Mar 2017. The issue first appeared after the client was rebooted to apply March 2017 Microsoft Patches to the server when the user backup jobs kicked off.
03-27-2017 11:42 AM
As I indicated in my original post for privacy concerns I replaced the 1st 3 octets of the IP Addresses with placeholders. I used X.X.X for one subnet and Y.Y.Y for a different subnet that appeared in the results of the bptestbpcd -verbose -debug -client fxxxxxxx4 command. Why Y.Y.Y.52 showed up in the test from the Master\Media server to the client I do not know since that IP Address has never been a Master or a Media server in the environment.
03-27-2017 12:43 PM
I found the solution to the issue and the HTML link is given below. I have modified the text slightly to put a (-) sign in front of the client in the bpclient commands as it would not work without it. Even thought NBU Admin console showed the client hostname I had to ADD the client hostname with the communication parameters as shown below then to be sure I did the UPDATE command. Once they were done I performed the
https://www.veritas.com/support/en_US/article.000108081
Solution
<Install_Path>\VERITAS\NetBackup\bin\admincmd\bptestbpcd -client <CLIENTNAME> -connect_options 1 0 2
Note: If the bptestbpcd fails to connect, review the bptestbpcd output using verbose switch to troubleshoot further.
<Install_Path>\VERITAS\NetBackup\bin\admincmd\
a. bpclient -update -client <CLIENTNAME> -connect_options 1 0 2
b. If the client is not listed in the client list need to add it with
bpclient -add -client <CLIENTNAME> -connect_options 1 0 2
Test the connection again using the following command :
bptestbpcd -client <CLIENTNAME>
Actual process utilized
C:\Program Files\Veritas\NetBackup\bin\admincmd>bpclient -add -client fxxxxxxxx4 -connect_options 1 0 2
C:\Program Files\Veritas\NetBackup\bin\admincmd>bptestbpcd -client fxxxxxxxx4
1 0 2
X.X.X.71:61813 -> X.X.X.78:13782
X.X.X.71:1276 <- X.X.X.78:50150
X.X.X.71:2467 <- X.X.X.78:50151
C:\Program Files\Veritas\NetBackup\bin\admincmd>bpclient -update -client fxxxxxxxx4 -connect_options 1 0 2
C:\Program Files\Veritas\NetBackup\bin\admincmd>bptestbpcd -client fxxxxxxxx4
1 0 2
X.X.X.71:62228 -> X.X.X.78:13782
X.X.X.71:1283 <- X.X.X.78:50487
X.X.X.71:2859 <- X.X.X.78:50488
C:\Program Files\Veritas\NetBackup\bin>bptestbpcd -verbose -debug -client fxxxxxxxx4
14:07:06.990 [11992.17684] <2> bptestbpcd: VERBOSE = 0
14:07:06.990 [11992.17684] <2> read_client: dname=., offline=0, online_at=0 offline_at=0
14:07:06.990 [11992.17684] <2> read_client: dname=.., offline=0, online_at=0 offline_at=0
14:07:07.005 [11992.17684] <2> read_client: dname=CO_0, offline=0, online_at=0 offline_at=0
14:07:07.005 [11992.17684] <2> read_client: dname=host_info, offline=0, online_at=0 offline_at=0
14:07:07.005 [11992.17684] <2> read_client: dname=OA_0, offline=0, online_at=0 offline_at=0
14:07:07.005 [11992.17684] <2> db_freeEXDB_INFO: ?
14:07:07.021 [11992.17684] <2> logconnections: BPCD CONNECT FROM X.X.X.71.62810 TO X.X.X.78.13782 fd = 940
1 0 2
X.X.X.71:62810 -> X.X.X.78:13782
X.X.X.71:3045 <- X.X.X.78:50627
14:07:07.286 [11992.17684] <2> bpcr_get_peername_rqst: Server peername length = 8
14:07:07.505 [11992.17684] <2> bpcr_get_hostname_rqst: Server hostname length = 10
14:07:07.723 [11992.17684] <2> bpcr_get_clientname_rqst: Server clientname length = 10
14:07:07.926 [11992.17684] <2> bpcr_get_version_rqst: bpcd version: 07500007
14:07:08.144 [11992.17684] <2> bpcr_get_platform_rqst: Server platform length = 7
14:07:08.363 [11992.17684] <2> bpcr_get_version_rqst: bpcd version: 07500007
14:07:08.628 [11992.17684] <2> bpcr_patch_version_rqst: theRest == > <
14:07:08.628 [11992.17684] <2> bpcr_get_version_rqst: bpcd version: 07500007
14:07:08.877 [11992.17684] <2> bpcr_patch_version_rqst: theRest == > <
14:07:08.877 [11992.17684] <2> bpcr_get_version_rqst: bpcd version: 07500007
PEER_NAME = gxxxxxx2
HOST_NAME = fxxxxxxxx4
CLIENT_NAME = fxxxxxxxx4
VERSION = 0x07500007
PLATFORM = win_x64
PATCH_VERSION = 7.5.0.7
SERVER_PATCH_VERSION = 7.5.0.7
MASTER_SERVER = gxxxxxx2
EMM_SERVER = gxxxxxx2
NB_MACHINE_TYPE = CLIENT
X.X.X.71:4868 <- X.X.X.78:50630
<2>bptestbpcd: EXIT status = 0
14:07:09.205 [11992.17684] <2> bptestbpcd: EXIT status = 0
C:\Program Files\Veritas\NetBackup\bin>