FT media server and SAN client backup job has failed with EC58 and 54

[NBU_13]

Hi

I have issue with FT media server and SAN client backup issue

I configured FT media server and SAN client, services are active on FT media server and SAN client

while trying to restart the job, it is failed with EC58 and also it is showing as FT client has no devices configured.

I added SAN client in master server host properties and done changes in bpcd port and then I try to restarted job, now its failed with EC54 and also it is showing same FT client has no devices configured.

can anyone guide me, to fix this issue.

Thanks in Advance.

[Marianne]

Please tell us what exactly was done to bpcd port?

I added SAN client in master server host properties and done changes in bpcd port

Do the following to troubleshoot connection:

Create bpcd log folder on client.

Run this command on media server:
bptestbpcd -client <client-name> -verbose -debug

Post output of command as well as bpcd log on client (rename log file to bpcd.txt and post as File attachment).

Please also give all relevant info:
OS on media server and client
NBU version on media server and client.

[NBU_13]

Hi,

I configured FT media server (Linux 5 update 5) and SAN client (windows 2008) I restarted test backup, it failed with below error :

22-Oct-12 16:40 - Info nbjm(pid=2088) starting backup job (jobid=2916752) for client soman-7, policy GlobalCrossing-solmannw7-FS, schedule Full-Semanal  

22-Oct-12 16:40 - Info nbjm(pid=2088) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=2916752, request id:{90E2D731-18C7-4C93-9438-724BFAF871CE})  

22-Oct-12 16:40 - requesting resource media9s-hcart3-robot-tld-9

22-Oct-12 16:40 - requesting resource masters.NBU_CLIENT.MAXJOBS.solman-nw7

22-Oct-12 16:40 - requesting resource masters.NBU_POLICY.MAXJOBS.GlobalCrossing-solmannw7-FS

22-Oct-12 16:40 - awaiting resource media9s-hcart3-robot-tld-9 - FT client has no devices configured

22-Oct-12 16:40 - granted resource masters.NBU_CLIENT.MAXJOBS.solman-nw7

22-Oct-12 16:40 - granted resource masters.NBU_POLICY.MAXJOBS.GlobalCrossing-solmanw7-FS

22-Oct-12 16:40 - granted resource JPL603

22-Oct-12 16:40 - granted resource HP.ULTRIUM5-SCSI.002

22-Oct-12 16:40 - granted resource media9s-hcart3-robot-tld-9

22-Oct-12 16:40 - estimated 73351024 Kbytes needed

22-Oct-12 16:40 - Info nbjm(pid=2088) started backup job for client solman7, policy GlobalCrossing-solmannw7-FS, schedule Full-Semanal on storage unit media9s-hcart3-robot-tld-9

22-Oct-12 16:40 - started process bpbrm (9200)

22-Oct-12 16:45 - end writing

22-Oct-12 17:50 - Info bpbrm(pid=9200) starting bptm           

22-Oct-12 17:50 - Info bpbrm(pid=9200) Started media manager using bpcd successfully       

22-Oct-12 17:55 - Error bpbrm(pid=9200) cannot connect to solman-nw7, Operation now in progress (115)    

can't connect to client(58)

So, I done changes as below for client

1. Launch the NetBackup Administration Console, connecting to the master server
2. Expand Host Properties in the left pane
3. Select Master Server in the left pane
4. Click the name of the master server in the right pane
5. Select the Client Attributes section
6. Add the name of the client in question if it isn't listed
7. In the Connect Options tab for the client, make the following changes:
BPCD connect back -> "Random port"
Ports -> "Reserved port"
Daemon connection port -> "Daemon port only"

then its failed with EC54 , refer below logs

22-Oct-12 17:47 - Info nbjm(pid=2088) starting backup job (jobid=2916991) for client solman7, policy GlobalCrossing-solmannw7-FS, schedule Full-Semanal  

22-Oct-12 17:47 - Info nbjm(pid=2088) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=2916991, request id:{EA5AD29E-A214-4A73-A530-7AE49CFF3BE4})  

22-Oct-12 17:47 - requesting resource media9s-hcart3-robot-tld-9

22-Oct-12 17:47 - requesting resource masters.NBU_CLIENT.MAXJOBS.solman7

22-Oct-12 17:47 - requesting resource masters.NBU_POLICY.MAXJOBS.GlobalCrossing-solmannw7-FS

22-Oct-12 17:47 - awaiting resource media9s-hcart3-robot-tld-9 - FT client has no devices configured

22-Oct-12 17:47 - granted resource masters.NBU_CLIENT.MAXJOBS.solman-nw7

22-Oct-12 17:47 - granted resource masters.NBU_POLICY.MAXJOBS.GlobalCrossing-solmannw7-FS

22-Oct-12 17:47 - granted resource JPL603

22-Oct-12 17:47 - granted resource HP.ULTRIUM5-SCSI.002

22-Oct-12 17:47 - granted resource media9s-hcart3-robot-tld-9

22-Oct-12 17:47 - estimated 73351024 Kbytes needed

22-Oct-12 17:47 - Info nbjm(pid=2088) started backup job for client solman7, policy GlobalCrossing-solmannw7-FS, schedule Full-Semanal on storage unit media9s-hcart3-robot-tld-9

22-Oct-12 17:47 - started process bpbrm (10598)

22-Oct-12 17:47 - mounting JPL603

22-Oct-12 17:48 - mounted; mount time: 00:00:58

22-Oct-12 17:48 - positioning JPL603 to file 5

22-Oct-12 17:52 - connecting

22-Oct-12 17:57 - end writing

22-Oct-12 18:56 - Info bpbrm(pid=10598) starting bptm           

22-Oct-12 18:56 - Info bpbrm(pid=10598) Started media manager using bpcd successfully       

22-Oct-12 18:56 - Info bpbrm(pid=10598) solman7 is the host to backup data from     

22-Oct-12 18:56 - Info bpbrm(pid=10598) telling media manager to start backup on client     

22-Oct-12 18:56 - Info bptm(pid=10601) using 65536 data buffer size        

22-Oct-12 18:56 - Info bptm(pid=10601) using 12 data buffers         

22-Oct-12 18:56 - Info bpbrm(pid=10598) spawning a brm child process        

22-Oct-12 18:56 - Info bpbrm(pid=10598) child pid: 10607          

22-Oct-12 18:56 - Info bptm(pid=10601) start backup           

22-Oct-12 18:56 - Info bptm(pid=10601) Waiting for mount of media id JPL603 (copy 1) on server media9stgo. 

22-Oct-12 18:57 - Info bptm(pid=10601) media id JPL603 mounted on drive index 0, drivepath /dev/nst0, drivename HP.ULTRIUM5-SCSI.002, copy 1

22-Oct-12 19:01 - Error bpbrm(pid=10607) cannot connect to solman-nw7, status = 25      

22-Oct-12 19:01 - Info bpbrm(pid=10598) sending bpsched msg: CONNECTING TO CLIENT FOR solman-nw7_1350938828     

22-Oct-12 19:02 - Error bptm(pid=10606) cannot create data socket, Connection timed out      

22-Oct-12 19:07 - Error bpbrm(pid=10607) timed out trying to connect to solman7      

22-Oct-12 19:07 - Info bpbrm(pid=10598) sending message to media manager: STOP BACKUP solman-nw7_1350938828     

22-Oct-12 19:07 - Info bpbrm(pid=10598) media manager for backup id solman7_1350938828 exited with status 150: termination requested by administrator

timed out connecting to client(54)

I ran bpclntcmd -pn, -hn, ip on client, master and media server, everything working fine, but

I ran bptestbpcd -client <client-name> on media server, its not giving any output

but bptestbpcd -client <client-name> is gave output from master server

NBU version is 7.1 on both media and client

[Marianne]

WHY did you do this??

So, I done changes as below for client

1. Launch the NetBackup Administration Console, connecting to the master server
2. Expand Host Properties in the left pane
3. Select Master Server in the left pane
4. Click the name of the master server in the right pane
5. Select the Client Attributes section
6. Add the name of the client in question if it isn't listed
7. In the Connect Options tab for the client, make the following changes:
BPCD connect back -> "Random port"
Ports -> "Reserved port"
Daemon connection port -> "Daemon port only"

There is no need to change port connectivity in Host Properties. They should be left as default unless you have a good reason (I cannot think of any...)
NBU 6.x to 7.0 will use port 13724 (vnetd) for connection to client as well as for client connect-back.
NBU 7.0.1 onwards will first try port 1556 (pbx) and if that fails, port 13724.

Please run bptestbpcd with -verbose and -debug on the media server. You will see the port number used by the media server for connection attempt. All connection attempts will be reported until the media server manages to connect or eventually times out (check Client Connect Timeout on the media server. ensure that it is no more than 300).

Check to see if anything is logged in Client's bpcd log (connection attempt from media server).

Forget about FT configurarion until connectivity issues between media server and client are fixed.

Please also post output of the following:

On media server:
bpclntcmd -hn <client>
bpclntcmd -ip <client-ip>

On client, check Server Entries in BAR GUI under File -> Specify NetBackup Machines - confirm Media server hostname is there, then double-check lookup :
bpclntcmd -hn <media-server>
bpclntcmd -ip <media-server-ip>
 

[NBU_13]

Hi,

Please refer below output from media server and client.

From MediaServer Media9stgo

[root@media9stgo bin]# ./bpclntcmd -pn

expecting response from server masterstgo

media9stgo media9stgo 192.168.83.26 47519

[root@media9stgo bin]# ./bpclntcmd -hn solman-nw7

host solman-nw7: solman-nw7 at 192.168.83.239

aliases:     solman-nw7     192.168.83.239

[root@media9stgo bin]# ./bpclntcmd -ip 192.168.83.239

host 192.168.83.239: solman-nw7 at 192.168.83.239

aliases:     solman-nw7     192.168.83.239

[root@media9stgo bin]#

[root@media9stgo admincmd]# ./bptestbpcd -client solman-nw7 -verbose -debug

14:22:46.359 [20749] <2> bptestbpcd: VERBOSE = 0

14:22:46.360 [20749] <2> ConnectionCache::connectAndCache: Acquiring new connection for host masterstgo, query type 223

14:22:46.364 [20749] <2> vnet_pbxConnect: pbxConnectEx Succeeded

14:22:46.364 [20749] <2> logconnections: BPDBM CONNECT FROM 192.168.83.26.36808 TO 192.168.83.11.1556 fd = 3

14:22:46.650 [20749] <2> db_CLIENTsend: reset client protocol version from 0 to 7

14:22:46.997 [20749] <2> db_end: Need to collect reply

14:22:46.998 [20749] <2> db_freeEXDB_INFO: ?

14:22:46.999 [20749] <2> logconnections: BPCD CONNECT FROM 192.168.83.26.717 TO 192.168.83.239.13782 fd = 3

0 0 2

192.168.83.26:717 -> 192.168.83.239:13782

192.168.83.26:746 <- 192.168.83.239:3104

14:22:47.201 [20749] <2> bpcr_get_peername_rqst: Server peername length = 10

14:22:47.202 [20749] <2> bpcr_get_hostname_rqst: Server hostname length = 10

14:22:47.203 [20749] <2> bpcr_get_clientname_rqst: Server clientname length = 10

14:22:47.203 [20749] <2> bpcr_get_version_rqst: bpcd version: 07100004

14:22:47.204 [20749] <2> bpcr_get_platform_rqst: Server platform length = 7

14:22:47.204 [20749] <2> bpcr_get_version_rqst: bpcd version: 07100004

14:22:47.205 [20749] <2> bpcr_patch_version_rqst: theRest == > <

14:22:47.206 [20749] <2> bpcr_get_version_rqst: bpcd version: 07100004

14:22:47.246 [20749] <2> bpcr_patch_version_rqst: theRest == > <

14:22:47.247 [20749] <2> bpcr_get_version_rqst: bpcd version: 07100004

PEER_NAME = media9stgo

HOST_NAME = solman-nw7

CLIENT_NAME = solman-nw7

VERSION = 0x07100004

PLATFORM = win_x86

PATCH_VERSION = 7.1.0.4

SERVER_PATCH_VERSION = 7.1.0.4

MASTER_SERVER = masterstgo

EMM_SERVER = masterstgo

NB_MACHINE_TYPE = CLIENT

192.168.83.26:739 <- 192.168.83.239:3105

<2>bptestbpcd: EXIT status = 0

14:22:47.292 [20749] <2> bptestbpcd: EXIT status = 0

[root@media9stgo admincmd]#

From Client Solman-nw7

C:\Program Files\Veritas\NetBackup\bin>bpclntcmd -pn

expecting response from server masterstgo

solman-nw7 solman-nw7 192.168.83.239 2967

C:\Program Files\Veritas\NetBackup\bin>bpclntcmd -hn media9stgo

host media9stgo: media9stgo at 192.168.83.26

aliases:     media9stgo     192.168.83.26

C:\Program Files\Veritas\NetBackup\bin>bpclntcmd -ip 192.168.83.26

host 192.168.83.26: media9stgo at 192.168.83.26

aliases:     media9stgo     192.168.83.26

[Marianne]

<2>bptestbpcd: EXIT status = 0

Connection seems to be successful.

We now need client's bpcd log to confirm.

We still have no idea why you have changed NBU connection defaults to pre-6.0 connection type?

You can see that there is no attempt to connect to pbx or vnetd - only bpcd (13782) and client is connecting back on random reserved port:

192.168.83.26:717 -> 192.168.83.239:13782

192.168.83.26:746 <- 192.168.83.239:3104

I am pretty sure that FT Client needs pbx. Please go back to NBU defaults?

[NBU_13]

Hi,

Now, backup job is running fine through LAN cable, once I disconnect LAN cable, it is failing with EC58, with error code 58,(cannot connect to client)

FT Media server can able to get resource, like drive, tape for backup, once try to connect SAN client, it get failed

23-Oct-12 16:43 - Info bptm(pid=21944) Waiting for mount of media id JPL603 (copy 1) on server media9stgo. 

23-Oct-12 16:43 - Error bpbrm(pid=21958) cannot connect to solman-nw7, status = 25      

23-Oct-12 16:43 - Info bpbrm(pid=21941) sending bpsched msg: CONNECTING TO CLIENT FOR solman-nw7_1351017119     

23-Oct-12 16:44 - Info bptm(pid=21944) media id JPL603 mounted on drive index 0, drivepath /dev/nst0, drivename HP.ULTRIUM5-SCSI.002, copy 1

23-Oct-12 16:44 - Error bpbrm(pid=21958) cannot connect to solman-nw7, Operation now in progress (115)    

23-Oct-12 16:44 - Info bpbrm(pid=21941) sending message to media manager: STOP BACKUP solman-nw7_1351017119     

23-Oct-12 16:45 - Info bpbrm(pid=21941) media manager for backup id solman-nw7_1351017119 exited with status 150: termination requested by administrator

can't connect to client(58)

23-Oct-12 16:49 - Error bptm(pid=21957) cannot create data socket, Connection timed out      

Anyone guide me, on this issue 

[Marianne]

Back to my previous post:

We still have no idea why you have changed NBU connection defaults to pre-6.0 connection type?

You can see that there is no attempt to connect to pbx or vnetd - only bpcd (13782) and client is connecting back on random reserved port..

I am pretty sure that FT Client needs pbx. Please go back to NBU defaults?

once I disconnect LAN cable.... 

You cannot do that!!

Initial hand-shanking using TCP/IP is still needed!!! Network is also needed for meta-data transfer.

[NBU_13]

ok, Let me change default and then, I will try for test backup.

Thanks

[NBU_13]

Hi,

Regarding FT media server and SAN client, Now backup jobs is running with LAN, its not using SAN for backup, I checked both FT media server and SAN client services are running fine and also in master server host properties, I changed default for port connection, but backup is not using SAN.

Anyone help on this issue.

Thanks.

C:\Program Files\Veritas\NetBackup\bin>bpclntcmd -sanclient
1

C:\Program Files\Veritas\NetBackup\bin>bpclntcmd -pn
expecting response from server masterstgo
solman-nw7 solman-nw7 192.168.83.239 4634

C:\Program Files\Veritas\NetBackup\bin>bpclntcmd -hn solman-nw7
host solman-nw7: solman-nw7 at 192.168.83.239
aliases:     solman-nw7     192.168.83.239

C:\Program Files\Veritas\NetBackup\bin>bpclntcmd -ip 192.168.83.239
host 192.168.83.239: solman-nw7 at 192.168.83.239
aliases:     solman-nw7     192.168.83.239

C:\Program Files\Veritas\NetBackup\bin>
C:\Program Files\Veritas\NetBackup\bin>bpclntcmd -hn media9stgo
host media9stgo: media9stgo at 192.168.83.26
aliases:     media9stgo     192.168.83.26

C:\Program Files\Veritas\NetBackup\bin>bpclntcmd -ip 192.168.83.26
host 192.168.83.26: media9stgo at 192.168.83.26
aliases:     media9stgo     192.168.83.26

[root@media9stgo bin]# ./bpps -x
NB Processes
------------
root      4576     1  0 Oct12 ?        01:08:12 /usr/openv/netbackup/bin/nbftsrvr
root      4616  4576  0 Oct12 ?        00:00:01 /usr/openv/netbackup/bin/nbfdrv64 -m=0x438002 -v=1 -s=256K
root      9719     1  0 Oct22 ?        00:00:00 /usr/openv/netbackup/bin/vnetd -standalone
root      9722     1  0 Oct22 ?        00:00:00 /usr/openv/netbackup/bin/bpcd -standalone
root      9904     1  0 Oct22 ?        00:00:00 /usr/openv/netbackup/bin/bpcompatd
root      9914     1  0 Oct22 ?        00:00:21 /usr/openv/netbackup/bin/nbrmms
root      9958     1  0 Oct22 ?        00:00:05 /usr/openv/netbackup/bin/nbsl
root     10013     1  0 Oct22 ?        00:00:04 /usr/openv/netbackup/bin/nbsvcmon

MM Processes
------------
root      9891     1  0 Oct22 pts/0    00:00:03 /usr/openv/volmgr/bin/ltid
root      9897     1  0 Oct22 pts/0    00:00:03 vmd
root      9994  9891  0 Oct22 pts/0    00:00:00 tldd
root     10036  9891  0 Oct22 pts/0    00:00:00 avrd
root     10039     1  0 Oct22 pts/0    00:00:00 tldcd
root     13855  9994  0 11:56 pts/0    00:00:00 tldd

Shared Symantec Processes
-------------------------
root      4406     1  0 Oct12 ?        00:00:02 /opt/VRTSpbx/bin/pbx_exchange

[root@media9stgo admincmd]# cd ..
[root@media9stgo bin]# ./bpclntcmd -self
yp_get_default_domain failed: (12) Local domain name not set
NIS does not seem to be running: (1) Request arguments bad
gethostname() returned: media9stgo
host media9stgo: media9stgo at 127.0.0.1
aliases:     media9stgo     127.0.0.1

[root@media9stgo bin]# lspci -v | grep QL
0a:00.0 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)
        Subsystem: QLogic Corp. Unknown device 815c

[root@media9stgo admincmd]# ./bptestbpcd -verbose -debug
14:32:31.122 [16297] <2> bptestbpcd: VERBOSE = 0
14:32:31.124 [16297] <2> vnet_pbxConnect: pbxConnectEx Succeeded
14:32:31.124 [16297] <2> logconnections: BPCD CONNECT FROM 127.0.0.1.49619 TO 127.0.0.1.1556 fd = 3
14:32:31.125 [16297] <2> vnet_pbxConnect: pbxConnectEx Succeeded
14:32:31.126 [16297] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1776: 0: via PBX: VNETD CONNECT FROM 127.0.0.1.33599 TO 127.0.0.1.1556 fd = 4
14:32:31.126 [16297] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.445: 0: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a
14:32:31.166 [16297] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.462: 0: ipc_string: /tmp/vnet-16299351186351166144000000000-ei4fGL
1 1 1
127.0.0.1:49619 -> 127.0.0.1:1556
127.0.0.1:33599 -> 127.0.0.1:1556
14:32:31.246 [16297] <2> bpcr_get_peername_rqst: Server peername length = 10
14:32:31.247 [16297] <2> bpcr_get_hostname_rqst: Server hostname length = 10
14:32:31.247 [16297] <2> bpcr_get_clientname_rqst: Server clientname length = 10
14:32:31.247 [16297] <2> bpcr_get_version_rqst: bpcd version: 07100004
14:32:31.247 [16297] <2> bpcr_get_platform_rqst: Server platform length = 14
14:32:31.247 [16297] <2> bpcr_get_version_rqst: bpcd version: 07100004
14:32:31.288 [16297] <2> bpcr_patch_version_rqst: theRest == > <
14:32:31.288 [16297] <2> bpcr_get_version_rqst: bpcd version: 07100004
14:32:31.328 [16297] <2> bpcr_patch_version_rqst: theRest == > <
14:32:31.328 [16297] <2> bpcr_get_version_rqst: bpcd version: 07100004
PEER_NAME = media9stgo
HOST_NAME = media9stgo
CLIENT_NAME = media9stgo
VERSION = 0x07100004
PLATFORM = linuxR_x86_2.6
PATCH_VERSION = 7.1.0.4
SERVER_PATCH_VERSION = 7.1.0.4
MASTER_SERVER = masterstgo
EMM_SERVER = masterstgo
NB_MACHINE_TYPE = MEDIA_SERVER
<2>bptestbpcd: EXIT status = 0
14:32:31.340 [16297] <2> bptestbpcd: EXIT status = 0
[root@media9stgo admincmd]# telnet 192.168.83.239 13782
Trying 192.168.83.239...
Connected to solman-nw7 (192.168.83.239).
Escape character is '^]'.

Connection closed by foreign host.
[root@media9stgo admincmd]# telnet 192.168.83.239 13724
Trying 192.168.83.239...
Connected to solman-nw7 (192.168.83.239).
Escape character is '^]'.

Connection closed by foreign host.
[root@media9stgo admincmd]# telnet 192.168.83.239 1556
Trying 192.168.83.239...
Connected to solman-nw7 (192.168.83.239).
Escape character is '^]'.

[NBU_13]

Hi,

Thanks...

FT media server and SAN client issue has resolved,

backup jobs are running through FT.

But, speed is not much good compare LAN, is there any way from netbackup to increase speed ?

[Shruti_Yadav]

Could you Please let us know, how you fixed it?

[Ankit_Maheshwar]

I read all..

Bpcd is ok..---> No conection issue

bpclntcmd is ok

So where was the issue?

[NBU_13 · Accepted solution]

Hi,

Please take a look on FT media server bp.conf file

1st - master server name

2nd - media server name