ERR - Cannot write to STDOUT. Errno = 110: Connection timed out

[T_N]

Hi All;

My backup system is running on 7.7.2 ( master server running on cluster, 9 media servers), they are linux servers. Media servers (1, 2, 3) are configured with tape drive and Data Domain. Media server 4 is configured with tape drive. I have a issue when I tried to backup media server 4 & 9 to data domain even I tried backup them with tape drive of media server 1 but if I backed up media server 4 with tape drive itself then it worked well. I tried to increase CLIENT_READ_TIMEOUT to 3600 on master & media server but it keeps failing. 

I created bpbkar log , it shows :

18:30:28.796 [28904] <4> check_file_sparseness: Device changing from 0 to 2049
18:46:00.727 [28904] <16> flush_archive(): ERR - Cannot write to STDOUT. Errno = 110: Connection timed out
18:46:00.727 [28904] <16> bpbkar Exit: ERR - bpbkar FATAL exit status = 24: socket write failed
18:46:00.727 [28904] <4> bpbkar Exit: INF - EXIT STATUS 24: socket write failed
18:46:00.727 [28904] <8> bpbkar Exit: WRN - Error closing stdout
18:46:00.727 [28904] <16> bpbkar Exit: ERR - read server exit status = 6: the backup failed to back up the requested files
18:46:00.727 [28904] <2> bpbkar manage_fscp_info: Start
18:46:00.727 [28904] <2> bpbkar manage_fscp_info: status=6, checkpoint_restart=900
18:46:00.728 [28904] <2> tracklog_close: closed current tracklog
18:46:00.728 [28904] <2> tracklog_close: closed previous tracklog

Thank you.

[Marianne]

I would like to see all entries for specific job in all of these logs:
bpbkar
bpbrm
bptm
(Assuming problem is with media server backing up itself)

Level 3 should be fine.

[T_N]

if media server backs up by itself, it worked well

[Marianne]

So, logs will be:
bpbkar on the client
bptm and bpbrm on the media server.

[T_N]

This is bpbkar log, I figure out how to get file from linux media server to my Mac laptop :)

[Marianne]

If media server backup of itself is successful, then bpbkar on the media server is not going to help.

You need bpbkar on the client where the backup has failed.
All 3 logs that is involved in the backup process are needed. I have listed them in my previous post.

You will see PIDs for all of these processes in Job Details.
These PIDs are represented in [ ] brackets in the relevant logs.
We need to follow the backup flow through the logs using the PIDs.

If you need assistance from the community, please post the text in Job Details along with all relevant logs.
Please copy logs to .txt files.
.txt files can be viewed online without needing to download them 1st.

[T_N]

I found one thing, if that media server backs up itself, it completed successfully. I ran backup test to backup 3 media servers 1, 2, 3 through DataDomain. They completed if they backed up by theirself . I attach bpbkar, bpbrm & bpdm log files. Thank you.

[Marianne]

@T_N

It seems that we don't understand one another very well....

My time to try and assist here is very limited. 
So, I can only offer to go through the logs for one set of logs that represent a failed backup.

I need the following :

1. Job details for a failed backup (I need the PIDs for the relevant processes)
2. bpbkar log of the FAILED client
3. bptm on the media server where the backup failed (NOT bpdm. Disk backups also write to bptm.)
4. bpbrm on the media server where the backup failed

That is what I need to try and assist. Nothing more, nothing less.

[T_N]

PID : 67427, I attach the logs, thank you.

[Marianne]

PLEASE post ALL text in Job details. There should be at least 3 PIDs -
One for bpbrm, one for bptm, one for bpbkar.

Honestly.... I'm about to give up....

[T_N]

Client_backup_job_faile detail: 

06/21/2018 21:31:21 - Info nbjm (pid=37151) starting backup job (jobid=129572) for client cark2nnobkup-media9, policy test, schedule Full

06/21/2018 21:31:21 - Info nbjm (pid=37151) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=129572, request id:{70BF71E6-759A-11E8-B186-02D2BD4E7F33})

06/21/2018 21:31:21 - requesting resource cark2-adv-disk1

06/21/2018 21:31:21 - requesting resource cark2nnobkup-master_cluster.NBU_CLIENT.MAXJOBS.cark2nnobkup-media9

06/21/2018 21:31:21 - requesting resource cark2nnobkup-master_cluster.NBU_POLICY.MAXJOBS.test

06/21/2018 21:31:21 - granted resource  cark2nnobkup-master_cluster.NBU_CLIENT.MAXJOBS.cark2nnobkup-media9

06/21/2018 21:31:21 - granted resource  cark2nnobkup-master_cluster.NBU_POLICY.MAXJOBS.test

06/21/2018 21:31:21 - granted resource  MediaID=@aaaab;DiskVolume=cark2-dd890-1;DiskPool=cark2-dd890-1_dp;Path=cark2-dd890-1;StorageServer=cark2-dd890-1;MediaServer=cark2nnobkup-media1

06/21/2018 21:31:21 - granted resource  cark2-adv-disk1

06/21/2018 21:31:21 - estimated 0 kbytes needed

06/21/2018 21:31:21 - Info nbjm (pid=37151) started backup (backupid=cark2nnobkup-media9_1529616681) job for client cark2nnobkup-media9, policy test, schedule Full on storage unit cark2-adv-disk1

06/21/2018 21:31:21 - started process bpbrm (pid=67427)

06/21/2018 21:31:21 - connecting

06/21/2018 21:31:22 - connected; connect time: 0:00:00

06/21/2018 21:31:53 - begin writing

06/21/2018 21:32:17 - Info bpbrm (pid=67427) cark2nnobkup-media9 is the host to backup data from

06/21/2018 21:32:17 - Info bpbrm (pid=67427) reading file list for client

06/21/2018 21:32:17 - Info bpbrm (pid=67427) starting bpbkar on client

06/21/2018 21:32:17 - Info bpbkar (pid=41276) Backup started

06/21/2018 21:32:17 - Info bpbrm (pid=67427) bptm pid: 67428

06/21/2018 21:32:17 - Info bptm (pid=67428) start

06/21/2018 21:32:28 - Info bptm (pid=67428) using 524288 data buffer size

06/21/2018 21:32:28 - Info bptm (pid=67428) using 64 data buffers

06/21/2018 21:32:49 - Info bptm (pid=67428) start backup

06/21/2018 21:32:49 - Info bptm (pid=67428) backup child process is pid 67429

06/21/2018 23:31:31 - end writing; write time: 1:59:38

06/21/2018 23:32:25 - Error bptm (pid=67429) system call failed - Connection reset by peer (at ../child.c.1308)

06/21/2018 23:32:25 - Error bptm (pid=67429) unable to perform read from client socket, connection may have been broken

06/21/2018 23:32:27 - Info bptm (pid=67428) EXITING with status 42 <----------

06/21/2018 23:32:27 - Info bpbkar (pid=41276) done. status: 42: network read failed

network read failed  (42)

Media server bptm log detail: 

fd =-467415038 length =524288 offset =29146742784

06:39:22.535 [26934] <2> 129495:bptm:26934:cark2nnobkup-media1: /usr/openv/lib/ost-plugins/libstspiDataDomain.so:stspi_write_image STS_EOK image handle

[T_N]

Media server bpbrm log detail:  part 1:

21:32:16.968 [67427.67427] <4> bpbrm main: logging will continue into new log at midnight

21:32:16.968 [67427.67427] <2> bpbrm main: INITIATING (VERBOSE = 5): version NetBackup 7.7.2 2016011116

21:32:16.968 [67427.67427] <2> logparams: -backup -S cark2nnobkup-master_cluster -c cark2nnobkup-media9 -ct 0 -ru root -cl test -sched Full -bt 1529616681 -dt 0 -st 0 -b cark2nnobkup-media9_1529616681 -mediasvr cark2nnobkup-media1 -jobid 129572 -jobgrpid 129572 -masterversion 771000 -maxfrag 524288 -bpstart_time 1529616981 -reqid -1525972491 -mt 0 -to 0 -stunit cark2-adv-disk1 -rl 0 -rp 604800 -eari 0 -p cark2-dd890-1 -mst 6 -flags 0 -storagesvr cark2-dd890-1 -sts_type DataDomain -use_ofb -use_otm -jm -secure 1 -kl 28 -rg root -fso -ckpt_time 900 -connect_options 16974338

21:32:16.968 [67427.67427] <4> bpbrm main: ensured stderr cannot be used

21:32:16.968 [67427.67427] <2> vnet_check_resilient_socket: [vnet_nbrntd.c:804] the socket is  0 0x0

21:32:16.968 [67427.67427] <2> vnet_check_resilient_socket: [vnet_nbrntd.c:804] the socket is  2 0x2

21:32:16.969 [67427.67427] <2> bprd_connect_need_auth: Ignoring reserved port request.

21:32:16.969 [67427.67427] <2> bprd_connect_need_auth: Ignoring local_name cark2nnobkup-media1.

21:32:16.969 [67427.67427] <2> bprd_connect_need_auth: Ignoring owner_name root.

21:32:16.969 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:16.969 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:16.969 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:16.969 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:16.969 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:16.969 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:16.969 [67427.67427] <2> async_connect: [vnet_connect.c:1570] connect in progress 1 0x1

21:32:16.969 [67427.67427] <2> vnet_pbxConnect: pbxConnectEx Succeeded

21:32:16.969 [67427.67427] <2> do_pbx_service: [vnet_connect.c:2235] via PBX bprd CONNECT FROM 192.168.168.4.43482 TO 192.168.168.3.1556 fd = 5

21:32:16.969 [67427.67427] <2> async_connect: [vnet_connect.c:1762] connect async CONNECT FROM 192.168.168.4.43482 TO 192.168.168.3.1556 fd = 5

21:32:16.970 [67427.67427] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO cark2nnobkup-master_cluster 192.168.168.3 bprd VIA pbx

21:32:16.970 [67427.67427] <2> logconnections: BPRD CONNECT FROM 192.168.168.4.43482 TO 192.168.168.3.1556 fd = 5

21:32:16.970 [67427.67427] <2> vnet_check_vxss_client_magic_with_info_ex: [vnet_vxss_helper.c:1112] Ignoring VxSS authentication 2 0x2

21:32:17.004 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.004 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.004 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.004 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.004 [67427.67427] <2> async_connect: [vnet_connect.c:1570] connect in progress 1 0x1

21:32:17.004 [67427.67427] <2> vnet_pbxConnect: pbxConnectEx Succeeded

21:32:17.005 [67427.67427] <8> do_pbx_service: [vnet_connect.c:2227] via PBX VNETD CONNECT FROM 192.168.168.4.36935 TO 192.168.168.3.1556 fd = 5

21:32:17.005 [67427.67427] <2> async_connect: [vnet_connect.c:1762] connect async CONNECT FROM 192.168.168.4.36935 TO 192.168.168.3.1556 fd = 5

21:32:17.005 [67427.67427] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO cark2nnobkup-master_cluster 192.168.168.3 vnetd VIA pbx

21:32:17.005 [67427.67427] <8> vnet_vnetd_get_master_useat_info: [vnet_vnetd.c:3135] VN_REQUEST_GET_SECURITY_INFO 9 0x9

21:32:17.045 [67427.67427] <8> vnet_vnetd_disconnect: [vnet_vnetd.c:192] VN_REQUEST_DISCONNECT 1 0x1

21:32:17.049 [67427.67427] <2> brm_update_local_resiliency: changed = 0

21:32:17.049 [67427.67427] <2> bpbrm main: max_entries_per_add = 5000

21:32:17.049 [67427.67427] <2> verify_client: ../bpbrm.c.42396: *connect_opts: 16974338 16974338 0x01030202

21:32:17.049 [67427.67427] <8> vnet_get_user_credential_path: [vnet_vxss.c:1474] status 35 0x23

21:32:17.049 [67427.67427] <8> vnet_check_user_certificate: [vnet_vxss_helper.c:3643] vnet_get_user_credential_path failed 35 0x23

21:32:17.049 [67427.67427] <2> ConnectionCache::connectAndCache: Acquiring new connection for host cark2nnobkup-master_cluster, query type 223

21:32:17.049 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.049 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.049 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.049 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.049 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.049 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.049 [67427.67427] <2> async_connect: [vnet_connect.c:1570] connect in progress 1 0x1

21:32:17.050 [67427.67427] <2> vnet_pbxConnect: pbxConnectEx Succeeded

21:32:17.050 [67427.67427] <2> do_pbx_service: [vnet_connect.c:2235] via PBX bpdbm CONNECT FROM 192.168.168.4.37967 TO 192.168.168.3.1556 fd = 5

21:32:17.050 [67427.67427] <2> async_connect: [vnet_connect.c:1762] connect async CONNECT FROM 192.168.168.4.37967 TO 192.168.168.3.1556 fd = 5

21:32:17.050 [67427.67427] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO cark2nnobkup-master_cluster 192.168.168.3 bpdbm VIA pbx

21:32:17.050 [67427.67427] <2> logconnections: BPDBM CONNECT FROM 192.168.168.4.37967 TO 192.168.168.3.1556 fd = 5

21:32:17.050 [67427.67427] <2> vnet_check_vxss_client_magic_with_info_ex: [vnet_vxss_helper.c:1112] Ignoring VxSS authentication 2 0x2

21:32:17.051 [67427.67427] <2> db_CLIENTsend: reset client protocol version from 0 to 9

21:32:17.051 [67427.67427] <2> db_CLIENTsend: prec->offline=0

21:32:17.051 [67427.67427] <2> db_CLIENTsend: prec->online_at=0

21:32:17.051 [67427.67427] <2> db_CLIENTsend: prec->offline_at=0

21:32:17.051 [67427.67427] <2> db_CLIENTsend: prec->offlineres=0

21:32:17.051 [67427.67427] <2> db_CLIENTsend: prec->onlineres_at=0

21:32:17.051 [67427.67427] <2> db_CLIENTsend: prec->offlineres_at=0

21:32:17.051 [67427.67427] <2> db_CLIENTsend: sending 66 bytes

21:32:17.094 [67427.67427] <2> db_CLIENTreceive: clientrec_protocol_version=9

21:32:17.094 [67427.67427] <2> db_CLIENTreceive: num_fi_info=0

21:32:17.094 [67427.67427] <2> db_CLIENTreceive: num_exdb_info=0

21:32:17.094 [67427.67427] <2> db_CLIENTreceive: offline=0

21:32:17.094 [67427.67427] <2> db_CLIENTreceive: online_at=0

21:32:17.094 [67427.67427] <2> db_CLIENTreceive: offline_at=0

21:32:17.094 [67427.67427] <2> db_CLIENTreceive: offlineres=0 onlineres_at=0 offlineres_at=0

21:32:17.094 [67427.67427] <2> db_end: Need to collect reply

21:32:17.094 [67427.67427] <2> isDONE: reply DONE 0 <?xml version="1.0" encoding="utf-8"?><StatusMsgDoc><StatusMsg NBUStatus="0" Severity="0" VXULoid="271" VXULtid="0"><MsgString>the&#32;requested&#32;operation&#32;was&#32;successfully&#32;completed</MsgString></StatusMsg></StatusMsgDoc>

21:32:17.094 [67427.67427] <2> isDONE: reply DONE 0

21:32:17.094 [67427.67427] <2> verify_client: ../bpbrm.c.42546: OST Proxy is Disabled: 0 0 0x00000000

21:32:17.094 [67427.67427] <2> verify_client: ../bpbrm.c.42605: Client Direct is Disabled:: 0 0 0x00000000

21:32:17.094 [67427.67427] <2> bpbrm start_bpcd_stat: ../bpbrm.c.23312: bpcd_client: cark2nnobkup-media9

21:32:17.094 [67427.67427] <2> bpbrm start_bpcd_stat: ../bpbrm.c.23315: bpcd_client_hostname: cark2nnobkup-media9

21:32:17.094 [67427.67427] <2> local_bpcr_connect: bpcr.c.345: connect_opts = 0x01000100 connect_opts2 = 0x01000100

21:32:17.094 [67427.67427] <2> local_bpcr_connect: bpcr.c.352: connect_opts = 0x01000100

21:32:17.094 [67427.67427] <2> local_bpcr_connect: bpcr.c.391: daemon_port_type = 0

21:32:17.095 [67427.67427] <2> retry_getaddrinfo: [vnet_addrinfo.c:910] using SVC=veritas_pbx PORT=1556

21:32:17.095 [67427.67427] <2> retry_getaddrinfo: [vnet_addrinfo.c:914] adjusted service name 1556

21:32:17.095 [67427.67427] <2> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1738] found via getaddrinfo NAME=cark2nnobkup-media9 SVC=veritas_pbx

21:32:17.096 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.096 [67427.67427] <2> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1738] found via getaddrinfo NAME=192.168.168.12 SVC=NULL

21:32:17.096 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.097 [67427.67427] <2> retry_getaddrinfo: [vnet_addrinfo.c:910] using SVC=vnetd PORT=13724

21:32:17.097 [67427.67427] <2> retry_getaddrinfo: [vnet_addrinfo.c:914] adjusted service name 13724

21:32:17.097 [67427.67427] <2> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1738] found via getaddrinfo NAME=cark2nnobkup-media9 SVC=vnetd

21:32:17.097 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.097 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.097 [67427.67427] <2> retry_getaddrinfo: [vnet_addrinfo.c:910] using SVC=bpcd PORT=13782

21:32:17.097 [67427.67427] <2> retry_getaddrinfo: [vnet_addrinfo.c:914] adjusted service name 13782

21:32:17.097 [67427.67427] <2> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1738] found via getaddrinfo NAME=cark2nnobkup-media9 SVC=bpcd

21:32:17.097 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.097 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.097 [67427.67427] <2> async_connect: [vnet_connect.c:1570] connect in progress 1 0x1

21:32:17.098 [67427.67427] <2> vnet_pbxConnect: pbxConnectEx Succeeded

21:32:17.098 [67427.67427] <2> do_pbx_service: [vnet_connect.c:2235] via PBX bpcd CONNECT FROM 192.168.168.4.56905 TO 192.168.168.12.1556 fd = 5

21:32:17.098 [67427.67427] <2> async_connect: [vnet_connect.c:1762] connect async CONNECT FROM 192.168.168.4.56905 TO 192.168.168.12.1556 fd = 5

21:32:17.098 [67427.67427] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO cark2nnobkup-media9 192.168.168.12 bpcd VIA pbx

21:32:17.098 [67427.67427] <2> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1662] file cache has expired NAME=NULL SVC=veritas_pbx

21:32:17.099 [67427.67427] <2> retry_getaddrinfo: [vnet_addrinfo.c:910] using SVC=veritas_pbx PORT=1556

21:32:17.099 [67427.67427] <2> retry_getaddrinfo: [vnet_addrinfo.c:914] adjusted service name 1556

21:32:17.099 [67427.67427] <2> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1738] found via getaddrinfo NAME=NULL SVC=veritas_pbx

21:32:17.100 [67427.67427] <2> logconnections: BPCD CONNECT FROM 192.168.168.4.56905 TO 192.168.168.12.1556 fd = 5

21:32:17.100 [67427.67427] <2> retry_getaddrinfo: [vnet_addrinfo.c:910] using SVC=veritas_pbx PORT=1556

21:32:17.100 [67427.67427] <2> retry_getaddrinfo: [vnet_addrinfo.c:914] adjusted service name 1556

21:32:17.100 [67427.67427] <2> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1738] found via getaddrinfo NAME=192.168.168.12 SVC=veritas_pbx

21:32:17.101 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.101 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.101 [67427.67427] <2> retry_getaddrinfo: [vnet_addrinfo.c:910] using SVC=vnetd PORT=13724

21:32:17.101 [67427.67427] <2> retry_getaddrinfo: [vnet_addrinfo.c:914] adjusted service name 13724

21:32:17.101 [67427.67427] <2> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1738] found via getaddrinfo NAME=192.168.168.12 SVC=vnetd

21:32:17.101 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.101 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.101 [67427.67427] <2> async_connect: [vnet_connect.c:1570] connect in progress 1 0x1

21:32:17.101 [67427.67427] <2> vnet_pbxConnect: pbxConnectEx Succeeded

21:32:17.102 [67427.67427] <8> do_pbx_service: [vnet_connect.c:2227] via PBX VNETD CONNECT FROM 192.168.168.4.45353 TO 192.168.168.12.1556 fd = 6

21:32:17.102 [67427.67427] <2> async_connect: [vnet_connect.c:1762] connect async CONNECT FROM 192.168.168.4.45353 TO 192.168.168.12.1556 fd = 6

21:32:17.102 [67427.67427] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO 192.168.168.12 192.168.168.12 vnetd VIA pbx

[T_N]

Media server bpbrm log detail:  part 2:

21:32:17.103 [67427.67427] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:440] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa

21:32:17.142 [67427.67427] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:457] ipc_string /usr/openv/var/tmp/vnet-41271529616737141194000016812-exNMNj

21:32:17.182 [67427.67427] <2> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:468] hash_str1 8fbe79c4d47a2cd449834db148ba8cdf

21:32:17.222 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.222 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.222 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.222 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.222 [67427.67427] <2> async_connect: [vnet_connect.c:1570] connect in progress 1 0x1

21:32:17.223 [67427.67427] <2> vnet_pbxConnect: pbxConnectEx Succeeded

21:32:17.223 [67427.67427] <8> do_pbx_service: [vnet_connect.c:2227] via PBX VNETD CONNECT FROM 192.168.168.4.55348 TO 192.168.168.3.1556 fd = 7

21:32:17.223 [67427.67427] <2> async_connect: [vnet_connect.c:1762] connect async CONNECT FROM 192.168.168.4.55348 TO 192.168.168.3.1556 fd = 7

21:32:17.223 [67427.67427] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO cark2nnobkup-master_cluster 192.168.168.3 vnetd VIA pbx

21:32:17.223 [67427.67427] <8> vnet_vnetd_get_master_useat_info: [vnet_vnetd.c:3135] VN_REQUEST_GET_SECURITY_INFO 9 0x9

21:32:17.264 [67427.67427] <8> vnet_vnetd_disconnect: [vnet_vnetd.c:192] VN_REQUEST_DISCONNECT 1 0x1

21:32:17.264 [67427.67427] <2> bpcr_get_version_rqst: bpcd version: 07720000

21:32:17.264 [67427.67427] <2> ddns_check: Entering

21:32:17.265 [67427.67427] <2> bpcr_get_hostname_rqst: Server hostname length = 19

21:32:17.265 [67427.67427] <2> bpbrm start_bpcd_stat: DATA_SOCK from bpcr = 5

21:32:17.265 [67427.67427] <2> bpbrm start_bpcd_stat: NAME_SOCK from bpcr = 6

21:32:17.265 [67427.67427] <2> verify_client: ../bpbrm.c.42750: hostname: cark2nnobkup-media1

21:32:17.265 [67427.67427] <2> verify_client: ../bpbrm.c.42751: is_on: 0 0 0x00000000

21:32:17.265 [67427.67427] <2> bpcr_get_version_rqst: bpcd version: 07720000

21:32:17.265 [67427.67427] <2> verify_client: ../bpbrm.c.42754: bpcr_update_resiliency : 0 0 0x00000000

21:32:17.265 [67427.67427] <2> verify_client: ../bpbrm.c.42779: *connect_opts: 16843008 16843008 0x01010100

21:32:17.265 [67427.67427] <2> read_legacy_touch_file: Found /usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS_DISK; requested from (miscsvr.c.521).

21:32:17.266 [67427.67427] <2> read_legacy_touch_file: 524288 read ; requested from (miscsvr.c.521).

21:32:17.266 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.266 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.266 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.266 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.266 [67427.67427] <2> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1662] file cache has expired NAME=cark2nnobkup-master_cluster SVC=bpjobd

21:32:17.269 [67427.67427] <2> retry_getaddrinfo: [vnet_addrinfo.c:888] using IANA SVC=bpjobd PORT=13723

21:32:17.269 [67427.67427] <2> retry_getaddrinfo: [vnet_addrinfo.c:914] adjusted service name 13723

21:32:17.269 [67427.67427] <2> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1738] found via getaddrinfo NAME=cark2nnobkup-master_cluster SVC=bpjobd

21:32:17.269 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.269 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.269 [67427.67427] <2> async_connect: [vnet_connect.c:1570] connect in progress 1 0x1

21:32:17.270 [67427.67427] <2> vnet_pbxConnect: pbxConnectEx Succeeded

21:32:17.270 [67427.67427] <2> do_pbx_service: [vnet_connect.c:2235] via PBX bpjobd CONNECT FROM 192.168.168.4.50722 TO 192.168.168.3.1556 fd = 5

21:32:17.270 [67427.67427] <2> async_connect: [vnet_connect.c:1762] connect async CONNECT FROM 192.168.168.4.50722 TO 192.168.168.3.1556 fd = 5

21:32:17.270 [67427.67427] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO cark2nnobkup-master_cluster 192.168.168.3 bpjobd VIA pbx

21:32:17.270 [67427.67427] <2> job_connect: SO_KEEPALIVE set on socket 5 for client cark2nnobkup-master_cluster

21:32:17.270 [67427.67427] <2> logconnections: BPJOBD CONNECT FROM 192.168.168.4.50722 TO 192.168.168.3.1556 fd = 5

21:32:17.270 [67427.67427] <2> job_authenticate_connection: VxSS authentication check for now...

21:32:17.270 [67427.67427] <8> vnet_get_user_credential_path: [vnet_vxss.c:1474] status 35 0x23

21:32:17.270 [67427.67427] <8> vnet_check_user_certificate: [vnet_vxss_helper.c:3643] vnet_get_user_credential_path failed 35 0x23

21:32:17.270 [67427.67427] <2> vnet_check_vxss_client_magic_with_info_ex: [vnet_vxss_helper.c:1112] Ignoring VxSS authentication 2 0x2

21:32:17.271 [67427.67427] <2> job_connect: Connected to the host cark2nnobkup-master_cluster contype 53 jobid <129572> socket <5>

21:32:17.271 [67427.67427] <2> job_connect: Connected on port 50722

21:32:17.271 [67427.67427] <2> set_job_details: Tfile (129572): LOG 1529616737 4 bpbrm 67427 cark2nnobkup-media9 is the host to backup data from

21:32:17.271 [67427.67427] <2> send_job_file: job ID 129572, ftype = 3 msg len = 81, msg = LOG 1529616737 4 bpbrm 67427 cark2nnobkup-media9 is the host to backup data from

21:32:17.271 [67427.67427] <2> set_job_details: Tfile (129572): LOG 1529616737 4 bpbrm 67427 reading file list for client

21:32:17.271 [67427.67427] <2> send_job_file: job ID 129572, ftype = 3 msg len = 58, msg = LOG 1529616737 4 bpbrm 67427 reading file list for client

21:32:17.271 [67427.67427] <2> bpbrm main: File read = "/etc/hosts"

21:32:17.271 [67427.67427] <2> bpbrm main: File read = "CONTINUE"

21:32:17.271 [67427.67427] <2> bpbrm main: send_3_0_client_params = 1

21:32:17.272 [67427.67427] <2> local_bpcr_connect: bpcr.c.345: connect_opts = 0x01000100 connect_opts2 = 0x01000100

21:32:17.272 [67427.67427] <2> local_bpcr_connect: bpcr.c.352: connect_opts = 0x01000100

21:32:17.272 [67427.67427] <2> local_bpcr_connect: bpcr.c.391: daemon_port_type = 0

21:32:17.272 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.272 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.272 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.272 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.272 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.272 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.272 [67427.67427] <2> async_connect: [vnet_connect.c:1570] connect in progress 1 0x1

21:32:17.272 [67427.67427] <2> vnet_pbxConnect: pbxConnectEx Succeeded

21:32:17.272 [67427.67427] <2> do_pbx_service: [vnet_connect.c:2235] via PBX bpcd CONNECT FROM 192.168.168.4.42625 TO 192.168.168.12.1556 fd = 6

21:32:17.272 [67427.67427] <2> async_connect: [vnet_connect.c:1762] connect async CONNECT FROM 192.168.168.4.42625 TO 192.168.168.12.1556 fd = 6

21:32:17.272 [67427.67427] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO cark2nnobkup-media9 192.168.168.12 bpcd VIA pbx

21:32:17.272 [67427.67427] <2> logconnections: BPCD CONNECT FROM 192.168.168.4.42625 TO 192.168.168.12.1556 fd = 6

21:32:17.272 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.272 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.272 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.272 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.272 [67427.67427] <2> async_connect: [vnet_connect.c:1570] connect in progress 1 0x1

21:32:17.273 [67427.67427] <2> vnet_pbxConnect: pbxConnectEx Succeeded

21:32:17.274 [67427.67427] <8> do_pbx_service: [vnet_connect.c:2227] via PBX VNETD CONNECT FROM 192.168.168.4.40474 TO 192.168.168.12.1556 fd = 7

21:32:17.274 [67427.67427] <2> async_connect: [vnet_connect.c:1762] connect async CONNECT FROM 192.168.168.4.40474 TO 192.168.168.12.1556 fd = 7

21:32:17.274 [67427.67427] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO 192.168.168.12 192.168.168.12 vnetd VIA pbx

21:32:17.274 [67427.67427] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:440] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa

21:32:17.314 [67427.67427] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:457] ipc_string /usr/openv/var/tmp/vnet-41273529616737312551000016812-visbkk

21:32:17.353 [67427.67427] <2> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:468] hash_str1 114679b6250f3f6d2fee43ab85c54dc0

21:32:17.393 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.393 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.393 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.393 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.393 [67427.67427] <2> async_connect: [vnet_connect.c:1570] connect in progress 1 0x1

21:32:17.394 [67427.67427] <2> vnet_pbxConnect: pbxConnectEx Succeeded

21:32:17.394 [67427.67427] <8> do_pbx_service: [vnet_connect.c:2227] via PBX VNETD CONNECT FROM 192.168.168.4.44548 TO 192.168.168.3.1556 fd = 9

21:32:17.394 [67427.67427] <2> async_connect: [vnet_connect.c:1762] connect async CONNECT FROM 192.168.168.4.44548 TO 192.168.168.3.1556 fd = 9

21:32:17.394 [67427.67427] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO cark2nnobkup-master_cluster 192.168.168.3 vnetd VIA pbx

21:32:17.394 [67427.67427] <8> vnet_vnetd_get_master_useat_info: [vnet_vnetd.c:3135] VN_REQUEST_GET_SECURITY_INFO 9 0x9

21:32:17.435 [67427.67427] <8> vnet_vnetd_disconnect: [vnet_vnetd.c:192] VN_REQUEST_DISCONNECT 1 0x1

21:32:17.435 [67427.67427] <2> bpcr_get_version_rqst: bpcd version: 07720000

21:32:17.435 [67427.67427] <2> bpcr_get_version_rqst: bpcd version: 07720000

21:32:17.435 [67427.67427] <2> bpcr_get_charset_rqst: bpcd charset: 00000001

[T_N]

Media server bpbrm log detail:  part 3:

21:32:17.435 [67427.67427] <8> vnet_get_user_credential_path: [vnet_vxss.c:1474] status 35 0x23

21:32:17.435 [67427.67427] <8> vnet_check_user_certificate: [vnet_vxss_helper.c:3643] vnet_get_user_credential_path failed 35 0x23

21:32:17.435 [67427.67427] <2> ConnectionCache::connectAndCache: Acquiring new connection for host cark2nnobkup-master_cluster, query type 89

21:32:17.435 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.435 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.435 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.435 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.435 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.435 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.436 [67427.67427] <2> async_connect: [vnet_connect.c:1570] connect in progress 1 0x1

21:32:17.436 [67427.67427] <2> vnet_pbxConnect: pbxConnectEx Succeeded

21:32:17.436 [67427.67427] <2> do_pbx_service: [vnet_connect.c:2235] via PBX bpdbm CONNECT FROM 192.168.168.4.52732 TO 192.168.168.3.1556 fd = 6

21:32:17.436 [67427.67427] <2> async_connect: [vnet_connect.c:1762] connect async CONNECT FROM 192.168.168.4.52732 TO 192.168.168.3.1556 fd = 6

21:32:17.436 [67427.67427] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO cark2nnobkup-master_cluster 192.168.168.3 bpdbm VIA pbx

21:32:17.436 [67427.67427] <2> logconnections: BPDBM CONNECT FROM 192.168.168.4.52732 TO 192.168.168.3.1556 fd = 6

21:32:17.436 [67427.67427] <2> vnet_check_vxss_client_magic_with_info_ex: [vnet_vxss_helper.c:1112] Ignoring VxSS authentication 2 0x2

21:32:17.438 [67427.67427] <2> db_end: Need to collect reply

21:32:17.459 [67427.67427] <2> isDONE: reply DONE 0 <?xml version="1.0" encoding="utf-8"?><StatusMsgDoc><StatusMsg NBUStatus="0" Severity="0" VXULoid="271" VXULtid="0"><MsgString>the&#32;requested&#32;operation&#32;was&#32;successfully&#32;completed</MsgString></StatusMsg></StatusMsgDoc>

21:32:17.459 [67427.67427] <2> isDONE: reply DONE 0

21:32:17.459 [67427.67427] <2> bpbrm start_bpcd_stat: ../bpbrm.c.23312: bpcd_client: cark2nnobkup-media9

21:32:17.459 [67427.67427] <2> bpbrm start_bpcd_stat: ../bpbrm.c.23315: bpcd_client_hostname: cark2nnobkup-media9

21:32:17.459 [67427.67427] <2> local_bpcr_connect: bpcr.c.345: connect_opts = 0x01010100 connect_opts2 = 0x01000100

21:32:17.459 [67427.67427] <2> local_bpcr_connect: bpcr.c.352: connect_opts = 0x01010100

21:32:17.459 [67427.67427] <2> local_bpcr_connect: bpcr.c.391: daemon_port_type = 1

21:32:17.459 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.459 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.459 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.459 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.459 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.459 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.459 [67427.67427] <2> async_connect: [vnet_connect.c:1570] connect in progress 1 0x1

21:32:17.460 [67427.67427] <2> vnet_pbxConnect: pbxConnectEx Succeeded

21:32:17.460 [67427.67427] <2> do_pbx_service: [vnet_connect.c:2235] via PBX bpcd CONNECT FROM 192.168.168.4.47030 TO 192.168.168.12.1556 fd = 6

21:32:17.460 [67427.67427] <2> async_connect: [vnet_connect.c:1762] connect async CONNECT FROM 192.168.168.4.47030 TO 192.168.168.12.1556 fd = 6

21:32:17.460 [67427.67427] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO cark2nnobkup-media9 192.168.168.12 bpcd VIA pbx

21:32:17.460 [67427.67427] <2> logconnections: BPCD CONNECT FROM 192.168.168.4.47030 TO 192.168.168.12.1556 fd = 6

21:32:17.460 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.460 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.460 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.460 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.460 [67427.67427] <2> async_connect: [vnet_connect.c:1570] connect in progress 1 0x1

21:32:17.460 [67427.67427] <2> vnet_pbxConnect: pbxConnectEx Succeeded

21:32:17.461 [67427.67427] <8> do_pbx_service: [vnet_connect.c:2227] via PBX VNETD CONNECT FROM 192.168.168.4.48475 TO 192.168.168.12.1556 fd = 7

21:32:17.461 [67427.67427] <2> async_connect: [vnet_connect.c:1762] connect async CONNECT FROM 192.168.168.4.48475 TO 192.168.168.12.1556 fd = 7

21:32:17.461 [67427.67427] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO 192.168.168.12 192.168.168.12 vnetd VIA pbx

21:32:17.461 [67427.67427] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:440] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa

21:32:17.501 [67427.67427] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:457] ipc_string /usr/openv/var/tmp/vnet-41275529616737500270000016812-XhcFTk

21:32:17.541 [67427.67427] <2> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:468] hash_str1 660771741dcd069e5d0aa2e825fbf006

21:32:17.581 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.581 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.581 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.581 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.581 [67427.67427] <2> async_connect: [vnet_connect.c:1570] connect in progress 1 0x1

21:32:17.582 [67427.67427] <2> vnet_pbxConnect: pbxConnectEx Succeeded

21:32:17.583 [67427.67427] <8> do_pbx_service: [vnet_connect.c:2227] via PBX VNETD CONNECT FROM 192.168.168.4.43823 TO 192.168.168.3.1556 fd = 9

21:32:17.583 [67427.67427] <2> async_connect: [vnet_connect.c:1762] connect async CONNECT FROM 192.168.168.4.43823 TO 192.168.168.3.1556 fd = 9

21:32:17.583 [67427.67427] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO cark2nnobkup-master_cluster 192.168.168.3 vnetd VIA pbx

21:32:17.583 [67427.67427] <8> vnet_vnetd_get_master_useat_info: [vnet_vnetd.c:3135] VN_REQUEST_GET_SECURITY_INFO 9 0x9

21:32:17.623 [67427.67427] <8> vnet_vnetd_disconnect: [vnet_vnetd.c:192] VN_REQUEST_DISCONNECT 1 0x1

21:32:17.623 [67427.67427] <2> bpcr_get_version_rqst: bpcd version: 07720000

21:32:17.623 [67427.67427] <2> ddns_check: Entering

21:32:17.624 [67427.67427] <2> bpcr_get_hostname_rqst: Server hostname length = 19

21:32:17.624 [67427.67427] <2> bpbrm start_bpcd_stat: DATA_SOCK from bpcr = 6

21:32:17.624 [67427.67427] <2> bpbrm start_bpcd_stat: NAME_SOCK from bpcr = 7

21:32:17.624 [67427.67427] <2> bpcr_get_version_rqst: bpcd version: 07720000

21:32:17.624 [67427.67427] <2> bpcr_get_platform_rqst: Server platform length = 17

21:32:17.624 [67427.67427] <2> bpcr_check_for_use_ofb_support: bpcd platform linuxR_x86_2.6.18

21:32:17.624 [67427.67427] <2> set_job_details: Tfile (129572): LOG 1529616737 4 bpbrm 67427 starting bpbkar on client

21:32:17.624 [67427.67427] <2> send_job_file: job ID 129572, ftype = 3 msg len = 55, msg = LOG 1529616737 4 bpbrm 67427 starting bpbkar on client

21:32:17.624 [67427.67427] <2> bpbrm main: Going to execute cmd </usr/openv/netbackup/bin/bpbkar bpbkar -r 604800 -ru root -dt 0 -to 0 -bpstart_time 1529616981 -clnt cark2nnobkup-media9 -class test -sched Full -st FULL -bpstart_to 300 -bpend_to 300 -read_to 3600 -ckpt_time 900 -blks_per_buffer 1024 -use_otm -fso -b cark2nnobkup-media9_1529616681 -kl 28 -use_ofb> on host <cark2nnobkup-media9>

21:32:17.624 [67427.67427] <2> bpbrm main: received bpcd success message

21:32:17.637 [67427.67427] <2> bpbrm main: client_pid=41276

21:32:17.637 [67427.67427] <2> set_job_details: Tfile (129572): LOG 1529616737 4 bpbkar 41276 Backup started

21:32:17.637 [67427.67427] <2> send_job_file: job ID 129572, ftype = 3 msg len = 45, msg = LOG 1529616737 4 bpbkar 41276 Backup started

21:32:17.637 [67427.67427] <2> bpbrm main: from client cark2nnobkup-media9: read client start message

21:32:17.637 [67427.67427] <2> bpbrm spawn_child: /usr/openv/netbackup/bin/bptm bptm -w -c cark2nnobkup-media9 -dpath cark2-dd890-1 -stunit cark2-adv-disk1 -cl test -bt 1529616681 -b cark2nnobkup-media9_1529616681 -st 0 -cj 1 -reqid -1525972491 -jm -brm -hostname cark2nnobkup-media9 -ru root -rclnt cark2nnobkup-media9 -rclnthostname cark2nnobkup-media9 -rl 0 -rp 604800 -sl Full -ct 0 -maxfrag 524288 -eari 0 -v -mediasvr cark2nnobkup-media1 -no_callback -connect_options 0x01010100 -jobid 129572 -jobgrpid 129572 -masterversion 771000 -bpbrm_shm_id 580354054 -blks_per_buffer 1024

21:32:17.638 [67427.67427] <2> set_job_details: Tfile (129572): LOG 1529616737 4 bpbrm 67427 bptm pid: 67428

21:32:17.638 [67427.67427] <2> send_job_file: job ID 129572, ftype = 3 msg len = 45, msg = LOG 1529616737 4 bpbrm 67427 bptm pid: 67428

21:32:17.638 [67427.67427] <2> bpbrm write_continue_backup: wrote CONTINUE BACKUP on COMM_SOCK <6>

21:32:17.638 [67427.67427] <2> write_file_names: buffering file name '/etc/hosts' for output

21:32:17.638 [67427.67427] <2> write_file_names: successfully wrote buffer to COMM_SOCK

21:32:17.638 [67427.67427] <2> bpbrm main: wrote CONTINUE on COMM_SOCK

21:32:17.638 [67427.67427] <2> bpbrm main: Value of PFI = 0

21:32:17.638 [67427.67427] <2> bpbrm main: ESTIMATE -1 -1 cark2nnobkup-media9_1529616681

21:32:17.638 [67427.67427] <2> bpbrm send_ping: PING

21:32:17.650 [67427.67427] <2> bpbrm main: ADDED FILES TO DB FOR cark2nnobkup-media9_1529616681 1 /etc/hosts

21:32:17.651 [67427.67427] <8> vnet_get_user_credential_path: [vnet_vxss.c:1474] status 35 0x23

21:32:17.651 [67427.67427] <8> vnet_check_user_certificate: [vnet_vxss_helper.c:3643] vnet_get_user_credential_path failed 35 0x23

21:32:17.651 [67427.67427] <2> ConnectionCache::connectAndCache: Acquiring new connection for host cark2nnobkup-master_cluster, query type 78

21:32:17.651 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.651 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.651 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.651 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.651 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

21:32:17.651 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

21:32:17.651 [67427.67427] <2> async_connect: [vnet_connect.c:1570] connect in progress 1 0x1

21:32:17.652 [67427.67427] <2> vnet_pbxConnect: pbxConnectEx Succeeded

21:32:17.652 [67427.67427] <2> do_pbx_service: [vnet_connect.c:2235] via PBX bpdbm CONNECT FROM 192.168.168.4.35345 TO 192.168.168.3.1556 fd = 9

21:32:17.652 [67427.67427] <2> async_connect: [vnet_connect.c:1762] connect async CONNECT FROM 192.168.168.4.35345 TO 192.168.168.3.1556 fd = 9

21:32:17.652 [67427.67427] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO cark2nnobkup-master_cluster 192.168.168.3 bpdbm VIA pbx

21:32:17.652 [67427.67427] <2> logconnections: BPDBM CONNECT FROM 192.168.168.4.35345 TO 192.168.168.3.1556 fd = 9

21:32:17.652 [67427.67427] <2> vnet_check_vxss_client_magic_with_info_ex: [vnet_vxss_helper.c:1112] Ignoring VxSS authentication 2 0x2

[T_N]

Media server bpbrm log detail:  part 4:

21:32:17.653 [67427.67427] <2> db_end: Need to collect reply

21:32:17.717 [67427.67427] <2> isDONE: reply DONE 0 <?xml version="1.0" encoding="utf-8"?><StatusMsgDoc><StatusMsg NBUStatus="0" Severity="0" VXULoid="271" VXULtid="0"><MsgString>the&#32;requested&#32;operation&#32;was&#32;successfully&#32;completed</MsgString></StatusMsg></StatusMsgDoc>

21:32:17.717 [67427.67427] <2> isDONE: reply DONE 0

21:32:17.717 [67427.67427] <2> bpbrm wait_for_child: start

21:32:49.302 [67427.67427] <2> Bpbrm_siginfo_print: 0: delay 0 signo SIGUSR1:10 code 0 pid 67428

21:32:49.302 [67427.67427] <2> bpbrm mm_sig: received ready signal from media manager

23:32:27.125 [67427.67427] <2> bpbrm wait_for_child: child exit_status = 42 signal_status = 0

23:32:27.125 [67427.67427] <2> bpbrm kill_child_process: start

23:32:27.125 [67427.67427] <2> bpbrm Exit: attempting to send mail to root on cark2nnobkup-media9

23:32:27.125 [67427.67427] <2> local_bpcr_connect: bpcr.c.345: connect_opts = 0x01010100 connect_opts2 = 0x01000100

23:32:27.125 [67427.67427] <2> local_bpcr_connect: bpcr.c.352: connect_opts = 0x01010100

23:32:27.125 [67427.67427] <2> local_bpcr_connect: bpcr.c.391: daemon_port_type = 1

23:32:27.127 [67427.67427] <2> retry_getaddrinfo: [vnet_addrinfo.c:910] using SVC=veritas_pbx PORT=1556

23:32:27.127 [67427.67427] <2> retry_getaddrinfo: [vnet_addrinfo.c:914] adjusted service name 1556

23:32:27.127 [67427.67427] <2> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1738] found via getaddrinfo NAME=cark2nnobkup-media9 SVC=veritas_pbx

23:32:27.128 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

23:32:27.128 [67427.67427] <2> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1738] found via getaddrinfo NAME=192.168.168.12 SVC=NULL

23:32:27.128 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

23:32:27.128 [67427.67427] <2> retry_getaddrinfo: [vnet_addrinfo.c:910] using SVC=vnetd PORT=13724

23:32:27.128 [67427.67427] <2> retry_getaddrinfo: [vnet_addrinfo.c:914] adjusted service name 13724

23:32:27.129 [67427.67427] <2> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1738] found via getaddrinfo NAME=cark2nnobkup-media9 SVC=vnetd

23:32:27.129 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

23:32:27.129 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

23:32:27.129 [67427.67427] <2> retry_getaddrinfo: [vnet_addrinfo.c:910] using SVC=bpcd PORT=13782

23:32:27.129 [67427.67427] <2> retry_getaddrinfo: [vnet_addrinfo.c:914] adjusted service name 13782

23:32:27.129 [67427.67427] <2> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1738] found via getaddrinfo NAME=cark2nnobkup-media9 SVC=bpcd

23:32:27.129 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

23:32:27.129 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

23:32:27.129 [67427.67427] <2> async_connect: [vnet_connect.c:1570] connect in progress 1 0x1

23:32:27.130 [67427.67427] <2> vnet_pbxConnect: pbxConnectEx Succeeded

23:32:27.130 [67427.67427] <2> do_pbx_service: [vnet_connect.c:2235] via PBX bpcd CONNECT FROM 192.168.168.4.41256 TO 192.168.168.12.1556 fd = 6

23:32:27.130 [67427.67427] <2> async_connect: [vnet_connect.c:1762] connect async CONNECT FROM 192.168.168.4.41256 TO 192.168.168.12.1556 fd = 6

23:32:27.130 [67427.67427] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO cark2nnobkup-media9 192.168.168.12 bpcd VIA pbx

23:32:27.130 [67427.67427] <2> logconnections: BPCD CONNECT FROM 192.168.168.4.41256 TO 192.168.168.12.1556 fd = 6

23:32:27.131 [67427.67427] <2> retry_getaddrinfo: [vnet_addrinfo.c:910] using SVC=veritas_pbx PORT=1556

23:32:27.131 [67427.67427] <2> retry_getaddrinfo: [vnet_addrinfo.c:914] adjusted service name 1556

23:32:27.131 [67427.67427] <2> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1738] found via getaddrinfo NAME=192.168.168.12 SVC=veritas_pbx

23:32:27.131 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

23:32:27.131 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

23:32:27.132 [67427.67427] <2> retry_getaddrinfo: [vnet_addrinfo.c:910] using SVC=vnetd PORT=13724

23:32:27.132 [67427.67427] <2> retry_getaddrinfo: [vnet_addrinfo.c:914] adjusted service name 13724

23:32:27.132 [67427.67427] <2> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1738] found via getaddrinfo NAME=192.168.168.12 SVC=vnetd

23:32:27.132 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

23:32:27.132 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

23:32:27.132 [67427.67427] <2> async_connect: [vnet_connect.c:1570] connect in progress 1 0x1

23:32:27.133 [67427.67427] <2> vnet_pbxConnect: pbxConnectEx Succeeded

23:32:27.134 [67427.67427] <8> do_pbx_service: [vnet_connect.c:2227] via PBX VNETD CONNECT FROM 192.168.168.4.32959 TO 192.168.168.12.1556 fd = 9

23:32:27.134 [67427.67427] <2> async_connect: [vnet_connect.c:1762] connect async CONNECT FROM 192.168.168.4.32959 TO 192.168.168.12.1556 fd = 9

23:32:27.134 [67427.67427] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO 192.168.168.12 192.168.168.12 vnetd VIA pbx

23:32:27.134 [67427.67427] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:440] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa

23:32:27.174 [67427.67427] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:457] ipc_string /usr/openv/var/tmp/vnet-41424529623947172263000016818-PdTuqa

23:32:27.213 [67427.67427] <2> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:468] hash_str1 4018546727e0e4af1e9372f16c7bf99d

23:32:27.253 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

23:32:27.253 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

23:32:27.253 [67427.67427] <2> vnet_sortaddrs: [vnet_addrinfo.c:3966] sorted addrs: 1 0x1

23:32:27.253 [67427.67427] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4897] using interface  ANY

23:32:27.254 [67427.67427] <2> async_connect: [vnet_connect.c:1570] connect in progress 1 0x1

23:32:27.254 [67427.67427] <2> vnet_pbxConnect: pbxConnectEx Succeeded

23:32:27.255 [67427.67427] <8> do_pbx_service: [vnet_connect.c:2227] via PBX VNETD CONNECT FROM 192.168.168.4.55944 TO 192.168.168.3.1556 fd = 10

23:32:27.255 [67427.67427] <2> async_connect: [vnet_connect.c:1762] connect async CONNECT FROM 192.168.168.4.55944 TO 192.168.168.3.1556 fd = 10

23:32:27.255 [67427.67427] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO cark2nnobkup-master_cluster 192.168.168.3 vnetd VIA pbx

23:32:27.255 [67427.67427] <8> vnet_vnetd_get_master_useat_info: [vnet_vnetd.c:3135] VN_REQUEST_GET_SECURITY_INFO 9 0x9

23:32:27.295 [67427.67427] <8> vnet_vnetd_disconnect: [vnet_vnetd.c:192] VN_REQUEST_DISCONNECT 1 0x1

23:32:27.296 [67427.67427] <2> bpcr_get_version_rqst: bpcd version: 07720000

23:32:27.296 [67427.67427] <2> bpbrm Exit: OUT_SOCK from bpcr = 6

23:32:27.296 [67427.67427] <2> bpbrm Exit: IN_SOCK from bpcr = 9

23:32:27.296 [67427.67427] <2> bpcr_get_version_rqst: bpcd version: 07720000

23:32:27.296 [67427.67427] <2> bpcr_get_version_rqst: bpcd version: 07720000

23:32:27.301 [67427.67427] <2> bpcr_get_version_rqst: bpcd version: 07720000

23:32:27.305 [67427.67427] <2> set_job_details: Tfile (129572): LOG 1529623947 4 bpbkar 41276 done. status: 42: network read failed

23:32:27.305 [67427.67427] <2> send_job_file: job ID 129572, ftype = 3 msg len = 68, msg = LOG 1529623947 4 bpbkar 41276 done. status: 42: network read failed

23:32:27.305 [67427.67427] <2> bpbrm Exit: client backup EXIT STATUS 42: network read failed

23:32:27.306 [67427.67427] <2> vnet_close_socket_safely: [vnet.c:1106] safe close 0 0x0

23:32:27.306 [67427.67427] <2> job_monitoring_exex: ACK disconnect

23:32:27.306 [67427.67427] <2> job_disconnect: Disconnected

[Marianne]

Seems there is still a misunderstanding....

Look at job details for the PIDs you need to trace through the logs:

bpbrm (pid=67427)

Info bpbkar (pid=41276)

Info bptm (pid=67428) backup child process is pid 67429

The bpbkar and bpbrm logs that you provided are correct, but not bptm.
The 2 bptm logs that you have posted only contain logging for early morning. 
You need to find the log for evening activity : 21:32 to 23:31.

The initial connectivity was good between :
MediaServer=cark2nnobkup-media1
client = cark2nnobkup-media9

06/21/2018 21:31:21 - connecting 
06/21/2018 21:31:22 - connected; connect time: 0:00:00
06/21/2018 21:31:53 - begin writing

Seems you did a test, just backing up /etc/hosts. This is not good choice, since /etc/hosts is just a symbolic link.

Anyhow, we see in bpbkar log that client cark2nnobkup-media9 finished sending data almost instantaneous:

21:32:17.649 [41276] <4> bpbkar expand_wildcards: end backup for filelist /etc/hosts
21:32:17.649 [41276] <4> bpbkar main: INF - Client completed sending data for backup
21:32:17.649 [41276] <2> bpbkar main: INF - Total Size:3858
21:32:17.649 [41276] <2> bpbkar delete_old_files_recur: INF - removing /usr/openv/netbackup/hardlink_info/root/hardlinks_1527012305
21:32:17.649 [41276] <4> bpbkar Exit: INF - bpbkar exit normal
21:32:17.649 [41276] <4> bpbkar Exit: INF - EXIT STATUS 0: the requested operation was successfully completed

From bpbrm, we see that the media server never received an update from bpbkar. 
It simply timed out after 2 hours:

21:32:49.302 [67427.67427] <2> bpbrm mm_sig: received ready signal from media manager
23:32:27.125 [67427.67427] <2> bpbrm wait_for_child: child exit_status = 42 signal_status = 0

You need to check bptm log to see what happened on the data path between the client and the media server in the 2 hours. 

Best that you get your network team to monitor comms between cark2nnobkup-media1 and cark2nnobkup-media9 to find out why initial comms was successful, but why there seems to be an issue during the data transfer.

[T_N]

I ran bptestbpcd on cark2nnobkup-media (media serrver), I don't see any error

[root@cark2nnobkup-media1 ~]# bptestbpcd -client cark2nnobkup-media9 -debug
21:08:23.712 [26057] <2> bptestbpcd: VERBOSE = 0
21:08:23.712 [26057] <8> vnet_get_user_credential_path: [vnet_vxss.c:1474] status 35 0x23
21:08:23.712 [26057] <8> vnet_check_user_certificate: [vnet_vxss_helper.c:3643] vnet_get_user_credential_path failed 35 0x23
21:08:23.712 [26057] <2> ConnectionCache::connectAndCache: Acquiring new connection for host cark2nnobkup-master_cluster, query type 223
21:08:23.714 [26057] <2> vnet_pbxConnect: pbxConnectEx Succeeded
21:08:23.714 [26057] <2> logconnections: BPDBM CONNECT FROM 192.168.168.4.56338 TO 192.168.168.3.1556 fd = 4
21:08:23.717 [26057] <2> db_CLIENTsend: reset client protocol version from 0 to 9
21:08:23.760 [26057] <2> db_end: Need to collect reply
21:08:23.768 [26057] <2> vnet_pbxConnect: pbxConnectEx Succeeded
21:08:23.769 [26057] <2> logconnections: BPCD CONNECT FROM 192.168.168.4.52522 TO 192.168.168.12.1556 fd = 4
21:08:23.770 [26057] <2> vnet_pbxConnect: pbxConnectEx Succeeded
21:08:23.772 [26057] <8> do_pbx_service: [vnet_connect.c:2227] via PBX VNETD CONNECT FROM 192.168.168.4.51192 TO 192.168.168.12.1556 fd = 5
21:08:23.773 [26057] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:440] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
21:08:23.812 [26057] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:457] ipc_string /usr/openv/var/tmp/vnet-52182529960903810282000017076-oxhC4A
21:08:23.893 [26057] <2> vnet_pbxConnect: pbxConnectEx Succeeded
21:08:23.894 [26057] <8> do_pbx_service: [vnet_connect.c:2227] via PBX VNETD CONNECT FROM 192.168.168.4.33076 TO 192.168.168.3.1556 fd = 6
21:08:23.894 [26057] <8> vnet_vnetd_get_master_useat_info: [vnet_vnetd.c:3135] VN_REQUEST_GET_SECURITY_INFO 9 0x9
21:08:23.934 [26057] <8> vnet_vnetd_disconnect: [vnet_vnetd.c:192] VN_REQUEST_DISCONNECT 1 0x1
21:08:23.935 [26057] <2> bpcr_get_version_rqst: bpcd version: 07720000
1 1 1
192.168.168.4:52522 -> 192.168.168.12:1556
192.168.168.4:51192 -> 192.168.168.12:1556
21:08:23.936 [26057] <2> vnet_pbxConnect: pbxConnectEx Succeeded
21:08:23.937 [26057] <8> do_pbx_service: [vnet_connect.c:2227] via PBX VNETD CONNECT FROM 192.168.168.4.43074 TO 192.168.168.12.1556 fd = 6
21:08:23.937 [26057] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:440] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
21:08:23.977 [26057] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:457] ipc_string /usr/openv/var/tmp/vnet-52183529960903975303000017076-VO3NzB
192.168.168.4:43074 -> 192.168.168.12:1556
<2>bptestbpcd: EXIT status = 0
21:08:24.017 [26057] <2> bptestbpcd: EXIT status = 0

[Marianne]

The problem is not with initial connectivity.

Please read through my previous post again.

[T_N]

I ran sas to get network information but it failed 

[root@cark2nnobkup-media9 sas-linux]# ./sas-linux -t 192.168.168.12 -b 20

-bash: ./sas-linux: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory

[root@cark2nnobkup-media9 sas-linux]# 

I installed glibc.i686 but still have error

[root@cark2nnobkup-media9 sas-linux]# yum -y install glibc.i686

[root@cark2nnobkup-media9 sas-linux]# ./sas-linux -t 192.168.168.12 -b 20 

./sas-linux: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory

[root@cark2nnobkup-media9 sas-linux]# 

I have no idea how to fix that, I open  new case to Veritas Support and still working with them.  My team says he had a similar issue with another data center 

where reading from one media server to another to write to tape was timing out.  

Veritas Support advise he did this  nbemcmd -changesetting -common_server_for_dup preferred -machinename xxxxx

I don't run this command line yet, still waiting for Veritas support advise on it. Any Advise on this command line ? Thank you.

[T_N · Accepted solution]

Hi All;

Finally I found the root cause and resolved the issue. It is MTU issue. Thank you everyone.