Socket write failed(24)

[knox890]

Team,

I am running backups on a windows 2008 server and I got an error syaing socket write failed(24)

The connection between master server and client is by using a private IP and this can be verified by running ping and bptestbpcd. All connections are okay but i am still having error. 

Size of file per drive = 350GB. It was previously running correctly but then right now, its having errors.

Can anyone help me with this?

May 31, 2012 9:12:49 AM - requesting resource atpsk01-hcart3-robot-tld-0

May 31, 2012 9:12:49 AM - requesting resource ATPBCKSP01.NBU_CLIENT.MAXJOBS.p1cl01n3

May 31, 2012 9:12:49 AM - requesting resource ATPBCKSP01.NBU_POLICY.MAXJOBS.Weekly_LNotes_P1P3

May 31, 2012 9:12:50 AM - granted resource  ATPBCKSP01.NBU_CLIENT.MAXJOBS.p1cl01n3

May 31, 2012 9:12:50 AM - granted resource  ATPBCKSP01.NBU_POLICY.MAXJOBS.Weekly_LNotes_P1P3

May 31, 2012 9:12:50 AM - granted resource  P10466

May 31, 2012 9:12:50 AM - granted resource  HP.ULTRIUM3-SCSI.005

May 31, 2012 9:12:50 AM - granted resource  atpsk01-hcart3-robot-tld-0

May 31, 2012 9:12:50 AM - estimated 1178294689 kbytes needed

May 31, 2012 9:12:51 AM - started process bpbrm (pid=23051)

May 31, 2012 9:13:26 AM - connecting

May 31, 2012 9:13:27 AM - connected; connect time: 0:00:00

May 31, 2012 9:13:35 AM - mounting P10466

May 31, 2012 9:14:17 AM - mounted P10466; mount time: 0:00:42

May 31, 2012 9:14:17 AM - positioning P10466 to file 4

May 31, 2012 9:15:35 AM - Warning bptm (pid=23058) read error on media id P10466, drive index 5 reading header block, No space left on device

May 31, 2012 9:16:50 AM - positioned P10466; position time: 0:02:33

May 31, 2012 9:16:50 AM - begin writing

May 31, 2012 9:16:50 AM - current media P10466 complete, requesting next media Any

May 31, 2012 9:16:51 AM - granted resource  P10733

May 31, 2012 9:16:51 AM - granted resource  HP.ULTRIUM3-SCSI.004

May 31, 2012 9:16:51 AM - granted resource  atpsk01-hcart3-robot-tld-0

May 31, 2012 9:16:52 AM - mounting P10733

May 31, 2012 9:17:34 AM - mounted P10733; mount time: 0:00:42

May 31, 2012 9:17:34 AM - positioning P10733 to file 1

May 31, 2012 9:17:42 AM - positioned P10733; position time: 0:00:08

May 31, 2012 9:17:42 AM - begin writing

May 31, 2012 10:12:07 AM - Critical bpbrm (pid=23051) from client p1cl01n3: FTL - socket write failed

May 31, 2012 10:12:10 AM - Error bptm (pid=23058) media manager terminated by parent process

May 31, 2012 10:12:55 AM - end writing; write time: 0:55:13

socket write failed  (24)

[Jackie007]

Check this forums.

https://www-secure.symantec.com/connect/forums/socket-write-failed24

http://www.symantec.com/business/support/index?pag...

[knox890]

Thanks Jackie007,

i tried that one too and still i am receiving socket write failed error.

can anyone help?

[Marianne]

Which OS is client p1cl01n3? NBU version?

Do you have bpbkar log on client p1cl01n3?
bpbrm and bptm logs on media server?

Above logs will tell you what happened in this hour:

May 31, 2012 9:17:42 AM - begin writing

May 31, 2012 10:12:07 AM - Critical bpbrm (pid=23051) from client p1cl01n3: FTL - socket write failed

[knox890]

P1CL01N3 = Windows 2k8 R2 Server

NBU Version = Netbackup 6.5.2

im not sure if i can locate bpbkar log on client p1cl01n3

but here's the log file for 

12:56:39.576 [6873] <2> bpbrm main: INITIATING (VERBOSE = 5): version NetBackup 6.5 2010042404

12:56:39.576 [6873] <2> logparams: -backup -S ATPBCKSP01 -c p1cl01n3 -ct 13 -ru root -cl Weekly_LNotes_P1P3 -sched Weekly_Full -bt 1338440198 -dt 0 -st 0 -b p1cl01n3_1338440198 -mediasvr atpsk01 -jobid 98075 -jobgrpid 98075 -masterversion 650000 -maxfrag 1048576 -reqid -1338361039 -mt 2 -to 0 -stunit atpsk01-hcart3-robot-tld-0 -rl 9 -rp 2147483647 -cj 6 -D 20 -rt 8 -rn 0 -pool Infinity_Pool -use_ofb -use_otm -WOFB_enabled 1 -WOFB_fim 1 -nbu -jm -secure 1 -kl 30 -rg root -fso -connect_options 16974338 

12:56:39.577 [6873] <2> verify_client: bpbrm.c.33198: *connect_opts: 16974338 16974338 0x01030202

12:56:39.578 [6873] <2> read_client: ?

12:56:39.578 [6873] <2> read_host_info: ?

12:56:39.578 [6873] <2> read_host_info: CONNECT_NON_RES_PORT 2

12:56:39.578 [6873] <2> read_host_info: DYNAMIC_ADDRESS 0

12:56:39.578 [6873] <2> read_host_info: FREE_BROWSE 0

12:56:39.578 [6873] <2> read_host_info: LIST_RESTORE 1

12:56:39.579 [6873] <2> read_host_info: MAX_JOBS_THIS_CLIENT 0

12:56:39.579 [6873] <2> read_host_info: NO_CALLBACK 2

12:56:39.579 [6873] <2> read_host_info: WOFB_ENABLED 1

12:56:39.579 [6873] <2> read_host_info: WOFB_FIM 1

12:56:39.579 [6873] <2> read_host_info: WOFB_USAGE 0

12:56:39.579 [6873] <2> read_host_info: WOFB_ERROR 0

12:56:39.579 [6873] <2> read_host_info: DAEMON_PORT_TYPE 3

12:56:39.579 [6873] <2> read_sec_info: ?

12:56:39.579 [6873] <2> read_sec_info: no sec_info: /usr/openv/netbackup/db/client/p1cl01n3/sec_info does not exist

12:56:39.579 [6873] <2> get_fi_info1: ?

12:56:39.579 [6873] <2> read_fi_info: ?

12:56:39.579 [6873] <2> read_fi_info: no fi_info files processed

12:56:39.579 [6873] <2> db_freeVALID_HOST: *plist is NULL

12:56:39.579 [6873] <2> db_freeSEC_INFO: pinfo is NULL

12:56:39.579 [6873] <2> db_freeFI_INFO: plist is NULL

12:56:39.579 [6873] <2> bpbrm start_bpcd_stat: bpbrm.c.19676: bpcd_client: p1cl01n3

12:56:39.579 [6873] <2> bpbrm start_bpcd_stat: bpbrm.c.19679: bpcd_client_hostname: p1cl01n3

12:56:39.581 [6873] <2> hosts_equal: Comparing hosts <p1cl01n3> and <p1cl01n3>

12:56:39.581 [6873] <2> hosts_equal: names are the same

12:56:39.621 [6873] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2048: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006

12:56:39.621 [6873] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2062: service: bpcd

12:56:39.821 [6873] <2> logconnections: BPCD CONNECT FROM 192.168.1.44.4960 TO 192.168.1.89.13724

12:56:39.823 [6873] <2> vnet_connect_to_vnetd_extra: vnet_vnetd.c.182: msg: VNETD CONNECT FROM 192.168.1.44.4880 TO 192.168.1.89.13724 fd = 8

12:56:39.859 [6873] <2> vnet_vnetd_connect_forward_socket_begin: vnet_vnetd.c.535: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a

12:56:40.070 [6873] <2> vnet_vnetd_connect_forward_socket_begin: vnet_vnetd.c.552: ipc_string: 61735

12:56:40.182 [6873] <2> bpcr_get_hostname_rqst: Server hostname length = 8

12:56:40.182 [6873] <2> bpbrm start_bpcd_stat: DATA_SOCK from bpcr = 7

12:56:40.182 [6873] <2> bpbrm start_bpcd_stat: NAME_SOCK from bpcr = 8

12:56:40.183 [6873] <2> verify_client: bpbrm.c.33390: *client_hostname: p1cl01n3

12:56:40.183 [6873] <2> verify_client: bpbrm.c.33392: *connect_opts: 16843008 16843008 0x01010100

12:56:40.183 [6873] <2> read_client: ?

12:56:40.183 [6873] <2> read_host_info: ?

12:56:40.183 [6873] <2> read_host_info: CONNECT_NON_RES_PORT 2

12:56:40.183 [6873] <2> read_host_info: DYNAMIC_ADDRESS 0

12:56:40.183 [6873] <2> read_host_info: FREE_BROWSE 0

12:56:40.183 [6873] <2> read_host_info: LIST_RESTORE 1

12:56:40.183 [6873] <2> read_host_info: MAX_JOBS_THIS_CLIENT 0

12:56:40.183 [6873] <2> read_host_info: NO_CALLBACK 2

12:56:40.183 [6873] <2> read_host_info: WOFB_ENABLED 1

12:56:40.183 [6873] <2> read_host_info: WOFB_FIM 1

12:56:40.183 [6873] <2> read_host_info: WOFB_USAGE 0

12:56:40.183 [6873] <2> read_host_info: WOFB_ERROR 0

12:56:40.183 [6873] <2> read_host_info: DAEMON_PORT_TYPE 3

12:56:40.183 [6873] <2> read_sec_info: ?

12:56:40.183 [6873] <2> read_sec_info: no sec_info: /usr/openv/netbackup/db/client/p1cl01n3/sec_info does not exist

12:56:40.183 [6873] <2> get_fi_info1: ?

12:56:40.183 [6873] <2> read_fi_info: ?

12:56:40.184 [6873] <2> read_fi_info: no fi_info files processed

12:56:40.184 [6873] <2> db_freeVALID_HOST: *plist is NULL

12:56:40.184 [6873] <2> db_freeSEC_INFO: pinfo is NULL

12:56:40.184 [6873] <2> db_freeFI_INFO: plist is NULL

12:56:40.184 [6873] <2> hosts_equal: Comparing hosts <p1cl01n3> and <p1cl01n3>

12:56:40.184 [6873] <2> hosts_equal: names are the same

12:56:40.215 [6873] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2048: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006

12:56:40.215 [6873] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2062: service: bpcd

12:56:40.430 [6873] <2> logconnections: BPCD CONNECT FROM 192.168.1.44.4877 TO 192.168.1.89.13724

12:56:40.431 [6873] <2> vnet_connect_to_vnetd_extra: vnet_vnetd.c.182: msg: VNETD CONNECT FROM 192.168.1.44.4869 TO 192.168.1.89.13724 fd = 8

12:56:40.461 [6873] <2> vnet_vnetd_connect_forward_socket_begin: vnet_vnetd.c.535: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a

12:56:40.663 [6873] <2> vnet_vnetd_connect_forward_socket_begin: vnet_vnetd.c.552: ipc_string: 61738

12:56:40.722 [6873] <2> local_getAllBEorNBHostInfo: CLIENT_CMD_SOCK from bpcr = 7

12:56:40.782 [6873] <2> bpcr_get_version_rqst: bpcd version: 06520000

12:56:40.842 [6873] <2> hosts_equal: Comparing hosts <atpbackupsp0144p> and <p1cl01n3>

12:56:40.843 [6873] <2> hosts_equal: host1 atpbackupsp0144p atpbackupsp0144p addr 192.168.1.44 (0xc0a8012c)

12:56:40.843 [6873] <2> hosts_equal: host2 p1cl01n3 p1cl01n3 addr 192.168.1.89 (0xc0a80159)

12:56:40.843 [6873] <2> hosts_equal: hostnames DO NOT compare

12:56:40.843 [6873] <2> hosts_equal: name1=atpbackupsp0144p  name2=p1cl01n3

12:56:40.843 [6873] <2> hosts_equal: hostbuf=atpbackupsp0144p  hostentry->h_name=p1cl01n3

12:56:40.843 [6873] <2> hosts_equal: there were NO aliases for p1cl01n3

12:56:40.843 [6873] <2> hosts_equal: there were NO aliases for atpbackupsp0144p

12:56:40.843 [6873] <2> hosts_equal: addresses DO NOT match

12:56:40.843 [6873] <2> hosts_equal: atpbackupsp0144p=0xc0a8012c p1cl01n3=0xc0a80159

12:56:40.843 [6873] <2> local_getAllBEorNBHostInfo: hostname comparison failed

12:56:40.843 [6873] <2> hosts_equal: Comparing hosts <atpbackupsp01> and <p1cl01n3>

12:56:40.843 [6873] <2> hosts_equal: host1 atpbackupsp01 ATPBACKUPSP01 addr 10.199.9.86 (0xac70956)

12:56:40.843 [6873] <2> hosts_equal: host2 p1cl01n3 p1cl01n3 addr 192.168.1.89 (0xc0a80159)

12:56:40.843 [6873] <2> hosts_equal: hostnames DO NOT compare

12:56:40.843 [6873] <2> hosts_equal: name1=atpbackupsp01  name2=p1cl01n3

12:56:40.843 [6873] <2> hosts_equal: hostbuf=ATPBACKUPSP01  hostentry->h_name=p1cl01n3

12:56:40.843 [6873] <2> hosts_equal: there were NO aliases for p1cl01n3

12:56:40.843 [6873] <2> hosts_equal: compare p1cl01n3 with alias ATPBCKSP01

12:56:40.843 [6873] <2> hosts_equal: compare p1cl01n3 with alias ATPSK01

12:56:40.843 [6873] <2> hosts_equal: compare p1cl01n3 with alias loghost

12:56:40.843 [6873] <2> hosts_equal: compare p1cl01n3 with alias ATPBACKUPSP01.amkor.com

12:56:40.843 [6873] <2> hosts_equal: addresses DO NOT match

12:56:40.843 [6873] <2> hosts_equal: atpbackupsp01=0xac70956 p1cl01n3=0xc0a80159

12:56:40.843 [6873] <2> local_getAllBEorNBHostInfo: hostname comparison failed

12:56:40.843 [6873] <2> hosts_equal: Comparing hosts <atpsk01> and <p1cl01n3>

12:56:40.844 [6873] <2> hosts_equal: host1 atpsk01 ATPBACKUPSP01 addr 10.199.9.86 (0xac70956)

12:56:40.844 [6873] <2> hosts_equal: host2 p1cl01n3 p1cl01n3 addr 192.168.1.89 (0xc0a80159)

12:56:40.844 [6873] <2> hosts_equal: hostnames DO NOT compare

12:56:40.844 [6873] <2> hosts_equal: name1=atpsk01  name2=p1cl01n3

12:56:40.844 [6873] <2> hosts_equal: hostbuf=ATPBACKUPSP01  hostentry->h_name=p1cl01n3

12:56:40.844 [6873] <2> hosts_equal: there were NO aliases for p1cl01n3

12:56:40.844 [6873] <2> hosts_equal: compare p1cl01n3 with alias ATPBCKSP01

12:56:40.844 [6873] <2> hosts_equal: compare p1cl01n3 with alias ATPSK01

12:56:40.844 [6873] <2> hosts_equal: compare p1cl01n3 with alias loghost

12:56:40.844 [6873] <2> hosts_equal: compare p1cl01n3 with alias ATPBACKUPSP01.amkor.com

12:56:40.844 [6873] <2> hosts_equal: addresses DO NOT match

12:56:40.844 [6873] <2> hosts_equal: atpsk01=0xac70956 p1cl01n3=0xc0a80159

12:56:40.844 [6873] <2> local_getAllBEorNBHostInfo: hostname comparison failed

12:56:40.844 [6873] <2> hosts_equal: Comparing hosts <atpbcksp01> and <p1cl01n3>

12:56:40.844 [6873] <2> hosts_equal: host1 atpbcksp01 ATPBACKUPSP01 addr 10.199.9.86 (0xac70956)

12:56:40.844 [6873] <2> hosts_equal: host2 p1cl01n3 p1cl01n3 addr 192.168.1.89 (0xc0a80159)

12:56:40.844 [6873] <2> hosts_equal: hostnames DO NOT compare

12:56:40.844 [6873] <2> hosts_equal: name1=atpbcksp01  name2=p1cl01n3

12:56:40.844 [6873] <2> hosts_equal: hostbuf=ATPBACKUPSP01  hostentry->h_name=p1cl01n3

12:56:40.844 [6873] <2> hosts_equal: there were NO aliases for p1cl01n3

12:56:40.844 [6873] <2> hosts_equal: compare p1cl01n3 with alias ATPBCKSP01

12:56:40.844 [6873] <2> hosts_equal: compare p1cl01n3 with alias ATPSK01

12:56:40.844 [6873] <2> hosts_equal: compare p1cl01n3 with alias loghost

12:56:40.844 [6873] <2> hosts_equal: compare p1cl01n3 with alias ATPBACKUPSP01.amkor.com

12:56:40.844 [6873] <2> hosts_equal: addresses DO NOT match

12:56:40.844 [6873] <2> hosts_equal: atpbcksp01=0xac70956 p1cl01n3=0xc0a80159

12:56:40.844 [6873] <2> local_getAllBEorNBHostInfo: hostname comparison failed

12:56:40.844 [6873] <2> hosts_equal: Comparing hosts <atpbackupsp011p> and <p1cl01n3>

12:56:40.844 [6873] <2> hosts_equal: host1 atpbackupsp011p atpbackupsp011p addr 10.155.2.1 (0xa9b0201)

12:56:40.844 [6873] <2> hosts_equal: host2 p1cl01n3 p1cl01n3 addr 192.168.1.89 (0xc0a80159)

12:56:40.844 [6873] <2> hosts_equal: hostnames DO NOT compare

12:56:40.844 [6873] <2> hosts_equal: name1=atpbackupsp011p  name2=p1cl01n3

12:56:40.844 [6873] <2> hosts_equal: hostbuf=atpbackupsp011p  hostentry->h_name=p1cl01n3

12:56:40.845 [6873] <2> hosts_equal: there were NO aliases for p1cl01n3

12:56:40.845 [6873] <2> hosts_equal: there were NO aliases for atpbackupsp011p

12:56:40.845 [6873] <2> hosts_equal: addresses DO NOT match

12:56:40.845 [6873] <2> hosts_equal: atpbackupsp011p=0xa9b0201 p1cl01n3=0xc0a80159

12:56:40.845 [6873] <2> local_getAllBEorNBHostInfo: hostname comparison failed

12:56:40.845 [6873] <2> hosts_equal: Comparing hosts <atpsk01> and <p1cl01n3>

12:56:40.845 [6873] <2> hosts_equal: host1 atpsk01 ATPBACKUPSP01 addr 10.199.9.86 (0xac70956)

12:56:40.845 [6873] <2> hosts_equal: host2 p1cl01n3 p1cl01n3 addr 192.168.1.89 (0xc0a80159)

12:56:40.845 [6873] <2> hosts_equal: hostnames DO NOT compare

12:56:40.845 [6873] <2> hosts_equal: name1=atpsk01  name2=p1cl01n3

12:56:40.845 [6873] <2> hosts_equal: hostbuf=ATPBACKUPSP01  hostentry->h_name=p1cl01n3

12:56:40.845 [6873] <2> hosts_equal: there were NO aliases for p1cl01n3

12:56:40.845 [6873] <2> hosts_equal: compare p1cl01n3 with alias ATPBCKSP01

12:56:40.845 [6873] <2> hosts_equal: compare p1cl01n3 with alias ATPSK01

12:56:40.845 [6873] <2> hosts_equal: compare p1cl01n3 with alias loghost

12:56:40.845 [6873] <2> hosts_equal: compare p1cl01n3 with alias ATPBACKUPSP01.amkor.com

12:56:40.845 [6873] <2> hosts_equal: addresses DO NOT match

12:56:40.845 [6873] <2> hosts_equal: atpsk01=0xac70956 p1cl01n3=0xc0a80159

12:56:40.845 [6873] <2> local_getAllBEorNBHostInfo: hostname comparison failed

12:56:40.902 [6873] <2> bpcr_get_version_rqst: bpcd version: 06520000

12:56:40.902 [6873] <2> local_getAllBEorNBHostInfo: rqstType == BPCD_GET_NB_VERSION_RQST

12:56:41.022 [6873] <2> bpcr_get_platform_rqst: Server client platform length = 7

12:56:41.023 [6873] <2> xlatePlatformName: my version = 650000, client version = 650000, check current list first

12:56:41.023 [6873] <2> xlatePlatformName: BUILD_PLATFORM name to xlate = win_x64

12:56:41.023 [6873] <2> local_getAllBEorNBHostInfo: platform of client p1cl01n3 is PC-x64, WindowsXP

12:56:41.082 [6873] <2> bpcr_get_uname_rqst: bpcr.c: 3875 count=24, buffer=UNAME_SYSNAME = UNKNOWN 

12:56:41.082 [6873] <2> bpcr_get_uname_rqst: bpcr.c: 3875 count=26, buffer=UNAME_NODENAME = P1CL01N3 

12:56:41.082 [6873] <2> bpcr_get_uname_rqst: bpcr.c: 3875 count=18, buffer=UNAME_RELEASE = 6 

12:56:41.082 [6873] <2> bpcr_get_uname_rqst: bpcr.c: 3875 count=18, buffer=UNAME_VERSION = 1 

12:56:41.082 [6873] <2> bpcr_get_uname_rqst: bpcr.c: 3875 count=52, buffer=UNAME_MACHINE = Unknown - Unknown -  Revision x2e06 

12:56:41.083 [6873] <2> bpcr_get_uname_rqst: bpcr.c: 3875 count=3, buffer=END

12:56:41.083 [6873] <2> local_getAllBEorNBHostInfo: dbmisc.c: 1784, uname -sr=UNKNOWN 6 

12:56:41.142 [6873] <2> bpcr_get_version_rqst: bpcd version: 06520000

12:56:41.263 [6873] <2> getPatchLevel: CLIENT_CMD_SOCK from bpcr = 7

12:56:41.322 [6873] <2> getPatchLevel: got client socket

12:56:41.322 [6873] <2> getPatchLevel: file num = 3, file size = 112

12:56:41.322 [6873] <2> bpcr_read_rqst: completed send of file number 3

12:56:41.322 [6873] <2> bpcr_read_rqst: completed send of transfer_count = 16383

12:56:41.322 [6873] <2> bpcr_read_rqst: Before readclientstatus on socket 8

12:56:41.382 [6873] <2> bpcr_read_rqst: Num bytes to be sent by client = 112

12:56:41.382 [6873] <2> getPatchLevel: total chars read = 112

12:56:41.382 [6873] <2> getPatchLevel: level = >6.5.2<

12:56:41.443 [6873] <2> bpbrm main: Client NBU version: 650000

12:56:41.468 [6873] <2> bpbrm main: send_3_0_client_params = 1

12:56:41.468 [6873] <2> bpbrm main: backup_cmd = /usr/openv/netbackup/bin/bpbkar bpbkar -r 2147483647 -ru root -dt 0 -to 0 -clnt p1cl01n3 -class Weekly_LNotes_P1P3 -sched Weekly_Full -st FULL -bpstart_to 300 -bpend_to 300 -read_to 3600 -blks_per_buffer 128 -use_otm -fso -b p1cl01n3_1338440198 -kl 30 -WOFB_enabled -WOFB_fim 1 -WOFB_usage 0 -WOFB_error 0 -ct 13

12:56:41.469 [6873] <2> hosts_equal: Comparing hosts <p1cl01n3> and <ATPBCKSP01>

12:56:41.469 [6873] <2> hosts_equal: host1 p1cl01n3 p1cl01n3 addr 192.168.1.89 (0xc0a80159)

12:56:41.469 [6873] <2> hosts_equal: host2 ATPBCKSP01 ATPBACKUPSP01 addr 10.199.9.86 (0xac70956)

12:56:41.469 [6873] <2> hosts_equal: hostnames DO NOT compare

12:56:41.469 [6873] <2> hosts_equal: name1=p1cl01n3  name2=ATPBCKSP01

12:56:41.469 [6873] <2> hosts_equal: hostbuf=p1cl01n3  hostentry->h_name=ATPBACKUPSP01

12:56:41.469 [6873] <2> hosts_equal: compare p1cl01n3 with alias ATPBCKSP01

12:56:41.469 [6873] <2> hosts_equal: compare p1cl01n3 with alias ATPSK01

12:56:41.469 [6873] <2> hosts_equal: compare p1cl01n3 with alias loghost

12:56:41.469 [6873] <2> hosts_equal: compare p1cl01n3 with alias ATPBACKUPSP01.amkor.com

12:56:41.469 [6873] <2> hosts_equal: there were NO aliases for p1cl01n3

12:56:41.469 [6873] <2> hosts_equal: addresses DO NOT match

12:56:41.469 [6873] <2> hosts_equal: p1cl01n3=0xc0a80159 ATPBCKSP01=0xac70956

12:56:41.469 [6873] <2> bpbrm start_bpcd_stat: bpbrm.c.19676: bpcd_client: p1cl01n3

12:56:41.469 [6873] <2> bpbrm start_bpcd_stat: bpbrm.c.19679: bpcd_client_hostname: p1cl01n3

12:56:41.469 [6873] <2> hosts_equal: Comparing hosts <p1cl01n3> and <p1cl01n3>

12:56:41.469 [6873] <2> hosts_equal: names are the same

12:56:41.504 [6873] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2048: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006

12:56:41.504 [6873] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2062: service: bpcd

12:56:41.709 [6873] <2> logconnections: BPCD CONNECT FROM 192.168.1.44.4917 TO 192.168.1.89.13724

12:56:41.709 [6873] <2> vnet_connect_to_vnetd_extra: vnet_vnetd.c.182: msg: VNETD CONNECT FROM 192.168.1.44.4959 TO 192.168.1.89.13724 fd = 8

12:56:41.745 [6873] <2> vnet_vnetd_connect_forward_socket_begin: vnet_vnetd.c.535: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a

12:56:41.942 [6873] <2> vnet_vnetd_connect_forward_socket_begin: vnet_vnetd.c.552: ipc_string: 61740

12:56:42.062 [6873] <2> bpcr_get_hostname_rqst: Server hostname length = 8

12:56:42.062 [6873] <2> bpbrm start_bpcd_stat: DATA_SOCK from bpcr = 7

12:56:42.062 [6873] <2> bpbrm start_bpcd_stat: NAME_SOCK from bpcr = 8

12:56:42.124 [6873] <2> bpcr_get_version_rqst: bpcd version: 06520000

12:56:42.182 [6873] <2> bpcr_get_platform_rqst: Server client platform length = 7

12:56:42.182 [6873] <2> bpcr_check_for_use_ofb_support: bpcd platform win_x64

12:56:42.182 [6873] <2> bpbrm main: received bpcd success message

12:56:45.184 [6873] <2> get_readline_type: Returning BPBRM_READ_INFORM_WHEN_DONE

12:56:45.242 [6873] <2> bpbrm main: from client p1cl01n3: read client start message

12:56:45.242 [6873] <2> bpbrm spawn_child: /usr/openv/netbackup/bin/bptm bptm -w -c p1cl01n3 -den 20 -rt 8 -rn 0 -stunit atpsk01-hcart3-robot-tld-0 -cl Weekly_LNotes_P1P3 -bt 1338440198 -b p1cl01n3_1338440198 -st 0 -cj 6 -p Infinity_Pool -reqid -1338361039 -jm -brm -hostname p1cl01n3 -ru root -rclnt p1cl01n3 -rclnthostname p1cl01n3 -rl 9 -rp 2147483647 -sl Weekly_Full -ct 13 -maxfrag 1048576 -v -mediasvr atpsk01 -no_callback -connect_options 0x01010100 -jobid 98075 -jobgrpid 98075 -masterversion 650000 -bpbrm_shm_id 201326666 -blks_per_buffer 128

12:56:45.244 [6873] <2> bpbrm write_continue_backup: wrote CONTINUE BACKUP on COMM_SOCK <7>

12:56:45.245 [6873] <2> write_file_names: buffering file name '/E' for output

12:56:45.245 [6873] <2> write_file_names: buffering file name '/G' for output

12:56:45.245 [6873] <2> write_file_names: buffering file name '/I' for output

12:56:45.245 [6873] <2> write_file_names: buffering file name '/K' for output

12:56:45.245 [6873] <2> write_file_names: buffering file name '/M' for output

12:56:45.245 [6873] <2> write_file_names: buffering file name '/O' for output

12:56:45.245 [6873] <2> write_file_names: buffering file name '/W' for output

12:56:45.245 [6873] <2> write_file_names: successfully wrote buffer to COMM_SOCK

12:56:45.245 [6873] <2> bpbrm main: wrote CONTINUE on COMM_SOCK

12:56:46.643 [6873] <2> bpbrm main: ESTIMATE -1 -1 p1cl01n3_1338440198

12:56:46.702 [6873] <4> bpbrm main: from client p1cl01n3: TRV - object not found for file system backup: E:

12:56:46.703 [6873] <4> db_error_add_to_file: from client p1cl01n3: TRV - object not found for file system backup: E:

12:56:46.719 [6873] <4> bpbrm main: from client p1cl01n3: TRV - object not found for file system backup: G:

12:56:46.719 [6873] <4> db_error_add_to_file: from client p1cl01n3: TRV - object not found for file system backup: G:

12:56:46.731 [6873] <4> bpbrm main: from client p1cl01n3: TRV - object not found for file system backup: I:

12:56:46.731 [6873] <4> db_error_add_to_file: from client p1cl01n3: TRV - object not found for file system backup: I:

12:56:46.743 [6873] <4> bpbrm main: from client p1cl01n3: TRV - object not found for file system backup: K:

12:56:46.743 [6873] <4> db_error_add_to_file: from client p1cl01n3: TRV - object not found for file system backup: K:

12:56:46.755 [6873] <2> bpbrm main: from client p1cl01n3: INF - Enabling volume snapshots for (M:), please wait...

12:57:10.969 [6873] <2> bpbrm main: from client p1cl01n3: INF - FIS_ID=p1cl01n3_1338440198

12:57:11.025 [6873] <2> bpbrm main: from client p1cl01n3: INF - Volume snapshots enabled

12:57:11.086 [6873] <2> bpbrm main: ADDED FILES TO DB FOR p1cl01n3_1338440198 1 /M/

12:58:11.369 [6873] <2> Bpbrm_siginfo_print: 0: delay 0 signo SIGUSR1:16 code 0 pid 6876

12:58:11.369 [6873] <2> bpbrm mm_sig: received ready signal from media manager

12:58:11.386 [6873] <2> bpbrm main: ADDED FILES TO DB FOR p1cl01n3_1338440198 20 /M/DATA/12182011_lotus/domino/p1ms05cl/data/activity.ntf

12:58:16.125 [6873] <2> bpbrm main: ADDED FILES TO DB FOR p1cl01n3_1338440198 79 /M/DATA/12182011_lotus/domino/p1ms05cl/data/domino/html/download/filesets/l_JAVA_APPLETS

12:58:21.121 [6873] <2> bpbrm main: ADDED FILES TO DB FOR p1cl01n3_1338440198 125 /M/DATA/12182011_lotus/domino/p1ms05cl/data/domino/html/inotes5/help/H_KEYBOARD_SHORTCUTS.html

12:58:26.387 [6873] <2> bpbrm main: ADDED FILES TO DB FOR p1cl01n3_1338440198 225 /M/DATA/12182011_lotus/domino/p1ms05cl/data/domino/html/sametime/stlinks/es/placeHeader.html

12:58:35.752 [6873] <2> bpbrm main: ADDED FILES TO DB FOR p1cl01n3_1338440198 550 /M/DATA/12182011_lotus/domino/p1ms05cl/data/domino/icons/actn111.gif

12:58:45.701 [6873] <2> bpbrm main: ADDED FILES TO DB FOR p1cl01n3_1338440198 500 /M/DATA/12182011_lotus/domino/p1ms05cl/data/espana.dic

12:59:02.800 [6873] <2> bpbrm main: ADDED FILES TO DB FOR p1cl01n3_1338440198 500 /M/DATA/12182011_lotus/domino/p1ms05cl/data/uk.dic

i know that this is from a different time frame but i am running tests right now

[muhanad_daher]

1-create a file NET_BUFFER_SZ (netbackup/NET_BUFFER_SZ), and the value is 262144.

2- Change client read timeout parameter from 300 to 9600

3- Change Communication buffer size from 32 Kb to 128 KB. Go to Host Properties > Clients > Client Properties > Windows Client > ClientSettings > Communication buffer size = 128

4- In case there is an Antivirus running, turn it off for troubleshooting proposes.

5- Disable autotuning and chimney features, from command prompt run:

netsh int tcp set global autotuning=disabled

netsh int tcp set global chimney=disabled

i hope that will help you.

[knox890]

thanks! but im not sure of the netsh command since the two are connected via PRIVATE IP and the other NICs might be affected(production)

what does item #1 do?

i did #2 #3

[muhanad_daher]

the first point increase the network buffer

for more info (http://www.symantec.com/business/support/index?page=content&id=TECH1724)

Also stop a firewall.

[knox890]

done setting the buffer.

firewall is not working either

chimney offload state is set to automatic.. should i change this to DISABLED instead? 

also found a new error.

14:07:35.455 [9034] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2048: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006

14:07:35.455 [9034] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2062: service: bpdbm

14:07:35.458 [9034] <2> logconnections: BPDBM CONNECT FROM 10.199.9.86.4952 TO 10.199.9.86.13724

14:12:17.082 [9034] <32> bpbrm main: from client p1cl01n3: FTL - socket write failed

14:12:17.116 [9034] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2048: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006

14:12:17.117 [9034] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2062: service: bpjobd

14:12:17.161 [9034] <2> job_connect: SO_KEEPALIVE set on socket 9 for client ATPBCKSP01

14:12:17.161 [9034] <2> logconnections: BPJOBD CONNECT FROM 10.199.9.86.4824 TO 10.199.9.86.13724

14:12:17.162 [9034] <2> job_authenticate_connection: ignoring VxSS authentication check for now...

14:12:17.162 [9034] <2> job_connect: Connected to the host ATPBCKSP01 contype 10 jobid <98076> socket <9>

14:12:17.162 [9034] <2> job_connect: Connected on port 4824

14:12:17.163 [9034] <2> get_long: (2) premature end of file (byte 1)

14:12:17.163 [9034] <2> job_monitoring_exex: Failed to get frame type

14:12:17.163 [9034] <2> job_disconnect: Disconnected

14:12:17.163 [9034] <4> db_error_add_to_file: from client p1cl01n3: FTL - socket write failed

14:12:17.394 [9034] <2> bpbrm main: client p1cl01n3 EXIT STATUS = 24: socket write failed

[mph999]

See this post:

https://www-secure.symantec.com/connect/forums/netbackup-basics-and-how-make-your-life-easier

Part (L)

DO NOT MAKE MULTIPLE CHANGES TO THE SYSTEM IS ONE GO.  IF THE SINGLE CHANGE YOU MAKE DOES NOT FIX THE ISSUE< PUT IT BACK.

1-create a file NET_BUFFER_SZ (netbackup/NET_BUFFER_SZ), and the value is 262144.

This is not likely to fix a status 24 - it will effect performance however.   I have never seen this fix a status 24 in real life.

2- Change client read timeout parameter from 300 to 9600

This again is very unlikely to make a difference - put it back to 300.

3- Change Communication buffer size from 32 Kb to 128 KB. Go to Host Properties > Clients > Client Properties > Windows Client > ClientSettings > Communication buffer size = 128

This one is listed as a possible cause of status 24 - again, in real life I have only ever seen it affect performance, and never fix, or cause a status 24.

These two are valid.

4- In case there is an Antivirus running, turn it off for troubleshooting proposes.

5- Disable autotuning and chimney features, from command prompt run:

netsh int tcp set global autotuning=disabled

netsh int tcp set global chimney=disabled

Martin

[mph999]

Here is just about every possible cause of Status 24 that I am aware of.  Apologies, but from the NBU side, it is virtually impossible to troubleshoot, as we have no details of what has happened, apart from the fact the network is unavailable.  The big clue, is the Network is unavailable, so this is not likely to be a NetBackup issue.

Really, all we can do is a 'process of elimination'.

Please confirm these details.

1.  How many clients have this error 

2.  Did this client previously work

3.  What was changed 

4.  Does it write some data then fail

5.  Does it fail at the very beginning of the job

6.  Does it always fail at the same point

7.  Operating system of client

8.  Operating system of media server

9.  NetBackup version

10. Logs from media server - bptm and bpbrm, from client bpbkar, bpcd

In my experience, Status 24 is hardly ever NBU (in fact, I don't think I have ever seen a status 24 failure caused by NetBackup myself)

Something below normally fixes it ...  Yes, it is a lot to read, and will probably tyake a number of hours to go through.

If this is a Windows client, a very common cause is the TCP Chimmey settings  - http://www.symantec.com/docs/TECH55653

I have given a number of technotes below (the odd one may be 'internal' only) , and have show a summary of the solutions, as well as the odd extra note.

http://www.symantec.com/docs/TECH76201

Possible solution to Status 24 by increasing TCP receive buffer space 

http://www.symantec.com/docs/TECH34183 

this Technote, although written for Solaris, shows how TCP tunings can 

cause status 24s. I am sure your system admins will be aware of the 

corresponding setting for the windows operating system. 

http://www.symantec.com/docs/TECH55653 

This technote is very important. It covers many many issues that can 

occur when either TCP Chimney Offload, TCP/IP Offload Engine (TOE) or TCP 

Segmentation Offload (TSO) are enabled. It is recommend to disable 

these, as per the technote. 

I also understand that we have previously seen MS Patch KB92222 resolve status 24 issues.

http://www.symantec.com/docs/TECH150369

A write operation to a socket failed, these are possible cause for this issue:

A high network load.

Intermittent connectivity.

Packet reordering.

Duplex Mismatch between client and master server NICs.

Small network buffer size

http://support.microsoft.com/kb/942861 

SOLUTION/WORKAROUND:

Contact the hardware vendor of the NIC for the latest updates for their product as a resolution.

This problem occurs when the TCP Chimney Offload feature is enabled on the NetBackup Windows 2003 client.  Disable this feature to workaround this problem.

To do this, at a command prompt, enter the following:

Netsh int ip set chimney DISABLED

http://www.symantec.com/docs/TECH127930

The above messages almost always indicate a networking issue of some sort. In this case it was due to a faulty switch. There are rare occasions when the above messages are not caused by a networking issue, such as those addressed in http://www.symantec.com/docs/TECH72115. 

(TECH72115 is not relevant to you, this was an issue with a SAN client, fixed in 6.5.4)

But note, the technote says the issue is 'almost always' network related, this can also include operating system settings.

http://www.symantec.com/docs/TECH145223

The issue was with the idle timeout setting on the firewall that was too low to allow backups and/or restores to complete. With the DMZ media server backing up a DMZ client the media server sends only the occasional meta data updates back to the master server in order to update the images catalog. If that TCP socket connection between the media server and master server is idle for a longer period than the firewall's idle timeout the firewall breaks the connection between the media server and master servder and thus the media server breaks the connection to the client producing the socket error.

Increasing the idle timeout setting on the firewall to a value larger than the amount of time a typical backups takes to complete should resolve the issue.

Also increasing the frequency of the TCP keepalive packets can also help maintain the socket during idle periods from the server's defaults.

Although you may not have a firewall between the client and the media server, this solution is another demonstation that the issue is network related, as opposed to NetBackup.

http://www.symantec.com/docs/S:TECH130343  (Internal technote)

The issue was found to be due to NIC card Network congestion (that is, network overloaded)

http://www.symantec.com/docs/TECH135924  (I think this one I sent previously, shows the MS fix for the issue)

In this instance, the problem was isolated to this single machine making the point of failure isolated to the problematic new host.

If the problem is due to an unidentified corruption / misconfiguration in the new media server's TCP Stack and Winsock environment (as was the case in this example), executing these two commands, followed by a reboot will resolve the problem:

netsh int ip reset resetlog.txt   Microsoft Reference:  http://support.microsoft.com/kb/299357 

netsh winsock reset catalog    Microsoft Reference:  http://technet.microsoft.com/en-us/library/cc759700(WS.10).aspx 

NOTE: The above two commands will reset the Windows TCP Stack as well as the Windows Winsock environment back to the default values.  This means that if the host is configured with a static IP Address and other customized TCP settings, they will be lost and will need to be re-entered after the reboot.  The default TCP setting is to use DHCP and the host will be using DHCP upon booting up.

http://www.symantec.com/docs/TECH76201

Possible solution to Status 24 by increasing TCP receive buffer space 

http://www.symantec.com/docs/TECH34183 

this Technote, although written for Solaris, shows how TCP tunings can 

cause status 24s. I am sure your system admins will be aware of the 

corresponding setting for the windows operating system. 

http://www.symantec.com/docs/TECH55653 

This technote is very important. It covers many many issues that can 

occur when either TCP Chimney Offload, TCP/IP Offload Engine (TOE) or TCP 

Segmentation Offload (TSO) are enabled. It is recommend to disable 

these, as per the technote. 

I understand that we have previously seen MS Patch KB92222 resolve status 24 issues.

There are  2 possible issues that could be NBU related that could cause this :

1.  Client NBU version is higher than the media serevr

2.  Make sure the comunications buffer is not too high (http://www.symantec.com/docs/TECH60570

)

What to do next:

http://www.symantec.com/docs/TECH135924  (mentioned before, MS suggested fix)

http://www.symantec.com/docs/TECH60570  (communications buffer, mentioned above)

http://www.symantec.com/docs/TECH60844

If these do not resolve the situation, I would recommend you talk with the Operating system vendor.  In summary, apart from the Client version of software and the communication buffer size (set in host properties) I can find no other cause that could be NBU.  However, from the very detailed research I have done, I can find many many causes that are the network or operating system.

[Marianne]

We need ALL THREE logs to troubleshoot. Log folders do not exist by default, they need to be created. Bpbkar log will tell us what happened on the client during backup process. Bptm on media server will log actual data received from client. Bpbrm will log comms and metadata info. Please ensure that all of these log folders exist before running another backup. Please rename log files to reflect process name (e.g bpbkar.txt) and post all 3 covering the same backup period as attachments (not in body of post, please). One more thing: W2008 R2 is only supported as from NBU 6.5.5. You will need to upgrade your environment ASAP.

[knox890]

everyone! 

appreciate your inputs! i am new to netbackup and your inputs are valuable to me. i will to provide you with more information as we go on with this problem. to answer your inputs...

1.  How many clients have this error 

P1CL01N3 client only

2.  Did this client previously work

Yes, it was previously working

3.  What was changed 

None, 

4.  Does it write some data then fail

Yes, 

5.  Does it fail at the very beginning of the job

No, it writes some data then fail.

6.  Does it always fail at the same point

Random

7.  Operating system of client

Windows 2008 Server R2

8.  Operating system of media server

SUNOS 5.10 UNIX

9.  NetBackup version

NBU 6.5.6 - correction for earlier

10. Logs from media server - bptm and bpbrm, from client bpbkar, bpcd

--will provide this later

--

on a lighter note,

i ran a backup with a selected folder only from the client and it successfully finished.

but when i tried a folder with 200GB, it then failed again.

--

since this is running on a private IP here are the configuration of the etc/hosts from the client

192.168.1.44 atpbackupsp0144p

10.155.2.1 atpbackupsp011p

#10.199.9.86 atpbackupsp01 atpsk01 atpbcksp01

[Marianne]

Some additional points from my side:

TCP Chimney on Windows systems should be disabled on W2003, but enabled on W2008.
http://www.symantec.com/docs/TECH145791
For Windows 2008 R2 platforms, performance gains have been realized when TCP Chimney Offload is enabled. Microsoft article 912222 discusses TCP Chimney Offload and RSS and the purpose of each.

Please also tell us more about your backup job: 

logparams: -backup -S ATPBCKSP01 -c p1cl01n3 -ct 13 -ru root -cl Weekly_LNotes_P1P3 -sched Weekly_Full -bt 1338440198 -dt 0 -st 0 -b p1cl01n3_1338440198 -mediasvr atpsk01 -jobid 98075 -jobgrpid 98075 -masterversion 650000 -maxfrag 1048576 -reqid -1338361039 -mt 2 -to 0 -stunit atpsk01-hcart3-robot-tld-0 -rl 9 -rp 2147483647 -cj 6 -D 20 -rt 8 -rn 0 -pool Infinity_Pool -use_ofb -use_otm -WOFB_enabled 1 -WOFB_fim 1 -nbu -jm -secure 1 -kl 30 -rg root -fso -connect_options 16974338

Are you trying to do Lotus Notes backups with a normal MS-Windows-NT policy? or is this a proper Lotus Notes policy type? (We cannot see policy type from the log parameters.)

[knox890]

we are doing this for our normal notes backup "MS-Windows-NT"

for the chimney, i dont want to touch on this since the other servers namely P1CL01N1, P1CL01N2 are working perfectly fine.

they all have the same configuration but it just happened that we received error 24 for P1CL01N3

[muhanad_daher]

Hi Martin,

this procedures from symantec was send to me when have the same issue and solved my problem before month ago.

[mph999]

Hi Muhanad,

Thanks for making that clear - I disagree with what was sent to you.  What was the case number ?

Here you have 5 possibly fixes ...

1-create a file NET_BUFFER_SZ (netbackup/NET_BUFFER_SZ), and the value is 262144.

2- Change client read timeout parameter from 300 to 9600

3- Change Communication buffer size from 32 Kb to 128 KB. Go to Host Properties > Clients > Client Properties > Windows Client > ClientSettings > Communication buffer size = 128

4- In case there is an Antivirus running, turn it off for troubleshooting proposes.

5- Disable autotuning and chimney features, from command prompt run:

netsh int tcp set global autotuning=disabled

netsh int tcp set global chimney=disabled

Which one fixed the issue, because I don't think you needed to do all 5 of them to make it work ?

Therefore, if you do all 5, maybe one of them fixes the issue and x4 are changes you don't actually need to make, and anyone of these x4 changes can cause further issues.

(1) and (2) I have never seen fix a status 24 issue, and I know (3) is listed as a possible fix, but when I actually tested this (on a working system) I found it caused performance issues, but didn't actually cause a status 24.

Therefore, it is reasonable to suggest that if I can't make it cause a status 24, it's probably not going to fix it ...

As per this :

https://www-secure.symantec.com/connect/forums/netbackup-basics-and-how-make-your-life-easier

Multiple changes should never be made, unless directed to do so by support.

I have explained why in the post.

Martin

[knox890]

Just an update, 

the three servers are on a clustered environment whereas 3 NBU clients are installed and running on a private IP

p1cl01n1 = 192.168.1.87

p1cl01n2 = 192.168.1.88

p1cl01n3 = 192.168.1.89

i have one policy created for these 3 clients namely Weekly_LNotes_P1P3 that runs a full backup. 

Backup selection are selected drives which only important files resides.

Are there some issues on this setup?

[Marianne]

Hopefully you have added hosts entries in each of these machines with different hostnames for the different IP's associated with each NIC?

If media server has 2 NICs for public and private backup clients, ensure that media has different hostnames for each NIC. See http://www.symantec.com/docs/TECH54733

If you post logs requested yesterday, we can trace comms path as well.

Please also share complete list of public hostnames & IPs as well as list of private hostnames and IPs.
This will help to identify if incorrect interface comms is attempted.

[knox890]

here are the attachments/logs