Windows client backup failed with 58,40,24,13

[wannawin]

Hi Team

Master Server: AIX, NBU Version: 7.1.0.4

Media Server: AIX, NBU Version: 7.1.0.1

Client: Windows Server 2003, NBU Version 7.1.0.1

Backup is going through Media server only.

Differential backup completed successfully but full backup from last one month is failing with differenr error codes , sometimeds it is failing with 13.. sometimes with 58.. sometimes with 24 and 40.

There are 4-6 client whose Full backups are  failing with same error codes having same versions and going through media server only.

Please let me know which logs you want so that i will attached that logs.

Thanks in Advance

[Nicolai]

Try to add CLIENT_READ_TIMEOUT = 1800 to bp.conf on master and media server. No re-start is required.

http://www.symantec.com/docs/HOWTO33199

To configure the setting on the client use:

NetBackup Management > Host Properties > Double-click on master server> Timeouts > Client read timeout.

[Marianne]

Best to open separate discussions for each of these errors.

What they have in common is that they are all network-related. All of which are outside of NBU control.

You need to carefully check info in Details tab of each failed backup to try and pinpoint where the issue is - media server or the clients.

Is there a firewall in the picture?
All clients connected to same network switch?
Anything else in common that may indicate an issue?

Confirmed that TCP Chimney is disabled on all W2003 clients?
Latest MS hotfixes installed?

Logs needed (depending on the error):

media server: bpbrm and bptm

client: bpcd and bpbkar 

[Nicolai]

Any "timer expired" messages 

Backup fails with status 13 after bpbrm reports; socket read failed: errno = 62 - Timer expired

http://www.symantec.com/docs/TECH71821

[wannawin]

Hello ,

Below are the detail logs.

Full backup of same client is failing with error code 58, 13, 24.

06/21/2014 10:50:16 - Info nbjm (pid=5701736) starting backup job (jobid=4149393) for client chiissp01ui, policy CERT_PROD_WIN_clients, schedule MTHLY_FULL
06/21/2014 10:50:16 - Info nbjm (pid=5701736) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=4149393, request id:{BA7FADAE-F903-11E3-A72B-0B8FEDF90000})
06/21/2014 10:50:16 - requesting resource ictcvnb35
06/21/2014 10:50:16 - requesting resource sctclbk02.NBU_CLIENT.MAXJOBS.chiissp01ui
06/21/2014 10:50:16 - requesting resource sctclbk02.NBU_POLICY.MAXJOBS.CERT_PROD_WIN_clients
06/21/2014 10:50:16 - granted resource  sctclbk02.NBU_CLIENT.MAXJOBS.chiissp01ui
06/21/2014 10:50:16 - granted resource  sctclbk02.NBU_POLICY.MAXJOBS.CERT_PROD_WIN_clients
06/21/2014 10:50:16 - granted resource  NA0300
06/21/2014 10:50:16 - granted resource  Shared-2
06/21/2014 10:50:16 - granted resource  ictcvnb35
06/21/2014 10:50:16 - estimated 18501926 kbytes needed
06/21/2014 10:50:16 - Info nbjm (pid=5701736) started backup job for client chiissp01ui, policy CERT_PROD_WIN_clients, schedule MTHLY_FULL on storage unit ictcvnb35
06/21/2014 10:50:29 - Info bpbrm (pid=7864452) telling media manager to start backup on client
06/21/2014 10:50:29 - Info bptm (pid=11141366) using 262144 data buffer size
06/21/2014 10:50:29 - Info bptm (pid=11141366) using 32 data buffers
06/21/2014 10:50:29 - Info bpbrm (pid=7864452) spawning a brm child process
06/21/2014 10:50:29 - Info bpbrm (pid=7864452) child pid: 11206880
06/21/2014 10:51:11 - Info bpbrm (pid=7864452) sending bpsched msg: CONNECTING TO CLIENT FOR chiissp01ui_1403328016
06/21/2014 10:51:14 - connecting
06/21/2014 10:51:23 - Info bpbrm (pid=7864452) start bpbkar on client
06/21/2014 10:51:27 - connected; connect time: 0:00:00
06/21/2014 10:51:27 - begin writing
06/21/2014 10:51:28 - Info bpbkar (pid=31212) Backup started
06/21/2014 10:51:28 - Info bpbrm (pid=7864452) Sending the file list to the client
06/21/2014 10:55:40 - Info bpbrm (pid=7864452) media manager for backup id chiissp01ui_1403328016 exited with status 0: the requested operation was successfully completed
06/21/2014 12:55:49 - Error bpbrm (pid=11206880) socket read failed: errno = 73 - A connection with a remote socket was reset by that socket.
06/21/2014 12:55:56 - end writing; write time: 2:04:29
file read failed  (13)

06/21/2014 10:31:20 - Info nbjm (pid=5701736) starting backup job (jobid=4149389) for client chiissp01ui, policy CERT_PROD_WIN_clients, schedule MTHLY_FULL
06/21/2014 10:31:20 - Info nbjm (pid=5701736) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=4149389, request id:{1563525A-F901-11E3-98EE-596F22BC0000})
06/21/2014 10:31:20 - requesting resource ictcvnb35
06/21/2014 10:31:20 - requesting resource sctclbk02.NBU_CLIENT.MAXJOBS.chiissp01ui
06/21/2014 10:31:20 - requesting resource sctclbk02.NBU_POLICY.MAXJOBS.CERT_PROD_WIN_clients
06/21/2014 10:31:20 - granted resource  sctclbk02.NBU_CLIENT.MAXJOBS.chiissp01ui
06/21/2014 10:31:20 - granted resource  sctclbk02.NBU_POLICY.MAXJOBS.CERT_PROD_WIN_clients
06/21/2014 10:31:20 - granted resource  NA0300
06/21/2014 10:31:20 - granted resource  Shared-2
06/21/2014 10:31:20 - granted resource  ictcvnb35
06/21/2014 10:31:20 - estimated 18501926 kbytes needed
06/21/2014 10:31:20 - Info nbjm (pid=5701736) started backup job for client chiissp01ui, policy CERT_PROD_WIN_clients, schedule MTHLY_FULL on storage unit ictcvnb35
06/21/2014 10:40:12 - Error bpbrm (pid=7864452) cannot connect to chiissp01ui, A connect operation on a socket is not complete. (55)
06/21/2014 10:40:16 - end writing
can't connect to client  (58)

06/21/2014 08:30:01 - Info nbjm (pid=5701736) starting backup job (jobid=4149249) for client chiissp01ui, policy CERT_PROD_WIN_clients, schedule MTHLY_FULL
06/21/2014 08:30:01 - Info nbjm (pid=5701736) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=4149249, request id:{22DF8C2A-F8F0-11E3-9A27-1694DC5F0000})
06/21/2014 08:30:01 - requesting resource ictcvnb35
06/21/2014 08:30:01 - requesting resource sctclbk02.NBU_CLIENT.MAXJOBS.chiissp01ui
06/21/2014 08:30:01 - requesting resource sctclbk02.NBU_POLICY.MAXJOBS.CERT_PROD_WIN_clients
06/21/2014 08:30:08 - awaiting resource ictcvnb35. Waiting for resources.
          Reason: Drives are in use, Media server: ictcvnb35,
          Robot Type(Number): ACS(0), Media ID: N/A, Drive Name: N/A,
          Volume Pool: Certegy, Storage Unit: ictcvnb35, Drive Scan Host: N/A,
          Disk Pool: N/A, Disk Volume: N/A
06/21/2014 10:00:03 - Info bpbrm (pid=7864452) chiissp01ui is the host to backup data from
06/21/2014 10:00:03 - Waiting for scan drive stop Shared-2, Media server: ictcvnb35
06/21/2014 10:00:06 - granted resource  sctclbk02.NBU_CLIENT.MAXJOBS.chiissp01ui
06/21/2014 10:00:06 - granted resource  sctclbk02.NBU_POLICY.MAXJOBS.CERT_PROD_WIN_clients
06/21/2014 10:00:06 - granted resource  NA0280
06/21/2014 10:00:06 - granted resource  Shared-2
06/21/2014 10:00:06 - granted resource  ictcvnb35
06/21/2014 10:00:06 - estimated 18501926 kbytes needed
06/21/2014 10:00:06 - Info nbjm (pid=5701736) started backup job for client chiissp01ui, policy CERT_PROD_WIN_clients, schedule MTHLY_FULL on storage unit ictcvnb35
06/21/2014 10:00:06 - started process bpbrm (pid=7864452)
06/21/2014 10:00:36 - Info bpbrm (pid=7864452) starting bptm
06/21/2014 10:00:36 - Info bpbrm (pid=7864452) Started media manager using bpcd successfully
06/21/2014 10:00:43 - Info bpbrm (pid=7864452) telling media manager to start backup on client
06/21/2014 10:00:43 - Info bptm (pid=11141366) using 262144 data buffer size
06/21/2014 10:00:43 - Info bptm (pid=11141366) using 32 data buffers
06/21/2014 10:00:49 - Info bptm (pid=11141366) start backup
06/21/2014 10:00:49 - Info bptm (pid=11141366) Waiting for mount of media id NA0280 (copy 1) on server ictcvnb35.
06/21/2014 10:00:49 - Info bpbrm (pid=7864452) spawning a brm child process
06/21/2014 10:00:49 - Info bpbrm (pid=7864452) child pid: 6291496
06/21/2014 10:00:53 - mounting NA0280
06/21/2014 10:00:55 - Error bptm (pid=11141366) error requesting media, TpErrno = Robot operation failed
06/21/2014 10:00:55 - Warning bptm (pid=11141366) media id NA0280 load operation reported an error
06/21/2014 10:00:56 - Info bpbrm (pid=7864452) sending bpsched msg: CONNECTING TO CLIENT FOR chiissp01ui_1403325006
06/21/2014 10:00:58 - current media NA0280 complete, requesting next media Any
06/21/2014 10:01:00 - connecting
06/21/2014 10:01:14 - Info bpbrm (pid=7864452) start bpbkar on client
06/21/2014 10:01:18 - connected; connect time: 0:00:00
06/21/2014 10:01:19 - Info bpbkar (pid=31356) Backup started
06/21/2014 10:01:19 - Info bpbrm (pid=7864452) Sending the file list to the client
06/21/2014 10:01:28 - Waiting for scan drive stop Shared-2, Media server: ictcvnb35
06/21/2014 10:01:30 - granted resource  NA0300
06/21/2014 10:01:30 - granted resource  Shared-2
06/21/2014 10:01:30 - granted resource  ictcvnb35
06/21/2014 10:01:51 - Info bptm (pid=11141366) Waiting for mount of media id NA0300 (copy 1) on server ictcvnb35.
06/21/2014 10:01:55 - mounting NA0300
06/21/2014 10:02:37 - Info bptm (pid=11141366) media id NA0300 mounted on drive index 19, drivepath /dev/rmt30.1, drivename Shared-2, copy 1
06/21/2014 10:02:41 - mounted NA0300; mount time: 0:00:46
06/21/2014 10:02:41 - positioning NA0300 to file 1
06/21/2014 10:02:49 - positioned NA0300; position time: 0:00:08
06/21/2014 10:02:49 - begin writing
06/21/2014 10:04:28 - Critical bpbrm (pid=6291496) from client chiissp01ui: FTL - socket write failed
06/21/2014 10:04:28 - Error bpbrm (pid=6291496) could not send server status message
06/21/2014 10:04:41 - Info bpbrm (pid=7864452) media manager for backup id chiissp01ui_1403325006 exited with status 150: termination requested by administrator
06/21/2014 10:04:45 - end writing; write time: 0:01:56
socket write failed  (24)

backup is going through media server only.

All clients are connected to same network switch

How to disable chimney on W2003?

[Nicolai]

Last job is a fialure because of media mount error 

Error bptm (pid=11141366) error requesting media, TpErrno = Robot operation failed

Did you add the CLIENT_READ_TIMEOUT to bp.conf on master and media ?

[wannawin]

Hi ,

Disable Chimney setting but it failed again with error code 40.

06/25/2014 00:24:09 - Info nbjm (pid=5701736) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=4151814, request id:{EC7C39C4-FBD0-11E3-BB15-584E5E7D0000})
06/25/2014 00:24:09 - requesting resource ictcvnb35
06/25/2014 00:24:09 - requesting resource sctclbk02.NBU_CLIENT.MAXJOBS.chiissp01ui
06/25/2014 00:24:09 - requesting resource sctclbk02.NBU_POLICY.MAXJOBS.CERT_PROD_WIN_clients
06/25/2014 00:24:09 - granted resource  sctclbk02.NBU_CLIENT.MAXJOBS.chiissp01ui
06/25/2014 00:24:09 - granted resource  sctclbk02.NBU_POLICY.MAXJOBS.CERT_PROD_WIN_clients
06/25/2014 00:24:09 - granted resource  Z60020
06/25/2014 00:24:09 - granted resource  Shared-2
06/25/2014 00:24:09 - granted resource  ictcvnb35
06/25/2014 00:24:09 - estimated 18526707 kbytes needed
06/25/2014 00:24:09 - Info nbjm (pid=5701736) started backup job for client chiissp01ui, policy CERT_PROD_WIN_clients, schedule WKLY_FULL on storage unit ictcvnb35
06/25/2014 00:24:11 - Info bpbrm (pid=11272208) telling media manager to start backup on client
06/25/2014 00:24:11 - Info bptm (pid=6946948) using 262144 data buffer size
06/25/2014 00:24:11 - Info bptm (pid=6946948) using 32 data buffers
06/25/2014 00:24:12 - Info bpbrm (pid=11272208) spawning a brm child process
06/25/2014 00:24:12 - Info bpbrm (pid=11272208) child pid: 14417978
06/25/2014 00:24:14 - Info bpbrm (pid=11272208) sending bpsched msg: CONNECTING TO CLIENT FOR chiissp01ui_1403636049
06/25/2014 00:24:15 - Info bpbrm (pid=11272208) start bpbkar on client
06/25/2014 00:24:21 - Info bpbkar (pid=43268) Backup started
06/25/2014 00:24:21 - Info bpbrm (pid=11272208) Sending the file list to the client
06/25/2014 00:27:23 - Error bpbrm (pid=14417978) could not write snapshot identifier to stderr
06/25/2014 00:27:23 - Error bpbrm (pid=14417978) could not write snapshot identifier to stderr
06/25/2014 00:27:24 - Error bpbrm (pid=14417978) could not write FILE ADDED message to stderr
06/25/2014 00:27:29 - Error bpbrm (pid=14417978) could not write FILE ADDED message to stderr
06/25/2014 00:27:38 - Error bpbrm (pid=14417978) could not write FILE ADDED message to stderr
06/25/2014 00:27:54 - Error bpbrm (pid=14417978) could not write FILE ADDED message to stderr
06/25/2014 00:28:09 - Error bpbrm (pid=14417978) could not write FILE ADDED message to stderr
06/25/2014 00:28:43 - Error bpbrm (pid=14417978) could not write FILE ADDED message to stderr
06/25/2014 00:28:50 - Error bpbrm (pid=14417978) could not write FILE ADDED message to stderr
06/25/2014 00:28:57 - Error bpbrm (pid=14417978) could not write FILE ADDED message to stderr
06/25/2014 00:35:08 - Error bpbrm (pid=14417978) could not write FILE ADDED message to stderr
06/25/2014 00:35:58 - Error bpbrm (pid=14417978) could not write FILE ADDED message to stderr
06/25/2014 00:41:35 - Error bptm (pid=6946948) media manager terminated by parent process
06/25/2014 00:41:35 - Critical bpbrm (pid=14417978) from client chiissp01ui: FTL - socket write failed
06/25/2014 00:41:35 - Info bpbrm (pid=11272208) sending media manager msg: STOP BACKUP chiissp01ui_1403636049
06/25/2014 00:42:01 - Info bpbrm (pid=11272208) got ERROR 82 from media manager
06/25/2014 00:42:01 - Info bpbrm (pid=11272208) killing bpbrm child 7733358.
06/25/2014 00:42:01 - Info bpbrm (pid=11272208) killing bpbrm child 13631602.
06/25/2014 00:42:01 - Info bpbrm (pid=11272208) killing bpbrm child 10879188.
06/25/2014 00:48:40 - Info nbjm (pid=5701736) starting backup job (jobid=4151814) for client chiissp01ui, policy CERT_PROD_WIN_clients, schedule WKLY_FULL
network connection broken  (40)

[wannawin]

Hi Team,

I enabled Multistreaming and all stream failed with error code 24 and 40.