network error, Backup going to hung state

[AnandhaKannan_D]

Hi,

Recently am facing backup failure for the few clients before it worked fine. Pls provide solution to rectify it

Client OS :Windows 2012 R2 Std

Master Server: AIX

10/16/2015 02:00:30 - Info nbjm (pid=28967070) starting backup job (jobid=5681) for client inchnexch06, policy InChnExch06, schedule Daily_Backup
10/16/2015 02:00:30 - Info nbjm (pid=28967070) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=5681, request id:{940A9B18-737B-11E5-B494-5972290A0000})
10/16/2015 02:00:30 - requesting resource Any
10/16/2015 02:00:30 - requesting resource InChnM22.NBU_CLIENT.MAXJOBS.inchnexch06
10/16/2015 02:00:30 - requesting resource InChnM22.NBU_POLICY.MAXJOBS.InChnExch06
10/16/2015 02:00:32 - granted resource  InChnM22.NBU_CLIENT.MAXJOBS.inchnexch06
10/16/2015 02:00:32 - granted resource  InChnM22.NBU_POLICY.MAXJOBS.InChnExch06
10/16/2015 02:00:32 - granted resource  DFH699
10/16/2015 02:00:32 - granted resource  IBM.ULT3580-TD5.003
10/16/2015 02:00:32 - granted resource  InChnM22-hcart2-robot-tld-1
10/16/2015 02:00:33 - estimated 16288 kbytes needed
10/16/2015 02:00:33 - Info nbjm (pid=28967070) started backup (backupid=inchnexch06_1444941032) job for client inchnexch06, policy InChnExch06, schedule Daily_Backup on storage unit InChnM22-hcart2-robot-tld-1
10/16/2015 02:00:36 - started process bpbrm (pid=9109606)
10/16/2015 02:00:40 - Info bpbrm (pid=9109606) starting bptm
10/16/2015 02:00:40 - Info bpbrm (pid=9109606) Started media manager using bpcd successfully
10/16/2015 02:00:43 - Info bpbrm (pid=9109606) inchnexch06 is the host to backup data from
10/16/2015 02:00:43 - Info bptm (pid=5767368) using 524288 data buffer size
10/16/2015 02:00:43 - Info bpbrm (pid=9109606) telling media manager to start backup on client
10/16/2015 02:00:43 - Info bptm (pid=5767368) using 64 data buffers
10/16/2015 02:00:44 - Info bpbrm (pid=9109606) spawning a brm child process
10/16/2015 02:00:44 - Info bptm (pid=5767368) start backup
10/16/2015 02:00:44 - Info bpbrm (pid=9109606) child pid: 27852982
10/16/2015 02:00:45 - Info bptm (pid=5767368) Waiting for mount of media id DFH699 (copy 1) on server 192.168.16.21.
10/16/2015 02:00:45 - mounting DFH699
10/16/2015 02:00:46 - Info bpbrm (pid=9109606) sending bpsched msg: CONNECTING TO CLIENT FOR inchnexch06_1444941032
10/16/2015 02:00:46 - connecting
10/16/2015 02:00:48 - Info bpbrm (pid=9109606) start bpbkar on client
10/16/2015 02:00:48 - connected; connect time: 0:00:00
10/16/2015 02:00:58 - Info bpbkar (pid=38796) Backup started
10/16/2015 02:00:58 - Info bpbrm (pid=9109606) Sending the file list to the client
10/16/2015 02:00:58 - Info bpbkar (pid=38796) change time comparison:<disabled>
10/16/2015 02:00:58 - Info bpbkar (pid=38796) archive bit processing:<enabled>
10/16/2015 02:00:58 - Info bpbkar (pid=38796) not using change journal data for <D:\ApplicationLogs>: not enabled
10/16/2015 02:01:14 - Info bptm (pid=5767368) media id DFH699 mounted on drive index 5, drivepath /dev/rmt5.1, drivename IBM.ULT3580-TD5.003, copy 1
10/16/2015 02:01:15 - mounted DFH699; mount time: 0:00:30
10/16/2015 02:01:15 - positioning DFH699 to file 32
10/16/2015 02:01:28 - positioned DFH699; position time: 0:00:13
10/16/2015 02:01:28 - begin writing
10/16/2015 04:24:32 - Error bptm (pid=22085794) system call failed - A connection with a remote socket was reset by that socket. (at child.c.1306)
10/16/2015 04:24:32 - Error bptm (pid=22085794) unable to perform read from client socket, connection may have been broken
10/16/2015 04:24:54 - Info bpbrm (pid=9109606) media manager for backup id inchnexch06_1444941032 exited with status 42: network read failed
10/16/2015 04:24:54 - end writing; write time: 2:23:26
10/16/2015 04:25:58 - Info bpbrm (pid=8192188) Starting delete snapshot processing
10/16/2015 04:31:14 - Error bpbrm (pid=8192188) from client inchnexch06: Get bpfis state from InChnM22 failed. status = 25
10/16/2015 04:41:45 - Info bpfis (pid=23900) Backup started
10/16/2015 04:41:45 - Critical bpbrm (pid=8192188) from client inchnexch06: cannot open C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\bpfis.fim.inchnexch06_1444941032.1.0
10/16/2015 04:47:01 - Info bpfis (pid=23900) done. status: 4207
10/16/2015 04:47:01 - Info bpfis (pid=23900) done. status: 4207: Could not fetch snapshot metadata or state files
network read failed  (42)

[AnandhaKannan_D]

NB Version : 7.6.0.1

[Michal_Mikulik1]

Hello,

from hostnames I deduce it is an MS Exchange backup. There will be some problem on OS/VSS level on the client. Refer to or provide Event Logs from client.

[sdo]

Did it move any data before it failed?  If so, maybe the MS Exchange client is clustered?  If so, was it failed-over during the backup?

[Marianne · Accepted solution]

Show network admins this error : "system call failed - A connection with a remote socket was reset by that socket. (at child.c.1306)" Configuring KeepAlive on the client may help. Please also confirm port connectivity and forward and reverse name lookup between master and client. I have personally experienced how backup actually completed but connectivity issue prevented client communicating snapshot status with the master and then failed with a snapshot error.

[AnandhaKannan_D]

After doing some modification in host file backup is getting success. But only full backup is succeeded. Incremental backup started begins writing but it didnt take any byte count for more than 2 hours. But full backup completed in 45 mins.

What could be wrong in incremental backup....There is no much changes in file (approx 75MB changes) after full backup also...

I tried by enabling change journal option, even the same result.

===============================================================

0/22/2015 16:51:02 - Info nbjm (pid=13107208) starting backup job (jobid=5810) for client inchnexch03.tcs.com, policy InChnExch03, schedule Daily_Backup
10/22/2015 16:51:02 - Info nbjm (pid=13107208) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=5810, request id:{FA774152-78AE-11E5-85EC-781F03010000})
10/22/2015 16:51:02 - requesting resource Any
10/22/2015 16:51:02 - requesting resource InChnM22.NBU_CLIENT.MAXJOBS.inchnexch03.tcs.com
10/22/2015 16:51:02 - requesting resource InChnM22.NBU_POLICY.MAXJOBS.InChnExch03
10/22/2015 16:51:02 - granted resource  InChnM22.NBU_CLIENT.MAXJOBS.inchnexch03.tcs.com
10/22/2015 16:51:02 - granted resource  InChnM22.NBU_POLICY.MAXJOBS.InChnExch03
10/22/2015 16:51:02 - granted resource  DFH683
10/22/2015 16:51:02 - granted resource  IBM.ULT3580-TD5.000
10/22/2015 16:51:02 - granted resource  InChnM22-hcart2-robot-tld-1
10/22/2015 16:51:03 - estimated 0 kbytes needed
10/22/2015 16:51:03 - Info nbjm (pid=13107208) started backup (backupid=inchnexch03.tcs.com_1445512862) job for client inchnexch03.tcs.com, policy InChnExch03, schedule Daily_Backup on storage unit InChnM22-hcart2-robot-tld-1
10/22/2015 16:51:04 - Info bpbrm (pid=8650842) inchnexch03.tcs.com is the host to backup data from
10/22/2015 16:51:04 - Info bptm (pid=30277792) using 524288 data buffer size
10/22/2015 16:51:04 - Info bpbrm (pid=8650842) telling media manager to start backup on client
10/22/2015 16:51:04 - Info bptm (pid=30277792) using 64 data buffers
10/22/2015 16:51:05 - Info bpbrm (pid=8650842) spawning a brm child process
10/22/2015 16:51:05 - Info bpbrm (pid=8650842) child pid: 29687968
10/22/2015 16:51:07 - Info bpbrm (pid=8650842) sending bpsched msg: CONNECTING TO CLIENT FOR inchnexch03.tcs.com_1445512862
10/22/2015 16:51:07 - connecting
10/22/2015 16:51:08 - Info bpbrm (pid=8650842) start bpbkar on client
10/22/2015 16:51:08 - connected; connect time: 0:00:00
10/22/2015 16:51:08 - begin writing
10/22/2015 16:51:19 - Info bpbkar (pid=52372) Backup started
10/22/2015 16:51:19 - Info bpbrm (pid=8650842) Sending the file list to the client
10/22/2015 16:51:19 - Info bpbkar (pid=52372) change time comparison:<disabled>
10/22/2015 16:51:19 - Info bpbkar (pid=52372) archive bit processing:<disabled>
10/22/2015 16:51:19 - Info bpbkar (pid=52372) will attempt to use change journal data for <D:\ApplicationLogs>
10/22/2015 17:07:19 - Info bpbkar (pid=52372) not using change journal data for <D:\ApplicationLogs>: snapshot has not been applied (unable to track open files)
10/22/2015 17:07:19 - Info bpbkar (pid=52372) not using change journal data for enumeration for <D:\ApplicationLogs> but will use it for change detection

[sdo]

It looks like you are performing a plain file system backup of MS Exchange application transaction logs folder.  Is that correct?

If yes, then are you sure that you want to do that?

If MS Exchange is "up" at the time that the backup runs then it is highly likely that this "backup data" will either be corrupt, or incomplete, and un-usable for restore/recovery.

In which case an MS Exchange "database agent backup" is the way forward, i.e. cease the backup that you are attempting... read the NetBackup for MS Exchange Admin Guide and re-implement your backups of this server.