network connection broken(40) - Error bpbrm(pid=142309) could not write FILE ADDED message to OUTSOCK

[yobole]

Guys I have been battling this for weeks and losin the will the live . I have a 2.5TB i need to bring into netbackup . This is the first long back and have created a 100GB volume to handle the NB VSS snapshot for the volume . I calculated thus should take about  4 to 5 days

The backup is however failing with

error network connection broken(40) 19/05/2015 17:24:51 - Error bpbrm(pid=142309) could not write FILE ADDED message to OUTSOCK

The eventaully the snapshot error fails as the 100GB snapshot volmue gets filled up

network connection broken(40)
19/05/2015 17:24:51 - Error bpbrm(pid=142309) could not write FILE ADDED message to OUTSOCK     
19/05/2015 17:27:06 - Error bpbrm(pid=142309) could not write FILE ADDED message to OUTSOCK     
19/05/2015 17:40:37 - Error bpbrm(pid=142309) could not write FILE ADDED message to OUTSOCK     
19/05/2015 17:41:37 - Error bpbrm(pid=142309) could not write FILE ADDED message to OUTSOCK     
 
19/05/2015 18:26:37 - Error bpbrm(pid=142309) could not write FILE ADDED message to OUTSOCK     
19/05/2015 18:47:53 - Warning bpbrm(pid=142309) from client ser01bas1uk: WRN - can't open directory: D:\UK\Pro\ Photos & survey (WIN32 2: Unknown error)
19/05/2015 18:47:54 - Error bpbrm(pid=142309) from client ser01bas1uk: ERR - Snapshot Error while opening directory: GLOBALROOT\Device\HarddiskVolumeShadowCopy11\UK\Pro\Photos & survey
19/05/2015 18:48:03 - Critical bpbrm(pid=142309) from client ser01bas1uk: FTL - Backup operation aborted!     
19/05/2015 18:48:04 - Info bpbkar(pid=5636) accelerator sent 1554310496256 bytes out of 1547439793152 bytes to server, optimization 0.0%
19/05/2015 18:48:04 - Error bpbrm(pid=142309) could not write BPBRM_OPTIMIZATION_RATE message to OUTSOCK,error code: 32    
19/05/2015 18:48:05 - Info bpbkar(pid=5636) bpbkar waited 4699057 times for empty buffer, delayed 12859503 times.   
19/05/2015 18:50:20 - Info uklon1bak01(pid=142355) StorageServer=PureDisk:uklon1bak01; Report=PDDO Stats for (uklon1bak01): scanned: 1511310541 KB, CR sent: 117166375 KB, CR sent over FC: 0 KB, dedup: 92.2%, cache disabled
19/05/2015 18:51:30 - Error bptm(pid=142355) media manager terminated by parent process       
19/05/2015 18:51:33 - Error bpbrm(pid=142309) could not write EXIT STATUS to OUTSOCK      
19/05/2015 18:51:33 - Info bpbkar(pid=5636) done. status: 156: snapshot error encountered      

Media server is a Netbackup appliance 7.6.0.4 successfully backing up loads of other clients..

[sdo]

What do these show:

> bpclient -L -client ser01bas1uk

> bpgetconfig -M ser01bas1uk | find /i "buff"

.

The Win32 status 2 is "file not found", from:

https://msdn.microsoft.com/en-us/library/windows/desktop/ms681382(v=vs.85).aspx

.

I'm wondering if you were to use '-WOFB_error 1' on the master server client attributes, i.e. tell NetBackup to continue, even if the snapshot fails, whether you could get a backup of most files.  Perhaps the snapshot full/fail is causing the 'file not found' error, i.e. the 'file not found' error could be an artifact of the VSS full/fail, i.e.:

> bpclient -L -client ser01bas1uk

> bpclient -update -client ser01bas1uk -WOFB_error 1

> bpclient -L -client ser01bas1uk

.

But having a 100GB VSS delta change log fill up... smacks of a very busy file system.

Have you also considered FlashBackup?  (if you have the required license)

[yobole]

Output of BPclient have been attached

I tried flashbackup and got a Succesfull full backup after 4 days , However the subsequent daily diff was failing with error code 14 which Symantec were struggling to resolve and other full backup was taking 4 days which does not fit into my weekend backup time window for this remote server

[sdo]

Ok - that client has '-WOFB_error 0' which is 'on error abort backup'.  Perhaps try again with the '-WOFB_error 1' setting above, which is 'on error continue backup'.

[Marianne · Accepted solution]

This looks like an issue with connectivity between the media server and the master:

Error bpbrm(pid=142309) could not write FILE ADDED message to OUTSOCK      

Verbose log of bpbrm on the media server is the first place I would look for issues.

Other posts with similar issues:

could not write FILE ADDED message to OUTSOCK 

Backup failing with status 40 

status code 40 

NetBackup Timeouts

status 636 / status 42 errors - backing up to MSDP pool 

[yobole]

After must support with symantec and still no joy we have now decided to try the backup going to our DR MSDP ( windows server ) to rule out the contention on the main MSDP Pool ( appliance)

[Marianne]

Have you had a look at anything I posted above? Symantec cannot really solve network issues in the environment....

[yobole]

Thanks Marianne yes looked at them  and fully agree with you .. Running AppCritical report

From media server to client there appears to be marginal performance (24.4Mb/s).  However we are seeing very poor performance from the client to the media server.  

• Total Capacity is at 22.1Mb/s
• Seeing 8% data loss suggesting heavy congestion.  
• Seeing heavy utilization at the target destination (the media server.)

Currently enggaing wth the network team to check on the reasons for the packet loss.  8% could easily explain why we are having problems writing to the media server.