Forum Discussion

yobole's avatar
yobole
Level 6
10 years ago

network connection broken(40) - Error bpbrm(pid=142309) could not write FILE ADDED message to OUTSOCK

Guys I have been battling this for weeks and losin the will the live . I have a 2.5TB i need to bring into netbackup . This is the first long back and have created a 100GB volume to handle the NB VSS snapshot for the volume . I calculated thus should take about  4 to 5 days

The backup is however failing with

error network connection broken(40) 19/05/2015 17:24:51 - Error bpbrm(pid=142309) could not write FILE ADDED message to OUTSOCK

 

The eventaully the snapshot error fails as the 100GB snapshot volmue gets filled up

 

network connection broken(40)
19/05/2015 17:24:51 - Error bpbrm(pid=142309) could not write FILE ADDED message to OUTSOCK     
19/05/2015 17:27:06 - Error bpbrm(pid=142309) could not write FILE ADDED message to OUTSOCK     
19/05/2015 17:40:37 - Error bpbrm(pid=142309) could not write FILE ADDED message to OUTSOCK     
19/05/2015 17:41:37 - Error bpbrm(pid=142309) could not write FILE ADDED message to OUTSOCK     
 
19/05/2015 18:26:37 - Error bpbrm(pid=142309) could not write FILE ADDED message to OUTSOCK     
19/05/2015 18:47:53 - Warning bpbrm(pid=142309) from client ser01bas1uk: WRN - can't open directory: D:\UK\Pro\ Photos & survey (WIN32 2: Unknown error)
19/05/2015 18:47:54 - Error bpbrm(pid=142309) from client ser01bas1uk: ERR - Snapshot Error while opening directory: GLOBALROOT\Device\HarddiskVolumeShadowCopy11\UK\Pro\Photos & survey
19/05/2015 18:48:03 - Critical bpbrm(pid=142309) from client ser01bas1uk: FTL - Backup operation aborted!     
19/05/2015 18:48:04 - Info bpbkar(pid=5636) accelerator sent 1554310496256 bytes out of 1547439793152 bytes to server, optimization 0.0%
19/05/2015 18:48:04 - Error bpbrm(pid=142309) could not write BPBRM_OPTIMIZATION_RATE message to OUTSOCK,error code: 32    
19/05/2015 18:48:05 - Info bpbkar(pid=5636) bpbkar waited 4699057 times for empty buffer, delayed 12859503 times.   
19/05/2015 18:50:20 - Info uklon1bak01(pid=142355) StorageServer=PureDisk:uklon1bak01; Report=PDDO Stats for (uklon1bak01): scanned: 1511310541 KB, CR sent: 117166375 KB, CR sent over FC: 0 KB, dedup: 92.2%, cache disabled
19/05/2015 18:51:30 - Error bptm(pid=142355) media manager terminated by parent process       
19/05/2015 18:51:33 - Error bpbrm(pid=142309) could not write EXIT STATUS to OUTSOCK      
19/05/2015 18:51:33 - Info bpbkar(pid=5636) done. status: 156: snapshot error encountered      

Media server is a Netbackup appliance 7.6.0.4 successfully backing up loads of other clients..

 

 

7.6
Backing Up
Backup and Recovery
NetBackup
Windows Server (2003-2008)

7 Replies

Sort By
  • sdo's avatar
    sdo
    Moderator
    10 years ago

    What do these show:

    > bpclient -L -client ser01bas1uk

> bpgetconfig -M ser01bas1uk | find /i "buff"

    .

    The Win32 status 2 is "file not found", from:

    https://msdn.microsoft.com/en-us/library/windows/desktop/ms681382(v=vs.85).aspx

    .

    I'm wondering if you were to use '-WOFB_error 1' on the master server client attributes, i.e. tell NetBackup to continue, even if the snapshot fails, whether you could get a backup of most files.  Perhaps the snapshot full/fail is causing the 'file not found' error, i.e. the 'file not found' error could be an artifact of the VSS full/fail, i.e.:

    > bpclient -L -client ser01bas1uk

> bpclient -update -client ser01bas1uk -WOFB_error 1

> bpclient -L -client ser01bas1uk

    .

    But having a 100GB VSS delta change log fill up... smacks of a very busy file system.

    Have you also considered FlashBackup?  (if you have the required license)

  • yobole's avatar
    yobole
    Level 6
    10 years ago

    Output of BPclient have been attached

    I tried flashbackup and got a Succesfull full backup after 4 days , However the subsequent daily diff was failing with error code 14 which Symantec were struggling to resolve and other full backup was taking 4 days which does not fit into my weekend backup time window for this remote server

     

     

     


     

  • sdo's avatar
    sdo
    Moderator
    10 years ago

    Ok - that client has '-WOFB_error 0' which is 'on error abort backup'.  Perhaps try again with the '-WOFB_error 1' setting above, which is 'on error continue backup'.

  • Marianne's avatar
    Marianne
    Level 6
    10 years ago

    This looks like an issue with connectivity between the media server and the master:

    Error bpbrm(pid=142309) could not write FILE ADDED message to OUTSOCK      

    Verbose log of bpbrm on the media server is the first place I would look for issues.

    Other posts with similar issues:

    could not write FILE ADDED message to OUTSOCK 

    Backup failing with status 40 

    status code 40 

    NetBackup Timeouts

    status 636 / status 42 errors - backing up to MSDP pool 

  • yobole's avatar
    yobole
    Level 6
    10 years ago

    After must support with symantec and still no joy we have now decided to try the backup going to our DR MSDP ( windows server ) to rule out the contention on the main MSDP Pool ( appliance)

  • Marianne's avatar
    Marianne
    Level 6
    10 years ago
    Have you had a look at anything I posted above? Symantec cannot really solve network issues in the environment....
  • yobole's avatar
    yobole
    Level 6
    10 years ago

    Thanks Marianne yes looked at them  and fully agree with you .. Running AppCritical report

     

    From media server to client there appears to be marginal performance (24.4Mb/s).  However we are seeing very poor performance from the client to the media server.  

    • Total Capacity is at 22.1Mb/s
    • Seeing 8% data loss suggesting heavy congestion.  
    • Seeing heavy utilization at the target destination (the media server.)

    Currently enggaing wth the network team to check on the reasons for the packet loss.  8% could easily explain why we are having problems writing to the media server.