Flash Backup Failing With Status Code 14

[Jecintha]

Hello,

We have Netbackup 7.5 configured on a Win 2008 R2, 64 bit OS Server.  Flash Backup for a server with Win 2008 64 bit OS server has been failing for the past couple of days.  Please help with the steps to proceed further.

Regards,

Jecintha

[Marianne]

Please start by showing us all text in Details Tab of the failed job.

Status 14 is either network or file problem. Information in Details tab should tell us where to start with troubleshooting.

[Jecintha]

Hello Marrianne,

Thanks for the update.  Please refer to the below logs:

7/18/2013 10:35:22 AM - Info nbjm(pid=2400) starting backup job (jobid=24910) for client "nodename", policy "policy name", schedule Incremental 
7/18/2013 10:35:22 AM - estimated 0 Kbytes needed
7/18/2013 10:35:22 AM - Info nbjm(pid=2400) started backup (backupid=ICCPIMDB01_1374123922) job for client ICCPIMDB01, policy "policy name", schedule Incremental on storage unit MUMESM01-hcart-robot-tld-0
7/18/2013 10:35:22 AM - started process bpbrm (3548)
7/18/2013 10:35:26 AM - Info bpbrm(pid=3548) starting bptm          
7/18/2013 10:35:26 AM - Info bpbrm(pid=3548) Started media manager using bpcd successfully      
7/18/2013 10:35:28 AM - Info bpbrm(pid=3548) ICCPIMDB01 is the host to backup data from    
7/18/2013 10:35:28 AM - Info bpbrm(pid=3548) telling media manager to start backup on client    
7/18/2013 10:35:28 AM - Info bptm(pid=1208) using 262144 data buffer size       
7/18/2013 10:35:28 AM - Info bptm(pid=1208) using 64 data buffers        
7/18/2013 10:35:29 AM - Info bptm(pid=1208) start backup          
7/18/2013 10:35:29 AM - Info bptm(pid=1208) Waiting for mount of media id X271L4 (copy 2) on server MUMESM01.
7/18/2013 10:35:34 AM - mounting X271L4
7/18/2013 10:35:36 AM - Info bpbrm(pid=8504) sending bpsched msg: CONNECTING TO CLIENT FOR ICCPIMDB01_1374123922    
7/18/2013 10:35:36 AM - connecting
7/18/2013 10:35:37 AM - Info bptm(pid=5652) setting receive network buffer to 1049600 bytes     
7/18/2013 10:35:37 AM - Info bpbrm(pid=8504) start bpbkar32 on client        
7/18/2013 10:35:37 AM - connected; connect time: 00:00:01
7/18/2013 10:35:43 AM - Info bpbkar32(pid=5016) Backup started          
7/18/2013 10:35:43 AM - Info bpbrm(pid=8504) Sending the file list to the client     
7/18/2013 10:36:16 AM - Info bptm(pid=1208) media id X271L4 mounted on drive index 5, drivepath {1,0,9,0}, drivename IBM.ULT3580-TD4.001, copy 2
7/18/2013 10:36:17 AM - mounted; mount time: 00:00:43
7/18/2013 10:36:17 AM - positioning X271L4 to file 219
7/18/2013 10:37:39 AM - positioned X271L4; position time: 00:01:22
7/18/2013 10:37:39 AM - begin writing
7/18/2013 11:45:20 AM - Critical bpbrm(pid=8504) from client ICCPIMDB01: FTL - socket write failed    
7/18/2013 11:45:20 AM - Critical bpbrm(pid=8504) from client ICCPIMDB01: FTL - socket write failed    
7/18/2013 11:45:20 AM - Critical bpbrm(pid=8504) from client ICCPIMDB01: FTL - tar file write error (14)  
7/18/2013 11:45:22 AM - Error bpbrm(pid=8504) could not send server status message      
7/18/2013 11:45:23 AM - Critical bpbrm(pid=8504) unexpected termination of client ICCPIMDB01       
7/18/2013 11:45:23 AM - Info bpbrm(pid=3548) child done, status 14        
7/18/2013 11:45:23 AM - Info bpbrm(pid=3548) sending message to media manager: STOP BACKUP ICCPIMDB01_1374123922    
7/18/2013 11:45:30 AM - Info bpbrm(pid=3548) media manager for backup id ICCPIMDB01_1374123922 exited with status 150: termination requested by administrator
7/18/2013 11:45:30 AM - end writing; write time: 01:07:51
file write failed(14)

Regards,

Jecintha

[Marianne · Accepted solution]

Critical bpbrm(pid=8504) from client ICCPIMDB01: FTL - socket write failed 
Critical bpbrm(pid=8504) from client ICCPIMDB01: FTL - tar file write error (14) 
Critical bpbrm(pid=8504) unexpected termination of client ICCPIMDB01 

If this was working previously, it means that 'something' has happened on the client in the meantime to break it.

Have you tried to do a 'normal' MS-Windows backup of a folder just to see if the problem is only seen with FlashBackup or with all backups?

A couple of things to check as per the Status Code Guide:

* Make sure that routers, bridges, and other network devices are all at "full" duplex.

* Use a "sniffer" program to determine the number of packets being rejected or re-requested.

* On Windows systems, the client bpbkar log may contain a 10054 Connection Reset Error (usually indicates a hardware error). Somewhere between the NetBackup client and server, the connection was reset. When NetBackup receives this error, it cannot continue the backup. This error has been attributed to the following:

* A hiccup in the network.

* A bad network interface card on a NetBackup client.

* A bad network interface card on the NetBackup server.

* Faulty routers.

* Any other applications that interfere with NetBackup connections.

To find out if anything was actually written before the backup failed, ensure that all of the following log folders exist:

on media server: bpbrm and bptm

on client: bpbkar

[CRZ]

Just to add one thing to Marianne's excellent post, I'm sure you have noticed this in your logs already, but just in case you haven't...

7/18/2013 10:37:39 AM - begin writing
7/18/2013 11:45:20 AM - Critical bpbrm(pid=8504) from client ICCPIMDB01: FTL - socket write failed    
7/18/2013 11:45:20 AM - Critical bpbrm(pid=8504) from client ICCPIMDB01: FTL - socket write failed    
7/18/2013 11:45:20 AM - Critical bpbrm(pid=8504) from client ICCPIMDB01: FTL - tar file write error (14)

Seeing a log break of over an hour always gets me wondering - could something be timing out your connection?  Maybe a firewall?

When it worked, how long did it take backups to complete?  About the same time... or less than an hour?

[watsons]

Info bptm(pid=5652) setting receive network buffer to 1049600 bytes

The above message look suspicious to (maybe just) me too, usually I don't see a network buffer this high. But I will have to check my recent backups to see if this has be changed in default..

[Jecintha]

Hello All,

Thanks for the suggestions.  As per Marianne's suggestion,  I had tried enabling windows backup on the client and I see that it is running fine and is completed successfully.  The backup was running fine till the 9th of July and now we see the backup repeatedly failing.  We are permanantly deactiviating the Flashbackup and would start Normal Ms-win backup for this server.

Regards,

Jecintha