vmware backups are failing bulk with status 14 & 42 when using appliance as media server.

[ajiwww]

NBU versions are 7.5.6

Appliance version is 2.5.3 (this is using as a media server and backup is going to DD & Quantum disks)

Vcenter version is 5.1

Proxies are Windows server 2008 R2 64 Bit and they are at cloud.

Transport mode is NBD

The appliance was upgraded to 2.5.3 from 2.0.3. Since then, we are experiencing bulk backup failures with 14 & 42. The backup is writing some data and getting failed later. They will get completed on the conescutive attempts.

12/06/2013 09:21:39 - Info bpbkar (pid=4580) INF - Transport Type = nbd
12/06/2013 09:34:36 - Critical bpbrm (pid=8326) from client TOFRTS26.FPS.loc: FTL - socket write failed
12/06/2013 09:34:36 - Critical bpbrm (pid=8326) from client TOFRTS26.FPS.loc: FTL - socket write failed
12/06/2013 09:34:36 - Error bpbrm (pid=8326) from client TOFRTS26.FPS.loc: ERR - tar file write error (14)
12/06/2013 09:34:41 - Error bptm (pid=8327) media manager terminated by parent process
12/06/2013 09:35:08 - Error bpbrm (pid=8326) could not send server status message
12/06/2013 09:35:08 - Critical bpbrm (pid=8326) unexpected termination of client TOFRTS26.FPS.loc
12/06/2013 09:35:09 - Info bpbkar (pid=0) done. status: 14: file write failed

12/06/2013 09:05:23 - Info bpbkar (pid=5740) INF - Transport Type = nbd
12/06/2013 09:17:36 - Critical bpbrm (pid=5275) from client TOFRTS03.FPS.loc: FTL - socket write failed
12/06/2013 09:17:36 - Critical bpbrm (pid=5275) from client TOFRTS03.FPS.loc: FTL - socket write failed
12/06/2013 09:17:36 - Error bpbrm (pid=5275) from client TOFRTS03.FPS.loc: ERR - tar file write error (14)
12/06/2013 09:17:37 - Error bptm (pid=5410) system call failed - Connection reset by peer (at child.c.1298)
12/06/2013 09:17:37 - Error bptm (pid=5410) unable to perform read from client socket, connection may have been broken
12/06/2013 09:18:12 - Info bptm (pid=5297) EXITING with status 42 <----------
12/06/2013 09:18:13 - Error bpbrm (pid=5275) could not send server status message
12/06/2013 09:18:13 - Critical bpbrm (pid=5275) unexpected termination of client TOFRTS03.FPS.loc
12/06/2013 09:18:14 - Info bpbkar (pid=0) done. status: 42: network read failed
12/06/2013 09:18:14 - end writing; write time: 0:13:00
network read failed  (42)

Case is opened with Vendor and so far no help :(

Does anyone faced this after the upgrade of appliance. Suspecting some issues with network drivers after the upgrade of the appliance.

[SymTerry]

Hello,

if you could please attach the bpbkar and bpbrm log, it would be useful to look at them. You also mentioed you opened a case. Please PM me with the case number, and I will see if I can get you further help to resolve this issue.

[ajiwww]

The MTU value of appliance, we are not able to set it to 9000. So network congestion and packt dropes are the showing at the interface. Case number for that is # 05583051  

For status 14 & 42, case number is 05604547  

[ajiwww]

bpbrm  bpbkar log files are too large to attach as the number of job failures are more than 1000. i will try to get a PID from both and attach here.