VM Backup failing with Error 13

[monty_222]

12/22/2011 10:02:46 PM - requesting resource psddbu02-vcb-su

12/22/2011 10:02:46 PM - requesting resource ps23nb20.NBU_CLIENT.MAXJOBS.PSVI28HERM01

12/22/2011 10:02:46 PM - requesting resource ps23nb20.NBU_POLICY.MAXJOBS.VCB-Daily-NQ2

12/22/2011 10:02:46 PM - Info nbrb(pid=4608) Limit has been reached for the logical resource ps23nb20.NBU_POLICY.MAXJOBS.VCB-Daily-NQ2    

12/22/2011 10:27:15 PM - granted resource ps23nb20.NBU_CLIENT.MAXJOBS.PSVI28HERM01

12/22/2011 10:27:15 PM - granted resource ps23nb20.NBU_POLICY.MAXJOBS.VCB-Daily-NQ2

12/22/2011 10:27:15 PM - granted resource MediaID=@aaaaj;DiskVolume=psddbu02-vcb-lsu;DiskPool=psddbu02-vcb-dp;Path=psddbu02-vcb-lsu;StorageServer=psddbu02.informatica.com;MediaServer=ps23nb09

12/22/2011 10:27:15 PM - granted resource psddbu02-vcb-su

12/22/2011 10:27:15 PM - estimated 0 Kbytes needed

12/22/2011 10:27:16 PM - started process bpbrm (3248)

12/22/2011 10:27:17 PM - connecting

12/22/2011 10:27:17 PM - connected; connect time: 00:00:00

12/22/2011 10:27:21 PM - begin writing

12/22/2011 10:32:18 PM - Error bpbrm(pid=3248) db_FLISTsend failed: file read failed (13)       

12/22/2011 10:32:28 PM - end writing; write time: 00:05:07

file read failed(13)

[Mark_Solutions]

A little more details please ...

NBU versions, vSphere or VCB, vCenter / ESX versions and policy / snapshot settings

Also ESX datastore locations / type

Thanks

[MKT]

You can try updating the Client Read Timeout on the client/media server in case the processing just isn't going fast enough....

[Marianne]

PLEASE let us know all releveant info as per Mark's request....

I found this TN that could be relevant, but we need more detail: http://www.symantec.com/docs/TECH62391

[VirtualED]

Check to see if bpbkar32.exe is crashing on the VMware Backup Host.  It could be several issues, I suggest upgrading to the latest NetBackup to address all known issues.

7.1.0.3 is the latest version currently available.

[pikachu]

• OS of backup host
• 32bit or 64bit backup host os
• mapepd backup? unmapped backup?
• how much ram in the backup host
• what version of NBU is on the backup host?
• Do you have any vmware rollup patches installed on the backup host?

[monty_222]

• OS of backup host  - Windows 2003 r2 Standard Edition
• 32bit or 64bit backup host os - 64 bit 
• mapepd backup? unmapped backup? - mapped
• how much ram in the backup host - 4 Gb
• what version of NBU is on the backup host? - 7.0.1
• Do you have any vmware rollup patches installed on the backup host? - no .

Also taking help from marianne Post about the  Technote .

It is a V Sphere ..attached is the Policy setting 

 

[falti_manullang]

Disable Snapshot mount point (VCB).

[pikachu]

VCB was really dependent on the boot.ini to be able to do SFR aka Mapped Backups.

Note: If you upgrade a Windows 2003 machine to Windows 2008, sometimes the boot.ini does NOT get deleted and the next vstorage backup will go crazy. Difficult to identify if you have never run into that issue before.

[pikachu]

OS of backup host  - Windows 2003 r2 Standard Edition

32bit or 64bit backup host os - 64 bit 

mapepd backup? unmapped backup? - mapped

=> try unmapped ; does that work?

how much ram in the backup host - 4 Gb

=> depending on how many vms you backup and how big they are... that might not be enough.

what version of NBU is on the backup host? - 7.0.1

Do you have any vmware rollup patches installed on the backup host? - no .
=> you HAVE to install eebinstaller.2105102.14.AMD64

Bonus EEB: eebinstaller.2193082.1.AMD64

[pikachu]

That has no impact even if it is selected. Still... make sure VCB is NOT installed anymore on the backup host. Check add/remove programs, task manager (vmount2) and make sure that c:\program files*\vmware\vmware consolidated backup host is gone.

[VirtualED]

The error:

db_FLISTsend failed: file read failed (13)

Means that bpbrm send some information about the data to the master server bpdbm and it did not like it.  In most cases, this is the Volume Recognition is failing on the Virtual Machine.  It is strongly advise that you upgrade from 7.0.1 to 7.1.0.2 or 7.1.0.3 as they are several issues fixed around this area.

[Mark_Solutions]

I see that you have quiece disabled - are you sure you want this setting as the data may not be consistent.

I would tidy up the snapshot options first:

Ensble quiece, remove the VCB entries, select a data transfer method rather than leave it as "Try All"

Make sure that VMTools is up to date on the host

Increase the client read timeout for the Media Servers to 1800

Try again and let us know how it goes

(but update everything as soon as you can)

Hope this helps

[monty_222]

Thanks Guys , I had made changes as per suggestions from Mark and will try running the backup again to see how it goes.

[monty_222]

I made the changes but the backup still fails with error 13 .

1/9/2012 2:37:27 PM - requesting resource psddbu02-vcb-su

1/9/2012 2:37:27 PM - requesting resource ps23nb20.NBU_CLIENT.MAXJOBS.psvi28herm01.informatica.com

1/9/2012 2:37:27 PM - requesting resource ps23nb20.NBU_POLICY.MAXJOBS.VCB-Daily-NQ

1/9/2012 2:37:27 PM - granted resource ps23nb20.NBU_CLIENT.MAXJOBS.psvi28herm01.informatica.com

1/9/2012 2:37:27 PM - granted resource ps23nb20.NBU_POLICY.MAXJOBS.VCB-Daily-NQ

1/9/2012 2:37:27 PM - granted resource MediaID=@aaaaj;DiskVolume=psddbu02-vcb-lsu;DiskPool=psddbu02-vcb-dp;Path=psddbu02-vcb-lsu;StorageServer=psddbu02.informatica.com;MediaServer=ps23nb09

1/9/2012 2:37:27 PM - granted resource psddbu02-vcb-su

1/9/2012 2:37:27 PM - estimated 0 Kbytes needed

1/9/2012 2:37:28 PM - started process bpbrm (9124)

1/9/2012 2:37:30 PM - connecting

1/9/2012 2:37:30 PM - connected; connect time: 00:00:00

1/9/2012 2:37:33 PM - begin writing

1/9/2012 2:40:37 PM - Error bpbrm(pid=9124) db_FLISTsend failed: file read failed (13)       

1/9/2012 2:40:46 PM - end writing; write time: 00:03:13

file read failed(13)

[Mark_Solutions]

I see that it fails after just 3 minutes this time so that tends to rule out the timeout issues (unless you have short client read timeouts set)

If this host fully resolvable (ping / reverse DNS lookup etc.) from the VMWare Backup Host? Although it doesnt back it up over the network it does do a resolution check first.

If this is not the case we need some logs now so set up the following, re-run the failing backup and post the logs

install_path\NetBackup\logs\bpbrm on the Media server
install_path\NetBackup\logs\bptm on the Media server
install_path\NetBackup\logs\bpfis on the VMWare Backup Host (Media Server)
install_path\NetBackup\logs\bpcd on the VMWare Backup Host (Media Server)
install_path\NetBackup\logs\bpbkar on the VMWare Backup Host (Media Server)

[monty_222]

Had to open a case with Symantec as still i am clueless after making all the changes still continues the error .

[Marianne]

Have you installed the VMWare EEB yet?

(ET2105102) Issues resolved and hotfix for NetBackup for VMWare backups
 http://www.symantec.com/docs/TECH141502

[monty_222]

Thanks Marianne, Will download the EEB installer and update the forum how does it goes.

[pikachu · Accepted solution]

Contact support to get the latest rollup.

71:

eebinstaller.2299772.3

701:

eebinstaller.2105102.14

700:

eebinstaller.2060693.11