vmware - flashbackup (2 - mapped fullVM) v6.5.1 fails (ERR - Unable to read metadata for index)

[kyle_beckham]

We are trying to evaluate netbackup v6.5.1 for use with vmware/vcb.  We can get netbackup to interface with vmware/vcb, snapshots are taken, full vm image is dumped on the proxy server. But then when netbackup i assume if doing the mapping of the vm it fails with the errors below. We tried this on two different small vm's.

Seems like netbackup starts to map the backup and then just fails, finished but never really backup anything.

We are also seeing the second error

1/1/2008 1:39:47 PM - estimated 0 kbytes needed
1/1/2008 1:39:47 PM - started process bpbrm (5140)
1/1/2008 1:39:47 PM - connecting
1/1/2008 1:39:47 PM - connected; connect time: 00:00:00
1/1/2008 1:39:50 PM - begin writing
1/1/2008 1:40:34 PM - Error bpbrm(pid=5140) from client MYSQL1: ERR - Unable to read metadata for index: 121943, VFM error = 1.
1/1/2008 1:40:34 PM - Error bpbrm(pid=5140) from client MYSQL1: ERR - Unable to read metadata for index: 121945, VFM error = 1.
1/1/2008 1:40:34 PM - Error bpbrm(pid=5140) from client MYSQL1: ERR - Unable to read metadata for index: 121946, VFM error = 1.
1/1/2008 1:40:34 PM - Error bpbrm(pid=5140) from client MYSQL1: ERR - Unable to read metadata for index: 121948, VFM error = 1.
1/1/2008 1:40:34 PM - Error bpbrm(pid=5140) from client MYSQL1: ERR - Unable to read metadata for index: 121950, VFM error = 1.
1/1/2008 1:40:34 PM - Error bpbrm(pid=5140) from client MYSQL1: ERR - Unable to read metadata for index: 121951, VFM error = 1.
1/1/2008 1:40:34 PM - Error bpbrm(pid=5140) from client MYSQL1: ERR - Unable to read metadata for index: 121953, VFM error = 1.
1/1/2008 1:40:34 PM - Error bpbrm(pid=5140) from client MYSQL1: ERR - Unable to read metadata for index: 121956, VFM error = 1.
1/1/2008 1:40:34 PM - Error bpbrm(pid=5140) from client MYSQL1: ERR - Unable to read metadata for index: 121966, VFM error = 1.
1/1/2008 1:40:34 PM - Error bpbrm(pid=5140) from client MYSQL1: ERR - Unable to read metadata for index: 121971, VFM error = 1.
1/1/2008 1:40:34 PM - Error bpbrm(pid=5140) from client MYSQL1: ERR - Unable to read metadata for index: 121972, VFM error = 1.
1/1/2008 1:40:34 PM - Error bpbrm(pid=5140) from client MYSQL1: ERR - Unable to read metadata for index: 121973, VFM error = 1.
1/1/2008 1:40:34 PM - Error bpbrm(pid=5140) from client MYSQL1: ERR - Unable to read metadata for index: 121982, VFM error = 1.
1/1/2008 1:55:08 PM - end writing; write time: 00:15:18
the requested operation was partially successful(1)

The job was successfully completed, but some files may have been
busy or unaccessible. See the problems report or the client's logs for more details.

=============================

[kyle_beckham]

The second error we are seeing in the event log is:

The description for Event ID ( 1 ) in Source ( VxMS NTFS Mapping Plug-in ) cannot be found.
The local computer may not have the necessary registry information or message DLL files to
 display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve
this description; see Help and Support for details. The following information is part of the event:

Error: 0x00000001, in file .\NtfsMftParser.cpp, line 4988

[schmaustech]

Did that host have a dynamic volume on it?

[marcus_lebisk]

Hi Guys, I facing almost the same problem. On my case the backup starts but due this error I'm getting error 13 - Cannot read from socket.

Any ideas?

Thanks

Marcus

10/03/2009 07:52:54 - estimated 0 kbytes needed
10/03/2009 07:52:55 - started process bpbrm (7248)
10/03/2009 07:52:55 - connecting
10/03/2009 07:52:56 - connected; connect time: 00:00:01
10/03/2009 07:53:02 - begin writing
10/03/2009 07:53:30 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read next index. VFM error = 6
10/03/2009 07:53:32 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 61, VFM error = 2058.
10/03/2009 07:53:45 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 33, VFM error = 2058.
10/03/2009 07:53:59 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
10/03/2009 07:53:59 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 3590, VFM error = 2058.
10/03/2009 07:54:00 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 4865, VFM error = 2058.
10/03/2009 07:54:00 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 7297, VFM error = 2058.
10/03/2009 07:54:01 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 25524, VFM error = 2058.
10/03/2009 07:54:01 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 4865, VFM error = 2058.
10/03/2009 07:54:02 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 61, VFM error = 2058.
10/03/2009 07:54:02 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 4781, VFM error = 2058.
10/03/2009 07:54:02 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 61, VFM error = 2058.
10/03/2009 07:54:03 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 4650, VFM error = 2058.
10/03/2009 07:54:03 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 5751, VFM error = 2058.
10/03/2009 07:54:04 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 8497, VFM error = 2058.
10/03/2009 07:54:04 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
10/03/2009 07:54:04 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 58, VFM error = 2058.
10/03/2009 07:54:05 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
10/03/2009 07:54:05 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
10/03/2009 07:54:06 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
10/03/2009 07:54:06 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 123, VFM error = 2058.
10/03/2009 07:54:07 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
10/03/2009 07:54:07 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
10/03/2009 07:54:08 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 5901, VFM error = 2058.
10/03/2009 07:54:08 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 5900, VFM error = 2058.
10/03/2009 07:54:08 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 57, VFM error = 2058.
10/03/2009 07:54:09 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 39, VFM error = 2058.
10/03/2009 07:54:09 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 12195, VFM error = 2058.
10/03/2009 07:54:10 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 3715, VFM error = 2058.
10/03/2009 07:54:10 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 3715, VFM error = 2058.
10/03/2009 07:54:11 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 7238, VFM error = 2058.
10/03/2009 07:54:11 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 6001, VFM error = 2058.
10/03/2009 07:54:12 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 6001, VFM error = 2058.
10/03/2009 07:54:12 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 7202, VFM error = 2058.
10/03/2009 07:54:12 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 6031, VFM error = 2058.
10/03/2009 07:54:13 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 6095, VFM error = 2058.
10/03/2009 07:54:13 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 6095, VFM error = 2058.
10/03/2009 07:54:14 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 6095, VFM error = 2058.
10/03/2009 07:54:14 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 57, VFM error = 2058.
10/03/2009 07:54:15 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 6110, VFM error = 2058.
10/03/2009 07:54:15 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 6110, VFM error = 2058.
10/03/2009 07:54:16 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 5957, VFM error = 2058.
10/03/2009 07:54:16 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 5957, VFM error = 2058.
10/03/2009 07:54:16 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 46627, VFM error = 2058.
10/03/2009 07:54:17 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 5723, VFM error = 2058.
10/03/2009 07:54:17 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
10/03/2009 07:54:18 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
10/03/2009 07:54:18 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
10/03/2009 07:54:19 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
10/03/2009 07:54:19 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 32, VFM error = 2058.
10/03/2009 07:54:20 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
10/03/2009 07:54:20 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
10/03/2009 07:54:21 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
10/03/2009 07:54:21 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
10/03/2009 07:54:21 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
10/03/2009 07:54:22 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
10/03/2009 07:54:22 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
10/03/2009 07:54:23 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
10/03/2009 07:54:23 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
10/03/2009 07:54:23 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
10/03/2009 07:54:24 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
10/03/2009 07:54:25 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
10/03/2009 07:54:25 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 16834, VFM error = 2058.
10/03/2009 07:54:26 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 54671, VFM error = 2058.
10/03/2009 07:54:26 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
10/03/2009 07:54:26 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 71, VFM error = 2058.
VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 75881, VFM error = 2058.
10/03/2009 07:54:59 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - Unable to read metadata for index: 81147, VFM error = 2058.
10/03/2009 07:54:59 - Error bpbrm(pid=7248) from client VCB_GUEST_MACHINE: ERR - failure reading raw device: \\.\C: (WIN32 999: Error performing inpage operation. )
10/03/2009 08:00:00 - Error bpbrm(pid=7248) could not send server status message       
10/03/2009 08:00:01 - end writing; write time: 00:06:59
file read failed(13)

[marcus_lebisk]

Just doing an addendun, no DYNAMIC VOLUME on it.

Thanks

[CRZ]

Are you also at 6.5.1?

If so, how much convincing would it take to get you to try 6.5.3 or 6.5.3.1?

[schmaustech]

What are you seeing in the bpfis and bpbkar logs on the proxy server?  If you do not have logging directories created, create them:

C:\Program Files\VERITAS\Netbackup\logs\bpbkar

C:\Program Files\VERITAS\Netbackup\logs\bpfis

Have you manually tried to run a VCBMounter.exe command for that host from the proxy server?

Regards,

Benjamin Schmaus

[Nickyxx9]

some of our VMs were failing with the repeated error "Unable to read next index. VFM error = 6".

The bpbkar logs on the proxy server showed errors similar to those mentioned in http://seer.entsupport.symantec.com/docs/311422.htm and the registry fix from that article fixed the problem.

Not exactly the same problem but might be worth a try!

[Pravs]

Nicky,

if you are getting the errors for all indexes. you can try the work around to set the registry  as mentioned in the TechNote. are you at 6.5.1?
let us know, if you are getting any error in Application event logs at the time of backup. It should be on your proxy for VxMS application.
you can drop me PM as well, with the information like.
your NBU version, Error in event viewer, and bpbkar erros Also what type of vcb backup you are doing (type 1, 2  or 3)
Hope it helps.

[Srikanth_Gubbal]

i suggest you to upgrade to 6.5.3, since almost 100% problems with VCB and Netbackup were addressed in 6.5.3, i have using it successfully; as i faced similar issues in 6.5.1

[Pravs]

Yes Sri, you are right. We fixed a number of VCB isues in  NBU 6.5.3 including the one mentioned above. but this not the end of the world. Watch out for 6.5.4 FA/GA and you will be surprised to see more and more features pouring in. Happy Virtulized world :)

[Nickyxx9]

Hi Pravs -

We're already at version 6.5.3, and we're doing type 3 backups.  The problem was with all indexes and was, as you say, fixed by the registry change in this technote:

http://seer.entsupport.symantec.com/docs/311422.htm

So our main issue is already resolved.

We do still occasionally see VM backups reporting partial success because they have failed on a few indexes - this is intermittent so not a huge problem.

Thanks,

Nicky

[Pravs]

Hi Nicky,

I have posted couple of articles and blogs related to Flashbackup performnce and you may consider to read them. good that your major issue is solved.

If you are seeing error just for few random indexes, then chances are high that your drive is highly fragmented and you have not run chkdsk from a long on VM.

though articles talk about FlashBackup only but it's fairly applicable to VCB type 0 and Type 3 backups where same mechanism is used for Mapping. though Type 2 works slightly different.

you migt be intersted in this article as well. 

https://www-secure.symantec.com/connect/articles/what-so-special-about-directread-registry-entry-flashbackup

I am thinking to write couple of more articles related to FBU mapping and such failures of indexes. My Bad. I hope, I will get time soon to complete them.