issue with BE2010R3 and lock files not being release after successful incremental backup

[halforl]

Wondering whether anyone has solved this before:

• BE2010R3 with all latest hotifixes.
• After a successful incremental VM GRT backup, the verify fails immediately with "unable to open a disk of the virtual machine", and "data for VMVCB::\\.....*.vmdk is corrupt.".
• Subsequent independent verify and duplication jobs also fail.
• Seems to only affect incrementals, but happening on multiple servers.

After turning on debug logs it appears the file "*.vmdk.lck" is not being released by the BE2010R3 server after the initial backup:

BEREMOTE: [08/05/13 11:06:24] [3152]     [fsys\shared]        - VDDK-Log: AIOMGR: AIOMgr_OpenWithRetry: Descriptor file 'U:\B2D-GRT-U\IMG000929\vc01.vmdk' locked (try 4)
BEREMOTE: [08/05/13 11:06:27] [3152]     [fsys\shared]        - VDDK-Log: FILE: WaitForPossession timeout on 'U:\B2D-GRT-U\IMG000929\vc01.vmdk.lck\M39856.lck' due to a local process (1880)
BEREMOTE: [08/05/13 11:06:27] [3152]     [fsys\shared]        - VDDK-Warn: FILE: FileIO_Lock on 'U:\B2D-GRT-U\IMG000929\vc01.vmdk' failed: Lock timed out
BEREMOTE: [08/05/13 11:06:27] [3152]     [fsys\shared]        - VDDK-Log: DISKLIB-DSCPTR: DescriptorDetermineType: failed to open 'U:\B2D-GRT-U\IMG000929\vc01.vmdk': Failed to lock the file (400000003)
BEREMOTE: [08/05/13 11:06:27] [3152]     [fsys\shared]        - VDDK-Log: DISKLIB-LINK  : "U:\B2D-GRT-U\IMG000929\vc01.vmdk" : failed to open (Failed to lock the file).
BEREMOTE: [08/05/13 11:06:27] [3152]     [fsys\shared]        - VDDK-Log: DISKLIB-CHAIN : "U:\B2D-GRT-U\IMG000929\vc01.vmdk" : failed to open (Failed to lock the file).
BEREMOTE: [08/05/13 11:06:27] [3152]     [fsys\shared]        - VDDK-Log: DISKLIB-LIB   : Failed to open 'U:\B2D-GRT-U\IMG000929\vc01.vmdk' with flags 0x1a Failed to lock the file (16392).
BEREMOTE: [08/05/13 11:06:27] [5080]     [fsys\vmvcb\pdi]     - Could not open the local disk U:\B2D-GRT-U\IMG000929\vc01.vmdk. Error Text: 'The file is already in use' Error: '70403103916047'
BEREMOTE: [08/05/13 11:06:27] [5080]     [ndmp\loops]         - Unswitched return from FS_ReadObj() = 0xe0009585

Local process 1880 is "beremote" on the BE2010R3 server. Viewing the IMG directory on the B2D folder above, that lock file does exist and was created with a timestamp at the START of the original verification stage of the backup job - never been released.

Looks very similar to a BE2012 bug that has been resolved:  http://www.symantec.com/business/support/index?page=content&id=TECH166644

But no hotfix for BE2010R3, only BE2012.

A manual restart of the remote agent per the hotfix suggestion of the client did not resolve issue, but not surprised, as per above the local media server is holding the image open.

Any help or workarounds appreciated.

[VJware]

Have you tried restarting the Remote Agent service on the media server ? The idea is that restarting this service should clear the .lck file from the IMG folder post which you should run a backup again.

If you have already tried this & the issue still persists, please log a formal support case.

[Sush---]

Hello Halforl,

   You may use the workaround in the technote that you provided and consider that Remote agent as Remote agent on the Media server itself

Workaround:

Restart the Backup Exec Remote Agent service and confirm if the ".LCK" file is cleared from the IMG folder. If the file is cleared then follow the steps below.

1. Add a post command in the Incremental job to restart the Backup Exec Remote Agent service once the job completes. Refer to article www.symantec.com/docs/HOWTO22999 for details on pre and post commands.
2. Modify the Duplicate job to run on schedule instead of linking it to run immediately after the incremental backup.

Also in addition to this you may refer the following technote

http://www.symantec.com/docs/TECH200110 : Verification of a virtual machine backed up during a VMware backup fails with "Unable to open a disk of the virtual machine."

I have refered the above technote because I can see Engineering team is working on the fix for that issue. So getting more information from your environment when you open a new case will definately help so that we can provide the fix.

Thanks,

-Sush...