Highlighted

BE 2014 SP2 Hyper-V Backup Failed to Mount one or more virtual disk images RDX drive

Hi,

We are having an issue at a number of our customers sites that backup Hyper-V VM's using Backup Exec 2014 SP1 & SP2 to RDX Cartridges. The backups complete with exceptions that the virtual disk images cannot be mounted. This stops the ability of being able to perform a GRT restore of individual items from the backup. The odd thing is that the virtual disk images can be manully mounted by windows directly from the RDX cartridge.

I have attached a debug log from one of the machines for reference, and have also found the following post on the Symantec forum:

http://www.symantec.com/business/support/index?page=content&id=TECH211717

http://www.symantec.com/business/support/index?page=content&id=TECH223818

The problem in the log looks to be as follows:

[8664] 2014-12-20T04:28:05.666 [engidrapi]          - CDiskLayoutImplementation::DisableAutoMount: Disabling AUTOMOUNT
[8664] 2014-12-20T04:28:05.666 [mounter]            - MSVSVirtualSystem::MountDisks() - Enter
[8664] 2014-12-20T04:28:05.667 [mounter]            - MSVSVirtualSystem::MountDisks: ***** This is Windows-8 or later, forcing AutoMount to be on *****
[8664] 2014-12-20T04:28:05.667 [engidrapi]          - CDiskLayoutImplementation::SetAutoMount: Enabling AUTOMOUNT
[8664] 2014-12-20T04:28:05.668 [mounter]            - MSVSVirtualSystem::MountDisks: checking '\\.\PHYSICALDRIVE0'
[8664] 2014-12-20T04:28:05.730 [mounter]            - MSVSVirtualSystem::MountDisks: MBR disk signature 0x00000000
[8664] 2014-12-20T04:28:05.731 [mounter]            - MSVSVirtualSystem::MountDisks: checking '\\.\PHYSICALDRIVE1'
[8664] 2014-12-20T04:28:05.749 [mounter]            - MSVSVirtualSystem::MountDisks: MBR disk signature 0x6f68ba71
[8664] 2014-12-20T04:28:05.749 [mounter]            - MSVSVirtualSystem::MountDisks: checking '\\.\PHYSICALDRIVE2'
[8664] 2014-12-20T04:28:05.755 [mounter]            - MSVSVirtualSystem::MountDisks: MBR disk signature 0x90c0919b
[8664] 2014-12-20T04:28:05.773                      - bevs::IVhdFileUtil::load_VHD_structures: the VHD footer failed checksum verification
[8664] 2014-12-20T04:28:05.775 [mounter]            - MSVSVirtualSystem::MountDisks: file "M:\IMG000076\Virtual Files\E\Virtual Hard Disks\GCN-SBS.vhdx" Signature: 0X90C0919B Partition Count: 0 Dynamic: False GPT-Dynamic: False Storage Pool: False
[8664] 2014-12-20T04:28:05.775                      - About to mount M:\IMG000076\Virtual Files\E\Virtual Hard Disks\GCN-SBS.vhdx
[8664] 2014-12-20T04:28:05.785                      - Failed to OpenVirtualDisk rc=1392 0x570
[8664] 2014-12-20T04:28:05.785                      - Retry attempt 1 for mount of M:\IMG000076\Virtual Files\E\Virtual Hard Disks\GCN-SBS.vhdx
[8664] 2014-12-20T04:28:05.785                      - About to mount M:\IMG000076\Virtual Files\E\Virtual Hard Disks\GCN-SBS.vhdx
[8664] 2014-12-20T04:28:05.794                      - Failed to OpenVirtualDisk rc=1392 0x570
[8664] 2014-12-20T04:28:05.794                      - Retry attempt 2 for mount of M:\IMG000076\Virtual Files\E\Virtual Hard Disks\GCN-SBS.vhdx
[8664] 2014-12-20T04:28:05.795                      - About to mount M:\IMG000076\Virtual Files\E\Virtual Hard Disks\GCN-SBS.vhdx
[8664] 2014-12-20T04:28:05.804                      - Failed to OpenVirtualDisk rc=1392 0x570
[8664] 2014-12-20T04:28:05.804                      - Retry attempt 3 for mount of M:\IMG000076\Virtual Files\E\Virtual Hard Disks\GCN-SBS.vhdx
[8664] 2014-12-20T04:28:05.804                      - About to mount M:\IMG000076\Virtual Files\E\Virtual Hard Disks\GCN-SBS.vhdx
[8664] 2014-12-20T04:28:05.813                      - Failed to OpenVirtualDisk rc=1392 0x570
[8664] 2014-12-20T04:28:05.814                      - Retry attempt 4 for mount of M:\IMG000076\Virtual Files\E\Virtual Hard Disks\GCN-SBS.vhdx
[8664] 2014-12-20T04:28:05.814                      - About to mount M:\IMG000076\Virtual Files\E\Virtual Hard Disks\GCN-SBS.vhdx
[8664] 2014-12-20T04:28:05.823                      - Failed to OpenVirtualDisk rc=1392 0x570
[8664] 2014-12-20T04:28:05.823                      - Retry attempt 5 for mount of M:\IMG000076\Virtual Files\E\Virtual Hard Disks\GCN-SBS.vhdx
[8664] 2014-12-20T04:28:05.823                      - About to mount M:\IMG000076\Virtual Files\E\Virtual Hard Disks\GCN-SBS.vhdx
[8664] 2014-12-20T04:28:05.833                      - Failed to OpenVirtualDisk rc=1392 0x570
[8664] 2014-12-20T04:28:05.833                      - Retry attempt 6 for mount of M:\IMG000076\Virtual Files\E\Virtual Hard Disks\GCN-SBS.vhdx
[8664] 2014-12-20T04:28:05.833                      - About to mount M:\IMG000076\Virtual Files\E\Virtual Hard Disks\GCN-SBS.vhdx
[8664] 2014-12-20T04:28:05.843                      - Failed to OpenVirtualDisk rc=1392 0x570
[8664] 2014-12-20T04:28:05.843                      - Retry attempt 7 for mount of M:\IMG000076\Virtual Files\E\Virtual Hard Disks\GCN-SBS.vhdx
[8664] 2014-12-20T04:28:05.843                      - About to mount M:\IMG000076\Virtual Files\E\Virtual Hard Disks\GCN-SBS.vhdx
[8664] 2014-12-20T04:28:05.852                      - Failed to OpenVirtualDisk rc=1392 0x570
[8664] 2014-12-20T04:28:05.852                      - Retry attempt 8 for mount of M:\IMG000076\Virtual Files\E\Virtual Hard Disks\GCN-SBS.vhdx
[8664] 2014-12-20T04:28:05.853                      - About to mount M:\IMG000076\Virtual Files\E\Virtual Hard Disks\GCN-SBS.vhdx
[8664] 2014-12-20T04:28:05.862                      - Failed to OpenVirtualDisk rc=1392 0x570
[8664] 2014-12-20T04:28:05.862                      - Retry attempt 9 for mount of M:\IMG000076\Virtual Files\E\Virtual Hard Disks\GCN-SBS.vhdx
[8664] 2014-12-20T04:28:05.862                      - About to mount M:\IMG000076\Virtual Files\E\Virtual Hard Disks\GCN-SBS.vhdx
[8664] 2014-12-20T04:28:05.871                      - Failed to OpenVirtualDisk rc=1392 0x570
[8664] 2014-12-20T04:28:05.871 [mounter]            - MSVSVirtualSystem::MountDisk() - mount of "M:\IMG000076\Virtual Files\E\Virtual Hard Disks\GCN-SBS.vhdx" failed with error code: 1392
[8664] 2014-12-20T04:28:05.872 [mounter]            - 	Total Virtual Disks Mounted: 0
[8664] 2014-12-20T04:28:05.872 [mounter]            - MSVSVirtualSystem::MountDisks() - Exit
[8664] 2014-12-20T04:28:05.873 [engidrapi]          - CDiskLayoutImplementation::SetAutoMount: Enabling AUTOMOUNT
[8664] 2014-12-20T04:28:05.873 [mounter]            - VDiskMounter::MountAllDisks:failed VirtualSystem::Initialize()
[8664] 2014-12-20T04:28:05.873 [mounter]            - Error mounting disks (0xe0009741)

 

However as stated above, we can manually mount the vhdx file directly from the RDX cartridge and copy any files needed to be restored this way. We are not however able to restore individual mail items from and exchange store, without starting up a seperate virtual machine with the VHD attached.

Does anyone else have the same issue? Any idea's on how to resolve this?

So far we have 3 customers with the same issue, all using BE 2014 backing up Hyper-V VM's on Server 2012 R2.

Thanks

Alex

 

 

11 Replies

Is the vhdx greater than 2TB

Is the vhdx greater than 2TB (Max Capacity not current use)?

AND/OR

Is the logical sector size inside the vhdx 4096?

The VHDX being backed up is

The VHDX being backed up is 420GB. (physical size)

The Volume within the VHDx is 420GB

The volume's sector size within the VM is as follows:

Bytes Per Sector  :               512
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024

 

The RDX volume's sector information is as follows:

Bytes Per Sector  :               512
Bytes Per Physical Sector :       512
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024

 

 

if you have enough space to

if you have enough space to try the backup to a fixed local disk instead of the RDX does that work?

Also are these full backups

Also are these full backups or incrementals (GRT mounts will not work if Incrementals are being done to different cartridges from the previous backups in the chain)

I will try the backup to a

I will try the backup to a local disk and see if that goes through ok.

They are full backups sorry, should have included that in the first post.

Hi, I've tried a backup to

Hi, I've tried a backup to local disk and this has completed succsfully and I can see the GRT information for the virtual machine in the restore wizzard.

So the problem seems to be with backups going to the RDX disk?

I think it might be a good

I think it might be a good idea if you log a formal support case for us to look into this more thoroughly.

 

I do have one extra (long shot) thought - are the sector sizes of the volumes containing the vhdx files on the hyper-V server the same as the sector size on the RDX cartridges? (and in fact the same as the test local disk/volume you used to prove that there was some problem with RDX.)

I will log a support call

I will log a support call thanks.

I checked and from what FSutil is telling me the bytes per sector on all of the drives are exactly the same. However the Bytes per Physical Sector" on the RDX drive is 4096 (compared to 512 on the c: and e: volumes)

The below is the output of fsutil,

c: doesn't have any data on it, but i didn't know if the mount location made any difference when it tries to mount the vhdx

d: is the RDX drives volume

e: is the volume with the running vhdx on it

C:\Users\Administrator>fsutil fsinfo ntfsinfo c:
NTFS Volume Serial Number :       0x407f3de20af3e7fb
NTFS Version   :                  3.1
LFS Version    :                  2.0
Number Sectors :                  0x0000000022e07d73
Total Clusters :                  0x00000000045c0fae
Free Clusters  :                  0x000000000393f975
Total Reserved :                  0x0000000000001c40
Bytes Per Sector  :               512
Bytes Per Physical Sector :       512
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length :           0x000000000c280000
Mft Start Lcn  :                  0x0000000000000004
Mft2 Start Lcn :                  0x00000000022e07d7
Mft Zone Start :                  0x000000000000c280
Mft Zone End   :                  0x000000000000cb80
Resource Manager Identifier :     641C824F-6663-11E4-939D-806E6F6E6963

C:\Users\Administrator>fsutil fsinfo ntfsinfo d:
NTFS Volume Serial Number :       0x6038cd6e38cd43ae
NTFS Version   :                  3.1
LFS Version    :                  1.1
Number Sectors :                  0x0000000074703fff
Total Clusters :                  0x000000000e8e07ff
Free Clusters  :                  0x0000000004c8174f
Total Reserved :                  0x0000000000000000
Bytes Per Sector  :               512
Bytes Per Physical Sector :       4096
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length :           0x0000000000040000
Mft Start Lcn  :                  0x00000000000c0000
Mft2 Start Lcn :                  0x00000000074703ff
Mft Zone Start :                  0x00000000000c0040
Mft Zone End   :                  0x00000000000cc820
Resource Manager Identifier :     5BE6E7B3-762F-11E4-80C1-9C8E9963E4E1

C:\Users\Administrator>fsutil fsinfo ntfsinfo e:
NTFS Volume Serial Number :       0xdc420a85420a649c
NTFS Version   :                  3.1
LFS Version    :                  2.0
Number Sectors :                  0x000000008bb83fff
Total Clusters :                  0x00000000117707ff
Free Clusters  :                  0x000000000821f5fd
Total Reserved :                  0x0000000000002680
Bytes Per Sector  :               512
Bytes Per Physical Sector :       512
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length :           0x00000000002c0000
Mft Start Lcn  :                  0x00000000000c0000
Mft2 Start Lcn :                  0x0000000000000002
Mft Zone Start :                  0x00000000000c02c0
Mft Zone End   :                  0x00000000000cc840
Resource Manager Identifier :     586BBF86-6669-11E4-80B8-9C8E9963E4E1

i'm going to have a look and see if theres a way to change the bytes per physical sector on the RDX volume. I suspect not, but its the only thing I can see thats different.

Cheers

Alex

I couldn't find a way of

I couldn't find a way of modifiying the bytes per physical sector on the RDX volume. I have logged a new support incident reference number:

08137855

If its at all possible can a reference to this post be added to the support ticket.
 

Thanks

Alex

 

OK - just be aware it needs

OK - just be aware it needs to be troubelshot to confirm what the issue is, and whilst it might be the sector size differences between where the vhd fiels are stored and the RDX drive it also might be something else.

Sorry not got back to this

Sorry not got back to this sooner.

After a number of months of support calls, we finially have a workaround for this issue.

Support have advised that this issue is because the RDX cartridges are advanced 4K sector sized disks. This can be seen from using: "fsutil fsinfo ntfsinfo <RDX drive letter>"

The bytes Per Physical Sector is 4096. Support have advised that in order for the virtual disk mounter to be able to mount the vhd(x) that the sector size must match that of the original volume in which the vhd(x) was originally located. In our instance all of the other drives in the server have 512 bytes per physical sector.

The workaround we have been given is to disable the auto discovery for the virtual disk mounter and to manually set the mount point for the VHD(x) so it is on a drive with the same bytes per physical sector. The registry keys we have set are as follows:

DWORD - HKEY_LOCAL_MACHINE\Software\Symantec\Backup Exec For Windows\Backup Exec\Engine\Virtual Disk Mounter\Auto Discovery = 0

STRING - HKEY_LOCAL_MACHINE\Software\Symantec\Backup Exec For Windows\Backup Exec\Engine\Virtual Disk Mounter\Delta Space = D:\

STRING - HKEY_LOCAL_MACHINE\Software\Symantec\Backup Exec For Windows\Backup Exec\Engine\Virtual Disk Mounter\Mount Point Root = D:\

STRING - HKEY_LOCAL_MACHINE\Software\Symantec\Backup Exec For Windows\Backup Exec\Engine\Virtual Disk Mounter\Virt Space = D:\

D:\ is the volume where the virtual machines and virtual hard disks are located.

After adding the registry keys we needed to re-start the backup exec services. Once done the GRT catalogue was succesful and we can perform granular restores from these RDX cartridges.

Further to this, we have discovered that not all manufacturers of RDX cartridge use Advanced 4K disks, so in some instances GRT backups with RDX drives will work without problem.

We have implemented this at a couple of sites and it appears to work.

Cheers

Alex