Forum Discussion

SteveLaw's avatar
SteveLaw
Level 5
9 years ago

Multipathing software required for ESX datastores SAN volumes?

Hi,

I have a NBU VMWare proxy server, NB 7.5.0.6 on Windows 2008, that has SAN visibility of the ESX datastores and has been happily backing up VMs to Data Domain for over a year. Recently an additional ESX farm was built and a SAN fabric change was made to grant the same NBU VMWare proxy visibility of the datastores hosted by these new ESX hosts. However I can't backup these new VMs  - I get the error "Error opening the snapshot disks using given transport mode: Status 23". 

I checked Disk Manager on the VM proxy media server and there's a whole load of disks present that show as offline with the message "The disk is offline because it has a redundant path with another device". I raised a call with our SAN team but they passed it back saying it was a problem with the multipathing on the Windows server. I thought this was odd as we've been using that server to backup VMs from the original ESX farm without a problem, so why would multipathing be an issue for these additional luns? 

I checked the server and Microsoft MPIO is installed but it doesn't appear to be managing any luns. On the mpio command line the commands mpclaim.exe -s -d (which is supposed to show the disks being managed by mpio) and mpclaim.exe -s (Queries connected enterprise storage and displays as vendor-product id strings) show mpio isn't doing anything:

mpio_0.JPG

So my question is (finally!): do you need multipathing drivers like Powerpath or MPIO for a Win2008 R2 Netbackup VMWare Proxy server to be able to read and backup from the ESX datastores? I don't appear to have any multipathing but it has been working on one ESX farm for a long time.

Thanks 
 

 

 

 

 

 

 

  • @Steve can tell you that I have used vmware backup with multipathing in Netbackup 7.5, without issues other than the SAN restore thing which had to do with the api rather than the number paths.

    Think you best bet of resolving this is together with your SAN, Windows & VM administrator

    Looking at the diskpart output the disks 35,59,83,107 could be the same LUN/datastore again assuming 4 paths through the SAN fabrics, and that disk 11 is from the old VM farm.

    Just checked on a server, the path to the "low" level disk drives is:

    Computer Management -> System Tools -> Device Manager -> Disk Drives

     

  • From these posts:

    https://www.veritas.com/community/forums/linux-vmware-backup-host-multi-pathing-support

    https://www.veritas.com/community/forums/suse-linux-112-vmware-backup

    ...it is clear that the the document "DOCUMENTATION: Support for NetBackup 7.x in virtual environments", here:

    https://www.veritas.com/support/en_US/article.TECH127089

    ...used to list that there were some restrictions around multi-pathing for NetBackup with some Linux OS.  However this document no longer carries any reference to this, and so would appear to imply that these restrictions are no longer valid (but probably one can only assume for the most recent versions of NetBackup - because it doesn't say when the restrictions were lifted).  I'm aware that similar restrictions used to once also exist for Windows based VMware backup hosts.  But when was that restriction lifted?  And for which version of NetBackup?

    .

    And yet at the time of the postings, from the version of the doc at the time, it was clear that multi-pathing was supported for Windows based VMware backup hosts - but which versions?

    .

    Do you see how your question is very difficult to answer because you are on v7.5.0.6 and so it's not easy to determine whether your version matrix can support multi-path.  Maybe it can, and so all you need to do is install a DSM and/or multi-path driver and you'll be ok.

    But I don't know this for sure.  Maybe it does, maybe it doesn't for your old version.

    Hence I was suggesting to dig back through the old VDDK release notes to see when the support was introduced by VMware.

    Remember, this isn't a NetBackup question per-se, it's a VMware "interop" question.

26 Replies

  • @Steve can tell you that I have used vmware backup with multipathing in Netbackup 7.5, without issues other than the SAN restore thing which had to do with the api rather than the number paths.

    Think you best bet of resolving this is together with your SAN, Windows & VM administrator

    Looking at the diskpart output the disks 35,59,83,107 could be the same LUN/datastore again assuming 4 paths through the SAN fabrics, and that disk 11 is from the old VM farm.

    Just checked on a server, the path to the "low" level disk drives is:

    Computer Management -> System Tools -> Device Manager -> Disk Drives

     

  • From these posts:

    https://www.veritas.com/community/forums/linux-vmware-backup-host-multi-pathing-support

    https://www.veritas.com/community/forums/suse-linux-112-vmware-backup

    ...it is clear that the the document "DOCUMENTATION: Support for NetBackup 7.x in virtual environments", here:

    https://www.veritas.com/support/en_US/article.TECH127089

    ...used to list that there were some restrictions around multi-pathing for NetBackup with some Linux OS.  However this document no longer carries any reference to this, and so would appear to imply that these restrictions are no longer valid (but probably one can only assume for the most recent versions of NetBackup - because it doesn't say when the restrictions were lifted).  I'm aware that similar restrictions used to once also exist for Windows based VMware backup hosts.  But when was that restriction lifted?  And for which version of NetBackup?

    .

    And yet at the time of the postings, from the version of the doc at the time, it was clear that multi-pathing was supported for Windows based VMware backup hosts - but which versions?

    .

    Do you see how your question is very difficult to answer because you are on v7.5.0.6 and so it's not easy to determine whether your version matrix can support multi-path.  Maybe it can, and so all you need to do is install a DSM and/or multi-path driver and you'll be ok.

    But I don't know this for sure.  Maybe it does, maybe it doesn't for your old version.

    Hence I was suggesting to dig back through the old VDDK release notes to see when the support was introduced by VMware.

    Remember, this isn't a NetBackup question per-se, it's a VMware "interop" question.

  • If anyone out there knows when NetBackup began support for multi-path on Windows based VMware backup hosts (i.e. NetBackup Media Server, and/or NetBackup Enterprise Client) via SAN FC to VMFS datastores - please do say!    It would save Steve a lot of time.  Thanks :)

  • Steve, when you say this:

    "...and I'm seeing many more offline than online disks."

    ...we need to remind ourselves of exactly what it is you are seeing.  I think you are seeing "multiple paths to LUNs", not "disks".   There's a big difference.  Remember Windows LDM (Logical Disk Manager) is a misnomer, or, a hang-up from days gone by when disks really were actual individual physical spinning disks, and not LUNs abstracted from storage arrays or RAID HBA cards.  It's such a basic un-intelligent GUI, but then it has to be.  MS can't build in default functionality for every storage array vendor.  Hence the storage array vendors issue their own DSM (Device Support Modules) to help Windows understand what it sees, and to help you manage your LUNs.

    .

    In "Device Manager" under "Disks" you see actual LUNs.

    Now, without the required DSM and/or multi-path driver, then "diskpart" which is actually the CLI equivalent of these GUIs:

    Windows 2008 R2:  Server Manager > Storage > Disk Management

    Windows 2012 R2:  Server Manager > Tools > Computer Manager > Storage > Disk Management

    ...then without a DSM and/or multi-path driver, then Windows is blind, it just sees LUNs what Windows unfortunately refers to as "Disks" because Windows doesn't know any better, because Windows has not been educated in to understanding how these LUNs all tie together.  i.e. Without a DSM then "Device Manager > Disks" AND ALSO "Server Manager > Storage > Disk Management" will both see the same number of objects that each of these views both (unfortunately) refer to as "disks".

    .

    In a properly configured server the number of so called "disk" objects seen in "Device Manager > Disks" would very likely NOT be the same number of "disk" objects seen by "Server Manager -> Storage > Disk Management".   i.e. in a properly configured server, then "Device Manager > Disks" are the LUNs, and "Server Manager > Storage > Disk Manager" are the actual "volumes".

    .

    IMO, two areas in the Windows GUIs should be renamed:

    "Device Manager > Disks" should be relabelled to read "Device Manager > Disk LUNs"

    ...and:

    "Server Manager > Storage > Disk Manager" should be relabelled to read "Server Manager > Storage > Disk Volume Manager".

    .

    What I'm trying to say is that without a storage array vendor DSM and/or multi-path driver then Windows cannot distinguish between LUNs and Volumes - Windows cannot tell the difference because Windows does not understand the vendor's "LUN to volume" translation constructs.   And so, Microsoft in their wisdom, just refer to all of these different types of objects as "disks", which is not very helpful to anyone.

     

    .

    If you follow your storage array vendor documentation and install the required modules/software/management/DSM/drivers etc, then you should see additional "snap-ins" in the Windows MMC that is "server manager" or "computer manager" or whatever Microsoft want to call it at the time - which will then allow you to manage the multiple paths to the "LUNs" that sit underneath the "Volume".

     

  • Hi,

    I raised a call with Netbackup and they have found that the cause of the backup failures is NOT the multipathing issue and the offline disks. The problem is incompatibility of our Netbackup 7.5.0.6 with the new ESX farm on which the vCenter server is v6.0. The error in the vxms debug logs shows this: 

    ....backline colleague confirmed that because of incompatibly, the handshake will not complete.
     
    VXMS log snippit:
     
    10:04:12.0109 : g_vixInterfaceLogger:bvix.cpp:1835 <INFO> : diskLibPlugin: 2016-02-16T10:04:12.109Z [01504 error 'Default'] SSLStreamImpl::DoClientHandshake (0000000007c37930) SSL_connect failed with Unexpected EOF
     
    10:04:12.0109 : g_vixInterfaceLogger:bvix.cpp:1835 <INFO> : diskLibPlugin: 2016-02-16T10:04:12.109Z [01504 trivia 'Default'] Setting error in state 1 : class Vmacore::Ssl::SSLException(SSL Exception: Unexpected EOF)

     

    Therefore I need to upgrade Netbackup master and media servers to v7.6.1 or more. On the multipathing front I asked "It seems that originally Windows servers running as Netbackup VM proxies with San access to ESX datastores DID NOT support multipathing, but more recently they do. Can you tell me which versions, and if multipathing is supported for a 7.5.0.6 vm proxy server?". They replied:

    "NetBackup 7.5.0.6 or NetBackup 7.6.1.2 should be fine.  It is the OS that handle multi-pathing.  So NetBackup rely on the OS for this function."

    So it seems that I should be able to install multipathing software without an issue on the VM Proxy server that can see the ESX datastores. (Although it will be nerve-wracking - I'm still worried that somehow making any changes on these servers will corrupt the datastores and kill a hundred vms...!) 

    Thanks for all your help and suggestions! I'll split the resolution between sdo and Michael.

  • Well, that's good news that you got to the bottom of the issue.

    And, also good to have it confirmed that you are good to install the array DSM and/or MP software.  I'd preferably do that first before upgrading NetBackup, and definitely not both at the same time.

    Just out of interest what is the storage array model and firmware version?  Someone here may venture to assist you in selecting the correct DSM/MP software, or maybe open a pre-emptive case with the array vendor?