Forum Discussion

SteveLaw's avatar
SteveLaw
Level 5
9 years ago

Multipathing software required for ESX datastores SAN volumes?

Hi,

I have a NBU VMWare proxy server, NB 7.5.0.6 on Windows 2008, that has SAN visibility of the ESX datastores and has been happily backing up VMs to Data Domain for over a year. Recently an additional ESX farm was built and a SAN fabric change was made to grant the same NBU VMWare proxy visibility of the datastores hosted by these new ESX hosts. However I can't backup these new VMs  - I get the error "Error opening the snapshot disks using given transport mode: Status 23". 

I checked Disk Manager on the VM proxy media server and there's a whole load of disks present that show as offline with the message "The disk is offline because it has a redundant path with another device". I raised a call with our SAN team but they passed it back saying it was a problem with the multipathing on the Windows server. I thought this was odd as we've been using that server to backup VMs from the original ESX farm without a problem, so why would multipathing be an issue for these additional luns? 

I checked the server and Microsoft MPIO is installed but it doesn't appear to be managing any luns. On the mpio command line the commands mpclaim.exe -s -d (which is supposed to show the disks being managed by mpio) and mpclaim.exe -s (Queries connected enterprise storage and displays as vendor-product id strings) show mpio isn't doing anything:

mpio_0.JPG

So my question is (finally!): do you need multipathing drivers like Powerpath or MPIO for a Win2008 R2 Netbackup VMWare Proxy server to be able to read and backup from the ESX datastores? I don't appear to have any multipathing but it has been working on one ESX farm for a long time.

Thanks 
 

 

 

 

 

 

 

7.5
Backup and Recovery
NetBackup
Windows Server (2003-2008)
  • Michael_G_Ander's avatar
    Michael_G_Ander
    9 years ago

    @Steve can tell you that I have used vmware backup with multipathing in Netbackup 7.5, without issues other than the SAN restore thing which had to do with the api rather than the number paths.

    Think you best bet of resolving this is together with your SAN, Windows & VM administrator

    Looking at the diskpart output the disks 35,59,83,107 could be the same LUN/datastore again assuming 4 paths through the SAN fabrics, and that disk 11 is from the old VM farm.

    Just checked on a server, the path to the "low" level disk drives is:

    Computer Management -> System Tools -> Device Manager -> Disk Drives

     

  • sdo's avatar
    sdo
    9 years ago

    From these posts:

    https://www.veritas.com/community/forums/linux-vmware-backup-host-multi-pathing-support

    https://www.veritas.com/community/forums/suse-linux-112-vmware-backup

    ...it is clear that the the document "DOCUMENTATION: Support for NetBackup 7.x in virtual environments", here:

    https://www.veritas.com/support/en_US/article.TECH127089

    ...used to list that there were some restrictions around multi-pathing for NetBackup with some Linux OS.  However this document no longer carries any reference to this, and so would appear to imply that these restrictions are no longer valid (but probably one can only assume for the most recent versions of NetBackup - because it doesn't say when the restrictions were lifted).  I'm aware that similar restrictions used to once also exist for Windows based VMware backup hosts.  But when was that restriction lifted?  And for which version of NetBackup?

    .

    And yet at the time of the postings, from the version of the doc at the time, it was clear that multi-pathing was supported for Windows based VMware backup hosts - but which versions?

    .

    Do you see how your question is very difficult to answer because you are on v7.5.0.6 and so it's not easy to determine whether your version matrix can support multi-path.  Maybe it can, and so all you need to do is install a DSM and/or multi-path driver and you'll be ok.

    But I don't know this for sure.  Maybe it does, maybe it doesn't for your old version.

    Hence I was suggesting to dig back through the old VDDK release notes to see when the support was introduced by VMware.

    Remember, this isn't a NetBackup question per-se, it's a VMware "interop" question.

26 Replies

Sort By
Show More
  • sdo's avatar
    sdo
    Moderator
    9 years ago

    Ah - sorry - some of the old multiple view "disks" are offline and should remain offline in my opinion - but some of the new LUNs are also offline, and one of the LUNs from each of these new disks needs to be onlined - but which one?   Is that the issue?

  • Michael_G_Ander's avatar
    Michael_G_Ander
    Level 6
    9 years ago

    If using multipath, there should proably be no offline drives, but only about 48 in all if I guess right that each VM farm has 24 datastores. The extraneous drives usually disappears when the multipath driver load on the disk device in the Windows system.

  • sdo's avatar
    sdo
    Moderator
    9 years ago

    My argument then is that 24 "volumes" should be online, and 24 should be offline.  Because, without a proper multi-path driver then how will the strict order of SCSI IO sequence be maintained.  Not having a multi-path driver loaded is just like having two separate hosts access the same LUN.  Dangerous if not supported.

    Michael - please shoot me down - but won't dual "online" paths (i.e. 48 paths) to what Windows thinks are 48 different volumes (because multi-path is not installed) - won't this represent a risk?

  • sdo's avatar
    sdo
    Moderator
    9 years ago

    Steve - when it's installed properly then Windows LDM will take the higher ground and see 24 volumes.  And what you should have is a vendor DSM, like an MMC snap-in, installed... which is the multi-path manager so to speak - some vendors blur the abstraction between DSM and driver - and some vendors do not - and so you need the proper vendor DSM so that you can then know whether to use a vendor MP driver or a generic one provided by the OS vendor - and with all this then correctly manage avtive/active or active/passive, and/or ALUA, and set multi-path properties like shortest-path-first, round-robin, least-recently-used etc.   There's a whole bunch of issues you can cause by not having this setup properly, like LUN trespass, and, golly the term slips me now, but NetApp have a word which for LUNs not accessed by active path, and that are accessed via passive path - which causes horrible slowness in NetApp, and all because you don't have the DSM loaded.  And I think VNX can suffer the same horrible slow IO too when passive path is used - basically the controller have to stop, talk to each other, and pass data between themselves - all because you are talking to the wrong path - and when you have 48 paths visible to 24 LUNs then there's nothing easily manageable in Windows to let you decide which of each pair is the correct one to keep online, let alone even identify the 24 pairs easily (unless every LUN is a different size then ot's easy to match up the pairs - but you still won't know which is the correct active path).  What an absolute effort all this is.

    And all this is nonsense if your NetBackup version does not support multi-pathed SAN LUN VMFS datastores - in which case you really should single path present, and so Windows will see 24 LUNs and not 48.

  • Michael_G_Ander's avatar
    Michael_G_Ander
    Level 6
    9 years ago

    Got the 48 online disks from 24 online disks, probably old VM farm datastores and (120-24) offline disk/4 paths (assuming 2 fabrics and dual port controllers with a port connect to each fabric as per SAN best pratice)  = 24 actually luns/datastores from the new VM farm proably.

    Will have to wait for Steve to tell if I have guessed right about the number datastores

  • sdo's avatar
    sdo
    Moderator
    9 years ago

    Steve - just a reminder, that the answer to these questions should help in some way to determine your options and next steps:

    1) Find out exactly which verision of VDDK (for VADP of VMDK) began support for multi-pathing FC SAN based VM style backups.

    2) Find out which version of NetBackup carries this minimum version of VDDK which supports multi-pathing.

  • SteveLaw's avatar
    SteveLaw
    Level 5
    9 years ago

    Hi,

    I have way more than 24 disks offline. Here's the output from diskpart list:

     

    DISKPART> list disk

      Disk ###  Status         Size     Free     Dyn  Gpt
      --------  -------------  -------  -------  ---  ---
      Disk 0    Online          136 GB      0 B
      Disk 1    Online         1024 GB      0 B        *
      Disk 2    Online         1250 GB      0 B        *
      Disk 3    Online         1100 GB      0 B        *
      Disk 4    Online         1073 GB      0 B        *
      Disk 5    Online         1024 GB      0 B        *
      Disk 6    Online         1024 GB      0 B        *
      Disk 7    Online         1024 GB      0 B        *
      Disk 8    Online         1073 GB      0 B        *
      Disk 9    Online         1024 GB      0 B        *
      Disk 10   Online         1024 GB      0 B        *
      Disk 11   Online         4611 GB      0 B        *
      Disk 12   Online         1024 GB      0 B        *
      Disk 13   Online         1024 GB      0 B        *
      Disk 14   Online         2048 GB      0 B        *
      Disk 15   Online         1024 GB      0 B        *
      Disk 16   Online          650 GB      0 B        *
      Disk 17   Online         1536 GB      0 B        *
      Disk 18   Online         1024 GB      0 B        *
      Disk 19   Online         3072 GB      0 B        *
      Disk 20   Online          500 GB  1024 KB
      Disk 21   Online          400 GB  1024 KB
      Disk 22   Online          100 GB  1024 KB
      Disk 23   Online         1024 MB  1984 KB
      Disk 24   Online         1024 MB  1984 KB
      Disk 25   Offline        1024 GB      0 B        *
      Disk 26   Offline        1250 GB      0 B        *
      Disk 27   Offline        1100 GB      0 B        *
      Disk 28   Offline        1073 GB      0 B        *
      Disk 29   Offline        1024 GB      0 B        *
      Disk 30   Offline        1024 GB      0 B        *
      Disk 31   Offline        1024 GB      0 B        *
      Disk 32   Offline        1073 GB      0 B        *
      Disk 33   Offline        1024 GB      0 B        *
      Disk 34   Offline        1024 GB      0 B        *
      Disk 35   Offline        4611 GB      0 B        *
      Disk 36   Offline        1024 GB      0 B        *
      Disk 37   Offline        1024 GB      0 B        *
      Disk 38   Offline        2048 GB      0 B        *
      Disk 39   Offline        1024 GB      0 B        *
      Disk 40   Offline         650 GB      0 B        *
      Disk 41   Offline        1536 GB      0 B        *
      Disk 42   Offline        1024 GB      0 B        *
      Disk 43   Offline        3072 GB      0 B        *
      Disk 44   Offline         500 GB  1024 KB
      Disk 45   Offline         400 GB  1024 KB
      Disk 46   Offline         100 GB  1024 KB
      Disk 47   Offline        1024 MB  1984 KB
      Disk 48   Offline        1024 MB  1984 KB
      Disk 49   Offline        1024 GB      0 B        *
      Disk 50   Offline        1250 GB      0 B        *
      Disk 51   Offline        1100 GB      0 B        *
      Disk 52   Offline        1073 GB      0 B        *
      Disk 53   Offline        1024 GB      0 B        *
      Disk 54   Offline        1024 GB      0 B        *
      Disk 55   Offline        1024 GB      0 B        *
      Disk 56   Offline        1073 GB      0 B        *
      Disk 57   Offline        1024 GB      0 B        *
      Disk 58   Offline        1024 GB      0 B        *
      Disk 59   Offline        4611 GB      0 B        *
      Disk 60   Offline        1024 GB      0 B        *
      Disk 61   Offline        1024 GB      0 B        *
      Disk 62   Offline        2048 GB      0 B        *
      Disk 63   Offline        1024 GB      0 B        *
      Disk 64   Offline         650 GB      0 B        *
      Disk 65   Offline        1536 GB      0 B        *
      Disk 66   Offline        1024 GB      0 B        *
      Disk 67   Offline        3072 GB      0 B        *
      Disk 68   Offline         500 GB  1024 KB
      Disk 69   Offline         400 GB  1024 KB
      Disk 70   Offline         100 GB  1024 KB
      Disk 71   Offline        1024 MB  1984 KB
      Disk 72   Offline        1024 MB  1984 KB
      Disk 73   Offline        1024 GB      0 B        *
      Disk 74   Offline        1250 GB      0 B        *
      Disk 75   Offline        1100 GB      0 B        *
      Disk 76   Offline        1073 GB      0 B        *
      Disk 77   Offline        1024 GB      0 B        *
      Disk 78   Offline        1024 GB      0 B        *
      Disk 79   Offline        1024 GB      0 B        *
      Disk 80   Offline        1073 GB      0 B        *
      Disk 81   Offline        1024 GB      0 B        *
      Disk 82   Offline        1024 GB      0 B        *
      Disk 83   Offline        4611 GB      0 B        *
      Disk 84   Offline        1024 GB      0 B        *
      Disk 85   Offline        1024 GB      0 B        *
      Disk 86   Offline        2048 GB      0 B        *
      Disk 87   Offline        1024 GB      0 B        *
      Disk 88   Offline         650 GB      0 B        *
      Disk 89   Offline        1536 GB      0 B        *
      Disk 90   Offline        1024 GB      0 B        *
      Disk 91   Offline        3072 GB      0 B        *
      Disk 92   Offline         500 GB  1024 KB
      Disk 93   Offline         400 GB  1024 KB
      Disk 94   Offline         100 GB  1024 KB
      Disk 95   Offline        1024 MB  1984 KB
      Disk 96   Offline        1024 MB  1984 KB
      Disk 97   Offline        1024 GB      0 B        *
      Disk 98   Offline        1250 GB      0 B        *
      Disk 99   Offline        1100 GB      0 B        *
      Disk 100  Offline        1073 GB      0 B        *
      Disk 101  Offline        1024 GB      0 B        *
      Disk 102  Offline        1024 GB      0 B        *
      Disk 103  Offline        1024 GB      0 B        *
      Disk 104  Offline        1073 GB      0 B        *
      Disk 105  Offline        1024 GB      0 B        *
      Disk 106  Offline        1024 GB      0 B        *
      Disk 107  Offline        4611 GB      0 B        *
      Disk 108  Offline        1024 GB      0 B        *
      Disk 109  Offline        1024 GB      0 B        *
      Disk 110  Offline        2048 GB      0 B        *
      Disk 111  Offline        1024 GB      0 B        *
      Disk 112  Offline         650 GB      0 B        *
      Disk 113  Offline        1536 GB      0 B        *
      Disk 114  Offline        1024 GB      0 B        *
      Disk 115  Offline        3072 GB      0 B        *
      Disk 116  Offline         500 GB  1024 KB
      Disk 117  Offline         400 GB  1024 KB
      Disk 118  Offline         100 GB  1024 KB
      Disk 119  Offline        1024 MB  1984 KB
      Disk 120  Offline        1024 MB  1984 KB
     

    As I say I've had an original ESX farm presented and working for well over a year - I can't remember if that showed some offline disks too. But now we've had a second ESX farm presented to the Netbackup VM Proxy server and I can't back up the VMs within it and I'm seeing many more offline than online disks.  

  • SteveLaw's avatar
    SteveLaw
    Level 5
    9 years ago

    Hi sdo,

    You wrote:

    1) Find out exactly which verision of VDDK (for VADP of VMDK) began support for multi-pathing FC SAN based VM style backups.

    2) Find out which version of NetBackup carries this minimum version of VDDK which supports multi-pathing.

     

    The NetBackup7.5_AdminGuide_VMware.pdf for NB7.5 mentions VDDK a few times but briefly in a discussion of the helper virtual machine required in earlier ESX versions, and in relation to slow restores. But no other useful info. I don't know how to answer your questions. Is it the case that once you didn't need multi-pathing but in later versions of VDDK / Netbackup you do?   

  • Steve_Law's avatar
    Steve_Law
    Level 4
    9 years ago

    Now I'm really nervous. The VM proxy media server was rebooted after routine windows patching and disk manager is prompting to initialize disks:

    diskmgrprompt_1.JPG

    I'm not going to 'initialize' them for fear this might corrupt the ESX datastores. Interesting though that it should pick out those four disks out of the dozens of offline ones. 

     

     

  • sdo's avatar
    sdo
    Moderator
    9 years ago

    This post contains a list of some older NetBackup versions and VDDK versions, and a list (up to a point) of some of the VDDK release notes.

    https://www.veritas.com/community/forums/limit-size-virtual-machine-while-backup-using-vadp

    .

    You need to identify from the VDDK release notes which of versions began support for multi-pathing of VADP backups.  Then tie that VDDK version back to a NetBackup version, and you'll know whether you version of NetBackup supports multi-pathing or not, and then you'll know what action you might want to take.