Using third party Multipathing with VCS LVM agents

[mikebounds]

Using third party Multipathing with VCS LVM agents is mostly not clear in the documentation.  With SF, you can understand why 3rd party Multipathing needs to be tested to be supported as SF integrates at a low-level with the storage and multipathing effect this, but with VCS, VCS integrates with LVM at a high level, activating and deactivating he volume groups (equivalent to importing and deporting Veritas diskgroups), so it is first unclear why any 3rd party multipathing software should not be supported, except for perhaps O/S multipathing which is tightly integrated with LVM where the command to activate or deactivate the diskgroup MAY be different if multipathing is involved.

For AIX, the HCL specifically mentions VCS LVM agents with third party Multipathing:

The VCS LVM agent supports the EMC PowerPath third-party driver on EMC's Symmetrix 8000 and DMX series arrays.
The VCS LVM agent supports the HITACHI HDLM third-party driver on Hitachi USP/NSC/USPV/USPVM, 9900V series arrays.

but VCS LVM agents are not mentioned for Linux or HP-ux - these should be, even if this is to say "no third party Multipathing is supported with VCS LVM agents"

In the Linux 5.1 bundled agents guide, it IS clear that no third party Multipathing is supported with VCS LVM agents:

You cannot use the DiskReservation agent to reserve disks that have multiple

paths. The LVMVolumeGroup and the LVMLogicalVolume agents can only be

used with the DiskReservation agent, Symantec does not support the

configuration of logical volumes on disks that have multiple paths

However, it is not so clear with 6.0 which says:

No fixed dependencies exist for LVMVolumeGroup Agent. When you create a

volume group on disks with single path, Symantec recommends that you use the

DiskReservation agent

So in 6.0, DiskReservation is optional, not mandatory as in 5.1, but the bundled agents guide does not say why the DiskReservation agent is mandatory in 5.1 and it does not elaborate why it is recommended in 6.0 - i.e. the 6.0 bundled agents guide does not explain the benefits of using the DiskReservation agent or the issues you may encounter if you don't use the DiskReservation agent.

The 6.0 bundled agents guide says for the DiskReservation agent:

In case of Veritas Dynamic Multi-Pathing, the LVMVolumeGroup and the

LVMLogicalVolume agents can be used without the DiskReservation agent

This says you can use Veritas Dynamic Multi-Pathing with LVM, but it doesn't explicitly say you can't use other multipathing software, and for the LVMVolumeGroup agent, the 6.0 bundled agents guide gives examples using multipathing, but it does not say if this is Veritas Dynamic Multi-Pathing or third party.  To me, as the examples say multipathing and NOT specifically Veritas DMP, this implies ALL multipathing is supported, but it is not clear.

So it seems that for 5.1 and before, for Linux and HP-ux (if you go by HCL), all disk multipathing was not supported with VCS LVM agents, but in the 10 years I worked for a consultant at Symantec, not one customer did NOT use disk multipathing - this is essential redundancy and there is no point having LVM agents if they don't support disk multipathing, so I am not sure it can be correct that multipathing is not supported with VCS LVM agents.  This is redeemed in part by the recent introduction of Veritas Dynamic Multi-Pathing being available as a separate product which doesn't require SF and can be used on non-VxVM disks.

So can the support be clarified for 5.1 and 6.0 of the support of third party Multipathing with VCS LVM agents on this forum and the documents updated to make what is supported clearer.

Thanks

Mike

[Diana_Pham]

Hi Mike

Thank you for your feedback. This issue has been forwarded to the appropriate writer for the VCS Bundled Agents Reference Guide and will be addressed in the next release.

Thanks,

Diana Pham

[AHerr]

Hi Mike,

What is written is correct.  If it is not clear, I appologize.  Symantec could not possibly test all Multipathing products with each of their different versions along with new and older OS versions of the local volume managers.  As such we do not claim they are supported.  This does not mean that we will not help with issues with the agents.  We will absolutley work through the support process.  We just do not have the time or people to go through and update our HCL for native products in combination of multipathing software versions.  The commands involved in LVM control are pretty standard and do not expect them to change outside of an OS release, but that does not mean a specific OS/Multipathing software combination will have problems.  When we release VCS we test our agents for supported OS versions.  With SF and DMP, we do a tremendous amount of testing and ensure the multipathing works with the commands we issue.  The support statement is basically saying that it would not be possible to do that same level of testing without a specific agent for the Multipathing vendor + LVM combination.  We have an agent that works with the native volume manager, LVM, to bring their file systems and logical volumes online.  If you add a Multipathing software into the stack and it changes the way LVM works or adds an issue into the stack, we may require the Multipathing software to be removed to troubleshoot the issue or the customer may have to troubleshoot the issue themselves. 

With the agent description discrepancies, It appears Diana will work through getting them resolved.

To ask you a question Mike, what are your expectations for support with multipathing and the LVM agent?  Should Symantec troubleshoot a 3^rd party multipathing solution?

Thanks,
Anthony

[mikebounds]

So are you saying multipathing is not supported with VCS 6.0 agents.  As the agent guide says you no longer need to use DiskReservation agent (like you had to in 5.1) and it gives example showing multipathing, this sort of implies it is supported, unless multipathing examples only applies to DMP, but the guide certainly does not say this.

I still don't understand why this has to be tested - let me draw an analogy.  Supposing someone is using the Oracle agent and they install some other software and this causes Oracle not to be able to start, so they think the Oracle agent is broken as the VCS resource won't online.  If they were to log a call, then my expectation is one of the first things that is asked (if the logs don't show what the issue is) is does it start manually outsdie of VCS control and if it doesn't then the issue is not VCS and the customer needs to resolve the issue.  This is how I see the LVM agents - if some other software is installed (like third party multipathing) and this causes problems activating or deactivating (or importing and deporting) the Logical Volume Group, then this is not an issue with the agent - it is an issue between LVM and the third party multipathing softtware.  The only exception is if using third party multipathing causes you to activate or deactivate the Logical Volume Group differently (maybe a different flag would be required, but this seems unlikely) or if the command the agent uses to monitor the Logical Volume Group has different output (with all disks paths up or one down) - and for this I can only see this possibly happening with O/S multipathing software because with hardware multipathing like EMC Powerpath, I believe the sinlge multipathed disk is added to the Logical Volume Group (not the separate paths) and so the disks looks no different to LVM than any other disk and if a path is lost, then LVM is unware.

Therefore unless, I am wrong and the LVM agents does some lower level stuff (agent docs for linux just say the vggimport, vgexport and vgdisplay commands are used), then I think all hardware multipathing should be supported and preferably O/S multipathing and I wouldn't have thought this requires much testing (mainly need to just make sure monitor entry point works if command shows differerent output if multiple paths are used, in particular if one is disconnected).

Mike

[AHerr]

Hi Mike,

I am saying exactly what the docs say, Multipathing is not supported with the LVM agent.  I am not disputing what you say in your response, there should be no issue with multipathing software and the operations of VG commands.  This is why there has not been an issue in the past.  That is not to say that there will not be in the future.  We test what we can and do not claim support for what we do not explicitly test.  We do not design, implement or sell Powerpath, so if it breaks LVM, we should we support it?  This statement is not against PP, this is the same for all external software multipathing vendors working with LVM at the host level for all OS's we support LVM.  If the implementation of the product impacts the behaivor of LVM, then we may ask you to remove it and try the cluster LVM agents without the multipathing software.

Any support for multipathing software will have to come from your vendor as we are only running the base VG commands to import and activate.

Regards,
Anthony

[mikebounds]

The docs say multipathing is not supported for 5.1, but they say they are supported for 6.0 - see extract from 6.0 Linux Bundled Agents Guide:

Linux configuration 3

In this example, the volume group testvg_1 is created on disks that have multiple

paths and have LVM tagging enabled.

  LVMVolumeGroup lvg1 (

  VolumeGroup = testvg_1

  EnableLVMTagging = 1

  )

Linux Configuration 4

In this example, the volume groups testvg_1 and testvg_2 are created on disks

that have multiple paths. LVM tagging is enabled for testvg_1 and disabled for

test_vg2.

So basically, in 5.1, the guide says the DiskReservation only supports single paths and, 

1. "The LVMVolumeGroup and the LVMLogicalVolume agents can only be used with the DiskReservation agent"
2. Explicity states "Symantec does not support the configuration of logical volumes on disks that have multiple paths" (as a consequence of 1)

The changes in 6.0 are:

1. LVMVolumeGroup no longer has to use DiskReservation agent 
2. Statement saying "Symantec does not support the configuration of logical volumes on disks that have multiple paths" is removed (which I presumed was a consequence of 1 - i.e removal of contraint to use DIskReservtion agent, removes contraint of using single paths)

It could be the examples should say disks using specifically DMP as oppose to any multipathing solution, but, even this been the case, this is just an example - nowhere does it state that multipathing is not supported with LVMVolumeGroup agent.  It doesn't explicitly state "LVMVolumeGroup agent supports Multipathing", but it does explicitly state that Diskgroup agent supported Multipathing also.

In any case, I think you are saying 6.0 should not have removed statement "Symantec does not support the configuration of logical volumes on disks that have multiple paths" as multipaths are still not supported in 6.0, but it should technically work, so I will pass this info on to the post https://www-secure.symantec.com/connect/forums/two-node-vcs-linux-cluster-hp-ca-replication which is seeing:

VCS ERROR V-16-10031-14001 (nmsomu01) LVMVolumeGroup:nmsomuVG:online:Activation of volume group failed.

with disks using multiple paths linux device mapper.

I don't understand the rest of what you have said:

We do not design, implement or sell Powerpath, so if it breaks LVM, we should we support it? 

No, if powerpath breaks LVM, this is a powerpath or LVM issue, it is nothing to do with VCS - it is only an issue with VCS if you can import a Volume Group manually, but it doesn't online in VCS, but I don't see how this could occur, because agent is running the same command as manual operation.

If the implementation of the product impacts the behaivor of LVM, then we may ask you to remove it and try the cluster LVM agents without the multipathing software.

If a product impacts the behaivor of LVM, then the customer will want to remove (or resolve issue) regardess of whether VCS is installed or not.

Any support for multipathing software will have to come from your vendor as we are only running the base VG commands to import and activate.

Of course, LVM has to support multipathing software, and Linux LVM supports Linux Device Mapper, and as agent is "only running the base VG commands to import and activate", I don't understand why agent cares whether Multipaths are used and why each each multipathing solution needs to be tested and explicitly supported for it to be used.

The bottom line is that multipathing software may break LVM, but this is an issue between LVM and multipathing software and has nothing to do with VCS, but I can't see how multipathing software will change the base VG commands the agents need to run.

Mike