09-14-2012 03:31 AM
Hi All,
would like to have a solution for below issue which i am facing during my twno node linux cluster for HP CA replication.
I have configured a two node cluster, each cluster resides in separate sites with dedicated HP P6300 storage. We had configured the storages,Fabric etc according to recommdation. I had installed the vcs agent for hp ca storage.
We have assigned a vdisk from primary array to Primary node and created a DR group for the vdisk for replication accroass to DR site array and presented the same vdisk to DR node.
We have configured LVMVolumeGroup, LVMLogicalVolume and Mount resource in VCS Service Group.however whenever i try o bring the Mount resourece to online the service group going to faulty state for some reason.
also i unable to integrate the EVACA resource in VCS Cluster for some reason.
Any advice highly appreciated.
Thanks.
Solved! Go to Solution.
09-20-2012 06:52 AM
I think it should work with device mapper - the support from Symantec, just means they test it and it is very unlikely they would have to change any code for it to work as it is very unlikley vgimport/vgchange commands will change just because Disk Multipathing is involved. So I don't why Symantec are bothered whether Disk Multipathing is used from 6.0 where the DiskReservation agent is no longer required.
Mike
09-14-2012 10:20 PM
Hi,
Can you provide more details plz
OS version
VCS version ?
snippet for engine log which demonstrates the error during the resource fail ..
G
09-15-2012 03:08 AM
Hi Gaurav,
Please fid the following,
OS Version - RHEL 5.8
VCS Version - 6.0
I am sending the logs shortly.
Thanks.
09-15-2012 06:04 AM
Hi Gaurav,
Please note my operating system is RHEL 5.8 x86_64.
Please find the following outout from logs, It is mounting the FS but it is throwing error " lvol0 is already mounted '.
2012/09/15 15:55:39 VCS ERROR V-16-10031-14001 (nmsomu01) LVMVolumeGroup:nmsomuVG:online:Activation of volume group failed.
2012/09/15 15:55:40 VCS INFO V-16-1-10298 Resource nmsomuVG (Owner: Unspecified, Group: nmsomuSG) is online on nmsomu01 (VCS initiated)
2012/09/15 15:56:34 VCS INFO V-16-1-50135 User root fired command: hares -clear nmsomuMNT from localhost
2012/09/15 15:56:34 VCS INFO V-16-1-10307 Resource nmsomuMNT (Owner: Unspecified, Group: nmsomuSG) is offline on nmsomu01 (Not initiated by VCS)
2012/09/15 15:56:34 VCS INFO V-16-1-10307 Resource nmsomuMNT (Owner: Unspecified, Group: nmsomuSG) is offline on nmsomu02 (Not initiated by VCS)
2012/09/15 15:56:34 VCS INFO V-16-1-50135 (nmsomu01) User root@vcsvom.qtel.com.qa fired command:/opt/VRTSvcs/bin/hares -clear nmsomuMNT from VOM-CS
2012/09/15 15:57:18 VCS INFO V-16-1-50135 User root fired command: hares -online nmsomuMNT nmsomu01 from localhost
2012/09/15 15:57:18 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group nmsomuSG on all nodes
2012/09/15 15:57:18 VCS NOTICE V-16-1-10301 Initiating Online of Resource nmsomuMNT (Owner: Unspecified, Group: nmsomuSG) on System nmsomu01
2012/09/15 15:57:18 VCS INFO V-16-1-50135 (nmsomu01) User root@vcsvom.qtel.com.qa fired command:/opt/VRTSvcs/bin/hares -online nmsomuMNT -sys nmsomu01 from VOM-CS
2012/09/15 15:57:18 VCS NOTICE V-16-10031-5511 (nmsomu01) Mount:nmsomuMNT:online:Trying force mount...
2012/09/15 15:57:18 VCS NOTICE V-16-10031-5516 (nmsomu01) Mount:nmsomuMNT:online:Running fsck...
2012/09/15 15:57:18 VCS WARNING V-16-10031-5521 (nmsomu01) Mount:nmsomuMNT:online:Could not mount the block device /dev/vg-omu/lvol0.
2012/09/15 15:57:19 VCS INFO V-16-2-13716 (nmsomu01) Resource(nmsomuMNT): Output of the completed operation (online)
==============================================
mount: /dev/vg-omu/lvol0 already mounted or /backup busy
mount: according to mtab, /dev/mapper/vg--omu-lvol0 is already mounted on /backup
fsck 1.39 (29-May-2006)
e2fsck 1.39 (29-May-2006)
Warning! /dev/vg-omu/lvol0 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
/dev/vg-omu/lvol0: clean, 11/6553600 files, 251733/13107200 blocks
mount: /dev/vg-omu/lvol0 already mounted or /backup busy
mount: according to mtab, /dev/mapper/vg--omu-lvol0 is already mounted on /backup
==============================================
2012/09/15 15:59:20 VCS ERROR V-16-2-13066 (nmsomu01) Agent is calling clean for resource(nmsomuMNT) because the resource is not up even after online completed.
2012/09/15 15:59:21 VCS INFO V-16-2-13068 (nmsomu01) Resource(nmsomuMNT) - clean completed successfully.
2012/09/15 15:59:21 VCS INFO V-16-2-13071 (nmsomu01) Resource(nmsomuMNT): reached OnlineRetryLimit(0).
09-15-2012 08:10 AM
The LVM and mount resources are not working because you have not integrated your EVACA resource yet. With normal EVACA replication, one side is read-write (RW) and the other read-only (RO) and so the EVACA resource is required to make the storage RW when you failover. Without this the volume group cannot be activated and the mount resource cannot be mounted RW and hence the messages in the log:
LVMVolumeGroup:nmsomuVG:online:Activation of volume group failed
So you need to get the EVACA resource working.
Mike
09-15-2012 11:49 AM
Hi Mike,
Thanks for your comment,
I have tried to integrate the VCS with HP CA according to the document vcs_hp_eva_ca_install.
I have configured the HP CA agents in the cluster nodes and configured the HP SSSU utililty to supply the
EVA Management and array information to VCS.
but my issue is i need to mention the array information for each site in main.cf file.
i did try to mention in main.cf file but it did not work. how can i achieve it ?
Any advice highly appreciated.
Thanks.
09-15-2012 12:29 PM
What error did you get when you tried to configure array name?
Mike
09-16-2012 12:53 AM
Hi Mike,
I have integrated the EVACA in VCS and able to bring the EVACA resource online. However still i am facing the same issue to bring the mount resource online. it says " /dev/vg-omu/lvol0 already mounted or /backup busy "
Please find the logs for your reference
2012/09/16 10:36:56 VCS INFO V-16-1-50135 User root fired command: hagrp -online nmsomuSG nmsomu01 from localhost
2012/09/16 10:36:56 VCS NOTICE V-16-1-10166 Initiating manual online of group nmsomuSG on system nmsomu01
2012/09/16 10:36:56 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group nmsomuSG on all nodes
2012/09/16 10:36:56 VCS NOTICE V-16-1-10301 Initiating Online of Resource nmomuEVACA (Owner: Unspecified, Group: nmsomuSG) on System nmsomu01
2012/09/16 10:36:56 VCS INFO V-16-1-50135 (nmsomu01) User root@vcsvom.qtel.com.qa(unixpwd) fired command:/opt/VRTSvcs/bin/hagrp -online nmsomuSG -sys nmsomu01 from VOM-CS
2012/09/16 10:36:59 VCS INFO V-16-20073-26 (nmsomu01) EVACA:nmomuEVACA:online:The replication role of DR group DRG_VCS_OMU is source; no action is required.
2012/09/16 10:37:00 VCS INFO V-16-1-10298 Resource nmomuEVACA (Owner: Unspecified, Group: nmsomuSG) is online on nmsomu01 (VCS initiated)
2012/09/16 10:37:00 VCS NOTICE V-16-1-10301 Initiating Online of Resource nmsomuVG (Owner: Unspecified, Group: nmsomuSG) on System nmsomu01
2012/09/16 10:37:01 VCS ERROR V-16-10031-14001 (nmsomu01) LVMVolumeGroup:nmsomuVG:online:Activation of volume group failed.
2012/09/16 10:37:02 VCS INFO V-16-1-10298 Resource nmsomuVG (Owner: Unspecified, Group: nmsomuSG) is online on nmsomu01 (VCS initiated)
2012/09/16 10:37:02 VCS NOTICE V-16-1-10447 Group nmsomuSG is online on system nmsomu01
2012/09/16 10:38:33 VCS INFO V-16-1-50135 User root fired command: hares -add nmsomuLV LVMLogicalVolume nmsomuSG from localhost
2012/09/16 10:38:33 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/LVMLogicalVolume/LVMLogicalVolumeAgent for resource type LVMLogicalVolume successfully started at Sun Sep 16 10:38:33 2012
2012/09/16 10:38:33 VCS INFO V-16-1-50135 User root fired command: hares -modify nmsomuLV LogicalVolume lvol0 from localhost
2012/09/16 10:38:33 VCS INFO V-16-1-50135 User root fired command: hares -modify nmsomuLV VolumeGroup vg-omu from localhost
2012/09/16 10:38:46 VCS INFO V-16-1-50135 User root fired command: hares -add nmsomuMNT Mount nmsomuSG from localhost
2012/09/16 10:38:46 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Sun Sep 16 10:38:46 2012
2012/09/16 10:38:46 VCS INFO V-16-1-50135 User root fired command: hares -modify nmsomuMNT BlockDevice /dev/vg-omu/lvol0 from localhost
2012/09/16 10:38:46 VCS INFO V-16-1-50135 User root fired command: hares -modify nmsomuMNT MountPoint /backup from localhost
2012/09/16 10:38:46 VCS INFO V-16-1-50135 User root fired command: hares -modify nmsomuMNT FSType ext3 from localhost
2012/09/16 10:38:48 VCS INFO V-16-1-50135 User root fired command: hares -modify nmsomuMNT FsckOpt -n from localhost
2012/09/16 10:39:21 VCS INFO V-16-1-50135 User root fired command: hares -link nmsomuLV nmsomuVG 0 0 from localhost
2012/09/16 10:39:22 VCS INFO V-16-1-50135 User root fired command: hares -link nmsomuMNT nmsomuLV 0 0 from localhost
2012/09/16 10:39:57 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources nmsomuSG from localhost
2012/09/16 10:39:57 VCS INFO V-16-1-10297 Resource nmsomuLV (Owner: Unspecified, Group: nmsomuSG) is online on nmsomu01 (First probe)
2012/09/16 10:39:57 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group nmsomuSG on all nodes
2012/09/16 10:39:57 VCS INFO V-16-1-10304 Resource nmsomuMNT (Owner: Unspecified, Group: nmsomuSG) is offline on nmsomu01 (First probe)
2012/09/16 10:39:58 VCS INFO V-16-1-10304 Resource nmsomuLV (Owner: Unspecified, Group: nmsomuSG) is offline on nmsomu02 (First probe)
2012/09/16 10:39:58 VCS INFO V-16-1-10304 Resource nmsomuMNT (Owner: Unspecified, Group: nmsomuSG) is offline on nmsomu02 (First probe)
2012/09/16 10:40:20 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost
2012/09/16 10:41:00 VCS INFO V-16-1-50135 User root fired command: hares -online nmsomuMNT nmsomu01 from localhost
2012/09/16 10:41:00 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group nmsomuSG on all nodes
2012/09/16 10:41:00 VCS NOTICE V-16-1-10301 Initiating Online of Resource nmsomuMNT (Owner: Unspecified, Group: nmsomuSG) on System nmsomu01
2012/09/16 10:41:00 VCS INFO V-16-1-50135 (nmsomu01) User root@vcsvom.qtel.com.qa(unixpwd) fired command:/opt/VRTSvcs/bin/hares -online nmsomuMNT -sys nmsomu01 from VOM-CS
2012/09/16 10:41:01 VCS NOTICE V-16-10031-5511 (nmsomu01) Mount:nmsomuMNT:online:Trying force mount...
2012/09/16 10:41:01 VCS NOTICE V-16-10031-5516 (nmsomu01) Mount:nmsomuMNT:online:Running fsck...
2012/09/16 10:41:02 VCS WARNING V-16-10031-5521 (nmsomu01) Mount:nmsomuMNT:online:Could not mount the block device /dev/vg-omu/lvol0.
2012/09/16 10:41:03 VCS INFO V-16-2-13716 (nmsomu01) Resource(nmsomuMNT): Output of the completed operation (online)
==============================================
mount: /dev/vg-omu/lvol0 already mounted or /backup busy
mount: according to mtab, /dev/mapper/vg--omu-lvol0 is already mounted on /backup
fsck 1.39 (29-May-2006)
e2fsck 1.39 (29-May-2006)
Warning! /dev/vg-omu/lvol0 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
/dev/vg-omu/lvol0 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (12855467, counted=12855466).
Fix? no
Free inodes count wrong (6553589, counted=6553587).
Fix? no
/dev/vg-omu/lvol0: ********** WARNING: Filesystem still has errors **********
/dev/vg-omu/lvol0: 11/6553600 files (9.1% non-contiguous), 251733/13107200 blocks
mount: /dev/vg-omu/lvol0 already mounted or /backup busy
mount: according to mtab, /dev/mapper/vg--omu-lvol0 is already mounted on /backup
==============================================
2012/09/16 10:42:02 VCS INFO V-16-1-50135 User root fired command: hares -modify ... -add nmomuEVACA ResourceInfo ReplicationStatus
DR Group Role: Source
Operational State: good
Write mode: Synchronous
Failsafe: disable from localhost
2012/09/16 10:43:03 VCS ERROR V-16-2-13066 (nmsomu01) Agent is calling clean for resource(nmsomuMNT) because the resource is not up even after online completed.
2012/09/16 10:43:04 VCS INFO V-16-2-13068 (nmsomu01) Resource(nmsomuMNT) - clean completed successfully.
2012/09/16 10:43:04 VCS INFO V-16-2-13071 (nmsomu01) Resource(nmsomuMNT): reached OnlineRetryLimit(0).
09-16-2012 02:01 AM
You don't appear to have a DiskReservation resource - see extract from Linux Bundled agents guide:
LVMVolumeGroup resources depend on DiskReservation resources. If an LVMVolumeGroup does not have a corresponding DiskReservation resource on which it depends, the LVMVolumeGroup does not function
09-16-2012 05:39 AM
Hi Mike,
Thanks for your reply.
Please note that we are using linux device mapper for multipathing.
Please correct me if i am wrong, According to bundle agent document disk reservation is not required for multipathing disks,
So can i configure disk reservation for linux device mapper managed disk /dev/mpath/mpath1 also ?
I have only one LUN ( /dev/mpath/mapth1 ) for vg-omu volume group.
Pv Name : /dev/mpath/mapth1
VG Name : vg-omu
LV Name : lvol0
Thanks.
09-16-2012 10:11 AM
Extract I gave earlier was from 5.1 bundled agents guide which also says:
You cannot use the DiskReservation agent to reserve disks that have multiplepaths. The LVMVolumeGroup and the LVMLogicalVolume agents can only beused with the DiskReservation agent, Symantec does not support theconfiguration of logical volumes on disks that have multiple paths. To ensuredata protection on such a configuration, Symantec recommends the use ofVeritas Volume Manager (VxVM) disk groups.
No fixed dependencies exist for LVMVolumeGroup Agent. When you create avolume group on disks with single path, Symantec recommends that you use theDiskReservation agent.
You cannot use the DiskReservation agent to reserve disks that have multiplepaths. In case of Veritas Dynamic Multi-Pathing, the LVMVolumeGroup and theLVMLogicalVolume agents can be used without the DiskReservation agent
Enabling volume group activation protection for Veritas Dynamic Multi-PathingOn each node in the cluster, perform the following procedure to enable activationprotection for volume groups on Red Hat and SUSE systems.To enable volume group activation protection1 On each node in the cluster, edit /etc/lvm/lvm.conf , and add the following line:tags { hosttags = 1 }2 On each node in the cluster, create the file lvm_`uname –n`.conf in the /etc/lvm/ directory.3 Add the following line to the file you created in step 2:activation { volume_list="@node" }where node is the value of the uname -n command.
The VCS LVM agent supports the EMC PowerPath third-party driver on EMC's Symmetrix 8000 and DMX series arrays.The VCS LVM agent supports the HITACHI HDLM third-party driver on Hitachi USP/NSC/USPV/USPVM, 9900V series arrays.
09-20-2012 01:11 AM
I posted query on support for third party Multipathing with VCS LVM agents at https://www-secure.symantec.com/connect/forums/using-third-party-multipathing-vcs-lvm-agents and it seems that multipaths are not supported for 6.0 VCS LVM agent, but it should technically work.
Have you tried manually activating volume group. I guess as there are no errors about the import, the vgimport was successful in the agent and it is the "vgchange -a" that fails, so see if it works from the command line.
Mike
09-20-2012 02:40 AM
Hi Mike,
Yes, You are correct. I was asking to symantec support also and they said i have to have dmp.
native device mapper may be supported in upcoming version.
i tried your suggestion to vgimport for vgomu VG but it is saying write failure as below.
[root@ ~]# vgimport vgomu
/dev/mpath/36001438009b04d840000400001030000: write failed after 0 of 4096 at 24576: Operation not permitted.
I have a doupt that this lun was not writable coz my replication from PR storage to DR storage is read only. (available options are none and read only )
so the hpevaca agent should take care of it that the DR storage should able to write when we do failover.
Thanks.
09-20-2012 02:49 AM
Hi mike,
I would like to inform you that node1 is running now, i can able to bring the service group online.
however i unable to do failover the service group to node2.
Thanks.
09-20-2012 03:00 AM
If you can't run vgimport then issue is with hpevaca agent which is not making LUNs writeable.
Have you localised the attributes for the hpevaca resource so that you are specifying a different array for each node - if not, this will be why it doesn't work on node2. If you don't know how to localise attributes then please give extract from main.cf (in /etc/VRTSvcs/conf/config) of your hpevaca resource and whether you want to use GUI or command line.
Mike
09-20-2012 03:27 AM
You need to localise at least LocalEVAName, so if the local array for node 2 is Q_NMS2_EVA, then you need to run:
haconf -makerw hares -local nmomuEVACA LocalEVAName hares -modify nmomuEVACA LocalEVAName Q_NMS2_EVA -sys nmsomu02 hares -dump -makero
If any other attributes are local to that node, you need to change these too, so for instance if you have a different ManagementServer for each node, then you need to locaise this attribute too.
Mike
09-20-2012 03:32 AM
Hi Mike,
It means i have to have EVACA resource which contains info about PR storage for node1 only and
I ahve create another resource for EVACA lets say EVACA2 along with Secondary storage info and restrict it to node 2 only orrect ?
09-20-2012 03:39 AM
No, you have one resource with localised attributes, so when you have run the commands in the last post, your resource in main.cf will look like:
EVACA nmomuEVACA (
ManagementServer = "172.16.236.200"
LocalEVAName@nmsomu01 = Q_NMS1_EVA
LocalEVAName@nmsomu02 = Q_NMS2_EVA
DRGroupName = DRG_VCS_OMU
SSSUPath = "/opt/Hewlett-Packard/sssu_linux_x64"
)
Mike
09-20-2012 06:44 AM
Hi Mike,
It is working but i have tested it with other setup where i have installed dmp also!
I will configure it with device mapper tomorrow and confirm you.
It was great and really appriciated your efforts to solve this issue.
Thank you very much!
09-20-2012 06:52 AM
I think it should work with device mapper - the support from Symantec, just means they test it and it is very unlikely they would have to change any code for it to work as it is very unlikley vgimport/vgchange commands will change just because Disk Multipathing is involved. So I don't why Symantec are bothered whether Disk Multipathing is used from 6.0 where the DiskReservation agent is no longer required.
Mike