cancel
Showing results for 
Search instead for 
Did you mean: 

I/O Fencing configuration on 2 node cluster

Assaf_Leibovitc
Level 4
Hi All,

We are installing a 2 node cluster in an automatic process.
I read the Cluster I/O fencing configuration manual and I would like to ask about hastop and hastart sequences during configuration.

The first node is installed and only after it finished we start installing the second one.
I read that I must hastop -all before coordinator disks import/deport , vxfendg and vxfenmode update and vxfen start.
there's no hastart command documented there.

After that we have the main.cf update , again with hastop -all and hastart.

On the second node - do I have to issue hastop -all to?
what about the hastop -all before coordinator disks import/deport , is it relevant when there's only one node alive?

Thanks
24 REPLIES 24

Gene_Henriksen
Level 6
Accredited Certified
To install Fencing, you could do it several ways:
1) after both nodes are installed and running
2) after 1 node is installed

Either way, you need to create a DG with 3 LUNs (minimum size is 10MB) that will be accessible to both nodes. Deport the DG. Create the vxfendg file. You can then shut down had, edit the main.cf and include the line for SCSI3 PR. Then start vxfen and restart VCS (hastart).

The reason for the stop of VCS is that ALL disk groups used by the cluster in IO Fencing mode must be imported with SCSI3 PR.

You could do this on the first node and set up the /etc/vxfendg file on the second node, install VCS and it will join the cluster and get a copy of the main.cf from the first node. Since the vxfendg file was already there, vxfen will start.

IF the vxfendg file is not present, you will see a message as vxfen starts, finds no vxfendg file and exits.

Assaf_Leibovitc
Level 4
Thanks for the reply...
So, Is this setup ok:
Only the primary node is up with cluster running, the second is not installed yet.
I configured the io fencing disk group and deported it
created the vxfendg, vxfenmode files
started the vxfen daemon
STOP the local cluster, configure main.cf, START the cluster

Second node installed with cluster running
I create the vxfendg, vxfenmode files
Start the vxfen daemon
STOP the local cluster, configure the same main.cf, START the cluster

Is this ok?

Thanks

Gene_Henriksen
Level 6
Accredited Certified
If you configure node one first, when you install VCS on node 2 and VCS starts, it will get a copy of main.cf from node 1, so there is no reason to do the second stop. VCS keeps all nodes who join the cluster updated with a copy from a running system. If node 2 can see node1 when VCS starts, then it will request a copy of the configuration.

If you have already configured the vxfen files prior to installing node 2, then when LLT, GAB and vxfen start, then fencing will be in place.

Apparently you are not using the installer script to install these nodes. If you do, you need to install both at the same time. If you install them separately, the /etc/llthosts files will not have entries for both nodes and the /etc/gabtab will not have gabconfig -c -n2, it will have -n1, this is not good.

Is there a reason for not running the installer and letting it build the files properly?

Assaf_Leibovitc
Level 4
Hi Gene,
At the moment, the I/O fencing configuration is done after the cluster is installed and up, so on node 2 , the cluster will be up when I configure the vxfenmode and vxfendg...
So, I understand that the main.cf will be updated from the primary node but still I'll have to stop and start vxfen daemon and the local system so it will use I/O Fencing?

we are not using the installer but install packages manually and wrap the whole process with an automatic script.

Gene_Henriksen
Level 6
Accredited Certified
IFyou have all the files in place before starting llt, gab and vxfen on the second node, then there is no reast to stop and restart it. The main.cf it will get will already have the modification made on node 1.

You can then use the vxfenadm command to display the keys registered on the disks to see that both nodes are registered on the vxfen DG and only the node that has a DG imported will have keys on the disks for that DG.

If you read the online script for DiskGroup (/opt/VRTSvcs/bin/DiskGroup) you will see how it imports the DG with reservations.

Assaf_Leibovitc
Level 4
Hi,

We configure VxVM after the cluster is up so we can register the disk groups and volumes online during configuration, that's why I can configure the coordinator disk group and all I/O Fencing files only after the cluster is up...

that's why I don't have the vxfendg and vxfenmode ready before the cluster is up on both nodes...

So Stop/Start for the vxfen and cluster is a must for me as I understand

Gene_Henriksen
Level 6
Accredited Certified
Node 2 cannot join the cluster if node 1 is running IO Fencing and node 2 isn't. It will not be able to register with the vxfen DG, therefore it cannot join. If VCS sees the directive for fencing and vxfen isn't running, it won't go.

Why can't you copy the vxfen files from node1 to node2?

To do it your way, once you get node 1 up amd configured, then perhaps you should stop VCS on it while you configure node2. Just make sure the node with the modified main.cf comes online first with VCS.

Gene_Henriksen
Level 6
Accredited Certified
Assaf, I think you are making this overly complicated. Install all the software on both nodes. If there is any question about it running, bring up VCS and then take it back down.

Create the vxfen DG on one system and deport it, on the other system run "vxdctl enable" and be sure you can see it ("vxdisk -o alldgs list"). Then create the vxfendg and vxfenmode files. edit the main.cf on one system, start vxfen, start vcs. Check to see that VCS came up. start vxfen on the other node, then start VCS.

Assaf_Leibovitc
Level 4
Hi,

This is the first time we try an Oracle RAC with VCS configuration, we didn't need to use I/O fencing before...

we build automatic kit, no manual intervention can be done during installation, that's why I try to find a way to line up things in a working order without any need to do something in the middle...

I didn't understand clearly:
while node 1 is configured and up, node 2 will fail to start at all when we will install the VCS packages and try to start it? (without vxfendg and vxfenmode)
or it will just not be able to join only the vxfen DG until we will stop it, update the relevant files and start it again?

Message was edited by:
Assaf LeibovitchMessage was edited by:
Assaf Leibovitch

Gene_Henriksen
Level 6
Accredited Certified
Correct. If one node is registering with the vxfen operating and the other one comes up and cannot register, VCS should not start. VCS checks to see if vxfen has registered if it is expecting SCSI3 PR to be in effect.

Gene_Henriksen
Level 6
Accredited Certified
By the way, you can use the install scripts with answer files so they could be part of your overall install script.

Assaf_Leibovitc
Level 4
Thanks a lot Gene,

So I guess vxfendg and vxfenmode copy before node 2 VCS is up is a must for me...

Well, it's time for me to logoff and sleep on it... :)


Thanks again and have a great day

Gene_Henriksen
Level 6
Accredited Certified
And don't forget you can reward points for answers.

Assaf_Leibovitc
Level 4
Hi Gene, do you know VxVM well also?

I have a problem with the alloc= on vxassist growby... it doesn't seem to work for me... i have to disks, want to use only the first but still it takes space from the second one...

Gene_Henriksen
Level 6
Accredited Certified
Yep I do VM. And occasionally VxFS. Mostly I do VCS, VVR and GCO.

Did you grow the volume too big to fit on one disk?

vxassist -g growto myvol 10g mydisk1

or

set disk2 to "reserved" and then grow the volume. With a reserved disk, it is not supposed to be used in space allocation unless specifically named.

Assaf_Leibovitc
Level 4
Hi,

I have 2 disks under veritas, each a 10g size disks
oracle_data
oracle_backup

I create the first volume with 2g size:
/usr/sbin/vxassist -g oracle_dg make oracle_data_vol 4190109 layout=concat logtype=dco dcoversion=20 drl=on oracle_data

than grow it with 2g size:

/usr/sbin/vxassist -g oracle_dg growby oracle_data_vol 4190109 oracle_data

when running vxprint, I see the the log plex is on oracle_backup... and not on oracle_data as it should.

Gene_Henriksen
Level 6
Accredited Certified
A little different problem. Check to see if the log is mirrored (2 copies). You can limit the number of copies of the log with nlog=1

Why do you need logs? The volume isn't mirrored? DRL is for resyncing (it is a bitmap) dirty regions that may not have gotten update on both plexes of a mirror. This will give you a performance hit for no reason. Also dco logs are for fast resync when you have split of a mirrored copy. Again, no mirror, no reason for log.

Instead of bothering with 4190109, just type 2g. Much easier.

If you use the VEA Gui, you can drag and drop these object onto any disk in the DG. or you could move them with CLI.

Assaf_Leibovitc
Level 4
Forgot to mention, I have the 2 disks to serve as source and snapshot.

I use the vxsnap refresh feature.

After I do growby for 2 times with 2 gb on every attempt with the oracle_data as storage space I get:
v oracle_data_vol fsgen ENABLED 12570336 - ACTIVE - -
pl oracle_data_vol-01 oracle_data_vol ENABLED 12570336 - ACTIVE - -
sd oracle_data-01 oracle_data_vol-01 ENABLED 4190112 0 - - -
sd oracle_data-03 oracle_data_vol-01 ENABLED 8380224 4190112 - - -
dc oracle_data_vol_dco oracle_data_vol - - - - - -
v oracle_data_vol_dcl gen ENABLED 1136 - ACTIVE - -
pl oracle_data_vol_dcl-01 oracle_data_vol_dcl ENABLED 1136 - ACTIVE - -
sd oracle_backup-02 oracle_data_vol_dcl-01 ENABLED 1136 0 - - -

This prevents me from using the second disk as mirror... :(

Gene_Henriksen
Level 6
Accredited Certified
you can use VEA easily to drag the SD back to the desired disk.

What type of snapshot are you using? Space Optimized snaps do not require being on a separate disk.

(go to www.symantec.com/yellowbooks and download Local Copy Serivces PDF for more info on snapshots)