I/O Fencing configuration on 2 node cluster

To install Fencing, you could do it several ways:
1) after both nodes are installed and running
2) after 1 node is installed

Either way, you need to create a DG with 3 LUNs (minimum size is 10MB) that will be accessible to both nodes. Deport the DG. Create the vxfendg file. You can then shut down had, edit the main.cf and include the line for SCSI3 PR. Then start vxfen and restart VCS (hastart).

The reason for the stop of VCS is that ALL disk groups used by the cluster in IO Fencing mode must be imported with SCSI3 PR.

You could do this on the first node and set up the /etc/vxfendg file on the second node, install VCS and it will join the cluster and get a copy of the main.cf from the first node. Since the vxfendg file was already there, vxfen will start.

IF the vxfendg file is not present, you will see a message as vxfen starts, finds no vxfendg file and exits.

Thanks for the reply...
So, Is this setup ok:
Only the primary node is up with cluster running, the second is not installed yet.
I configured the io fencing disk group and deported it
created the vxfendg, vxfenmode files
started the vxfen daemon
STOP the local cluster, configure main.cf, START the cluster

Second node installed with cluster running
I create the vxfendg, vxfenmode files
Start the vxfen daemon
STOP the local cluster, configure the same main.cf, START the cluster

Is this ok?

Thanks

If you configure node one first, when you install VCS on node 2 and VCS starts, it will get a copy of main.cf from node 1, so there is no reason to do the second stop. VCS keeps all nodes who join the cluster updated with a copy from a running system. If node 2 can see node1 when VCS starts, then it will request a copy of the configuration.

If you have already configured the vxfen files prior to installing node 2, then when LLT, GAB and vxfen start, then fencing will be in place.

Apparently you are not using the installer script to install these nodes. If you do, you need to install both at the same time. If you install them separately, the /etc/llthosts files will not have entries for both nodes and the /etc/gabtab will not have gabconfig -c -n2, it will have -n1, this is not good.

Is there a reason for not running the installer and letting it build the files properly?

Hi Gene,
At the moment, the I/O fencing configuration is done after the cluster is installed and up, so on node 2 , the cluster will be up when I configure the vxfenmode and vxfendg...
So, I understand that the main.cf will be updated from the primary node but still I'll have to stop and start vxfen daemon and the local system so it will use I/O Fencing?

we are not using the installer but install packages manually and wrap the whole process with an automatic script.

IFyou have all the files in place before starting llt, gab and vxfen on the second node, then there is no reast to stop and restart it. The main.cf it will get will already have the modification made on node 1.

You can then use the vxfenadm command to display the keys registered on the disks to see that both nodes are registered on the vxfen DG and only the node that has a DG imported will have keys on the disks for that DG.

If you read the online script for DiskGroup (/opt/VRTSvcs/bin/DiskGroup) you will see how it imports the DG with reservations.

Hi,

We configure VxVM after the cluster is up so we can register the disk groups and volumes online during configuration, that's why I can configure the coordinator disk group and all I/O Fencing files only after the cluster is up...

that's why I don't have the vxfendg and vxfenmode ready before the cluster is up on both nodes...

So Stop/Start for the vxfen and cluster is a must for me as I understand

Node 2 cannot join the cluster if node 1 is running IO Fencing and node 2 isn't. It will not be able to register with the vxfen DG, therefore it cannot join. If VCS sees the directive for fencing and vxfen isn't running, it won't go.

Why can't you copy the vxfen files from node1 to node2?

To do it your way, once you get node 1 up amd configured, then perhaps you should stop VCS on it while you configure node2. Just make sure the node with the modified main.cf comes online first with VCS.

Assaf, I think you are making this overly complicated. Install all the software on both nodes. If there is any question about it running, bring up VCS and then take it back down.

Create the vxfen DG on one system and deport it, on the other system run "vxdctl enable" and be sure you can see it ("vxdisk -o alldgs list"). Then create the vxfendg and vxfenmode files. edit the main.cf on one system, start vxfen, start vcs. Check to see that VCS came up. start vxfen on the other node, then start VCS.

Hi,

This is the first time we try an Oracle RAC with VCS configuration, we didn't need to use I/O fencing before...

we build automatic kit, no manual intervention can be done during installation, that's why I try to find a way to line up things in a working order without any need to do something in the middle...

I didn't understand clearly:
while node 1 is configured and up, node 2 will fail to start at all when we will install the VCS packages and try to start it? (without vxfendg and vxfenmode)
or it will just not be able to join only the vxfen DG until we will stop it, update the relevant files and start it again?

Message was edited by:
Assaf LeibovitchMessage was edited by:
Assaf Leibovitch

Correct. If one node is registering with the vxfen operating and the other one comes up and cannot register, VCS should not start. VCS checks to see if vxfen has registered if it is expecting SCSI3 PR to be in effect.