How to add virtual fencing disk to KVM guest

[Juha_2]

Hi,

I'm trying to install Veritas Storage Foundation Cluster File System HA 6.0.3 to 6 KVM RHEL guests.

I installed RHEL6.4 and created 3 virtual fencing disks using qemu-img. I added them as SCSI disks to the node and did the vxdisksetup for them. Now when I tried to configure fencing, I get the error below.

I'm wondering if anyone has installed cluster file system into KVM guest environment and could tell me how to add virtual fencing disks so that fencing configuration would be successful.

This is what I tried on the KVM:

    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/home/VM/VMImages/IOFencing1.img'/>
      <target dev='sda' bus='scsi'/>
      <shareable/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/home/VM/VMImages/IOFencing2.img'/>
      <target dev='sdb' bus='scsi'/>
      <shareable/>
      <alias name='scsi0-0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/home/VM/VMImages/IOFencing3.img'/>
      <target dev='sdc' bus='scsi'/>
      <shareable/>
      <alias name='scsi0-0-0-2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>

Here is how the disks look like:

[root@node1 installsfcfsha601-201605190528YQa]# vxdisk -o alldgs list
DEVICE       TYPE            DISK         GROUP        STATUS
disk_0       auto:cdsdisk    -            -            online
disk_1       auto:cdsdisk    -            -            online
disk_2       auto:cdsdisk    -            -            online
disk_3       auto:none       -            -            online invalid
disk_4       auto:none       -            -            online invalid
vda          auto:none       -            -            online invalid
[root@node1 installsfcfsha601-201605190528YQa]# 

[root@node1 installsfcfsha601-201605190528YQa]# cat /etc/vxfentab 
#
# /etc/vxfentab: 
# DO NOT MODIFY this file as it is generated by the
# VXFEN rc script from the file /etc/vxfendg.
#
/dev/vx/rdmp/disk_0 QEMU%5FQEMU%20HARDDISK%5FDISKS%5Fdrive-scsi0-0-0-0
/dev/vx/rdmp/disk_1 QEMU%5FQEMU%20HARDDISK%5FDISKS%5Fdrive-scsi0-0-0-1
/dev/vx/rdmp/disk_2 QEMU%5FQEMU%20HARDDISK%5FDISKS%5Fdrive-scsi0-0-0-2
[root@node1 installsfcfsha601-201605190528YQa]# 

The error I get:

kernel:      I/O Fencing DISABLED!VXFEN INFO V-11-1-35 Fencing driver going into RUNNING state
kernel: GAB INFO V-15-1-20032 Port b closed
kernel: GAB INFO V-15-1-20229 Client VxFen deiniting GAB API
kernel: VXFEN INFO V-11-1-36 VxFEN configured at protocol version 30
kernel: GAB INFO V-15-1-20230 Client VxFen inited GAB API with handle ffff8803b8288ac0
kernel: GAB INFO V-15-1-20036 Port b[VxFen (refcount 2)] gen   93aa25 membership 0
kernel: GAB INFO V-15-1-20038 Port b[VxFen (refcount 2)] gen   93aa25 k_jeopardy ;12345
kernel: GAB INFO V-15-1-20040 Port b[VxFen (refcount 2)] gen   93aa25    visible ;12345
kernel: VXFEN WARNING V-11-1-12 Potentially a
kernel:        preexisting split-brain.
kernel:        Dropping out of cluster.
kernel:        Refer to user documentation for
kernel:        steps required to clear preexisting
kernel:        split-brain.

[Juha_2]

So I guess it seems that the virtual SCSI disks are not SCSI-3 compliant. How do I create SCSI-3 compliant virtual scsi disks?

[root@node1 ~]# vxfentsthdw

Veritas vxfentsthdw version 6.0.300.000-GA Linux

The utility vxfentsthdw works on the two nodes of the cluster.
The utility verifies that the shared storage one intends to use is
configured to support I/O fencing.  It issues a series of vxfenadm
commands to setup SCSI-3 registrations on the disk, verifies the
registrations on the disk, and removes the registrations from the disk.

******** WARNING!!!!!!!! ********

THIS UTILITY WILL DESTROY THE DATA ON THE DISK!! 

Do you still want to continue : [y/n] (default: n) y
The logfile generated for vxfentsthdw is /var/VRTSvcs/log/vxfen/vxfentsthdw.log.16323

Enter the first node of the cluster:
node1
Enter the second node of the cluster:
node2

Enter the disk name to be checked for SCSI-3 PGR on node node1 in the format:
        for dmp: /dev/vx/rdmp/sdx
        for raw: /dev/sdx
Make sure it is the same disk as seen by nodes node1 and node2
/dev/vx/rdmp/disk_0

Enter the disk name to be checked for SCSI-3 PGR on node node2 in the format:
        for dmp: /dev/vx/rdmp/sdx
        for raw: /dev/sdx
Make sure it is the same disk as seen by nodes node1 and node2
/dev/vx/rdmp/disk_0

***************************************************************************

Testing node1 /dev/vx/rdmp/disk_0 node2 /dev/vx/rdmp/disk_0

Evaluate the disk before testing  ........................... Pre-existing keys
RegisterIgnoreKeys on disk /dev/vx/rdmp/disk_0 from node node1 ....... Failed

RegisterIgnoreKeys to disk  failed on node node1.
        Disk  is not SCSI-3 compliant on node node1.
        Execute the utility vxfentsthdw again and if failure persists contact
        the vendor for support in enabling SCSI-3 persistent reservations

Removing test keys and temporary files, if any...
[root@node1 ~]#