cancel
Showing results for 
Search instead for 
Did you mean: 

I/O fencing not working

Fugitive
Level 4

I have 2 solaris systems running VCS5.1 MP1 , solaris 10u9 , vxfencing is configured but when i test network failure , both the systems panic

 

What could be the reason for it  Is there anything i 'm doing wrong. 

 

 

Following are the o/p of the fencing configurations 

 

node1# cat /etc/vxfentab

 

#
# /etc/vxfentab:
# DO NOT MODIFY this file as it is generated by the
# VXFEN rc script from the file /etc/vxfendg.
#
/dev/vx/rdmp/disk_4s2
/dev/vx/rdmp/disk_5s2
/dev/vx/rdmp/disk_6s2
node1# cat /etc/vxfenmode
#
# vxfen_mode determines in what mode VCS I/O Fencing should work.
#
# available options:
# scsi3      - use scsi3 persistent reservation disks
# customized - use script based customized fencing
# sybase     - use scsi3 disks in kernel but coordinate membership with Sybase ASE
# disabled   - run the driver but don't do any actual fencing
#
vxfen_mode=scsi3
 
#
# scsi3_disk_policy determines the way in which I/O Fencing communicates with
# the coordination disks.
#
# available options:
# dmp - use dynamic multipathing
# raw - connect to disks using the native interface
#
scsi3_disk_policy=dmp
node1# cat /etc/vxfendg
oddfendg
node1#
 
 
 
vxfenadm -s all -f /etc/vxfentab
 
Device Name: /dev/vx/rdmp/disk_4s2
Total Number Of Keys: 2
key[0]:
        [Numeric Format]:  86,70,48,48,48,49,48,48
        [Character Format]: VF000100
   *    [Node Format]: Cluster ID: 1     Node ID: 0   Node Name: node1
key[1]:
        [Numeric Format]:  86,70,48,48,48,49,48,49
        [Character Format]: VF000101
   *    [Node Format]: Cluster ID: 1     Node ID: 1   Node Name: node2
 
Device Name: /dev/vx/rdmp/disk_5s2
Total Number Of Keys: 2
key[0]:
        [Numeric Format]:  86,70,48,48,48,49,48,48
        [Character Format]: VF000100
   *    [Node Format]: Cluster ID: 1     Node ID: 0   Node Name: node1
key[1]:
        [Numeric Format]:  86,70,48,48,48,49,48,49
        [Character Format]: VF000101
   *    [Node Format]: Cluster ID: 1     Node ID: 1   Node Name: node2
 
Device Name: /dev/vx/rdmp/disk_6s2
Total Number Of Keys: 2
key[0]:
        [Numeric Format]:  86,70,48,48,48,49,48,48
        [Character Format]: VF000100
   *    [Node Format]: Cluster ID: 1     Node ID: 0   Node Name: node1
key[1]:
        [Numeric Format]:  86,70,48,48,48,49,48,49
        [Character Format]: VF000101
   *    [Node Format]: Cluster ID: 1     Node ID: 1   Node Name: node2
1 ACCEPTED SOLUTION

Accepted Solutions

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Have you run the same tests on your fencing disks?

/opt/VRTSvcs/vxfen/bin/vxfentsthdw -r -g oddfendg

If that also completes successfully, you might want to double-check DMP support/configuration.

Start by verifying array settings (extract from Hardware TN:(

View solution in original post

7 REPLIES 7

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Please post /var/adm/messages for both nodes.

Please post files as attachments.

Gaurav_S
Moderator
Moderator
   VIP    Certified
Also post # grep -i usefence /etc/VRTSvcs/conf/config/main.cf what array are you using ? does the array supports SCSI3 PR ? Did u ran vxfentsthdw before using fencing ? did all the tests passed in it ? In case you try to use vxfentsthdw now, use it on a blank Lun as this test may be destructive to data.. G

Fugitive
Level 4

Hi Marianne,

 

I have attached the excerpt of the messages file from both the systems .. from the moment one NIC fails between the 2 nodes and jeopardy membership is formed and then to network partition. Let me know if more details are required.

 

Thanks.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

I agree with Gaurav - you need to run vxfentsthdw tests to confirm that all data and fencing disks pass all tests.

This seems to be the problem (both nodes reported similar errors):

VxVM vxdmp V-5-0-0 dmp_pr_do_preempt: failed on path (30/0x140) with status = 0x2 sense

You can run vxfentsthdw on data as well as fencing disks in non-destructive mode, either one disk at a time or all disks in disk group (-r for non-destructive, -n to use rsh (drop this option if using ssh), -g  for diskgroup):

/opt/VRTSvcs/vxfen/bin/vxfentsthdw -r -n -g <diskgroup>

Fugitive
Level 4

 

Its on Clarrion array. 
 
#haclus -value UseFence
SCSI3
# grep -i fence /etc/VRTSvcs/conf/config/main.cf
        UseFence = SCSI3
 
 
 
sol10arch01 # /opt/VRTSvcs/vxfen/bin/vxfentsthdw -r -g oradg
 
Veritas vxfentsthdw version 5.1SP1 Solaris
 
 
The utility vxfentsthdw works on the two nodes of the cluster.
The utility verifies that the shared storage one intends to use is
configured to support I/O fencing.  It issues a series of vxfenadm
commands to setup SCSI-3 registrations on the disk, verifies the
registrations on the disk, and removes the registrations from the disk.
 
The logfile generated for vxfentsthdw is /var/VRTSvcs/log/vxfen/vxfentsthdw.log.8120
 
Enter the first node of the cluster:
sol10arch01
Enter the second node of the cluster:
 sol10arch02  
 
********************************************
 
Testing  sol10arch01  /dev/vx/rdmp/disk_3s2  sol10arch02   /dev/vx/rdmp/disk_1s2
 
Evaluate the disk before testing  ........................ No Pre-existing keys
RegisterIgnoreKeys on disk /dev/vx/rdmp/disk_3s2 from node  sol10arch01   Passed
Verify registrations for disk /dev/vx/rdmp/disk_3s2 on node  sol10arch01   Passed
RegisterIgnoreKeys on disk /dev/vx/rdmp/disk_1s2 from node  sol10arch02    Passed
Verify registrations for disk /dev/vx/rdmp/disk_1s2 on node  sol10arch02    Passed
Unregister keys on disk /dev/vx/rdmp/disk_3s2 from node  sol10arch01  . Passed
Verify registrations for disk /dev/vx/rdmp/disk_1s2 on node  sol10arch02    Passed
Unregister keys on disk /dev/vx/rdmp/disk_1s2 from node  sol10arch02   . Passed
Check to verify there are no keys from node  sol10arch01  ............. Passed
Check to verify there are no keys from node  sol10arch02   ............. Passed
RegisterIgnoreKeys on disk /dev/vx/rdmp/disk_3s2 from node  sol10arch01   Passed
Verify registrations for disk /dev/vx/rdmp/disk_3s2 on node  sol10arch01   Passed
Read from disk /dev/vx/rdmp/disk_3s2 on node  sol10arch01  ............ Passed
Write to disks from node  sol10arch01  ................................ Skipped
Read from disk /dev/vx/rdmp/disk_1s2 on node  sol10arch02   ............ Passed
Write to disks from node  sol10arch02   ................................ Skipped

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Have you run the same tests on your fencing disks?

/opt/VRTSvcs/vxfen/bin/vxfentsthdw -r -g oddfendg

If that also completes successfully, you might want to double-check DMP support/configuration.

Start by verifying array settings (extract from Hardware TN:(

Gaurav_S
Moderator
Moderator
   VIP    Certified

Agree with Marianne ... get the storage config checked again like zoning /masking etc .. array should be tuned as per Symantec requirements mentioned in hardware technotes ... also would recommend few more basic stuff..

- You have pasted output of files from one node, have a check on both nodes to see that contents of file are same.. check following:

/etc/vxfenmode

/etc/vxfendg

/etc/vxfentab  ---- (don't go with just disk names, you should see with serial number that disks are same, you can use "vxdmpinq or vxscsiinq" commands for this located in /etc/vx/diag.d ) file should have same disks visible from both nodes.

 

G