cancel
Showing results forΒ 
Search instead forΒ 
Did you mean:Β 

Test disk with io fencing with vxfentsthdw

donypie
Level 4
Hi all, When I test disk with vxfentsthdw, I have a step in error. Could you tell me what I have to do to solve it ? ------------------------------------------------------------------------------------------------------------ /opt/VRTS/bin/vxfentsthdw VERITAS vxfentsthdw version 5.1.00.00-GA Linux The utility vxfentsthdw works on the two nodes of the cluster. The utility verifies that the shared storage one intends to use is configured to support I/O fencing. It issues a series of vxfenadm commands to setup SCSI-3 registrations on the disk, verifies the registrations on the disk, and removes the registrations from the disk. ******** WARNING!!!!!!!! ******** THIS UTILITY WILL DESTROY THE DATA ON THE DISK!! Do you still want to continue : [y/n] (default: n) y The logfile generated for vxfentsthdw is /var/VRTSvcs/log/vxfen/vxfentsthdw.log.1648 Enter the first node of the cluster: linsys33 Enter the second node of the cluster: linsys34 Enter the disk name to be checked for SCSI-3 PGR on node linsys33 in the format: for dmp: /dev/vx/rdmp/sdx for raw: /dev/sdx Make sure its the same disk as seen by nodes linsys33 and linsys34 /dev/vx/rdmp/disk_0 Enter the disk name to be checked for SCSI-3 PGR on node linsys34 in the format: for dmp: /dev/vx/rdmp/sdx for raw: /dev/sdx Make sure its the same disk as seen by nodes linsys33 and linsys34 /dev/vx/rdmp/disk_0 *************************************************************************** Testing linsys33 /dev/vx/rdmp/disk_0 linsys34 /dev/vx/rdmp/disk_0 Evaluate the disk before testing ........................ No Pre-existing keys RegisterIgnoreKeys on disk /dev/vx/rdmp/disk_0 from node linsys33 ...... Passed Verify registrations for disk /dev/vx/rdmp/disk_0 on node linsys33 ..... Passed RegisterIgnoreKeys on disk /dev/vx/rdmp/disk_0 from node linsys34 ...... Passed Verify registrations for disk /dev/vx/rdmp/disk_0 on node linsys34 ..... Passed Unregister keys on disk /dev/vx/rdmp/disk_0 from node linsys33 ......... Passed Verify registrations for disk /dev/vx/rdmp/disk_0 on node linsys34 ..... Passed Unregister keys on disk /dev/vx/rdmp/disk_0 from node linsys34 ......... Passed RegisterIgnoreKeys on disk /dev/vx/rdmp/disk_0 from node linsys33 ...... Passed Verify registrations for disk /dev/vx/rdmp/disk_0 on node linsys33 ..... Passed Read from disk /dev/vx/rdmp/disk_0 on node linsys33 .................... Passed Write to disk /dev/vx/rdmp/disk_0 from node linsys33 ................... Passed Read from disk /dev/vx/rdmp/disk_0 on node linsys34 .................... Passed Write to disk /dev/vx/rdmp/disk_0 from node linsys34 ................... Passed Reserve disk /dev/vx/rdmp/disk_0 from node linsys33 .................... Passed Verify reservation for disk /dev/vx/rdmp/disk_0 on node linsys33 ....... Passed Read from disk /dev/vx/rdmp/disk_0 on node linsys33 .................... Passed Read from disk /dev/vx/rdmp/disk_0 on node linsys34 .................... Passed Write to disk /dev/vx/rdmp/disk_0 from node linsys33 ................... Passed Expect no writes for disk /dev/vx/rdmp/disk_0 on node linsys34 ......... Passed RegisterIgnoreKeys on disk /dev/vx/rdmp/disk_0 from node linsys34 ...... Passed Verify registrations for disk /dev/vx/rdmp/disk_0 on node linsys33 ..... Passed Verify registrations for disk /dev/vx/rdmp/disk_0 on node linsys34 ..... Passed Write to disk /dev/vx/rdmp/disk_0 from node linsys33 ................... Passed Write to disk /dev/vx/rdmp/disk_0 from node linsys34 ................... Passed Preempt and abort key KeyA using key KeyB on node linsys34 ............. Passed Test to see if I/O on node linsys33 terminated ......................... Passed RegisterIgnoreKeys on disk /dev/vx/rdmp/disk_0 from node linsys33 ...... Passed Verify registrations for disk /dev/vx/rdmp/disk_0 on node linsys33 ..... Passed Preempt key KeyC using key KeyB on node linsys34 ....................... Passed Test to see if I/O on node linsys33 terminated ......................... Passed Verify registrations for disk /dev/vx/rdmp/disk_0 on node linsys33 ..... Passed Verify registrations for disk /dev/vx/rdmp/disk_0 on node linsys34 ..... Passed Verify reservation for disk /dev/vx/rdmp/disk_0 on node linsys33 ....... Passed Verify reservation for disk /dev/vx/rdmp/disk_0 on node linsys34 ....... Passed Remove key KeyB on node linsys34 ....................................... Passed Check to verify there are no keys from node linsys33 ................... Passed Check to verify there are no keys from node linsys34 ................... Passed RegisterIgnoreKeys on disk /dev/vx/rdmp/disk_0 from node linsys33 ...... Passed Verify registrations for disk /dev/vx/rdmp/disk_0 on node linsys33 ..... Passed RegisterIgnoreKeys on disk /dev/vx/rdmp/disk_0 from node linsys33 ...... Passed Verify registrations for disk /dev/vx/rdmp/disk_0 on node linsys33 ..... Passed Clear PGR on node linsys33 ............................................. Passed Check to verify there are no keys from node linsys33 ................... Failed Removing test keys and temporary files, if any... ------------------------------------------------------------------------------------------------------- When I check the log file, I have : [root@linsys33 rhel5_x86_64]# view /var/VRTSvcs/log/vxfen/vxfentsthdw.log.1648 Thu Jun 10 16:04:51 CEST 2010 write_disk() passed for /dev/vx/rdmp/disk_0\n Thu Jun 10 16:04:53 CEST 2010 nowrite_disk() passed for /dev/vx/rdmp/disk_0\n Thu Jun 10 16:04:53 CEST 2010 registerignore() passed for /dev/vx/rdmp/disk_0\n Thu Jun 10 16:04:54 CEST 2010 chk_registrations() passed for /dev/vx/rdmp/disk_0\n Thu Jun 10 16:04:55 CEST 2010 chk_registrations() passed for /dev/vx/rdmp/disk_0\n Thu Jun 10 16:05:03 CEST 2010 write_disk() passed for /dev/vx/rdmp/disk_0\n Thu Jun 10 16:05:05 CEST 2010 write_disk() passed for /dev/vx/rdmp/disk_0\n Thu Jun 10 16:05:27 CEST 2010 preempt_abort() passed for linsys34\n Thu Jun 10 16:05:29 CEST 2010 chk_dd_process() failed for linsys33\n Thu Jun 10 16:05:31 CEST 2010 registerignore() passed for /dev/vx/rdmp/disk_0\n Thu Jun 10 16:05:32 CEST 2010 chk_registrations() passed for /dev/vx/rdmp/disk_0\n Thu Jun 10 16:05:55 CEST 2010 preempt() passed for linsys34\n Thu Jun 10 16:05:57 CEST 2010 chk_dd_process() failed for linsys33\n Thu Jun 10 16:05:58 CEST 2010 chk_registrations() passed for /dev/vx/rdmp/disk_0\n Thu Jun 10 16:05:58 CEST 2010 chk_registrations() passed for /dev/vx/rdmp/disk_0\n Thu Jun 10 16:05:59 CEST 2010 chk_reservations() passed for /dev/vx/rdmp/disk_0\n Thu Jun 10 16:05:59 CEST 2010 chk_reservations() passed for /dev/vx/rdmp/disk_0\n Thu Jun 10 16:05:59 CEST 2010 delete_keys() passed for linsys34\n Thu Jun 10 16:06:00 CEST 2010 chk_nokeys() passed for linsys33\n Thu Jun 10 16:06:00 CEST 2010 chk_nokeys() passed for linsys34\n Thu Jun 10 16:06:00 CEST 2010 registerignore() passed for /dev/vx/rdmp/disk_0\n Thu Jun 10 16:06:01 CEST 2010 chk_registrations() passed for /dev/vx/rdmp/disk_0\n Thu Jun 10 16:06:04 CEST 2010 registerignore() passed for /dev/vx/rdmp/disk_0\n Thu Jun 10 16:06:05 CEST 2010 chk_registrations() passed for /dev/vx/rdmp/disk_0\n Thu Jun 10 16:06:06 CEST 2010 clear() passed for linsys33\n Thu Jun 10 16:06:07 CEST 2010 chk_nokeys() failed for linsys33\n Thu Jun 10 16:06:07 CEST 2010 SCSI Registration Keys for linsys33 are..:- VXFEN vxfenadm WARNING V-11-2-2414 This option is deprecated and would be removed with the next release. Please use the -s option. Device Name: /dev/vx/rdmp/disk_0 Total Number Of Keys: 1 key[0]: Key Value [Numeric Format]: 75,101,121,66,45,45,45,45 Key Value [Character Format]: KeyB---- Thu Jun 10 16:06:08 CEST 2010 SCSI Reservation Keys for linsys33 are..:- Device Name: /dev/vx/rdmp/disk_0 Total Number Of Keys: 0 No keys... /dev/vx/rdmp/disk_0 Thu Jun 10 16:06:12 CEST 2010 SCSI Registration Keys for linsys34 are..:- VXFEN vxfenadm WARNING V-11-2-2414 This option is deprecated and would be removed with the next release. Please use the -s option. Device Name: /dev/vx/rdmp/disk_0 Total Number Of Keys: 1 key[0]: Key Value [Numeric Format]: 75,101,121,66,45,45,45,45 Key Value [Character Format]: KeyB---- Thu Jun 10 16:06:12 CEST 2010 SCSI Reservation Keys for linsys34 are..:- Device Name: /dev/vx/rdmp/disk_0 Total Number Of Keys: 0 No keys... /dev/vx/rdmp/disk_0 Thanks a lot for your help. Kind regards, Pierre.
1 ACCEPTED SOLUTION

Accepted Solutions

donypie
Level 4
Gaurav,

I've configured io fencing manually and it works :

[root@linsys33 config]# /opt/VRTS/bin/vxfenadm -d
 
I/O Fencing Cluster Information:
================================
 Fencing Protocol Version: 201
 Fencing Mode: SCSI3
 Fencing SCSI3 Disk Policy: dmp
 Cluster Members:
        * 0 (linsys33)
          1 (linsys34)
 RFSM State Information:
        node   0 in state  8 (running)
        node   1 in state  8 (running)

Kind regards,
Pierre.
 

View solution in original post

10 REPLIES 10

donypie
Level 4
I have to add that I use iscsi targets.

Thanks.
Pierre.

Gaurav_S
Moderator
Moderator
   VIP    Certified
Hi Donypie,

As far my knowledge, SF5.0 MP3 onwards iSCSI was supported & it looks like you are using version SF 5.1 on linux.. (please confirm if this is incorrect)

Looking at logs above, I somehow believe, the issue is only with linsys33 node

Write to disk /dev/vx/rdmp/disk_0 from node linsys33 ................... Passed
Write to disk /dev/vx/rdmp/disk_0 from node linsys34 ................... Passed
Preempt and abort key KeyA using key KeyB on node linsys34 ............. Passed    <<<<<<<<<<

Preempt key KeyC using key KeyB on node linsys34 ....................... Passed  <<<<<<<<<<<<<<
Test to see if I/O on node linsys33 terminated ......................... Passed

Remove key KeyB on node linsys34 ....................................... Passed <<<<<<<<<<<<<<<<

you can see the 3 highlighted steps above, all the key writing happend with linsys34 only. I would expect same steps to happen with linsys33 as well...  what I can suggest at this point is

a) confirm exact version of volume manager you are using.
b) carry the test once again, this time give first node as linsys34 & second node as linsys33 & paste the outputs.

Also, I am presuming that Disk_0 is the same disk across the nodes (I hope you have verified by the serial number of the LUN)

Gaurav


donypie
Level 4
Hi Gaurav,

Thanks a lot for your answer. Yes, I'm runnig SF 5.1 on a RedHat EL 5.5.  Disks are the same on each node :


[root@linsys33 ~]# /opt/VRTS/bin/vxfenadm -i /dev/vx/rdmp/disk_0
Vendor id       : ROCKET
Product id      : IMAGEFILE
Revision        : 0001
Serial Number   : 1780C3DEFC044C56
 
[root@linsys34 ~]#  /opt/VRTS/bin/vxfenadm -i /dev/vx/rdmp/disk_0
Vendor id       : ROCKET
Product id      : IMAGEFILE
Revision        : 0001
Serial Number   : 1780C3DEFC044C56

 
When I try to rerun the tools with the same disk, I receive an error message :

Testing linsys34 /dev/vx/rdmp/disk_0 linsys33 /dev/vx/rdmp/disk_0
Evaluate the disk before testing  ........................... Pre-ex
I/O fencing appears to be configured.
Please shutdown fencing then continue.

[root@linsys33 ~]# /etc/init.d/vxfen stop
Stopping vxfen..
ERROR: failed to stop vxfen
ERROR: Module vxfen is in use
VxFEN:Error: cannot unload module
 


Can you please tell me how I can stop vxfen ?

I've tried with another disk, disk_1 and i've specified this time linsys34 as first node and linsys33 as second node. I've now an error for linsys34 !.


[root@linsys33 bin]# /opt/VRTS/bin/vxfentsthdw
VERITAS vxfentsthdw version 5.1.00.00-GA Linux

The utility vxfentsthdw works on the two nodes of the cluster.
The utility verifies that the shared storage one intends to use is
configured to support I/O fencing.  It issues a series of vxfenadm
commands to setup SCSI-3 registrations on the disk, verifies the
registrations on the disk, and removes the registrations from the disk.

******** WARNING!!!!!!!! ********
THIS UTILITY WILL DESTROY THE DATA ON THE DISK!!
Do you still want to continue : [y/n] (default: n) y
The logfile generated for vxfentsthdw is /var/VRTSvcs/log/vxfen/vxfentsthdw.log.7809
Enter the first node of the cluster:
linsys34
Enter the second node of the cluster:
linsys33
Enter the disk name to be checked for SCSI-3 PGR on node linsys34 in the format:
        for dmp: /dev/vx/rdmp/sdx
        for raw: /dev/sdx
Make sure its the same disk as seen by nodes linsys34 and linsys33
/dev/vx/rdmp/disk_1
Enter the disk name to be checked for SCSI-3 PGR on node linsys33 in the format:
        for dmp: /dev/vx/rdmp/sdx
        for raw: /dev/sdx
Make sure its the same disk as seen by nodes linsys34 and linsys33
/dev/vx/rdmp/disk_1
***************************************************************************
Testing linsys34 /dev/vx/rdmp/disk_1 linsys33 /dev/vx/rdmp/disk_1
Evaluate the disk before testing  ........................ No Pre-existing keys
RegisterIgnoreKeys on disk /dev/vx/rdmp/disk_1 from node linsys34 ...... Passed
Verify registrations for disk /dev/vx/rdmp/disk_1 on node linsys34 ..... Passed
RegisterIgnoreKeys on disk /dev/vx/rdmp/disk_1 from node linsys33 ...... Passed
Verify registrations for disk /dev/vx/rdmp/disk_1 on node linsys33 ..... Passed
Unregister keys on disk /dev/vx/rdmp/disk_1 from node linsys34 ......... Passed
Verify registrations for disk /dev/vx/rdmp/disk_1 on node linsys33 ..... Passed
Unregister keys on disk /dev/vx/rdmp/disk_1 from node linsys33 ......... Passed
RegisterIgnoreKeys on disk /dev/vx/rdmp/disk_1 from node linsys34 ...... Passed
Verify registrations for disk /dev/vx/rdmp/disk_1 on node linsys34 ..... Passed
Read from disk /dev/vx/rdmp/disk_1 on node linsys34 .................... Passed
Write to disk /dev/vx/rdmp/disk_1 from node linsys34 ................... Passed
Read from disk /dev/vx/rdmp/disk_1 on node linsys33 .................... Passed
Write to disk /dev/vx/rdmp/disk_1 from node linsys33 ................... Passed
Reserve disk /dev/vx/rdmp/disk_1 from node linsys34 .................... Passed
Verify reservation for disk /dev/vx/rdmp/disk_1 on node linsys34 ....... Passed
Read from disk /dev/vx/rdmp/disk_1 on node linsys34 .................... Passed
Read from disk /dev/vx/rdmp/disk_1 on node linsys33 .................... Passed
Write to disk /dev/vx/rdmp/disk_1 from node linsys34 ................... Passed
Expect no writes for disk /dev/vx/rdmp/disk_1 on node linsys33 ......... Passed
RegisterIgnoreKeys on disk /dev/vx/rdmp/disk_1 from node linsys33 ...... Passed
Verify registrations for disk /dev/vx/rdmp/disk_1 on node linsys34 ..... Passed
Verify registrations for disk /dev/vx/rdmp/disk_1 on node linsys33 ..... Passed
Write to disk /dev/vx/rdmp/disk_1 from node linsys34 ................... Passed
Write to disk /dev/vx/rdmp/disk_1 from node linsys33 ................... Passed
Preempt and abort key KeyA using key KeyB on node linsys33 ............. Passed
Test to see if I/O on node linsys34 terminated ......................... Passed
RegisterIgnoreKeys on disk /dev/vx/rdmp/disk_1 from node linsys34 ...... Passed
Verify registrations for disk /dev/vx/rdmp/disk_1 on node linsys34 ..... Passed
Preempt key KeyC using key KeyB on node linsys33 ....................... Passed
Test to see if I/O on node linsys34 terminated ......................... Passed
Verify registrations for disk /dev/vx/rdmp/disk_1 on node linsys34 ..... Passed
Verify registrations for disk /dev/vx/rdmp/disk_1 on node linsys33 ..... Passed
Verify reservation for disk /dev/vx/rdmp/disk_1 on node linsys34 ....... Passed
Verify reservation for disk /dev/vx/rdmp/disk_1 on node linsys33 ....... Passed
Remove key KeyB on node linsys33 ....................................... Passed
Check to verify there are no keys from node linsys34 ................... Passed
Check to verify there are no keys from node linsys33 ................... Passed
RegisterIgnoreKeys on disk /dev/vx/rdmp/disk_1 from node linsys34 ...... Passed
Verify registrations for disk /dev/vx/rdmp/disk_1 on node linsys34 ..... Passed
RegisterIgnoreKeys on disk /dev/vx/rdmp/disk_1 from node linsys34 ...... Passed
Verify registrations for disk /dev/vx/rdmp/disk_1 on node linsys34 ..... Passed
Clear PGR on node linsys34 ............................................. Passed
Check to verify there are no keys from node linsys34 ................... Failed
Removing test keys and temporary files, if any...


What can I do to solve this problem ?  I'm sure my iscsi targets are "scsi-3 pr" compatible.

Thanks for your help.
Kind regards,
Pierre.

 

Gaurav_S
Moderator
Moderator
   VIP    Certified
Hi Pierre,

To answer first part:

a) To stop vxfen module, have you stopped all clients above vxfen ? for e.g, port "h" HAD is a client. You can only stop fencing once all other ports except port "a" GAB are stopped. So you will need to stop cluster using "hastop" (with appropriate option) before you can stop fencing. If you are running CFS/CVM OR SFRAC, you will need to stop other ports as well i.e ODM, VCSMM, CVM, VXFSCKD etc.

b) Regarding actual problem, we will need to dig more.... During this test it errored for linsys34. can you check whether actually any keys are lying on the disk ?

# vxfenadm -g /dev/vx/rdsk/Disk_1
# vxfenadm -r /dev/vx/rdsk/Disk_1

Also can you paste

# cat /etc/vxfenmode

Gaurav

donypie
Level 4
Gaurav,

Find here answers :

[root@linsys33 bin]# /opt/VRTS/bin/vxfenadm -g /dev/vx/rdmp/disk_1
Reading SCSI Registration Keys...
Device Name: /dev/vx/rdmp/disk_1
Total Number Of Keys: 1
key[0]:
        Key Value [Numeric Format]:  69,114,97,115,101,114,45,45
        Key Value [Character Format]: Eraser--
 /opt/VRTS/bin/vxfenadm -g /dev/vx/rdmp/disk_1
VXFEN vxfenadm WARNING V-11-2-2414 This option is deprecated and would be removed with the next release.
Please use the -s option.

[root@linsys33 bin]# /opt/VRTS/bin/vxfenadm -r /dev/vx/rdmp/disk_1
Reading SCSI Reservation Information...
Device Name: /dev/vx/rdmp/disk_1
Total Number Of Keys: 0
No keys...

[root@linsys33 bin]# cat /etc/vxfenmode
#
# vxfen_mode determines in what mode VCS I/O Fencing should work.
#
# available options:
# scsi3      - use scsi3 persistent reservation disks
# customized - use script based customized fencing
# sybase     - use scsi3 disks in kernel but coordinate membership with Sybase ASE
# disabled   - run the driver but don't do any actual fencing
#
vxfen_mode=disabled



Thanks a lot.
Regards,
Pierre.
 

Gaurav_S
Moderator
Moderator
   VIP    Certified

Hi There...

So 2 things...

For sure vxfentsthdw is having issues, it should have cleared the "Eraser" key which it has not.... so for sure there is some issue....

Secondly currently you are running with vxfen_mode=disabled, you would need to copy /etc/vxfen.d/vxfenmode_dmp OR vxfenmode_raw to /etc/vxfenmode (depending on which mode you want to use), for details on how to setup fencing, refer VCS user's guide

https://vos.symantec.com/documents

anyways vxfenmode is irrespective of vxfentsthdw, it should still pass all the tests.

Well to be honest, it wouldn't be recommended to use iofencing unless vxfentsthdw passes all the tests... why is it not clearing the last "Eraser" key, not sure..... I would recommend to escalate the case to Symantec support as this may need additional debugging...

Still if you want to try something at your end, I would recommend following:

a) Check Hardware Compatbility list (HCL) for 5.1 & ensure that components you are using are supported (storage, FC card etc), HCL can be found at link given above.
b) If it is under supported matrix, check array settings from HCL to ensure array modes are setup correctly.
c) If you find array mode is correct, make sure LUNS are correctly masked & all SAN/ISCSI/port settings are good.
d) You can try provisioning a  fresh 100MB test lun & run tests on that & see if it succeeds.


Gaurav

donypie
Level 4
Gaurav,

Ok I will try it without checking disk with vxfentsthdw. I will update this post with my results in a few days.

Thanks.
Have  a nice week-end
Regards,
Pierre.

 

donypie
Level 4
Gaurav,

I've configured io fencing manually and it works :

[root@linsys33 config]# /opt/VRTS/bin/vxfenadm -d
 
I/O Fencing Cluster Information:
================================
 Fencing Protocol Version: 201
 Fencing Mode: SCSI3
 Fencing SCSI3 Disk Policy: dmp
 Cluster Members:
        * 0 (linsys33)
          1 (linsys34)
 RFSM State Information:
        node   0 in state  8 (running)
        node   1 in state  8 (running)

Kind regards,
Pierre.
 

Gaurav_S
Moderator
Moderator
   VIP    Certified

Hi Pierre,

Good to hear that its working...

Did you carried out failover testing by failing both LLT links ?


Gaurav

donypie
Level 4
Hi Gaurav,

Not yet but that will be the following test :)

It will be a good idea to document what I'm doing in a blog (create a vcs cluster with virtualbox VM's, iscsi and global cluster option).

Pierre.