solution needed for vxfen issue

[symsonu]

there is a two node cluster and we split two node cluster for upgrade. The isolated node is not coming up as vxfen is not starting

/02/01 11:52:05 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2013/02/01 11:52:20 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2013/02/01 11:52:35 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2013/02/01 11:52:50 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2013/02/01 11:53:05 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2013/02/01 11:53:20 VCS CRITICAL V-16-1-10031 VxFEN driver not configured. VCS Stopping. Manually restart VCS after configuring fencing
^C

IOFENCING configuration seems okay on node 2 as /etc/vxfentab has the entry for  co-ordinator disks and /etc/vxfendg has diskgroup entry vxfendg2 and these disks and diskgroup are visible too.

DEVICE       TYPE            DISK         GROUP        STATUS
c0t5006016047201339d0s2 auto:sliced     -            (ossdg)      online
c0t5006016047201339d1s2 auto:sliced     -            (sybasedg)   online
c0t5006016047201339d2s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d3s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d4s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d5s2 auto:sliced     -            (vxfendg2)   online
c0t5006016047201339d6s2 auto:sliced     -            (vxfendg2)   online
c0t5006016047201339d7s2 auto:sliced     -            -            online
c2t0d0s2     auto:SVM        -            -            SVM
c2t1d0s2     auto:SVM        -            -            SVM

On checking vxfen.log

nvoked S97vxfen. Starting
Fri Feb  1 11:50:37 CET 2013 starting vxfen.. 
Fri Feb  1 11:50:37 CET 2013 calling start_fun
Fri Feb  1 11:50:38 CET 2013 found vxfenmode file
Fri Feb  1 11:50:38 CET 2013 calling generate_vxfentab
Fri Feb  1 11:50:38 CET 2013 checking for /etc/vxfendg
Fri Feb  1 11:50:38 CET 2013 found  /etc/vxfendg.
Fri Feb  1 11:50:38 CET 2013 calling generate_disklist
Fri Feb  1 11:50:38 CET 2013 Starting vxfen.. Done. 
Fri Feb  1 11:50:38 CET 2013 starting in vxfen-startup
Fri Feb  1 11:50:38 CET 2013 calling regular vxfenconfig
Fri Feb  1 11:50:38 CET 2013 return value from above operation is 1
Fri Feb  1 11:50:38 CET 2013 output was VXFEN vxfenconfig ERROR V-11-2-1003 At least three coordinator disks must be defined
Log Buffer: 0xfffffffff4041090


refadm2-oss1{root} # cat /etc/vxfendg
vxfendg2

and there are below mentioned two disks in vxfendg2

c0t5006016047201339d5s2 auto:sliced     -            (vxfendg2)   online
c0t5006016047201339d6s2 auto:sliced     -            (vxfendg2)   online

is it due to two  disks in coordinator diskgroup? Is it a known issue ?

[arangari]

could you please provide output of 'gabconfig -a'  from both the nodes?

[symsonu]

refadm2-oss1{root} # gabconfig -a
GAB Port Memberships
===============================================================
Port a gen   d22e01 membership 0

refadm2-oss1{root} # gabconfig -a
GAB Port Memberships
===============================================================
Port a gen   d22e01 membership 0

Both the nodes are split and cannot see eachother.

[mikebounds]

The coordinator diskgroup must have at least 3 disks.  It is ok if a node can only see 2 of the 3 disks, must the diskgroup must contain 3 disks, and it looks as though vxfendg only contains 2 disks.

What version of VCS is this as with upgrades FROM 5.1 or higher you can do rolling upgrades, which means you do not need to split cluster - you can upgrade one node and VCS on the other node can still communicate with the other node even though it is on a different version.

Mike

[symsonu]

version is 5.1SP1RP2P2.

how can i add the third disk in the coordinator diskgroup?

[symsonu]

I am trying to add   c0t5006016047201339d7s2 in vxfendg2 by importing vxfendg2.but it failed.

vxdisk -o alldgs list
DEVICE       TYPE            DISK         GROUP        STATUS
c0t5006016047201339d0s2 auto:sliced     -            (ossdg)      online
c0t5006016047201339d1s2 auto:sliced     -            (sybasedg)   online
c0t5006016047201339d2s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d3s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d4s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d5s2 auto:sliced     -            (vxfendg2)   online
c0t5006016047201339d6s2 auto:sliced     -            (vxfendg2)   online
c0t5006016047201339d7s2 auto:sliced     -            -            online
c2t0d0s2     auto:SVM        -            -            SVM
c2t1d0s2     auto:SVM        -            -            SVM
refadm2-oss1{root} # vxdg -t import vxfendg2
VxVM vxdg ERROR V-5-1-10978 Disk group vxfendg2: import failed:  
Disk for disk group not found
refadm2-oss1{root} #  
 

[mikebounds]

You could try to import with "-f" to see if there is just one disk that can't be seen or if it is all the disks.

Mike

[symsonu]

refadm2-oss1{root} # vxdg -t import vxfendg2
VxVM vxdg ERROR V-5-1-10978 Disk group vxfendg2: import failed:  
Disk for disk group not found
refadm2-oss1{root} # vxdg -tfC import vxfendg2
VxVM vxdg WARNING V-5-1-560 Disk c0t5006016C47201339d7: Not found, last known location: c0t5006016047201339d7s2
refadm2-oss1{root} # vxdg list
NAME         STATE           ID
vxfendg2     enabled              1340186515.29.refadm1-oss1
refadm2-oss1{root} # vxdisk -o alldgs list
DEVICE       TYPE            DISK         GROUP        STATUS
c0t5006016047201339d0s2 auto:sliced     -            (ossdg)      online
c0t5006016047201339d1s2 auto:sliced     -            (sybasedg)   online
c0t5006016047201339d2s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d3s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d4s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d5s2 auto:sliced     c0t5006016C47201339d5  vxfendg2     online
c0t5006016047201339d6s2 auto:sliced     c0t5006016C47201339d6  vxfendg2     online
c0t5006016047201339d7s2 auto:sliced     -            -            online
c2t0d0s2     auto:SVM        -            -            SVM
c2t1d0s2     auto:SVM        -            -            SVM
-            -         c0t5006016C47201339d7 vxfendg2     failed was:c0t5006016047201339d7s2
refadm2-oss1{root} #  
 

[symsonu]

Can you pls suggest me the action plan ?

[Marianne]

Try 'vxreattach' command.

See http://www.symantec.com/docs/TECH175491

[symsonu]

refadm2-oss1{root} # /etc/vx/bin/vxreattach -c c0t5006016C47201339d7
VxVM vxreattach ERROR V-5-2-238 No matching Volume Manager disk and device IDs found for c0t5006016047201339d7s2
refadm2-oss1{root} # /usr/lib/vxvm/bin/vxreattach  
refadm2-oss1{root} # vxdisk -o alldgs list
DEVICE       TYPE            DISK         GROUP        STATUS
c0t5006016047201339d0s2 auto:sliced     -            (ossdg)      online
c0t5006016047201339d1s2 auto:sliced     -            (sybasedg)   online
c0t5006016047201339d2s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d3s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d4s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d5s2 auto:sliced     c0t5006016C47201339d5  vxfendg2     online
c0t5006016047201339d6s2 auto:sliced     c0t5006016C47201339d6  vxfendg2     online
c0t5006016047201339d7s2 auto:sliced     -            -            online
c2t0d0s2     auto:SVM        -            -            SVM
c2t1d0s2     auto:SVM        -            -            SVM
-            -         c0t5006016C47201339d7 vxfendg2     failed was:c0t5006016047201339d7s2
refadm2-oss1{root} #  
 

[Marianne · Accepted solution]

Try: 

vxdg -k -g <group> adddisk <disk>=<device> 

vxdg -k -g vxfendg2 adddisk c0t5006016047201339d7s2=c0t5006016047201339d7s2

(We don't normally use Device name for Disk name, but it seems this is what you've done for other disks in vxfendg2).

[AHerr]

You could: 1) disable fencing (/opt/VRTS/install/installvcs -fencing) and then enable fencing. 2) run vxfenswap to change fencing devices (https://sort.symantec.com/public/documents/sfha/6.0.1/linux/productguides/html/sfsybasece_admin/ch02s03s05.htm)

[symsonu]

Hello,

what does -k  option specifies in  below coomand

vxdg -k -g vxfendg2 adddisk c0t5006016047201339d7s2=c0t5006016047201339d7s2

I mean the use of -k option please.

[g_lee]

re: vxdg -k option - from manpage:

https://sort.symantec.com/public/documents/sfha/5.1sp1/solaris/manualpages/html/manpages/volume_manager/html/man1m/vxdg.1m.html

----------
KEYWORDS

adddisk

[...]

If the -k option is specified, the disk media name must represent a disk media record that was previously dissociated from its disk access record with -k rmdisk; otherwise, a new disk media record is created to represent the disk. With the -k option, plexes requiring recovery are flagged as stale.
----------

minor amendment to Marianne's command as the other dmnames don't use s2:

# vxdg -g vxfendg2 -k adddisk c0t5006016047201339d7=c0t5006016047201339d7s2

If the adddisk doesn't work, please provide the output from:

# vxdisk list c0t5006016047201339d7s2

[symsonu]

I have added the disk with below command

vxdg -g vxfendg2 -o coordinator -kf adddisk c0t5006016C47201339d7=c0t5006016047201339d7s2.

a slight modification needed to add -o coordinator as its a coordinator diskgroup

Thanks all for the excellent support

[g_lee]

Please mark Marianne's post as solution since she pointed out the command.