cancel
Showing results for 
Search instead for 
Did you mean: 

solution needed for vxfen issue

symsonu
Level 6

 

there is a two node cluster and we split two node cluster for upgrade. The isolated node is not coming up as vxfen is not starting



/02/01 11:52:05 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2013/02/01 11:52:20 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2013/02/01 11:52:35 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2013/02/01 11:52:50 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2013/02/01 11:53:05 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2013/02/01 11:53:20 VCS CRITICAL V-16-1-10031 VxFEN driver not configured. VCS Stopping. Manually restart VCS after configuring fencing
^C


IOFENCING configuration seems okay on node 2 as /etc/vxfentab has the entry for  co-ordinator disks and /etc/vxfendg has diskgroup entry vxfendg2 and these disks and diskgroup are visible too.

DEVICE       TYPE            DISK         GROUP        STATUS
c0t5006016047201339d0s2 auto:sliced     -            (ossdg)      online
c0t5006016047201339d1s2 auto:sliced     -            (sybasedg)   online
c0t5006016047201339d2s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d3s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d4s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d5s2 auto:sliced     -            (vxfendg2)   online
c0t5006016047201339d6s2 auto:sliced     -            (vxfendg2)   online
c0t5006016047201339d7s2 auto:sliced     -            -            online
c2t0d0s2     auto:SVM        -            -            SVM
c2t1d0s2     auto:SVM        -            -            SVM

On checking vxfen.log

 

nvoked S97vxfen. Starting
Fri Feb  1 11:50:37 CET 2013 starting vxfen.. 
Fri Feb  1 11:50:37 CET 2013 calling start_fun
Fri Feb  1 11:50:38 CET 2013 found vxfenmode file
Fri Feb  1 11:50:38 CET 2013 calling generate_vxfentab
Fri Feb  1 11:50:38 CET 2013 checking for /etc/vxfendg
Fri Feb  1 11:50:38 CET 2013 found  /etc/vxfendg.
Fri Feb  1 11:50:38 CET 2013 calling generate_disklist
Fri Feb  1 11:50:38 CET 2013 Starting vxfen.. Done. 
Fri Feb  1 11:50:38 CET 2013 starting in vxfen-startup
Fri Feb  1 11:50:38 CET 2013 calling regular vxfenconfig
Fri Feb  1 11:50:38 CET 2013 return value from above operation is 1
Fri Feb  1 11:50:38 CET 2013
output was VXFEN vxfenconfig ERROR V-11-2-1003 At least three coordinator disks must be defined
Log Buffer: 0xfffffffff4041090


refadm2-oss1{root} # cat /etc/vxfendg
vxfendg2

and there are below mentioned two disks in vxfendg2

c0t5006016047201339d5s2 auto:sliced     -            (vxfendg2)   online
c0t5006016047201339d6s2 auto:sliced     -            (vxfendg2)   online

 

is it due to two  disks in coordinator diskgroup? Is it a known issue ?

 


 

1 ACCEPTED SOLUTION

Accepted Solutions

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Try: 

vxdg -k -g <group> adddisk <disk>=<device> 

vxdg -k -g vxfendg2 adddisk c0t5006016047201339d7s2=c0t5006016047201339d7s2

(We don't normally use Device name for Disk name, but it seems this is what you've done for other disks in vxfendg2).

View solution in original post

16 REPLIES 16

arangari
Level 5

could you please provide output of 'gabconfig -a'  from both the nodes?

 

 

 

symsonu
Level 6

refadm2-oss1{root} # gabconfig -a
GAB Port Memberships
===============================================================
Port a gen   d22e01 membership 0

refadm2-oss1{root} # gabconfig -a
GAB Port Memberships
===============================================================
Port a gen   d22e01 membership 0

Both the nodes are split and cannot see eachother.

mikebounds
Level 6
Partner Accredited

The coordinator diskgroup must have at least 3 disks.  It is ok if a node can only see 2 of the 3 disks, must the diskgroup must contain 3 disks, and it looks as though vxfendg only contains 2 disks.

 

What version of VCS is this as with upgrades FROM 5.1 or higher you can do rolling upgrades, which means you do not need to split cluster - you can upgrade one node and VCS on the other node can still communicate with the other node even though it is on a different version.

Mike

symsonu
Level 6

version is 5.1SP1RP2P2.

 

how can i add the third disk in the coordinator diskgroup?

symsonu
Level 6

I am trying to add   c0t5006016047201339d7s2 in vxfendg2 by importing vxfendg2.but it failed.

vxdisk -o alldgs list
DEVICE       TYPE            DISK         GROUP        STATUS
c0t5006016047201339d0s2 auto:sliced     -            (ossdg)      online
c0t5006016047201339d1s2 auto:sliced     -            (sybasedg)   online
c0t5006016047201339d2s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d3s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d4s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d5s2 auto:sliced     -            (vxfendg2)   online
c0t5006016047201339d6s2 auto:sliced     -            (vxfendg2)   online
c0t5006016047201339d7s2 auto:sliced     -            -            online
c2t0d0s2     auto:SVM        -            -            SVM
c2t1d0s2     auto:SVM        -            -            SVM
refadm2-oss1{root} # vxdg -t import vxfendg2
VxVM vxdg ERROR V-5-1-10978 Disk group vxfendg2: import failed:  
Disk for disk group not found
refadm2-oss1{root} #  
 

mikebounds
Level 6
Partner Accredited

You could try to import with "-f" to see if there is just one disk that can't be seen or if it is all the disks.

Mike

symsonu
Level 6

 

 

refadm2-oss1{root} # vxdg -t import vxfendg2
VxVM vxdg ERROR V-5-1-10978 Disk group vxfendg2: import failed:  
Disk for disk group not found
refadm2-oss1{root} # vxdg -tfC import vxfendg2
VxVM vxdg WARNING V-5-1-560 Disk c0t5006016C47201339d7: Not found, last known location: c0t5006016047201339d7s2
refadm2-oss1{root} # vxdg list
NAME         STATE           ID
vxfendg2     enabled              1340186515.29.refadm1-oss1
refadm2-oss1{root} # vxdisk -o alldgs list
DEVICE       TYPE            DISK         GROUP        STATUS
c0t5006016047201339d0s2 auto:sliced     -            (ossdg)      online
c0t5006016047201339d1s2 auto:sliced     -            (sybasedg)   online
c0t5006016047201339d2s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d3s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d4s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d5s2 auto:sliced     c0t5006016C47201339d5  vxfendg2     online
c0t5006016047201339d6s2 auto:sliced     c0t5006016C47201339d6  vxfendg2     online
c0t5006016047201339d7s2 auto:sliced     -            -            online
c2t0d0s2     auto:SVM        -            -            SVM
c2t1d0s2     auto:SVM        -            -            SVM
-            -         c0t5006016C47201339d7 vxfendg2     failed was:c0t5006016047201339d7s2
refadm2-oss1{root} #  
 

symsonu
Level 6

Can you pls suggest me the action plan ?

 

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

symsonu
Level 6

refadm2-oss1{root} # /etc/vx/bin/vxreattach -c c0t5006016C47201339d7
VxVM vxreattach ERROR V-5-2-238 No matching Volume Manager disk and device IDs found for c0t5006016047201339d7s2
refadm2-oss1{root} # /usr/lib/vxvm/bin/vxreattach  
refadm2-oss1{root} # vxdisk -o alldgs list
DEVICE       TYPE            DISK         GROUP        STATUS
c0t5006016047201339d0s2 auto:sliced     -            (ossdg)      online
c0t5006016047201339d1s2 auto:sliced     -            (sybasedg)   online
c0t5006016047201339d2s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d3s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d4s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d5s2 auto:sliced     c0t5006016C47201339d5  vxfendg2     online
c0t5006016047201339d6s2 auto:sliced     c0t5006016C47201339d6  vxfendg2     online
c0t5006016047201339d7s2 auto:sliced     -            -            online
c2t0d0s2     auto:SVM        -            -            SVM
c2t1d0s2     auto:SVM        -            -            SVM
-            -         c0t5006016C47201339d7 vxfendg2     failed was:c0t5006016047201339d7s2
refadm2-oss1{root} #  
 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Try: 

vxdg -k -g <group> adddisk <disk>=<device> 

vxdg -k -g vxfendg2 adddisk c0t5006016047201339d7s2=c0t5006016047201339d7s2

(We don't normally use Device name for Disk name, but it seems this is what you've done for other disks in vxfendg2).

AHerr
Level 5
Employee Accredited Certified
You could: 1) disable fencing (/opt/VRTS/install/installvcs -fencing) and then enable fencing. 2) run vxfenswap to change fencing devices (https://sort.symantec.com/public/documents/sfha/6.0.1/linux/productguides/html/sfsybasece_admin/ch02s03s05.htm)

symsonu
Level 6

Hello,

 

what does -k  option specifies in  below coomand

 

vxdg -k -g vxfendg2 adddisk c0t5006016047201339d7s2=c0t5006016047201339d7s2

 

I mean the use of -k option please.

 

 

g_lee
Level 6

re: vxdg -k option - from manpage:

https://sort.symantec.com/public/documents/sfha/5.1sp1/solaris/manualpages/html/manpages/volume_manager/html/man1m/vxdg.1m.html

----------
KEYWORDS

adddisk

[...]

If the -k option is specified, the disk media name must represent a disk media record that was previously dissociated from its disk access record with -k rmdisk; otherwise, a new disk media record is created to represent the disk. With the -k option, plexes requiring recovery are flagged as stale.
----------

minor amendment to Marianne's command as the other dmnames don't use s2:

# vxdg -g vxfendg2 -k adddisk c0t5006016047201339d7=c0t5006016047201339d7s2

If the adddisk doesn't work, please provide the output from:

# vxdisk list c0t5006016047201339d7s2

symsonu
Level 6

I have added the disk with below command

 

vxdg -g vxfendg2 -o coordinator -kf adddisk c0t5006016C47201339d7=c0t5006016047201339d7s2.

 

a slight modification needed to add -o coordinator as its a coordinator diskgroup

Thanks all for the excellent support

 

 

g_lee
Level 6

Please mark Marianne's post as solution since she pointed out the command.