Forum Discussion

symsonu's avatar
symsonu
Level 6
12 years ago

solution needed for vxfen issue

 

there is a two node cluster and we split two node cluster for upgrade. The isolated node is not coming up as vxfen is not starting



/02/01 11:52:05 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2013/02/01 11:52:20 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2013/02/01 11:52:35 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2013/02/01 11:52:50 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2013/02/01 11:53:05 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2013/02/01 11:53:20 VCS CRITICAL V-16-1-10031 VxFEN driver not configured. VCS Stopping. Manually restart VCS after configuring fencing
^C

IOFENCING configuration seems okay on node 2 as /etc/vxfentab has the entry for  co-ordinator disks and /etc/vxfendg has diskgroup entry vxfendg2 and these disks and diskgroup are visible too.

DEVICE       TYPE            DISK         GROUP        STATUS
c0t5006016047201339d0s2 auto:sliced     -            (ossdg)      online
c0t5006016047201339d1s2 auto:sliced     -            (sybasedg)   online
c0t5006016047201339d2s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d3s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d4s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d5s2 auto:sliced     -            (vxfendg2)   online
c0t5006016047201339d6s2 auto:sliced     -            (vxfendg2)   online
c0t5006016047201339d7s2 auto:sliced     -            -            online
c2t0d0s2     auto:SVM        -            -            SVM
c2t1d0s2     auto:SVM        -            -            SVM

On checking vxfen.log

 

nvoked S97vxfen. Starting
Fri Feb  1 11:50:37 CET 2013 starting vxfen.. 
Fri Feb  1 11:50:37 CET 2013 calling start_fun
Fri Feb  1 11:50:38 CET 2013 found vxfenmode file
Fri Feb  1 11:50:38 CET 2013 calling generate_vxfentab
Fri Feb  1 11:50:38 CET 2013 checking for /etc/vxfendg
Fri Feb  1 11:50:38 CET 2013 found  /etc/vxfendg.
Fri Feb  1 11:50:38 CET 2013 calling generate_disklist
Fri Feb  1 11:50:38 CET 2013 Starting vxfen.. Done. 
Fri Feb  1 11:50:38 CET 2013 starting in vxfen-startup
Fri Feb  1 11:50:38 CET 2013 calling regular vxfenconfig
Fri Feb  1 11:50:38 CET 2013 return value from above operation is 1
Fri Feb  1 11:50:38 CET 2013 output was VXFEN vxfenconfig ERROR V-11-2-1003 At least three coordinator disks must be defined
Log Buffer: 0xfffffffff4041090


refadm2-oss1{root} # cat /etc/vxfendg
vxfendg2

and there are below mentioned two disks in vxfendg2

c0t5006016047201339d5s2 auto:sliced     -            (vxfendg2)   online
c0t5006016047201339d6s2 auto:sliced     -            (vxfendg2)   online

 

is it due to two  disks in coordinator diskgroup? Is it a known issue ?

 


 

business continuity
Cluster Server
Error messages
Troubleshooting
  • Marianne's avatar
    Marianne
    12 years ago

    Try: 

    vxdg -k -g <group> adddisk <disk>=<device> 

    vxdg -k -g vxfendg2 adddisk c0t5006016047201339d7s2=c0t5006016047201339d7s2

    (We don't normally use Device name for Disk name, but it seems this is what you've done for other disks in vxfendg2).

  • arangari's avatar
    arangari
    Level 5
    12 years ago

    could you please provide output of 'gabconfig -a'  from both the nodes?

     

     

     

  • symsonu's avatar
    symsonu
    Level 6
    12 years ago

    refadm2-oss1{root} # gabconfig -a
    GAB Port Memberships
    ===============================================================
    Port a gen   d22e01 membership 0

    refadm2-oss1{root} # gabconfig -a
    GAB Port Memberships
    ===============================================================
    Port a gen   d22e01 membership 0

    Both the nodes are split and cannot see eachother.

  • mikebounds's avatar
    mikebounds
    Level 6
    12 years ago

    The coordinator diskgroup must have at least 3 disks.  It is ok if a node can only see 2 of the 3 disks, must the diskgroup must contain 3 disks, and it looks as though vxfendg only contains 2 disks.

     

    What version of VCS is this as with upgrades FROM 5.1 or higher you can do rolling upgrades, which means you do not need to split cluster - you can upgrade one node and VCS on the other node can still communicate with the other node even though it is on a different version.

    Mike

  • symsonu's avatar
    symsonu
    Level 6
    12 years ago

    version is 5.1SP1RP2P2.

     

    how can i add the third disk in the coordinator diskgroup?

  • symsonu's avatar
    symsonu
    Level 6
    12 years ago

    I am trying to add   c0t5006016047201339d7s2 in vxfendg2 by importing vxfendg2.but it failed.

    vxdisk -o alldgs list
    DEVICE       TYPE            DISK         GROUP        STATUS
    c0t5006016047201339d0s2 auto:sliced     -            (ossdg)      online
    c0t5006016047201339d1s2 auto:sliced     -            (sybasedg)   online
    c0t5006016047201339d2s2 auto:sliced     -            (vxfendg1)   online
    c0t5006016047201339d3s2 auto:sliced     -            (vxfendg1)   online
    c0t5006016047201339d4s2 auto:sliced     -            (vxfendg1)   online
    c0t5006016047201339d5s2 auto:sliced     -            (vxfendg2)   online
    c0t5006016047201339d6s2 auto:sliced     -            (vxfendg2)   online
    c0t5006016047201339d7s2 auto:sliced     -            -            online
    c2t0d0s2     auto:SVM        -            -            SVM
    c2t1d0s2     auto:SVM        -            -            SVM
    refadm2-oss1{root} # vxdg -t import vxfendg2
    VxVM vxdg ERROR V-5-1-10978 Disk group vxfendg2: import failed:  
    Disk for disk group not found
    refadm2-oss1{root} #  
     

  • mikebounds's avatar
    mikebounds
    Level 6
    12 years ago

    You could try to import with "-f" to see if there is just one disk that can't be seen or if it is all the disks.

    Mike

  • symsonu's avatar
    symsonu
    Level 6
    12 years ago

     

     

    refadm2-oss1{root} # vxdg -t import vxfendg2
    VxVM vxdg ERROR V-5-1-10978 Disk group vxfendg2: import failed:  
    Disk for disk group not found
    refadm2-oss1{root} # vxdg -tfC import vxfendg2
    VxVM vxdg WARNING V-5-1-560 Disk c0t5006016C47201339d7: Not found, last known location: c0t5006016047201339d7s2
    refadm2-oss1{root} # vxdg list
    NAME         STATE           ID
    vxfendg2     enabled              1340186515.29.refadm1-oss1
    refadm2-oss1{root} # vxdisk -o alldgs list
    DEVICE       TYPE            DISK         GROUP        STATUS
    c0t5006016047201339d0s2 auto:sliced     -            (ossdg)      online
    c0t5006016047201339d1s2 auto:sliced     -            (sybasedg)   online
    c0t5006016047201339d2s2 auto:sliced     -            (vxfendg1)   online
    c0t5006016047201339d3s2 auto:sliced     -            (vxfendg1)   online
    c0t5006016047201339d4s2 auto:sliced     -            (vxfendg1)   online
    c0t5006016047201339d5s2 auto:sliced     c0t5006016C47201339d5  vxfendg2     online
    c0t5006016047201339d6s2 auto:sliced     c0t5006016C47201339d6  vxfendg2     online
    c0t5006016047201339d7s2 auto:sliced     -            -            online
    c2t0d0s2     auto:SVM        -            -            SVM
    c2t1d0s2     auto:SVM        -            -            SVM
    -            -         c0t5006016C47201339d7 vxfendg2     failed was:c0t5006016047201339d7s2
    refadm2-oss1{root} #  
     

  • symsonu's avatar
    symsonu
    Level 6
    12 years ago

    refadm2-oss1{root} # /etc/vx/bin/vxreattach -c c0t5006016C47201339d7
    VxVM vxreattach ERROR V-5-2-238 No matching Volume Manager disk and device IDs found for c0t5006016047201339d7s2
    refadm2-oss1{root} # /usr/lib/vxvm/bin/vxreattach  
    refadm2-oss1{root} # vxdisk -o alldgs list
    DEVICE       TYPE            DISK         GROUP        STATUS
    c0t5006016047201339d0s2 auto:sliced     -            (ossdg)      online
    c0t5006016047201339d1s2 auto:sliced     -            (sybasedg)   online
    c0t5006016047201339d2s2 auto:sliced     -            (vxfendg1)   online
    c0t5006016047201339d3s2 auto:sliced     -            (vxfendg1)   online
    c0t5006016047201339d4s2 auto:sliced     -            (vxfendg1)   online
    c0t5006016047201339d5s2 auto:sliced     c0t5006016C47201339d5  vxfendg2     online
    c0t5006016047201339d6s2 auto:sliced     c0t5006016C47201339d6  vxfendg2     online
    c0t5006016047201339d7s2 auto:sliced     -            -            online
    c2t0d0s2     auto:SVM        -            -            SVM
    c2t1d0s2     auto:SVM        -            -            SVM
    -            -         c0t5006016C47201339d7 vxfendg2     failed was:c0t5006016047201339d7s2
    refadm2-oss1{root} #