Forum Discussion

Pranali's avatar
Pranali
Level 3
15 years ago

Reservation conflict on all LUNs of the cluster nodes.

Hi All,

        We have recently done the solaris live upgrade from solaris 9 to solaris 10 along with the maintenance patch live upgrade from 5.0MP1 to 5.0MP3.After making the alternate boot environment active we got the reservation conflict error for all the LUNs and also it created multipathing
problem. below are the logs from server, I would be thankfull for any suggetions or recommandations.

Aug 16 03:28:27 P0111CRMDB scsi: [ID 107833 kern.notice]        Sense Key: Unit Attention
Aug 16 03:28:27 P0111CRMDB scsi: [ID 107833 kern.notice]        ASC: 0x2a (reservations released), ASCQ: 0x4, FRU: 0x0
Aug 16 03:28:27 P0111CRMDB gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port w gen  103c72d membership 0
Aug 16 03:28:27 P0111CRMDB gab: [ID 674723 kern.notice] GAB INFO V-15-1-20038 Port w gen  103c72d k_jeopardy ;1
Aug 16 03:28:27 P0111CRMDB gab: [ID 513393 kern.notice] GAB INFO V-15-1-20040 Port w gen  103c72d    visible ;1
Aug 16 03:28:27 P0111CRMDB gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port v gen  103c72b membership 0
Aug 16 03:28:27 P0111CRMDB gab: [ID 674723 kern.notice] GAB INFO V-15-1-20038 Port v gen  103c72b k_jeopardy ;1
Aug 16 03:28:27 P0111CRMDB gab: [ID 513393 kern.notice] GAB INFO V-15-1-20040 Port v gen  103c72b    visible ;1
Aug 16 03:28:28 P0111CRMDB vxfs: [ID 779698 kern.notice] GLM recovery : gen 103c727 mbr 1 0 0 0 flags 0
Aug 16 03:28:28 P0111CRMDB vxfs: [ID 702911 kern.notice] NOTICE: msgcnt 2 mesg 125: V-2-125: GLM restart callback, protocol f
lag 0
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.warning] WARNING: /ssm@0,0/pci@1a,600000/SUNW,qlc@1,1/fp@0,0/ssd@w50060e8005
438657,90 (ssd618):
Aug 16 03:28:28 P0111CRMDB      Error for Command: read(10)                Error Level: Retryable
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Requested Block: 12813040                  Error Block: 12813
040
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Vendor: HITACHI                            Serial Number: 50
04386244F
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Sense Key: Unit Attention
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        ASC: 0x2a (reservations released), ASCQ: 0x4, FRU: 0x0
Aug 16 03:28:28 P0111CRMDB vxvm:vxconfigd: [ID 702911 daemon.notice] V-5-1-7899 CVM_VOLD_CHANGE command received
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.warning] WARNING: /ssm@0,0/pci@1a,600000/SUNW,qlc@1,1/fp@0,0/ssd@w50060e8005
438657,a9 (ssd987):
Aug 16 03:28:28 P0111CRMDB      Error for Command: write(10)               Error Level: Retryable
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Requested Block: 55124544                  Error Block: 55124
544
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Vendor: HITACHI                            Serial Number: 50
043862106
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Sense Key: Unit Attention
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        ASC: 0x2a (reservations released), ASCQ: 0x4, FRU: 0x0
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.warning] WARNING: /ssm@0,0/pci@1a,600000/SUNW,qlc@1,1/fp@0,0/ssd@w50060e8005
438657,94 (ssd614):
Aug 16 03:28:28 P0111CRMDB      Error for Command: write(10)               Error Level: Retryable
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Requested Block: 65844000                  Error Block: 65844
000
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Vendor: HITACHI                            Serial Number: 50
043862450
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Sense Key: Unit Attention
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        ASC: 0x2a (reservations released), ASCQ: 0x4, FRU: 0x0
Aug 16 03:28:28 P0111CRMDB vxvm:vxconfigd: [ID 702911 daemon.notice] V-5-1-13170 Preempting CM NID 1
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.warning] WARNING: /ssm@0,0/pci@18,600000/SUNW,qlc@1,1/fp@0,0/ssd@w50060e8005
438645,97 (ssd7):
Aug 16 03:28:28 P0111CRMDB      Error for Command: read(10)                Error Level: Retryable
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Requested Block: 69328                     Error Block: 69328
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Vendor: HITACHI                            Serial Number: 50
043862051
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Sense Key: Unit Attention
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        ASC: 0x2a (reservations released), ASCQ: 0x4, FRU: 0x0
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.warning] WARNING: /ssm@0,0/pci@1a,600000/SUNW,qlc@1,1/fp@0,0/ssd@w50060e8005
438657,ae (ssd982):
Aug 16 03:28:28 P0111CRMDB      Error for Command: read(10)                Error Level: Retryable
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Requested Block: 14741104                  Error Block: 14741
104
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Vendor: HITACHI                            Serial Number: 50
043862507
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Sense Key: Unit Attention
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        ASC: 0x2a (reservations released), ASCQ: 0x4, FRU: 0x0
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.warning] WARNING: /ssm@0,0/pci@18,600000/SUNW,qlc@1,1/fp@0,0/ssd@w50060e8005
438647,93 (ssd170):
Aug 16 03:28:28 P0111CRMDB      Error for Command: write(10)               Error Level: Retryable
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Requested Block: 65844000                  Error Block: 65844
000
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Vendor: HITACHI                            Serial Number: 50
043862050
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Sense Key: Unit Attention
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        ASC: 0x2a (reservations released), ASCQ: 0x4, FRU: 0x0
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.warning] WARNING: /ssm@0,0/pci@18,600000/SUNW,qlc@1,1/fp@0,0/ssd@w50060e8005
438647,8c (ssd177):
Aug 16 03:28:28 P0111CRMDB      Error for Command: read(10)                Error Level: Retryable
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Requested Block: 69475488                  Error Block: 69475
488
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Vendor: HITACHI                            Serial Number: 50
04386244E
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Sense Key: Unit Attention
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        ASC: 0x2a (reservations released), ASCQ: 0x4, FRU: 0x0
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.warning] WARNING: /ssm@0,0/pci@1a,600000/SUNW,qlc@1,1/fp@0,0/ssd@w50060e8005
438657,97 (ssd611):
Aug 16 03:28:28 P0111CRMDB      Error for Command: write(10)               Error Level: Retryable
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Requested Block: 69328                     Error Block: 69328
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Vendor: HITACHI                            Serial Number: 50
043862051
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Sense Key: Unit Attention
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        ASC: 0x2a (reservations released), ASCQ: 0x4, FRU: 0x0
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.warning] WARNING: /ssm@0,0/pci@1a,600000/SUNW,qlc@1,1/fp@0,0/ssd@w50060e8005
438657,84 (ssd630):
Aug 16 03:28:28 P0111CRMDB      Error for Command: read(10)                Error Level: Retryable
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Requested Block: 39521056                  Error Block: 39521
056
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Vendor: HITACHI                            Serial Number: 50
04386244C
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Sense Key: Unit Attention
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        ASC: 0x2a (reservations released), ASCQ: 0x4, FRU: 0x0
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.warning] WARNING: /ssm@0,0/pci@18,600000/SUNW,qlc@1,1/fp@0,0/ssd@w50060e8005
438647,ab (ssd913):
Aug 16 03:28:28 P0111CRMDB      Error for Command: read(10)                Error Level: Retryable
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Requested Block: 31795296                  Error Block: 31795
296
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Vendor: HITACHI                            Serial Number: 50
043862906
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        Sense Key: Unit Attention
Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.notice]        ASC: 0x2a (reservations released), ASCQ: 0x4, FRU: 0x0
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.warning] WARNING: /ssm@0,0/pci@1a,600000/SUNW,qlc@1,1/fp@0,0/ssd@w50060e8005
438657,ac (ssd984):
Aug 16 03:28:29 P0111CRMDB      Error for Command: read(10)                Error Level: Retryable
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        Requested Block: 14757984                  Error Block: 14757
984
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        Vendor: HITACHI                            Serial Number: 50
043862D06
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        Sense Key: Unit Attention
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        ASC: 0x2a (reservations released), ASCQ: 0x4, FRU: 0x0
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.warning] WARNING: /ssm@0,0/pci@1a,600000/SUNW,qlc@1,1/fp@0,0/ssd@w50060e8005
438657,7f (ssd635):
Aug 16 03:28:29 P0111CRMDB      Error for Command: read(10)                Error Level: Retryable
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        Requested Block: 39857696                  Error Block: 39857
696
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        Vendor: HITACHI                            Serial Number: 50
04386204B
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        Sense Key: Unit Attention
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        ASC: 0x2a (reservations released), ASCQ: 0x4, FRU: 0x0
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.warning] WARNING: /ssm@0,0/pci@1a,600000/SUNW,qlc@1,1/fp@0,0/ssd@w50060e8005
438657,87 (ssd627):
Aug 16 03:28:29 P0111CRMDB      Error for Command: read(10)                Error Level: Retryable
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        Requested Block: 17532480                  Error Block: 17532
480
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        Vendor: HITACHI                            Serial Number: 50
04386204D
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        Sense Key: Unit Attention
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        ASC: 0x2a (reservations released), ASCQ: 0x4, FRU: 0x0
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.warning] WARNING: /ssm@0,0/pci@1a,600000/SUNW,qlc@1,1/fp@0,0/ssd@w50060e8005
438657,86 (ssd628):
Aug 16 03:28:29 P0111CRMDB      Error for Command: read(10)                Error Level: Retryable
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        Requested Block: 39816976                  Error Block: 39816
976
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        Vendor: HITACHI                            Serial Number: 50
043862C4C
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        Sense Key: Unit Attention
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        ASC: 0x2a (reservations released), ASCQ: 0x4, FRU: 0x0
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.warning] WARNING: /ssm@0,0/pci@18,600000/SUNW,qlc@1,1/fp@0,0/ssd@w50060e8005
438645,80 (ssd30):
Aug 16 03:28:29 P0111CRMDB      Error for Command: read(10)                Error Level: Retryable
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        Requested Block: 50250576                  Error Block: 50250
576
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        Vendor: HITACHI                            Serial Number: 50
04386244B
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        Sense Key: Unit Attention
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        ASC: 0x2a (reservations released), ASCQ: 0x4, FRU: 0x0
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.warning] WARNING: /ssm@0,0/pci@1a,600000/SUNW,qlc@1,1/fp@0,0/ssd@w50060e8005
438655,93 (ssd456):
Aug 16 03:28:29 P0111CRMDB      Error for Command: write(10)               Error Level: Retryable
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        Requested Block: 65844240                  Error Block: 65844
240
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        Vendor: HITACHI                            Serial Number: 50
043862050
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        Sense Key: Unit Attention
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        ASC: 0x2a (reservations released), ASCQ: 0x4, FRU: 0x0
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.warning] WARNING: /ssm@0,0/pci@18,600000/SUNW,qlc@1,1/fp@0,0/ssd@w50060e8005
438645,92 (ssd12):
Aug 16 03:28:29 P0111CRMDB      Error for Command: write(10)               Error Level: Retryable
Aug 16 03:28:29 P0111CRMDB scsi: [ID 107833 kern.notice]        Requested Block: 65843280                  Error Block: 65843
280
 
business continuity
Cluster Server
Clustering
High availability
  • Gaurav_S's avatar
    Gaurav_S
    Moderator
    15 years ago
    Hi Pranali,

    Can you elaborate on following ?

    a) I believe you are using VCS, can you confirm that you are using VCS & not Sun cluster ?
    b) Do you use IO Fencing with VCS ?
    c) Why do you think that its a DMP problem as well ?
    d) I see above, port v & port w are having issues, that means CVM is facing some issues as well.... is CVM healthy ? 

    To confirm last point, can u paste following outputs from any node in cluster...

    # gabconfig -a
    # modinfo | egrep 'gab|llt|vx'
    # hastatus -sum
    # grep -i usefence /etc/VRTSvcs/conf/config/main.cf


    Gaurav
  • Nishil's avatar
    Nishil
    Level 2
    15 years ago

    Hi,

    Can you provide the below output also.

    /etc/vx/dmpevents.log
  • Zahid_Haseeb's avatar
    Zahid_Haseeb
    Moderator
    15 years ago
    First it seems that the HeartBeats are not working

    Aug 16 03:28:27 P0111CRMDB gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port w gen  103c72d membership 0
    Aug 16 03:28:27 P0111CRMDB gab: [ID 674723 kern.notice] GAB INFO V-15-1-20038 Port w gen  103c72d k_jeopardy ;1
    Aug 16 03:28:27 P0111CRMDB gab: [ID 513393 kern.notice] GAB INFO V-15-1-20040 Port w gen  103c72d    visible ;1
    Aug 16 03:28:27 P0111CRMDB gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port v gen  103c72b membership 0
    Aug 16 03:28:27 P0111CRMDB gab: [ID 674723 kern.notice] GAB INFO V-15-1-20038 Port v gen  103c72b k_jeopardy ;1
    Aug 16 03:28:27 P0111CRMDB gab: [ID 513393 kern.notice] GAB INFO V-15-1-20040 Port v gen  103c72b    visible ;1

    vxconfigd


    The vxconfigd is responsible for managing disk....

    Aug 16 03:28:28 P0111CRMDB vxvm:vxconfigd: [ID 702911 daemon.notice] V-5-1-13170 Preempting CM NID 1

    Qlogic card


    Your Qlogic card is doing some problem. i think you cards which connect your SAN drive with your Server is not able to establish the connection.


    Aug 16 03:28:28 P0111CRMDB scsi: [ID 107833 kern.warning] WARNING:/ssm@0,0/pci@18,600000/SUNW,qlc@1,1/fp@0,0/ssd@w50060e8005
    438645,97 (ssd7):


    i think you try to configure your QAlogic cards and try to reconfigure your SAN

  • Pranali's avatar
    Pranali
    Level 3
    15 years ago

    Thanks all for your reply ..

    Yes Gaurav we are using VCS with  IO fencing. when we were observing these errors in message file the same time we checked for the multipathing status of the luns, In the path status of the Luns it was not showing the O.K. status bt as"Reservation Conflict" . Also the CVM is healthy .

    Following o/p as requisted:

     gabconfig -a
    GAB Port Memberships
    ===============================================================
    Port a gen  103c72a membership 01
    Port b gen  103c72a membership 01
    Port d gen  103c72d membership 01
    Port f gen  103c728 membership 01
    Port h gen  103c72e membership 01
    Port o gen  103c72f membership 01
    Port v gen  103c72c membership 01
    Port w gen  103c72e membership 01
    root@P0111CRMDB     # modinfo | egrep 'gab|llt|vx'
     30  13392a0  3e4e0 289   1  vxdmp (VxVM 5.0MP3: DMP Driver)
     32 7be00000 209248 290   1  vxio (VxVM 5.0MP3 I/O driver)
     34 7bfe90f0    c78 291   1  vxspec (VxVM 5.0MP3 control/status driv)
    223 7aa9c000  26470 286   1  llt (LLT 5.0MP3)
    224 7aabc000  4a028 287   1  gab (GAB device 5.0MP3)
    225 7b23dd18  51c10 288   1  vxfen (VRTS Fence 5.0MP3)
    226 7b79f3c0    cb0 292   1  vxportal (VxFS 5.0_REV-5.0MP3A25_sol port)
    227 7a600000 1d89e0  21   1  vxfs (VxFS 5.0_REV-5.0MP3A25_sol SunO)
    228 7af76000  21ec0 294   1  vxglm (VxGLM 5.0MP3 (SunOS 5.10))
    231 7aff8000   5418 297   1  vxgms (VxGMS 5.0MP3 (SunOS))

    root@P0111CRMDB     # grep -i usefence /etc/VRTSvcs/conf/config/main.cf
    root@P0111CRMDB     # hastatus -sum

     

    -- SYSTEM STATE
    -- System               State                Frozen
    A  P0111CRMDB           RUNNING              0
    A  P2861PHGSB           RUNNING              0
    -- GROUP STATE
    -- Group           System               Probed     AutoDisabled    State
    B  ClusterService  P0111CRMDB           Y          N               ONLINE
    B  ClusterService  P2861PHGSB           Y          N               OFFLINE
    B  PHONEGENRAC     P0111CRMDB           Y          N               ONLINE
    B  PHONEGENRAC     P2861PHGSB           Y          N               STARTING|PARTIAL
    B  cvm             P0111CRMDB           Y          N               ONLINE
    B  cvm             P2861PHGSB           Y          N               ONLINE
    -- RESOURCES FAILED
    -- Group           Type                 Resource             System
    C  PHONEGENRAC     Netlsnr              PRODCLFY-LSNR-1      P2861PHGSB
    C  PHONEGENRAC     Netlsnr              PRODCLFY-LSNR-2      P2861PHGSB
    C  PHONEGENRAC     Netlsnr              PRODCLFY-LSNR-3      P2861PHGSB
    C  PHONEGENRAC     Netlsnr              PRODCLFY-LSNR-4      P2861PHGSB
    C  PHONEGENRAC     Netlsnr              PRODCLFY-LSNR-5      P2861PHGSB
    C  PHONEGENRAC     Netlsnr              PRODCLFY-LSNR-6      P2861PHGSB
    C  PHONEGENRAC     Netlsnr              PRODCLFY-LSNR-7      P2861PHGSB
    C  PHONEGENRAC     Netlsnr              PRODCLFY-LSNR-8      P2861PHGSB
    C  PHONEGENRAC     Netlsnr              PRODCLFY-LSNR-9      P2861PHGSB

    The above listners are showing as faulted as they are not recognizing the passwords, but working fine.

     
  • Gaurav_S's avatar
    Gaurav_S
    Moderator
    15 years ago

    Pranali,

    IOFencing modules are loaded however it is NOT configured for your cluster.... As you can see

    root@P0111CRMDB # grep -i usefence /etc/VRTSvcs/conf/config/main.cf
    root@P0111CRMDB     #

    If IOFencing is configured to use for cluster, main.cf should have a line saying "UseFence = SCSI3", which is not there in your cluster... So I am guessing IOFencing might be running in disabled mode... can confirm with following:

    # cat /etc/vxfenmode
    # /sbin/vxfenadm -d

    Coming to original problem of reservation conficts, I was trying to understand if any reservations being placed by IOFencing module..... Can you confirm following:

    a) IOFencing was never in use for this cluster  (you can check if any old main.cf contains "usefence" attribute.. or you will need to check keys on the disks using

    # /sbin/vxfenadm -g /dev/rdsk/ <cxtxdxs2>

    b) Was there anything changed from Storage end ? what Storage array you are using ?

    c) Are you seeing these reservation conflicts only for this node OR same messages are coming for other node as well.....


    Gaurav
  • Pranali's avatar
    Pranali
    Level 3
    15 years ago

    Hi Gaurav,

         I agree that the entry is not there in main.cf, but when i check vxfenmode it has following entry:

    vxfen_mode=scsi3

    also
    root@P2861PHGSB     # vxfenadm -d
    bash: vxfenadm: command not found
    root@P2861PHGSB     # /sbin/vxfenadm -d

    I/O Fencing Cluster Information:
    ================================
     Fencing Protocol Version: 201
     Fencing Mode: SCSI3
     Fencing SCSI3 Disk Policy: dmp
     Cluster Members:
              0 (P0111CRMDB)
            * 1 (P2861PHGSB)
     RFSM State Information:
            node   0 in state  8 (running)
            node   1 in state  8 (running)
    But in haclus -display it shows  UseFence            NONE

    a) IOFencing was never in use for this cluster  (you can check if any old main.cf contains "usefence" attribute.. or you will need to check keys on the disks using

    From begining (in old main.cf) we dont have the usefence entry.Also we have the reservation keys :

    root@P0111CRMDB     # /sbin/vxfenadm -g /dev/rdsk/c5t50060E8005438645d194s2
     
    Reading SCSI Registration Keys...
    Device Name: /dev/rdsk/c5t50060E8005438645d194s2
    Total Number Of Keys: 8
    key[0]:
            Key Value [Numeric Format]:  66,80,71,82,48,48,48,48
            Key Value [Character Format]: BPGR0000
    key[1]:
            Key Value [Numeric Format]:  65,80,71,82,48,48,48,48
            Key Value [Character Format]: APGR0000
    key[2]:
            Key Value [Numeric Format]:  66,80,71,82,48,48,48,48
            Key Value [Character Format]: BPGR0000
    key[3]:
            Key Value [Numeric Format]:  65,80,71,82,48,48,48,48
            Key Value [Character Format]: APGR0000
    key[4]:
            Key Value [Numeric Format]:  65,80,71,82,48,48,48,48
            Key Value [Character Format]: APGR0000
    key[5]:
            Key Value [Numeric Format]:  66,80,71,82,48,48,48,48
            Key Value [Character Format]: BPGR0000
    key[6]:
            Key Value [Numeric Format]:  65,80,71,82,48,48,48,48
            Key Value [Character Format]: APGR0000
    key[7]:
            Key Value [Numeric Format]:  66,80,71,82,48,48,48,48
            Key Value [Character Format]: BPGR0000
    root@P0111CRMDB     #

    ) Are you seeing these reservation conflicts only for this node OR same messages are coming for other node as well.....
    -->Yes these messages are coming for both the nodes and because of this it is taking upto 45 mins to start the vxconfigd , and i/o fencing.

    b) Was there anything changed from Storage end ? what Storage array you are using ?
    -->We are using HITACHI-OPEN-V-SUN-6004 and  there are no changes made from storage .


     
  • Gaurav_S's avatar
    Gaurav_S
    Moderator
    15 years ago

    Hi Pranali,

    good that signifies what is the problem...

    -- Sometime this cluster had IOFencing used & thats why cluster has IOFencing keys...

    -- Since main.cf doesn't contain the UseFence=SCSI3, then its for sure, cluster is not using IOFencing.... (I hope cluster config is saved with haconf -dump -makero)

    To solve this issue:

    a) Either you start using IOFencing again.... so make changes to main.cf accordingly .... You will need to restart VCS for this change to take effect....

    b) If you don't want to use IOFencing.... you can clear the keys by using "vxfenclearpre" command....

    See VCS users guide to understand how to use vxfenclearpre .....

    Hope this will solve the issue..

    Gaurav
  • Pranali's avatar
    Pranali
    Level 3
    15 years ago

    Thanks all for your reply and support, 

        The resevation conflict errors are stoped coming on both the node after making changes in to following storage parameters as recommanded by symantec.

    Host Mode Option : 02 (Veritas Storage Foundation for Oracle Rac)..................................Need to set

    refered doc:ftp://exftpp.symantec.com/pub/support/products/Foundation_Suite/283282.pdf

    Thanks,
    Pranali.

  • Gaurav_S's avatar
    Gaurav_S
    Moderator
    15 years ago

    Good to know....  Additionally make sure that IOFencing is corrected (adding usefence to main.cf)  ... if both network links fail, there might be chance of data corruption....

    Good luck


    Gaurav