Forum Discussion

joaotelles's avatar
joaotelles
Level 4
11 years ago

VCS 5.1 - DG import fails to the new node

Hi,

 

I have recently added a new node to a cluster and it joined the cluster with no problem... but whe I tried to import a DG to the node it fails...

Note that this DG is available and imported in a different node.

Here si the log when it WORKED in a existing node:

2014/01/16 00:49:42 VCS NOTICE V-16-1-10301 Initiating Online of Resource md_DG (Owner: unknown, Group: mdm) on System D-node6
2014/01/16 00:49:44 VCS WARNING V-16-10001-1014 (D-node6) DiskGroup:md_DG:online:Diskgroups will be imported without reservations
2014/01/16 00:49:46 VCS NOTICE V-16-10001-1009 (D-node6) DiskGroup:md_DG:online:vxdg import succeeded on Disk Group md_DG
2014/01/16 00:49:47 VCS NOTICE V-16-10001-1010 (D-node6) DiskGroup:md_DG:online:Volumes in Disk Group md_DG are started. Any mirrors are updated in background
 

And here is when it FAILED in the new node:

2014/01/16 00:45:11 VCS NOTICE V-16-1-10301 Initiating Online of Resource md_DG (Owner: unknown, Group: mdm) on System d-node9
2014/01/16 00:45:12 VCS WARNING V-16-10001-1014 (d-node9) DiskGroup:md_DG:online:Diskgroups will be imported without reservations
2014/01/16 00:45:12 VCS WARNING V-16-10001-1016 (d-node9) DiskGroup:md_DG:online:vxdg import (clear flag) failed.
2014/01/16 00:45:12 VCS WARNING V-16-10001-1017 (d-node9) DiskGroup:md_DG:online:Trying force import for the diskgroup.
2014/01/16 00:45:12 VCS ERROR V-16-10001-1003 (d-node9) DiskGroup:md_DG:online:** ERROR: vxdg import (force) failed on Disk Group md_DG
2014/01/16 00:45:12 VCS ERROR V-16-10001-1004 (d-node9) DiskGroup:md_DG:online:** ERROR: vxdg import failed on Disk Group md_DG after vxdctl enable
2014/01/16 00:45:13 VCS INFO V-16-2-13716 (d-node9) Resource(mdm_DG): Output of the completed operation (online)
==============================================
VxVM vxdg ERROR V-5-1-10978 Disk group md_DG: import failed:
No valid disk found containing disk group
VxVM vxdg ERROR V-5-1-10978 Disk group md_DG: import failed:
No valid disk found containing disk group
VxVM vxdg ERROR V-5-1-10978 Disk group md_DG: import failed:
No valid disk found containing disk group
==============================================
 

Looking into the old dicussions I have tried this but still didnt work...: changed before importing

# vxdmpadm settune dmp_cache_open=off
Tunable value will be changed immediately
# vxdmpadm gettune all | grep cache
dmp_cache_open                          off               on
 

But I got the same error.

Same thing o var/adm/mesages:

Jan 16 00:45:12 d-node9 Had[17566]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1003 (dp-node9) DiskGroup:mdm_DG:online:** ERROR: vxdg import (force) failed on Disk Group mdm_DG
Jan 16 00:45:12 d-node9 vxdmp: [ID 631182 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 removed disk array DISKS, datype = Disk
Jan 16 00:45:12 d-node9 vxdmp: [ID 803759 kern.notice] NOTICE: VxVM vxdmp V-5-0-34 added disk array DISKS, datype = Disk
Jan 16 00:45:12 d-node9 Had[17566]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1004 (dp-node9) DiskGroup:mdm_DG:online:** ERROR: vxdg import failed on Disk Group mdm_DG after vxdctl enable
 

Any suggestion?

 

Tks,

Joao

 

 

 

  • Veritas Volume Manager only sees the disks the O/S sees, so if Solaris "format" command only see the 7 disks from oradg01 and the 2 internal disks and "format" cannot see the 2 disks for the md_DG diskgroup, then you need to fix issue with your O/S not seeing the disks, which could be an issue with the O/S, the FC switches or the disk array.

    Mike

  • Hi Joao,

    Can yo give more details , is this a CFS/CVM setup or its a failover setup ? Please give OS & SF versions as well, also to start, paste following outputs

    1. attach main.cf

    2. # gabconfig -a

    3. # cat /etc/VRTSvcs/conf/config/main.cf | grep -i usefence

    4. # hastatus -sum

     

    G

  • Can you provide output of "vxdisk -o alldgs list" on problem node - does it show the md_DG diskgroup?

    Mike

  • From the faulty node:

    # vxdisk -o alldgs list
    DEVICE       TYPE            DISK         GROUP        STATUS
    disk_0       auto            -            -            error
    disk_1       auto            -            -            error
    st2540-0_0   auto:cdsdisk    -            (oradg01)    online
    st2540-0_1   auto:cdsdisk    -            (oradg01)    online
    st2540-0_2   auto:cdsdisk    -            (oradg01)    online
    st2540-0_3   auto:cdsdisk    -            (oradg01)    online
    st2540-0_4   auto:cdsdisk    -            (oradg01)    online
    st2540-0_5   auto:cdsdisk    -            (oradg01)    online
    st2540-0_6   auto:cdsdisk    -            (oradg01)    online
     

    From the old/working node:

    vxdisk -o alldgs list
    DEVICE       TYPE            DISK         GROUP        STATUS
    disk_0       auto:SVM        -            -            SVM
    disk_1       auto:SVM        -            -            SVM
    st2540-0_0   auto:cdsdisk    -            (mastera_DG) online
    st2540-0_1   auto:cdsdisk    bomserver_DG01  bomserver_DG online
    st2540-0_2   auto:cdsdisk    wsm_DG01     wsm_DG       online
    st2540-0_3   auto:cdsdisk    -            (hds_DG)     online
    st2540-0_4   auto:cdsdisk    -            -            online
    st2540-0_5   auto:cdsdisk    -            -            online
    st2540-0_6   auto:cdsdisk    -            -            online
    st2540-0_7   auto:cdsdisk    -            -            online
    st2540-0_8   auto:cdsdisk    -            -            online
    st2540-0_9   auto:cdsdisk    -            -            online
    st2540-0_10  auto:cdsdisk    -            -            online
    st2540-0_11  auto:cdsdisk    -            (sfm1a_DG)   online
    st2540-0_12  auto:cdsdisk    -            (wig2a_DG)   online
    st2540-0_13  auto:cdsdisk    wig3a_DG01   wig3a_DG     online
    st2540-0_14  auto:cdsdisk    wig5a_DG01   wig5a_DG     online
    st2540-0_15  auto:cdsdisk    md_DG01     md_DG       online
    st2540-0_16  auto:cdsdisk    md_DG02     md_DG       online
     

    =====

     

    From the faulty node:

    # gabconfig -a
    GAB Port Memberships
    ===============================================================
    Port a gen   40691c membership 012 4
    Port h gen   406934 membership 012 4
     

    I dont use fence.

    # cat /etc/VRTSvcs/conf/config/main.cf | grep -i usefence
    #

    hastatus -sum

    -- SYSTEM STATE
    -- System               State                Frozen

    A  D-node4             RUNNING              0
    A  D-node5             RUNNING              0
    A  D-node6             RUNNING              0
    A  d-node9             RUNNING              0
     

    B  md             D-node5             Y          N               OFFLINE
    B  md             D-node6             Y          N               ONLINE
    B  md             d-node9             Y          N               OFFLINE


     

  • On the new/faulty node:

    # vxdisk -o alldgs list
    DEVICE       TYPE            DISK         GROUP        STATUS
    disk_0       auto            -            -            error
    disk_1       auto            -            -            error
    st2540-0_0   auto:cdsdisk    -            (oradg01)    online
    st2540-0_1   auto:cdsdisk    -            (oradg01)    online
    st2540-0_2   auto:cdsdisk    -            (oradg01)    online
    st2540-0_3   auto:cdsdisk    -            (oradg01)    online
    st2540-0_4   auto:cdsdisk    -            (oradg01)    online
    st2540-0_5   auto:cdsdisk    -            (oradg01)    online
    st2540-0_6   auto:cdsdisk    -            (oradg01)    online
     

    On the old/working node:

    #  vxdisk -o alldgs list
    DEVICE       TYPE            DISK         GROUP        STATUS
    disk_0       auto:SVM        -            -            SVM
    disk_1       auto:SVM        -            -            SVM
    st2540-0_0   auto:cdsdisk    -            (mastera_DG) online
    st2540-0_1   auto:cdsdisk    bomserver_DG01  bomserver_DG online
    st2540-0_2   auto:cdsdisk    wsm_DG01     wsm_DG       online
    st2540-0_3   auto:cdsdisk    -            (hds_DG)     online
    st2540-0_4   auto:cdsdisk    -            -            online
    st2540-0_5   auto:cdsdisk    -            -            online
    st2540-0_6   auto:cdsdisk    -            -            online
    st2540-0_7   auto:cdsdisk    -            -            online
    st2540-0_8   auto:cdsdisk    -            -            online
    st2540-0_9   auto:cdsdisk    -            -            online
    st2540-0_10  auto:cdsdisk    -            -            online
    st2540-0_11  auto:cdsdisk    -            (sfm1a_DG)   online
    st2540-0_12  auto:cdsdisk    -            (wig2a_DG)   online
    st2540-0_13  auto:cdsdisk    wig3a_DG01   wig3a_DG     online
    st2540-0_14  auto:cdsdisk    wig5a_DG01   wig5a_DG     online
    st2540-0_15  auto:cdsdisk    md_DG01     mdm_DG       online
    st2540-0_16  auto:cdsdisk    md_DG02     mdm_DG       online

    =====

    On the new node:

    # gabconfig -a
    GAB Port Memberships
    ===============================================================
    Port a gen   40691c membership 012 4
    Port h gen   406934 membership 012 4
     

    # cat /etc/VRTSvcs/conf/config/main.cf | grep -i usefence
    #
     

    hastatus -sum

    -- SYSTEM STATE
    -- System               State                Frozen

    A  D-node4             RUNNING              0
    A  D-node5             RUNNING              0
    A  D-node6             RUNNING              0
    A  d-node9             RUNNING              0

    -- GROUP STATE
    -- Group           System               Probed     AutoDisabled    State

    B  ClusterService  D-node4             Y          N               OFFLINE
    B  ClusterService  D-node5             Y          N               ONLINE
    B  ClusterService  D-node6             Y          N               OFFLINE
    B  ClusterService  d-node9             Y          N               OFFLINE
    B  bomserver       D-node5             Y          N               OFFLINE
    B  bomserver       D-node6             Y          N               ONLINE
    B  bomserver       d-node9             Y          N               OFFLINE
    B  hds             D-node5             Y          N               ONLINE
    B  hds             D-node6             Y          N               OFFLINE
    B  mastera         D-node5             Y          N               ONLINE
    B  mastera         D-node6             Y          N               OFFLINE
    B  md             D-node5             Y          N               OFFLINE
    B  md             D-node6             Y          N               ONLINE
    B  md             d-node9             Y          N               OFFLINE
    B  nic             D-node4             Y          N               ONLINE
    B  nic             D-node5             Y          N               ONLINE
    B  nic             D-node6             Y          N               ONLINE
    B  nic             d-node9             Y          N               ONLINE
    B  oracle          D-node4             Y          N               ONLINE
    B  sfm1a           D-node5             Y          N               ONLINE
    B  sfm1a           D-node6             Y          N               OFFLINE
    B  sysmona         D-node4             Y          N               ONLINE
    B  sysmona         D-node5             Y          N               ONLINE
    B  sysmona         D-node6             Y          N               ONLINE
    B  wig2a           D-node5             Y          N               ONLINE
    B  wig2a           D-node6             Y          N               OFFLINE
    B  wig3a           D-node5             Y          N               OFFLINE
    B  wig3a           D-node6             Y          N               ONLINE|STOPPING
    B  wig3a           d-node9             Y          N               OFFLINE
    B  wig5a           D-node5             Y          N               OFFLINE
    B  wig5a           D-node6             Y          N               ONLINE
    B  wig5a           d-node9             Y          N               OFFLINE
    B  wsm             D-node5             Y          N               OFFLINE
    B  wsm             D-node6             Y          N               ONLINE
    B  wsm             d-node9             Y          N               OFFLINE
     

     

  • Faulty node is not having disks or md_dg visible, it can only see oradg01 in deported state, you need to make sure that below two disks are visible to faulty node as well ...

    st2540-0_15  auto:cdsdisk    md_DG01     md_DG       online
    st2540-0_16  auto:cdsdisk    md_DG02     md_DG       online

    don't rely on the names st2540xxxxx, this name may be different on different nodes, you can run a /etc/vx/diag.d/vxscsiinq on good node to find serial number of below luns & make sure that same luns are visible to faulty node as well

     

    G

  • Just did for the good node on the md_dg devices but I got no serial number...

     

    # /etc/vx/diag.d/vxscsiinq st2540-0_15

    Inquiry for /dev/vx/rdmp/st2540-0_15, evpd 0x0, page code 0x0
            Vendor id                        : SUN
            Product id                       : LCSM100_F
            Revision                         : 0670
            Serial Number                    :

    # /etc/vx/diag.d/vxscsiinq st2540-0_16

    Inquiry for /dev/vx/rdmp/st2540-0_16, evpd 0x0, page code 0x0
            Vendor id                        : SUN
            Product id                       : LCSM100_F
            Revision                         : 0670
            Serial Number                    :
     

    Is this expected...? this is the good node... I will try to get the list of LUNs visible on the faulty one now.

  • I would expect to get a serial number ... can you try the same with raw devices once

    Find raw device with "vxdisk -e list" & you would get the native name in right column ... you can then run a vxscsiinq on raw device with /dev/rdsk/cxtxdx number & see the serial number

     

    G

  • Did as you told but no luck. Only for the disk at c0. Not for the c6 disk.

    I have tried to access the CAM to see these disks but the diskarray is out of commuication to the CAM.. maybe this is the problem?

    # vxdisk -e list
    DEVICE       TYPE           DISK        GROUP        STATUS               OS_NATIVE_NAME   ATTR
    disk_0       auto:SVM       -            -           SVM                  c0t5000CCA0153684F4d0s2 -
    disk_1       auto:SVM       -            -           SVM                  c0t5000CCA015393DACd0s2 -
    st2540-0_0   auto:cdsdisk   -            -           online               c6t202400A0B84930EDd4s2 -
    st2540-0_1   auto:cdsdisk   bomserver_DG01  bomserver_DG online               c6t202400A0B84930EDd5s2 -
    st2540-0_2   auto:cdsdisk   wsm_DG01     wsm_DG      online               c6t202400A0B84930EDd0s2 -
    st2540-0_3   auto:cdsdisk   -            -           online               c6t202400A0B84930EDd1s2 -
    st2540-0_4   auto:cdsdisk   -            -           online               c6t202400A0B84930EDd2s2 -
    st2540-0_5   auto:cdsdisk   -            -           online               c6t202400A0B84930EDd3s2 -
    st2540-0_6   auto:cdsdisk   -            -           online               c6t202400A0B84930EDd6s2 -
    st2540-0_7   auto:cdsdisk   -            -           online               c6t202400A0B84930EDd7s2 -
    st2540-0_8   auto:cdsdisk   -            -           online               c6t202400A0B84930EDd8s2 -
    st2540-0_9   auto:cdsdisk   -            -           online               c6t202400A0B84930EDd9s2 -
    st2540-0_10  auto:cdsdisk   -            -           online               c6t202400A0B84930EDd10s2 -
    st2540-0_11  auto:cdsdisk   -            -           online               c6t202400A0B84930EDd11s2 -
    st2540-0_12  auto:cdsdisk   -            -           online               c6t202400A0B84930EDd12s2 -
    st2540-0_13  auto:cdsdisk   wig3a_DG01   wig3a_DG    online               c6t202400A0B84930EDd13s2 -
    st2540-0_14  auto:cdsdisk   wig5a_DG01   wig5a_DG    online               c6t202400A0B84930EDd14s2 -
    st2540-0_15  auto:cdsdisk   mdm_DG01     mdm_DG      online               c6t202400A0B84930EDd15s2 -
    st2540-0_16  auto:cdsdisk   mdm_DG02     mdm_DG      online               c6t202400A0B84930EDd16s2 -
    root@DP-node6 # etc/vx/diag.d/vxscsiinq c6t202400A0B84930EDd15s2

    Inquiry for /dev/rdsk/c6t202400A0B84930EDd15s2, evpd 0x0, page code 0x0
            Vendor id                        : SUN
            Product id                       : LCSM100_F
            Revision                         : 0670
            Serial Number                    :
    root@DP-node6 # etc/vx/diag.d/vxscsiinq c6t202400A0B84930EDd16s2

    Inquiry for /dev/rdsk/c6t202400A0B84930EDd16s2, evpd 0x0, page code 0x0
            Vendor id                        : SUN
            Product id                       : LCSM100_F
            Revision                         : 0670
            Serial Number                    :
    root@DP-node6 # etc/vx/diag.d/vxscsiinq c0t5000CCA0153684F4d0s2

    Inquiry for /dev/rdsk/c0t5000CCA0153684F4d0s2, evpd 0x0, page code 0x0
            Vendor id                        : HITACHI
            Product id                       : H103030SCSUN300G
            Revision                         : A2A8
            Serial Number                    : 1105GYZHBE
     

  • Veritas Volume Manager only sees the disks the O/S sees, so if Solaris "format" command only see the 7 disks from oradg01 and the 2 internal disks and "format" cannot see the 2 disks for the md_DG diskgroup, then you need to fix issue with your O/S not seeing the disks, which could be an issue with the O/S, the FC switches or the disk array.

    Mike