How to fix the 'Duplicate Device' error for one of the disks

[ShelleyZhou]

We have VxVM4.1 running on Solaris 5.8 SPARC server.

After swapping one faulty disk, I couldn't re-associate the newly added disk with the plexes. The current status is like below:

root@agwha01(agwha01) # vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
c1t0d0s2     auto:sliced     rootmirror   rootdg       online
c1t0d0s2     auto:sliced     rootdisk     rootdg       online
c1t1d0s2     auto:sliced     -            -            error
c1t2d0s2     auto:sliced     -            -            online
c1t3d0s2     auto:cdsdisk    beadisk      wlint_dg     online
emcpower0s2  auto:sliced     Disk_6       ora_db_dg    online shared
emcpower1s2  auto:sliced     Disk_7       ora_db_dg    online shared
emcpower2s2  auto:sliced     Disk_8       ora_bin_dg   online shared
emcpower3s2  auto:sliced     Disk_3       smq_dg       online shared
emcpower4s2  auto:sliced     Disk_4       wl_dg        online shared
emcpower5s2  auto:sliced     Disk_5       ora_db_dg    online shared
emcpower6s2  auto:sliced     -            -            online
emcpower7s2  auto:sliced     -            -            online
emcpower8s2  auto:sliced     -            -            online
-            -         rootdsk      rootdg       failed was:c1t3d0s2
 

It's showed 'Duplicate Device' for c1t1d0s2. More detailed information is included in the attachment.

I really appreciate if somebody could give me suggestion on how to associate rootmirror  with c1t1d0s2 and rootdsk with c1t2d0s2.

Thank you very in advance!

Best Regards

Shelley zhou

[g_lee]

Shelley,

What procedure/steps did you use to swap the failed disk?

The output/symptoms look similar to the following, although you've already done a vxconfigd -k to try to fix the incorrect naming issue.

vxdisk list output erroneously lists duplicate or ghost entries during dynamic lun addition after a vxdctl enable command is executed
http://www.symantec.com/business/support/index?page=content&id=TECH52756

Can you try:

1. Stop the event source daemon (vxddladm stop eventsource)

2. devfsadm -C  (to ensure any leftover OS dev paths are cleaned up)

3. Remove /etc/vx/disk.info and /etc/vx/disk.info.old

4. Then perform vxconfigd -k

Note: this still may not work if the disk was not replaced correctly / has already been picked up incorrectly. If there are still duplicate devices, or vxdisk -e list still shows incorrect mappings after the vxconfigd restart, then the only recourse would be to reboot the server to get rid of the duplicates / allow the disk to be put back into the diskgroup.

[ShelleyZhou]

Hi G_lee,

I followed you suggestion and restarted vxconfigd with '-k' option. But afterwards the 'duplicate device' error moved from c1t1d0 to another disk c1t0d0. Below is current situation. How could I associate rootmirror  with c1t1d0s2 and rootdsk with c1t2d0s2 ? Should I start the event source deamon now?

root@agwha01(agwha01) # vxdisk -e list
DEVICE       TYPE      DISK         GROUP        STATUS       OS_NATIVE_NAME
c1t0d0s2     auto      -             -            error        c1t0d0s2
c1t0d0s2     auto      rootdisk      rootdg       online       c1t0d0s2
c1t1d0s2     auto      -             -            online       c1t1d0s2
c1t2d0s2     auto      -             -            online       c1t2d0s2
c1t3d0s2     auto      beadisk       wlint_dg     online       c1t3d0s2
emcpower0s2  auto      Disk_6        ora_db_dg    online       emcpower0c
emcpower1s2  auto      Disk_7        ora_db_dg    online       emcpower1c
emcpower2s2  auto      Disk_8        ora_bin_dg   online       emcpower2c
emcpower3s2  auto      Disk_3        smq_dg       online       emcpower3c
emcpower4s2  auto      Disk_4        wl_dg        online       emcpower4c
emcpower5s2  auto      Disk_5        ora_db_dg    online       emcpower5c
emcpower6s2  auto      -             -            online       emcpower6c
emcpower7s2  auto      -             -            online       emcpower7c
emcpower8s2  auto      -             -            online       emcpower8c
-            -         rootmirror   rootdg       failed was:c1t0d0s2
-            -         rootdsk      rootdg       failed was:c1t3d0s2

root@agwha01(agwha01) # vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
c1t0d0s2     auto:sliced     -            -            error
c1t0d0s2     auto:sliced     rootdisk     rootdg       error
c1t1d0s2     auto:sliced     -            -            online
c1t2d0s2     auto:sliced     -            -            online
c1t3d0s2     auto:cdsdisk    beadisk      wlint_dg     online
emcpower0s2  auto:sliced     Disk_6       ora_db_dg    online shared
emcpower1s2  auto:sliced     Disk_7       ora_db_dg    online shared
emcpower2s2  auto:sliced     Disk_8       ora_bin_dg   online shared
emcpower3s2  auto:sliced     Disk_3       smq_dg       online shared
emcpower4s2  auto:sliced     Disk_4       wl_dg        online shared
emcpower5s2  auto:sliced     Disk_5       ora_db_dg    online shared
emcpower6s2  auto:sliced     -            -            online
emcpower7s2  auto:sliced     -            -            online
emcpower8s2  auto:sliced     -            -            online
-            -         rootmirror   rootdg       failed was:c1t0d0s2
-            -         rootdsk      rootdg       failed was:c1t3d0s2

root@agwha01(agwha01) # vxdisk list c1t0d0s2
Device:    c1t0d0s2
devicetag: c1t0d0
type:      auto
info:      format=sliced,privoffset=1,pubslice=4,privslice=3
flags:     online error private
pubpaths:  block=/dev/vx/dmp/c1t0d0s4 char=/dev/vx/rdmp/c1t0d0s4
privpaths: block=/dev/vx/dmp/c1t0d0s3 char=/dev/vx/rdmp/c1t0d0s3
errno:     Duplicate device
Multipathing information:
numpaths:   1
c1t0d0s2        state=enabled
root@agwha01(agwha01) # bash
root@agwha01(agwha01) # vxdisk list c1t1d0s2
Device:    c1t1d0s2
devicetag: c1t1d0
type:      auto
hostid:
disk:      name= id=1288356218.48.dmp
group:     name= id=
info:      format=sliced,privoffset=1,pubslice=4,privslice=3
flags:     online ready private autoconfig autoimport
pubpaths:  block=/dev/vx/dmp/c1t1d0s4 char=/dev/vx/rdmp/c1t1d0s4
privpaths: block=/dev/vx/dmp/c1t1d0s3 char=/dev/vx/rdmp/c1t1d0s3
version:   2.1
iosize:    min=512 (bytes) max=2048 (blocks)
public:    slice=4 offset=0 len=143328960 disk_offset=20352
private:   slice=3 offset=1 len=9919 disk_offset=10176
update:    time=1288356218 seqno=0.1
ssb:       actual_seqno=0.0
headers:   0 248
configs:   count=1 len=7293
logs:      count=1 len=1105
Defined regions:
 config   priv 000017-000247[000231]: copy=01 offset=000000 disabled
 config   priv 000249-007310[007062]: copy=01 offset=000231 disabled
 log      priv 007311-008415[001105]: copy=01 offset=000000 disabled
Multipathing information:
numpaths:   1
c1t1d0s2        state=enabled

root@agwha01(agwha01) # vxprint -g rootdg -hvt
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO

v  rootvol      -            ENABLED  ACTIVE   83890944 ROUND     -        root
pl rootvol-01   rootvol      ENABLED  ACTIVE   83890944 CONCAT    -        RW
sd rootdisk-01  rootvol-01   rootdisk 0        83890944 0         c1t0d0   ENA
pl rootvol-02   rootvol      DISABLED NODEVICE 83890944 CONCAT    -        RW
sd rootdsk-02   rootvol-02   rootdsk  33550272 83890944 0         -        RLOC

v  swapvol      -            DISABLED ACTIVE   33550272 ROUND     -        swap
pl swapvol-01   swapvol      DISABLED NODEVICE 33550272 CONCAT    -        RW
sd rootdsk-01   swapvol-01   rootdsk  0        33550272 0         -        NDEV

v  var          -            ENABLED  ACTIVE   25897920 ROUND     -        fsgen
pl var-01       var          ENABLED  ACTIVE   25897920 CONCAT    -        RW
sd rootdisk-02  var-01       rootdisk 83890944 25897920 0         c1t0d0   ENA
pl var-02       var          DISABLED NODEVICE 25897920 CONCAT    -        WO
sd rootdsk-03   var-02       rootdsk  117441216 25897920 0        -        RLOC
 

[g_lee]

Shelley,

Normally I'd say try to remove the duplicate disk with vxdisk rm <disk>; however in this case the duplicate disk is your [encapsulated] root disk - so this is probably not a good idea as it may cause loss of access to the root disk/further problems.

At this point I think the best thing would be to reboot the system as the devices obviously haven't been picked up properly on replacement. Remove disk.info and disk.info.old prior to rebooting so it will recreate the file rather than using old/bad mappings. Once the disks are seen correctly on reboot, then you can reassociate/reattach the rootmirror.

If you want to know why it hasn't been picked up properly, please provide details of how the disk was replaced (ie: the sequence of commands/tasks run to remove the disk). Additionally, you mentioned the version was VxVM 4.1; this is quite an old version, and there are known problems with disk mapping/disk.info file if running early patch levels/unpatched.

[avsrini]

Hi Shelly,

Current state doesn't looks good. If you have a support contract, pls open a case with us.

If not, collect VRTSexplorer output (/opt/VRTSspt/VRTSexplorer) and move the output off the server

to your PC and do a reconfig reboot.

Also arrange to install the latest patch for 4.1. We are not sure what VxVM patch is installed on the system.

A reconfig reboot should clean up the messed up device tree though.

If the system is V440 / 880, which has internal dual fiber channel disks, there is a specific procedure

to be followed in replacing the disks. Pls check sunsolve for keyword "V880 replace internal disk"

Regards

Srini

[ShelleyZhou]

Hello,

After several run of 'vxconfigd -k', the 'Duplicate Device' error has gone and the path name in Solaris matches now with VxVM DAname, but new issue (even more serious) has appeared. See current status:

root@agwha01(agwha01) # vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
c1t0d0s2     auto:sliced     rootdisk     rootdg       online dgdisabled
c1t1d0s2     auto:sliced     -            -            online
c1t2d0s2     auto:sliced     -            -            online
c1t3d0s2     auto:cdsdisk    beadisk      wlint_dg     online

root@agwha01(agwha01) # vxdg -q list
rootdg       disabled             1143624634.1025.agwha01
ora_bin_dg   enabled,shared       1143637000.1275.agwha01
ora_db_dg    enabled,shared       1143637246.1283.agwha01
smq_dg       enabled,shared       1143638214.1305.agwha01
wl_dg        enabled,shared       1143637932.1295.agwha01
wlint_dg     enabled,cds          1202324469.27.agwha01

The rootdg has got disabled somehow, and rootmirror and rootdsk have disappeared. Even though the system is up and running fine so far (which is a mystery for me), I'm starting to panic :(

How can I get rootdg enabled? It should not be very difficult to re-create roodmirror and rootdsk, I believe as I have copy of their structures (such as subdisks, plexes, volumes etc).

Unfortunately we have no support agreement for Veritas. Does Synmantec provide 'one time' support (or Time Based support)?

Thanks and look forward for suggestions/advices!

Best Regards

Shelley

[g_lee · Accepted solution]

Shelley,

If a dg goes into dgdisabled state, normally you would deport it and re-import the dg to get it back to enabled state.

This cannot be done in this case as the dg that is in dgdisabled state is rootdg / encapsulated rootdisk so it can't be deported.

The server needs to be rebooted (as previously mentioned) - after reboot rootdg should be in enabled state again, and hopefully the devices will be seen correctly, so you can then add the replaced disk back into the diskgroup to mirror.

Do not keep running vxconfigd -k as this is not helping / may make things even worse.

Sequence of steps:

1. Take VRTSexplorer as suggested by Srini

2. Remove /etc/vx/disk.info /etc/vx/disk.info.old

3. touch /reconfigure ### so server will reconfig reboot

4. reboot the system (init 6)

5. Once the system is back up, you should be able to see the disks (check vxdisk list, vxdisk -e list to ensure they map correctly).

If the devices are seen correctly, use vxdiskadm to replace the disk and re-mirror the volumes.

regards,
Grace