10-28-2010 07:46 AM
We have VxVM4.1 running on Solaris 5.8 SPARC server.
After swapping one faulty disk, I couldn't re-associate the newly added disk with the plexes. The current status is like below:
root@agwha01(agwha01) # vxdisk list
DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 auto:sliced rootmirror rootdg online
c1t0d0s2 auto:sliced rootdisk rootdg online
c1t1d0s2 auto:sliced - - error
c1t2d0s2 auto:sliced - - online
c1t3d0s2 auto:cdsdisk beadisk wlint_dg online
emcpower0s2 auto:sliced Disk_6 ora_db_dg online shared
emcpower1s2 auto:sliced Disk_7 ora_db_dg online shared
emcpower2s2 auto:sliced Disk_8 ora_bin_dg online shared
emcpower3s2 auto:sliced Disk_3 smq_dg online shared
emcpower4s2 auto:sliced Disk_4 wl_dg online shared
emcpower5s2 auto:sliced Disk_5 ora_db_dg online shared
emcpower6s2 auto:sliced - - online
emcpower7s2 auto:sliced - - online
emcpower8s2 auto:sliced - - online
- - rootdsk rootdg failed was:c1t3d0s2
It's showed 'Duplicate Device' for c1t1d0s2. More detailed information is included in the attachment.
I really appreciate if somebody could give me suggestion on how to associate rootmirror with c1t1d0s2 and rootdsk with c1t2d0s2.
Thank you very in advance!
Best Regards
Shelley zhou
Solved! Go to Solution.
11-03-2010 07:39 AM
Shelley,
If a dg goes into dgdisabled state, normally you would deport it and re-import the dg to get it back to enabled state.
This cannot be done in this case as the dg that is in dgdisabled state is rootdg / encapsulated rootdisk so it can't be deported.
The server needs to be rebooted (as previously mentioned) - after reboot rootdg should be in enabled state again, and hopefully the devices will be seen correctly, so you can then add the replaced disk back into the diskgroup to mirror.
Do not keep running vxconfigd -k as this is not helping / may make things even worse.
Sequence of steps:
1. Take VRTSexplorer as suggested by Srini
2. Remove /etc/vx/disk.info /etc/vx/disk.info.old
3. touch /reconfigure ### so server will reconfig reboot
4. reboot the system (init 6)
5. Once the system is back up, you should be able to see the disks (check vxdisk list, vxdisk -e list to ensure they map correctly).
If the devices are seen correctly, use vxdiskadm to replace the disk and re-mirror the volumes.
regards,
Grace
10-29-2010 01:15 AM
Shelley,
What procedure/steps did you use to swap the failed disk?
The output/symptoms look similar to the following, although you've already done a vxconfigd -k to try to fix the incorrect naming issue.
vxdisk list output erroneously lists duplicate or ghost entries during dynamic lun addition after a vxdctl enable command is executed
http://www.symantec.com/business/support/index?page=content&id=TECH52756
Can you try:
1. Stop the event source daemon (vxddladm stop eventsource)
2. devfsadm -C (to ensure any leftover OS dev paths are cleaned up)
3. Remove /etc/vx/disk.info and /etc/vx/disk.info.old
4. Then perform vxconfigd -k
Note: this still may not work if the disk was not replaced correctly / has already been picked up incorrectly. If there are still duplicate devices, or vxdisk -e list still shows incorrect mappings after the vxconfigd restart, then the only recourse would be to reboot the server to get rid of the duplicates / allow the disk to be put back into the diskgroup.
10-29-2010 06:03 AM
Hi G_lee,
I followed you suggestion and restarted vxconfigd with '-k' option. But afterwards the 'duplicate device' error moved from c1t1d0 to another disk c1t0d0. Below is current situation. How could I associate rootmirror with c1t1d0s2 and rootdsk with c1t2d0s2 ? Should I start the event source deamon now?
root@agwha01(agwha01) # vxdisk -e list
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME
c1t0d0s2 auto - - error c1t0d0s2
c1t0d0s2 auto rootdisk rootdg online c1t0d0s2
c1t1d0s2 auto - - online c1t1d0s2
c1t2d0s2 auto - - online c1t2d0s2
c1t3d0s2 auto beadisk wlint_dg online c1t3d0s2
emcpower0s2 auto Disk_6 ora_db_dg online emcpower0c
emcpower1s2 auto Disk_7 ora_db_dg online emcpower1c
emcpower2s2 auto Disk_8 ora_bin_dg online emcpower2c
emcpower3s2 auto Disk_3 smq_dg online emcpower3c
emcpower4s2 auto Disk_4 wl_dg online emcpower4c
emcpower5s2 auto Disk_5 ora_db_dg online emcpower5c
emcpower6s2 auto - - online emcpower6c
emcpower7s2 auto - - online emcpower7c
emcpower8s2 auto - - online emcpower8c
- - rootmirror rootdg failed was:c1t0d0s2
- - rootdsk rootdg failed was:c1t3d0s2
root@agwha01(agwha01) # vxdisk list
DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 auto:sliced - - error
c1t0d0s2 auto:sliced rootdisk rootdg error
c1t1d0s2 auto:sliced - - online
c1t2d0s2 auto:sliced - - online
c1t3d0s2 auto:cdsdisk beadisk wlint_dg online
emcpower0s2 auto:sliced Disk_6 ora_db_dg online shared
emcpower1s2 auto:sliced Disk_7 ora_db_dg online shared
emcpower2s2 auto:sliced Disk_8 ora_bin_dg online shared
emcpower3s2 auto:sliced Disk_3 smq_dg online shared
emcpower4s2 auto:sliced Disk_4 wl_dg online shared
emcpower5s2 auto:sliced Disk_5 ora_db_dg online shared
emcpower6s2 auto:sliced - - online
emcpower7s2 auto:sliced - - online
emcpower8s2 auto:sliced - - online
- - rootmirror rootdg failed was:c1t0d0s2
- - rootdsk rootdg failed was:c1t3d0s2
root@agwha01(agwha01) # vxdisk list c1t0d0s2
Device: c1t0d0s2
devicetag: c1t0d0
type: auto
info: format=sliced,privoffset=1,pubslice=4,privslice=3
flags: online error private
pubpaths: block=/dev/vx/dmp/c1t0d0s4 char=/dev/vx/rdmp/c1t0d0s4
privpaths: block=/dev/vx/dmp/c1t0d0s3 char=/dev/vx/rdmp/c1t0d0s3
errno: Duplicate device
Multipathing information:
numpaths: 1
c1t0d0s2 state=enabled
root@agwha01(agwha01) # bash
root@agwha01(agwha01) # vxdisk list c1t1d0s2
Device: c1t1d0s2
devicetag: c1t1d0
type: auto
hostid:
disk: name= id=1288356218.48.dmp
group: name= id=
info: format=sliced,privoffset=1,pubslice=4,privslice=3
flags: online ready private autoconfig autoimport
pubpaths: block=/dev/vx/dmp/c1t1d0s4 char=/dev/vx/rdmp/c1t1d0s4
privpaths: block=/dev/vx/dmp/c1t1d0s3 char=/dev/vx/rdmp/c1t1d0s3
version: 2.1
iosize: min=512 (bytes) max=2048 (blocks)
public: slice=4 offset=0 len=143328960 disk_offset=20352
private: slice=3 offset=1 len=9919 disk_offset=10176
update: time=1288356218 seqno=0.1
ssb: actual_seqno=0.0
headers: 0 248
configs: count=1 len=7293
logs: count=1 len=1105
Defined regions:
config priv 000017-000247[000231]: copy=01 offset=000000 disabled
config priv 000249-007310[007062]: copy=01 offset=000231 disabled
log priv 007311-008415[001105]: copy=01 offset=000000 disabled
Multipathing information:
numpaths: 1
c1t1d0s2 state=enabled
root@agwha01(agwha01) # vxprint -g rootdg -hvt
V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO
v rootvol - ENABLED ACTIVE 83890944 ROUND - root
pl rootvol-01 rootvol ENABLED ACTIVE 83890944 CONCAT - RW
sd rootdisk-01 rootvol-01 rootdisk 0 83890944 0 c1t0d0 ENA
pl rootvol-02 rootvol DISABLED NODEVICE 83890944 CONCAT - RW
sd rootdsk-02 rootvol-02 rootdsk 33550272 83890944 0 - RLOC
v swapvol - DISABLED ACTIVE 33550272 ROUND - swap
pl swapvol-01 swapvol DISABLED NODEVICE 33550272 CONCAT - RW
sd rootdsk-01 swapvol-01 rootdsk 0 33550272 0 - NDEV
v var - ENABLED ACTIVE 25897920 ROUND - fsgen
pl var-01 var ENABLED ACTIVE 25897920 CONCAT - RW
sd rootdisk-02 var-01 rootdisk 83890944 25897920 0 c1t0d0 ENA
pl var-02 var DISABLED NODEVICE 25897920 CONCAT - WO
sd rootdsk-03 var-02 rootdsk 117441216 25897920 0 - RLOC
10-29-2010 06:47 AM
Shelley,
Normally I'd say try to remove the duplicate disk with vxdisk rm <disk>; however in this case the duplicate disk is your [encapsulated] root disk - so this is probably not a good idea as it may cause loss of access to the root disk/further problems.
At this point I think the best thing would be to reboot the system as the devices obviously haven't been picked up properly on replacement. Remove disk.info and disk.info.old prior to rebooting so it will recreate the file rather than using old/bad mappings. Once the disks are seen correctly on reboot, then you can reassociate/reattach the rootmirror.
If you want to know why it hasn't been picked up properly, please provide details of how the disk was replaced (ie: the sequence of commands/tasks run to remove the disk). Additionally, you mentioned the version was VxVM 4.1; this is quite an old version, and there are known problems with disk mapping/disk.info file if running early patch levels/unpatched.
11-01-2010 10:18 PM
Hi Shelly,
Current state doesn't looks good. If you have a support contract, pls open a case with us.
If not, collect VRTSexplorer output (/opt/VRTSspt/VRTSexplorer) and move the output off the server
to your PC and do a reconfig reboot.
Also arrange to install the latest patch for 4.1. We are not sure what VxVM patch is installed on the system.
A reconfig reboot should clean up the messed up device tree though.
If the system is V440 / 880, which has internal dual fiber channel disks, there is a specific procedure
to be followed in replacing the disks. Pls check sunsolve for keyword "V880 replace internal disk"
Regards
Srini
11-03-2010 03:32 AM
Hello,
After several run of 'vxconfigd -k', the 'Duplicate Device' error has gone and the path name in Solaris matches now with VxVM DAname, but new issue (even more serious) has appeared. See current status:
root@agwha01(agwha01) # vxdisk list
DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 auto:sliced rootdisk rootdg online dgdisabled
c1t1d0s2 auto:sliced - - online
c1t2d0s2 auto:sliced - - online
c1t3d0s2 auto:cdsdisk beadisk wlint_dg online
root@agwha01(agwha01) # vxdg -q list
rootdg disabled 1143624634.1025.agwha01
ora_bin_dg enabled,shared 1143637000.1275.agwha01
ora_db_dg enabled,shared 1143637246.1283.agwha01
smq_dg enabled,shared 1143638214.1305.agwha01
wl_dg enabled,shared 1143637932.1295.agwha01
wlint_dg enabled,cds 1202324469.27.agwha01
The rootdg has got disabled somehow, and rootmirror and rootdsk have disappeared. Even though the system is up and running fine so far (which is a mystery for me), I'm starting to panic :(
How can I get rootdg enabled? It should not be very difficult to re-create roodmirror and rootdsk, I believe as I have copy of their structures (such as subdisks, plexes, volumes etc).
Unfortunately we have no support agreement for Veritas. Does Synmantec provide 'one time' support (or Time Based support)?
Thanks and look forward for suggestions/advices!
Best Regards
Shelley
11-03-2010 07:39 AM
Shelley,
If a dg goes into dgdisabled state, normally you would deport it and re-import the dg to get it back to enabled state.
This cannot be done in this case as the dg that is in dgdisabled state is rootdg / encapsulated rootdisk so it can't be deported.
The server needs to be rebooted (as previously mentioned) - after reboot rootdg should be in enabled state again, and hopefully the devices will be seen correctly, so you can then add the replaced disk back into the diskgroup to mirror.
Do not keep running vxconfigd -k as this is not helping / may make things even worse.
Sequence of steps:
1. Take VRTSexplorer as suggested by Srini
2. Remove /etc/vx/disk.info /etc/vx/disk.info.old
3. touch /reconfigure ### so server will reconfig reboot
4. reboot the system (init 6)
5. Once the system is back up, you should be able to see the disks (check vxdisk list, vxdisk -e list to ensure they map correctly).
If the devices are seen correctly, use vxdiskadm to replace the disk and re-mirror the volumes.
regards,
Grace