04-06-2011 12:52 PM
I am trying to setup multipathing with EMC Clariion. The problem is that vxdisk list fabric_0 only shows one path. The EMC array is in auto-trespass mode. This is solaris 8 and format shows two paths.
# vxdisk list fabric_2
Device: fabric_2
devicetag: fabric_2
type: sliced
hostid: ncsun1
disk: name=disk05 id=1302111549.6037.ncsun1
group: name=rootdg id=1072877341.1025.nc1
info: privoffset=1
flags: online ready private autoconfig autoimport imported
pubpaths: block=/dev/vx/dmp/fabric_2s4 char=/dev/vx/rdmp/fabric_2s4
privpaths: block=/dev/vx/dmp/fabric_2s3 char=/dev/vx/rdmp/fabric_2s3
version: 2.2
iosize: min=512 (bytes) max=2048 (blocks)
public: slice=4 offset=0 len=1048494080
private: slice=3 offset=1 len=32511
update: time=1302111558 seqno=0.5
headers: 0 248
configs: count=1 len=23969
logs: count=1 len=3631
Defined regions:
config priv 000017-000247[000231]: copy=01 offset=000000 enabled
config priv 000249-023986[023738]: copy=01 offset=000231 enabled
log priv 023987-027617[003631]: copy=01 offset=000000 enabled
Multipathing information:
numpaths: 1
c10t500601613B241045d5s2 state=enabled
formt
8. c10t500601613B241045d0 <DGC-RAID5-0428 cyl 63998 alt 2 hd 256 sec 64>
/ssm@0,0/pci@19,700000/SUNW,qlc@2/fp@0,0/ssd@w500601613b241045,0
16. c16t500601603B241045d0 <DGC-RAID5-0428 cyl 63998 alt 2 hd 256 sec 64>
/ssm@0,0/pci@18,700000/SUNW,qlc@1/fp@0,0/ssd@w500601603b241045,0
vxdisk -o alldgs list show both paths. Two things here it should only show one of the paths and also the second path it shows with diskgroup in ( ). Another issue is why the disk dont show up as EMC_0 or similiar.
*****The server has T3 connect and EMC which we are migrating from T3 to EMC. The EMC is the fabric naming convention.
# vxdisk -o alldgs list
DEVICE TYPE DISK GROUP STATUS
T30_0 sliced disk01 rootdg online
T30_1 sliced disk02 rootdg online
T31_0 sliced disk03 rootdg online
T31_1 sliced disk04 rootdg online
T32_0 sliced rootdg00 rootdg online
T32_1 sliced rootdg01 rootdg online
c1t0d0s2 sliced - - error
c1t1d0s2 sliced - - error
fabric_0 sliced - - error
fabric_1 sliced - - error
fabric_2 sliced disk05 rootdg online
fabric_3 sliced disk06 rootdg online
fabric_4 sliced disk07 rootdg online
fabric_5 sliced disk08 rootdg online
fabric_6 sliced disk09 rootdg online
fabric_7 sliced disk10 rootdg online
fabric_8 sliced - - error
fabric_9 sliced - - error
fabric_10 sliced - (rootdg) online
fabric_11 sliced - (rootdg) online
fabric_12 sliced - (rootdg) online
fabric_13 sliced - (rootdg) online
fabric_14 sliced - (rootdg) online
fabric_15 sliced - (rootdg) online
Here is the ASL...There is no APM prior to version Veritas 4.0.
vxddladm listsupport
snippet for brevity.....
libvxDGCclariion.so A/P DGC CLARiiON
The c10 and c16 are the paths for the EMC
# vxdmpadm listctlr all
CTLR-NAME ENCLR-TYPE STATE ENCLR-NAME
=====================================================
c1 OTHER_DISKS ENABLED OTHER_DISKS
c10 OTHER_DISKS ENABLED OTHER_DISKS
c16 OTHER_DISKS ENABLED OTHER_DISKS
# vxdmpadm getsubpaths ctlr=c10
NAME STATE PATH-TYPE DMPNODENAME ENCLR-TYPE ENCLR-NAME
======================================================================
c10t500601613B241045d7s2 ENABLED - fabric_0 OTHER_DISKS OTHER_DISKS
c10t500601613B241045d6s2 ENABLED - fabric_1 OTHER_DISKS OTHER_DISKS
c10t500601613B241045d5s2 ENABLED - fabric_2 OTHER_DISKS OTHER_DISKS
c10t500601613B241045d4s2 ENABLED - fabric_3 OTHER_DISKS OTHER_DISKS
c10t500601613B241045d3s2 ENABLED - fabric_4 OTHER_DISKS OTHER_DISKS
c10t500601613B241045d2s2 ENABLED - fabric_5 OTHER_DISKS OTHER_DISKS
c10t500601613B241045d1s2 ENABLED - fabric_6 OTHER_DISKS OTHER_DISKS
c10t500601613B241045d0s2 ENABLED - fabric_7 OTHER_DISKS OTHER_DISKS
# vxdmpadm getsubpaths ctlr=c16
NAME STATE PATH-TYPE DMPNODENAME ENCLR-TYPE ENCLR-NAME
======================================================================
c16t500601603B241045d7s2 ENABLED - fabric_8 OTHER_DISKS OTHER_DISKS
c16t500601603B241045d6s2 ENABLED - fabric_9 OTHER_DISKS OTHER_DISKS
c16t500601603B241045d5s2 ENABLED - fabric_10 OTHER_DISKS OTHER_DISKS
c16t500601603B241045d4s2 ENABLED - fabric_11 OTHER_DISKS OTHER_DISKS
c16t500601603B241045d3s2 ENABLED - fabric_12 OTHER_DISKS OTHER_DISKS
c16t500601603B241045d2s2 ENABLED - fabric_13 OTHER_DISKS OTHER_DISKS
c16t500601603B241045d1s2 ENABLED - fabric_14 OTHER_DISKS OTHER_DISKS
c16t500601603B241045d0s2 ENABLED - fabric_15 OTHER_DISKS OTHER_DISKS
Thanks for any help
Solved! Go to Solution.
04-08-2011 10:09 AM
Ok here is an idea...If i add a smaller disk to rootdg and then remove the current disk from rootdg leaving the smaller disk in rootdg, think that will work? If not we will just use rootdg as it is and put a filesystem on that dg. As you see there are no volumes in rootdg on the one disk. Also before i got everything working i placed vxconfigd in disable mode "vxconfigd -k -m disable". I then ran vxdctl enable and know if i do type "vxdctl mode" it shows it is enable. Is it ok now to run stop vxconfigd and restart as normal "vxconfigd". All the volumes are now enabled and fsck'd and filesystems are mounted. If not we will just use rootdg as it is and create a volume and filesystem on that dg.
# vxprint -g data01 -v
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
v NB5.1_Client fsgen ENABLED 2097152 - ACTIVE - -
v NCFADS_Scripts fsgen ENABLED 4194304 - ACTIVE - -
v NDH_Home fsgen ENABLED 146800640 - ACTIVE - -
v Perl_5.8.3 fsgen ENABLED 2097152 - ACTIVE - -
v Python_2.3.4 fsgen ENABLED 2097152 - ACTIVE - -
v Python_2.3.4_Beta fsgen ENABLED 1048576 - ACTIVE - -
v SAS_ODBC fsgen ENABLED 4194304 - ACTIVE - -
v SAS82 fsgen ENABLED 4194304 - ACTIVE - -
v SybData2 fsgen ENABLED 12582912 - ACTIVE - -
v Sybase12V5 fsgen ENABLED 8388608 - ACTIVE - -
v archive gen ENABLED 8388608 - ACTIVE - -
v home gen ENABLED 8388608 - ACTIVE - -
v java fsgen ENABLED 3145728 - ACTIVE - -
v mfserver gen ENABLED 2097152 - ACTIVE - -
v ncdss gen ENABLED 629145600 - ACTIVE - -
v ncusers fsgen ENABLED 314572800 - ACTIVE - -
v nharch gen ENABLED 4194304 - ACTIVE - -
v sasprod gen ENABLED 125829120 - ACTIVE - -
v sastemp gen ENABLED 41943040 - ACTIVE - -
v sas8 gen ENABLED 2097152 - ACTIVE - -
v sas81 gen ENABLED 2097152 - ACTIVE - -
v sybdebug fsgen ENABLED 5242880 - ACTIVE - -
v sybtest fsgen ENABLED 12582912 - ACTIVE - -
# vxdisk list
DEVICE TYPE DISK GROUP STATUS
Disk_0 sliced disk01 rootdg online
Disk_1 sliced - - error
Disk_2 sliced disk05 data01 online
Disk_3 sliced disk06 data01 online
Disk_4 sliced disk07 data01 online
Disk_5 sliced disk08 data01 online
Disk_6 sliced disk09 data01 online
Disk_7 sliced disk10 data01 online
Any final thoughts here?
04-06-2011 02:15 PM
OS can see both paths seems something is with the ASL....Oh and let me add I am not using navisphere or powerpath....
# luxadm probe
No Network Array enclosures found in /dev/es
Found Fibre Channel device(s):
Node WWN:50020f200000bd4f Device Type:Disk device
Logical Path:/dev/rdsk/c7t60020F200000C2DC44EF5AF7000B560Dd0s2
Node WWN:50020f200000bd4f Device Type:Disk device
Logical Path:/dev/rdsk/c7t60020F200000C2DC44EF593900005A36d0s2
Node WWN:50020f200001062f Device Type:Disk device
Logical Path:/dev/rdsk/c7t60020F20000104AE3EB2EF4A000C3565d0s2
Node WWN:50020f200001062f Device Type:Disk device
Logical Path:/dev/rdsk/c7t60020F20000104AE3EB2EF5A00058465d0s2
Node WWN:50020f2000010c33 Device Type:Disk device
Logical Path:/dev/rdsk/c7t60020F20000107DC3EB2F021000BEB2Dd0s2
Node WWN:50020f2000010c33 Device Type:Disk device
Logical Path:/dev/rdsk/c7t60020F20000107DC3EB2F0310003CB97d0s2
Node WWN:50060160bb201045 Device Type:Disk device
Logical Path:/dev/rdsk/c10t500601613B241045d0s2
Logical Path:/dev/rdsk/c16t500601603B241045d0s2
Node WWN:50060160bb201045 Device Type:Disk device
Logical Path:/dev/rdsk/c10t500601613B241045d1s2
Logical Path:/dev/rdsk/c16t500601603B241045d1s2
Node WWN:50060160bb201045 Device Type:Disk device
Logical Path:/dev/rdsk/c10t500601613B241045d2s2
Logical Path:/dev/rdsk/c16t500601603B241045d2s2
Node WWN:50060160bb201045 Device Type:Disk device
Logical Path:/dev/rdsk/c10t500601613B241045d3s2
Logical Path:/dev/rdsk/c16t500601603B241045d3s2
Node WWN:50060160bb201045 Device Type:Disk device
Logical Path:/dev/rdsk/c10t500601613B241045d4s2
Logical Path:/dev/rdsk/c16t500601603B241045d4s2
Node WWN:50060160bb201045 Device Type:Disk device
Logical Path:/dev/rdsk/c10t500601613B241045d5s2
Logical Path:/dev/rdsk/c16t500601603B241045d5s2
Node WWN:50060160bb201045 Device Type:Disk device
Logical Path:/dev/rdsk/c10t500601613B241045d6s2
Logical Path:/dev/rdsk/c16t500601603B241045d6s2
Node WWN:50060160bb201045 Device Type:Disk device
Logical Path:/dev/rdsk/c10t500601613B241045d7s2
Logical Path:/dev/rdsk/c16t500601603B241045d7s2
#
04-07-2011 07:18 AM
Hello,
So its version 4.0 for vxvm, can you confirm the exact version ?
# pkginfo -l VRTSvxvm
Also, I see that you have ASL however somehow I believe either ASL is in excluded state or else its not claiming the devices...
Can you paste some outputs:
# vxdmpadm listexclude all
# ls -l /etc/vx/*exclude*
if you find any exclude file, paste the contents of them...
Also, can you attach output of:
# /etc/vx/diag.d/vxasldebug (if path is wrong, check in other directories for same command, not sure of exact path in 4.0)
Paths not being shown & vxdisk list showing as "fabric" both are relating to ASL issue only .. Tell me this also:
1. is this a new installation ? was it working before ?
2. whether ASL was installed manually or it came pre-installed ?
Gaurav
04-07-2011 09:27 AM
Hi Gaurav,
Sorry i thought i mention the version in my previous but i see i didnt. Its 3.5 see below:
# pkginfo -l VRTSvxvm
PKGINST: VRTSvxvm
NAME: VERITAS Volume Manager, Binaries
CATEGORY: system
ARCH: sparc
VERSION: 3.5,REV=06.21.2002.23.14
BASEDIR: /
VENDOR: VERITAS Software
DESC: Virtual Disk Subsystem
PSTAMP: VERITAS-3.5s_PointPatch1.3:26-July-2002
INSTDATE: Dec 30 2003 20:16
HOTLINE: 800-342-0652
EMAIL: support@veritas.com
STATUS: completely installed
FILES: 600 installed pathnames
21 shared pathnames
9 linked files
76 directories
337 executables
159005 blocks used (approx)
# vxdmpadm listexclude all
Devices excluded from VxVM:
--------------------------
Paths : None
Controllers : None
VID:PID : None
Devices excluded from multipathing by vxdmp:
-------------------------------------------
Paths : None
VID:PID : None
Pathgroups : None
# ls -l /etc/vx/*exclude*
-rw-r--r-- 1 root other 59 Dec 31 2003 /etc/vx/vxdmp.exclude
-rw-r--r-- 1 root other 59 Dec 31 2003 /etc/vx/vxvm.exclude
# cat /etc/vx/vxdmp.exclude
exclude_all 0
paths
#
controllers
#
product
#
pathgroups
#
# cat /etc/vx/vxvm.exclude
exclude_all 0
paths
#
controllers
#
product
#
pathgroups
Output for asldebug is attached in gzip format
This is a new installation of using EMC storage. Before the storage was using T3 SAN. The ASL library was installed via pkgadd just two days ago. The T3 are using mpxio for multipathing.
Here is a T3 disk its using mpxio as you can see by the long naming scheme...not concerned with the T3:
# vxdisk list T30_0
Device: T30_0
devicetag: T30_0
type: sliced
hostid: ncsun1
disk: name=disk01 id=1072877353.1041.nc1
group: name=rootdg id=1072877341.1025.nc1
flags: online ready private autoconfig autoimport imported
pubpaths: block=/dev/vx/dmp/T30_0s4 char=/dev/vx/rdmp/T30_0s4
privpaths: block=/dev/vx/dmp/T30_0s3 char=/dev/vx/rdmp/T30_0s3
version: 2.2
iosize: min=512 (bytes) max=2048 (blocks)
public: slice=4 offset=0 len=1000595456
private: slice=3 offset=1 len=57343
update: time=1302107429 seqno=0.343
headers: 0 248
configs: count=1 len=42303
logs: count=1 len=6409
Defined regions:
config priv 000017-000247[000231]: copy=01 offset=000000 enabled
config priv 000249-042320[042072]: copy=01 offset=000231 enabled
log priv 042321-048729[006409]: copy=01 offset=000000 enabled
Multipathing information:
numpaths: 1
c7t60020F20000104AE3EB2EF5A00058465d0s2 state=enabled type=primary
Here is EMC:
# vxdisk list fabric_2
Device: fabric_2
devicetag: fabric_2
type: sliced
hostid: ncsun1
disk: name=disk05 id=1302111549.6037.ncsun1
group: name=rootdg id=1072877341.1025.nc1
info: privoffset=1
flags: online ready private autoconfig autoimport imported
pubpaths: block=/dev/vx/dmp/fabric_2s4 char=/dev/vx/rdmp/fabric_2s4
privpaths: block=/dev/vx/dmp/fabric_2s3 char=/dev/vx/rdmp/fabric_2s3
version: 2.2
iosize: min=512 (bytes) max=2048 (blocks)
public: slice=4 offset=0 len=1048494080
private: slice=3 offset=1 len=32511
update: time=1302111558 seqno=0.5
headers: 0 248
configs: count=1 len=23969
logs: count=1 len=3631
Defined regions:
config priv 000017-000247[000231]: copy=01 offset=000000 enabled
config priv 000249-023986[023738]: copy=01 offset=000231 enabled
log priv 023987-027617[003631]: copy=01 offset=000000 enabled
Multipathing information:
numpaths: 1
c10t500601613B241045d5s2 state=enabled
04-07-2011 09:58 AM
Ok, first surprise to me that its a new installation & you have installed 3.5 which is End of Life Ages ago .. Even 4.0 & in last OCt even 4.1 was end of life .... so any particular reason you are using such an old version ?
I would very strongly recommend you to install latest version (5.1SP1 preferrable) .. I am saying this because even if your 3.5 setup gets to work, Symantec won't support it on any case unless your organization have support exception with them...
Ok coming to original problem .. my suspection was right ... you have ASL installed however it doesn't claim the Luns.. from the attached output:
checking asl library outputs
----------------------------------------------------------
**********************************************************
libvxDGCclariion.so
**********************************************************
libvxDGCclariion.so:vendor_info()
VID : DGC
PID : CLARiiON
ATYPE : A/P
ANAME : EMC_CLARiiON
libvxDGCclariion.key()
ASL Name : libvxDGCclariion.so
Feature Needed : 95
VxVM Version Needed : 32
Device : /dev/rdsk/c10t500601613B241045d0s2
Vendor Identification : DGC
Product Identification : RAID 5
Revision : 0428
Serial Number : 0C0000D4FCCL
libvxDGCclariion.so:claim_device() : ERROR
Device : /dev/rdsk/c10t500601613B241045d1s2
Vendor Identification : DGC
Product Identification : RAID 5
Revision : 0428
Serial Number : 100000DC30CL
libvxDGCclariion.so:claim_device() : ERROR
Device : /dev/rdsk/c10t500601613B241045d2s2
Vendor Identification : DGC
Product Identification : RAID 5
Revision : 0428
Serial Number : 140000E59FCL
libvxDGCclariion.so:claim_device() : ERROR
Device : /dev/rdsk/c10t500601613B241045d3s2
Vendor Identification : DGC
Product Identification : RAID 5
Revision : 0428
Serial Number : 160000E5A4CL
libvxDGCclariion.so:claim_device() : ERROR
Device : /dev/rdsk/c10t500601613B241045d4s2
Vendor Identification : DGC
Product Identification : RAID 5
Revision : 0428
Serial Number : 0E0000D843CL
libvxDGCclariion.so:claim_device() : ERROR
Device : /dev/rdsk/c10t500601613B241045d5s2
Vendor Identification : DGC
Product Identification : RAID 5
Revision : 0428
Serial Number : 120000E16ECL
libvxDGCclariion.so:claim_device() : ERROR
Device : /dev/rdsk/c10t500601613B241045d6s2
Vendor Identification : DGC
Product Identification : RAID 5
Revision : 0428
Serial Number : 5A00004614CL
libvxDGCclariion.so:claim_device() : ERROR
Device : /dev/rdsk/c10t500601613B241045d7s2
Vendor Identification : DGC
Product Identification : RAID 5
Revision : 0428
Serial Number : 6000004CC9CL
libvxDGCclariion.so:claim_device() : ERROR
Device : /dev/rdsk/c16t500601603B241045d0s2
Vendor Identification : DGC
Product Identification : RAID 5
Revision : 0428
Serial Number : 0C0000D4FCCL
libvxDGCclariion.so:claim_device() : ERROR
Device : /dev/rdsk/c16t500601603B241045d1s2
Vendor Identification : DGC
Product Identification : RAID 5
Revision : 0428
Serial Number : 100000DC30CL
libvxDGCclariion.so:claim_device() : ERROR
Device : /dev/rdsk/c16t500601603B241045d2s2
Vendor Identification : DGC
Product Identification : RAID 5
Revision : 0428
Serial Number : 140000E59FCL
libvxDGCclariion.so:claim_device() : ERROR
Device : /dev/rdsk/c16t500601603B241045d3s2
Vendor Identification : DGC
Product Identification : RAID 5
Revision : 0428
Serial Number : 160000E5A4CL
libvxDGCclariion.so:claim_device() : ERROR
Device : /dev/rdsk/c16t500601603B241045d4s2
Vendor Identification : DGC
Product Identification : RAID 5
Revision : 0428
Serial Number : 0E0000D843CL
libvxDGCclariion.so:claim_device() : ERROR
you can see all disks in error state ... so now questions are:
-- are you sure you installed correct version of ASL which is supported by 3.5 ?
-- did you reboot the server after ASL installation ? I doubt if 3.5 will accept ASL changes online ?
--again, if you have any latest version of vxvm, it will have clariion ASL inbuilt & will pickup your devices automatically. ...
Think over it ..
G
04-07-2011 01:44 PM
I may have mislead you...this is not a new install ...the only new thing is adding the EMC clariion tray to the server. We are not able to upgrade veritas at this time so trying to make this work as is with the new addition of the EMC tray.
I installed from here:
-- are you sure you installed correct version of ASL which is supported by 3.5 ?
http://www.symantec.com/business/support/index?page=content&id=TECH22609
Seems to be the correct one.... 3.5 solaris
-- did you reboot the server after ASL installation ? I doubt if 3.5 will accept ASL changes online ?
I have rebooted a couple of times along with doing reboot reconfigure
--again, if you have any latest version of vxvm, it will have clariion ASL inbuilt & will pickup your devices automatically. ...
Not able to upgrade because not worth the cost as customer is going away in a year so really trying to get this to work on 3.5
Thanks for the quick response....
04-07-2011 03:16 PM
Well we decided since we mirrored the T3 plex with EMC plex under one volume and one disk group we would go ahead and remove the T3 plex and leave the EMC plex in place. This would leave just the one EMC plex and no T3 volumes exist now. Upon reboot vxconfigd does not start because of message below
# vxconfigd
vxvm:vxconfigd: NOTICE: Unable to resolve duplicate diskid.
Please refer to release notes and admin guide for possible action/solution.
Following are the disks with duplicate diskid:
Vendor: DGC Product: RAID - fabric_2, fabric_10
vxvm:vxconfigd: NOTICE: Unable to resolve duplicate diskid.
Please refer to release notes and admin guide for possible action/solution.
Following are the disks with duplicate diskid:
Vendor: DGC Product: RAID - fabric_3, fabric_11
vxvm:vxconfigd: NOTICE: Unable to resolve duplicate diskid.
Please refer to release notes and admin guide for possible action/solution.
Following are the disks with duplicate diskid:
Vendor: DGC Product: RAID - fabric_4, fabric_12
vxvm:vxconfigd: NOTICE: Unable to resolve duplicate diskid.
Please refer to release notes and admin guide for possible action/solution.
Following are the disks with duplicate diskid:
Vendor: DGC Product: RAID - fabric_5, fabric_13
vxvm:vxconfigd: NOTICE: Unable to resolve duplicate diskid.
Please refer to release notes and admin guide for possible action/solution.
Following are the disks with duplicate diskid:
Vendor: DGC Product: RAID - fabric_6, fabric_14
vxvm:vxconfigd: NOTICE: Unable to resolve duplicate diskid.
Please refer to release notes and admin guide for possible action/solution.
Following are the disks with duplicate diskid:
Vendor: DGC Product: RAID - fabric_7, fabric_15
vxvm:vxconfigd: ERROR: enable failed: Error in disk group configuration copies
Unexpected kernel error in configuration update; transactions are disabled.
Any idea how to resolve this one? I think it all is an issue with the DGC library as you have stated.
Thanks
04-07-2011 09:57 PM
well true ..
try out these steps:
# rm /dev/vx/dmp/*
# rm /dev/vx/rdmp/*
# rm /dev/rdsk/*
# rm /dev/rdsk/*
# devfsadm -Cv
# /sbin/vxconfigd -k -m disable
check if vxconfigd started
# vxdctl mode (see if it returns disabled)
Try to enable it..
# vxdctl enable
Also, can u paste output of:
pkginfo -l CLR-ASL
Gaurav
04-07-2011 10:03 PM
Did all the steps above and the results are below....
# vxdctl enable
vxvm:vxconfigd: ERROR: enable failed: Error in disk group configuration copies
No valid disk found containing disk group; transactions are disabled.
vxvm:vxdctl: ERROR: enable failed: Error in disk group configuration copies
# pkginfo -l CLR-ASL
PKGINST: CLR-ASL
NAME: EMC CLARIION Array Support Library
CATEGORY: system
ARCH: sparc
VERSION: 1.04,REV=07.25.2002.17.36
BASEDIR: /etc/vx
VENDOR: EMC Corp
DESC: EMC CLARIION Array Support Library
PSTAMP: VERSION-1.04:14-February-2002
INSTDATE: Apr 07 2011 18:00
HOTLINE: 800-782-4362
STATUS: completely installed
FILES: 4 installed pathnames
2 shared pathnames
2 directories
2 executables
68 blocks used (approx)
I know the data is still out there because a vxinstall will let me do a vxdisk -e list and it reveals the disk belong to rootdg. Hopefully were are not toast.
04-07-2011 10:08 PM
Is there another form of communication you are willing to do(ie IM(googletalk, yahoo), phone, etc...) Just thought i would ask. If so i will put my outside email account on here and we can continue that way or we can continue like we are doing now. Thanks for assistance...
04-07-2011 10:44 PM
Data is still out there however its ASL which has caused issues to device names here.. & yes vxinstall will not touch out data region...
If you get vxconfigd to start , I would suggest to first exclude your ASL ...
# vxddladm excludearray libname=libvxDGCclariion.so
sorry m typing this thru mobile so bit slow..
G
04-07-2011 10:54 PM
I have tried to run vxinstall but didnt follow through with completion in fear the process would wipe out my public region where the data lives. I really wasnt sure but i think vxinstall only touches the private region. Your thought on whether i should try to run vxinstall. My root drive is not part of vxvm.
I can run vxinstall and then in another window i am able to run vxdisk list and it displays this:
# vxdisk -e list
DEVICE TYPE DISK GROUP STATUS c#t#d#_NAME
Disk_0 sliced disk01 rootdg online c10t500601613B241045d7s2
Disk_1 sliced - - error c10t500601613B241045d6s2
Disk_2 sliced - - online c10t500601613B241045d5s2
Disk_3 sliced - - online c10t500601613B241045d4s2
Disk_4 sliced - - online c10t500601613B241045d3s2
Disk_5 sliced - - online c10t500601613B241045d2s2
Disk_6 sliced - - online c10t500601613B241045d1s2
Disk_7 sliced - - online c10t500601613B241045d0s2
c1t0d0s2 sliced - - error c1t0d0s2
c1t1d0s2 sliced - - error c1t1d0s2
# vxdisk -o alldgs list
DEVICE TYPE DISK GROUP STATUS
Disk_0 sliced disk01 rootdg online
Disk_1 sliced - - error
Disk_2 sliced - (rootdg) online
Disk_3 sliced - (rootdg) online
Disk_4 sliced - (rootdg) online
Disk_5 sliced - (rootdg) online
Disk_6 sliced - (rootdg) online
Disk_7 sliced - (rootdg) online
c1t0d0s2 sliced - - error
c1t1d0s2 sliced - - error
But HEY LOOK BELOW I WAS ABLE TO GET THE PATHS TO SHOW CORRECTLY!!!!!!.....So now the problem is getting vxconfigd to be enabled.
# vxdisk list Disk_0
Device: Disk_0
devicetag: Disk_0
type: sliced
hostid: ncsun1
disk: name=disk01 id=1302235001.1049.ncsun1
group: name=rootdg id=1302234996.1025.ncsun1
flags: online ready private autoconfig autoimport imported
pubpaths: block=/dev/vx/dmp/Disk_0s4 char=/dev/vx/rdmp/Disk_0s4
privpaths: block=/dev/vx/dmp/Disk_0s3 char=/dev/vx/rdmp/Disk_0s3
version: 2.2
iosize: min=512 (bytes) max=2048 (blocks)
public: slice=4 offset=0 len=629094400
private: slice=3 offset=1 len=20479
update: time=1302241535 seqno=0.17
headers: 0 248
configs: count=1 len=15096
logs: count=1 len=2287
Defined regions:
config priv 000017-000247[000231]: copy=01 offset=000000 enabled
config priv 000249-015113[014865]: copy=01 offset=000231 enabled
log priv 015114-017400[002287]: copy=01 offset=000000 enabled
Multipathing information:
numpaths: 2
c10t500601613B241045d7s2 state=enabled
c16t500601603B241045d7s2 state=enabled
So the focus has turned from dmp to vxconfigd....
04-07-2011 11:10 PM
so what does # vxdctl mode shows now ?
04-08-2011 04:37 AM
WOW!!!!!!
# vxdctl mode
mode: enabled
So now the question is how do i get the disk back in rootdg. The ones that show (rootdg) are the original disk that should be in rootdg. Do i just perform vxdisksetup -i on all the disk that show (rootdg)?
# vxdisk -o alldgs list
DEVICE TYPE DISK GROUP STATUS
Disk_0 sliced disk01 rootdg online
Disk_1 sliced - - error
Disk_2 sliced - (rootdg) online
Disk_3 sliced - (rootdg) online
Disk_4 sliced - (rootdg) online
Disk_5 sliced - (rootdg) online
Disk_6 sliced - (rootdg) online
Disk_7 sliced - (rootdg) online
c1t0d0s2 sliced - - error
c1t1d0s2 sliced - - error
# vxdisk -e list
DEVICE TYPE DISK GROUP STATUS c#t#d#_NAME
Disk_0 sliced disk01 rootdg online c10t500601613B241045d7s2
Disk_1 sliced - - error c10t500601613B241045d6s2
Disk_2 sliced - - online c10t500601613B241045d5s2
Disk_3 sliced - - online c10t500601613B241045d4s2
Disk_4 sliced - - online c10t500601613B241045d3s2
Disk_5 sliced - - online c10t500601613B241045d2s2
Disk_6 sliced - - online c10t500601613B241045d1s2
Disk_7 sliced - - online c10t500601613B241045d0s2
c1t0d0s2 sliced - - error c1t0d0s2
c1t1d0s2 sliced - - error c1t1d0s2
04-08-2011 04:46 AM
NO, don't run vxdisksetup .. if offsets are given wrong you may landup in loosing all data..
Why we see less disks ? From the output of first post
fabric_0 sliced - - error
fabric_1 sliced - - error
fabric_2 sliced disk05 rootdg online
fabric_3 sliced disk06 rootdg online
fabric_4 sliced disk07 rootdg online
fabric_5 sliced disk08 rootdg online
fabric_6 sliced disk09 rootdg online
fabric_7 sliced disk10 rootdg online
fabric_8 sliced - - error
fabric_9 sliced - - error
fabric_10 sliced - (rootdg) online
fabric_11 sliced - (rootdg) online
fabric_12 sliced - (rootdg) online
fabric_13 sliced - (rootdg) online
fabric_14 sliced - (rootdg) online
fabric_15 sliced - (rootdg) online
now we see only
# vxdisk -e list
DEVICE TYPE DISK GROUP STATUS c#t#d#_NAME
Disk_0 sliced disk01 rootdg online c10t500601613B241045d7s2
Disk_1 sliced - - error c10t500601613B241045d6s2
Disk_2 sliced - - online c10t500601613B241045d5s2
Disk_3 sliced - - online c10t500601613B241045d4s2
Disk_4 sliced - - online c10t500601613B241045d3s2
Disk_5 sliced - - online c10t500601613B241045d2s2
Disk_6 sliced - - online c10t500601613B241045d1s2
Disk_7 sliced - - online c10t500601613B241045d0s2
c1t0d0s2 sliced - - error c1t0d0s2
c1t1d0s2 sliced - - error c1t1d0s2
why less disks ? moreover did u excluded the array library ?
we need to ensure that, the disks which were imported in rootdg before i.e
fabric_2 sliced disk05 rootdg online
fabric_3 sliced disk06 rootdg online
fabric_4 sliced disk07 rootdg online
fabric_5 sliced disk08 rootdg online
fabric_6 sliced disk09 rootdg online
fabric_7 sliced disk10 rootdg online
are the same ones now, which are shown as
Disk_0 sliced disk01 rootdg online
Disk_1 sliced - - error
Disk_2 sliced - (rootdg) online
Disk_3 sliced - (rootdg) online
Disk_4 sliced - (rootdg) online
Disk_5 sliced - (rootdg) online
Disk_6 sliced - (rootdg) online
Disk_7 sliced - (rootdg) online
so problem is, rootdg is imported & deported too (yeah sounds crazy) ..but u can't deport rootdg & reimport it ... vxvm will expect the same disks to be present .
can u confirm if these are the same disks which were visible & imported before ?
G
04-08-2011 04:54 AM
What i did to get the paths to show correctly was vxddladm
# /etc/vx/diag.d/vxdmpinq /dev/rdsk/c10t500601613B241045d0s2
Inquiry for /dev/rdsk/c10t500601613B241045d0s2, evpd 0x0, page code 0x0
Vendor id : DGC
Product id : RAID 5
Revision : 0428
Serial Number : 0C0000D4FCCL
# /etc/vx/diag.d/vxdmpinq /dev/rdsk/c10t500601613B241045d1s2
Inquiry for /dev/rdsk/c10t500601613B241045d1s2, evpd 0x0, page code 0x0
Vendor id : DGC
Product id : RAID 5
Revision : 0428
Serial Number : 100000DC30CL
# vxddladm addjbod vid=DGC
# vxddladm listjbod
VID PID Opcode Page Code Page Offset SNO length
==========================================================================
SEAGATE ALL PIDs 18 -1 36 12
SUN SESS01 18 -1 36 12
DGC ALL PIDs 18 -1 36 12
# vxdmpadm listctlr all
CTLR-NAME ENCLR-TYPE STATE ENCLR-NAME
=====================================================
c1 OTHER_DISKS ENABLED OTHER_DISKS
c10 Disk ENABLED Disk
c16 Disk ENABLED Disk
04-08-2011 05:18 AM
Here is a copy of an old vxdisk list prior to the T3 SAN being removed. I have output of various vxprint, vxdisk, etc... of previous setup. You can see the fabric devices were part of rootdg
(root)> vxdisk -o alldgs list
DEVICE TYPE DISK GROUP STATUS
T30_0 sliced disk01 rootdg online
T30_1 sliced disk02 rootdg online
T31_0 sliced disk03 rootdg online
T31_1 sliced disk04 rootdg online
T32_0 sliced rootdg00 rootdg online
T32_1 sliced rootdg01 rootdg online
c1t0d0s2 sliced - - error
c1t1d0s2 sliced - - error
fabric_0 sliced - - error
fabric_1 sliced - - error
fabric_2 sliced disk05 rootdg online
fabric_3 sliced disk06 rootdg online
fabric_4 sliced disk07 rootdg online
fabric_5 sliced disk08 rootdg online
fabric_6 sliced disk09 rootdg online
fabric_7 sliced disk10 rootdg online
fabric_8 sliced - - error
fabric_9 sliced - - error
fabric_10 sliced - (rootdg) online
fabric_11 sliced - (rootdg) online
fabric_12 sliced - (rootdg) online
fabric_13 sliced - (rootdg) online
fabric_14 sliced - (rootdg) online
fabric_15 sliced - (rootdg) online
Also prior vxdisk list -e
(root)> vxdisk -e list
DEVICE TYPE DISK GROUP STATUS c#t#d#_NAME
T30_0 sliced disk01 rootdg online c7t60020F20000104AE3EB2EF5A00058465d0s2
T30_1 sliced disk02 rootdg online c7t60020F20000104AE3EB2EF4A000C3565d0s2
T31_0 sliced disk03 rootdg online c7t60020F20000107DC3EB2F0310003CB97d0s2
T31_1 sliced disk04 rootdg online c7t60020F20000107DC3EB2F021000BEB2Dd0s2
T32_0 sliced rootdg00 rootdg online c7t60020F200000C2DC44EF5AF7000B560Dd0s2
T32_1 sliced rootdg01 rootdg online c7t60020F200000C2DC44EF593900005A36d0s2
c1t0d0s2 sliced - - error c1t0d0s2
c1t1d0s2 sliced - - error c1t1d0s2
fabric_0 sliced - - error c10t500601613B241045d7s2
fabric_1 sliced - - error c10t500601613B241045d6s2
fabric_2 sliced disk05 rootdg online c10t500601613B241045d5s2
fabric_3 sliced disk06 rootdg online c10t500601613B241045d4s2
fabric_4 sliced disk07 rootdg online c10t500601613B241045d3s2
fabric_5 sliced disk08 rootdg online c10t500601613B241045d2s2
fabric_6 sliced disk09 rootdg online c10t500601613B241045d1s2
fabric_7 sliced disk10 rootdg online c10t500601613B241045d0s2
fabric_8 sliced - - error c16t500601603B241045d7s2
fabric_9 sliced - - error c16t500601603B241045d6s2
fabric_10 sliced - - online c16t500601603B241045d5s2
fabric_11 sliced - - online c16t500601603B241045d4s2
fabric_12 sliced - - online c16t500601603B241045d3s2
fabric_13 sliced - - online c16t500601603B241045d2s2
fabric_14 sliced - - online c16t500601603B241045d1s2
fabric_15 sliced - - online c16t500601603B241045d0s2
04-08-2011 05:45 AM
These disk were part of rootdg before as rootdg was the only diskgroup. The disk that shows up correctly is a new disk that added with no worries about losing data. The other disk that have (rootdg) have data on them. So i am 100% sure they belong to rootdg. We see less disk in output now because the extra disk we were seeing before came from the other path for disk 0-7. Doing the vxddladm add DGC corrected my path issue which was the original problem. So now vxdisk shows the correct number of disks. So now when i do vxdisk list Disk_0 i see two paths instead of 1. Next step?
04-08-2011 05:52 AM
devans3428,
In this comment ( https://www-secure.symantec.com/connect/forums/dynamic-multipath-using-emc-storage-veritas-35#comment-5410891 ), you mentioned:
"Well we decided since we mirrored the T3 plex with EMC plex under one volume and one disk group we would go ahead and remove the T3 plex and leave the EMC plex in place. This would leave just the one EMC plex and no T3 volumes exist now."
Did you cleanly remove the T3 disks from the dg (vxdg -g <dg> rmdisk <disk>) before they were removed/unpresented from the OS?
Your original dg (on T3) appears to have had 6 disks. VxVM keeps 5 diskgroup configuration copies by default (may be 4 or 5 on 3.5, can't remember off the top of my head since it's a v old version!). Unless you've performed an explicit config change to place an active config copy every disk, this means only 5 disks will have an active dg config (or up to 5 if the dg had less than 5 disks). If you didn't remove the T3 disks cleanly (ie: at VxVM level) before removing physically and rebooting, it's possible these disks had the active copies, so this would be why there are no valid copies found on the EMC disks now.
If the disks weren't removed cleanly, then you may need to rebuild the config from previous output (if you'd saved it before), or worst case you may need to put the T3 disks back. Please confirm how the disks were removed from the OS so we can eliminate this as a possibility/determine how to proceed.
04-08-2011 06:03 AM
Yes i did vxdg -g rootdg rmdisk T3_device and what was left in rootdg were the EMC luns....
Also i am past the valid copies now as vxdctl is in enable mode. Now i just need to get the old disk back under rootdg.
thanks