vxdisk list error after vxdctl enable

[Mahmoud_Moussa]

Dear All, 

  Kindly I have the following system  
Solaris 8 with Veritas Volume manager 3.5

I had two disk three disks failing 
two of them in a volume tdisk4 and the third one was part of another volume named tdisk5 

the thing is that I successfully replaced the three disks and now the systems saw them and I can manage them but when I try to put the new discovered luns under VxVm control it failed

I issue the command 

vxdiskconfig

or 

vxdctl enable

then when I issue the command 

vxdisk list

It shows that they have error status 
they are given a name T3_0 , T30_1 , T30_2 , but with error status 

What could be the possible cause for that 
vxdisk  list T30_0 for example it shows that the error is " Device path is not valid"
However the lun is seen successsfully from the Solaris OS prompt and I can edit its slices , .. etc 

Thanks for your help in advance 

[Mahmoud_Moussa]

Kindly fine below the output of the some commands to give you a further information

root@apps # vxdisk list

DEVICE       TYPE      DISK         GROUP        STATUS

T30_0        sliced    -            -            error

T30_1        sliced    -            -            error

T30_2        sliced    -            -            error

T30_3        sliced    -            -            error

T30_4        sliced    t3dg01       t3dg         online

T30_5        sliced    t3dg202      t3dg2        online

T30_6        sliced    t3dg200      t3dg2        online

T30_7        sliced    t3dg201      t3dg2        online

T30_8        sliced    t3dg300      t3dg3        online

T30_9        sliced    t3dg301      t3dg3        online

c1t0d0s2     sliced    rootdisk     rootdg       online

c1t1d0s2     sliced    rootmirr     rootdg       online

-            -         t3dg00       t3dg         failed was:T30_2

-            -         t3dg02       t3dg         failed was:T30_0

-            -         t3dg203      t3dg2        removed was:T30_1

root@apps # vxprint

Disk group: rootdg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0

dg rootdg       rootdg       -        -        -        -        -       -

dm rootdisk     c1t0d0s2     -        71124291 -        -        -       -

dm rootmirr     c1t1d0s2     -        71124291 -        -        -       -

v  rootvol      root         ENABLED  16779312 -        ACTIVE   -       -

pl rootvol-01   rootvol      ENABLED  16779312 -        ACTIVE   -       -

sd rootdisk-B0  rootvol-01   ENABLED  1        0        -        -       Block0

sd rootdisk-02  rootvol-01   ENABLED  16779311 1        -        -       -

pl rootvol-02   rootvol      ENABLED  16779312 -        ACTIVE   -       -

sd rootmirr-01  rootvol-02   ENABLED  16779312 0        -        -       -

v  swapvol      swap         ENABLED  16779312 -        ACTIVE   -       -

pl swapvol-01   swapvol      ENABLED  16779312 -        ACTIVE   -       -

sd rootdisk-01  swapvol-01   ENABLED  16779312 0        -        -       -

pl swapvol-02   swapvol      ENABLED  16779312 -        ACTIVE   -       -

sd rootmirr-02  swapvol-02   ENABLED  16779312 0        -        -       -

v  var          fsgen        ENABLED  8389656  -        ACTIVE   -       -

pl var-01       var          ENABLED  8389656  -        ACTIVE   -       -

sd rootdisk-03  var-01       ENABLED  8389656  0        -        -       -

pl var-02       var          ENABLED  8389656  -        ACTIVE   -       -

sd rootmirr-03  var-02       ENABLED  8389656  0        -        -       -

Disk group: t3dg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0

dg t3dg         t3dg         -        -        -        -        -       -

dm t3dg00       -            -        -        -        NODEVICE -       -

dm t3dg01       T30_4        -        161491456 -       -        -       -

dm t3dg02       -            -        -        -        NODEVICE -       -

v  t3disk4      fsgen        DISABLED 532715520 -       ACTIVE   -       -

pl t3disk4-01   t3disk4      DISABLED 532718592 -       NODEVICE -       -

sd t3dg00-01    t3disk4-01   DISABLED 209735680 0       NODEVICE -       -

sd t3dg01-01    t3disk4-01   ENABLED  161491456 209735680 -      -       -

sd t3dg02-01    t3disk4-01   DISABLED 161491456 371227136 NODEVICE -     -

Disk group: t3dg2

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0

dg t3dg2        t3dg2        -        -        -        -        -       -

dm t3dg200      T30_6        -        104866048 -       -        -       -

dm t3dg201      T30_7        -        209735680 -       -        -       -

dm t3dg202      T30_5        -        104866048 -       -        -       -

dm t3dg203      -            -        -        -        REMOVED  -       -

v  t3disk5      fsgen        DISABLED 629202944 -       ACTIVE   -       -

pl t3disk5-01   t3disk5      DISABLED 629203456 -       REMOVED  -       -

sd t3dg200-01   t3disk5-01   ENABLED  104866048 0       -        -       -

sd t3dg201-01   t3disk5-01   ENABLED  209735680 104866048 -      -       -

sd t3dg202-01   t3disk5-01   ENABLED  104866048 314601728 -      -       -

sd t3dg203-01   t3disk5-01   DISABLED 209735680 419467776 REMOVED -      -

Disk group: t3dg3

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0

dg t3dg3        t3dg3        -        -        -        -        -       -

dm t3dg300      T30_8        -        209735680 -       -        -       -

dm t3dg301      T30_9        -        209735680 -       -        -       -

v  t3disk6      fsgen        ENABLED  419471360 -       ACTIVE   -       -

pl t3disk6-01   t3disk6      ENABLED  419471360 -       ACTIVE   -       -

sd t3dg300-01   t3disk6-01   ENABLED  209735680 0       -        -       -

sd t3dg301-01

[gaurav_dong]

Hi,

Can u paste the "vxdisk -eo alldgs list" output ?

Gaurav D

[Mahmoud_Moussa]

I'm not now in front of the server but I  remember the output of the command

vxdisk -e list

which added the last column of the path of each lun and the path was correct 

also the command of vxdisk list T30_0 

for example shows the two redundant paths for the lun and both were enabled
 

eventhough the error says device path is not valid

!!!!

hope this answer helps you and if you still need the output of the command  above I will past its complete output tomorrow :)

Thanks :)

[Marianne]

You need to initialize the new disks and then add them to the respective diskgroups:

vxdisksetup -i T30_0

vxdg -g t3dg -k adddisk  t3dg02=T30_0

Since your volumes have no redundancy, you will need to restore from backup.

[Mahmoud_Moussa]

Dear Marianne

I run the command 

vxdisksetup -i T30_0

but it says 

Error Device T30_0 online failed 
Device path not valid

do you have any clue why ?

Thanks 

[gaurav_dong]

can you post the output

# vxdisk list T30_0

[Daniel_Matheus]

Hi Mahmoud,

I guess the OS enumerated the new LUNs with the old device ID's or your OS device tree is not cleaned up properly.

-Cleanup Solaris device tree
 # devfsadm -Cv
 # vxddladm stop eventsource

-Cleanup powerpath device tree (Only needed if EMC Power Path is installed and configured)
 # powercf -q
 # powermt config
 # powermt check

-Cleanup VxVM device tree
 # mv /etc/vx/disk.info /etc/vx/disk.info.old
 # mv /etc/vx/array.info /etc/vx/array.info.old
 # rm /dev/vx/dmp/*
 # rm /dev/vx/rdmp/*
 # vxdctl enable
 # vxddladm start eventsource

Then try again to initialize the disks and add them to the respective diskgroups.

Another option would be to reboot the system, this will also clean up the device tree

[Mahmoud_Moussa]

VxVm incident 105173

On T3 array,Volume Manager may get the following failure when trying

to add a disk (typically from vxinstall or vxdisksetup):

vxvm:vxdisk: ERROR: Device XXXX: online failed

Device path not valid

This can happen in cases where the T3 disk was re-partitioned (or

re-formatted) prior to disk(s) being added.

I found this peice in a pdf document on Oracle website but I didn't find any solution 

this is the exact same situation so any 

Dear Daniel , 

I couldn't find vxddladm stop eventsource 

also the paths you've given here is not found

/etc/vx/

I only have diag.d & ddl.support in the mentioned directory

the output of 

vxdisk list T30_0

Device T30_0

Devicetag T30_0

type sliced

flags online error private autoconfig

errno Device path not valid

Multipathing information

numpaths 2

c3t50020F230000FC25d9s2 state=enabled type=primary

c4t50020F230000FBBAd9s2 state=enabled type=secondary

[Daniel_Matheus]

Please apologize, I missed the 3.5 version.

The persistent feature which uses the disk.info and array.infow as introduced in 4.1, hence you can't find the files in 3.5.

vxddladm is a command that resides in the sbin directory.

Can you try the above procedure without the renaming of the *.info files?

[Mahmoud_Moussa]

vxddladm doesn't work with the option given above 

vxddladm stop eventsource is not working

It seems to be a wrong option 

It's worth mentioning now that T30_0 suddenly work and now it's online :D

What do you think happened here ?

[Mahmoud_Moussa]

Do I need to do anything regarding the format ? of any of the disks ?

[Daniel_Matheus]

If you ran all the other steps apart from the vxddladm command, then I assume your device tree is clean and the dmp nodes are referencing correct OS devices.

Sorry for wrong option, but it's been a couple years I touched a 3.5 installation ;)

As the disks are all sliced, the new disks format should also be sliced, if that is your question.

If this is a cluster, please also make sure that the devices are correctly discovered on all nodes.

[Marianne · Accepted solution]

Do I need to do anything regarding the format ? of any of the disks ?

Newly added disks/luns need to be labeled using the Solaris 'format' menu.

[Mahmoud_Moussa]

Dear All thanks all for your help

Finally I solved the problem 

each lun has two paths 

I just found that one of the paths the disk is formatted by the veritas volume manager and the other not

so I prtvtoc the vtoc from this one labled by VxVm and applied to the other path of each lun , hence it was changed in the vxdisk list from error status to online invalid

from this point I continued successfully to initialize the disk and add it back to all the defected disk groups ,.... etc until I finally recover the volumes and all now are enabled active 

Of course I lost all the data and now I'm restoring the data from the backup the customer have using Veritas Netbakup 5.1 which is a very old version 

Wish me luck restoring the backup because it's a very old data and and very old backup solution as you can see :)

Thanks again so much for your help