cancel
Showing results for 
Search instead for 
Did you mean: 

To form the cluster to share device

Home_224
Level 6

Dear All,

 

The problem is complex , l wish that there is professional expert to assist with me 

The system build in the VCS and VXVM on solairs 9  , I plan to form the cluster to connect the san storage, it is no record /etc/vxfentab , vxfendg and vxdisk -o alldg list , no vfendrootdg , so it failed to start the vxfen daemon.    

At present, I have no connect to extend FC storage and VCS each node.  I check the system has connect to share disk group sitdg but no idea it is remove from the vxvm and vcs. 

root@devuardbs01 # vxdisk -o alldgs list
DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 auto:sliced rootdg01 rootdg online
c1t1d0s2 auto:sliced rootdg02 rootdg online

root@devuardbs01 # hastatus -sum

-- SYSTEM STATE
-- System State Frozen

A devuardbs01 RUNNING 0
A devuardbs02 RUNNING 0

-- GROUP STATE
-- Group System Probed AutoDisabled State

B Oracle_MultiNIC devuardbs01 Y N ONLINE
B Oracle_MultiNIC devuardbs02 Y N ONLINE
B Oracle_dev_SG devuardbs01 Y N OFFLINE|FAULTED
B Oracle_dev_SG devuardbs02 Y N OFFLINE|FAULTED
B Oracle_sit_SG devuardbs01 Y N OFFLINE|FAULTED
B Oracle_sit_SG devuardbs02 Y N OFFLINE|FAULTED

-- RESOURCES FAILED
-- Group Type Resource System

C Oracle_dev_SG DiskGroup Oracle_devdg_Diskgroup devuardbs01
C Oracle_dev_SG DiskGroup Oracle_devdg_Diskgroup devuardbs02
C Oracle_sit_SG DiskGroup Oracle_sitdg_Diskgroup devuardbs01
C Oracle_sit_SG DiskGroup Oracle_sitdg_Diskgroup devuardbs02

My question is how to enable the VCS and enable disk group  to access the same disk group right now.   Can you anyone to assist with us ?

Thanks,

 

Alfred 

 

 

14 REPLIES 14

frankgfan
Moderator
Moderator
   VIP   

Hello,

From the outputs posted, the host machine (VCS node) lost external storage.

First thing first, does Solaris see the external storage?

 

Can you run the commands below and post the outputs?

 

1. echo | format

2. modinfo | grep vx

3. ls -l /etc/vx/*.exclude

4. vxdctl license

5. uptime

Regards,

Frank

 

Hi Frank,

root@devuardbs01 # modinfo | grep vx
17 11c81b2 21ddf9 275 1 vxio (VxVM 4.1_MP2_RP6 I/O driver)
18 13c5103 36263 274 1 vxdmp (VxVM 4.1_MP2_RP6: DMP Driver)
46 7809cb53 14a1 276 1 vxspec (VxVM 4.1_MP2_RP6 control/status)
133 7804d431 cc3 277 1 vxportal (VxFS 4.1_REV-4.1MP2RP5b_sol por)
134 784b8000 17254d 8 1 vxfs (VxFS 4.1_REV-4.1MP2RP5b_sol Sun)
302 78838000 3c6d1 282 1 vxfen (VRTS Fence 4.1MP2)
root@devuardbs01 # ls -l /etc/vx/*.exclude
/etc/vx/*.exclude: No such file or directory
root@devuardbs01 # ls -l /etc/vx/*.exclude
/etc/vx/*.exclude: No such file or directory
root@devuardbs01 # vxdctl license
All features are available:
Mirroring
Root Mirroring
Concatenation
Disk-spanning
Striping
RAID-5
VxSmartSync
Array Snapshot Integration Feature
Clustering-full
FastResync
DGSJ
DMP (multipath enabled)
CDS
Dynamic LUN Expansion
Hardware assisted copy
root@devuardbs01 # uptime
2:41pm up 1 day(s), 2:30, 1 user, load average: 0.03, 0.04, 0.05
root@devuardbs01 # echo | format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
0. c1t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@1c,600000/scsi@2/sd@0,0
1. c1t1d0 <HITACHI-HUS103073FL3800-SL01 cyl 52541 alt 2 hd 3 sec 907>
/pci@1c,600000/scsi@2/sd@1,0
2. c3t40d0 <SUN-StorEdge3510-415F cyl 24998 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff89cc3e,0
Specify disk (enter its number): Specify disk (enter its number):
root@devuardbs01 #

----------------------------------------------------------------------

root@devuardbs02 # vxdctl -c mode
mode: enabled: cluster inactive
root@devuardbs02 # modinfo | grep vx
17 11c81b2 21ddf9 275 1 vxio (VxVM 4.1_MP2_RP6 I/O driver)
18 13c5103 36263 274 1 vxdmp (VxVM 4.1_MP2_RP6: DMP Driver)
32 7809cb53 14a1 276 1 vxspec (VxVM 4.1_MP2_RP6 control/status)
133 782b7431 cc3 277 1 vxportal (VxFS 4.1_REV-4.1MP2RP5b_sol por)
134 784ae000 17254d 8 1 vxfs (VxFS 4.1_REV-4.1MP2RP5b_sol Sun)
root@devuardbs02 # ls -l /etc/vx/*.exclude
/etc/vx/*.exclude: No such file or directory
root@devuardbs02 # vxdctl license
All features are available:
Mirroring
Root Mirroring
Concatenation
Disk-spanning
Striping
RAID-5
VxSmartSync
Array Snapshot Integration Feature
Clustering-full
FastResync
DGSJ
DMP (multipath enabled)
CDS
Dynamic LUN Expansion
Hardware assisted copy
root@devuardbs02 # echo | format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
0. c1t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@1c,600000/scsi@2/sd@0,0
1. c1t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@1c,600000/scsi@2/sd@1,0
2. c3t44d0 <SUN-StorEdge3510-415F cyl 24998 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@1/fp@0,0/ssd@w256000c0ffc9cc3e,0
Specify disk (enter its number): Specify disk (enter its number):

Hi Fank,

I already connect the host to service SE3510 but no idea how to add disk to VCS.

Disk group: uatdg

dg uatdg default default 5000 1524026009.10.devuardbs02

dm uatdg01 c3t44d0s2 auto 2048 51193600 -

v uat_vol01 - ENABLED ACTIVE 2097152 SELECT - fsgen
pl uat_vol01-01 uat_vol01 ENABLED ACTIVE 2097152 CONCAT - RW
sd uatdg01-01 uat_vol01-01 uatdg01 0 2097152 0 c3t44d0 ENA

 

dg uatdg default default 16000 1524027509.10.devuardbs01

dm uatdg01 c3t40d0s2 auto 2048 51193600 -

v uat_vol01 - ENABLED ACTIVE 2097152 SELECT - fsgen
pl uat_vol01-01 uat_vol01 ENABLED ACTIVE 2097152 CONCAT - RW
sd uatdg01-01 uat_vol01-01 uatdg01 0 2097152 0 c3t40d0 ENA

root@devuardbs01 # vxdctl -c mode
mode: enabled: cluster inactive

root@devuardbs02 # vxdctl -c mode
mode: enabled: cluster inactive

 

frankgfan
Moderator
Moderator
   VIP   

in your previous vxdisk -o alldgs list output, there were only two internal; disks below on c1:

root@devuardbs01 # vxdisk -o alldgs list
DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 auto:sliced rootdg01 rootdg online
c1t1d0s2 auto:sliced rootdg02 rootdg online

now OS sees 3 disks and one of them is on c3

c3t44d0

 

and on other node

c3t40d0

 

Can you please run the commands below on both the nodes and post the output

 

1. vxdctl enable

2. vxdisk -o alldgs list

3. vxprint -ht

4. hastatus -sum

5. gabconfig -a

Veritas 4.1 is no longer a supported version, please upgrade to a supported version (which is FREE of charge).

Regards,

Frank

Mouse
Moderator
Moderator
Partner    VIP    Accredited Certified

If you have connected the storage back you need to make sure the disks are visible in format output (just run format) or you may need to run devfsadm/luxadm or even reboot the system.

If disks are visible in the format utility, run 'vxdctl enable' or vxdisk scandisks and then vxdisk list and make sure the disks are visible.

If they are, you can try to run hagrp -online and bring the service group up

Hi Frank,

rovxdctl enable

 vxdisk -o alldgs list
DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 auto:sliced rootdg01 rootdg online
c1t1d0s2 auto:sliced rootdg02  rootdg online
c3t40d0s2 auto:cdsdisk - - online
root@devuardbs01 # vxprint -ht
Disk group: rootdg

dg rootdg default default 112000 1323751603.17.devuardbs01

dg rootdg default default 126000 1321933389.13.devuardbs02

dm rootdg01 c1t0d0s2 auto 20351 143328960 -
dm rootdg02 c1t1d0s2 auto 20351 143328960 -

v rootvol - ENABLED ACTIVE 20972736 ROUND - root
pl rootvol-01 rootvol ENABLED ACTIVE 20972736 CONCAT - RW
sd rootdg01-B0 rootvol-01 rootdg01 143328959 1 0 c1t0d0 ENA
sd rootdg01-02 rootvol-01 rootdg01 0 20972735 1 c1t0d0 ENA
pl rootvol-02 rootvol ENABLED ACTIVE 20972736 CONCAT - RW
sd rootdg02-01 rootvol-02 rootdg02 0 20972736 0 c1t1d0s2 ENA

v swapvol - ENABLED ACTIVE 16780224 ROUND - swap
pl swapvol-01 swapvol ENABLED ACTIVE 16780224 CONCAT - RW
sd rootdg01-01 swapvol-01 rootdg01 20972735 16780224 0 c1t0d0 ENA
pl swapvol-02 swapvol ENABLED ACTIVE 16780224 CONCAT - RW
sd rootdg02-02 swapvol-02 rootdg02 20972736 16780224 0 c1t1d0s2 ENA

v usr - ENABLED ACTIVE 20972736 ROUND - fsgen
pl usr-01 usr ENABLED ACTIVE 20972736 CONCAT - RW
sd rootdg01-04 usr-01 rootdg01 58725695 20972736 0 c1t0d0 ENA
pl usr-02 usr ENABLED ACTIVE 20972736 CONCAT - RW
sd rootdg02-04 usr-02 rootdg02 58725696 20972736 0 c1t1d0s2 ENA

v u01 - ENABLED ACTIVE 20972736 ROUND - fsgen
pl u01-01 u01 ENABLED ACTIVE 20972736 CONCAT - RW
sd rootdg01-03 u01-01 rootdg01 79698431 20972736 0 c1t0d0 ENA
pl u01-02 u01 ENABLED ACTIVE 20972736 CONCAT - RW
sd rootdg02-05 u01-02 rootdg02 79698432 20972736 0 c1t1d0s2 ENA

v var - ENABLED ACTIVE 20972736 ROUND - fsgen
pl var-01 var ENABLED ACTIVE 20972736 CONCAT - RW
sd rootdg01-05 var-01 rootdg01 37752959 20972736 0 c1t0d0 ENA
pl var-02 var ENABLED ACTIVE 20972736 CONCAT - RW
sd rootdg02-03 var-02 rootdg02 37752960 20972736 0 c1t1d0s2 ENA

-- SYSTEM STATE

-- System State Frozen

A devuardbs01 RUNNING 0
A devuardbs02 RUNNING 0

-- GROUP STATE
-- Group System Probed AutoDisabled State

B Oracle_MultiNIC devuardbs01 Y N ONLINE
B Oracle_MultiNIC devuardbs02 Y N ONLINE
B Oracle_dev_SG devuardbs01 Y N OFFLINE|FAULTED
B Oracle_dev_SG devuardbs02 Y N OFFLINE|FAULTED
B Oracle_sit_SG devuardbs01 Y N OFFLINE|FAULTED
B Oracle_sit_SG devuardbs02 Y N OFFLINE|FAULTED

-- RESOURCES FAILED
-- Group Type Resource System

C Oracle_dev_SG DiskGroup Oracle_devdg_Diskgroup devuardbs01
C Oracle_dev_SG DiskGroup Oracle_devdg_Diskgroup devuardbs02
C Oracle_sit_SG DiskGroup Oracle_sitdg_Diskgroup devuardbs01
C Oracle_sit_SG DiskGroup Oracle_sitdg_Diskgroup devuardbs02
root@devuardbs01 # gabconfig -a
GAB Port Memberships
===============================================================
Port a gen 1a061d membership 01
Port h gen 1a0620 membership 01

Hi Mouse,

I can see the disk on the OS and VXVM but no idea how to share it on the node 1 /2 .  I want to add the new disk group to share on the cluster for lab test.  

 

Hi Frank,

Due to the Production and UAT is using the same MP4.1 version on VXVM and VCS,  so we need to the lab test in the UAT, if it upgrade the new version on UAT, it would not be the same PRODUCTION , so end users may doubt.  

I find the package VRTScavf and VRTSglm package mainly for share disk on VCS, I download it and patchadd but there is some dependency patches, consultant for veritas but not feedback. 

 

I failed to run the command cfscluster config  and vxdctl -c mode , 

frankgfan
Moderator
Moderator
   VIP   

from the output below

vxdisk -o alldgs list
DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 auto:sliced rootdg01 rootdg online
c1t1d0s2 auto:sliced rootdg02  rootdg online
c3t40d0s2 auto:cdsdisk - - online

on VxVM level, the disk group on external storage no longer exist.  Its gone.  The cause could be one of the followings

 

1. a wrong disk was allocated to the cluster

or

2.  the disk c3t40d0 was accidentally re-initialized so the original  disk private region was wiped.

in 4,.1m there is a vxconfigrestore command which can be used to restore a coruppted disk private region.

 

Try the command below to see how it goes

 

#vxconfigrestore -p <disk group name>

Do you have a previous saved vxprint -ht output?  or a VRTSexplorer output?

Regards,

Frank

 

frankgfan
Moderator
Moderator
   VIP   

Is the dg in question a cvm dg or a local dg?  if its a local dg, not to worry about vxdctl -c mode and cfsclustadm, there are for shared dg.

Frank

Hi Frankgan,

I don't have no idea the reason there is no VCS and offline the external DG , I never touch this box before when I force to do vcs on this server again.  However, I will try to get the VRTSexplorer for both node or one of node to attach and printout the old vxprint -ht and vfstab here. 

I find there was two diskgroups , devdg and sitdg from hastatus -sum information which connect to extenal SE3510 by FC . But no idea what happen previously the disk group faulted and offline.

Now I create the same name devdg and want to hagrp -clear and hagrp -online to resume the share disk group, but the disk size is not to the same as previous diskgroup .

Anyway I will update some information tomorrow as I am in HK time zone. 

Thank you Frank !

 

 

Hi Frank,

 root@devuardbs01 # vxprint -ht
Disk group: rootdg

dg rootdg default default 112000 1323751603.17.devuardbs01

dm rootdg01 c1t0d0s2 auto 20351 143328960 -
dm rootdg02 c1t1d0s2 auto 21767 142942293 -

v rootvol - ENABLED ACTIVE 20972736 ROUND - root
pl rootvol-01 rootvol ENABLED ACTIVE 20972736 CONCAT - RW
sd rootdg01-01 rootvol-01 rootdg01 0 20972736 0 c1t0d0 ENA
pl rootvol-02 rootvol ENABLED ACTIVE 20973468 CONCAT - RW
sd rootdg02-01 rootvol-02 rootdg02 0 20973468 0 c1t1d0 ENA

v swapvol - ENABLED ACTIVE 16780224 ROUND - swap
pl swapvol-01 swapvol ENABLED ACTIVE 16780224 CONCAT - RW
sd rootdg01-02 swapvol-01 rootdg01 20972736 16780224 0 c1t0d0 ENA
pl swapvol-02 swapvol ENABLED ACTIVE 16780407 CONCAT - RW
sd rootdg02-02 swapvol-02 rootdg02 20973468 16780407 0 c1t1d0 ENA

v usr - ENABLED ACTIVE 20972736 ROUND - fsgen
pl usr-01 usr ENABLED ACTIVE 20972736 CONCAT - RW
sd rootdg01-04 usr-01 rootdg01 58725696 20972736 0 c1t0d0 ENA
pl usr-02 usr ENABLED ACTIVE 20973468 CONCAT - RW
sd rootdg02-04 usr-02 rootdg02 58727343 20973468 0 c1t1d0 ENA

v u01 - ENABLED ACTIVE 20972736 ROUND - fsgen
pl u01-01 u01 ENABLED ACTIVE 20972736 CONCAT - RW
sd rootdg01-05 u01-01 rootdg01 79698432 20972736 0 c1t0d0 ENA
pl u01-02 u01 ENABLED ACTIVE 20973468 CONCAT - RW
sd rootdg02-05 u01-02 rootdg02 79700811 20973468 0 c1t1d0 ENA

v var - ENABLED ACTIVE 20972736 ROUND - fsgen
pl var-01 var ENABLED ACTIVE 20972736 CONCAT - RW
sd rootdg01-03 var-01 rootdg01 37752960 20972736 0 c1t0d0 ENA
pl var-02 var ENABLED ACTIVE 20973468 CONCAT - RW
sd rootdg02-03 var-02 rootdg02 37753875 20973468 0 c1t1d0 ENA

Disk group: devdg

dg devdg default default 19000 1524132403.14.devuardbs01

dm devdg01 c3t40d0s2 auto 2048 51193600 -

dg rootdg default default 126000 1321933389.13.devuardbs02

dm rootdg01 c1t0d0s2 auto 20351 143328960 -
dm rootdg02 c1t1d0s2 auto 20351 143328960 -

v rootvol - ENABLED ACTIVE 20972736 ROUND - root
pl rootvol-01 rootvol ENABLED ACTIVE 20972736 CONCAT - RW
sd rootdg01-B0 rootvol-01 rootdg01 143328959 1 0 c1t0d0 ENA
sd rootdg01-02 rootvol-01 rootdg01 0 20972735 1 c1t0d0 ENA
pl rootvol-02 rootvol ENABLED ACTIVE 20972736 CONCAT - RW
sd rootdg02-01 rootvol-02 rootdg02 0 20972736 0 c1t1d0s2 ENA

v swapvol - ENABLED ACTIVE 16780224 ROUND - swap
pl swapvol-01 swapvol ENABLED ACTIVE 16780224 CONCAT - RW
sd rootdg01-01 swapvol-01 rootdg01 20972735 16780224 0 c1t0d0 ENA
pl swapvol-02 swapvol ENABLED ACTIVE 16780224 CONCAT - RW
sd rootdg02-02 swapvol-02 rootdg02 20972736 16780224 0 c1t1d0s2 ENA

v usr - ENABLED ACTIVE 20972736 ROUND - fsgen
pl usr-01 usr ENABLED ACTIVE 20972736 CONCAT - RW
sd rootdg01-04 usr-01 rootdg01 58725695 20972736 0 c1t0d0 ENA
pl usr-02 usr ENABLED ACTIVE 20972736 CONCAT - RW
sd rootdg02-04 usr-02 rootdg02 58725696 20972736 0 c1t1d0s2 ENA

v u01 - ENABLED ACTIVE 20972736 ROUND - fsgen
pl u01-01 u01 ENABLED ACTIVE 20972736 CONCAT - RW
sd rootdg01-03 u01-01 rootdg01 79698431 20972736 0 c1t0d0 ENA
pl u01-02 u01 ENABLED ACTIVE 20972736 CONCAT - RW
sd rootdg02-05 u01-02 rootdg02 79698432 20972736 0 c1t1d0s2 ENA

v var - ENABLED ACTIVE 20972736 ROUND - fsgen
pl var-01 var ENABLED ACTIVE 20972736 CONCAT - RW
sd rootdg01-05 var-01 rootdg01 37752959 20972736 0 c1t0d0 ENA
pl var-02 var ENABLED ACTIVE 20972736 CONCAT - RW
sd rootdg02-03 var-02 rootdg02 37752960 20972736 0 c1t1d0s2 ENA

Disk group: devdg

dg devdg default default 8000 1524190608.12.devuardbs02

dm devdg01 c3t44d0s2 auto 2048 51193600 -

 

 

Hi Frank,

Once I online the share diskgroup, check the log, it shows the message here 

2018/04/20 14:35:56 VCS INFO V-16-1-10305 Resource Oracle_dev_IPMultiNICB (Owner: unknown, Group: Oracle_dev_SG) is offline on devuardbs02 (VCS initiated)
2018/04/20 14:35:59 VCS INFO V-16-1-10305 Resource Oracle_devdg_Diskgroup (Owner: unknown, Group: Oracle_dev_SG) is offline on devuardbs02 (VCS initiated)
2018/04/20 14:35:59 VCS ERROR V-16-1-10205 Group Oracle_dev_SG is faulted on system devuardbs02
2018/04/20 14:35:59 VCS NOTICE V-16-1-10446 Group Oracle_dev_SG is offline on system devuardbs02
2018/04/20 14:35:59 VCS INFO V-16-1-10493 Evaluating devuardbs01 as potential target node for group Oracle_dev_SG
2018/04/20 14:35:59 VCS INFO V-16-1-50010 Group Oracle_dev_SG is online or faulted on system devuardbs01
2018/04/20 14:35:59 VCS INFO V-16-1-10493 Evaluating devuardbs02 as potential target node for group Oracle_dev_SG
2018/04/20 14:35:59 VCS INFO V-16-1-50010 Group Oracle_dev_SG is online or faulted on system devuardbs02
2018/04/20 14:35:59 VCS INFO V-16-6-15004 (devuardbs01) hatrigger:Failed to send trigger for nofailover; script doesn't exist
2018/04/20 14:35:59 VCS INFO V-16-6-15004 (devuardbs02) hatrigger:Failed to send trigger for postoffline; script doesn't exist

frankgfan
Moderator
Moderator
   VIP   

Sorry for the delay in reply as I was away the past to weeks.

 

The error below


2018/04/20 14:35:59 VCS ERROR V-16-1-10205 Group Oracle_dev_SG is faulted on system devuardbs02

shows the dg is fauletd on node *02.

Can you run the commands below and post the output?

 

1. gabconfig -a     

2. hastatus -sum

3. vxdg list

Regards,

 

Frank