jani
15 years agoLevel 3
Problem using iSCSI and Storage Foundation 5.0MP3
Hi,
I'm trying to create two node cluster using iSCSI as a disk array. My OS is Solaris 10 (sol-10-u8-ga-x86-dvd.iso).
My iSCSI targets are created using solaris iscstadm tool. Creation goes ok and also my two cluster nodes can see the iSCSI disks.
Server used to create iSCSI targets is a separate (third) Solaris server.
In my system if DMP is activated I can only see aluadis0_0
-bash-3.00# vxdisk list
DEVICE TYPE DISK GROUP STATUS
aluadisk0_0 auto:none - - online invalid
c0d0s2 auto:none - - online invalid
If I disable DMP then I can see all the disks on both nodes
-bash-3.00# vxdisk list
DEVICE TYPE DISK GROUP STATUS
c0d0s2 auto:none - - online invalid
fabric_0 auto:none - - online invalid
fabric_1 auto:none - - online invalid
fabric_2 auto:none - - online invalid
fabric_3 auto:none - - online invalid
fabric_4 auto:none - - online invalid
fabric_5 auto:none - - online invalid
fabric_6 auto:none - - online invalid
fabric_7 auto:none - - online invalid
And I'm able to see all these disks on both cluster nodes also using format command
-bash-3.00# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0d0 <DEFAULT cyl 1250 alt 2 hd 255 sec 63>
/pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0
1. c2t600144F04B583D7000001E3788F54300d0 <DEFAULT cyl 771 alt 2 hd 128 sec 32>
/scsi_vhci/disk@g600144f04b583d7000001e3788f54300
2. c2t600144F04B583DCF00001E3788F54300d0 <DEFAULT cyl 770 alt 2 hd 128 sec 32>
/scsi_vhci/disk@g600144f04b583dcf00001e3788f54300
3. c2t600144F04B583DD000001E3788F54300d0 <DEFAULT cyl 197 alt 2 hd 64 sec 32>
/scsi_vhci/disk@g600144f04b583dd000001e3788f54300
4. c2t600144F04B583DD100001E3788F54300d0 <DEFAULT cyl 197 alt 2 hd 64 sec 32>
/scsi_vhci/disk@g600144f04b583dd100001e3788f54300
5. c2t600144F04B583DD200001E3788F54300d0 <DEFAULT cyl 771 alt 2 hd 128 sec 32>
/scsi_vhci/disk@g600144f04b583dd200001e3788f54300
6. c2t600144F04B583DD300001E3788F54300d0 <DEFAULT cyl 771 alt 2 hd 128 sec 32>
/scsi_vhci/disk@g600144f04b583dd300001e3788f54300
7. c2t600144F04B583DE100001E3788F54300d0 <DEFAULT cyl 197 alt 2 hd 64 sec 32>
/scsi_vhci/disk@g600144f04b583de100001e3788f54300
8. c2t600144F04B583FB500001E3788F54300d0 <DEFAULT cyl 771 alt 2 hd 128 sec 32>
/scsi_vhci/disk@g600144f04b583fb500001e3788f54300
With format command disks are presented in the same order on both servers.
The problem is that disks are in different order in my servers. For example disk fabric_1 on server1 is fabric_3 on server2. Also the order changes if I reboot the servers or if I do changes to the disks.
How to make veritas to find the disks in the same order everytime I boot servers and also how to make my two servers to find iSCSI devices in the same order every time?
Thanks in advance
br, J
I'm trying to create two node cluster using iSCSI as a disk array. My OS is Solaris 10 (sol-10-u8-ga-x86-dvd.iso).
My iSCSI targets are created using solaris iscstadm tool. Creation goes ok and also my two cluster nodes can see the iSCSI disks.
Server used to create iSCSI targets is a separate (third) Solaris server.
In my system if DMP is activated I can only see aluadis0_0
-bash-3.00# vxdisk list
DEVICE TYPE DISK GROUP STATUS
aluadisk0_0 auto:none - - online invalid
c0d0s2 auto:none - - online invalid
If I disable DMP then I can see all the disks on both nodes
-bash-3.00# vxdisk list
DEVICE TYPE DISK GROUP STATUS
c0d0s2 auto:none - - online invalid
fabric_0 auto:none - - online invalid
fabric_1 auto:none - - online invalid
fabric_2 auto:none - - online invalid
fabric_3 auto:none - - online invalid
fabric_4 auto:none - - online invalid
fabric_5 auto:none - - online invalid
fabric_6 auto:none - - online invalid
fabric_7 auto:none - - online invalid
And I'm able to see all these disks on both cluster nodes also using format command
-bash-3.00# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0d0 <DEFAULT cyl 1250 alt 2 hd 255 sec 63>
/pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0
1. c2t600144F04B583D7000001E3788F54300d0 <DEFAULT cyl 771 alt 2 hd 128 sec 32>
/scsi_vhci/disk@g600144f04b583d7000001e3788f54300
2. c2t600144F04B583DCF00001E3788F54300d0 <DEFAULT cyl 770 alt 2 hd 128 sec 32>
/scsi_vhci/disk@g600144f04b583dcf00001e3788f54300
3. c2t600144F04B583DD000001E3788F54300d0 <DEFAULT cyl 197 alt 2 hd 64 sec 32>
/scsi_vhci/disk@g600144f04b583dd000001e3788f54300
4. c2t600144F04B583DD100001E3788F54300d0 <DEFAULT cyl 197 alt 2 hd 64 sec 32>
/scsi_vhci/disk@g600144f04b583dd100001e3788f54300
5. c2t600144F04B583DD200001E3788F54300d0 <DEFAULT cyl 771 alt 2 hd 128 sec 32>
/scsi_vhci/disk@g600144f04b583dd200001e3788f54300
6. c2t600144F04B583DD300001E3788F54300d0 <DEFAULT cyl 771 alt 2 hd 128 sec 32>
/scsi_vhci/disk@g600144f04b583dd300001e3788f54300
7. c2t600144F04B583DE100001E3788F54300d0 <DEFAULT cyl 197 alt 2 hd 64 sec 32>
/scsi_vhci/disk@g600144f04b583de100001e3788f54300
8. c2t600144F04B583FB500001E3788F54300d0 <DEFAULT cyl 771 alt 2 hd 128 sec 32>
/scsi_vhci/disk@g600144f04b583fb500001e3788f54300
With format command disks are presented in the same order on both servers.
The problem is that disks are in different order in my servers. For example disk fabric_1 on server1 is fabric_3 on server2. Also the order changes if I reboot the servers or if I do changes to the disks.
How to make veritas to find the disks in the same order everytime I boot servers and also how to make my two servers to find iSCSI devices in the same order every time?
Thanks in advance
br, J
- It's not possible to force the disks to be discovered in a particular order. However, if you want the order to remain the same after reboots, refer to the following links to set persistent naming:
VxVM 5.0MP3 (Solaris) Administrator's Guide - pp 106-108
http://sfdoccentral.symantec.com/sf/5.0MP3/solaris/pdf/vxvm_admin.pdf
html versions of relevant sections:
Changing the disk-naming scheme
http://sfdoccentral.symantec.com/sf/5.0MP3/solaris/html/vxvm_admin/ch02s05.htm
Displaying the disk-naming scheme
http://sfdoccentral.symantec.com/sf/5.0MP3/solaris/html/vxvm_admin/ch02s05s01.htm
Regenerating persistent device names (for explanation of impact/implications of persistent naming)
http://sfdoccentral.symantec.com/sf/5.0MP3/solaris/html/vxvm_admin/ch02s05s02.htm
The following technote is also helpful (refers to 4.1 but still applicable to later versions)
TN 278998: VERITAS Volume Manager 4.1 persistently labels disk devices. The 'vxdisk list' command will no longer reflect underlying OS device name changes
http://seer.entsupport.symantec.com/docs/278998.htm
This still won't guarantee/cause the disks to be picked up in the same order across both systems though. To match up, use vxdisk -e list to display the underlying OS name to match up manually.
If the names absolutely must match on both nodes, you can try the following unsupported workaround/hack to the disk.info file. Note this is not supported so it's at your own risk, and any new disks added later may/will still be in different order.
1. If VCS is running, freeze any service groups that have volume manager resources (preferably stop VCS all together/do this in an outage window as it's entirely possible this could lead to outage if there are errors in the file/any problems)
# hagrp -freeze <group> ### repeat for all groups with volume, dg, etc resources
2. Ensure persistent naming is on/enabled
# vxddladm get namingscheme
3. Back up the /etc/vx/disk.info file
# cp -p /etc/vx/disk.info /other/tmp/location
4. Edit the disk.info file to so the disks in the desired order. Ensure there are no duplicates/no typos in the file - ie: try to limit edits to the numbers to avoid introducing any errors
5. Restart vxconfigd using vxconfigd -k or reboot the server
# vxconfigd -k
6. Unfreeze any groups that were frozen in step 1
Note: for the issue of DMP only seeing one device instead of eight, have you checked if your configuration/settings are supported in the HCL/Hardware technote? If they are listed as supported it may be worth logging a case to see why the disks aren't displaying correctly with DMP enabled.
Hardware Compatibility List for SF HA 5.0 [...] - http://support.veritas.com/docs/283161
ftp://exftpp.symantec.com/pub/support/products/Foundation_Suite/283161.pdf
SF HA Hardware TechNote for 5.0 MP1 and 5.0 MP3 (AIX, Solaris)[etc] - http://support.veritas.com/docs/283282
ftp://exftpp.symantec.com/pub/support/products/Foundation_Suite/283282.pdf