Problem using iSCSI and Storage Foundation 5.0MP3

[jani]

Hi,

I'm trying to create two node cluster using iSCSI as a disk array. My OS is Solaris 10 (sol-10-u8-ga-x86-dvd.iso).
My iSCSI targets are created using solaris iscstadm tool. Creation goes ok and also my two cluster nodes can see the iSCSI disks.
Server used to create iSCSI targets is a separate (third) Solaris server.

In my system if DMP is activated I can only see aluadis0_0
-bash-3.00# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
aluadisk0_0  auto:none       -            -            online invalid
c0d0s2       auto:none       -            -            online invalid

If I disable DMP then I can see all the disks on both nodes
-bash-3.00# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
c0d0s2       auto:none       -            -            online invalid
fabric_0     auto:none       -            -            online invalid
fabric_1     auto:none       -            -            online invalid
fabric_2     auto:none       -            -            online invalid
fabric_3     auto:none       -            -            online invalid
fabric_4     auto:none       -            -            online invalid
fabric_5     auto:none       -            -            online invalid
fabric_6     auto:none       -            -            online invalid
fabric_7     auto:none       -            -            online invalid

And I'm able to see all these disks on both cluster nodes also using format command
-bash-3.00# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c0d0 <DEFAULT cyl 1250 alt 2 hd 255 sec 63>
          /pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0
       1. c2t600144F04B583D7000001E3788F54300d0 <DEFAULT cyl 771 alt 2 hd 128 sec 32>
          /scsi_vhci/disk@g600144f04b583d7000001e3788f54300
       2. c2t600144F04B583DCF00001E3788F54300d0 <DEFAULT cyl 770 alt 2 hd 128 sec 32>
          /scsi_vhci/disk@g600144f04b583dcf00001e3788f54300
       3. c2t600144F04B583DD000001E3788F54300d0 <DEFAULT cyl 197 alt 2 hd 64 sec 32>
          /scsi_vhci/disk@g600144f04b583dd000001e3788f54300
       4. c2t600144F04B583DD100001E3788F54300d0 <DEFAULT cyl 197 alt 2 hd 64 sec 32>
          /scsi_vhci/disk@g600144f04b583dd100001e3788f54300
       5. c2t600144F04B583DD200001E3788F54300d0 <DEFAULT cyl 771 alt 2 hd 128 sec 32>
          /scsi_vhci/disk@g600144f04b583dd200001e3788f54300
       6. c2t600144F04B583DD300001E3788F54300d0 <DEFAULT cyl 771 alt 2 hd 128 sec 32>
          /scsi_vhci/disk@g600144f04b583dd300001e3788f54300
       7. c2t600144F04B583DE100001E3788F54300d0 <DEFAULT cyl 197 alt 2 hd 64 sec 32>
          /scsi_vhci/disk@g600144f04b583de100001e3788f54300
       8. c2t600144F04B583FB500001E3788F54300d0 <DEFAULT cyl 771 alt 2 hd 128 sec 32>
          /scsi_vhci/disk@g600144f04b583fb500001e3788f54300

With format command disks are presented in the same order on both servers.

The problem is that disks are in different order in my servers. For example disk fabric_1 on server1 is fabric_3 on server2. Also the order changes if I reboot the servers or if I do changes to the disks.

How to make veritas to find the disks in the same order everytime I boot servers and also how to make my two servers to find iSCSI devices in the same order every time?

Thanks in advance
br, J

[g_lee · Accepted solution]

It's not possible to force the disks to be discovered in a particular order. However, if you want the order to remain the same after reboots, refer to the following links to set persistent naming:

VxVM 5.0MP3 (Solaris) Administrator's Guide - pp 106-108
http://sfdoccentral.symantec.com/sf/5.0MP3/solaris/pdf/vxvm_admin.pdf

html versions of relevant sections:
Changing the disk-naming scheme
http://sfdoccentral.symantec.com/sf/5.0MP3/solaris/html/vxvm_admin/ch02s05.htm

Displaying the disk-naming scheme
http://sfdoccentral.symantec.com/sf/5.0MP3/solaris/html/vxvm_admin/ch02s05s01.htm

Regenerating persistent device names (for explanation of impact/implications of persistent naming)
http://sfdoccentral.symantec.com/sf/5.0MP3/solaris/html/vxvm_admin/ch02s05s02.htm

The following technote is also helpful (refers to 4.1 but still applicable to later versions)
TN 278998: VERITAS Volume Manager 4.1 persistently labels disk devices. The 'vxdisk list' command will no longer reflect underlying OS device name changes
http://seer.entsupport.symantec.com/docs/278998.htm

This still won't guarantee/cause the disks to be picked up in the same order across both systems though. To match up, use vxdisk -e list to display the underlying OS name to match up manually.

If the names absolutely must match on both nodes, you can try the following unsupported workaround/hack to the disk.info file. Note this is not supported so it's at your own risk, and any new disks added later may/will still be in different order.

1. If VCS is running, freeze any service groups that have volume manager resources (preferably stop VCS all together/do this in an outage window as it's entirely possible this could lead to outage if there are errors in the file/any problems)
# hagrp -freeze <group> ### repeat for all groups with volume, dg, etc resources
2. Ensure persistent naming is on/enabled
# vxddladm get namingscheme
3. Back up the /etc/vx/disk.info file
# cp -p /etc/vx/disk.info /other/tmp/location
4. Edit the disk.info file to so the disks in the desired order. Ensure there are no duplicates/no typos in the file - ie: try to limit edits to the numbers to avoid introducing any errors
5. Restart vxconfigd using vxconfigd -k or reboot the server
# vxconfigd -k
6. Unfreeze any groups that were frozen in step 1

Note: for the issue of DMP only seeing one device instead of eight, have you checked if your configuration/settings are supported in the HCL/Hardware technote? If they are listed as supported it may be worth logging a case to see why the disks aren't displaying correctly with DMP enabled.

Hardware Compatibility List for SF HA 5.0 [...] - http://support.veritas.com/docs/283161
ftp://exftpp.symantec.com/pub/support/products/Foundation_Suite/283161.pdf

SF HA Hardware TechNote for 5.0 MP1 and 5.0 MP3 (AIX, Solaris)[etc] - http://support.veritas.com/docs/283282
ftp://exftpp.symantec.com/pub/support/products/Foundation_Suite/283282.pdf

[Marianne]

Have you checked the Hardware Compatibilty Guide? There are a limited number of arrays for which iSCSI is supported and some of them need an ASL.
Only if the required DMP ASL is installed can the array be used with DMP enabled.

[jani]

Thanks for the tips.

I think my HW is not part of the HCL.  I don't have any actual array just iSCSI configured to solaris server and I'm using just one disk for this. This is also the reason why I disabled the DMP.

I configured my cluster to use the persistent disk naming and it seems to work at least after one reboot. Now I'm able to continue my "project".

Thank you for your help
Jani