09-08-2011 12:16 PM
Hello everyone,
I'm following the steps in the Veritas Cluster Server Installation Guide Solaris 5.1, Chapter 7 Configuring VCS Clusters for Data Integrity the section Verifying that the nodes have access to the same disk. The instructions are to run the "vxfenadm -i diskpath" command and confirm that the serial number is equivalent between the two nodes. All disks are actually LUNs from my SAN and all report the same Vendor ID, Product ID and Revision but not all my disks are reporting the same serial number. Below is a table of my disks and the serial number that get reported. The first 26 characters of each serial number are the same so I have removed them to make the table a bit clearer.
Disk |
Node 1 Serial No |
Node 2 Serial No |
/dev/rdsk/c3t50001FE150109D38d1s0 |
160000 |
160000 |
/dev/rdsk/c3t50001FE150109D38d2s0 |
600000 |
190000 |
/dev/rdsk/c3t50001FE150109D38d3s0 |
190000 |
310000 |
/dev/rdsk/c3t50001FE150109D38d4s0 |
310000 |
360000 |
/dev/rdsk/c3t50001FE150109D38d5s0 |
360000 |
3B0000 |
/dev/rdsk/c3t50001FE150109D38d6s0 |
3B0000 |
600000 |
As you can see, disk D38d2 from node 1 shows the serial number that matches disk D38d6 from node 2. My question is: How to I correct the configuration so that the same disk names on each node actually point to the same LUNs and return the same serial numbers?
Thanks in advance for any help.
Ken
Solved! Go to Solution.
09-12-2011 08:00 AM
It turns out that it's OK if the same disk name on on different servers accesses different LUNs on the SAN so long as the Veritas device names on the different servers access the same LUNs. The path is:
Device > Disk > LUN
So long as the same device accesses the same LUN between nodes, it doesn't really matter what the name of the disk is. Below is the information from my system. Notice that the serial numbers match between nodes which means the same device name is using the same LUN on each node even though the disk name is different.
Node 1 | Node 2 | |||
Serial # | Disk | Device | Disk | Serial # |
160000 | c3t50001FE150109D38d1s2 | eva80000_0 | c3t50001FE150109D38d1s2 | 160000 |
190000 | c3t50001FE150109D38d3s2 | eva80000_1 | c3t50001FE150109D38d2s2 | 190000 |
310000 | c3t50001FE150109D38d4s2 | eva80000_2 | c3t50001FE150109D38d3s2 | 310000 |
360000 | c3t50001FE150109D38d5s2 | eva80000_3 | c3t50001FE150109D38d4s2 | 360000 |
3B0000 | c3t50001FE150109D38d6s2 | eva80000_4 | c3t50001FE150109D38d5s2 | 3B0000 |
600000 | c3t50001FE150109D38d2s2 | eva80000_5 | c3t50001FE150109D38d6s2 | 600000 |
Ken
09-08-2011 01:06 PM
The order of the third column within /etc/vx/disk.info seems to be messed up. The 3rd, 4th and 5th columns of the disk.info files from Node 1 are:
0x4d80030 0x2 eva80000_0
0x4d80048 0x2 eva80000_1
0x4d80040 0x2 eva80000_2
0x4d80020 0x2 eva80000_3
0x4d80028 0x2 eva80000_4
0x4d80038 0x2 eva80000_5
The same columns from disk.info on node 2 are:
0x4d80040 0x2 eva80000_0
0x4d80048 0x2 eva80000_1
0x4d80028 0x2 eva80000_2
0x4d80038 0x2 eva80000_3
0x4d80030 0x2 eva80000_4
0x4d80020 0x2 eva80000_5
Can anyone tell me the best way to correct this info?
Ken
09-08-2011 01:16 PM
More information:
Node 1: vxdisk -e list
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME ATTR
eva80000_0 auto:cdsdisk - - online c4t50001FE150109D39d1s2 -
eva80000_1 auto:cdsdisk - - online c4t50001FE150109D3Dd3s2 -
eva80000_2 auto:cdsdisk - - online c4t50001FE150109D3Dd4s2 -
eva80000_3 auto:cdsdisk - - online c4t50001FE150109D3Bd5s2 -
eva80000_4 auto:cdsdisk - - online c4t50001FE150109D39d6s2 -
eva80000_5 auto:cdsdisk - - online c4t50001FE150109D39d2s2 -
Node 2: vxdisk -e list
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME ATTR
eva80000_0 auto:cdsdisk - - online c4t50001FE150109D39d1s2 -
eva80000_1 auto:cdsdisk - - online c4t50001FE150109D39d2s2 -
eva80000_2 auto:cdsdisk - - online c4t50001FE150109D3Fd3s2 -
eva80000_3 auto:cdsdisk - - online c4t50001FE150109D3Fd4s2 -
eva80000_4 auto:cdsdisk - - online c4t50001FE150109D3Dd5s2 -
eva80000_5 auto:cdsdisk - - online c4t50001FE150109D39d6s2 -
Notice on Node 2 how the disks are listed in order (1 to 6) but on Node 1 the disks are listed in a messed-up order (1, 3-6, 2). Again, any idea how to fixe this? I'm assuming that if I can fix this order, the mis-matched serial number issue will be resolved.
Ken
09-08-2011 10:46 PM
Hi Ken,
this happens depending on how OS device tree was built up, the order in which OS tree scans the disk, same is developed for veritas device tree as well..
To fix this, you will need to use parameter from vxddladm,
vxddladm set namingscheme=ebn persistence=yes use_avid=yes
in above, ebn is enclosure based naming, avid is array volume ID, so your Luns will be scanned & assigned the names depending on array volume ID (which is same for all nodes) .. so device names should be consistent ..
hope this helps
G
09-09-2011 07:19 AM
I ran the command "vxddladm set namingscheme=ebn persistence=yes use_avid=yes" on both servers - it did not give any output. I then rebooted both servers and checked the serial numbers returned by "vxfenadm -i /dev/rdsk/<disk_name>".
Nothing has changed. The one disk that reported the same serial number on both nodes is still correct and all other disks are mis-matched.
Do you have any other suggestions?
Ken
09-09-2011 08:18 AM
Regarding the avid command, one possibility is that AVID (Array Volume ID) is not available with that array/ASL as it requires vendor specfic knowledge/interaction coded into the library
If not available, VxVM will have to auto generate the disk ids _1, _2 etc.
09-09-2011 09:04 AM
Our SAN is an HP EVA 8000.
Is there a way to override the assignment of disk IDs?
09-09-2011 01:58 PM
FYI: I've opened a case with Symantec support and have generated and uploaded vxexplorer files from each of my servers for their review. I'll post updates here as they occur.
Ken
09-12-2011 08:00 AM
It turns out that it's OK if the same disk name on on different servers accesses different LUNs on the SAN so long as the Veritas device names on the different servers access the same LUNs. The path is:
Device > Disk > LUN
So long as the same device accesses the same LUN between nodes, it doesn't really matter what the name of the disk is. Below is the information from my system. Notice that the serial numbers match between nodes which means the same device name is using the same LUN on each node even though the disk name is different.
Node 1 | Node 2 | |||
Serial # | Disk | Device | Disk | Serial # |
160000 | c3t50001FE150109D38d1s2 | eva80000_0 | c3t50001FE150109D38d1s2 | 160000 |
190000 | c3t50001FE150109D38d3s2 | eva80000_1 | c3t50001FE150109D38d2s2 | 190000 |
310000 | c3t50001FE150109D38d4s2 | eva80000_2 | c3t50001FE150109D38d3s2 | 310000 |
360000 | c3t50001FE150109D38d5s2 | eva80000_3 | c3t50001FE150109D38d4s2 | 360000 |
3B0000 | c3t50001FE150109D38d6s2 | eva80000_4 | c3t50001FE150109D38d5s2 | 3B0000 |
600000 | c3t50001FE150109D38d2s2 | eva80000_5 | c3t50001FE150109D38d6s2 | 600000 |
Ken
09-12-2011 08:06 AM
Hi
Look up the "assign names" option in the man page for vxddladm.
You should be able to use vxgetdmpnames to produce an output of your current mappings, edit it then use it in conjunction with the assign names option.
There should be a KM article on this, will see if I can find it and post it here
cheers
09-12-2011 09:42 AM
I'd be interested in seeing the KM on this however I'm comfortable enough that I'll continue on with my installation. FYI: I don't see "vxgetdmpnames" on my installation so there may be an additional install required to get this to work.
> vxgetdmpnames bash: vxgetdmpnames: command not found
Ken
09-13-2011 01:58 AM
Hi,
Below is a SORT link to the 5.1SP1/Solaris vxvm admin guide (html) showing the topic of interest
https://sort.symantec.com/public/documents/sfha/5.1sp1/solaris/productguides/html/vxvm_admin/ch04s07.htm
The vxgetdmpnames utility shoul be in /etc/vx/bin, it is only in the recent releases.
Also found this KM article
http://www.symantec.com/docs/TECH69648
cheers