Problem with iSCSI and DMP in Storage Foundation 5

[Roger_Zimmerman]

I suffer from one problem with iSCSI software targets and SF5 on Solaris 10 Sparc. I need to have some iSCSI connected drives, not that big nor very fast. But many.

I tried a lot of software targets on Linux (Openfiler and Ubuntu) and Solaris (Open Solaris Nevada) all with the same effect: DMP makes from all seen iSCSI drives only one logical drive (e.g. Disk_8) with some pathes (as many iSCSI mapped drives your OS can see). I learned that this is normal because DMP is looking for the SCSI Serial Number on the drives, which is not transferred via iSCSI or at least not shown by the iSCSI initiator, and not for the UID, which is shown correctly.

This results in the following behavior: as long as there is only one iSCSI drive seen in the OS also SF5 runs correctly, but if there are more than one mapped iSCSI drives for the very same system DMP is mixing up all drives to one logical with multiple pathes. NOT USABLE AT ALL.

Does anybody got the combination iSCSI software target (Linux or Solaris on Intel) - iSCSI software initiator (Solrais 10 Sparc) - multiple iSCSI drives - Storage Foundation 5 running? I'm in urgent need for a solution.

Thanks for any ideas

[Douglas_Snyder]

Hello Roger;

That issue is addressed in MP1 for SF5.  You can download the MP here: http://www.symantec.com/enterprise/support/downloads.jsp?pid=15107

Additional information about the MP is located here: http://www.symantec.com/enterprise/support/documentation.jsp?pid=15107

Hope this helps.

Doug Snyder

Sr. Systems Engineer

Symantec Corp.

[Eugene_R]

I was able to get arround the problem by using Qlogic 4052c HBA card, thus replacing software with hardware based initiator.  The downside of this is that the card is a bit expansinve ~$700. 

[Stuball]

Roger -

I used the SourceForge iSCSI Enterprise Target for the target but I had to modify the source code so that the TestUnitReady and the SCSI inquiry would return the correct strings - including the LUNs Vendor Product and Sserial Number. When the ietd daemon loads, it would query the LUNs and load a structure so that the default VID PID and Serial were replaced with the info from the structure loaded. Remember that the target still refers to a list of "devices" so I could preload the SCSI inquiry command and just dump it into the structure when an actual query was made from the initiator - made the memory map of the ietd daemon a little larger but what the heck. This was tested successfully with the vxdmpinq command. Having multiple paths to the target(s) then allows the DMP to work. From memory though and the release notes for *nix this is unsupported.

I had a debian kernel (2.6.14) for the target and RHEL4 for the initiators. On the RHEL4 I was running VxVM 4.1 and could create diskgroups etc etc and also run VCS ( Version 5 ) but again this is unsupported.

Although the initiators can use multiple paths I didn;t end up setting up multiple paths using the Linux device mapper but this OR multiple targets with correct names passwords and matching initiator / target pairs would allow the VxVM DMP to work.

Stuart

[Roger_Zimmerman]

Eugene, i was able to configure all the needs with a hardware controller too, but the main goal of all the game is to give some trainees an idea of what is iSCSI and the interoperability with standard operating systems and volume managers. It works very fine in other operating systems than Solaris and even in Solaris with a hardware based solution. But the costs for the controllers are higher than the benefit of learning in this case. but thnx anyway.

[Roger_Zimmerman]

---

Douglas Snyder wrote:

Hello Roger;

That issue is addressed in MP1 for SF5.  You can download the MP here: http://www.symantec.com/enterprise/support/downloads.jsp?pid=15107

Additional information about the MP is located here: http://www.symantec.com/enterprise/support/documentation.jsp?pid=15107

Hope this helps.

Doug Snyder

Sr. Systems Engineer

Symantec Corp.

---

@Doug:

it dosn't help in this case, i tried all the possible configurations again (that's the rason for the delay in reply). The problem still exists: the iSCSI target is not giving you the SCSI Serial number of the iSCSI LUN's, so the DMP is seeing only many things with the same name, and this is six times space. So the basic thing is, Solaris (also in 08/07) has a iSCSI initiator that is not giving you the SCSI Device Serial Number from the iSCSI target LUN's. And this is misinterpreted by the DMP as only one Device with multiple pathes to.

Also the mentioned updates and patches do not address this problem.

Can i remove DMP completely from the device path? So, ignore the possibility to have multiple pathes to the device, but make sure that the device is coming directly from the iSCSI driver layer?  (I'm afraid not, but give it a chance..:-))

Sturat wrote:

Roger -

I used the SourceForge iSCSI Enterprise Target for the target but I had to modify the source code so that the TestUnitReady and the SCSI inquiry would return the correct strings - including the LUNs Vendor Product and Sserial Number. When the ietd daemon loads, it would query the LUNs and load a structure so that the default VID PID and Serial were replaced with the info from the structure loaded. Remember that the target still refers to a list of "devices" so I could preload the SCSI inquiry command and just dump it into the structure when an actual query was made from the initiator - made the memory map of the ietd daemon a little larger but what the heck. This was tested successfully with the vxdmpinq command. Having multiple paths to the target(s) then allows the DMP to work. From memory though and the release notes for *nix this is unsupported.

I had a debian kernel (2.6.14) for the target and RHEL4 for the initiators. On the RHEL4 I was running VxVM 4.1 and could create diskgroups etc etc and also run VCS ( Version 5 ) but again this is unsupported.

Although the initiators can use multiple paths I didn;t end up setting up multiple paths using the Linux device mapper but this OR multiple targets with correct names passwords and matching initiator / target pairs would allow the VxVM DMP to work.

Stuart

---

@Stuart:

Ok, with Linux there is no problem at all, there are iSCSI targets (like OpenFiler) what can export the SCSI Device Serial Number, an initiators what can supply all that stuff, and many volume managers that handle all the things in the right way. But not in Solaris (now 10 08/07). There you cannot have the correct things.

My problem is to find a workaround in the standard components of Sun Solaris (no, i don't have the sources) or even in Storage Foundation (maybe the cut of DMP from the device path).

The reason for this is that the rest of the testing and training environment is based completely on Sun Solaris and Storage Foundation 5.

@all:

It is very interesting that there are many solutions for the initial problem, but not on Sun Solaris, only with Linux and hardware things. Is iSCSI (completely in Software) in Solaris environments together with Volume Manager not used at all? Is my aproach to low cost environments that completely wrong? (If so, maybe i have to give this complete project away. But i think, this too interesting to be blown up.)

[Eugene_R]

It may sound rather discouraging, but full functionality of software based iSCSI initiator on Sun Solaris is apparently not there yet.  It is my understanding, that unless the initiator software component reports all that there is to be known about the target properties, there will be missing pieces that may brake the functionality of other software components that depend on it, such as Veritas DMP.

Another example of me running into the same sort of problem was trying to get backup software to use iSCSI addressable tape drive libraries.  They also present themselves as iSCSI targets. Veritas Backup needed serial numbers of the tape drives in order to track licensing and to set unique device ID's of the tape drives etc.

It just did not work with a software initiator solution because of not fully described devices.

Some day Sun will fix that...

[Ed_Cook]

Hello,

A workaround to this problem would be exclude it from being used under DMP.

example:

AVAILABLE DISK SELECTIONS:
       0. c1t0d0 <FUJITSU-MAY2073RCSUN72G-0401 cyl 14087 alt 2 hd 24 sec 424>
          /pci@1e,600000/pci@0/pci@a/pci@0/pci@8/scsi@1/sd@0,0
       1. c1t1d0 <FUJITSU-MAY2073RCSUN72G-0501 cyl 14087 alt 2 hd 24 sec 424>
          /pci@1e,600000/pci@0/pci@a/pci@0/pci@8/scsi@1/sd@1,0
       2. c2t010000144F72234600002A00471CA4E6d0 <SUN-SOLARIS-1 cyl 32766 alt 2 hd 4 sec 80>
          /scsi_vhci/ssd@g010000144f72234600002a00471ca4e6
       3. c2t010000144F72234600002A00471CAB95d0 <SUN-SOLARIS-1 cyl 32766 alt 2 hd 4 sec 80>
          /scsi_vhci/ssd@g010000144f72234600002a00471cab95
Specify disk (enter its number):

Before excluding the iSCSI device from DMP:

DEVICE       TYPE            DISK         GROUP        STATUS
Disk_0       auto:cdsdisk    -            -            online
c1t0d0s2     auto:sliced     rootdg01     rootdg       online
c1t1d0s2     auto:none       -            -            online invalid

#vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
Disk_0       auto:cdsdisk    -            -            online
c1t0d0s2     auto:sliced     rootdg01     rootdg       online
c1t1d0s2     auto:none       -            -            online invalid
[root@WALV215-12:vx]#vxdisk list Disk_0
Device:    Disk_0
devicetag: Disk_0
type:      auto
hostid:
disk:      name= id=1193066152.13.WALV215-12
group:     name= id=1193060046.8.WALV215-12
info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags:     online ready private autoconfig
pubpaths:  block=/dev/vx/dmp/Disk_0s2 char=/dev/vx/rdmp/Disk_0s2
guid:      {ad88e736-1dd1-11b2-8805-00144f6ffd5a}
udid:      SUN%5FSOLARIS%5FDISKS%5F00000001
site:      -
version:   3.1
iosize:    min=512 (bytes) max=2048 (blocks)
public:    slice=2 offset=65792 len=10419328 disk_offset=0
private:   slice=2 offset=256 len=65536 disk_offset=0
update:    time=1193230940 seqno=0.17
ssb:       actual_seqno=0.0
headers:   0 240
configs:   count=1 len=48144
logs:      count=1 len=7296
Defined regions:
 config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
 config   priv 000256-048207[047952]: copy=01 offset=000192 enabled
 log      priv 048208-055503[007296]: copy=01 offset=000000 enabled
 lockrgn  priv 055504-055647[000144]: part=00 offset=000000
Multipathing information:
numpaths:   2
c2t010000144F72234600002A00471CAB95d0s2 state=enabled
c2t010000144F72234600002A00471CA4E6d0s2 state=enabled

After:

#more vxdmp.exclude
exclude_all 0
paths
c2t010000144F72234600002A00471CA4E6d0s2 /scsi_vhci/ssd@g010000144f72234600002a00471ca4e6
c2t010000144F72234600002A00471CAB95d0s2 /scsi_vhci/ssd@g010000144f72234600002a00471cab95
#
controllers
#
product
#
pathgroups
#

DEVICE       TYPE            DISK         GROUP        STATUS
c1t0d0s2     auto:sliced     rootdg01     rootdg       online
c1t1d0s2     auto:none       -            -            online invalid
fabric_0     auto:cdsdisk    -            -            online
fabric_1     auto:cdsdisk    -            -            online

Ed

[Roger_Zimmerman]

Hi, Ed,

sorry for the delay but I wanted to test all of the details very well. Yes, it works with the exclusion from DMP (normally I should have known this, shame :-). Great. At least for most of the features of Storage Foundation.

The only thing I found was the incompatibility with SCSI 3 persitent reservation, which is not working with iSCSI Targets (at least on Solaris 10 and Open Solaris 11). Ok, but that's a minor point. And I dont think that this will be in the driver (very soon :-).

The other thing I'm working on this days is the different ID's on the disks in a cluster environment. On one host it shows correctly, on the other hosts there ist a difference in informations on the disk decription and the operating system informations, ending in the status "online,incompatible" for the disks. But thats only cosmetic, the switch from one host to another including the import of the disk groups works fine. I think I will write a preonline trigger for the cluster to prevent confusing my trainees. Or even for spending some time in long winter nigths.

Again thanks for the help

Roger