Volume Manager 5.0 Solaris 10 AVT Issue

[mikeyd810]

We've run in to an issue we just can't seem to resolve.  We're runing SUN T5220(solaris 10) servers with Veritas Volume Manager 5.0 installed, SAN disk is from a SUN/STK Flexline 280 & 380.  Things were running fine for a few months but a couple weeks back we started getting AVT transfers back and forth to the point where we had to disconnect one of the two fiber cables for disk on each server to keep things up and from bouncing around.  Of course the problem with this is we're not redundant now, every time we try and plug in the cable and reboot the system just hangs trying to come up and from what we can tell it's due to the volumes on the array performing AVT's back and forth.  Switch wise and array wise everything is working and online, no down ports or controllers.  We've had both vendors/support teams verify the hardware and configuration.

We've tried different versions of the STK ASL with no success, any ideas on what the heck is going on here?  We've searched around to see if these issues have been encoutered or there were any bugs fixed in the various MP1 and MP3 RP's but we couldn't find anything that really matched this situation.

$ pkginfo -l VRTSvxvm
PKGINST: VRTSvxvm
NAME: Binaries for VERITAS Volume Manager by Symantec
CATEGORY: system
ARCH: sparc
VERSION: 5.0,REV=05.11.2006.17.55
BASEDIR: /
VENDOR: Symantec Corporation
DESC: Virtual Disk Subsystem
PSTAMP: Veritas-5.0-2007-01-11a
INSTDATE: Nov 10 2008 13:23
HOTLINE: http://support.veritas.com/phonesup/phonesup_ddProduct_.htm
EMAIL: support@veritas.com
STATUS: completely installed
FILES: 940 installed pathnames
29 shared pathnames
13 linked files
106 directories
403 executables
378158 blocks used (approx)


$ pkginfo -l VRTSSTKasl
PKGINST: VRTSSTKasl
NAME: OpenStorage and BladeController/FlexLine Array Support Librarry
CATEGORY: system
ARCH: sparc
VERSION: 1.0,REV=06.21.2007.12.53
BASEDIR: /etc/vx
VENDOR: VERITAS Software
DESC: Array Support library for OpenStorage & BladeController/FlexLine Array.
PSTAMP: VERSION-1.0:8-January-1999
INSTDATE: Apr 24 2009 14:31
HOTLINE: 800-342-0652
EMAIL: support@veritas.com
STATUS: completely installed
FILES: 5 installed pathnames
3 shared pathnames
3 directories
2 executables
30 blocks used (approx)

[Gaurav_S]

Hello,

As you are mentioning that  you have involved both the vendors to check configuration, just to double check, on the ASL technotes, usually there is mentioning of Array Settings compatible with that ASL, have you confirmed that you are running with recommended array settings ?

Veritas DMP has to do work for handling the trespass while storage has to control the trespass so what I could think the root cause should be somewhere from array side.

Also, are you seeing any messages on the console/dmesg ?

Gaurav

[mikeyd810]

ASL's say to have the array in AVT mode which it is.  Host groups have their HBA's set to Veritas DMP.

[Gaurav_S]

ok,

do you see any messages like "disabled DMP node belonging to path....  " or "enabled DMP node belonging to path...." ?

What tuneables have been set for restored daemon ? checking the health of DMP paths is in the hands of restored daemon,

# vxdmpadm stat restored


Thanks

Gaurav

[mikeyd810]

A few weeks when this happened we had disabled and enabled messages, what happened was one of our disk array controllers rebooted during the night around 1AM.

Apr 2 01:00:01 ustlscbu001 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 32/0x720 belonging to the dmpnode 305/0x188
Apr 2 01:00:01 ustlscbu001 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 32/0x578 belonging to the dmpnode 305/0x168
Apr 2 01:00:01 ustlscbu001 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 32/0xf20 belonging to the dmpnode 305/0x198
Apr 2 01:00:01 ustlscbu001 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 32/0xf40 belonging to the dmpnode 305/0x178
Apr 2 01:00:01 ustlscbu001 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 32/0x1428 belonging to the dmpnode 305/0x198
Apr 2 01:00:01 ustlscbu001 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 32/0x1458 belonging to the dmpnode 305/0x168
Apr 2 01:00:01 ustlscbu001 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 32/0x558 belonging to the dmpnode 305/0x188
Apr 2 01:00:01 ustlscbu001 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 32/0x730 belonging to the dmpnode 305/0x178
Apr 2 01:00:02 ustlscbu001 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 32/0x1438 belonging to the dmpnode 305/0x188
Apr 2 01:00:02 ustlscbu001 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 32/0x568 belonging to the dmpnode 305/0x178
Apr 2 01:00:02 ustlscbu001 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 32/0x548 belonging to the dmpnode 305/0x198
Apr 2 01:00:02 ustlscbu001 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 32/0xf50 belonging to the dmpnode 305/0x168
Apr 2 01:00:02 ustlscbu001 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 32/0x1448 belonging to the dmpnode 305/0x178
Apr 2 01:00:02 ustlscbu001 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 32/0x740 belonging to the dmpnode 305/0x168
Apr 2 01:00:02 ustlscbu001 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 32/0xf30 belonging to the dmpnode 305/0x188
Apr 2 01:00:02 ustlscbu001 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 32/0x710 belonging to the dmpnode 305/0x198
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 32/0x730 belonging to the dmpnode 305/0x178
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 638735 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 failback issued for 305/0x178
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 32/0x740 belonging to the dmpnode 305/0x168
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 638735 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 failback issued for 305/0x168
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 32/0x720 belonging to the dmpnode 305/0x188
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 638735 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 failback issued for 305/0x188
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 32/0x710 belonging to the dmpnode 305/0x198
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 638735 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 failback issued for 305/0x198
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 32/0x578 belonging to the dmpnode 305/0x168
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 638735 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 failback issued for 305/0x168
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 32/0x568 belonging to the dmpnode 305/0x178
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 638735 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 failback issued for 305/0x178
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 32/0x558 belonging to the dmpnode 305/0x188
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 638735 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 failback issued for 305/0x188
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 32/0x548 belonging to the dmpnode 305/0x198
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 638735 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 failback issued for 305/0x198
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 32/0xf40 belonging to the dmpnode 305/0x178
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 638735 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 failback issued for 305/0x178
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 32/0xf30 belonging to the dmpnode 305/0x188
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 638735 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 failback issued for 305/0x188
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 32/0xf50 belonging to the dmpnode 305/0x168
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 638735 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 failback issued for 305/0x168
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 32/0xf20 belonging to the dmpnode 305/0x198
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 638735 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 failback issued for 305/0x198
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 32/0x1458 belonging to the dmpnode 305/0x168
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 638735 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 failback issued for 305/0x168
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 32/0x1428 belonging to the dmpnode 305/0x198
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 638735 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 failback issued for 305/0x198
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 32/0x1438 belonging to the dmpnode 305/0x188
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 638735 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 failback issued for 305/0x188
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 32/0x1448 belonging to the dmpnode 305/0x178
Apr 2 01:04:28 ustlscbu001 vxdmp: [ID 638735 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 failback issued for 305/0x178


Pretty standard for the DMP restore daemon options.

ustlscbu001:root> vxdmpadm stat restored
The number of daemons running : 1
The interval of daemon: 300
The policy of daemon: check_disabled

[Gaurav_S]

Thanks for pasting the outputs...

Well getting so support would be best option thats what I can say for now  (if you already involved them & still issue not found then its a bad luck :(  )

Though I strongly feel this is something which needs to be corrected from Array.....


Gaurav