DMP I/O error is disabling the NETAPP paths via DMP

[Ramesh_Kasiredd]

For everty roo boot the VXVM volume is going to DEPORT state.

I am importing it manually.

I saw the below intresting logs on "dmpevents.log"

Thu Oct 10 16:53:56.317: Enabled Disk array other_disks

Thu Oct 10 16:53:56.317: Enabled Disk array fas32200

Thu Oct 10 16:53:56.318: Added Dmpnode c0t1d0s2

Thu Oct 10 16:53:56.318: Added Dmpnode fas32200_0

Thu Oct 10 16:53:56.318: Dmpnode c0t0d0s2 has migrated from enclosure - to other_disks

Thu Oct 10 16:53:56.318: Disabled Disk array -

Thu Oct 10 16:53:56.367: I/O error occured on Path c3t60A980004431702F643F44436B446B46d0s2 belonging to Dmpnode fas32200_0

Thu Oct 10 16:53:56.370: I/O analysis done as DMP_PATH_OKAY on Path c3t60A980004431702F643F44436B446B46d0s2 belonging to Dmpnode fas32200_0

Thu Oct 10 16:53:56.370: I/O retry(4) on Path c3t60A980004431702F643F44436B446B46d0s2 belonging to Dmpnode fas32200_0

Thu Oct 10 16:53:56.373: I/O error occured on Path c3t60A980004431702F643F44436B446B46d0s2 belonging to Dmpnode fas32200_0

Thu Oct 10 16:53:56.374: SCSI error occured on Path c3t60A980004431702F643F44436B446B46d0s2: opcode=0x12 reported generic device failure (status=0x0, key=0x0, asc=0x50, ascq=0x50)

Thu Oct 10 16:53:56.374: I/O analysis done as DMP_PATH_FAILURE on Path c3t60A980004431702F643F44436B446B46d0s2 belonging to Dmpnode fas32200_0

Thu Oct 10 16:53:56.374: Disabled Path c3t60A980004431702F643F44436B446B46d0s2 belonging to Dmpnode fas32200_0 due to path failure

Thu Oct 10 16:53:56.374: Disabled Dmpnode fas32200_0

Thu Oct 10 16:53:56.374: Disabled Disk array fas32200

Thu Oct 10 16:53:56.374: I/O error occured (errno=0x6) on Dmpnode fas32200_0

Thu Oct 10 16:53:59.558: Enabled Disk array fas32200

Thu Oct 10 16:53:59.558: Enabled Path c3t60A980004431702F643F44436B446B46d0s2 belonging to Dmpnode fas32200_0

Thu Oct 10 16:53:59.558: Enabled Dmpnode fas32200_0

Thu Oct 10 16:53:59.559: Failover initiated for Dmpnode fas32200_0 without quiescing

Thu Oct 10 16:53:59.559: CURPRI set to primary for Dmpnode fas32200_0 without quiescing

Thu Oct 10 16:54:04.000: Initiated SAN topology discovery

Thu Oct 10 16:54:05.000: Completed SAN topology discovery

Thu Oct 10 22:06:11.460: Marked as idle Path c3t60A980004431702F643F44436B446B46d0s2 belonging to Dmpnode fas32200_0

Thu Oct 10 22:07:22.012: Unmarked as idle Path c3t60A980004431702F643F44436B446B46d0s2 belonging to Dmpnode fas32200_0

Thu Oct 10 22:07:24.012: Lost 5120 DMP I/O statistics records

Thu Oct 10 22:07:25.012: Lost 2071 DMP I/O statistics records

Thu Oct 10 22:08:28.011: Marked as idle Path c3t60A980004431702F643F44436B446B46d0s2 belonging to Dmpnode fas32200_0

[Gaurav_S]

Hello,

It is clear from the errors that DMP/VxVM is trying to make IO to the device however is failing intermittently ..

Questions:

1. What version of VxVM is in use ?

2. What Storage array is in use ?

3. attach the output of # iostat -En

4. paste output of # vxdmpadm listenclosure all

5. Any analysis done from Storage / SAN switch side to know if there are any IO errors noticed ? I would expect the SAN switch logs should show some errors.

6. Any other multipathing software underneath DMP ?  mpxio or powerpath ?

G

[Ramesh_Kasiredd]

1. What version of VxVM is in use ?

             VERSION:  5.1,REV=10.06.2009.21.48

2. What Storage array is in use ?

             NETAPP

3. attach the output of # iostat -En

              c0t0d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0

Vendor: ATA      Product: ST31000524NS     Revision: SN12 Serial No:

Size: 1000.20GB <1000204885504 bytes>

Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0

Illegal Request: 49 Predictive Failure Analysis: 0

c0t1d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0

Vendor: ATA      Product: ST31000524NS     Revision: SN12 Serial No:

Size: 1000.20GB <1000204885504 bytes>

Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0

Illegal Request: 29 Predictive Failure Analysis: 0

c3t60A980004431702F643F44436B446B46d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0

Vendor: NETAPP   Product: LUN              Revision: 811a Serial No:

Size: 536.95GB <536952700416 bytes>

Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0

Illegal Request: 1 Predictive Failure Analysis: 0

4 . paste output of # vxdmpadm listenclosure all

ENCLR_NAME        ENCLR_TYPE     ENCLR_SNO      STATUS       ARRAY_TYPE     LUN_COUNT

=======================================================================================

fas32200          FAS3220        700000265539         CONNECTED    ALUA        1

other_disks       OTHER_DISKS    OTHER_DISKS          CONNECTED    OTHER_DISKS   2

5. Any analysis done from Storage / SAN switch side to know if there are any IO errors noticed ? I would expect the SAN switch logs should show some errors.

        A) Contacted SAN admin, every thing look fine.

6 . Any other multipathing software underneath DMP ?  mpxio or powerpath ?

         A) mpxio

[ursi]

hmm -- just my two cents:

This looks like a SCSI error while trying to write to the disk c3t60A980004431702F643F44436B446B46d0s2 (nice name). From the ASC it says it is an "Write Append Error" but I could not find the additional ASCQ Qualifier "50" for that error code.

If it is a SCSI error it seems unappropriate to disable the path since the path (to the broken (virtual) disk) is just fine.

[AlexeyL · Accepted solution]

It's possible that MPxIO and DMP interfere with each other. In my environment I tend to disable MPxIO for controllers/disks managed by DMP. Here is some info from Symantec about interoperability of MPxIO and DMP.

http://www.symantec.com/business/support/index?page=content&id=TECH204060

Also if you could provide the following:

1) mpathadm list lu

2) vxdmpadm getsubpaths