Forum Discussion

Kumaran's avatar
Kumaran
Level 3
13 years ago

QDepth tuneup for DMP

Hi

Am I right thinking setting the following in /etc/system which sets the QDepth and IO Timeout globally for the HBA's 

set sd:sd_io_time=0x3c

 

set sd:sd_max_throttle=8
 
can be controlled at array level using vxdmpadm as per these tunable parameters
 
 
and will this overide the system settings only for the specific Array and use the defaults or forced from /etc/system values for the rest?
 
Any other insights on this would be great.
 
Thanks
 
business continuity
Configuring
Performance
Storage Foundation
Troubleshooting
  • hafiz_rehman's avatar
    hafiz_rehman
    13 years ago

    Hi Kumaran,

    Above both are two different things, set sd:sd_io_time is tuning parameter for SCSI Disk Driver which work at Operating System level and are global for all arrays. dmp_queue_depth is tuneable which work at DMP layer , which mean that The maximum number of queued I/O requests on a path during I/O throttling.

    More information about I/O Throttling please refer to https://sort.symantec.com/public/documents/sf/5.0/solaris/html/vxvm_admin/ag_ch_dmp_vm31.html#458394

    Following are further notes about I/O throttling and what they mean. NOTE: below is from vxdmpadm man page.

     throttle

    Enables I/O throttling for a path.DMP continues to schedule I/O requests on a path that seems to have failed, but on which an I/O error has not been reported, until the specified limit is reached. DMP then prevents new I/O requests being scheduled on the path, and attempts to reschedule failed I/O requests on other paths. Error detection can be based on the change in response times or pending I/O counts, as follows:

     

    I would also recommend you to review the article http://www.symantec.com/docs/TECH32080 which explain how I/O are processed in vxio driver.

    Article http://www.symantec.com/docs/HOWTO45767 will explain more about improving efficiency of DMP in co-existence with SD driver.

     

    I hope above will help.

     

    Regards,

    Hafiz

  • Kumaran's avatar
    Kumaran
    Level 3
    13 years ago

    Looks the default  tuneable value for dmp_queue_depth is "32" but doesn't looks to be enforced eventhough the DOC says it overrides the OS defaults which is /etc/system value

     set sd:sd_max_throttle=8

    root@server01:/root# vxdmpadm gettune dmp_queue_depth

                Tunable               Current Value  Default Value
    ------------------------------    -------------  -------------
    dmp_queue_depth                          32               32.
     
    root@server01:/root# vxdmpadm getattr enclosure df8000 recoveryoption
    ENCLR-NAME      RECOVERY-OPTION      DEFAULT[VAL]        CURRENT[VAL]
    ==============================================================================
    df8000          Throttle             Nothrottle[0]       Queuedepth[16]
    df8000          Error-Retry          Fixed-Retry[5]      Fixed-Retry[5]
     
    The reason I say it isn't enforced is, IOSTAT shows, queues of maximum 256 per lun:
     
    root@server01:/root# awk '{print $5","$6","$NF}' iostat_stripe64|egrep -v "wait|c1t0d0|c1t2d0|^$"|sort -unr|more
    217.3,256.0,c3t50060E80100D213Dd10
    216.5,256.0,c2t50060E80100D2135d10
    178.1,256.0,c2t50060E80100D2135d13
    177.2,256.0,c3t50060E80100D213Dd13
    176.7,256.0,c2t50060E80100D2135d13
    176.5,256.0,c2t50060E80100D2135d13
    175.9,256.0,c3t50060E80100D213Dd13
    175.8,256.0,c2t50060E80100D2135d13
    175.7,256.0,c3t50060E80100D213Dd13
    175.1,256.0,c2t50060E80100D2135d11
    174.6,256.0,c3t50060E80100D213Dd13
    174.2,256.0,c3t50060E80100D213Dd13
    174.0,256.0,c3t50060E80100D213Dd13
    173.7,256.0,c2t50060E80100D2135d11
    173.6,256.0,c2t50060E80100D2135d13
    173.5,256.0,c3t50060E80100D213Dd13
    173.4,256.0,c2t50060E80100D2135d11
    173.2,256.0,c2t50060E80100D2135d11
    173.0,256.0,c2t50060E80100D2135d13
    172.8,256.0,c3t50060E80100D213Dd13
    172.5,256.0,c3t50060E80100D213Dd13
    172.3,256.0,c2t50060E80100D2135d13
     
    NOTE: 256 in the above is the OS default of 256 and I've not enforced set sd:sd_max_throttle=8
    Hope this additional info prompts someone to have a look and comment.

     

  • Kumaran's avatar
    Kumaran
    Level 3
    13 years ago

    more evidence after setting sd:sd_max_throttle=16

     

     

    root@server01:/root# more iostat_stripe64_20120524_3.csv
    440.5,16.0,c2t50060E80100D2135d14
    438.2,16.0,c2t50060E80100D2135d14
    435.9,16.0,c2t50060E80100D2135d14
    433.6,16.0,c2t50060E80100D2135d14
    432.4,16.0,c2t50060E80100D2135d14
    431.0,16.0,c2t50060E80100D2135d14
    430.9,16.0,c2t50060E80100D2135d14
    430.0,16.0,c2t50060E80100D2135d14
    429.7,16.0,c3t50060E80100D213Dd14
    429.0,16.0,c2t50060E80100D2135d14
    427.9,16.0,c3t50060E80100D213Dd14
    427.8,16.0,c3t50060E80100D213Dd14
    427.6,16.0,c3t50060E80100D213Dd14
    427.5,16.0,c2t50060E80100D2135d14
    427.1,16.0,c2t50060E80100D2135d14
    427.0,16.0,c2t50060E80100D2135d14
    426.6,16.0,c2t50060E80100D2135d14
    426.5,16.0,c3t50060E80100D213Dd14
    426.3,16.0,c3t50060E80100D213Dd14
    426.0,16.0,c3t50060E80100D213Dd14
    425.8,16.0,c3t50060E80100D213Dd14
    425.7,16.0,c2t50060E80100D2135d14
     
  • hafiz_rehman's avatar
    hafiz_rehman
    Level 3
    13 years ago

    Hi Kumaran,

    Above both are two different things, set sd:sd_io_time is tuning parameter for SCSI Disk Driver which work at Operating System level and are global for all arrays. dmp_queue_depth is tuneable which work at DMP layer , which mean that The maximum number of queued I/O requests on a path during I/O throttling.

    More information about I/O Throttling please refer to https://sort.symantec.com/public/documents/sf/5.0/solaris/html/vxvm_admin/ag_ch_dmp_vm31.html#458394

    Following are further notes about I/O throttling and what they mean. NOTE: below is from vxdmpadm man page.

     throttle

    Enables I/O throttling for a path.DMP continues to schedule I/O requests on a path that seems to have failed, but on which an I/O error has not been reported, until the specified limit is reached. DMP then prevents new I/O requests being scheduled on the path, and attempts to reschedule failed I/O requests on other paths. Error detection can be based on the change in response times or pending I/O counts, as follows:

     

    I would also recommend you to review the article http://www.symantec.com/docs/TECH32080 which explain how I/O are processed in vxio driver.

    Article http://www.symantec.com/docs/HOWTO45767 will explain more about improving efficiency of DMP in co-existence with SD driver.

     

    I hope above will help.

     

    Regards,

    Hafiz