Highlighted

QDepth tuneup for DMP

Hi

Am I right thinking setting the following in /etc/system which sets the QDepth and IO Timeout globally for the HBA's 

set sd:sd_io_time=0x3c

 

set sd:sd_max_throttle=8
 
can be controlled at array level using vxdmpadm as per these tunable parameters
 
 
and will this overide the system settings only for the specific Array and use the defaults or forced from /etc/system values for the rest?
 
Any other insights on this would be great.
 
Thanks
 
1 Solution

Accepted Solutions
Highlighted
Accepted Solution!

Hi Kumaran, Above both are

Hi Kumaran,

Above both are two different things, set sd:sd_io_time is tuning parameter for SCSI Disk Driver which work at Operating System level and are global for all arrays. dmp_queue_depth is tuneable which work at DMP layer , which mean that The maximum number of queued I/O requests on a path during I/O throttling.

More information about I/O Throttling please refer to https://sort.symantec.com/public/documents/sf/5.0/solaris/html/vxvm_admin/ag_ch_dmp_vm31.html#458394

Following are further notes about I/O throttling and what they mean. NOTE: below is from vxdmpadm man page.

 throttle

Enables I/O throttling for a path.DMP continues to schedule I/O requests on a path that seems to have failed, but on which an I/O error has not been reported, until the specified limit is reached. DMP then prevents new I/O requests being scheduled on the path, and attempts to reschedule failed I/O requests on other paths. Error detection can be based on the change in response times or pending I/O counts, as follows:

 

I would also recommend you to review the article http://www.symantec.com/docs/TECH32080 which explain how I/O are processed in vxio driver.

Article http://www.symantec.com/docs/HOWTO45767 will explain more about improving efficiency of DMP in co-existence with SD driver.

 

I hope above will help.

 

Regards,

Hafiz

View solution in original post

4 Replies
Highlighted

Looks the default  tuneable

Looks the default  tuneable value for dmp_queue_depth is "32" but doesn't looks to be enforced eventhough the DOC says it overrides the OS defaults which is /etc/system value

 set sd:sd_max_throttle=8

root@server01:/root# vxdmpadm gettune dmp_queue_depth

            Tunable               Current Value  Default Value
------------------------------    -------------  -------------
dmp_queue_depth                          32               32.
 
root@server01:/root# vxdmpadm getattr enclosure df8000 recoveryoption
ENCLR-NAME      RECOVERY-OPTION      DEFAULT[VAL]        CURRENT[VAL]
==============================================================================
df8000          Throttle             Nothrottle[0]       Queuedepth[16]
df8000          Error-Retry          Fixed-Retry[5]      Fixed-Retry[5]
 
The reason I say it isn't enforced is, IOSTAT shows, queues of maximum 256 per lun:
 
root@server01:/root# awk '{print $5","$6","$NF}' iostat_stripe64|egrep -v "wait|c1t0d0|c1t2d0|^$"|sort -unr|more
217.3,256.0,c3t50060E80100D213Dd10
216.5,256.0,c2t50060E80100D2135d10
178.1,256.0,c2t50060E80100D2135d13
177.2,256.0,c3t50060E80100D213Dd13
176.7,256.0,c2t50060E80100D2135d13
176.5,256.0,c2t50060E80100D2135d13
175.9,256.0,c3t50060E80100D213Dd13
175.8,256.0,c2t50060E80100D2135d13
175.7,256.0,c3t50060E80100D213Dd13
175.1,256.0,c2t50060E80100D2135d11
174.6,256.0,c3t50060E80100D213Dd13
174.2,256.0,c3t50060E80100D213Dd13
174.0,256.0,c3t50060E80100D213Dd13
173.7,256.0,c2t50060E80100D2135d11
173.6,256.0,c2t50060E80100D2135d13
173.5,256.0,c3t50060E80100D213Dd13
173.4,256.0,c2t50060E80100D2135d11
173.2,256.0,c2t50060E80100D2135d11
173.0,256.0,c2t50060E80100D2135d13
172.8,256.0,c3t50060E80100D213Dd13
172.5,256.0,c3t50060E80100D213Dd13
172.3,256.0,c2t50060E80100D2135d13
 
NOTE: 256 in the above is the OS default of 256 and I've not enforced set sd:sd_max_throttle=8
Hope this additional info prompts someone to have a look and comment.

 

Highlighted

more evidence after setting

more evidence after setting sd:sd_max_throttle=16

 

 

root@server01:/root# more iostat_stripe64_20120524_3.csv
440.5,16.0,c2t50060E80100D2135d14
438.2,16.0,c2t50060E80100D2135d14
435.9,16.0,c2t50060E80100D2135d14
433.6,16.0,c2t50060E80100D2135d14
432.4,16.0,c2t50060E80100D2135d14
431.0,16.0,c2t50060E80100D2135d14
430.9,16.0,c2t50060E80100D2135d14
430.0,16.0,c2t50060E80100D2135d14
429.7,16.0,c3t50060E80100D213Dd14
429.0,16.0,c2t50060E80100D2135d14
427.9,16.0,c3t50060E80100D213Dd14
427.8,16.0,c3t50060E80100D213Dd14
427.6,16.0,c3t50060E80100D213Dd14
427.5,16.0,c2t50060E80100D2135d14
427.1,16.0,c2t50060E80100D2135d14
427.0,16.0,c2t50060E80100D2135d14
426.6,16.0,c2t50060E80100D2135d14
426.5,16.0,c3t50060E80100D213Dd14
426.3,16.0,c3t50060E80100D213Dd14
426.0,16.0,c3t50060E80100D213Dd14
425.8,16.0,c3t50060E80100D213Dd14
425.7,16.0,c2t50060E80100D2135d14
 
Highlighted
Accepted Solution!

Hi Kumaran, Above both are

Hi Kumaran,

Above both are two different things, set sd:sd_io_time is tuning parameter for SCSI Disk Driver which work at Operating System level and are global for all arrays. dmp_queue_depth is tuneable which work at DMP layer , which mean that The maximum number of queued I/O requests on a path during I/O throttling.

More information about I/O Throttling please refer to https://sort.symantec.com/public/documents/sf/5.0/solaris/html/vxvm_admin/ag_ch_dmp_vm31.html#458394

Following are further notes about I/O throttling and what they mean. NOTE: below is from vxdmpadm man page.

 throttle

Enables I/O throttling for a path.DMP continues to schedule I/O requests on a path that seems to have failed, but on which an I/O error has not been reported, until the specified limit is reached. DMP then prevents new I/O requests being scheduled on the path, and attempts to reschedule failed I/O requests on other paths. Error detection can be based on the change in response times or pending I/O counts, as follows:

 

I would also recommend you to review the article http://www.symantec.com/docs/TECH32080 which explain how I/O are processed in vxio driver.

Article http://www.symantec.com/docs/HOWTO45767 will explain more about improving efficiency of DMP in co-existence with SD driver.

 

I hope above will help.

 

Regards,

Hafiz

View solution in original post

Highlighted

Thanks! that was helpful and

Thanks! that was helpful and a good read!