Best practice “Maximum I/O streams per volume” with Disk Pools

[rpandey]

My backups are failed with below error - 

awaiting resource CSD-SYD-STU-PD Reason: Maximum I/O stream count has been reached for disk volume, Media Server: ABC1257, 
     Robot Number: NONE, Robot Type: NONE, Media ID: N/A, Drive Name: N/A, 
     Volume Pool: CSD-SYD, Storage Unit: CSD-SYD-STU-PD, Drive Scan Host: N/A

Limit has been reached for the logical resource csd-nbumaster.NBU_POLICY.MAXJOBS.CSD-SYD-CA  
client backup was not attempted because backup window closed(196)

Environment:

Netbackup 7.6.0.2 & Windows Server 2008.

Current Setting :

Limit I/O streams = 32 per volume

Maximum concurrent jobs=32

Maximum fragment size=51200 MB

[sksujeet]

As it clearly says Maximum I/O stream count has been reached for disk volume for Storage Unit: CSD-SYD-STU-PD

As you have set this to Limit I/O streams = 32 per volume and if there would be more jobs running then they will remain in queued state unless it finishes the current job which are running. And if the jobs remain in queued state and if backup window closes then jobs will fail with 196.

You have to check what kind of disk pool you are using and if that i/o stream can be increased, make sure you check the hardware vendor for it or it might thrash the disk.

Check you backup window, if it is too small you can increase it.

From the Netbackup 7.5 SLP best pratice manual:

http://www.symantec.com/docs/HOWTO73205

Use “Maximum I/O streams per volume” with Disk Pools

Disk storage units allow you to limit the number of concurrent write jobs that use the storage unit, however there are no limits on the number of read jobs (restores and duplications) that may be accessing the same disk pool at the same time and it is also possible to configure multiple storage units to access the same disk pool simultaneously. This can give rise to unexpected I/O contention on the disk pool. By setting the Maximum I/O streams per volume option on the Disk Pool, you can limit the total number of jobs that access the disk pool concurrently, regardless of the job type. While the disk pool is maxed out with backup jobs that are writing images to the device, no duplication jobs are allowed to start reading from the device. (The one exception to this rule is that restore jobs are allowed to read from the Disk Pool even if the maximum configured number of input and output streams is already being utilized.)

When you enable the Maximum I/O streams per volume option on the Disk Pool the default number of streams is 2. Although the optimal number of streams per volume will vary depending on the disk type, a general guide line to minimize contention would be to divide the maximum concurrent jobs count for all the storage units using the disk pool by the number of volumes in the disk pool.

[Marianne]

There is no 'best practice' for this value.

The amount of concurrent writes that can be handled depends on media server specs and lun layout on the disk volume.

You can try to increase the value in small increments - e.g. 2 per night.
Carefully monitor resources on media server with increased streams.
At the point where you see the mediaserver 'battling', go back to previous setting.

It may be time to stand up a new media server...

Have a look at this featured post:

Updated NetBackup Backup Planning and Performance Tuning Guide for Release 7.5 and Release 7.6 

[rpandey]

Thanks for the quick response.

I forgot to mention - The assigned disk is SATA and 7200 rpm

[sksujeet · Accepted solution]

you can use process monitor or any 3rd party tool at the time of backup to see the read/write iops on the disk to see if it can take any more load on this or it is already taking too much load.

If you read through above 2 posts you should be able to sort this out.

[Nicolai · Accepted solution]

Be careful not to overload SATA disk. They have limited amount of I/O per spindle.

Also the number of spindles has a impact on performance. 20 spindles will be able to drive better performance than only 5.

This is why there is no "general best practice", it all comes down to the disk subsystem, type of disks, number of spindles and RAID type.