12 - Docker Persistent Storage and Quality of Service with Next Generation InfoScale Storage Solutions

DISCLAIMER: Quality of Service (QoS) feature is not available in the current version of the GA product.

NOTE: QoS is now available with InfoScale 7.1. This article contains more up to date information

What is this about?

InfoScale Storage is continually evolving to adapt to new demands driven by the needs of our customers most critical applications and strategic initiatives. One such initiative we are seeing more of is the desire to build more value through delivery of applications in a more agile and dynamic environment. This can be done in a few ways, but container based architectures with lower costs, better performance, and more flexibility than hypervisor based options are gaining traction. While some apps are stately and don’t have large storage requirements, persistent storage is a clear need for containers to be the de-facto architectures moving forward. With the scale and rapid deployment capabilities allowed by containers, managing persistent storage and guaranteeing that each container does not consume all IO resources can be a challenge.

Therefore, one storage solution that will be able to control each container performance needs will enable enterprises in their journey into a more agile, consolidated and faster environment.

InfoScale storage will be introducing a new QoS feature which, in combination with the available docker plug-in, will be a perfect fit for persistent storage in a container driven world.

With the integration of QoS and InfoScale Storage plug-in for Docker customers will be able to get predictable SLAs, preventing noisy neighbour problems by throttling non-critical container applications.

Although at this moment we have to script a few steps in this process, this could be easily added into our plug-in for Docker for fully automation.



 I am permitting myself to show you what we are working on and I will be more than happy to hear your feedback about that.

In this demo, I am going to create six containers running MySQL, and another six that will be generating a workload in the database ones, generating some IO.

The script is creating a container that will automatically create a volume using the veritas driver and mapping it to each container. That volume will have the database content. The volume name will be mysql followed by a number:

    docker run -d --name mysql$i -h mysql -e MYSQL_ROOT_PASSWORD=mysql -v $PWD:/etc/mysql/conf.d -v mysql$i:/var/lib/mysql --volume-driver veritas  mysql:5.6

I am also creating a new container that will link to the MySQL one and that will create some IO:

    docker run --rm --link mysql$i:mysql tpcc-mysql /tpcc-mysql/tpcc_start -hmysql -dtpcc -uroot -pmysql -w$WAREHOUSES -c16 -r10 -l1200 > tpcc-mysql$i.log &


For each persistent volume that the plug-in has created, I have added a command in the script which will be mapping each volume to a different SLA group, so I will be able to control later what is the maximum IOPS that the volume will be serving:

        vxvolgrp -g dockerdg make grp_mysql$i mysql$i

Once my test is up and running, these are the containers running:

[root@target-3 ~]# docker ps -a

CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS               NAMES

28cd6360cd67        tpcc-mysql          "/entrypoint.sh /tpcc"   26 seconds ago       Up 14 seconds       3306/tcp            admiring_hawking

ca2436842160        tpcc-mysql          "/entrypoint.sh /tpcc"   26 seconds ago       Up 13 seconds       3306/tcp            focused_wilson

83db39e0f289        tpcc-mysql          "/entrypoint.sh /tpcc"   26 seconds ago       Up 16 seconds       3306/tcp            sick_thompson

d8f49271ed28        tpcc-mysql          "/entrypoint.sh /tpcc"   27 seconds ago       Up 16 seconds       3306/tcp            goofy_leakey

bb72002aeeb6        tpcc-mysql          "/entrypoint.sh /tpcc"   27 seconds ago       Up 15 seconds       3306/tcp            focused_almeida

2c5ac0fd05ad        tpcc-mysql          "/entrypoint.sh /tpcc"   27 seconds ago       Up 16 seconds       3306/tcp            adoring_raman

9db1de997e7e        mysql:5.6           "/entrypoint.sh mysql"   59 seconds ago       Up 57 seconds       3306/tcp            mysql5

12925e8629ec        mysql:5.6           "/entrypoint.sh mysql"   About a minute ago   Up 59 seconds       3306/tcp            mysql4

4b6b1ac52976        mysql:5.6           "/entrypoint.sh mysql"   About a minute ago   Up About a minute   3306/tcp            mysql3

66bd37c25294        mysql:5.6           "/entrypoint.sh mysql"   About a minute ago   Up About a minute   3306/tcp            mysql2

065f3ae133bb        mysql:5.6           "/entrypoint.sh mysql"   About a minute ago   Up About a minute   3306/tcp            mysql1

6f5089ccafef        mysql:5.6           "/entrypoint.sh mysql"   About a minute ago   Up About a minute   3306/tcp            mysql0

Once each container is generation IO, I can measure the performance for each of the volumes that was automatically created by the plug-in:

                                         OPERATIONS          BYTES           AVG TIME(ms)

TYP NAME                                 READ     WRITE      READ     WRITE   READ  WRITE

vol mysql0                                  7      1818       28k    30312k   0.23   9.69

vol mysql1                                  8      1808       33k    30017k   0.15  11.05

vol mysql2                                  6      1647       27k    27678k   0.15  11.04

vol mysql3                                  4      1535       17k    25831k   0.38  10.08

vol mysql4                                  7      1898       30k    31720k   0.16   9.91

vol mysql5                                  7      1547       28k    25105k   0.14  11.18

We can see how the number of Operations per second (IOPS) are almost all writes and they are in the range of 1500 to 1800.

The future QoS feature will provide a vxvolgrp command to set what is the maximum number of IOs per second a volume can provide. Previously, I had mapped each of the volumes to a different group (with the grp_ prefix). This is very flexible and I can combine several volumes into one SLA group or have more granularity as I am doing here.

While the test is running, I am going to tune the MAXIOPS to 1000 for the first three volumes:

[root@target-3 ~]# vxvolgrp -g dockerdg set grp_mysql0 maxiops=1000

[root@target-3 ~]# vxvolgrp -g dockerdg set grp_mysql1 maxiops=1000

[root@target-3 ~]# vxvolgrp -g dockerdg set grp_mysql2 maxiops=1000

And we can see how automatically the IOPS for those volumes are limited to 1000:

                                         OPERATIONS          BYTES           AVG TIME(ms)

TYP NAME                                 READ     WRITE      READ     WRITE   READ  WRITE

vol mysql0                                  3      1000       14k    16628k   0.17  11.57

vol mysql1                                  1      1000        4k    17317k   0.20  11.61

vol mysql2                                  1      1000        7k    16251k   0.22  10.57

vol mysql3                                  5      1912       23k    31210k   0.24   5.65

vol mysql4                                  4      1759       18k    29017k   0.22   7.93

vol mysql5                                  6      1769       24k    30696k   0.20   7.82


Those parameters can be modified anytime. For example I am going to clear the first two:

[root@target-3 ~]# vxvolgrp -g dockerdg clear grp_mysql0 maxiops

[root@target-3 ~]# vxvolgrp -g dockerdg clear grp_mysql1 maxiops

And I can see how the first two volumes are back to their normal performance while the third one is still limited to 1000 IOPS:

                                         OPERATIONS          BYTES           AVG TIME(ms)

TYP NAME                                 READ     WRITE      READ     WRITE   READ  WRITE

vol mysql0                                  7      1707       29k    27821k   0.22  11.41

vol mysql1                                  6      1873       26k    30634k   0.27  10.54

vol mysql2                                  3      1000       14k    16353k   0.17   8.86

vol mysql3                                  3      1314       15k    20942k   0.11   9.29

vol mysql4                                  4      1758       18k    28662k   0.09   9.46

vol mysql5                                  4      1801       17k    30283k   0.05  10.89



As you can see, even in its early stages, InfoScale Storage provides an agile framework to run containers that need persistent storage while providing Quality of Service for container based services and applications. There are some points for integration here where we can facilitate that storage consumption. For example, either via a configuration file or docker command, the user should be able to specify the SLA needed (even for type of storage to be consumed) without knowing the details of the storage systems. We would love to hear your ideas, needs and feedback about that!