7 cgroup subsys "blkio" implements the block io controller. There seems to be
8 a need of various kinds of IO control policies (like proportional BW, max BW)
9 both at leaf nodes as well as at intermediate nodes in a storage hierarchy.
10 Plan is to use the same cgroup based management interface for blkio controller
11 and based on user options switch IO policies in the background.
13 One IO control policy is throttling policy which can be used to
14 specify upper IO rate limits on devices. This policy is implemented in
15 generic block layer and can be used on leaf nodes as well as higher
16 level logical devices like device mapper.
20 Throttling/Upper Limit policy
21 -----------------------------
22 - Enable Block IO controller::
26 - Enable throttling in block layer::
28 CONFIG_BLK_DEV_THROTTLING=y
30 - Mount blkio controller (see cgroups.txt, Why are cgroups needed?)::
32 mount -t cgroup -o blkio none /sys/fs/cgroup/blkio
34 - Specify a bandwidth rate on particular device for root group. The format
35 for policy is "<major>:<minor> <bytes_per_second>"::
37 echo "8:16 1048576" > /sys/fs/cgroup/blkio/blkio.throttle.read_bps_device
39 Above will put a limit of 1MB/second on reads happening for root group
40 on device having major/minor number 8:16.
42 - Run dd to read a file and see if rate is throttled to 1MB/s or not::
44 # dd iflag=direct if=/mnt/common/zerofile of=/dev/null bs=4K count=1024
47 4194304 bytes (4.2 MB) copied, 4.0001 s, 1.0 MB/s
49 Limits for writes can be put using blkio.throttle.write_bps_device file.
54 Throttling implements hierarchy support; however,
55 throttling's hierarchy support is enabled iff "sane_behavior" is
56 enabled from cgroup side, which currently is a development option and
57 not publicly available.
59 If somebody created a hierarchy like as follows::
67 Throttling with "sane_behavior" will handle the
68 hierarchy correctly. For throttling, all limits apply
69 to the whole subtree while all statistics are local to the IOs
70 directly generated by tasks in that cgroup.
72 Throttling without "sane_behavior" enabled from cgroup side will
73 practically treat all groups at same level as if it looks like the
78 root test1 test2 test3
80 Various user visible config options
81 ===================================
83 - Block IO controller.
85 CONFIG_BFQ_CGROUP_DEBUG
86 - Debug help. Right now some additional stats file show up in cgroup
87 if this option is enabled.
89 CONFIG_BLK_DEV_THROTTLING
90 - Enable block device throttling support in block layer.
92 Details of cgroup files
93 =======================
94 Proportional weight policy files
95 --------------------------------
97 - Specifies per cgroup weight. This is default weight of the group
98 on all the devices until and unless overridden by per device rule.
99 (See blkio.weight_device).
100 Currently allowed range of weights is from 10 to 1000.
102 - blkio.weight_device
103 - One can specify per cgroup per device rules using this interface.
104 These rules override the default value of group weight as specified
107 Following is the format::
109 # echo dev_maj:dev_minor weight > blkio.weight_device
111 Configure weight=300 on /dev/sdb (8:16) in this cgroup::
113 # echo 8:16 300 > blkio.weight_device
114 # cat blkio.weight_device
118 Configure weight=500 on /dev/sda (8:0) in this cgroup::
120 # echo 8:0 500 > blkio.weight_device
121 # cat blkio.weight_device
126 Remove specific weight for /dev/sda in this cgroup::
128 # echo 8:0 0 > blkio.weight_device
129 # cat blkio.weight_device
133 - blkio.leaf_weight[_device]
134 - Equivalents of blkio.weight[_device] for the purpose of
135 deciding how much weight tasks in the given cgroup has while
136 competing with the cgroup's child cgroups. For details,
137 please refer to Documentation/block/cfq-iosched.txt.
140 - disk time allocated to cgroup per device in milliseconds. First
141 two fields specify the major and minor number of the device and
142 third field specifies the disk time allocated to group in
146 - number of sectors transferred to/from disk by the group. First
147 two fields specify the major and minor number of the device and
148 third field specifies the number of sectors transferred by the
149 group to/from the device.
151 - blkio.io_service_bytes
152 - Number of bytes transferred to/from the disk by the group. These
153 are further divided by the type of operation - read or write, sync
154 or async. First two fields specify the major and minor number of the
155 device, third field specifies the operation type and the fourth field
156 specifies the number of bytes.
159 - Number of IOs (bio) issued to the disk by the group. These
160 are further divided by the type of operation - read or write, sync
161 or async. First two fields specify the major and minor number of the
162 device, third field specifies the operation type and the fourth field
163 specifies the number of IOs.
165 - blkio.io_service_time
166 - Total amount of time between request dispatch and request completion
167 for the IOs done by this cgroup. This is in nanoseconds to make it
168 meaningful for flash devices too. For devices with queue depth of 1,
169 this time represents the actual service time. When queue_depth > 1,
170 that is no longer true as requests may be served out of order. This
171 may cause the service time for a given IO to include the service time
172 of multiple IOs when served out of order which may result in total
173 io_service_time > actual time elapsed. This time is further divided by
174 the type of operation - read or write, sync or async. First two fields
175 specify the major and minor number of the device, third field
176 specifies the operation type and the fourth field specifies the
177 io_service_time in ns.
180 - Total amount of time the IOs for this cgroup spent waiting in the
181 scheduler queues for service. This can be greater than the total time
182 elapsed since it is cumulative io_wait_time for all IOs. It is not a
183 measure of total time the cgroup spent waiting but rather a measure of
184 the wait_time for its individual IOs. For devices with queue_depth > 1
185 this metric does not include the time spent waiting for service once
186 the IO is dispatched to the device but till it actually gets serviced
187 (there might be a time lag here due to re-ordering of requests by the
188 device). This is in nanoseconds to make it meaningful for flash
189 devices too. This time is further divided by the type of operation -
190 read or write, sync or async. First two fields specify the major and
191 minor number of the device, third field specifies the operation type
192 and the fourth field specifies the io_wait_time in ns.
195 - Total number of bios/requests merged into requests belonging to this
196 cgroup. This is further divided by the type of operation - read or
197 write, sync or async.
200 - Total number of requests queued up at any given instant for this
201 cgroup. This is further divided by the type of operation - read or
202 write, sync or async.
204 - blkio.avg_queue_size
205 - Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y.
206 The average queue size for this cgroup over the entire time of this
207 cgroup's existence. Queue size samples are taken each time one of the
208 queues of this cgroup gets a timeslice.
210 - blkio.group_wait_time
211 - Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y.
212 This is the amount of time the cgroup had to wait since it became busy
213 (i.e., went from 0 to 1 request queued) to get a timeslice for one of
214 its queues. This is different from the io_wait_time which is the
215 cumulative total of the amount of time spent by each IO in that cgroup
216 waiting in the scheduler queue. This is in nanoseconds. If this is
217 read when the cgroup is in a waiting (for timeslice) state, the stat
218 will only report the group_wait_time accumulated till the last time it
219 got a timeslice and will not include the current delta.
222 - Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y.
223 This is the amount of time a cgroup spends without any pending
224 requests when not being served, i.e., it does not include any time
225 spent idling for one of the queues of the cgroup. This is in
226 nanoseconds. If this is read when the cgroup is in an empty state,
227 the stat will only report the empty_time accumulated till the last
228 time it had a pending request and will not include the current delta.
231 - Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y.
232 This is the amount of time spent by the IO scheduler idling for a
233 given cgroup in anticipation of a better request than the existing ones
234 from other queues/cgroups. This is in nanoseconds. If this is read
235 when the cgroup is in an idling state, the stat will only report the
236 idle_time accumulated till the last idle period and will not include
240 - Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y. This
241 gives the statistics about how many a times a group was dequeued
242 from service tree of the device. First two fields specify the major
243 and minor number of the device and third field specifies the number
244 of times a group was dequeued from a particular device.
247 - Recursive version of various stats. These files show the
248 same information as their non-recursive counterparts but
249 include stats from all the descendant cgroups.
251 Throttling/Upper limit policy files
252 -----------------------------------
253 - blkio.throttle.read_bps_device
254 - Specifies upper limit on READ rate from the device. IO rate is
255 specified in bytes per second. Rules are per device. Following is
258 echo "<major>:<minor> <rate_bytes_per_second>" > /cgrp/blkio.throttle.read_bps_device
260 - blkio.throttle.write_bps_device
261 - Specifies upper limit on WRITE rate to the device. IO rate is
262 specified in bytes per second. Rules are per device. Following is
265 echo "<major>:<minor> <rate_bytes_per_second>" > /cgrp/blkio.throttle.write_bps_device
267 - blkio.throttle.read_iops_device
268 - Specifies upper limit on READ rate from the device. IO rate is
269 specified in IO per second. Rules are per device. Following is
272 echo "<major>:<minor> <rate_io_per_second>" > /cgrp/blkio.throttle.read_iops_device
274 - blkio.throttle.write_iops_device
275 - Specifies upper limit on WRITE rate to the device. IO rate is
276 specified in io per second. Rules are per device. Following is
279 echo "<major>:<minor> <rate_io_per_second>" > /cgrp/blkio.throttle.write_iops_device
281 Note: If both BW and IOPS rules are specified for a device, then IO is
282 subjected to both the constraints.
284 - blkio.throttle.io_serviced
285 - Number of IOs (bio) issued to the disk by the group. These
286 are further divided by the type of operation - read or write, sync
287 or async. First two fields specify the major and minor number of the
288 device, third field specifies the operation type and the fourth field
289 specifies the number of IOs.
291 - blkio.throttle.io_service_bytes
292 - Number of bytes transferred to/from the disk by the group. These
293 are further divided by the type of operation - read or write, sync
294 or async. First two fields specify the major and minor number of the
295 device, third field specifies the operation type and the fourth field
296 specifies the number of bytes.
298 Common files among various policies
299 -----------------------------------
301 - Writing an int to this file will result in resetting all the stats