Skip to content

feat(docker): add blkio throttling support to scale mode#935

Open
acouvreur wants to merge 1 commit into
mainfrom
add-blkio-throttling
Open

feat(docker): add blkio throttling support to scale mode#935
acouvreur wants to merge 1 commit into
mainfrom
add-blkio-throttling

Conversation

@acouvreur

Copy link
Copy Markdown
Member

Summary

Extends scale mode to support block I/O throttling via BlkioWeight and per-device blkio fields. When a container goes idle, Sablier can now apply blkio constraints; when the container wakes up, the original (active) profile is restored — all without restarting the container.

New labels

Label Example value Description
sablier.idle.blkio-weight 100 Global blkio weight (10–1000)
sablier.active.blkio-weight 500
sablier.idle.blkio-weight-device /dev/vda:100 Per-device weight (comma-separated)
sablier.idle.blkio-device-read-bps /dev/vda:10m Read bandwidth limit
sablier.idle.blkio-device-write-bps /dev/vda:10m Write bandwidth limit
sablier.idle.blkio-device-read-iops /dev/vda:100 Read IOPS limit
sablier.idle.blkio-device-write-iops /dev/vda:100 Write IOPS limit

Multiple devices can be specified with commas: /dev/vda:10m,/dev/vdb:5m.

Changes

  • pkg/sablier/instance.goResourceProfile gains six blkio fields; ScaleConfigFromLabels parses 10 new labels; HasResources covers all fields
  • pkg/provider/docker/container_scale.goapplyResources builds container.Resources with all blkio fields; new parseBpsRate / parseIOpsRate / parseBlkioWeight helpers (human-friendly units: 10m = 10 MB/s)
  • pkg/provider/docker/container_start.go / container_stop.go — call applyResources with active/idle profile
  • Tests — unit tests for all parsers; integration tests via DinD for BlkioWeight (pass) and device throttle (skip with explanation when unsupported, see below)

Known upstream limitation

BlkioDeviceReadBps, BlkioDeviceWriteBps, BlkioDeviceReadIOps, BlkioDeviceWriteIOps, and BlkioWeightDevice are currently silently ignored by the Docker daemon when updating a running container — the API returns 200 OK but the cgroup is not updated.

Root cause: daemon/update_linux.go:toContainerdResources only maps BlkioWeight into specs.LinuxBlockIO; the five per-device fields are never converted.

Tracked upstream: moby/moby#52650

The fields are included in Sablier today so they become effective as soon as the upstream issue is resolved, without requiring a Sablier release. Integration tests skip (rather than fail) in environments where Docker silently drops the update.

Add BlkioWeight and per-device blkio fields to ResourceProfile so that
scale mode can throttle block I/O when a container goes idle and restore
it when the container wakes up.

Changes:
- pkg/sablier/instance.go: add BlkioWeight, BlkioWeightDevice,
  BlkioDeviceReadBps, BlkioDeviceWriteBps, BlkioDeviceReadIOps,
  BlkioDeviceWriteIOps to ResourceProfile; add parseWeightDevices and
  parseThrottleDevices helpers; wire 10 new sablier labels in
  ScaleConfigFromLabels; update HasResources to include all blkio fields
- pkg/provider/docker/container_scale.go: refactor applyResources to
  build container.Resources with all six blkio fields; add parseBpsRate,
  parseIOpsRate, parseBlkioWeight helpers
- pkg/provider/docker/container_start.go: call applyResources with
  active profile
- pkg/provider/docker/container_stop.go: call applyResources with idle
  profile
- pkg/provider/docker/container_scale_unit_test.go: unit tests for
  parseBlkioWeight, parseBpsRate, parseIOpsRate, parseCPUNano,
  parseMemoryBytes
- pkg/sablier/scale_test.go: unit tests for new label parsing
- pkg/provider/docker/container_scale_test.go: integration tests using
  DinD; BlkioWeight tests pass; device throttle tests skip in
  environments where Docker silently drops the update (moby/moby#52650)
- pkg/provider/docker/testcontainers_test.go: enable cgroupv2 io
  controller in DinD after startup

Note: BlkioDeviceReadBps, BlkioDeviceWriteBps, BlkioDeviceReadIOps,
BlkioDeviceWriteIOps, and BlkioWeightDevice are accepted by the API but
currently silently ignored by the Docker daemon when updating a running
container (see moby/moby#52650). The fields are included so that they
become effective once the upstream issue is resolved.
@github-actions github-actions Bot added the provider Issue related to a provider label May 19, 2026
@github-actions

Copy link
Copy Markdown
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Diff between sablier and sablier                                                                                        │
├──────────┬──────────────────────────────────────────────────────────────────────────────┬──────────┬──────────┬─────────┤
│ PERCENT  │ NAME                                                                         │ OLD SIZE │ NEW SIZE │ DIFF    │
├──────────┼──────────────────────────────────────────────────────────────────────────────┼──────────┼──────────┼─────────┤
│ +100%    │ google.golang.org/grpc                                                       │          │ 1.2 MB   │ +1.2 MB │
│ +138.36% │ go.opentelemetry.io/otel                                                     │ 407 kB   │ 971 kB   │ +564 kB │
│ +67.40%  │ github.com/sablierapp/sablier                                                │ 325 kB   │ 544 kB   │ +219 kB │
│ +12.38%  │ google.golang.org/protobuf                                                   │ 1.7 MB   │ 2.0 MB   │ +216 kB │
│ +13.82%  │ golang.org/x/net                                                             │ 789 kB   │ 898 kB   │ +109 kB │
│ +3.21%   │ runtime                                                                      │ 3.2 MB   │ 3.3 MB   │ +103 kB │
│ +100%    │ go.opentelemetry.io/proto/otlp                                               │          │ 84 kB    │ +84 kB  │
│ +4.08%   │ <unnamed:generated>                                                          │ 938 kB   │ 976 kB   │ +38 kB  │
│ +2.14%   │ net                                                                          │ 1.7 MB   │ 1.7 MB   │ +36 kB  │
│ +100%    │ go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin │          │ 36 kB    │ +36 kB  │
│ +8.15%   │ slices                                                                       │ 294 kB   │ 318 kB   │ +24 kB  │
│ +0.09%   │ k8s.io/api                                                                   │ 17 MB    │ 17 MB    │ +15 kB  │
│ +10.69%  │ sync                                                                         │ 106 kB   │ 117 kB   │ +11 kB  │
│ +5.05%   │ github.com/pelletier/go-toml/v2                                              │ 217 kB   │ 228 kB   │ +11 kB  │
│ +0.07%   │ k8s.io/client-go                                                             │ 14 MB    │ 14 MB    │ +10 kB  │
│ +91.81%  │ errors                                                                       │ 11 kB    │ 20 kB    │ +9.8 kB │
│ +100%    │ github.com/grpc-ecosystem/grpc-gateway/v2                                    │          │ 9.8 kB   │ +9.8 kB │
│ +30.61%  │ vendor/golang.org/x/net/dns/dnsmessage                                       │ 29 kB    │ 38 kB    │ +8.8 kB │
│ +13.38%  │ golang.org/x/sys                                                             │ 41 kB    │ 46 kB    │ +5.4 kB │
│ +0.29%   │ crypto                                                                       │ 1.9 MB   │ 1.9 MB   │ +5.4 kB │
│ +100%    │ google.golang.org/genproto/googleapis/api                                    │          │ 5.2 kB   │ +5.2 kB │
│ +100%    │ google.golang.org/genproto/googleapis/rpc                                    │          │ 5.1 kB   │ +5.1 kB │
│ +1.98%   │ github.com/gabriel-vasile/mimetype                                           │ 184 kB   │ 187 kB   │ +3.6 kB │
│ +2.66%   │ html                                                                         │ 136 kB   │ 140 kB   │ +3.6 kB │
│ +1.94%   │ time                                                                         │ 157 kB   │ 160 kB   │ +3.0 kB │
│ +1.10%   │ os                                                                           │ 210 kB   │ 213 kB   │ +2.3 kB │
│ +0.13%   │ k8s.io/apimachinery                                                          │ 1.8 MB   │ 1.8 MB   │ +2.3 kB │
│ +3.76%   │ io                                                                           │ 54 kB    │ 56 kB    │ +2.0 kB │
│ +0.49%   │ github.com/go-playground/validator/v10                                       │ 326 kB   │ 328 kB   │ +1.6 kB │
│ +0.47%   │ github.com/diskfs/go-diskfs                                                  │ 329 kB   │ 330 kB   │ +1.5 kB │
│ +44.83%  │ github.com/docker/go-units                                                   │ 3.4 kB   │ 5.0 kB   │ +1.5 kB │
│ +1.36%   │ internal/poll                                                                │ 95 kB    │ 96 kB    │ +1.3 kB │
│ +0.24%   │ github.com/json-iterator/go                                                  │ 463 kB   │ 464 kB   │ +1.1 kB │
│ +0.29%   │ math                                                                         │ 305 kB   │ 306 kB   │ +899 B  │
│ +18.90%  │ internal/singleflight                                                        │ 4.2 kB   │ 5.1 kB   │ +803 B  │
│ +0.27%   │ text/template                                                                │ 292 kB   │ 293 kB   │ +787 B  │
│ +100%    │ github.com/cenkalti/backoff/v5                                               │          │ 786 B    │ +786 B  │
│ +0.25%   │ github.com/google/go-cmp                                                     │ 297 kB   │ 298 kB   │ +733 B  │
│ +0.21%   │ github.com/gin-gonic/gin                                                     │ 337 kB   │ 337 kB   │ +709 B  │
│ +0.09%   │ github.com/goccy/go-yaml                                                     │ 702 kB   │ 703 kB   │ +666 B  │
│ +0.05%   │ github.com/quic-go/quic-go                                                   │ 1.3 MB   │ 1.3 MB   │ +641 B  │
│ +0.99%   │ strings                                                                      │ 58 kB    │ 58 kB    │ +571 B  │
│ +13.69%  │ internal/syscall/unix                                                        │ 4.1 kB   │ 4.6 kB   │ +557 B  │
│ +1.23%   │ bytes                                                                        │ 41 kB    │ 41 kB    │ +502 B  │
│ +0.53%   │ github.com/go-viper/mapstructure/v2                                          │ 88 kB    │ 89 kB    │ +469 B  │
│ +10.56%  │ go.shape                                                                     │ 4.2 kB   │ 4.6 kB   │ +441 B  │
│ +0.08%   │ encoding                                                                     │ 417 kB   │ 418 kB   │ +350 B  │
│ +0.35%   │ syscall                                                                      │ 97 kB    │ 97 kB    │ +336 B  │
│ +0.45%   │ github.com/spf13/viper                                                       │ 73 kB    │ 73 kB    │ +331 B  │
│ +0.67%   │ context                                                                      │ 47 kB    │ 48 kB    │ +319 B  │
│ +0.09%   │ reflect                                                                      │ 340 kB   │ 340 kB   │ +307 B  │
│ +0.11%   │ sigs.k8s.io/structured-merge-diff/v6                                         │ 276 kB   │ 276 kB   │ +290 B  │
│ +0.06%   │ k8s.io/kube-openapi                                                          │ 466 kB   │ 467 kB   │ +286 B  │
│ +12.25%  │ iter                                                                         │ 2.3 kB   │ 2.6 kB   │ +280 B  │
│ +0.21%   │ github.com/emicklei/go-restful/v3                                            │ 133 kB   │ 134 kB   │ +275 B  │
│ +0.15%   │ regexp                                                                       │ 185 kB   │ 186 kB   │ +273 B  │
│ +0.10%   │ github.com/spf13/cobra                                                       │ 239 kB   │ 239 kB   │ +243 B  │
│ +0.07%   │ github.com/prometheus/client_golang                                          │ 309 kB   │ 310 kB   │ +223 B  │
│ +0.03%   │ go.mongodb.org/mongo-driver/v2                                               │ 672 kB   │ 672 kB   │ +186 B  │
│ +0.06%   │ go.yaml.in/yaml/v3                                                           │ 312 kB   │ 312 kB   │ +179 B  │
│ +0.06%   │ gopkg.in/yaml.v3                                                             │ 305 kB   │ 306 kB   │ +174 B  │
│ +0.08%   │ golang.org/x/text                                                            │ 162 kB   │ 162 kB   │ +129 B  │
│ +0.74%   │ github.com/pmezard/go-difflib                                                │ 17 kB    │ 17 kB    │ +124 B  │
│ +0.04%   │ go.yaml.in/yaml/v2                                                           │ 275 kB   │ 275 kB   │ +105 B  │
│ +0.11%   │ golang.org/x/crypto                                                          │ 91 kB    │ 91 kB    │ +101 B  │
│ +0.15%   │ github.com/prometheus/common                                                 │ 68 kB    │ 68 kB    │ +101 B  │
│ +0.08%   │ log                                                                          │ 126 kB   │ 126 kB   │ +96 B   │
│ +0.21%   │ sigs.k8s.io/randfill                                                         │ 36 kB    │ 37 kB    │ +78 B   │
│ +0.09%   │ mime                                                                         │ 78 kB    │ 79 kB    │ +72 B   │
│ +0.02%   │ github.com/fxamacker/cbor/v2                                                 │ 300 kB   │ 300 kB   │ +70 B   │
│ +0.77%   │ github.com/tniswong/go.rfcx                                                  │ 9.1 kB   │ 9.2 kB   │ +70 B   │
│ +0.35%   │ path                                                                         │ 20 kB    │ 20 kB    │ +69 B   │
│ +0.08%   │ github.com/gorilla/websocket                                                 │ 84 kB    │ 84 kB    │ +68 B   │
│ +0.12%   │ github.com/davecgh/go-spew                                                   │ 54 kB    │ 54 kB    │ +66 B   │
│ +0.14%   │ flag                                                                         │ 46 kB    │ 46 kB    │ +65 B   │
│ +0.03%   │ sigs.k8s.io/json                                                             │ 173 kB   │ 173 kB   │ +58 B   │
│ +0.15%   │ github.com/lmittmann/tint                                                    │ 36 kB    │ 36 kB    │ +56 B   │
│ +0.15%   │ k8s.io/utils                                                                 │ 32 kB    │ 32 kB    │ +50 B   │
│ +0.04%   │ fmt                                                                          │ 106 kB   │ 106 kB   │ +44 B   │
│ +0.47%   │ github.com/sagikazarmark/locafero                                            │ 7.7 kB   │ 7.7 kB   │ +36 B   │
│ +0.14%   │ sigs.k8s.io/yaml                                                             │ 25 kB    │ 25 kB    │ +36 B   │
│ +0.10%   │ vendor/golang.org/x/net/http2/hpack                                          │ 35 kB    │ 35 kB    │ +35 B   │
│ +0.34%   │ expvar                                                                       │ 9.4 kB   │ 9.4 kB   │ +32 B   │
│ +0.04%   │ github.com/go-logr/logr                                                      │ 59 kB    │ 59 kB    │ +24 B   │
│ +0.01%   │ github.com/spf13/pflag                                                       │ 302 kB   │ 302 kB   │ +22 B   │
│ +0.01%   │ github.com/modern-go/reflect2                                                │ 148 kB   │ 149 kB   │ +21 B   │
│ +0.01%   │ go.opentelemetry.io/auto/sdk                                                 │ 89 kB    │ 89 kB    │ +11 B   │
│ +0.01%   │ github.com/docker/go-connections                                             │ 6.8 kB   │ 6.8 kB   │ +1 B    │
│ -0.00%   │ github.com/quic-go/qpack                                                     │ 22 kB    │ 22 kB    │ -1 B    │
│ -0.01%   │ github.com/pkg/errors                                                        │ 12 kB    │ 12 kB    │ -1 B    │
│ -0.02%   │ compress/zlib                                                                │ 9.8 kB   │ 9.8 kB   │ -2 B    │
│ -0.02%   │ vendor/golang.org/x/net/http/httpproxy                                       │ 14 kB    │ 14 kB    │ -3 B    │
│ -0.01%   │ github.com/sourcegraph/conc                                                  │ 41 kB    │ 41 kB    │ -3 B    │
│ -0.02%   │ embed                                                                        │ 12 kB    │ 12 kB    │ -3 B    │
│ -0.01%   │ sort                                                                         │ 29 kB    │ 29 kB    │ -3 B    │
│ -0.03%   │ github.com/go-logr/stdr                                                      │ 16 kB    │ 16 kB    │ -5 B    │
│ -0.01%   │ internal/strconv                                                             │ 43 kB    │ 43 kB    │ -6 B    │
│ -0.06%   │ github.com/gin-contrib/sse                                                   │ 9.6 kB   │ 9.6 kB   │ -6 B    │
│ -0.01%   │ internal/abi                                                                 │ 74 kB    │ 74 kB    │ -6 B    │
│ -0.47%   │ github.com/mailru/easyjson                                                   │ 1.5 kB   │ 1.5 kB   │ -7 B    │
│ -0.06%   │ text/tabwriter                                                               │ 14 kB    │ 14 kB    │ -8 B    │
│ -0.03%   │ github.com/samber/slog-gin                                                   │ 32 kB    │ 32 kB    │ -9 B    │
│ -0.05%   │ github.com/djherbis/times                                                    │ 19 kB    │ 19 kB    │ -10 B   │
│ -0.03%   │ github.com/distribution/reference                                            │ 39 kB    │ 39 kB    │ -12 B   │
│ -0.02%   │ vendor/golang.org/x/text/unicode/norm                                        │ 50 kB    │ 50 kB    │ -12 B   │
│ -0.04%   │ unique                                                                       │ 34 kB    │ 34 kB    │ -14 B   │
│ -0.23%   │ internal/cpu                                                                 │ 6.1 kB   │ 6.1 kB   │ -14 B   │
│ -0.03%   │ github.com/prometheus/procfs                                                 │ 43 kB    │ 43 kB    │ -14 B   │
│ -0.09%   │ golang.org/x/time                                                            │ 18 kB    │ 18 kB    │ -16 B   │
│ -0.59%   │ golang.org/x/sync                                                            │ 2.7 kB   │ 2.7 kB   │ -16 B   │
│ -0.04%   │ bufio                                                                        │ 38 kB    │ 38 kB    │ -17 B   │
│ -0.03%   │ github.com/leodido/go-urn                                                    │ 55 kB    │ 55 kB    │ -18 B   │
│ -0.08%   │ github.com/spf13/afero                                                       │ 21 kB    │ 21 kB    │ -18 B   │
│ -0.12%   │ hash                                                                         │ 15 kB    │ 15 kB    │ -18 B   │
│ -0.00%   │ github.com/moby/moby/client                                                  │ 434 kB   │ 434 kB   │ -20 B   │
│ -0.08%   │ vendor/golang.org/x/crypto/cryptobyte                                        │ 28 kB    │ 28 kB    │ -21 B   │
│ -0.09%   │ github.com/opencontainers/go-digest                                          │ 23 kB    │ 23 kB    │ -22 B   │
│ -0.09%   │ github.com/jinzhu/copier                                                     │ 27 kB    │ 26 kB    │ -23 B   │
│ -0.34%   │ github.com/modern-go/concurrent                                              │ 7.0 kB   │ 7.0 kB   │ -24 B   │
│ -0.07%   │ github.com/containerd/errdefs                                                │ 33 kB    │ 33 kB    │ -25 B   │
│ -0.08%   │ gopkg.in/inf.v0                                                              │ 35 kB    │ 35 kB    │ -30 B   │
│ -0.03%   │ k8s.io/klog/v2                                                               │ 124 kB   │ 124 kB   │ -40 B   │
│ -0.04%   │ github.com/spf13/cast                                                        │ 107 kB   │ 107 kB   │ -44 B   │
│ -0.11%   │ internal/runtime/maps                                                        │ 44 kB    │ 44 kB    │ -46 B   │
│ -0.51%   │ github.com/go-openapi/swag                                                   │ 10 kB    │ 10 kB    │ -53 B   │
│ -0.04%   │ github.com/moby/moby/api                                                     │ 149 kB   │ 149 kB   │ -66 B   │
│ -1.15%   │ vendor/golang.org/x/sys/cpu                                                  │ 6.4 kB   │ 6.4 kB   │ -74 B   │
│ -0.02%   │ github.com/luthermonson/go-proxmox                                           │ 333 kB   │ 333 kB   │ -77 B   │
│ -0.14%   │ vendor/golang.org/x/crypto/chacha20poly1305                                  │ 71 kB    │ 71 kB    │ -101 B  │
│ -0.39%   │ github.com/prometheus/client_model                                           │ 47 kB    │ 47 kB    │ -183 B  │
│ -0.06%   │ github.com/google/gnostic-models                                             │ 1.6 MB   │ 1.6 MB   │ -900 B  │
│ -27.44%  │ go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp                │ 74 kB    │ 54 kB    │ -20 kB  │
├──────────┼──────────────────────────────────────────────────────────────────────────────┼──────────┼──────────┼─────────┤
│ +23.88%  │ .rodata                                                                      │ 2.1 MB   │ 2.6 MB   │ +505 kB │
│ +6.05%   │ .data                                                                        │ 196 kB   │ 208 kB   │ +12 kB  │
│ +1.86%   │ .noptrdata                                                                   │ 451 kB   │ 459 kB   │ +8.4 kB │
├──────────┼──────────────────────────────────────────────────────────────────────────────┼──────────┼──────────┼─────────┤
│ +5.57%   │ sablier                                                                      │ 59 MB    │ 62 MB    │ +3.3 MB │
│          │ sablier                                                                      │          │          │         │
└──────────┴──────────────────────────────────────────────────────────────────────────────┴──────────┴──────────┴─────────┘

@github-actions

Copy link
Copy Markdown

Test Results

✅ All tests passed! | 542 tests in 211.922s

⚠️ 3 test(s) were flaky (failed then passed on rerun)

  • github.com/sablierapp/sablier/pkg/provider/dockerswarm/TestDockerSwarmProvider_InstanceEvents_Created
  • github.com/sablierapp/sablier/pkg/provider/podman/TestPodmanProvider_GetState
  • github.com/sablierapp/sablier/pkg/provider/podman/TestPodmanProvider_GetState/running_container_with_sablier_labels
    View HTML Test Report

@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
74.3% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

provider Issue related to a provider

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant