Repository navigation
CI never resolves dependencies at their declared lower bounds, so stale floors ship unnoticed #107
Description
Activity
The audit landed for
faststream-outboxacross modern-python/faststream-outbox#185, modern-python/faststream-outbox#186 and modern-python/faststream-outbox#187. Six declared floors were wrong. Three findings from doing it by hand change what the job in this issue has to look like.1. An install-only job would miss half of them
Where each floor actually surfaced:
floor was now surfaced at asyncpg >=0.29marked 0.29 / 0.30 / 0.31 install — sdist build fails on 3.13+ typing-extensions >=4.12.0>=4.12.2install — drags pydantic to a pydantic-core with no cp313 wheel pydantic (3.13/3.14 branches) undeclared >=2.8/>=2.12install — pydantic-core wheel coverage pydantic (the v2 requirement itself) undeclared >=2tests — installs fine, then AttributeError: '_PydanticBody' object has no attribute 'model_dump'sqlalchemy >=2.0marked >=2.0.31on 3.13+import — installs fine, then SQLAlchemy's own TypingOnlyassertion firesopentelemetry >=1.20>=1.21tests — get_meter() got an unexpected keyword argument 'schema_url'fastapi >=0.95>=0.113tests — TypeError: must be called with a dataclass type or instanceInstall alone catches three, and for pydantic only the wheel-coverage half — it would have left the package still declaring a range that admits pydantic 1.x, under which publishing a model raises. The job has to install, import, and run the suite, in that order, or it reports a floor clean that is not.
2. The test tooling it installs cannot simply be "current"
The two-step shape proposed in the issue body works — runtime deps at
lowest-direct, test tooling at default resolution — but the second step needs care.starlette1.6 switchedTestClienttohttpx2, andfaststream-outbox's dev group correctly declareshttpx2>=2.2. At the fastapi floor, though, the resolved starlette is 0.38, which knows nothing abouthttpx2and callshttpx.Client(app=...)— removed in httpx 0.28. Installing currenthttpxmakes every FastAPI test fail with:TypeError: Client.__init__() got an unexpected keyword argument 'app'That looks exactly like a floor problem and is not one. It cost me a wrong bisect before I noticed; the real boundary only appeared with
httpx>=0.27,<0.28.So the tooling has to be something the resolved stack can use, which is not always the newest. In practice that means either pinning the tooling per repo, or reading a failure in this job with the question "is this the floor, or my tooling?" in hand.
3. Findings arrive one at a time, so budget for iteration
A resolver stops at the first wall. Each fix only exposes the next, and there is no way to see the whole list up front. This took four passes for one package:
asyncpg → 3.13/3.14 install fails typing-extensions → still fails, now on pydantic-core pydantic → installs, sqlalchemy fails at import sqlalchemy → imports, 12 tests fail opentelemetry + fastapi → 624 passed on every interpreterWorth knowing before anyone schedules this across 24 repos: the first green run is several rounds away per repo, not one.
Rollout note
faststream-outboxis now green at the lower bound — the full suite, Postgres integration tests included, 624 passed on 3.11, 3.12, 3.13, 3.14 and 3.14t. That makes it a good repo to build the job against, since a correct job must now pass on it and a broken one will not.The other 23 have not been audited. Six wrong floors in the first package looked at is not a reassuring base rate.
Audit finished across the remaining 23 repos. 13 had at least one broken floor, on top of the six already fixed in
faststream-outbox.Filed
repo issue declaration fails at db-retry modern-python/db-retry#52 sqlalchemy[asyncio](no bound) → 0.1.0install, 3.11 and 3.14 semvertag modern-python/semvertag#78 semver(no bound) → 0.0.1install, 3.11 and 3.14 eof-fixer modern-python/eof-fixer#48 pathspec(no bound)import that-depends modern-python/that-depends#251 fastapi,faststream(no bounds)import modern-di-grpc modern-python/modern-di-grpc#26 grpcio>=1.48,<2install, 3.10 and 3.14 compose2pod modern-python/compose2pod#126 PyYAML>=6install, 3.14 httpware modern-python/httpware#132 msgspec>=0.18install, 3.14 modern-di-aiogram modern-python/modern-di-aiogram#28 aiogram>=3.2,<4install, 3.14 faststream-redis-timers modern-python/faststream-redis-timers#88 typing-extensions>=4.12.0install, 3.14 lite-bootstrap modern-python/lite-bootstrap#242 litestar>=2.19import modern-di-arq modern-python/modern-di-arq#28 arq>=0.25,<1tests modern-di-typer modern-python/modern-di-typer#53 typer>=0.9,<1tests modern-di-starlette modern-python/modern-di-starlette#34 starlette>=0.40,<2tests Clean at the lower bound — install, import and full suite:
faststream-concurrent-aiokafka,modern-di,modern-di-aiohttp,modern-di-celery,modern-di-faststream,modern-di-fastapi,modern-di-flask,modern-di-litestar,modern-di-pytest,modern-di-taskiq.What the spread says about the job
Four repos declare no lower bound at all.
db-retryresolves SQLAlchemy to 0.1.0, from 2006;semvertagresolvessemverto 0.0.1. These are not floors that went stale, they are floors that were never written, and a bare package name in a dependency list reads as deliberate until someone resolves it downward. Cheap to catch and worth a separate lint: every direct dependency should carry a lower bound.The stage split holds, and matters. Of the 13: 6 fail at install, 3 at import, 4 only in the suite. An install-only job would find fewer than half. This is the same ratio as
faststream-outboxand is now measured across 23 repos rather than one.modern-di-grpcfails at both ends of its range.grpcio>=1.48,<2does not install on 3.10 or 3.14, so there is no interpreter where the declared minimum works. A job that runs only on the newest Python would report this as a 3.14 wheel-coverage problem and miss that it is broader.The httpx/httpx2 trap is real and it bit twice more.
modern-di-fastapiandmodern-di-starletteboth first appeared as failures purely because the harness installed currenthttpx. Re-run withhttpx>=0.27,<0.28, fastapi went clean and starlette's failures proved genuine. Anyone building this job should expect to spend time telling floor problems apart from tooling problems — the two look identical from the test output.Caveat on the clean list
The clean results are the optimistic reading. I did not re-check each one against the tooling-compatibility trap above, so a repo listed clean could still be hiding a floor problem behind tooling that happened to work. The 13 failures are confirmed; the 10 passes are "passed under one reasonable tooling choice".
Scope this implies
14 of 24 repos are affected.
faststream-outboxtook four rounds and three PRs on its own, because a resolver stops at the first wall and each fix only exposes the next. Several of the repos above will behave the same way, and none of the 13 has had its correct floor bisected yet — the issues describe the defect and how to find the floor, not the answer.Split the bare-dependency check out into #108. It is a static parse of
pyproject.tomlwith no resolver, interpreter matrix or test run behind it, so it does not belong in the same job as this one.One finding from writing it up that changes the numbers here: the audit undercounted. A resolver stops at the first wall, so it reported one broken dependency per repo. The static scan shows
db-retryhas three unbounded dependencies andsemvertaghas five, not the one each that the per-repo issues describe. Fixing only what the audit surfaced would leave most of them in place.Two updates, both from
compose2pod— which now has a working floors job, built in modern-python/compose2pod#129 by another session and released as 0.6.1.A reference implementation exists
uv pip install --resolution lowest-direct --only-binary PyYAML ".[yaml]" uv run --no-project python -c "import compose2pod, yaml; print(yaml.__version__)"
on 3.10 and 3.14. That covers the install and import classes. It does not cover the third — the suite at the floor — which is where 5 of the 14 broken repos landed, so the gap in this issue is narrower than my earlier comment implied but real. Correcting my own framing: I described the alternative as "install-only", and a job that ends in an import is strictly better than that.
--only-binaryis load-bearing, and I would have got this wrongThe measurement that produced that flag, on Python 3.14:
PyYAML 6.0 install: no wheel-only: no PyYAML 6.0.1 install: YES wheel-only: no PyYAML 6.0.2 install: YES wheel-only: no PyYAML 6.0.3 install: yes wheel-only: YES6.0.1 and 6.0.2 install by compiling the sdist. A bisect that accepts install success picks
>=6.0.1, which passes locally and in CI and still fails in any minimal image with no toolchain. I was about to do exactly that before the other session shared the numbers.So a floors job that installs without
--only-binarycan go green on a broken floor. Every wheel-coverage floor in this issue's scope should be chosen from PyPI wheel tags, with the install as corroboration rather than as the evidence.Run it where CI runs, not on a laptop
This cut both ways in one afternoon:
- My audit ran on macOS arm64 and over-reported: I filed
grpcio>=1.48as broken on 3.10 as well as 3.14. It is not. grpcio 1.48.1 ships no macOS arm64 wheel but does ship cp310 for Linux, and CI on fix(deps): mark the grpcio floor and raise modern-di's modern-di-grpc#27 has since confirmed it. Corrected there. - Trusting a local compile instead of a runner under-reports, which is the PyYAML case above.
Neither error is visible from inside the machine that made it. The job belongs in CI on the same OS the rest of the matrix uses, and any hand audit should be read as provisional until a runner agrees.
Scope note
The other session is not extending the floors job beyond
compose2pod; the workflow files in the remaining 23 repos are unclaimed. #108 (the bare-dependency lint) is now satisfied by every repo, since the four offenders are fixed and merged.- My audit ran on macOS arm64 and over-reported: I filed
Done. The job exists, is a standard, and is deployed everywhere it applies.
Where it landed
floorsis onmainin 23 of 24 repos. The twenty-fourth ismodern-di, which CI6 exempts by name — it declares no runtime dependencies and no extras, so there are no floors to install:dependencies = [] # and no [project.optional-dependencies]
A job there would resolve the empty set. That is a deliberate omission, not a gap.
that-dependswas the last real one and is now done in modern-python/that-depends#253. Its two extras carried floors nothing installed. One caveat recorded there: 3.14 is dropped under CI6's carve-out, because the dev group'smkdocspullswatchdog, which ships no cp314 wheel, so--no-buildclimbs tomkdocs 2.0.devand the leg would measure upstream's packaging rather than our floors. Moving the docs tooling out of the dev group, which every other repo already does and TL3 implies, would restore it. That is athat-dependsfollow-up, not a blocker here.What the standard ended up saying that this issue did not
Two corrections to my own earlier comments on this issue, both worth keeping:
Wheel-only is not sufficient on its own. I wrote that
--only-binarywas load-bearing. It is, and it is also not enough: it makes the wheel-less version ineligible andlowest-directthen climbs to the lowest version that has a wheel, so the job passes on a version the repo never declared. That is why CI6 prescribes two steps — pin the floors with builds allowed, then install those pins wheel-only. compose2pod#135 is the case that found it and compose2pod#139 the fix.Dev-group floors turned out to be unnecessary. I argued the 179 unbounded dev entries blocked the simple form and would have to be bounded first.
-r pyproject.tomlwith--group devsidesteps it: only the project's own dependencies get pinned, so the harness resolves at newest and needs no floors. The whole objection dissolved.And #108
Closed as not-planned, correctly. A bare dependency turns the floors job red on its own — the compile step pins the oldest release on PyPI and the wheel-only install then fails on it. No lint needed. If astral-sh/uv#12566 lands, the message gets specific for free.
Closing this. The remaining work is per-repo, not systemic.
What
No repo in the org ever installs its dependencies at the lower bound of what it declares. Every gate — PR checks and the daily scheduled check alike — resolves highest. A floor that is a lie is therefore invisible until a user hits it.
24 repos have a
_checks.yml. None of them has a lower-bound job.Why now — three instances in one day, 2026-09-20
faststream-outboxdeclaredfaststream>=0.7.1while the code needed 0.7.6.import faststream_outboxfailed outright on 0.7.5 withTypeError: TestOutboxBroker.__init_subclass__() takes no keyword arguments. Caught only because someone asked whether the work was backward compatible; fixed in chore(deps): adapt to faststream 0.7.6 and raise its floor faststream-outbox#182.faststream-redis-timersandfaststream-concurrent-aiokafkawere both capped<0.7.6and then re-floored within the hour (redis-timers#84 → Relative links in README.md 404 on the PyPI package page #85, aiokafka#80 → docs: stop asserting the PR-body shape in AGENTS.md #79). Floor-range edits made under time pressure, with nothing verifying either end.faststream-outboxdeclaresasyncpg>=0.29and claims support for Python 3.11–3.14.asyncpg 0.29.0ships wheels for cp38–cp312 only, but declaresrequires-python >=3.8.0, so on 3.13+ uv falls through to the sdist and the build fails. The first release with cp313 wheels is 0.30.0. That floor is unsatisfiable on half the supported Python range, today, and nothing reports it.Item 3 was found by running the proposed job by hand. It is not hypothetical.
Proposal
A
lowestjob in_checks.yml, alongside the existing matrix. The naive form does not work — see below — so the shape that does:Verified on
faststream-outboxat Python 3.11:Every runtime dependency sits on its declared floor while the test tooling stays current.
Why not just set
UV_RESOLUTION=lowest-directon the existing jobBecause
lowest-directapplies to every direct dependency, dev groups included, and all 24 repos declare dev/lint dependencies with no version specifier at all:that-dependshttpware,modern-di-faststreammodern-di,modern-di-grpcdb-retry,modern-di-aiohttp,modern-di-fastapi,modern-di-starlette,modern-di-typer,semvertagpytestwith no floor resolves to 2.0.0, which fails to build, and the job dies before it tests anything. Same forasgi-lifespan(0.0.1) and friends. Splitting the install in two sidesteps this entirely and keeps the job about what it is actually for: the runtime contract we publish.Adding floors across every dev group would also work, but it is 24 repos of churn to fix a problem the two-step install does not have.
Caveats worth deciding on
faststream-outbox,faststream-redis-timers,faststream-concurrent-aiokafka,db-retry) need their Postgres/Redis/redpanda service block on this job too, or the job runs unit tests only. Unit-only still catches the import-time failures, which is what items 1 and 3 were.faststream-outboxis the obvious candidate: it is the repo where two of the three instances were found, and item 3 means it will fail immediately with something real.Related