Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .socraticodecontextartifacts.json
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@
{
"name": "dependencies-reference",
"path": "./docs/DEPENDENCIES.md",
"description": "Where co-core comes from and its pin (#183). co-core, co-core-aio and co-core-sync resolve from cannobserv's private co-pypi Artifact Registry index, declared explicit with a [tool.uv.sources] entry each, the lock carrying every file's sha256; it replaced the GCS wheelhouse and scripts/sync_wheelhouse.py. Which commands need the index token (any uv lock, a cold-cache uv sync) and which don't (uv run, uv lock --check, the unit's ExecStart, even offline). The wrapper scripts/co-pypi.sh: one read-only key, REPLICATOR_CO_PYPI_CREDENTIALS, never the writer's GOOGLE_APPLICATION_CREDENTIALS; an allowlisted environment so neither the writer's key path nor the repo .env PATs reach uv; the token never printed. CI's WIF step hands the token to uv sync alone. Moving the pin: edit, wrapped uv lock, `co-pypi.sh wheels OLD NEW DIR`, then scripts/co_core_closure_diff.py OLD NEW --wheels DIR. Then the pin itself (moved from docs/DEPLOYMENT.md): pin the current minor, why the patch floor is load-bearing under extra=\"ignore\", the 0.10 -> 0.13 jump taken on evidence, and the import-closure diff every pin move takes, its verdicts and the two lazy __getattr__ sites it lists as expected (#145). Last, the uv_build build-backend bound (#195): one minor, the host's uv's; ci.yml's setup-uv version: inputs repeat it verbatim; bump when uv sync warns the bound excludes the running uv; the warning is noise (uv still direct-builds, no fetch); required-version rejected because it would refuse the unit's uv run."
"description": "Where co-core comes from and its pin (#183). co-core, co-core-aio and co-core-sync resolve from cannobserv's private co-pypi Artifact Registry index, declared explicit with a [tool.uv.sources] entry each, the lock carrying every file's sha256; it replaced the GCS wheelhouse and scripts/sync_wheelhouse.py. Which commands need the index token (any uv lock, a cold-cache uv sync) and which don't (uv run, uv lock --check, the unit's ExecStart, even offline). The wrapper scripts/co-pypi.sh is deploying-releases' shared copy, never edited (#201): one read-only key, CO_PYPI_KEY else /etc/replicator/co-pypi-reader.json, never the writer's GOOGLE_APPLICATION_CREDENTIALS; an allowlisted environment so neither the writer's key path nor the repo .env PATs reach uv; the token never printed; run from the lock's non-default co-pypi group. CI's WIF step hands the token to uv sync alone. Moving the pin: edit, wrapped uv lock, scripts/co_pypi_wheels.py under the wrapper (uv has no download command), then scripts/co_core_closure_diff.py OLD NEW --wheels DIR. Then the pin itself (moved from docs/DEPLOYMENT.md): pin the current minor, why the patch floor is load-bearing under extra=\"ignore\", the 0.10 -> 0.13 jump taken on evidence, and the import-closure diff every pin move takes, its verdicts and the two lazy __getattr__ sites it lists as expected (#145). Last, the uv_build build-backend bound (#195): one minor, the host's uv's; ci.yml's setup-uv version: inputs repeat it verbatim; bump when uv sync warns the bound excludes the running uv; the warning is noise (uv still direct-builds, no fetch); required-version rejected because it would refuse the unit's uv run."
},
{
"name": "memory-protection-reference",
Expand Down Expand Up @@ -148,7 +148,7 @@
{
"name": "co-pypi-wrapper",
"path": "./scripts/co_pypi_token.py",
"description": "The co-pypi index wrapper (#183), run as `bash scripts/co-pypi.sh uv …` or `… wheels OLD NEW DIR`. Mints an Artifact Registry access token from the read-only key REPLICATOR_CO_PYPI_CREDENTIALS names — never GOOGLE_APPLICATION_CREDENTIALS, which on this host is the worker's writer — and execs uv with UV_INDEX_CO_PYPI_USERNAME/PASSWORD from an allowlisted environment (co-pypi.sh applies the same list to the outer uv run), so the writer's key path and the repo .env PATs never reach uv. The wheels verb fetches co-core wheels for scripts/co_core_closure_diff.py, sha256-checked against the index, following same-host redirects only. The token is never printed, logged, put in argv or written. CI never runs it: its WIF step hands the token to uv sync."
"description": "The co-pypi index wrapper (#183), deploying-releases' shared marked copy since #201 (install.sh --co-pypi; never edited, tests/test_co_pypi.py checks the marker), run as `bash scripts/co-pypi.sh uv …`. Mints an Artifact Registry access token from the read-only key CO_PYPI_KEY names, else /etc/<[project].name>/co-pypi-reader.json (here /etc/replicator/co-pypi-reader.json) — never GOOGLE_APPLICATION_CREDENTIALS, which on this host is the worker's writer — and execs uv with UV_INDEX_CO_PYPI_USERNAME/PASSWORD from an allowlisted environment (co-pypi.sh applies the same list to the outer uv run, which runs the lock's co-pypi group --frozen), so the writer's key path and the repo .env PATs never reach uv. The token is never printed, logged, put in argv or written. Wheel files for the closure diff come from scripts/co_pypi_wheels.py, run under it. CI never runs it: its WIF step hands the token to uv sync."
},
{
"name": "redis-floor-guard",
Expand All @@ -168,7 +168,7 @@
{
"name": "release-deploy-hooks",
"path": "./scripts/deploy_hooks.sh",
"description": "This repo's half of the release deploy (#151), sourced by scripts/deploy.sh, the deploying-releases core (gregoryfoster/skills#372) installed as a marked copy and never edited. CI=github anonymous with the four ci.yml job display names, UV_WRAPPER=scripts/co-pypi.sh (this repo's own until #201) with REPLICATOR_CO_PYPI_CREDENTIALS defaulted to /etc/replicator/co-pypi-reader.json, src and scripts compiled, src.worker.main and scripts.inflight import-probed, five HOST_CONFIGS compared never installed, KEEP=0 for disk. post_build_check runs the release's own #52 writer guard (src/worker/checkout.py) so a release that would refuse its writers never goes live; verify proves replicator.service with verify_started_on, then its \"worker ready\" journal line naming the build. Held by tests/deploy/. Design: docs/plans/2026-11-14-releases-design.md."
"description": "This repo's half of the release deploy (#151), sourced by scripts/deploy.sh, the deploying-releases core (gregoryfoster/skills#372) installed as a marked copy and never edited. CI=github anonymous with the four ci.yml job display names, UV_WRAPPER=scripts/co-pypi.sh (the shared wrapper since #201, whose key defaults to /etc/replicator/co-pypi-reader.json, so no key variable is exported), src and scripts compiled, src.worker.main and scripts.inflight import-probed, five HOST_CONFIGS compared never installed, KEEP=0 for disk. post_build_check runs the release's own #52 writer guard (src/worker/checkout.py) so a release that would refuse its writers never goes live; verify proves replicator.service with verify_started_on, then its \"worker ready\" journal line naming the build. Held by tests/deploy/. Design: docs/plans/2026-11-14-releases-design.md."
}
]
}
3 changes: 1 addition & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ reads `XAUTOCLAIM`'s three-element reply, added in Redis server 7.0.

```bash
# co-core, co-core-aio and co-core-sync come from the private co-pypi index, whose token
# the wrapper mints from REPLICATOR_CO_PYPI_CREDENTIALS (docs/DEPENDENCIES.md).
# the shared wrapper mints from /etc/replicator/co-pypi-reader.json (docs/DEPENDENCIES.md).
. scripts/load_env.sh
bash scripts/co-pypi.sh uv sync
uv run pre-commit install
Expand Down Expand Up @@ -118,7 +118,6 @@ rather than deleting bytes a consumer was promised.
| Variable | Set on this VM | Purpose |
|---|---|---|
| `GOOGLE_APPLICATION_CREDENTIALS` | `/etc/replicator/co-gcs-replicator-writer.json` | The worker's ADC — the writer SA (`co-gcs-replicator-writer@co-gcs`, #114), never the co-pypi reader |
| `REPLICATOR_CO_PYPI_CREDENTIALS` | `/etc/replicator/co-pypi-reader.json` | Read-only key for the co-pypi index, read by `scripts/co-pypi.sh` in the deploy shell, never by the worker |
| `REPLICATOR_REDIS_URL` | `redis://replicator:<password>@broker:6379/0` | Change-bus client URL — the `replicator` ACL user on `co-broker`, over the tailnet |
| `REPLICATOR_BLOB_BACKEND` | `gcs` | Temp blobs live in an object store — **not** the `local` default |
| `REPLICATOR_BLOB_BUCKET` | `co-gcs-blobs` | The temp-blob bucket that backend writes into |
Expand Down
2 changes: 1 addition & 1 deletion docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ src/worker/checkout.py — Is this checkout main's code? Asked before a write id
src/core/logging.py — build_json_formatter() + ColorMessageFilter + configure_logging() + get_logger()
src/core/log_config.json — uvicorn --log-config; routes uvicorn's own loggers through that formatter
src/core/config.py — Settings / env access (see Environment Variables)
scripts/ — co-pypi.sh + co_pypi_token.py, co_core_closure_diff.py, check_redis_floor.sh, rcli.sh, load_env.sh, deploy.sh (the deploying-releases core, never edited) + deploy_hooks.sh, notify_failure.sh, browser_exec.sh, seed_fetch.py, seed_capture.py, inflight.py, rehearse_reconnect.sh
scripts/ — co-pypi.sh + co_pypi_token.py (deploying-releases' copies, never edited), co_pypi_wheels.py, co_core_closure_diff.py, check_redis_floor.sh, rcli.sh, load_env.sh, deploy.sh (the deploying-releases core, never edited) + deploy_hooks.sh, notify_failure.sh, browser_exec.sh, seed_fetch.py, seed_capture.py, inflight.py, rehearse_reconnect.sh
scripts/rcli.sh — sourced: redis-cli with the URL's credential in REDISCLI_AUTH, never argv (#127)
scripts/load_env.sh — sourced: parses both env files into the shell, never runs them (#170)
scripts/notify_failure.sh — the OnFailure= handler's body: records the incident, dispatches it when configured (#94)
Expand Down
5 changes: 3 additions & 2 deletions docs/COMMANDS.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,11 @@
## Setup

```bash
. scripts/load_env.sh # REPLICATOR_CO_PYPI_CREDENTIALS, the index's read-only key
bash scripts/co-pypi.sh uv sync # co-core comes from the private co-pypi index
bash scripts/co-pypi.sh uv lock # every relock, even a PyPI-only one
bash scripts/co-pypi.sh wheels OLD NEW DIR # every pin move: fetch, then diff the closure
# every pin move: fetch the wheels, then diff the closure
bash scripts/co-pypi.sh uv run --no-project python scripts/co_pypi_wheels.py \
--dest DIR --version OLD --version NEW co-core co-core-aio co-core-sync
uv run --no-project python scripts/co_core_closure_diff.py OLD NEW --wheels DIR
uv run pre-commit install
```
Expand Down
61 changes: 36 additions & 25 deletions docs/DEPENDENCIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,39 +24,43 @@ package: uv asks it for those three only, so neither side can shadow the other.
| any `uv lock`, even a PyPI-only bump; `uv sync` on a cold cache | **required**: a 401 on co-core without it |

So every release build, a dev venv's sync and every relock go through the wrapper.
`scripts/deploy.sh` runs it as `UV_WRAPPER`, and `scripts/deploy_hooks.sh` names the
key, so a deploy needs no `load_env.sh` (#151):
`scripts/deploy.sh` runs it as `UV_WRAPPER` (#151); neither it nor a shell needs an
env file for the key:

```bash
. scripts/load_env.sh # REPLICATOR_CO_PYPI_CREDENTIALS
bash scripts/co-pypi.sh uv sync --frozen # a dev venv
bash scripts/co-pypi.sh uv lock
```

**The wrapper.** `scripts/co-pypi.sh` runs `scripts/co_pypi_token.py`, which mints an
access token from the read-only key `REPLICATOR_CO_PYPI_CREDENTIALS` names
(`co-pypi-reader@`, `/etc/replicator/co-pypi-reader.json`) and `exec`s uv with
`UV_INDEX_CO_PYPI_USERNAME` / `_PASSWORD`. Each rule is tested
(`tests/test_co_pypi_token.py`):

- **One key, no fallback.** Never `GOOGLE_APPLICATION_CREDENTIALS`: on this host
that is the worker's *writer* (#114). Unset, the wrapper exits 2 and names the
variable.
**The wrapper is the cohort's, never edited here** (#201). `scripts/co-pypi.sh` and
`scripts/co_pypi_token.py` are deploying-releases' marked copies, installed with
`install.sh --co-pypi`; a change goes upstream, then a reinstall, and
`tests/test_co_pypi.py` fails on a hand edit. The minter mints an access token from
the read-only `co-pypi-reader@` key and `exec`s uv with `UV_INDEX_CO_PYPI_USERNAME` /
`_PASSWORD`. Its rules are tested in the skill's own suite:

- **One key, never the writer's.** `CO_PYPI_KEY` when set, else
`/etc/<[project].name>/co-pypi-reader.json` — here `/etc/replicator/co-pypi-reader.json`,
so nothing sets it. Never `GOOGLE_APPLICATION_CREDENTIALS`: on this host that is the
worker's *writer* (#114).
- **An allowlisted environment.** The deploy shell carries the writer's key path
and the repo `.env`'s org-wide PATs, and uv (with the `uv_build` backend it runs)
starts with neither. The shell script and the Python hold the same list; a
variable uv turns out to need goes on both.
starts with neither.
- **The token is never printed, logged, put in argv or written.** Anything but
`uv` or `wheels` is refused before a mint.
`uv` is refused before a mint.
- **It runs from the lock's `co-pypi` group** (`google-auth[requests]`), `--frozen`,
so nothing in the process holding the key resolves at run time. The group is not a
default one: neither a release nor CI syncs it. ruff excludes the minter, whose
marker a `--fix` would break.

**CI never runs it.** Each job's read-only WIF step mints the token
(`token_format: access_token`, id `co_pypi`) and writes no credentials file; the
`uv sync` step alone receives it, masked (`tests/ci/test_workflow_pins.py`).

**Limits.** The token reaches what the wrapped uv spawns (the build backend),
bounded by the reader's role and the token's hour. The `wheels` verb sends it to
`urllib`, for the index host alone and across same-host redirects only: the index
answers a file with a 307.
**Limits.** The token reaches what the wrapped uv spawns (the build backend, and
the wheel fetcher below), bounded by the reader's role and the token's hour. The
fetcher sends it to `urllib`, for the index host alone and across same-host
redirects only: the index answers a file with a 307.

## Moving the pin

Expand All @@ -66,19 +70,26 @@ answers a file with a 307.
3. Diff the import closure ([below](#the-co-core-pin-and-why-the-patch-floor-is-load-bearing)):

```bash
bash scripts/co-pypi.sh wheels OLD NEW /tmp/co-core-wheels
bash scripts/co-pypi.sh uv run --no-project python scripts/co_pypi_wheels.py \
--dest /tmp/co-core-wheels --version OLD --version NEW co-core co-core-aio co-core-sync
uv run --no-project python scripts/co_core_closure_diff.py OLD NEW --wheels /tmp/co-core-wheels
```

4. Paste the report into the pin comment and raise the floor by the rules below.

`wheels` checks each file against the index's sha256. The closure diff's
real-wheel test reads `REPLICATOR_TEST_CLOSURE_WHEELS`, a directory `wheels 0.36.0
0.37.0` filled, and skips without it, in CI too.
**The fetcher** (#201) exists because uv has no download command and leaves no `.whl`
behind [measured, uv 0.13.5]. It runs under the wrapper and reads the pair the wrapper
hands uv, so it holds no key of its own; packages and versions are arguments, the
index is `pyproject.toml`'s `co-pypi`. Each file is checked against the index's sha256
and renamed into place; a missing version or a mismatch exits 1 naming the file. It is
standard library only and names nothing of this repo's, built to be proposed to
deploying-releases beside the wrapper. The closure diff's real-wheel test reads
`REPLICATOR_TEST_CLOSURE_WHEELS`, a directory it filled at 0.36.0 and 0.37.0, and
skips without it, in CI too.

**Ask the index, not a cache.** The #77 review reported 0.13.1 "not mirrored" from a
local `ls` while the index already carried it. Anything local is a cache; `wheels`
or `uv lock` asks the index.
local `ls` while the index already carried it. Anything local is a cache; the
fetcher or `uv lock` asks the index.

## The co-core pin, and why the patch floor is load-bearing

Expand Down
Loading
Loading