Skip to content

[Epic] Parking lot (burn-down done 2026-11-16): live stability, sibling waits, operational conformance #154

Open

Description

@gregoryfoster

The whole open backlog of CannObserv/replicator, burned down one child at a time: live stability first, then the work sibling repos wait on, then operational conformance. Done when the host's disk is under a canonical GC, the capture consumer is live, the co-core pin tracks the current minor, and Dependabot proposes action bumps. Every other open issue is then either parked with a trigger or waits on a named external with an observable unblock condition.

Order

The operator's rule, verbatim: "Prioritize bug fixes, issues other repos are waiting on, operational conformance and stability. Older issues may need to be re-evaluated for continued relevance."

Tiebreaks the coordinator added, numbered so the operator can challenge one:

  1. A hazard measured live on the host ranks as a bug. Host disk GC: adopt provisioner#61's canonical GC + headroom sensor on co-replicator (root hit 97%) #148 is the only one: root at 81% and rising (see Measured state).
  2. Unblocked before blocked. A blocked child's interim (a hand step) can still run ahead of its turn.
  3. A multi-minor co-core pin move lands alone, in an isolated venv, before any behaviour that needs the new version (the Converge storage on the cluster content-addressed model #114 rule, kept from [Epic] content.fetch-policy producer moves to Archiver #135).
  4. A shared runtime is built once, by the child with the most sibling waits. Viewport capture: consume the capture command, render headless, store, publish the stored-capture fact #138 builds the browser runtime and Headless-browser fetch path for JS-rendered pages (re-sited from watcher#3) #63 reuses it.
  5. Context freshness: the child whose surface just changed goes next.

Overrides:

Boundary decisions

Exit criterion

#148, #153, #138, #76, and the bugs #159 and #156, are closed and verified live (bugs added 2026-10-26). Every other open issue is parked with a trigger, or waits on a named external with an observable unblock condition.

Verify recipe

Tier 1 — live stability

Tier 2 — sibling waits

Tier 3 — operational conformance

Waits

Ref Who waits On what Asked Unblock condition
gregoryfoster/skills#372, gregoryfoster/skills#345 #151 the release-deploy core and the deploy_command knob on gregoryfoster/skills#469, 2026-10-23 met 2026-11-14: Part A shipped (f728abd5) and vendored at 55ede55f (9575b27); #151 shipped on it. gregoryfoster/skills#372 stays open for its own close-out
CannObserv/cannobserv#554 #169 (the uv half, parked) an index Dependabot's uv updater can resolve on the issue the index is served and declared as a registries: entry
CannObserv/broker#116 us (verification) Archiver's half: broker reads the group again after the first capture command. Replicator's half read back clean on 2026-10-27 2026-10-26, on the issue broker's second read-back, after CannObserv/archiver#380's first command
CannObserv/archiver#380 us (verification); them for the rest Archiver issuing content.capture go-ahead posted 2026-10-26 the first stored a capture and published capture_stored in our journal
CannObserv/archiver#411 them #138 (met) and their own design — —
CannObserv/archiver#379 them #63 — met 2026-11-06: deployed and announced (issuecomment-6677665321)
CannObserv/watcher#414 them #63's support announcement (co-core's MUST: no issuer sets render before it) filed 2026-10-31 by the cannobserv coordinator met 2026-11-06: announced (issuecomment-6677665074); the MUST is discharged
CannObserv/watcher#412 them #161 (met); hardening, their call filed 2026-10-31 (operator), after CannObserv/watcher#402 (the #156 port, closed 2026-10-28) —
CannObserv/cannobserv#602 #63 co-core's render field on content.fetch filed 2026-10-28 (operator) met 2026-11-01: v0.40.1 released (tag 0106cf45) and on the index [measured]
gregoryfoster/skills#536 a future adoption issue here the reviewing-code* variants folded into one stack-detecting reviewing-code (gregoryfoster/skills#537's D6) raised by #174's implementer, 2026-11-01 merged upstream and vendored: then file the adoption issue (operator), covering the skills/ pair and docs/SKILLS.md:58
CannObserv/broker#85 (the Valkey 7.2 window) #178 broker grants `+client setinfotoreplicator` in the window, then posts its notice answered 2026-11-03 on #178
CannObserv/cannobserv#635 them ContentFetchCommandEmit refusing any header but user-agent beside render="browser" (#63's call 1A) filed 2026-11-06 (operator) — (meanwhile Replicator refuses it as invalid_request_options)
gregoryfoster/skills#551 #164 the skill's pin move to disk GC VERSION 4 (only the newest staged VS Code build kept); built and held on wave8/551-version-4 since 2026-11-03, its held question answered "no re-stage" the same day nudged 2026-11-06 (operator, issuecomment-6697525603) met 2026-11-08: shipped upstream 2026-11-07 (b51f3c24), vendored in edb63bc, _pin.sh reads DISK_GC_VERSION=4 [measured]
CannObserv/cannobserv#640 #163 B (its argument) co-core's FetchPolicyState docstring stops naming a still-publishing Watcher (stale since CannObserv/broker#95) filed 2026-11-06 (operator) met 2026-11-10: v0.42.0 (ce0277f5) rewords it to name both sources of an unordered frame; in our floor since #184 (0.45.0 now) [read 2026-11-13]. B's other conditions are unchanged (Parked)
CannObserv/watcher#420 #185 (the Reader row) co-gcs-blob-reader@ loses its objectViewer binding on co-gcs-blobs and is deleted filed by Watcher 2026-11-09 met 2026-11-15: closed 19:56Z; both bindings removed, the account deleted, read-back on the issue; undo until ~2026-12-15 (uniqueId on the issue)

Parked

Each row is brief-grade: a trigger may reach an implementer with no hand-off. When any upstream release lands (a co-core minor, a skills release), re-read every row, not only the one whose trigger fired. Rows re-read 2026-11-16 [measured unless marked].

Ref Trigger (observable) When it fires Hazards Evidence Last re-read
#203 Drift check for installed units and host configs gregoryfoster/skills#641 ships its drift check, or splits the host half into its own issue becomes the adoption: install the module with its marker (adopt unmodified), set the per-repo inputs (RUNTIME_PATHS, the monitor id, the grace) in scripts/deploy_hooks.sh or its sibling, the units from the template; the operator creates the co-status monitor and check-in key, the tailnet rule to tag:status:9000, then ENABLE_NEW GitHub's unauthenticated 60/h per address is shared with the deploy's CI gate; a host-half verdict must page only for units (production not running what was deployed), not for a host config waiting on a hand cp the scope ask issuecomment-6906658975 (2026-11-16), no reply yet; today the core installs differing units at each deploy and notes differing HOST_CONFIGS 2026-11-16
#169 Dependabot: add the uv ecosystem all three: CannObserv/cannobserv#615 decides the PyPI name registration (or the operator accepts the PyPI-first limit); an org owner admits this repository to the private registry (cannobserv's docs/DEPENDABOT.md); #183's index move (met, bc4c5ec) add the uv ecosystem to .github/dependabot.yml with a registries: entry for co-pypi, keep the co-core current-minor caps as ignores, extend the tests/ci guards dependency confusion is the reason CannObserv/cannobserv#615 exists: unregistered names on PyPI; the updater must hold only the reader identity, never the writer's; a proposal crossing the pin's minor breaks the "pin the current minor" rule CannObserv/cannobserv#615 open, its owner runbook paused 2026-11-14 (nothing uploaded); dependabot.yml's reason names this row 2026-11-16
#163 B Fetch policy: retire the unordered path all three: co-core decodes FetchPolicyState.generation as required (still Generation | None at 0.48.2, changes.py:1974), or the operator chooses Replicator's own refusal; Archiver at co-core ≥ 0.41 (still >=0.38,<0.39); retention clear of Watcher's frames (only broker's XRANGE can say) remove the occurred_at branch, the ratchet, _log_out_of_order's None branch, POLITENESS :72, and the Watcher scenarios and docstrings (test_policy.py:540-542, test_policy_reader.py:578-657) a frame without a generation after the change must be refused loudly, never applied; the replay at boot reads the whole stream, so retention matters _log_out_of_order's None branch: 0 firings since 2026-10-22; trigger A shipped 2026-11-06 (7460a5c) 2026-11-16
#64 WACZ/WARC capture research CannObserv/archiver#408's design settles the producer and the trigger rescope #64 to the design's Replicator half, or close it for the issue the design files; run the charter's three tests (docs/contracts/replicator-boundaries.md) on the capability first a new command stream means a design doc, a contract doc, broker grants (broker's) and a stream-kind entry; a WACZ is a browser workload, so it inherits #138's renderer confinements and memory bounds CannObserv/archiver#408 open, 0 comments, last update 2026-10-20 2026-11-16

Sibling replies owed

Verification debt

What is unverified Why it was skipped How to discharge it Discharged by
A real capture end to end (capture_stored on content.artifacts) Replicator can't XADD to content.capture; Archiver hasn't issued yet the journal shows stored a capture and published capture_stored, and XPENDING content.capture replicator.capture returns to 0 discharged 2026-11-11: 01M75H603VX307CE6YRMTJV3NR, stored a capture and published capture_stored at 14:46:26Z, Archiver applied it 6.4 ms earlier (their record on #138, now closed); a second at 19:11:09Z; the worker's own pending count across its groups reads 0 (scripts/inflight.py, 23:2xZ) [measured]
replicator.capture read back, and ACL LOG clean XINFO GROUPS is denied to replicator broker posts its acladmin read-back discharged 2026-10-27: CannObserv/broker#116 (issuecomment-6450135332): the group exists at 0-0, created before any entry, 4 replicator readers blocked, no replicator entry in ACL LOG. consumers 0 is expected until the first delivery
The manifest's re-embed after #151 another session's SocratiCode server held the index and context locks the capped update from docs/COMMANDS.md, run from the main checkout, then codebase_context_search shows the new descriptions discharged 2026-11-15: the #201 implementer's server's watcher ran the incremental update; codebase_context_search returns #201's new DEPENDENCIES.md and COMMANDS.md text [measured by the coordinator]
99-co-replicator-memory.conf's installed copy predates #112's comment edit; every deploy reports it values identical, so not urgent; a root write the operator's sudo cp /srv/replicator/live/deploy/99-co-replicator-memory.conf /etc/sysctl.d/, then the next deploy reports no HOST_CONFIGS drift discharged 2026-11-15: installed by the #204 implementer with the operator's authorization at 00:30:30Z; its sha256 equals the repo's (6b42b277…), and the live values match [measured]
The first pr-mode context-cadence run (#208) merged after the render; the schedule hadn't fired yet a PR from context/weekly-measurement titled chore: weekly context measurement, no direct bot commit on main; merged with a merge commit (close and reopen for CI first). Expected 2026-11-17 ~21:00–22:30Z, else 2026-11-24 discharged 2026-11-18: the 2026-11-17T23:53Z run opened PR #210 (only the three .skills/context-* files), main did not move; its own pull_request run (actor github-actions[bot]) failed with 0 jobs; the coordinator closed and reopened it (operator's request), CI green by job on dba8b20, merged with a merge commit 796186f at 05:53:41Z [measured]
The refresh hook's first pr-mode run (#208) today's ran at 08:43Z, direct .git/skills-update.log shows the pr path at the first session on 2026-11-18; the first real PR from skills/submodule-refresh comes with the next upstream change; main gets no hook commit discharged 2026-11-18: the 13:12Z run took the pr path (log: restored both submodules to the recorded pointer, proposed d4b291c on origin/main) and opened PR #212, only the gregoryfoster-skills pointer, CI green by job; main got no hook commit [measured]

Accepted limitations

  • CR 10 of Viewport capture: consume the capture command, render headless, store, publish the stored-capture fact #138: each render's SOCKS listener is unauthenticated, and a local process could reach it. Accepted 2026-10-26.
  • CR 11 of Viewport capture: consume the capture command, render headless, store, publish the stored-capture fact #138: Playwright's Node driver runs unconfined in the worker/ cgroup. A page can't reach it. Accepted 2026-10-26.
  • From Retention sweep can rmdir a shard the store just created, failing the write (transient; flakes test_run_dispatches_to_the_byte_path) #159, local backend only (production uses the object store), accepted 2026-10-26: a forward clock step larger than the grace can lose the one retry, and is then retried at the reclaim; emptied shards linger for one grace per level; the INFO line "a sweep removed the shard mid-store" is the only signal of a lost race.
  • From Retention sweep can rmdir a shard the store just created, failing the write (transient; flakes test_run_dispatches_to_the_byte_path) #159: REPLICATOR_BLOB_TEMP_GRACE_SECONDS is unbounded, and at 0 the shard grace and the one-retry argument both fail. Not filed: it needs a misconfiguration on a local-backend host. Re-entry trigger: a host moving to the local backend, or a boot-time bound proposed for another reason. Proposed 2026-10-26.
  • The co-core upstreaming of Retention sweep can rmdir a shard the store just created, failing the write (transient; flakes test_run_dispatches_to_the_byte_path) #159's retry is a draft on the issue, not filed. Worth it only if another cohort repo sweeps a co-core local tree. Operator's call.
  • From Adopt Dependabot: nothing proposes crossing this repo's dependency caps #76: the browser job has no Assert wheelhouse auth is configured step, so an auth failure on a Dependabot run in that job can't be read against the verdict table. The other three jobs carry it. Re-entry trigger: the next ci.yml edit. Recorded 2026-10-30. Settled 2026-11-10 by Resolve co-core from the co-pypi Artifact Registry index, retiring the GCS wheelhouse (#169 part 1) #183: browser gained the WIF-provider assert.
  • From Adopt Dependabot: nothing proposes crossing this repo's dependency caps #76: context-cadence.yml runs on ubuntu-latest with a tag-pinned actions/checkout@v5. It's generated by install-cadence.sh and excluded from Dependabot, so a fix is an upstream installer change. Not filed: the operator declined, 2026-10-30.
  • From CannObserv/replicator#162: the skills-refresh hook commits and pushes a submodule bump without running the suite, and 4cac2a2 pushed a red main. That's an upstream ask (gregoryfoster/skills: run the repo's tests before pushing). Not filed: the operator declined, 2026-10-30.
  • From CannObserv/replicator#162: install-cadence.sh's rendered workflow warns only on the policy file. The drafted upstream ask: warn on any docs[].over_budget, and list near_budget. Not filed: the operator declined, 2026-10-30.
  • From CannObserv/replicator#171 (CR 4): the drift test checks that documented caps agree, not that every documented SocratiCode launch is capped. A bare node "$SOCRATICODE_DRIVER" index . line passes. The operator declined a guard, 2026-10-30.
  • From CannObserv/replicator#171: two tracked-file readers, tests/test_doc_sensitive_paths.py::read_tracked_files (paths) and tests/_tracked.py (text). Merge them if a third caller appears. Recorded 2026-10-30.
  • From CannObserv/replicator#161, accepted 2026-10-31:
    • a set REPLICATOR_BLOCKED_DESTINATIONS replaces the table and drops all 12 rows (the unwrap still holds);
    • a few globally reachable sub-entries are refused: PCP and TURN anycast in 192.0.0.0/24, the 6a44 relay 192.88.99.2, and all of ORCHIDv2;
    • 2001::/23 is no row;
    • a site-chosen NAT64 prefix is still unchecked, as after CannObserv/replicator#156;
    • the rest of the IANA registry (AS112, AMT, anycast services, DETs, 100:0:0:1::/64, the old ORCHID) is left out, with reasons on the issue.
  • From CannObserv/replicator#172, accepted 2026-11-01: the pin comment's 0.39 entry is condensed under "Previously:", and the verbatim 0.37.0 → 0.39.1 closure report is in git at 37b6eca:pyproject.toml.
  • From CannObserv/replicator#174, accepted 2026-11-01:
    • .skills/import-targets is unset, so the gate's import check skips: import src loads only src/__init__.py, and the suite imports every module. Recorded in docs/SKILLS.md.
    • No .skills/test-workers: on 2 vCPUs the resolver goes serial on its own, and pytest-xdist isn't installed.
    • Step 6, the CI census, is deferred to upstream's prescriptions, because ci.yml overlaps CannObserv/replicator#151 and CannObserv/replicator#169.
    • Skipped: a committed red (the doctor holds while the old path is named; shown in-session), and the run-everything gate policy (nothing failed).
  • From CannObserv/replicator#173, accepted 2026-11-02:
    • a peer whose TLS breaks on every attempt now retries indefinitely (transient is exempt from the ceiling), as ConnectError does; before, it closed handler_error after about 4 minutes. The outcome reference's "network error … retry indefinitely" row already covers it;
    • only _fetch is covered: capture's TLS errors go through Playwright;
    • no real-TLS test in the suite (the guard refuses loopback); the escape was measured once by a scratch script.
  • From CannObserv/replicator#175, accepted 2026-11-03: production's refused-dead-letter alternation holds with about 59 s of margin at the defaults, but nothing refreshes a claim mid-handler, so a handler that takes about 59 s or more to reach a permanent failure under a sustained cap, or a REPLICATOR_CLAIM_MIN_IDLE_MS set near the backoff base, reaches the restart [inferred from code, not measured]. Stated in docs/CONVENTIONS.md. The OOM test now takes about 5 s.
  • From CannObserv/replicator#177, accepted 2026-11-03: until CannObserv/replicator#63, a render="browser" command is decoded and still fetched over HTTP. co-core's MUST (no issuer sets render before Replicator's announcement) guards it.
  • From CannObserv/replicator#63, accepted 2026-11-05 (design approved; listed on the issue):
    • inherited from CannObserv/replicator#138: CR 10 (the per-render listener), CR 11 (the unconfined Node driver, which now relays the document, bounded by the in-page measure and _raise_for_size), the CannObserv/replicator#156 and #161 guard limits, and unpaced subresources;
    • the document is the DOM at load; later script content is absent until co-core has a wait field; the serialized DOM keeps the origin's <meta charset>;
    • unclassified (handler_error at the ceiling): an unbidden 304 (ERR_ABORTED), a navigation during serialization, an unmet load, and the deadline under browser (call 4);
    • a browser fetch can wait behind a capture inside its own 120 s deadline (call 5);
    • a parked browser command on a host with the flag off logs one WARNING per reclaim (CR 6, kept as the only signal);
    • skipped: no live browser command yet, and no single end-to-end dev-worker run; the lock test proves itself by failing without the lock.
  • From CannObserv/replicator#179, accepted 2026-11-06:
    • option (b) departs from co-core 0.41.0's BlobStore docstring (a wrapping subclass wraps the twins); before the first twin call, move the retry onto both twins and drop the plain-verb overrides; the guard goes red on a twin reference or override;
    • the guard sees literal names only (attribute or exact string) and scans src/, not scripts/;
    • pre-existing, unfiled: build_system.requires = ["uv-build>=0.12.0,<0.13"] warns under the host's uv 0.13.5.
  • From CannObserv/replicator#180, accepted 2026-11-07: no hang reproduced (the guard plus a green run is the evidence); the wheelhouse sync and uv sync are bounded only by their job (30 min in test and browser); an apt outage is a red run at 15 min, re-run by a person.
  • From CannObserv/replicator#182, accepted 2026-11-08:
    • /etc/replicator/.env destinations stay unscrubbed (REPLICATOR_BLOB_DIR, REPLICATOR_LOG_LEVEL, REPLICATOR_WHEELHOUSE_CREDENTIALS; REPLICATOR_REDIS_URL, which the recipe unsets; REPLICATOR_NOTIFY_* ×6, which tests/test_notify_failure.py scrubs itself): they choose no code path, and no test reads them; corrected 2026-11-10: tests/test_inflight.py:113 does read REPLICATOR_REDIS_URL, and prints it on failure (CannObserv/replicator#187);
    • the guard catches bool settings only, so a future path-choosing Literal (as REPLICATOR_BLOB_BACKEND is) is added to PRODUCTION_ENV by hand;
    • a Settings() validation failure in a loaded shell echoes a truncated copy of the env input into pytest output;
    • skipped: -m browser, and a re-check of the 2026-11-03 occurrence (2 failed, names lost).
  • From CannObserv/replicator#164, accepted 2026-11-09: the disk-gc code change (+92/−8) was taken on the pinned digests and the upstream release, not reviewed here; the first timer run of VERSION 4 wasn't observed (the hand start stands in); the GC lists the SocratiCode plugin cache at 1.14.0 as installed while the npx pin is 1.16.1 (outside the child).
  • CannObserv/replicator#186 (2026-11-10): the cold-start half rests on a scratch 3 s delayed-launch run, not a cold host; posix_fadvise eviction of the shell and driver didn't slow run 1. The test checks the sandbox's settled state, not how fast seccomp engages (6–55 ms measured). A process seen only before its argv reads --type=renderer is never checked. The 429 test no longer exercises the real clock; test_pacing.py and test_handler_pacing.py cover the arithmetic on fake clocks.
  • From CannObserv/replicator#183, accepted 2026-11-10: every uv lock, and a uv sync from a cold cache, needs the wrapper (uv run, uv lock --check and a warm sync don't); the token reaches the uv_build backend uv spawns, and the wheels command's urllib (index host only); a forgotten deploy sync runs the old venv, as before; #136's real-wheel closure check skips unless REPLICATOR_TEST_CLOSURE_WHEELS names a fetched directory (CI 22 → 23 skips; locally the second skip); the CI bounds are unchanged on 2 runs of the new shape; no CI run with a cold uv cache.
  • From CannObserv/replicator#187, accepted 2026-11-10: the credential scrub covers Settings fields only (REPLICATOR_NOTIFY_*, REPLICATOR_CO_PYPI_CREDENTIALS and the repo .env PATs still reach a loaded test's environment; their own tests filter them); detection is by field name; the scrub is function-scoped (no import-, module- or session-scope Settings today); the NOGROUP fail-open was shown by a unit test, not a scratch redis; -m integration not run locally.
  • From CannObserv/replicator#189, accepted 2026-11-12: nothing keeps the lifecycle rule, the IAM condition and the temp constant in step except the boot line and the provisioning record (docs/INFRASTRUCTURE.md); 7-day soft delete bills each reaped temp capture up to 7 more days; the two pre-split captures stay in co-gcs-blobs and co-gcs-replicator, metadata only; a capture and a fetched blob with the same digest are two objects. Archiver adopted a kept digest across kinds until CannObserv/archiver#499 (live 03:26:19Z); the first post-split capture hit exactly that case and got its own copy.
  • From CannObserv/replicator#188, accepted 2026-11-13: fetch policy poll cycle recovered is unit-tested only; its first live read-back is the next broker-unreachable window (Watching). after_failures can exceed the cycle WARNINGs before it when a run includes stuck-cursor passes (ERROR-logged), unreachable through today's driver. No -m integration run (test_policy_integration.py untouched).
  • From CannObserv/replicator#201, accepted 2026-11-15: -m integration not run (nothing under src/ changed); the retired test_co_pypi_token.py's two real-loader cases (a mint from the named key with a writer key in ADC; a {} key through the real loader) are covered neither here nor upstream, offered on gregoryfoster/skills#652; tests/test_dependency_index.py overlaps test_co_pypi.py's index and lock checks, left alone; the ship's first deploy runs uv run --frozen --only-group co-pypi against the main checkout's dev venv (the wrapper's --project is its own parent), a no-op there.
  • From CannObserv/replicator#202, accepted 2026-11-15: RestrictNamespaces= stays off (it breaks Chromium's user-namespace sandbox, measured: No usable sandbox!; now pinned out by a test) and ProtectControlGroups= stays off (Delegate= writes cgroupfs, CannObserv/observo#696); scripts/rehearse_reconnect.sh's scratch unit stays User=exedev (it runs the dev checkout); the Playwright driver keeps the worker's environment (CR 11 of CannObserv/replicator#138); a session started before a group change keeps its boot-time groups (exe-init starts the VS Code server), bridged with sg replicator -c 'bash -c "…"' until a reboot; a Playwright bump reinstalls the shell as replicator from the live release's venv (DEPLOYMENT.md, The service user).
  • From CannObserv/replicator#206, accepted 2026-11-16: -m integration, -m browser and -m gcs ran locally on cae0dce, before CR 1 (096afd3, a pyproject.toml comment only); CI re-ran browser and gcs on it.
  • From CannObserv/replicator#185, accepted 2026-11-16: docs/INFRASTRUCTURE.md is 10011 tokens exact against its 10000 budget (10101 before; the offline estimate 9925 reads under), left to a curation pass (CR 3, held); the Readers row is the bucket policy as read 2026-11-16, and nothing re-checks it; the deselected marks weren't run locally (docs and a docstring only).

Standing checks

  • df -h / at every check-in. The GC and its sensor own the disk since CannObserv/replicator#148: the monitor co-replicator-disk-headroom pages at 85% and clears at 80%. No more hand reclaims; a page is read with the skill's verifying.md. Last read 2026-11-10T21:1xZ: see Measured state. Before each patching run (gated at 85%), re-read it, and GC VERSION 4 is in (CannObserv/replicator#164).
  • The journal for stored a capture: met 2026-11-11 (first production capture; Verification debt row discharged). Broker took part in the go-live live, per CannObserv/replicator#138's record [stated].
  • Before any pin-move brief, and at each idle check: list the co-pypi index with no version bound ($SCRATCH/idx.py as co-pypi-reader@, or a lock probe at <1). Added 2026-11-13: the CannObserv/replicator#184 refreshes didn't, and missed 0.43.0 and 0.44.0.
  • Weekly since 2026-11-06 (operator); next due about 2026-11-21 (root 30 GiB since CannObserv/replicator#204: 84.0% of 29.42G usable is about 24.7G, so ~9.5G of headroom at 51.8%, measured 2026-11-15T04:1xZ): the patching-hosts run (knob posture scheduled, cadence weekly). The last full run was CannObserv/replicator#198 on 2026-11-14 (92 packages, 32 s down); before it, CannObserv/replicator#131 on 2026-10-06. The dry run needs --refresh-into too (gregoryfoster/skills#629), and the gate's real ceiling is 84.0% because the sensor rounds up. Best after the 05:15Z GC, which leaves the most headroom. Read the host memory before proposing it. Since CannObserv/replicator#148: the knob's quiet 05:15-06:15 covers the GC's run, and the 85% line gates each step (gregoryfoster/skills#390). The run also covers the pinned browser's system libraries, which ldd checks.
  • Idle check weekdays at 04:03 PT (operator, 2026-11-09; was every 12 h, rescheduled 2026-11-20): a recurring session cron 3 12 * * 1-5 (UTC; 3 11 once PDT starts in March). The scheduler adds up to 30 min to a recurring job, so it fires by about 04:33 PT. It's read-only, a 5–10 line report, and re-creates itself at each run, because a recurring job expires after 7 days. A session restart drops it: re-create it from the saved prompt. Its issue query reads above Last issue seen as well as updated-since: the 2026-11-10 run missed CannObserv/replicator#185, filed before the previous report.

Watching

  • Broker-unreachable windows. One on 2026-10-22, 04:13–04:16Z: 8 failed cycles, then recovered with no restart. One on 2026-11-11, 00:15:42–00:16:04Z, the co-broker reboot (CannObserv/broker#141): recovered, no restart; it surfaced CannObserv/replicator#188. The next window is the first live read-back of the tail's fetch policy poll cycle recovered (shipped 762fd1a): it should follow the tail's fetch policy poll cycle failed WARNINGs. Threshold: a window that exhausts the boot budget, or recurs daily. Since 2026-10-24.
  • VS Code server builds. The live build is the client's own: 1.140.0 until at least 2026-11-03, 1.145.0 (a717861) on 2026-11-06, then 1.145.2 (c4453f8) after the operator's window restart that evening. The CLI's agent host stages each new release about 6 h apart (1.145.1 at 06:33Z, 1.145.2 at 12:33Z on 11-06), and nothing starts them; GC VERSION 2 removes them daily and they're fetched again. Paged once, 2026-10-28T17:00:24Z (85%, two staged), recovered by a hand-started GC run. The fix, VERSION 4, is installed since 2026-11-09T00:27Z (CannObserv/replicator#164). The first report with a staged build should read kept vscode-servers … (<v>, the newest, newer than the live <v>). The first VERSION 4 'newest kept' line came on 2026-11-11's 05:27Z run (dcd3ad6, 1.145.3, newer than the live 1.145.2), as designed. Since then a third build (6904719) has been staged, about 660M each, and root is at 83% (22:47Z); the next run should drop dcd3ad6. Threshold: root ≥ 85%.
  • CI bounds after CannObserv/replicator#183 (the wheelhouse sync is gone): re-size once about 20 green main runs exist on the new shape; the rule already gives lint 5, test 20, browser 20, gcs 5 (ci.yml's jobs: comment). Not filed. Since 2026-11-10.
  • Upstream from CannObserv/replicator#151: gregoryfoster/skills#649 (the service template starts Python through the live link) and gregoryfoster/skills#650 (hardlink identical files between releases: about one release, ~240M, back). The next vendor bump brings Wave 8 set D (55ede55f..8f2dbfae): no core change, three new tests/deploy unit tests that pass on our units off-host (the 22:52Z note on CannObserv/replicator#151). Since 2026-11-14.
  • gregoryfoster/skills#652 (the co-pypi wheel fetcher, proposed from CannObserv/replicator#201) and gregoryfoster/skills#454 (the closure diff). When gregoryfoster/skills#652 ships, reinstall with install.sh --co-pypi --replace; until then --check calls scripts/co_pypi_wheels.py foreign. Since 2026-11-15.
  • The disk GC skipped 2026-11-15: CannObserv/replicator#202's reboot at 05:33:12Z came before the timer's randomized fire (05:15 + up to 30 min), and the timer is Persistent=false (the shared GC's setting), so no catch-up. Last run 2026-11-14T05:27Z (197.5M); next 2026-11-16T05:18Z, the first after CannObserv/replicator#204's grow. Root 55%, so harmless. A reboot inside 05:15–05:45Z skips that day; the patching knob's quiet window already avoids it. Since 2026-11-15.
  • Docs near their 10000-token budget by the offline estimate (2026-11-16): TESTING 9992, the replicate issuer contract 9988, the outcome reference 9985, CONVENTIONS 9968, INFRASTRUCTURE 9925 (exact 10011, over), STREAMS 9902. The exact count runs higher than the estimate here (INFRASTRUCTURE at 2.52 bytes/token against the repo's 2.65), so others may be over too. A curating-context pass owns it; no gate fails today.
  • pr merge mode since 2026-11-17 (CannObserv/replicator#208): two standing bot PRs, from skills/submodule-refresh (the refresh hook) and context/weekly-measurement (the cadence), each merged by an operator with a merge commit; until merged, the vendored skills don't advance and the weekly row isn't on main. Read their state at each check-in. A rollback target is now a merge commit (docs/DEPLOYMENT.md). Threshold: either PR open more than 7 days, or CI red on it. Since 2026-11-17.
  • gregoryfoster/skills#675 (the cadence template's actions/checkout SHA pin) and gregoryfoster/skills#674 (the knob-flipping branch carries its own re-render), both from CannObserv/replicator#208. When gregoryfoster/skills#675 ships and is vendored, re-run install-cadence.sh from the main checkout, through a PR, and check --check's checkout pin: line. Since 2026-11-17.

Measured state

  • Production runs a release as replicator (CannObserv/replicator#202): /srv/replicator/live → releases/59fb24082f7e (deployed 2026-11-15T23:44:13Z), previous f2a43335c6b0; both units NoNewPrivileges, ProtectSystem=strict, ProtectHome, PrivateTmp; NRestarts 0, pid cwd the release, nothing mapped from /home. capture, browser_fetch and persist are enabled. co-core 0.48.2 ×3. main is at d549a06, clean, at origin/main; live trails it by docs and a docstring (CannObserv/replicator#185, not deployed). Only INFO since the start. Deploys: scripts/deploy.sh (core version 1, KEEP=0), also via shipping-work; the co-pypi wrapper is the shared one (version 1). Measured 2026-11-16T20:4xZ.

  • Fetch policy: the 2026-11-15T23:44Z start's replay read 503 messages, tracked_hosts 10. without a generation never logged since 2026-10-22.

  • Browser: REPLICATOR_BROWSER_FETCH_ENABLED=true. No render="browser" command yet. Captures: 3 stored; the two pre-split (2026-11-11) are in co-gcs-blobs and co-gcs-replicator; since 02:41Z they go to co-gcs-captures (temp/, kept under blobs/), the first 01M76WR77JYHK5QFS31W3JXQ44 at 03:27:48Z.

  • Disk: root 30 GiB since CannObserv/replicator#204: 55% (df -h) at 2026-11-16T15:47Z; two releases at 248M each; sensor alert_at 85, clear_at 80; the GC ran 2026-11-16T05:18Z, exit 0, 166.0M removed, 0 failed.

  • co-core: 0.48.2 ×3 from the co-pypi index (Path C, CannObserv/replicator#183), pin >=0.48.0,<0.49 (CannObserv/replicator#206), locked with URLs and sha256. The index's newest is 0.48.2 ×3 [measured 2026-11-15T23:5xZ]. uv lock --check clean. uv audit --frozen clean, 0 open Dependabot alerts (2026-11-03).

  • CI: four jobs bounded (CannObserv/replicator#180); no wheelhouse sync since CannObserv/replicator#183; uv sync gets a masked WIF access token. main's tip d549a06: all four green. uv: host 0.13.5, CI pinned to >=0.13.0,<0.14 (0.13.8), the uv_build bound's range; no uv sync warns since CannObserv/replicator#195 (d5926ab, run 43512090888).

  • Context, exact: POLITENESS 8106; outcome reference 9985 of 10000, CONVENTIONS 9968, STREAMS 9875, COMMANDS 9840, the issuer contract 9839, ENVIRONMENT 9209, the issuer reference 8465, AGENTS.md 5969 of 6000; after CannObserv/replicator#183, AGENTS.md 5897 and DEPLOYMENT 8014 [stated by the implementer].

  • Host addressing: eth0 10.42.0.42/16; no IPv6 default route.

  • Env loading: scripts/load_env.sh is the single entry point.

  • Tests: PRODUCTION_ENV scrubs every bool setting, guarded (CannObserv/replicator#182); every Settings credential too (CannObserv/replicator#187), so a loaded shell matches a clean one with the live broker URL set. Baseline on d549a06: 2062 passed, 1 skipped (test_co_core_closure_diff.py:428; test_deploy.py:895 runs and passes from this post-boot session, which counts as an exe.dev session), 105 deselected; tests/ci 44; tests/deploy 32 on the host; -m browser 31 [the implementer's, on 7b338ff].

Overlaps

Surface Items
the fetch outcome reference (9985), docs/CONVENTIONS.md (9968), STREAMS (9875), the issuer contract (9839) any new row needs a curating-context pass first
AGENTS.md, docs/DEPLOYMENT.md, the manifest none open; AGENTS.md has 59 tokens of headroom since CannObserv/replicator#202 [stated]
docs/POLITENESS.md, src/worker/policy.py CannObserv/replicator#163 B (parked; A shipped 7460a5c)
pyproject.toml, uv.lock, docs/DEPENDENCIES.md, tests/test_dependency_index.py the next pin move; CannObserv/replicator#169 (parked)
.github/ (ci.yml, dependabot.yml, tests/ci guards; Dependabot action PRs land here too) CannObserv/replicator#169 (parked)
skills-vendor/ submodule (the hook bumps it on its own, untested: 4cac2a2 pushed red; later bumps green) the next bump (Wave 8 set D, re-copy tests/deploy), the gregoryfoster/skills#536 adoption
docs/SKILLS.md (about 9314) the gregoryfoster/skills#536 adoption (:58)
src/core/bus_client.py (build_bus_client) CannObserv/replicator#178 (no change unless broker withholds the grant)
docs/INFRASTRUCTURE.md (the production temp store table) a curation pass: 10011 exact of 10000 after CannObserv/replicator#185 (Watching)
src/worker/policy.py CannObserv/replicator#188, CannObserv/replicator#163 B (parked)
scripts/deploy_hooks.sh, deploy/ units CannObserv/replicator#203 (parked on gregoryfoster/skills#641; its adoption adds units and inputs)

Log

  • 2026-10-24: created. 8 open: 5 in tiers, plus 1 new child (CannObserv/replicator#153); 3 parked (CannObserv/replicator#64, CannObserv/replicator#135 and its CannObserv/replicator#144); 0 closures proposed. Reworks: CannObserv/replicator#63 (selection settled), CannObserv/replicator#76 (cohort shape). CannObserv/provisioner#61 closed today, so CannObserv/replicator#148 is half unblocked. Gate green on 42b728c. Next: CannObserv/replicator#153.
  • 2026-10-24: filed CannObserv/replicator#153 (co-core catch-up; operator-approved) and posted its brief. CannObserv/replicator#148 interim reclaim: 81% → 77% (SocratiCode 1.14.0 _npx and Claude Code 2.1.301; CachedExtensionVSIXs kept, because the live VS Code server holds a handle in it). Replies posted on CannObserv/archiver#377 and CannObserv/cannobserv#495 (operator consent). Next: CannObserv/replicator#153.
  • 2026-10-25: check-in 1: CannObserv/replicator#153 verified, deployed and closed (PR CannObserv/replicator#155, the only new number). The baseline is 1468/1, with the skip identical. My [extract] claim was corrected in place on CannObserv/replicator#153's body and brief. Harvest: the For #138 notes are in CannObserv/replicator#138's brief. Blockers: CannObserv/provisioner#85 closed (CannObserv/replicator#148's line rewritten); gregoryfoster/skills#425, gregoryfoster/skills#469, CannObserv/cannobserv#554, CannObserv/watcher#374 and CannObserv/archiver#366 still open. Found: the vendored mcp-driver.mjs has no update subcommand; filing upstream is pending the operator. Next: CannObserv/replicator#138.
  • 2026-10-25: filed gregoryfoster/skills#483 (operator): mcp-driver.mjs needs an update subcommand, so a capped codebase_update can run outside a session. Next: CannObserv/replicator#138.
  • 2026-10-26: check-in 2: CannObserv/replicator#138 verified, deployed and closed (PRs CannObserv/replicator#157 and CannObserv/replicator#158, 49 commits); capture is live and the debt is listed. New: CannObserv/replicator#156 and CannObserv/replicator#159 (bugs, Tier 1) and CannObserv/replicator#160 (filed, Tier 3; red Dependabot security-update runs since 09:15Z). gregoryfoster/skills#425 closed, so CannObserv/replicator#148 is unblocked once the skill is vendored. CannObserv/archiver#366 is deployed dark; F1 is still open. Corrected in place: CannObserv/replicator#159 isn't live in production (no sweeper under the object store). Harvest notes are on CannObserv/replicator#63, CannObserv/replicator#76, CannObserv/replicator#148 and CannObserv/replicator#156. Next: CannObserv/replicator#159.
  • 2026-10-26: check-in 3: CannObserv/replicator#159 verified, deployed and closed (9b37afe..66321b5, 5 commits, fast-forward); the CI flake is fixed; the baseline is 1691/1, with the skip identical. No new numbers. Harvest: #159's limits and its unbounded-grace note went under Accepted limitations, with a re-entry trigger. Blockers: CannObserv/watcher#374 part 1 shipped (Watcher's producer stopped at 18:55Z), and F1 is still open; CannObserv/provisioner#85 confirmed closed. Pre-flight for CannObserv/replicator#148 found no co-status tenant or ACL rule, so CannObserv/status#67 was filed (operator), with the edge in the same request, the CannObserv/notifier#70 way. Root 80%. Next: CannObserv/replicator#148.
  • 2026-10-27: check-in 4: CannObserv/replicator#148 verified and closed (66321b5..6010a66; the GC and sensor are live, the monitor is enabled, the rehearsal was confirmed). CannObserv/status#67 closed. No new numbers. Since check-in 3, this thread also coordinates CannObserv/replicator#135 (operator, 2026-10-26): the read-back is posted, CannObserv/replicator#144 is unparked, and the Parked rows are reworded. The #148 brief was revised after the Status agreement (a disabled monitor still pages; the hook vendored the skill on its own). Pre-flight for CannObserv/replicator#156 found the same gap in Watcher's copy of the guard; filed after the fix (operator). Watching: the daily 76→81% re-stage (gregoryfoster/skills#504). Baseline 1691/1. Next: CannObserv/replicator#156, then CannObserv/replicator#144.
  • 2026-10-27: check-in 5: CannObserv/replicator#156 verified and closed (6010a66..6efad10, 6 commits; the guard probed live in the service venv). Filed CannObserv/watcher#402 (the port), and CannObserv/replicator#161 and CannObserv/replicator#162 into Tier 3 (operator). CannObserv/broker#116: Replicator's half read back clean, and that debt row is discharged. Corrected: check-in 2's doc counts were estimates; COMMANDS.md is 10096 exact and unchanged since 9b37afe. The CannObserv/replicator#144 pre-flight was re-run and its brief posted: the unordered path stays (CannObserv/cannobserv#579 open), and the manifest needs editing too. Baseline 1735/1. Next: CannObserv/replicator#144.
  • 2026-10-28: check-in 6: CannObserv/replicator#144 verified and closed (6efad10..02aa79f, after the bot's aa4c697; live, replay clean, no F1-as-future wording). That emptied CannObserv/replicator#135, which closed with its summary, CannObserv/watcher#368 notified, and its process log filed as gregoryfoster/skills#513. Filed CannObserv/replicator#163 (two triggers, parked). Corrected in place: the CannObserv/replicator#144 brief's COMMANDS.md pointer, and the coordinator's own claim 'noted on gregoryfoster/skills#483' (nothing was posted; posting waits for the operator). Tier 3 order decided (operator). The CannObserv/replicator#160 pre-flight found virtualenv 21.15.1 moving packaging too, and the Dependabot runs stopped. CannObserv/replicator#63 is still blocked (no render-field issue filed). Baseline 1735/1. Next: CannObserv/replicator#160.
  • 2026-10-28: check-in 7: CannObserv/replicator#160 verified and closed (02aa79f..f3db89b, lock only; uv audit clean; no restart). Corrected in place on CannObserv/replicator#160: the brief's 21.14.x option and the body's 21.14.9 floor. Operator asks done: posted on gregoryfoster/skills#483 (one line reference corrected in place), and filed CannObserv/cannobserv#602 (the render field), which now blocks CannObserv/replicator#63. Disk paged at 17:00Z (85%: a third VS Code build); decided GC now plus a VS Code reload (operator). Pre-flight for CannObserv/replicator#76 found two cohort hazards (setup-uv's prune-cache default on a public repo, ubuntu-latest → 26.04); brief posted. Baseline 1735/1. Next: CannObserv/replicator#76.
  • 2026-10-28: the disk page resolved. The operator rebooted the VM; the worker came back clean on f3db89b, with every cgroup and limit as before. The reboot didn't switch VS Code builds: the client is on 1.140.0, and the staged builds are the agent host's unused downloads (corrected the coordinator's own pre-reboot claim, in memory and here). GC started by hand (operator-authorized): 1.4G removed, 73%, recovered 22:20:25Z. Measurement and a VERSION 3 concern posted on gregoryfoster/skills#504. New: CannObserv/replicator#164 (GC VERSION 3), slotted in Tier 3 and held on the VS Code client's version. Next: CannObserv/replicator#76.
  • 2026-10-28: decided (operator): CannObserv/replicator#164 waits for the operator's VS Code client update, then runs after CannObserv/replicator#76. Next: CannObserv/replicator#76.
  • 2026-10-30: check-in 8: CannObserv/replicator#76 verified and closed (f3db89b..cda2861; Dependabot proposed and landed three action bumps, all green; OIDC minted on its runs). The uv half was re-filed as CannObserv/replicator#169 and parked on CannObserv/cannobserv#554. CannObserv/replicator#165 (new) was closed as met by the hook's b520e7f, its preflight passing (operator). Every child the exit criterion names is now closed and verified; the remaining unblocked children are CannObserv/replicator#162, CannObserv/replicator#161 and CannObserv/replicator#164 (held on the VS Code client). Corrected in place: CannObserv/replicator#162's body (a generated workflow can't carry a per-doc check). Baseline 1756/0. Next: CannObserv/replicator#162.
  • 2026-10-30: check-in 9: CannObserv/replicator#162 verified and closed (cda2861..ec96147, after the hook's red 4cac2a2; CI green by job, exact counts match, suite 1756/0, re-embed done by the watcher). New: CannObserv/replicator#170 and CannObserv/replicator#171, both upstream follow-ups. CannObserv/replicator#171's steps had already landed, so it was re-scoped to its drift test. Decided (operator): CannObserv/replicator#170 → CannObserv/replicator#171 → CannObserv/replicator#161. Two upstream asks drafted, not filed. CannObserv/replicator#164 still waits on the VS Code client. Next: CannObserv/replicator#170.
  • 2026-10-30: check-in 10: CannObserv/replicator#170 verified and closed (ec96147..ccd034f, 7 commits; CI green by job, 1783/0, AGENTS.md 5960, rcli PING from a clean shell). Nothing new. Decided (operator): CannObserv/replicator#161 takes all 12 ranges, with one doc clause. Next: CannObserv/replicator#171.
  • 2026-10-30: check-in 11: CannObserv/replicator#171 verified and closed (ccd034f..be3d735; CI green by job, 1793/0, COMMANDS 9800, re-embed by the watcher). co-core v0.40.0 shipped, so CannObserv/replicator#172 was filed into Tier 2 (operator). gregoryfoster/skills#469 closed as a duplicate; CannObserv/replicator#151 now waits on gregoryfoster/skills#372 and gregoryfoster/skills#345. CannObserv/cannobserv#602 was decided and briefed (v0.40.1, behind CannObserv/cannobserv#609). Declined (operator): the uncapped-launch guard and the three upstream asks. Next: CannObserv/replicator#161, then CannObserv/replicator#172.
  • 2026-10-31: check-in 12: CannObserv/replicator#161 verified and closed (be3d735..37b6eca; CI green by job, 1819/0, live since 16:00Z; all 12 rows and embedded forms probed refused in the service venv; exact budgets match). The SOCKS-path test gap closes by construction (one blocked_networks source), so the verify recipe now separates rows from forms. Filed (operator): CannObserv/watcher#412 (the 12 rows) and CannObserv/replicator#173 (raw ssl.SSLError, Tier 1, after CannObserv/replicator#172); 100:0:0:1::/64 left out. CannObserv/cannobserv#602 cleared to build; no 0.40.1 yet. Next: CannObserv/replicator#172, briefed.
  • 2026-11-01: check-in 13: CannObserv/replicator#172 verified and closed (37b6eca..2a6e0f2; CI green by job, 1819/0, co-core 0.40.0 ×3 live, replay clean; the re-embed was already done by the coordinator session's watcher). New: CannObserv/replicator#174 (unified shipping-work, Tier 3, and now a prerequisite of CannObserv/replicator#151 via gregoryfoster/skills#345) and CannObserv/replicator#175 (a stale integration test, Tier 1). v0.40.1 is on the index, so CannObserv/replicator#63 is unblocked; CannObserv/watcher#414 waits on it. uv audit found 4 pyasn1 advisories, folded into CannObserv/replicator#173. Decided (operator): CannObserv/replicator#174 → CannObserv/replicator#173 → CannObserv/replicator#175 → CannObserv/replicator#63. Next: CannObserv/replicator#174, briefed.
  • 2026-11-02: check-in 14: CannObserv/replicator#174 verified and closed (2a6e0f2..337623a: the hook's bump, 3 commits, then the hook's second bump; CI green by job on both; 1819/0; rcli PING from a clean shell; no restart). My brief's "breaks at release 2" was corrected in place on the issue. No new numbers. Harvest: a For #151 note, the gregoryfoster/skills#536 adoption added to Waits, and #174's limits. Blockers unchanged: gregoryfoster/skills#345 is done on a branch, not merged. Next: CannObserv/replicator#173, briefed.
  • 2026-11-03: check-in 15: CannObserv/replicator#173 verified and closed (337623a..0a3c5af; CI green by job; live on 1029e6a, replay clean; pyasn1 0.6.5, audit clean, alerts #23–#26 fixed). New: CannObserv/replicator#176 (Dependabot setup-uv 10.3.3), merged by the coordinator as 64c92c0 on the operator's decision. Harvest: the OOM integration flake into CannObserv/replicator#175's brief (traced), For #63 notes, #173's limits; an unidentified default-suite flake (1 of 10 runs) added to Watching. Next: CannObserv/replicator#175, briefed.
  • 2026-11-03: check-in 16: CannObserv/replicator#175 verified and closed (64c92c0..81396a4; CI green by job; -m integration 57/57 twice on a scratch broker; no restart). CONVENTIONS is at 9968 exact. Decided (operator): split the 0.40.1 floor from CannObserv/replicator#63, filed as CannObserv/replicator#177 (Tier 2, next). Harvest: For #63 notes, #175's limit. Blockers unchanged: CannObserv/replicator#151 (upstream), CannObserv/replicator#164 (the VS Code client is still 1.140.0). Next: CannObserv/replicator#177, briefed; then CannObserv/replicator#63, the last unblocked child.
  • 2026-11-03: check-in 17: CannObserv/replicator#177 verified and closed (81396a4..93fb513; CI green by job; live on 93fb513, 0.40.1 ×3, replay clean; re-embed confirmed). My brief's decode site was corrected in place, twice. New: CannObserv/replicator#178 (broker's CLIENT SETINFO question), answered on the operator's decision: take the grant. Next: CannObserv/replicator#63, the last unblocked child, briefed: design doc first, and announcement replies posted by the coordinator with consent.
  • 2026-11-06: check-in 18: CannObserv/replicator#63 verified and closed (93fb513..0de3ce5; CI green by job; default 1902/0, browser 31, integration 57 re-run by the coordinator; live on 0de3ce5 with browser_fetch: enabled). With the operator's go-ahead: announced on CannObserv/watcher#414 and CannObserv/archiver#379, filed CannObserv/cannobserv#635 and CannObserv/replicator#179 (co-core 0.41, from the cannobserv coordinator's epic comment). Re-scan: CannObserv/replicator#163 un-parked (trigger A met: CannObserv/broker#95 closed); CannObserv/replicator#169 re-read (CannObserv/cannobserv#554 closed, CannObserv/cannobserv#615 open), still parked. Next: CannObserv/replicator#179, briefed; then CannObserv/replicator#163 A.
  • 2026-11-06: check-in 19: CannObserv/replicator#179 verified (53918a7..74c69a7; CI green by job; default 1907/0, browser 31, integration 57 re-run by the coordinator; live on 74c69a7, 0.41.0 ×3). New: CannObserv/replicator#180 (CI timeouts, Tier 3). Operator: patching cadence weekly (knob cadence weekly, beb6d3b); .env.bak-63 deleted; CannObserv/replicator#164 rescoped to GC VERSION 4, waiting on gregoryfoster/skills#551. gregoryfoster/skills#345 vendored, so CannObserv/replicator#151 waits on gregoryfoster/skills#372 alone. Next: CannObserv/replicator#163 A, briefed; then CannObserv/replicator#180.
  • 2026-11-06: check-in 20: CannObserv/replicator#163 trigger A verified (beb6d3b..7460a5c; CI green by job; default 1907/0, boundaries 97 re-run by the coordinator; no restart); B parked again with the report's notes. No new numbers. With the operator's go-ahead: nudged gregoryfoster/skills#551, filed CannObserv/cannobserv#640. VS Code client now 1.145.2. Next: CannObserv/replicator#180, briefed, the last unblocked child.
  • 2026-11-07: check-in 21: CannObserv/replicator#180 verified (7460a5c..7b21e86; CI green by job; tests/ci 27, default 1914 re-run by the coordinator; no restart). The watched flake recurred once (5 failed, names lost), 14 clean after. Filed CannObserv/replicator#181 (operator). CannObserv/replicator#169 re-read: Watcher is on the co-pypi index, Replicator's reader identity can read it; un-parking part 1 proposed. Blockers unchanged; gregoryfoster/skills#551 scheduled upstream. Next: CannObserv/replicator#181.
  • 2026-11-08: check-in 22: CannObserv/replicator#181 verified (7b21e86..83af705; CI green by job; tests/ci 37, default 1913/1 in a clean shell; no restart). The watched flake solved: PRODUCTION_ENV misses the capture and browser-fetch flags, so pytest after load_env.sh fails 5 tests every time; my #180 verification corrected in place. Filed CannObserv/replicator#182 (bug, next) and CannObserv/replicator#183 (#169 part 1, the co-pypi index) on the operator's decision. Next: CannObserv/replicator#182.
  • 2026-11-08: check-in 23: CannObserv/replicator#182 verified (83af705..86c8986; CI green by job; default 1914/1 in a clean shell and after load_env.sh, tests/ci 37, ruff clean; no restart); the Verify recipe's clean-shell caveat is retired. No new numbers. gregoryfoster/skills#551 shipped and was vendored (edb63bc), so CannObserv/replicator#164 is unblocked; pre-flight plan update 2 → 4, read-only. Decided (operator): CannObserv/replicator#164 → CannObserv/replicator#183. Next: CannObserv/replicator#164, briefed.
  • 2026-11-09: check-in 24: CannObserv/replicator#164 verified (host-only: GC VERSION 4 at 00:27Z, re-plan current, timers enabled, first run exit 0, root 74%; no repo change, no restart). co-core 0.42.0 found on the index; filed CannObserv/replicator#184 after CannObserv/replicator#183 (operator). Pre-flight for CannObserv/replicator#183 found the reader-key hazard (ADC here is the writer) and 9 more wheelhouse sites; brief posted. Next: CannObserv/replicator#183.
  • 2026-11-10: check-in 25 (operator: file and pre-flight the browser CI failure): browser red on b5d4f2f, the hook's submodule bump; reproduced 3 in 30 locally, traced to the sandbox test's poll; filed CannObserv/replicator#186, ordered before CannObserv/replicator#183 (operator). Found CannObserv/replicator#185 (docs, from Watcher, filed 2026-11-09), waiting on CannObserv/watcher#420. Idle checks every 12 h set up 2026-11-09. Next: CannObserv/replicator#186.
  • 2026-11-10: check-in 26: CannObserv/replicator#186 verified and closed (b5d4f2f..c74be97, 4 test commits, fast-forward; step 1 showed a test race, not a sandbox gap; scope extended to the 429 pacer flake (operator)); CI green by job on c74be97; coordinator re-runs 15/15, -m browser 31, 1914/1; no restart. No new numbers. Harvest: the For #183 note and a pre-flight refresh on CannObserv/replicator#183; #186's limits under Accepted limitations. Blockers unchanged (gregoryfoster/skills#372, CannObserv/broker#85, CannObserv/cannobserv#615, CannObserv/watcher#420: no new comments). Next: CannObserv/replicator#183, briefed.
  • 2026-11-10: check-in 27: CannObserv/replicator#183 verified and closed (c74be97..294e489, merged as bc4c5ec; co-core from the co-pypi index, the wheelhouse retired, reader key renamed REPLICATOR_CO_PYPI_CREDENTIALS); CI green by job; live since 18:36:53Z, 0.41.0 ×3, replay clean, 12 blobs stored; coordinator 1959/2 clean and loaded (minus the broker URL). New: CannObserv/replicator#187 (a loaded-shell test prints the broker password), Tier 1, next (operator), briefed. Harvest: For #184 and a pre-flight on CannObserv/replicator#184; #183's limits; #76's browser assert limit settled; #182's 'no test reads them' corrected in place; CannObserv/replicator#169 re-read (one of three conditions met). Blockers unchanged. Next: CannObserv/replicator#187.
  • 2026-11-10: check-in 28: CannObserv/replicator#187 verified and closed (294e489..10d9ef3; credential settings scrubbed and guarded, no URL in a diff, inflight.py reads its file alone); CI green by job; coordinator 1961/2 clean and loaded with the live URL (0 password hits); no restart. Corrected in place: my brief's 'a stale shell value fails closed' (it could open the patching gate; the implementer found it). No new numbers. Archiver's capture issuance is starting (CannObserv/replicator#138): the capture debt row gains its discharge path; the stale capture-contract Status line folds into CannObserv/replicator#184 (operator). Blockers unchanged. Next: CannObserv/replicator#184.
  • 2026-11-11: check-in 29 (operator: pre-flight CannObserv/replicator#189): Archiver's D4 asks for a captures-only bucket; pre-flight and brief posted (design doc first; one bucket's two tiers conflict on soft delete, update and the lifecycle scope; persist routes by source; replicate will refuse captures). Also slotted CannObserv/replicator#188 (from the 00:15Z co-broker reboot). The capture debt row is discharged (first production capture 14:46:26Z, closed with Archiver on CannObserv/replicator#138). Order (operator): CannObserv/replicator#189 → CannObserv/replicator#184 → CannObserv/replicator#188. Root 83% (three VS Code builds). Next: CannObserv/replicator#189.
  • 2026-11-12: check-in 30: CannObserv/replicator#189 verified and closed (10d9ef3..6e0b19f, PR CannObserv/replicator#190 merged as ba37286; captures in co-gcs-captures, provisioned by the operator, object-level grant probe PASS); CI green by job; live since 02:41:35Z; the first post-split capture read back in temp/ and kept in blobs/, raw buckets untouched; coordinator 2001/2 clean and loaded. Reply posted on CannObserv/archiver#380. CannObserv/replicator#184's fold-in voided (done by #189) and its pre-flight refreshed. Next: CannObserv/replicator#184.
  • 2026-11-13: check-in 31: CannObserv/replicator#184 verified and closed as 0.41 → 0.44 (6e0b19f..6eb65b8, PR CannObserv/replicator#191; widened at review because 0.43.0 and 0.44.0 were already on the index); CI green by job; live since 2026-11-12T23:18:59Z on 0.44.0 ×3, replay clean; coordinator 2001/2 clean and loaded. Corrected in place: my stale index claim on CannObserv/replicator#184 (retitled); a standing check now lists the index unbounded. CannObserv/replicator#188 briefed (a second silent recovery path at policy.py:575). Root 78%. Next: CannObserv/replicator#188.
  • 2026-11-13: idle check found co-core 0.45.0 ×3 on the index (tagged 05:36Z). Filed CannObserv/replicator#192 (operator), closure diff version-only [measured]; slotted after CannObserv/replicator#188. Nothing else moved: build 6eb65b8, NRestarts 0, 0 WARNING/ERROR; blockers unchanged; root 81% (extension update). Next: CannObserv/replicator#188.
  • 2026-11-13: check-in 32: CannObserv/replicator#188 verified and closed (6eb65b8..762fd1a, PR CannObserv/replicator#193, the only new number); CI green by job; live since 19:00:28Z, replay 501/10 clean; coordinator 2005/2 clean and loaded. The recovery line is unit-tested only (Accepted limitations, Watching). Filed CannObserv/processor#134 and CannObserv/watcher#436 (operator): same gap, traced. CannObserv/replicator#192 pre-flighted: index still 0.45.0; test_dependency_index.py's >=0.45 bound added to scope. Blockers unchanged. Next: CannObserv/replicator#192.
  • 2026-11-13: check-in 33: CannObserv/replicator#192 verified and closed (762fd1a..da5a19d, PR CannObserv/replicator#194); CI green by job; live since 20:48:46Z on 0.45.0 ×3, replay 503/10 clean; coordinator 2005/2 clean and loaded; index still 0.45.0. New: CannObserv/replicator#195 (filed by the implementer, operator's call), slotted Tier 3 and pre-flighted; it's the last unblocked child. Blockers unchanged. Next: CannObserv/replicator#195.
  • 2026-11-13: check-in 34: CannObserv/replicator#195 verified and closed (da5a19d..d5926ab, PR CannObserv/replicator#196; CI's uv pinned to the bound's range, warning gone in CI and on the host; no restart); coordinator 2008/2 clean and loaded, tests/ci 44. New: CannObserv/replicator#197 (archiver's patch-window notice), closed as no impact with the operator's consent, correction on CannObserv/archiver#375. Blockers re-read: CannObserv/cannobserv#640 met (v0.42.0); the rest unchanged. No child unblocked; the hand-off is the blockers.
  • 2026-11-14: check-in 35: the weekly patching run CannObserv/replicator#198 verified and closed (92 packages, reboot 00:50:45Z, 32 s down, 0 pending after); 3d16e77 (knob exception) and the hook's a46e650, CI green by job on both; coordinator 2009/1, tests/ci 44. gregoryfoster/skills#372 pre-flighted upstream (a plan first), still open. Still no child unblocked: the hand-off is the blockers; first to clear is CannObserv/watcher#420 → CannObserv/replicator#185, no earlier than 2026-11-16.
  • 2026-11-14: check-in 36: nothing merged since a46e650; the 13:57Z idle check found co-core 0.46.0 and 0.46.1 ×3 on the index, and gregoryfoster/skills#372 Part A shipped (f728abd5), unblocking CannObserv/replicator#151. Filed CannObserv/replicator#199 (operator), closure diff version-only [measured]. Decided (operator): CannObserv/replicator#199 → CannObserv/replicator#151. Coordinator note on CannObserv/replicator#151: the vendored pin 4f26e316 lacks the skill; KEEP sized against 3.6G free. Next: CannObserv/replicator#199.
  • 2026-11-14: check-in 37: CannObserv/replicator#199 verified and closed (a46e650..91e87dc, PR CannObserv/replicator#200); CI green by job; live since 19:01:13Z on 0.46.1 ×3, replay 509/10 clean; coordinator 2009/1 clean and loaded, tests/ci 44. My brief's wp_view_*_url claim corrected in place (not in Replicator's wheels). Decided (operator): keep the closure diff; the shared co-pypi wrapper is CannObserv/replicator#201 (filed), after CannObserv/replicator#151, its fetcher built upstream-ready. Next: CannObserv/replicator#151, briefed.
  • 2026-11-14: check-in 38: CannObserv/replicator#151 verified and closed (91e87dc..b61703a; CI green by job on M and C): replicator runs a root-owned release under /srv/replicator, rollback drilled, coordinator 2028/1 and tests/deploy 32; my KEEP arithmetic corrected in place; agent memory retired. Root is 83.72%: decided (operator) to grow the disk, filed CannObserv/replicator#204 into Tier 1. New follow-ups CannObserv/replicator#202 and CannObserv/replicator#203 into Tier 3. For #201 note posted. Next: CannObserv/replicator#204.
  • 2026-11-15: check-in 39: CannObserv/replicator#204 verified (the operator's grow to 30 GiB, reboot 23:55:50Z, no root commands needed; b61703a..7706bf0 docs, deployed 00:13Z; CI green by job on 7706bf0); root 54.46%, the worker on 7706bf05cc16, coordinator 2028/1, tests/deploy 32. The sysctl drop-in's drift discharged (operator-authorized install). Patching has ~8.9G headroom. Next: CannObserv/replicator#201, pre-flighted.
  • 2026-11-15: check-in 40: CannObserv/replicator#201 verified (7706bf0..a8900f5, PR CannObserv/replicator#205; CI green by job on a8900f5; live a8900f5d15a2, the first production mint through the shared wrapper; coordinator 2044/1, the new fetcher's 12 wheels and the dry pin move reproduced). The manifest re-embed debt discharged. Upstream: gregoryfoster/skills#652 filed, gregoryfoster/skills#454 commented (operator). Found: the worker's uid has passwordless sudo, unsandboxed. Decided (operator): CannObserv/replicator#202's shared-group model; CannObserv/replicator#202 next, briefed.
  • 2026-11-15: check-in 41: CannObserv/replicator#202 verified (a8900f5..f2a4333; CI green by job; the worker runs as replicator, confined, after the 05:33Z reboot; a production capture after the switch; coordinator 2062/1, host deploy tests 181). Filed CannObserv/replicator#206 (operator): co-core 0.46 → 0.48, widened to 0.48.2 at pre-flight; closure diff Drive-only. The GC skipped 2026-11-15 (the reboot, Persistent=false). Next: CannObserv/replicator#206.
  • 2026-11-16: check-in 42: CannObserv/replicator#206 verified (f2a4333..59fb240; CI green by job; live on 0.48.2 ×3 as replicator; coordinator 2062/1 and 181 counting one session-premise test run in the foreground; the code index caught up). CannObserv/watcher#420 closed 2026-11-15 (gate waived), so CannObserv/replicator#185 is unblocked; pre-flight posted (two more stale sites; the Reader row needs the operator's policy read). New: #207 (the PR, closed). Next: CannObserv/replicator#185.
  • 2026-11-16: check-in 42, continued: the operator chose to park CannObserv/replicator#203 on gregoryfoster/skills#641; the scope ask is posted there and the park note on CannObserv/replicator#203. CannObserv/replicator#185 is the last unblocked child.
  • 2026-11-16: check-in 43: CannObserv/replicator#185 verified (59fb240..d549a06; CI green by job; docs and a docstring; Readers from the operator's live policy; coordinator 2062/1; not deployed, by the operator). The disk GC ran 05:18Z (exit 0). The exit criterion is met; no child is unblocked.
  • 2026-11-16: check-in 44: the operator chose parking-lot mode. Retitled; every parked row rewritten brief-grade and re-read (co-core 0.48.2 still decodes generation as optional; Archiver still on 0.38; CannObserv/archiver#408 and CannObserv/cannobserv#615 unchanged).
  • 2026-11-16: the run journaled at the park (operator's approval): process-log entry filed upstream as gregoryfoster/skills#668.
  • 2026-11-17: check-in 45: the operator asked for PR mode; filed and pre-flighted CannObserv/replicator#208. Idle check 08:43Z normal: the hook's 85f8d7b green by job; live 59fb24082f7e, NRestarts 0; co-core 0.48.2 still newest.
  • 2026-11-17: check-in 46: CannObserv/replicator#208 verified (85f8d7b..a072ba5, the first GitHub merge commit; push CI green by job); not deployed (operator). Two debt rows (first pr-mode cadence and hook runs), one Watching row (the standing bot PRs). CR 6 asked.
  • 2026-11-17: CR 6 (operator's answer): filed upstream as gregoryfoster/skills#675.
  • 2026-11-18: check-in 47: merged the cadence PR #210 (operator's request; close/reopen for CI, green by job, merge commit 796186f); the first pr-mode cadence row discharged. Filed and briefed CannObserv/replicator#211 (co-core 0.48 → 0.50) out of the parking lot; parked rows re-read against 0.50.0: CannObserv/replicator#163 B unchanged.
  • 2026-11-18: check-in 48: CannObserv/replicator#211 verified and live (753ecb9609a0, co-core 0.50.0 ×3, the first GitHub-merge-commit deploy); the hook's first pr-mode run opened PR #212 (debt discharged); two brief claims corrected in place.
  • 2026-11-18: merged the refresh PR #212 (operator's request): skills 063ba5d → 817963f, merge commit 64a2c7a, its push CI green by job; main checkout pulled and the submodule checked out to the pointer, doctor exit 0, clean. Not deployed: no shipped code changed.
  • 2026-11-20: idle check rescheduled to weekdays 04:03 PT (operator); status read 17:59Z: two one-cycle broker read timeouts (2026-11-19 09:30Z, 2026-11-20 09:29Z), each recovered in 6 s; co-core 0.50.2 on the index, version-only over this closure; refresh PR #214 open, CI green.

Activity

  1. gregoryfoster commented on Oct 26, 2026

    @gregoryfoster
    MemberAuthor

    #135 is now this thread's to coordinate (operator, 2026-10-26). The hand-off is #135 (comment); #135's body has been updated through epic-edit.sh and reads back clean.

  2. added a commit that references this issue on Nov 3, 2026
  3. gregoryfoster commented on Nov 5, 2026

    @gregoryfoster
    MemberAuthor

    cannobserv v0.41.0: two lines for Replicator

    v0.41.0 is tagged at cannobserv f7285e1e. Replicator pins >=0.40.1,<0.41 (main 7c151486), so taking it is a floor move.

    • The BlobStore reporting twins (cannobserv#530). store_reporting / store_file_reporting return Stored(uri, created), and the base store / store_file now return self.…_reporting(...).uri by dispatch. SweepSafeLocalBlobStore (src/storage/local_store.py:33-39) overrides store / store_file only, and it keeps working unchanged, since super().store still runs inside its retry. Before Replicator calls a twin, move the retry onto store_reporting / store_file_reporting and drop the two plain-verb overrides. Keeping both nests the retry, so one write is attempted up to four times. A call to a twin on today's subclass skips the retry. The test-only ThreadRecordingStore (tests/worker/test_storage_offloop.py), which overrides store alone, wouldn't record a reporting call.
    • FetchPolicyEmit.generation is required (cannobserv#579). This touches only the Emit side. Replicator decodes FetchPolicyState, which still takes a frame without one, and fetch_policy_supersedes keeps its ratchet. Replicator has nothing to change.
  4. changed the title [-][Epic] Backlog burn-down: live stability, sibling waits, operational conformance (2026-10)[/-] [+][Epic] Parking lot (burn-down done 2026-11-16): live stability, sibling waits, operational conformance[/+] on Nov 16, 2026
  5. added a commit that references this issue on Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions