refactor: split packages and add release machinery#5
Merged
Conversation
Rework the OSS release machinery to mirror FlowMesh's pattern: the
``lumilake`` distribution becomes a thin metapackage with extras
``[sdk]``/``[cli]``/``[deploy]``/``[hook]``/``[all]`` that pull in
four workspace wheels; the server runtime lives in ``src/lumilake_server/``
and ships only inside the Docker image, never on PyPI.
Workspace:
- packages/sdk publishes the ``lumilake`` Python module (Client, envs,
log). Other workspace packages depend on it as the shared base.
- packages/cli publishes ``lumilake_cli`` (Typer entry point ``lumilake``).
- packages/deploy publishes ``lumilake_deploy`` (Docker / FlowMesh glue).
- packages/hook publishes ``lumilake_hook`` (lumid-hooks bindings).
- Server code moved from ``src/lumilake/*`` to ``src/lumilake_server/*``;
it stays out of every wheel.
Image release:
- New ``.github/workflows/image-publish.yml`` builds linux/amd64+arm64
via buildx and pushes to ``ghcr.io/mlsys-io/lumilake_server`` with
tags ``:<sha>`` per push, ``:dev`` on main, and ``:<semver>`` + ``:latest``
on tag-v*. GHCR auth uses the workflow's ``GITHUB_TOKEN``.
- ``scripts/dev/sync_requirements.py`` derives
``src/lumilake_server/requirements.txt`` from the ``server-runtime``
uv dependency group so the Dockerfile's pin set tracks ``uv.lock``.
- Dockerfile rewritten to install the published ``lumilake-sdk`` wheel
plus the generated requirements, copy ``src/lumilake_server/`` onto
``PYTHONPATH``, and run ``python -m lumilake_server.main``.
- docker-compose ``server`` service now pulls
``\${LUMILAKE_REGISTRY:-ghcr.io/mlsys-io}/lumilake_server:\${LUMILAKE_IMAGE_TAG:-latest}``
(no ``build:`` stanza).
CLI:
- ``LUMILAKE_REGISTRY`` env var (default ``ghcr.io/mlsys-io``) selects
the image registry.
- New ``lumilake deploy pull`` command pulls the published image.
- ``lumilake deploy up`` no longer auto-builds — it fails fast if the
image is missing, pointing the user at ``pull`` or ``build``.
- ``lumilake deploy build`` retains source builds (re-tagged for GHCR).
PyPI publish:
- ``.github/workflows/package-build.yml`` now builds all workspace
packages (``uv build --all-packages``) and, on a tag-v*, uploads the
dist artifact and publishes via OIDC trusted publishing
(``pypa/gh-action-pypi-publish``).
- ``scripts/ci/check_package_build.py`` validates the metapackage is
code-free, each workspace wheel ships its top-level module, none of
them contain ``lumilake_server/``, and smoke-installs every extra.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>
Post-review tightening of the GHCR + PyPI release flow introduced in 5d5ed47. Adopts FlowMesh's release-day machinery and moves the deploy templates into ``lumilake-deploy`` package data so PyPI users can run ``lumilake deploy`` without a source checkout. Release machinery (FlowMesh-aligned): - ``docs/RELEASE.md`` runbook covers PyPI Trusted Publisher pending- publisher setup, GHCR visibility + repo-link manual steps, ``pypi`` / ``testpypi`` / ``ghcr`` environment approvals, and the rollback recipes (yank, ``.postN``, "PyPI succeeded but image broken"). - ``scripts/ci/check_release_version.py`` cross-checks every ``version = ...`` string across the 5 pyprojects + 2 ``__version__`` literals against the release tag. - ``scripts/dev/bump_version.py`` writes them in one pass; release.yml re-runs ``--check`` after the version validator for belt-and-suspenders. - ``scripts/ci/check_image_release.py`` verifies the published image's per-platform OCI labels match the release tag + commit, polls for the manifest (race-safe against image-publish.yml), and is the gate for the ``:latest`` retag. - ``scripts/ci/retag_image_release.py`` writes ``:latest`` via ``docker buildx imagetools create``, with downgrade and missing-label-protection guards. - ``scripts/ci/filter_distributions.py`` trims dist files for staggered Trusted Publisher onboarding. - ``scripts/ci/append_release_digests.py`` sentinel-replace edits the Release body with the digest table, idempotent across re-runs. - New ``.github/workflows/release.yml`` (PyPI publish; GitHub release event, ``pypi`` environment approval) + ``release-images.yml`` (``:latest`` retag; ``ghcr`` environment approval). Both run on ``self-hosted`` to match FlowMesh. - ``image-publish.yml`` keeps pushing ``:sha`` / ``:dev`` / ``:vX.Y.Z`` per commit/tag but no longer pushes ``:latest`` (release-images owns it). Adds a post-push native-arch smoke (``import lumilake, lumilake_hook, lumilake_server``). - ``package-build.yml`` reduced to PR/main build + verify + ``twine check``; PyPI publish moved entirely to ``release.yml``. Packaged deploy assets: - ``.env.example`` and ``docker-compose.yml`` moved into ``packages/deploy/src/lumilake_deploy/assets/`` (compose renamed ``compose.yml``); root ``.env.example`` is now a symlink for editor / IDE convenience, matching FlowMesh. - New ``lumilake_deploy.assets`` module exposes ``asset_path`` / ``env_example_path`` / ``compose_path`` via ``importlib.resources`` so the deploy CLI works from any directory after a ``pip install``. - ``deploy init`` reads the packaged template; ``setup.py`` / ``stop.py`` run ``docker compose -f <compose_path()> --project-directory <deploy_dir>``. ``build_server_image`` now raises a friendly error when invoked without a source checkout's Dockerfile. ``-C`` / ``--project-dir`` (+ ``LUMILAKE_DEPLOY_DIR``) for the deploy CLI: every subcommand of ``lumilake deploy`` reads the env / compose project dir from the new option / env var, defaulting to cwd. Removes the source-checkout-required walker. CLI weight split: ``lumilake-cli`` drops ``lumilake-deploy`` from hard deps, exposes it as ``[deploy]`` extra. ``lumilake[cli]`` keeps the full lifecycle. ``commands/__init__.py`` uses ``find_spec`` to gate the deploy subcommands (real ImportError inside .deploy bubbles up loudly). Thin-install users hit an install hint for every deploy subcommand instead of typer's bare "No such command". SDK polish: - ``packages/sdk/src/lumilake/py.typed`` marker added. - ``resources/deploy.py`` switches the backend gate to ``find_spec``, reads ``ENV_FILE_NAME`` from the packaged constants, and resolves FlowMesh container names from ``.env.flowmesh`` via the shared ``lumilake_deploy.containers`` helper (was hardcoded ``flowmesh_node_*``). Exports ``SERVICE_NAMES`` (the slug-prefixed container names are per-deploy and resolved at call time). stop.py ``--wipe-archive`` derives the FlowMesh state-volume names from the same slug helper so an operator who ran ``init --flowmesh`` with the default ``lumilake`` suffix actually has their FlowMesh volumes removed. Per-package ``LICENSE`` files committed (workspace wheels were previously shipping without LICENSE due to a parent-traversal ``license-files``). ``check_package_build.py`` SHA-verifies every wheel's bundled LICENSE matches the repo root; it also catches flat-file leaks and enforces ``lumilake_server`` absent from every wheel. Server runtime / Dockerfile: non-root ``lumilake`` user (uid 10001), ``tini`` PID-1 + exec-form ``CMD``, ``COPY LICENSE ./LICENSE``, installs ``lumilake-sdk`` + ``lumilake-hook`` in-tree + flat ``requirements.txt`` generated by ``scripts/dev/sync_requirements.py`` (env-marker safe, fails loudly on unparsable specs). CI consistency: - All 9 ``setup-uv`` invocations now pin both ``version: "0.11.8"`` and ``python-version: "3.12"``. - ``env-examples.yml`` adds ``--extra deploy`` so the script can ``import lumilake_deploy.doctor``. - ``lint-typecheck.yml`` and ``unit-tests.yml`` add ``--frozen``. - ``security.yml`` bandit scans every package source root + ``scripts/``; pip-audit splits into image-runtime (server ``requirements.txt``) and wheel-install closure (``uv export --all-extras --no-default-groups --no-emit-workspace``). Docs: - ``README.md`` Quick Start + Deployment, ``docs/CLI.md``, ``docs/ENV.md``, ``CONTRIBUTING.md`` consistently use ``lumilake deploy -C <path>`` (or ``LUMILAKE_DEPLOY_DIR``), with explicit ``uv run`` vs PyPI install guidance. - ``docs/ARCHITECTURE.md``, ``docs/OPS.md``, ``docs/PLUGINS.md``, ``docs/CODE_STYLE.md``, ``docs/SDK.md`` updated for the workspace split (no more ``lumilake.{server,runtime,ops,utils,...}`` paths). - PR template + repository-layout block updated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>
Pedantic-persona zizmor (the form CI runs under ``--persona pedantic --format github``) was failing with two findings on the image-publish workflow: * ``cache-poisoning`` (high) — ``astral-sh/setup-uv`` enables its cache by default. In a workflow that pushes signed artifacts to GHCR, a poisoned cache could influence the publishing step. Set ``enable-cache: false`` on this workflow's setup-uv invocation; the other setup-uv calls (lint, tests, env-examples, security, etc.) aren't publishing workflows and keep caching for speed. * ``undocumented-permissions`` (low) — ``packages: write`` at job level had no explanatory comment. Added the same style of inline comment release-images.yml already uses (`docker/login-action + build-push-action push to GHCR`). ``zizmor --persona pedantic --format github`` exits 0 after the fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>
timzsu
force-pushed
the
release/ghcr-machinery
branch
from
May 14, 2026 07:32
24bed07 to
a6692e5
Compare
Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>
timzsu
force-pushed
the
release/ghcr-machinery
branch
from
May 14, 2026 07:47
a6692e5 to
52f3224
Compare
timzsu
changed the title
Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>
timzsu
force-pushed
the
release/ghcr-machinery
branch
from
May 14, 2026 07:59
5196951 to
8dfe349
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Stand up the OSS release machinery for Lumilake. After this PR the project can be released to PyPI (multiple wheels, OIDC trusted publishing) and GHCR (server image) from a tagged commit, and the CLI works from a plain
pip install lumilake[cli]without a source checkout. Mirrors FlowMesh's release-day flow where it makes sense so the two projects converge on one operator playbook.Changes
Three orthogonal pieces, in order of blast radius:
lumilakepackage -> metapackage + 4 published wheels (lumilake-sdk,lumilake-cli,lumilake-deploy,lumilake-hook) + an image-onlylumilake_serverruntime that never lands on PyPI. SDK is the shared base; CLI and deploy depend on it.lumilake[cli]keeps the full deploy lifecycle; thinpip install lumilake-cliskips Docker/deploy deps and prints install hints fordeploy *subcommands.image-publish.ymlbuilds PR images without publishing, and pushes per-commit/per-tag images onmain/v*(:sha/:dev/:vX.Y.Z, never:latest).release.ymldoes PyPI/TestPyPI OIDC publish after building and validating all distributions.release-images.ymlverifies the pre-pushed image labels and retags:latestwithghcrenvironment approval and downgrade protection.check_release_version.pycross-checks 5 pyprojects + 2__version__literals;bump_version.pywrites them in one pass.--project-dir..env.exampleandcompose.ymlship insidelumilake-deployas package data, resolved viaimportlib.resources.lumilake deploy -C <dir>(orLUMILAKE_DEPLOY_DIR) selects where env files live; no source checkout needed. SDK resolves FlowMesh container names from.env.flowmesh's slug;down --wipe-archiveuses the same slug for volume cleanup.Design
pypi/testpypienvironments. GHCR image push uses the workflowGITHUB_TOKENwith job-levelpackages: write; PR image builds never receive that permission and never push. The:latestretag is separated behind theghcrenvironment.release: published. PyPI immutability + GHCR:latestretag-after-verify means a half-published release degrades gracefully (re-runrelease-images.ymlfor image-only failure; yank +.postNfor PyPI).release-images.ymlpolls the registry up to 30 min to absorb the race withimage-publish.yml.src/lumilake_server/is intentionally not published.check_package_build.pyenforces that any wheel containinglumilake_server.*fails CI; it also SHA-checks each wheel's bundled LICENSE matches the repo root.check_release_version.py, sentinel-replace digest table append, downgrade-protected retag, two-tier CLI/deploy weight). Cheaper to lift than reinvent; identical operator UX across the two projects.Test Plan
uv build --all-packages,twine check, andcheck_package_build.pyvalidate the metapackage + 4 workspace wheels.lumilake deploy -C <dir>, non-checkout deploy behavior, and FlowMesh slug-derived container/volume names.main/tag pushes log in to GHCR and smoke-test the pushed image.v*tag because PyPI Trusted Publisher and GHCR package visibility are operator-side setup.Test Result
The release workflows themselves will only get a true end-to-end exercise on the first
v*tag; their dependency on PyPI Trusted Publisher config + GHCR package visibility/repository access settings is operator-side and documented indocs/RELEASE.md.Follow-up before first release
lumilake,lumilake-sdk,lumilake-cli,lumilake-deploy,lumilake-hook(env=pypi, repo=mlsys-io/lumilake_OSS, workflow=release.yml).testpypi.pypi/testpypi/ghcrGitHub environments. Add reviewer approval where the repository/org plan exposes environment protection rules.image-publish.ymlpush: flip GHCR package visibility to public + link to repo (Manage Actions access -> Write).Pre-submission Checklist
CONTRIBUTING.md.uv run pre-commit run --all-filesand fixed any issues.uv run pytest tests/ --ignore=tests/serverpasses locally.[BREAKING]and described migration steps above.