refactor: split packages and add release machinery (#5)

* refactor: split workspace, publish server image to GHCR

Rework the OSS release machinery to mirror FlowMesh's pattern: the
``lumilake`` distribution becomes a thin metapackage with extras
``[sdk]``/``[cli]``/``[deploy]``/``[hook]``/``[all]`` that pull in
four workspace wheels; the server runtime lives in ``src/lumilake_server/``
and ships only inside the Docker image, never on PyPI.

Workspace:
- packages/sdk publishes the ``lumilake`` Python module (Client, envs,
  log). Other workspace packages depend on it as the shared base.
- packages/cli publishes ``lumilake_cli`` (Typer entry point ``lumilake``).
- packages/deploy publishes ``lumilake_deploy`` (Docker / FlowMesh glue).
- packages/hook publishes ``lumilake_hook`` (lumid-hooks bindings).
- Server code moved from ``src/lumilake/*`` to ``src/lumilake_server/*``;
  it stays out of every wheel.

Image release:
- New ``.github/workflows/image-publish.yml`` builds linux/amd64+arm64
  via buildx and pushes to ``ghcr.io/mlsys-io/lumilake_server`` with
  tags ``:<sha>`` per push, ``:dev`` on main, and ``:<semver>`` + ``:latest``
  on tag-v*. GHCR auth uses the workflow's ``GITHUB_TOKEN``.
- ``scripts/dev/sync_requirements.py`` derives
  ``src/lumilake_server/requirements.txt`` from the ``server-runtime``
  uv dependency group so the Dockerfile's pin set tracks ``uv.lock``.
- Dockerfile rewritten to install the published ``lumilake-sdk`` wheel
  plus the generated requirements, copy ``src/lumilake_server/`` onto
  ``PYTHONPATH``, and run ``python -m lumilake_server.main``.
- docker-compose ``server`` service now pulls
  ``\${LUMILAKE_REGISTRY:-ghcr.io/mlsys-io}/lumilake_server:\${LUMILAKE_IMAGE_TAG:-latest}``
  (no ``build:`` stanza).

CLI:
- ``LUMILAKE_REGISTRY`` env var (default ``ghcr.io/mlsys-io``) selects
  the image registry.
- New ``lumilake deploy pull`` command pulls the published image.
- ``lumilake deploy up`` no longer auto-builds — it fails fast if the
  image is missing, pointing the user at ``pull`` or ``build``.
- ``lumilake deploy build`` retains source builds (re-tagged for GHCR).

PyPI publish:
- ``.github/workflows/package-build.yml`` now builds all workspace
  packages (``uv build --all-packages``) and, on a tag-v*, uploads the
  dist artifact and publishes via OIDC trusted publishing
  (``pypa/gh-action-pypi-publish``).
- ``scripts/ci/check_package_build.py`` validates the metapackage is
  code-free, each workspace wheel ships its top-level module, none of
  them contain ``lumilake_server/``, and smoke-installs every extra.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>

* refactor: harden release machinery and packaged deploy assets

Post-review tightening of the GHCR + PyPI release flow introduced in
5d5ed47. Adopts FlowMesh's release-day machinery and moves the deploy
templates into ``lumilake-deploy`` package data so PyPI users can run
``lumilake deploy`` without a source checkout.

Release machinery (FlowMesh-aligned):
- ``docs/RELEASE.md`` runbook covers PyPI Trusted Publisher pending-
  publisher setup, GHCR visibility + repo-link manual steps,
  ``pypi`` / ``testpypi`` / ``ghcr`` environment approvals, and the
  rollback recipes (yank, ``.postN``, "PyPI succeeded but image
  broken").
- ``scripts/ci/check_release_version.py`` cross-checks every
  ``version = ...`` string across the 5 pyprojects + 2 ``__version__``
  literals against the release tag.
- ``scripts/dev/bump_version.py`` writes them in one pass; release.yml
  re-runs ``--check`` after the version validator for belt-and-suspenders.
- ``scripts/ci/check_image_release.py`` verifies the published image's
  per-platform OCI labels match the release tag + commit, polls for the
  manifest (race-safe against image-publish.yml), and is the gate for
  the ``:latest`` retag.
- ``scripts/ci/retag_image_release.py`` writes ``:latest`` via
  ``docker buildx imagetools create``, with downgrade and
  missing-label-protection guards.
- ``scripts/ci/filter_distributions.py`` trims dist files for staggered
  Trusted Publisher onboarding.
- ``scripts/ci/append_release_digests.py`` sentinel-replace edits the
  Release body with the digest table, idempotent across re-runs.
- New ``.github/workflows/release.yml`` (PyPI publish; GitHub release
  event, ``pypi`` environment approval) + ``release-images.yml``
  (``:latest`` retag; ``ghcr`` environment approval). Both run on
  ``self-hosted`` to match FlowMesh.
- ``image-publish.yml`` keeps pushing ``:sha`` / ``:dev`` / ``:vX.Y.Z``
  per commit/tag but no longer pushes ``:latest`` (release-images owns
  it). Adds a post-push native-arch smoke (``import lumilake,
  lumilake_hook, lumilake_server``).
- ``package-build.yml`` reduced to PR/main build + verify + ``twine
  check``; PyPI publish moved entirely to ``release.yml``.

Packaged deploy assets:
- ``.env.example`` and ``docker-compose.yml`` moved into
  ``packages/deploy/src/lumilake_deploy/assets/`` (compose renamed
  ``compose.yml``); root ``.env.example`` is now a symlink for editor /
  IDE convenience, matching FlowMesh.
- New ``lumilake_deploy.assets`` module exposes ``asset_path`` /
  ``env_example_path`` / ``compose_path`` via ``importlib.resources``
  so the deploy CLI works from any directory after a ``pip install``.
- ``deploy init`` reads the packaged template; ``setup.py`` / ``stop.py``
  run ``docker compose -f <compose_path()> --project-directory
  <deploy_dir>``. ``build_server_image`` now raises a friendly error
  when invoked without a source checkout's Dockerfile.

``-C`` / ``--project-dir`` (+ ``LUMILAKE_DEPLOY_DIR``) for the deploy
CLI: every subcommand of ``lumilake deploy`` reads the env / compose
project dir from the new option / env var, defaulting to cwd. Removes
the source-checkout-required walker.

CLI weight split: ``lumilake-cli`` drops ``lumilake-deploy`` from hard
deps, exposes it as ``[deploy]`` extra. ``lumilake[cli]`` keeps the
full lifecycle. ``commands/__init__.py`` uses ``find_spec`` to gate the
deploy subcommands (real ImportError inside .deploy bubbles up loudly).
Thin-install users hit an install hint for every deploy subcommand
instead of typer's bare "No such command".

SDK polish:
- ``packages/sdk/src/lumilake/py.typed`` marker added.
- ``resources/deploy.py`` switches the backend gate to ``find_spec``,
  reads ``ENV_FILE_NAME`` from the packaged constants, and resolves
  FlowMesh container names from ``.env.flowmesh`` via the shared
  ``lumilake_deploy.containers`` helper (was hardcoded
  ``flowmesh_node_*``). Exports ``SERVICE_NAMES`` (the slug-prefixed
  container names are per-deploy and resolved at call time).

stop.py ``--wipe-archive`` derives the FlowMesh state-volume names from
the same slug helper so an operator who ran ``init --flowmesh`` with
the default ``lumilake`` suffix actually has their FlowMesh volumes
removed.

Per-package ``LICENSE`` files committed (workspace wheels were
previously shipping without LICENSE due to a parent-traversal
``license-files``). ``check_package_build.py`` SHA-verifies every
wheel's bundled LICENSE matches the repo root; it also catches
flat-file leaks and enforces ``lumilake_server`` absent from every
wheel.

Server runtime / Dockerfile: non-root ``lumilake`` user (uid 10001),
``tini`` PID-1 + exec-form ``CMD``, ``COPY LICENSE ./LICENSE``,
installs ``lumilake-sdk`` + ``lumilake-hook`` in-tree + flat
``requirements.txt`` generated by ``scripts/dev/sync_requirements.py``
(env-marker safe, fails loudly on unparsable specs).

CI consistency:
- All 9 ``setup-uv`` invocations now pin both ``version: "0.11.8"`` and
  ``python-version: "3.12"``.
- ``env-examples.yml`` adds ``--extra deploy`` so the script can
  ``import lumilake_deploy.doctor``.
- ``lint-typecheck.yml`` and ``unit-tests.yml`` add ``--frozen``.
- ``security.yml`` bandit scans every package source root +
  ``scripts/``; pip-audit splits into image-runtime (server
  ``requirements.txt``) and wheel-install closure
  (``uv export --all-extras --no-default-groups
  --no-emit-workspace``).

Docs:
- ``README.md`` Quick Start + Deployment, ``docs/CLI.md``, ``docs/ENV.md``,
  ``CONTRIBUTING.md`` consistently use ``lumilake deploy -C <path>``
  (or ``LUMILAKE_DEPLOY_DIR``), with explicit ``uv run`` vs PyPI
  install guidance.
- ``docs/ARCHITECTURE.md``, ``docs/OPS.md``, ``docs/PLUGINS.md``,
  ``docs/CODE_STYLE.md``, ``docs/SDK.md`` updated for the workspace
  split (no more ``lumilake.{server,runtime,ops,utils,...}`` paths).
- PR template + repository-layout block updated.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>

* ci: clear zizmor findings on image-publish.yml

Pedantic-persona zizmor (the form CI runs under
``--persona pedantic --format github``) was failing with two findings
on the image-publish workflow:

* ``cache-poisoning`` (high) — ``astral-sh/setup-uv`` enables its cache
  by default. In a workflow that pushes signed artifacts to GHCR, a
  poisoned cache could influence the publishing step. Set
  ``enable-cache: false`` on this workflow's setup-uv invocation; the
  other setup-uv calls (lint, tests, env-examples, security, etc.)
  aren't publishing workflows and keep caching for speed.
* ``undocumented-permissions`` (low) — ``packages: write`` at job
  level had no explanatory comment. Added the same style of inline
  comment release-images.yml already uses
  (`docker/login-action + build-push-action push to GHCR`).

``zizmor --persona pedantic --format github`` exits 0 after the fix.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>

* patch image publish workflow

Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>

* docs: clarify image publish workflow

Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>

---------

Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: timzsu

.env.example

@@ -0,0 +1 @@
+packages/deploy/src/lumilake_deploy/assets/.env.example

.env.example

@@ -19,8 +19,8 @@ Scope (optional): deploy, server, cli, runtime, docker, ...
 
 <!-- List modified files or groups of files with a brief explanation. -->
 <!--
-- `src/lumilake/cli/commands/deploy.py` — added `--since` flag to `deploy logs`
-- `src/lumilake/deploy/docker_client.py` — thread `since` through to docker-py
+- `packages/cli/src/lumilake_cli/commands/deploy.py` — added `--since` flag to `deploy logs`
+- `packages/deploy/src/lumilake_deploy/docker_client.py` — thread `since` through to docker-py
 - `tests/cli/test_deploy_impl.py` — cover the new flag
 -->
 

.github/PULL_REQUEST_TEMPLATE.md

@@ -22,6 +22,7 @@ jobs:
       - uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b  # v8.1.0
         with:
           version: "0.11.8"
+          python-version: "3.12"
       - name: Validate PR title
         env:
           PR_TITLE: ${{ github.event.pull_request.title }}

.github/workflows/check-pr-title.yml

@@ -33,9 +33,10 @@ jobs:
         uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b  # v8.1.0
         with:
           version: "0.11.8"
+          python-version: "3.12"
 
       - name: Install dependencies
-        run: uv sync --frozen
+        run: uv sync --frozen --extra deploy
 
       - name: Verify env example files
         run: uv run scripts/dev/check_env_examples.py

.github/workflows/env-examples.yml

@@ -0,0 +1,176 @@
+name: Image Publish
+
+on:
+  push:
+    branches:
+      - main
+    tags:
+      - 'v*'
+    paths-ignore:
+      - '**.md'
+      - 'docs/**'
+  pull_request:
+    paths-ignore:
+      - '**.md'
+      - 'docs/**'
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+env:
+  REGISTRY: ghcr.io
+  IMAGE_NAME: mlsys-io/lumilake_server
+
+jobs:
+  build-pr:
+    name: Build image
+    if: github.event_name == 'pull_request'
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    steps:
+      - name: Checkout
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6
+        with:
+          persist-credentials: false
+
+      - name: Set up Buildx
+        uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435  # v3.11.1
+
+      - name: Compute image tags
+        id: tags
+        env:
+          SHA: ${{ github.sha }}
+        run: |
+          set -eu
+          short_sha="${SHA::7}"
+          echo "tags=${REGISTRY}/${IMAGE_NAME}:pr-${short_sha}" >> "$GITHUB_OUTPUT"
+
+      - name: Set up uv
+        uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b  # v8.1.0
+        with:
+          version: "0.11.8"
+          python-version: "3.12"
+          # No uv cache restore in a publishing workflow (cache-poisoning).
+          enable-cache: false
+
+      - name: Sync server requirements from uv.lock
+        run: uv run --frozen scripts/dev/sync_requirements.py --write
+
+      - name: Verify lockfile sync was a no-op
+        run: |
+          if ! git diff --quiet src/lumilake_server/requirements.txt; then
+            echo "src/lumilake_server/requirements.txt drifted from uv.lock." >&2
+            echo "Run: uv run scripts/dev/sync_requirements.py --write" >&2
+            git --no-pager diff src/lumilake_server/requirements.txt
+            exit 1
+          fi
+
+      - name: Build image
+        id: build
+        uses: docker/build-push-action@1dc73863535b631f98b2378be8619f83b136f4a0  # v6.17.0
+        with:
+          context: .
+          file: ./Dockerfile
+          push: false
+          tags: ${{ steps.tags.outputs.tags }}
+          provenance: false
+          build-args: |
+            BUILD_VERSION=${{ github.ref_name }}
+            BUILD_REF=${{ github.sha }}
+            BUILD_CREATED=${{ github.event.repository.updated_at }}
+
+  build-and-push:
+    name: Build and push multi-arch image
+    if: github.event_name == 'push'
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      packages: write  # docker/login-action + build-push-action push to GHCR
+    steps:
+      - name: Checkout
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6
+        with:
+          persist-credentials: false
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@29109295f81e9208d7d86ff1c6c12d2833863392  # v3.6.0
+
+      - name: Set up Buildx
+        uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435  # v3.11.1
+
+      - name: Log in to GHCR
+        uses: docker/login-action@184bdaa0721073962dff0199f1fb9940f07167d1  # v3.5.0
+        with:
+          registry: ${{ env.REGISTRY }}
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Compute image tags
+        id: tags
+        env:
+          REF_NAME: ${{ github.ref_name }}
+          REF_TYPE: ${{ github.ref_type }}
+          SHA: ${{ github.sha }}
+        run: |
+          set -eu
+          short_sha="${SHA::7}"
+          tags=("${REGISTRY}/${IMAGE_NAME}:${short_sha}")
+          # :latest is owned by release-images.yml (gated on env approval).
+          if [[ "${REF_TYPE}" == "tag" && "${REF_NAME}" == v* ]]; then
+            tags+=("${REGISTRY}/${IMAGE_NAME}:${REF_NAME}")
+          elif [[ "${REF_NAME}" == "main" ]]; then
+            tags+=("${REGISTRY}/${IMAGE_NAME}:dev")
+          fi
+          ( IFS=,; echo "tags=${tags[*]}" ) >> "$GITHUB_OUTPUT"
+
+      - name: Set up uv
+        uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b  # v8.1.0
+        with:
+          version: "0.11.8"
+          python-version: "3.12"
+          # No uv cache restore in a publishing workflow (cache-poisoning).
+          enable-cache: false
+
+      - name: Sync server requirements from uv.lock
+        run: uv run --frozen scripts/dev/sync_requirements.py --write
+
+      - name: Verify lockfile sync was a no-op
+        run: |
+          if ! git diff --quiet src/lumilake_server/requirements.txt; then
+            echo "src/lumilake_server/requirements.txt drifted from uv.lock." >&2
+            echo "Run: uv run scripts/dev/sync_requirements.py --write" >&2
+            git --no-pager diff src/lumilake_server/requirements.txt
+            exit 1
+          fi
+
+      - name: Build and push
+        id: build
+        uses: docker/build-push-action@1dc73863535b631f98b2378be8619f83b136f4a0  # v6.17.0
+        with:
+          context: .
+          file: ./Dockerfile
+          platforms: linux/amd64,linux/arm64
+          push: true
+          tags: ${{ steps.tags.outputs.tags }}
+          provenance: false
+          build-args: |
+            BUILD_VERSION=${{ github.ref_name }}
+            BUILD_REF=${{ github.sha }}
+            BUILD_CREATED=${{ github.event.repository.updated_at }}
+
+      - name: Smoke-test pushed image
+        env:
+          SHA: ${{ github.sha }}
+        run: |
+          set -eu
+          short_sha="${SHA:0:7}"
+          image="${REGISTRY}/${IMAGE_NAME}:${short_sha}"
+          # Plain imports only — routes/jobs.py module-load init needs S3/DB.
+          docker pull "$image"
+          docker run --rm --entrypoint /usr/bin/tini "$image" -- \
+            python -c "import lumilake, lumilake_hook, lumilake_server; print('image-smoke-ok')"

.github/workflows/image-publish.yml

@@ -26,10 +26,12 @@ jobs:
       - name: Set up uv
         uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b  # v8.1.0
         with:
+          version: "0.11.8"
           python-version: "3.12"
 
       - name: Install dependencies
-        run: uv sync --all-groups
+        # --all-extras: mypy needs every workspace wheel for cross-imports.
+        run: uv sync --frozen --all-extras --all-groups
 
       - name: Run pre-commit hooks
         run: uv run pre-commit run --all-files --show-diff-on-failure

.github/workflows/lint-typecheck.yml

@@ -21,7 +21,7 @@ concurrency:
 
 jobs:
   package-build:
-    name: Build and smoke-test package
+    name: Build and smoke-test packages
     runs-on: self-hosted
     steps:
       - name: Checkout
@@ -33,12 +33,16 @@ jobs:
         uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b  # v8.1.0
         with:
           version: "0.11.8"
+          python-version: "3.12"
 
       - name: Install build environment
-        run: uv sync --all-groups --all-extras --frozen
+        run: uv sync --all-packages --all-extras --all-groups --frozen
 
       - name: Build distributions
-        run: uv build --out-dir dist
+        run: uv build --all-packages --out-dir dist
 
       - name: Verify distributions
         run: uv run scripts/ci/check_package_build.py --dist dist
+
+      - name: Check distribution metadata
+        run: uvx twine==6.2.0 check dist/*