Skip to content

ci: add nightly integration smoke test against lecture-python-programming #393

Description

@mmcky

Context

PR #390 migrated visual regression testing from `lecture-python-programming` to the curated `quantecon-book-theme-fixtures` repo, pinning fixtures + their build dependencies (jupyter-book, sphinx, extensions) to a specific SHA + version set. This was the right call for stability — visual diffs now attribute cleanly to theme changes.

However, this introduces a new blind spot: version drift between the pinned fixtures stack and what real QuantEcon lectures actually use today. If jupyter-book or sphinx ships a backwards-incompatible change, our CI won't notice because the fixtures repo stays on the old pinned version — but real lecture sites consuming this theme will break.

The existing `lecture-python-programming` build was useful as a real-world canary for this kind of drift. We removed it from the per-PR critical path (too slow, too noisy) but should bring it back as a non-blocking nightly smoke test.

Proposed solution

Add `.github/workflows/integration.yml` — runs nightly on a cron, builds `lecture-python-programming` from scratch with the latest released theme, fails loudly if the build breaks but does not gate any PR.

Sketch

name: Integration Smoke Test (lecture-python-programming)
on:
  schedule:
    - cron: \"0 6 * * *\"      # daily at 06:00 UTC
  workflow_dispatch:

jobs:
  smoke:
    runs-on: ubuntu-latest
    continue-on-error: true   # explicitly non-blocking
    steps:
      - uses: actions/checkout@v6

      - name: Fetch lecture-python-programming
        run: git clone --branch quantecon-book-theme https://github.com/QuantEcon/lecture-python-programming

      - name: Setup Anaconda
        uses: conda-incubator/setup-miniconda@v4
        with:
          auto-update-conda: true
          python-version: \"3.13\"
          environment-file: lecture-python-programming/environment.yml
          activate-environment: lecture-python-programming

      - name: Install theme from main
        shell: bash -l {0}
        run: pip install .

      - name: Build HTML
        shell: bash -l {0}
        run: |
          cd lecture-python-programming
          jb build lectures --path-output ./

      # Optional: open an issue if the build fails, so we don't have to
      # manually scan failed workflow runs
      - name: Open issue on failure
        if: failure()
        uses: actions/github-script@v9
        with:
          script: |
            const date = new Date().toISOString().split('T')[0];
            await github.rest.issues.create({
              owner: context.repo.owner,
              repo: context.repo.repo,
              title: \`Integration smoke failure: \${date}\`,
              body: \`Nightly lecture-python-programming build broke on \${date}.\\n\\nWorkflow run: \${context.serverUrl}/\${context.repo.owner}/\${context.repo.repo}/actions/runs/\${context.runId}\\n\\nLikely cause: jupyter-book / sphinx / extension version drift. Investigate and either bump the fixtures pin in \\\`quantecon-book-theme-fixtures\\\` or fix the theme to match.\`,
              labels: ['ci', 'integration-smoke']
            });

Why this matters

Without this:

  • A jb/sphinx version bump that breaks the theme could ship to PyPI and only get noticed when a lecture maintainer reports a broken build
  • The fixtures pin would mask the issue indefinitely on PR CI
  • Drift accumulates silently between releases

With this:

  • Nightly heartbeat against the real, current lecture-python-programming environment
  • Failures are visible (via the auto-opened issue) without spamming PR reviewers
  • Fix-forward path is clear: either bump `FIXTURES_SHA` in this repo (after refreshing the fixtures repo), or patch the theme to handle the new jb/sphinx version

Tradeoffs

Pros:

  • Closes the drift blind spot introduced by the fixtures migration
  • Non-blocking — doesn't slow down or destabilize PR workflow
  • Cheap: one nightly run ≈ 6 min of CI time
  • Self-documenting failure mode via auto-opened issue

Cons:

  • Bot-opened issues can be noisy if drift is frequent (mitigation: dedupe by checking for an existing open `integration-smoke` issue before creating a new one)
  • Requires occasional manual triage to bump fixtures or patch the theme
  • Conda + jb build is slow; can't avoid this since we're explicitly testing the full real environment

Acceptance criteria

  • `.github/workflows/integration.yml` exists and runs nightly
  • Workflow uses `continue-on-error: true` at the job level so failures don't show as a red ✗ on the repo's main branch status
  • Failure auto-opens an issue labelled `ci` + `integration-smoke` (or comments on an existing open one rather than spamming)
  • `workflow_dispatch` trigger included for manual investigation
  • Documented in `docs/developer/visual-testing.md` (the "Relationship to lecture-python-programming" section) and in `quantecon-book-theme-fixtures/MIGRATION.md`

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions