ci: propose each new taxonomy version of the occupation names deck

.github/workflows/ci.yml

@@ -208,7 +208,8 @@ jobs:
           cache-dependency-path: scripts/requirements-ci.txt
       - run: pip install -r scripts/requirements-ci.txt
       # The script's own tests (scripts/test_shacl_crosscheck.py), on
-      # small documents of their own, before it checks the real ones.
+      # small documents of their own, before it checks the real ones, and
+      # the taxonomy deck generator's (scripts/test_taxonomy_deck.py).
       - run: python -m unittest discover -s scripts
       - run: python scripts/shacl_crosscheck.py
 

.github/workflows/taxonomy.yml

@@ -0,0 +1,212 @@
+# SPDX-FileCopyrightText: 2026 Anton Wiklund and Solid Memo contributors
+#
+# SPDX-License-Identifier: MIT
+
+name: Taxonomy
+
+# Once a week, bring the deck of new and changed occupation names up to the
+# newest version of Arbetsförmedlingen's labor market taxonomy
+# (docs/deck-library.md#the-taxonomy-deck). When the taxonomy has a version
+# the deck's newest version lacks, the workflow opens an issue for it, makes
+# the deck's next version with scripts/taxonomy_deck.py, checks it as CI's
+# library and crosscheck jobs do, and opens a pull request that closes the
+# issue, or updates the one already open. The pull request is opened with
+# the taxonomy app's token, so CI runs on it like on anyone's; merging it
+# through the merge queue publishes the version. A failure opens an issue.
+
+on:
+  schedule:
+    - cron: "17 6 * * 1"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: taxonomy
+  cancel-in-progress: false
+
+env:
+  BRANCH: taxonomy-deck
+
+# bash with pipefail, so a step whose script fails in a pipe fails too.
+defaults:
+  run:
+    shell: bash
+
+jobs:
+  # One small request: the taxonomy's versions, against the newest the deck
+  # records. A pull request already open for the newest version is left as
+  # it is, so the deck is not made again every week until it is merged.
+  check:
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    permissions:
+      contents: read
+      pull-requests: read
+    outputs:
+      update: ${{ steps.open.outputs.update }}
+      newest: ${{ steps.check.outputs.newest }}
+      next: ${{ steps.check.outputs.next }}
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.15"
+      - id: check
+        run: python3 scripts/taxonomy_deck.py --check | tee -a "$GITHUB_OUTPUT"
+      - id: open
+        env:
+          GH_TOKEN: ${{ github.token }}
+          UPDATE: ${{ steps.check.outputs.update }}
+          NEWEST: ${{ steps.check.outputs.newest }}
+        run: |
+          title="$(gh pr list --head "$BRANCH" --state open --json title --jq '.[0].title // ""')"
+          if [ "$UPDATE" = true ] && [[ "$title" == *"taxonomy version $NEWEST "* ]]; then
+            echo "::notice::The pull request for taxonomy version $NEWEST is open already: merge it, or close it to have it made again."
+            echo "update=false" >> "$GITHUB_OUTPUT"
+          else
+            echo "update=$UPDATE" >> "$GITHUB_OUTPUT"
+          fi
+
+  update:
+    needs: check
+    if: needs.check.outputs.update == 'true'
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    # The taxonomy app's key lives here, and the environment admits only
+    # main (docs/deck-library.md#the-taxonomy-app).
+    environment: labour-market-deck
+    permissions:
+      contents: read
+    env:
+      NEWEST: ${{ needs.check.outputs.newest }}
+      NEXT: ${{ needs.check.outputs.next }}
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.15"
+      # A published taxonomy version never changes, so its snapshot is kept;
+      # the script still checks each against the checksum the deck records.
+      - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        with:
+          path: ~/.cache/taxonomy
+          key: taxonomy-${{ needs.check.outputs.newest }}
+          restore-keys: taxonomy-
+      # The deck's newest version first: its cards must come out the same
+      # from the taxonomy versions it records, or the script, the Python it
+      # runs on or a published taxonomy version has changed, and the next
+      # version would say what it should not.
+      - run: python3 scripts/taxonomy_deck.py --verify --cache-dir ~/.cache/taxonomy
+      - run: >-
+          python3 scripts/taxonomy_deck.py --to "$NEWEST" --cache-dir ~/.cache/taxonomy
+          --notes-file "$RUNNER_TEMP/notes.txt"
+
+      # The checks of CI's library and crosscheck jobs, before anything is
+      # pushed; the pull request then runs all of CI. Nothing installed here
+      # runs with a token that can write (docs/releasing.md#the-deploy-key).
+      - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
+        with:
+          node-version-file: .nvmrc
+          cache: npm
+      - run: npm ci --ignore-scripts
+      - run: npm run format:turtle
+      - run: npm run library
+      - run: npm run library:check -- --base HEAD
+      - run: pip install -r "$GITHUB_WORKSPACE/scripts/requirements-ci.txt"
+        working-directory: ${{ runner.temp }}
+      - run: python scripts/shacl_crosscheck.py
+
+      # The token is made only now, for the steps that open the issue, push
+      # and open the pull request, and lasts an hour.
+      - id: app
+        uses: actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510 # v3.3.0
+        with:
+          client-id: ${{ vars.LABOUR_MARKET_DECK_CLIENT_ID }}
+          private-key: ${{ secrets.LABOUR_MARKET_DECK_PRIVATE_KEY }}
+          permission-contents: write
+          permission-issues: write
+          permission-pull-requests: write
+      # Every commit names its issue (CONTRIBUTING.md#commit-messages): the
+      # taxonomy version's own, opened here if it is not open yet. The pull
+      # request closes it, and those of earlier versions still open, since
+      # the new version of the deck covers them too. The open issues are
+      # listed, not searched: GitHub's search finds a new issue only later.
+      - id: issue
+        env:
+          GH_TOKEN: ${{ steps.app.outputs.token }}
+        run: |
+          title="Taxonomy version $NEWEST is out"
+          gh api --paginate "repos/$GITHUB_REPOSITORY/issues?state=open&per_page=100" \
+            --jq '.[] | select(.pull_request | not) | select(.title | test("^Taxonomy version [0-9]+ is out$")) | "\(.number) \(.title)"' \
+            > "$RUNNER_TEMP/issues.txt"
+          number="$(awk -v title="$title" '{ n = $1; $1 = ""; if (substr($0, 2) == title) { print n; exit } }' "$RUNNER_TEMP/issues.txt")"
+          if [ -z "$number" ]; then
+            url="$(gh issue create --title "$title" --label deck --body "Arbetsförmedlingen has published version $NEWEST of its labor market taxonomy. The [taxonomy workflow]($GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID) makes version $NEXT of the deck of new and changed occupation names from it, in a pull request that closes this issue (docs/deck-library.md#the-taxonomy-deck).")"
+            number="${url##*/}"
+          fi
+          {
+            echo "closes<<EOF"
+            { echo "$number"; cut -d ' ' -f 1 "$RUNNER_TEMP/issues.txt"; } | sort -nu | sed 's/^/Closes: #/'
+            echo "EOF"
+          } >> "$GITHUB_OUTPUT"
+      - env:
+          GH_TOKEN: ${{ steps.app.outputs.token }}
+          APP: ${{ steps.app.outputs.app-slug }}
+          CLOSES: ${{ steps.issue.outputs.closes }}
+        run: |
+          id="$(gh api "/users/$APP%5Bbot%5D" --jq .id)"
+          export GIT_AUTHOR_NAME="${APP}[bot]" GIT_AUTHOR_EMAIL="${id}+${APP}[bot]@users.noreply.github.com"
+          export GIT_COMMITTER_NAME="$GIT_AUTHOR_NAME" GIT_COMMITTER_EMAIL="$GIT_AUTHOR_EMAIL"
+          title="feat(decks): add taxonomy version $NEWEST to the occupation names deck"
+          why="Arbetsförmedlingen has published version $NEWEST of its labor market taxonomy. This is version $NEXT of the deck of new and changed occupation names, made from it by the weekly taxonomy workflow, as the deck promises one version per taxonomy version (docs/deck-library.md#the-taxonomy-deck)."
+          git switch -c "$BRANCH"
+          git add decks
+          # The body wrapped at 72 columns (docs/style.md#commits); the pull
+          # request's, Markdown, is not.
+          git commit -m "$title" -m "$(fold -s -w 72 <<< "$why" | sed 's/ *$//')" \
+            -m "$(fold -s -w 72 "$RUNNER_TEMP/notes.txt" | sed 's/ *$//')" -m "$CLOSES"
+          git push --force "https://x-access-token:$GH_TOKEN@github.com/$GITHUB_REPOSITORY.git" "HEAD:refs/heads/$BRANCH"
+          body="$(cat <<EOF
+          ## What and why
+
+          $why
+
+          $(cat "$RUNNER_TEMP/notes.txt")
+
+          Made by [the taxonomy workflow]($GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID), which first made the cards of the deck's newest version again from the taxonomy and found them unchanged, then checked the new version as CI's library and crosscheck jobs do. Merging it publishes the version.
+
+          $CLOSES
+          EOF
+          )"
+          if [ "$(gh pr list --head "$BRANCH" --state open --json number --jq length)" = "0" ]; then
+            gh pr create --head "$BRANCH" --title "$title" --body "$body" --label deck
+          else
+            gh pr edit "$BRANCH" --title "$title" --body "$body"
+          fi
+
+  failed:
+    needs: [check, update]
+    if: failure()
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    permissions:
+      issues: write
+    steps:
+      - env:
+          GH_TOKEN: ${{ github.token }}
+          GH_REPO: ${{ github.repository }}
+        run: |
+          title="The weekly taxonomy deck update failed"
+          run="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"
+          open="$(gh issue list --state open --search "\"$title\" in:title" --json number --jq '.[0].number // ""')"
+          if [ -n "$open" ]; then
+            gh issue comment "$open" --body "It failed again: $run"
+          else
+            gh issue create --title "$title" --body "See $run. The script stops when the deck's newest version does not come out the same from the taxonomy, when a published taxonomy version has changed, or when the taxonomy is not as it expects; the checks stop when the new version would not be valid (docs/deck-library.md#the-taxonomy-deck)."
+          fi

GOVERNANCE.md

@@ -142,7 +142,10 @@ are pinned to a commit, not a tag, which their owners could move, and
 Renovate updates them as it does every other dependency. Renovate
 cannot approve its own pull requests: once `main` requires an approving
 review, a maintainer approves each update, and Renovate merges it after
-that.
+that. The weekly Taxonomy workflow likewise proposes each new version of
+the deck it generates as a pull request, which CI checks like any other
+and nothing merges on its own
+([docs/deck-library.md](docs/deck-library.md#the-taxonomy-deck)).
 
 ## Deploying
 

cspell.config.yaml

@@ -125,6 +125,7 @@ words:
   - xone
   # Products, projects, organizations and people
   - antwika
+  - Arbetsförmedlingen
   - clownface
   - cmark
   - Codeberg

docs/deck-library.md

@@ -244,6 +244,121 @@ escaped (a carriage return as `\r`), and so is any quote that would end
 or run into a delimiter, so the text reads back the same. Every other
 literal stays on one line.
 
+## The taxonomy deck
+
+The deck of new and changed Swedish occupation names,
+[`swedish-labour-market-taxonomy-occupation-names`](../decks/swedish-labour-market-taxonomy-occupation-names/),
+is generated, not written: one version per version of the labor market
+taxonomy of Arbetsförmedlingen, the Swedish Public Employment Service,
+which its JobTech taxonomy API serves under CC0 1.0, as the deck is.
+[scripts/taxonomy_deck.py](../scripts/taxonomy_deck.py) makes it, with
+Python's standard library alone. Its docstring says which names become
+cards, and why, and how it handles the taxonomy's inconsistencies.
+
+The script reads the deck's newest version, `v<N>.ttl`, and writes
+`v<N+1>.ttl` beside it, covering every taxonomy version up to the newest
+(or `--to`). From `v<N>.ttl` it carries over, verbatim, what it cannot
+make itself: the title, description and keywords in every language of
+the app, the creator's name (never an email), and the review rounds
+that made those texts, by which the new version's `<#generation>` is
+`prov:wasInformedBy`. It keeps every card of `v<N>.ttl`, and retires one
+that no rule makes any longer. It writes the rest of what
+[a new version](#publishing-a-new-version) needs: `dcat:version`,
+`dcat:prev` and `dcat:previousVersion`, `dcterms:issued` (`--issued`,
+by default today in UTC), the version notes, the publisher, the series,
+the [contact](#contact) and the Turtle distribution.
+
+A version is reproducible. Its [provenance](#provenance) records the
+script at the commit it ran from, with its SHA-256 and the Python and
+Unicode versions, the command line, and every taxonomy version read,
+with the query that fetched it and a checksum of the answer. Running
+that command at that commit makes the version again, byte for byte. So
+the script refuses to run unless it is committed and unmodified
+(`--allow-uncommitted` writes a draft), and stops when a taxonomy
+version the deck records no longer matches its checksum.
+
+- `--check` writes nothing, and prints the newest taxonomy version, the
+  newest one the deck covers, whether there is a version to add, and
+  the deck version that would hold it, as `key=value` lines.
+- `--verify` writes nothing, makes the cards of the deck's newest
+  version again from the taxonomy versions it records, and fails unless
+  they come out byte for byte the same.
+
+To make a version by hand, from the repository root on an up-to-date
+`main`:
+
+```sh
+python3 scripts/taxonomy_deck.py --verify --cache-dir ~/.cache/taxonomy
+python3 scripts/taxonomy_deck.py --cache-dir ~/.cache/taxonomy
+npm run format:turtle
+npm run library
+npm run library:check -- --base origin/main
+```
+
+The script prints what it included, left out and adjusted, then the
+version notes. Commit the new version with the index as a
+`feat(decks)` commit that names its issue
+([CONTRIBUTING.md](../CONTRIBUTING.md#commit-messages)), as for
+[any new version](#publishing-a-new-version).
+
+The [Taxonomy workflow](../.github/workflows/taxonomy.yml) does this
+every Monday, and by hand (Actions › Taxonomy › Run workflow). Its
+`check` job runs `--check`. When the taxonomy has a version the deck
+lacks, its `update` job runs `--verify`, makes the next version, and
+checks it as CI's `library` and `crosscheck` jobs do. Only then does it
+open the issue "Taxonomy version N is out" (label `deck`), unless one is
+open already, and push a commit to the branch `taxonomy-deck` that
+closes it, and the issues of earlier taxonomy versions still open, with
+`Closes:` lines. It opens a pull request from that branch, or updates
+the one already open. A pull request already open for the newest
+taxonomy version is left alone: merge it, or close it to have it made
+again. Nothing merges it on its own. Once it is merged through the merge
+queue ([testing.md](testing.md#the-merge-queue)), the next release
+publishes the version, as it does any other. When a job fails, the
+`failed` job opens the issue "The weekly taxonomy deck update failed",
+or comments on the one open, with a link to the run.
+
+### The taxonomy app
+
+The `update` job pushes and opens the pull request with the token of a
+GitHub App, not the workflow's own. A pull request opened with the
+workflow's token starts no workflow, so CI would never run on it,
+`ci-ok` would never report, and it could not be merged. The App's token
+starts workflows as anyone's does. Only this workflow can use it: the
+App's private key is a secret of the `labour-market-deck` environment,
+which allows only `main`, as the [deploy key](releasing.md#the-deploy-key)'s
+environment does.
+
+A maintainer sets it up once:
+
+1. Make a GitHub App owned by the `solid-memo` organization
+   (Organization settings › Developer settings › GitHub Apps › New
+   GitHub App), named `solid-memo-labour-market-deck`, with no webhook ("Active" unchecked), and the
+   repository permissions Contents, Issues and Pull requests, each
+   "Read and write" (Metadata "Read" comes with them), and no other.
+2. Install it on `solid-memo/website` alone ("Only select
+   repositories").
+3. Generate a private key in the App's settings.
+4. Make the environment `labour-market-deck` (Settings › Environments
+   › New environment), or open it if a run has made it already: GitHub
+   makes it, with no rule, when a job first names it. Under "Deployment
+   branches and tags", choose "Selected branches and tags" and add the
+   branch `main`, and nothing else. Add the App's client ID as its
+   environment variable `LABOUR_MARKET_DECK_CLIENT_ID`, and the private
+   key as its environment secret `LABOUR_MARKET_DECK_PRIVATE_KEY`. Then
+   delete the key's file, and leave no repository variable or secret of
+   either name: a job whose environment lacks one reads the
+   repository's.
+
+The job makes the token only after the script and the checks have run,
+for the steps that open the issue, push and open the pull request, and
+the token lasts an hour. The App needs no bypass of the ruleset on
+`main`: it pushes only the branch `taxonomy-deck`. Until the App is set
+up, the `update` job fails at the step that makes the token, and the
+failure issue links that run. To change the key, generate a new one in
+the App's settings, replace the environment's secret, then delete the
+old key there.
+
 ## Authoring Markdown
 
 A card, a course step or a chapter may be written in Markdown

docs/glossary.md

@@ -224,8 +224,10 @@ it most. The specifications' versions and status are in
   [testing.md](testing.md).
 - **Environment**: a GitHub setting that a job of a workflow names, and
   that lets it run only on the branches it allows: `github-pages`, to
-  deploy the site, and `release`, whose secret is the release's deploy
-  key, both for `main` alone. [releasing.md](releasing.md#the-deploy-key).
+  deploy the site, `release`, whose secret is the release's deploy key,
+  and `labour-market-deck`, whose secret is the taxonomy app's key, all
+  three for `main` alone. [releasing.md](releasing.md#the-deploy-key),
+  [deck-library.md](deck-library.md#the-taxonomy-app).
 - **ETag**: a server's tag for one version of a document, which changes
   whenever the document does. The app sends it back, so that it does
   not write over a newer version or read again what it already has.