Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .conformance-catalog-ref
Original file line number Diff line number Diff line change
@@ -1 +1 @@
b4c758a7dac698d7fcacd32dafcd4bb2f5dbddaf
583a6d92412543ea352251c88f15f2c5a39d2593
361 changes: 361 additions & 0 deletions .github/scripts/conformance-case-body-drift.sh

Large diffs are not rendered by default.

660 changes: 660 additions & 0 deletions .github/scripts/conformance-case-body-drift.test.sh

Large diffs are not rendered by default.

123 changes: 123 additions & 0 deletions .github/scripts/conformance-registered-case-ids.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
#!/usr/bin/env python3
"""Print the conformance case ids this SDK registers, one per line.

This is the repo-specific half of the case-body drift check: the id source
depends on how this repo's harness records registrations, so it lives here and
``conformance-case-body-drift.sh`` stays generic.

The ids come out of ``conformance-report.json``, which ``conformance-tests/
conftest.py`` writes from ``pytest_sessionfinish``. That is the harness's own
record of what registered, so it cannot disagree with what the suite actually
did -- the reason for reading the report rather than grepping
``@pytest.mark.conformance(...)`` out of the test sources, which would be a
second, weaker id extractor that reports what it matched and stays silent about
what it missed.

The report lists one entry per CATALOG case, so presence in it is not
registration. ``nodeid`` is: ``pytest_runtest_logreport`` sets it when a marked
test runs, and the placeholder entries ``pytest_sessionfinish`` synthesizes for
uncovered catalog cases are ``{"case_id": ..., "status": "not_run"}`` with no
``nodeid`` key at all. Reading ``nodeid`` rather than ``status`` matters -- a
registered case whose test failed or skipped is still registered, and still
needs its body watched, but its status is not ``passed``.

Requires the suite to have run, so the report on disk belongs to this commit.

Reading the report with Python rather than jq keeps the consumer in the same
language as the producer: one parse, and every shape guard below phrased against
the structure ``conftest.py`` actually writes.

Inputs (environment):
CONFORMANCE_REPORT path to conformance-report.json
(default: $GITHUB_WORKSPACE/conformance-report.json)

Exit status:
0 ids printed on stdout
1 the report is missing, unreadable, or holds no registered case
"""

from __future__ import annotations

import json
import os
import sys
from pathlib import Path
from typing import Any, NoReturn, cast


def fail(message: str) -> NoReturn:
print(f"::error::{message}", file=sys.stderr)
raise SystemExit(1)


def as_object(value: Any, what: str) -> dict[str, Any]:
"""Return a decoded JSON value as an object, or fail naming what was read.

The cast is sound after the isinstance: JSON object keys are always strings.
"""
if not isinstance(value, dict):
fail(f"registered case ids: {what} is not a JSON object.")
return cast("dict[str, Any]", value)


def main() -> None:
workspace = os.environ.get("GITHUB_WORKSPACE", ".")
report_path = Path(os.environ.get("CONFORMANCE_REPORT", f"{workspace}/conformance-report.json"))

if not report_path.is_file():
fail(
f"registered case ids: '{report_path}' does not exist. conformance-tests/conftest.py "
"writes it from pytest_sessionfinish, so either the suite did not run or it failed "
"before the report was written."
)

try:
decoded: Any = json.loads(report_path.read_text(encoding="utf-8"))
except (OSError, ValueError) as exc:
fail(f"registered case ids: '{report_path}' could not be read as JSON: {exc}")

payload = as_object(decoded, f"the top level of '{report_path}'")

cases: Any = payload.get("cases")
if not isinstance(cases, list) or not cases:
fail(f"registered case ids: '{report_path}' has no non-empty 'cases' array.")

ids: list[str] = []
for raw_entry in cast("list[Any]", cases):
entry = as_object(raw_entry, f"a case entry in '{report_path}'")

# An entry with a case_id that is not a string, or empty, would silently
# drop out of the filter below and take a real registration with it.
case_id: Any = entry.get("case_id")
if not isinstance(case_id, str) or not case_id:
fail(
f"registered case ids: '{report_path}' holds a case entry with a missing or "
"non-string case_id."
)

# Absent or empty is the placeholder shape: a catalog case with no
# marker behind it. Present but not a string is a report this script
# will not guess at, because the guess would be about which cases stop
# being guarded.
nodeid: Any = entry.get("nodeid")
if nodeid is None or nodeid == "":
continue
if not isinstance(nodeid, str):
fail(
f"registered case ids: '{report_path}' holds a non-string nodeid on case "
f"'{case_id}'."
)
ids.append(case_id)

if not ids:
fail(
f"registered case ids: '{report_path}' records no case with a nodeid, so no "
"@pytest.mark.conformance marker registered. Any check restricted to this list "
"would be vacuously green."
)

print("\n".join(ids))


if __name__ == "__main__":
main()
76 changes: 76 additions & 0 deletions .github/scripts/fetch-conformance-catalog.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
#!/usr/bin/env bash
#
# Fetch the conformance catalog at the revision this repo pins.
#
# Generic: nothing here is specific to one workflow or one caller's layout.
# Every workflow in this repo that needs the pinned catalog runs this one
# script, so a guard tightened here is tightened for all of them.
#
# The catalog lives in github.com/AuthPlane/conformance — a public repo, updated
# independently of this one — so cloning its default branch would let a catalog
# change turn an unrelated PR red here. The ref is pinned instead, single-sourced
# from the tracked .conformance-catalog-ref at the repo root: bump it when
# adopting new catalog cases, together with the coverage for them, so a catalog
# change can never break CI on its own.
#
# This script exists because the read/guard/fetch sequence is needed by more than
# one workflow (ci.yml, release.yml and conformance-catalog-drift.yml). Kept
# inline in each, the guard could be tightened in one and not the others; the pin
# would be single-sourced but the logic reading it would not.
#
# Clones into $RUNNER_TEMP — outside $GITHUB_WORKSPACE — so the catalog stays out
# of the working tree: it must never be picked up by this repo's own build, test
# or coverage tooling, and `git add -A` in the release commit must never stage it
# as an embedded gitlink.
#
# Plain git over HTTPS is enough: the repo is public and read-only here, so there
# is no token to plumb and no third-party action surface to SHA-pin.
#
# Requires: GITHUB_WORKSPACE, RUNNER_TEMP.
#
# Optional: CONFORMANCE_CATALOG_DEST overrides the clone directory. A caller that
# needs the pinned catalog and the catalog tip side by side in the same job
# cannot let both land on the default path. Every other caller leaves it unset
# and gets $RUNNER_TEMP/conformance.

set -euo pipefail

: "${GITHUB_WORKSPACE:?GITHUB_WORKSPACE must be set}"
: "${RUNNER_TEMP:?RUNNER_TEMP must be set}"

REF_FILE="$GITHUB_WORKSPACE/.conformance-catalog-ref"
DEST="${CONFORMANCE_CATALOG_DEST:-$RUNNER_TEMP/conformance}"
CATALOG_REPO="https://github.com/AuthPlane/conformance.git"
CATALOG_FILE="oauth-sdk-conformance-catalog.yaml"

if [[ ! -f "$REF_FILE" ]]; then
echo "::error::$REF_FILE is missing; the conformance catalog revision is unpinned"
exit 1
fi

CONFORMANCE_CATALOG_REF="$(tr -d '[:space:]' < "$REF_FILE")"

# Guard against un-pinning BEFORE the fetch: the ref must be a full commit SHA,
# not a branch or tag name, either of which would silently track a moving target.
if ! grep -Eq '^[0-9a-f]{40}$' <<< "$CONFORMANCE_CATALOG_REF"; then
echo "::error::.conformance-catalog-ref must be a 40-hex commit SHA, got '$CONFORMANCE_CATALOG_REF'"
exit 1
fi

git init -q "$DEST"
if ! git -C "$DEST" fetch --depth=1 "$CATALOG_REPO" "$CONFORMANCE_CATALOG_REF"; then
echo "::error::Pinned conformance catalog ref $CONFORMANCE_CATALOG_REF is unreachable"
exit 1
fi
git -C "$DEST" checkout -q FETCH_HEAD

# The alignment assertion hard-fails when CONFORMANCE_CATALOG_PATH points at a
# missing file, but it reports that as a harness problem rather than drift. Fail
# here instead, where the cause is unambiguous: the fetch succeeded and the
# catalog still is not where every caller expects it.
if [[ ! -f "$DEST/$CATALOG_FILE" ]]; then
echo "::error::$CATALOG_FILE is not in the catalog at $CONFORMANCE_CATALOG_REF; the fetch succeeded but produced no catalog in $DEST"
exit 1
fi

echo "Conformance catalog checked out at $CONFORMANCE_CATALOG_REF in $DEST"
30 changes: 10 additions & 20 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,28 +26,18 @@ jobs:
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2

# AuthPlane/conformance is the public sibling repo carrying the shared
# oauth-sdk-conformance-catalog.yaml. Cloned to $RUNNER_TEMP — outside
# $GITHUB_WORKSPACE — so the catalog stays out of the working tree,
# matching release.yml's invariant. Plain git clone is enough:
# actions/checkout disallows paths outside the workspace, and we don't
# need its auth/persist-credentials features for a public read-only repo.
# oauth-sdk-conformance-catalog.yaml, pinned by SHA in the tracked
# .conformance-catalog-ref (read from the checked-out workspace, so the
# Checkout step above must precede this one).
#
# The read/guard/fetch sequence lives in the script rather than inline
# here: release.yml and the drift workflow need the same three lines, and
# inline in each the 40-hex-SHA guard could be tightened in one copy and
# not the others. The script clones to $RUNNER_TEMP — outside
# $GITHUB_WORKSPACE — so the catalog stays out of the working tree.
- name: Clone shared conformance catalog (out of tree)
if: matrix.package == 'root'
run: |
# Conformance catalog pinned by SHA, single-sourced from the tracked
# .conformance-catalog-ref at the repo root (read from the checked-out
# workspace, so the Checkout step above must precede this one). Bump
# that file when adopting new catalog cases, together with the SDK-side
# conformance coverage, so a catalog change can never break CI on its
# own. Source: github.com/AuthPlane/conformance.
CONFORMANCE_CATALOG_REF="$(cat "$GITHUB_WORKSPACE/.conformance-catalog-ref")"
grep -Eq '^[0-9a-f]{40}$' <<<"$CONFORMANCE_CATALOG_REF" \
|| { echo "::error::.conformance-catalog-ref must be a 40-hex commit SHA"; exit 1; }
git init -q "$RUNNER_TEMP/conformance"
git -C "$RUNNER_TEMP/conformance" \
fetch --depth=1 https://github.com/AuthPlane/conformance.git "$CONFORMANCE_CATALOG_REF" \
|| { echo "::error::Pinned conformance catalog ref $CONFORMANCE_CATALOG_REF is unreachable"; exit 1; }
git -C "$RUNNER_TEMP/conformance" checkout -q FETCH_HEAD
run: .github/scripts/fetch-conformance-catalog.sh

- name: Setup Python
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
Expand Down
Loading
Loading