feat: daily digest of contributor pull requests

README.md

@@ -653,6 +653,97 @@ tried. `session_util` grew a `blind15_id` covering this in
 [`7e8d126`](https://github.com/session-foundation/libsession-python/commit/7e8d126), which
 no packaged release carries yet.
 
+## Contributor Pull Requests
+
+[`github_prs/digest.py`](github_prs/digest.py) posts one message each weekday morning
+listing the open pull requests in session-foundation's repositories whose author is not
+a maintainer and which have moved in the last three days — the ones nobody on the team
+has a reason to already know about:
+
+```
+**Contributor pull requests** · last 3 days
+🟢 **2** new · ✏️ **1** updated
+**36** open from contributors across the org.
+
+**session-desktop**
+🟢 [#1958](…) @KyWB · 12h · 💬1 · Fix issue #563
+✏️ [#1904](…) @scrense-hash · 3h · 💬2 · feat: add SOCKS5 proxy support
+```
+
+🟢 is a PR the digest has never reported; ✏️ is one it has, which has moved since. A PR
+that has not moved is left out entirely, however wide the window. The backlog line
+counts every open contributor PR regardless, so a quiet day still says how much is
+waiting.
+
+### Weekdays, and the state file
+
+The timer runs `Mon..Fri`, so Monday's run has to cover the weekend — hence a 72-hour
+window rather than a daily one. That window overlaps itself by two days on every run,
+and [`--state`](github_prs/digest.py) is what stops the overlap being noise: it records
+which PRs reached Discord and what each one's `updated_at` was at the time.
+
+`updated_at` moves on *any* change, including one that touches several PRs at once — a
+label sweep, a base branch renamed — so those resurface once even though nobody worked
+on them. The accurate alternative is the head SHA and the comment counts, which are not
+in the search result and cost a request per PR that moved; this is the cheaper half of
+that trade, taken deliberately.
+
+Only what Discord accepted is recorded, so a run that fails on its second message
+re-reports that message's PRs tomorrow rather than losing them. Every way of failing to
+read the state file — missing, unreadable, written by another version — treats every PR
+in the window as new: noisy once, never wrong, which is what makes it a cache rather
+than something to back up.
+
+One search fetches every open PR in the org and the window is applied to the result
+here rather than in the query — that is what buys the backlog count for the cost of a
+single query. Past GitHub's 1000-result search ceiling the digest says the counts are a
+floor instead of failing.
+
+### Who is a maintainer
+
+[`github_prs/maintainers.txt`](github_prs/maintainers.txt), one login per line, matched
+case-insensitively. Bot accounts need no entry — every account GitHub types as a `Bot`
+is dropped, so a renamed Dependabot stays out on its own.
+
+Neither of the two things GitHub could answer this with is a substitute. Org membership
+covers six accounts, two of which are not in the review loop; push access is held by a
+dozen more as outside collaborators, several of them contractors whose PRs are exactly
+what the digest is for. Both would get it wrong in both directions, so the list is
+written by hand — and goes stale silently, since a new maintainer's PRs are reported as
+a stranger's until someone adds them.
+
+### What is left out
+
+Forks, archived repositories and private repositories, by checking the search results
+against the org's repository list rather than by name — so a repository created today
+is covered today and a fork of an upstream project never is. There is no flag to widen
+that: private repositories stay out whatever the token can see.
+
+| flag | |
+| --- | --- |
+| `--window-hours N` | how far back a PR must have moved to be considered (default 72) |
+| `--state PATH` | dedup state; without it every PR in the window is new |
+| `--state-retention-days N` | drop state entries older than this (default 30) |
+| `--dry-run` | print the Discord payload, post nothing |
+| `--org`, `--token`, `--webhook` | override the environment |
+| `--maintainers PATH` | a different list |
+
+| env var | |
+| --- | --- |
+| `GITHUB_PRS_TOKEN` | read-only token; no scope at all is needed, the digest reads public repositories only |
+| `GITHUB_PRS_DISCORD_WEBHOOK_URL` | the channel it posts to (not needed with `--dry-run`) |
+| `GITHUB_PRS_ORG` | optional; defaults to `session-foundation` |
+
+It runs on the same box as the Zendesk digest, under its own user and its own
+environment file — see [deploy/README.md](deploy/README.md). Its HTTP retries,
+Discord posting and dedup state are the same code the Zendesk digest uses, in
+[shared/](shared/).
+
+```sh
+cd github_prs && python -m unittest discover
+cd shared && python -m unittest discover
+```
+
 ## Workflow Failure Notificaiton
 
 If a workflow fails and is in the list of workflows monitored by the failure notificaiton workflow, the failure notificaiton workflow will send a message to a discord webhook.

crowdin/report_multiple_translations.py

@@ -40,16 +40,17 @@
 import collections
 import concurrent.futures
 import datetime as dt
-import email.utils
 import json
 import os
 import subprocess
 import sys
 import threading
-import time
 
 import requests
 
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+from shared import discord, retry  # noqa: E402
+
 API = "https://api.crowdin.com/api/v2"
 DEFAULT_PROJECT = "618696"
 KEYRING_ATTRS = ["service", "crowdin", "key", "translation-api-token"]
@@ -81,49 +82,10 @@ def get_token(cli_token):
              "or store it via secret-tool).")
 
 
-def parse_retry_after(value, fallback, cap=30):
-    """Seconds to wait for a Retry-After header, parsed defensively.
-
-    Supports both numeric-seconds and HTTP-date forms (RFC 7231); falls back to
-    `fallback` when the header is missing or unparseable, and clamps the result
-    to `cap` so a bogus/huge value can't stall the run."""
-    wait = fallback
-    if value is not None:
-        try:
-            wait = float(value)
-        except (TypeError, ValueError):
-            try:
-                when = email.utils.parsedate_to_datetime(value)
-                if when.tzinfo is None:
-                    when = when.replace(tzinfo=dt.timezone.utc)
-                wait = (when - dt.datetime.now(dt.timezone.utc)).total_seconds()
-            except (TypeError, ValueError):
-                wait = fallback
-    if wait < 0:
-        wait = fallback
-    return min(wait, cap)
-
-
-def request_with_retry(session, method, url, max_retries=10, **kw):
-    """Request with backoff on 429/5xx AND on network errors (flaky DNS/connection)."""
-    delay = 0.5
-    last_exc = None
-    for _ in range(max_retries):
-        try:
-            r = session.request(method, url, timeout=60, **kw)
-        except requests.exceptions.RequestException as e:
-            last_exc = e
-            time.sleep(delay)
-            delay = min(delay * 2, 30)
-            continue
-        if r.status_code == 429 or r.status_code >= 500:
-            time.sleep(parse_retry_after(r.headers.get("Retry-After"), delay))
-            delay = min(delay * 2, 30)
-            continue
-        r.raise_for_status()
-        return r
-    if last_exc:
-        raise last_exc
+def request_with_retry(session, method, url, **kw):
+    """A Crowdin API call. A non-retryable 4xx raises, so a caller can read the
+    body of what it asked for without checking the status first."""
+    r = retry.request_with_retry(session, method, url, attempts=10, timeout=60, **kw)
     r.raise_for_status()
     return r
 
@@ -366,33 +328,22 @@ def pack_embeds(embeds):
 
 
 def post_to_discord(webhook_url, messages):
-    # Use a fresh, unauthenticated session -- the Crowdin Bearer token must never
-    # be sent to Discord. request_with_retry raises on any non-retryable 4xx, so
-    # we translate that into the concise failure message here.
+    # A fresh, unauthenticated session: the Crowdin Bearer token must never be
+    # sent to Discord.
     with requests.Session() as webhook_session:
-        for payload in messages:
-            try:
-                request_with_retry(webhook_session, "POST", webhook_url, json=payload)
-            except requests.exceptions.RequestException as e:
-                resp = getattr(e, "response", None)
-                if resp is not None:
-                    detail = f"Discord webhook failed ({resp.status_code}): {resp.text[:300]}"
-                else:
-                    detail = f"Discord webhook failed: {e}"
-                # A rich payload can be rejected outright (e.g. an embed exceeded
-                # Discord's size limits). Before crashing, best-effort post a plain
-                # warning so the failure is at least visible in the channel; if even
-                # that fails, fall through to the sys.exit below.
-                try:
-                    request_with_retry(webhook_session, "POST", webhook_url, json={
-                        "content": "⚠️ Crowdin multiple-translations report failed to post "
-                                   "its results (a message was rejected by Discord). "
-                                   "Re-run `report_multiple_translations.py --json` for the "
-                                   "full list.",
-                    })
-                except requests.exceptions.RequestException:
-                    pass
-                sys.exit(detail)
+        posted = discord.post_to_discord(webhook_session, webhook_url, messages)
+        if posted == len(messages):
+            return
+        # A rich payload can be rejected outright, an embed over Discord's size
+        # limits say. A plain warning at least makes the failure visible in the
+        # channel; if that fails too, the exit below still says so.
+        discord.post_to_discord(webhook_session, webhook_url, [{
+            "content": "⚠️ Crowdin multiple-translations report failed to post "
+                       "its results (a message was rejected by Discord). "
+                       "Re-run `report_multiple_translations.py --json` for the "
+                       "full list.",
+        }])
+        sys.exit(f"Discord accepted {posted} of {len(messages)} messages.")
 
 
 # --------------------------------------------------------------------------- #

crowdin/test_report_multiple_translations.py

@@ -0,0 +1,75 @@
+"""
+    cd crowdin && python -m unittest discover
+"""
+import contextlib
+import io
+import os
+import sys
+import unittest
+
+import requests
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+import report_multiple_translations as report  # noqa: E402
+from shared.testing import FakeResponse, FakeSession, NoSleep, Patched  # noqa: E402
+
+
+class ApiResponse(FakeResponse):
+    def raise_for_status(self):
+        if self.status_code >= 400:
+            raise requests.HTTPError(f"{self.status_code}", response=self)
+
+
+class WebhookSession(FakeSession):
+    def __enter__(self):
+        return self
+
+    def __exit__(self, *exc):
+        return False
+
+
+class TestRequestWithRetry(unittest.TestCase):
+    def test_a_client_error_raises_rather_than_returning(self):
+        """Callers read the body straight off the response, so a 4xx has to stop them."""
+        with self.assertRaises(requests.HTTPError):
+            report.request_with_retry(FakeSession([ApiResponse({}, status_code=404)]),
+                                      "GET", "https://x")
+
+    def test_crowdins_budget_is_ten_attempts_at_sixty_seconds(self):
+        session = FakeSession([ApiResponse({}, status_code=503)] * 9 + [ApiResponse({"ok": 1})])
+        with NoSleep():
+            self.assertEqual(report.request_with_retry(session, "GET", "https://x").json(), {"ok": 1})
+        self.assertEqual(len(session.calls), 10)
+        self.assertEqual({kw["timeout"] for _, _, kw in session.calls}, {60})
+
+
+class TestPostToDiscord(unittest.TestCase):
+    def post(self, responses, messages):
+        session = WebhookSession(responses)
+        with Patched(report.requests, Session=lambda: session), \
+                contextlib.redirect_stdout(io.StringIO()):
+            report.post_to_discord("https://hook", messages)
+        return session
+
+    def test_every_message_accepted_posts_nothing_else(self):
+        session = self.post([FakeResponse({}, status_code=204)] * 2, [{"embeds": []}] * 2)
+        self.assertEqual(len(session.calls), 2)
+
+    def test_a_rejection_posts_a_plain_warning_then_exits(self):
+        responses = [FakeResponse({}, status_code=204), FakeResponse({}, status_code=400),
+                     FakeResponse({}, status_code=204)]
+        with self.assertRaises(SystemExit) as caught:
+            self.post(responses, [{"embeds": []}] * 3)
+        self.assertIn("1 of 3", str(caught.exception))
+
+    def test_the_warning_is_plain_content_the_webhook_cannot_reject_for_size(self):
+        session = WebhookSession([FakeResponse({}, status_code=400), FakeResponse({}, status_code=204)])
+        with Patched(report.requests, Session=lambda: session), \
+                contextlib.redirect_stdout(io.StringIO()), self.assertRaises(SystemExit):
+            report.post_to_discord("https://hook", [{"embeds": [{"title": "x" * 9000}]}])
+        self.assertEqual(list(session.calls[1][2]["json"]), ["content"])
+
+
+if __name__ == "__main__":
+    unittest.main()