Skip to content

sources: add github_repo_events — one record per issue/PR event, with the login that acted - #513

Open
polyglotAI-bot wants to merge 1 commit into
ClickHouse:mainfrom
polyglotAI-bot:polyglot/github-repo-events
Open

polyglotAI-bot wants to merge 1 commit into
ClickHouse:mainfrom
polyglotAI-bot:polyglot/github-repo-events

Conversation

@polyglotAI-bot

Copy link
Copy Markdown
Contributor

Summary

Adds github_repo_events, an opt-in source that emits one record per issue/PR event and names the one login that acted: "alice requested a review from you on PR #7", "bob assigned you issue #9". The allow_actors guardrail on it then judges who did the thing, not who is involved in the thread.

The github (notifications) source cannot do this, and no enrichment can make it:

This source reads GET /repos/{owner}/{repo}/issues/events instead: newest first, full history, PRs included, one entry per event.

Who acted is not the actor field

Verified against live repositories:

Event Who acted What actor holds
review_requested review_requester the same login
review_request_removed review_requester (the account that removed the request, not the original requester) the same login
assigned / unassigned assigner the assignee
mentioned not recorded anywhere the user who was mentioned

So each supported type reads a fixed field (_EVENT_SPECS) and never falls back to actor. For the two review types, an event whose actor differs from review_requester is dropped. Any event without its field is dropped, never credited to a guess. mentioned cannot be attributed, so it is not supported: @-mentions stay on the notification source, which carries the comment and its author.

Behaviour

  • Config (sync.github_repo_events, off by default):
    • repos — required;
    • events — default [review_requested, assigned], also review_request_removed, unassigned;
    • only_targeting_me — default true: only events whose target is the authenticated account;
    • allow_actors / deny_actors;
    • username — set when the token cannot read /user, e.g. a GitHub App token;
    • the usual schedule, batch_size, condense.
  • Not emitted:
    • Team requests, under only_targeting_me. A team request names the team, not its members, and a member cannot be identified.
    • The account's own actions. github_events covers those.
  • Records hold only fields of the event:
    • actor, event (also as reason, for the existing GitHub renderer), target, target_type, number, subject_type, and a URL built from the repo and the number.
    • No issue title, body, state or labels. Those change after the event, and the inbox re-surfaces a stored record whose content changes. Leaving them out also keeps the author's text away from a consuming agent.
  • Cursor: a JSON object with one position (UTC timestamp) per repo. A run reads the events created after it.
    • After a read: the position becomes the newest event seen, held back to no later than the read's server time (Date header) minus 10 minutes, and it never moves backwards.
      • So the last 10 minutes are read again, and an event the API lists late is still caught. An event already stored re-reads byte-identical, which the inbox skips as unchanged.
      • A quiet repo settles on its newest event and reads nothing twice.
    • Failures: a repo whose fetch fails keeps its own position, and a failed later page emits nothing and keeps the position.
    • No backfill: a repo without a position (first run, or newly added to repos) starts at its newest event. An empty repo starts at the server's time minus 10 minutes.
  • Cost:
    • Conditional reads: after a poll that emitted nothing for a repo, the next poll sends If-None-Match. An unchanged first page answers 304, which does not count against the rate limit (verified: a repeat poll used 0).
    • Stale ETags: the ETag is sent only while the cursor still has the position that read produced. So a 304 cannot hide records whose persist failed.
    • Busy repos: each event embeds its issue, so a busy repo's first page changes often and most polls are full requests. The docs say so.
  • With the github source also on: a review request can reach the inbox twice. The docs suggest one source per consumer, or deny_reasons: ["review_requested", "assign"] on github.

Scope

  • New: nerve/sources/github_repo_events.py.
  • Wiring: GitHubRepoEventsSyncConfig in nerve/config.py, registry wiring with the actor guardrail.
  • Docs: docs/sources.md (new section, config block, and a pointer from the existing actor-guardrail section), config.example.yaml, docs/config.md, template source lists.
  • Web UI: icon, colour and renderer mapping for the new source name.
  • Installer output unchanged on purpose: the source is off by default, so nerve init has nothing to write (the golden-output tests stay as they are).
  • Not included:
    • @-mention attribution: it would need /repos/{o}/{r}/issues/comments.
    • Webhook intake: exact and real time, but it needs a public endpoint. The record shape is the same as a webhook intake would produce.
    • A shared gh helper across the GitHub sources: each source keeps its own, as they do today.

Test plan

  • tests/test_github_repo_events_source.py — 45 tests, organized by area:
    • Attribution: the doer field for each type, including assigned where actor is the assignee; a missing doer is dropped; a review actor/requester mismatch is dropped.
    • Filtering: targets on and off, team targets, own actions, configured and unsupported types.
    • Record content: no issue text, and a URL that does not change.
    • Cursor: baseline, the pre-baseline leak, an empty repo starting at the server clock, the overlap re-read (identical, and catching a late event), a position held back to the read time, a position that never moves backwards on a skewed clock, positions and event timestamps normalized to UTC, an unreadable cursor.
    • ETags: the 304 path, no ETag after an emitting poll, a quiet repo regaining 304s, no ETag for a foreign cursor.
    • Failures: a failed repo keeps its position; a newly configured repo baselines.
    • Pagination: reading to the floor, the page cap, a failed later page.
    • Account: an unknown login, login caching, username, repo dedupe.
    • gh plumbing: 304 exit handling, the Date header (including a zoneless one), kill on timeout and on cancellation.
    • Wiring: config and registry.
    • End to end: SourceRunner + database, where a re-read event keeps its rowid (not re-surfaced).
  • Mutation check: 23 deliberate breaks of the guards above, each caught by a test.
  • Live, read-only: against three public repositories.
    • A baseline emitted nothing.
    • From a cursor two days back it emitted exactly the five review requests and assignments aimed at the account, each credited to the maintainer who made it.
    • A repeat poll right after a baseline used 0 rate limit.
    • After emitting, the next poll was a full read that emitted nothing, and the poll after it used 0 again.
  • Full suite: 5792 passed, 39 skipped. npm run build passes.
  • Pre-PR review: two read-only subagent reviews.
    • The first found one blocking issue: a baseline at the newest event let the first window emit up to 10 minutes of pre-existing history. Now fixed and tested.
    • Its other points are also fixed: review_request_removed attribution (verified live, plus the actor check), ETag kept only after an empty poll, cursor normalization, a built URL, the server clock for an empty repo, a guard for non-string timestamps, kill on cancellation, username, docs corrections, and guidance for using it alongside github.
    • The second confirmed those fixes and found no blocking issues. It also prompted the final position rule. Previously the overlap was measured from the newest event, so a quiet repo whose last event was relevant re-read it on every poll and never got a 304 again. The overlap is now measured from the read's server time. The same pass normalized event timestamps and zoneless Date headers, and made the page-cap warning name the time range it skips.
    • Accepted, not changed:
      • A late-listed event with exactly the same second as a repo's position is not read.
      • asyncio.shield around the final proc.wait() was not added: the process is already killed, so a second cancellation there leaves nothing running.

🤖 Generated with Claude Code

… the login that acted

The notifications source cannot say who did something: a notification is one
record per thread, with no actor and no event id, it merges every event
between two polls into one update, and its reason keeps its value after later
activity. So its actor guardrail can only judge who is involved in a thread,
and comment-less events (assignments, review requests) can only be attributed
by guessing from the thread's history.

github_repo_events reads GET /repos/{owner}/{repo}/issues/events and emits one
record per event, naming the one login that acted, so allow_actors judges who
did it. The doer is not the `actor` field for every type: an assigned event
puts the assignee in `actor` and the doer in `assigner`, and a mentioned event
records no author at all. Each supported type (review_requested,
review_request_removed, assigned, unassigned) therefore reads a fixed field
and never falls back to `actor`; an event without it is dropped.

- Opt-in (sync.github_repo_events), default events review_requested and
  assigned, only events aimed at the authenticated account by default, never
  the account's own actions; optional username for tokens without /user.
- Records hold only immutable event fields, so a re-read is byte-identical
  and the inbox does not re-surface it; no issue title or body.
- Per-repo JSON cursor: the position is the newest event seen, held back to
  the read's server time minus 10 minutes, so late-listed events are caught
  and a quiet repo reads nothing twice. No backfill on the first run or for a
  newly added repo; a failed repo keeps its position.
- Conditional requests: a 304 costs no rate limit; the ETag is only reused
  after a poll that emitted nothing and for the position it produced.

Docs, example config, template source lists and web UI mappings updated.
Installer output is unchanged (the source is off by default).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant