Skip to content

[FEAT repo-guard] migrate repo+team bulk fetching to GraphQL to eliminate REST N+1 pattern #149

Description

@onuryilmaz

Part of epic #150. Sibling task: #141 (ETag caching for the REST endpoints that remain after this migration).

Scope

This issue is narrowly focused on one call site: ExtendedList() in internal/github/repos.go. It replaces the REST N+1 pattern (1 + N calls for an org with N repos) with a single paginated GraphQL query.

Other GraphQL opportunities (org members, team members) are explicitly not part of this issue — REST cost there is already low and is addressed by ETag caching in #141.

A detailed design has been drafted in docs/graphql-migration-plan.md.


Problem

ExtendedList() performs the following on every reconcile cycle:

GET /orgs/{org}/repos          → 1 call (paginated, 100 per page)
  for each repo:
    GET /repos/{org}/{repo}/teams  → N calls (one per repo)

For an org with 300 repositories this is 301+ REST calls per reconcile. Each call counts against the primary rate limit and returns 200 OK even when nothing changed.

Proposed Solution

Replace the REST N+1 pattern with a single paginated GraphQL query that fetches all repositories and their team permissions at once.

GraphQL query

query OrgRepositoriesWithTeams($org: String!, $repoCursor: String, $teamCursor: String) {
  organization(login: $org) {
    repositories(first: 100, after: $repoCursor, orderBy: {field: NAME, direction: ASC}) {
      pageInfo { hasNextPage endCursor }
      nodes {
        name
        isPrivate
        teams(first: 100, after: $teamCursor) {
          pageInfo { hasNextPage endCursor }
          edges {
            permission   # ADMIN | MAINTAIN | WRITE | TRIAGE | READ
            node { slug }
          }
        }
      }
    }
  }
}

Permission enum mapping

GraphQL enum REST string GithubTeamPermission
ADMIN admin admin
MAINTAIN maintain maintain
WRITE push push
TRIAGE triage triage
READ pull pull

WRITEpush is the critical mapping difference.

Implementation Plan

  1. Add graphqlClient *githubv4.Client to DefaultRepositoryProvider (uses github.com/shurcooL/githubv4, already in go.mod).
  2. Construct the GraphQL client in NewRepositoryProvider reusing the same authenticated HTTP client used by go-github (same pattern as users.go:149).
  3. New file internal/github/repos_graphql.go containing query struct definitions and pagination logic.
  4. New method ExtendedListGraphQL(ctx) on RepositoryProvider returning the same []GithubRepository shape as today.
  5. Add a permission converter graphqlPermissionToTeamPermission.
  6. Add GraphQL-specific rate-limit error detection in internal/controller/utils.go (the shurcooL/githubv4 library returns graphql.Errors rather than HTTP 403).
  7. Wire GithubOrganizationReconciler.Reconcile() to call ExtendedListGraphQL instead of ExtendedList (optionally behind a feature flag during rollout).

Files to Change

File Change
internal/github/repos.go Add graphqlClient field; update NewRepositoryProvider; add ExtendedListGraphQL; add permission converter
internal/github/repos_graphql.go New file: GraphQL query struct + pagination loop
internal/controller/githuborganization_controller.go Replace ExtendedList call
internal/controller/utils.go GraphQL rate-limit error detection
internal/github/repos_test.go Unit tests for converter + mock-based tests

What Does NOT Change

  • repoChangeCalculator — operates on []GithubRepository; unaffected by data source.
  • GithubRepository / GithubTeamWithPermission types — no schema changes.
  • Mutation calls (RepositoryTeamAdd, RepositoryTeamRemove) — REST is correct for writes; GraphQL mutations for team-repo permissions are not available in the current GitHub GraphQL schema.
  • GithubTeamRepository (exception CRDs) — fetched from Kubernetes.

Out of Scope (handled elsewhere)

Expected API Call Reduction

Scenario REST calls today GraphQL calls after
100 repos, ≤100 teams each 1 + 100 = 101 1–2
300 repos, ≤100 teams each 3 + 300 = 303 3–4
500 repos, ≤100 teams each 5 + 500 = 505 5–6

GitHub GraphQL is point-based (5,000 points/hour for App installations); a single read-only query costs 1 point regardless of node count, so the reduction in primary rate-limit pressure is dramatic.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    backlogReady for sprint planning; triggers project additionenhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    Status
    Sprint Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions