diff --git a/.changeset/checkpointed-registry-ingestion.md b/.changeset/checkpointed-registry-ingestion.md new file mode 100644 index 00000000..c2e17ce0 --- /dev/null +++ b/.changeset/checkpointed-registry-ingestion.md @@ -0,0 +1,14 @@ +--- +"chkit": patch +"@chkit/plugin-ingest": patch +--- + +Allow registry resources to declare cursor checkpoints and timestamp windows alongside full scans. Upgrade the eight raw API integrations with provider-specific sync tokens, overlapping incremental ranges, and safe restart recovery, while preserving historical registry releases. Use one pipeline per installation with independent resource or configured collection streams, editable configuration, injectable clients, and meaningful resource readers. Attio creates independent object-type and list streams using existing full-sync pagination, with People and Companies enabled by default. GitHub separates eight raw resource types per repository, including independent issues, PRs, comments, reviews, and SHA/message commits with warehouse join keys; it no longer fetches PR files or assembles child arrays. Resources without reliable change filters use built-in full-sync completion; Calendar tokens and Lemlist activity intervals retain their necessary recovery state. Meet and Slack use recent rediscovery and coarse completed-page checkpoints, replaying unfinished children without retained-work queues. Circleback independently reads meetings, complete transcript snapshots, action items, people, and companies with replay-safe full pagination. + +Update the ingestion authoring skill with the registry sync quality bar, provider capability selection, meaningful stream boundaries, normalized raw ingestion by default with transformations and joins in ClickHouse, existing pagination/checkpoint primitives, coarse replay where affordable, explicit freshness coverage, durable recovery rules, and required failure/restart verification. + +Linear syncs issues, comments, projects, project updates, cycles, users, teams, issue relations, and issue history into nine independent raw destinations. Seven resources own update-time windows; relations and history use complete full reads, with history discovering all issues independently of parent progress. Preserve raw relationship IDs and complete issue label names as strings, and leave joins and metrics to ClickHouse. Populate new destinations independently and migrate retained legacy nested comments deliberately. + +Treat repeated pagination continuations as permanent provider-protocol errors so reader retries cannot repeatedly replay a cyclic collection. + +Add per-integration changelog metadata, shown in registry inspection and integration guides. Keep one draft release per integration in each PR, refresh it in place until merge, and validate release artifacts against the PR base to protect published versions and changelog history. diff --git a/.changeset/circleback-independent-raw-resources.md b/.changeset/circleback-independent-raw-resources.md new file mode 100644 index 00000000..ff0ffe89 --- /dev/null +++ b/.changeset/circleback-independent-raw-resources.md @@ -0,0 +1,5 @@ +--- +"chkit": patch +--- + +Split Circleback into independently selectable meetings, meeting transcripts, action items, people, and companies with raw destinations and replay-safe full reads. Cover all verified action-item statuses and ownership, keep transcript availability separate from successful content, retrieve complete same-resource person/company details, and represent applied tags as string names. Keep joins and transformations in ClickHouse and consolidate these changes into the integration's existing draft release. diff --git a/.changeset/google-calendar-raw-events.md b/.changeset/google-calendar-raw-events.md new file mode 100644 index 00000000..f92d99b8 --- /dev/null +++ b/.changeset/google-calendar-raw-events.md @@ -0,0 +1,5 @@ +--- +"chkit": patch +--- + +Document and verify Google Calendar's events-only raw contract: native nested event fields remain unchanged, and sparse cancellations replace recurring exceptions under stable identities. Preserve the existing recoverable page and sync-token protocol in the integration's single draft release. diff --git a/.changeset/google-meet-core-raw-resources.md b/.changeset/google-meet-core-raw-resources.md new file mode 100644 index 00000000..f5c68b1b --- /dev/null +++ b/.changeset/google-meet-core-raw-resources.md @@ -0,0 +1,5 @@ +--- +"chkit": patch +--- + +Add independent Google Meet participant, participant-session, and recording metadata streams alongside conferences, transcripts, and transcript entries. Rediscover recent conferences with a 24-hour lookback and ordinary nested pagination. Checkpoint completed discovery pages and replay their unfinished children instead of retaining conferences or per-child positions. Keep scoped identities, date-bound backfills, and bounded token recovery in the existing draft release. Recording rows retain links without downloading media; artifacts that appear after a conference leaves the lookback require an explicit backfill. diff --git a/.changeset/lemlist-core-raw-resources.md b/.changeset/lemlist-core-raw-resources.md new file mode 100644 index 00000000..09bdeda2 --- /dev/null +++ b/.changeset/lemlist-core-raw-resources.md @@ -0,0 +1,5 @@ +--- +"chkit": patch +--- + +Expand Lemlist to seven independent raw streams: campaigns, contacts, companies, campaign leads, activities, inbox conversations, and inbox messages. Use complete campaign-lead exports and full contact payloads, keep inbox reads read-only, and retain acknowledged incremental activity intervals. Preserve native relationship keys for warehouse joins and refresh the single integration draft release. diff --git a/.changeset/page-aware-pagination.md b/.changeset/page-aware-pagination.md new file mode 100644 index 00000000..95ec57c0 --- /dev/null +++ b/.changeset/page-aware-pagination.md @@ -0,0 +1,7 @@ +--- +"@chkit/plugin-ingest": patch +--- + +Change `paginate()` to yield full `{ items, next, metadata? }` pages, including empty and terminal pages. This is a breaking return-type change: readers must consume `page.items` instead of treating a page as an item array. Metadata passes through unchanged and readers can explicitly map it to checkpoint state, which the existing executor commits after destination acknowledgement. Reject repeated continuations before yielding candidate progress, including a continuation back to the initial cursor. Migrate registry readers and use page metadata for Google Calendar sync-token checkpoints without changing their saved state format. + +Simplify Calendar token recovery into the same checkpointed pagination loop and Slack temporary message pagination through full pages with native completion metadata. Remove provider cursor-equality checks now covered by the shared paginator, retaining provider validation, durable timestamp frontiers, and bounded recovery. diff --git a/.changeset/slack-independent-raw-messages.md b/.changeset/slack-independent-raw-messages.md new file mode 100644 index 00000000..0f0e42f9 --- /dev/null +++ b/.changeset/slack-independent-raw-messages.md @@ -0,0 +1,5 @@ +--- +"chkit": patch +--- + +Document Slack's three raw resources and verify messages-first ingestion and metadata execution that leaves message progress unchanged. Default message bootstrap and overlap to 24 hours, with optional explicit historical bootstrap and isolated backfills. Complete each history page's replies before checkpointing its timestamp frontier; interruptions replay the unfinished page instead of retaining thread queues. Remove automatic historical reconciliation: edits and late replies on roots outside the chosen window are excluded. Replies remain individual native messages joined through thread references in ClickHouse. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 26104484..8911f652 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -15,6 +15,8 @@ jobs: steps: - name: Checkout uses: actions/checkout@v4 + with: + fetch-depth: 0 - name: Setup Bun uses: ./.github/actions/setup diff --git a/CLAUDE.md b/CLAUDE.md index 7c275d5d..61aec71a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -101,6 +101,12 @@ A test may skip itself on `CHKIT_E2E_TARGET=obsessiondb` only for a known Obsess Uses changesets. Run `bun run changeset` to create a changeset, then `bun run version-packages` to bump versions. +### Registry integration releases + +Keep at most one new release artifact per integration in a PR. Versions already on the merged base are immutable; artifacts added by the unmerged PR are drafts. Further edits refresh that same draft version and its current `meta.chkit.changelog` entry, without adding intermediate versions. Preserve earlier released changelog entries. + +Regenerate with `bun run registry:release -- --base origin/main ` and validate with `bun run check:registry-releases -- --base origin/main`; use the actual PR base if different. Omit provider names to refresh all drafts. Do not hand-edit generated artifacts. `bun run verify` checks release history and source/artifact agreement. + ### Releasing (GitHub Actions) Versions are semver; all 10 published packages share one version (a changesets `fixed` group). A release takes two steps, so the version reaches git before npm and any failure can be re-run: diff --git a/apps/docs/src/components/RegistryApps.astro b/apps/docs/src/components/RegistryApps.astro index 0a73066a..7f50caf6 100644 --- a/apps/docs/src/components/RegistryApps.astro +++ b/apps/docs/src/components/RegistryApps.astro @@ -1,5 +1,5 @@ --- -import { registryApps } from '../data/registry'; +import { registryApps, registryStrategyLabel } from '../data/registry'; ---