Add recoverable raw resource streams and integration changelogs

.changeset/checkpointed-registry-ingestion.md

@@ -0,0 +1,14 @@
+---
+"chkit": patch
+"@chkit/plugin-ingest": patch
+---
+
+Allow registry resources to declare cursor checkpoints and timestamp windows alongside full scans. Upgrade the eight raw API integrations with provider-specific sync tokens, overlapping incremental ranges, and safe restart recovery, while preserving historical registry releases. Use one pipeline per installation with independent resource or configured collection streams, editable configuration, injectable clients, and meaningful resource readers. Attio creates independent object-type and list streams using existing full-sync pagination, with People and Companies enabled by default. GitHub separates eight raw resource types per repository, including independent issues, PRs, comments, reviews, and SHA/message commits with warehouse join keys; it no longer fetches PR files or assembles child arrays. Resources without reliable change filters use built-in full-sync completion; Calendar tokens and Lemlist activity intervals retain their necessary recovery state. Meet and Slack use recent rediscovery and coarse completed-page checkpoints, replaying unfinished children without retained-work queues. Circleback independently reads meetings, complete transcript snapshots, action items, people, and companies with replay-safe full pagination.
+
+Update the ingestion authoring skill with the registry sync quality bar, provider capability selection, meaningful stream boundaries, normalized raw ingestion by default with transformations and joins in ClickHouse, existing pagination/checkpoint primitives, coarse replay where affordable, explicit freshness coverage, durable recovery rules, and required failure/restart verification.
+
+Linear syncs issues, comments, projects, project updates, cycles, users, teams, issue relations, and issue history into nine independent raw destinations. Seven resources own update-time windows; relations and history use complete full reads, with history discovering all issues independently of parent progress. Preserve raw relationship IDs and complete issue label names as strings, and leave joins and metrics to ClickHouse. Populate new destinations independently and migrate retained legacy nested comments deliberately.
+
+Treat repeated pagination continuations as permanent provider-protocol errors so reader retries cannot repeatedly replay a cyclic collection.
+
+Add per-integration changelog metadata, shown in registry inspection and integration guides. Keep one draft release per integration in each PR, refresh it in place until merge, and validate release artifacts against the PR base to protect published versions and changelog history.

.changeset/circleback-independent-raw-resources.md

@@ -0,0 +1,5 @@
+---
+"chkit": patch
+---
+
+Split Circleback into independently selectable meetings, meeting transcripts, action items, people, and companies with raw destinations and replay-safe full reads. Cover all verified action-item statuses and ownership, keep transcript availability separate from successful content, retrieve complete same-resource person/company details, and represent applied tags as string names. Keep joins and transformations in ClickHouse and consolidate these changes into the integration's existing draft release.

.changeset/google-calendar-raw-events.md

@@ -0,0 +1,5 @@
+---
+"chkit": patch
+---
+
+Document and verify Google Calendar's events-only raw contract: native nested event fields remain unchanged, and sparse cancellations replace recurring exceptions under stable identities. Preserve the existing recoverable page and sync-token protocol in the integration's single draft release.

.changeset/google-meet-core-raw-resources.md

@@ -0,0 +1,5 @@
+---
+"chkit": patch
+---
+
+Add independent Google Meet participant, participant-session, and recording metadata streams alongside conferences, transcripts, and transcript entries. Rediscover recent conferences with a 24-hour lookback and ordinary nested pagination. Checkpoint completed discovery pages and replay their unfinished children instead of retaining conferences or per-child positions. Keep scoped identities, date-bound backfills, and bounded token recovery in the existing draft release. Recording rows retain links without downloading media; artifacts that appear after a conference leaves the lookback require an explicit backfill.

.changeset/lemlist-core-raw-resources.md

@@ -0,0 +1,5 @@
+---
+"chkit": patch
+---
+
+Expand Lemlist to seven independent raw streams: campaigns, contacts, companies, campaign leads, activities, inbox conversations, and inbox messages. Use complete campaign-lead exports and full contact payloads, keep inbox reads read-only, and retain acknowledged incremental activity intervals. Preserve native relationship keys for warehouse joins and refresh the single integration draft release.

.changeset/page-aware-pagination.md

@@ -0,0 +1,7 @@
+---
+"@chkit/plugin-ingest": patch
+---
+
+Change `paginate()` to yield full `{ items, next, metadata? }` pages, including empty and terminal pages. This is a breaking return-type change: readers must consume `page.items` instead of treating a page as an item array. Metadata passes through unchanged and readers can explicitly map it to checkpoint state, which the existing executor commits after destination acknowledgement. Reject repeated continuations before yielding candidate progress, including a continuation back to the initial cursor. Migrate registry readers and use page metadata for Google Calendar sync-token checkpoints without changing their saved state format.
+
+Simplify Calendar token recovery into the same checkpointed pagination loop and Slack temporary message pagination through full pages with native completion metadata. Remove provider cursor-equality checks now covered by the shared paginator, retaining provider validation, durable timestamp frontiers, and bounded recovery.

.changeset/slack-independent-raw-messages.md

@@ -0,0 +1,5 @@
+---
+"chkit": patch
+---
+
+Document Slack's three raw resources and verify messages-first ingestion and metadata execution that leaves message progress unchanged. Default message bootstrap and overlap to 24 hours, with optional explicit historical bootstrap and isolated backfills. Complete each history page's replies before checkpointing its timestamp frontier; interruptions replay the unfinished page instead of retaining thread queues. Remove automatic historical reconciliation: edits and late replies on roots outside the chosen window are excluded. Replies remain individual native messages joined through thread references in ClickHouse.

.github/workflows/ci.yml

@@ -15,6 +15,8 @@ jobs:
     steps:
       - name: Checkout
         uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
 
       - name: Setup Bun
         uses: ./.github/actions/setup

CLAUDE.md

@@ -101,6 +101,12 @@ A test may skip itself on `CHKIT_E2E_TARGET=obsessiondb` only for a known Obsess
 
 Uses changesets. Run `bun run changeset` to create a changeset, then `bun run version-packages` to bump versions.
 
+### Registry integration releases
+
+Keep at most one new release artifact per integration in a PR. Versions already on the merged base are immutable; artifacts added by the unmerged PR are drafts. Further edits refresh that same draft version and its current `meta.chkit.changelog` entry, without adding intermediate versions. Preserve earlier released changelog entries.
+
+Regenerate with `bun run registry:release -- --base origin/main <provider>` and validate with `bun run check:registry-releases -- --base origin/main`; use the actual PR base if different. Omit provider names to refresh all drafts. Do not hand-edit generated artifacts. `bun run verify` checks release history and source/artifact agreement.
+
 ### Releasing (GitHub Actions)
 
 Versions are semver; all 10 published packages share one version (a changesets `fixed` group). A release takes two steps, so the version reaches git before npm and any failure can be re-run:

apps/docs/src/components/RegistryApps.astro

@@ -1,5 +1,5 @@
 ---
-import { registryApps } from '../data/registry';
+import { registryApps, registryStrategyLabel } from '../data/registry';
 ---
 
 <ul class="registry-apps not-content" aria-label="Available ClickHouse integrations">
@@ -10,7 +10,7 @@ import { registryApps } from '../data/registry';
 					<div class="app-logo">{app.logo ? <img src={app.logo} alt={`${app.title} logo`} width="37" height="30" loading="lazy" /> : <span aria-hidden="true">{app.title.slice(0, 1)}</span>}</div>
 					<div>
 						<h3><a href={app.href}>{app.title}</a></h3>
-						<p class="app-meta">{app.resources.length} resources · {app.strategies.map((strategy) => strategy === 'full' ? 'Full reads' : strategy).join(', ')} · {app.language === 'typescript' ? 'TypeScript' : app.language}</p>
+						<p class="app-meta">{app.resources.length} resources · {app.strategies.map(registryStrategyLabel).join(', ')} · {app.language === 'typescript' ? 'TypeScript' : app.language}</p>
 					</div>
 					<span class="app-version">v{app.version}</span>
 				</div>

apps/docs/src/components/RegistryReference.astro

@@ -1,5 +1,5 @@
 ---
-import { registryApps, type RegistryReferenceSection } from '../data/registry';
+import { registryApps, registryStrategyLabel, type RegistryReferenceSection } from '../data/registry';
 
 interface Props {
 	name: string;
@@ -11,7 +11,7 @@
 if (!app) throw new Error(`Unknown registry app: ${name}`);
 ---
 
 {section === 'overview' && (
 	<table>
 		<thead><tr><th>Property</th><th>Included behavior</th></tr></thead>
 		<tbody>
@@ -22,7 +22,7 @@
 			<tr><td>Environment variables</td><td>{app.authentication?.env.map((env, index) => <>{index > 0 && ', '}<code>{env}</code></>) ?? 'See provider setup'}</td></tr>
 			<tr><td>ClickHouse</td><td><code>{app.clickhouse}</code></td></tr>
 			<tr><td>Coverage</td><td>{app.resources.length} synced resources, {app.views?.length ?? 0} derived views</td></tr>
-			<tr><td>Sync strategy</td><td>{app.strategies.map((strategy) => strategy === 'full' ? 'Full read on every run' : strategy).join(', ')}</td></tr>
+			<tr><td>Sync strategy</td><td>{app.strategies.map(registryStrategyLabel).join(', ')}</td></tr>
 		</tbody>
 	</table>
 )}
@@ -67,3 +67,9 @@
 		</tbody>
 	</table>
 )}
+{section === 'changelog' && app.changelog?.map((release) => (
+	<>
+		<h3>Version {release.version}</h3>
+		<ul>{release.changes.map((change) => <li>{change}</li>)}</ul>
+	</>
+))}

apps/docs/src/content/docs/api-sync/agent-skill.md

@@ -42,10 +42,11 @@ Include provider documentation, example responses, freshness needs, and any rest
 
 ## Skill scope
 
-The skill covers the authoring steps and requirements for bounded readers, checkpoint ordering, write identity, and scheduling. Use the linked docs for complete examples and tradeoffs.
+The skill covers bounded readers, provider-specific sync strategies, durable checkpoint ordering, write identity, and scheduling. For official registry integrations, it requires a source contract and recovery design, raw payload preservation, explicit freshness coverage, and portable interruption/replay tests. Prefer coarse checkpoints and replay when affordable; reconcile older changes when the chosen coverage requires it. Sync tokens, timestamp windows, and checkpointed full scans follow the provider's actual capabilities. Use the linked docs for complete examples and tradeoffs.
 
 ## Related pages
 
 - [Destinations and transformations](/api-sync/destinations/): choose storage and mapping for the source.
 - [Quickstart](/api-sync/quickstart/): a working first source.
+- [Publish a registry](/api-sync/registry-authoring/): metadata, portable tests, and immutable releases.
 - [For AI agents](/ai-agents/): general chkit setup and schema workflows.

apps/docs/src/content/docs/api-sync/incremental-syncs.md

@@ -42,7 +42,7 @@ const tickets = defineStream({
       },
     })
     for await (const page of pages) {
-      yield { rows: rawRows(page, (ticket) => ticket.id) }
+      yield { rows: rawRows(page.items, (ticket) => ticket.id) }
     }
   },
 })
@@ -65,10 +65,10 @@ Use `timestampWindow({ from })` when `start` and `overlapMs` do not express the
 
 Choose `cursorState` when the provider guarantees resumption across runs. `parse` validates the saved state. `context.state` and `context.selection` expose it to the reader; neither contains uncommitted progress.
 
-This standalone fixture demonstrates a provider with stable, increasing change sequence numbers. Replace the fixture read with a documented provider operation inside `context.attempt`:
+This standalone fixture demonstrates a provider with stable, increasing change sequence numbers. `paginate` owns each request attempt and exposes its candidate checkpoint in metadata. Replace the fixture body of `fetchPage` with a documented provider operation, forwarding its signal:
 
 ```ts
-import { cursorState, definePipeline, defineStream, rawRows, rawTable } from '@chkit/plugin-ingest'
+import { cursorState, definePipeline, defineStream, paginate, rawRows, rawTable } from '@chkit/plugin-ingest'
 
 const changes = [
   { sequence: 1, id: 'a', subject: 'Opened' },
@@ -90,16 +90,24 @@ const tickets = defineStream({
     },
   }),
   async *read(context) {
-    let after = context.state ?? 0
-    while (true) {
-      const page = await context.attempt(async (signal) => {
+    const pages = paginate({
+      context,
+      initial: context.state ?? 0,
+      label: 'read changes',
+      fetchPage: async (after = 0, signal) => {
         signal.throwIfAborted()
-        return changes.filter((change) => change.sequence > after).slice(0, 2)
-      }, { label: 'read changes' })
-      const last = page.at(-1)
-      if (!last) return
-      yield { rows: rawRows(page, (ticket) => ticket.id), state: last.sequence }
-      after = last.sequence
+        const remaining = changes.filter((change) => change.sequence > after)
+        const items = remaining.slice(0, 2)
+        const checkpoint = items.at(-1)?.sequence ?? after
+        return {
+          items,
+          next: remaining.length > items.length ? checkpoint : undefined,
+          metadata: { checkpoint },
+        }
+      },
+    })
+    for await (const page of pages) {
+      yield { rows: rawRows(page.items, (ticket) => ticket.id), state: page.metadata?.checkpoint }
     }
   },
 })
@@ -109,6 +117,15 @@ export const fixture = definePipeline({ id: 'fixture', streams: [tickets] })
 
 `state` is the complete resume state safe after all rows through that chunk are saved. It can be a scalar or an object such as `{ snapshot, cursor }`. Omit it if the source cannot guarantee a safe boundary. An empty chunk with state can record progress when no destination rows are needed.
 
+`next` and checkpoint metadata have separate jobs. For a provider sync-token protocol, the page adapter can return:
+
+| Page | `next` request continuation | `metadata.checkpoint` candidate |
+| --- | --- | --- |
+| More pages remain | `'page-2'` | `{ syncToken: 'sync-old', pageToken: 'page-2' }` |
+| Final page, possibly empty | `undefined` | `{ syncToken: 'sync-new' }` |
+
+Start a resumed pagination sequence with the saved `pageToken` and keep its input `syncToken` fixed. Only the acknowledged terminal candidate replaces that sync token and clears the page position. Construct fresh complete checkpoint objects; metadata alone does not commit progress. [Google Calendar](/integrations/google-calendar/) implements this protocol with `paginate`, `cursorState`, and provider-specific expiry recovery.
+
 A pagination token may expire between runs. Confirm expiration, snapshot lifetime, ordering, and terminal-token behavior before storing one. Resumption must not skip unfinished pages or repeatedly reuse an expired cursor.
 
 ## Custom strategies and changes to state

apps/docs/src/content/docs/api-sync/operations.md

@@ -100,7 +100,7 @@ chkit ingest status --tag pipeline:helpdesk --json
 | Symptom | Check |
 |---|---|
 | No pipeline / no matching streams | Entry exports and exact tag spelling |
-| No checkpoint after a successful run | Full syncs have no incremental bookmark |
+| No checkpoint after a successful run | The bundled `fullSync()` strategy has no checkpoint; custom full-scan strategies may record completion |
 | Window restarts from the beginning | Timestamp progress commits after the whole window; reduce work or use safe intermediate provider state |
 | Duplicated logical records | Destination keys/versioning, query reconciliation, deduplication settings, and volatile mapped fields |
 | Incompatible checkpoint | Strategy ID/version or state format changed |

apps/docs/src/content/docs/api-sync/readers.md

@@ -46,18 +46,15 @@ export async function fetchTicketPage(
 
 This helper performs one request. Call it through `context.attempt` or from `paginate`'s `fetchPage` callback, as shown in [Incremental syncs](/api-sync/incremental-syncs/#timestamp-windows). Validate untrusted response shapes at the source boundary in production; a TypeScript assertion does not validate JSON.
 
-## Pagination is not a checkpoint
+## Pages and checkpoint metadata
 
-Use `paginate` when only the items matter. It supports an `initial` continuation, yields nonempty item arrays, and stops when `next` is `undefined` or `null`. It rejects repeated continuations to avoid infinite loops.
+`paginate` yields the complete `{ items, next, metadata? }` returned by `fetchPage`, including empty and terminal pages. Read rows from `page.items`. `initial` and `next` support offset, string-token, and compound-object continuations; `next: undefined` ends pagination. Repeated continuations, including a return to `initial`, fail with `IngestConfigError` before the invalid page reaches the reader.
 
-A page cursor can be temporary, tied to one snapshot, or expire before the next run. Only persist it with `cursorState` when the provider guarantees it is valid for later executions. Otherwise page through a timestamp window and commit progress after that whole window succeeds.
-
-<details>
-<summary>When a manual pagination loop is needed</summary>
+`next` controls the next request. `metadata` carries provider details such as a candidate checkpoint, replacement sync token, or page identity. The helper preserves metadata without interpreting or persisting it. With `cursorState`, explicitly map a safe, complete checkpoint from the page into `SourceChunk.state`; the executor commits it after the rows are saved. Empty pages can carry meaningful progress through the same path. See the complete [provider cursor example](/api-sync/incremental-syncs/#provider-cursors-and-compound-state).
 
-Use a manual loop when page metadata must become a chunk `id` or durable `state`, or when an empty page carries meaningful progress. The helper yields item arrays, so it does not expose the returned continuation to the consumer or persist it as a checkpoint. Wrap each manual page request in `context.attempt`.
+A page cursor can be temporary, tied to one snapshot, or expire before the next run. Only persist it with `cursorState` when the provider guarantees it is valid for later executions. Otherwise page through a timestamp window and commit progress after that whole window succeeds.
 
-</details>
+For a sync-token API, a nonterminal page can retain the input sync token and save its next-page position in metadata. Its terminal page has no `next`, but can carry a replacement sync token. Metadata and pagination therefore have independent types: `paginate<TItem, TCursor, TMetadata>`. Protocol-specific token recovery remains the reader's responsibility; use `context.attempt` for requests outside `paginate` and avoid wrapping its `fetchPage` in another attempt.
 
 ## Bound work and separate accounts