Skip to content

Add recoverable raw resource streams and integration changelogs - #261

Merged
replicas-connector[bot] merged 18 commits into
mainfrom
feat/checkpointed-registry-ingestion
Oct 5, 2026
Merged

replicas-connector[bot] merged 18 commits into
mainfrom
feat/checkpointed-registry-ingestion

Conversation

@replicas-connector

@replicas-connector replicas-connector Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Upgrade all eight registry integrations to one pipeline per installation with independently selectable raw resource streams and injectable clients: Attio resource types/lists, eight GitHub resources, nine Linear resources, five Circleback resources, Calendar events, six Meet resources, Slack channels/users/messages, and seven Lemlist resources.

Change paginate() to yield full { items, next, metadata? } pages including empty terminal pages, simplify Calendar recovery and provider cursor handling, and use recent 24-hour Meet/Slack rediscovery with sink-acknowledged page checkpoints and replay of unfinished child reads, while retaining complete Circleback/Lemlist coverage, read-only inbox retrieval, and transcript availability records that preserve successful content.

Make normalized raw sync with later ClickHouse joins/transforms the authoring default, prefer affordable coarse replay with explicit freshness coverage over retained-work queues, preserve native fields and join keys with labels/tags as strings, omit PR files and binary downloads, document identity/state migrations and provider limits, and add cumulative changelogs while retaining one editable 0.2.0 draft per integration and protecting published history.

Validation passed all 48 bun run verify tasks, including 211 registry tests with 1,778 assertions, 73 ingestion-plugin tests, relocated installed consumers, live local ClickHouse/Kafka checks, packed dependencies, release/history guards and documentation builds; skill validation and changed-integration dead-code checks also pass.


Created with Replicas

Replicas Workspace

Co-Authored-By: Marc Höffl marc.hoeffl@gmail.com
Co-Authored-By: replicas-connector[bot] <replicas-connector[bot]@users.noreply.github.com>

Implement provider-specific tokens, incremental windows, durable scan progress,
and safe restart recovery across the eight raw ingestion integrations.
Update registry metadata, immutable releases, documentation, and the ingestion
authoring skill with the stable sync quality bar.

Co-Authored-By: Marc Höffl <marc.hoeffl@gmail.com>
Co-Authored-By: replicas-connector[bot] <replicas-connector[bot]@users.noreply.github.com>
Give configured object types and lists separate streams using the existing
full-sync and pagination primitives. Remove the custom scan checkpoint ledger,
preserve raw tables and row identities, and document the explicit configuration
and stream identity migration in a new immutable template release.

Co-Authored-By: Marc Höffl <marc.hoeffl@gmail.com>
Co-Authored-By: replicas-connector[bot] <replicas-connector[bot]@users.noreply.github.com>
Comment thread registry/attio/tests/fixtures.ts Fixed
Comment thread registry/attio/tests/fixtures.ts Fixed
@replicas-connector

replicas-connector Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor Author

CI/CD Failures

Workflow Commit Status Notes
fallow e51ddb0 Fixed Made unused fixture exports private; published immutable Attio 0.2.1 in 6d87111.

Keep Companies and Sales fixture constants private and publish the corrected
bundle as Attio 0.2.1, preserving the immutable 0.2.0 artifact.

Co-Authored-By: Marc Höffl <marc.hoeffl@gmail.com>
Co-Authored-By: replicas-connector[bot] <replicas-connector[bot]@users.noreply.github.com>
Consolidate Attio's unmerged releases into one 0.2.0 draft. Add cumulative
integration changelog metadata and render it in CLI inspection and guides.
Refresh drafts against the merged base and validate one new artifact per
integration, immutable published history, and current source agreement.
Document the lifecycle in authoring guidance and verify recovery and release
protection through fixture and installed-consumer tests.

Co-Authored-By: Marc Höffl <marc.hoeffl@gmail.com>
Co-Authored-By: replicas-connector[bot] <replicas-connector[bot]@users.noreply.github.com>
@replicas-connector replicas-connector Bot changed the title Add recoverable syncs to raw registry integrations Add recoverable registry syncs and integration changelogs Oct 4, 2026
Group each installation in one pipeline, bind reader configuration, and share pagination while preserving provider recovery state and raw storage. Refresh the existing draft releases and teach the same contracts in the authoring skill.

Co-Authored-By: Marc Höffl <marc.hoeffl@gmail.com>
Co-Authored-By: replicas-connector[bot] <replicas-connector[bot]@users.noreply.github.com>
Give issues, PRs, discussions, reviews, commits, and stargazers separate
raw destinations and checkpoints. Child readers discover their own parents;
commit reads paginate GraphQL SHA/message data without fetching files.
Teach the raw resource model and retain the integration's single PR draft.

Co-Authored-By: Marc Höffl <marc.hoeffl@gmail.com>
Co-Authored-By: replicas-connector[bot] <replicas-connector[bot]@users.noreply.github.com>
@replicas-connector replicas-connector Bot changed the title Add recoverable registry syncs and integration changelogs Add recoverable raw resource streams and integration changelogs Oct 5, 2026
Make direct raw resource ingestion the authoring default. Preserve provider
fields and relationship keys, and perform transformations, joins, enrichment,
and denormalization afterward in ClickHouse. Qualify alternate storage models
as explicit user choices and align the reference and registry authoring guide.

Co-Authored-By: Marc Höffl <marc.hoeffl@gmail.com>
Co-Authored-By: replicas-connector[bot] <replicas-connector[bot]@users.noreply.github.com>
Sync issues, comments, projects, project updates, cycles, users, teams,
issue relations, and issue history with independent recovery and raw tables.
Keep complete issue labels as strings and refresh the single 0.2.0 draft.

Co-Authored-By: Marc Höffl <marc.hoeffl@gmail.com>
Co-Authored-By: replicas-connector[bot] <replicas-connector[bot]@users.noreply.github.com>
replicas-connector Bot and others added 5 commits October 5, 2026 03:29
Use full-sync pagination with complete resource details, string tags, independent transcript content and availability, and native warehouse join keys.

Co-Authored-By: Marc Höffl <marc.hoeffl@gmail.com>
Co-Authored-By: replicas-connector[bot] <replicas-connector[bot]@users.noreply.github.com>
Keep events as the only resource and verify native nested fields, recurring exceptions, and sparse cancellation replacement.

Co-Authored-By: Marc Höffl <marc.hoeffl@gmail.com>
Co-Authored-By: replicas-connector[bot] <replicas-connector[bot]@users.noreply.github.com>
Add participants, participant sessions, and recording metadata with acknowledged nested progress, retained conference polling, and bounded token recovery.

Co-Authored-By: Marc Höffl <marc.hoeffl@gmail.com>
Co-Authored-By: replicas-connector[bot] <replicas-connector[bot]@users.noreply.github.com>
Keep channels, users, and messages with replies as independent resources and prove metadata runs cannot advance pending message work.

Co-Authored-By: Marc Höffl <marc.hoeffl@gmail.com>
Co-Authored-By: replicas-connector[bot] <replicas-connector[bot]@users.noreply.github.com>
Add complete contacts, companies, campaign leads, inbox conversations, and inbox messages with scoped native identities, safe full replay, and read-only retrieval.

Co-Authored-By: Marc Höffl <marc.hoeffl@gmail.com>
Co-Authored-By: replicas-connector[bot] <replicas-connector[bot]@users.noreply.github.com>
Change paginate to yield items, continuation, and typed metadata for every
page, including empty and terminal results. Reject continuation cycles before
exposing candidate progress, and keep durable commits with the existing
cursorState and executor primitives.

Migrate registry readers and use page metadata for Calendar token checkpoints
without changing saved state. Update authoring guidance and API examples,
and refresh the existing integration drafts without adding versions.

Co-Authored-By: Marc Höffl <marc.hoeffl@gmail.com>
Co-Authored-By: replicas-connector[bot] <replicas-connector[bot]@users.noreply.github.com>
Handle Calendar token resets as checkpointed continuations in one paginator.
Reuse Slack pagination for empty message pages while retaining native
completion metadata, durable timestamp frontiers, and bounded cursor replay.

Remove duplicate GitHub, Circleback, and Linear cursor equality checks now
covered before yield by the shared paginator. Preserve provider response
validation and existing checkpoint formats, update authoring guidance, and
refresh each affected integration's existing draft release.

Co-Authored-By: Marc Höffl <marc.hoeffl@gmail.com>
Co-Authored-By: replicas-connector[bot] <replicas-connector[bot]@users.noreply.github.com>
replicas-connector Bot and others added 3 commits October 5, 2026 08:12
Reread recent and ongoing conferences with a 24-hour overlap. Commit discovery only after all child rows land, replay unfinished pages, and remove retained conference and child-work bookkeeping.

Co-Authored-By: Marc Höffl <marc.hoeffl@gmail.com>
Co-Authored-By: replicas-connector[bot] <replicas-connector[bot]@users.noreply.github.com>
Use recent overlapping message windows and commit history progress after selected replies finish. Replay unfinished pages, remove durable thread queues and automatic historical reconciliation, and document recent-root coverage.

Co-Authored-By: Marc Höffl <marc.hoeffl@gmail.com>
Co-Authored-By: replicas-connector[bot] <replicas-connector[bot]@users.noreply.github.com>
Recommend existing pagination and acknowledged checkpoints with affordable replay. Choose explicit freshness coverage before adding retained work, and document raw replacement without body hashes or insert-only staging.

Co-Authored-By: Marc Höffl <marc.hoeffl@gmail.com>
Co-Authored-By: replicas-connector[bot] <replicas-connector[bot]@users.noreply.github.com>
@replicas-connector
replicas-connector Bot merged commit 86aff37 into main Oct 5, 2026
8 checks passed
@replicas-connector
replicas-connector Bot deleted the feat/checkpointed-registry-ingestion branch October 5, 2026 08:32

This branch was successfully deployed

1 active deployment
preview — 49bea965 Deployed Oct 5, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant