|
| 1 | +# @secondlayer/subgraphs |
| 2 | + |
| 3 | +Typed on-chain indexing for Stacks. Declare event filters + column schema with `defineSubgraph()`; the runtime decodes blocks, matches filters, runs your handlers inside a transactional context, and exposes the result as a Postgres schema you query over REST or SQL. |
| 4 | + |
| 5 | +Subgraph rows fan out to HTTP subscribers through a post-flush outbox emitter — signed Standard Webhooks POSTs with retries, circuit breaker, and replay. |
| 6 | + |
| 7 | +## Install |
| 8 | + |
| 9 | +```bash |
| 10 | +bun add @secondlayer/subgraphs |
| 11 | +``` |
| 12 | + |
| 13 | +## Quick Start |
| 14 | + |
| 15 | +```typescript |
| 16 | +import { defineSubgraph } from "@secondlayer/subgraphs"; |
| 17 | + |
| 18 | +export default defineSubgraph({ |
| 19 | + name: "token-transfers", |
| 20 | + version: "1.0.0", |
| 21 | + sources: { |
| 22 | + // Named event sources — the key becomes the handler name. |
| 23 | + transfer: { |
| 24 | + type: "ft_transfer", |
| 25 | + assetIdentifier: "SP2X0TZ59D5SZ8ACQ6YMCHHNR2ZN51Z32E2CJ173.stx-token::stx", |
| 26 | + }, |
| 27 | + }, |
| 28 | + schema: { |
| 29 | + transfers: { |
| 30 | + columns: { |
| 31 | + sender: { type: "principal" }, |
| 32 | + recipient: { type: "principal" }, |
| 33 | + amount: { type: "uint" }, |
| 34 | + }, |
| 35 | + // Auto-added: _block_height, _tx_id, _created_at |
| 36 | + }, |
| 37 | + }, |
| 38 | + handlers: { |
| 39 | + async transfer(event, ctx) { |
| 40 | + ctx.insert("transfers", { |
| 41 | + sender: event.sender, |
| 42 | + recipient: event.recipient, |
| 43 | + amount: event.amount, |
| 44 | + }); |
| 45 | + }, |
| 46 | + }, |
| 47 | +}); |
| 48 | +``` |
| 49 | + |
| 50 | +Deploy via CLI (`sl subgraphs deploy path/to/definition.ts`), SDK (`sl.subgraphs.deploy({...})`), or MCP (`subgraphs_deploy`). The dashboard is read-only — creation always happens through an API surface. |
| 51 | + |
| 52 | +## Exports |
| 53 | + |
| 54 | +| Subpath | Description | |
| 55 | +| --- | --- | |
| 56 | +| `.` | `defineSubgraph`, `validateSubgraphDefinition`, `deploySchema`, `diffSchema`, `reindexSubgraph`, `backfillSubgraph`, `generateSubgraphSQL`, `pgSchemaName` | |
| 57 | +| `./types` | All schema + filter + handler types (`SubgraphDefinition`, `SubgraphFilter`, `StxTransferFilter`, etc.) | |
| 58 | +| `./schema` | Generator + deployer internals | |
| 59 | +| `./validate` | Shape + filter validation for deploys | |
| 60 | +| `./runtime/source-matcher` | Pure fn: match txs+events against a `SubgraphFilter` — used by the processor hot path | |
| 61 | +| `./runtime/replay` | `replaySubscription({ accountId, subscriptionId, fromBlock, toBlock })` — re-enqueue historical rows as outbox entries | |
| 62 | + |
| 63 | +## Runtime components |
| 64 | + |
| 65 | +The runtime ships behind these entrypoints (import from the package root): |
| 66 | + |
| 67 | +- `startSubgraphProcessor(opts?)` — boots the block processor. LISTENs on `indexer:new_block`, matches sources, runs handlers, flushes writes inside a transaction, and emits outbox rows for matching subscriptions. Also boots the emitter worker. |
| 68 | +- `processBlock(subgraph, name, height, opts?)` — single-block entry point used by catch-up, reindex, and tests. |
| 69 | +- `catchUpSubgraph(def, name)` — drains pending blocks up to chain tip. |
| 70 | +- `reindexSubgraph(def, opts)` — drop + rebuild schema tables from a start block. Breaking schema changes trigger this automatically on deploy. |
| 71 | + |
| 72 | +## Subscription emitter |
| 73 | + |
| 74 | +Every row written through `ctx.insert()` / `ctx.upsert()` is atomically enqueued to `subscription_outbox` for every active subscription whose filter matches — inside the same transaction as the flush, so a processor crash rolls back both. |
| 75 | + |
| 76 | +The emitter drains the outbox via `LISTEN subscriptions:new_outbox` and `FOR UPDATE SKIP LOCKED` batch claims. Live deliveries win a 90/10 split over replays. Each row dispatches through the format builder matching the subscription's `format` column (`standard-webhooks`, `inngest`, `trigger`, `cloudflare`, `cloudevents`, `raw`). Retries follow `30s → 2m → 10m → 1h → 6h → 24h → 72h`. Twenty consecutive failures trips the per-sub circuit breaker and pauses the subscription. |
| 77 | + |
| 78 | +Delivery bodies and response previews land in `subscription_deliveries`. Rows whose retries exhaust mark `status = 'dead'` in the outbox and surface in the dashboard's dead-letter queue for one-click requeue. |
| 79 | + |
| 80 | +## Environment |
| 81 | + |
| 82 | +| Variable | Default | Description | |
| 83 | +| --- | --- | --- | |
| 84 | +| `SECONDLAYER_EMIT_OUTBOX` | `true` | Set `false` to bypass outbox emission on every block (kill-switch). | |
| 85 | +| `SECONDLAYER_ALLOW_PRIVATE_EGRESS` | `false` | Allow the emitter to deliver to private IP ranges (localhost, 10/8, 172.16/12, 192.168/16, link-local, v6 mapped). Leave off in production. | |
| 86 | +| `SECONDLAYER_SECRETS_KEY` | — | 32-byte hex key for the AES-GCM envelope around subscription signing secrets. OSS mode auto-generates + persists to `.env.local`. | |
| 87 | + |
| 88 | +## Postgres + pool mode |
| 89 | + |
| 90 | +The emitter holds a persistent `LISTEN` on `subscriptions:new_outbox` and `subscriptions:changed`, so it MUST connect through a session-mode pool. pgbouncer in transaction mode silently breaks it. Run the emitter against a session-mode port (`pool_mode = session`), or connect directly to Postgres as the default docker-compose setup does. |
| 91 | + |
| 92 | +## License |
| 93 | + |
| 94 | +MIT |
0 commit comments