From ffb6a5160cd78821648a38014cc17e90a8965ea5 Mon Sep 17 00:00:00 2001 From: dillonstreator Date: Tue, 3 Nov 2026 16:16:32 -0600 Subject: [PATCH 1/5] feat: deferred delivery with handler defaults and publish-time override Add `defer` to handler config (static `{ delayMs } | { until }` or a function of the event) and a `defer` publish option that overrides it for every handler of that event. The processor resolves the deferral at fan-out into `run_after` on each new delivery; `delayMs` counts from `event.created_at` so fan-out retries are deterministic. - `TxOBEvent.defer` persisted; new `defer JSONB` column on `txob_events` - `TxOBNewDelivery.run_after: Date | null` (breaking for custom stores) - all stores honor `run_after` on fan-out; redis routes to the delayed zset - processor schedules a poll at the earliest due time; `scheduleTick` skips timers longer than `pollingIntervalMs` (also avoids setTimeout overflow) - pg fan-out passes `run_after` as `text[]`: postgres.js under Drizzle cannot encode `timestamptz[]` parameters - docs, example, migration notes, changeset --- .changeset/deferred-delivery.md | 7 ++ MIGRATING.md | 15 ++++ README.md | 36 +++++++++- architecture.mmd | 6 +- examples/pg/processor.ts | 8 +++ src/defer.test.ts | 48 +++++++++++++ src/defer.ts | 51 +++++++++++++ src/dynamodb/store.ts | 9 ++- src/handlers.ts | 14 ++++ src/index.ts | 1 + src/memory/store.ts | 15 +++- src/mongodb/store.ts | 5 +- src/outbox.ts | 8 +++ src/pg/drizzle.ts | 1 + src/pg/kysely.ts | 1 + src/pg/schema.ts | 2 + src/pg/store.ts | 20 ++++-- src/processor.ts | 33 +++++++-- src/redis/scripts.ts | 26 +++++-- src/redis/store.ts | 3 + src/store.ts | 2 + src/types.ts | 11 +++ test/contract.ts | 109 ++++++++++++++++++++++++++++ test/units.test.ts | 123 ++++++++++++++++++++++++++++++++ 24 files changed, 524 insertions(+), 30 deletions(-) create mode 100644 .changeset/deferred-delivery.md create mode 100644 src/defer.test.ts create mode 100644 src/defer.ts diff --git a/.changeset/deferred-delivery.md b/.changeset/deferred-delivery.md new file mode 100644 index 0000000..bd5e2d2 --- /dev/null +++ b/.changeset/deferred-delivery.md @@ -0,0 +1,7 @@ +--- +"txob": minor +--- + +Deferred delivery: `defer: { delayMs } | { until }` on handler definitions sets when deliveries become claimable, and `publish(..., { defer })` overrides it per event. Resolved at fan-out into the delivery's `run_after`; `delayMs` counts from the event's `created_at`. + +**Breaking for Postgres and custom stores:** `txob_events` gains a `defer JSONB` column (`ALTER TABLE txob_events ADD COLUMN IF NOT EXISTS defer JSONB` on existing installs), `TxOBEvent` gains `defer`, and `TxOBNewDelivery` gains `run_after`. See MIGRATING.md. diff --git a/MIGRATING.md b/MIGRATING.md index c57b265..f276122 100644 --- a/MIGRATING.md +++ b/MIGRATING.md @@ -1,3 +1,18 @@ +# Migrating from 0.5 to 0.6 + +0.6 adds deferred delivery (`defer` on handler definitions and `publish`). Events gain a `defer` field, and new deliveries carry a `run_after`. + +- **Postgres:** add the column to existing tables. `migrate` and `getMigrationSql` include it for new installs only: + + ```sql + ALTER TABLE txob_events ADD COLUMN IF NOT EXISTS defer JSONB; + ``` + + Drizzle users regenerate their migration from `txobTables()`. + +- **MongoDB, Redis, DynamoDB:** no migration; old events read back with `defer: null`. +- **Custom stores:** persist and return `event.defer` from `insertEvent`, and apply `run_after` (a `Date`, or `null` for "now") when `completeFanout` creates deliveries. The store contract in `test/contract.ts` checks both. + # Migrating from 0.4 to 0.5 0.5 reworks the processing model and is not backwards compatible. The ideas you're used to stay the same: typed event schemas, a handler map keyed by event type, at-least-once delivery. Storage, the processing model and most APIs change. diff --git a/README.md b/README.md index 6bf4171..62ab736 100644 --- a/README.md +++ b/README.md @@ -133,6 +133,8 @@ try { `publish` validates the data against the event's schema (throwing `TxOBValidationError` on failure) and inserts it using the connection you pass, so it commits or rolls back with your data. +Options: `id` (default: a random UUID), `correlation_id`, and `defer` to hold every delivery of the event back, overriding the handlers' own `defer` (see [Deferred delivery](#deferred-delivery)). + **4. Process them:** ```typescript @@ -239,15 +241,39 @@ Each handler is either a function or a config object: timeoutMs: 30_000, // abort the handler's signal after this long priority: (event) => (event.data.plan === "enterprise" ? 10 : 0), // or a number; higher first + + defer: { delayMs: 60_000 }, // or { until: Date }, or a function of the event; see below } ``` - **Concurrency, per-key concurrency and rate limits are global.** `concurrency: 5` means at most 5 deliveries of the handler run at once across all processors. Running deliveries are counted by their live leases, so a processor that crashes keeps holding its share until its leases expire (at most `leaseMs`). - **Claims for a limited handler are serialized** (a transaction-scoped advisory lock on Postgres, a short-lived lock document on MongoDB). Claims take milliseconds, but if one handler sees very high claim rates, prefer a higher `concurrency` on the processor over many tiny claims. -- **`priority` and the `perKey` key are computed at fan-out** and stored with the delivery, so they're evaluated once per event, not per attempt. +- **`priority`, the `perKey` key and `defer` are computed at fan-out** and stored with the delivery, so they're evaluated once per event, not per attempt. - **`timeoutMs` cancels the handler's context.** When it elapses, the `signal` passed to the handler is aborted with a `TxOBTimeoutError` reason. Honoring it is up to the handler: pass the signal to `fetch`, database drivers and so on, or check `signal.aborted`. Whatever the handler then does decides the outcome. If it throws, the attempt fails and is retried; if it finishes successfully anyway, the delivery is completed. The lease stays renewed while the handler runs. - **Process-wide defaults** go in `handlerDefaults: { retry, timeoutMs }`. +### Deferred delivery + +`defer` holds a delivery back until a point in time. It takes `{ delayMs }`, counted from the event's `created_at`, or `{ until: Date }`. The handler definition sets the default; `publish` can override it per event: + +```typescript +// Handler default: send the reminder a day after the order, unless the event says otherwise +OrderPlaced: { + sendReminder: { + defer: (event) => (event.data.reminderAt ? { until: event.data.reminderAt } : { delayMs: 86_400_000 }), + handler: async (event) => {}, + }, +}, + +// Publish-time override: every handler of this event waits 10 minutes +await outbox.publish(tx, "OrderPlaced", data, { defer: { delayMs: 600_000 } }); +``` + +- **Resolved at fan-out.** The processor turns `defer` into the delivery's `run_after` when it creates the delivery, so a change to a handler's `defer` applies to events fanned out afterwards. `delayMs` counts from `created_at`, not from fan-out, so a processor that was down for an hour doesn't push "in 5 minutes" to "in 65 minutes", and repeated fan-outs of the same event agree. +- **The publish-time `defer` wins** over the handler's for every handler of that event. The event's `defer` is also available to handler `defer` functions and handlers as `event.defer`, so a handler can opt out of honoring it. +- **Due deliveries are picked up promptly.** The processor that fanned the event out polls when the earliest deferred delivery is due; beyond `pollingIntervalMs`, the regular poll finds it. +- **Priority still applies.** A deferred delivery competes with everything else that is due once its time comes. + ### Tuning at runtime Limits aren't stored on queued deliveries. The processor passes its current settings with every claim, so raising a limit applies to work that is already queued, whether you redeploy with new values or change them live: @@ -274,7 +300,7 @@ processor.getConfig(); // effective settings per handler, after overrides - **Typed:** event types, handler names and values are checked against your handler map. - **Declarative and idempotent:** `handlers` replaces any previous overrides, and handlers it doesn't mention revert to their definitions. Applying the same config twice changes nothing. Invalid config throws and leaves the current settings untouched. - **When it takes effect:** from the next claim. Attempts already running keep the settings they started with. -- **What can't be changed live:** `priority` and the `perKey` key function. They are computed when an event fans out, so changes apply to events fanned out afterwards; the per-key _limit_ is tunable via `perKeyConcurrency`. +- **What can't be changed live:** `priority`, `defer` and the `perKey` key function. They are computed when an event fans out, so changes apply to events fanned out afterwards; the per-key _limit_ is tunable via `perKeyConcurrency`. - **Scope:** `configure` changes this processor only. Global limits are checked with the claiming processor's settings, so apply the same config to every processor (shared config, a feature flag, or a broadcast to all instances) to keep them in agreement. ### Errors @@ -427,6 +453,10 @@ SELECT event_id, handler, attempts, last_error FROM txob_deliveries WHERE status -- Retry a dead delivery UPDATE txob_deliveries SET status = 'pending', attempts = 0, run_after = now() WHERE event_id = $1 AND handler = $2; + +-- Deliveries waiting on a deferral +SELECT event_id, handler, run_after FROM txob_deliveries +WHERE status = 'pending' AND lease_token IS NULL AND run_after > now(); ``` ## MongoDB @@ -622,7 +652,7 @@ A store implements `TxOBStore`, where `TTx` is whatever handle your writes 1. **An atomic write with business data**, for `insertEvent`. Without this you don't have an outbox. 2. **Compare-and-set**, for lease-fenced `completeDelivery`, `failDelivery`, `extendLeases` and `releaseDeliveries`. -3. **An ordered query on due time**, for `claimEventsForFanout` and `claimDeliveries`. +3. **An ordered query on due time**, for `claimEventsForFanout` and `claimDeliveries`. `completeFanout` must honor each new delivery's `run_after`, and `insertEvent` must persist `event.defer` and hand it back on later reads. The behavioral contract every store must satisfy lives in this repo at [`test/contract.ts`](./test/contract.ts) (it is not part of the npm package); copy it and run `runStoreContract(name, setup)` against your store. [`src/memory/store.ts`](./src/memory/store.ts) is a compact reference implementation. diff --git a/architecture.mmd b/architecture.mmd index 4bd5887..e18e68c 100644 --- a/architecture.mmd +++ b/architecture.mmd @@ -2,13 +2,13 @@ graph TB subgraph Application["Your Application"] AppTx["BEGIN"] AppData["Write business data"] - AppPublish["outbox.publish(tx, type, data)
validates with the event's schema"] + AppPublish["outbox.publish(tx, type, data, { defer? })
validates with the event's schema"] AppCommit["COMMIT"] AppTx --> AppData --> AppPublish --> AppCommit end subgraph Database["Store (Postgres, MongoDB, Redis, DynamoDB, ...)"] - Events[("txob_events
id, type, data, correlation_id
fanned_out_at, fanout lease")] + Events[("txob_events
id, type, data, correlation_id, defer
fanned_out_at, fanout lease")] Deliveries[("txob_deliveries
one per (event, handler)
status, attempts, priority,
concurrency_key, run_after,
lease_token, last_error")] Notify["Wakeup source
pg_notify, change stream, pub/sub"] end @@ -20,7 +20,7 @@ graph TB subgraph Processor["Event Processor (run N for scale)"] Wakeup["Wakeup emitter
(optional; polling otherwise)"] Poll["Poll (interval, wakeup,
completion, backoff timers)"] - Fanout["Fan-out
lease unfanned events of handled types,
create deliveries + mark fanned out
(one statement, idempotent)"] + Fanout["Fan-out
lease unfanned events of handled types,
resolve priority, concurrency key, run_after (defer),
create deliveries + mark fanned out
(one statement, idempotent)"] Claim["Claim deliveries, priority order
within limits counted across ALL processors:
concurrency, per-key concurrency, rate limit
(serialized per limited handler),
set lease + token, attempts+1"] Run["Run handler
signal aborted on: shutdown | timeoutMs | lease lost
handler's own result decides the outcome"] Heartbeat["Heartbeat
renew leases every leaseMs/3"] diff --git a/examples/pg/processor.ts b/examples/pg/processor.ts index 3faef05..7a28933 100644 --- a/examples/pg/processor.ts +++ b/examples/pg/processor.ts @@ -46,6 +46,14 @@ const handlerMap = createEventHandlerMap({ }, }, + // Runs an hour after the event was published, unless publish() asked for a different delay + followUp: { + defer: { delayMs: 3_600_000 }, + handler: async (event) => { + console.log(`${event.id} followUp ${JSON.stringify(event.defer)}`); + }, + }, + // Per-account updates must not race: one at a time per account, accounts in parallel updateAccountStats: { concurrency: { diff --git a/src/defer.test.ts b/src/defer.test.ts new file mode 100644 index 0000000..a8f88d4 --- /dev/null +++ b/src/defer.test.ts @@ -0,0 +1,48 @@ +import { describe, expect, it } from "vitest"; +import { normalizeDefer, resolveDefer } from "./defer.js"; + +describe("normalizeDefer", () => { + it("passes through null and well-formed values, dropping extra keys", () => { + expect(normalizeDefer(null, "p")).toBeNull(); + expect(normalizeDefer(undefined, "p")).toBeNull(); + expect(normalizeDefer({ delayMs: 0 }, "p")).toEqual({ delayMs: 0 }); + expect(normalizeDefer({ delayMs: 5, extra: 1 } as any, "p")).toEqual({ + delayMs: 5, + }); + const until = new Date("2030-01-01T00:00:00Z"); + expect(normalizeDefer({ until }, "p")).toEqual({ until }); + }); + + it("revives an ISO string for until, as stored in JSON", () => { + const until = new Date("2030-01-01T00:00:00Z"); + expect(normalizeDefer({ until: until.toISOString() } as any, "p")).toEqual({ + until, + }); + }); + + it.each([ + [{ delayMs: -1 }, "non-negative"], + [{ delayMs: Infinity }, "non-negative"], + [{ delayMs: "5" }, "non-negative"], + [{ until: "nope" }, "valid date"], + [{ until: new Date(NaN) }, "valid date"], + [{}, "{ delayMs } or { until }"], + [5, "must be an object"], + ])("rejects %j", (defer, message) => { + expect(() => normalizeDefer(defer as any, "p")).toThrow(message); + expect(() => normalizeDefer(defer as any, "p")).toThrow("p:"); + }); +}); + +describe("resolveDefer", () => { + const created_at = new Date("2030-01-01T00:00:00Z"); + + it("counts delayMs from created_at and passes until through", () => { + expect(resolveDefer(null, { created_at })).toBeNull(); + expect(resolveDefer({ delayMs: 60_000 }, { created_at })).toEqual( + new Date("2030-01-01T00:01:00Z"), + ); + const until = new Date("2031-01-01T00:00:00Z"); + expect(resolveDefer({ until }, { created_at })).toBe(until); + }); +}); diff --git a/src/defer.ts b/src/defer.ts new file mode 100644 index 0000000..c301c17 --- /dev/null +++ b/src/defer.ts @@ -0,0 +1,51 @@ +import type { TxOBDefer, TxOBEvent } from "./types.js"; + +/** + * Validates a deferral and strips anything else, so what gets persisted is exactly + * `{ delayMs }` or `{ until }`. Accepts an ISO string for `until`, as read back from JSON. + */ +export const normalizeDefer = ( + defer: TxOBDefer | null | undefined, + path: string, +): TxOBDefer | null => { + if (defer === null || defer === undefined) return null; + if (typeof defer !== "object") { + throw new Error(`${path}: defer must be an object, received ${defer}`); + } + if ("until" in defer) { + const until = + defer.until instanceof Date + ? defer.until + : new Date(defer.until as unknown as string); + if (Number.isNaN(until.getTime())) { + throw new Error(`${path}: defer.until must be a valid date`); + } + return { until }; + } + if ("delayMs" in defer) { + const { delayMs } = defer; + if ( + !(typeof delayMs === "number" && Number.isFinite(delayMs)) || + delayMs < 0 + ) { + throw new Error( + `${path}: defer.delayMs must be a non-negative number, received ${delayMs}`, + ); + } + return { delayMs }; + } + throw new Error(`${path}: defer must be { delayMs } or { until }`); +}; + +/** + * The earliest claim time a deferral stands for, or `null` for "now". `delayMs` counts from + * `created_at` so repeated fan-outs of the same event agree. + */ +export const resolveDefer = ( + defer: TxOBDefer | null, + event: Pick, +): Date | null => { + if (defer === null) return null; + if ("until" in defer) return defer.until; + return new Date(event.created_at.getTime() + defer.delayMs); +}; diff --git a/src/dynamodb/store.ts b/src/dynamodb/store.ts index 706af05..1876e4a 100644 --- a/src/dynamodb/store.ts +++ b/src/dynamodb/store.ts @@ -110,6 +110,7 @@ const toEvent = (item: Item): TxOBEvent => ({ data: JSON.parse(item.data), correlation_id: item.correlation_id ?? null, created_at: new Date(item.created_at), + defer: item.defer ? JSON.parse(item.defer) : null, }); const jitter = () => @@ -559,6 +560,7 @@ export const createDynamoStore = ( data: JSON.stringify(event.data), correlation_id: event.correlation_id, created_at: event.created_at.getTime(), + ...(event.defer ? { defer: JSON.stringify(event.defer) } : {}), fanout_type: event.type, fanout_run_after: event.created_at.getTime(), fanout_attempts: 0, @@ -663,8 +665,9 @@ export const createDynamoStore = ( ), ); await Promise.all( - deliveries.map(async (d) => { + deliveries.map(async ({ run_after, ...d }) => { const id = deliveryId(d); + const runAfter = run_after?.getTime() ?? now; try { await doc.send( new PutCommand({ @@ -674,10 +677,10 @@ export const createDynamoStore = ( ...d, status: "pending", attempts: 0, - run_after: now, + run_after: runAfter, created_at: now, queue_pk: queuePk(d.event_type, d.handler, d.priority), - queue_sk: queueSk(now, id), + queue_sk: queueSk(runAfter, id), }, ConditionExpression: "attribute_not_exists(pk)", }), diff --git a/src/handlers.ts b/src/handlers.ts index 22ad2bc..bb5638e 100644 --- a/src/handlers.ts +++ b/src/handlers.ts @@ -1,4 +1,6 @@ +import { normalizeDefer } from "./defer.js"; import type { + TxOBDefer, TxOBEventFromSchemas, TxOBEventSchemaMap, TxOBEventTypeOf, @@ -68,6 +70,12 @@ export type TxOBHandlerConfig = { timeoutMs?: number; /** Higher runs first. Computed when the event is fanned out. Default: 0. */ priority?: number | ((event: TEvent) => number); + /** + * Delay before deliveries of this handler become claimable. `delayMs` counts from the + * event's `created_at`. Resolved when the event is fanned out, and overridden by a `defer` + * passed to `publish`. A function receives the event (including its `defer`). + */ + defer?: TxOBDefer | ((event: TEvent) => TxOBDefer | null | undefined); }; export type TxOBHandlerDefinition = @@ -130,6 +138,7 @@ export type CompiledHandler = { backoff: (ctx: TxOBBackoffContext) => number | Date; timeoutMs?: number; priority: (event: any) => number; + defer: (event: any) => TxOBDefer | null | undefined; }; const assertPositiveInt = (value: number, path: string): void => { @@ -230,6 +239,10 @@ export const compileHandlerMap = ( } const priority = config.priority ?? 0; + const defer = + typeof config.defer === "function" + ? config.defer + : normalizeDefer(config.defer, `${path}.defer`); compiledHandlers.push({ eventType, @@ -242,6 +255,7 @@ export const compileHandlerMap = ( backoff: retry.backoff ?? defaultBackoff, timeoutMs, priority: typeof priority === "function" ? priority : () => priority, + defer: typeof defer === "function" ? defer : () => defer, }); } compiled.set(eventType, compiledHandlers); diff --git a/src/index.ts b/src/index.ts index 0e69ae2..29b3a0c 100644 --- a/src/index.ts +++ b/src/index.ts @@ -1,5 +1,6 @@ export * from "./processor.js"; export * from "./outbox.js"; +export * from "./defer.js"; export * from "./handlers.js"; export * from "./store.js"; export * from "./types.js"; diff --git a/src/memory/store.ts b/src/memory/store.ts index 4e234ca..69bdc94 100644 --- a/src/memory/store.ts +++ b/src/memory/store.ts @@ -108,6 +108,7 @@ export const createMemoryStore = (): MemoryStore => { data: event.data, correlation_id: event.correlation_id, created_at: event.created_at, + defer: event.defer, }), }; }; @@ -128,13 +129,22 @@ export const createMemoryStore = (): MemoryStore => { getEvents: () => [...events.values()].map( - ({ id, type, data, correlation_id, created_at, fanned_out_at }) => + ({ + id, + type, + data, + correlation_id, + created_at, + defer, + fanned_out_at, + }) => structuredClone({ id, type, data, correlation_id, created_at, + defer, fanned_out_at, }), ), @@ -164,6 +174,7 @@ export const createMemoryStore = (): MemoryStore => { data: e.data, correlation_id: e.correlation_id, created_at: e.created_at, + defer: e.defer, fanout_attempts: e.fanout_attempts, fanout_lease_token: e.fanout_lease_token, }); @@ -179,7 +190,7 @@ export const createMemoryStore = (): MemoryStore => { ...d, status: "pending", attempts: 0, - run_after: Date.now(), + run_after: d.run_after?.getTime() ?? Date.now(), lease_token: null, last_error: null, created_at: sequence++, diff --git a/src/mongodb/store.ts b/src/mongodb/store.ts index 6cbc10f..a1be1a5 100644 --- a/src/mongodb/store.ts +++ b/src/mongodb/store.ts @@ -26,6 +26,7 @@ type EventDoc = { data: TxOBEvent["data"]; correlation_id: string | null; created_at: Date; + defer: TxOBEvent["defer"]; fanned_out_at: Date | null; fanout_attempts: number; fanout_run_after: Date; @@ -82,6 +83,7 @@ const toEvent = (doc: EventDoc): TxOBEvent => ({ data: doc.data, correlation_id: doc.correlation_id, created_at: doc.created_at, + defer: doc.defer ?? null, }); const isDuplicateKey = (error: unknown): boolean => @@ -262,6 +264,7 @@ export const createMongoStore = ( data: event.data, correlation_id: event.correlation_id, created_at: event.created_at, + defer: event.defer, fanned_out_at: null, fanout_attempts: 0, fanout_run_after: event.created_at, @@ -323,7 +326,7 @@ export const createMongoStore = ( ...d, status: "pending", attempts: 0, - run_after: now, + run_after: d.run_after ?? now, lease_token: null, last_error: null, created_at: now, diff --git a/src/outbox.ts b/src/outbox.ts index 635958d..dc7fdf2 100644 --- a/src/outbox.ts +++ b/src/outbox.ts @@ -1,8 +1,10 @@ import { randomUUID } from "node:crypto"; import type { StandardSchemaV1 } from "@standard-schema/spec"; +import { normalizeDefer } from "./defer.js"; import { TxOBValidationError } from "./error.js"; import type { TxOBStore } from "./store.js"; import type { + TxOBDefer, TxOBEvent, TxOBEventSchemaMap, TxOBEventTypeOf, @@ -14,6 +16,11 @@ export type TxOBPublishOpts = { /** Defaults to a random UUID. */ id?: string; correlation_id?: string | null; + /** + * Delay every delivery of this event, overriding the handlers' `defer`. `delayMs` counts + * from publish time (`created_at`). + */ + defer?: TxOBDefer | null; }; export type TxOBPublish = < @@ -94,6 +101,7 @@ export const createOutbox = < >, correlation_id: publishOpts?.correlation_id ?? null, created_at: new Date(), + defer: normalizeDefer(publishOpts?.defer ?? null, `publish('${type}')`), }; await store.insertEvent(tx, event); return event; diff --git a/src/pg/drizzle.ts b/src/pg/drizzle.ts index 8d4ad87..4f3d88c 100644 --- a/src/pg/drizzle.ts +++ b/src/pg/drizzle.ts @@ -75,6 +75,7 @@ export const txobTables = (opts: PgTableOpts = {}) => { data: jsonb("data").notNull(), correlation_id: text("correlation_id"), created_at: timestamptz("created_at").notNull().defaultNow(), + defer: jsonb("defer"), fanned_out_at: timestamptz("fanned_out_at"), fanout_attempts: integer("fanout_attempts").notNull().default(0), fanout_run_after: timestamptz("fanout_run_after").notNull().defaultNow(), diff --git a/src/pg/kysely.ts b/src/pg/kysely.ts index b5845b3..1338bcf 100644 --- a/src/pg/kysely.ts +++ b/src/pg/kysely.ts @@ -64,6 +64,7 @@ export interface TxobEventsTable { data: unknown; correlation_id: string | null; created_at: Generated; + defer: unknown | null; fanned_out_at: Timestamp | null; fanout_attempts: Generated; fanout_run_after: Generated; diff --git a/src/pg/schema.ts b/src/pg/schema.ts index e6e5c57..fcde6f2 100644 --- a/src/pg/schema.ts +++ b/src/pg/schema.ts @@ -55,6 +55,8 @@ CREATE TABLE IF NOT EXISTS ${events} ( data JSONB NOT NULL, correlation_id TEXT, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + -- deferral requested at publish time, resolved at fan-out + defer JSONB, fanned_out_at TIMESTAMPTZ, fanout_attempts INTEGER NOT NULL DEFAULT 0, -- when the event may next be fanned out; doubles as the fan-out lease expiry diff --git a/src/pg/store.ts b/src/pg/store.ts index 10e2aed..55257d0 100644 --- a/src/pg/store.ts +++ b/src/pg/store.ts @@ -91,6 +91,7 @@ type DeliveryRow = { data: TxOBEvent["data"]; correlation_id: string | null; event_created_at: Date; + defer: TxOBEvent["defer"]; }; const toDelivery = (row: DeliveryRow): TxOBDelivery => ({ @@ -107,6 +108,7 @@ const toDelivery = (row: DeliveryRow): TxOBDelivery => ({ data: toJson(row.data), correlation_id: row.correlation_id, created_at: toDate(row.event_created_at), + defer: toJson(row.defer) ?? null, }, }); @@ -219,7 +221,7 @@ WITH slots AS ( WHERE s.rate_max IS NOT NULL AND r.event_type = s.event_type AND r.handler = s.handler AND r.claimed_at <= now() - ${ms("s.rate_ms")} ) -SELECT claimed.*, e.type, e.data, e.correlation_id, e.created_at AS event_created_at +SELECT claimed.*, e.type, e.data, e.correlation_id, e.created_at AS event_created_at, e.defer FROM claimed JOIN ${events} e ON e.id = claimed.event_id ORDER BY claimed.priority DESC`; @@ -237,7 +239,7 @@ RETURNING 1`; insertEvent: async (tx, event) => { await db.queryIn( tx, - `INSERT INTO ${events} (id, type, data, correlation_id, created_at) VALUES ($1, $2, $3::text::jsonb, $4, $5)`, + `INSERT INTO ${events} (id, type, data, correlation_id, created_at, defer) VALUES ($1, $2, $3::text::jsonb, $4, $5, $6::text::jsonb)`, [ event.id, event.type, @@ -245,6 +247,7 @@ RETURNING 1`; event.correlation_id, // ISO strings: some drivers (e.g. Drizzle on postgres.js) don't encode Dates event.created_at.toISOString(), + event.defer === null ? null : JSON.stringify(event.defer), ], ); }, @@ -264,13 +267,14 @@ FROM ( FOR UPDATE SKIP LOCKED ) claimable WHERE e.id = claimable.id -RETURNING e.id, e.type, e.data, e.correlation_id, e.created_at, e.fanout_attempts, e.fanout_lease_token`, +RETURNING e.id, e.type, e.data, e.correlation_id, e.created_at, e.defer, e.fanout_attempts, e.fanout_lease_token`, [types, limit, leaseMs], ); return rows.map((row) => ({ ...row, data: toJson(row.data), created_at: toDate(row.created_at), + defer: toJson(row.defer) ?? null, })) as TxOBFanoutEvent[]; }, @@ -280,18 +284,22 @@ RETURNING e.id, e.type, e.data, e.correlation_id, e.created_at, e.fanout_attempt await db.query( ` WITH inserted AS ( - INSERT INTO ${deliveries} (event_id, event_type, handler, priority, concurrency_key) - SELECT * FROM unnest($1::text[], $2::text[], $3::text[], $4::int[], $5::text[]) + INSERT INTO ${deliveries} (event_id, event_type, handler, priority, concurrency_key, run_after) + SELECT event_id, event_type, handler, priority, concurrency_key, COALESCE(run_after::timestamptz, now()) + -- run_after as text: postgres.js (under Drizzle) can't encode timestamptz[] parameters + FROM unnest($1::text[], $2::text[], $3::text[], $4::int[], $5::text[], $6::text[]) + AS n (event_id, event_type, handler, priority, concurrency_key, run_after) ON CONFLICT (event_id, handler) DO NOTHING ) UPDATE ${events} SET fanned_out_at = now(), fanout_lease_token = NULL -WHERE id = ANY($6::text[])`, +WHERE id = ANY($7::text[])`, [ newDeliveries.map((d) => d.event_id), newDeliveries.map((d) => d.event_type), newDeliveries.map((d) => d.handler), newDeliveries.map((d) => d.priority), newDeliveries.map((d) => d.concurrency_key), + newDeliveries.map((d) => d.run_after?.toISOString() ?? null), event_ids, ], ); diff --git a/src/processor.ts b/src/processor.ts index da69021..82d58f0 100644 --- a/src/processor.ts +++ b/src/processor.ts @@ -5,6 +5,7 @@ import { TxOBTimeoutError, TxOBValidationError, } from "./error.js"; +import { normalizeDefer, resolveDefer } from "./defer.js"; import { compileHandlerMap, type CompiledHandler, @@ -478,11 +479,15 @@ export class EventProcessor< const eventIds: string[] = []; const deliveries: TxOBNewDelivery[] = []; + let firstDue: Date | null = null; for (const event of events) { try { + // Stores hand `defer` back as plain JSON; revive `until` and reject corrupt values + const eventDefer = normalizeDefer(event.defer, `${event.type}.defer`); + const revived = { ...event, defer: eventDefer }; const eventDeliveries = (this.handlers.get(event.type) ?? []).map( (handler): TxOBNewDelivery => { - const priority = handler.priority(event); + const priority = handler.priority(revived); if ( !Number.isInteger(priority) || Math.abs(priority) > maxPriority @@ -491,18 +496,31 @@ export class EventProcessor< `${event.type}.${handler.name}: priority must be a 32-bit integer, got ${priority}`, ); } - const key = handler.perKey ? handler.perKey.key(event) : null; + const key = handler.perKey ? handler.perKey.key(revived) : null; if (key !== null && typeof key !== "string") { throw new Error( `${event.type}.${handler.name}: concurrency key must be a string or null, got ${typeof key}`, ); } + // The publisher's request wins over the handler's default + const path = `${event.type}.${handler.name}`; + const defer = + eventDefer ?? + normalizeDefer(handler.defer(revived), `${path}.defer`); + const run_after = resolveDefer(defer, event); + if ( + run_after !== null && + (firstDue === null || run_after < firstDue) + ) { + firstDue = run_after; + } return { event_id: event.id, event_type: event.type, handler: handler.name, priority, concurrency_key: key, + run_after, }; }, ); @@ -518,7 +536,7 @@ export class EventProcessor< fanoutAttempts: event.fanout_attempts, error, }, - "error computing deliveries for event (invalid priority or concurrency key)", + "error computing deliveries for event (invalid priority, concurrency key or defer)", ); } } @@ -535,6 +553,8 @@ export class EventProcessor< {}, eventIds.length, ); + // Deferred deliveries don't trigger a wakeup when they become due + if (firstDue !== null) this.scheduleTick(firstDue); } setTelemetrySpanAttributes(span, { [TxOBTelemetryAttributeKey.EventsFannedOut]: eventIds.length, @@ -834,15 +854,18 @@ export class EventProcessor< : TxOBTelemetryDeliveryOutcome.LeaseLost; } - /** Poll when a backed-off delivery becomes due instead of waiting for the next interval. */ + /** Poll when a deferred or backed-off delivery becomes due instead of waiting for the next interval. */ private scheduleTick(at: Date): void { if (this.state !== "started") return; + const delayMs = at.getTime() - Date.now(); + // A regular poll gets there first; also keeps clear of setTimeout's ~24.8 day limit + if (delayMs > this.opts.pollingIntervalMs) return; const timer = setTimeout( () => { this.timers.delete(timer); this.requestTick(); }, - Math.max(at.getTime() - Date.now(), 0), + Math.max(delayMs, 0), ); timer.unref?.(); this.timers.add(timer); diff --git a/src/redis/scripts.ts b/src/redis/scripts.ts index 4d35296..144d686 100644 --- a/src/redis/scripts.ts +++ b/src/redis/scripts.ts @@ -7,7 +7,7 @@ * The default prefix carries a hash tag so all of txob's keys live in one cluster slot. * * Layout (P = prefix): - * P e: hash: type, data, correlation_id, created_at, + * P e: hash: type, data, correlation_id, created_at, defer, * fanout_attempts, fanout_lease_token, fanned_out_at * P fanout: zset of unfanned events, by created_at * P fanoutleased: zset of events leased for fan-out, by lease expiry @@ -115,14 +115,18 @@ local function before(score_a, member_a, score_b, member_b) end `; -/** ARGV: P, id, type, data, created_at, correlation_id ('' for none, else '=' .. id) */ +/** + * ARGV: P, id, type, data, created_at, correlation_id ('' for none, else '=' .. id), + * defer ('' for none, else '=' .. JSON) + */ export const insertEventScript = `${common} -local id, t, data, created, corr = ARGV[2], ARGV[3], ARGV[4], ARGV[5], ARGV[6] +local id, t, data, created, corr, defer = ARGV[2], ARGV[3], ARGV[4], ARGV[5], ARGV[6], ARGV[7] local ek = P .. 'e:' .. id -- MULTI can't abort, so a duplicate id is ignored rather than rejected if redis.call('EXISTS', ek) == 1 then return 0 end redis.call('HSET', ek, 'type', t, 'data', data, 'created_at', created, 'fanout_attempts', 0) if corr ~= '' then redis.call('HSET', ek, 'correlation_id', string.sub(corr, 2)) end +if defer ~= '' then redis.call('HSET', ek, 'defer', string.sub(defer, 2)) end redis.call('ZADD', P .. 'fanout:' .. t, created, id) redis.call('ZADD', P .. 'events', created, id) wakeup() @@ -166,7 +170,7 @@ return out /** * ARGV: P, n, ...event ids (n), then per delivery: event_id, type, handler, priority, - * concurrency key ('' for none, else '=' .. key) + * concurrency key ('' for none, else '=' .. key), run_after ms ('' for now) */ export const completeFanoutScript = `${common} local now = now_ms() @@ -174,7 +178,7 @@ local n = tonumber(ARGV[2]) local created = 0 local i = 3 + n while i <= #ARGV do - local eid, t, h, pr, ck = ARGV[i], ARGV[i + 1], ARGV[i + 2], ARGV[i + 3], ARGV[i + 4] + local eid, t, h, pr, ck, ra = ARGV[i], ARGV[i + 1], ARGV[i + 2], ARGV[i + 3], ARGV[i + 4], ARGV[i + 5] local id = eid .. ':' .. h local dk = P .. 'd:' .. id if redis.call('EXISTS', dk) == 0 then @@ -182,10 +186,18 @@ while i <= #ARGV do 'attempts', 0, 'priority', pr, 'created_at', now) if ck ~= '' then redis.call('HSET', dk, 'concurrency_key', string.sub(ck, 2)) end redis.call('ZADD', P .. 'deliveries', now, id) - make_ready(dk, id, now) + local run_after = now + if ra ~= '' then run_after = tonumber(ra) end + if run_after > now then + -- Waits in the delayed queue until due, like a retry + redis.call('HSET', dk, 'run_after', run_after) + redis.call('ZADD', slot_key('delayed', t, h), run_after, id) + else + make_ready(dk, id, now) + end created = created + 1 end - i = i + 5 + i = i + 6 end for j = 3, 2 + n do local id = ARGV[j] diff --git a/src/redis/store.ts b/src/redis/store.ts index eb2b0ed..0be0918 100644 --- a/src/redis/store.ts +++ b/src/redis/store.ts @@ -55,6 +55,7 @@ const toEvent = (id: string, hash: Hash): TxOBEvent => ({ data: JSON.parse(hash.data), correlation_id: hash.correlation_id ?? null, created_at: new Date(Number(hash.created_at)), + defer: hash.defer ? JSON.parse(hash.defer) : null, }); const nullable = (value: string | null) => (value === null ? "" : `=${value}`); @@ -153,6 +154,7 @@ export const createRedisStore = ( JSON.stringify(event.data), String(event.created_at.getTime()), nullable(event.correlation_id), + nullable(event.defer === null ? null : JSON.stringify(event.defer)), ]); }, @@ -188,6 +190,7 @@ export const createRedisStore = ( d.handler, d.priority, nullable(d.concurrency_key), + d.run_after?.getTime() ?? "", ]), ]); }, diff --git a/src/store.ts b/src/store.ts index 59d076e..358261e 100644 --- a/src/store.ts +++ b/src/store.ts @@ -11,6 +11,8 @@ export type TxOBNewDelivery = { handler: string; priority: number; concurrency_key: string | null; + /** Earliest claim time. `null` means claimable right away. */ + run_after: Date | null; }; /** diff --git a/src/types.ts b/src/types.ts index d0c6717..d3c8519 100644 --- a/src/types.ts +++ b/src/types.ts @@ -18,6 +18,12 @@ export type TxOBSchemaOutput = export type TxOBEventTypeOf = keyof TEventSchemas & string; +/** + * When deliveries become claimable. `delayMs` counts from the event's `created_at`, so it + * resolves to the same time however often fan-out runs; `until` is an absolute time. + */ +export type TxOBDefer = { delayMs: number } | { until: Date }; + export type TxOBEvent< TEventType extends string = string, TData = TxOBEventData, @@ -27,6 +33,11 @@ export type TxOBEvent< data: TData; correlation_id: string | null; created_at: Date; + /** + * Deferral requested when the event was published. Overrides the handlers' own `defer` + * for every delivery of this event. `null` when none was requested. + */ + defer: TxOBDefer | null; }; export type TxOBEventFromSchemas< diff --git a/test/contract.ts b/test/contract.ts index 6f6f1fd..6a8c514 100644 --- a/test/contract.ts +++ b/test/contract.ts @@ -190,6 +190,7 @@ export const runStoreContract = ( handler: "email", priority: 0, concurrency_key: null, + run_after: null, }; await h.store.completeFanout({ event_ids: [event.id], @@ -221,6 +222,7 @@ export const runStoreContract = ( handler: "a", priority: i, concurrency_key: null, + run_after: null, }, { event_id: e.id, @@ -228,6 +230,7 @@ export const runStoreContract = ( handler: "b", priority: 0, concurrency_key: null, + run_after: null, }, ]), }); @@ -279,6 +282,7 @@ export const runStoreContract = ( handler: "capped", priority: 0, concurrency_key: null, + run_after: null, }, { event_id: e.id, @@ -286,6 +290,7 @@ export const runStoreContract = ( handler: "keyed", priority: 0, concurrency_key: e.data.accountId, + run_after: null, }, { event_id: e.id, @@ -293,6 +298,7 @@ export const runStoreContract = ( handler: "rated", priority: 0, concurrency_key: null, + run_after: null, }, ]), }); @@ -370,6 +376,7 @@ export const runStoreContract = ( handler: `capped${round}`, priority: 0, concurrency_key: null, + run_after: null, }, { event_id: e.id, @@ -377,6 +384,7 @@ export const runStoreContract = ( handler: `keyed${round}`, priority: 0, concurrency_key: e.data.accountId, + run_after: null, }, ]), }); @@ -433,6 +441,7 @@ export const runStoreContract = ( handler: "keyed", priority: 0, concurrency_key: e.data.accountId, + run_after: null, })), }); const slots = [ @@ -490,6 +499,7 @@ export const runStoreContract = ( handler: "a", priority: 0, concurrency_key: null, + run_after: null, })), }); const slots = [ @@ -529,6 +539,7 @@ export const runStoreContract = ( handler: "a", priority: 0, concurrency_key: null, + run_after: null, }, ], }); @@ -569,6 +580,7 @@ export const runStoreContract = ( handler: "a", priority: 0, concurrency_key: null, + run_after: null, })), }); void event; @@ -593,6 +605,54 @@ export const runStoreContract = ( ).toHaveLength(1); }); + it("holds deliveries with a future run_after until it passes", async () => { + const [event] = await publishUsers(1); + const soon = new Date(Date.now() + 400); + await h.store.completeFanout({ + event_ids: [event.id], + deliveries: [ + { + event_id: event.id, + event_type: "UserCreated", + handler: "later", + priority: 0, + concurrency_key: null, + run_after: soon, + }, + { + event_id: event.id, + event_type: "UserCreated", + handler: "now", + priority: 0, + concurrency_key: null, + run_after: null, + }, + ], + }); + const claim = () => + h.store.claimDeliveries({ + slots: ["later", "now"].map((handler) => ({ + event_type: "UserCreated", + handler, + limit: 10, + })), + limit: 10, + leaseMs: 10_000, + }); + expect((await claim()).map((d) => d.handler)).toEqual(["now"]); + // The store's clock decides when it is due; allow for skew against this process + let claimed: TxOBDelivery[] = []; + await waitFor( + async () => { + claimed = await claim(); + return claimed.length > 0; + }, + { intervalMs: 50 }, + ); + expect(Date.now()).toBeGreaterThanOrEqual(soon.getTime() - 100); + expect(claimed.map((d) => d.handler)).toEqual(["later"]); + }); + it("fences writes with the lease token", async () => { const [event] = await publishUsers(1); await h.store.completeFanout({ @@ -604,6 +664,7 @@ export const runStoreContract = ( handler: "a", priority: 0, concurrency_key: null, + run_after: null, }, ], }); @@ -910,6 +971,7 @@ export const runStoreContract = ( data: { userId: 42 }, correlation_id: null, created_at: new Date(), + defer: null, }), ); await waitFor(async () => (await deliveriesBy("dead")).length === 1); @@ -1088,6 +1150,53 @@ export const runStoreContract = ( expect(order[0]).toBe("o3"); }); + it("defers deliveries, with the publish-time defer overriding the handler's", async () => { + const ran: Record = {}; + const record = + (name: string) => + async (event: TxOBEvent): Promise => { + (ran[name] ??= []).push({ at: Date.now(), defer: event.defer }); + }; + startProcessor({ + handlerMap: { + ...baseHandlers, + UserCreated: { + immediate: record("immediate"), + delayed: { defer: { delayMs: 500 }, handler: record("delayed") }, + }, + }, + }); + const publishedAt = Date.now(); + const [plain] = await publishUsers(1); + // Overrides the handler default, so `delayed` runs right away for this one + const [rushed] = await h.transaction(async (tx) => [ + await outbox.publish( + tx, + "UserCreated", + { userId: "rush", email: "r@x.io" }, + { defer: { delayMs: 0 } }, + ), + ]); + await waitFor( + async () => (await deliveriesBy("completed")).length === 3, + { timeoutMs: 2_000 }, + ); + // Only plain.delayed is still waiting + expect(ran.immediate).toHaveLength(2); + expect(ran.delayed).toHaveLength(1); + expect(ran.delayed[0].defer).toEqual({ delayMs: 0 }); + await waitFor( + async () => (await deliveriesBy("completed")).length === 4, + { timeoutMs: 5_000 }, + ); + expect(ran.delayed).toHaveLength(2); + expect(ran.delayed[1].at - publishedAt).toBeGreaterThanOrEqual(490); + expect(ran.delayed[1].defer).toBeNull(); + expect((await h.getEvents()).map((e) => e.id).sort()).toEqual( + [plain.id, rushed.id].sort(), + ); + }); + it("leaves events of types the processor doesn't handle unfanned", async () => { const { Unhandled: _, ...withoutUnhandled } = baseHandlers; startProcessor({ diff --git a/test/units.test.ts b/test/units.test.ts index 118aa2b..56a2c10 100644 --- a/test/units.test.ts +++ b/test/units.test.ts @@ -58,6 +58,33 @@ describe("compileHandlerMap", () => { ); }); + it("normalizes a static defer and keeps a defer function", () => { + const until = new Date("2030-01-01T00:00:00Z"); + const compiled = compileHandlerMap({ + A: { + none: fn, + fixed: { handler: fn, defer: { delayMs: 500 } }, + absolute: { handler: fn, defer: { until: until.toISOString() as any } }, + dynamic: { handler: fn, defer: (e: any) => ({ delayMs: e.data.n }) }, + }, + }); + const [none, fixed, absolute, dynamic] = compiled.get("A")!; + expect(none.defer({})).toBeNull(); + expect(fixed.defer({})).toEqual({ delayMs: 500 }); + expect(absolute.defer({})).toEqual({ until }); + expect(dynamic.defer({ data: { n: 7 } })).toEqual({ delayMs: 7 }); + }); + + it.each([ + [{ handler: fn, defer: { delayMs: -1 } }, "defer.delayMs"], + [{ handler: fn, defer: { until: "soon" } }, "defer.until"], + [{ handler: fn, defer: {} }, "defer must be"], + ])("rejects invalid defer %#", (definition, message) => { + expect(() => compileHandlerMap({ A: { h: definition as any } })).toThrow( + message, + ); + }); + it("rejects handler names containing ':'", () => { expect(() => compileHandlerMap({ A: { "a:b": fn } })).toThrow( "cannot contain ':'", @@ -285,6 +312,102 @@ describe("EventProcessor with a faulty store", () => { ); }); + it("leaves an event unfanned when a handler's defer function returns an invalid value", async () => { + const { base, processor, errors } = await setup(() => ({}), { + E: { + h: { + defer: () => ({ delayMs: -5 }), + handler: async () => {}, + }, + }, + }); + await waitFor(() => errors.length > 0); + await processor.stop(); + expect(base.getEvents()[0].fanned_out_at).toBeNull(); + expect(base.getDeliveries()).toHaveLength(0); + expect(errors).toContainEqual( + expect.objectContaining({ + error: expect.objectContaining({ + message: expect.stringContaining("E.h.defer: defer.delayMs"), + }), + }), + ); + }); + + it("resolves run_after at fan-out: publish defer wins over the handler's, delayMs counts from created_at", async () => { + const seen: unknown[] = []; + const { base, processor } = await setup(() => ({}), { + E: { + fixed: { defer: { delayMs: 60_000 }, handler: async () => {} }, + fromEvent: { + defer: (event) => (event.defer ? null : { delayMs: 30_000 }), + handler: async () => {}, + }, + none: async (event) => void seen.push(event.defer), + }, + }); + const outbox = createOutbox({ eventSchemas, store: base }); + const until = new Date(Date.now() + 120_000); + const [overridden, absolute] = await base.transaction(async (tx) => [ + await outbox.publish(tx, "E", { k: "o" }, { defer: { delayMs: 5_000 } }), + await outbox.publish(tx, "E", { k: "u" }, { defer: { until } }), + ]); + await waitFor(() => base.getEvents().every((e) => e.fanned_out_at)); + await processor.stop(); + + const runAfter = (eventId: string, handler: string) => + base + .getDeliveries() + .find((d) => d.event_id === eventId && d.handler === handler)! + .run_after; + const plain = base.getEvents().find((e) => e.data.k === "a")!; + expect(runAfter(plain.id, "fixed")).toBe( + plain.created_at.getTime() + 60_000, + ); + expect(runAfter(plain.id, "fromEvent")).toBe( + plain.created_at.getTime() + 30_000, + ); + expect( + base + .getDeliveries() + .find((d) => d.event_id === plain.id && d.handler === "none")?.status, + ).toBe("completed"); + for (const handler of ["fixed", "fromEvent", "none"]) { + expect(runAfter(overridden.id, handler)).toBe( + overridden.created_at.getTime() + 5_000, + ); + expect(runAfter(absolute.id, handler)).toBe(until.getTime()); + } + expect(base.getEvents().find((e) => e.id === absolute.id)?.defer).toEqual({ + until, + }); + }); + + it("runs a deferred delivery when it becomes due, ahead of the next poll", async () => { + const ran: number[] = []; + const base = createMemoryStore(); + const processor = createEventProcessor({ + store: base, + eventSchemas, + handlerMap: { E: { h: async () => void ran.push(Date.now()) } }, + pollingIntervalMs: 60_000, + wakeupEmitter: base.wakeupEmitter, + logger: silentLogger, + }); + processor.start(); + const outbox = createOutbox({ eventSchemas, store: base }); + const publishedAt = Date.now(); + await base.transaction((tx) => + outbox.publish(tx, "E", { k: "d" }, { defer: { delayMs: 300 } }), + ); + // The insert wakes the processor up, which fans the event out; from then on only the + // due-time timer can start the delivery before the 60s poll + await waitFor(() => ran.length === 1, { timeoutMs: 5_000 }); + await processor.stop(); + expect(ran[0] - publishedAt).toBeGreaterThanOrEqual(290); + expect(ran[0] - publishedAt).toBeLessThan(3_000); + }); + it("aborts handlers whose lease could not be renewed before it expired", async () => { let abortReason: unknown; const { processor } = await setup( From ea368a1a24591e6d5d7d89e75d3d7a5b206c4d21 Mon Sep 17 00:00:00 2001 From: dillonstreator Date: Tue, 3 Nov 2026 17:12:00 -0600 Subject: [PATCH 2/5] refactor: store publish-time defer as a defer_until timestamp Resolve the publish `defer` option at publish time into `event.defer_until` instead of persisting the config object. The column is a plain TIMESTAMPTZ, validation happens once in `publish`, and stores carry a Date like `created_at` rather than a JSON blob. Handler defers still resolve at fan-out. --- .changeset/deferred-delivery.md | 4 ++-- MIGRATING.md | 8 ++++---- README.md | 6 +++--- architecture.mmd | 2 +- examples/pg/processor.ts | 4 +++- src/defer.test.ts | 2 +- src/defer.ts | 4 ++-- src/dynamodb/store.ts | 7 +++++-- src/handlers.ts | 2 +- src/memory/store.ts | 8 ++++---- src/mongodb/store.ts | 6 +++--- src/outbox.ts | 14 +++++++++----- src/pg/drizzle.ts | 2 +- src/pg/kysely.ts | 2 +- src/pg/schema.ts | 4 ++-- src/pg/store.ts | 14 +++++++------- src/processor.ts | 21 +++++++++++---------- src/redis/scripts.ts | 8 ++++---- src/redis/store.ts | 4 ++-- src/types.ts | 6 +++--- test/contract.ts | 14 +++++++++----- test/units.test.ts | 12 +++++++----- 22 files changed, 85 insertions(+), 69 deletions(-) diff --git a/.changeset/deferred-delivery.md b/.changeset/deferred-delivery.md index bd5e2d2..5584fae 100644 --- a/.changeset/deferred-delivery.md +++ b/.changeset/deferred-delivery.md @@ -2,6 +2,6 @@ "txob": minor --- -Deferred delivery: `defer: { delayMs } | { until }` on handler definitions sets when deliveries become claimable, and `publish(..., { defer })` overrides it per event. Resolved at fan-out into the delivery's `run_after`; `delayMs` counts from the event's `created_at`. +Deferred delivery: `defer: { delayMs } | { until }` on handler definitions sets when deliveries become claimable, and `publish(..., { defer })` overrides it per event. The publish-time defer is resolved into `event.defer_until` when published; handler defers are resolved at fan-out. Either way the result lands in the delivery's `run_after`, and `delayMs` counts from the event's `created_at`. -**Breaking for Postgres and custom stores:** `txob_events` gains a `defer JSONB` column (`ALTER TABLE txob_events ADD COLUMN IF NOT EXISTS defer JSONB` on existing installs), `TxOBEvent` gains `defer`, and `TxOBNewDelivery` gains `run_after`. See MIGRATING.md. +**Breaking for Postgres and custom stores:** `txob_events` gains a `defer_until TIMESTAMPTZ` column (`ALTER TABLE txob_events ADD COLUMN IF NOT EXISTS defer_until TIMESTAMPTZ` on existing installs), `TxOBEvent` gains `defer_until`, and `TxOBNewDelivery` gains `run_after`. See MIGRATING.md. diff --git a/MIGRATING.md b/MIGRATING.md index f276122..5e902df 100644 --- a/MIGRATING.md +++ b/MIGRATING.md @@ -1,17 +1,17 @@ # Migrating from 0.5 to 0.6 -0.6 adds deferred delivery (`defer` on handler definitions and `publish`). Events gain a `defer` field, and new deliveries carry a `run_after`. +0.6 adds deferred delivery (`defer` on handler definitions and `publish`). Events gain a `defer_until` timestamp, and new deliveries carry a `run_after`. - **Postgres:** add the column to existing tables. `migrate` and `getMigrationSql` include it for new installs only: ```sql - ALTER TABLE txob_events ADD COLUMN IF NOT EXISTS defer JSONB; + ALTER TABLE txob_events ADD COLUMN IF NOT EXISTS defer_until TIMESTAMPTZ; ``` Drizzle users regenerate their migration from `txobTables()`. -- **MongoDB, Redis, DynamoDB:** no migration; old events read back with `defer: null`. -- **Custom stores:** persist and return `event.defer` from `insertEvent`, and apply `run_after` (a `Date`, or `null` for "now") when `completeFanout` creates deliveries. The store contract in `test/contract.ts` checks both. +- **MongoDB, Redis, DynamoDB:** no migration; old events read back with `defer_until: null`. +- **Custom stores:** persist `event.defer_until` (a `Date` or `null`) from `insertEvent` and return it on reads, and apply `run_after` (a `Date`, or `null` for "now") when `completeFanout` creates deliveries. The store contract in `test/contract.ts` checks both. # Migrating from 0.4 to 0.5 diff --git a/README.md b/README.md index 62ab736..251a6d1 100644 --- a/README.md +++ b/README.md @@ -269,8 +269,8 @@ OrderPlaced: { await outbox.publish(tx, "OrderPlaced", data, { defer: { delayMs: 600_000 } }); ``` -- **Resolved at fan-out.** The processor turns `defer` into the delivery's `run_after` when it creates the delivery, so a change to a handler's `defer` applies to events fanned out afterwards. `delayMs` counts from `created_at`, not from fan-out, so a processor that was down for an hour doesn't push "in 5 minutes" to "in 65 minutes", and repeated fan-outs of the same event agree. -- **The publish-time `defer` wins** over the handler's for every handler of that event. The event's `defer` is also available to handler `defer` functions and handlers as `event.defer`, so a handler can opt out of honoring it. +- **Resolved once, to a timestamp.** `publish` resolves its `defer` immediately and stores it on the event as `defer_until`. The processor resolves a handler's `defer` at fan-out, so a change to it applies to events fanned out afterwards. Either way the result is the delivery's `run_after`. `delayMs` counts from `created_at`, not from fan-out, so a processor that was down for an hour doesn't push "in 5 minutes" to "in 65 minutes", and repeated fan-outs of the same event agree. +- **The publish-time `defer` wins** over the handler's for every handler of that event. The resolved time is available to handler `defer` functions and handlers as `event.defer_until`, so a handler can opt out of honoring it. - **Due deliveries are picked up promptly.** The processor that fanned the event out polls when the earliest deferred delivery is due; beyond `pollingIntervalMs`, the regular poll finds it. - **Priority still applies.** A deferred delivery competes with everything else that is due once its time comes. @@ -652,7 +652,7 @@ A store implements `TxOBStore`, where `TTx` is whatever handle your writes 1. **An atomic write with business data**, for `insertEvent`. Without this you don't have an outbox. 2. **Compare-and-set**, for lease-fenced `completeDelivery`, `failDelivery`, `extendLeases` and `releaseDeliveries`. -3. **An ordered query on due time**, for `claimEventsForFanout` and `claimDeliveries`. `completeFanout` must honor each new delivery's `run_after`, and `insertEvent` must persist `event.defer` and hand it back on later reads. +3. **An ordered query on due time**, for `claimEventsForFanout` and `claimDeliveries`. `completeFanout` must honor each new delivery's `run_after`, and `insertEvent` must persist `event.defer_until` and hand it back on later reads. The behavioral contract every store must satisfy lives in this repo at [`test/contract.ts`](./test/contract.ts) (it is not part of the npm package); copy it and run `runStoreContract(name, setup)` against your store. [`src/memory/store.ts`](./src/memory/store.ts) is a compact reference implementation. diff --git a/architecture.mmd b/architecture.mmd index e18e68c..54e9ec4 100644 --- a/architecture.mmd +++ b/architecture.mmd @@ -8,7 +8,7 @@ graph TB end subgraph Database["Store (Postgres, MongoDB, Redis, DynamoDB, ...)"] - Events[("txob_events
id, type, data, correlation_id, defer
fanned_out_at, fanout lease")] + Events[("txob_events
id, type, data, correlation_id, defer_until
fanned_out_at, fanout lease")] Deliveries[("txob_deliveries
one per (event, handler)
status, attempts, priority,
concurrency_key, run_after,
lease_token, last_error")] Notify["Wakeup source
pg_notify, change stream, pub/sub"] end diff --git a/examples/pg/processor.ts b/examples/pg/processor.ts index 7a28933..e47574f 100644 --- a/examples/pg/processor.ts +++ b/examples/pg/processor.ts @@ -50,7 +50,9 @@ const handlerMap = createEventHandlerMap({ followUp: { defer: { delayMs: 3_600_000 }, handler: async (event) => { - console.log(`${event.id} followUp ${JSON.stringify(event.defer)}`); + console.log( + `${event.id} followUp ${event.defer_until?.toISOString()}`, + ); }, }, diff --git a/src/defer.test.ts b/src/defer.test.ts index a8f88d4..25a6982 100644 --- a/src/defer.test.ts +++ b/src/defer.test.ts @@ -13,7 +13,7 @@ describe("normalizeDefer", () => { expect(normalizeDefer({ until }, "p")).toEqual({ until }); }); - it("revives an ISO string for until, as stored in JSON", () => { + it("accepts an ISO string for until", () => { const until = new Date("2030-01-01T00:00:00Z"); expect(normalizeDefer({ until: until.toISOString() } as any, "p")).toEqual({ until, diff --git a/src/defer.ts b/src/defer.ts index c301c17..47da5af 100644 --- a/src/defer.ts +++ b/src/defer.ts @@ -1,8 +1,8 @@ import type { TxOBDefer, TxOBEvent } from "./types.js"; /** - * Validates a deferral and strips anything else, so what gets persisted is exactly - * `{ delayMs }` or `{ until }`. Accepts an ISO string for `until`, as read back from JSON. + * Validates a deferral and strips anything else, leaving exactly `{ delayMs }` or `{ until }`. + * Accepts an ISO string for `until`. */ export const normalizeDefer = ( defer: TxOBDefer | null | undefined, diff --git a/src/dynamodb/store.ts b/src/dynamodb/store.ts index 1876e4a..f2d86bd 100644 --- a/src/dynamodb/store.ts +++ b/src/dynamodb/store.ts @@ -110,7 +110,8 @@ const toEvent = (item: Item): TxOBEvent => ({ data: JSON.parse(item.data), correlation_id: item.correlation_id ?? null, created_at: new Date(item.created_at), - defer: item.defer ? JSON.parse(item.defer) : null, + defer_until: + item.defer_until === undefined ? null : new Date(item.defer_until), }); const jitter = () => @@ -560,7 +561,9 @@ export const createDynamoStore = ( data: JSON.stringify(event.data), correlation_id: event.correlation_id, created_at: event.created_at.getTime(), - ...(event.defer ? { defer: JSON.stringify(event.defer) } : {}), + ...(event.defer_until + ? { defer_until: event.defer_until.getTime() } + : {}), fanout_type: event.type, fanout_run_after: event.created_at.getTime(), fanout_attempts: 0, diff --git a/src/handlers.ts b/src/handlers.ts index bb5638e..6e05f57 100644 --- a/src/handlers.ts +++ b/src/handlers.ts @@ -73,7 +73,7 @@ export type TxOBHandlerConfig = { /** * Delay before deliveries of this handler become claimable. `delayMs` counts from the * event's `created_at`. Resolved when the event is fanned out, and overridden by a `defer` - * passed to `publish`. A function receives the event (including its `defer`). + * passed to `publish`. A function receives the event (including its `defer_until`). */ defer?: TxOBDefer | ((event: TEvent) => TxOBDefer | null | undefined); }; diff --git a/src/memory/store.ts b/src/memory/store.ts index 69bdc94..5bdfb3e 100644 --- a/src/memory/store.ts +++ b/src/memory/store.ts @@ -108,7 +108,7 @@ export const createMemoryStore = (): MemoryStore => { data: event.data, correlation_id: event.correlation_id, created_at: event.created_at, - defer: event.defer, + defer_until: event.defer_until, }), }; }; @@ -135,7 +135,7 @@ export const createMemoryStore = (): MemoryStore => { data, correlation_id, created_at, - defer, + defer_until, fanned_out_at, }) => structuredClone({ @@ -144,7 +144,7 @@ export const createMemoryStore = (): MemoryStore => { data, correlation_id, created_at, - defer, + defer_until, fanned_out_at, }), ), @@ -174,7 +174,7 @@ export const createMemoryStore = (): MemoryStore => { data: e.data, correlation_id: e.correlation_id, created_at: e.created_at, - defer: e.defer, + defer_until: e.defer_until, fanout_attempts: e.fanout_attempts, fanout_lease_token: e.fanout_lease_token, }); diff --git a/src/mongodb/store.ts b/src/mongodb/store.ts index a1be1a5..d634604 100644 --- a/src/mongodb/store.ts +++ b/src/mongodb/store.ts @@ -26,7 +26,7 @@ type EventDoc = { data: TxOBEvent["data"]; correlation_id: string | null; created_at: Date; - defer: TxOBEvent["defer"]; + defer_until: Date | null; fanned_out_at: Date | null; fanout_attempts: number; fanout_run_after: Date; @@ -83,7 +83,7 @@ const toEvent = (doc: EventDoc): TxOBEvent => ({ data: doc.data, correlation_id: doc.correlation_id, created_at: doc.created_at, - defer: doc.defer ?? null, + defer_until: doc.defer_until ?? null, }); const isDuplicateKey = (error: unknown): boolean => @@ -264,7 +264,7 @@ export const createMongoStore = ( data: event.data, correlation_id: event.correlation_id, created_at: event.created_at, - defer: event.defer, + defer_until: event.defer_until, fanned_out_at: null, fanout_attempts: 0, fanout_run_after: event.created_at, diff --git a/src/outbox.ts b/src/outbox.ts index dc7fdf2..9c0e779 100644 --- a/src/outbox.ts +++ b/src/outbox.ts @@ -1,6 +1,6 @@ import { randomUUID } from "node:crypto"; import type { StandardSchemaV1 } from "@standard-schema/spec"; -import { normalizeDefer } from "./defer.js"; +import { normalizeDefer, resolveDefer } from "./defer.js"; import { TxOBValidationError } from "./error.js"; import type { TxOBStore } from "./store.js"; import type { @@ -17,8 +17,8 @@ export type TxOBPublishOpts = { id?: string; correlation_id?: string | null; /** - * Delay every delivery of this event, overriding the handlers' `defer`. `delayMs` counts - * from publish time (`created_at`). + * Delay every delivery of this event, overriding the handlers' `defer`. Resolved now into + * `event.defer_until`; `delayMs` counts from publish time (`created_at`). */ defer?: TxOBDefer | null; }; @@ -90,6 +90,7 @@ export const createOutbox = < if (!schema) { throw new Error(`unknown event type '${type}'`); } + const created_at = new Date(); const event: TxOBEvent< typeof type, TxOBSchemaOutput @@ -100,8 +101,11 @@ export const createOutbox = < TEventSchemas[typeof type] >, correlation_id: publishOpts?.correlation_id ?? null, - created_at: new Date(), - defer: normalizeDefer(publishOpts?.defer ?? null, `publish('${type}')`), + created_at, + defer_until: resolveDefer( + normalizeDefer(publishOpts?.defer ?? null, `publish('${type}')`), + { created_at }, + ), }; await store.insertEvent(tx, event); return event; diff --git a/src/pg/drizzle.ts b/src/pg/drizzle.ts index 4f3d88c..889ada2 100644 --- a/src/pg/drizzle.ts +++ b/src/pg/drizzle.ts @@ -75,7 +75,7 @@ export const txobTables = (opts: PgTableOpts = {}) => { data: jsonb("data").notNull(), correlation_id: text("correlation_id"), created_at: timestamptz("created_at").notNull().defaultNow(), - defer: jsonb("defer"), + defer_until: timestamptz("defer_until"), fanned_out_at: timestamptz("fanned_out_at"), fanout_attempts: integer("fanout_attempts").notNull().default(0), fanout_run_after: timestamptz("fanout_run_after").notNull().defaultNow(), diff --git a/src/pg/kysely.ts b/src/pg/kysely.ts index 1338bcf..3abbf85 100644 --- a/src/pg/kysely.ts +++ b/src/pg/kysely.ts @@ -64,7 +64,7 @@ export interface TxobEventsTable { data: unknown; correlation_id: string | null; created_at: Generated; - defer: unknown | null; + defer_until: Timestamp | null; fanned_out_at: Timestamp | null; fanout_attempts: Generated; fanout_run_after: Generated; diff --git a/src/pg/schema.ts b/src/pg/schema.ts index fcde6f2..7a634de 100644 --- a/src/pg/schema.ts +++ b/src/pg/schema.ts @@ -55,8 +55,8 @@ CREATE TABLE IF NOT EXISTS ${events} ( data JSONB NOT NULL, correlation_id TEXT, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), - -- deferral requested at publish time, resolved at fan-out - defer JSONB, + -- earliest claim time for this event's deliveries, from publish's defer option + defer_until TIMESTAMPTZ, fanned_out_at TIMESTAMPTZ, fanout_attempts INTEGER NOT NULL DEFAULT 0, -- when the event may next be fanned out; doubles as the fan-out lease expiry diff --git a/src/pg/store.ts b/src/pg/store.ts index 55257d0..d89cdf5 100644 --- a/src/pg/store.ts +++ b/src/pg/store.ts @@ -91,7 +91,7 @@ type DeliveryRow = { data: TxOBEvent["data"]; correlation_id: string | null; event_created_at: Date; - defer: TxOBEvent["defer"]; + defer_until: Date | null; }; const toDelivery = (row: DeliveryRow): TxOBDelivery => ({ @@ -108,7 +108,7 @@ const toDelivery = (row: DeliveryRow): TxOBDelivery => ({ data: toJson(row.data), correlation_id: row.correlation_id, created_at: toDate(row.event_created_at), - defer: toJson(row.defer) ?? null, + defer_until: row.defer_until === null ? null : toDate(row.defer_until), }, }); @@ -221,7 +221,7 @@ WITH slots AS ( WHERE s.rate_max IS NOT NULL AND r.event_type = s.event_type AND r.handler = s.handler AND r.claimed_at <= now() - ${ms("s.rate_ms")} ) -SELECT claimed.*, e.type, e.data, e.correlation_id, e.created_at AS event_created_at, e.defer +SELECT claimed.*, e.type, e.data, e.correlation_id, e.created_at AS event_created_at, e.defer_until FROM claimed JOIN ${events} e ON e.id = claimed.event_id ORDER BY claimed.priority DESC`; @@ -239,7 +239,7 @@ RETURNING 1`; insertEvent: async (tx, event) => { await db.queryIn( tx, - `INSERT INTO ${events} (id, type, data, correlation_id, created_at, defer) VALUES ($1, $2, $3::text::jsonb, $4, $5, $6::text::jsonb)`, + `INSERT INTO ${events} (id, type, data, correlation_id, created_at, defer_until) VALUES ($1, $2, $3::text::jsonb, $4, $5, $6)`, [ event.id, event.type, @@ -247,7 +247,7 @@ RETURNING 1`; event.correlation_id, // ISO strings: some drivers (e.g. Drizzle on postgres.js) don't encode Dates event.created_at.toISOString(), - event.defer === null ? null : JSON.stringify(event.defer), + event.defer_until?.toISOString() ?? null, ], ); }, @@ -267,14 +267,14 @@ FROM ( FOR UPDATE SKIP LOCKED ) claimable WHERE e.id = claimable.id -RETURNING e.id, e.type, e.data, e.correlation_id, e.created_at, e.defer, e.fanout_attempts, e.fanout_lease_token`, +RETURNING e.id, e.type, e.data, e.correlation_id, e.created_at, e.defer_until, e.fanout_attempts, e.fanout_lease_token`, [types, limit, leaseMs], ); return rows.map((row) => ({ ...row, data: toJson(row.data), created_at: toDate(row.created_at), - defer: toJson(row.defer) ?? null, + defer_until: row.defer_until === null ? null : toDate(row.defer_until), })) as TxOBFanoutEvent[]; }, diff --git a/src/processor.ts b/src/processor.ts index 82d58f0..744b316 100644 --- a/src/processor.ts +++ b/src/processor.ts @@ -482,12 +482,9 @@ export class EventProcessor< let firstDue: Date | null = null; for (const event of events) { try { - // Stores hand `defer` back as plain JSON; revive `until` and reject corrupt values - const eventDefer = normalizeDefer(event.defer, `${event.type}.defer`); - const revived = { ...event, defer: eventDefer }; const eventDeliveries = (this.handlers.get(event.type) ?? []).map( (handler): TxOBNewDelivery => { - const priority = handler.priority(revived); + const priority = handler.priority(event); if ( !Number.isInteger(priority) || Math.abs(priority) > maxPriority @@ -496,18 +493,22 @@ export class EventProcessor< `${event.type}.${handler.name}: priority must be a 32-bit integer, got ${priority}`, ); } - const key = handler.perKey ? handler.perKey.key(revived) : null; + const key = handler.perKey ? handler.perKey.key(event) : null; if (key !== null && typeof key !== "string") { throw new Error( `${event.type}.${handler.name}: concurrency key must be a string or null, got ${typeof key}`, ); } // The publisher's request wins over the handler's default - const path = `${event.type}.${handler.name}`; - const defer = - eventDefer ?? - normalizeDefer(handler.defer(revived), `${path}.defer`); - const run_after = resolveDefer(defer, event); + const run_after = + event.defer_until ?? + resolveDefer( + normalizeDefer( + handler.defer(event), + `${event.type}.${handler.name}.defer`, + ), + event, + ); if ( run_after !== null && (firstDue === null || run_after < firstDue) diff --git a/src/redis/scripts.ts b/src/redis/scripts.ts index 144d686..e410f91 100644 --- a/src/redis/scripts.ts +++ b/src/redis/scripts.ts @@ -7,7 +7,7 @@ * The default prefix carries a hash tag so all of txob's keys live in one cluster slot. * * Layout (P = prefix): - * P e: hash: type, data, correlation_id, created_at, defer, + * P e: hash: type, data, correlation_id, created_at, defer_until, * fanout_attempts, fanout_lease_token, fanned_out_at * P fanout: zset of unfanned events, by created_at * P fanoutleased: zset of events leased for fan-out, by lease expiry @@ -117,16 +117,16 @@ end /** * ARGV: P, id, type, data, created_at, correlation_id ('' for none, else '=' .. id), - * defer ('' for none, else '=' .. JSON) + * defer_until ('' for none, else '=' .. ms since epoch) */ export const insertEventScript = `${common} -local id, t, data, created, corr, defer = ARGV[2], ARGV[3], ARGV[4], ARGV[5], ARGV[6], ARGV[7] +local id, t, data, created, corr, defer_until = ARGV[2], ARGV[3], ARGV[4], ARGV[5], ARGV[6], ARGV[7] local ek = P .. 'e:' .. id -- MULTI can't abort, so a duplicate id is ignored rather than rejected if redis.call('EXISTS', ek) == 1 then return 0 end redis.call('HSET', ek, 'type', t, 'data', data, 'created_at', created, 'fanout_attempts', 0) if corr ~= '' then redis.call('HSET', ek, 'correlation_id', string.sub(corr, 2)) end -if defer ~= '' then redis.call('HSET', ek, 'defer', string.sub(defer, 2)) end +if defer_until ~= '' then redis.call('HSET', ek, 'defer_until', string.sub(defer_until, 2)) end redis.call('ZADD', P .. 'fanout:' .. t, created, id) redis.call('ZADD', P .. 'events', created, id) wakeup() diff --git a/src/redis/store.ts b/src/redis/store.ts index 0be0918..92d2644 100644 --- a/src/redis/store.ts +++ b/src/redis/store.ts @@ -55,7 +55,7 @@ const toEvent = (id: string, hash: Hash): TxOBEvent => ({ data: JSON.parse(hash.data), correlation_id: hash.correlation_id ?? null, created_at: new Date(Number(hash.created_at)), - defer: hash.defer ? JSON.parse(hash.defer) : null, + defer_until: hash.defer_until ? new Date(Number(hash.defer_until)) : null, }); const nullable = (value: string | null) => (value === null ? "" : `=${value}`); @@ -154,7 +154,7 @@ export const createRedisStore = ( JSON.stringify(event.data), String(event.created_at.getTime()), nullable(event.correlation_id), - nullable(event.defer === null ? null : JSON.stringify(event.defer)), + nullable(event.defer_until?.getTime().toString() ?? null), ]); }, diff --git a/src/types.ts b/src/types.ts index d3c8519..81580d8 100644 --- a/src/types.ts +++ b/src/types.ts @@ -34,10 +34,10 @@ export type TxOBEvent< correlation_id: string | null; created_at: Date; /** - * Deferral requested when the event was published. Overrides the handlers' own `defer` - * for every delivery of this event. `null` when none was requested. + * Earliest time deliveries of this event may be claimed, resolved at publish time from the + * `defer` option. Overrides the handlers' own `defer`. `null` when none was requested. */ - defer: TxOBDefer | null; + defer_until: Date | null; }; export type TxOBEventFromSchemas< diff --git a/test/contract.ts b/test/contract.ts index 6a8c514..9535693 100644 --- a/test/contract.ts +++ b/test/contract.ts @@ -971,7 +971,7 @@ export const runStoreContract = ( data: { userId: 42 }, correlation_id: null, created_at: new Date(), - defer: null, + defer_until: null, }), ); await waitFor(async () => (await deliveriesBy("dead")).length === 1); @@ -1151,11 +1151,15 @@ export const runStoreContract = ( }); it("defers deliveries, with the publish-time defer overriding the handler's", async () => { - const ran: Record = {}; + const ran: Record = + {}; const record = (name: string) => async (event: TxOBEvent): Promise => { - (ran[name] ??= []).push({ at: Date.now(), defer: event.defer }); + (ran[name] ??= []).push({ + at: Date.now(), + defer_until: event.defer_until, + }); }; startProcessor({ handlerMap: { @@ -1184,14 +1188,14 @@ export const runStoreContract = ( // Only plain.delayed is still waiting expect(ran.immediate).toHaveLength(2); expect(ran.delayed).toHaveLength(1); - expect(ran.delayed[0].defer).toEqual({ delayMs: 0 }); + expect(ran.delayed[0].defer_until).toEqual(rushed.created_at); await waitFor( async () => (await deliveriesBy("completed")).length === 4, { timeoutMs: 5_000 }, ); expect(ran.delayed).toHaveLength(2); expect(ran.delayed[1].at - publishedAt).toBeGreaterThanOrEqual(490); - expect(ran.delayed[1].defer).toBeNull(); + expect(ran.delayed[1].defer_until).toBeNull(); expect((await h.getEvents()).map((e) => e.id).sort()).toEqual( [plain.id, rushed.id].sort(), ); diff --git a/test/units.test.ts b/test/units.test.ts index 56a2c10..53bb1d0 100644 --- a/test/units.test.ts +++ b/test/units.test.ts @@ -340,10 +340,10 @@ describe("EventProcessor with a faulty store", () => { E: { fixed: { defer: { delayMs: 60_000 }, handler: async () => {} }, fromEvent: { - defer: (event) => (event.defer ? null : { delayMs: 30_000 }), + defer: (event) => (event.defer_until ? null : { delayMs: 30_000 }), handler: async () => {}, }, - none: async (event) => void seen.push(event.defer), + none: async (event) => void seen.push(event.defer_until), }, }); const outbox = createOutbox({ eventSchemas, store: base }); @@ -378,9 +378,11 @@ describe("EventProcessor with a faulty store", () => { ); expect(runAfter(absolute.id, handler)).toBe(until.getTime()); } - expect(base.getEvents().find((e) => e.id === absolute.id)?.defer).toEqual({ - until, - }); + expect( + base.getEvents().find((e) => e.id === absolute.id)?.defer_until, + ).toEqual(until); + // Only the undeferred event ran before stop + expect(seen).toEqual([null]); }); it("runs a deferred delivery when it becomes due, ahead of the next poll", async () => { From 203bf995697c19ede724a9b9e8d0f5db93f8d389 Mon Sep 17 00:00:00 2001 From: dillonstreator Date: Wed, 4 Nov 2026 19:42:54 -0600 Subject: [PATCH 3/5] ci: fail fast and dump logs when MongoDB does not start --- .github/workflows/coverage.yml | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index 6f7e222..8d93398 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -38,11 +38,17 @@ jobs: - name: Start MongoDB replica set run: | docker run -d --name mongo -p 27017:27017 mongo:8 --replSet rs0 --bind_ip_all + fail() { echo "::error::$1"; docker ps -a; docker logs mongo; exit 1; } for i in $(seq 1 30); do docker exec mongo mongosh --quiet --eval 'rs.initiate({_id:"rs0",members:[{_id:0,host:"localhost:27017"}]}).ok' && break + [ "$i" = 30 ] && fail "rs.initiate never succeeded" sleep 1 done - until docker exec mongo mongosh --quiet --eval 'rs.status().members[0].stateStr' | grep -q PRIMARY; do sleep 1; done + for i in $(seq 1 30); do + docker exec mongo mongosh --quiet --eval 'rs.status().members[0].stateStr' | grep -q PRIMARY && exit 0 + sleep 1 + done + fail "replica set never became PRIMARY" - name: Start Redis and DynamoDB Local run: | From a8d20dde9b332d812ab9d864d05368b6c4f0a261 Mon Sep 17 00:00:00 2001 From: dillonstreator Date: Wed, 4 Nov 2026 19:43:08 -0600 Subject: [PATCH 4/5] ci: pin runner to ubuntu-24.04 until mongod 8 starts on the 26.04 kernel --- .github/workflows/coverage.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index 8d93398..aea296b 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -4,7 +4,7 @@ on: [push, pull_request] jobs: build: - runs-on: ubuntu-latest + runs-on: ubuntu-24.04 # ubuntu-26.04 kernel 7.0.x cannot run mongod 8 (rseq ABI); revisit when runner kernel >= 7.0.14 services: postgres: From 8581d28604a083c0ae32085b0db9ee8e9dee8a48 Mon Sep 17 00:00:00 2001 From: dillonstreator Date: Wed, 4 Nov 2026 19:47:17 -0600 Subject: [PATCH 5/5] ci: test against mongo:9 while mongo:8 images refuse the runner kernel mongod 8.0.21-8.0.36 and 8.3.0-8.3.15 exit at startup on the GitHub runners' Ubuntu kernels: their kernel 6.19+ guard does not recognise Ubuntu's version numbering as already carrying the rseq fix (SERVER-131779). The exemption ships in 8.3.16 / 8.0.37, which Docker Hub has not published yet; 9.0.4 has it. Also drops the ubuntu-24.04 pin: that image's kernel is affected too. --- .github/workflows/coverage.yml | 6 ++++-- test/mongodb.test.ts | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index aea296b..6e24acc 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -4,7 +4,7 @@ on: [push, pull_request] jobs: build: - runs-on: ubuntu-24.04 # ubuntu-26.04 kernel 7.0.x cannot run mongod 8 (rseq ABI); revisit when runner kernel >= 7.0.14 + runs-on: ubuntu-latest services: postgres: @@ -37,7 +37,9 @@ jobs: - name: Start MongoDB replica set run: | - docker run -d --name mongo -p 27017:27017 mongo:8 --replSet rs0 --bind_ip_all + # mongo:9 until the mongo:8 image reaches 8.3.16 / 8.0.37: earlier mongod 8 builds refuse to + # start on the runners' Ubuntu kernels (SERVER-131779) + docker run -d --name mongo -p 27017:27017 mongo:9 --replSet rs0 --bind_ip_all fail() { echo "::error::$1"; docker ps -a; docker logs mongo; exit 1; } for i in $(seq 1 30); do docker exec mongo mongosh --quiet --eval 'rs.initiate({_id:"rs0",members:[{_id:0,host:"localhost:27017"}]}).ok' && break diff --git a/test/mongodb.test.ts b/test/mongodb.test.ts index 935175c..ebb8ebb 100644 --- a/test/mongodb.test.ts +++ b/test/mongodb.test.ts @@ -11,7 +11,7 @@ import { runStoreContract } from "./contract.js"; /** * Integration tests against a real MongoDB replica set. Skipped unless TXOB_TEST_MONGO_URL is * set, e.g. - * docker run --rm -p 27019:27017 mongo:8 --replSet rs0 --bind_ip_all + * docker run --rm -p 27019:27017 mongo:9 --replSet rs0 --bind_ip_all * docker exec mongosh --eval 'rs.initiate({_id:"rs0",members:[{_id:0,host:"localhost:27017"}]})' * TXOB_TEST_MONGO_URL="mongodb://localhost:27019/?directConnection=true" yarn test */