Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changeset/deferred-delivery.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
"txob": minor
---

Deferred delivery: `defer: { delayMs } | { until }` on handler definitions sets when deliveries become claimable, and `publish(..., { defer })` overrides it per event. The publish-time defer is resolved into `event.defer_until` when published; handler defers are resolved at fan-out. Either way the result lands in the delivery's `run_after`, and `delayMs` counts from the event's `created_at`.

**Breaking for Postgres and custom stores:** `txob_events` gains a `defer_until TIMESTAMPTZ` column (`ALTER TABLE txob_events ADD COLUMN IF NOT EXISTS defer_until TIMESTAMPTZ` on existing installs), `TxOBEvent` gains `defer_until`, and `TxOBNewDelivery` gains `run_after`. See MIGRATING.md.
12 changes: 10 additions & 2 deletions .github/workflows/coverage.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,12 +37,20 @@ jobs:

- name: Start MongoDB replica set
run: |
docker run -d --name mongo -p 27017:27017 mongo:8 --replSet rs0 --bind_ip_all
# mongo:9 until the mongo:8 image reaches 8.3.16 / 8.0.37: earlier mongod 8 builds refuse to
# start on the runners' Ubuntu kernels (SERVER-131779)
docker run -d --name mongo -p 27017:27017 mongo:9 --replSet rs0 --bind_ip_all
fail() { echo "::error::$1"; docker ps -a; docker logs mongo; exit 1; }
for i in $(seq 1 30); do
docker exec mongo mongosh --quiet --eval 'rs.initiate({_id:"rs0",members:[{_id:0,host:"localhost:27017"}]}).ok' && break
[ "$i" = 30 ] && fail "rs.initiate never succeeded"
sleep 1
done
until docker exec mongo mongosh --quiet --eval 'rs.status().members[0].stateStr' | grep -q PRIMARY; do sleep 1; done
for i in $(seq 1 30); do
docker exec mongo mongosh --quiet --eval 'rs.status().members[0].stateStr' | grep -q PRIMARY && exit 0
sleep 1
done
fail "replica set never became PRIMARY"

- name: Start Redis and DynamoDB Local
run: |
Expand Down
15 changes: 15 additions & 0 deletions MIGRATING.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,18 @@
# Migrating from 0.5 to 0.6

0.6 adds deferred delivery (`defer` on handler definitions and `publish`). Events gain a `defer_until` timestamp, and new deliveries carry a `run_after`.

- **Postgres:** add the column to existing tables. `migrate` and `getMigrationSql` include it for new installs only:

```sql
ALTER TABLE txob_events ADD COLUMN IF NOT EXISTS defer_until TIMESTAMPTZ;
```

Drizzle users regenerate their migration from `txobTables()`.

- **MongoDB, Redis, DynamoDB:** no migration; old events read back with `defer_until: null`.
- **Custom stores:** persist `event.defer_until` (a `Date` or `null`) from `insertEvent` and return it on reads, and apply `run_after` (a `Date`, or `null` for "now") when `completeFanout` creates deliveries. The store contract in `test/contract.ts` checks both.

# Migrating from 0.4 to 0.5

0.5 reworks the processing model and is not backwards compatible. The ideas you're used to stay the same: typed event schemas, a handler map keyed by event type, at-least-once delivery. Storage, the processing model and most APIs change.
Expand Down
36 changes: 33 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,8 @@ try {

`publish` validates the data against the event's schema (throwing `TxOBValidationError` on failure) and inserts it using the connection you pass, so it commits or rolls back with your data.

Options: `id` (default: a random UUID), `correlation_id`, and `defer` to hold every delivery of the event back, overriding the handlers' own `defer` (see [Deferred delivery](#deferred-delivery)).

**4. Process them:**

```typescript
Expand Down Expand Up @@ -239,15 +241,39 @@ Each handler is either a function or a config object:
timeoutMs: 30_000, // abort the handler's signal after this long

priority: (event) => (event.data.plan === "enterprise" ? 10 : 0), // or a number; higher first

defer: { delayMs: 60_000 }, // or { until: Date }, or a function of the event; see below
}
```

- **Concurrency, per-key concurrency and rate limits are global.** `concurrency: 5` means at most 5 deliveries of the handler run at once across all processors. Running deliveries are counted by their live leases, so a processor that crashes keeps holding its share until its leases expire (at most `leaseMs`).
- **Claims for a limited handler are serialized** (a transaction-scoped advisory lock on Postgres, a short-lived lock document on MongoDB). Claims take milliseconds, but if one handler sees very high claim rates, prefer a higher `concurrency` on the processor over many tiny claims.
- **`priority` and the `perKey` key are computed at fan-out** and stored with the delivery, so they're evaluated once per event, not per attempt.
- **`priority`, the `perKey` key and `defer` are computed at fan-out** and stored with the delivery, so they're evaluated once per event, not per attempt.
- **`timeoutMs` cancels the handler's context.** When it elapses, the `signal` passed to the handler is aborted with a `TxOBTimeoutError` reason. Honoring it is up to the handler: pass the signal to `fetch`, database drivers and so on, or check `signal.aborted`. Whatever the handler then does decides the outcome. If it throws, the attempt fails and is retried; if it finishes successfully anyway, the delivery is completed. The lease stays renewed while the handler runs.
- **Process-wide defaults** go in `handlerDefaults: { retry, timeoutMs }`.

### Deferred delivery

`defer` holds a delivery back until a point in time. It takes `{ delayMs }`, counted from the event's `created_at`, or `{ until: Date }`. The handler definition sets the default; `publish` can override it per event:

```typescript
// Handler default: send the reminder a day after the order, unless the event says otherwise
OrderPlaced: {
sendReminder: {
defer: (event) => (event.data.reminderAt ? { until: event.data.reminderAt } : { delayMs: 86_400_000 }),
handler: async (event) => {},
},
},

// Publish-time override: every handler of this event waits 10 minutes
await outbox.publish(tx, "OrderPlaced", data, { defer: { delayMs: 600_000 } });
```

- **Resolved once, to a timestamp.** `publish` resolves its `defer` immediately and stores it on the event as `defer_until`. The processor resolves a handler's `defer` at fan-out, so a change to it applies to events fanned out afterwards. Either way the result is the delivery's `run_after`. `delayMs` counts from `created_at`, not from fan-out, so a processor that was down for an hour doesn't push "in 5 minutes" to "in 65 minutes", and repeated fan-outs of the same event agree.
- **The publish-time `defer` wins** over the handler's for every handler of that event. The resolved time is available to handler `defer` functions and handlers as `event.defer_until`, so a handler can opt out of honoring it.
- **Due deliveries are picked up promptly.** The processor that fanned the event out polls when the earliest deferred delivery is due; beyond `pollingIntervalMs`, the regular poll finds it.
- **Priority still applies.** A deferred delivery competes with everything else that is due once its time comes.

### Tuning at runtime

Limits aren't stored on queued deliveries. The processor passes its current settings with every claim, so raising a limit applies to work that is already queued, whether you redeploy with new values or change them live:
Expand All @@ -274,7 +300,7 @@ processor.getConfig(); // effective settings per handler, after overrides
- **Typed:** event types, handler names and values are checked against your handler map.
- **Declarative and idempotent:** `handlers` replaces any previous overrides, and handlers it doesn't mention revert to their definitions. Applying the same config twice changes nothing. Invalid config throws and leaves the current settings untouched.
- **When it takes effect:** from the next claim. Attempts already running keep the settings they started with.
- **What can't be changed live:** `priority` and the `perKey` key function. They are computed when an event fans out, so changes apply to events fanned out afterwards; the per-key _limit_ is tunable via `perKeyConcurrency`.
- **What can't be changed live:** `priority`, `defer` and the `perKey` key function. They are computed when an event fans out, so changes apply to events fanned out afterwards; the per-key _limit_ is tunable via `perKeyConcurrency`.
- **Scope:** `configure` changes this processor only. Global limits are checked with the claiming processor's settings, so apply the same config to every processor (shared config, a feature flag, or a broadcast to all instances) to keep them in agreement.

### Errors
Expand Down Expand Up @@ -427,6 +453,10 @@ SELECT event_id, handler, attempts, last_error FROM txob_deliveries WHERE status
-- Retry a dead delivery
UPDATE txob_deliveries SET status = 'pending', attempts = 0, run_after = now()
WHERE event_id = $1 AND handler = $2;

-- Deliveries waiting on a deferral
SELECT event_id, handler, run_after FROM txob_deliveries
WHERE status = 'pending' AND lease_token IS NULL AND run_after > now();
```

## MongoDB
Expand Down Expand Up @@ -622,7 +652,7 @@ A store implements `TxOBStore<TTx>`, where `TTx` is whatever handle your writes

1. **An atomic write with business data**, for `insertEvent`. Without this you don't have an outbox.
2. **Compare-and-set**, for lease-fenced `completeDelivery`, `failDelivery`, `extendLeases` and `releaseDeliveries`.
3. **An ordered query on due time**, for `claimEventsForFanout` and `claimDeliveries`.
3. **An ordered query on due time**, for `claimEventsForFanout` and `claimDeliveries`. `completeFanout` must honor each new delivery's `run_after`, and `insertEvent` must persist `event.defer_until` and hand it back on later reads.

The behavioral contract every store must satisfy lives in this repo at [`test/contract.ts`](./test/contract.ts) (it is not part of the npm package); copy it and run `runStoreContract(name, setup)` against your store. [`src/memory/store.ts`](./src/memory/store.ts) is a compact reference implementation.

Expand Down
6 changes: 3 additions & 3 deletions architecture.mmd
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@ graph TB
subgraph Application["Your Application"]
AppTx["BEGIN"]
AppData["Write business data"]
AppPublish["outbox.publish(tx, type, data)<br/>validates with the event's schema"]
AppPublish["outbox.publish(tx, type, data, { defer? })<br/>validates with the event's schema"]
AppCommit["COMMIT"]
AppTx --> AppData --> AppPublish --> AppCommit
end

subgraph Database["Store (Postgres, MongoDB, Redis, DynamoDB, ...)"]
Events[("txob_events<br/>id, type, data, correlation_id<br/>fanned_out_at, fanout lease")]
Events[("txob_events<br/>id, type, data, correlation_id, defer_until<br/>fanned_out_at, fanout lease")]
Deliveries[("txob_deliveries<br/>one per (event, handler)<br/>status, attempts, priority,<br/>concurrency_key, run_after,<br/>lease_token, last_error")]
Notify["Wakeup source<br/>pg_notify, change stream, pub/sub"]
end
Expand All @@ -20,7 +20,7 @@ graph TB
subgraph Processor["Event Processor (run N for scale)"]
Wakeup["Wakeup emitter<br/>(optional; polling otherwise)"]
Poll["Poll (interval, wakeup,<br/>completion, backoff timers)"]
Fanout["Fan-out<br/>lease unfanned events of handled types,<br/>create deliveries + mark fanned out<br/>(one statement, idempotent)"]
Fanout["Fan-out<br/>lease unfanned events of handled types,<br/>resolve priority, concurrency key, run_after (defer),<br/>create deliveries + mark fanned out<br/>(one statement, idempotent)"]
Claim["Claim deliveries, priority order<br/>within limits counted across ALL processors:<br/>concurrency, per-key concurrency, rate limit<br/>(serialized per limited handler),<br/>set lease + token, attempts+1"]
Run["Run handler<br/>signal aborted on: shutdown | timeoutMs | lease lost<br/>handler's own result decides the outcome"]
Heartbeat["Heartbeat<br/>renew leases every leaseMs/3"]
Expand Down
10 changes: 10 additions & 0 deletions examples/pg/processor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,16 @@ const handlerMap = createEventHandlerMap({
},
},

// Runs an hour after the event was published, unless publish() asked for a different delay
followUp: {
defer: { delayMs: 3_600_000 },
handler: async (event) => {
console.log(
`${event.id} followUp ${event.defer_until?.toISOString()}`,
);
},
},

// Per-account updates must not race: one at a time per account, accounts in parallel
updateAccountStats: {
concurrency: {
Expand Down
48 changes: 48 additions & 0 deletions src/defer.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
import { describe, expect, it } from "vitest";
import { normalizeDefer, resolveDefer } from "./defer.js";

describe("normalizeDefer", () => {
it("passes through null and well-formed values, dropping extra keys", () => {
expect(normalizeDefer(null, "p")).toBeNull();
expect(normalizeDefer(undefined, "p")).toBeNull();
expect(normalizeDefer({ delayMs: 0 }, "p")).toEqual({ delayMs: 0 });
expect(normalizeDefer({ delayMs: 5, extra: 1 } as any, "p")).toEqual({
delayMs: 5,
});
const until = new Date("2030-01-01T00:00:00Z");
expect(normalizeDefer({ until }, "p")).toEqual({ until });
});

it("accepts an ISO string for until", () => {
const until = new Date("2030-01-01T00:00:00Z");
expect(normalizeDefer({ until: until.toISOString() } as any, "p")).toEqual({
until,
});
});

it.each([
[{ delayMs: -1 }, "non-negative"],
[{ delayMs: Infinity }, "non-negative"],
[{ delayMs: "5" }, "non-negative"],
[{ until: "nope" }, "valid date"],
[{ until: new Date(NaN) }, "valid date"],
[{}, "{ delayMs } or { until }"],
[5, "must be an object"],
])("rejects %j", (defer, message) => {
expect(() => normalizeDefer(defer as any, "p")).toThrow(message);
expect(() => normalizeDefer(defer as any, "p")).toThrow("p:");
});
});

describe("resolveDefer", () => {
const created_at = new Date("2030-01-01T00:00:00Z");

it("counts delayMs from created_at and passes until through", () => {
expect(resolveDefer(null, { created_at })).toBeNull();
expect(resolveDefer({ delayMs: 60_000 }, { created_at })).toEqual(
new Date("2030-01-01T00:01:00Z"),
);
const until = new Date("2031-01-01T00:00:00Z");
expect(resolveDefer({ until }, { created_at })).toBe(until);
});
});
51 changes: 51 additions & 0 deletions src/defer.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
import type { TxOBDefer, TxOBEvent } from "./types.js";

/**
* Validates a deferral and strips anything else, leaving exactly `{ delayMs }` or `{ until }`.
* Accepts an ISO string for `until`.
*/
export const normalizeDefer = (
defer: TxOBDefer | null | undefined,
path: string,
): TxOBDefer | null => {
if (defer === null || defer === undefined) return null;
if (typeof defer !== "object") {
throw new Error(`${path}: defer must be an object, received ${defer}`);
}
if ("until" in defer) {
const until =
defer.until instanceof Date
? defer.until
: new Date(defer.until as unknown as string);
if (Number.isNaN(until.getTime())) {
throw new Error(`${path}: defer.until must be a valid date`);
}
return { until };
}
if ("delayMs" in defer) {
const { delayMs } = defer;
if (
!(typeof delayMs === "number" && Number.isFinite(delayMs)) ||
delayMs < 0
) {
throw new Error(
`${path}: defer.delayMs must be a non-negative number, received ${delayMs}`,
);
}
return { delayMs };
}
throw new Error(`${path}: defer must be { delayMs } or { until }`);
};

/**
* The earliest claim time a deferral stands for, or `null` for "now". `delayMs` counts from
* `created_at` so repeated fan-outs of the same event agree.
*/
export const resolveDefer = (
defer: TxOBDefer | null,
event: Pick<TxOBEvent, "created_at">,
): Date | null => {
if (defer === null) return null;
if ("until" in defer) return defer.until;
return new Date(event.created_at.getTime() + defer.delayMs);
};
12 changes: 9 additions & 3 deletions src/dynamodb/store.ts
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,8 @@ const toEvent = (item: Item): TxOBEvent => ({
data: JSON.parse(item.data),
correlation_id: item.correlation_id ?? null,
created_at: new Date(item.created_at),
defer_until:
item.defer_until === undefined ? null : new Date(item.defer_until),
});

const jitter = () =>
Expand Down Expand Up @@ -559,6 +561,9 @@ export const createDynamoStore = (
data: JSON.stringify(event.data),
correlation_id: event.correlation_id,
created_at: event.created_at.getTime(),
...(event.defer_until
? { defer_until: event.defer_until.getTime() }
: {}),
fanout_type: event.type,
fanout_run_after: event.created_at.getTime(),
fanout_attempts: 0,
Expand Down Expand Up @@ -663,8 +668,9 @@ export const createDynamoStore = (
),
);
await Promise.all(
deliveries.map(async (d) => {
deliveries.map(async ({ run_after, ...d }) => {
const id = deliveryId(d);
const runAfter = run_after?.getTime() ?? now;
try {
await doc.send(
new PutCommand({
Expand All @@ -674,10 +680,10 @@ export const createDynamoStore = (
...d,
status: "pending",
attempts: 0,
run_after: now,
run_after: runAfter,
created_at: now,
queue_pk: queuePk(d.event_type, d.handler, d.priority),
queue_sk: queueSk(now, id),
queue_sk: queueSk(runAfter, id),
},
ConditionExpression: "attribute_not_exists(pk)",
}),
Expand Down
14 changes: 14 additions & 0 deletions src/handlers.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
import { normalizeDefer } from "./defer.js";
import type {
TxOBDefer,
TxOBEventFromSchemas,
TxOBEventSchemaMap,
TxOBEventTypeOf,
Expand Down Expand Up @@ -68,6 +70,12 @@ export type TxOBHandlerConfig<TEvent> = {
timeoutMs?: number;
/** Higher runs first. Computed when the event is fanned out. Default: 0. */
priority?: number | ((event: TEvent) => number);
/**
* Delay before deliveries of this handler become claimable. `delayMs` counts from the
* event's `created_at`. Resolved when the event is fanned out, and overridden by a `defer`
* passed to `publish`. A function receives the event (including its `defer_until`).
*/
defer?: TxOBDefer | ((event: TEvent) => TxOBDefer | null | undefined);
};

export type TxOBHandlerDefinition<TEvent> =
Expand Down Expand Up @@ -130,6 +138,7 @@ export type CompiledHandler = {
backoff: (ctx: TxOBBackoffContext<any>) => number | Date;
timeoutMs?: number;
priority: (event: any) => number;
defer: (event: any) => TxOBDefer | null | undefined;
};

const assertPositiveInt = (value: number, path: string): void => {
Expand Down Expand Up @@ -230,6 +239,10 @@ export const compileHandlerMap = (
}

const priority = config.priority ?? 0;
const defer =
typeof config.defer === "function"
? config.defer
: normalizeDefer(config.defer, `${path}.defer`);

compiledHandlers.push({
eventType,
Expand All @@ -242,6 +255,7 @@ export const compileHandlerMap = (
backoff: retry.backoff ?? defaultBackoff,
timeoutMs,
priority: typeof priority === "function" ? priority : () => priority,
defer: typeof defer === "function" ? defer : () => defer,
});
}
compiled.set(eventType, compiledHandlers);
Expand Down
1 change: 1 addition & 0 deletions src/index.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
export * from "./processor.js";
export * from "./outbox.js";
export * from "./defer.js";
export * from "./handlers.js";
export * from "./store.js";
export * from "./types.js";
Expand Down
Loading
Loading