Add a conformance harness for River's Go, Rust, and JavaScript clients by bgentry · Pull Request #1435 · riverqueue/river

bgentry · 2026-10-05T15:18:51Z

Before this merges

Rebase onto master after the Rust 0.2.0 and 0.3.0 releases, the start_with_graceful_stop and WorkerRegistry to Workers renames, the JavaScript MPL-2.0 license change, and the JavaScript 0.2.0 release. The Rust adapter takes the workspace version (0.3.0) and uses Workers, and the JavaScript adapter is licensed MPL-2.0 and follows JavaScript's own version line (0.2.0) like the other packages.
Record the Rust behavior changes under Unreleased in the Rust changelog: i32 attempt counts with PostgreSQL clamping (a breaking type change) and JobCancelError.
CI green on the rebased branch, including the new Conformance jobs for Go, Rust, and JavaScript.
Decide whether the breaking i32 attempt count change should ship in a Rust 0.4.0 release soon after this merges, or wait for more changes.
Record the five JavaScript Go-parity fixes under Unreleased in the JavaScript changelog. JavaScript 0.2.0 shipped without them, so each now changes released behavior.
Decide whether JavaScript's next release is 0.2.1 or 0.3.0: rejecting an empty insertMany batch can break a caller that relied on getting [] back.

River now has three implementations, Go, Rust, and JavaScript, and an application can run any mix of them against one database. They only work together if they agree on everything they share there: the rows they write, the unique keys they compute, the notifications they send, who holds leadership, and how a stuck job gets rescued. Each port's own tests can't prove that, because they never run next to another implementation. For example, River Go can insert a job with 40,000 maximum attempts on SQLite, but the Rust client read it back as 32,767 and couldn't insert one at all, and the JavaScript client rejected it too. A JavaScript leader whose term had been taken over kept running maintenance until its own deadline passed. None of that shows up until two implementations share a database.

This adds a harness that puts them in one. River Go is the reference, and a candidate (Go itself, Rust, or JavaScript) runs each scenario with it: one inserts and the other works, one leads and the other takes over, one cancels a job the other is running, and both must leave the database as Go alone would. Every scenario runs on PostgreSQL and SQLite.

Implementations talk to the harness through a 14-method JSON-RPC contract over stdin and stdout (conformance/protocol), and the harness drives them through a typed Go client with one method per contract method. Each language has one adapter serving both drivers, since River's driver interface already hides the difference: Go's in conformance/cmd, Rust's as the unpublished riverqueue-conformance crate, and JavaScript's as the private @riverqueue/conformance workspace package. The harness doesn't ask adapters what's in the database. It reads rows, leadership, queues, and notifications itself with SQL, and injects faults the same way (expiring a leader, terminating connections, holding locks). Each scenario gets its own schema on PostgreSQL or its own file on SQLite, so they all run in parallel. Go test names are the scenario list, with no registry or manifest beside them.

There are two tiers. On a pull request, the Conformance workflow runs one job per language against Go on PostgreSQL 18 and then SQLite: Go against itself, and Rust and JavaScript against Go. Go is the reference and generates the ports' fixtures, so a small first job diffs the change and runs a language's job when that language's files change or when the Go side does (Go code and modules, SQL, the conformance module, the workflow, or the Makefile); a docs- or CI-only change runs none of them, and the existing port fixture job runs under the same Go condition. The scenarios take about 30 seconds, so each job is a few minutes, mostly building the adapter. A nightly workflow adds process kills, database faults, rolling deploys, and connection and batching checks on PostgreSQL 14 through 18 for each language, a fleet of all three engines, JavaScript against Rust as the reference, throughput and latency against Go, and an hour-long soak. Dispatching it as a release candidate soaks for 5.5 hours, the most GitHub's six-hour job limit allows, and fails on a performance regression. A scheduled run skips itself when nothing the suite exercises has changed since the last successful one, and a run started by hand always runs.

It builds on the nested conformance module from #1451, which holds the fixtures generated from River Go that the ports' own tests read and is kept out of River's Go module zips.

It replaces the earlier version of this PR. That one was about 32,000 lines here plus about 14,000 in the ports' adapters, most of it a 68-method contract with separate PostgreSQL and SQLite copies of every adapter, a scenario registry, a feature inventory, JSON schemas with their own validator, and committed generated JSON. Its PR tier took over an hour per language.

	Before	Now
Harness, contract, and Go adapter	~32,000 lines	~7,600 lines
Rust adapter	~4,300 lines	~1,500 lines
JavaScript adapter	~10,200 lines	~1,300 lines
Contract methods	68	14
PR tier per language	~80 minutes	a few minutes

Coverage carries over. Each of the old suite's 167 scenarios maps to a harness scenario on the pull request or nightly tier, to the Go-generated fixtures, or, for behavior one implementation shows alone (a worker's outcomes, queue administration, a maintenance service's batching), to that port's own tests. Where a port's tests didn't already check those, this adds tests that do, against real drivers, in their own commits.

Running the ports against Go found these differences, fixed in one commit per port:

JavaScript: insertMany accepts an empty batch, where Go fails it with "no jobs to insert".
JavaScript: a job can't have more than 32,767 maximum attempts, even on SQLite, which Go allows. On PostgreSQL, whose columns are 16-bit, Go's drivers clamp a larger value to 32,767 on insert, and the PostgreSQL and Prisma drivers now do too instead of failing.
JavaScript: a failed resumable step records a wrapped error instead of the step's own, as Go does.
JavaScript: a leader whose renewal finds its term gone keeps leading until its local deadline, instead of stepping down at once.
JavaScript: raising a full queue's capacity with updateQueue starts nothing until a running job finishes.
Rust: attempt counts are 16-bit, so a job with more than 32,767 attempts can't be inserted or read on SQLite. They're now i32, like Go's int, and a PostgreSQL insert clamps a larger max_attempts to 32,767 as Go's drivers do.
Rust: a worker can't cancel its job with a reason, as Go's JobCancel(err) does. A new JobCancelError does, and the attempt records the text Go writes.

The harness keeps the attempt count fixes in place: on SQLite, a job one implementation inserts with 40,000 maximum attempts is listed and worked by the other, and rows already past 32,767 attempts are worked, failed, retried, and listed with their attempts, maximum attempts, and recorded error attempt numbers unchanged. On PostgreSQL, the same insert succeeds in every implementation, stores and lists 32,767, and is worked by the other.

The commits go in order: the contract and Go's adapter, the pull request tier, and the nightly tier; then for JavaScript its Go-parity fixes, its adapter, and tests of behavior the harness doesn't cover; then for Rust its Go-parity fixes, and its adapter with its port-native tests; and then CI.

The cross-language conformance suite talks to each implementation through an adapter process. Define the contract it speaks as Go types in a dependency-free `protocol` package: fourteen JSON-RPC methods (`handshake`, `migrate`, `insert`, `list`, `cancel`, `retry`, `queue`, `request_resign`, `tx_begin`, `tx_end`, `start`, `stop`, `stats`, and `release`), the job shape adapters report, the built-in worker's behaviors, and the error codes. Operations that may run in a caller's transaction take an optional `tx` name instead of having transactional mirrors, and the harness reads rows and injects faults with SQL itself, so the contract has no raw-row, fault, reset, or deterministic-value methods. Add River Go's adapter, the reference the other implementations are checked against. It's one handler generic over the driver's transaction type, so PostgreSQL (pgx) and SQLite share every method. It inserts `conformance_echo` jobs, runs a worker client whose jobs follow their `behavior` arg, records the events and counters scenarios observe, and can hold a client's first claim on a barrier through a pilot plugin.

Each River implementation tests itself, but nothing checks that a Go process and a Rust or JavaScript process sharing one database agree: that one works the other's jobs, reads its rows, honors its unique keys, wakes on its notifications, and follows its leadership. Add a harness that runs those scenarios between River Go and a candidate implementation's adapter. The harness builds and starts adapters, drives them through a typed client for the contract, and reads and writes the database itself: rows, leaders, queues, migrations, notifications (`LISTEN` on PostgreSQL, the outbox on SQLite), and `pg_stat_activity` by each process's `application_name`. `EachDriver` and `EachDirection` run every scenario on PostgreSQL and SQLite, with each implementation in each role, in a database of its own: a schema reached through the adapters' `search_path`, or a SQLite file. With nothing shared, scenarios run in parallel, and Go against Go finishes in about 25 seconds. The scenarios cover insert-then-work in both directions, golden row comparisons of what each implementation stores when it inserts, works, claims, snoozes, discards, and rescues the same jobs, exact large numbers and IDs, batches, transactions, unique keys and conflicts, list cursors, migrations and custom schemas, notifications and their payloads, remote cancellation, queue control, leadership, claim order and competition, kind handling across a fleet, rescue and scheduling of the other's jobs, resumable cursors, and reserved metadata. Behavior one implementation exhibits alone stays in that implementation's own tests. `RIVER_CONFORMANCE` names the candidate (`go`, `rust`, or `js`); unset, every scenario skips, so `make test` is unaffected. `make test/conformance` runs the suite, against Go itself by default. The harness builds each adapter once per run and runs the built program directly, so killing an adapter kills the adapter itself. Rust's binary is found under `CARGO_TARGET_DIR` when it's set, and JavaScript's adapter is built after the root `riverqueue` package, which pnpm's `@riverqueue/conformance...` filter doesn't select. Each `Implementation` carries an exported `Build` function returning the directory its adapter runs in and the command that starts it, and `UseImplementations` replaces the set the `RIVER_CONFORMANCE` variables select from. Another module can use them, along with `RunBuild`, to run its own scenarios against adapters of its own; River's suite keeps its `go`, `rust`, and `js` implementations unchanged.

Some cross-language checks are too slow or disruptive for every pull request but still matter before a release: implementations must survive faults while sharing a database, stay within reach of River Go's performance, and run together for long periods. Add a nightly tier to the conformance harness. With `RIVER_CONFORMANCE_NIGHTLY` set, it kills processes holding running attempts and leadership so the other implementation rescues them and takes over, replaces every process in turn as a rolling deploy would, terminates listener and pool connections, takes the database away behind a TCP fault proxy, fails and blocks completions with triggers and row locks, holds SQLite's write lock past the busy timeout, hands workers rows they can't decode, and runs both implementations on PostgreSQL disguised as YugabyteDB. It also checks that completions are batched, that connections stay bounded under pool pressure, and that throughput and p95 latency stay within each implementation's bounds relative to Go's. `RIVER_CONFORMANCE_SOAK` runs mixed traffic with periodic leader restarts for the given duration, and `RIVER_CONFORMANCE_PEER` adds a third implementation for fleet scenarios. Pairs of non-Go implementations run the whole suite with `RIVER_CONFORMANCE_REFERENCE`. `make test/conformance/nightly` runs the tier.

Running the conformance harness with JavaScript as the candidate shows a few places where River for JavaScript and River for Go store or do different things for the same job, so a mixed fleet behaves differently depending on which process handles a job. Fix each to match Go: - `insertMany` rejects an empty batch with a `ValidationError` carrying Go's "no jobs to insert", like `InsertMany` and `InsertManyTx`, instead of resolving to `[]`. - `maxAttempts` is no longer capped at 32767, like Go's `int`: SQLite stores wider attempt counts, and the PostgreSQL and Prisma drivers clamp `max_attempts` and `priority` to their `smallint` columns on insert, like Go's pgx and `database/sql` drivers. - A failed resumable step records the step's own error, like Go's `ResumableStep`, instead of wrapping it in a `LifecycleError`. - A leader whose renewal finds no term to renew gives up leadership at once without resigning, like Go's elector, instead of running maintenance until its local deadline passes. - `updateQueue` raising a full queue's `maxWorkers` starts jobs immediately: a producer with every worker busy also waits for the queue's wake-up, not only for a running job to finish. Tests cover the empty batch, a job with 40,000 maximum attempts read back unchanged on SQLite and as 32,767 through the PostgreSQL and Prisma drivers, and the resumable step's recorded error. The changelog lists each fix under Unreleased.

The conformance harness launches an adapter per implementation, but JavaScript has none, so `RIVER_CONFORMANCE=js` fails at the build. Add the private, never-published `@riverqueue/conformance` workspace package. It implements the contract in `conformance/protocol`: strict JSON-RPC over stdin and stdout, the 14 methods with Go's error codes, the built-in worker's 10 behaviors, named barriers, the claim barrier (a `PilotClient` whose first claim holds its jobs), and stats from River's events and hooks. One `Adapter` class runs over River's driver interface, so PostgreSQL (node-postgres, one driver per schema) and SQLite (`node:sqlite`, one handle per open transaction) differ only in how they connect and begin transactions. `make test/conformance/js` builds the root `riverqueue` package and then the adapter with its workspace dependencies, and runs the pull request tier with JavaScript as the candidate. The package joins the workspace's build, lint, format, and type-check scripts.

The cross-language harness checks only what two implementations do together. Behavior River for JavaScript shows alone is tested mostly with fake drivers or not at all, and the Go-generated fixtures don't check what the drivers write: that notification payloads match Go's, that PostgreSQL's spaced `json_build_object` payloads read like Go's compact ones, or that the retry delay without jitter is exactly Go's base delay. Check the topics and payload fields the SQLite driver writes, and the PostgreSQL driver sends, for each of Go's notification goldens; that every payload reader parses the spaced form of each golden; and that the default retry policy schedules exactly Go's minimum delay for every retry case when the jitter is zero. Test single-implementation behavior against the real SQLite driver: bulk and transactional job updates and deletes, worker outcomes and runtime faults, aborted and transactional completions, queue reads, dynamic queue reconfiguration and pause, error handler cancellation, extension ordering, resumable retries and validation, timeouts, shutdown classification, graceful stop, job and queue cleaner retention, the rescuer paging past a full batch, leadership term replacement, and periodic and scheduled jobs. On PostgreSQL, test leader renewal while the job cleaner waits on a row lock and the reindexer skipping missing and artifact indexes. The PostgreSQL notification check is an integration test, so `make test/js/integration` now generates the fixtures first, and the integration CI job sets up Go and generates them as the unit test job does. `make test/js/conformance` also runs the payload reader test.

Running the conformance harness with Rust as the candidate shows two places where River Rust can't do what River Go does with the same job. River Go stores `attempt` and `max_attempts` as Go `int`s. On SQLite, whose columns are native integers, a Go client inserts a job with 40,000 maximum attempts and reads it back unchanged; on PostgreSQL, whose columns are `smallint`, Go's pgx and `database/sql` drivers clamp `max_attempts` to 32,767 on insert instead of failing. River Rust models both as `i16`, so it can't insert such a job at all and reads one Go wrote on SQLite as 32,767. A River Go worker cancels its job with `JobCancel(err)`, and the attempt records `JobCancelError: <err>`. A Rust worker can only return `WorkOutcome::Cancel`, which records the fixed message "job cancelled by worker", so a Rust-worked job loses the reason a Go-worked one keeps. Make attempt counts `i32` across the crates: `JobRow::attempt` and `max_attempts`, `AttemptError::attempt` and `AttemptError::new`, `InsertOpts::max_attempts`/`with_max_attempts`, `InsertParams::max_attempts`, `ClientBuilder::default_max_attempts`, `MAX_ATTEMPTS_DEFAULT`, the derive macro's `max_attempts` bound, and `riverqueue-test`'s builders. SQLite rows decode the native integers directly instead of saturating them into `i16`; a value beyond `i32` makes the row undecodable like other columns River can't represent. PostgreSQL rows widen the `smallint` columns, and a PostgreSQL insert clamps `max_attempts` to 32,767 like Go's drivers. Add `JobCancelError`, an error a worker returns to cancel its job with a reason. Anywhere in the worker error's source chain, it cancels the job whatever attempts it has left, skips the error handler as Go does, and records `JobCancelError: <reason>`, the same text Go writes. `WorkOutcome::Cancel` keeps cancelling without a reason. New tests insert and list a job with 40,000 maximum attempts on SQLite, check PostgreSQL stores and lists it as 32,767, and check a worker's `JobCancelError` cancels the job and records its reason.

The conformance harness launches an adapter per implementation, but Rust has none, so `RIVER_CONFORMANCE=rust` fails at the build. And behavior one implementation shows alone belongs in that implementation's own tests rather than the harness, where the Rust suite misses several assertions. Add `riverqueue-conformance`, an unpublished workspace binary that serves the 14-method contract in `conformance/protocol` for River's Rust client. One `Server<B: Backend>` handles every method. `Backend` holds only what differs between PostgreSQL and SQLite: opening the pool, the schema a client uses, beginning a transaction and handing it to River's `.tx` requests, and migrating. Everything else goes through River's database-independent `Client`. The built-in worker implements the ten behaviors, cancelling with a `JobCancelError` reason as Go's adapter does with `JobCancel`. A pilot implements the claim barrier, and a hook, an error handler, and an event subscription record the stats. Jobs are reported with their stored args and metadata as raw JSON, so numbers no float can hold, such as `1e400`, keep their exact text as they do in Go's adapter. `make test/conformance/rust` and `make test/conformance/rust/nightly` build the adapter first so a compile error is reported once. `cargo package --workspace` includes `publish = false` crates, so `check/rust/package` excludes the adapter by name; `cargo publish --workspace` already skips it. Add or extend port-native tests for single-client behavior: - Worker outcomes, including a `JobCancelError` reason, panics, job timeouts, resumable step validation, and remote cancellation of a snoozed and refetched attempt, in a new `worker_outcomes` test. - An error handler cancelling a job with attempts left, and insert middleware wrapping the insert hook before any work hook runs. - Dynamic queues: a removed queue's job stays available, and a queue added at runtime pauses, resumes, and is removed. - Queue get and list reflecting updates, and pause, resume, and update of unknown queue names (with spaces, 129 characters) reporting not found, plus `|` in queue names. - Transactions: commit of a PostgreSQL transaction aborted by a failed statement rolls back, transactional completion records no errors, and update and delete-many in a transaction stay invisible until commit. - Leadership: a same-ID term replacement is followed by a fresh term, with run-on-start periodic jobs inserted once per term and completed. - Maintenance: the queue cleaner keeps recently updated and actively worked queues, the reindexer skips missing indexes, and the rescuer pages past a full default batch of timeout-disabled jobs to discard an unregistered kind. - Go-generated fixtures: the client's default retry policy stays within Go's bounds, and notification topics and payload fields match Go's on PostgreSQL and in SQLite's outbox. `make test/rust/conformance` also runs the SQLite notification check, so a Go change to notification payloads is checked against Rust.

The harness proves River Go, Rust, and JavaScript agree when they share a database, but nothing runs it, so a change to any of them can break the agreement unnoticed. Each pull request runs one job per candidate in the `Conformance` workflow, every scenario on PostgreSQL 18 and then SQLite: Go against itself, and Rust and JavaScript against Go. Go is the reference and generates the ports' fixtures, so a first job diffs the change and runs a candidate when its own files change or when Go code, modules, SQL, the conformance module, the workflow, or the `Makefile` do. A change that touches none of them, like docs or other CI, runs none, and the existing port fixture job runs under the same Go condition. The pull request tier adds at most three runners, each a few minutes, and no matrix. A new `Conformance nightly` workflow, on a schedule and by hand, runs what's too slow or noisy for pull requests: - The nightly tier (process kills, database faults, rolling deploys, completion batching, pool pressure) for each candidate on PostgreSQL 14 through 18, with SQLite only on 18. - A three-engine fleet of Go, Rust, and JavaScript, and the ordinary suite with JavaScript against Rust as the reference. - Throughput and p95 latency of Rust and JavaScript relative to Go, reported without failing the run on shared runners. - A one-hour soak through all three engines. Its `release_candidate` input soaks for 5.5 hours instead, the most a GitHub-hosted job's six-hour limit leaves room for, and makes a throughput or latency regression fail the run. A scheduled run first compares its commit with the last successful scheduled run's and skips the rest of the workflow when nothing the suite exercises changed: Go code and modules, SQL, the conformance module, the Rust and JavaScript ports, the setup actions, the `Makefile`, and the workflow itself. If there's no earlier success, its commit isn't in history, or the lookup fails, the run goes ahead. A run started by hand always runs, and the job summary says why each run ran or skipped.

bgentry mentioned this pull request Oct 5, 2026

Add River for Rust #1436

Closed

bgentry force-pushed the bg/conformance-harness branch 4 times, most recently from 3a61e95 to 6e5ad81 Compare October 6, 2026 04:26

bgentry changed the base branch from master to bg/conformance-fixtures October 6, 2026 04:26

bgentry changed the title ~~Add a cross-language conformance suite~~ Add a conformance harness for River's Go, Rust, and JavaScript clients Oct 6, 2026

bgentry force-pushed the bg/conformance-harness branch from 6e5ad81 to 903f1ab Compare October 6, 2026 12:38

bgentry force-pushed the bg/conformance-fixtures branch from db624b4 to 864b4af Compare October 6, 2026 12:38

bgentry force-pushed the bg/conformance-harness branch 2 times, most recently from 274a4b1 to d948cfc Compare October 6, 2026 13:34

bgentry mentioned this pull request Oct 6, 2026

Flaky Rust test: notification_poll_failures_do_not_stop_the_client hits database is locked #1455

Closed

bgentry force-pushed the bg/conformance-harness branch 2 times, most recently from d5649aa to 755282d Compare October 6, 2026 15:17

brandur force-pushed the bg/conformance-fixtures branch from 864b4af to 35949ea Compare October 6, 2026 16:56

Base automatically changed from bg/conformance-fixtures to master October 6, 2026 17:05

bgentry force-pushed the bg/conformance-harness branch 6 times, most recently from 411b482 to 270627b Compare October 7, 2026 13:52

bgentry marked this pull request as ready for review October 7, 2026 17:03

bgentry force-pushed the bg/conformance-harness branch from 270627b to 00a938b Compare October 7, 2026 17:23

bgentry added 6 commits October 7, 2026 12:26

bgentry added 3 commits October 7, 2026 12:26

bgentry force-pushed the bg/conformance-harness branch from 00a938b to fb45ae1 Compare October 7, 2026 17:36

Navigation Menu

Repository navigation

Add a conformance harness for River's Go, Rust, and JavaScript clients - #1435

Add a conformance harness for River's Go, Rust, and JavaScript clients#1435
bgentry wants to merge 9 commits into
masterfrom
bg/conformance-harness

bgentry commented Oct 5, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

bgentry commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before this merges

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

bgentry commented Oct 5, 2026 •

edited

Loading