Skip to content

Hold the audit export behind the oldest open transaction - #595

Merged
haksungjang merged 2 commits into
mainfrom
fix/587-audit-export-commit-time
Oct 25, 2026
Merged

haksungjang merged 2 commits into
mainfrom
fix/587-audit-export-commit-time

Conversation

@haksungjang

@haksungjang haksungjang commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Closes #587

An audit row's created_at is now(), the start of its transaction rather than its commit. A transaction open longer than AUDIT_EXPORT_LAG_SECONDS therefore committed rows dated behind a cursor that had already passed them. clock_timestamp() would still stamp them before the commit, so it does not close the gap. Committing audit rows in their own short transaction would break the rule that an audit row and its change commit or roll back together. A fixed lag needs an upper bound on transaction length, and there is none: the scan retention sweep, for one, deletes many scans with cascades in a single transaction. So each batch now reads pg_stat_activity (after pg_stat_clear_snapshot()) and takes only rows dated strictly before the oldest open client transaction. The lag floor is measured on the database clock, read in the statement before that one. Rows from transactions that commit later are then either inside that bound or visible to the next READ COMMITTED snapshot. There is no schema change.

A role without pg_read_all_stats is not shown xact_start for other roles' sessions. In a role-separated deployment those include the owner role (the anonymisation script) and superusers. They still appear with their pid, and their virtualxid lock is listed in pg_locks. While any such transaction is open, the export reads nothing. This is safe by induction: the transaction was not open at the previous read, so all of its rows are dated after anything already sent. Vacuums are excluded through pg_stat_progress_vacuum. The lag now has a floor of 1 s, because a transaction is dated slightly before it appears in the activity list. With track_activities off the export reads only when nothing is open, and logs a WARNING. Prepared transactions are outside the guarantee; the portal does not use them. The guides now state the guarantee and these limits instead of saying "no row is lost".

When a run is held back with rows waiting, it ends with status held_back and the real pending count, and logs audit_export_held_back_by_open_transaction at WARNING. Every run logs audit_export_run_finished. Server-side tcp_keepalives_* are set after connect on both the worker and API engines (not as startup parameters, which some poolers reject). A refusal is logged instead of failing the connection. A unit test ties the "four minutes" in both guides (EN and KO) to _RUN_BUDGET_SECONDS.

New integration tests run the issue's sequence through the task, and run the same sequence with a hidden superuser transaction read by a freshly created plain role. They also cover a committed row at the exact start instant, a vacuum hidden from the role, a stale activity copy, the export's own session, a worker clock running ahead, a long transaction with nothing waiting (no warning, idle), track_activities off, and keepalives on both engines, including a rejected setting. Each guarded line was mutated back, and in every case the matching test failed.

audit_logs.created_at is now(), the transaction start, so a transaction open
longer than AUDIT_EXPORT_LAG_SECONDS committed rows behind the export cursor
and they were never sent. Each batch now reads only rows dated before the
oldest open transaction in pg_stat_activity, measured on the database clock.
Worker connections also set server-side TCP keepalives, and a guard ties the
documented run budget to _RUN_BUDGET_SECONDS.
A role without pg_read_all_stats is not shown xact_start for other roles, so
an owner or superuser transaction writing audit rows could still be passed.
Such a transaction is detected by its virtualxid lock and the export reads
nothing until it ends. Idle runs now report real pending and a held_back
status, keepalives also cover the API engine, and the lag floor is one second.
@haksungjang
haksungjang force-pushed the fix/587-audit-export-commit-time branch from 4bd5655 to de8e0b3 Compare October 25, 2026 00:40
@haksungjang
haksungjang merged commit a8c5eb0 into main Oct 25, 2026
34 checks passed
@haksungjang
haksungjang deleted the fix/587-audit-export-commit-time branch October 25, 2026 01:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Audit export: rows dated by transaction start can fall behind the cursor

1 participant