Skip to content

Tell failed work apart by cause, dismiss it, and retry plan refusals on upgrade - #298

Merged
dillonstreator merged 2 commits into
mainfrom
claude/admin-queue-failed-work-439a3e
Nov 18, 2026
Merged

dillonstreator merged 2 commits into
mainfrom
claude/admin-queue-failed-work-439a3e

Conversation

@dillonstreator

Copy link
Copy Markdown
Member

Why

The admin Failed work list mixed failures a Retry can fix (disk full, model down) with ones it cannot (a dead link, a video with no audio, nothing worth clipping, a plan refusal), offered Retry on all of them, and had no way to clear the ones nobody will retry. The pipeline already knew which errors were non-retryable (NonRetryableError) but only used that to skip pg-boss retries; the row kept just the message. And a video refused for its length stayed failed after the owner upgraded.

What

  • Cause on every failure. projects / clips get failure_cause (platform | input | quota, FailureCause in @reelsaw/shared). failedRow(err) in pipeline/common.ts is now the only way a row becomes failed; NonRetryableError carries failureCause (input by default, quota for QuotaExceededError), anything else is platform. Migration 0059 backfills existing failures from their recorded wording (the minutes refusal, no audio, no usable clips, dead/private links, live streams); the rest read as platform.
  • Dismiss. failure_dismissed_at hides a failure from the admin list only; the owner's project, error and Retry are untouched, and a new failure clears it. Restore brings it back. dismissFailures in ops/failures.ts, event failures.dismissed.
  • Bulk retry by filter takes platform failures unless a cause is named; retrying exact ids still works for any cause.
  • Plan refusals retry on upgrade. retryRefusedWork on user.limits_changed (open) and billing.subscribed / billing.changed / billing.complimentary (ee) resumes the owner's quota failures from the last 30 days through retryProject; ingest re-checks the minutes, so a change that still doesn't cover the video fails it once more. The handler's key is null when the account has none, so most billing events enqueue nothing.
  • Surfaces. Settings → System → Queue shows the cause per row, a cause filter, Retry only where it helps and Dismiss otherwise, Dismiss all / matching, and a dismissed view with Restore. MCP: dismiss_failures, cause on list_failures / retry_failures, cause / retryable / dismissedAt on items. CLI: dismiss-failures, retry-failures --cause. README, TROUBLESHOOTING and a changeset updated.

Testing

  • ops-failures.test.ts: cause classification, cause filter, bulk retry skipping input/quota, dismiss/restore/re-fail, and the quota auto-retry (recent only, not other causes); handler map and MCP definition tests updated. Run against a scratch Postgres with this branch's migrations; server and web typecheck, lint clean.
  • Deployed the branch locally: a real dead-link project failed at ingest with cause input; dismiss, show dismissed and restore behaved; raising the owner's monthly minutes resumed only the plan-limit failure.

…on upgrade

Every project or clip that fails now records why: platform (the instance's
fault, worth a Retry after the fix), input (the video, link or settings,
which fail the same way again) or quota (the owner's plan). One helper,
failedRow, is the only way a row becomes failed, and a NonRetryableError
names its own cause. A migration fills in the cause of existing failures.

Admins can dismiss failures from Settings > System > Queue (and restore
them), without touching the owner's project. Bulk retry by filter takes
platform failures unless a cause is named. Plan refusals from the last 30
days are retried when the owner subscribes, changes plan, gets a
complimentary plan or has their limits raised.

Same actions over MCP (dismiss_failures, cause filters) and the CLI
(dismiss-failures, retry-failures --cause).

Signed-off-by: dillonstreator <dillonstreator@gmail.com>
@vercel

vercel Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
reelsaw-website Ready Ready Preview Nov 18, 2026 6:51pm UTC

Request Review

…pgrade retry

- An expired source and "no usable clips" are platform failures: the
  project's Retry, or a model fix, can clear them. NonRetryableError takes
  its cause as a constructor argument. Retrying a failed clip of a project
  whose source expired goes through the project, which fetches it again.
- The plan-change retry follows only changes to the minutes limits, and
  does nothing when even a one-minute video would be refused now.
- retryProject takes a per-project retry lock and, for admin and automatic
  retries, only resumes a project that is still failed, so two retries
  never both start a run.
- Issue-report import and the failed dev scenario record a cause; a test
  fails when a status "failed" write leaves the cause out.
- No failures.dismissed event when a dismiss changes nothing; the MCP
  schemas use FAILURE_CAUSES.

Signed-off-by: dillonstreator <dillonstreator@gmail.com>
@dillonstreator
dillonstreator merged commit 3e3c69c into main Nov 18, 2026
15 checks passed
@dillonstreator dillonstreator mentioned this pull request Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant