Skip to content

Resume or take over a team reminder that didn't finish, and stop retrying a broken webhook - #151

Open
wayx112 wants to merge 3 commits into
Team6083:mainfrom
wayx112:fix/reminder-retries
Open

wayx112 wants to merge 3 commits into
Team6083:mainfrom
wayx112:fix/reminder-retries

Conversation

@wayx112

@wayx112 wayx112 commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Three ways a team reminder run went wrong once its first try didn't finish cleanly (#146, #148):

  1. A failed send released the run, so the next call started again from the first message. A long reminder split over several messages posted the parts that had already gone out a second time.
  2. A wrong webhook was retried on every scheduler call until midnight, logging an error each time.
  3. If the process died between the claim and the end of the run (timeout, restart, out of memory), the row stayed claimed and the rule was skipped for the rest of the day, with nothing recording why.

They all come down to how a run is claimed and taken over, so they are fixed together here (this includes #149, now closed).

ReminderDelivery   + remindedUserIds, failures, nextAttemptAt; result can be "failed" or "gaveUp"

claimRun
  insert row as claimed                         → new run
  row "failed" and its wait is over             → take over
  row "claimed" for more than 10 min            → take over, failures + 1 (the process died)
  otherwise (fresh claim, sent, gaveUp, …)      → skip
  each takeover is one conditional write, so only one call gets it; failures ≥ 5 → "gaveUp"

send a part  → ok: add its members to remindedUserIds
failure      → "failed", failures + 1, wait 5 / 10 / 20 / 40 min; the 5th failure → "gaveUp" for the day
takeover     → work the message out again without the members already reminded, and send it
new webhook  → today's failed or gaveUp run: failures reset, retried by the next call
  • Resume: the row records the ids of the members in every part that went out, and any takeover, after a failure or a crash, works the message out again without them. Members who filled in meanwhile drop out, and nobody shifts into a part that is skipped, since nothing is skipped by position. This is at least once: if the process dies, or recording fails, right after a part posts, its members are reminded again (README says so).
  • Back-off and give up: MAX_FAILURES = 5 a day. An abandoned run counts as a failure, so one that keeps dying is given up like one that keeps failing. The run result is then gaveUp (README and OpenAPI updated).
  • Failures before sending (reading members fails) count as a failed try too.
  • Finished runs: a sent or nobodyMissing row still blocks the day, however old.
  • New webhook: setting a webhook resets today's failed or gaveUp run (failures to 0, next try now) and keeps who was reminded, so fixing a broken webhook helps the same day.

Closes #146. Closes #148.

Evidence

Against a local MongoDB replica set, with a 1,200-member team (two messages) and fetch stubbed:

part 1 ok (990 members), part 2 fails      → failed, 990 recorded, next try in 5 min
2 min later                                 → nothing sent
a member from part 1 fills in, 5 min later  → the other 210 sent, none of part 1 again; all 1,200 reminded once
gaveUp, then an admin sets a new webhook     → failed, failures 0, the 1,200 kept
webhook always 404                          → failed ×4 at 0/5/15/35 min, gaveUp at 75 min, nothing at 200 min; 5 posts
run died after part 1                       → nothing at 5 min; at 11 min taken over, only the rest sent, failures 1
run abandoned a fifth time                  → gaveUp, nothing posted
Test Files  25 passed (25)
     Tests  339 passed (339)

New or changed tests:

  • Failed sends: wait then retry, longer waits and giving up after five, reading members failing counts as a try, a retry after a later part fails that reminds nobody from the earlier part again, and a member of the earlier part filling in before the retry still leaves everyone else reminded.
  • Abandoned runs: a fresh claim is skipped, an abandoned run is taken over and leaves out who it reminded, one abandoned a fifth time is given up, and a sent run is never taken over.
  • A new webhook resets today's given-up run and keeps who was reminded, and leaves a sent run alone.

tsc --noEmit, ESLint on the changed files and npm run build pass.

Merge danger

Door: two-way. New fields have defaults, and rows from before only matter while a run is in progress.

Blast radius: team reminders only.

  • Run npx prisma db push when deploying.

🤖 Generated with Claude Code

wayx112 and others added 2 commits November 15, 2026 19:50
A failed send released the run, so the next call started again from the first message: a long reminder split over
several messages posted the parts that had gone out a second time. And a wrong webhook was retried on every
scheduler call until midnight, logging an error each time.

The delivery row now records how many messages went out and how often the run failed. A failure keeps the row as
failed with a wait of 5, 10, 20, then 40 minutes before a later call may take it over (in one conditional write),
and that call sends only the parts after the ones already out. The fifth failure gives the rule up for the day, and
the run says gaveUp.

Closes Team6083#148

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
If the process died between claiming a run and finishing it (timeout, restart, out of memory), the row stayed
claimed and the rule was skipped for the rest of the day. A run still claimed after 10 minutes, longer than a run
takes, is now taken over by the next call in one conditional write. It counts as a failure and continues after the
parts already sent, so a run that keeps dying is given up like one that keeps failing, and a crash doesn't repeat
messages. A sent or nobodyMissing run still blocks the day.

This was Team6083#149, folded in here since both change claimRun.

Closes Team6083#146

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wayx112 wayx112 changed the title Resume a failed team reminder and stop retrying a broken webhook Resume or take over a team reminder that didn't finish, and stop retrying a broken webhook Nov 15, 2026

@kennhung kennhung left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The claim, back-off and give-up logic looks right: takeovers are single conditional writes, and the 5/10/20/40 minute waits are well tested. One blocking issue on resume (inline), plus a couple of non-blocking notes.

This PR, #150 and #152 all touch team-reminder-dto.ts and its test, so expect a conflict resolution on whichever merges later. Suggested order: #150, #152, #151.

Comment thread src/lib/data/team-reminder-dto.ts Outdated
const message = await evaluateRule(condition, { team, tree, date, appUrl });
if (message) await deliver(channel, message);
result = message ? "sent" : "nobodyMissing";
if (message) await deliver(channel, message, claim.partsSent, parts => recordParts(rule.id, date, parts));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking. On resume the message is recomputed and the first partsSent parts are skipped by index. But slackPayloads packs bullets into parts by size, so when someone in part 1 fills in meanwhile, everyone after them shifts forward. A member who was in old part 2 and never got reminded can land in new part 1 and be skipped. They are not reminded today.

The PR description presents "members who filled in drop out of the remaining parts" as a benefit, but the index-based skip breaks it. Suggestion: record the user ids already reminded on the delivery row and exclude them after recomputing, or store the payloads on first try and resume from those.

Comment thread src/lib/reminders/delivery.ts Outdated
signal: AbortSignal.timeout(10_000),
});
if (!response.ok) throw new Error(`HTTP ${response.status}`);
await sent(i + 1);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this is at-least-once, not exactly-once. If the process dies, or recordParts fails, after the POST succeeds, the part is sent again on takeover (and if recordParts fails, failRun will likely fail too, leaving the row claimed until the 10 minute takeover). The window is small and acceptable, but the PR description says part 1 is not repeated, so a line in the README would help.

Also, a run that is slow rather than dead can be taken over after STALE_CLAIM_MS while still posting. Unlikely, so fine.

Comment thread src/lib/data/team-reminder-dto.ts Outdated
const message = await evaluateRule(condition, { team, tree, date, appUrl });
if (message) await deliver(channel, message);
result = message ? "sent" : "nobodyMissing";
if (message) await deliver(channel, message, claim.partsSent, parts => recordParts(rule.id, date, parts));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion, not blocking: after gaveUp, fixing the webhook does not help until tomorrow. Consider resetting today's delivery row when an admin changes the webhook.

A retry worked the message out again and skipped the first parts by index, but parts are packed by size. If
someone in part 1 filled in meanwhile, everyone after them shifted forward, and a member from the old part 2 could
land in the new part 1 and be skipped, never reminded that day. The delivery row now records the ids of the members
in each part that went out, and a retry leaves them out before building the message, so everyone not yet reminded
is reminded and nobody twice.

Setting a new webhook also gives today's failed or given-up run a fresh try (failures reset, retried by the next
call), keeping who was reminded. The README says resuming is at least once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wayx112

wayx112 commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, all addressed in b3c76f0. Description updated.

Blocking: resume by who, not by position. You're right, skipping by index broke as soon as someone filled in. Now:

  • The delivery row has remindedUserIds instead of partsSent. After each part posts, the ids of the members in it are pushed onto it.
  • A takeover, after a failure or a crash, works the message out again with those members left out (skipUserIds in the rule context), then sends the whole result. Nothing is skipped by position, so nobody can shift into a skipped part.
  • New test for your case: part 1 goes out, part 2 fails, a member of part 1 fills in, then the retry. Everyone else is still reminded, and nobody twice.
  • Checked against a local MongoDB with 1,200 members: part 1 had 990, part 2 failed, one of the 990 filled in, and the retry sent the other 210. All 1,200 were reminded exactly once.

At least once: agreed. The README now says that if the process dies, or recording fails, right after a part posts, those members are reminded again on the retry. The description no longer says "not repeated" without that caveat.

Webhook after gaveUp: done. Setting a new webhook updates today's failed or gaveUp row back to failed with failures 0 and the next try now, keeping remindedUserIds, so the next call retries only the members not yet reminded. A sent run is left alone.

Tests (339), tsc --noEmit, ESLint and npm run build pass. prisma db push is still needed for the new fields.

@wayx112
wayx112 requested a review from kennhung November 15, 2026 23:51

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants