Repository navigation
Conversation
A failed send released the run, so the next call started again from the first message: a long reminder split over several messages posted the parts that had gone out a second time. And a wrong webhook was retried on every scheduler call until midnight, logging an error each time. The delivery row now records how many messages went out and how often the run failed. A failure keeps the row as failed with a wait of 5, 10, 20, then 40 minutes before a later call may take it over (in one conditional write), and that call sends only the parts after the ones already out. The fifth failure gives the rule up for the day, and the run says gaveUp. Closes Team6083#148 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
If the process died between claiming a run and finishing it (timeout, restart, out of memory), the row stayed claimed and the rule was skipped for the rest of the day. A run still claimed after 10 minutes, longer than a run takes, is now taken over by the next call in one conditional write. It counts as a failure and continues after the parts already sent, so a run that keeps dying is given up like one that keeps failing, and a crash doesn't repeat messages. A sent or nobodyMissing run still blocks the day. This was Team6083#149, folded in here since both change claimRun. Closes Team6083#146 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kennhung
left a comment
There was a problem hiding this comment.
The claim, back-off and give-up logic looks right: takeovers are single conditional writes, and the 5/10/20/40 minute waits are well tested. One blocking issue on resume (inline), plus a couple of non-blocking notes.
This PR, #150 and #152 all touch team-reminder-dto.ts and its test, so expect a conflict resolution on whichever merges later. Suggested order: #150, #152, #151.
| const message = await evaluateRule(condition, { team, tree, date, appUrl }); | ||
| if (message) await deliver(channel, message); | ||
| result = message ? "sent" : "nobodyMissing"; | ||
| if (message) await deliver(channel, message, claim.partsSent, parts => recordParts(rule.id, date, parts)); |
There was a problem hiding this comment.
Blocking. On resume the message is recomputed and the first partsSent parts are skipped by index. But slackPayloads packs bullets into parts by size, so when someone in part 1 fills in meanwhile, everyone after them shifts forward. A member who was in old part 2 and never got reminded can land in new part 1 and be skipped. They are not reminded today.
The PR description presents "members who filled in drop out of the remaining parts" as a benefit, but the index-based skip breaks it. Suggestion: record the user ids already reminded on the delivery row and exclude them after recomputing, or store the payloads on first try and resume from those.
| signal: AbortSignal.timeout(10_000), | ||
| }); | ||
| if (!response.ok) throw new Error(`HTTP ${response.status}`); | ||
| await sent(i + 1); |
There was a problem hiding this comment.
Non-blocking: this is at-least-once, not exactly-once. If the process dies, or recordParts fails, after the POST succeeds, the part is sent again on takeover (and if recordParts fails, failRun will likely fail too, leaving the row claimed until the 10 minute takeover). The window is small and acceptable, but the PR description says part 1 is not repeated, so a line in the README would help.
Also, a run that is slow rather than dead can be taken over after STALE_CLAIM_MS while still posting. Unlikely, so fine.
| const message = await evaluateRule(condition, { team, tree, date, appUrl }); | ||
| if (message) await deliver(channel, message); | ||
| result = message ? "sent" : "nobodyMissing"; | ||
| if (message) await deliver(channel, message, claim.partsSent, parts => recordParts(rule.id, date, parts)); |
There was a problem hiding this comment.
Suggestion, not blocking: after gaveUp, fixing the webhook does not help until tomorrow. Consider resetting today's delivery row when an admin changes the webhook.
A retry worked the message out again and skipped the first parts by index, but parts are packed by size. If someone in part 1 filled in meanwhile, everyone after them shifted forward, and a member from the old part 2 could land in the new part 1 and be skipped, never reminded that day. The delivery row now records the ids of the members in each part that went out, and a retry leaves them out before building the message, so everyone not yet reminded is reminded and nobody twice. Setting a new webhook also gives today's failed or given-up run a fresh try (failures reset, retried by the next call), keeping who was reminded. The README says resuming is at least once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks, all addressed in b3c76f0. Description updated. Blocking: resume by who, not by position. You're right, skipping by index broke as soon as someone filled in. Now:
At least once: agreed. The README now says that if the process dies, or recording fails, right after a part posts, those members are reminded again on the retry. The description no longer says "not repeated" without that caveat. Webhook after gaveUp: done. Setting a new webhook updates today's Tests (339), |
Summary
Three ways a team reminder run went wrong once its first try didn't finish cleanly (#146, #148):
claimedand the rule was skipped for the rest of the day, with nothing recording why.They all come down to how a run is claimed and taken over, so they are fixed together here (this includes #149, now closed).
MAX_FAILURES= 5 a day. An abandoned run counts as a failure, so one that keeps dying is given up like one that keeps failing. The run result is thengaveUp(README and OpenAPI updated).sentornobodyMissingrow still blocks the day, however old.failedorgaveUprun (failures to 0, next try now) and keeps who was reminded, so fixing a broken webhook helps the same day.Closes #146. Closes #148.
Evidence
Against a local MongoDB replica set, with a 1,200-member team (two messages) and
fetchstubbed:New or changed tests:
tsc --noEmit, ESLint on the changed files andnpm run buildpass.Merge danger
Door: two-way. New fields have defaults, and rows from before only matter while a run is in progress.
Blast radius: team reminders only.
npx prisma db pushwhen deploying.🤖 Generated with Claude Code