Skip to content

fix(api): recover scans stranded in executing by a dead worker #12008

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Subhro-ai wants to merge 2 commits into
base: master
Choose a base branch
from
Open

fix(api): recover scans stranded in executing by a dead worker #12008

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
46775bf
fix(api): recover scans stranded in executing by a dead worker
Subhro-ai Jul 22, 2026
0586609
fix(api): harden stale-scan cleanup per review
Subhro-ai Jul 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions api/changelog.d/scan-stuck-queued.fixed.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Recover scans stranded in `executing` by a dead worker and dispatch the next queued scan so a provider's scans no longer stay stuck in `Queued`
48 changes: 48 additions & 0 deletions api/src/backend/api/migrations/0098_scan_cleanup_periodic_task.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
from django.db import migrations

TASK_NAME = "scan-cleanup-stale-scans"
INTERVAL_MINUTES = 15


def create_periodic_task(apps, schema_editor):
IntervalSchedule = apps.get_model("django_celery_beat", "IntervalSchedule")
PeriodicTask = apps.get_model("django_celery_beat", "PeriodicTask")

schedule, _ = IntervalSchedule.objects.get_or_create(
every=INTERVAL_MINUTES,
period="minutes",
)

PeriodicTask.objects.update_or_create(
name=TASK_NAME,
defaults={
"task": TASK_NAME,
"interval": schedule,
"enabled": True,
},
)


def delete_periodic_task(apps, schema_editor):
IntervalSchedule = apps.get_model("django_celery_beat", "IntervalSchedule")
PeriodicTask = apps.get_model("django_celery_beat", "PeriodicTask")

PeriodicTask.objects.filter(name=TASK_NAME).delete()

# Clean up the schedule if no other task references it
IntervalSchedule.objects.filter(
every=INTERVAL_MINUTES,
period="minutes",
periodictask__isnull=True,
).delete()


class Migration(migrations.Migration):
dependencies = [
("api", "0097_attack_paths_scan_db_defaults"),
("django_celery_beat", "0019_alter_periodictasks_options"),
]

operations = [
migrations.RunPython(create_periodic_task, delete_periodic_task),
]
11 changes: 11 additions & 0 deletions api/src/backend/config/django/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,17 @@
"ATTACK_PATHS_SCAN_STALE_THRESHOLD_MINUTES", 2880
) # 48h

# Stale scan cleanup. A scan whose worker dies mid-run is stranded in
# `executing`; because scans run one-at-a-time per provider (#11848), that also
# blocks every queued scan for the provider. The periodic `scan-cleanup-stale-scans`
# task fails such scans and drains the queue. A scan on a live worker is only
# reaped after the stale ceiling (matched to the long hard time-limit); a scan
# whose worker is gone is reaped after the shorter inactivity window.
SCAN_INACTIVITY_THRESHOLD_MINUTES = env.int("SCAN_INACTIVITY_THRESHOLD_MINUTES", 30)
SCAN_STALE_THRESHOLD_MINUTES = env.int(
"SCAN_STALE_THRESHOLD_MINUTES", 2880
) # 48h
Comment on lines +324 to +327

@coderabbitai coderabbitai Bot Jul 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Validate cleanup thresholds during startup.

Zero or negative values make current scans immediately stale, while a stale threshold below the inactivity threshold reverses the documented recovery policy. Reject invalid configuration before the watchdog can fail active scans.

Proposed validation 
 SCAN_STALE_THRESHOLD_MINUTES = env.int(
     "SCAN_STALE_THRESHOLD_MINUTES", 2880
 )  # 48h
+
+if SCAN_INACTIVITY_THRESHOLD_MINUTES <= 0:
+    raise ValueError("SCAN_INACTIVITY_THRESHOLD_MINUTES must be positive")
+if SCAN_STALE_THRESHOLD_MINUTES < SCAN_INACTIVITY_THRESHOLD_MINUTES:
+    raise ValueError(
+        "SCAN_STALE_THRESHOLD_MINUTES must be greater than or equal to "
+        "SCAN_INACTIVITY_THRESHOLD_MINUTES"
+    )
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
SCAN_INACTIVITY_THRESHOLD_MINUTES = env.int("SCAN_INACTIVITY_THRESHOLD_MINUTES", 30)
SCAN_STALE_THRESHOLD_MINUTES = env.int(
"SCAN_STALE_THRESHOLD_MINUTES", 2880
) # 48h
SCAN_INACTIVITY_THRESHOLD_MINUTES = env.int("SCAN_INACTIVITY_THRESHOLD_MINUTES", 30)
SCAN_STALE_THRESHOLD_MINUTES = env.int(
"SCAN_STALE_THRESHOLD_MINUTES", 2880
) # 48h
if SCAN_INACTIVITY_THRESHOLD_MINUTES <= 0:
raise ValueError("SCAN_INACTIVITY_THRESHOLD_MINUTES must be positive")
if SCAN_STALE_THRESHOLD_MINUTES < SCAN_INACTIVITY_THRESHOLD_MINUTES:
raise ValueError(
"SCAN_STALE_THRESHOLD_MINUTES must be greater than or equal to "
"SCAN_INACTIVITY_THRESHOLD_MINUTES"
)
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@api/src/backend/config/django/base.py` around lines 324 - 327, Validate
SCAN_INACTIVITY_THRESHOLD_MINUTES and SCAN_STALE_THRESHOLD_MINUTES during Django
configuration startup, rejecting zero or negative values and any stale threshold
lower than the inactivity threshold. Raise the configuration error through the
existing startup validation mechanism before watchdog processing begins.


# Selects where the persistent attack-paths graph is stored. The scan
# temporary database is always Neo4j; only the sink is configurable.
# Valid values: "neo4j" (default, OSS and local dev), "neptune" (hosted).
Expand Down
1 change: 1 addition & 0 deletions api/src/backend/tasks/jobs/orphan_recovery.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ def reenqueueable_tasks() -> set[str]:
"scan-perform-scheduled",
"attack-paths-scan-perform",
"attack-paths-cleanup-stale-scans",
"scan-cleanup-stale-scans",
"reconcile-orphan-tasks",
}

Expand Down
Loading
Loading