fix(queueserver): recover from partial bootstrap failures on env-open

[Anthony Sligar (sligara7)]

What

When the configuration-service registry is empty, the first env-open inserts every device the worker knows about. If any one of those inserts failed, the registry was left partially populated AND the next env-open never tried to fix it — the system stayed in a permanent partial state until an operator manually intervened.

Why it happened

sync_devices_on_env_open used a bare asyncio.gather over the per-device upserts. Two problems:

1. The first failure cancelled the sibling tasks, so the registry only got some of the devices.
2. On the next env-open, the "is registry empty?" gate saw a non-empty registry, skipped the bootstrap, and the incremental change feed was never going to deliver the missing devices (they were never there to begin with — there was nothing to "change").

Fix

Move the bootstrap into a small helper that:

1. Runs gather with return_exceptions=True so survivors land instead of being cancelled.
2. Collects the names that failed.
3. Retries just those names exactly once.
4. If anything is still failing after the retry, raises ConfigServiceError naming every still-missing device and its underlying exception.

The retry is bounded (one extra attempt) so a genuinely broken upstream surfaces immediately rather than looping. The loud raise upholds the no-silent-fallback rule.

Symmetric in shape with the env-open lock recovery that just landed in #59.

Tests

In tests/manager/test_config_service.py:

• Updated test_sync_propagates_bootstrap_failure — the failure now has to repeat on retry to escape, and the wrapping error is ConfigServiceError, not the raw ConfigServiceHTTPError from the underlying POST.
• New test_sync_bootstrap_retries_partial_failure_and_succeeds — m1 succeeds, m2 fails-then-succeeds; no exception escapes, both devices in the registry, cursor read happens.
• New test_sync_bootstrap_raises_loudly_with_all_unrecoverable_failures — both devices fail both attempts; raised message names every still-missing device.
• New test_sync_bootstrap_does_not_retry_when_first_attempt_succeeds — sanity guard against the retry path being entered unnecessarily.

Verified the three new/updated tests fail without the fix and pass with it.

Full tests/manager/test_config_service.py + test_config_service_integration.py: 94 passed.

[Copilot]

Pull request overview

This PR hardens queueserver env-open synchronization with the configuration-service by preventing “partially bootstrapped” registries when the initial empty-registry bootstrap hits transient per-device failures.

Changes:

• Introduces _bootstrap_with_retry() to run per-device upserts with asyncio.gather(..., return_exceptions=True), retry failed devices once, and raise a single ConfigServiceError enumerating remaining failures.
• Updates/adds unit tests to cover: partial-failure recovery, bounded retry behavior, and loud error reporting when failures persist.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
backend/queueserver_service/queueserver_service/manager/config_service.py	Adds bootstrap-with-retry helper and routes env-open bootstrap through it to avoid sibling-task cancellation and permanent partial registry states after transient failures.
backend/queueserver_service/tests/manager/test_config_service.py	Updates existing bootstrap-failure test and adds new tests for partial-failure retry success, bounded retries, and multi-failure loud errors.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.