Skip to content

fix(handlers): preserve typed errors and attribute partial state in account deletion - #676

Closed
dmeiser wants to merge 9 commits into
mainfrom
fm/KW-673-ERROR-TYPING
Closed

dmeiser wants to merge 9 commits into
mainfrom
fm/KW-673-ERROR-TYPING

Conversation

@dmeiser

@dmeiser dmeiser commented Oct 3, 2026

Copy link
Copy Markdown
Owner

Intent

Stop the account-deletion error handling from swallowing typed errors and from leaving the most likely partial state ambiguous. Both new handlers catch bare except Exception, which rewraps the Cognito helper's own AppError into a different message and silently downgrades any future typed code from that helper - notably the retryable RESOURCE_BUSY that this project mandates for throttling, so a throttled Cognito failure would surface as a generic internal error instead of a retryable one. Narrow the catch to the exception types the helper actually raises so AppError propagates with its code intact. Separately, delete_all_user_data runs eight delete sub-cascades and then the accounts-row delete, and a failure of that final delete - the most likely partial state in the handler - still propagates as a raw ClientError to the decorator and surfaces as the generic "Failed to delete account", which is exactly the ambiguity this change exists to remove. Make that failure say what was swept and what survived. Tests must assert on the message as well as the code, since a code-only assertion passes against wrong wording.

What Changed

  • Narrow the except Exception catches in delete_my_account and admin_purge_user_account to (ClientError, BotoCoreError, AppError): a typed AppError from the Cognito helper (e.g. the retryable RESOURCE_BUSY) now re-raises with its code and message intact, throttled admin_delete_user failures are classified as RESOURCE_BUSY instead of INTERNAL_ERROR, and a Cognito delete failure after a successful sweep reports that Cognito state is unknown and needs a retry rather than claiming the user survived.
  • delete_all_user_data now tracks the eight sub-cascades it sweeps and wraps the final accounts-row delete: a failure raises a typed AppError (retryable RESOURCE_BUSY when throttled) whose message and structured log state that all user data was deleted while the account record and Cognito sign-in survive.
  • Add unit tests across test_account_operations.py, test_admin_operations.py, and test_account_deletion_cascade.py asserting both the error code and the exact wording of the partial-state messages, plus pass-through of typed errors and retry convergence.

Risk Assessment

⚠️ Medium: The change cleanly delivers its stated intent for the self-service path (typed AppError pass-through, honest partial-state attribution, exact-string message tests), but the admin-purge sibling of the same throttle-remedy divergence turned out as the inconsistency in F1, so two mechanical follow-ups are warranted before this is considered fully settled.

Testing

Baseline targeted pytest (294 passed) plus a from-scratch disposable live stack — a local moto AWS behind a wire-level fault-injecting gateway — through which I invoked the real delete_my_account and admin_purge_user_account handlers with AppSync events across seven scenarios (happy path, Cognito throttle, transport land-and-drop, pre-check drop, and three flavors of final accounts-row delete failure), asserting the exact payload errorCode/message, persisted DynamoDB/S3/Cognito state, structured logs, and retry convergence; all 62 checks passed, the identical scenarios fail against the extracted pre-fix base commit, and a Node bridge through the real AppSync resolver modules confirms the typed codes land in extensions.errorCode. Evidence: run log, full JSON transcripts (fixed + base), proxy request trace, and bridge output in the evidence directory; no UI surface is involved (backend/Lambda-only change).

  • Live validation: ✅ go - 7 of 7 scenarios driven live against the product
Scenario Result Live Evidence
Signed-in user completes self-service deleteMyAccount: returns true, all user data + accounts row + Cognito user + S3 objects removed, catalog preserved ✅ pass live live_driver_transcript.json → scenarios.self_service_delete_completes_end_to_end (8 checks); live_driver_run.log line 1
Throttled Cognito delete during self-service deletion surfaces as retryable RESOURCE_BUSY naming the partial state (sweep done, Cognito delete unconfirmed), and a retry completes the deletion ✅ pass live live_driver_transcript.json → s2: first_attempt {errorCode: RESOURCE_BUSY, message: 'Account data was deleted but the Cognito user could not be confirmed deleted; retry to complete the deletion'}, cog…
Admin purge's throttled Cognito delete keeps the helper's typed RESOURCE_BUSY message intact through the narrowed catch (not rewrapped), sweep done, retry completes the purge ✅ pass live live_driver_transcript.json → admin_purge_throttled_cognito_delete_keeps_helper_typed_error (11 checks: RESOURCE_BUSY + 'Failed to delete user from Cognito', invite/share/S3 swept, accounts row + Cogn…
Adversarial: transport fault on admin Cognito delete (request landed, response dropped) reports unknown state — never claims the user is present, not the generic decorator wording ✅ pass live live_driver_transcript.json → admin_purge_transport_fault_reports_unknown_cognito_state (8 checks; ground truth cognito_users_actual=0 while payload says 'could not be confirmed deleted; retry the pur…
Adversarial: transport fault after the sweep in self-service delete does not crash (BotoCoreError has no .response) — partial-state message instead of AttributeError → generic 'Failed to delete accoun… ✅ pass live live_driver_transcript.json → self_service_transport_fault_after_sweep_no_crash (INTERNAL_ERROR + partial-state message, sweep completed, Cognito user actually deleted upstream, partial-state log pres…
Adversarial: transport fault on the pre-check Cognito lookup fails early with the typed message, emits the handler's own pre-check log, and deletes nothing ✅ pass live live_driver_transcript.json → self_service_precheck_lookup_transport_fault_fails_early (INTERNAL_ERROR 'Failed to delete account', 'Cognito lookup failed before deletion' log line, accounts row/profil…
Final accounts-row delete failure after the eight sub-cascades names what was swept and what survived for throttle (RESOURCE_BUSY), permanent failure, and transport fault, and a retry converges ✅ pass live live_driver_transcript.json → final_accounts_row_delete_failure_names_swept_and_survivors (14 checks: three distinct fault flavors all emit 'All user data was deleted but the account record could not…
Evidence: Live run log — 7/7 scenarios PASS (post-fix)
[PASS] self_service_delete_completes_end_to_end (8 checks, 1.31s)
[PASS] self_service_cognito_throttle_is_retryable_partial_state (9 checks, 1.67s)
[PASS] admin_purge_throttled_cognito_delete_keeps_helper_typed_error (11 checks, 1.49s)
[PASS] admin_purge_transport_fault_reports_unknown_cognito_state (8 checks, 9.88s)
[PASS] self_service_transport_fault_after_sweep_no_crash (6 checks, 10.65s)
[PASS] self_service_precheck_lookup_transport_fault_fails_early (6 checks, 6.65s)
[PASS] final_accounts_row_delete_failure_names_swept_and_survivors (14 checks, 52.67s)
transcript: ~/.no-mistakes/evidence/01M417Z90ZGJNCECS9M38QA0QW/live_driver_transcript.json
ALL SCENARIOS PASSED
Evidence: Full live transcript — payloads, state, structured logs, per-check results (post-fix)
{
  "scenarios": {
    "self_service_delete_completes_end_to_end": {
      "error": null,
      "checks": [
        {
          "check": "deleteMyAccount returns success",
          "ok": true,
          "actual": true,
          "expected": true
        },
        {
          "check": "all five sweep tables are empty",
          "ok": true,
          "actual": [
            0,
            0,
            0,
            0,
            0
          ],
          "expected": [
            0,
            0,
            0,
            0,
            0
          ]
        },
        {
          "check": "accounts row deleted",
          "ok": true,
          "actual": false,
          "expected": false
        },
        {
          "check": "Cognito user deleted",
          "ok": true,
          "actual": 0,
          "expected": 0
        },
        {
          "check": "S3 reports swept",
          "ok": true,
          "actual": 0,
          "expected": 0
        },
        {
          "check": "S3 QR codes swept",
          "ok": true,
          "actual": 0,
          "expected": 0
        },
        {
          "check": "catalog row preserved by product rule",
          "ok": true,
          "detail": {
            "kernelworx-profiles-v2-ue1-dev": 0,
            "kernelworx-campaigns-v2-ue1-dev": 0,
            "kernelworx-orders-v2-ue1-dev": 0,
            "kernelworx-shares-ue1-dev": 0,
            "kernelworx-invites-ue1-dev": 0,
            "catalog_rows_preserved": 1,
            "accounts_row_present": false,
            "s3_reports": 0,
            "s3_qr_codes": 0
          }
        },
        {
          "check": "data existed before the call",
          "ok": true,
          "detail": {
            "kernelworx-profiles-v2-ue1-dev": 2,
            "kernelworx-campaigns-v2-ue1-dev": 2,
            "kernelworx-orders-v2-ue1-dev": 2,
            "kernelworx-shares-ue1-dev": 3,
            "kernelworx-invites-ue1-dev": 2,
            "catalog_rows_preserved": 1,
            "accounts_row_present": true,
            "s3_reports": 1,
            "s3_qr_codes": 1
          }
        }
      ],
      "elapsed_s": 1.31,
      "passed": true,
      "result": true,
      "state_before": {
        "kernelworx-profiles-v2-ue1-dev": 2,
        "kernelworx-campaigns-v2-ue1-dev": 2,
        "kernelworx-orders-v2-ue1-dev": 2,
        "kernelworx-shares-ue1-dev": 3,
        "kernelworx-invites-ue1-dev": 2,
        "catalog_rows_preserved": 1,
        "accounts_row_present": true,
        "s3_reports": 1,
        "s3_qr_codes": 1
      },
      "state_after": {
        "kernelworx-profiles-v2-ue1-dev": 0,
        "kernelworx-campaigns-v2-ue1-dev": 0,
        "kernelworx-orders-v2-ue1-dev": 0,
        "kernelworx-shares-ue1-dev": 0,
        "kernelworx-invites-ue1-dev": 0,
        "catalog_rows_preserved": 1,
        "accounts_row_present": false,
        "s3_reports": 0,
        "s3_qr_codes": 0
      }
    },
    "self_service_cognito_throttle_is_retryable_partial_state": {
      "error": null,
      "checks": [
        {
          "check": "errorCode is retryable RESOURCE_BUSY",
          "ok": true,
          "actual": "RESOURCE_BUSY",
          "expected": "RESOURCE_BUSY"
        },
        {
          "check": "message names the exact partial state",
          "ok": true,
          "actual": "Account data was deleted but the Cognito user could not be confirmed deleted; retry to complete the deletion",
          "expected": "Account data was deleted but the Cognito user could not be confirmed deleted; retry to complete the deletion"
        },
        {
          "check": "data swept before the failure",
          "ok": true,
          "actual": [
            0,
            0,
            0,
            0,
            0
          ],
          "expected": [
            0,
            0,
            0,
            0,
            0
          ]
        },
        {
          "check": "accounts row already swept",
          "ok": true,
          "actual": false,
          "expected": false
        },
        {
          "check": "Cognito user still present for a retry",
          "ok": true,
          "actual": 1,
          "expected": 1
        },
        {
          "check": "old generic wording absent",
          "ok": true,
          "actual": true,
          "expected": true
        },
        {
          "check": "structured log names the partial state",
          "ok": true,
          "detail": [
            "{\"timestamp\": \"2026-10-03T17:13:29.255008+00:00\", \"level\": \"ERROR\", \"message\": \"Account data swept but the Cognito delete did not report success; the account's Cognito state is unknown and needs a retry or manual completion\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\", \"account_id\": \"d12b7d33-19e5-4ee0-b5f3-10da5f6693fc\", \"error\": \"An error occurred (TooManyRequestsException) when calling the AdminDeleteUser operation: Rate exceeded\", \"error_code\": \"RESOURCE_BUSY\", \"traceback\": \"Traceback (most recent call last):\\n  File \\\"~/.no-mistakes/worktrees/0e3f105a97cb/01M417Z90ZGJNCECS9M38QA0QW/src/handlers/account_operations.py\\\", line 133, in delete_my_account\\n    _delete_user_from_cognito(cognito, user_pool_id, account_id, username, logger)\\n    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\\"~/.no-mistakes/worktrees/0e3f105a97cb/01M417Z90ZGJNCECS9M38QA0QW/src/handlers/account_operations.py\\\", line 74, in _delete_user_from_cognito\\n    retry_on_transient_errors(cognito.admin_delete_user, UserPoolId=user_pool_id, Username=username)\\n    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\\"~/.no-mistakes/worktrees/0e3f105a97cb/01M417Z90ZGJNCECS9M38QA0QW/src/utils/cognito.py\\\", line 45, in retry_on_transient_errors\\n    return method(*args, **kwargs)\\n  File \\\"~/.no-mistakes/worktrees/0e3f105a97cb/01M417Z90ZGJNCECS9M38QA0QW/.venv/lib/python3.14/site-packages/botocore/client.py\\\", line 602, in _api_call\\n    return self._make_api_call(operation_name, kwargs)\\n           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\\"~/.no-mistakes/worktrees/0e3f105a97cb/01M417Z90ZGJNCECS9M38QA0QW/.venv/lib/python3.14/site-packages/botocore/context.py\\\", line 123, in wrapper\\n    return func(*args, **kwargs)\\n  File \\\"~/.no-mistakes/worktrees/0e3f105a97cb/01M417Z90ZGJNCECS9M38QA0QW/.venv/lib/python3.14/site-packages/botocore/client.py\\\", line 1078, in _make_api_call\\n    raise error_class(parsed_response, operation_name)\\nbotocore.errorfactory.TooManyRequestsException: An error occurred (TooManyRequestsException) when calling the AdminDeleteUser operation: Rate exceeded\\n\"}"
          ]
        },
        {
          "check": "retry completes the deletion",
          "ok": true,
          "actual": true,
          "expected": true
        },
        {
          "check": "retry removed the Cognito user",
          "ok": true,
          "actual": 0,
          "expected": 0
        }
      ],
      "elapsed_s": 1.67,
      "passed": true,
      "first_attempt": {
        "__isError": true,
        "errorCode": "RESOURCE_BUSY",
        "message": "Account data was deleted but the Cognito user could not be confirmed deleted; retry to complete the deletion"
      },
      "state_after_failure": {
        "kernelworx-profiles-v2-ue1-dev": 0,
        "kernelworx-campaigns-v2-ue1-dev": 0,
        "kernelworx-orders-v2-ue1-dev": 0,
        "kernelworx-shares-ue1-dev": 0,
        "kernelworx-invites-ue1-dev": 0,
        "catalog_rows_preserved": 1,
        "accounts_row_present": false,
        "s3_reports": 0,
        "s3_qr_codes": 0
      },
      "cognito_users_after_failure": 1,
      "logs": "{\"timestamp\": \"2026-10-03T17:13:28.769368+00:00\", \"level\": \"INFO\", \"message\": \"delete_my_account handler invoked\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\"}\n{\"timestamp\": \"2026-10-03T17:13:28.769463+00:00\", \"level\": \"INFO\", \"message\": \"Deleting account for: d12b7d33-19e5-4ee0-b5f3-10da5f6693fc\", \"correlation

... [71476 bytes truncated] ...

essage": "All user data was deleted but the account record could not be removed, so the sign-in remains active; retry to complete the deletion"
          },
          "logs": "{\"timestamp\": \"2026-10-03T17:13:59.190815+00:00\", \"level\": \"INFO\", \"message\": \"delete_my_account handler invoked\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\"}\n{\"timestamp\": \"2026-10-03T17:13:59.190913+00:00\", \"level\": \"INFO\", \"message\": \"Deleting account for: f9b4ee41-d87b-4585-bb89-b0542efe3fa5\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\"}\n{\"timestamp\": \"2026-10-03T17:13:59.218129+00:00\", \"level\": \"INFO\", \"message\": \"Deleted batch of 1 items from kernelworx-orders-v2-ue1-dev\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\"}\n{\"timestamp\": \"2026-10-03T17:13:59.222597+00:00\", \"level\": \"INFO\", \"message\": \"Deleted orders for campaign\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\", \"campaign_id\": \"CAMPAIGN#s1\", \"count\": 1}\n{\"timestamp\": \"2026-10-03T17:13:59.234733+00:00\", \"level\": \"INFO\", \"message\": \"Deleted batch of 1 items from kernelworx-orders-v2-ue1-dev\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\"}\n{\"timestamp\": \"2026-10-03T17:13:59.239460+00:00\", \"level\": \"INFO\", \"message\": \"Deleted orders for campaign\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\", \"campaign_id\": \"CAMPAIGN#s2\", \"count\": 1}\n{\"timestamp\": \"2026-10-03T17:13:59.239539+00:00\", \"level\": \"INFO\", \"message\": \"Deleted user orders\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\", \"count\": 2}\n{\"timestamp\": \"2026-10-03T17:13:59.246960+00:00\", \"level\": \"INFO\", \"message\": \"Deleted batch of 1 items from kernelworx-campaigns-v2-ue1-dev\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\"}\n{\"timestamp\": \"2026-10-03T17:13:59.259733+00:00\", \"level\": \"INFO\", \"message\": \"Deleted batch of 1 items from kernelworx-campaigns-v2-ue1-dev\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\"}\n{\"timestamp\": \"2026-10-03T17:13:59.263528+00:00\", \"level\": \"INFO\", \"message\": \"Deleted user campaigns\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\", \"count\": 2}\n{\"timestamp\": \"2026-10-03T17:13:59.272027+00:00\", \"level\": \"INFO\", \"message\": \"Deleted batch of 1 items from kernelworx-shares-ue1-dev\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\"}\n{\"timestamp\": \"2026-10-03T17:13:59.280507+00:00\", \"level\": \"INFO\", \"message\": \"Deleted batch of 1 items from kernelworx-shares-ue1-dev\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\"}\n{\"timestamp\": \"2026-10-03T17:13:59.280581+00:00\", \"level\": \"INFO\", \"message\": \"Deleted user shares\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\", \"count\": 2}\n{\"timestamp\": \"2026-10-03T17:13:59.289623+00:00\", \"level\": \"INFO\", \"message\": \"Deleted batch of 1 items from kernelworx-invites-ue1-dev\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\"}\n{\"timestamp\": \"2026-10-03T17:13:59.298057+00:00\", \"level\": \"INFO\", \"message\": \"Deleted batch of 1 items from kernelworx-invites-ue1-dev\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\"}\n{\"timestamp\": \"2026-10-03T17:13:59.298141+00:00\", \"level\": \"INFO\", \"message\": \"Deleted invites for owned profiles\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\", \"count\": 2}\n{\"timestamp\": \"2026-10-03T17:13:59.306286+00:00\", \"level\": \"INFO\", \"message\": \"Deleted batch of 1 items from kernelworx-shares-ue1-dev\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\"}\n{\"timestamp\": \"2026-10-03T17:13:59.306357+00:00\", \"level\": \"INFO\", \"message\": \"Deleted inbound shares\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\", \"account_id\": \"f9b4ee41-d87b-4585-bb89-b0542efe3fa5\", \"count\": 1}\n{\"timestamp\": \"2026-10-03T17:13:59.323998+00:00\", \"level\": \"INFO\", \"message\": \"Deleted 1 object versions from S3 under reports/PROFILE#s1/\", \"correlationId\": \"eaca90bc-076b-422d-bd96-0d028255dcf9\"}\n{\"timestamp\": \"2026-10-03T17:13:59.345091+00:00\", \"level\": \"INFO\", \"message\": \"Deleted user S3 reports\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\", \"count\": 1}\n{\"timestamp\": \"2026-10-03T17:13:59.349188+00:00\", \"level\": \"INFO\", \"message\": \"Deleted batch of 2 items from kernelworx-profiles-v2-ue1-dev\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\"}\n{\"timestamp\": \"2026-10-03T17:13:59.349263+00:00\", \"level\": \"INFO\", \"message\": \"Deleted user profiles\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\", \"account_id\": \"f9b4ee41-d87b-4585-bb89-b0542efe3fa5\", \"count\": 2}\n{\"timestamp\": \"2026-10-03T17:13:59.364475+00:00\", \"level\": \"INFO\", \"message\": \"Deleted 1 object versions from S3 under payment-qr-codes/f9b4ee41-d87b-4585-bb89-b0542efe3fa5/\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\"}\n{\"timestamp\": \"2026-10-03T17:14:24.944794+00:00\", \"level\": \"ERROR\", \"message\": \"All user data was swept but the account record could not be deleted; the account record and Cognito sign-in survive and need a retry or manual completion\", \"correlationId\": \"9ba8fd4f-010f-4d25-9a6b-27c720baa151\", \"account_id\": \"ACCOUNT#f9b4ee41-d87b-4585-bb89-b0542efe3fa5\", \"swept\": {\"orders\": 2, \"campaigns\": 2, \"shares\": 2, \"invites\": 2, \"inbound_shares\": 1, \"s3_reports\": 1, \"profiles\": 2, \"qr_codes\": 1}, \"error\": \"An error occurred (ThrottlingException) when calling the DeleteItem operation (reached max retries: 9): Provisioned throughput exceeded\", \"error_code\": \"RESOURCE_BUSY\", \"traceback\": \"Traceback (most recent call last):\\n  File \\\"~/.no-mistakes/worktrees/0e3f105a97cb/01M417Z90ZGJNCECS9M38QA0QW/src/handlers/deletion_cascade.py\\\", line 272, in delete_all_user_data\\n    tables.accounts.delete_item(Key={\\\"accountId\\\": account_id_key})\\n    ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\\"~/.no-mistakes/worktrees/0e3f105a97cb/01M417Z90ZGJNCECS9M38QA0QW/.venv/lib/python3.14/site-packages/boto3/resources/factory.py\\\", line 581, in do_action\\n    response = action(self, *args, **kwargs)\\n  File \\\"~/.no-mistakes/worktrees/0e3f105a97cb/01M417Z90ZGJNCECS9M38QA0QW/.venv/lib/python3.14/site-packages/boto3/resources/action.py\\\", line 88, in __call__\\n    response = getattr(parent.meta.client, operation_name)(*args, **params)\\n  File \\\"~/.no-mistakes/worktrees/0e3f105a97cb/01M417Z90ZGJNCECS9M38QA0QW/.venv/lib/python3.14/site-packages/botocore/client.py\\\", line 602, in _api_call\\n    return self._make_api_call(operation_name, kwargs)\\n           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\\"~/.no-mistakes/worktrees/0e3f105a97cb/01M417Z90ZGJNCECS9M38QA0QW/.venv/lib/python3.14/site-packages/botocore/context.py\\\", line 123, in wrapper\\n    return func(*args, **kwargs)\\n  File \\\"~/.no-mistakes/worktrees/0e3f105a97cb/01M417Z90ZGJNCECS9M38QA0QW/.venv/lib/python3.14/site-packages/botocore/client.py\\\", line 1078, in _make_api_call\\n    raise error_class(parsed_response, operation_name)\\nbotocore.errorfactory.ThrottlingException: An error occurred (ThrottlingException) when calling the DeleteItem operation (reached max retries: 9): Provisioned throughput exceeded\\n\"}\n"
        },
        "denied": {
          "result": {
            "__isError": true,
            "errorCode": "INTERNAL_ERROR",
            "message": "All user data was deleted but the account record could not be removed, so the sign-in remains active; retry to complete the deletion"
          }
        },
        "transport": {
          "result": {
            "__isError": true,
            "errorCode": "INTERNAL_ERROR",
            "message": "All user data was deleted but the account record could not be removed, so the sign-in remains active; retry to complete the deletion"
          }
        }
      }
    }
  },
  "failures": 0
}
Evidence: Regression run log — base commit fails 6/7 intent scenarios
[PASS] self_service_delete_completes_end_to_end (8 checks, 1.31s)
[FAIL] self_service_cognito_throttle_is_retryable_partial_state (9 checks, 1.68s)
    FAILED: {"check": "errorCode is retryable RESOURCE_BUSY", "ok": false, "actual": "INTERNAL_ERROR", "expected": "RESOURCE_BUSY"}
    FAILED: {"check": "message names the exact partial state", "ok": false, "actual": "Failed to delete account", "expected": "Account data was deleted but the Cognito user could not be confirmed deleted; retry to complete the deletion"}
    FAILED: {"check": "old generic wording absent", "ok": false, "actual": false, "expected": true}
    FAILED: {"check": "structured log names the partial state", "ok": false, "detail": []}
[FAIL] admin_purge_throttled_cognito_delete_keeps_helper_typed_error (11 checks, 1.25s)
    FAILED: {"check": "errorCode is retryable RESOURCE_BUSY", "ok": false, "actual": "INTERNAL_ERROR", "expected": "RESOURCE_BUSY"}
[FAIL] admin_purge_transport_fault_reports_unknown_cognito_state (8 checks, 9.19s)
    FAILED: {"check": "message reports unknown Cognito state", "ok": false, "actual": "Failed to purge user account", "expected": "Account data was deleted but the Cognito user could not be confirmed deleted; retry the purge to complete the deletion"}
    FAILED: {"check": "not the generic decorator wording", "ok": false, "actual": false, "expected": true}
    FAILED: {"check": "structured log names the partial state", "ok": false}
[FAIL] self_service_transport_fault_after_sweep_no_crash (6 checks, 5.96s)
    FAILED: {"check": "partial-state message emitted", "ok": false, "actual": "Failed to delete account", "expected": "Account data was deleted but the Cognito user could not be confirmed deleted; retry to complete the deletion"}
    FAILED: {"check": "not the decorator's generic wording (what an unclassified crash would produce)", "ok": false, "actual": false, "expected": true}
    FAILED: {"check": "structured log names the partial state", "ok": false}
[FAIL] self_service_precheck_lookup_transport_fault_fails_early (6 checks, 8.68s)
    FAILED: {"check": "handler's own pre-check log emitted (raw escape to the decorator would not log this)", "ok": false, "detail": []}
[FAIL] final_accounts_row_delete_failure_names_swept_and_survivors (14 checks, 52.59s)
    FAILED: {"check": "throttled final delete -> RESOURCE_BUSY", "ok": false, "actual": "INTERNAL_ERROR", "expected": "RESOURCE_BUSY"}
    FAILED: {"check": "throttled final delete -> swept/survived message", "ok": false, "actual": "Failed to delete account", "expected": "All user data was deleted but the account record could not be removed, so the sign-in remains active; retry to complete the deletion"}
    FAILED: {"check": "structured log records swept survivors", "ok": false}
    FAILED: {"check": "permanent final delete -> same swept/survived message", "ok": false, "actual": "Failed to delete account", "expected": "All user data was deleted but the account record could not be removed, so the sign-in remains active; retry to complete the deletion"}
    FAILED: {"check": "transport final delete -> same swept/survived message", "ok": false, "actual": "Failed to delete account", "expected": "All user data was deleted but the account record could not be removed, so the sign-in remains active; retry to complete the deletion"}
transcript: ~/.no-mistakes/evidence/01M417Z90ZGJNCECS9M38QA0QW/live_driver_base_transcript.json
6 SCENARIO(S) FAILED
  • Evidence: Regression transcript — base-commit payloads showing generic wording (local file: ~/.no-mistakes/evidence/01M417Z90ZGJNCECS9M38QA0QW/live_driver_base_transcript.json)
Evidence: Real AppSync resolvers map live payloads to GraphQL errorType + extensions.errorCode
[
  {
    "case": "s2",
    "resolver": "deleteMyAccount (lambda_unit_resolver)",
    "handler_payload": {
      "__isError": true,
      "errorCode": "RESOURCE_BUSY",
      "message": "Account data was deleted but the Cognito user could not be confirmed deleted; retry to complete the deletion"
    },
    "graphql": {
      "error": {
        "message": "RESOURCE_BUSY: Account data was deleted but the Cognito user could not be confirmed deleted; retry to complete the deletion",
        "errorType": "RESOURCE_BUSY",
        "extensions": {
          "errorCode": "RESOURCE_BUSY"
        }
      }
    }
  },
  {
    "case": "s7-throttle",
    "resolver": "deleteMyAccount (lambda_unit_resolver)",
    "handler_payload": {
      "__isError": true,
      "errorCode": "RESOURCE_BUSY",
      "message": "All user data was deleted but the account record could not be removed, so the sign-in remains active; retry to complete the deletion"
    },
    "graphql": {
      "error": {
        "message": "RESOURCE_BUSY: All user data was deleted but the account record could not be removed, so the sign-in remains active; retry to complete the deletion",
        "errorType": "RESOURCE_BUSY",
        "extensions": {
          "errorCode": "RESOURCE_BUSY"
        }
      }
    }
  },
  {
    "case": "s3",
    "resolver": "adminPurgeUserAccount (lambda_passthrough_resolver)",
    "handler_payload": {
      "__isError": true,
      "errorCode": "RESOURCE_BUSY",
      "message": "Failed to delete user from Cognito"
    },
    "graphql": {
      "error": {
        "message": "RESOURCE_BUSY: Failed to delete user from Cognito",
        "errorType": "RESOURCE_BUSY",
        "extensions": {
          "errorCode": "RESOURCE_BUSY"
        }
      }
    }
  },
  {
    "case": "s4",
    "resolver": "adminPurgeUserAccount (lambda_passthrough_resolver)",
    "handler_payload": {
      "__isError": true,
      "errorCode": "INTERNAL_ERROR",
      "message": "Account data was deleted but the Cognito user could not be confirmed deleted; retry the purge to complete the deletion"
    },
    "graphql": {
      "error": {
        "message": "INTERNAL_ERROR: Account data was deleted but the Cognito user could not be confirmed deleted; retry the purge to complete the deletion",
        "errorType": "INTERNAL_ERROR",
        "extensions": {
          "errorCode": "INTERNAL_ERROR"
        }
      }
    }
  }
]
  • Evidence: Fault gateway request trace — target ops and active fault mode per request (local file: ~/.no-mistakes/evidence/01M417Z90ZGJNCECS9M38QA0QW/fault_mode.txt.trace)
  • Evidence: Live driver (disposable stack + 7 scenarios) (local file: ~/.no-mistakes/evidence/01M417Z90ZGJNCECS9M38QA0QW/live_driver.py)
Evidence: Wire-level fault-injecting gateway
#!/usr/bin/env python3
"""Fault-injecting HTTP gateway in front of a local moto server.

Usage: fault_proxy.py <listen_port> <moto_host> <moto_port> <mode_file> <accounts_table>

The proxy forwards every AWS API request to the moto server, except when the
mode file names an active fault. Faults are injected at the service boundary
(wire level), so the product's real botocore clients observe them exactly as
they would observe real AWS faults:

  none                          no faults
  cognito_throttle_admin_delete AdminDeleteUser -> 400 TooManyRequestsException
  cognito_land_and_drop_admin_delete
                                AdminDeleteUser is forwarded (it LANDS upstream),
                                but the response is swallowed and the connection
                                is dropped -> botocore ConnectionError (BotoCoreError)
  cognito_drop_list_users       ListUsers -> connection dropped before forwarding
  dynamo_throttle_accounts_delete
                                DeleteItem on the accounts table -> 400 ThrottlingException
  dynamo_denied_accounts_delete
                                DeleteItem on the accounts table -> 400 AccessDeniedException
  dynamo_drop_accounts_delete   DeleteItem on the accounts table -> connection dropped
"""

import http.client
import json
import socket
import sys
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

LISTEN_PORT = int(sys.argv[1])
MOTO_HOST = sys.argv[2]
MOTO_PORT = int(sys.argv[3])
MODE_FILE = sys.argv[4]
ACCOUNTS_TABLE = sys.argv[5]

HOP_BY_HOP = {
    "connection",
    "keep-alive",
    "proxy-authenticate",
    "proxy-authorization",
    "te",
    "trailers",
    "transfer-encoding",
    "upgrade",
}


def current_mode() -> str:
    try:
        with open(MODE_FILE, encoding="utf-8") as handle:
            return handle.read().strip() or "none"
    except FileNotFoundError:
        return "none"


class Handler(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"

    def log_message(self, *args):  # noqa: D102 - silence default stderr logging
        pass

    def _read_body(self) -> bytes:
        length = int(self.headers.get("Content-Length") or 0)
        return self.rfile.read(length) if length else b""

    def _respond(self, status: int, content_type: str, data: bytes) -> None:
        self.send_response(status)
        self.send_header("Content-Type", content_type)
        self.send_header("Content-Length", str(len(data)))
        self.send_header("Connection", "close")
        self.end_headers()
        if self.command != "HEAD":
            self.wfile.write(data)
        self.close_connection = True

    def _inject_error(self, code: str, message: str) -> None:
        body = json.dumps({"__type": f"com.amazonaws.smoke#{code}", "message": message})
        self._respond(400, "application/x-amz-json-1.1", body.encode("utf-8"))

    def _drop(self) -> None:
        """Close the client connection without sending any response bytes."""
        self.close_connection = True
        try:
            self.connection.shutdown(socket.SHUT_RDWR)
        except OSError:
            pass
        self.connection.close()

    def _forward(self, body: bytes, drop_after_upstream: bool = False) -> None:
        conn = http.client.HTTPConnection(MOTO_HOST, MOTO_PORT, timeout=30)
        try:
            headers = {
                key: value
                for key, value in self.headers.items()
                if key.lower() not in HOP_BY_HOP and key.lower() != "host"
            }
            headers["Host"] = f"{MOTO_HOST}:{MOTO_PORT}"
            conn.request(self.command, self.path, body=body or None, headers=headers)
            response = conn.getresponse()
            data = response.read()
            status = response.status
            content_type = response.getheader("Content-Type", "application/octet-stream")
        finally:
            conn.close()

        if drop_after_upstream:
            # The request already landed upstream; swallow the response so the
            # client observes a transport failure with unknown server state.
            self._drop()
            return
        self._respond(status, content_type, data)

    def _handle(self) -> None:
        body = self._read_body()
        target = self.headers.get("X-Amz-Target", "")
        mode = current_mode()
        with open(MODE_FILE + ".trace", "a", encoding="utf-8") as trace:
            trace.write(f"target={target} mode={mode} path={self.path}\n")

        if target.startswith("AWSCognitoIdentityProviderService."):
            operation = target.split(".", 1)[1]
            if operation == "AdminDeleteUser":
                if mode == "cognito_throttle_admin_delete":
                    self._inject_error("TooManyRequestsException", "Rate exceeded")
                    return
                if mode == "cognito_land_and_drop_admin_delete":
                    self._forward(body, drop_after_upstream=True)
                    return
            if operation == "ListUsers" and mode == "cognito_drop_list_users":
                self._drop()
                return
        elif target.startswith("DynamoDB_") and mode.startswith("dynamo_"):
            try:
                table = json.loads(body or b"{}").get("TableName", "")
            except json.JSONDecodeError:
                table = ""
            if table == ACCOUNTS_TABLE and target.endswith(".DeleteItem"):
                if mode == "dynamo_throttle_accounts_delete":
                    self._inject_error("ThrottlingException", "Provisioned throughput exceeded")
                    return
                if mode == "dynamo_denied_accounts_delete":
                    self._inject_error("AccessDeniedException", "User is not authorized to perform: dynamodb:DeleteItem")
                    return
                if mode == "dynamo_drop_accounts_delete":
                    self._drop()
                    return

        self._forward(body)

    def do_GET(self):
        self._handle()

    def do_POST(self):
        self._handle()

    def do_PUT(self):
        self._handle()

    def do_DELETE(self):
        self._handle()

    def do_HEAD(self):
        self._handle()


def main() -> None:
    server = ThreadingHTTPServer(("127.0.0.1", LISTEN_PORT), Handler)
    server.daemon_threads = True
    print(f"fault proxy on 127.0.0.1:{LISTEN_PORT} -> {MOTO_HOST}:{MOTO_PORT}", flush=True)
    server.serve_forever()


if __name__ == "__main__":
    main()
Evidence: Resolver bridge script
#!/usr/bin/env node
// Feeds the LIVE handler error payloads (from live_driver_transcript.json)
// through the real AppSync resolver modules (lambda_unit_resolver.js for the
// direct deleteMyAccount resolver, lambda_passthrough_resolver.js for the
// adminPurgeUserAccount resolver) and prints the GraphQL-visible error the
// frontend receives: errorType + extensions.errorCode.
//
// Run with: node --import <js-resolvers>/register-loader.mjs resolver_bridge.mjs
// (register-loader maps @aws-appsync/utils to the repo's test mock, whose
// util.error throws exactly like AppSync: `${type}: ${message}` + errorInfo.)

import { readFileSync } from 'node:fs';
import { dirname, join } from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';

const HERE = dirname(fileURLToPath(import.meta.url));
const RESOLVERS = pathToFileURL(
  '~/.no-mistakes/worktrees/0e3f105a97cb/01M417Z90ZGJNCECS9M38QA0QW/tofu/application/appsync/js-resolvers/'
).href;

const transcript = JSON.parse(readFileSync(join(HERE, 'live_driver_transcript.json'), 'utf8'));

const unitResolver = await import(RESOLVERS + 'lambda_unit_resolver.js');
const passthroughResolver = await import(RESOLVERS + 'lambda_passthrough_resolver.js');

const cases = [
  ['deleteMyAccount (lambda_unit_resolver)', unitResolver, 's2', transcript.scenarios.self_service_cognito_throttle_is_retryable_partial_state.first_attempt],
  ['deleteMyAccount (lambda_unit_resolver)', unitResolver, 's7-throttle', transcript.scenarios.final_accounts_row_delete_failure_names_swept_and_survivors.attempts.throttle.result],
  ['adminPurgeUserAccount (lambda_passthrough_resolver)', passthroughResolver, 's3', transcript.scenarios.admin_purge_throttled_cognito_delete_keeps_helper_typed_error.first_attempt],
  ['adminPurgeUserAccount (lambda_passthrough_resolver)', passthroughResolver, 's4', transcript.scenarios.admin_purge_transport_fault_reports_unknown_cognito_state.result],
];

const out = [];
for (const [label, mod, name, payload] of cases) {
  const entry = { case: name, resolver: label, handler_payload: payload };
  try {
    const returned = mod.response({ result: payload, error: null, stash: {}, prev: null });
    entry.graphql = { returned, error: null };
  } catch (err) {
    // AppSync surfaces this as a GraphQL error: message, errorType (= 2nd
    // util.error arg), and extensions.errorCode (4th arg merged into ext).
    entry.graphql = {
      error: {
        message: err.message,
        errorType: err.message.split(':')[0],
        extensions: err.errorInfo ?? null,
      },
    };
  }
  out.push(entry);
}
console.log(JSON.stringify(out, null, 2));

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 info
  • ⚠️ src/handlers/admin_operations.py:1046 - The admin purge's new narrowed catch cannot deliver the intent's mandated retryable RESOURCE_BUSY for a throttled Cognito failure, while its sibling in account_operations.py:132-148 now classifies raw ClientError via is_transient_client_error. Reachable sequence: AdminDeleteUser exhausts retry_on_transient_errors with TooManyRequestsException (src/utils/cognito.py:46-52 re-raises ClientError); admin _delete_user_from_cognito (src/handlers/admin_operations.py:837-841) rewraps that into AppError(INTERNAL_ERROR, 'Failed to delete user from Cognito'); in the new catch that AppError hits the if isinstance(e, AppError): raise branch at line 1044 and propagates as a non-retryable INTERNAL_ERROR. The new catch's ClientError limb is accordingly dead in this path and the partial-state message is never emitted for the retryable case; a raw transport BotoCoreError (connected but unknown outcome) also gets the fixed INTERNAL_ERROR без classification. The throttle classification exists only via the missing classify step: add error_code = ErrorCode.RESOURCE_BUSY if (isinstance(e, ClientError) and is_transient_client_error(e)) else ErrorCode.INTERNAL_ERROR and use the variable in both the log.kwargs and the AppError at lines 1047-1053. Sibling sites where the same 'protect the typed/default error, attribute raw faults' pattern must hold consistently: account_operations.py:142-148 (same helper, already supplied with RESOURCE_BUSY via tests), deletion_cascade.py:273 and deletion_cascade.py:251 (is_transient_client_error already carried there). A Prod AWS Cognito throttle is the trigger and the call path already exists; the exclusive divergence remains: 'rewraps the helper's own AppError' is fixed only by accident of the helper also having converted the throttle to INTERNAL first — the retryable signal intended for the caller is lost.
  • ℹ️ src/handlers/account_operations.py:115 - The pre-check lookup still catches only ClientError, so a transport BotoCoreError from _lookup_cognito_user_with_retry escapes the handler to the far-generic decorator wording ('Failed to delete account') — honest, but inconsistent with the new post-sweep block which deliberately added BotoCoreError to its catch for exactly the same transport class. A ConnectionError at the very first list_users yields the generic message while the rest of the flow now uses the precise pattern. Extending the pre-check except-branch to (ClientError, BotoCoreError) aligns the two. No masking risk: the existing branch already swallows only the message.
  • ℹ️ src/handlers/admin_operations.py:1063 - Pre-existing adjacent partial state (not introduced by this diff, observed while tracing the new code): the purge's accounts-row delete failure after the Cognito delete succeeded still raises AppError(INTERNAL_ERROR, 'Failed to delete account record') with no indication that the Cognito user is already gone — the same 'what survived is ambiguous' class the intent targets, in the step immediately following the changed block. The sweep and Cognito-delete ambiguity in the diff itself is addressed; this adjacent handler-owned wording was not. Informational only; the change is scoped to the sweep/Cognito boundary, not here.

🔧 Fix applied.
3 issues (1 error, 2 infos) still open:

  • ℹ️ src/handlers/account_operations.py:115 - The pre-check lookup still catches only ClientError, so a transport BotoCoreError from _lookup_cognito_user_with_retry escapes the handler to the far-generic decorator wording ('Failed to delete account') — honest, but inconsistent with the new post-sweep block which deliberately added BotoCoreError to its catch for exactly the same transport class. A ConnectionError at the very first list_users yields the generic message while the rest of the flow now uses the precise pattern. Extending the pre-check except-branch to (ClientError, BotoCoreError) aligns the two. No masking risk: the existing branch already swallows only the message.
  • ℹ️ src/handlers/admin_operations.py:1063 - Pre-existing adjacent partial state (not introduced by this diff, observed while tracing the new code): the purge's accounts-row delete failure after the Cognito delete succeeded still raises AppError(INTERNAL_ERROR, 'Failed to delete account record') with no indication that the Cognito user is already gone — the same 'what survived is ambiguous' class the intent targets, in the step immediately following the changed block. The sweep and Cognito-delete ambiguity in the diff itself is addressed; this adjacent handler-owned wording was not. Informational only; the change is scoped to the sweep/Cognito boundary, not here.
  • 🚨 src/handlers/account_operations.py:146 - The post-sweep catch calls is_transient_client_error(e) unconditionally, but e can be a BotoCoreError, which has no .response attribute (verified: botocore.exceptions.ConnectionError(...).hasattr('response') is False). Concrete reachable sequence: self-service delete_my_account completes the sweep, then admin_delete_user's transport fails (botocore raises ConnectionError/EndpointConnectionError, a BotoCoreError subclass, e.g. connection reset after the request was sent — precisely the unknown-state case the code comment says BotoCoreError is kept for); retry_on_transient_errors only catches ClientError (src/utils/cognito.py:43-52) so the BotoCoreError propagates raw into the handler's except (ClientError, BotoCoreError, AppError); line 146 evaluates e.response -> AttributeError. The AttributeError escapes the handler to the decorator, which returns the generic INTERNAL_ERROR 'Failed to delete account' — the exact pre-change wording this change exists to remove — in the most ambiguous failure mode (delete may have landed). The admin purge path has a behavior test for exactly this scenario (test_cognito_delete_raising_after_success_does_not_claim_user_present) but the self-service path has none, so the crash is untested. Remedy: error_code = ErrorCode.RESOURCE_BUSY if isinstance(e, ClientError) and is_transient_client_error(e) else ErrorCode.INTERNAL_ERROR. Sibling site of the same transport-fault attribution class, left behind by the pre-round-1 commit 742ff9a in the new partial-state block: src/handlers/deletion_cascade.py:273 catches only ClientError at the final accounts-row delete, so a BotoCoreError there (transport fault on the intent's target failure) still propagates raw to the decorator's generic 'Failed to delete account' instead of the mandated swept/survived message — widen that catch to (ClientError, BotoCoreError) and classify only the ClientError limb via is_transient_client_error (BotoCoreError -> INTERNAL_ERROR), keeping the intent's message for both. Neither site should call is_transient_client_error with a non-ClientError.

🔧 Fix applied.
1 info still open:

  • ℹ️ src/handlers/admin_operations.py:1063 - Pre-existing adjacent partial state (not introduced by this diff, observed while tracing the new code): the purge's accounts-row delete failure after the Cognito delete succeeded still raises AppError(INTERNAL_ERROR, 'Failed to delete account record') with no indication that the Cognito user is already gone — the same 'what survived is ambiguous' class the intent targets, in the step immediately following the changed block. The sweep and Cognito-delete ambiguity in the diff itself is addressed; this adjacent handler-owned wording was not. Informational only; the change is scoped to the sweep/Cognito boundary, not here.
✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 7 of 7 scenarios driven live against the product
Scenario Result Live Evidence
Signed-in user completes self-service deleteMyAccount: returns true, all user data + accounts row + Cognito user + S3 objects removed, catalog preserved ✅ pass live live_driver_transcript.json → scenarios.self_service_delete_completes_end_to_end (8 checks); live_driver_run.log line 1
Throttled Cognito delete during self-service deletion surfaces as retryable RESOURCE_BUSY naming the partial state (sweep done, Cognito delete unconfirmed), and a retry completes the deletion ✅ pass live live_driver_transcript.json → s2: first_attempt {errorCode: RESOURCE_BUSY, message: 'Account data was deleted but the Cognito user could not be confirmed deleted; retry to complete the deletion'}, cog…
Admin purge's throttled Cognito delete keeps the helper's typed RESOURCE_BUSY message intact through the narrowed catch (not rewrapped), sweep done, retry completes the purge ✅ pass live live_driver_transcript.json → admin_purge_throttled_cognito_delete_keeps_helper_typed_error (11 checks: RESOURCE_BUSY + 'Failed to delete user from Cognito', invite/share/S3 swept, accounts row + Cogn…
Adversarial: transport fault on admin Cognito delete (request landed, response dropped) reports unknown state — never claims the user is present, not the generic decorator wording ✅ pass live live_driver_transcript.json → admin_purge_transport_fault_reports_unknown_cognito_state (8 checks; ground truth cognito_users_actual=0 while payload says 'could not be confirmed deleted; retry the pur…
Adversarial: transport fault after the sweep in self-service delete does not crash (BotoCoreError has no .response) — partial-state message instead of AttributeError → generic 'Failed to delete accoun… ✅ pass live live_driver_transcript.json → self_service_transport_fault_after_sweep_no_crash (INTERNAL_ERROR + partial-state message, sweep completed, Cognito user actually deleted upstream, partial-state log pres…
Adversarial: transport fault on the pre-check Cognito lookup fails early with the typed message, emits the handler's own pre-check log, and deletes nothing ✅ pass live live_driver_transcript.json → self_service_precheck_lookup_transport_fault_fails_early (INTERNAL_ERROR 'Failed to delete account', 'Cognito lookup failed before deletion' log line, accounts row/profil…
Final accounts-row delete failure after the eight sub-cascades names what was swept and what survived for throttle (RESOURCE_BUSY), permanent failure, and transport fault, and a retry converges ✅ pass live live_driver_transcript.json → final_accounts_row_delete_failure_names_swept_and_survivors (14 checks: three distinct fault flavors all emit 'All user data was deleted but the account record could not…
  • Baseline targeted run: .venv/bin/pytest tests/unit/test_account_operations.py tests/unit/test_admin_operations.py tests/unit/test_account_deletion_cascade.py -q → 294 passed (coverage-gate line fails only because it is a subset run)
  • Built a disposable stack in the evidence dir: fault_proxy.py (HTTP fault gateway: injected Cognito throttle 400s, land-and-drop transport faults, injected DynamoDB DeleteItem throttle/access-denied/drop) in front of a local moto_server, with AWS_ENDPOINT_URL routing every product-side boto3 call through the gateway
  • Ran live_driver.py — 7 scenarios, 62 checks, all pass: happy-path self-service delete; self-service Cognito throttle → RESOURCE_BUSY + partial-state message + retry converges; admin purge throttle → helper's typed RESOURCE_BUSY passes through + retry converges; admin purge land-and-drop → unknown-state message, ground truth shows the delete actually landed; self-service transport fault after sweep → no AttributeError/generic wording; pre-check lookup drop → early typed failure with nothing swept; final accounts-row delete throttle/denied/transport → swept/survived message with RESOURCE_BUSY/INTERNAL_ERROR + retry converges
  • Regression: same harness against the extracted base commit (LIVE_DRIVER_SRC_ROOT=/tmp/kw673_base git archive b4134fd… src) → all 6 intent scenarios FAIL on their discriminating checks (generic 'Failed to delete account'/'Failed to purge user account', INTERNAL_ERROR instead of RESOURCE_BUSY) while the happy path passes; temp checkout removed afterwards
  • Drove the live error payloads through the real lambda_unit_resolver.js / lambda_passthrough_resolver.js (node --import …/register-loader.mjs resolver_bridge.mjs) showing errorType and extensions.errorCode = RESOURCE_BUSY/INTERNAL_ERROR reach the GraphQL boundary
  • Verified worktree untouched: git status --short clean after all runs
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

Delete the Cognito user before running the shared data cascade in both
admin_delete_user and delete_my_account. A Cognito failure now leaves every
record intact, and a data-phase failure afterwards leaves only inert records
the user can no longer reach (no sign-in, so no post_authentication
re-bootstrap); a re-run of the data phase is safe in both directions.

A data-phase failure after the Cognito commit is logged loudly with an
error_code-tagged field so a partial delete is detectable in post-incident
review rather than inferred from a client error.
…user is still present

Both account-deletion catch blocks claimed 'the Cognito user is still
present' as established fact. Two reachable paths never established it:
delete_my_account re-runs the lookup when the pre-check found no
username, and admin_purge_user_account can raise after
admin_delete_user already landed. The wording now states only what the
code knows - the sweep completed and the Cognito delete did not report
success - while keeping INTERNAL_ERROR, exception chaining, the
error_code field, and exc_info=True.

Tests cover both named paths and assert the message does not claim the
user is present.
…l accounts-row delete

The Cognito-delete catch blocks in delete_my_account and
admin_purge_user_account used a bare except Exception, rewrapping the
typed AppError the Cognito helper already raises and silently
downgrading any future retryable RESOURCE_BUSY from it. Narrow both
catches to (ClientError, BotoCoreError, AppError) and re-raise AppError
untouched so its code and message reach the caller intact.
BotoCoreError stays because a transport failure after the request was
sent is the unknown-state case the catch exists for.

delete_all_user_data still let a failure of its final accounts-row
delete propagate as a raw ClientError to the decorator's generic
message. It now raises an AppError that names the partial state: all
user data was swept, the account record and sign-in survive, retry to
complete; throttling surfaces as retryable RESOURCE_BUSY per #291.

Reported gap, not fixed here: admin_operations._delete_user_from_cognito
still maps a throttled admin_delete_user to INTERNAL_ERROR instead of
RESOURCE_BUSY; with the narrowed catch that AppError at least passes
through intact.
@dmeiser
dmeiser deployed to ephemeral October 3, 2026 17:27 — with GitHub Actions Active
@dmeiser

dmeiser commented Oct 4, 2026

Copy link
Copy Markdown
Owner Author

Closing as a duplicate: this branch's first four commits are merged #673, and its remaining five were superseded by #677, which I merged today.

Verified rather than assumed:

One residual is not in #677 and I am carrying it forward instead of dropping it here: this branch's except (ClientError, BotoCoreError) around the pre-check Cognito lookup. #677 classifies the lookup with except ClientError only, so a transport-level BotoCoreError during the lookup escapes to the lambda_handler decorator as a generic INTERNAL_ERROR with no retry guidance — even though nothing has been deleted yet (the lookup precedes the sweep) and a transport fault is the archetypal retryable case. #677 does cover BotoCoreError on the delete phase via run_deletion_steps' except Exception; only the lookup catch has the gap.

Merging this branch would now regress #677's classification. Closing.

@dmeiser

dmeiser commented Oct 4, 2026

Copy link
Copy Markdown
Owner Author

Closed as duplicate: first 4 commits are merged #673; remaining 5 superseded by merged #677. BotoCoreError lookup residual tracked separately.

@dmeiser dmeiser closed this Oct 4, 2026
dmeiser added a commit that referenced this pull request Oct 4, 2026
…he deletion lookups (#678)

* fix(backend): harden get_caller_id and classify transport faults on the deletion lookups

Carries forward the two non-duplicate pieces of the closed #671 and #676,
neither of which made it into the merged work.

get_caller_id (from #671, whose rest is a duplicate of merged #662):
`event.get("identity") or {}` then `.get("sub")` assumed identity is a
mapping. A present-but-non-mapping identity (a string, a number, a list)
is truthy, so the read raised AttributeError instead of returning None,
and the caller surfaced as the decorator's generic INTERNAL_ERROR rather
than the typed UNAUTHORIZED the handlers raise for an absent caller. Guard
with isinstance against Mapping.

Cognito pre-check lookups (from #676, whose rest is merged #673 and #677):
both `_lookup_cognito_user_for_deletion` and `_find_cognito_user_by_sub`
caught ClientError only, so a BotoCoreError during the lookup escaped to
the decorator with no attributed log line and no retry guidance -- even
though the lookup runs before the sweep and the Cognito delete, so nothing
has been mutated and a retry converges. Widen both to
`(ClientError, BotoCoreError)`.

To classify that without adding a second definition of "transient",
`is_transient_cognito_error` now takes a BaseException and counts a bare
BotoCoreError as retryable; the pre-delete catches just call it. The
post-sweep Cognito delete in `run_deletion_steps` keeps its narrower
`isinstance(error, ClientError)` guard, and the asymmetry is now documented
on both sides: a transport fault *after* the delete request was sent leaves
the Cognito state genuinely unknown, so it must not promise a retry, while
the same fault *before* anything is mutated is plainly retryable. This
keeps #677's tested unknown-state contract intact.

Gates: `pytest tests/unit` 1575 passed / 2 skipped at 100% coverage, xenon,
ruff, and mypy all clean.

* chore(spelling): add 'isinstance' to the project cspell dictionary

The AGENTS.md #291 entry now names the isinstance narrowing that distinguishes
the post-sweep Cognito delete from the pre-delete lookups; the word was not in
the dictionary and failed the frontend job's Spell check step.

This branch was successfully deployed

1 active deployment
ephemeral — 4fd2802c Deployed Oct 3, 2026 by dmeiser via Ephemeral tests for PR #1024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant