Skip to content

feat(hosted): serve the rollback witness from the deployment-test control plane - #434

Merged
Lokesh7025 merged 2 commits into
RCfrom
remote/aws-witness
Sep 25, 2026
Merged

Lokesh7025 merged 2 commits into
RCfrom
remote/aws-witness

Conversation

@Lokesh7025

@Lokesh7025 Lokesh7025 commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The hosted-path deployment test served pairing rendezvous and relay traffic but returned 404 on /v1/e2ee/witness. As a result, no real E2EE endpoint could register or advance against AWS, and a phone could not pair through it. This PR serves the rollback witness from the deployment-test control plane.

  • Assembly: services/aws-control-plane/src/witness-deployment-test.ts creates a WitnessGateway over three in-process WitnessReplicas. It is enabled only when AXL_TEST_WITNESS_KEYS is set; without it the witness path stays unrouted, as it is today.
  • Keys: each replica signs with its own Ed25519 key from a new axl/hosted-path/witness-keys secret. deploy.sh generates the secret once, and Terraform passes it to the control-plane task.
  • Admission: admission, registration, and binding lookup accept only the single deployment-test account, matched against the request lineage.
  • Bootstrap: empty replicas accept exactly one registration. The first signed read of that lineage then drives the ordinary recovery exchange (each replica's recovery head, then recovery from the other two receipts). After that the replicas accept further lineages.
  • Trust export: witness-trust.sh writes the public trust in the canonical ReplicaTrustSet configuration form for AXL_E2EE_DEPLOYMENT_TEST_TRUST_FILE. Only public keys leave the secret.
  • Health: /healthz reports whether the witness is served, and the smoke test requires it.

Known limitation

Replica state lives only in process memory. A replica can recover persisted state only from fresh endpoint-signed reads covering every lineage, and nothing orchestrates that across a restart yet. A restarted or redeployed control plane therefore starts every replica empty. Endpoints registered before the restart then fail closed and must pair again. The stack README documents this.

This is not the production witness topology: one process, one account, one failure domain. Production gates are unchanged.

Test plan

  • pnpm --filter @axl/aws-control-plane typecheck and build
  • terraform fmt and terraform validate; bash -n on the stack scripts
  • Local cross-check against the native Node test artifact:
    • daemon registration certified by all three replicas using the TS-encoded trust (decoded by Rust ReplicaTrustSet::decode_config)
    • a device registering before the first read is refused with witness_unavailable
    • the first daemon read moves the replicas from bootstrap to ready
    • a device registers after that
    • a foreign account is rejected with forbidden
  • Deploy to the deployment-test stack and run smoke-test.sh

Also fixed: UUIDv7 identities

E2EE pairing accepts only UUIDv7 installation, crypto session, and device identities. deploy.sh generated the stack's installation and device IDs with uuid4, so a real endpoint using the stack's installation failed with identity_mismatch. It now generates both as UUIDv7. An existing secret keeps its values, so an older stack needs its runtime secret rotated to pick this up.

Deployed

  • Deployed to the axl-deploy deployment-test stack (ap-south-2) from this branch.
  • The runtime secret was rotated to UUIDv7 identities, and both services were redeployed.
  • smoke-test.sh passes; /healthz reports "witness":true.
  • Live check: a native daemon endpoint with the trust exported by witness-trust.sh registered through the AWS witness (certified by all three replicas), then read and reconciled ready.

@github-actions

Copy link
Copy Markdown

⚠️ Deprecation Warning: The deny-licenses option is deprecated for possible removal in the next major release. For more information, see issue 997.

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@coderabbitai

coderabbitai Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: cde6aa74-962c-49bb-91c1-c39cb25051c6

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…trol plane

The hosted-path deployment test routes pairing and relay traffic, but
/v1/e2ee/witness returned 404, so no real E2EE endpoint could register
or advance against it and a phone could not pair through AWS.

The deployment-test control plane now assembles a WitnessGateway over
three in-process WitnessReplicas when AXL_TEST_WITNESS_KEYS is set:

- Each replica signs with its own Ed25519 key from the new
  axl/hosted-path/witness-keys secret, which deploy.sh generates once.
  Admission, registration, and binding lookup accept only the single
  deployment-test account, matched against the request lineage.
- Replica state is process memory. A replica can recover persisted
  state only from fresh endpoint-signed reads covering every lineage, so
  a restarted control plane starts empty instead and earlier endpoints
  fail closed until they pair again. The README says so.
- Empty replicas bootstrap with one registration. The first signed read
  of that lineage drives the ordinary recovery exchange (each replica's
  recovery head, then recovery from the other two receipts), after which
  the replicas are ready for further lineages.
- witness-trust.sh writes the public trust in the canonical
  ReplicaTrustSet configuration form for deployment-test client builds.
  Only public keys leave the secret.
- /healthz reports whether the witness is served and the smoke test
  requires it.

Checked locally against the native daemon and device endpoints built
from the encoded trust: registration, bootstrap to ready on the first
read, a second lineage after ready, and rejection of a foreign account.

This is not the production witness topology: one process, one account,
one failure domain.

Signed-off-by: Lokesh <lokeshselvam7025@gmail.com>
E2EE pairing accepts only UUIDv7 installation, crypto session, and
device identities, but deploy.sh generated the deployment-test
installation and device IDs with uuid4. A real endpoint registering with
the stack's installation failed with identity_mismatch, so the stack
could never pair an E2EE device. Generate both as UUIDv7; the account
and relay instance IDs are not E2EE identities and stay uuid4.

An existing secret keeps its values: deploy.sh only creates a missing
secret, so rotate an older stack's secret to pick this up.

Signed-off-by: Lokesh <lokeshselvam7025@gmail.com>
@Lokesh7025
Lokesh7025 merged commit 7804d92 into RC Sep 25, 2026
34 checks passed
@Lokesh7025
Lokesh7025 deleted the remote/aws-witness branch September 25, 2026 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant