Add black box e2e#28
Merged
Merged
Conversation
Runs the production stack and drives the real web UI end to end against the physical ESP32-C3 on the Pi test stand: - web/src/tests/teststand: TS Playwright suite that seeds a session (the one allowed DB shortcut), creates a device + MQTT token via the UI, injects that config onto the device over mpremote, asserts the Online badge, clicks Trigger Success and verifies the success pin via gpioget. - blackbox-e2e/docker-compose.teststand.yml: overlay that runs the real OAuth path (TEST_MODE off) since auth is seeded directly. - .github/workflows/teststand-e2e.yml: runs on the self-hosted [test-stand] runner, reusing the proven start-ap.sh (AP) and run_tests.sh (device flash). The web half (auth seed → device online → trigger publish) was validated locally against the production build; the flash + GPIO steps require the stand. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The script already sudo's its own nmcli/rfkill (covered by NOPASSWD); the outer 'sudo bash' required a password for bash and failed. Drop it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The hardware test provisions firmware via mpremote, so the worker service is unnecessary; bringing up only 'app' avoids pulling its multi-GB image. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…config
- GPIO success level is HIGH (relay ties contact to 3.3V); idle reads LOW
('inactive' on the stand), so active level is 1, not 0.
- Inject QUICK_START + MUTE_BUZZER so the device skips the boot animation that
trips the MicroPython task watchdog on boot (observed WDT reset before WiFi).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…llow Web Serial access
Three independent bugs kept the device from ever showing Online:
1. start-ap.sh brought the AP up as WPA1/TKIP (NetworkManager default
without an explicit proto). ESP-IDF's default authmode threshold is
WPA2, so the ESP32-C3 refused to join (wlan.status() = 211
NO_AP_FOUND_IN_AUTHMODE_THRESHOLD). Force RSN/CCMP — verified on the
stand: join now takes ~2s.
2. The worker bakes its MQTT_URL env into the firmware, and the blackbox
stack gave it the docker-internal mqtt://admin:admin@mqtt:1883. The
device can't resolve `mqtt`, and the firmware URL parser crashed on
the user:pass@ form (int("admin@mqtt:1883")). Override MQTT_URL for
the worker with the AP gateway IP (mirrors PUBLIC_MQTT_URL in
docker-stack.yml) and make the parser strip userinfo.
3. The test asserted getByText('online', { exact: true }) but the badge
renders 'Online' — exact match is case-sensitive, so the assertion
could never pass.
CI feedback loop: build web/worker images in parallel via a matrix and
add a ghcr registry layer cache so the ~8 min ESP-IDF image build only
reruns when the Dockerfile changes; drop xvfb (suite is headless);
diagnostics now dump the device health.log, AP leases and worker logs
instead of the stale params.json.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…kers docker image prune only removes dangling images; the uniquely-tagged test-<sha> worker images (~6GB each) accumulated until the Pi hit ENOSPC mid-job. Remove test images from other commits, keep base images cached. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pre-pull deletion of old test images also deleted the ~6GB of ESP-IDF layers the new worker image shares with them, forcing a full re-pull (33 min on the Pi). Clean up after the run instead, keeping the current sha's images so the next run pulls only changed layers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The worker picked up the build job but the test only waited 120s for "Firmware ready!" — an arm64 MicroPython build on the Pi takes longer. Wait up to the worker's own BUILD_TIMEOUT (600s) and size the overall test timeout accordingly. Also read dnsmasq leases with sudo in the failure diagnostics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Surface which build is actually running on the device and what WiFi SSID
it has baked in, to root-cause the teststand e2e "No WiFi configured"
failure (build-vs-flash).
- config.py: GIT_COMMIT template placeholder + get_git_commit(), worker
substitutes {{GIT_COMMIT}} from its COMMIT_SHA env (same mechanism as
{{WIFI_SSID}}); overridable via config_dev.py for local dev.
- health_log: git_commit in boot log + health snapshot.
- wifi_manager: log raw_ssid on the "No WiFi configured" path.
- Pass COMMIT_SHA to the worker in docker-stack.yml, the teststand
compose overlay, and the teststand-e2e .env.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- playwright.teststand: video/trace 'on' (was retain-on-failure) so the recording is kept for passing runs too. - teststand-e2e: run the diagnostics step (docker logs, gpioget, mpremote, health.log) on always() not just failure(), and drop the --tail caps for full container logs. - Upload raw test-results/ (.webm + trace zips) alongside the HTML report. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- The web app now populates commitSha into the worker job using env.COMMIT_SHA. - The worker uses the commitSha from the mongo job payload instead of the local env. - Added erase_flash before write_flash in blackbox tests to ensure clean state. - Purge mpy-cross caches in build.sh to prevent using old frozen module artifacts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.