Skip to content

feat(cloud): add TestingBot real device cloud provider - #283

Open
jochen-testingbot wants to merge 1 commit into
AlphaWaveSystems:mainfrom
jochen-testingbot:feat/testingbot-cloud-provider
Open

jochen-testingbot wants to merge 1 commit into
AlphaWaveSystems:mainfrom
jochen-testingbot:feat/testingbot-cloud-provider

Conversation

@jochen-testingbot

Copy link
Copy Markdown

What

Adds TestingBot as a sixth cloud device farm, alongside BrowserStack, Sauce Labs, LambdaTest, AWS Device Farm and Firebase Test Lab.

Disclosure: I work at TestingBot. This is a vendor integration for our own platform, written against the public API — happy to adjust anything that doesn't fit how you'd like third-party providers handled here.

Why

TestingBot is a real device cloud in the same category as the three farms already supported, and CloudProvider is a clean extension point, so this slots in without disturbing existing providers.

How

internal/cloud/testingbot.go implements the existing CloudProvider interface. The only change to shared code is one case in NewProvider — nothing on the cloud path in internal/cli/test.go moves.

Method Endpoint
UploadApp POST /v1/storage (multipart file) → tb://<appkey> for appium:app
ListDevices GET /v1/devices → name / platform_name / version
StartSession / StopSession Appium W3C hub at hub.testingbot.com, credentials in tb:options
GetSessionArtifacts GET /v1/tests/<id>/assets → video + screenshots

Endpoints and payload shapes were taken from the TestingBot OpenAPI spec (api.testingbot.com/v1/openapi.json) and the Appium support docs, not from memory.

Two decisions worth your review:

  • Credential naming. Resolves key/secret (TestingBot's own naming, matching TESTINGBOT_KEY/TESTINGBOT_SECRET) and falls back to the generic username/access_key. That keeps --cloud-key/--cloud-secret and probe.yaml working without editing the shared credential map in test.go, which would have touched all five existing providers. Happy to flip to generic-only if you'd rather keep one convention.
  • Relay mode only. ForwardPort passes through when RelayURL is set and otherwise returns an error pointing at --relay, exactly as Sauce Labs and LambdaTest do. Direct forwarding would need the TestingBot Tunnel binary; that can be a follow-up if it's wanted.

Artifact collection polls up to 30s because TestingBot processes assets asynchronously — same shape as the existing BrowserStack video poll.

Tests

internal/cloud/testingbot_test.go covers credential resolution (both naming schemes and the failure cases), upload, device listing, capability construction for Android and iOS, the WebDriver error path, relay/direct ForwardPort, artifact polling, and session teardown.

To test against httptest I made the base and hub URLs struct fields defaulting to the package constants. The other providers hardcode consts and currently have no tests; I kept that change inside testingbot.go rather than refactoring them.

  • make test — all 17 packages pass
  • go build ./... and go vet clean; gofmt clean on the added files (firebase.go and lambdatest.go are gofmt-dirty on main already; left alone to keep this focused)

Not exercised against a live TestingBot account in this branch: request/response shapes are spec-verified, not wire-verified.

Also updated

README provider table, probe.yaml.example, the --cloud-provider flag help, the cloud.provider config comment, and the VS Code extension (setting enum, positional enumDescriptions, run-profile dropdown, extension README setup section).

No CHANGELOG entry — happy to add one under [Unreleased] in whatever wording you prefer if that's part of your flow.

Adds TestingBot as a sixth cloud device farm alongside BrowserStack,
Sauce Labs, LambdaTest, AWS Device Farm and Firebase Test Lab. It
implements the existing CloudProvider interface, so nothing on the
shared cloud path in internal/cli/test.go changes.

- UploadApp: POST /v1/storage (multipart "file"), returns the tb:// URL
  used as the appium:app capability.
- ListDevices: GET /v1/devices, mapping platform_name/version onto the
  common Device shape.
- StartSession/StopSession: Appium W3C hub at hub.testingbot.com, with
  credentials passed in tb:options as TestingBot documents.
- GetSessionArtifacts: GET /v1/tests/<id>/assets for video and
  screenshots, polled up to 30s since assets are processed
  asynchronously.

Credentials resolve from "key"/"secret" (TestingBot's own naming),
falling back to the generic "username"/"access_key" so --cloud-key and
--cloud-secret keep working without touching the CLI credential map.

Like Sauce Labs and LambdaTest, the provider runs in relay mode; direct
port forwarding would require the TestingBot Tunnel binary and returns
an error pointing at --relay.

Endpoints and payloads were taken from the TestingBot OpenAPI spec
(api.testingbot.com/v1/openapi.json) and the Appium support docs.

Signed-off-by: Jochen Delabie <jochen@testingbot.com>
@jochen-testingbot
jochen-testingbot requested a review from a team as a code owner September 22, 2026 12:27
@github-actions

Copy link
Copy Markdown


Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.


I have read the CLA Document and I hereby sign the CLA


You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.

@jochen-testingbot

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant