Skip to content

test: cover enterprise licensing end to end - #7377

Closed
otavio wants to merge 3 commits into
masterfrom
test/e2e-licensing
Closed

otavio wants to merge 3 commits into
masterfrom
test/e2e-licensing

Conversation

@otavio

@otavio otavio commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

Summary

Covers the [testcontainers] items of shellhub-io/team#243 Domain 21 (Licensing) and runs them in CI. This is the bottom layer of the per-domain e2e stack. The firewall, recording and billing domains that follow reuse its enterprise harness and CI job.

An enterprise stack could only run under the one real license SHELLHUB_LICENSE_FILE named, so expired, limited or feature-less licenses were out of reach. A run now signs its own:

StartRun(IssuingLicenses(issuer))      key in tests/.stack/license-issuer.pem, reused across runs
  server image                         go build -ldflags "$GO_LDFLAGS"
    -X github.com/shellhub-io/cloud/pkg/lic.PublicKey=<issuer public key>
  Up(enterprise stack)
    WithLicense(l) | FullLicense()     signed, written to .stack/licenses/<stack>.dat, mounted
    WithoutLicense()                   no file, empty ADMIN_API_LICENSE_FILE
    WithEveryAddressIn("BR")           mmdbwriter GeoLite2 DBs placing ::/0 in BR
  UseLicense / InstalledLicense / RemoveLicenses / LicenseRows       admin API + licenses table
  WriteLicenseFile / RestartServer / RestartFailingServer            startup loading

The production build leaves GO_LDFLAGS empty. Runs without an issuer, the Playwright suite among them, still read SHELLHUB_LICENSE_FILE.

 qa.yml
   validate (tests - postgres)      go test -skip ^TestEnterprise ./...
+  validate (tests - enterprise)    cloud checkout, branch-matched like ui-e2e.yml
+                                   go test -run ^TestEnterprise ./...   (no license secret)

Issue lines covered

Line Item Test
754 Firewall routes blocked without feature TestEnterpriseLicensing/feature gating/firewall routes answer 402 without the feature
755 Firewall routes allowed with feature TestEnterpriseLicensing/feature gating/firewall routes answer with the feature
756 Session recording routes blocked without feature TestEnterpriseLicensing/feature gating/session recording routes answer 402 without the feature
757 Session recording routes allowed with feature TestEnterpriseLicensing/feature gating/session recording routes answer with the feature
760 Expired license blocks feature-gated routes TestEnterpriseLicensing/feature gating/an expired license answers 402 on every gated route
761 License endpoint itself skips license check TestEnterpriseLicensing/feature gating/the license endpoint answers under an expired license, .../the license endpoint installs a license when none is installed
764 Accept device within license limit TestEnterpriseLicensing/device acceptance/accepting a device within the license limit succeeds
765 Accept device at exact limit blocked TestEnterpriseLicensing/device acceptance/accepting a device with the fleet at the license limit is refused
766 Accept device with unlimited license TestEnterpriseLicensing/device acceptance/accepting a device under an unlimited license succeeds
767 Accept device with no license blocked TestEnterpriseLicensing/device acceptance/accepting a device with no license installed is refused
768 Accept device with expired license blocked TestEnterpriseLicensing/device acceptance/accepting a device under an expired license is refused
769 Accept device with not-yet-started license blocked TestEnterpriseLicensing/device acceptance/accepting a device under a license not yet started is refused
773 SSH connect within license limit TestEnterpriseLicensing/device connection/connecting with the fleet within the license limit succeeds
774 SSH connect over license limit blocked TestEnterpriseLicensing/device connection/connecting with the fleet over the license limit is refused
775 SSH connect with unlimited license TestEnterpriseLicensing/device connection/connecting under an unlimited license succeeds
776 SSH connect with no license blocked TestEnterpriseLicensing/device connection/connecting with no license installed is refused
777 SSH connect with expired license blocked TestEnterpriseLicensing/device connection/connecting under an expired license is refused
778 SSH connect with not-yet-started license allowed TestEnterpriseLicensing/device connection/connecting under a license not yet started succeeds
779 Community edition skips license check already on master: every community SSH test connects with no license and no evaluator, e.g. TestSSHConnectionBehaviour/each cipher the server offers is the one negotiated
785 Allowed region passes TestEnterpriseLicensing/device connection/connecting from a region the license allows succeeds
786 Disallowed region blocked TestEnterpriseLicensing/device connection/connecting from a region the license does not allow is refused
787 Empty allowed_regions means global TestEnterpriseLicensing/device connection/connecting under a license that names no region succeeds from anywhere
790 Valid RSA signature accepted TestEnterpriseLicensing/signature/a license the issuer signed is installed
791 Tampered license body rejected TestEnterpriseLicensing/signature/a license whose body changed after signing is refused
792 Truncated license rejected TestEnterpriseLicensing/signature/a license shorter than its signature is refused
793 Invalid base64 rejected TestEnterpriseLicensing/signature/a license that is not base64 is refused
796 License file loaded on startup TestEnterpriseLicensing/license file/the server installs the license file on startup
797 License file replaces different DB license TestEnterpriseLicensing/license file/a restart with another license file replaces the installed license
798 License file same as DB is no-op TestEnterpriseLicensing/license file/a restart with the same license file installs nothing
799 Empty license path is no-op TestEnterpriseWithoutLicenseFile
800 Invalid license file rejected on startup TestEnterpriseLicensing/license file/an invalid license file stops the server from starting

Not covered here:

  • 758 and 759, billing routes blocked or allowed by the feature. These can't be reached. cloud registers /api/billing on router.Group("/api/billing") without the license middlewares, in internal/billing/routes/routes.go, and has since the monolith conversion in cloud a0681806. The billing branch of LicenseCheckFeature is dead code. Billing routes also exist only in the cloud edition.
  • 770, accept device fails open on store error. A test proved this is a product bug. validateDeviceAcceptance read the license inside UpdateDeviceStatus's transaction, so a failed read aborted the transaction and the accept answered 500. The fix and this case's test are in fix(server): check the license before accepting a device opens its tx #7379, stacked on this PR.

Evidence

  • Before: the Go suite had no enterprise test, and an enterprise stack refused to start without a real license file. qa.yml ran the community stack only.
  • After: both enterprise tests pass locally in the test runner:
$ go test -run '^TestEnterprise' -v .
--- PASS: TestEnterpriseLicensing (59.20s)          30 cases
--- PASS: TestEnterpriseWithoutLicenseFile (15.79s)
ok  github.com/shellhub-io/shellhub/tests  81.265s

$ go test -skip '^TestEnterprise' ./...             the community leg as CI runs it
ok  github.com/shellhub-io/shellhub/tests              1698.166s
ok  github.com/shellhub-io/shellhub/tests/environment  37.559s

golangci-lint v2.14.0 reports 0 issues on tests/ with --max-issues-per-linter=0 --max-same-issues=0, go mod tidy from an empty go.sum reproduces the committed one, and actionlint passes qa.yml.

The review ran three rounds of /code-review:

  • Round 1. Spec: 31 met, 2 struck, 1 held out, and CI partial because a fork PR skipped silently. Standards: 3 doc-contract violations plus judgement calls. The fixes added a fork notice, made Up clean up through one deferred call, made Down remove files even when network removal fails, made License+Unlicensed an error, stopped WriteLicenseFile from rewriting a real license file, and restructured the refused-connection table. Rejected, with reasons: the SQL row-count helpers, because no API exposes stored rows; the in-package key comparison; the .stack literal in main_test.go; and the early go.mod check, which the enterprise leg needs.
  • Round 2. Spec clean. Standards: a partial license file could leak on a failed write, and the spelling mixed licence and license. Both fixed.
  • Round 3. Clean.

Merge Danger

Door: two-way

The only production-facing change is the GO_LDFLAGS build argument in server/Dockerfile, which is empty unless a build passes it.

Blast Radius: CI

The new job lengthens QA by one enterprise build and two stacks. It depends on the CLOUD_DISPATCH_APP_* secrets, the same ones ui-e2e.yml uses. The community leg now skips ^TestEnterprise.

Part of shellhub-io/team#243.

@otavio otavio changed the title test/e2e licensing test: cover enterprise licensing end to end Oct 7, 2026
@otavio
otavio added this pull request to stack #7380 October 7, 2026 20:00
@otavio
otavio marked this pull request as ready for review October 7, 2026 20:02
@otavio
otavio requested review from a team as code owners October 7, 2026 20:02
@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Claude finished @otavio's task in 8m 21s —— View job


Code Review Complete

Reviewed 26 files and posted 2 inline issues:

  • Standards: 1 violation and 1 judgement call.
  • Spec: did not run.
  • Correctness: no findings.

Standards

The worst violation is in the doc comment on Up (tests/environment/stack.go:85-91). Its error contract leaves out the Unlicensed cases. A community stack built with WithoutLicense() also fails, and an unlicensed enterprise stack never reads SHELLHUB_LICENSE_FILE.

Spec

The Spec axis did not run. The PR cites shellhub-io/team#243, which is in a repo this run cannot read, and it names no issue in this repo.

Additional notes

  • Correctness found nothing. I checked the fork gating in qa.yml, how HEAD_REF reaches the script (via env, quoted), and the token scope. I checked that the -skip ^TestEnterprise leg excludes both enterprise tests and that an empty GO_LDFLAGS is a no-op in the production server/Dockerfile. I also confirmed the -X target is a real string var in cloud's pkg/lic, and checked cleanup ordering, context.WithoutCancel in cleanups, and the device-limit boundaries against cloud's evaluator.
  • One case I could not confirm. The acceptance cases assume cloud's CountRegisteredDevices counts only accepted devices, but I could not find its implementation.
  • Two judgement calls dropped. Both repeat round-1 decisions the PR description already defends: the private-key comparison in TestLoadLicenseIssuer, and having License and Unlicensed together return an error.

To request another review round, comment /review.

Comment thread tests/environment/stack.go Resolved
Comment thread tests/environment/configurator.go Resolved
otavio added 3 commits October 8, 2026 11:43
An enterprise stack verified licenses against the production key compiled into
cloud's pkg/lic, so a test could only run under the one real license a developer
or CI secret supplied. Expired, limited, region-bound or feature-less licenses
were out of reach, and so was most of the licensing domain.

A run started with IssuingLicenses now owns a test RSA key. The harness builds
the server image with -X github.com/shellhub-io/cloud/pkg/lic.PublicKey set to
its public half, through a new GO_LDFLAGS build argument that the production
build leaves empty. The key lives in tests/.stack/license-issuer.pem and runs
reuse it. A fresh key per run would change the build argument and recompile the
enterprise server every time. GO_LDFLAGS is declared right before the build
step so the module download layers stay cached, and the harness sets it only
for enterprise and cloud builds.

Each enterprise stack then writes its license to a file of its own and mounts
it. The enterprise test overlay maps an unset SHELLHUB_LICENSE_FILE to
/dev/null and to an empty ADMIN_API_LICENSE_FILE, which is how WithoutLicense
starts a server that loads no license at all. Runs without an issuer, the
Playwright suite among them, keep reading SHELLHUB_LICENSE_FILE as before.

WithEveryAddressIn mounts GeoLite2 country and city databases, written with
mmdbwriter, that place ::/0 in one country. They include reserved networks and
turn IPv4 aliasing off because a test connects from a private address, which a
stock database leaves unlocated. The locator needs both files to start, and
the coordinates do not matter, so they stay at zero. MAXMIND_MIRROR points at
an unreachable host only to make the server enable the locator. The fetcher
skips every database already present, so it never contacts the host.

RestartFailingServer waits for the container's on-failure policy to give up
rather than for the API, since a server refusing its license file exits.
UseLicense, RestartServer and DeleteDevice outlive the test's context, because
Go cancels that context before cleanups run and these helpers restore state in
a t.Cleanup. A paid edition now fails early, naming CloudDir's go.mod, when the
cloud source is missing, instead of deep inside the image build.
TestEnterpriseLicensing runs one enterprise stack under licenses the run
signs, and swaps the installed license per case through the admin API.
TestEnterpriseWithoutLicenseFile starts a second stack with no license file.

The tests pin two behaviours that read like accidents and are not.

- The server logs a refusal for a region the license does not allow as a
  firewall block. Only the firewall evaluator checks allowed_regions, so the
  region cases run in a legacy-mode namespace with the firewall feature
  licensed.
- A license that has not started refuses device acceptance but lets a
  connection through. CanConnectDevice skips IsStarted on purpose, so that a
  future start date cannot lock an instance out of the devices it already has.

Each acceptance case deletes the pending device it enrolled when it ends,
because startAcceptedAgent expects exactly one pending device. Their MAC
addresses use 02:00:00:00:21:xx, a locally administered prefix numbered after
the domain, so they never merge with another case's device.

Covers shellhub-io/team#243 Domain 21 Feature Gating, Device Acceptance,
Device Connection, License Regions, License Signature and License File
Loading, except the billing gating and fail-open cases.
The testcontainers suite ran on the community stack only, so the licensing
tests, and the firewall and recording tests that follow them, never ran in
CI. A new job checks out the cloud source the way ui-e2e.yml does. It matches
the PR's branch name, falls back to master, and runs ^TestEnterprise. It needs
no license secret, because the run signs its own licenses.

A pull request from a fork cannot mint the cloud token, so the job skips its
steps and leaves a notice instead of failing every outside contribution. The
community leg skips ^TestEnterprise so a run without the cloud source does
not try to build an enterprise stack.
@otavio
otavio force-pushed the test/e2e-licensing branch from a9c1786 to dcb8514 Compare October 8, 2026 14:45
@otavio

otavio commented Oct 8, 2026

Copy link
Copy Markdown
Member Author

/review

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Code Review Complete

The automated review ran but did not post an updated summary — this usually means no new issues were found since the previous review. If you've pushed changes and want a fresh pass, comment /review.

View job

@otavio
otavio removed this pull request from stack #7380 October 8, 2026 15:58
@otavio

otavio commented Oct 8, 2026

Copy link
Copy Markdown
Member Author

Consolidated into #7390, which carries this PR's commits so the whole stack reviews and merges together.

@otavio otavio closed this Oct 8, 2026
@otavio
otavio deleted the test/e2e-licensing branch October 8, 2026 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant