Repository navigation
feat(cli): add Windows support and automated CI test coverage #2350
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
568e170
7fc9dec
3753b26
7221504
2d5cf87
a58522c
ed5c88a
0c16b4d
c3c213b
5c9b266
6e8753d
64df2b8
def5b00
7e64a68
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| * text=auto eol=lf | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,105 @@ | ||
| name: Test | ||
|
|
||
| on: | ||
| pull_request: | ||
| push: | ||
| branches: [main] | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| # Lint, unit and integration tests run on Cloud Build (cloudbuild-test.yaml); this job only | ||
| # builds the packages and the smoke test bundle that the Windows job needs. | ||
| build-linux: | ||
| name: Build packages and smoke test bundle (Linux) | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 60 | ||
| permissions: | ||
| contents: read | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 | ||
|
|
||
| - name: Setup Bazel | ||
| uses: bazelbuild/setup-bazelisk@b39c379c82683a5f25d34f0d062761f62693e0b2 # v3.0.0 | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not blocking here, but lets keep in mind that no Bazel cache is configured, so every PR does a cold build (60 min timeout). Let's consider adding a disk/repository cache (e.g.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed, new issue will be created. For now the Linux job takes about a minute, so it isn't urgent. |
||
|
|
||
| # MODULE.bazel sets `use_home_npmrc = True`, so aspect_rules_js requires ~/.npmrc to exist. | ||
| # An empty file keeps the default public npm registry (Cloud Build writes its own). | ||
| - name: Create an empty ~/.npmrc | ||
| run: touch ~/.npmrc | ||
|
|
||
| - name: Build package tarballs and the Windows smoke test bundle | ||
| run: | | ||
| bazel build \ | ||
| //packages/@dataform/cli:package_tar \ | ||
| //packages/@dataform/core:package_tar \ | ||
| //cli/tests/windows:smoke_test_bundle | ||
|
|
||
| - name: Upload CLI tarball | ||
| uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1 | ||
| with: | ||
| name: dataform-cli-tarball | ||
| path: bazel-bin/packages/@dataform/cli/package.tar.gz | ||
|
|
||
| - name: Upload Core tarball | ||
| uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1 | ||
| with: | ||
| name: dataform-core-tarball | ||
| path: bazel-bin/packages/@dataform/core/package.tar.gz | ||
|
|
||
| - name: Upload Windows smoke test bundle | ||
| uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1 | ||
| with: | ||
| name: windows-smoke-test | ||
| path: bazel-bin/cli/tests/windows/smoke_test_bundle.js | ||
|
|
||
| test-windows: | ||
|
rafal-hawrylak marked this conversation as resolved.
|
||
| name: CLI smoke tests (Windows, Node ${{ matrix.node-version }}) | ||
| needs: build-linux | ||
| runs-on: windows-latest | ||
| timeout-minutes: 30 | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| # Oldest supported Node.js LTS and the current one (also used by the Bazel toolchain). | ||
| node-version: [22, 24] | ||
| permissions: | ||
| contents: read | ||
| steps: | ||
| - name: Setup Node.js | ||
| uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 | ||
| with: | ||
| node-version: ${{ matrix.node-version }} | ||
|
|
||
| - name: Download CLI tarball | ||
| uses: actions/download-artifact@cc203385981b70ca67e1cc392babf9cc229d5806 # v4.1.9 | ||
| with: | ||
| name: dataform-cli-tarball | ||
| path: dist/cli | ||
|
|
||
| - name: Download Core tarball | ||
| uses: actions/download-artifact@cc203385981b70ca67e1cc392babf9cc229d5806 # v4.1.9 | ||
| with: | ||
| name: dataform-core-tarball | ||
| path: dist/core | ||
|
|
||
| - name: Download Windows smoke test bundle | ||
| uses: actions/download-artifact@cc203385981b70ca67e1cc392babf9cc229d5806 # v4.1.9 | ||
| with: | ||
| name: windows-smoke-test | ||
| path: dist/smoke | ||
|
|
||
| - name: Install @dataform/cli globally from tarball | ||
| run: npm install -g dist/cli/package.tar.gz | ||
|
|
||
| - name: Run Windows smoke tests in PowerShell | ||
| shell: powershell | ||
| run: | | ||
| node dist/smoke/smoke_test_bundle.js --shell powershell --core-tarball dist/core/package.tar.gz | ||
|
|
||
| - name: Run Windows smoke tests in cmd.exe | ||
| shell: cmd | ||
| run: | | ||
| node dist\smoke\smoke_test_bundle.js --shell cmd --core-tarball dist\core\package.tar.gz | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -19,7 +19,7 @@ export abstract class BaseWorker<TResponse, TMessage = any> { | |
| ): Promise<TResponse> { | ||
| const forkScript = this.resolveScript(); | ||
| const child = fork(forkScript, [], { | ||
| stdio: [0, 1, 2, "ipc", "pipe"], | ||
| stdio: [0, 1, 2, "ipc"], | ||
|
apilaskowski marked this conversation as resolved.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Removing the extra
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed: nothing in |
||
| }); | ||
|
|
||
| return new Promise((resolve, reject) => { | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -96,7 +96,7 @@ export async function compile( | |
| compiledGraph = dataform.CompiledGraph.create(decodedResult.compile.compiledGraph); | ||
|
|
||
| if (workflowSettingsDataformCoreVersion) { | ||
| fs.rmSync(temporaryProjectPath, { recursive: true }); | ||
| fs.rmSync(temporaryProjectPath, { recursive: true, force: true, maxRetries: 3 }); | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Nice to have]
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| } | ||
|
|
||
| return compiledGraph; | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| load("//packages:index.bzl", "pkg_bundle") | ||
| load("//tools:ts_library.bzl", "ts_library") | ||
|
|
||
| package(default_visibility = ["//visibility:public"]) | ||
|
|
||
| ts_library( | ||
| name = "smoke_test_lib", | ||
| srcs = ["smoke_test.ts"], | ||
| deps = [ | ||
| "//:node_modules/@types/node", | ||
| ], | ||
| ) | ||
|
|
||
| # Self-contained CommonJS bundle (Node built-ins only) that the Windows CI runner executes with | ||
| # plain `node`; it is built on the Linux job and shipped as a workflow artifact. | ||
| pkg_bundle( | ||
| name = "smoke_test_bundle", | ||
| allow_node_builtins = True, | ||
| entry_point = "smoke_test.js", | ||
| externals = [], | ||
| deps = [":smoke_test_lib"], | ||
| ) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[Nice to have] Repo-wide
* text=auto eol=lfwill likely trigger a one-time renormalization diff for existing files, and may need exceptions for any files that must keep CRLF. Maybe worth a note in the PR or a follow-upgit add --renormalize ..There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Only one tracked file had CRLF (
vscode/workflow_settings_yaml.schema.json), and I renormalized it in this PR, so there's no follow-up diff. No files need to keep CRLF (there are no.bat/.cmd/.ps1files in the repo).