Skip to content

Run Polars TPC benchmarks in CI - #22848

Open
Matt711 wants to merge 74 commits into
NVIDIA:mainfrom
Matt711:ci/polars/run-tpc-bench
Open

Run Polars TPC benchmarks in CI#22848
Matt711 wants to merge 74 commits into
NVIDIA:mainfrom
Matt711:ci/polars/run-tpc-bench

Conversation

@Matt711

@Matt711 Matt711 commented Jun 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Description

This PR adds TPC-H and TPC-DS validation tests to the cudf-polars CI. The tests run all 22 TPC-H queries and all 99 TPC-DS queries against the SPMD streaming engine and validate the results against DuckDB at SF=1. Each query is a separate pytest node and each iteration is reported as a subtest so CI output shows exactly which query and which iteration failed.

Changes

  • New CI job: tpc-tests-cudf-polars takes around 10 minutes to finish
  • TPC-DS query changes mainly around validating with decimal data (xref [Story] Decimal support in cudf-polars/libcudf for PDS-DS #20587). I also checked that we still pass validation with floats.
  • Benchmark runner refactoring mainly around making it easier to build pytest group options from the benchmark runner options. So all benchmark runner options (--path, --io-mode, --max-rows-per-partition, etc.) are available as pytest CLI arguments.
  • --io-mode lukewarm has been corrected to match the Presto/Velox convention: the page cache is dropped once before the first query so the run starts from a known cold baseline, then the cache is left to warm naturally.
  • Fix for None boundary comparison in asserts.py: assert_tpch_result_equal emitted a UserWarning when the sort tie boundary value was None, because it called pl.col(col).lt(None). The fix treats a None boundary as meaning all rows are ties by appending pl.lit(False) instead.

Contributes to #23380
Closes #22868

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@Matt711 Matt711 added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Jun 11, 2026
@copy-pr-bot

copy-pr-bot Bot commented Jun 11, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added Python Affects Python cuDF API. cudf-polars Issues specific to cudf-polars labels Jun 11, 2026
@Matt711

Matt711 commented Jun 11, 2026

Copy link
Copy Markdown
Member Author

/ok to test c94a0dc

@GPUtester GPUtester moved this to In Progress in cuDF Python Jun 11, 2026
mroeschke
mroeschke reviewed Jun 11, 2026
View reviewed changes
Comment thread ci/run_cudf_polars_tpc.sh Outdated
Comment thread ci/run_cudf_polars_tpc.sh Outdated
@Matt711 Matt711 linked an issue Jun 12, 2026 that may be closed by this pull request
Add TPC benchmarks to CI #22868
Open
@davidwendt

davidwendt commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Could you consider adding this to the nightly tests instead of pr.yaml?

@Matt711

Matt711 commented Jun 25, 2026

Copy link
Copy Markdown
Member Author

Could you consider adding this to the nightly tests instead of pr.yaml?

Yes I think that is reasonable, even if just until we get these CI sorted out (I believe we will).

@Matt711

Matt711 commented Jun 25, 2026

Copy link
Copy Markdown
Member Author

/ok to test c72714b

@Matt711

Matt711 commented Jun 25, 2026

Copy link
Copy Markdown
Member Author

/ok to test 4dc4c16

@Matt711 Matt711 mentioned this pull request Jun 29, 2026
Merged
3 tasks
@Matt711

Matt711 commented Jun 29, 2026

Copy link
Copy Markdown
Member Author

/ok to test 494fa40

rapids-bot Bot pushed a commit that referenced this pull request Jul 2, 2026
@Matt711
Add instructions on running PDS-H (#23025)
226a17a 
Documentation for humans and agents to refer to for reproducing our benchmarks. 

I punted on PDS-DS primarily because the data generation step is not as simple as PDS-H. I'll add instructions in a follow-up. First I want to close out #20587, so data generation really is just using duckdb's wrapper of `dsdgen`, rather than having to convert to floats.

#20587 should mostly be done, since the work we did for PDS-H with decimals should cover most of the cases we'll encounter for PDS-DS. FYI: I'm testing PDS-DS with decimals in #22848.

Contributes to #17640

Authors:
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - Matthew Roeschke (https://github.com/mroeschke)

URL: #23025
@Matt711

Matt711 commented Jul 2, 2026

Copy link
Copy Markdown
Member Author

/ok to test d9d00b4

@Matt711

Matt711 commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

/ok to test 3d86c03

@Matt711 Matt711 closed this Sep 10, 2026
@github-project-automation github-project-automation Bot moved this from In Progress to Done in cuDF Python Sep 10, 2026
@Matt711 Matt711 reopened this Sep 10, 2026
@github-project-automation github-project-automation Bot moved this from Done to In Progress in cuDF Python Sep 10, 2026
@Matt711

Matt711 commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

/ok to test 3d86c03

jameslamb
jameslamb previously approved these changes Sep 10, 2026
View reviewed changes

@jameslamb jameslamb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving for CI/packaging codeowners so you have what you need when this is ready, but please do address the suggestions in my review.

Comment thread ci/run_cudf_polars_tpc.sh Outdated
Comment thread dependencies.yaml Outdated
Comment thread cpp/examples/versions.cmake
Comment thread ci/run_cudf_polars_pytests.sh
Comment thread .github/workflows/pr.yaml Outdated
Comment thread .github/workflows/pr.yaml

@jameslamb jameslamb Sep 10, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only want to run these tests nightly

Just reminding you that right now this PR doesn't have any changes in build.yaml. That's where you'll need updates if you want this to run nightly.

Matt711 and others added 3 commits September 10, 2026 09:28
@Matt711 @jameslamb
Update ci/run_cudf_polars_tpc.sh
5c684f9 
Co-authored-by: James Lamb <jaylamb20@gmail.com>
@Matt711 @jameslamb
Update .github/workflows/pr.yaml
3306d18 
Co-authored-by: James Lamb <jaylamb20@gmail.com>
@Matt711 @jameslamb
Update dependencies.yaml
bb32435 
Co-authored-by: James Lamb <jaylamb20@gmail.com>
@Matt711
Matt711 marked this pull request as draft September 10, 2026 17:08
@Matt711

Matt711 commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

We discussed this a little offline and decided to retarget to next release. Not urgent, but we definitely want to get it in early in 26.12 (which we can thanks to the reviewers :)).

@Matt711
Matt711 dismissed stale reviews from jameslamb and mroeschke via bb32435 September 16, 2026 04:20
@Matt711
Matt711 changed the base branch from release/26.10 to main September 29, 2026 15:41
@Matt711
Matt711 force-pushed the ci/polars/run-tpc-bench branch from 893c358 to f81258b Compare October 7, 2026 15:39
@Matt711

Matt711 commented Oct 24, 2026

Copy link
Copy Markdown
Member Author

/ok to test 6eed9b9

@Matt711

Matt711 commented Oct 25, 2026

Copy link
Copy Markdown
Member Author

/ok to test 7178d2f

@Matt711
Matt711 marked this pull request as ready for review October 25, 2026 17:50
@Matt711
Matt711 removed the request for review from a team October 25, 2026 17:52
@davidwendt davidwendt added the DO NOT MERGE Hold off on merging; see PR for details label Oct 25, 2026
@davidwendt

davidwendt commented Oct 25, 2026

Copy link
Copy Markdown
Contributor

Looks like you forgot to address this comment: https://github.com/NVIDIA/cudf/pull/22848/changes#r3856198100

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@coderabbitai coderabbitai[bot] coderabbitai[bot] left review comments
@jameslamb jameslamb jameslamb left review comments
@mroeschke mroeschke mroeschke left review comments
@KyleFromNVIDIA KyleFromNVIDIA Awaiting requested review from KyleFromNVIDIA KyleFromNVIDIA was automatically assigned from rapidsai/packaging-codeowners
@madsbk madsbk Awaiting requested review from madsbk
@wence- wence- Awaiting requested review from wence-

At least 1 approving review is required to merge this pull request.

Assignees

@Matt711 Matt711

Labels

CMake CMake build issue cudf-polars Issues specific to cudf-polars DO NOT MERGE Hold off on merging; see PR for details improvement Improvement / enhancement to an existing function Java Affects Java cuDF API. libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change Python Affects Python cuDF API.

Projects

cuDF Python
Status: In Progress

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add TPC benchmarks to CI

9 participants

@Matt711 @davidwendt @vyasr @madsbk @wence- @jameslamb @mroeschke @GPUtester @josephine-wolf-oberholtzer