Skip to content

Add more options to the BulkDumping workload to support restoring a full database in a test. - #14086

Merged
brownleej merged 4 commits into
apple:mainfrom
brownleej:extend-bulkdumping-workload
Sep 23, 2026
Merged

brownleej merged 4 commits into
apple:mainfrom
brownleej:extend-bulkdumping-workload

Conversation

@brownleej

Copy link
Copy Markdown
Contributor

This PR adds more options to the BulkDumping workload, in order to support dumping and loading full databases, and loading a database from a prior dump. I think this will be useful when running longer workloads through the multitest harness.

This leave the existing functionality intact, and I did a quick simulation run to verify:

Run test:BulkDumpingWorkload Done.

1 tests passed; 0 tests failed.

Unseed: 13729
Elapsed: 241.733437 simsec, 19.436907 real seconds

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@saintstack saintstack left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good. A few things in the below probably worth fixing 'cos will save you headache this is used over in the internal test harness.

BulkDumpState bulkDumpJob,
KeyRangeRef bulkDumpJobRange,
std::map<Key, Value> kvs) {
bool bulkDumpRangeContainBulkLoadRange = true; // Will set to false if the bulk load job range is not

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we skip generateInitialData, this could end up as set incorrectly....
Do bulkDumpRangeContainBulkLoadRange = bulkDumpJobRange.contains(bulkLoadJobRange); ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good point, and I'm not sure how best to handle the different flags in their interaction with that check. With the current logic in this PR, when we skip generateInitialData, it also skips any check on the ranges of the restored data, and that's not ideal. However, the current implementation of this check relies on having the original KV data in the kvs map, and that doesn't get populated when generateInitialData is false, and is probably not feasible to populate when we're running on a non-empty database with a large amount of data. Given that, I'm leaning toward having those checks get skipped, and having bulkDumpRangeContainBulkLoadRange left set to true. What do you think?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whatever suits you John. You are taking this dumb workload made for a particular simple run and trying to backfill steps/options for use in a different context. You have a +1 from @sbodagala ... go for it.

runLoad = getOption(options, "runLoad"_sr, true);
generateInitialData = getOption(options, "generateInitialData"_sr, true);
sourceJobId = getOption(options, "sourceJobId"_sr, ""_sr).toString();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we skip the dump, source jobid shouldn't be empty... Its an error if it is...Means one wasn't passed. Add an assert?

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@brownleej brownleej closed this Sep 22, 2026
@brownleej brownleej reopened this Sep 22, 2026
@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux RHEL 9

  • Commit ID: bc04feb
  • Duration 0:48:45
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: bc04feb
  • Duration 0:55:24
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: bc04feb
  • Duration 1:24:13
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos-m1 on macOS 14.x

  • Commit ID: bc04feb
  • Duration 1:25:31
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: bc04feb
  • Duration 1:44:54
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: bc04feb
  • Duration 2:15:00
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos on macOS 14.x

  • Commit ID: bc04feb
  • Duration 4:45:56
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@brownleej
brownleej merged commit 80a1b64 into apple:main Sep 23, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants