test(verify): sweep a configuration into the variants a parity run needs by kz930 · Pull Request #8388 · apache/texera

kz930 · 2026-09-03T21:44:41Z

What changes were proposed in this PR?

One configuration exercises one path through an operator. A parity run wants
more than that: every value an enum offers, the optional knobs a default leaves
empty, and a text knob carrying a value that breaks a Python literal if it was
spliced rather than escaped.

This adds the sweep on top of the generator that derives the configuration. Four
entry points return variants: one for an operator configured from its
annotations, and three for a hand-written fixture, which needs them more, since
the configuration it pins is otherwise the only one its operator ever runs.

The sweep moves one knob at a time rather than crossing them, so a failure names
the knob that caused it. The exceptions are the two variants that move every
knob of a kind at once, where bisecting by hand costs less than a run per field:
all optional knobs filled together, and all free-text knobs carrying the hostile
value together, numbered so no two knobs carry the same text. Two knobs sharing
one value is not a smaller version of the same test. Where both name an output
column, it writes the same column twice and the run fails for a reason that has
nothing to do with escaping.

Any related issues, documentation, discussions?

Part of #8325, 22 of 27; that issue lists the set in order. This is the second
half of the configuration generator, and #8361 is the first. The two are one
Scala object, so this branch carries that one's commit as well: the sweep does
not compile without the members it defines. Until #8361 merges the file view
here shows both halves, and once it does this narrows to the sweep on its own.
The last commit is the sweep, and it is the one to read.

Closes #8424, the task this change is the whole of.

How was this PR tested?

Four tests read the entry points rather than the members behind them: an enum
offered once per value with each variant labelled by the knob it moved, a
boolean flipped in a variant of its own, an optional knob the base leaves empty
and this fills, and two free-text knobs that come out holding different text.

The sweep is exercised end to end once the last pieces of the set land, when
every operator runs through the engine and through its generated script and the
two answers are compared. These four cover it in the meantime, and they name the
knob rather than the operator when they fail.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

An operator is configured by a form, and the form is generated from the descriptor's own annotations. The same annotations are enough to fill it without a human: an enum offers its constants, a column knob takes a column of the type its rule allows, a numeric knob its declared bounds, and a conditional field is filled only where the branch that shows it is taken. This is one file because it is one decision made repeatedly — what a knob may hold — and splitting it would put half of that decision on either side of a boundary. `@SampleColumn` names the column a knob means where a type cannot say it: the opening price among four numbers, the country code among several strings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A candlestick reads four prices and the day they belong to, which a type cannot tell apart; a choropleth reads a country code the same way, and a value to shade it by that a type rule does narrow. Both are what the annotation and the rule are for. These read operators that land before this change does, so they belong here rather than being added back afterwards. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The generator did two jobs in one file: derive a valid configuration for an operator from the annotations it already carries, and sweep that configuration into the variants a parity run needs. At two thousand lines it was the largest change in the export series and the only one that could not be read in one sitting. This change leaves the first job. The sweep follows as its own change on top of it, restoring the members verbatim. The two jobs share no member. The base pass reaches its values through buildObject, decide and valueFor; the sweep reaches its own through rowFills, columnFill and leafFill, and the four entry points that return variants are the only callers of either. Nothing was rewritten to make the seam: whole members moved out, and every line that stays is the line that was there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

One configuration exercises one path through an operator. A parity run wants more than that: every value an enum offers, the optional knobs a default leaves empty, and a text knob carrying a value that breaks a Python literal if it was spliced rather than escaped. This adds the sweep on top of the generator that derives the configuration. Four entry points return variants: one for an operator configured from its annotations, and three for a hand-written fixture, which needs them more, since the configuration it pins is otherwise the only one its operator ever runs. The sweep moves one knob at a time rather than crossing them, so a failure names the knob that caused it. The exceptions are the two variants that move every knob of a kind at once, where bisecting by hand costs less than a run per field: all optional knobs filled together, and all free-text knobs carrying the hostile value together, numbered so no two knobs carry the same text. The four tests read the entry points rather than the members behind them: an enum offered once per value, a boolean flipped in a variant of its own, an optional knob the base leaves empty and this fills, and two free-text knobs that come out holding different text. Every member here was in the change before this one and is restored verbatim. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

github-actions · 2026-09-03T21:44:59Z

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

Contributors with relevant context: @sshiv012
You can notify them by mentioning @sshiv012 in a comment.

codecov-commenter · 2026-09-03T21:48:06Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.55%. Comparing base (40b507d) to head (8c41b1d).
⚠️ Report is 19 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8388      +/-   ##
============================================
- Coverage     92.60%   92.55%   -0.05%     
+ Complexity     5043     5039       -4     
============================================
  Files          1252     1220      -32     
  Lines         53587    50774    -2813     
  Branches       6673     6185     -488     
============================================
- Hits          49622    46995    -2627     
+ Misses         2317     2212     -105     
+ Partials       1648     1567      -81

Flag	Coverage Δ		*Carryforward flag
access-control-service	`77.38% <ø> (ø)`
agent-service	`99.36% <ø> (+0.19%)`	⬆️	Carriedforward from 80d432f
amber	`88.09% <ø> (+<0.01%)`	⬆️
computing-unit-managing-service	`60.48% <ø> (ø)`
config-service	`87.37% <ø> (ø)`
file-service	`82.00% <ø> (+0.47%)`	⬆️
frontend	`96.86% <ø> (+0.27%)`	⬆️	Carriedforward from 80d432f
notebook-migration-service	`83.73% <ø> (ø)`
pyamber	`98.57% <ø> (+0.05%)`	⬆️	Carriedforward from 80d432f
workflow-compiling-service	`77.19% <ø> (+3.09%)`	⬆️	Carriedforward from 80d432f

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

`variantsOf` swept an already-configured op for its enums alone. Every caller wants the optional and hostile-text variants with them, which is the entry point beside it, so this one was never called. Its scaladoc explained itself by contrast with that one, so the explanation moves there rather than going with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… above Splitting the sweep out of this file moved members by their declaration, which left the block of documentation above three of them behind. Each then sat over the member that followed, describing something else: a JSON pointer helper's text over the one that reads a field's key, a nested-row accessor's over the one that reads a field's declared example, and a free-scalar test's over the type-id registry. None of them is wrong about the member it was written for. They travel with it. The class's own header and the column-resolution comment lose the parts a reader gets from the code beside them: an assembly strategy the next twenty lines carry out, and a numbered list of the tiers an `orElse` chain already reads as. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The optional-scalar fill repeated, sentence for sentence, why the column fill beside it exists. It points there instead. The pointer helper's one-line description said what its one-line body says. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

carloea2

I checked the sweep part of this stack. It looks good.

A free-text knob already gets a hostile value. A column knob carries a name the generated code writes into a subscript, which is a separate splice site, so it gets a run of its own: each knob moves to a hostile-named column of the type it already reads, one per type per port so two knobs never collide. Matched on the type the knob HOLDS rather than the types its rule admits: a filter's value is parsed as the held column's type, and moving the knob to a string would fail on this generator rather than on any escaping. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kz930 · 2026-09-11T01:19:05Z

@carloea2 this moved after your approval, one commit: every column knob now points at a hostile-named column. Would you take another look when you have a moment?

ConfigGenerator names @SampleColumn in five places, and the annotation itself was in the change that introduces the trait, where nothing reads it. It is test-only metadata: it names the column in the shared fixture that should fill an @AutofillAttributeName field, so a parity run reads a realistic value rather than whatever the first column happens to hold. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The annotation is applied on operator fields, so it has to exist before the first change that annotates one, which is ten steps earlier than this. It goes there instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

carloea2

The generated variants now cover hostile column names and the schema annotations. Looks good.

The column pickers read a field's `attributeTypeRules` and then, finding no column of an accepted type, took another one anyway. That writes a configuration nobody would: the operator states which types the field takes, and a run over a column outside them reports the operator as covered on a configuration it does not describe. Both pickers now say why instead, naming the types the field accepts and the columns the port actually holds. Neither is reached against today's fixtures, every type any rule names having a column in each of them, so the whole job of this is to speak up the day one of them narrows. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The char delimiter constant lists the picker's presets under `examples`, and a field that offers more than one example is swept like an enum. Old CSV's default is already a comma, so its optionals variant read the field as unset and filled the canonical "1", which split the fixture's numbers apart. A value with no letter or digit is labelled by its code points, since the runners fold every other character of a label to `_` and the five delimiters shared one directory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions · 2026-10-08T04:23:47Z

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 12 worse · ⚪ 3 noise (<±5%) · 0 without baseline

Compared against main 40b507d benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	621	0.379	15,804/21,997/21,997 us	🔴 +10.1% / 🔴 +39.9%
🔴	bs=100 sw=10 sl=64	1,567	0.957	60,944/87,729/87,729 us	🔴 +18.3% / 🟢 +53.0%
🔴	bs=1000 sw=10 sl=64	1,988	1.214	494,855/596,247/596,247 us	🔴 +10.3% / 🟢 +87.8%

Baseline details

Latest main 40b507d from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	621 tuples/sec	668 tuples/sec	790.31 tuples/sec	-7.0%	-21.4%
bs=10 sw=10 sl=64	MB/s	0.379 MB/s	0.408 MB/s	0.482 MB/s	-7.1%	-21.4%
bs=10 sw=10 sl=64	p50	15,804 us	14,786 us	12,375 us	+6.9%	+27.7%
bs=10 sw=10 sl=64	p95	21,997 us	19,972 us	15,719 us	+10.1%	+39.9%
bs=10 sw=10 sl=64	p99	21,997 us	19,972 us	19,058 us	+10.1%	+15.4%
bs=100 sw=10 sl=64	throughput	1,567 tuples/sec	1,677 tuples/sec	1,025 tuples/sec	-6.6%	+52.9%
bs=100 sw=10 sl=64	MB/s	0.957 MB/s	1.023 MB/s	0.626 MB/s	-6.5%	+53.0%
bs=100 sw=10 sl=64	p50	60,944 us	57,963 us	98,759 us	+5.1%	-38.3%
bs=100 sw=10 sl=64	p95	87,729 us	74,137 us	105,327 us	+18.3%	-16.7%
bs=100 sw=10 sl=64	p99	87,729 us	74,137 us	120,491 us	+18.3%	-27.2%
bs=1000 sw=10 sl=64	throughput	1,988 tuples/sec	2,058 tuples/sec	1,059 tuples/sec	-3.4%	+87.7%
bs=1000 sw=10 sl=64	MB/s	1.214 MB/s	1.256 MB/s	0.646 MB/s	-3.3%	+87.8%
bs=1000 sw=10 sl=64	p50	494,855 us	480,340 us	965,886 us	+3.0%	-48.8%
bs=1000 sw=10 sl=64	p95	596,247 us	540,437 us	1,007,533 us	+10.3%	-40.8%
bs=1000 sw=10 sl=64	p99	596,247 us	540,437 us	1,031,622 us	+10.3%	-42.2%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,322.15,200,128000,621,0.379,15804.12,21997.23,21997.23
1,100,10,64,20,1275.98,2000,1280000,1567,0.957,60944.11,87729.41,87729.41
2,1000,10,64,20,10058.61,20000,12800000,1988,1.214,494854.82,596246.71,596246.71

The scan sources take apache#8341's resolution: the Limit 0 sample now lives in the shared inferSampleSize, and the CSV delimiters use the picker constant. JSONL reads its sample through inferSampleSize too. CSVOld's optionals variant is red until the delimiter examples land in apache#8388. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

carloea2

The delimiter sweep works: both CSV sources offer all five presets, and their directory labels stay distinct. All five sweep tests and scalafmtCheckAll pass on the unchanged branch.

Command: sbt -batch "WorkflowCompilingService/testOnly *ConfigGeneratorSpec -- -oDF -z generateVariants" "scalafmtCheckAll".

The full suite has ten passes and the two missing chart-metadata failures already described in #8361. This branch also carries an older copy of that base generator. Please update the stack after addressing the incompatible join-key selection reported there. No separate finding in the new sweep changes.

Hash Join ties its build key's type to the probe key's through "const": {"$data": ...}, which the generator never read, so two ports listing their columns in different orders joined an INTEGER to a STRING, and two ports sharing no type still produced a configuration. Read the link from both ends and intersect it with the enum rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…guration-variants Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

kz930 · 2026-10-13T05:43:39Z

@carloea2 Merged #8361's current head (ab851d6) into this branch as 8c41b1d, so the generator here now carries the $data join-key fix. ConfigGeneratorSpec gives 15 passes, and the two failures are the same chart-metadata ones.

Merging the configuration branch in left listColumnFill's comment on generateVariants as well, so it appeared twice and generateVariants lost its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

kz930 and others added 5 commits September 2, 2026 16:30

chore: the annotation travels with the operators, not the generator

607c1e8

github-actions Bot assigned kz930 Sep 3, 2026

github-actions Bot added feature platform Non-amber Scala service paths labels Sep 3, 2026

kz930 mentioned this pull request Sep 3, 2026

A workflow cannot be taken out of Texera as a script that runs on its own #8325

Open

13 of 31 tasks

kz930 and others added 3 commits September 3, 2026 15:00

carloea2 approved these changes Sep 4, 2026

View reviewed changes

kz930 marked this pull request as draft September 4, 2026 17:12

kz930 mentioned this pull request Sep 4, 2026

test(verify): derive an operator's configuration from its schema #8361

Draft

kz930 force-pushed the feat/verify-sweep-configuration-variants branch from 1ed4742 to ed6b700 Compare September 9, 2026 06:50

kz930 and others added 2 commits September 10, 2026 21:05

carloea2 approved these changes Sep 18, 2026

View reviewed changes

kz930 and others added 4 commits September 18, 2026 23:56

test(verify): say that a list knob whose rule matches nothing is refused

80d432f

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Merge upstream/main into feat/verify-sweep-configuration-variants

faa3c91

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions Bot added the common label Oct 8, 2026

carloea2 reviewed Oct 13, 2026

View reviewed changes

kz930 and others added 2 commits October 12, 2026 21:43

Merge feat/verify-derive-a-configuration into feat/verify-sweep-confi…

8c41b1d

…guration-variants Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs(verify): put generateVariants' own comment back above it

fb31555

Merging the configuration branch in left listColumnFill's comment on generateVariants as well, so it appeared twice and generateVariants lost its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Navigation Menu

Repository navigation

test(verify): sweep a configuration into the variants a parity run needs - #8388

test(verify): sweep a configuration into the variants a parity run needs#8388
kz930 wants to merge 18 commits into
apache:mainfrom
kz930:feat/verify-sweep-configuration-variants

kz930 commented Sep 3, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 3, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Sep 3, 2026 •

edited

Loading

Uh oh!

carloea2 left a comment

Uh oh!

kz930 commented Sep 11, 2026 •

edited

Loading

Uh oh!

carloea2 left a comment

Uh oh!

github-actions Bot commented Oct 8, 2026 •

edited

Loading

Uh oh!

carloea2 left a comment

Uh oh!

kz930 commented Oct 13, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

kz930 commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this PR?

Any related issues, documentation, discussions?

How was this PR tested?

Was this PR authored or co-authored using generative AI tooling?

Uh oh!

github-actions Bot commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Automated Reviewer Suggestions

Uh oh!

codecov-commenter commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

kz930 commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Benchmark changes need a look

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

kz930 commented Oct 13, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kz930 commented Sep 3, 2026 •

edited

Loading

github-actions Bot commented Sep 3, 2026 •

edited

Loading

codecov-commenter commented Sep 3, 2026 •

edited

Loading

kz930 commented Sep 11, 2026 •

edited

Loading

github-actions Bot commented Oct 8, 2026 •

edited

Loading