Repository navigation
Conversation
…ython script A workflow can be read in the editor but not taken away: there is no form of it that runs anywhere else, so a user who wants to keep a pipeline, hand it to someone without Texera, or step through it in a notebook has nothing to take. This adds the seam for one and the first few operators through it. An operator says how it reads outside the engine by implementing `StandaloneCodeGenerator`, returning a block of pandas that names its inputs and outputs as `in1df` / `out1df`. The translator walks the plan in topological order, gives every port a variable, substitutes those placeholders, and prints the leaves; `inAlldf` stands for the whole list of upstreams, which is what a variadic port like Union's needs, since any fixed count the code stated would be wrong for some workflow. An operator with no generator yet leaves a commented TODO rather than a line that looks like it works. `GET /workflow-to-python` on the compiling service returns the script for a plan it is given. Five operators implement it here — Distinct, Limit, Projection, Filter and Union — chosen to cover the shapes the translator has to handle: a single input, a config-driven one, one that renames columns, one that builds a predicate, and the variadic port. The rest of the operator set follows in later changes. `pyStringLiteral` renders a value as a Python literal with the escaping that keeps a quote or a newline in a column name from ending the literal early. The generators cannot use the runtime's decode expression, which needs an operator instance to decode through. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…re the files The standalone export claims that a generated script does what the operator does. Nothing checks it. This adds the two runners that make the claim checkable, and the file format they meet in. `OpExecHarness` runs a LogicalOp the way the engine does — compiling it to a physical plan and driving the executor — but outside a workflow, against JSONL files rather than a live upstream. `PyOpExecHarness` does the same for a Python operator, through the worker the engine uses. `StandaloneRunner` takes the other path: it asks the operator for its standalone code, wraps it in a script that binds `in1df` from the same files, and runs it. `TupleIO` is what the two meet in. A JSONL row carries values and no types, so the schema travels beside it in a sidecar; without one, a column written as INTEGER reads back as a number and the two paths disagree over a difference neither operator made. Both runners produce files, not assertions, so what to make of a difference is left to a later change. What is here is enough to run one operator both ways and see that the answers match, which is what the spec does with Distinct. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…he operator it came from The standalone export claims a generated script does what the operator does. This is what checks it, for every operator, on every configuration the operator offers. An operator is run twice. `OpExecHarness` drives it the way the engine does, compiled to a physical plan but outside a workflow, reading JSONL files rather than a live upstream; `PyOpExecHarness` does the same for a Python operator through the worker the engine uses. `StandaloneRunner` takes the other path, wrapping the operator's standalone code in a script that binds the same files. Both write files, and `Comparator` reads them back: order-insensitive by default, since the engine interleaves across workers and only the sort family promises an order. A visualization is compared as a figure rather than as a frame. What to run an operator ON is decided rather than written by hand for each. `ConfigGenerator` reads the operator's own schema — its enums, defaults, declared ranges and column pickers — and produces a base configuration plus one variant per branch the operator offers, so a switch nobody thought to try is still tried. `CanonicalFixture` is the table they run against, one column per shape an operator might ask for. `CuratedHandlers` is the escape hatch for an operator whose input cannot be derived, and `TransformVerificationRunner` decides which of the three tiers each operator takes and reports what it could not run and why. `LogicalOp.orderSensitive` and `@SampleColumn` are the two things the operators had to say for this to read them: whether row order is part of the contract, and which column a field should be pointed at when the first unused one would be a poor choice. Most of the operator set does not implement the generator yet — it arrives a family at a time — and the runner reports each of those rather than passing over it. The tier assertions for a family land with the change that gives that family its generator. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…f the one without The split this change relies on was declared but never wired. The specs carry `@IntegrationTest` and `build.sbt` reads `WCS_TEST_FILTER` to act on it, but nothing set that variable, so the filter was a no-op and the specs that fork Python ran in the job that provisions none — failing on `No module named 'pandas'` rather than on anything they were testing. The platform job now sets `skip-integration`, which excludes them. The platform-integration job sets `integration-only` and provisions what they need: Python 3.12, amber's requirements, protoc, and the generated proto bindings, which are gitignored and so have to be regenerated before a forked driver can import pyamber. Every step is guarded on the service, so the other entries in that matrix are untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The comment claimed a knob carrying a `defaultValue` is left alone. It is not: a knob holding what a fresh instance holds counts as untouched, because a config where the user typed the default is indistinguishable from one where they typed nothing. Moving it off that value is the only way the branch for a knob that IS set gets exercised. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…-coordinate-systems
… Python Eighteen visualizations that draw in a space of their own rather than over a table's columns: the polar family — polar, radar in its two forms, wind rose — the ternary pair, the three-dimensional scatter, the contour and carpet plots, the quiver field, the map, the gantt and time series over a clock, the gauge and bullet indicators, the dumbbell and filled-area comparisons, and the volcano plot. Each is compared as a figure rather than as a frame, after both paths have drawn: a plot's meaning is in the numbers behind it, and comparing the rendered picture would fail over a layout detail that carries none. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
feature
dependencies
Contributor
Automated Reviewer SuggestionsBased on the
|
Codecov Report❌ Patch coverage is ❌ Your patch status has failed because the patch coverage (7.69%) is below the target coverage (60.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #8346 +/- ##
============================================
- Coverage 92.30% 92.14% -0.17%
+ Complexity 5046 5045 -1
============================================
Files 1260 1260
Lines 53887 53983 +96
Branches 6711 6714 +3
============================================
- Hits 49743 49741 -2
- Misses 2486 2582 +96
- Partials 1658 1660 +2
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 383 | 0.234 | 26,360/29,707/29,707 us | 🔴 +10.3% / 🔴 +131.4% |
| 🟢 | bs=100 sw=10 sl=64 | 800 | 0.488 | 124,616/155,996/155,996 us | 🟢 -9.9% / 🔴 +55.7% |
| ⚪ | bs=1000 sw=10 sl=64 | 909 | 0.555 | 1,102,799/1,191,694/1,191,694 us | ⚪ within ±5% / 🔴 +28.9% |
Baseline details
Latest main 7426431 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 383 tuples/sec | 423 tuples/sec | 912.36 tuples/sec | -9.5% | -58.0% |
| bs=10 sw=10 sl=64 | MB/s | 0.234 MB/s | 0.258 MB/s | 0.557 MB/s | -9.3% | -58.0% |
| bs=10 sw=10 sl=64 | p50 | 26,360 us | 23,892 us | 11,391 us | +10.3% | +131.4% |
| bs=10 sw=10 sl=64 | p95 | 29,707 us | 32,188 us | 14,608 us | -7.7% | +103.4% |
| bs=10 sw=10 sl=64 | p99 | 29,707 us | 32,188 us | 17,250 us | -7.7% | +72.2% |
| bs=100 sw=10 sl=64 | throughput | 800 tuples/sec | 817 tuples/sec | 1,161 tuples/sec | -2.1% | -31.1% |
| bs=100 sw=10 sl=64 | MB/s | 0.488 MB/s | 0.499 MB/s | 0.709 MB/s | -2.2% | -31.1% |
| bs=100 sw=10 sl=64 | p50 | 124,616 us | 119,724 us | 91,471 us | +4.1% | +36.2% |
| bs=100 sw=10 sl=64 | p95 | 155,996 us | 173,114 us | 100,192 us | -9.9% | +55.7% |
| bs=100 sw=10 sl=64 | p99 | 155,996 us | 173,114 us | 119,565 us | -9.9% | +30.5% |
| bs=1000 sw=10 sl=64 | throughput | 909 tuples/sec | 916 tuples/sec | 1,226 tuples/sec | -0.8% | -25.9% |
| bs=1000 sw=10 sl=64 | MB/s | 0.555 MB/s | 0.559 MB/s | 0.748 MB/s | -0.7% | -25.8% |
| bs=1000 sw=10 sl=64 | p50 | 1,102,799 us | 1,094,966 us | 887,282 us | +0.7% | +24.3% |
| bs=1000 sw=10 sl=64 | p95 | 1,191,694 us | 1,174,109 us | 924,352 us | +1.5% | +28.9% |
| bs=1000 sw=10 sl=64 | p99 | 1,191,694 us | 1,174,109 us | 951,858 us | +1.5% | +25.2% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,522.06,200,128000,383,0.234,26359.94,29706.80,29706.80
1,100,10,64,20,2500.98,2000,1280000,800,0.488,124616.47,155996.20,155996.20
2,1000,10,64,20,22009.20,20000,12800000,909,0.555,1102798.58,1191693.54,1191693.54The example of an operator with no generator was an operator another batch gives one to, so the assertion held only until that batch landed. A Python UDF holds whatever order these land in: its body is written by whoever drops the operator, so no generator can emit one. The word cloud assertion goes for the same reason, and the prediction op alone already covers what that test is for. The visualizations this batch adds are asserted where they route, which is the one thing the per-operator runs do not check, and a choropleth covers a column read two ways: a type rule narrows what can be shaded, an annotation names what holds the country code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Sep 2, 2026
Everything here that is not this batch's own operators belongs to apache#8327 and was carried only so the branch could compile and run its own tests before that one landed. Reviewing it twice costs more than the red build does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
github-actions
Bot
removed
dependencies
The rows these operators add to the runner and its config tests sit in files apache#8327 introduces, so they land once that does rather than as whole new files here. Each operator's own spec stays. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kz930
changed the title
carloea2
approved these changes
Sep 18, 2026
carloea2
left a comment
carloea2
left a comment
Contributor
There was a problem hiding this comment.
Charts that cannot be drawn now write an error page without ending the script. Looks good.
carloea2
suggested changes
Sep 18, 2026
carloea2
left a comment
carloea2
left a comment
Contributor
There was a problem hiding this comment.
Bullet and Gauge write one HTML chart per row but their JSON file contains only the first chart. Multirow output is incomplete. Please either write every figure to JSON or make HTML the only output, and add a test where the second row differs.
Bullet and Gauge draw one chart per row and wrote all of them to the HTML page, but only the first to the JSON. Those two are the file the verification compares them by, so every chart after the first went unread: the exported script could draw the wrong one and the run would still agree. A single figure still writes as a lone object, which is what every one-chart operator writes; more than one writes as the sequence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Both now write every chart to the JSON, which is the file the verification compares them by. HTML alone would not have closed it, since the runtime path writes one row per chart while the script writes one page holding all of them, so #8359 reads both sides as sequences instead. |
The operator draws inside a try and answers anything it did not foresee with a page saying what went wrong. The exported block had only the per-row catch, so a value column that is not in the table reached pandas as a KeyError and ended the whole run, where the chart alone should have failed. The same try wraps the block now. The spec runs the exported code over a missing column, an empty table, a table of nulls and a table it can draw, and holds each to the page the operator yields. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
While fixing the same class of bug on #8504 I found one here: the exported gauge chart had only the per-row catch, so a value column that is not in the table reached pandas as a KeyError and ended the whole run, where the operator answers it with a "General error" page. The block now sits inside the same try, and the spec runs the exported code over a missing column, an empty table, a table of nulls and a table it can draw, holding each to the page the operator yields. |
carloea2
approved these changes
Sep 19, 2026
carloea2
left a comment
carloea2
left a comment
Contributor
There was a problem hiding this comment.
The export now keeps all figures. The earlier missing chart finding is addressed. Looks good.
carloea2
reviewed
Sep 23, 2026
carloea2
left a comment
carloea2
left a comment
Contributor
There was a problem hiding this comment.
One export failure reproduced locally with the normal integer frame as a passing control. No full workflow run.
The exported script checked r and theta with np.issubdtype, which cannot interpret a nullable pandas dtype such as Int64 and raises. A frame read from Arrow carries those dtypes, so the export died instead of drawing. The check now uses pandas' own numeric test and still refuses booleans, as np.number did. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…-integration Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
carloea2
suggested changes
Oct 13, 2026
carloea2
left a comment
carloea2
left a comment
Contributor
There was a problem hiding this comment.
The Polar nullable-number fix works, including the boolean and text rejection controls. Testing used PlotlyStandaloneCode from merged #8344 locally.
Command: sbt -batch "WorkflowOperator/testOnly *PolarChartOpDescSpec *RadarChartOpDescSpec *RadarPlotOpDescSpec *WindRoseChartOpDescSpec *TernaryPlotOpDescSpec *TernaryContourOpDescSpec *GaugeChartOpDescSpec *BulletChartOpDescSpec -- -oDF" "scalafmtCheckAll".
Default Windows encoding gives 54 passes and one failure in the new Polar driver. With PYTHONUTF8=1, all 55 runnable tests and formatting pass. One existing test is pending in both runs. The branch still needs the merged shared helper before CI can compile it.
…-coordinate-systems
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
carloea2
approved these changes
Oct 14, 2026
carloea2
left a comment
carloea2
left a comment
Contributor
There was a problem hiding this comment.
The UTF-8 fix passes on Windows without forcing Python's encoding. All 55 runnable tests and formatting checks pass; the one existing pending test is unchanged. The remaining CI failure is LoopIntegrationSpec, covered by #8938. Code looks good. Leaving the draft status unchanged.
kz930
marked this pull request as ready for review
October 15, 2026 00:17
kz930
marked this pull request as draft
October 15, 2026 01:17
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kz930
marked this pull request as ready for review
October 15, 2026 22:51
aglinxinyuan
enabled auto-merge
October 19, 2026 21:04
github-merge-queue
Bot
removed this pull request from the merge queue due to no response for status checks
Oct 19, 2026
kz930
added a commit
to kz930/texera
that referenced
this pull request
Oct 19, 2026
…o kary-integration
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
Eight visualizations implement
StandaloneCodeGenerator: the polar family (polar, radar in its two forms, wind rose), the ternary pair, and the gauge and bullet indicators. What they share is the question they answer, which is where on an angle, a radius or a simplex a row lands.The ten drawn over a field or a sequence are in #8504. With those, #8344 and this one, every visualization Texera offers can be exported.
Each is compared as a figure rather than as a frame, after both paths have drawn. A plot's meaning is in the numbers behind it, so that is what the comparison reads; comparing the rendered picture instead would fail over a layout detail that carries no meaning.
Any related issues, documentation, discussions?
Part of #8325, 16 of 27; that issue lists the set in order. It needs #8327 for the trait, so it does not compile until that lands, and the rows these operators add to the verification runner follow with the harness rather than as whole new files here.
Closes #8419, the task this change is the whole of.
How was this PR tested?
Each operator asserts the block it emits in its own spec. Once the verification lands it is also run through the engine and through its generated script, on every configuration its schema offers, and the two answers compared; this branch is cut from main and does not carry that machinery, so those runs are not on this diff's CI.
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)