Skip to content

feat(workflow-operator): read a column named in a different case the way the engine does - #9035

Open
kz930 wants to merge 1 commit into
apache:mainfrom
kz930:feat/export-ignore-column-case
Open

kz930 wants to merge 1 commit into
apache:mainfrom
kz930:feat/export-ignore-column-case

Conversation

@kz930

@kz930 kz930 commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

The engine looks a column up through Schema, which ignores case. So a setting that names Name reads the column name, and the workflow runs in Texera. The exported script indexed the pandas frame with the name as written and stopped with a KeyError.

StandaloneCodeGenerator gains columnAsStored, which resolves a setting's column name against the input schema and returns the table's spelling. A name the schema does not hold comes back unchanged, so the script still fails where the run would. Seven operators read their column through it: Projection, Filter, Keyword Search, Regular Expression, Substring Search, Dictionary Matcher and Unnest String. Projection still names each output column by its alias, which falls back to the name as the setting writes it. That matches ProjectionOpExec.

Type Casting is the exception. Its schema propagation and its executor both match the column by its exact name, so in the engine a unit whose column differs in case casts nothing and the column passes through unchanged. The export now skips that unit too, instead of raising a KeyError.

Operators that run as Python in the engine already compare names exactly on both sides, so they are left alone.

Any related issues, documentation, discussions?

Closes #9004. @carloea2 found the mismatch while reviewing #8918.

#8980 rewrites Filter's export and looks up the declared type by the column's name. Whichever of the two merges second should look that type up under the resolved name as well.

How was this PR tested?

Each of the eight operators' OpDescSpec gains a test with a setting in a different case from the table. It asserts that the generated code reads the table's spelling. For Type Casting, the test also checks that the engine's output schema leaves that column's type unchanged, and that the export skips the unit. All of WorkflowOperator/test passes. Merged with the other open export PRs, the verification suite still passes all 150 cases (3 ignored, as before).

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5)

🤖 Generated with Claude Code

…way the engine does

The engine looks a column up through Schema, which ignores case, so a setting
that names `Name` reads the column `name`. The exported script indexed the
frame with the name as written and raised a KeyError.

StandaloneCodeGenerator.columnAsStored resolves a name against the input
schema to the table's spelling. Projection, Filter, Keyword Search, Regular
Expression, Substring Search, Dictionary Matcher and Unnest String read their
column through it. Projection still names its output by the alias, as the
engine does.

Type Casting matches a column by its exact name in the engine, so a unit whose
column differs in case casts nothing there. The script now skips that unit too.

Closes apache#9004

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @aglinxinyuan, @sshiv012
    You can notify them by mentioning @aglinxinyuan, @sshiv012 in a comment.

@kz930

kz930 commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

@carloea2 could you review this one when you have time? It fixes the column-case mismatch you found on #8918. The red CI is the repo-wide ampel-bootstrap action being blocked, not this change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Exported Python looks up a column with its exact case where the engine ignores case

1 participant