Skip to content

test(integration-tests): add RTAS cases - #744

Draft
mkuchenbecker wants to merge 32 commits into
mkuchenbecker/delta-harness-standard-dmlfrom
mkuchenbecker/delta-harness-rtas
Draft

mkuchenbecker wants to merge 32 commits into
mkuchenbecker/delta-harness-standard-dmlfrom
mkuchenbecker/delta-harness-rtas

Conversation

@mkuchenbecker

@mkuchenbecker mkuchenbecker commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

Stack context

These four same-repository draft PRs form the complete path from main
through RTAS. Each PR is based on the head branch of the preceding PR.

Order Capability Pull request
1 Integration test core #741
2 Coverage documentation #742
3 Standard DML #743
4 RTAS #744

main -> #741 ->
#742 ->
#743 ->
#744

Summary

This PR adds Replace Table As Select (RTAS) prepared tables and test cases on top
of the Standard DML layer. It adds 264 cases to the 130-case Standard DML
catalog, producing 394 combined cases.

This branch also fills the RTAS section in TEST-COVERAGE.md. The other
capability sections remain owned by their respective PRs.

Scope

This layer owns everything a CREATE OR REPLACE TABLE AS SELECT statement can
change about a table and everything the catalog governs about a table that
reached its state through a replace. It reuses every Standard DML operation as
data against replaced tables and excludes concurrency and locking, which belong
to the compatibility and streaming layer. ChangelogFixtures and
ConcurrencySupport live in this layer because RTAS is their first consumer.
TableMetadataFixtures provides the snapshot and catalog-name helpers shared
with later capability layers.

Family Coverage Cases
Reused Standard DML operations Every applicable operation on four replace-lineage prepared tables, plus the null-string DELETE and two partition-scoped writes. 212
Replace contract Enablement gates, same-shape replacement, schema discontinuities, partition replacement, property override and preservation, retention and tag preservation, time travel, rollback, changelog and incremental-read boundaries, rename ordering, sort order, identity preservation, and replace-versus-append behavior. 52
Total 264

Known limitations

Case Reason Skipped
rtas.schema.incompatibleType.notSilentlyLossy (Parquet, ORC) A bigint-to-int replacement succeeds and wraps an out-of-range key instead of being rejected or preserving the value. 2
rtas.concurrency.replaceVersusAppend (Parquet, ORC) A replace and an append can both report successful commits while the append's snapshot wins and loses the replace. 2

Every RTAS descendant inherits exactly these four skips; none of them adds a
new one.

Validation

Check Result
git diff --check against the parent tip Clean.
Added lines ASCII only.
Embedded catalog 394 combined cases: 390 passed, 4 skipped, 0 failed.

mkuchenbecker and others added 18 commits September 1, 2026 20:40
Add 264 replacement-specific cases on the 642-case foundation.

- run every reusable DML operation across four RTAS preparations
- cover schema, partition, policy, history, rename, order, and identity
- require precise lineage rejection and typed concurrency outcomes
- expose silent narrowing corruption as a known product bug

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep the replace-versus-append assertion limited to serializable outcomes while recording the rare lost-replace result as a known bug.

Both format cases remain available for re-enablement when concurrent RTAS commits report conflicts or preserve the replace.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Validate replace-table behavior by running its scenarios instead of
restating their generated catalog entries.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep replace-table behavior with the shared scenario suite while preserving
its package and published names.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rebase the replace-table capability on the focused 108-case DML
contract. Keep schema, catalog, and standard state-matrix breadth in
independent sibling reviews.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep RTAS as an additive capability over the stable foundation and the
complete standard DML contribution.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move partitioning, lineage, metadata, and rename helpers into the RTAS
layer and adopt the explicit DML operation naming from its parent.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mkuchenbecker and others added 4 commits September 17, 2026 17:51
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Describe replacement behavior and the prepared replacement lineages that reuse
the shared DML contracts without indexing tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mkuchenbecker and others added 7 commits September 18, 2026 09:55
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

# Conflicts:
#	integrations/spark/delta-harness/TEST-COVERAGE.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use integration-test terminology in RTAS comments and worker thread names.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move the replacement-table contributors into the renamed package and
preserve the same shared test definitions across local and remote runs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* preparations. The replace-table layer uses it to require rejection when a changelog range crosses a table
* replacement. The follow-up standard changelog scenario builds on the same operation definitions.
*/
trait ChangelogSupport extends RtasTableFixtures {

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shoudl not be an RTAS fixture, jsut a fixture.

trait / composition are preferred vs inheritance.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker mkuchenbecker changed the title test(delta-harness): add RTAS matrix test(delta-integration-tests): add RTAS cases Sep 21, 2026
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
mkuchenbecker marked this pull request as ready for review September 22, 2026 00:24
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker mkuchenbecker changed the title test(delta-integration-tests): add RTAS cases test(integration-tests): add RTAS cases Sep 22, 2026
@mkuchenbecker
mkuchenbecker marked this pull request as draft September 22, 2026 00:54

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant