Skip to content

test(integration-tests): cover compatibility and streaming - #749

Closed
mkuchenbecker wants to merge 33 commits into
mkuchenbecker/delta-harness-rtasfrom
mkuchenbecker/delta-harness-compatibility-streaming
Closed

test(integration-tests): cover compatibility and streaming#749
mkuchenbecker wants to merge 33 commits into
mkuchenbecker/delta-harness-rtasfrom
mkuchenbecker/delta-harness-compatibility-streaming

Conversation

@mkuchenbecker

@mkuchenbecker mkuchenbecker commented Sep 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

This PR adds concurrency, table locking, structured streaming, table-evolution
compatibility, and writer compatibility on top of RTAS. It contributes 114
cases to the 394-case RTAS catalog and produces 508 combined cases.

This branch also fills the Compatibility and streaming section in
TEST-COVERAGE.md.

Stack

Layer Parent Contribution Combined Pull request
RTAS Standard DML 264 394 #744
Compatibility and streaming RTAS 114 508 This PR

Scope

This layer owns the generated OpenHouse Tables API client, the table-lock
lifecycle, concurrency, streaming, and compatibility after table alterations.
DML and DDL semantics remain in the layers below it.

File Coverage Cases
ScenarioConcurrency.scala Two racing writers, where each writer commits or fails with a typed commit conflict. 4
ScenarioLocking.scala A REST lock rejects UPDATE and expire_snapshots while held, then admits them after DELETE. 2
ScenarioStreaming.scala Streaming read, append, restart after append, DELETE snapshot, and checkpoint-offset expiration. 10
ScenarioTableEvolutionCompatibility.scala INSERT, row-level DELETE, VERSION AS OF, rollback, expire_snapshots, and rewrite_data_files after four alteration kinds. 96
ScenarioWriterCompatibility.scala An explicit-column INSERT is accepted before ADD COLUMN and rejected afterward. 2
Total 114

Known limitations

This layer adds no known-bug skips. Its four combined-catalog skips are inherited
from RTAS.

Validation

Check Result
git diff --check against the parent tip Clean.
Added lines ASCII only.
Embedded catalog 508 combined cases: 504 passed, 4 skipped, 0 failed.

mkuchenbecker and others added 20 commits September 1, 2026 23:53
@mkuchenbecker
test(delta-harness): add compatibility scenarios
9235fb9 
Add concurrency, locking, streaming, table-evolution, and writer compatibility contracts as an extension of the focused RTAS catalog.

The scenarios prove exact writer outcomes, owned streaming resources, active snapshot stability, and observable follow-up behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
Merge RTAS race marker into compatibility
6cf1fc4 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
Merge foundation test cleanup into compatibility and streaming
6ee43b4 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
Merge RTAS test cleanup into compatibility and streaming
fe1f6b3 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
test(delta-harness): remove compatibility streaming catalog pin
a2f89ac 
Validate compatibility and streaming behavior by running its scenarios instead of restating
its generated catalog entries.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
Merge scenario source layout into compatibility and streaming
1bbfb58 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
refactor(delta-harness): relocate compatibility scenarios
78036e0 
Keep compatibility and streaming behavior with the shared scenario suite while preserving its
package and published names.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
Merge scenario ownership docs into compatibility and streaming
d41e824 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
refactor(delta-harness): use generated lock client
c262a65 
Replace the general JDK HTTP helper with a lock-specific client built on
the generated OpenHouse Tables API. Preserve response status and
diagnostic text while using the standard authentication and truststore
path in local and remote environments.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
Merge recut RTAS parent into compatibility
6a08beb 
Keep the generated lock client and compatibility scenarios on the
focused RTAS stack while removing unrelated foundation breadth.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
Merge core boundary cleanup into compatibility
ec3c91f 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

# Conflicts:
#	integrations/spark/delta-harness/src/main/scala/harness/openhouse/Framework.scala
#	integrations/spark/delta-harness/src/main/scala/harness/openhouse/LocalRunner.scala
#	integrations/spark/delta-harness/src/main/scala/harness/openhouse/scenarios/ScenarioKit.scala
@mkuchenbecker
merge(delta-harness): adopt revised RTAS stack
b92cab0 
Keep compatibility, locking, streaming, and concurrency coverage as an
additive layer over the revised RTAS catalog.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
Merge commit '74fae5e16d68434c2088c1fd8ab92d249266ab82' into mkuchenb…
e02029e 
…ecker/delta-harness-compatibility-streaming
@mkuchenbecker
Merge commit 'b784179951e36a64bf8926455799d09b4ef6c5ed' into mkuchenb…
802e701 
…ecker/delta-harness-compatibility-streaming

# Conflicts:
#	integrations/spark/delta-harness/src/test/scala/harness/framework/RunnerTest.scala
@mkuchenbecker
refactor(delta-harness): scope compatibility fixtures
47f91a2 
Keep layout expansion, side-table ownership, and table creation helpers
with the compatibility, locking, and streaming scenarios that use them.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
Merge incremental RTAS documentation
72ab344 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
docs(delta-harness): describe compatibility coverage
068ba24 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
Merge revised RTAS documentation
b4fe1de 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
Merge RTAS review links
7803b8e 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
Merge official RTAS PR links
d5213fe 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Sep 18, 2026
Draft
Closed
mkuchenbecker and others added 7 commits September 17, 2026 17:51
@mkuchenbecker
Merge RTAS documentation update
5073583 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
Merge RTAS coverage model
92bf575 
Describe compatibility and streaming through coherent concurrent state,
checkpoint recovery, table evolution, locking, and writer compatibility.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
Merge revised RTAS parent into compatibility
4c20ead 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
Merge concrete RTAS docs into compatibility
455b6a6 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
Merge coverage matrix rewrite into compatibility
b5d69a7 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

# Conflicts:
#	integrations/spark/delta-harness/TEST-COVERAGE.md
@mkuchenbecker
Merge integration test terminology into compatibility
ed0fa28 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
Merge revised integration test terminology into compatibility
071e630 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mkuchenbecker and others added 5 commits September 18, 2026 11:22
@mkuchenbecker
Merge Graphviz record dividers into compatibility
ec6c120 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
Merge Airflow graph simplification into compatibility
9a5cad5 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
Merge the Delta integration-test rename into compatibility
0134e2c 
Move compatibility, streaming, and locking contributors into the renamed
package while preserving their portable and embedded execution boundary.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
Merge renamed RTAS fixtures into compatibility tests
0639ecc 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
Merge shared table metadata fixtures
093692a 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker mkuchenbecker changed the title test(delta-harness): add compatibility and streaming scenarios test(delta-integration-tests): cover compatibility and streaming Sep 21, 2026
@mkuchenbecker mkuchenbecker reopened this Sep 21, 2026
@mkuchenbecker
Merge integration-test naming into compatibility tests
9d19bfc 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker mkuchenbecker changed the title test(delta-integration-tests): cover compatibility and streaming test(integration-tests): cover compatibility and streaming Sep 22, 2026
@mkuchenbecker mkuchenbecker reopened this Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@mkuchenbecker