Repository navigation
Add a generic bounded post commit operations framework - #729
Merged
abhisheknath2011 merged 6 commits intoSep 16, 2026
Merged
Conversation
mkuchenbecker
left a comment
Collaborator
There was a problem hiding this comment.
Can we see an example stacked on top?
Member
Author
Yes added example usage in the PR description. |
…the internal methods with warn
mkuchenbecker
previously approved these changes
Sep 16, 2026
mkuchenbecker
approved these changes
Sep 16, 2026
5 of 17 tasks
abhisheknath2011
added a commit
that referenced
this pull request
Sep 17, 2026
…gate (#713) ## Summary Rework of the Table Optimizer "Stats Bridge" commit-stats step on top of the merged generic post-commit framework (#729). This PR adds **only** the stats-specific pieces; all the seam/threading/timeout/wiring is provided by `PostCommitOperationRunner` from #729. Previously this PR was stacked on #712 and gated collection at the **database level**. That approach is dropped: #712 is superseded by #729, and gating moves to a **per-table property**. **New / kept** - `CommitStats` — neutral, immutable value object, free of optimizer-client types. Separates **current-state** totals (`numCurrentFiles`, `tableSizeBytes`) from a nested **`CommitStats.Delta`** (`numFilesAdded`/`numFilesDeleted`, `addedSizeBytes`/`deletedSizeBytes`), plus identity, location, version, and table properties. - `CommitStatsFactory` — extracts `CommitStats` from the committed `TableMetadata` (`SnapshotSummary` totals/deltas + OpenHouse canonical properties). Returns empty without a table UUID; for a commit that produced **no new snapshot** (metadata-only), current totals and `delta` are left **null** (a null delta means "unknown", not zero); tolerates malformed summary values. - `AbstractCommitStatsPublishOperation` — a `PostCommitOperation` (from #729) that extracts and publishes stats, gated per table. Abstract `publish(CommitStats)` is implemented by a `@Component` subclass in the LinkedIn repo; OSS registers no bean, so this is inert in OSS/dev. **Gating: database-level → table-property level** - Removed `TableStatsPublisher`, `NoOpTableStatsPublisher`, `CommitStatsCollectionGate`, `ConfigurableCommitStatsCollectionGate`, the `doCommit()`/`OpenHouseInternalCatalog` wiring, and the `cluster.tables.optimizer.commit-stats-collection.database-filter` config — all superseded by #729. - Collection is enabled per table via the user-settable property **`optimizer.commitStatsCollectionEnabled=true`**. The key is intentionally **not** `openhouse.`-prefixed: `BasePreservedKeyChecker` treats `openhouse.*` as reserved keys that table owners cannot set, which would make the gate un-settable. **Two-level enablement** (both required to publish): 1. Global runner switch `cluster.tables.postcommit.enabled=true` (default false, from #729). 2. Per-table opt-in `optimizer.commitStatsCollectionEnabled=true`. <!--- HINT: Replace #nnn with corresponding Issue number, if you are fixing an existing issue --> [Issue](https://github.com/linkedin/openhouse/issues/#nnn)] Briefly discuss the summary of the changes made in this pull request in 2-3 lines. ## Changes - [ ] Client-facing API Changes - [x] Internal API Changes - [ ] Bug Fixes - [x] New Features - [ ] Performance Improvements - [ ] Code Style - [x] Refactoring - [ ] Documentation - [x] Tests For all the boxes checked, please include additional details of the changes made in this pull request. ## Testing Done `./gradlew build` passed - `CommitStatsFactoryTest` — identity/properties-only, snapshot-summary mapping into nested delta, empty-without-UUID, malformed-value tolerance, null delta for snapshot-less commits. - `AbstractCommitStatsPublishOperationTest` — publishes only when the table property is `true` **and** a UUID is present; no publish when the property is absent/false or the table has no UUID. - `./gradlew :iceberg:openhouse:internalcatalog:test` (Java 17) passes. <!--- Check any relevant boxes with "x" --> - [ ] Manually Tested on local docker setup. Please include commands ran, and their output. - [x] Added new tests for the changes made. - [ ] Updated existing tests to reflect the changes made. - [ ] No tests added or updated. Please explain why. If unsure, please feel free to ask for help. - [ ] Some other form of testing like staging or soak time in production. Please explain. For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request. # Additional Information - [ ] Breaking Changes - [ ] Deprecations - [ ] Large PR broken into smaller PRs, and PR plan linked in the description.
7 of 17 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduce a generic, best-effort post-commit operations framework in the internal catalog. This is pure infrastructure that lets arbitrary logic run after a successful table commit without the catalog knowing what that logic does, and without any such operation being able to destabilize the commit path.
Motivation (per review on #712): rather than a single-purpose "stats publisher" seam, model post-commit work as a collection of N operations with bounded guarantees. The business logic (commit-stats publishing) is added in follow-up PRs — this PR contains only the framework and its wiring.
Details
New abstractions (
iceberg/openhouse/internalcatalog/.../internal/catalog/):PostCommitOperation— interface (getName(),execute(PostCommitContext));best-effort, interruptible contract that must never affect commit correctness.
PostCommitContext— immutable value object carrying the committedTableIdentifierand
TableMetadata; operations extract whatever they need.PostCommitOperationRunner(@component) owns the execution safety envelope:ThreadPoolExecutorwith a bounded queue;daemon, named threads that time out when idle (an unused runner holds zero threads).
AbortPolicysheds work (drop + metric) instead of blocking thecommitter or growing memory unbounded.
timeoutSchedulercancels/interrupts anyop exceeding the configured timeout, so a slow/hung op cannot pin a worker.
runAll()never throws.submitted / success / failed / timeout / rejectedcounters,tagged by operation name.
cluster.tables.postcommit.{max-threads,queue-capacity,operation-timeout-ms}(defaults
4 / 1000 / 10000ms).Wiring:
OpenHouseInternalCatalogautowires the runner and passes it intonewTableOps.OpenHouseInternalTableOperationsinvokesrunner.runAll(...)only in thedoCommitfinally SUCCESS branch, so operations fire only after a durably successful commit.
A backward-compatible 7-arg constructor keeps the runner optional (null-guarded), so
existing callers/tests are unaffected.
No behavioral change for OSS/dev: those deployments register zero
PostCommitOperationbeans, so
runAllis a no-op.Example operation (illustrative)
Adding post-commit behavior is just a Spring bean — no changes to the catalog or
OpenHouseInternalTableOperations. The runner auto-collects everyPostCommitOperationinto its
List, so presence of the bean is the only switch.The runner handles the hard parts around this bean: bounded async execution, the per-op
timeout (interrupts a stuck publish ), failure isolation, and success/failed/timeout/
rejected metrics tagged name="commit-stats-publish" . The real implementation lands in a
follow-up PR stacked on this one.
Issue] Briefly discuss the summary of the changes made in this
pull request in 2-3 lines.
Changes
For all the boxes checked, please include additional details of the changes made in this pull request.
Testing Done
PostCommitOperationRunnerTest: runs all registered ops, isolates a throwing op,enforces the per-op timeout (interrupts a slow op), drops overflow work when the pool is
saturated (rejected metric), and is a safe no-op for null context / empty op list.
OpenHouseInternalTableOperationsTestcases: post-commit ops run exactly once on asuccessful commit (context carries identity + committed metadata) and do NOT run on a
failed commit.
:iceberg:openhouse:internalcatalog:testsuite passes (Java 17).For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.
Additional Information
For all the boxes checked, include additional details of the changes made in this pull request.