Skip to content

Allow a retention time zone on native timestamp columns - #760

Merged
mkuchenbecker merged 1 commit into
mkuchenbecker/retention-timezone-supportfrom
mkuchenbecker/retention-timezone-native-column
Sep 22, 2026
Merged

mkuchenbecker merged 1 commit into
mkuchenbecker/retention-timezone-supportfrom
mkuchenbecker/retention-timezone-native-column

Conversation

@mkuchenbecker

Copy link
Copy Markdown
Collaborator

Stacked on #732 (mkuchenbecker/retention-timezone-support); review that PR first.

What this changes

Native timestamp retention columns hold a wall clock in an arbitrary producer zone, not UTC. This lets the retention time zone declare that zone, the same way it already declares the zone of a string label. Before this change, a zone on a time-partitioned (native timestamp) table was rejected.

  • The validator rejects a zone only when a string column's pattern already encodes a zone. It accepts a zone on a native timestamp column and on a zone-free string pattern.
  • No delete-SQL change: the app builds now in the declared zone, and the native date_trunc predicate already reads that zoned wall clock, so the cutoff is evaluated in the column's zone.

Known limitation

With backup enabled, the Iceberg backup filter compares an absolute instant while the native delete compares a wall clock, so for a native column with a non-UTC zone the two can differ by the offset. Backup is a secondary path and metadata-only is not a hard requirement, so this is left as a follow-up rather than reintroducing a computed cutoff.

Tests

  • Validator tests accept a zone on a native table and still reject a zone-bearing pattern.
  • A SparkJobUtil test pins that the native statement interpolates the zoned wall clock.
  • The no-zone regression suite is unchanged, so the UTC contract is unaffected.

Native timestamp columns hold a wall clock in an arbitrary producer zone, not
UTC, so the retention time zone declares that zone, the same way it declares the
zone of a string label. The validator no longer rejects a zone on a
time-partitioned (native timestamp) table; it rejects a zone only when a string
column's pattern already encodes one.

The delete SQL needs no change: the app builds now in the declared zone, and the
native date_trunc predicate already reads that zoned wall clock, so the cutoff is
evaluated in the column's zone. A SparkJobUtil test pins that the native
statement interpolates the zoned wall clock, and the validator tests now accept a
zone on a native table.

Known limitation: with backup enabled, the Iceberg backup filter compares an
absolute instant while the native delete compares a wall clock, so for a native
column with a non-UTC zone the two can differ by the offset. Backup is a
secondary path, so this is left as a follow-up rather than reintroducing a
computed cutoff.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mkuchenbecker
mkuchenbecker merged commit fa1cc67 into mkuchenbecker/retention-timezone-support Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant