Add optional time zone to retention policy (backend core)

[mkuchenbecker]

Summary

Adds an optional timeZone to the retention policy. When set, the retention age boundary is evaluated in that zone instead of UTC, applied to the boundary value rather than the column so the delete stays a metadata-only partition drop; a policy with no zone is unchanged. This PR is the OpenHouse backend core, opened as a draft; the SQL AT TIME ZONE grammar and the li-openhouse orchestration are follow-ups. The full design is below.

Design

Overview

Today retention evaluates its age boundary only in UTC. This change adds an optional time zone: when set, retention evaluates the boundary in that zone, and a policy with no zone is unchanged.

Requirements

The time zone is a requirement input; its business rationale is out of scope.

Tier	Requirement
Must	A policy may set a time zone: an IANA id such as America/Los_Angeles, or a fixed offset such as +05:30.
Must	Retention evaluates the boundary in that zone, using the zone's offset for the boundary date.
Must	Retention stays a metadata-only partition drop; no zone value causes a row-level rewrite.
Must	Any approximation retains more data, never less.
Must	No zone reproduces today's behavior.
Should	The boundary is exact where the partitioning allows.
Won't	Exact-to-the-second boundaries for native timestamp tables.
Won't	Per-partition zones, or new granularities.
Out of scope	Why a caller wants local-time retention.

Behavior

Set a zone with an AT TIME ZONE clause, with or without a column pattern:

ALTER TABLE db.t SET POLICY (RETENTION=30D AT TIME ZONE 'America/Los_Angeles');

ALTER TABLE db.t SET POLICY (RETENTION=30D AT TIME ZONE 'America/Los_Angeles')
  ON COLUMN datepartition WHERE PATTERN='yyyy-MM-dd';

The boundary is the start of the current period in the zone, moved back by the count: truncate the current time to the retention granularity, then subtract count periods. Retention keeps rows at or after the boundary and deletes rows before it, so the boundary is inclusive on the kept side and exclusive on the deleted side.

The kept window is the count most recent complete periods plus the current in-progress period, so it spans between count and count+1 periods. With 2-day retention one minute after local midnight, the current day is nearly empty, so retention keeps close to two full days; twelve hours later it keeps two days plus the half day so far. With hourly retention, each run that crosses a local hour advances the boundary by one hour and deletes the oldest kept hour.

Accuracy at the boundary depends on the column:

Column	Accuracy
String pattern	Exact.
Native timestamp	Snapped to the partition edge: keeps up to one extra partition, never deletes early.

The boundary uses the zone's offset in effect on the boundary date, so an IANA zone tracks daylight saving and a fixed offset does not.

To adopt, add the clause to an existing policy. Nothing else changes.

Decisions that affect behavior

The zone applies to the boundary value, not the column; the native-timestamp boundary snaps to a partition edge. Both keep retention metadata-only, which the rejected alternatives do not.

Decision	Rejected alternative and its cost
Zone on the boundary value	Zone on the column: every run rewrites the whole table.
Native boundary snapped to a partition edge	Exact local instant: every run rewrites the boundary partition.

Worked example

A daily table keeps 30 days in America/Los_Angeles, run at 2024-02-01T02:00Z, which is still January 31 in the zone. The boundary is local January 1 midnight, 2024-01-01T08:00Z, snapped to 2024-01-01T00:00Z, so the 2024-01-01 partition is kept. In UTC the run reads as February 1 and would drop it.

Changes

• Client-facing API Changes
• Internal API Changes
• Bug Fixes
• New Features
• Performance Improvements
• Code Style
• Refactoring
• Documentation
• Tests

New Features: an optional timeZone on the retention policy; the retention job evaluates its age boundary in that zone. Internal API Changes: SparkJobUtil.createDeleteStatement and createDeleteFilter, Operations.runRetention, RetentionSparkApp (--timeZone), RetentionConfig, TablesClient, and TableRetentionTask gain a timeZone parameter or field. Tests: added zone-aware unit tests; existing UTC tests are unchanged.

Testing Done

• Added new tests for the changes made.
• Updated existing tests to reflect the changes made.

Java 17 unit tests pass: SparkJobUtilTest (9, including the native snap, the string anchor, the Iceberg filter micros, and a fractional-hour zone), RetentionPolicySpecValidatorTest (new time-zone validation plus existing), AppsTest, and TableRetentionTaskTest. A blank zone reproduces today's SQL and Iceberg expression byte for byte, so existing behavior is unchanged.

Additional Information

• Breaking Changes
• Deprecations
• Large PR broken into smaller PRs, and PR plan linked in the description.

This feature is delivered in layers: this PR is the OpenHouse backend core, followed by the SQL AT TIME ZONE grammar in the Spark extensions, then the li-openhouse orchestration (the LinkedIn retention app, Central Policy Store mapping, and emitted stats).

🤖 Generated with GitHub Copilot CLI

[mkuchenbecker]

Time zone should require a SQL conf be set to enable for replicaiotn time zone support. This avoids an invalid time from impacting an existing table.

[mkuchenbecker]

Superseded by #732. Recreated with the head branch on linkedin/openhouse instead of the personal fork.