Skip to content

fix: preserve millisecond precision for UNIX_TIMESTAMP sort keys#369

Open
piket wants to merge 2 commits into
masterfrom
fix/EAPC-22316-sorted-fv-timestamp-precision
Open

fix: preserve millisecond precision for UNIX_TIMESTAMP sort keys#369
piket wants to merge 2 commits into
masterfrom
fix/EAPC-22316-sorted-fv-timestamp-precision

Conversation

@piket

@piket piket commented Jun 25, 2026

Copy link
Copy Markdown
Collaborator

SortedFeatureView rows whose UNIX_TIMESTAMP sort keys differed by less than one second were being assigned the same sort key score in Redis/Valkey (wrong ordering) and the same Cassandra clustering key (row overwrites).

Root cause: _python_datetime_to_int_timestamp() truncated all UNIX_TIMESTAMP values to integer seconds before storing in unix_timestamp_val. The * 1000 multiplications in zset_score() attempted ms precision but had no effect.

Fix:

  • Add _python_datetime_to_int_ms_timestamp() for millisecond conversion.
  • In _convert_arrow_fv_to_proto(), detect UNIX_TIMESTAMP sort key columns and use the ms function for those columns only; all other UNIX_TIMESTAMP columns remain at second precision (backward compatible).
  • Remove the * 1000 from zset_score() in Redis, Valkey, and Cassandra — unix_timestamp_val is now already ms for sort key columns.
  • Add a threshold discriminator (val > 1e11) in feast_value_type_to_python_type() so sort key ms values are correctly read back as datetimes without affecting the seconds interpretation for regular feature columns.

What this PR does / why we need it:

Which issue(s) this PR fixes:

Misc

piket and others added 2 commits June 25, 2026 13:15
…C-22316)

SortedFeatureView rows whose UNIX_TIMESTAMP sort keys differed by less
than one second were being assigned the same sort key score in Redis/Valkey
(wrong ordering) and the same Cassandra clustering key (row overwrites).

Root cause: _python_datetime_to_int_timestamp() truncated all UNIX_TIMESTAMP
values to integer seconds before storing in unix_timestamp_val. The * 1000
multiplications in zset_score() attempted ms precision but had no effect.

Fix:
- Add _python_datetime_to_int_ms_timestamp() for millisecond conversion.
- In _convert_arrow_fv_to_proto(), detect UNIX_TIMESTAMP sort key columns
  and use the ms function for those columns only; all other UNIX_TIMESTAMP
  columns remain at second precision (backward compatible).
- Remove the * 1000 from zset_score() in Redis, Valkey, and Cassandra —
  unix_timestamp_val is now already ms for sort key columns.
- Add a threshold discriminator (val > 1e11) in feast_value_type_to_python_type()
  so sort key ms values are correctly read back as datetimes without affecting
  the seconds interpretation for regular feature columns.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant