Skip to content

Decode 4-byte bounds of date columns promoted to timestamp - #4409

Open
rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/promoted-timestamp-bounds
Open

rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/promoted-timestamp-bounds

Conversation

@rahulsmahadev

Copy link
Copy Markdown
Contributor

Rationale for this change

Iceberg v3 allows date to be promoted to timestamp and timestamp_ns. The promotion does not rewrite the
bounds of existing data files, so a bound written before the promotion is still a 4-byte day count from
1970-01-01. The spec says readers must infer the write-time type from the length of the bound: in the
"Schema Evolution" section of format/spec.md, a 4-byte timestamp or timestamp_ns bound is a date.

from_bytes registers TimestampType and TimestampNanoType with the 8-byte decoder, so such a bound fails:

from pyiceberg.conversions import from_bytes, to_bytes
from pyiceberg.types import DateType, TimestampType

from_bytes(TimestampType(), to_bytes(DateType(), 18793))  # 2021-06-15
# on main: struct.error: unpack requires a buffer of 8 bytes
# with this change: 1623715200000000

On a v3 table with retained date bounds after promotion to timestamp, tbl.scan(row_filter=GreaterThan("d", "2022-01-01T00:00:00")).plan_files() raises this struct.error when evaluating those bounds. tbl.inspect.files() raises it after promotion to either timestamp or timestamp_ns.

The fix decodes a 4-byte bound of TimestampType and TimestampNanoType as a signed day count, scaled to microseconds (x 86,400,000,000) or nanoseconds (x 86,400,000,000,000), and returns the exact Python int. The 8-byte path is unchanged. The fallback decodes 4-byte bounds; unsupported lengths still raise struct.error. The change adds no per-file containers or caches. TimeType, TimestamptzType and TimestamptzNanoType keep their decoder, because the spec does not allow date to be promoted to them. The fix is in from_bytes, the one decoder that the inclusive and strict metrics evaluators and inspect all use, so no call site changes.

Planning compares such bounds exactly, because Python ints have no int64 limit. Metadata tables are different: converting a bound outside the Arrow range to a timestamp raises OverflowError. For date bounds promoted to timestamp_ns, representable midnight dates run from 1677-09-22 through 2262-04-11 inclusive; metadata inspection raises OverflowError for dates outside that interval. The spec's runtime failure rule (Schema Evolution) applies to promoted data values. It does not apply to bounds, because the bounds rules only require a lower bound at or below all values and an upper bound at or above all values, so a bound can be wider than the data (format/spec.md, the data file fields lower_bounds and upper_bounds). An out-of-range bound therefore does not show that a data value overflows.

This PR follows the spec rather than Java. Java's Conversions.fromByteBuffer
(api/src/main/java/org/apache/iceberg/types/Conversions.java, lines 171-179) returns the raw day count for a
4-byte TIMESTAMP or TIMESTAMP_NANO bound without scaling it. Comparing that count with a microsecond literal
could prune a file that has matching rows.

Alternatives considered: B, raise ValueError for bounds that do not fit in int64. That fails planning or inspection even when every row is representable, because a bound need not occur in the data. C, treat such bounds as unknown, or leave them unavailable in metadata tables. That avoids the errors, but it changes the evaluator and metadata contracts, so it is left for a follow-up. Actual-value overflow checks belong to date-promotion support.

This extends #3293, which added the same kind of 4-byte fallback for int to long and float to double bounds, and #4314, which added the inspect test for those types (it closes #3744). Neither covers date to timestamp.

Out of scope: normal schema promotion and reading date data files as timestamps remain unsupported (related closed PR #2586). Comparison filters on timestamp_ns remain unsupported for string, integer and datetime literals (#4078).

Are these changes tested?

Yes. test_from_bytes_date_bound_promoted_to_timestamp in tests/test_conversions.py checks exact micros and nanos for 4-byte day counts, including a date before 1970, and checks that the 8-byte encoding is unchanged. test_from_bytes_date_bound_promoted_to_timestamp_beyond_int64 checks the exact scaled value for the first day count outside the int64 range, in both directions. test_plan_and_inspect_after_date_to_timestamp_promotion in tests/expressions/test_evaluator.py uses the public API with each catalog fixture. Two filters plan exactly the expected data file each, and inspect.files() returns the exact bounds. It simulates the promotion with update_schema(allow_incompatible_changes=True), because PyIceberg cannot write v3 metadata yet (#1551), and the bound decoder does not depend on the format version. The new rows in test_inclusive_metrics_eval_bounds_after_promotion and test_strict_metrics_eval_bounds_after_promotion cover both metrics evaluators, including a day count before the epoch. The strict row compares against a literal after the epoch, so a missing scale fails it.

Without this change, the new tests fail with struct.error: unpack requires a buffer of 8 bytes, including the beyond-int64 test. With this change, they pass.

Are there any user-facing changes?

Yes. Filtered file planning for timestamp and metadata inspection for timestamp and timestamp_ns now decode retained 4-byte date bounds. This does not add date-to-timestamp data reading or nanosecond literal binding. Please add the changelog label.

This pull request and its description were written by Isaac.

Bounds written before a date to timestamp or timestamp_ns promotion keep their 4-byte day
count, but from_bytes used the 8-byte decoder for these types, so planning and inspect
raised struct.error. Scale the day count to micros or nanos and return the exact int, so
metrics comparisons stay exact (metadata tables can still raise OverflowError for bounds
outside the Arrow range).

Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

@rambleraptor, this extends the 4-byte bound fallback you added in #3293 to date -> timestamp, and @avison9, you added the inspect coverage in #4314. Could you take a look when you have a chance?

@avison9

avison9 commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Thanks for the ping! I pulled this locally and it looks good to me.

  • Confirmed the decoding matches the spec's bound-inference table (timestamp/timestamp_ns + 4 bytes → date), and that TimeType/TimestamptzType/TimestamptzNanoType are correctly left on the 8-byte path, since the spec doesn't allow promoting date to them.
  • Reverting the conversions.py change makes all 8 new tests fail with the original struct.error. Keeping the 4-byte branch but dropping the * _MICROS_PER_DAY / * _NANOS_PER_DAY scale also fails 7 of them, including the strict-evaluator row, so the scaling is properly covered.
  • Full unit suite passes and prek is clean on the changed files.

A few non-blocking suggestions from me regarding your fix:

  1. timestamp_ns through inspect: the description says inspect.files() now works after promotion to timestamp_ns, and I confirmed it does locally, but the end-to-end test only covers TimestampType. Could you parametrize the inspect.files() part over TimestampNanoType too? Planning can stay TimestampType-only because of timestamp_ns / timestamptz_ns: filter + pruning #4078.
  2. Test location: test_plan_and_inspect_after_date_to_timestamp_promotion mostly exercises inspect plus scan planning against a catalog, so it might fit better next to test_inspect_bounds_after_type_promotion in tests/table/test_inspect.py than in the evaluator unit tests. Up to you and the maintainers.
  3. Nit: the two new tests in tests/test_conversions.py loop over a cases list. The rest of that file uses @pytest.mark.parametrize, which also reports which case failed.

LGTM otherwise.

avison9

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Metadata inspection APIs fail with struct.error after int→long / float→double type promotion

2 participants