Repository navigation
Decode 4-byte bounds of date columns promoted to timestamp - #4409
Open
rahulsmahadev wants to merge 1 commit into
Open
rahulsmahadev wants to merge 1 commit into
rahulsmahadev wants to merge 1 commit into
Conversation
Bounds written before a date to timestamp or timestamp_ns promotion keep their 4-byte day count, but from_bytes used the 8-byte decoder for these types, so planning and inspect raised struct.error. Scale the day count to micros or nanos and return the exact int, so metrics comparisons stay exact (metadata tables can still raise OverflowError for bounds outside the Arrow range). Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
|
@rambleraptor, this extends the 4-byte bound fallback you added in #3293 to date -> timestamp, and @avison9, you added the inspect coverage in #4314. Could you take a look when you have a chance? |
Contributor
|
Thanks for the ping! I pulled this locally and it looks good to me.
A few non-blocking suggestions from me regarding your fix:
LGTM otherwise. avison9 |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Iceberg v3 allows
dateto be promoted totimestampandtimestamp_ns. The promotion does not rewrite thebounds of existing data files, so a bound written before the promotion is still a 4-byte day count from
1970-01-01. The spec says readers must infer the write-time type from the length of the bound: in the
"Schema Evolution" section of
format/spec.md, a 4-bytetimestamportimestamp_nsbound is adate.from_bytesregistersTimestampTypeandTimestampNanoTypewith the 8-byte decoder, so such a bound fails:On a v3 table with retained date bounds after promotion to
timestamp,tbl.scan(row_filter=GreaterThan("d", "2022-01-01T00:00:00")).plan_files()raises thisstruct.errorwhen evaluating those bounds.tbl.inspect.files()raises it after promotion to eithertimestamportimestamp_ns.The fix decodes a 4-byte bound of
TimestampTypeandTimestampNanoTypeas a signed day count, scaled to microseconds (x 86,400,000,000) or nanoseconds (x 86,400,000,000,000), and returns the exact Python int. The 8-byte path is unchanged. The fallback decodes 4-byte bounds; unsupported lengths still raisestruct.error. The change adds no per-file containers or caches.TimeType,TimestamptzTypeandTimestamptzNanoTypekeep their decoder, because the spec does not allowdateto be promoted to them. The fix is infrom_bytes, the one decoder that the inclusive and strict metrics evaluators andinspectall use, so no call site changes.Planning compares such bounds exactly, because Python ints have no int64 limit. Metadata tables are different: converting a bound outside the Arrow range to a timestamp raises
OverflowError. For date bounds promoted totimestamp_ns, representable midnight dates run from 1677-09-22 through 2262-04-11 inclusive; metadata inspection raisesOverflowErrorfor dates outside that interval. The spec's runtime failure rule (Schema Evolution) applies to promoted data values. It does not apply to bounds, because the bounds rules only require a lower bound at or below all values and an upper bound at or above all values, so a bound can be wider than the data (format/spec.md, the data file fieldslower_boundsandupper_bounds). An out-of-range bound therefore does not show that a data value overflows.This PR follows the spec rather than Java. Java's
Conversions.fromByteBuffer(
api/src/main/java/org/apache/iceberg/types/Conversions.java, lines 171-179) returns the raw day count for a4-byte
TIMESTAMPorTIMESTAMP_NANObound without scaling it. Comparing that count with a microsecond literalcould prune a file that has matching rows.
Alternatives considered: B, raise
ValueErrorfor bounds that do not fit in int64. That fails planning or inspection even when every row is representable, because a bound need not occur in the data. C, treat such bounds as unknown, or leave them unavailable in metadata tables. That avoids the errors, but it changes the evaluator and metadata contracts, so it is left for a follow-up. Actual-value overflow checks belong to date-promotion support.This extends #3293, which added the same kind of 4-byte fallback for int to long and float to double bounds, and #4314, which added the inspect test for those types (it closes #3744). Neither covers date to timestamp.
Out of scope: normal schema promotion and reading date data files as timestamps remain unsupported (related closed PR #2586). Comparison filters on
timestamp_nsremain unsupported for string, integer and datetime literals (#4078).Are these changes tested?
Yes.
test_from_bytes_date_bound_promoted_to_timestampintests/test_conversions.pychecks exact micros and nanos for 4-byte day counts, including a date before 1970, and checks that the 8-byte encoding is unchanged.test_from_bytes_date_bound_promoted_to_timestamp_beyond_int64checks the exact scaled value for the first day count outside the int64 range, in both directions.test_plan_and_inspect_after_date_to_timestamp_promotionintests/expressions/test_evaluator.pyuses the public API with each catalog fixture. Two filters plan exactly the expected data file each, andinspect.files()returns the exact bounds. It simulates the promotion withupdate_schema(allow_incompatible_changes=True), because PyIceberg cannot write v3 metadata yet (#1551), and the bound decoder does not depend on the format version. The new rows intest_inclusive_metrics_eval_bounds_after_promotionandtest_strict_metrics_eval_bounds_after_promotioncover both metrics evaluators, including a day count before the epoch. The strict row compares against a literal after the epoch, so a missing scale fails it.Without this change, the new tests fail with
struct.error: unpack requires a buffer of 8 bytes, including the beyond-int64 test. With this change, they pass.Are there any user-facing changes?
Yes. Filtered file planning for
timestampand metadata inspection fortimestampandtimestamp_nsnow decode retained 4-byte date bounds. This does not add date-to-timestamp data reading or nanosecond literal binding. Please add thechangeloglabel.This pull request and its description were written by Isaac.