Repository navigation
Fix scan filters at the edges of the date and float ranges - #4406
Anurag-S1ngh wants to merge 1 commit into
Conversation
Year and month partitions raised OverflowError when a filter projected past 9999-12-31 or before 0001-01-01, because the transform converted the day count to a datetime.date. Compute years and months from the day count with integer arithmetic, so projections just past the last representable date produce year 10000 and match no partition instead of raising. A Python float became a DoubleLiteral, and converting that to a float column turned inf into FloatAboveMax or FloatBelowMin, which bound the predicate to AlwaysFalse or AlwaysTrue. inf is a valid float value, so convert it to a FloatLiteral like any other float that fits.
rahulsmahadev
left a comment
There was a problem hiding this comment.
Thanks! I ran this locally against main (33b4ce5):
- The new tests fail on main and pass here (the parametrized cases inside the date range pass on both, as expected).
_civil_from_daysmatchesdatetime.datefor every day from 0001-01-01 to 9999-12-31, so results in that range are unchanged.- The float change also fixes deletes and range filters with an infinity. On main,
x < infbinds toAlwaysTrue, sotable.delete(LessThan("x", math.inf))on a float column deletes every row, including the inf, NaN and null rows; here it keeps exactly those three.<,<=,>and>=with an infinity all change this way, so they are worth listing in the description, with a test.
The one bug I found is in the negative-era branch of _civil_from_days (inline comment). A few more points:
- Java's
DoubleLiteral.to(FLOAT)inLiterals.javahas the same range check, so Java bindsfloat_col = <double infinity>toalwaysFalseas well. Worth saying in the description that this differs from Java. StringLiteral.to(FloatType)applies the same check, soEqualTo("x", "Infinity")still returns no rows andIn("x", ["Infinity", "1.5"])only1.5. Fine as a follow-up, but it is the same bug.- Nit:
2932897 * day_nanosintest_years_and_months_beyond_the_date_rangeis beyond int64, so notimestamp_nscan hold it (they end in 2262); that assert pins an unreachable case. - This conflicts with #4407 (and #4401, #4402) in
tests/io/test_pyarrow.py, since they all add tests right aftertest_append_to_table_bucketed_by_timeand edit the same import block.
| https://howardhinnant.github.io/date_algorithms.html#civil_from_days. | ||
| """ | ||
| z = days + 719468 | ||
| era = (z if z >= 0 else z - 146096) // 146097 |
There was a problem hiding this comment.
This is the C++ trick for floor division, but Python's // already floors, so for z < 0 (days before 0000-03-01) era comes out one too small. About 3% of those days get the wrong month, and some the wrong year. For example, -0001-12-31 (days=-719529) gives year -1970 and month -23640 instead of -1971 and -23641, while the writer puts that date in year partition -1971. So EqualTo("d", -719529) on a year-partitioned table projects to d_year == -1970 and prunes the file that holds the row; main raises OverflowError there instead.
| era = (z if z >= 0 else z - 146096) // 146097 | |
| era = z // 146097 |
With this change the function matches datetime on the whole representable range and gives the same values as Java's DateTimeUtil.daysToYears / daysToMonths before year 1, and the tests in this PR still pass. None of the new test days has z < 0; adding (-719529, -1971, -23641) to test_days_to_years_and_months would pin it.
Fokko
left a comment
There was a problem hiding this comment.
Thanks @Anurag-S1ngh! Two real user-facing bugs fixed here, and the 9999-12-31 "no end date" case is exactly the kind of value that shows up in production data.
I pulled the branch and verified:
tests/test_transforms.py,tests/expressions/test_projection.py,tests/utils/test_datetime.py,tests/expressions/test_literals.pyall pass- Brute-forced
_civil_from_daysagainst Python'sdatetimeover 20k random days in the valid range: identical year/month everywhere - Compared old vs new
days_to_months/micros_to_months/nanos_to_*over 5k random samples: results identical inside the representable range, so no behavior change for existing data - Reproduced both scan scenarios end to end:
EqualTo("x", inf)on a float column now returns the right row, and a year-partitioned table with9999-12-31scans correctly withLessThanOrEqualandGreaterThan - Checked pre-epoch timestamps (e.g. 1969-12-31 23:59:59.999999): months/years come out the same as before
One small question, non-blocking: in pyiceberg/utils/datetime.py, micros_to_months(micros) now does micros // 86_400_000_000 first. Since that's floor division, negative micros (pre-1970) floor toward negative infinity, which matches the old day-boundary behavior. Just double-checked it, all good.
CI is fully green. LGTM!
Summary
Filtered scans raised or returned wrong rows at two kinds of edge values.
OverflowErrorwhen a filter projected past 9999-12-31 or before 0001-01-01, because the transform converted the day count to adatetime.date. 9999-12-31 is a common "no end date" value.EqualTo,In, andNotEqualTooninfreturned wrong rows for float columns. A Python float becomes aDoubleLiteral, and converting that to a float column turnedinfintoFloatAboveMaxorFloatBelowMin, which bound the predicate toAlwaysFalseorAlwaysTrue.Changes
infand-infto aFloatLiteralfor float columns, like any other float that fits.Testing
inf/In filters on a float column and a year-partitioned table scanned at 9999-12-31.make testpasses (4703 passed, 5 skipped).Closes #4404