Repository navigation
Description
Apache Iceberg version
0.12.0 (latest release)
Please describe the bug 🐞
Filtered scans raise or return wrong rows at two kinds of edge values. Both bugs reproduce on 0.12.0 and on main (33b4ce5). A third bug, in truncate partitions, is in #4403.
import datetime as dt
import math
import tempfile
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.expressions import EqualTo, In, LessThanOrEqual, NotEqualTo
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import YearTransform
from pyiceberg.types import DateType, FloatType, LongType, NestedField
tmp = tempfile.mkdtemp()
catalog = SqlCatalog("demo", uri=f"sqlite:///{tmp}/catalog.db", warehouse=f"file://{tmp}/warehouse")
catalog.create_namespace("ns")
def table(name, column, spec, values):
schema = Schema(NestedField(1, "id", LongType(), required=True), NestedField(2, "x", column))
t = catalog.create_table(f"ns.{name}", schema=schema, partition_spec=spec)
t.append(pa.Table.from_pylist([{"id": i, "x": v} for i, v in enumerate(values)], schema=schema.as_arrow()))
return t
def ids(t, row_filter):
try:
return sorted(t.scan(row_filter=row_filter).to_arrow()["id"].to_pylist())
except OverflowError as e:
return repr(e)
# 1. year partitions and the last representable date
spec = PartitionSpec(PartitionField(source_id=2, field_id=1000, transform=YearTransform(), name="x_year"))
t = table("year", DateType(), spec, [dt.date(2024, 5, 1), dt.date(9999, 12, 31)])
print(ids(t, LessThanOrEqual("x", dt.date(9999, 12, 31)))) # OverflowError('date value out of range') -- expected [0, 1]
# 2. inf in a float column
t = table("floats", FloatType(), PartitionSpec(), [1.5, math.inf])
print(ids(t, EqualTo("x", math.inf))) # [] -- expected [1]
print(ids(t, NotEqualTo("x", math.inf))) # [0, 1] -- expected [0]
print(ids(t, In("x", {math.inf, 1.5}))) # [0] -- expected [0, 1]1. year and month partitions raise OverflowError at 9999-12-31 and 0001-01-01
LessThanOrEqual and GreaterThan raise at 9999-12-31. GreaterThanOrEqual and LessThan raise at 0001-01-01. Timestamp columns raise the same way at 9999-12-31 23:59:59.999999 and 0001-01-01 00:00:00. EqualTo and In work. day, hour, identity and bucket partitions return the right rows. 9999-12-31 is a common "no end date" value.
Cause. The projections step one day past the bound. The strict projection turns <= 9999-12-31 into < 10000-01-01 (transforms.py#L1066-L1070). The inclusive projection turns > 9999-12-31 into >= 10000-01-01 (transforms.py#L1041-L1046). Both then convert the day count to a Python date (datetime.py#L242-L243), and Python dates stop at year 9999. Java computes years and months with LocalDate (DateTimeUtil.java#L207-L225), which has a much wider range.
2. Filters on inf in a float column return wrong rows
EqualTo and In miss the inf row, and NotEqualTo keeps it. LessThanOrEqual(x, -math.inf) misses the -inf row. Every layout behaves the same way. double columns are not affected.
Cause. A Python float becomes a DoubleLiteral. DoubleLiteral.to(FloatType) returns FloatAboveMax for any value above the float32 maximum, and inf is above it. Binding then turns EqualTo into AlwaysFalse and NotEqualTo into AlwaysTrue. Java has the same conversion (Literals.java#L372-L380). Java engines such as Spark pass a Float for a float column, so they do not take this path. In PyIceberg, every Python float does. inf and -inf are valid float values, so they could convert to FloatLiteral.
Already reported
While probing, I also hit three known issues. I do not repeat them here.
- Visitor and evaluator edge cases can over-prune files or mishandle nulls #3498 item 3, the
TypeErrorfor a null identity partition under anOrfilter, still reproduces onmain. The stale bot closed its fix, Fix residual ordered comparisons for null identity partitions #3817. - [Python][C++][Dataset][Parquet] Row-group statistics drop NaN rows for != and negated comparisons on float columns arrow#51491: Parquet row-group statistics drop NaN rows for
!=andis_nan. - [C++][Python] pyarrow.compute.field("x").isin([0.0]) incorrectly filters out all rows when they all match arrow#46183:
isin([0.0])drops a row group whose values are all0.0.
How I found them
I used Temper to audit each table layout against the same scan guarantees. Every layout holds the same rows. The reference is the same filter written as plain Python over the appended rows. Temper helped identify these bugs. The table shows the result.
| Guarantee | unpartitioned | many files | identity | bucket | truncate | year | month | day | hour |
|---|---|---|---|---|---|---|---|---|---|
| random filters match Python | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| int and long extremes are found | ✅ | ✅ | ✅ | ✅ | ❌ #4403 | ✅ | ✅ | ✅ | ✅ |
| date and time sentinels scan | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ 1 | ❌ 1 | ✅ | ✅ |
float inf is found |
❌ 2 | ❌ 2 | ❌ 2 | ❌ 2 | ❌ 2 | ❌ 2 | ❌ 2 | ❌ 2 | ❌ 2 |
✅ passes. ❌ fails, marked with the bug number. The random row runs 300 seeded filters on each layout, and all of them matched.
The map is MATRIX.md, built from these tests and these declarations.
I can send a PR for each bug, with regression tests in the existing test files. I'm happy to split them into separate issues.
I used an AI assistant to run the tests and draft this issue. I verified each example on 0.12.0 and on main, with pyarrow 26.0.1 and Python 3.12.9.
Willingness to contribute
- I can contribute a fix for this bug independently
- I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- I cannot contribute a fix for this bug at this time
Activity