Skip to content

truncate partitions lose rows near the minimum int or long #4403

Open

Description

@turbolytics

Apache Iceberg version

0.12.0 (latest release)

Please describe the bug 🐞

A truncate partition loses rows whose value is near the minimum int or long. EqualTo, In and range filters such as LessThan(x, 0) all miss them. It reproduces on 0.12.0 and on main (33b4ce5).

import tempfile

import pyarrow as pa

from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.expressions import EqualTo, LessThan
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import TruncateTransform
from pyiceberg.types import IntegerType, LongType, NestedField

tmp = tempfile.mkdtemp()
catalog = SqlCatalog("demo", uri=f"sqlite:///{tmp}/catalog.db", warehouse=f"file://{tmp}/warehouse")
catalog.create_namespace("ns")

schema = Schema(NestedField(1, "id", LongType(), required=True), NestedField(2, "x", IntegerType()))
spec = PartitionSpec(PartitionField(source_id=2, field_id=1000, transform=TruncateTransform(1000), name="x_trunc"))
t = catalog.create_table("ns.trunc", schema=schema, partition_spec=spec)
rows = [{"id": 0, "x": -(2**31)}, {"id": 1, "x": -(2**31) + 100}, {"id": 2, "x": 5}]
t.append(pa.Table.from_pylist(rows, schema=schema.as_arrow()))


def ids(row_filter):
    return sorted(t.scan(row_filter=row_filter).to_arrow()["id"].to_pylist())


print(sorted(task.file.partition[0] for task in t.scan().plan_files()))  # [0, 2147483296]
print(ids(EqualTo("x", -(2**31))))  # [] -- expected [0]
print(ids(EqualTo("x", -(2**31) + 100)))  # [] -- expected [1]
print(ids(LessThan("x", 0)))  # [] -- expected [0, 1]

For truncate[1000] on an int, every value from -2147483648 to -2147483001 is affected. A long column behaves the same way near -(2**63).

Cause. The writer computes the partition value with pyiceberg-core. Its math wraps around, as Java's TruncateUtil.truncateInt does. So these rows land in partition 2147483296, a large positive value. Scan planning projects the filter with the Python transform, v - v % self._width, which does not wrap. EqualTo(x, -2147483648) projects to -2147484000, so it never matches the stored value. LessThan(x, 0) projects to x_trunc <= -1000, and the stored value is positive. Either way, the scan prunes the file.

What a fix could look like.

I'm not sure of the right fix for range filters. It may need a note in the spec. I can raise it in apache/iceberg if that helps.

How I found it

I used Temper to audit each table layout against the same scan guarantees. The reference is the same filter written as plain Python over the appended rows. Temper helped identify this bug. In MATRIX.md, only the truncate layout fails "int and long extremes are found". The tests are here.

Once the approach is agreed, I can send a PR for EqualTo and In, with regression tests in the existing test files.

I used an AI assistant to run the tests and draft this issue. I verified the example on 0.12.0 and on main, with pyarrow 26.0.1 and Python 3.12.9.

Willingness to contribute

  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions