Skip to content

Keep rows in truncate partitions near the minimum int or long - #4407

Open
Anurag-S1ngh wants to merge 1 commit into
apache:mainfrom
Anurag-S1ngh:fix/4403-truncate-min-wrap
Open

Anurag-S1ngh wants to merge 1 commit into
apache:mainfrom
Anurag-S1ngh:fix/4403-truncate-min-wrap

Conversation

@Anurag-S1ngh

Copy link
Copy Markdown
Contributor

Summary

The writer, like Java, truncates with two's complement arithmetic, so values from the minimum of the type wrap to a partition value at the top of the range. The Python transform did not wrap, so scan planning projected a different partition, pruned the file, and lost those rows for EqualTo, In, and range filters.

Changes

  • Wrap the Python truncate transform into the range of the source type, which fixes EqualTo and In.
  • Add the wrapped partition value to the inclusive projection of ranges that can match it. Greater-than filters use the first partition after the block that holds the minimum, because the boundary itself wraps.
  • Skip the inclusive projection in the residual evaluator when a transform projects to an expression instead of a single predicate. Residuals are an optimization, so falling back to the original predicate is safe.

The writer output is unchanged, matching Java and existing tables.

Testing

  • Added tests for the transform at the type minimum, the projections near the minimum, and an end-to-end scan of a truncate[1000] int partition with EqualTo, LessThan, and GreaterThan at the minimum.
  • make test passes (4695 passed, 5 skipped).

Closes #4403

The writer, like Java, truncates with two's complement arithmetic, so the
values from the minimum of the type wrap to a partition value at the top of
the range. The Python transform did not wrap, so scan planning projected a
different partition and pruned the file, losing those rows for EqualTo, In,
and range filters.

Wrap the Python transform into the range of the source type, and add the
wrapped partition value to the inclusive projection of ranges that can
match it. Greater-than filters need the first partition value after the
block that holds the minimum, because the boundary itself wraps. Skip the
inclusive projection in the residual evaluator when a transform projects to
an expression instead of a single predicate.
@Fokko
Fokko self-requested a review November 17, 2026 17:28

@rahulsmahadev rahulsmahadev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this. I ran it locally against main (33b4ce5):

  • The new and changed tests fail on main and pass here.
  • I swept =, !=, <, <=, >, >=, IN and NOT IN at both ends of int and long, with truncate[1000] and truncate[7], comparing to_arrow() and count() with the expected rows: 77 wrong results on main, none here.
  • It also fixes add_files for such a file. On main, adding a Parquet file with x = -2**31 to this spec fails with struct.error: 'i' format requires -2147483648 <= number <= 2147483647; here the file gets partition 2147483296, the value the writer uses. Maybe worth a line in the description.

One problem I think should be fixed here: the strict projection still assumes that truncate partitions sort like the source values, and ResidualEvaluator trusts it, so count() is wrong for files without column bounds (for example with write.metadata.metrics.default=counts):

  • New with this change: transform(MIN + 100) is now 2147483296, so the strict projection of x < MIN + 100 is x_trunc < 2147483296. That holds for every ordinary partition, so count() adds their rows.
  • Already on main: the strict projection of x > 0 is x_trunc > 0, which holds for the wrapped partition, so count() adds the rows near the minimum.
import tempfile

import pyarrow as pa

from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.expressions import GreaterThan, LessThan
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import TruncateTransform
from pyiceberg.types import IntegerType, NestedField

tmp = tempfile.mkdtemp()
catalog = SqlCatalog("demo", uri=f"sqlite:///{tmp}/catalog.db", warehouse=f"file://{tmp}/warehouse")
catalog.create_namespace("ns")
schema = Schema(NestedField(1, "x", IntegerType()))
spec = PartitionSpec(PartitionField(source_id=1, field_id=1000, transform=TruncateTransform(1000), name="x_trunc"))
t = catalog.create_table("ns.t", schema=schema, partition_spec=spec, properties={"write.metadata.metrics.default": "counts"})
t.append(pa.table({"x": pa.array([-(2**31), 5], pa.int32())}))

for row_filter in [LessThan("x", -(2**31) + 100), GreaterThan("x", 0)]:
    scan = t.scan(row_filter=row_filter)
    print(len(scan.to_arrow()), scan.count())  # expected 1 1 for both; this PR prints 1 2 for both, main 0 0 and 1 2

to_arrow() is right in both cases because it re-applies the row filter. For < and <= with a boundary in the block of the minimum there is no safe strict projection, so strict_project could return None there; for > and >= it needs to exclude the wrapped value, or also return None. A count() assert on a table with write.metadata.metrics.default=counts would cover both.

Smaller things:

  • Java: TruncateUtil.truncateInt wraps the same way, but ProjectionUtil.truncateInteger does not add the wrapped partition, so Java has the same gap for < and >. It would help to say in the description that the range projection goes beyond Java.
  • #4401, #4402, #4406 and this PR all add tests right after test_append_to_table_bucketed_by_time and edit the same import block in tests/io/test_pyarrow.py, so each pair conflicts. Placing each test next to related tests would let them merge in any order.

Comment thread pyiceberg/transforms.py
if boundary + 1 < next_block_start:
return Or(GreaterThanOrEqual(Reference(name), next_block_start), EqualTo(Reference(name), wrapped_minimum))
return _truncate_number(name, pred, transform)
elif isinstance(pred, BoundGreaterThanOrEqual):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No test covers this branch: with if boundary < next_block_start: replaced by if False:, every test still passes, although x >= MIN then skips all ordinary partitions. An assert next to the GreaterThan one in test_int_truncate_projection_near_minimum would cover it:

assert project(GreaterThanOrEqual("id", -(2**63) + 1)) == Or(
    GreaterThanOrEqual("id_trunc", -9223372036854775800), EqualTo("id_trunc", wrapped_minimum)
)

That test projects the LongType id column, so an IntegerType case would help too; only the scan test covers int.

inclusive_projection = part.transform.project(part.name, predicate)
inclusive_result = None
if inclusive_projection is not None:
if isinstance(inclusive_projection, UnboundPredicate):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TruncateTransform.project() can now return an Or, but Transform.project(), TruncateTransform.project() and _truncate_integer_projection() are still annotated UnboundPredicate | None, and this guard shows that callers relied on that. Widening the annotations to BooleanExpression | None would let mypy point at callers like this one.

#4278 rewrites this block into _eval_projection(projection: UnboundPredicate | None), which calls .bind() on the projection, so whichever of the two PRs merges second needs this guard there too.

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Anurag-S1ngh! Great catch, this is a subtle one: the writer (like Java) wraps the truncate of the type minimum via two's complement, so the partition value sits at the top of the range and scan planning silently pruned those files.

I pulled the branch and verified:

  • tests/test_transforms.py, tests/expressions/test_projection.py, tests/utils/test_datetime.py, tests/expressions/test_literals.py all pass
  • Reproduced the end-to-end scenario: TruncateTransform(1000) on an int column with value INT_MIN, scanned with EqualTo, In, LessThan, GreaterThan: correct rows come back
  • Checked the wrap math against Java's floorMod semantics for int32/int64 at several widths, all consistent
  • Confirmed the isinstance(inclusive_projection, UnboundPredicate) guard in visitors.py is needed and safe: the new Or(...) projections are BooleanExpressions, not predicates, and skipping the inclusive re-check there just falls back to the original predicate (which is correct, only less optimal)

One nit, non-blocking: in _truncate_integer_projection (pyiceberg/transforms.py, around line 1048), the not isinstance(pred.literal, LongLiteral) guard falls back to _truncate_number. Since bound int/long predicates always carry LongLiteral literals in practice, that branch is mostly defensive. If a DecimalLiteral ever landed here, pred.literal.value would be a Decimal and the comparison would still work, so no action needed, just noting the fallback covers it.

CI is fully green. LGTM!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

truncate partitions lose rows near the minimum int or long

3 participants