Skip to content

Build: Use stdlib compression.zstd on Python 3.14+ and build CPython 3.15 wheels - #4395

Open
raufer wants to merge 1 commit into
apache:mainfrom
raufer:python-3.15-stdlib-zstd
Open

raufer wants to merge 1 commit into
apache:mainfrom
raufer:python-3.15-stdlib-zstd

Conversation

@raufer

@raufer raufer commented Nov 16, 2026

Copy link
Copy Markdown

Rationale for this change

CPython 3.15 is out (3.15.2), but PyIceberg cannot be installed on it from wheels:

Python 3.14 added a zstd module to the standard library: compression.zstd, from PEP 784. PyIceberg only needs one-shot compress and decompress (the Avro codec and Puffin blobs), so on 3.14+ the third-party package is not needed at all.

What changes

  • pyiceberg/table/puffin.py uses compression.zstd.decompress on Python 3.14+. Below 3.14 it keeps zstandard.ZstdDecompressor().decompress.
  • pyiceberg/avro/codecs/zstandard_codec.py uses compression.zstd.compress and decompress on 3.14+. Below 3.14 the existing zstandard code is unchanged, only indented under the version check (git diff -w shows the real change).
  • In pyproject.toml, the dependency becomes zstandard>=0.13.0; python_version < '3.14'. uv.lock is regenerated with uv lock.
  • The wheel build adds 3.15: CIBW_PROJECT_REQUIRES_PYTHON becomes >=3.11,<3.16, and 3.15 joins the setup-python list. cibuildwheel v4.3.4 already builds cp315 by default.
  • CIBW_TEST_SKIP: "cp315-*" is set for now. The test step syncs the locked dev group, which pins pyarrow 25.0.1 and numpy 2.4.6 through pyspark[connect], and neither has a cp315 wheel. Moving those pins is a separate change, and the skip can go once it lands.

The version check follows the existing sys.version_info >= (3, 14) pattern (uuid7 in catalog/rest/__init__.py), so mypy checks each branch for the interpreter it runs on.

PEP 784 makes compression.zstd an optional module, built when libzstd is available; python-build-standalone (what uv installs) ships it on 3.14 and 3.15. A CPython built without it fails at import time, as a missing zstandard does today.

Are these changes tested?

The existing tests cover both paths: Puffin's test_read_compressed_zstd, and the zstd/zstandard manifest tests in tests/utils/test_manifest.py. Locally, on macOS arm64:

Python zstandard installed tests/avro tests/table/test_puffin.py tests/utils/test_manifest.py tests/table/test_metadata.py
3.13.7 yes (marker keeps it) 281 passed
3.14.10 no 281 passed
3.15.2 no 281 passed

ruff (v0.14.3) and mypy (1.18.2) are clean on the changed files under 3.13 and 3.15.

Are there any user-facing changes?

On Python 3.14+, pip install pyiceberg no longer installs zstandard. On 3.15, pyiceberg installs from a wheel.

🤖 Generated with Claude Code

…3.15 wheels

zstandard publishes no cp315 wheel; Python 3.14 ships compression.zstd (PEP 784).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up, nice to drop the hard zstandard dependency on 3.14+. Besides the inline comment:

  • CIBW_TEST_SKIP: "cp315-*" means we'd publish cp315 wheels where the compiled decoder_fast extension has never been imported. Fine to unblock this PR, but could you follow up with a cibuildwheel override for cp315-* that runs a minimal smoke test without the dev group (e.g. import pyiceberg.avro.decoder_fast and read a small zstd Avro file)? Until pyarrow/numpy ship cp315 wheels, that keeps us from shipping something completely untested.
  • We'd build 3.15 wheels, but there's no Programming Language :: Python :: 3.15 classifier in pyproject.toml, and lint-and-unit-test in python-ci.yml still stops at 3.14, so 3.15 isn't exercised in CI at all.

try:
from zstandard import ZstdCompressor, ZstdDecompressor
if sys.version_info >= (3, 14):
from compression import zstd

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PEP 784 makes compression.zstd optional; it's only built when libzstd is available (e.g. pyenv building 3.14 on a machine without the libzstd headers won't have it). On such a build this import raises at module load, and since pyiceberg/avro/codecs/__init__.py imports ZStandardCodec eagerly, import pyiceberg.manifest fails. That means reading any manifest breaks, not just zstd-compressed ones. Today the try/except defers the error until zstd is actually used.

On top of that, with the marker in pyproject.toml users can't recover by installing zstandard, since this branch never tries it.

Could we fall back instead: try compression.zstd, then zstandard, then the stub that raises on use? The same applies to puffin.py. It might be cleanest to put this in one small module that both use, so the import switch lives in one place.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants