Repository navigation
Conversation
…3.15 wheels zstandard publishes no cp315 wheel; Python 3.14 ships compression.zstd (PEP 784). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fokko
left a comment
There was a problem hiding this comment.
Thanks for picking this up, nice to drop the hard zstandard dependency on 3.14+. Besides the inline comment:
CIBW_TEST_SKIP: "cp315-*"means we'd publish cp315 wheels where the compileddecoder_fastextension has never been imported. Fine to unblock this PR, but could you follow up with a cibuildwheel override forcp315-*that runs a minimal smoke test without the dev group (e.g. importpyiceberg.avro.decoder_fastand read a small zstd Avro file)? Until pyarrow/numpy ship cp315 wheels, that keeps us from shipping something completely untested.- We'd build 3.15 wheels, but there's no
Programming Language :: Python :: 3.15classifier inpyproject.toml, andlint-and-unit-testinpython-ci.ymlstill stops at 3.14, so 3.15 isn't exercised in CI at all.
| try: | ||
| from zstandard import ZstdCompressor, ZstdDecompressor | ||
| if sys.version_info >= (3, 14): | ||
| from compression import zstd |
There was a problem hiding this comment.
PEP 784 makes compression.zstd optional; it's only built when libzstd is available (e.g. pyenv building 3.14 on a machine without the libzstd headers won't have it). On such a build this import raises at module load, and since pyiceberg/avro/codecs/__init__.py imports ZStandardCodec eagerly, import pyiceberg.manifest fails. That means reading any manifest breaks, not just zstd-compressed ones. Today the try/except defers the error until zstd is actually used.
On top of that, with the marker in pyproject.toml users can't recover by installing zstandard, since this branch never tries it.
Could we fall back instead: try compression.zstd, then zstandard, then the stub that raises on use? The same applies to puffin.py. It might be cleanest to put this in one small module that both use, so the import switch lives in one place.
Rationale for this change
CPython 3.15 is out (3.15.2), but PyIceberg cannot be installed on it from wheels:
zstandardis a core dependency, and it publishes no cp315 wheel: 0.25.0 (2025-09-14) is still its latest release. Its 3.15 PRs, Add Python 3.15 support indygreg/python-zstandard#357 and Build: Bump adlfs from 2024.1.0 to 2024.2.0 #372, are open. Without a C compiler,pip install pyicebergfails on 3.15.pyicebergitself ships no cp315 wheel either, becauseCIBW_PROJECT_REQUIRES_PYTHONstops at<3.15.Python 3.14 added a zstd module to the standard library:
compression.zstd, from PEP 784. PyIceberg only needs one-shot compress and decompress (the Avro codec and Puffin blobs), so on 3.14+ the third-party package is not needed at all.What changes
pyiceberg/table/puffin.pyusescompression.zstd.decompresson Python 3.14+. Below 3.14 it keepszstandard.ZstdDecompressor().decompress.pyiceberg/avro/codecs/zstandard_codec.pyusescompression.zstd.compressanddecompresson 3.14+. Below 3.14 the existingzstandardcode is unchanged, only indented under the version check (git diff -wshows the real change).pyproject.toml, the dependency becomeszstandard>=0.13.0; python_version < '3.14'.uv.lockis regenerated withuv lock.CIBW_PROJECT_REQUIRES_PYTHONbecomes>=3.11,<3.16, and 3.15 joins thesetup-pythonlist. cibuildwheel v4.3.4 already builds cp315 by default.CIBW_TEST_SKIP: "cp315-*"is set for now. The test step syncs the lockeddevgroup, which pins pyarrow 25.0.1 and numpy 2.4.6 throughpyspark[connect], and neither has a cp315 wheel. Moving those pins is a separate change, and the skip can go once it lands.The version check follows the existing
sys.version_info >= (3, 14)pattern (uuid7incatalog/rest/__init__.py), so mypy checks each branch for the interpreter it runs on.PEP 784 makes
compression.zstdan optional module, built when libzstd is available; python-build-standalone (what uv installs) ships it on 3.14 and 3.15. A CPython built without it fails at import time, as a missingzstandarddoes today.Are these changes tested?
The existing tests cover both paths: Puffin's
test_read_compressed_zstd, and thezstd/zstandardmanifest tests intests/utils/test_manifest.py. Locally, on macOS arm64:tests/avro tests/table/test_puffin.py tests/utils/test_manifest.py tests/table/test_metadata.pyruff (v0.14.3) and mypy (1.18.2) are clean on the changed files under 3.13 and 3.15.
Are there any user-facing changes?
On Python 3.14+,
pip install pyicebergno longer installszstandard. On 3.15, pyiceberg installs from a wheel.🤖 Generated with Claude Code