Skip to content

Deprecate implicit non-spec type changes in update_column - #4410

Open
rahulsmahadev wants to merge 2 commits into
apache:mainfrom
rahulsmahadev:rahul/update-column-promotions
Open

rahulsmahadev wants to merge 2 commits into
apache:mainfrom
rahulsmahadev:rahul/update-column-promotions

Conversation

@rahulsmahadev

Copy link
Copy Markdown
Contributor

Rationale for this change

update_column checks a type change with promote(), the read-side rule that the Avro and Parquet readers use. That rule is more lenient than the table spec: it also accepts string -> binary, binary -> string and fixed(16) -> uuid. So a populated string column can be changed to binary without any warning, and the commit succeeds:

tbl.update_schema().update_column("name", field_type=BinaryType()).commit()  # name: string

The spec's "Valid primitive type promotions" table does not list these pairs. Java's SchemaUpdate.updateColumn rejects them through TypeUtil.isPromotionAllowed, which allows equal types, int -> long, float -> double and decimal widening with the same scale.

This PR validates primitive type changes using Java's v1/v2 promotion rules, while keeping PyIceberg's existing unknown -> primitive behaviour. It does not add v3 date -> timestamp or date -> timestamp_ns support, and it does not broaden unknown-type conversions. Validation runs when field_type is supplied and allow_incompatible_changes is false. promote() is unchanged, because the Avro resolver, the Arrow projection, the union_by_name no-op check and append compatibility (_check_schema_compatible) still rely on its leniency.

Behaviour change: without allow_incompatible_changes=True, update_column warns before applying string -> binary, binary -> string or fixed[16] -> uuid. The warning is a DeprecationWarning that says the change is deprecated in 0.13.0 and will be removed in 0.14.0. A subsequent release will require the explicit opt-in. Every other invalid change still raises ValidationError, as on main. allow_incompatible_changes=True applies these three changes without a warning.

union_by_name warns when an incoming UUID field replaces an existing fixed[16] field, including nested struct fields, list elements and map values. String/binary unions remain unchanged, because union_by_name checks them with promote() before it calls update_column. An incoming fixed[16] field over an existing UUID field also remains a no-op.

Precedent: Fokko added the string <-> binary allowed-update tests in apache/iceberg#8374 (September 2023), and #207 only moved that test file. #986 added test_table_scan_default_to_large_types, which changed a string column to binary with update_column. #1669 later changed that test to create the column as binary, so the update became a no-op.

Alternative for maintainers: reject these three changes immediately instead of warning first. #3613 rejected decimal promotions that change the scale without a deprecation cycle, and Fokko approved it. This PR chooses a warning cycle, because these changes were tested as supported (apache/iceberg#8374), and AGENTS.md asks for a deprecation cycle before breaking public API. Choosing immediate rejection means replacing the warning branch in update_column with a ValidationError.

Other engines: in a local DuckDB 2.0.0 iceberg_scan reproduction, promoting a populated string column to binary made a non-ASCII value fail to read, and changed the four-character string \x41 to the single byte A. PyIceberg continued to read the original UTF-8 bytes.

Are these changes tested?

Yes. tests/test_schema.py covers the following.

  • test_update_column_rejects_invalid_promotions: the exact error message for each invalid change (for example int -> float, a decimal scale change and fixed[15] -> uuid), unchanged table metadata after the rejections, and the valid promotions committed and read back after a reload.
  • test_update_column_warns_for_non_spec_promotions: the three deprecated changes warn with the exact message, commit, and read back after a reload.
  • test_update_column_no_warning_with_allow_incompatible_changes: the explicit opt-in applies the same three changes without a warning.
  • test_union_by_name_fixed_uuid_and_string_binary: fixed(16) and UUID in both directions, string/binary unchanged, and a list element.
  • test_update_column_promotes_unknown_to_primitive_only: the existing unknown behaviour, on a v3 table.

On main, the three warning tests fail with DID NOT WARN and the union case fails because no warning is emitted. With this change they all pass. In tests/integration/test_rest_schema.py, test_allowed_updates and test_disallowed_updates expect the deprecation warning for string <-> binary, and test_alter_map_key still opts in with allow_incompatible_changes=True to reach the map key check.

Are there any user-facing changes?

Yes. Without allow_incompatible_changes=True, update_column emits a DeprecationWarning before applying string -> binary, binary -> string or fixed[16] -> uuid. Without this opt-in, union_by_name warns when incoming UUID fields replace existing fixed[16] fields. String/binary unions and incoming fixed[16] fields over existing UUID fields remain type no-ops. The deprecated implicit changes are scheduled to require the explicit opt-in in 0.14.0. Please add the changelog label.

This pull request and its description were written by Isaac.

update_column applied string <-> binary and fixed(16) -> uuid silently,
using the read-side promote(). Check the spec's promotion list instead.
Those three pairs now warn with a DeprecationWarning, and are rejected
in a later release unless allow_incompatible_changes=True is set.

Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

@geruh, could you review the compatibility transition for these previously accepted schema changes? @Anurag-S1ngh, tagging you as the latest author of table/update/schema.py in #4393. The warning cycle and the immediate-rejection alternative are in the description.

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant