Repository navigation
Default ascending sort fields to nulls-first - #4390
rahulsmahadev wants to merge 1 commit into
Conversation
asc() defaulted to nulls-last, and SortField with a str direction did too: the validator compared the raw str with the enum. Java and SortField default ascending fields to nulls-first, so default asc() to nulls-first and normalize the direction. desc() and explicit null orders are unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
|
@kevinjqliu could you take a look when you have a chance? You merged #2552, which added |
Fokko
left a comment
There was a problem hiding this comment.
Thanks @rahulsmahadev (and Isaac)! Great write up, and I agree with the reasoning: the spec does not define a default, so aligning with Java's SortField and SortOrderBuilder is the right call.
I checked out the branch and ran tests/table/test_sorting.py and tests/catalog/test_catalog_behaviors.py, all 768 pass, CI is green too.
The one line fix in set_null_order is a nice find, SortDirection("asc") handles both the str and the enum member so the validator now picks the same default either way.
Only note for release notes: update_sort_order().asc() without an explicit null_order changes behaviour for existing callers, thanks for calling that out and asking for the changelog label.
LGTM!
Rationale for this change
Two defaults give ascending sort fields the nulls-last order, which disagrees with Java and with
SortField's own default for ascending fields. The same ascending field can therefore be recorded with different null orders depending on how it is built.Table.update_sort_order().asc(name, transform)with nonull_orderrecords nulls-last:SortFieldbuilt with a str direction also gets nulls-last, while the enum member gets nulls-first:Java's
SortOrderBuilder.asc(String)usesNullOrder.NULLS_FIRST(its Javadoc says "ascending with nulls first"), anddesc(String)usesNULLS_LAST.SortFieldpicks nulls-first forSortDirection.ASCand nulls-last otherwise. Sodesc()agrees in all three places, and only the ascending default inUpdateSortOrder.asc()differs. The spec requiresnull-orderin every sort field and does not define a default, so Java andSortFieldare the reference.In
SortField, theset_null_ordervalidator chooses the default by comparing the rawdirectionwithSortDirection.ASC.SortDirectionis a plainEnum, so the str"asc"never matches and gets the descending default. Pydantic converts the value to the enum afterwards, so the field ends up ascending with nulls-last.This change:
UpdateSortOrder.asc()defaults toNullOrder.NULLS_FIRST.desc()keepsNULLS_LAST.set_null_orderconverts the direction toSortDirectionbefore it chooses the default.An explicit null order is used as given. Existing sort orders are not changed, because their null order is already stored. The
asc()default came in withUpdateSortOrderin #2552. Calls that omitnull_ordernow request nulls-first, also on existing tables, where a matching existing sort order may be reused. PassNullOrder.NULLS_LASTexplicitly to keep the previous behaviour.Are these changes tested?
Yes.
test_update_sort_order_default_null_orderintests/catalog/test_catalog_behaviors.pyruns through thecatalogfixture (memory, SQL, and SQL without rowcount catalogs). It commitsasc()anddesc()with no null order and checks the stored sort order, then commits explicit null orders and checks that they are kept.test_deserialize_sort_field_str_directionintests/table/test_sorting.pychecks that"asc"and"desc"validate to the sameSortFieldas the enum members, with the matching default null order. Both fail on main withNullOrder.NULLS_LASTwhereNullOrder.NULLS_FIRSTis expected, and both pass with this change. The integration tests intests/integration/test_sort_order_update.pypass explicit null orders, so they do not depend on the default.Are there any user-facing changes?
Yes.
update_sort_order().asc()without a null order now records nulls-first (it recorded nulls-last), andSortFieldwith a str direction now gets the same default as the enum member. Please add thechangeloglabel.This pull request and its description were written by Isaac.