Repository navigation
Register the S3 signer on a session instead of a shared client - #4405
Open
Anurag-S1ngh wants to merge 1 commit into
Open
Anurag-S1ngh wants to merge 1 commit into
Anurag-S1ngh wants to merge 1 commit into
Conversation
_s3 registered the remote signer on the botocore client, which s3fs caches and every thread shares. Unregistering the signer before registering it again left a window in which a request went out without an Authorization header, and the store answered 403 against a remote-signing catalog. Create an AioSession per filesystem, register the signer on it, and pass it to S3FileSystem so every filesystem keeps its own signer and never touches a shared client. The path without a signer is unchanged.
Fokko
self-requested a review
November 17, 2026 17:28
Fokko
reviewed
Nov 17, 2026
Fokko
left a comment
Contributor
There was a problem hiding this comment.
Thanks @Anurag-S1ngh! Good find, and the approach is right: the signer belongs on a session this filesystem owns, not on the shared cached client where unregister/re-register leaves an unsigned window.
I pulled the branch and verified:
test_s3_registers_the_signer_on_the_sessionandtest_s3_registers_the_signer_per_sessionpass locally- Confirmed the no-signer path is untouched:
_s3passes nosessionkwarg whenregister_eventsis empty, so s3fs keeps creating its own session exactly as before - Confirmed s3fs honors an injected session: when
sessionis passed,S3FileSystemskips creating its own (_session_is_owned = False) and uses the one we hand it, so theregister_lasthandlers are the ones that actually fire
One nit, non-blocking: session_kwargs is typed dict[str, str] and only carries profile today. If someone later needs to forward more session-level options (e.g. a custom endpoint or credentials), the type will need loosening. Fine as is for now.
The full tests/io/test_fsspec.py run needs a local MinIO and the adlfs/gcsfs extras, so I couldn't run every S3 test locally, but CI (including integration-test-s3) is fully green. LGTM!
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_s3registered the remote signer on the botocore client, which s3fs caches and every thread shares. Unregistering the signer before registering it again left a window in which a request went out without anAuthorizationheader, and the store answered 403 against a remote-signing catalog.Changes
AioSessionper filesystem, register the signer on its event emitter, and pass it toS3FileSystem.profileproperty to the session, since s3fs no longer creates the session.This is the approach of #3783, which was closed for inactivity, with regression tests for the signer itself.
Testing
S3FileSystemsigns abefore-sign.s3request, and one that two configurations get independent sessions and signers.make testpasses (4694 passed, 5 skipped).Note: #4270 also mentions this issue, but it fixes the duplicate header defect from #2422 and does not change the signer registration.
Closes #3896