Skip to content

fix(firestore): resume query streams correctly on the default retry - #19137

Open
Su-001 wants to merge 1 commit into
googleapis:mainfrom
Su-001:fix/firestore-stream-default-retry
Open

Su-001 wants to merge 1 commit into
googleapis:mainfrom
Su-001:fix/firestore-stream-default-retry

Conversation

@Su-001

@Su-001 Su-001 commented Nov 19, 2026

Copy link
Copy Markdown

Fixes #19129

When a Query.stream() or AggregationQuery.stream() call on the default retry hits a retryable gRPC error partway through, _retry_query_after_exception reads transport.run_query._retry. That is the raw gRPC stub, which has no _retry, so the caller gets an AttributeError instead of a resume or the original GoogleAPICallError. retry=None fails the same way.

The existing unit tests didn't catch this because they gave the stub a fake _retry attribute.

Changes

  • Read the default policy from transport._wrapped_methods[...]. retry=None now raises the original error.
  • Fixed the resume path, which had never run with the default retry:
    • Resumes are now spaced out with exponential backoff, and the stream gives up with the original error at the policy's deadline. Before, it could re-query in an endless loop.
    • If the stream breaks before any document was yielded, the original query is re-run. start_after(None) dropped the caller's cursor or raised ValueError.
    • Resumes ask for only the rest of the limit and set no offset. If the limit was already reached, the stream ends.
    • Aggregations are not re-run after their single result was yielded, so the result is never yielded twice.

Behaviour change

With the default retry, a stream that fails partway with ServiceUnavailable, InternalServerError or DeadlineExceeded now resumes after the last document instead of crashing.

Testing

  • The retry tests now use the client's real GAPIC transport. They fail on the old code.
  • Added tests for limit/offset, cursors, backoff, the deadline, and the aggregation cases.
  • Unit tests pass on Python 3.11 and 3.13. lint, mypy, and coverage of the changed code are clean.
  • Checked end to end over real gRPC against a fake Firestore server with errors injected partway through streams: 11/11 scenarios pass, and the old code fails 9 of the first 10.

Follow-ups (not in this PR)

  • A resumed query reads at a new point in time; it could be pinned to the first response's read_time.
  • explain_options is not passed to a resumed query.
  • The async query classes do not resume broken streams.

Query.stream() and AggregationQuery.stream() read the default retry policy from the raw gRPC stub, which has no _retry attribute. Any retryable error partway through a stream on the default retry therefore raised AttributeError instead of resuming or raising the original error. retry=None failed the same way.

Read the policy from the transport's wrapped method instead, and fix the resume path that this enables: back off and give up at the retry deadline instead of looping, keep the caller's cursor when nothing was yielded yet, resume with the remaining limit and no offset, and do not re-run an aggregation whose result was already yielded.

Fixes googleapis#19129
@Su-001
Su-001 requested a review from a team as a code owner November 19, 2026 12:33

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces robust stream resumption and exponential backoff mechanisms for Firestore queries and aggregation queries when encountering retriable errors. Specifically, it adds helper functions and classes (resolve_stream_retry and StreamResumeBackoff) to resolve retry policies from wrapped GAPIC methods and manage backoff timing and deadlines. It integrates these helpers into the stream generators for both queries and aggregation queries, ensuring that queries resume from the last yielded snapshot with adjusted limits and offsets, and that backoff resets upon making progress. Comprehensive unit tests have also been added to verify these behaviors. I have no additional feedback to provide as there are no review comments.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant