Description
Determine this is the right repository
- I determined this is the correct repository in which to report this bug.
This is a client library bug, not a Spanner service bug. The service correctly ends the stream with an error, and StreamedResultSet raises it. The DB-API Cursor then swallows it.
Summary of the issue
Context
We read Spanner through SQLAlchemy (sqlalchemy-spanner), which fetches rows through the DB-API cursor's fetchall() / fetchmany() / fetchone(). Sometimes a query's result stream fails after the first chunk of rows has already arrived, for example with DeadlineExceeded on a long read or Aborted in a read-write transaction. When that happens, the cursor returns the rows read so far as if they were the complete result. Nothing is raised or logged, so the caller can't tell a partial result from a complete one.
The cause is in Cursor._fetch (google/cloud/spanner_dbapi/cursor.py). It wraps the read in except Exception as e: exception = e, records the exception for the transaction-retry bookkeeping (add_fetch_statement_for_retry), and then return rows without re-raising. The same code is on main today. Some older releases (for example 3.51 and 3.63) had a raise, but a return rows inside the finally discarded it, with the same effect.
Not every error is affected. _restart_on_unavailable resumes UNAVAILABLE and RST_STREAM before they reach the cursor. Every other error that reaches the cursor mid-stream is swallowed: DeadlineExceeded, Aborted, ResourceExhausted, Cancelled, non-resumable InternalServerError, session NotFound, and so on.
Aborted behaves worse than the others:
- In a read-write connection,
_fetchcallstransaction_helper.retry_transaction(). That replays the transaction's statements on new cursors (_cursor_map). Thewhile Trueloop then reads on from the original cursor's broken stream, so_fetchreturns[]. - In a read-only connection the
Abortedis swallowed like the other errors, and_fetchreturns[].
Expected Behavior:
When the result stream fails partway, fetchall() / fetchmany() / fetchone() either raise the error or (for Aborted in a read-write transaction) retry and return the complete result. A caller never gets a silently truncated result.
Actual Behavior:
They return the rows read before the failure (or [] for Aborted) and raise nothing.
API client name and version
google-cloud-spanner 3.72.0 (also reproduced on 3.68.0; same _fetch on main)
Reproduction steps: code
file: repro_partial_fetch.py
"""Minimal reproduction: spanner_dbapi Cursor.fetchall()/fetchmany() return a partial
result, with no exception, when the result stream fails after some rows were read.
No Spanner instance is needed: the cursor's row iterator is replaced by one that yields
three rows and then raises, the way StreamedResultSet does when the gRPC stream breaks.
"""
from unittest import mock
from google.api_core import exceptions
from google.cloud.spanner_dbapi.cursor import Cursor
from google.cloud.spanner_v1 import __version__
def rows_then(error):
yield ["row-1"]
yield ["row-2"]
yield ["row-3"]
raise error
def cursor_over(stream, read_only):
connection = mock.MagicMock(is_closed=False, read_only=read_only)
cursor = Cursor(connection)
cursor._itr = stream
return cursor
print("google-cloud-spanner", __version__)
for read_only in (True, False):
for error in (
exceptions.DeadlineExceeded("Deadline Exceeded"),
exceptions.ResourceExhausted("quota"),
exceptions.Aborted("Transaction was aborted."),
):
cursor = cursor_over(rows_then(error), read_only)
try:
rows = cursor.fetchall()
print(f"read_only={read_only}: {type(error).__name__} after 3 rows -> fetchall() returned {rows!r}, no exception")
except Exception as raised:
print(f"read_only={read_only}: {type(error).__name__} after 3 rows -> raised {type(raised).__name__}")Reproduction steps: supporting files
None. Needs only pip install google-cloud-spanner==3.72.0. No Spanner instance or emulator is required.
Reproduction steps: actual results
google-cloud-spanner 3.72.0
read_only=True: DeadlineExceeded after 3 rows -> fetchall() returned [['row-1'], ['row-2'], ['row-3']], no exception
read_only=True: ResourceExhausted after 3 rows -> fetchall() returned [['row-1'], ['row-2'], ['row-3']], no exception
read_only=True: Aborted after 3 rows -> fetchall() returned [], no exception
read_only=False: DeadlineExceeded after 3 rows -> fetchall() returned [['row-1'], ['row-2'], ['row-3']], no exception
read_only=False: ResourceExhausted after 3 rows -> fetchall() returned [['row-1'], ['row-2'], ['row-3']], no exception
read_only=False: Aborted after 3 rows -> fetchall() returned [], no exception
Reproduction steps: expected results
google-cloud-spanner 3.72.0
read_only=True: DeadlineExceeded after 3 rows -> raised DeadlineExceeded
read_only=True: ResourceExhausted after 3 rows -> raised ResourceExhausted
read_only=True: Aborted after 3 rows -> raised Aborted
read_only=False: DeadlineExceeded after 3 rows -> raised DeadlineExceeded
read_only=False: ResourceExhausted after 3 rows -> raised ResourceExhausted
read_only=False: Aborted after 3 rows -> raised Aborted (or the transaction is retried and the full result returned)
OS & version + platform
macOS 27.2 (arm64) for the repro. The code path is platform-independent; we run on Linux (Google App Engine and Cloud Run).
Python environment
Python 3.12.13
Python dependencies
Package Version
---------------------- -------
google-api-core 2.44.0
google-cloud-spanner 3.72.0
grpcio 1.85.1
proto-plus 1.29.0
protobuf 7.36.2
(In production, also sqlalchemy-spanner 1.19.0 and SQLAlchemy 2.0.)
Additional context
Impact: SQLAlchemy reads through these cursor methods, so application code gets short or empty query results that look complete. Code that acts on what's missing from a result can then delete or overwrite good data. Examples are a sync job that removes rows not returned, or a check that a record doesn't exist yet.
Suggested fix: keep recording the exception for the retry bookkeeping, then re-raise it. For Aborted in a read-write transaction, either raise it to the caller, or, after retry_transaction(), continue from the replayed cursor's result instead of the original cursor's broken stream.
Workaround we're using in the meantime: replace Cursor._fetch with a version that records the exception via add_fetch_statement_for_retry and then re-raises.
Activity