Skip to content

test(gapic-generator): add Showcase operational error logging compliance suite - #19141

Draft
chalmerlowe wants to merge 1 commit into
feat/otel-logging-m3-s7-grpc-rpc-methodfrom
feat/otel-logging-m6-compliance
Draft

chalmerlowe wants to merge 1 commit into
feat/otel-logging-m3-s7-grpc-rpc-methodfrom
feat/otel-logging-m6-compliance

Conversation

@chalmerlowe

@chalmerlowe chalmerlowe commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Context

Operational error logging (F4 wire-attempt DEBUG logs and F5 client-method WARNING logs) needs end-to-end verification against a live GAPIC Showcase server across synchronous and asynchronous gRPC and HTTP/REST transports, matching the declarative matrix approach used for OpenTelemetry tracing compliance (test_span_compliance.py).

In addition, aligning google-api-core status-code extraction in #19144 (canonical INVALID_ARGUMENT for HTTP 400 BadRequest, and omitting rpc.response.status_code on client-side HTTP timeouts with no server response) brings Tier 3 tracing spans (F2.3, F2.4) in line with the canonical observability test plan as well.

Bug: b/479827845

Solution

  • Showcase Compliance Harness (packages/gapic-generator/tests/system/showcase_harness.py):
    • Centralizes CSV matrix loading (load_csv_matrix), sync/async Showcase client construction (construct_showcase_client), google.rpc.ErrorInfo status construction, deterministic Echo/Sequence scenario execution (execute_showcase_scenario, execute_showcase_scenario_async), async gRPC inspection task draining, and InMemoryLogCapture.
  • Showcase Logging Compliance Matrix & Suite (packages/gapic-generator/tests/system/):
    • Adds logging_requirements_matrix.csv (28 atomic rows covering F4.1–F5.14 across HTTP/REST and gRPC, using rpc.method, canonical google.rpc.Code status strings, omitted httpRequest.status / rpc.response.status_code on client timeouts, and exact status.message parity across sync and async).
    • Adds test_logging_compliance.py validating both raw logging.LogRecord attributes and StructuredLogFormatter JSON output across disabled-flag suppression on failing RPCs, happy-path silence, retry recovery suppression, L4 DEBUG wire attempts, L3 WARNING terminal failures, active T3/T4 OpenTelemetry span correlation, ErrorInfo fallback boundaries, and async transport parity (grpc_asyncio and rest_asyncio).
  • Tracing Matrix Alignment (packages/gapic-generator/tests/system/telemetry_requirements_matrix.csv):
  • CI Artifact Upload (.github/workflows/gapic-generator-tests.yml):
    • Configures SHOWCASE_RAW_LOGS_OUTPUT_PATH and uploads raw_logs_output.json in showcase and showcase-canary jobs alongside raw_spans_output.json, while keeping local test runs hermetic via tmp_path_factory.

Notes for Reviewers

  • Stacked on feat/otel-logging-m3-s7-grpc-rpc-method (fix(api_core): normalize observability log attributes and canonical status codes #19144) so this PR stays scoped strictly to packages/gapic-generator/ and its CI workflow.
  • Both test_logging_compliance.py and test_span_compliance.py wrap strict monorepo assertions in conftest.skip_on_core_mismatch() so standard showcase jobs running against PyPI google-api-core do not break before release, while showcase-canary (INSTALL_LOCAL_CORE=true) enforces all assertions.
  • For experimental rest_asyncio, Echo suppression, L3 server failures/trace correlation, L4 400 + T4 span correlation, and client timeouts are verified directly; SequenceService 503 retry scenarios are omitted for rest_asyncio because google-auth's AsyncAuthorizedSession performs internal 503 retries below GAPIC's retry and trace wrappers.

Verification

  • Verified packages/gapic-generator locally against live GAPIC Showcase (v0.44.2) in Docker across:
    • showcase_w_rest_async-3.14 (INSTALL_LOCAL_CORE=true)
    • showcase-3.10 (INSTALL_LOCAL_CORE=true)
    • showcase-3.14 against released PyPI google-api-core (non-canary compatibility check)
  • Relying on GitHub Actions CI for full multi-version verification.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a comprehensive system test suite for operational error logging compliance (F4 & F5) against a live Showcase server, alongside normalizing the rpc.method attribute in google-api-core's observability module to ensure leading slashes are stripped and edge cases are handled correctly. The review feedback highlights an improvement opportunity in the test harness to avoid multiplying lists of mutable protobuf messages, which creates shared references, and suggests using list comprehensions instead to ensure independent instances.

Comment on lines +330 to +337
responses = [
Sequence.Response(
status=status_pb2.Status(
code=code_pb2.UNAVAILABLE,
message="Persistent outage",
)
)
] * EXHAUSTED_RETRY_ATTEMPTS

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Multiplying a list containing a mutable protobuf message object (e.g., [Sequence.Response(...)] * N) creates a list of references to the exact same object. If any part of the test or client code mutates one of these responses, all elements in the list will be affected, which can lead to subtle bugs or flaky tests. Using a list comprehension ensures that a distinct, independent instance is created for each element.

Suggested change
responses = [
Sequence.Response(
status=status_pb2.Status(
code=code_pb2.UNAVAILABLE,
message="Persistent outage",
)
)
] * EXHAUSTED_RETRY_ATTEMPTS
responses = [
Sequence.Response(
status=status_pb2.Status(
code=code_pb2.UNAVAILABLE,
message="Persistent outage",
)
)
for _ in range(EXHAUSTED_RETRY_ATTEMPTS)
]

Comment on lines +357 to +366
responses = [
Sequence.Response(
status=status_pb2.Status(
code=code_pb2.UNAVAILABLE,
message="Temporary glitch",
)
)
] * recovery_failed_attempts + [
Sequence.Response(status=status_pb2.Status(code=code_pb2.OK))
]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Multiplying a list containing a mutable protobuf message object (e.g., [Sequence.Response(...)] * N) creates a list of references to the exact same object. If any part of the test or client code mutates one of these responses, all elements in the list will be affected, which can lead to subtle bugs or flaky tests. Using a list comprehension ensures that a distinct, independent instance is created for each element.

        responses = [
            Sequence.Response(
                status=status_pb2.Status(
                    code=code_pb2.UNAVAILABLE,
                    message="Temporary glitch",
                )
            )
            for _ in range(recovery_failed_attempts)
        ] + [
            Sequence.Response(status=status_pb2.Status(code=code_pb2.OK))
        ]

@chalmerlowe
chalmerlowe force-pushed the feat/otel-logging-m6-compliance branch from 7f09887 to 00492f5 Compare November 19, 2026 18:07
@chalmerlowe
chalmerlowe changed the base branch from feat/otel-logging-m3-s6-http-error-info to feat/otel-logging-m3-s7-grpc-rpc-method November 19, 2026 18:07
@chalmerlowe
chalmerlowe force-pushed the feat/otel-logging-m6-compliance branch from 00492f5 to bb492a4 Compare November 19, 2026 18:18
@chalmerlowe
chalmerlowe force-pushed the feat/otel-logging-m3-s7-grpc-rpc-method branch from 591ef2b to facba85 Compare November 19, 2026 20:23
@chalmerlowe
chalmerlowe force-pushed the feat/otel-logging-m6-compliance branch from bb492a4 to 8d664f4 Compare November 19, 2026 20:25
@chalmerlowe
chalmerlowe force-pushed the feat/otel-logging-m3-s7-grpc-rpc-method branch from facba85 to 210b44b Compare November 19, 2026 21:17
…nce suite

Add the 28-row operational error logging matrix (F4.1-F5.14), Showcase compliance harness, sync/async compliance tests, and CI raw_logs_output.json artifact upload.

Bug: b/479827845
@chalmerlowe
chalmerlowe force-pushed the feat/otel-logging-m6-compliance branch from 8d664f4 to 30e2042 Compare November 19, 2026 21:21

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant