Skip to content

Trial Windows build output reuse for cross-platform CI - #12410

Draft
Amaury Levé (Evangelink) wants to merge 4 commits into
mainfrom
dev/amauryleve/cross-platform-build-reuse
Draft

Amaury Levé (Evangelink) wants to merge 4 commits into
mainfrom
dev/amauryleve/cross-platform-build-reuse

Conversation

@Evangelink

@Evangelink Amaury Levé (Evangelink) commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Pull request

Summary

Linux/macOS CI currently rebuilds MSTest, MTP, and extensions even though Windows produces the packages that ship. Add a disabled-by-default UseWindowsBuildOutputs trial to compare running the same Windows-built outputs across operating systems without changing test selection.

Windows exports configuration-matched binaries, dependencies, test data, packages, and coverage settings in a commit-bound, SHA-256-verified archive. Public Linux/macOS jobs and the official all-products Linux job download that run's exact producer attempt, restore their own SDK/runtime/workload and NuGet dependencies, rebase coverage paths and source-root mapping sidecars, and use DLL-based no-build execution. Generated acceptance applications still build on the destination OS. Existing native-build paths, SkipTests, and affected-test collection/fallback remain available.

The focused Linux coverage issue is now fixed: stale Windows source-root mappings caused the misleading no_symbols skip. The same Windows-built test selection now collects 18.9% line coverage on Linux, matching the Windows runner summary. The trial stays off pending full acceptance, macOS, and hosted transport validation; no compute saving has been measured.

Design and decision record

Windows build output reuse: research and decisions records the artifact/lifecycle contract, alternatives, successful and unsuccessful experiments, and rollout prerequisites.

Keep the existing project-based runner and reporting configuration rather than introducing a second module-selection runner. Consumers wait for the existing Windows job, so less aggregate computation does not necessarily mean shorter elapsed CI time. Missing or invalid artifacts fail closed; no silent rebuild fallback is introduced. Symbol extraction and symbol-format experiments that did not fix coverage were removed from the delivered change.

The importer also rebases the physical project/source/package paths in .msCoverageExtensionSourceRootsMapping_*, preserving virtual map keys and encoding. Hidden sidecars are included on Unix. DLL/PDB bytes and all four existing SymbolsRestrict* defaults remain unchanged.

Demonstration

Infrastructure-only change; screenshots are not applicable. The initial composed experiment, before the source-root repair, used SDK 12.0.100-alpha.1.26556.115, Release/net11.0, and a Windows-produced Microsoft.Testing.Platform.3.0.0-ci.nupkg payload:

Producer / execution DeadlineHelperTests Coverage
Windows / Windows 35 passed 19.0% line coverage
Linux / Linux, clean native checkout 35 passed 19.0% line coverage
Windows / Linux, old importer 35 passed No coverable data; skipReason: no_symbols

Both data rows of TryGetDotnetPathByArchitecture_LinuxConfiguredElfMuxer_IsRejectedWithoutFallback also passed on Linux from the Windows-built assembly; they were present and skipped on Windows as expected. All 2,722 transported DLLs remained byte-identical. Restore and no-build test binlog replay found zero Csc/Vbc/Fsc tasks. TRX, CTRF, and JUnit reports were produced; coverage-file existence was not accepted as proof of nonempty coverage.

Source-root repair, validated 2026-11-12: a fresh Windows producer from f50f73e66dd136c88c213795ffcf207ad8bf3321, with CodeCoverage 18.13.0-preview.26560.3, was compared using the same archive with old and repaired importers. All 35 selected deadline cases passed on Windows and both Linux variants, with exact TRX names reconciled. The old importer still produced empty coverage; the repaired importer produced an 18.9% runner line-coverage summary, matching the current Windows baseline. All 2,836 DLL/PDB files stayed byte-identical, and four restore/test binlog replays found zero compiler tasks. The repaired Cobertura contains 5,223 covered line entries out of 25,000, including correctly relocated source paths. This is local composed evidence, not hosted CI or a full-matrix result.

Validation and regression proof

Rechecked before committing the original trial as 4d62268d8d5ce932013b07e745846211cbe324ed:

Command Result
.\eng\test-windows-build-outputs.ps1 Exit 0; originally 12 transport cases passed. The same cases also passed on Linux during implementation.
.\eng\validate-affected-tests.ps1 -SelfTest Exit 0; originally 71 affected-test/collection/rollback wiring cases passed.
git diff --check Exit 0.

YAML syntax check, exit 0, parsed 13 pipeline/template files:

python -c "import yaml; from pathlib import Path; paths = [Path('azure-pipelines.yml'), Path('azure-pipelines-official.yml'), *Path('eng').joinpath('pipelines','steps').glob('*.yml')]; [yaml.safe_load(p.read_text(encoding='utf-8-sig')) for p in paths]; print(f'Parsed {len(paths)} pipeline YAML files')"

Earlier real product builds/pack commands and the detailed Windows/Linux comparisons are recorded in the decision record. The original source baseline was 32a6ed6e1511ae9b61c9ce307e22ed5594b239cc; the original implementation patch has SHA-256 FD084840F3EAB0B94FD597F068DBB54DD30F8303C31C6F8209D8E6B25B6C266B.

The source-root repair is committed as d73ad4e8c3cb942a00ef547f9849dbb2f081dc2d. Its exact tested patch on f50f73e66dd136c88c213795ffcf207ad8bf3321 has SHA-256 74158C913CE0ABF21C83BA10137DDF6DA4C6783BF47A20393F378987F243EDCF. .\eng\test-windows-build-outputs.ps1 now passes 13 cases on Windows and Linux; .\eng\validate-affected-tests.ps1 -SelfTest passes the updated 80 cases. The new mapping regression fails against the old importer at the intended stale-source-path assertion. Windows build/pack and both real Linux no-build test commands exited zero; exact commands and limitations are in the maintained decision record.

Remaining gaps: the focused Linux source-resolution defect is repaired, but full artifact-based acceptance execution, macOS execution, and live Azure/1ES artifact transport have not been verified. No CI performance improvement is claimed. Keep the trial disabled until these checks succeed; do not suppress coverage or accept empty results to enable it.

Test-value evidence

Contract / test Without the behavior Correct implementation
Reject stale producer commits / RejectsStaleSourceBeforeExtraction Executed: removing only the commit comparison in an isolated helper copy made the unchanged test fail its rejection assertion, exit 1. Executed: passed as part of the transport suite, exit 0.
Rebase nested Windows coverage paths / WindowsCoveragePathsUseConsumerDirectorySeparators Executed: the initial prefix-only implementation failed the exact consumer source-root regex assertion on Linux. Executed: passed on Windows and Linux after normalizing the rebased path separators.
Rebase coverage source maps / CoverageSourceMappingsUseConsumerPaths Executed: the unchanged importer failed the expected consumer repository/source mapping assertion. Executed: passed on Windows and Linux, preserving virtual keys and UTF-16 encoding, including nested package caches.
Preserve binary/package/test-data/symbol bytes / RoundTripPreservesBinariesPackagesAndTestData Counterfactual not executed for this row. Executed: exact marker assertions passed on Windows and Linux.
Preserve composed execution and coverage Executed: the old importer ran all 35 selected cases but produced no coverable data. Executed: the repaired importer ran the same 35 cases and collected real source coverage with an 18.9% runner summary; all 2,836 DLL/PDB files remained unchanged. Broader rollout is not yet validated.

Detailed reports/binlogs remain local session evidence, not publicly hosted artifacts or a CI verification claim.

Add an opt-in same-run Windows Release payload for public Linux/macOS and official Linux test jobs. Verify payload provenance, preserve target-OS restore and test selection, and document the unresolved coverage parity gap with the trial disabled by default.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 11, 2026 20:29
Resolve the pipeline conflicts by keeping the Windows-output reuse trial alongside main's affected-test provider comment and report-comparison validation. Preserve the auto-merged report and affected-test pipeline updates.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Note

This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.

Relocate coverage source-map sidecars as well as coverage.config, preserving virtual path keys and encoding. Include hidden output metadata on Unix and cover repository, nested package-cache, and cross-OS mappings. Verify the same Windows-built tests collect coverage on Linux without changing DLLs, PDBs, or symbol-access defaults.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 12, 2026 12:33
@github-actions github-actions Bot added the state/needs-review Awaiting review from the team. label Nov 12, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧪 Test quality review — PR #12410

Compared merge base 50b513022be7c356cb342866a3b755a4b18bf97d with head d73ad4e8c3cb942a00ef547f9849dbb2f081dc2d. Evidence applies to this head, not every intermediate commit.

Test-value evidence

This PR changes only pipeline YAML, PowerShell scripts under eng/ and a decision record. It has no changed C# or MSTest test methods.

Contract Test Prediction Evidence status
Reject stale producer commits eng/test-windows-build-outputs.ps1 (RejectsStaleSourceBeforeExtraction) Static: the case should fail if the commit comparison is removed and pass with it. Unverified. The PR description reports execution, but no accessible records were checked here.
Rebase coverage paths and source mappings eng/test-windows-build-outputs.ps1 (WindowsCoveragePathsUseConsumerDirectorySeparators, CoverageSourceMappingsUseConsumerPaths) Static: the cases should fail with a prefix-only or unrebased importer. Unverified. The PR description reports red and green runs, but no accessible records were checked here.
Preserve binary, package and test-data bytes RoundTripPreservesBinariesPackagesAndTestData Expected to pass with the correct implementation. The PR description says no counterfactual was run.
Pipeline wiring (UseWindowsBuildOutputs, which defaults to off) None found by static inspection Not assessed. Hosted Azure/1ES transport and macOS execution are unvalidated, as the PR states.

This workflow ran no code. Green CI or an author claim is not paired execution proof.

🧵 Parallel safety

NOT_APPLICABLE. No changed tests or parallelization configuration were in scope.

Per-test assessment

No changed C# test methods were found, so nothing is graded (Not applicable). The PowerShell harness was not graded against the C# rubric.

I have no inline findings.

This advisory review is heuristic and non-blocking. Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.

🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~20 AIC · ⌖ ~1 AIC · ⊞ 8.1K

@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 12, 2026
@github-actions github-actions Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 12, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The cross-platform reuse path has critical blockers: coverage validation requires unrestored project assets, acceptance tests lack required staged packages, and Windows-compiled newline expectations fail on Linux/macOS.

3 open issues

Review effort: Balanced

Comment thread azure-pipelines-official.yml Resolved
Comment thread eng/windows-build-outputs.ps1 Resolved
@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 12, 2026
@github-actions github-actions Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 12, 2026
@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 12, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. labels Nov 12, 2026
@github-actions

This comment has been minimized.

Scope official coverage validation to the restored non-Windows graph, transport staged acceptance packages, and normalize reporter test line endings across execution hosts.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 12, 2026 13:51
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/approved Proposal approved; ready for implementation. labels Nov 12, 2026
@microsoft-github-policy-service microsoft-github-policy-service Bot added the state/needs-review Awaiting review from the team. label Nov 12, 2026
@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 12, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. state/approved Proposal approved; ready for implementation. labels Nov 12, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧪 Test quality review — PR #12410

Compared merge base 50b513022be7c356cb342866a3b755a4b18bf97d with head 5f4624db93a4a2ca61fdff1a494bcdaf6b5e5bb5. Evidence applies to this head only.

Test-value evidence

Contract Scenario / expected outcome Test and decisive assertion Counterfactual Evidence status
Terminal reporter output assertions are line-ending agnostic (test-only refactor) Raw-string expectations and output compared after CRLF is normalized to LF Five Assert.AreEqual(expected.Replace(...), ShowEscape(output.Replace(...))) calls in TerminalTestReporterTests Behavior-preserving: expected to pass before and after. Real output changes still fail the equality. Static prediction only. No paired execution records were available.
Pipeline and eng/*.ps1 changes (Windows build-output reuse, opt-in UseWindowsBuildOutputs, default off) Not applicable to unit-level counterfactuals eng/test-windows-build-outputs.ps1 is wired into the pipeline as a validation step Not assessed: requires an AzDO run Unverified (non-executable here). Existing validation script not inspected in depth.

🧵 Parallel safety

STATUS: PARTIAL. The specialist subtask was not run in this session, so coverage is incomplete and this is not an all-clear. The only changed tests are pure string normalizations of local variables. No shared state, resource, or parallelization configuration changed (config_count=0).

Per-test assessment

Pseudo-mutation evidence below consists of static predictions, not executed mutation results.

Result Quality Test Notes How to improve
Pass A (90–100) mod TerminalTestReporterTests.NonAnsiTerminal_OutputFormattingIsCorrect Exact multiline equality is kept, with line endings normalized. None
Pass A (90–100) mod TerminalTestReporterTests.InProcessRetry_AnnotatesAttemptsAndCountsTestOnce Exact multiline equality is kept, with line endings normalized. None
Pass A (90–100) mod TerminalTestReporterTests.SimpleAnsiTerminal_OutputFormattingIsCorrect Exact multiline equality is kept, with line endings normalized. None
Pass A (90–100) mod TerminalTestReporterTests.AnsiTerminal_OutputFormattingIsCorrect Exact multiline equality is kept, with line endings normalized. None
Pass A (90–100) mod TerminalTestReporterTests.AnsiTerminal_OutputProgressFrameIsCorrect Exact multiline equality is kept, with line endings normalized. None

No inline findings. The normalization matches the repository guidance for platform-native newlines.

<sub>This advisory review is heuristic and non-blocking.

Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.</sub>

🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~31 AIC · ⌖ ~1 AIC · ⊞ 8.1K

@microsoft-github-policy-service microsoft-github-policy-service Bot added the state/needs-review Awaiting review from the team. label Nov 12, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. state/approved Proposal approved; ready for implementation. labels Nov 12, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Reusing the Windows-built test assembly on Linux/macOS leaves three newline-sensitive assertions comparing CRLF expectations with LF output, causing test failures until both operands are normalized.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Normalize line endings in reused helper test assertions

eng/​pipelines/​steps/​restore-windows-build-outputs.yml:31

With a CRLF Windows checkout, the reused Microsoft.Testing.Platform.MSBuild.UnitTests assembly still has three newline-sensitive assertions in FailedTestHelperTests.cs:95–145. On Linux/macOS, the compiled raw-string expectations retain CRLF while the generated or normalized messages use LF, so these selected tests fail despite correct output. Normalize both operands in the comparisons at lines 95–103, 117–124, and 138–145, as this PR already does for the terminal-reporter tests.

3 resolved since last review

Review effort: Balanced

@github-actions

This comment has been minimized.

@github-actions

Copy link
Copy Markdown
Contributor

Final test-triage resolution — Azure Pipelines build 1677661

Triggering check: microsoft.testfx (Build Windows Release). Mode: full. Evidence is incomplete (1 artifact download failure, history download budget exhausted), so this is an inconclusive resolution, not a clean bill of health.

Failure signatures (Linux Release / "Test" task, exit code 2; the Windows "Test affected changes" task reports the same tests)

  1. OpenTelemetry acceptance tests (OpenTelemetryTests, OpenTelemetryActivityTopologyTests, 17 results) fail in ClassInitialize. The dotnet build of the generated ApplicationOwnedOpenTelemetry asset fails with NU1015: The following PackageReference item(s) do not have a version specified: Microsoft.Extensions.Hosting. This is deterministic and a test-asset/package-version issue, not a flake. Confidence: high on the immediate cause. The link to this PR's changes (build output reuse, central package versions) is likely but unverified.
  2. PlaywrightSdkTests.EnablePlaywrightProperty_* (4 results: Runner and VSTest) fail with Unexpected exit code / missing expected output. The root cause is not visible in the compact evidence, so confidence is low.
  3. Windows App Model diagnostics probes (RetryFailsFirstAttempt, RetryTestMethod_PassesOnSecondApp, NativeTests.Outcome) fail with deliberate probe messages such as "This probe deliberately fails the first retry attempt". These look like intentional diagnostic probes, not product regressions, and are not treated as actionable here.

No retry/flaky recovery, crash/hang, or duration-regression evidence was found. No issue is created: this is a PR-specific failure and no recurrence is proven (history was incomplete).

Next step: check whether the PR changes how Microsoft.Extensions.Hosting gets its version in generated test assets (for example, Directory.Packages / central package management or reused build output). Then inspect the Playwright test logs from the Test task and rerun the Linux leg.

🤖 Automated content by GitHub Copilot. Generated by the Pipeline Test Triage workflow. · copilot · auto · ~38 AIC · ⌖ ~1 AIC · ⊞ 10.1K

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants