Skip to content

feat(aarch64): add GDB debugging for Linux KVM - #2016

Open
cshung wants to merge 2 commits into
hyperlight-dev:mainfrom
cshung:cshung/kvm-aarch64-gdb
Open

cshung wants to merge 2 commits into
hyperlight-dev:mainfrom
cshung:cshung/kvm-aarch64-gdb

Conversation

@cshung

@cshung cshung commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

ARM64 Linux KVM GDB

KVM guests support register and memory access, two hardware breakpoints,
BRK software breakpoints, and hardware single-step.
KVM reports debug stops through KVM_EXIT_DEBUG.
Debug state survives vCPU reset, and register access respects the active stack pointer.

Commit layers

This PR is C, one KVM-specific commit above shared support A in #2050.
WHP support in #1907 is an independent sibling, not a prerequisite.

KVM-only comparison.

Shared page-table translation, guest-page-sized memory access, snapshot
recovery, cancellation and crash-detach handling belong to #2050.
This layer contains KVM integration, platform enablement and native GDB CI.

The graph and update rules are in
docs/gdb-commit-layers.md.
The full PR diff includes the unmerged shared prerequisite.

Validation

  • Formatting and ARM64 GDB-enabled cross-Clippy pass with default and KVM-only features.
  • KVM backend source matches the tested combined tree.
  • Shared support passes native ARM64 MSHV formatting, Clippy, build, the full
    just test recipe, 26 debugger/backend tests and seven real GDB client tests.
  • Earlier Raspberry Pi 4 validation passed 16 targeted native unit tests and six
    real GDB client tests. Native KVM ARM64 runtime tests were not repeated for these
    shared fixes.

Pi validation uses a separate, uncommitted ARMv8.0 allocator fix and guests built
with RUSTFLAGS="-C target-feature=-lse". Pi compatibility changes are outside this PR.

November 16 rebase validation

Base: 5fe5c8f8. Each layer is one signed commit. A and E are patch-equivalent.
The shared ARM64 test helpers use upstream's guest startup and memory-layout APIs.

  • All six layers pass formatting and Linux ARM64 cross-Clippy.
  • KVM-only and MSHV-only ARM64 GDB configurations pass cross-Clippy.
  • WHP passes Windows ARM64 cross-Clippy and Linux just clippyw.
  • The final MSHV GDB tip passes Windows x64 formatting, Clippy, build, the full
    just test recipe and three focused debugger tests. Rust and C guests are
    rebuilt in debug and release for the guest startup ABI.

Native ARM64 runtime results above predate this rebase. They were not repeated
against the guest startup ABI in this base.

@cshung cshung added C-enhancement For PRs adding features, improving functionality, docs, tests, etc. S-needs-review PR is ready for (re-)review labels Nov 3, 2026
Copilot AI balanced review requested due to automatic review settings November 3, 2026 00:34
@cshung
cshung marked this pull request as draft November 3, 2026 00:36
@cshung cshung removed the S-needs-review PR is ready for (re-)review label Nov 3, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical flaws allow guest modification of executable trampoline code and debugger memory corruption, while breakpoint cleanup and snapshot-ordering issues can leave sandboxes unrecoverable.

5 open issues
What changed in this PR

Adds ARM64 guest debugging on Linux/KVM, alongside the shared debugger infrastructure and Windows/WHP support from #1907.

Changes:

  • Supports ARM64 register and memory access, software and hardware breakpoints, and hardware single-step.
  • Adds instruction-cache synchronization, debug-state preservation across reset, and snapshot safeguards.
  • Adds debugger tests and documents platform capabilities.
File Description
src/​hyperlight_host/​src/​sandbox/​uninitialized_evolve.rs Initializes ARM64 debug vectors during VM setup.
src/​hyperlight_host/​src/​sandbox/​snapshot/​memory/​backing.rs Exposes backing ranges for cache synchronization.
src/​hyperlight_host/​src/​sandbox/​initialized.rs Guards snapshots and restores against active software breakpoints.
src/​hyperlight_host/​src/​mem/​mgr.rs Exposes checked host memory ranges to the debugger.
src/​hyperlight_host/​src/​mem/​exe.rs Carries debug-vector addresses in load metadata.
src/​hyperlight_host/​src/​mem/​elf.rs Finds and relocates the debug-vector symbol.
src/​hyperlight_host/​src/​hypervisor/​virtual_machine/​whp/​x86_64.rs Adapts address translation to the shared interface.
src/​hyperlight_host/​src/​hypervisor/​virtual_machine/​whp/​aarch64.rs Implements WHP debugging, cache synchronization, and tests.
src/​hyperlight_host/​src/​hypervisor/​virtual_machine/​mshv/​x86_64.rs Adapts address translation to the shared interface.
src/​hyperlight_host/​src/​hypervisor/​virtual_machine/​mod.rs Adds ARM64 debug exits and initialization errors.
src/​hyperlight_host/​src/​hypervisor/​virtual_machine/​kvm/​x86_64.rs Adapts address translation to the shared interface.
src/​hyperlight_host/​src/​hypervisor/​virtual_machine/​kvm/​aarch64.rs Implements KVM debugging and active-stack-pointer handling.
src/​hyperlight_host/​src/​hypervisor/​regs/​aarch64/​whp_reg.rs Defines WHP debug register identifiers.
src/​hyperlight_host/​src/​hypervisor/​hyperlight_vm/​x86_64.rs Uses shared debugger handling and breakpoint storage.
src/​hyperlight_host/​src/​hypervisor/​hyperlight_vm/​mod.rs Integrates ARM64 stops, stepping, and mapping safeguards.
src/​hyperlight_host/​src/​hypervisor/​hyperlight_vm/​debug.rs Provides shared requests and software-breakpoint lifecycle handling.
src/​hyperlight_host/​src/​hypervisor/​hyperlight_vm/​aarch64.rs Connects ARM64 VM setup and execution to GDB.
src/​hyperlight_host/​src/​hypervisor/​gdb/​target.rs Adds ARM64 register conversion to the shared target.
src/​hyperlight_host/​src/​hypervisor/​gdb/​mod.rs Adds page-table translation and cache-flush support.
src/​hyperlight_host/​src/​hypervisor/​gdb/​event_loop.rs Reports ARM64 initial stops to GDB.
src/​hyperlight_host/​src/​hypervisor/​gdb/​arch/​x86_64.rs Houses x86 breakpoint constants and stop classification.
src/​hyperlight_host/​src/​hypervisor/​gdb/​arch/​aarch64.rs Defines ARM64 software breakpoints and alignment checks.
src/​hyperlight_host/​src/​hypervisor/​gdb/​arch.rs Selects architecture-specific debugger helpers.
src/​hyperlight_host/​examples/​guest-debugging/​main.rs Adds ARM64 breakpoint, stepping, and code-write integration tests.
src/​hyperlight_host/​build.rs Enables GDB for supported ARM64 platforms.
src/​hyperlight_guest_bin/​src/​arch/​aarch64/​exception/​entry.rs Exports debug vectors and forwards synchronous exceptions.
src/​hyperlight_common/​src/​arch/​aarch64/​exn.rs Defines the shared debug-exception HVC immediate.
docs/​how-to-debug-a-hyperlight-guest.md Documents ARM64 capabilities and guest-symbol requirements.
CHANGELOG.md Records Linux/KVM and Windows/WHP ARM64 debugging support.

Review effort: Balanced

@hyperlight-gh-bot

This comment has been minimized.

@cshung
cshung force-pushed the cshung/kvm-aarch64-gdb branch from a60be9b to 55da94f Compare November 3, 2026 01:58
@hyperlight-gh-bot

This comment has been minimized.

@cshung
cshung force-pushed the cshung/kvm-aarch64-gdb branch from 55da94f to 2418379 Compare November 3, 2026 19:28
@hyperlight-gh-bot

This comment has been minimized.

@hyperlight-gh-bot

This comment has been minimized.

@cshung
cshung force-pushed the cshung/kvm-aarch64-gdb branch from a26c2d9 to 11361ca Compare November 4, 2026 16:37
@hyperlight-gh-bot

This comment has been minimized.

@cshung
cshung force-pushed the cshung/kvm-aarch64-gdb branch from 11361ca to 4849110 Compare November 5, 2026 17:24
@hyperlight-gh-bot

This comment has been minimized.

@cshung
cshung force-pushed the cshung/kvm-aarch64-gdb branch 2 times, most recently from 3f280ae to 859ef39 Compare November 12, 2026 00:12
@hyperlight-gh-bot

This comment has been minimized.

@cshung
cshung marked this pull request as ready for review November 12, 2026 02:56
@cshung
cshung force-pushed the cshung/kvm-aarch64-gdb branch from 859ef39 to 68a8704 Compare November 12, 2026 18:15
@hyperlight-gh-bot

This comment has been minimized.

@yoshuawuyts yoshuawuyts added O-aarch64 Specific to 64-bit Arm processors O-kvm Linux KVM backend O-linux Linux host support S-needs-review PR is ready for (re-)review labels Nov 16, 2026
cshung and others added 2 commits November 16, 2026 10:02
Architecture adapters, software breakpoints, page-sized memory access and recovery handling are shared by backend layers. Guest debug vectors and common tests support ARM64. Backend layers control platform enablement. The commit-layer document defines dependencies and shared-update validation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>
KVM debug exits provide hardware breakpoints and single-step. Register access respects the active stack pointer. Native ARM64 GDB tests run in debug CI. Shared debugger behavior belongs to the parent commit.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>
@cshung
cshung force-pushed the cshung/kvm-aarch64-gdb branch from 68a8704 to 45810c6 Compare November 16, 2026 18:16
@hyperlight-gh-bot

Copy link
Copy Markdown

Benchmark Results

Measured commit: 45810c6b19e3
Baseline commit: 5fe5c8f8c44f

kvm / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 758.56 ns (➖ 1.00x faster)
vec_bytes 600.40 ns (➖ 1.00x slower)
372.95 µs (➖ 1.00x slower)

payload_allocation

slot_pool_segmented
262144 570.15 ns (➖ 1.02x slower)
65536 141.26 ns (➖ 1.01x faster)

sandboxes

create_initialized_and_drop
medium 13.20 ms (➖ 1.00x faster)

slot_pool

alloc_dealloc_128 alloc_dealloc_4096 alloc_dealloc_1500
7.68 ns (➖ 1.00x faster) 7.67 ns (➖ 1.00x faster) 7.69 ns (➖ 1.01x faster)

snapshot_files

load_snapshot_unverified
small 96.65 µs (➖ 1.01x faster)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
262144 7.08 µs (➖ 1.03x faster) 7.08 µs (➖ 1.28x faster)
65536 1.91 µs (➖ 1.11x faster) 2.05 µs (➖ 1.07x slower)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
262144 28.20 µs (➖ 1.03x slower)
65536 6.13 µs (➖ 1.03x faster) 6.45 µs (➖ 1.01x slower)
8192 1.08 µs (➖ 1.04x slower) 1.05 µs (➖ 1.01x faster)
kvm / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 763.48 ns (➖ 1.01x slower)
vec_bytes 614.49 ns (➖ 1.05x slower)
715.82 µs (➖ 1.03x faster)

payload_allocation

slot_pool_segmented
262144 566.82 ns (➖ 1.00x slower)
65536 194.64 ns (➖ 1.01x faster)

sandboxes

create_initialized_and_drop
medium 15.13 ms (➖ 1.01x slower)

slot_pool

alloc_dealloc_128 alloc_dealloc_4096 alloc_dealloc_1500
7.71 ns (➖ 1.01x slower) 7.71 ns (➖ 1.01x slower) 7.70 ns (➖ 1.01x slower)

snapshot_files

load_snapshot_unverified
small 55.91 µs (➖ 1.03x slower)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
262144 8.55 µs (➖ 1.00x faster) 8.59 µs (➖ 1.01x slower)
65536 2.40 µs (➖ 1.01x slower) 2.40 µs (➖ 1.00x slower)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
262144 32.61 µs (➖ 1.09x faster)
65536 7.95 µs (➖ 1.02x faster) 7.75 µs (➖ 1.02x faster)
8192 809.20 ns (➖ 1.01x slower) 813.01 ns (➖ 1.02x faster)
mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 898.57 ns (➖ 1.02x faster)
vec_bytes 699.79 ns (➖ 1.00x faster)
320.51 µs (➖ 1.01x faster)

payload_allocation

slot_pool_segmented
262144 680.02 ns (➖ 1.01x slower)
65536 182.24 ns (➖ 1.00x faster)

sandboxes

create_initialized_and_drop
medium 68.60 ms (➖ 1.00x faster)

slot_pool

alloc_dealloc_128 alloc_dealloc_4096 alloc_dealloc_1500
9.86 ns (➖ 1.00x faster) 9.87 ns (➖ 1.07x faster) 9.87 ns (➖ 1.00x faster)

snapshot_files

load_snapshot_unverified
small 106.36 µs (➖ 1.01x faster)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
262144 10.43 µs (➖ 1.01x slower) 9.29 µs (➖ 1.06x faster)
65536 2.47 µs (➖ 1.03x faster) 2.65 µs (➖ 1.04x faster)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
262144 35.03 µs (➖ 1.01x faster)
65536 7.94 µs (➖ 1.02x faster) 7.93 µs (➖ 1.07x faster)
8192 1.29 µs (➖ 1.00x faster) 1.30 µs (➖ 1.01x slower)
mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 922.87 ns (➖ 1.00x slower)
vec_bytes 659.82 ns (➖ 1.00x faster)
707.18 µs (➖ 1.04x slower)

payload_allocation

slot_pool_segmented
262144 665.27 ns (➖ 1.00x slower)
65536 257.06 ns (➖ 1.01x faster)

sandboxes

create_initialized_and_drop
medium 82.16 ms (➖ 1.09x slower)

slot_pool

alloc_dealloc_128 alloc_dealloc_4096 alloc_dealloc_1500
8.72 ns (➖ 1.00x slower) 8.92 ns (➖ 1.03x slower) 9.57 ns (➖ 1.10x slower)

snapshot_files

load_snapshot_unverified
small 54.12 µs (➖ 1.02x slower)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
262144 7.94 µs (➖ 1.01x slower) 7.90 µs (➖ 1.03x faster)
65536 2.27 µs (➖ 1.00x slower) 2.27 µs (➖ 1.00x faster)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
262144 39.37 µs (➖ 1.01x slower)
65536 7.47 µs (➖ 1.01x faster) 7.52 µs (➖ 1.00x faster)
8192 898.22 ns (➖ 1.02x faster) 916.69 ns (➖ 1.00x faster)
hyperv-ws2025 / amd (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 1.30 µs (➖ 1.06x slower)
vec_bytes 774.30 ns (➖ 1.01x faster)
2.06 ms (➖ 1.03x slower)

payload_allocation

slot_pool_segmented
262144 781.49 ns (➖ 1.01x faster)
65536 233.63 ns (➖ 1.02x slower)

sandboxes

create_initialized_and_drop
medium 13.16 ms (➖ 1.03x slower)

slot_pool

alloc_dealloc_128 alloc_dealloc_4096 alloc_dealloc_1500
10.04 ns (➖ 1.00x faster) 10.00 ns (➖ 1.00x slower) 10.18 ns (➖ 1.02x slower)

snapshot_files

load_snapshot_unverified
small 800.53 µs (➖ 1.05x slower)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
262144 9.19 µs (➖ 1.02x faster) 9.05 µs (➖ 1.05x faster)
65536 2.34 µs (➖ 1.01x faster) 2.34 µs (➖ 1.01x faster)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
262144 43.45 µs (➖ 1.09x faster)
65536 8.90 µs (➖ 1.14x faster) 9.45 µs (➖ 1.08x slower)
8192 1.27 µs (➖ 1.03x faster) 1.30 µs (➖ 1.04x slower)
hyperv-ws2025 / intel (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 1.25 µs (➖ 1.04x slower)
vec_bytes 772.06 ns (➖ 1.01x slower)
3.69 ms (➖ 1.01x slower)

payload_allocation

slot_pool_segmented
262144 795.85 ns (➖ 1.08x slower)
65536 212.57 ns (➖ 1.02x slower)

sandboxes

create_initialized_and_drop
medium 14.58 ms (➖ 1.01x slower)

slot_pool

alloc_dealloc_128 alloc_dealloc_4096 alloc_dealloc_1500
9.88 ns (➖ 1.02x slower) 10.06 ns (➖ 1.00x faster) 9.94 ns (➖ 1.03x slower)

snapshot_files

load_snapshot_unverified
small 670.83 µs (➖ 1.05x faster)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
262144 7.70 µs (➖ 1.00x slower) 7.45 µs (➖ 1.03x faster)
65536 2.31 µs (➖ 1.00x slower) 2.30 µs (➖ 1.00x faster)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
262144 48.48 µs (➖ 1.04x slower)
65536 8.38 µs (➖ 1.02x slower) 8.35 µs (➖ 1.03x faster)
8192 1.04 µs (➖ 1.14x faster) 1.12 µs (➖ 1.05x slower)

Reported by cargo ci bench-report --candidate run:43927116195 --config-file bench_report.toml.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

C-enhancement For PRs adding features, improving functionality, docs, tests, etc. O-aarch64 Specific to 64-bit Arm processors O-kvm Linux KVM backend O-linux Linux host support S-needs-review PR is ready for (re-)review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants