Skip to content

Fix: preserve data columns in tables with only row-scoped headers - #2582

Open
GoldArowana wants to merge 1 commit into
unclecode:developfrom
GoldArowana:fix/table-row-headers-6efbf124
Open

GoldArowana wants to merge 1 commit into
unclecode:developfrom
GoldArowana:fix/table-row-headers-6efbf124

Conversation

@GoldArowana

Copy link
Copy Markdown

Summary

An HTML data table can use <th scope="row"> or <th scope="rowgroup"> without defining column headers. The default extractor currently copies that first data row into headers and uses its width to truncate later rows. For example, a two-cell first row can cause a value in the third column of the next row to disappear.

Keep an explicitly row-scoped first row in the data grid and let the existing generic column labels reflect the full grid width. Explicit thead and mixed explicit column scopes retain their existing behavior. This follows up on the row-grid work in #2258.

List of files changed and why

  • crawl4ai/table_extraction.py distinguishes row-scoped labels from inferred column headers and uses that decision for has_headers.
  • tests/test_table_rowspan_grid.py verifies header metadata and preservation of wider rows for row and row-group scopes.

How Has This Been Tested?

  • On base 51ccab7, three regression assertions failed and eight existing grid tests passed. On head b2553e9, 24 table-grid and Markdown-table tests passed.
  • Eight independently authored offline controls passed on head, including the public LXML scraping result and min_cols. Three of those failed on base.
  • The tests used Python 3.12 and LXML 6.1.3. Codex authored, ran, and independently reviewed the patch. Browser and network tests were outside this offline validation.

Checklist

  • Checked Black formatting for the changed lines and ran git diff --check.
  • Reviewed the diff and recorded an independent Codex review.
  • Commented the implicit column-header decision.
  • Updated documentation. The existing distinction between column headers and data rows applies.
  • Added and updated regression tests.
  • Ran the focused new and existing tests locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant