hazo_scrape 1.6.0 → 1.6.1

package/CHANGE_LOG.md

@@ -1,5 +1,32 @@
 # hazo_scrape — Change Log
 
+## 1.6.1 — 2026-07-29
+
+Four `extractTable` correctness fixes. Kept as a **separate version** rather than folded into the (unpublished) 1.6.0: three of the four regress behaviour that shipped in **1.3.0** and is live on npm, so they deserve their own visible `Fixed` entry attributable to a version consumers can upgrade *to*, instead of being buried inside 1.6.0's `Added` list.
+
+### Fixed — grid mode duplicated the header row when `headerRow` matched nothing
+- `extractTable(html, { grid: { …, headerRow } })` with a typo'd/stale `headerRow` selector emitted the header as **data row 0**. The fallback correctly re-read row 0 as the header, but the "skip row 0" guard was gated on `grid.headerRow == null`, so it could never fire once `headerRow` was supplied. A `headerRow` that matches nothing now degrades to exactly the same behaviour as omitting it.
+
+### Fixed — the repeated-in-cell-header strip truncated legitimate values
+- The responsive-label strip (1.3.0) only tested `text.startsWith(header)`, which butchered real values: header `2025` + cell `2025-06-30` → `-06-30` (then `parseDate` → `null`); header `Total` + cell `Total dividend` → `dividend`. The strip now requires a genuine **DOM-concatenation seam**, matching the pattern it exists for (`<div>Payment date</div><div>2 July 2026</div>` → `"Payment date2 July 2026"`):
+  - the header must **not** be followed by whitespace — whitespace means the cell reads as one natural phrase, not two glued nodes; and
+  - the remainder must not continue the header's own token — if the header ends in a digit and the remainder opens with a digit or a numeric joiner (`-` `/` `.` `:`), the "header" is really a prefix of one longer number/date.
+- Everything else (letter → digit, letter → letter with no space) is still stripped, so the NAB responsive case is unchanged.
+
+### Fixed — row-header alignment when the header row has NO corner cell
+- 1.3.0's `td` → `th, td` change fixed tables that pair a `<th scope="row">` row label with a corner cell in the header row, but broke the equally common **cornerless** variant: header `<th>Ex date</th><th>Amount</th>` with row `<th scope="row">FY25 Final</th><td>1 Jan 2026</td><td>45c</td>` shifted every column right by one (`Ex date` read as `"FY25 Final"`). Data rows are now reconciled against the header width: when a row is wider than the header and the excess is fully explained by leading `<th>` cells, exactly that many leading row-header cells are dropped. **Both** shapes now map correctly; a merely ragged row (extra `<td>`s, or a leading `<td>`) is untouched.
+
+### Fixed — combine mode leaked unrelated tables past a non-table sibling
+- `combine` documented "stopping at the first sibling that doesn't match", but the walk used `nextAll('table')`, which **filters out** non-table siblings rather than stopping at them — so an intervening `<h2>`/`<p>` never triggered the `break` and a later, unrelated same-width table was absorbed as data. The walk now visits every following sibling and breaks on the first that is not a `<table>` (as well as on the first width mismatch), matching the documented contract.
+
+### Docs
+- `README.md`'s `extractTable` entry documented only `opts.select`. It now documents `grid`, `combine`, `cellIgnoreSelectors`, precedence between the modes, and row-header reconciliation; `mapRows`/`parseDate` gained the `dateYearPivot`/`yearPivot` options added in 1.6.0.
+
+### Known remaining (out of scope, pre-existing)
+- `parseNumber('1.234,56')` (European locale) is misread — only comma-as-thousands is handled.
+- Accounting-style negatives (`(1.23)`) are not recognised as `-1.23`.
+- `extractTableFromPdf` keeps a multi-page PDF's per-page repeated header as data rows.
+
 ## 1.6.0 — 2026-07-25
 
 ### Added — `parseNumber` tokenizes a trailing bare cent marker

package/README.md

@@ -61,13 +61,54 @@ console.log(rows[0].exDate[0].value); // "2026-03-05"
 
 ## API
 
-### `extractTable(html: string, opts?: { select?: string }): { headers: string[]; rows: string[][]; warning?: string }`
+### `extractTable(html, opts?): { headers: string[]; rows: string[][]; warning?: string }`
+
+```ts
+extractTable(html: string, opts?: {
+  select?: string;
+  grid?: { container?: string; row: string; cell: string; headerRow?: string };
+  combine?: { headerSelector: string; dropLabelRows?: boolean };
+  cellIgnoreSelectors?: string[];
+}): ExtractTableResult
+```
 
 Parses network-free HTML and returns the best-guess data table as raw header
-and row strings. Pass `opts.select` (a CSS selector) to target a specific
-`<table>` instead of letting the built-in scoring heuristic pick one. A
-`warning` is returned (never thrown) when no table-like element is found, or
-when `select` matches nothing.
+and row strings. With no options a scoring heuristic picks the most
+data-table-like `<table>` on the page. A `warning` is returned (never thrown)
+when no table-like element is found, or when a selector matches nothing.
+
+Precedence: `grid` > `combine` > `select`.
+
+- **`select`** — a CSS selector targeting a specific `<table>` instead of
+  letting the scoring heuristic choose.
+
+- **`grid`** — read a "table" built from styled `<div>`s (CSS grid/flex)
+  rather than a real `<table>`; common on IR pages whose real table is
+  JS-rendered. `row`/`cell` are required; `container` scopes the search (when
+  it matches several grids, the one with the most data rows wins); `headerRow`
+  identifies the header among the `row` matches (rows matching it are never
+  data). Omit `headerRow` — or give one that matches nothing — and the first
+  row is used as the header and excluded from the data.
+
+- **`combine`** — reconstruct one logical table from a header-only `<table>`
+  followed by one `<table>` per section (e.g. per year), a common
+  Computershare-style layout. Headers come from `headerSelector`'s first row;
+  data rows are gathered from that table plus each **immediately-following
+  sibling**, stopping at the first sibling that is not a `<table>` or whose
+  column count differs — so an unrelated later section can't leak in. Set
+  `dropLabelRows: true` to drop bare section-divider rows (only the first cell
+  non-empty, e.g. a lone `2025`).
+
+- **`cellIgnoreSelectors`** — CSS selectors whose matching descendants are
+  removed from every cell **and header** before its text is read, e.g.
+  `['sup']` so `100<sup>4</sup>` reads as `100`, not `1004`. Off by default
+  (cells are read verbatim); applies to all three modes; the removal happens
+  on a per-cell clone and never mutates the document.
+
+**Row-header columns.** Data cells are read as `th, td`, so an accessible
+table whose first column is a `<th scope="row">` label stays column-aligned.
+When the header row omits the matching corner cell, the leading row-header
+cell(s) are dropped so the row still lines up with the headers.
 
 ### `mapColumns(headers: string[], keywords: ColumnKeywords, opts?: { dateGuard?: boolean; required?: string[] }): ColumnMap`
 
@@ -77,14 +118,24 @@ the first. `opts.dateGuard` drops date-looking headers from any key whose
 `type !== 'date'`. `opts.required` lists keys that must have at least one
 candidate for `map.ok` to be `true`.
 
-### `mapRows(table: { headers; rows }, map: ColumnMap, keywords: ColumnKeywords, opts?: { dateFormats?: string[]; dateExtractLeading?: boolean }): MappedRow[]`
+### `mapRows(table, map, keywords, opts?): MappedRow[]`
+
+```ts
+mapRows(
+  table: { headers: string[]; rows: string[][] },
+  map: ColumnMap,
+  keywords: ColumnKeywords,
+  opts?: { dateFormats?: string[]; dateExtractLeading?: boolean; dateYearPivot?: number },
+): MappedRow[]
+```
 
 Joins a `ColumnMap` back onto `table.rows`, producing one `MappedRow` per data
 row. Each matched key holds an array of `Cell`s (one per candidate header),
-each parsed according to that key's declared `type`. `opts.dateFormats` and
-`opts.dateExtractLeading` are forwarded verbatim to `parseDate`'s `opts.formats`
-and `opts.extractLeading` for every date-typed cell; both default off, so
-omitting `opts` preserves the exact prior behaviour.
+each parsed according to that key's declared `type`. `opts.dateFormats`,
+`opts.dateExtractLeading` and `opts.dateYearPivot` are forwarded verbatim to
+`parseDate`'s `opts.formats`, `opts.extractLeading` and `opts.yearPivot` for
+every date-typed cell; all three default off, so omitting `opts` preserves the
+exact prior behaviour.
 
 ### `parseNumber(raw: string): number | null`
 
@@ -93,7 +144,7 @@ and `%`, then parses what remains as a number. Returns `null` when nothing
 parseable is left. Does **not** convert cents to dollars — domain scaling is
 the caller's job.
 
-### `parseDate(raw: string, opts?: { formats?: string[]; extractLeading?: boolean }): string | null`
+### `parseDate(raw: string, opts?: { formats?: string[]; extractLeading?: boolean; yearPivot?: number }): string | null`
 
 Parses a handful of common date text formats (ISO, `D Mon YYYY`, `Mon D,
 YYYY`, and `D/M/Y` slash dates) into an ISO `yyyy-mm-dd` string. Returns
@@ -108,6 +159,13 @@ junk is still rejected (the date must be the first token) and the date itself is
 still matched exactly and disambiguated by the same rules — so the "never guess"
 contract holds. For real IR pages that staple a label onto a date cell.
 
+`opts.yearPivot` (default off) enables 2-digit years, which are otherwise
+**rejected** (`null`) rather than guessed. It is the base of a sliding 100-year
+window: `parseDate('10 Mar 26', { yearPivot: 2000 })` → `'2026-03-10'`, while
+`yearPivot: 1950` reads `26`→2026 and `99`→1999. Applies to the named-month
+formats only (`D Mon YY`, `Mon D, YY`); slash dates stay 4-digit-only, since a
+2-digit slash year compounds day/month **and** century ambiguity.
+
 ### Types
 
 ```ts

package/dist/parse/extract_table.d.ts

@@ -25,7 +25,9 @@ export type GridSelector = {
     /**
      * Selector identifying the header row among the `row` matches. Rows that
      * match it are treated as headers (never data); the first such row supplies
-     * the column headers. Omit to treat the first row as the header.
+     * the column headers. Omit to treat the first row as the header — as does a
+     * selector that matches NOTHING (a typo'd/stale selector degrades to the
+     * omitted behaviour rather than emitting the header as data row 0).
      */
     headerRow?: string;
 };
@@ -36,8 +38,9 @@ export type GridSelector = {
  * Computershare-style IR pages (e.g. JB Hi-Fi's dividend history). The header
  * comes from `headerSelector`'s first row; data rows are gathered from that
  * table plus its immediately-following sibling <table>s that share its column
- * count (stopping at the first sibling that doesn't), so an unrelated later
- * table can't leak in.
+ * count. The walk stops at the first following sibling that is not a <table>
+ * (e.g. an <h2> introducing an unrelated section) or whose column count
+ * differs, so an unrelated later table can't leak in.
  */
 export type CombineSelector = {
     /** Selector matching the ONE table whose first row supplies the shared headers. */

package/dist/parse/extract_table.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"extract_table.d.ts","sourceRoot":"","sources":["../../src/parse/extract_table.ts"],"names":[],"mappings":"AAsBA,MAAM,MAAM,kBAAkB,GAAG;IAC/B,OAAO,EAAE,MAAM,EAAE,CAAC;IAClB,IAAI,EAAE,MAAM,EAAE,EAAE,CAAC;IACjB,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB,CAAC;AAEF;;;;;GAKG;AACH,MAAM,MAAM,YAAY,GAAG;IACzB;;;;;;OAMG;IACH,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,0FAA0F;IAC1F,GAAG,EAAE,MAAM,CAAC;IACZ,0DAA0D;IAC1D,IAAI,EAAE,MAAM,CAAC;IACb;;;;OAIG;IACH,SAAS,CAAC,EAAE,MAAM,CAAC;CACpB,CAAC;AAEF;;;;;;;;;GASG;AACH,MAAM,MAAM,eAAe,GAAG;IAC5B,mFAAmF;IACnF,cAAc,EAAE,MAAM,CAAC;IACvB;;;;;OAKG;IACH,aAAa,CAAC,EAAE,OAAO,CAAC;CACzB,CAAC;AAEF,MAAM,MAAM,mBAAmB,GAAG;IAChC,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,iGAAiG;IACjG,IAAI,CAAC,EAAE,YAAY,CAAC;IACpB;;;;OAIG;IACH,OAAO,CAAC,EAAE,eAAe,CAAC;IAC1B;;;;;;OAMG;IACH,mBAAmB,CAAC,EAAE,MAAM,EAAE,CAAC;CAChC,CAAC;AAgQF,wBAAgB,YAAY,CAAC,IAAI,EAAE,MAAM,EAAE,IAAI,CAAC,EAAE,mBAAmB,GAAG,kBAAkB,CAkCzF"}
+{"version":3,"file":"extract_table.d.ts","sourceRoot":"","sources":["../../src/parse/extract_table.ts"],"names":[],"mappings":"AAsBA,MAAM,MAAM,kBAAkB,GAAG;IAC/B,OAAO,EAAE,MAAM,EAAE,CAAC;IAClB,IAAI,EAAE,MAAM,EAAE,EAAE,CAAC;IACjB,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB,CAAC;AAEF;;;;;GAKG;AACH,MAAM,MAAM,YAAY,GAAG;IACzB;;;;;;OAMG;IACH,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,0FAA0F;IAC1F,GAAG,EAAE,MAAM,CAAC;IACZ,0DAA0D;IAC1D,IAAI,EAAE,MAAM,CAAC;IACb;;;;;;OAMG;IACH,SAAS,CAAC,EAAE,MAAM,CAAC;CACpB,CAAC;AAEF;;;;;;;;;;GAUG;AACH,MAAM,MAAM,eAAe,GAAG;IAC5B,mFAAmF;IACnF,cAAc,EAAE,MAAM,CAAC;IACvB;;;;;OAKG;IACH,aAAa,CAAC,EAAE,OAAO,CAAC;CACzB,CAAC;AAEF,MAAM,MAAM,mBAAmB,GAAG;IAChC,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,iGAAiG;IACjG,IAAI,CAAC,EAAE,YAAY,CAAC;IACpB;;;;OAIG;IACH,OAAO,CAAC,EAAE,eAAe,CAAC;IAC1B;;;;;;OAMG;IACH,mBAAmB,CAAC,EAAE,MAAM,EAAE,CAAC;CAChC,CAAC;AAkTF,wBAAgB,YAAY,CAAC,IAAI,EAAE,MAAM,EAAE,IAAI,CAAC,EAAE,mBAAmB,GAAG,kBAAkB,CAkCzF"}

package/dist/parse/extract_table.js

@@ -80,6 +80,34 @@ function scoreTable(tableSel) {
     const consistencyRatio = numDataRows > 0 ? numConsistent / numDataRows : 0;
     return numHeaderCells * 2 + (hasTh ? 5 : 0) + numDataRows * 3 + consistencyRatio * 10;
 }
+// Decide whether a leading copy of `header` inside `text` is a REPEATED DOM
+// LABEL (which must be stripped) or genuinely part of the value (which must
+// not). The responsive-table pattern that motivates the strip glues two sibling
+// nodes together with no separator — `<div>Payment date</div><div>2 July
+// 2026</div>` reads as `"Payment date2 July 2026"` — so the seam is the tell:
+//
+//   1. Whitespace after the header => prose, one natural phrase, not a seam.
+//      Header "Total" + "Total dividend" is a value that happens to start with
+//      the header word; stripping it would leave the bare word "dividend".
+//   2. The remainder must not continue the header's OWN token. If the header
+//      ends in a digit and the remainder opens with a digit or a numeric
+//      joiner (`-`, `/`, `.`, `:`), the "header" is really a prefix of one
+//      longer number/date — header "2025" + "2025-06-30" would be butchered
+//      into "-06-30".
+//
+// Everything else (letter -> digit, letter -> letter with no space) is the
+// concatenation signature the strip exists for and is stripped as before.
+function isRepeatedHeaderLabel(text, header) {
+    if (header === '' || text.length <= header.length || !text.startsWith(header))
+        return false;
+    const rest = text.slice(header.length);
+    if (/^\s/.test(rest))
+        return false; // (1) prose, not a DOM seam
+    const headerEndsDigit = /\d$/.test(header);
+    if (headerEndsDigit && /^[\d\-/.:]/.test(rest))
+        return false; // (2) same token cut in half
+    return rest.trim() !== '';
+}
 // Read the data rows (everything after the header row) of one real <table>,
 // aligning each cell against `headers` (for the repeated-in-cell-label strip)
 // and removing `ignore` descendants first. Shared by readTable and combine
@@ -97,18 +125,35 @@ function readTableDataRows(rows, headers, ignore, startIndex) {
         if (rowSel.children('td').length === 0)
             continue;
         const cellEls = rowSel.children('th, td');
+        // Row-header reconciliation. Two real shapes exist:
+        //   (a) the header row carries a (usually blank) CORNER cell for the
+        //       row-label column — widths already agree, keep every cell;
+        //   (b) the header row OMITS it — the row is then wider than the header
+        //       and reading the <th> would shift every column right by one.
+        // Detect (b) by an excess that is fully explained by leading <th> cells,
+        // and drop exactly that many. A row that is merely ragged (excess <td>s,
+        // or a leading <td>) is untouched.
+        let offset = 0;
+        if (headers.length > 0 && cellEls.length > headers.length) {
+            let leadingTh = 0;
+            while (leadingTh < cellEls.length && cellEls.eq(leadingTh).is('th'))
+                leadingTh += 1;
+            const excess = cellEls.length - headers.length;
+            if (excess <= leadingTh)
+                offset = excess;
+        }
         const cells = [];
-        for (let j = 0; j < cellEls.length; j++) {
+        for (let j = offset; j < cellEls.length; j++) {
             let text = readCellText(cellEls.eq(j), ignore);
             // Responsive-table pattern: some IR pages repeat the column header
             // INSIDE each cell as a (visually-hidden on desktop) label, e.g. NAB's
             // "<div>Payment date</div><div>2 July 2026</div>", whose .text()
             // collapses to "Payment date2 July 2026". Strip a leading copy of this
             // cell's own column header so downstream number/date parsing sees just
-            // the value. Guarded to an exact non-empty header prefix that leaves a
-            // non-empty remainder, so a legitimate value is never truncated.
-            const header = headers[j];
-            if (header != null && header !== '' && text.length > header.length && text.startsWith(header)) {
+            // the value — but only at a genuine DOM seam (see
+            // isRepeatedHeaderLabel), so a legitimate value is never truncated.
+            const header = headers[j - offset];
+            if (header != null && isRepeatedHeaderLabel(text, header)) {
                 text = text.slice(header.length).trim();
             }
             cells.push(text);
@@ -158,9 +203,15 @@ function readCombined($, combine, ignore) {
     // Following-sibling tables that share the header's column count; stop at the
     // first sibling that doesn't (or a non-table sibling) so an unrelated later
     // table with a coincidentally-equal width can't leak in.
-    const siblings = headerTable.nextAll('table');
+    // Walk EVERY following sibling, not just the <table> ones: filtering to
+    // tables first would silently skip over an intervening <h2>/<p>, so the
+    // documented "stop at the first sibling that doesn't match" could never fire
+    // and an unrelated later section's table could leak in.
+    const siblings = headerTable.nextAll();
     for (let i = 0; i < siblings.length; i++) {
         const sib = siblings.eq(i);
+        if (!sib.is('table'))
+            break;
         const sibRows = tableRows(sib);
         if (sibRows.length === 0)
             break;
@@ -208,8 +259,12 @@ function readGridContainer(container, grid, ignore) {
         const rowSel = allRows.eq(i);
         if (grid.headerRow != null && rowSel.is(grid.headerRow))
             continue; // never treat a header row as data
-        if (grid.headerRow == null && firstRowIsHeader && i === 0)
-            continue; // first row already consumed as header
+        // First row already consumed as header — either because no headerRow
+        // selector was given, OR because one was given and matched nothing (a
+        // typo'd/stale selector must degrade to "first row is the header", never
+        // silently emit the header as data row 0).
+        if (firstRowIsHeader && i === 0)
+            continue;
         const cells = gridRowCells(rowSel, grid.cell, ignore);
         if (cells.length === 0)
             continue;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "hazo_scrape",
-  "version": "1.6.0",
+  "version": "1.6.1",
   "description": "Generic source-agnostic web scraping engine with a network-free parse core.",
   "type": "module",
   "module": "./dist/index.js",
@@ -35,7 +35,7 @@
     "build:test-app": "npm run build && cd test-app && npm run build"
   },
   "peerDependencies": {
-    "hazo_core": "^1.0.1",
+    "hazo_core": "^1.3.0",
     "react": "^18.0.0 || ^19.0.0",
     "react-dom": "^18.0.0 || ^19.0.0"
   },