hazo_scrape 1.2.1 → 1.4.0

package/CHANGE_LOG.md

@@ -1,5 +1,24 @@
 # hazo_scrape — Change Log
 
+## 1.4.0 — 2026-07-24
+
+### Added — `parseDate` optional `opts.extractLeading` (+ `mapRows` `opts.dateExtractLeading`)
+- `parseDate(raw, { extractLeading: true })` accepts a date at the START of the cell even when trailing text follows it — e.g. `parseDate('25/06/2024 - special dividend', { formats: ['DD/MM/YYYY'], extractLeading: true })` now returns `'2024-06-25'` (previously `null`, since every date pattern was `^…$`-anchored). Only the leading date token is read; trailing text is ignored, never parsed. Leading junk is still rejected (the date must be the first token), a partial year is guarded against (`'25/06/20245'` → `null`), and the date itself is still matched exactly and disambiguated by the same rules — the "never guess" contract holds. Default off; omitting it preserves the exact prior whole-string behaviour.
+- `mapRows(table, map, keywords, { dateExtractLeading: true })` forwards this to `parseDate` for every date-typed cell (alongside the existing `dateFormats`). Purely additive.
+- Motivation: real IR pages (e.g. WBC's dividend history) staple a label onto a date cell — `"25/06/2024 - special dividend"` — which the strict parser correctly refused, dropping the payment date to `null`. A caller that KNOWS a source does this can now opt in via `SourceRegistryEntry.nuances` without this engine ever guessing.
+
+## 1.3.0 — 2026-07-24
+
+### Added — `extractTable` div-grid mode (`opts.grid`)
+- `extractTable(html, { grid })` reads a "table" built from styled `<div>`s (CSS grid/flex) instead of a real `<table>`. `GridSelector = { container?, row, cell, headerRow? }` — all plain CSS selectors. When `container` matches more than one grid, the wrapper with the most data rows wins (mirrors the existing pickBestTable "the real data grid is the biggest" bias). Motivation: IR pages like CBA's `dividend-information.html` render dividend history as `.complex-table` div-grids that a static fetch sees in markup but `extractTable` (table-only) could not read — surfacing as `0 rows × 0 cols`. Purely additive; omit `grid` for unchanged behavior. Takes precedence over `select`.
+
+### Fixed — `extractTable` row-header alignment + repeated in-cell header labels
+- Data-row cells are now read as `th, td` (not `td` alone), so an accessible table whose first column is a `<th scope="row">` row-label (e.g. NAB's payment-history table) stays column-aligned with its header row instead of shifting every column left by one. Pure-`<th>` section rows (zero `<td>`) are still skipped as before.
+- A leading copy of a cell's own column header baked INTO the cell is stripped (responsive-label pattern: NAB emits `<div>Payment date</div><div>2 July 2026</div>`, so `.text()` collapses to `"Payment date2 July 2026"`). Guarded to an exact non-empty header prefix leaving a non-empty remainder, so a legitimate value is never truncated. Together these were why NAB scraped 90 rows but mapped 0.
+
+### Fixed — `parseNumber` "cents" unit word
+- `parseNumber('85 cents')` now returns `85` (previously `null` — `Number('85 cents')` is `NaN`). Only TOKENIZES the minor-unit word; the cents→dollars SCALING remains the caller's job via a domain `amount_unit` (unchanged, per the file's no-cents-conversion contract). Word-boundaried, so `'recent'` is untouched.
+
 ## 1.2.1 — 2026-07-23
 
 ### Fixed — republish of 1.2.0's `dist`

package/README.md

@@ -77,11 +77,14 @@ the first. `opts.dateGuard` drops date-looking headers from any key whose
 `type !== 'date'`. `opts.required` lists keys that must have at least one
 candidate for `map.ok` to be `true`.
 
-### `mapRows(table: { headers; rows }, map: ColumnMap, keywords: ColumnKeywords): MappedRow[]`
+### `mapRows(table: { headers; rows }, map: ColumnMap, keywords: ColumnKeywords, opts?: { dateFormats?: string[]; dateExtractLeading?: boolean }): MappedRow[]`
 
 Joins a `ColumnMap` back onto `table.rows`, producing one `MappedRow` per data
 row. Each matched key holds an array of `Cell`s (one per candidate header),
-each parsed according to that key's declared `type`.
+each parsed according to that key's declared `type`. `opts.dateFormats` and
+`opts.dateExtractLeading` are forwarded verbatim to `parseDate`'s `opts.formats`
+and `opts.extractLeading` for every date-typed cell; both default off, so
+omitting `opts` preserves the exact prior behaviour.
 
 ### `parseNumber(raw: string): number | null`
 
@@ -90,13 +93,21 @@ and `%`, then parses what remains as a number. Returns `null` when nothing
 parseable is left. Does **not** convert cents to dollars — domain scaling is
 the caller's job.
 
-### `parseDate(raw: string, opts?: { formats?: string[] }): string | null`
+### `parseDate(raw: string, opts?: { formats?: string[]; extractLeading?: boolean }): string | null`
 
 Parses a handful of common date text formats (ISO, `D Mon YYYY`, `Mon D,
 YYYY`, and `D/M/Y` slash dates) into an ISO `yyyy-mm-dd` string. Returns
 `null` — never a guess — for anything unrecognised or genuinely ambiguous
 (e.g. `05/03/2026` with no `opts.formats` hint and both components `<= 12`).
 
+`opts.extractLeading` (default off) accepts a date at the **start** of the cell
+even when trailing text follows it — `parseDate('25/06/2024 - special dividend',
+{ formats: ['DD/MM/YYYY'], extractLeading: true })` → `'2024-06-25'`. Only the
+leading date token is read; the trailing text is ignored, never parsed. Leading
+junk is still rejected (the date must be the first token) and the date itself is
+still matched exactly and disambiguated by the same rules — so the "never guess"
+contract holds. For real IR pages that staple a label onto a date cell.
+
 ### Types
 
 ```ts

package/dist/parse/extract_table.d.ts

@@ -3,8 +3,36 @@ export type ExtractTableResult = {
     rows: string[][];
     warning?: string;
 };
+/**
+ * Div-grid mode selectors — for "tables" built from styled <div>s (CSS
+ * grid/flex) instead of a real <table>. A static HTML fetch sees these in
+ * the markup even when the page's data table proper is JS-rendered (e.g.
+ * CBA's `.complex-table`). All selectors are plain CSS.
+ */
+export type GridSelector = {
+    /**
+     * Element(s) that wrap one grid. When the selector matches MORE than one,
+     * the wrapper holding the most data rows wins — mirroring pickBestTable's
+     * "the real data grid is the biggest one" bias, so an upcoming-dividend
+     * teaser grid never shadows the full history grid. Omit to search the
+     * whole document.
+     */
+    container?: string;
+    /** Selector (searched under the container) matching every row — header and data alike. */
+    row: string;
+    /** Selector (searched under a row) matching each cell. */
+    cell: string;
+    /**
+     * Selector identifying the header row among the `row` matches. Rows that
+     * match it are treated as headers (never data); the first such row supplies
+     * the column headers. Omit to treat the first row as the header.
+     */
+    headerRow?: string;
+};
 export type ExtractTableOptions = {
     select?: string;
+    /** Read a <div>-based pseudo-table instead of a real <table>. Takes precedence over `select`. */
+    grid?: GridSelector;
 };
 export declare function extractTable(html: string, opts?: ExtractTableOptions): ExtractTableResult;
 //# sourceMappingURL=extract_table.d.ts.map

package/dist/parse/extract_table.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"extract_table.d.ts","sourceRoot":"","sources":["../../src/parse/extract_table.ts"],"names":[],"mappings":"AAsBA,MAAM,MAAM,kBAAkB,GAAG;IAC/B,OAAO,EAAE,MAAM,EAAE,CAAC;IAClB,IAAI,EAAE,MAAM,EAAE,EAAE,CAAC;IACjB,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB,CAAC;AAEF,MAAM,MAAM,mBAAmB,GAAG;IAChC,MAAM,CAAC,EAAE,MAAM,CAAC;CACjB,CAAC;AAwGF,wBAAgB,YAAY,CAAC,IAAI,EAAE,MAAM,EAAE,IAAI,CAAC,EAAE,mBAAmB,GAAG,kBAAkB,CAyBzF"}
+{"version":3,"file":"extract_table.d.ts","sourceRoot":"","sources":["../../src/parse/extract_table.ts"],"names":[],"mappings":"AAsBA,MAAM,MAAM,kBAAkB,GAAG;IAC/B,OAAO,EAAE,MAAM,EAAE,CAAC;IAClB,IAAI,EAAE,MAAM,EAAE,EAAE,CAAC;IACjB,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB,CAAC;AAEF;;;;;GAKG;AACH,MAAM,MAAM,YAAY,GAAG;IACzB;;;;;;OAMG;IACH,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,0FAA0F;IAC1F,GAAG,EAAE,MAAM,CAAC;IACZ,0DAA0D;IAC1D,IAAI,EAAE,MAAM,CAAC;IACb;;;;OAIG;IACH,SAAS,CAAC,EAAE,MAAM,CAAC;CACpB,CAAC;AAEF,MAAM,MAAM,mBAAmB,GAAG;IAChC,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,iGAAiG;IACjG,IAAI,CAAC,EAAE,YAAY,CAAC;CACrB,CAAC;AAwLF,wBAAgB,YAAY,CAAC,IAAI,EAAE,MAAM,EAAE,IAAI,CAAC,EAAE,mBAAmB,GAAG,kBAAkB,CA6BzF"}

package/dist/parse/extract_table.js

@@ -80,17 +80,96 @@ function readTable(tableSel) {
     const dataRowsSel = rows.slice(1);
     const dataRows = [];
     for (let i = 0; i < dataRowsSel.length; i++) {
-        const tds = dataRowsSel.eq(i).children('td');
-        if (tds.length === 0)
-            continue; // skip non-data rows (prototype behaviour)
+        const rowSel = dataRowsSel.eq(i);
+        // A genuine data row has at least one <td>; a pure-<th> row (a section
+        // divider spanning the table) is skipped — prototype behaviour. But the
+        // cells themselves are then read as `th, td`, not `td` alone, so a
+        // row-header cell (<th scope="row">, common in accessible IR tables
+        // whose first column labels the row) is kept and the row stays
+        // column-aligned with the header row (which is likewise read as th,td).
+        if (rowSel.children('td').length === 0)
+            continue;
+        const cellEls = rowSel.children('th, td');
         const cells = [];
-        for (let j = 0; j < tds.length; j++) {
-            cells.push(normalizeCell(tds.eq(j).text()));
+        for (let j = 0; j < cellEls.length; j++) {
+            let text = normalizeCell(cellEls.eq(j).text());
+            // Responsive-table pattern: some IR pages repeat the column header
+            // INSIDE each cell as a (visually-hidden on desktop) label, e.g. NAB's
+            // "<div>Payment date</div><div>2 July 2026</div>", whose .text()
+            // collapses to "Payment date2 July 2026". Strip a leading copy of this
+            // cell's own column header so downstream number/date parsing sees just
+            // the value. Guarded to an exact non-empty header prefix that leaves a
+            // non-empty remainder, so a legitimate value is never truncated.
+            const header = headers[j];
+            if (header != null && header !== '' && text.length > header.length && text.startsWith(header)) {
+                text = text.slice(header.length).trim();
+            }
+            cells.push(text);
         }
         dataRows.push(cells);
     }
     return { headers, rows: dataRows };
 }
+// Read one cell's normalized text list from a row selection, using a cell
+// sub-selector (div-grid mode — see GridSelector).
+function gridRowCells(rowSel, cellSelector) {
+    const cellEls = rowSel.find(cellSelector);
+    const cells = [];
+    for (let j = 0; j < cellEls.length; j++)
+        cells.push(normalizeCell(cellEls.eq(j).text()));
+    return cells;
+}
+// Read a single div-grid wrapper into the same {headers, rows} shape a real
+// <table> yields, so every downstream stage (mapColumns/mapRows) is oblivious
+// to which markup the data came from.
+function readGridContainer(container, grid) {
+    const allRows = container.find(grid.row);
+    if (allRows.length === 0)
+        return { headers: [], rows: [] };
+    let headers = [];
+    let firstRowIsHeader = false;
+    if (grid.headerRow != null) {
+        const headerRows = allRows.filter(grid.headerRow);
+        if (headerRows.length > 0)
+            headers = gridRowCells(headerRows.eq(0), grid.cell);
+    }
+    if (headers.length === 0) {
+        headers = gridRowCells(allRows.eq(0), grid.cell);
+        firstRowIsHeader = true;
+    }
+    const dataRows = [];
+    for (let i = 0; i < allRows.length; i++) {
+        const rowSel = allRows.eq(i);
+        if (grid.headerRow != null && rowSel.is(grid.headerRow))
+            continue; // never treat a header row as data
+        if (grid.headerRow == null && firstRowIsHeader && i === 0)
+            continue; // first row already consumed as header
+        const cells = gridRowCells(rowSel, grid.cell);
+        if (cells.length === 0)
+            continue;
+        dataRows.push(cells);
+    }
+    return { headers, rows: dataRows };
+}
+function readGrid($, grid) {
+    const containers = grid.container != null ? $(grid.container) : $.root();
+    if (grid.container != null && containers.length === 0) {
+        return { headers: [], rows: [], warning: `The selector "${grid.container}" matched no element on the page.` };
+    }
+    // When the container selector matches several grids, keep the one with the
+    // most data rows (same "the real data is the biggest grid" heuristic
+    // pickBestTable uses for real tables). Ties keep the first encountered.
+    let best = null;
+    for (let i = 0; i < containers.length; i++) {
+        const parsed = readGridContainer(containers.eq(i), grid);
+        if (best === null || parsed.rows.length > best.rows.length)
+            best = parsed;
+    }
+    if (best === null || (best.headers.length === 0 && best.rows.length === 0)) {
+        return { headers: [], rows: [], warning: NO_TABLE_WARNING };
+    }
+    return best;
+}
 function pickBestTable($) {
     const tables = $('table');
     let best = null;
@@ -111,6 +190,9 @@ function pickBestTable($) {
 }
 export function extractTable(html, opts) {
     const $ = load(html);
+    if (opts?.grid != null) {
+        return readGrid($, opts.grid);
+    }
     if (opts?.select != null) {
         const matched = $(opts.select);
         if (matched.length === 0) {

package/dist/parse/map_rows.d.ts

@@ -1,6 +1,7 @@
 import type { ColumnKeywords, ColumnMap, MappedRow } from './types.js';
 export interface MapRowsOptions {
     dateFormats?: string[];
+    dateExtractLeading?: boolean;
 }
 export declare function mapRows(table: {
     headers: string[];

package/dist/parse/map_rows.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"map_rows.d.ts","sourceRoot":"","sources":["../../src/parse/map_rows.ts"],"names":[],"mappings":"AA2BA,OAAO,KAAK,EAAQ,cAAc,EAAE,SAAS,EAAE,SAAS,EAAE,MAAM,YAAY,CAAC;AAI7E,MAAM,WAAW,cAAc;IAC7B,WAAW,CAAC,EAAE,MAAM,EAAE,CAAC;CACxB;AAaD,wBAAgB,OAAO,CACrB,KAAK,EAAE;IAAE,OAAO,EAAE,MAAM,EAAE,CAAC;IAAC,IAAI,EAAE,MAAM,EAAE,EAAE,CAAA;CAAE,EAC9C,GAAG,EAAE,SAAS,EACd,QAAQ,EAAE,cAAc,EACxB,IAAI,CAAC,EAAE,cAAc,GACpB,SAAS,EAAE,CAwBb"}
+{"version":3,"file":"map_rows.d.ts","sourceRoot":"","sources":["../../src/parse/map_rows.ts"],"names":[],"mappings":"AA2BA,OAAO,KAAK,EAAQ,cAAc,EAAE,SAAS,EAAE,SAAS,EAAE,MAAM,YAAY,CAAC;AAI7E,MAAM,WAAW,cAAc;IAC7B,WAAW,CAAC,EAAE,MAAM,EAAE,CAAC;IAMvB,kBAAkB,CAAC,EAAE,OAAO,CAAC;CAC9B;AAmBD,wBAAgB,OAAO,CACrB,KAAK,EAAE;IAAE,OAAO,EAAE,MAAM,EAAE,CAAC;IAAC,IAAI,EAAE,MAAM,EAAE,EAAE,CAAA;CAAE,EAC9C,GAAG,EAAE,SAAS,EACd,QAAQ,EAAE,cAAc,EACxB,IAAI,CAAC,EAAE,cAAc,GACpB,SAAS,EAAE,CAyBb"}

package/dist/parse/map_rows.js

@@ -26,17 +26,22 @@
 // the exact prior behavior — this is purely additive.
 import { parseNumber } from './parse_number.js';
 import { parseDate } from './parse_date.js';
-function cellValue(type, raw, dateFormats) {
+function cellValue(type, raw, dateFormats, dateExtractLeading) {
     if (type === 'number')
         return parseNumber(raw);
-    if (type === 'date')
-        return parseDate(raw, dateFormats ? { formats: dateFormats } : undefined);
+    if (type === 'date') {
+        return parseDate(raw, {
+            ...(dateFormats ? { formats: dateFormats } : {}),
+            ...(dateExtractLeading ? { extractLeading: true } : {}),
+        });
+    }
     const trimmed = raw.trim();
     return trimmed === '' ? null : trimmed;
 }
 export function mapRows(table, map, keywords, opts) {
     const keys = Object.keys(map.matched);
     const dateFormats = opts?.dateFormats;
+    const dateExtractLeading = opts?.dateExtractLeading;
     return table.rows.map((row) => {
         const mappedRow = {};
         for (const key of keys) {
@@ -48,7 +53,7 @@ export function mapRows(table, map, keywords, opts) {
                     header: candidate.header,
                     index: candidate.index,
                     raw,
-                    value: cellValue(type, raw, dateFormats),
+                    value: cellValue(type, raw, dateFormats, dateExtractLeading),
                 };
             });
         }

package/dist/parse/parse_date.d.ts

@@ -1,4 +1,5 @@
 export declare function parseDate(raw: string, opts?: {
     formats?: string[];
+    extractLeading?: boolean;
 }): string | null;
 //# sourceMappingURL=parse_date.d.ts.map

package/dist/parse/parse_date.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"parse_date.d.ts","sourceRoot":"","sources":["../../src/parse/parse_date.ts"],"names":[],"mappings":"AA6EA,wBAAgB,SAAS,CAAC,GAAG,EAAE,MAAM,EAAE,IAAI,CAAC,EAAE;IAAE,OAAO,CAAC,EAAE,MAAM,EAAE,CAAA;CAAE,GAAG,MAAM,GAAG,IAAI,CAoCnF"}
+{"version":3,"file":"parse_date.d.ts","sourceRoot":"","sources":["../../src/parse/parse_date.ts"],"names":[],"mappings":"AAuFA,wBAAgB,SAAS,CAAC,GAAG,EAAE,MAAM,EAAE,IAAI,CAAC,EAAE;IAAE,OAAO,CAAC,EAAE,MAAM,EAAE,CAAC;IAAC,cAAc,CAAC,EAAE,OAAO,CAAA;CAAE,GAAG,MAAM,GAAG,IAAI,CA4C7G"}

package/dist/parse/parse_date.js

@@ -8,6 +8,16 @@
 //     blindly assuming day-first. Slash dates are only resolved without a
 //     hint when one ordering is calendar-impossible (a component > 12), or
 //     when the caller supplies `opts.formats` to disambiguate.
+//
+// opts.extractLeading (added 2026-07-24): OFF by default. When true, a date
+// at the START of the cell is accepted even if trailing text follows it
+// (e.g. "25/06/2024 - special dividend" -> "2024-06-25"). Only the leading
+// date token is read; the trailing text is ignored, never parsed. Leading
+// junk is still rejected — the date must be the first token. This preserves
+// the "never guess" contract (the date itself is still matched exactly and
+// still disambiguated by the same rules) while tolerating real IR pages that
+// staple a label onto a date cell. Omitting it preserves exact prior
+// behaviour (a whole-string anchored match) — this is purely additive.
 const MONTHS = {
     jan: 1,
     feb: 2,
@@ -80,13 +90,20 @@ export function parseDate(raw, opts) {
     const s = String(raw).trim();
     if (s === '')
         return null;
+    // End-of-token anchor. Strict mode (default) anchors to end-of-string, so
+    // any trailing text fails the match. `extractLeading` relaxes that to "no
+    // further word char or slash follows the date" — enough to end the date
+    // token cleanly (a following space, hyphen-with-space, punctuation, or EOS)
+    // without letting a partial number ("2024" out of "20245") slip through.
+    const tail = opts?.extractLeading ? '(?![\\w/])' : '$';
+    const re = (body) => new RegExp(`^${body}${tail}`);
     // ISO: yyyy-mm-dd
-    let m = s.match(/^(\d{4})-(\d{2})-(\d{2})$/);
+    let m = s.match(re('(\\d{4})-(\\d{2})-(\\d{2})'));
     if (m) {
         return toIsoIfValid(Number(m[1]), Number(m[2]), Number(m[3]));
     }
     // "D Mon YYYY" / "D Month YYYY", e.g. "5 Mar 2026", "18 November 2025"
-    m = s.match(/^(\d{1,2})\s+([A-Za-z]{3,})\s+(\d{4})$/);
+    m = s.match(re('(\\d{1,2})\\s+([A-Za-z]{3,})\\s+(\\d{4})'));
     if (m) {
         const month = monthFromName(m[2]);
         if (month == null)
@@ -94,7 +111,7 @@ export function parseDate(raw, opts) {
         return toIsoIfValid(Number(m[3]), month, Number(m[1]));
     }
     // "Mon D, YYYY" / "Month D YYYY", comma optional, e.g. "Mar 5, 2026"
-    m = s.match(/^([A-Za-z]{3,})\s+(\d{1,2}),?\s+(\d{4})$/);
+    m = s.match(re('([A-Za-z]{3,})\\s+(\\d{1,2}),?\\s+(\\d{4})'));
     if (m) {
         const month = monthFromName(m[1]);
         if (month == null)
@@ -104,7 +121,7 @@ export function parseDate(raw, opts) {
     // Slash dates: D/M/Y or M/D/Y — ambiguous without a hint or a proof by
     // calendar impossibility (see resolveSlashDate). Two-digit years are
     // intentionally rejected: the year group below requires exactly 4 digits.
-    m = s.match(/^(\d{1,2})\/(\d{1,2})\/(\d{4})$/);
+    m = s.match(re('(\\d{1,2})\\/(\\d{1,2})\\/(\\d{4})'));
     if (m) {
         return resolveSlashDate(Number(m[1]), Number(m[2]), Number(m[3]), opts?.formats);
     }

package/dist/parse/parse_number.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"parse_number.d.ts","sourceRoot":"","sources":["../../src/parse/parse_number.ts"],"names":[],"mappings":"AAaA,wBAAgB,WAAW,CAAC,GAAG,EAAE,MAAM,GAAG,MAAM,GAAG,IAAI,CAqBtD"}
+{"version":3,"file":"parse_number.d.ts","sourceRoot":"","sources":["../../src/parse/parse_number.ts"],"names":[],"mappings":"AAaA,wBAAgB,WAAW,CAAC,GAAG,EAAE,MAAM,GAAG,MAAM,GAAG,IAAI,CA4BtD"}

package/dist/parse/parse_number.js

@@ -23,6 +23,12 @@ export function parseNumber(raw) {
     // Strip thousands separators and percent signs.
     s = s.replace(/,/g, '');
     s = s.replace(/%/g, '');
+    // Strip a trailing minor-unit word — some IR pages write an amount as
+    // "85 cents". This only TOKENIZES ("85 cents" -> "85"); the cents->dollars
+    // SCALING remains the caller's job via a domain amount_unit (see the file
+    // header's no-cents-conversion note). Word-boundaried so it never touches
+    // the digits or an unrelated word like "recent".
+    s = s.replace(/\bcents?\b/gi, '');
     s = s.trim();
     if (s === '')
         return null;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "hazo_scrape",
-  "version": "1.2.1",
+  "version": "1.4.0",
   "description": "Generic source-agnostic web scraping engine with a network-free parse core.",
   "type": "module",
   "module": "./dist/index.js",