page-markdown 0.1.5 → 0.3.0

package/README.md

@@ -3,8 +3,9 @@
 Faithful page-to-Markdown for humans and agents. Give it a web page's DOM and
 it returns clean Markdown plus metadata. It is written for browsers, for
 example an extension's content script, and runs in Node with a DOM library.
-It is tested in Node and in headless Chrome and Edge, but not yet in Firefox or
-Safari (see [Known limits](#known-limits-v01)). It was built for WATT3D's
+It is tested in Node, in headless Chrome and Edge, and in a real extension
+(HELIOS, in Brave), but not yet in Firefox or Safari (see
+[Known limits](#known-limits-v02)). It was built for WATT3D's
 browser extensions.
 
 ```
@@ -13,6 +14,7 @@ live Document
   → standardize   <pre> → fenced code with language; heading wrappers unwrapped; empty and
                   javascript: links removed; absolute links and images; lazy images resolved
   → extract       Mozilla Readability on a copy; full-page fallback when it finds too little;
+                  main-content fallback when it keeps only a small share of the page;
                   a dropped lead paragraph restored; a clean title heading
   → convert       Turndown + GitHub-flavoured tables, strikethrough and task lists; tables never
                   come out as raw HTML
@@ -42,7 +44,7 @@ import { pageToMarkdown } from "page-markdown";
 const result = pageToMarkdown(document);
 result.markdown;            // "# Title\n\n…"
 result.metadata;            // { url, canonicalUrl, title, author, published, siteName, lang, … }
-result.extraction.method;   // "readability" | "full_page"
+result.extraction.method;   // "readability" | "main_content" | "full_page" | custom method
 
 // In Node, with any DOM implementation (linkedom shown)
 import { parseHTML } from "linkedom";
@@ -55,15 +57,16 @@ pageToMarkdown(doc, { url: "https://example.com/article" });
 | Option | Default | Meaning |
 |---|---|---|
 | `url` | the document's URL | Base for absolute links and images |
-| `extract` | `"readable"` | `"readable"` (Readability, full-page fallback), `"full"`, or a function `(docCopy) => { html, method }` |
-| `minExtractChars` | `100` | Below this, readable extraction falls back to the full page |
+| `extract` | `"readable"` | `"readable"` (Readability with main-content or full-page fallback), `"full"`, or a function `(docCopy) => { html, method }` |
+| `minExtractChars` | `100` | Below this, Readability falls back to a sufficiently long `<main>` or `[role="main"]`, else the full page; custom extractors fall back to the full page |
+| `minExtractShare` | `0.4` | When Readability keeps less than this share of the page's main content (and misses at least 500 characters), the main content is used instead: forum threads and product grids. `0` turns it off |
 | `includeFullPage` | `false` | Also return `fullPageMarkdown` |
-| `includeTitle` | `true` | Start readable output with `# <headline>`: the page's H1, else its title without the site name |
+| `includeTitle` | `true` | Add `# <headline>` to Readability or main-content output when no H1 remains; skipped in `text_only` mode |
 | `recoverLead` | `true` | Restore a lead paragraph the extractor dropped, when it matches the meta description |
 | `pageFacts` | `false` | Append `## Page facts`: short label/value pairs outside the article (a sidebar's version or license, spec rows, infobox rows) that the extractor dropped. `extraction.pageFacts` counts them |
 | `removeHiddenText` | `false` | Remove text hidden from people but read by models: runs of tag characters, variation selectors, invisible operators and reserved tag-plane code points, found by [text-sanctifier](https://github.com/iWhatty/text-sanctifier)'s `findHiddenRuns`. Applies to `markdown` (fenced code included), `fullPageMarkdown` and `metadata`. The England, Scotland and Wales flags and a lone emoji or text presentation selector are kept. `omissions` gets one entry per kind (`hidden_tag`, `hidden_variation_selector`, `hidden_invisible_operator`, `hidden_reserved_tag_plane`) with `count` runs and `chars` code points removed from `markdown` |
 | `mode` | `"full"` | `"no_links"` keeps link text; `"text_only"` also drops structure. Fenced code is never changed |
-| `sanitize` | see `DEFAULT_SANITIZE_OPTIONS` | `stripAds`, `extraStripSelectors` (e.g. `["[data-app-overlay]"]`), shadow limits |
+| `sanitize` | see `DEFAULT_SANITIZE_OPTIONS` | `stripAds`, `extraStripSelectors` (e.g. `["[data-app-overlay]"]`), shadow limits, which apply per host in the page's own tree and don't charge slotted page content |
 | `readability` | `{}` | Passed to Readability |
 
 The stages are also exported for advanced use: `sanitizeToInertDocument`,
@@ -84,18 +87,65 @@ The stages are also exported for advanced use: `sanitizeToInertDocument`,
 | `metadata.description`, `author`, `published`, `modified`, `siteName`, `lang` | `string` | From meta tags and JSON-LD, `""` when the page doesn't state them. `siteName` falls back to the hostname |
 | `metadata.headline` | `string?` | The heading chosen for the capture |
 | `metadata.extractedTitle`, `byline` | `string?` | What the extractor reported |
-| `extraction.method` | `string` | `"readability"`, `"full_page"`, or the name a custom extractor gave |
-| `extraction.fallbackReason` | `string?` | Why extraction fell back to the full page: `extractor_found_nothing`, `extracted_text_below_minimum` or `extractor_error: …` |
+| `extraction.method` | `string` | `"readability"`, `"main_content"`, `"full_page"`, or the name a custom extractor gave |
+| `extraction.fallbackReason` | `string?` | Why extraction fell back: `extracted_small_share_of_page`, `extracted_site_menu` or `extracted_few_sections_of_page` use the main content; `extractor_found_nothing`, `extracted_text_below_minimum` or `extractor_error: ...` use a sufficiently long main-content region for Readability, else the full page. Custom extractors fall back to the full page |
 | `extraction.leadRecovered` | `true?` | A dropped lead paragraph was restored |
 | `extraction.pageFacts` | `number?` | How many facts `pageFacts` appended |
-| `omissions[]` | `{ kind, count, note, chars? }` | What was left out or removed. `kind` is `shadow_content`, `json_line` or a `hidden_*` kind; only `hidden_*` entries have `chars` |
-| `warnings[]` | `string` | Currently the full-page fallback, in words |
+| `omissions[]` | `{ kind, count, note, chars? }` | What was left out or removed. `kind` is `shadow_content`, `json_line`, `inactive_tabs`, `comments` or a `hidden_*` kind; only `hidden_*` entries have `chars`. All but `hidden_*` also add a closing line to `markdown`, as a fallback does, so a caller that passes on only the Markdown still says so |
+| `warnings[]` | `string` | Read these before trusting a capture: a fallback, or shadow content cut off at the capture limit, in which case the main content may be missing |
 | `stats.mode` | `string` | The output mode used |
-| `stats.extractedHtmlChars`, `convertedChars`, `markdownChars` | `number` | Sizes at each step. `markdownChars` is measured before `removeHiddenText` and the shadow omission notice |
-| `stats.standardization` | `object` | Counts of what standardizing changed: `lazyImagesResolved`, `inlineImagesDropped`, `scriptLinksUnwrapped`, `emptyLinksRemoved`, `preWrapped`, `headingWidgetsRemoved`, `headingsUnwrapped` |
+| `stats.extractedHtmlChars`, `convertedChars`, `markdownChars` | `number` | Sizes at each step. `markdownChars` is the returned Markdown's length |
+| `stats.pageTextChars` | `number` | Whitespace-normalized text length of the standardized page at capture time |
+| `stats.standardization` | `object` | Counts of what standardizing changed: `lazyImagesResolved`, `inlineImagesDropped`, `scriptLinksUnwrapped`, `invalidLinksUnwrapped`, `emptyLinksRemoved`, `preWrapped`, `headingWidgetsRemoved`, `headingsUnwrapped`; `omittedTabs` lists labels of inactive panels left out |
 
 A `?` marks a field that is absent, not empty, when it doesn't apply.
 
+### Concealed text (opt-in)
+
+Detection is off by default. Enable it for a live document:
+
+```js
+const result = pageToMarkdown(document, {
+  detectConcealedText: true,
+  includeFullPage: true,
+});
+```
+
+Supported computed styles separate concealed DOM text from the article and
+return it in `concealedText` as `{ reason, chars, text }` blocks. Text is capped
+at 2,048 UTF-16 units per block, 16,384 total and 100 entries;
+`concealedTextTruncated` reports capping. Omission counts and character totals
+are uncapped. A `Concealed text omitted:` notice appears in both `markdown`
+and `fullPageMarkdown` when text is captured.
+
+The seven reasons are `display_none`, `visibility_hidden`, `opacity_zero`,
+`font_size_zero`, `clipped`, `transparent_text` and `same_color`. Supported
+clips are zero-area legacy rectangles on absolute/fixed elements,
+`inset(50%)` and zero-radius circles. Colour checks compare computed sRGB
+rgb()/rgba() values for an element's own text against its own or nearest
+declared opaque background.
+
+`concealmentCheck` is `disabled` by default (no computed-style calls),
+`static-only` when enabled without a computed-style API, `computed` after a
+successful check, or `incomplete` if the capability probe or a consulted
+property fails. Detection applies to live documents only; enabled static
+input retains its previous output. A top-level `detectConcealedText` option
+takes precedence over `sanitize.detectConcealedText`, then the default false.
+
+Limits include white text with no declared opaque background, non-zero alpha,
+near-colour matches, and unparsed OKLCH and Display P3 colours. Intervening
+images or translucent backgrounds and uncertain text painting effects prevent
+colour equality detection. Existing short inactive tab, collapsed answer and
+code wrapper exceptions remain; independent concealment reasons still apply.
+Off-screen or covered text, attributes, metadata and custom-extractor output
+are outside this check.
+
+The scroll-reveal rule is **provisional**: zero-opacity content with an
+opacity-applicable transition or named animation of nonzero duration stays
+in place with a warning. A declaration does not prove it will appear, and a
+page can use one to evade detection. Real-site validation is still required.
+Chrome is the only browser tested; Firefox and Safari are untested.
+
 ### TypeScript
 
 Types ship with the package, generated from the JSDoc in `src`: `pageToMarkdown`,
@@ -124,7 +174,7 @@ const { markdown: safe, omissions } = pageToMarkdown(document, { removeHiddenTex
 ## Quality
 
 `npm test` runs the contract suite. It covers:
-- golden outputs for 34 fixtures;
+- golden outputs for 51 fixtures;
 - regressions for known converter bugs, including dropped lead paragraphs,
   raw HTML tables, flattened code in tables, site-suffixed titles and
   unescaped table pipes;
@@ -146,8 +196,11 @@ checks what Node can't show:
   children in rendered order;
 - closed shadow roots stay out of the capture.
 
-Set `PAGE_MARKDOWN_BROWSER` to choose the browser. The built page is kept at
-`.browser-test/index.html` for opening by hand.
+It launches a browser, so it runs only when a person sets
+`PAGE_MARKDOWN_ALLOW_BROWSER_LAUNCH=1` in their own shell; agents must not set it.
+Without the flag it builds `.browser-test/index.html`, which you can open in a
+browser that is already running, and exits with an error. Set
+`PAGE_MARKDOWN_BROWSER` to choose the browser.
 
 `npm run bench` scores the package against two baselines: Turndown over the page
 body, and Readability then Turndown. It runs 15 fixture checks and 117 probes on
@@ -159,38 +212,57 @@ recall against a hand-picked reference for each page. `npm run bench` only print
 |---|---|---|---|
 | Fixtures, written with the package | 15/15 | 7/15 | 3/15 |
 | Five pages the fixes were tuned on | 17/17 | 9/17 | 8/17 |
-| **11 held-out pages** | **84/100** | **74/100** | **61/100** |
-| Held-out word recall / precision | 86.3% / 95.5% | 85.4% / 95.6% | 98.2% / 72.1% |
-
-The held-out row is the one to weigh. On those pages the package leads plain
-Readability + Turndown on probes, and the two are level on word recall and
-precision. The pages are no longer fully unseen: the package scored 75/100 before
-two fixes that came from their failures, so read 84 as a little generous. What it
-still fails there:
-
-- **Most H2s dropped** on Docusaurus and Starlight, by Readability; Starlight also
-  keeps "Section titled" anchor text.
-- **Non-default tab panels dropped** (Starlight).
-- **Comments and product grids lost**, by Readability: 36% word recall on a
-  Lobsters thread, 50% on an IKEA category page.
-- **Promo boxes kept** inside a BBC Arabic article.
+| **11 pages, once held out** | **98/100** | **74/100** | **61/100** |
+| Their word recall / precision | 97.6% / 89.1% | 85.4% / 95.6% | 98.2% / 72.1% |
 
-Browser bundle: about 67 KB minified, 24 KB gzip (`npm run size`, ceiling 68 KB
-minified). Readability is most of it.
+Don't read 98 as a score on unseen pages. These probes were written before any
+engine ran and the package first scored 75/100, but most of the gain since comes
+from fixes for these same pages' failures. New pages are the real test.
 
-`npm run ci` runs the contract suite, the type check, the browser run and the size check, and must exit 0
-before a commit. It runs locally; the repo deliberately has no cloud CI.
+It gains recall (97.6% against 85.4%) by keeping more, and pays in precision
+(89.1% against 95.6%). The cost is mostly on the IKEA category page: 29.7% precision,
+because the main-content fallback keeps the page's banners and filters along with
+its products. What it still fails on these pages:
 
-## Known limits (v0.1)
+- **Promo boxes kept** inside a BBC Arabic article.
+
+Browser bundle: about 86.0 KB minified, 31.2 KB gzip (`npm run size`, ceiling 86 KB
+minified). Readability is most of it.
 
-- **Chromium only so far.** Tested in Node (linkedom) and in headless Chrome
-  and Edge, but not in Firefox or Safari, and not yet inside a real extension's
-  content script.
+`npm run ci` runs the contract suite, the type check and the size check, and must
+exit 0 before a commit. It launches no browser, so run the browser check yourself
+before a release when the DOM handling has changed. It runs locally; the repo
+deliberately has no cloud CI.
+
+## Known limits (v0.2)
+
+- **Chromium only so far.** Tested in Node (linkedom), in headless Chrome and
+  Edge, and on live pages in a real extension (HELIOS, in Brave), but not
+  in Firefox or Safari.
+- **Extraction can still be wrong without saying so.** The main-content fallback
+  and its warning catch a small share of the page, a kept site menu and lost
+  sections, not every wrong choice. If Readability picks the wrong part of a page
+  and that part is large, nothing flags it. No closing line doesn't mean nothing
+  was lost. A short post among long replies can be dropped: a Mastodon status lost
+  the post itself, and its cause isn't known yet.
+- **Answers hidden on an ordinary article can be lost.** Collapsed FAQ answers are
+  kept by the main-content fallback only; Readability removes hidden text.
+- **Bylines and dates can be missing from the Markdown.** Readability often leaves
+  them out of the article. Look in `metadata.author` and `metadata.published`.
 - **Some legacy markup loses paragraph breaks.** Readability can drop runs of
   `<br>` between inline elements in old HTML. The text survives; a paragraph
   break may not.
-- **Hidden content stays.** The pipeline reads the static DOM, so elements
-  hidden only by CSS remain unless extraction drops them.
+- **A capture is a snapshot.** It reads the DOM as it is at call time, and it
+  can't tell whether the page has finished rendering. A page read too early looks
+  like a normal, short result. To wait, re-capture until `stats.pageTextChars`
+  settles.
+- **Captures hold whatever is in the DOM,** including text that other extensions
+  add. Use `sanitize.extraStripSelectors` for what you know about.
+- **Hidden content stays where only CSS hides it.** The pipeline reads the static
+  DOM. The `hidden` attribute and inline `display: none` are understood; a tab
+  panel hidden by a stylesheet alone is kept unless its group names the selected
+  one. Tabs marked with neither `role` nor `aria-role` and hidden with
+  `class="hidden"` can be lost, with no closing line.
 - **`removeHiddenText` covers hidden runs only.** Zero-width spaces, bidi
   controls and look-alike spaces pass through. It also removes runs that are
   legitimate in some scripts: ideographic variation selectors in CJK names and