page-markdown 0.1.5 → 0.2.0

package/README.md

@@ -3,8 +3,9 @@
 Faithful page-to-Markdown for humans and agents. Give it a web page's DOM and
 it returns clean Markdown plus metadata. It is written for browsers, for
 example an extension's content script, and runs in Node with a DOM library.
-It is tested in Node and in headless Chrome and Edge, but not yet in Firefox or
-Safari (see [Known limits](#known-limits-v01)). It was built for WATT3D's
+It is tested in Node, in headless Chrome and Edge, and in a real extension
+(HELIOS, in Brave), but not yet in Firefox or Safari (see
+[Known limits](#known-limits-v02)). It was built for WATT3D's
 browser extensions.
 
 ```
@@ -13,6 +14,7 @@ live Document
   → standardize   <pre> → fenced code with language; heading wrappers unwrapped; empty and
                   javascript: links removed; absolute links and images; lazy images resolved
   → extract       Mozilla Readability on a copy; full-page fallback when it finds too little;
+                  main-content fallback when it keeps only a small share of the page;
                   a dropped lead paragraph restored; a clean title heading
   → convert       Turndown + GitHub-flavoured tables, strikethrough and task lists; tables never
                   come out as raw HTML
@@ -57,13 +59,14 @@ pageToMarkdown(doc, { url: "https://example.com/article" });
 | `url` | the document's URL | Base for absolute links and images |
 | `extract` | `"readable"` | `"readable"` (Readability, full-page fallback), `"full"`, or a function `(docCopy) => { html, method }` |
 | `minExtractChars` | `100` | Below this, readable extraction falls back to the full page |
+| `minExtractShare` | `0.4` | When Readability keeps less than this share of the page's main content (and misses at least 500 characters), the main content is used instead: forum threads and product grids. `0` turns it off |
 | `includeFullPage` | `false` | Also return `fullPageMarkdown` |
 | `includeTitle` | `true` | Start readable output with `# <headline>`: the page's H1, else its title without the site name |
 | `recoverLead` | `true` | Restore a lead paragraph the extractor dropped, when it matches the meta description |
 | `pageFacts` | `false` | Append `## Page facts`: short label/value pairs outside the article (a sidebar's version or license, spec rows, infobox rows) that the extractor dropped. `extraction.pageFacts` counts them |
 | `removeHiddenText` | `false` | Remove text hidden from people but read by models: runs of tag characters, variation selectors, invisible operators and reserved tag-plane code points, found by [text-sanctifier](https://github.com/iWhatty/text-sanctifier)'s `findHiddenRuns`. Applies to `markdown` (fenced code included), `fullPageMarkdown` and `metadata`. The England, Scotland and Wales flags and a lone emoji or text presentation selector are kept. `omissions` gets one entry per kind (`hidden_tag`, `hidden_variation_selector`, `hidden_invisible_operator`, `hidden_reserved_tag_plane`) with `count` runs and `chars` code points removed from `markdown` |
 | `mode` | `"full"` | `"no_links"` keeps link text; `"text_only"` also drops structure. Fenced code is never changed |
-| `sanitize` | see `DEFAULT_SANITIZE_OPTIONS` | `stripAds`, `extraStripSelectors` (e.g. `["[data-app-overlay]"]`), shadow limits |
+| `sanitize` | see `DEFAULT_SANITIZE_OPTIONS` | `stripAds`, `extraStripSelectors` (e.g. `["[data-app-overlay]"]`), shadow limits, which apply per host in the page's own tree and don't charge slotted page content |
 | `readability` | `{}` | Passed to Readability |
 
 The stages are also exported for advanced use: `sanitizeToInertDocument`,
@@ -84,14 +87,14 @@ The stages are also exported for advanced use: `sanitizeToInertDocument`,
 | `metadata.description`, `author`, `published`, `modified`, `siteName`, `lang` | `string` | From meta tags and JSON-LD, `""` when the page doesn't state them. `siteName` falls back to the hostname |
 | `metadata.headline` | `string?` | The heading chosen for the capture |
 | `metadata.extractedTitle`, `byline` | `string?` | What the extractor reported |
-| `extraction.method` | `string` | `"readability"`, `"full_page"`, or the name a custom extractor gave |
-| `extraction.fallbackReason` | `string?` | Why extraction fell back to the full page: `extractor_found_nothing`, `extracted_text_below_minimum` or `extractor_error: …` |
+| `extraction.method` | `string` | `"readability"`, `"main_content"`, `"full_page"`, or the name a custom extractor gave |
+| `extraction.fallbackReason` | `string?` | Why extraction fell back: `extracted_small_share_of_page`, `extracted_site_menu` or `extracted_few_sections_of_page` (to the main content), or `extractor_found_nothing`, `extracted_text_below_minimum` or `extractor_error: …` (to the full page) |
 | `extraction.leadRecovered` | `true?` | A dropped lead paragraph was restored |
 | `extraction.pageFacts` | `number?` | How many facts `pageFacts` appended |
-| `omissions[]` | `{ kind, count, note, chars? }` | What was left out or removed. `kind` is `shadow_content`, `json_line` or a `hidden_*` kind; only `hidden_*` entries have `chars` |
-| `warnings[]` | `string` | Currently the full-page fallback, in words |
+| `omissions[]` | `{ kind, count, note, chars? }` | What was left out or removed. `kind` is `shadow_content`, `json_line`, `inactive_tabs`, `comments` or a `hidden_*` kind; only `hidden_*` entries have `chars`. All but `hidden_*` also add a closing line to `markdown`, as a fallback does, so a caller that passes on only the Markdown still says so |
+| `warnings[]` | `string` | Read these before trusting a capture: a fallback, or shadow content cut off at the capture limit, in which case the main content may be missing |
 | `stats.mode` | `string` | The output mode used |
-| `stats.extractedHtmlChars`, `convertedChars`, `markdownChars` | `number` | Sizes at each step. `markdownChars` is measured before `removeHiddenText` and the shadow omission notice |
+| `stats.extractedHtmlChars`, `convertedChars`, `markdownChars` | `number` | Sizes at each step. `markdownChars` is the returned Markdown's length |
 | `stats.standardization` | `object` | Counts of what standardizing changed: `lazyImagesResolved`, `inlineImagesDropped`, `scriptLinksUnwrapped`, `emptyLinksRemoved`, `preWrapped`, `headingWidgetsRemoved`, `headingsUnwrapped` |
 
 A `?` marks a field that is absent, not empty, when it doesn't apply.
@@ -146,8 +149,11 @@ checks what Node can't show:
   children in rendered order;
 - closed shadow roots stay out of the capture.
 
-Set `PAGE_MARKDOWN_BROWSER` to choose the browser. The built page is kept at
-`.browser-test/index.html` for opening by hand.
+It launches a browser, so it runs only when a person sets
+`PAGE_MARKDOWN_ALLOW_BROWSER_LAUNCH=1` in their own shell; agents must not set it.
+Without the flag it builds `.browser-test/index.html`, which you can open in a
+browser that is already running, and exits with an error. Set
+`PAGE_MARKDOWN_BROWSER` to choose the browser.
 
 `npm run bench` scores the package against two baselines: Turndown over the page
 body, and Readability then Turndown. It runs 15 fixture checks and 117 probes on
@@ -159,38 +165,59 @@ recall against a hand-picked reference for each page. `npm run bench` only print
 |---|---|---|---|
 | Fixtures, written with the package | 15/15 | 7/15 | 3/15 |
 | Five pages the fixes were tuned on | 17/17 | 9/17 | 8/17 |
-| **11 held-out pages** | **84/100** | **74/100** | **61/100** |
-| Held-out word recall / precision | 86.3% / 95.5% | 85.4% / 95.6% | 98.2% / 72.1% |
-
-The held-out row is the one to weigh. On those pages the package leads plain
-Readability + Turndown on probes, and the two are level on word recall and
-precision. The pages are no longer fully unseen: the package scored 75/100 before
-two fixes that came from their failures, so read 84 as a little generous. What it
-still fails there:
-
-- **Most H2s dropped** on Docusaurus and Starlight, by Readability; Starlight also
-  keeps "Section titled" anchor text.
-- **Non-default tab panels dropped** (Starlight).
-- **Comments and product grids lost**, by Readability: 36% word recall on a
-  Lobsters thread, 50% on an IKEA category page.
-- **Promo boxes kept** inside a BBC Arabic article.
+| **11 pages, once held out** | **98/100** | **74/100** | **61/100** |
+| Their word recall / precision | 97.6% / 89.1% | 85.4% / 95.6% | 98.2% / 72.1% |
 
-Browser bundle: about 67 KB minified, 24 KB gzip (`npm run size`, ceiling 68 KB
-minified). Readability is most of it.
+Don't read 98 as a score on unseen pages. These probes were written before any
+engine ran and the package first scored 75/100, but most of the gain since comes
+from fixes for these same pages' failures. New pages are the real test.
 
-`npm run ci` runs the contract suite, the type check, the browser run and the size check, and must exit 0
-before a commit. It runs locally; the repo deliberately has no cloud CI.
+It gains recall (97.6% against 85.4%) by keeping more, and pays in precision
+(89.1% against 95.6%). The cost is mostly on the IKEA category page: 29.7% precision,
+because the main-content fallback keeps the page's banners and filters along with
+its products. What it still fails on these pages:
 
-## Known limits (v0.1)
+- **Promo boxes kept** inside a BBC Arabic article.
+
+Browser bundle: about 75 KB minified, 26 KB gzip (`npm run size`, ceiling 76 KB
+minified). Readability is most of it.
 
-- **Chromium only so far.** Tested in Node (linkedom) and in headless Chrome
-  and Edge, but not in Firefox or Safari, and not yet inside a real extension's
-  content script.
+`npm run ci` runs the contract suite, the type check and the size check, and must
+exit 0 before a commit. It launches no browser, so run the browser check yourself
+before a release when the DOM handling has changed. It runs locally; the repo
+deliberately has no cloud CI.
+
+## Known limits (v0.2)
+
+- **Chromium only so far.** Tested in Node (linkedom), in headless Chrome and
+  Edge, and on live pages in a real extension (HELIOS, in Brave), but not
+  in Firefox or Safari.
+- **Extraction can still be wrong without saying so.** The main-content fallback
+  and its warning catch a small share of the page, a kept site menu and lost
+  sections, not every wrong choice. If Readability picks the wrong part of a page
+  and that part is large, nothing flags it. No closing line doesn't mean nothing
+  was lost. A short post among long replies can be dropped: a Mastodon status lost
+  the post itself, and its cause isn't known yet.
+- **Answers hidden on an ordinary article can be lost.** Collapsed FAQ answers are
+  kept by the main-content fallback only; Readability removes hidden text.
+- **Bylines and dates can be missing from the Markdown.** Readability often leaves
+  them out of the article. Look in `metadata.author` and `metadata.published`.
+- **Note labels can be lost.** On Docker's docs, "Warning", "Note" and "Tip" are
+  dropped, though the blockquotes stay.
 - **Some legacy markup loses paragraph breaks.** Readability can drop runs of
   `<br>` between inline elements in old HTML. The text survives; a paragraph
   break may not.
-- **Hidden content stays.** The pipeline reads the static DOM, so elements
-  hidden only by CSS remain unless extraction drops them.
+- **A capture is a snapshot.** It reads the DOM as it is at call time, and it
+  can't tell whether the page has finished rendering. A page read too early looks
+  like a normal, short result. To wait, re-capture until `stats.pageTextChars`
+  settles.
+- **Captures hold whatever is in the DOM,** including text that other extensions
+  add. Use `sanitize.extraStripSelectors` for what you know about.
+- **Hidden content stays where only CSS hides it.** The pipeline reads the static
+  DOM. The `hidden` attribute and inline `display: none` are understood; a tab
+  panel hidden by a stylesheet alone is kept unless its group names the selected
+  one. Tabs marked with neither `role` nor `aria-role` and hidden with
+  `class="hidden"` can be lost, with no closing line.
 - **`removeHiddenText` covers hidden runs only.** Zero-width spaces, bidi
   controls and look-alike spaces pass through. It also removes runs that are
   legitimate in some scripts: ideographic variation selectors in CJK names and

package/dist/types/extract.d.ts

@@ -1,6 +1,7 @@
+export function isHiddenElement(el: any): any;
 /**
- * @typedef {{ html: string, method: "readability"|"full_page"|(string & {}), fallbackReason?: string, title?: string,
- *   byline?: string, published?: string, siteName?: string, lang?: string, excerpt?: string }} Extraction
+ * @typedef {{ html: string, method: "readability"|"main_content"|"full_page"|(string & {}), fallbackReason?: string, title?: string,
+ *   byline?: string, published?: string, siteName?: string, lang?: string, excerpt?: string, commentsOmitted?: boolean }} Extraction
  */
 /**
  * Receives a copy of the standardized document. Returning nothing, empty HTML
@@ -10,18 +11,21 @@
 /**
  * @param {Document} doc standardized inert document (not mutated)
  * @param {{ extract?: "readable"|"full"|Extractor,
- *   minExtractChars?: number, readability?: import("./types.js").ReadabilityOptions }} [options]
+ *   minExtractChars?: number, minExtractShare?: number, readability?: import("./types.js").ReadabilityOptions }} [options]
  * @returns {Extraction}
  */
 export function extractContent(doc: Document, options?: {
     extract?: "readable" | "full" | Extractor;
     minExtractChars?: number;
+    minExtractShare?: number;
     readability?: import("./types.js").ReadabilityOptions;
 }): Extraction;
 export const DEFAULT_MIN_EXTRACT_CHARS: 100;
+export const DEFAULT_MIN_EXTRACT_SHARE: 0.4;
+export const PAGE_CHROME: "nav, footer, form, [role='navigation'], [role='banner'], [role='contentinfo'], [role='search']";
 export type Extraction = {
     html: string;
-    method: "readability" | "full_page" | (string & {});
+    method: "readability" | "main_content" | "full_page" | (string & {});
     fallbackReason?: string;
     title?: string;
     byline?: string;
@@ -29,6 +33,7 @@ export type Extraction = {
     siteName?: string;
     lang?: string;
     excerpt?: string;
+    commentsOmitted?: boolean;
 };
 /**
  * Receives a copy of the standardized document. Returning nothing, empty HTML

package/dist/types/facts.d.ts

@@ -7,6 +7,7 @@ export function collectPageFacts(doc: Document, articleHtml: string): Array<{
     label: string;
     value: string;
 }>;
+export function escapeMarkdown(text: any): any;
 /**
  * "## Page facts" list, or "" when there are none.
  * @param {readonly import("./types.js").PageFact[]} facts

package/dist/types/index.d.ts

@@ -5,7 +5,10 @@
  * @property {"readable"|"full"|import("./extract.js").Extractor} [extract="readable"]
  *   "readable": Readability with full-page fallback; "full": the whole body;
  *   or a custom extractor receiving a copy of the standardized document.
- * @property {number} [minExtractChars=100] below this, fall back to the full page
+ * @property {number} [minExtractChars=100] below this, fall back to the page's <main>
+ *   if it marks one and it holds enough, else to the full page
+ * @property {number} [minExtractShare=0.4] when Readability keeps less than this
+ *   share of the page's main content, use the main content instead; 0 turns it off
  * @property {boolean} [includeFullPage=false] also return the full-page Markdown
  * @property {boolean} [includeTitle=true] start readable output with "# <headline>"
  * @property {boolean} [recoverLead=true] restore a dropped lead paragraph that matches the meta description
@@ -27,7 +30,7 @@
  * @returns {import("./types.js").PageMarkdownResult}
  */
 export function pageToMarkdown(input: Document | Element, options?: PageMarkdownOptions): import("./types.js").PageMarkdownResult;
-export const VERSION: "0.1.5";
+export const VERSION: "0.2.0";
 export { standardizeDocument } from "./standardize.js";
 export { collectMetadata } from "./metadata.js";
 export * from "./types.js";
@@ -43,9 +46,15 @@ export type PageMarkdownOptions = {
      */
     extract?: import("./extract.js").Extractor | "readable" | "full" | undefined;
     /**
-     * below this, fall back to the full page
+     * below this, fall back to the page's <main>
+     * if it marks one and it holds enough, else to the full page
      */
     minExtractChars?: number | undefined;
+    /**
+     * when Readability keeps less than this
+     * share of the page's main content, use the main content instead; 0 turns it off
+     */
+    minExtractShare?: number | undefined;
     /**
      * also return the full-page Markdown
      */
@@ -90,7 +99,7 @@ export type PageMarkdownOptions = {
     } | undefined;
 };
 export { sanitizeToInertDocument, DEFAULT_SANITIZE_OPTIONS, isAdvertisementElement, createInertDocument } from "./sanitize.js";
-export { extractContent, DEFAULT_MIN_EXTRACT_CHARS } from "./extract.js";
+export { extractContent, DEFAULT_MIN_EXTRACT_CHARS, DEFAULT_MIN_EXTRACT_SHARE } from "./extract.js";
 export { htmlToMarkdown, createTurndown, TURNDOWN_OPTIONS } from "./convert.js";
 export { cleanMarkdown, applyMode, mapOutsideFences } from "./clean.js";
 export { recoverLeadParagraph, chooseHeadline } from "./refine.js";

package/dist/types/sanitize.d.ts

@@ -9,8 +9,10 @@ export function createInertDocument(doc: Document): Document;
 /**
  * Clones `root` (a Document or Element) into a new inert document.
  * Open shadow roots replace their host's light children (native slots supply
- * assigned children). Limits apply only to shadow content; a host over the
- * limit is emptied and counted as an omission.
+ * assigned children). Limits apply only to shadow content, per host in the
+ * page's own tree, so slotted page content is not charged: docs sites put a
+ * whole article in one host's slot. A host over the limit keeps what fits and
+ * is counted as an omission.
  *
  * @param {Document|Element} root
  * @param {import("./types.js").SanitizeOptions} [options]

package/dist/types/standardize.d.ts

@@ -12,4 +12,5 @@ export function standardizeDocument(doc: Document, options?: {
     preWrapped: number;
     headingWidgetsRemoved: number;
     headingsUnwrapped: number;
+    omittedTabs: string[];
 };

package/dist/types/types.d.ts

@@ -37,7 +37,7 @@ export type PageMarkdownMetadata = CollectedMetadata & ExtractedMetadata;
 export type PageMarkdownExtraction = {
     method: Extraction["method"];
     /**
-     * why extraction fell back to the full page
+     * why extraction fell back to the full page or the main content
      */
     fallbackReason?: string | undefined;
     /**
@@ -107,6 +107,14 @@ export type PageMarkdownOmission = {
     kind: "json_line";
     count: number;
     note: string;
+} | {
+    kind: "inactive_tabs";
+    count: number;
+    note: string;
+} | {
+    kind: "comments";
+    count: number;
+    note: string;
 } | HiddenTextOmission;
 export type StandardizationStats = ReturnType<typeof import("./standardize.js").standardizeDocument>;
 export type PageMarkdownStats = {
@@ -114,6 +122,11 @@ export type PageMarkdownStats = {
     extractedHtmlChars: number;
     convertedChars: number;
     markdownChars: number;
+    /**
+     * the page's text at capture time: a capture is a
+     * snapshot of the DOM, so a caller can re-capture until this number settles
+     */
+    pageTextChars: number;
     standardization: StandardizationStats;
 };
 export type PageMarkdownResult = {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "page-markdown",
-  "version": "0.1.5",
+  "version": "0.2.0",
   "description": "Faithful page-to-Markdown for humans and agents: sanitize, extract the readable article, convert with GFM, and clean without losing content.",
   "type": "module",
   "exports": {
@@ -31,7 +31,7 @@
     "typecheck": "npm run build:types && node test/types/real-output.mjs && tsc -p test/types && tsc -p test/types/linkedom",
     "size": "node bench/size.mjs",
     "test:browser": "node test/browser/run.mjs",
-    "ci": "npm test && npm run typecheck && npm run test:browser && npm run size",
+    "ci": "npm test && npm run typecheck && npm run size",
     "prepack": "npm run build:types",
     "prepublishOnly": "npm run ci",
     "bench": "node bench/compare.mjs",

package/src/convert.js

@@ -14,6 +14,15 @@ export const TURNDOWN_OPTIONS = Object.freeze({
 });
 
 function gfmInlineRules(turndown) {
+  // Link text is one line: a link around blocks would otherwise split across lines.
+  turndown.addRule("pmInlineLink", {
+    filter: (node) => node.nodeName === "A" && Boolean(node.getAttribute("href")),
+    replacement(content, node) {
+      const href = node.getAttribute("href").replace(/([()])/g, "\\$1");
+      const title = (node.getAttribute("title") || "").replace(/\s*\n+\s*/g, " ").replace(/"/g, '\\"');
+      return `[${content.trim().replace(/\s*\n+\s*/g, " ")}](${href}${title ? ` "${title}"` : ""})`;
+    },
+  });
   turndown.addRule("pmStrikethrough", {
     filter: ["del", "s", "strike"],
     replacement: (content) => `~~${content}~~`,
@@ -187,10 +196,64 @@ function cellText(content) {
   return text.replace(/\|/g, "\\|");
 }
 
+// An icon's straight strokes, as lists of points; null if any stroke curves.
+function iconStrokes(svg) {
+  const strokes = [];
+  for (const shape of Array.from(svg.querySelectorAll?.("path, polyline, line") || [])) {
+    const at = (name) => shape.getAttribute(name);
+    const d = at("d") ?? `M${at("points") ?? `${at("x1")} ${at("y1")} ${at("x2")} ${at("y2")}`}`;
+    const tokens = d.match(/[a-zA-Z]|-?\d*\.?\d+/g) || [];
+    let x = 0;
+    let y = 0;
+    let command = "";
+    let points = [];
+    for (let i = 0; i < tokens.length;) {
+      if (/[a-zA-Z]/.test(tokens[i])) command = tokens[i++];
+      const relative = command > "Z";
+      const type = command.toUpperCase();
+      if (!"MLHV".includes(type)) return null;
+      if (type !== "V") x = Number(tokens[i++]) + (relative ? x : 0);
+      if (type !== "H") y = Number(tokens[i++]) + (relative ? y : 0);
+      if (type === "M") {
+        points = [];
+        strokes.push(points);
+        command = relative ? "l" : "L";
+      }
+      points.push([x, y]);
+    }
+  }
+  return strokes;
+}
+
+// What an icon's shape says, for the tick, dash and cross of comparison tables.
+function iconShape(strokes) {
+  const [first, second] = strokes || [];
+  if (strokes?.length === 1 && first.length === 2 && first[0][1] === first[1][1]) return "\u2013";
+  if (strokes?.length === 1 && first.length === 3) {
+    const [a, b, c] = first;
+    const arm = (p) => Math.hypot(p[0] - b[0], p[1] - b[1]);
+    const ratio = Math.max(arm(a), arm(c)) / Math.min(arm(a), arm(c));
+    if (b[1] > a[1] && b[1] > c[1] && (a[0] - b[0]) * (c[0] - b[0]) < 0 && ratio > 1.3) return "\u2713";
+  }
+  const slope = ([p, q]) => (q[0] - p[0]) * (q[1] - p[1]);
+  if (strokes?.length === 2 && first.length === 2 && second.length === 2 && slope(first) * slope(second) < 0) return "\u2717";
+  return "";
+}
+
+// A cell holding only an icon would be empty, and the table couldn't say which
+// plan has what: its accessible name, else its shape, else that an icon is there.
+function cellIcon(cell) {
+  const svg = cell.querySelector?.("svg");
+  if (!svg) return "";
+  return svg.getAttribute("aria-label")?.trim() || svg.querySelector("title")?.textContent?.trim()
+    || iconShape(iconStrokes(svg)) || "[icon]";
+}
+
 function tableRules(turndown) {
   turndown.addRule("pmTableCell", {
     filter: ["th", "td"],
-    replacement(content, node) {
+    replacement(rawContent, node) {
+      const content = rawContent.trim() ? rawContent : cellIcon(node);
       if (isBlockCell(node)) {
         // Header cells become the labels on the other cells.
         const label = cellLabel(node);

package/src/extract.js

@@ -1,10 +1,33 @@
 // Stage 3: pick the readable content. Default: Mozilla Readability on a copy
 // of the standardized clone, falling back to the whole body when it finds
-// nothing or too little.
+// nothing or too little, and to the page's main content when it keeps only a
+// small share of that (forum threads, product grids), brings the site's menu
+// along, or keeps too few of its sections (product pages).
 // The class only: skips isProbablyReaderable, which the pipeline does not use.
 import Readability from "@mozilla/readability/Readability.js";
+import { htmlText } from "./refine.js";
 
 export const DEFAULT_MIN_EXTRACT_CHARS = 100;
+export const DEFAULT_MIN_EXTRACT_SHARE = 0.4;
+// A small share of a short page may be all there is: don't fall back for less.
+const MIN_MISSED_CHARS = 500;
+export const PAGE_CHROME = "nav, footer, form, [role='navigation'], [role='banner'], [role='contentinfo'], [role='search']";
+const MAIN = "main, [role='main']";
+const COMMENTS = "#comments, #disqus_thread, .comments, .commentlist, .comment-list, .comment, [id^='comment-']";
+const HIDDEN_STYLE = /(?:^|;)\s*(?:display\s*:\s*none|visibility\s*:\s*hidden)/i;
+const TOGGLE = "button, summary, [aria-expanded]";
+// Shorter hidden text beside a button is a status message ("Added"), not an answer.
+const MIN_ANSWER_CHARS = 40;
+const MENU_SAMPLE_CHARS = 30;
+// Fewer h2s than this, and a page has no sections to lose.
+const MIN_SECTIONS = 3;
+const SECTION_SHARE_FACTOR = 1.5;
+
+// As far as a clone can tell, with no stylesheets. Not aria-hidden: shops put
+// the price people see in it, beside a longer one for screen readers.
+export function isHiddenElement(el) {
+  return el.hasAttribute("hidden") || HIDDEN_STYLE.test(el.getAttribute("style") || "");
+}
 
 function textLength(html = "") {
   return String(html).replace(/<[^>]*>/g, " ").replace(/\s+/g, " ").trim().length;
@@ -14,6 +37,86 @@ function bodyHtml(doc) {
   return (doc.body || doc.documentElement)?.innerHTML || "";
 }
 
+// An accordion's answer, hidden until its button is pressed: FAQs on shop pages.
+function isCollapsedAnswer(el) {
+  return Boolean(el.previousElementSibling?.matches(TOGGLE)) && (el.textContent || "").trim().length >= MIN_ANSWER_CHARS;
+}
+
+// <main> or else the body, without navigation, other page chrome or anything
+// hidden, except code behind a "Show code" toggle and collapsed answers. A header
+// inside <main> usually belongs to the content; one in the body doesn't.
+function mainContentRoot(doc) {
+  const main = doc.querySelector(MAIN);
+  const root = (main || doc.body || doc.documentElement).cloneNode(true);
+  for (const el of Array.from(root.querySelectorAll(`${PAGE_CHROME}, aside, template, [role='complementary']${main ? "" : ", header"}`))) el.remove();
+  for (const el of Array.from(root.querySelectorAll("[hidden], [style]"))) {
+    if (isHiddenElement(el) && !el.querySelector("pre") && !isCollapsedAnswer(el)) el.remove();
+  }
+  for (const link of Array.from(root.querySelectorAll("a[href^='#']"))) {
+    if (/^\s*skip\b/i.test(link.textContent || "")) link.remove();
+  }
+  return root;
+}
+
+const withoutSpaces = (text) => text.replace(/\s+/g, "");
+
+// An article above its comments, where Readability kept the article: what's left
+// without them. null for a thread, where what Readability kept is itself a comment.
+function articleWithoutComments(root, articleHtml) {
+  const comments = Array.from(root.querySelectorAll(COMMENTS));
+  if (!comments.length) return null;
+  const kept = withoutSpaces(htmlText(articleHtml));
+  const sample = kept.slice(kept.length >> 1, (kept.length >> 1) + 60);
+  if (comments.some((el) => withoutSpaces(el.textContent || "").includes(sample))) return null;
+  for (const el of comments) el.remove();
+  return root;
+}
+
+// Readability chose a container above <main>, and the site's menu came with it.
+function keptSiteMenu(doc, kept) {
+  const main = doc.querySelector(MAIN);
+  if (!main) return false;
+  const text = withoutSpaces(main.textContent || "");
+  if (!kept.includes(text.slice(text.length >> 1, (text.length >> 1) + MENU_SAMPLE_CHARS))) return false;
+  return Array.from(doc.querySelectorAll("nav, [role='navigation']")).some((nav) => {
+    const menu = withoutSpaces(nav.textContent || "").slice(0, MENU_SAMPLE_CHARS);
+    return !main.contains(nav) && menu.length === MENU_SAMPLE_CHARS && kept.includes(menu);
+  });
+}
+
+// A heading that is one link names another page, not a section of this one.
+function isStoryCardTitle(h2) {
+  const text = withoutSpaces(h2.textContent || "");
+  return Boolean(h2.closest("a")) || Array.from(h2.querySelectorAll("a:not([href^='#'])")).some((a) => withoutSpaces(a.textContent || "") === text);
+}
+
+function lostMostSections(root, kept) {
+  const headings = Array.from(root.querySelectorAll("h2")).filter((h2) => !isStoryCardTitle(h2)).map((h2) => withoutSpaces(h2.textContent || "")).filter(Boolean);
+  return headings.length >= MIN_SECTIONS && headings.filter((heading) => !kept.includes(heading)).length * 2 > headings.length;
+}
+
+// Layout utilities, which say nothing about content: Tailwind's scroll-mt-2 and
+// overflow-hidden, md:hidden.
+const LAYOUT_CLASS = /^(?:\S+:)?(?:-?scroll-[mp][a-z]?-\S+|overflow(?:-[xy])?-(?:hidden|scroll)|(?<=:)hidden)$/;
+
+// Readability scores class and id names, and removes what looks like clutter:
+// Docusaurus's heading "anchorTargetHideOnScrollNavbar" matches "scroll", and so
+// does the scroll-mt-2 on every code block of Docker's docs.
+function withoutMisleadingNames(doc) {
+  for (const heading of Array.from(doc.querySelectorAll("h1, h2, h3, h4, h5, h6"))) {
+    heading.removeAttribute("class");
+    heading.removeAttribute("id");
+  }
+  for (const el of Array.from(doc.querySelectorAll("[class]"))) {
+    const names = String(el.getAttribute("class")).split(/\s+/);
+    const kept = names.filter((name) => !LAYOUT_CLASS.test(name));
+    if (kept.length < names.length) el.setAttribute("class", kept.join(" "));
+  }
+  // Readability removes what's aria-hidden, and icons in table cells usually are.
+  for (const icon of Array.from(doc.querySelectorAll("td svg[aria-hidden], th svg[aria-hidden]"))) icon.removeAttribute("aria-hidden");
+  return doc;
+}
+
 function copyDocument(doc) {
   const copy = doc.cloneNode(true);
   if (copy && copy.nodeType === 9 && copy.documentElement) return copy;
@@ -23,8 +126,8 @@ function copyDocument(doc) {
 }
 
 /**
- * @typedef {{ html: string, method: "readability"|"full_page"|(string & {}), fallbackReason?: string, title?: string,
- *   byline?: string, published?: string, siteName?: string, lang?: string, excerpt?: string }} Extraction
+ * @typedef {{ html: string, method: "readability"|"main_content"|"full_page"|(string & {}), fallbackReason?: string, title?: string,
+ *   byline?: string, published?: string, siteName?: string, lang?: string, excerpt?: string, commentsOmitted?: boolean }} Extraction
  */
 
 /**
@@ -36,7 +139,7 @@ function copyDocument(doc) {
 /**
  * @param {Document} doc standardized inert document (not mutated)
  * @param {{ extract?: "readable"|"full"|Extractor,
- *   minExtractChars?: number, readability?: import("./types.js").ReadabilityOptions }} [options]
+ *   minExtractChars?: number, minExtractShare?: number, readability?: import("./types.js").ReadabilityOptions }} [options]
  * @returns {Extraction}
  */
 export function extractContent(doc, options = {}) {
@@ -52,7 +155,7 @@ export function extractContent(doc, options = {}) {
     if (typeof mode === "function") {
       result = mode(copyDocument(doc));
     } else {
-      const article = new Readability(copyDocument(doc), { keepClasses: true, ...options.readability }).parse();
+      const article = new Readability(withoutMisleadingNames(copyDocument(doc)), { keepClasses: true, ...options.readability }).parse();
       result = article && {
         html: article.content || "",
         method: "readability",
@@ -68,9 +171,41 @@ export function extractContent(doc, options = {}) {
     failure = `extractor_error: ${/** @type {any} */ (error)?.message || error}`;
   }
 
-  if (!result?.html) return { ...full(), fallbackReason: failure || "extractor_found_nothing" };
-  if (textLength(result.html) < minChars) {
-    return { ...full(), fallbackReason: "extracted_text_below_minimum", title: result.title };
+  // Too little from the default extractor: the page's own <main>, if it marks
+  // one and it holds enough, beats a full page that opens with site navigation.
+  const tooLittle = (fallbackReason, title) => {
+    if (typeof mode !== "function" && doc.querySelector(MAIN)) {
+      const html = mainContentRoot(doc).innerHTML || "";
+      if (textLength(html) >= minChars) return { html, method: "main_content", fallbackReason, title };
+    }
+    return { ...full(), fallbackReason, ...(title === undefined ? {} : { title }) };
+  };
+
+  if (!result?.html) return tooLittle(failure || "extractor_found_nothing");
+  if (textLength(result.html) < minChars) return tooLittle("extracted_text_below_minimum", result.title);
+  if (result.method === "readability") {
+    const share = options.minExtractShare ?? DEFAULT_MIN_EXTRACT_SHARE;
+    const kept = textLength(result.html);
+    const isSmall = (html) => {
+      const available = textLength(html);
+      return kept < available * share && available - kept >= MIN_MISSED_CHARS;
+    };
+    const keptText = withoutSpaces(htmlText(result.html));
+    // Section loss counts only where it takes more of the page than a related-links block.
+    const misses = (el) => {
+      const available = el.innerHTML || "";
+      return isSmall(available) || (kept < textLength(available) * share * SECTION_SHARE_FACTOR && lostMostSections(el, keptText));
+    };
+    const root = mainContentRoot(doc);
+    const html = root.innerHTML || "";
+    const reason = keptSiteMenu(doc, keptText) && textLength(html) >= minChars ? "extracted_site_menu"
+      : isSmall(html) ? "extracted_small_share_of_page"
+      : misses(root) ? "extracted_few_sections_of_page" : "";
+    if (reason) {
+      const article = reason === "extracted_site_menu" ? null : articleWithoutComments(root, result.html);
+      if (article && !misses(article)) return { ...result, commentsOmitted: true };
+      return { ...result, html, method: "main_content", fallbackReason: reason };
+    }
   }
   return result;
 }