page-markdown 0.2.0 → 0.3.0

package/README.md

@@ -44,7 +44,7 @@ import { pageToMarkdown } from "page-markdown";
 const result = pageToMarkdown(document);
 result.markdown;            // "# Title\n\n…"
 result.metadata;            // { url, canonicalUrl, title, author, published, siteName, lang, … }
-result.extraction.method;   // "readability" | "full_page"
+result.extraction.method;   // "readability" | "main_content" | "full_page" | custom method
 
 // In Node, with any DOM implementation (linkedom shown)
 import { parseHTML } from "linkedom";
@@ -57,11 +57,11 @@ pageToMarkdown(doc, { url: "https://example.com/article" });
 | Option | Default | Meaning |
 |---|---|---|
 | `url` | the document's URL | Base for absolute links and images |
-| `extract` | `"readable"` | `"readable"` (Readability, full-page fallback), `"full"`, or a function `(docCopy) => { html, method }` |
-| `minExtractChars` | `100` | Below this, readable extraction falls back to the full page |
+| `extract` | `"readable"` | `"readable"` (Readability with main-content or full-page fallback), `"full"`, or a function `(docCopy) => { html, method }` |
+| `minExtractChars` | `100` | Below this, Readability falls back to a sufficiently long `<main>` or `[role="main"]`, else the full page; custom extractors fall back to the full page |
 | `minExtractShare` | `0.4` | When Readability keeps less than this share of the page's main content (and misses at least 500 characters), the main content is used instead: forum threads and product grids. `0` turns it off |
 | `includeFullPage` | `false` | Also return `fullPageMarkdown` |
-| `includeTitle` | `true` | Start readable output with `# <headline>`: the page's H1, else its title without the site name |
+| `includeTitle` | `true` | Add `# <headline>` to Readability or main-content output when no H1 remains; skipped in `text_only` mode |
 | `recoverLead` | `true` | Restore a lead paragraph the extractor dropped, when it matches the meta description |
 | `pageFacts` | `false` | Append `## Page facts`: short label/value pairs outside the article (a sidebar's version or license, spec rows, infobox rows) that the extractor dropped. `extraction.pageFacts` counts them |
 | `removeHiddenText` | `false` | Remove text hidden from people but read by models: runs of tag characters, variation selectors, invisible operators and reserved tag-plane code points, found by [text-sanctifier](https://github.com/iWhatty/text-sanctifier)'s `findHiddenRuns`. Applies to `markdown` (fenced code included), `fullPageMarkdown` and `metadata`. The England, Scotland and Wales flags and a lone emoji or text presentation selector are kept. `omissions` gets one entry per kind (`hidden_tag`, `hidden_variation_selector`, `hidden_invisible_operator`, `hidden_reserved_tag_plane`) with `count` runs and `chars` code points removed from `markdown` |
@@ -88,17 +88,64 @@ The stages are also exported for advanced use: `sanitizeToInertDocument`,
 | `metadata.headline` | `string?` | The heading chosen for the capture |
 | `metadata.extractedTitle`, `byline` | `string?` | What the extractor reported |
 | `extraction.method` | `string` | `"readability"`, `"main_content"`, `"full_page"`, or the name a custom extractor gave |
-| `extraction.fallbackReason` | `string?` | Why extraction fell back: `extracted_small_share_of_page`, `extracted_site_menu` or `extracted_few_sections_of_page` (to the main content), or `extractor_found_nothing`, `extracted_text_below_minimum` or `extractor_error: …` (to the full page) |
+| `extraction.fallbackReason` | `string?` | Why extraction fell back: `extracted_small_share_of_page`, `extracted_site_menu` or `extracted_few_sections_of_page` use the main content; `extractor_found_nothing`, `extracted_text_below_minimum` or `extractor_error: ...` use a sufficiently long main-content region for Readability, else the full page. Custom extractors fall back to the full page |
 | `extraction.leadRecovered` | `true?` | A dropped lead paragraph was restored |
 | `extraction.pageFacts` | `number?` | How many facts `pageFacts` appended |
 | `omissions[]` | `{ kind, count, note, chars? }` | What was left out or removed. `kind` is `shadow_content`, `json_line`, `inactive_tabs`, `comments` or a `hidden_*` kind; only `hidden_*` entries have `chars`. All but `hidden_*` also add a closing line to `markdown`, as a fallback does, so a caller that passes on only the Markdown still says so |
 | `warnings[]` | `string` | Read these before trusting a capture: a fallback, or shadow content cut off at the capture limit, in which case the main content may be missing |
 | `stats.mode` | `string` | The output mode used |
 | `stats.extractedHtmlChars`, `convertedChars`, `markdownChars` | `number` | Sizes at each step. `markdownChars` is the returned Markdown's length |
-| `stats.standardization` | `object` | Counts of what standardizing changed: `lazyImagesResolved`, `inlineImagesDropped`, `scriptLinksUnwrapped`, `emptyLinksRemoved`, `preWrapped`, `headingWidgetsRemoved`, `headingsUnwrapped` |
+| `stats.pageTextChars` | `number` | Whitespace-normalized text length of the standardized page at capture time |
+| `stats.standardization` | `object` | Counts of what standardizing changed: `lazyImagesResolved`, `inlineImagesDropped`, `scriptLinksUnwrapped`, `invalidLinksUnwrapped`, `emptyLinksRemoved`, `preWrapped`, `headingWidgetsRemoved`, `headingsUnwrapped`; `omittedTabs` lists labels of inactive panels left out |
 
 A `?` marks a field that is absent, not empty, when it doesn't apply.
 
+### Concealed text (opt-in)
+
+Detection is off by default. Enable it for a live document:
+
+```js
+const result = pageToMarkdown(document, {
+  detectConcealedText: true,
+  includeFullPage: true,
+});
+```
+
+Supported computed styles separate concealed DOM text from the article and
+return it in `concealedText` as `{ reason, chars, text }` blocks. Text is capped
+at 2,048 UTF-16 units per block, 16,384 total and 100 entries;
+`concealedTextTruncated` reports capping. Omission counts and character totals
+are uncapped. A `Concealed text omitted:` notice appears in both `markdown`
+and `fullPageMarkdown` when text is captured.
+
+The seven reasons are `display_none`, `visibility_hidden`, `opacity_zero`,
+`font_size_zero`, `clipped`, `transparent_text` and `same_color`. Supported
+clips are zero-area legacy rectangles on absolute/fixed elements,
+`inset(50%)` and zero-radius circles. Colour checks compare computed sRGB
+rgb()/rgba() values for an element's own text against its own or nearest
+declared opaque background.
+
+`concealmentCheck` is `disabled` by default (no computed-style calls),
+`static-only` when enabled without a computed-style API, `computed` after a
+successful check, or `incomplete` if the capability probe or a consulted
+property fails. Detection applies to live documents only; enabled static
+input retains its previous output. A top-level `detectConcealedText` option
+takes precedence over `sanitize.detectConcealedText`, then the default false.
+
+Limits include white text with no declared opaque background, non-zero alpha,
+near-colour matches, and unparsed OKLCH and Display P3 colours. Intervening
+images or translucent backgrounds and uncertain text painting effects prevent
+colour equality detection. Existing short inactive tab, collapsed answer and
+code wrapper exceptions remain; independent concealment reasons still apply.
+Off-screen or covered text, attributes, metadata and custom-extractor output
+are outside this check.
+
+The scroll-reveal rule is **provisional**: zero-opacity content with an
+opacity-applicable transition or named animation of nonzero duration stays
+in place with a warning. A declaration does not prove it will appear, and a
+page can use one to evade detection. Real-site validation is still required.
+Chrome is the only browser tested; Firefox and Safari are untested.
+
 ### TypeScript
 
 Types ship with the package, generated from the JSDoc in `src`: `pageToMarkdown`,
@@ -127,7 +174,7 @@ const { markdown: safe, omissions } = pageToMarkdown(document, { removeHiddenTex
 ## Quality
 
 `npm test` runs the contract suite. It covers:
-- golden outputs for 34 fixtures;
+- golden outputs for 51 fixtures;
 - regressions for known converter bugs, including dropped lead paragraphs,
   raw HTML tables, flattened code in tables, site-suffixed titles and
   unescaped table pipes;
@@ -179,7 +226,7 @@ its products. What it still fails on these pages:
 
 - **Promo boxes kept** inside a BBC Arabic article.
 
-Browser bundle: about 75 KB minified, 26 KB gzip (`npm run size`, ceiling 76 KB
+Browser bundle: about 86.0 KB minified, 31.2 KB gzip (`npm run size`, ceiling 86 KB
 minified). Readability is most of it.
 
 `npm run ci` runs the contract suite, the type check and the size check, and must
@@ -202,8 +249,6 @@ deliberately has no cloud CI.
   kept by the main-content fallback only; Readability removes hidden text.
 - **Bylines and dates can be missing from the Markdown.** Readability often leaves
   them out of the article. Look in `metadata.author` and `metadata.published`.
-- **Note labels can be lost.** On Docker's docs, "Warning", "Note" and "Tip" are
-  dropped, though the blockquotes stay.
 - **Some legacy markup loses paragraph breaks.** Readability can drop runs of
   `<br>` between inline elements in old HTML. The text survives; a paragraph
   break may not.