page-markdown 0.1.3 → 0.1.4

package/CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## 0.1.4 — 2026-11-02
+
+- **text-sanctifier 1.3.8** (was 1.3.3). `removeHiddenText` now also removes
+  runs of reserved tag-plane code points (U+E0080–U+E00FF, U+E01F0–U+E0FFF),
+  which 1.3.3 did not find, and reports them as `hidden_reserved_tag_plane`.
+  The browser bundle grows to about 65.6 KB minified.
+- A test checks that every run kind text-sanctifier can report has an
+  omission note, so a new kind fails `npm run ci` instead of reaching users
+  with no note.
+- A test fails when any source, test, fixture or doc file holds a raw
+  invisible or control character. A test regex in 0.1.3 held four, written
+  by an editing tool that expanded `\u` escapes; they never shipped, since
+  tests are not in the package.
+
 ## 0.1.3 — 2026-10-30
 
 First release on npm. `npm run ci` (contract suite and size check) is the

package/README.md

@@ -56,7 +56,7 @@ pageToMarkdown(doc, { url: "https://example.com/article" });
 | `includeTitle` | `true` | Start readable output with `# <headline>`: the page's H1, else its title without the site name |
 | `recoverLead` | `true` | Restore a lead paragraph the extractor dropped, when it matches the meta description |
 | `pageFacts` | `false` | Append `## Page facts`: short label/value pairs outside the article (a sidebar's version or license, spec rows, infobox rows) that the extractor dropped. `extraction.pageFacts` counts them |
-| `removeHiddenText` | `false` | Remove text hidden from people but read by models: runs of tag characters, variation selectors and invisible operators, found by [text-sanctifier](https://github.com/iWhatty/text-sanctifier)'s `findHiddenRuns`. Applies to `markdown` (fenced code included), `fullPageMarkdown` and `metadata`. The England, Scotland and Wales flags and a lone emoji or text presentation selector are kept. `omissions` gets one entry per kind (`hidden_tag`, `hidden_variation_selector`, `hidden_invisible_operator`) with `count` runs and `chars` code points removed from `markdown` |
+| `removeHiddenText` | `false` | Remove text hidden from people but read by models: runs of tag characters, variation selectors, invisible operators and reserved tag-plane code points, found by [text-sanctifier](https://github.com/iWhatty/text-sanctifier)'s `findHiddenRuns`. Applies to `markdown` (fenced code included), `fullPageMarkdown` and `metadata`. The England, Scotland and Wales flags and a lone emoji or text presentation selector are kept. `omissions` gets one entry per kind (`hidden_tag`, `hidden_variation_selector`, `hidden_invisible_operator`, `hidden_reserved_tag_plane`) with `count` runs and `chars` code points removed from `markdown` |
 | `mode` | `"full"` | `"no_links"` keeps link text; `"text_only"` also drops structure. Fenced code is never changed |
 | `sanitize` | see `DEFAULT_SANITIZE_OPTIONS` | `stripAds`, `extraStripSelectors` (e.g. `["[data-app-overlay]"]`), shadow limits |
 | `readability` | `{}` | Passed to Readability |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "page-markdown",
-  "version": "0.1.3",
+  "version": "0.1.4",
   "description": "Faithful page-to-Markdown for humans and agents: sanitize, extract the readable article, convert with GFM, and clean without losing content.",
   "type": "module",
   "exports": {
@@ -46,7 +46,7 @@
   },
   "dependencies": {
     "@mozilla/readability": "0.6.0",
-    "text-sanctifier": "1.3.3",
+    "text-sanctifier": "1.3.8",
     "turndown": "7.2.4"
   },
   "devDependencies": {

package/src/hidden.js

@@ -1,6 +1,6 @@
 // Optional last stage: remove characters that hide text from people but not
-// from models (tag characters, variation-selector runs, invisible operators),
-// found by text-sanctifier's run finder. Legitimate sequences are kept: the
+// from models (tag characters, variation-selector runs, invisible operators,
+// reserved tag-plane code points), found by text-sanctifier's run finder. Legitimate sequences are kept: the
 // England, Scotland and Wales flags and a lone emoji or text presentation
 // selector. Unlike the clean stage, this also applies inside fenced code.
 import { findHiddenRuns } from "text-sanctifier/hidden";
@@ -9,6 +9,7 @@ const NOTES = {
   tag: "Hidden tag-character text removed",
   variation_selector: "Variation selector runs removed",
   invisible_operator: "Invisible operator runs removed",
+  reserved_tag_plane: "Reserved tag-plane runs removed",
 };
 
 /**

package/src/index.js

@@ -10,7 +10,7 @@ import { recoverLeadParagraph, chooseHeadline } from "./refine.js";
 import { collectPageFacts, pageFactsMarkdown } from "./facts.js";
 import { removeHiddenText, hiddenTextOmissions } from "./hidden.js";
 
-export const VERSION = "0.1.3";
+export const VERSION = "0.1.4";
 
 export { sanitizeToInertDocument, DEFAULT_SANITIZE_OPTIONS, isAdvertisementElement, createInertDocument } from "./sanitize.js";
 export { standardizeDocument } from "./standardize.js";