@v1nvn/readability 0.35.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (8) hide show
  1. package/README.md +355 -0
  2. package/dist/assets/cli-boGClDt0.js +71 -0
  3. package/dist/assets/cli-boGClDt0.js.map +1 -0
  4. package/dist/assets/extract-DaYydtVZ.js +2866 -0
  5. package/dist/assets/extract-DaYydtVZ.js.map +1 -0
  6. package/dist/index.js +2267 -0
  7. package/dist/index.js.map +1 -0
  8. package/package.json +66 -0
package/README.md ADDED
@@ -0,0 +1,355 @@
1
+ # readability
2
+
3
+ Turn **already-rendered HTML** (captured post-JavaScript from a browser or [chrome-devtools MCP](https://github.com/anthropic/claude-code-chrome-devtools)) into clean, LLM-friendly **Markdown + metadata**, using [Mozilla Readability](https://github.com/mozilla/readability), [Turndown](https://github.com/mixmark-io/turndown), and [DOMPurify](https://github.com/cure53/DOMPurify).
4
+
5
+ The key idea: **rendering and extraction are decoupled.** A real browser (chrome-devtools) owns rendering; this server only transforms HTML it reads from a file. **The server makes no outbound requests** — there is no `fetch`, no SSRF surface. Every HTML-input tool takes a `localPath` (a file on disk), never an inline string, so a full rendered page never enters the model context. The optional `baseUrl` is _origin context only_, used to absolutize relative links; it is never fetched.
6
+
7
+ Also a Claude Code plugin: `claude plugin install readability@agentic`.
8
+ User-facing docs: [root README](../../README.md).
9
+
10
+ ## Install
11
+
12
+ ```bash
13
+ npm install @v1nvn/readability@0.35.0
14
+ # or run on demand:
15
+ npx @v1nvn/readability@0.35.0
16
+ ```
17
+
18
+ Requires Node >= 22. Build from source:
19
+
20
+ ```bash
21
+ git clone <repo> && cd packages/readability
22
+ yarn install
23
+ yarn build # bundles to dist/index.js
24
+ node dist/index.js # starts the stdio MCP server
25
+ ```
26
+
27
+ ### Smithery
28
+
29
+ A `smithery.yaml` (stdio runtime) is included for [Smithery](https://smithery.ai) deployment. It pins the `stdio` startCommand (this server ships `StdioServerTransport` only — the HTTP container runtime cannot launch it) and surfaces `READABILITY_LOG_LEVEL` as the one config knob.
30
+
31
+ ## The chrome-devtools handoff
32
+
33
+ The motivating flow is two hops — each tool does the one thing it is best at:
34
+
35
+ ```js
36
+ // 1. In the chrome-devtools MCP, grab the RENDERED document (post-JS) and write
37
+ // it to a file via evaluate_script's `filePath` arg (the model emits only a
38
+ // path, never the page bytes):
39
+ mcp__chrome -
40
+ devtools__evaluate_script({
41
+ function: () => document.documentElement.outerHTML,
42
+ filePath: '/tmp/page.html',
43
+ });
44
+
45
+ // 2. Point readability at that file.
46
+ // `baseUrl` is OPTIONAL context (origin for absolutizing relative links) — never fetched.
47
+ mcp__readability__extract({ localPath: '/tmp/page.html', baseUrl: pageUrl });
48
+ ```
49
+
50
+ This matters most for SPAs and JS-augmented pages, where the initial HTML is an empty `<div id="root">` and only the post-JS DOM has the content.
51
+
52
+ ## MCP client config
53
+
54
+ Add to your MCP client config (Claude Code, Claude Desktop, etc.):
55
+
56
+ ```jsonc
57
+ {
58
+ "mcpServers": {
59
+ "readability": {
60
+ "command": "npx",
61
+ "args": ["-y", "@v1nvn/readability@0.35.0"],
62
+ },
63
+ },
64
+ }
65
+ ```
66
+
67
+ ## Tools
68
+
69
+ All eleven always-on tools return MCP **structured content** (`schemaVersion` plus a tool-specific payload of `metadata` / `diagnostics` / `items` / …) validated by a zod `outputSchema`, plus a human/LLM-readable payload in `content[0].text`. A sampling-capable host also sees a twelfth and a thirteenth — `summarize` and `suggest_preset` — registered after the `initialize` handshake when the client advertises the MCP `sampling` capability. Nothing throws across the wire — failures become `{ "isError": true }` results. Every input and output field carries a description in the tool's JSON schema, so clients can introspect each option without reading these docs.
70
+
71
+ **`localPath` — the only HTML input.** Every HTML-input tool takes a `localPath` pointing at a file holding the already-rendered (post-JavaScript) HTML. The server reads the bytes itself, so the page never enters the model context — the model emits only a path string. The motivating hop is chrome-devtools → readability: `evaluate_script` writes `document.documentElement.outerHTML` to a file via its `filePath` arg, then the tool reads that path. Resolved relative to the server process working directory; prefer absolute paths so the chrome-devtools capture and this read agree on location.
72
+
73
+ ### `extract` — primary tool
74
+
75
+ Extracts the main article from rendered HTML and returns Markdown + metadata + diagnostics.
76
+
77
+ | Option | Default | Description |
78
+ | ------------------------ | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
79
+ | `localPath` _(required)_ | — | Path to a file holding the rendered HTML (post-JS), e.g. `document.documentElement.outerHTML` written to disk by a browser/devtools capture. The server reads it so the page bytes never enter the model context. |
80
+ | `baseUrl` | — | Optional origin. **Never fetched**; used to absolutize relative links/images. |
81
+ | `format` | `markdown` | `markdown` \| `html` \| `text` \| `json`. `json` emits `{metadata, content, diagnostics}`. |
82
+ | `metadataMode` | `none` | `none` \| `yaml` \| `json` — prepend a metadata block to the markdown/text payload. |
83
+ | `extraction` | `balanced` | `balanced` \| `aggressive` \| `conservative` — maps to Readability's scorer knobs. |
84
+ | `selectors.include` | — | Restrict extraction to a subtree: `"main"`, `"article"`, `".post"`. |
85
+ | `selectors.exclude` | — | Strip boilerplate before Readability: `["nav", "footer", "[role=banner]"]`. |
86
+ | `maxNodes` | — | Perf/safety cap = Readability `maxElemsToParse`. |
87
+ | `minArticleLength` | — | Semantic alias for Readability `charThreshold`. |
88
+ | `gfm` | `true` | Tables, strikethrough, task lists. |
89
+ | `headingStyle` | `atx` | `atx` (`#`) \| `setext` (underlining). |
90
+ | `codeBlockStyle` | `fenced` | `fenced` (\`\`\`) \| `indented`. |
91
+ | `images` | `keep` | `keep` \| `drop` \| `src-only` (bare URL) \| `reference` (link-ref style). |
92
+ | `tables` | — | `gfm` (default, native) \| `csv` \| `json` — render `<table>` elements via a rowspan/colspan-aware matrix IR. `csv`/`json` emit fenced code blocks; `gfm` re-renders native tables so headerless and span-degenerate tables round-trip consistently. When unset, tables pass through Turndown's native rule. |
93
+ | `sanitize` | `true` | Run DOMPurify on the article HTML. |
94
+ | `maxChars` | — | Truncate the payload at a block boundary — **never inside a fenced code block**. |
95
+ | `wordsPerMinute` | `200` | For `readingTimeMin`. |
96
+ | `keepClasses` | `false` | Retain all classes (default strips non-language classes). |
97
+ | `cleanChrome` | `true` | Strip browser chrome (scrollbars, consent/cookie banners, fixed nav and overlays) before conversion — these poison Readability's density scoring. |
98
+ | `readabilityOverrides` | — | Escape hatch — passed verbatim to `new Readability(doc, …)`. Unstable. |
99
+ | `chunk` | — | Split the extracted markdown into token-bounded chunks (RAG/embedding-ready). `{maxTokens, overlap?, strategy?}` (strategy defaults to `semantic`) — when set, `structuredContent.chunks` is an array of `{index, text, tokenCount, headingContext}`. Only applies to `format:"markdown" \| "text"`; HTML/JSON payloads carry no markdown body to slice and leave `chunks` unset. |
100
+ | `imageInventory` | `false` | Emit `structuredContent.images` — an array of `{src, alt, width?, height?, caption}` for every `<img>` in the extracted article (absolute resolved srcs, placeholders skipped, caption from the enclosing `<figure>`'s `<figcaption>` else `alt`). Independent of the `images` inline-rendering option. |
101
+ | `debug` | `false` | Emit `diagnostics.trace` with per-stage `{stage, ms}` timings (`normalize`, `readability`, `sanitize`, `turndown`, `metadata`). Debug-only — `trace` is absent otherwise. |
102
+
103
+ **Fallback.** If Readability's `parse()` returns no article (e.g. an app shell or image-only page), a selector cascade salvages the first usable root — `article` → `main` → `[role=main]` → largest text-dense block → `body` — and reports `diagnostics.fallbackUsed: true` with `extractedNode` naming the root that was used.
104
+
105
+ **Site presets.** A preset pairs a site with layout fingerprints (`detectors`) and a `selectors`-shaped scope. When `baseUrl` names a site holding a preset, `extract` applies the scope through the same path as the `selectors` option — no second mechanism — after every detector still matches the document; a preset whose fingerprints no longer hit (site redesign) is discarded and the normal cascade runs, so a stale preset can never win. An explicit `selectors` argument beats the preset. `diagnostics.preset = {site, applied, reason?}` reports the outcome; the field is absent when the site has none. Other tools never resolve presets.
106
+
107
+ Presets load at server start from a local cache directory — one `<site>.json` file per site, read into the preset store before the first tool call:
108
+
109
+ ```json
110
+ {
111
+ "site": "www.dailymail.com",
112
+ "detectors": ["#js-article-text", ".artSplitter"],
113
+ "scope": {
114
+ "include": "div[itemprop=\"articleBody\"]",
115
+ "exclude": [".mol-video", ".vjs-video-container", ".artSplitter"]
116
+ }
117
+ }
118
+ ```
119
+
120
+ | Aspect | Behavior |
121
+ | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
122
+ | Directory | `$READABILITY_PRESETS_DIR`; else `$XDG_CACHE_HOME/readability/presets`; else `~/Library/Caches/readability/presets` (macOS) / `~/.cache/readability/presets`. Setting the variable to an empty string disables preset loading. |
123
+ | Bound | 64 files; beyond that the oldest by mtime are deleted at load — enforced when loading and again when `suggest_preset` writes. |
124
+ | Invalid files | Unreadable JSON, a wrong shape (no `detectors`, empty `scope`), or a non-hostname `site` is skipped with a warning; the remaining files still load. |
125
+ | Staleness | No clock-based expiry — detectors are re-checked against every page, so a preset survives until the site redesigns. |
126
+ | Writing | `suggest_preset` is the writer: an accepted, verified preset is stored in memory immediately and persisted as `<site>.json` after verification comes back clean — on the proposal page, plus the `secondPath` capture when one was given. A disabled directory (empty env value) skips the write and reports `persisted: false` — the in-memory preset still applies for the session. Presets can also land by hand in this exact shape. |
127
+ | Scope | Local to the user's machine; the server never fetches or phones home. |
128
+
129
+ **Metadata cascade.** Each metadata field is resolved by priority: **JSON-LD → OpenGraph → Twitter → `<meta>`/`<time>` → Readability → `<title>`** (first non-empty value wins). When the page carries schema.org JSON-LD, `metadata.structured` exposes the parsed primary object (Recipe/Product/Event/HowTo/Article…) with `@context` stripped and `@type` normalized, so non-article content rides on `extract` without a separate tool. Alongside the bibliographic fields, `metadata` carries `wordCount`, `readingTimeMin`, and `tokenEstimate` (with `estimator: "chars/4"` naming the heuristic) — an advisory count for context budgeting; the host re-counts before sending, so a model-specific tokenizer isn't worth the weight.
130
+
131
+ ### `html_to_markdown` — fragment path
132
+
133
+ Converts an arbitrary HTML fragment to Markdown **without** Readability scoring (e.g. a snippet already isolated via chrome-devtools). Same Turndown + DOMPurify path; reports `fallbackUsed: true`, `extractedNode: "fragment"`. Takes `localPath` plus the same `format`, `gfm`, `headingStyle`, `codeBlockStyle`, `images`, `tables`, `sanitize`, `maxChars`, `wordsPerMinute`, `selectors`, `baseUrl`, `metadataMode`, `cleanChrome`, and `debug` options as `extract`. Metadata is minimal (`baseUrl`, `wordCount`, `readingTimeMin`, and a title from the fragment's first heading).
134
+
135
+ ### `extract_section` — one section by selector or heading
136
+
137
+ Returns just one section of a document — "give me the Authentication section" on a long doc without paying for full extraction. A thin resolver over `extract`'s `selectors.include` path, not a new extractor: selector mode passes straight through, and heading mode wraps the matched subtree in `<section data-rdrm-section-scope>` before routing through the same `selectors.include` path.
138
+
139
+ | Option | Default | Description |
140
+ | ------------------------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
141
+ | `localPath` _(required)_ | — | Path to a file holding the rendered HTML (post-JS), e.g. `document.documentElement.outerHTML` written to disk by a browser/devtools capture. The server reads it so the page bytes never enter the model context. |
142
+ | `baseUrl` | — | Optional origin. **Never fetched**; used to absolutize relative links/images. |
143
+ | `selector` | — | CSS selector scoping extraction to one subtree; passed straight through as `selectors.include`. Provide exactly one of `selector`/`heading`. |
144
+ | `heading` | — | Heading text selecting one section; the section spans from this heading to the next same-or-higher-level heading. Case-insensitive; first exact match wins, falling back to the first substring contain. Provide exactly one of `selector`/`heading`. |
145
+
146
+ Output shape is the same as `extract` (`content`, `metadata`, `diagnostics`). Heading mode is equivalent to selector mode on the same subtree: `heading: "Authentication"` discovers the same boundary a wrapping `<section id="auth">…</section>` would expose via `selector: "#auth"`. A non-matching heading yields `{ "isError": true }` with `no heading matched: <query>`.
147
+
148
+ ### `extract_tables` — every table on the page
149
+
150
+ Extracts **every** `<table>` on the page — a `querySelectorAll('table')` walk (page-wide by default; narrow it with `selectors.include`) in front of the same rowspan/colspan-aware matrix serializer used by the `tables` option on `extract`. Runs **no** Readability, Turndown, sanitization, or `normalizeDocument` chrome-stripping, so it captures tables outside the article body (nav, aside, footer, boilerplate) that the `tables` option on `extract` never sees — the motivating case is wiki/doc/data pages whose content is table-heavy but whose article boundary hides most of them.
151
+
152
+ | Option | Default | Description |
153
+ | ------------------------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
154
+ | `localPath` _(required)_ | — | Path to a file holding the rendered HTML (post-JS), e.g. `document.documentElement.outerHTML` written to disk by a browser/devtools capture. The server reads it so the page bytes never enter the model context. |
155
+ | `baseUrl` | — | Optional origin. **Never fetched**; carried through to `metadata.baseUrl`. |
156
+ | `format` | `gfm` | `gfm` (default, native GFM table with a delimiter row) \| `csv` (RFC-4180-ish, quoted fields) \| `json` (array of row objects keyed by the header row). |
157
+ | `selectors` | — | Same `include`/`exclude` shape as `extract`. Scope the walk: `include:"#shareholding"` returns only tables inside that subtree; `exclude:[".ads"]` drops matches anywhere on the page. |
158
+
159
+ Output shape: `structuredContent.tables = [{index, rows, cols, markdown}]` — one entry per non-empty table in document order, where `rows`/`cols` are the matrix dimensions after rowspan/colspan resolution and `markdown` is the table rendered in the requested format. All entries' `markdown` are joined by blank lines into `content[0].text` (`"(no tables found)"` when the page has none). `metadata = {baseUrl?, format, tableCount}`. Empty `<table>` elements (no rows) are skipped, so `index` is contiguous over the emitted tables. Nested `<table>`s are emitted as their own entries in document order (the matrix walk excludes nested tables from a parent's matrix; `querySelectorAll` then returns the nested table separately).
160
+
161
+ ### `extract_grid` — CSS-grid / div tables (the div equivalent of extract_tables)
162
+
163
+ Detects and extracts a **CSS-grid / div "table"** — for SPA pages that render data into repeating `<div>` rows instead of `<table>` (analyst-estimate tables, financial summaries, comparison grids) where `extract_tables` returns nothing. Two modes: **auto-detect** finds the container whose direct children form the largest same-shape sibling group of **≥3** rows (each row a set of **≥2** direct element-children, outside `nav`/`header`/`footer`/`aside`); **selector mode** takes explicit `rowSelector` + `cellSelector` (cells scoped to each row subtree). The detected matrix is rendered through the **same** gfm/csv/json matrix renderer as `extract_tables`. Runs **no** Readability, Turndown, sanitization, or `normalizeDocument` chrome-stripping in selector mode (auto mode strips chrome internally so an article's nav doesn't look like a 4-row grid).
164
+
165
+ | Option | Default | Description |
166
+ | ------------------------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
167
+ | `localPath` _(required)_ | — | Path to a file holding the rendered HTML (post-JS), e.g. `document.documentElement.outerHTML` written to disk by a browser/devtools capture. The server reads it so the page bytes never enter the model context. |
168
+ | `baseUrl` | — | Optional origin. **Never fetched**; carried through to `metadata.baseUrl`. |
169
+ | `format` | `gfm` | `gfm` (default, native GFM table with a delimiter row) \| `csv` (RFC-4180-ish, quoted fields) \| `json` (array of row objects keyed by the header row — first row used as keys when non-empty, else `column_N`). |
170
+ | `selectors` | — | Same `include`/`exclude` shape as `extract`. Scope auto-detection: `include:"#estimates"` narrows the scan to that subtree. |
171
+ | `rowSelector` | — | CSS selector for repeating row containers. When set **with** `cellSelector`, selector mode is used (no auto-detection). Example: `'div[class*="estimate-row"]'`. |
172
+ | `cellSelector` | — | CSS selector for cells within each row (scoped to the row subtree). Required together with `rowSelector` — both-or-neither (setting only one is rejected). Example: `'div[class*="cell"]'`. |
173
+
174
+ Output shape: `structuredContent = {schemaVersion, content, grid, diagnostics, metadata}`. `grid = {rows, cols, markdown}` — the single detected grid (`rows`/`cols` 0 and `markdown` empty when nothing is detected), where `markdown` is the grid rendered in the requested format (ragged rows are padded to a dense rectangular matrix). `content[0].text` is the grid markdown, or `"(no repeating grid found)"`. `diagnostics = {detected, rowCount, colCount, containerSelector, rowTag, confidence, note}`: `detected:false` means no grid structure was found; `containerSelector`/`rowTag` name the winning cluster (or `rowSelector` in selector mode); `rowCount` counts emitted rows (a recovered header row included); `confidence` is `high` when ≥6 detected data rows, `medium` when ≥3, `low` otherwise (a recovered header is inference and does not raise it). `metadata = {baseUrl?, format, detected}`.
175
+
176
+ ### `extract_list` — feed/index/search pages
177
+
178
+ A **second engine** for pages Readability cannot turn into one article: HN-style feeds, search-result pages, blog indexes, product grids. Strips `nav`/`header`/`footer`/`aside` + ARIA chrome roles first (the false-positive guard so an article's nav menu doesn't look like a 4-item feed), then finds the container whose direct children form a same-shape sibling cluster of **≥3** elements each carrying a navigation anchor — the cluster with the most items wins. Runs **no** Readability, Turndown, sanitization, or `normalizeDocument` chrome-stripping (the detector scores against the very chrome-bearing structure the article normalizer would discard). Returns `detected:false` on article pages.
179
+
180
+ | Option | Default | Description |
181
+ | ------------------------ | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
182
+ | `localPath` _(required)_ | — | Path to a file holding the rendered HTML (post-JS), e.g. `document.documentElement.outerHTML` written to disk by a browser/devtools capture. The server reads it so the page bytes never enter the model context. |
183
+ | `baseUrl` | — | Optional origin. **Never fetched**; used to absolutize item `href`s. |
184
+ | `selectors` | — | Same `include`/`exclude` shape as `extract`. `include` picks which list the detector scores against — note the detector is comparative ("the cluster with the most items wins"), so pre-scoping to one container subverts that comparison; treat it as an "I know which list I want" escape hatch. |
185
+
186
+ Output shape: `structuredContent = {schemaVersion, content, items, diagnostics, metadata}`. `items = [{title, url, snippet, score}]` in document order — `snippet` is the item's text teaser (empty when the cluster has no per-item text), `score` is the detector's internal ranking weight. `diagnostics = {detected, itemCount, containerSelector, itemTag, confidence, note}`: `detected:false` means no list structure was found (`itemCount:0`, empty `items`, and a `note` explaining why); `containerSelector`/`itemTag` name the winning cluster; `confidence` is a rough quality signal. `metadata = {baseUrl}`.
187
+
188
+ ### `outline` — heading pre-check
189
+
190
+ Returns the document outline (`h1`–`h6` in document order with stable anchor ids) as a cheap "is this worth reading?" / "where's the section about X?" pre-check before paying for full extraction. Runs **no** Readability, Turndown, or sanitization — a pure heading walk over the normalized DOM.
191
+
192
+ | Option | Default | Description |
193
+ | ------------------------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
194
+ | `localPath` _(required)_ | — | Path to a file holding the rendered HTML (post-JS), e.g. `document.documentElement.outerHTML` written to disk by a browser/devtools capture. The server reads it so the page bytes never enter the model context. |
195
+ | `baseUrl` | — | Optional origin. **Never fetched**; carried through to `metadata.baseUrl`. |
196
+ | `selectors` | — | Same `include`/`exclude` shape as `extract`. `include:"main"` scopes the heading walk to that subtree, dropping nav/footer headings from the outline. |
197
+
198
+ Output shape: `structuredContent.outline = [{level, text, anchor}]` plus an indented-bullet TOC rendered into `content[0].text`, and `metadata = {title?, baseUrl?}` (`title` falls back from `<title>` to the first `<h1>`). Anchor precedence: the heading's own `id`, then a descendant permalink's `#fragment`, then a slug of the text (deduped `-1`, `-2`, … for generated slugs only — author ids are kept verbatim).
199
+
200
+ ### `extract_links` — anchor inventory for crawl/navigation
201
+
202
+ Returns a structured list of anchor links — `[{text, href, rel, isExternal}]` in document order — gathered from the raw parsed DOM. Runs **no** Readability, Turndown, sanitization, or `normalizeDocument` chrome-stripping, so nav/footer/main links survive (the crawl-relevant ones). Pairs with chrome-devtools for crawl/navigation decisions: the host picks the next page without re-parsing HTML.
203
+
204
+ | Option | Default | Description |
205
+ | ------------------------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
206
+ | `localPath` _(required)_ | — | Path to a file holding the rendered HTML (post-JS), e.g. `document.documentElement.outerHTML` written to disk by a browser/devtools capture. The server reads it so the page bytes never enter the model context. |
207
+ | `baseUrl` | — | Optional origin. **Never fetched**; absolutizes relative `href`s and drives `isExternal`. |
208
+ | `sameOriginOnly` | `false` | Drop cross-origin links; keep same-origin, relative, fragment, and non-http(s) (`mailto`/`tel`/`javascript`) links. |
209
+ | `selectors` | — | Same `include`/`exclude` shape as `extract`. DOM-level scope (e.g. `include:"#peers"`) applied before the link walk; composes with `sameOriginOnly`'s semantic filter. |
210
+
211
+ Output shape: `structuredContent.links = [{text, href, rel, isExternal}]` plus a `- [text](href)` rendering in `content[0].text`. `href` is absolutized against `baseUrl` (unchanged when `baseUrl` is absent or the pair fails to parse). `isExternal` is `true` only when `baseUrl` is provided **and** the absolutized `href` parses to a different HTTP(S) origin — relative, fragment, same-origin, `mailto:`/`tel:`/`javascript:`, and malformed hrefs are all `false`. `rel` is the raw attribute value (`"noopener noreferrer"`, `"nofollow"`, …) or `""` when absent. Anchors with no `href` are skipped; the rest are kept in document order with **no deduplication**.
212
+
213
+ ### `extract_metadata` — bibliographic pre-check
214
+
215
+ Returns only the bibliographic metadata — `title`, `byline`, `siteName`, `lang`, `publishedTime`, `excerpt`, `canonical`, `baseUrl` — without running Readability/Turndown, as a fast pre-check for crawlers and citation. Short-circuits the pipeline before the article body is scored; resolves the same metadata cascade as `extract` (JSON-LD → OpenGraph → Twitter → `<meta>`/`<time>` → `<title>`), plus `<link rel="canonical">` → `og:url` for `canonical`. The `baseUrl` field is the origin you passed in; `canonical` is the page's declared canonical — they often differ.
216
+
217
+ | Option | Default | Description |
218
+ | ------------------------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
219
+ | `localPath` _(required)_ | — | Path to a file holding the rendered HTML (post-JS), e.g. `document.documentElement.outerHTML` written to disk by a browser/devtools capture. The server reads it so the page bytes never enter the model context. |
220
+ | `baseUrl` | — | Optional origin. **Never fetched**; carried through to `metadata.baseUrl`. |
221
+
222
+ Output shape: `structuredContent.metadata = {title?, byline?, siteName?, lang?, publishedTime?, excerpt?, canonical?, baseUrl?}` plus a human-readable `key: value` rendering in `content[0].text`. Note: `wordCount`/`readingTimeMin`/`tokenEstimate` are **not** populated by this tool — they are meaningless without the extracted body.
223
+
224
+ ### `explain` — extraction post-mortem
225
+
226
+ Post-mortem diagnostics for an `extract` call: surfaces **why** Readability picked what it picked. Runs the same normalize + Readability pipeline as `extract` (no fallback cascade, no Turndown, no DOMPurify) and reads Readability's real per-candidate `contentScore` values off the DOM expando Readability stamps during scoring. Reach for it when `extract` lands on the wrong root or strips content you expected — it shows the scored runners-up so you can tune `selectors`/`extraction`/`minArticleLength`.
227
+
228
+ | Option | Default | Description |
229
+ | ------------------------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
230
+ | `localPath` _(required)_ | — | Path to a file holding the rendered HTML (post-JS), e.g. `document.documentElement.outerHTML` written to disk by a browser/devtools capture. The server reads it so the page bytes never enter the model context. |
231
+ | `baseUrl` | — | Optional origin. **Never fetched**; used for pagination/gating detection only. |
232
+ | `selectors` | — | Same `include`/`exclude` shape as `extract` — applied at the normalize step so the diagnosis matches what `extract` would see. |
233
+ | `topN` | `5` | Maximum scored candidate nodes to return (highest first); 1–20. |
234
+
235
+ Output shape: `structuredContent = {schemaVersion, content, chosenRoot, candidates, readerable, parseSucceeded, fallbackUsed, gating, pagination, removedNodes, snapshot}`. `chosenRoot` is Readability's raw top pick (before parent-walking/only-child post-processing); `candidates` is the ranked list (capped at `topN`) where each entry carries `{tag, id, className, selector, score, textLength}` — `score` is Readability's actual `contentScore`, not a self-rolled heuristic, and `selector` is a CSS-ish hint, **not** a unique locator (the score lives on a JS expando invisible to CSS). `removedNodes = {total, chrome, boilerplate}`. `gating`/`pagination` mirror `extract`'s diagnostics (`null` when none). `snapshot = {html, truncated}` is the post-normalize, pre-Readability HTML — "what Readability saw" — capped at 4000 chars. `parseSucceeded:false` is the signal that `extract` would have hit its fallback cascade; `fallbackUsed` is always `false` here (explain never runs the cascade).
236
+
237
+ ### `chunk_text` — chunk for RAG/embedding
238
+
239
+ Splits already-extracted text into token-bounded chunks, each carrying `index`, `text`, `tokenCount` (chars/4, same estimator as `metadata.tokenEstimate`), and `headingContext` (the heading hierarchy path in effect at the chunk's first unit — empty string when the chunk precedes any heading). Operates on any text — pair with `extract`'s `chunk` option when you want chunks inline with the extraction.
240
+
241
+ | Option | Default | Description |
242
+ | ------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
243
+ | `text` _(required)_ | — | Already-extracted text to split (e.g. markdown from `extract`). No HTML parsing or Readability scoring — the input is chunked verbatim. |
244
+ | `maxTokens` | `500` | Per-chunk token budget. No chunk exceeds this; oversized blocks are split by line, then hard-split. |
245
+ | `overlap` | `0` | Tokens to overlap between consecutive chunks (`>=0`). The trailing overlapChars of chunk N becomes the leading context of chunk N+1. |
246
+ | `strategy` | `semantic` | Chunking strategy. `semantic` (default) breaks on heading/section boundaries and never splits a fenced code block (an oversized code block is emitted as its own chunk that may exceed the budget — the deliberate tradeoff for keeping fences intact); `char` is the greedy char-bounded fallback that may split a code block. |
247
+
248
+ Output shape: `structuredContent.chunks = [{index, text, tokenCount, headingContext}]` in order, plus a readable numbered index in `content[0].text`. Empty array when the input has no non-whitespace content.
249
+
250
+ ### `summarize` — host-model summarization (sampling-gated)
251
+
252
+ Delegates summarization to the **host's** model via MCP `sampling/createMessage` — the server embeds no model and calls no provider directly. Only listed when the connected client advertises the `sampling` capability on `initialize` (registered after the handshake, so a non-sampling host never sees it on `tools/list`); otherwise invisible. The host picks the model and may prompt the user before each call (human-in-the-loop, per MCP). Hand it the output of `extract`/`extract_section`/`html_to_markdown`/`chunk_text` — or any markdown/text string.
253
+
254
+ | Option | Default | Description |
255
+ | ------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
256
+ | `text` _(required)_ | — | Markdown or text to summarize. Passed through to the host model verbatim; the server does not parse or modify it. |
257
+ | `maxTokens` | `4096` | Ceiling on tokens the host model may generate, forwarded as `sampling/createMessage` `maxTokens`. Thinking counts toward it on thinking models; summary length comes from the prompt. |
258
+
259
+ Output shape: a single `content[0].text` entry holding the host's summary. No `structuredContent` — the server returns whatever the host model produces. A non-text response from the host (e.g. an image) surfaces as `{ "isError": true }`.
260
+
261
+ ### `suggest_preset` — the site-preset suggest loop (sampling-gated)
262
+
263
+ For a page whose extraction was lost, asks the host's model to propose a site preset, validates it deterministically, applies it through the real pipeline, and — on convergence — stores it in memory and persists it to the preset cache directory. Call it after `extract` reports lost signals: gated content, `fallbackUsed`, a near-empty result, or visible debris (e.g. video-player control text). On a healthy page it refuses without any sampling call.
264
+
265
+ | Option | Default | Description |
266
+ | ------------------------ | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
267
+ | `localPath` _(required)_ | — | Path to a file holding the rendered HTML. |
268
+ | `baseUrl` _(required)_ | — | A URL of the site; its host names the preset. Refused when the page's `<link rel="canonical">` names a different host. |
269
+ | `secondPath` | — | A second capture of the same site. The converged preset must verify on it too before anything is persisted; without it the preset is verified on the proposal page alone. |
270
+ | `maxSamplingCalls` | `4` | Budget for host-model calls. Round one (include + detectors) and round two (excludes) each consume one; rejected proposals consume retries. A budget of 1 skips round two. |
271
+
272
+ The loop, in order:
273
+
274
+ 1. **Trigger.** A baseline `extract` runs with caching off; the lost verdict is `fallbackUsed`, a word count under 120, or a debris probe over the extracted text (`Loaded: N%` / `Duration Time m:ss` player controls, metered-barrier phrases). A gating signal alone never fires the loop — vendor SDK classes report on free articles. The baseline's canonical URL must name the same host as `baseUrl`, or the run refuses — a capture keyed to the wrong site can never mint a preset.
275
+ 2. **Round one.** The model sees a copy-safe text-chain outline (ancestor chains of every node with ≥200 chars of own text, real attributes, hash classes already stripped) and returns `{"detectors": […], "scope": {"include": "…", "exclude": […]}}`.
276
+ 3. **Validation.** Static: no positional pseudos, no `:contains`, no generated hash identifiers. DOM: detectors and `include` must match, proposed excludes must match, and no exclude may shadow the include root. Rejections are fed back verbatim for a retry.
277
+ 4. **Round two.** The accepted `include` is applied for real; the model then sees the included subtree's remaining text blocks (grouped by chain, with samples) and proposes excludes for the debris groups.
278
+ 5. **Verification.** The candidate preset is added to the store and a verification `extract` runs through the real preset path. Converged = the preset applied **and** the extraction comes back clean. With `secondPath`, the same verification runs on the second capture — detectors must match there and the extraction must come back clean — and a capture whose canonical names another site refuses the run. Only then is `<site>.json` persisted; non-convergence removes the preset and leaves no file.
279
+
280
+ Output: `trigger` (the lost-signal verdict), `preset` (when accepted), `verification` (`applied`, `converged`, `verifiedPages`, word counts before/after), `secondVerification` (the second capture's verdict, when `secondPath` was given), `persistence` (file written, or why not), and `sampling` (calls/budget/exhausted). Every model proposal is validated — a proposal the pipeline would silently misapply never reaches it.
281
+
282
+ ## Diagnostics
283
+
284
+ `structuredContent.diagnostics` exposes: `readerable`, `extractedNode`, `fallbackUsed`, `removedNodes` (element delta vs. the document), `chromeRemoved` and `imagesResolved` (pre-conversion cleanup counts), `boilerplateRemoved` (related-posts / newsletter-signup / read-next blocks stripped before conversion, footprint-guarded so article content is never deleted), `sanitization.{scripts,iframes}` (counted across the **whole** pipeline), `pagination` (`{type:"paginated"|"infinite", nextUrl?, selector?}` — detection only; the host drives loading, this server never fetches), `gated` (`{likely, reason}` — detection only; signals a likely paywall/metered gate so the host knows the extraction may be partial — this server never fetches or authenticates), `preset` (`{site, applied, reason?}` — `extract` only, present only when the document's site holds a preset: `reason:"overridden"` when an explicit `selectors` argument won, `"detectors-missed"` when a fingerprint no longer matched and the normal cascade ran), `truncated`, and `trace` (per-stage `{stage, ms}` timings — **debug-only**, emitted only when `debug:true` is passed to `extract`/`html_to_markdown`; absent otherwise). Stages are non-overlapping and ordered: `normalize`, `readability`, `sanitize`, `turndown`, `metadata` on the article path (`html_to_markdown` omits `readability`); on the fallback path a single `fallback` stage covers sanitize + turndown so the timings still sum to the pipeline's wall-clock.
285
+
286
+ ## Rich content
287
+
288
+ - **Code-block language tags.** Before Readability scores the document, real-world code-block conventions are canonicalized to `<pre><code class="language-X">` so the language survives Readability's class stripping and Turndown emits a tagged fence. Mapped conventions: GitHub `<div class="highlight highlight-source-js">` wrappers (`-shell`, `-python`, …), React/sandpack `<pre class="sp-javascript">`, and generic `lang-X` / `brush: X`. Common language tokens are added to Readability's `classesToPreserve` so ` ```js `/` ```shell ` land in the markdown instead of a bare fence; exotic languages fall back to an untagged fence. Automatic (no option); `html_to_markdown` is unaffected (it skips Readability).
289
+ - **Footnotes.** When an article pairs `<sup>` reference markers with a definitions list (`<ol class="footnotes">`, `<ol class="references">`, `[role="doc-endnotes"]`, or standalone `<li id="fn-…">`/`<li id="cite_note-…">`), both halves are auto-converted to Markdown footnote syntax — inline `[^N]` markers in place of the `<sup>` and an appended `[^N]: definition` block. The conversion is automatic (no option); when no footnote markup is detected, output is byte-identical to a plain turndown.
290
+ - **Math.** KaTeX (`<span class="katex">` with an `<annotation encoding="application/x-tex">`) and MathJax (`<script type="math/tex">` / `mode=display`) are auto-converted to `$…$` (inline) or `$$…$$` (display) LaTeX before turndown runs, so raw backslashes survive unescaped and the rendered spans never leak. The conversion is automatic (no option); when the source LaTeX is absent (a broken `.katex` with no annotation, an empty MathJax script), a `[?]` placeholder is emitted in its place — never a crash.
291
+
292
+ ## Payload size (stdio)
293
+
294
+ A full rendered SPA can be several MB as a string, and MCP tool args travel over JSON-RPC on stdio. Mitigations:
295
+
296
+ - **Scoped capture (recommended)** — real pages are large (hundreds of KB of `outerHTML`), so capture only what you need rather than the whole document. Via chrome-devtools `evaluate_script` (with its `filePath` arg), write `document.head.outerHTML` (for metadata) plus a content subtree such as `document.querySelector('article')?.outerHTML || document.querySelector('main')?.outerHTML` to a file, then point `extract` at it via `localPath`. `baseUrl` absolutizes relative links within whatever HTML is passed.
297
+ - **`selectors.include`** — scope to the article subtree, e.g. `"main"`, so only the relevant DOM is scored and serialized.
298
+ - **`maxChars`** — cap the returned payload; truncation lands at a block boundary and never splits a fenced code block.
299
+ - **`maxNodes`** — a hard cap on elements parsed (`Readability.maxElemsToParse`) for very large documents.
300
+
301
+ ## Resources (page cache)
302
+
303
+ `extract({cache:true})` caches the result and exposes it as an addressable MCP Resource at `readability://page/{hash}`. Subsequent `extract` calls with the same HTML (modulo volatile bytes — see below) and the same output options hit the cache instead of re-running the pipeline. The cache is in-memory, bounded (256 entries, LRU), and TTL'd (30 min).
304
+
305
+ - **`diagnostics.cache = {hit, normalizedHash, originalHash}`** appears on every cached `extract` result: `hit:true`/`false`, plus both hashes. The `normalizedHash` is what the key is built from; `originalHash` is the SHA-256 of the raw HTML. A miss where `normalizedHash` matches an existing entry but the lookup still missed points at an args-fingerprint mismatch (different `format`/`selectors`/…) rather than a genuinely different page — useful when debugging "should have hit."
306
+ - **Normalized-hash keying.** Before hashing, the HTML is volatility-normalized: inline `<script>` blocks, CSP `<meta>` tags, per-render `nonce=` attributes, build-tool generated attribute names (`data-v-…`, `data-css-…`, `data-svelte-…`, `data-h-…`), and React/Next generated `id`s (`:R1:`, `:r1:`, `__next_…`, `reactX_…`) are stripped, and whitespace runs are collapsed. The same page re-rendered with a fresh CSP nonce or a different build hash collapses to the same key.
307
+ - **Listing and reading.** `resources/list` enumerates current cache entries (`readability://page/{cacheKey}`, `text/markdown`); `resources/read` on a `readability://page/{hash}` URI returns the cached markdown (empty body if the entry has expired or been evicted).
308
+
309
+ ## CLI
310
+
311
+ `readability` also runs as a one-shot CLI for extracting from a local HTML file or stdin, with no MCP server in the loop:
312
+
313
+ ```bash
314
+ readability extract [file.html] [--format md|json|html] [--max-chars N]
315
+ ```
316
+
317
+ - `extract` is the only subcommand; everything after it is parsed as options. With no args at all (`readability`), the stdio MCP server starts instead.
318
+ - `file.html` is read from disk; when no file is given, HTML is read from **stdin**.
319
+ - `--format`: `md` (default, markdown) | `json` (the `structuredContent` object, pretty-printed) | `html` (the post-pipeline HTML). Internally `json` reuses the markdown pipeline and serializes the structured object on the way out.
320
+ - `--max-chars N` mirrors `extract`'s `maxChars` — truncate the payload at a block boundary, never inside a fenced code block.
321
+
322
+ ```bash
323
+ curl -s https://example.com | readability extract --format md
324
+ readability extract page.html --format json --max-chars 20000
325
+ cat saved.html | readability extract
326
+ ```
327
+
328
+ ## Development
329
+
330
+ ```bash
331
+ yarn typecheck # tsc --noEmit — from the repo root
332
+ yarn build # vite build -> dist/index.js
333
+ yarn lint # eslint — from the repo root
334
+ yarn test # vitest run
335
+ yarn test:update-goldens # UPDATE_GOLDENS=1 vitest run
336
+ ```
337
+
338
+ ## Benchmark
339
+
340
+ `yarn bench` prints a per-fixture metrics table (input nodes, markdown chars, token estimate, compression ratio, removed nodes, and preserved images/tables/links) plus a unified content delta against committed baselines under `test/bench/baseline/`. It also prints a **precision/recall table** of the extracted main content vs human-labeled boundaries (`test/bench/labels.ts` — one CSS selector per fixture naming its article container), with a macro-average aggregate row, and an **aggregate per-stage timing breakdown** averaged from the `debug` trace across fixtures. The bench runs in CI as a **non-blocking** job (`continue-on-error: true`), so a regression is surfaced, not gating; `bench.test.ts` additionally fails `yarn test` if the committed metrics or scores drift out of sync.
341
+
342
+ ```bash
343
+ yarn bench # print metrics + content deltas + PR/timing tables
344
+ BENCH_UPDATE=1 yarn bench # refresh baselines (do deliberately, like UPDATE_GOLDENS)
345
+ ```
346
+
347
+ Per-fixture fields: `inputNodes` (parsed element count), `markdownChars`/`tokens` (output size, chars/4), `compressionRatio` (output chars per input node), `removedNodes` (element delta across the pipeline), and `images`/`tables`/`links` (preserved content counts). PR fields: `precision` (fraction of extracted word tokens inside the labeled main content), `recall` (fraction of labeled tokens recovered), `f1` (harmonic mean), `extractedTokens`/`labeledTokens` (multiset sizes). Fixtures with no prose (the image-only `fallback` gallery) score N/A and are excluded from the aggregate. Site-preset scenarios (`test/bench/fixtures.ts` `PRESET_SCENARIOS`) are scored against the same human labels and land in `scores.json` as `<fixtureId>@preset`; they print as their own table with their own aggregate, never folded into the default-pipeline aggregate. Suggest-loop scenarios (`test/bench/suggest-scenarios.ts`) replay the proposals the loop accepted through its own validators (static lint, preset match contract, material budget) and land as `<fixtureId>@suggest` in a third table — the suggest-loop baseline.
348
+
349
+ ## Corpus fixtures
350
+
351
+ Captured real pages live beside the hand-built ones under `test/fixtures/<site>-<slug>/` (`saved.html` + `saved.test.ts`). New captures are slimmed with `node scripts/trim-capture.mjs <capture.html> test/fixtures/<dir>/saved.html` — executable scripts, style payloads, and base64 data URIs are dropped; every element and attribute is kept, so DOM-level behavior is unchanged. Each capture's `saved.test.ts` asserts the failure class it proves (current debris, gating signals, applied-selector behavior). Captured pages are composite — article plus feed furniture — so the list-detector's article-fixture guard excludes them and their detection behavior is asserted in their own test.
352
+
353
+ ## License
354
+
355
+ MIT
package/dist/assets/cli-boGClDt0.js ADDED
@@ -0,0 +1,71 @@
1
+ import { ft as describeError, j as readHtmlFile, mt as parseQuietly, t as extractArticleFromHtml } from "./extract-DaYydtVZ.js";
2
+ import { Command, InvalidArgumentError, Option } from "commander";
3
+ //#region src/cli.ts
4
+ var FORMATS = [
5
+ "html",
6
+ "json",
7
+ "md"
8
+ ];
9
+ function parseMaxChars(value) {
10
+ const n = Number(value);
11
+ if (!Number.isInteger(n)) throw new InvalidArgumentError("must be an integer");
12
+ return n;
13
+ }
14
+ function buildProgram() {
15
+ return new Command("readability extract").argument("[file]", "HTML file; stdin when omitted").addOption(new Option("--format <fmt>", "output format").choices(FORMATS).default("md")).option("--max-chars <n>", "truncate the output", parseMaxChars);
16
+ }
17
+ function parseArgs(argv) {
18
+ const program = parseQuietly(buildProgram(), argv.slice(1));
19
+ if (program === void 0) return;
20
+ const { format, maxChars } = program.opts();
21
+ return {
22
+ file: program.args.at(0),
23
+ format,
24
+ maxChars
25
+ };
26
+ }
27
+ async function readHtml(file, stream) {
28
+ if (file !== void 0) return readHtmlFile(file);
29
+ const chunks = [];
30
+ for await (const chunk of stream) if (typeof chunk === "string") chunks.push(chunk);
31
+ else chunks.push(Buffer.from(chunk).toString("utf8"));
32
+ return chunks.join("");
33
+ }
34
+ function payloadText(result) {
35
+ const first = result.content.at(0);
36
+ return first !== void 0 && "text" in first ? first.text : "";
37
+ }
38
+ async function runCli(argv) {
39
+ if (argv[0] !== "extract") {
40
+ process.stderr.write(buildProgram().helpInformation());
41
+ return 2;
42
+ }
43
+ const parsed = parseArgs(argv);
44
+ if (parsed === void 0) {
45
+ process.stderr.write(buildProgram().helpInformation());
46
+ return 2;
47
+ }
48
+ try {
49
+ const html = await readHtml(parsed.file, process.stdin);
50
+ const pipelineFormat = parsed.format === "html" ? "html" : "markdown";
51
+ const result = extractArticleFromHtml({
52
+ html,
53
+ format: pipelineFormat,
54
+ ...parsed.maxChars !== void 0 ? { maxChars: parsed.maxChars } : {}
55
+ });
56
+ if (result.isError) {
57
+ process.stderr.write(`${payloadText(result)}\n`);
58
+ return 1;
59
+ }
60
+ if (parsed.format === "json") process.stdout.write(`${JSON.stringify(result.structuredContent, null, 2)}\n`);
61
+ else process.stdout.write(`${payloadText(result)}\n`);
62
+ return 0;
63
+ } catch (err) {
64
+ process.stderr.write(`${describeError(err)}\n`);
65
+ return 1;
66
+ }
67
+ }
68
+ //#endregion
69
+ export { runCli };
70
+
71
+ //# sourceMappingURL=cli-boGClDt0.js.map
package/dist/assets/cli-boGClDt0.js.map ADDED
@@ -0,0 +1 @@
1
+ {"version":3,"file":"cli-boGClDt0.js","names":[],"sources":["../../src/cli.ts"],"sourcesContent":["import { parseQuietly } from '@v1nvn/agentic-core';\nimport { Command, InvalidArgumentError, Option } from 'commander';\n\nimport { describeError } from './errors.js';\nimport { extractArticleFromHtml } from './tools/extract.js';\nimport { readHtmlFile } from './tools/html-source.js';\n\nimport type { CallToolResult } from '@modelcontextprotocol/sdk/types.js';\nimport type { Readable } from 'node:stream';\n\ntype CliFormat = 'html' | 'json' | 'md';\n\nexport interface ParsedArgs {\n readonly file: string | undefined;\n readonly format: CliFormat;\n readonly maxChars: number | undefined;\n}\n\nconst FORMATS: readonly CliFormat[] = ['html', 'json', 'md'];\n\nfunction parseMaxChars(value: string): number {\n const n = Number(value);\n if (!Number.isInteger(n)) {\n throw new InvalidArgumentError('must be an integer');\n }\n return n;\n}\n\nexport function buildProgram(): Command {\n return new Command('readability extract')\n .argument('[file]', 'HTML file; stdin when omitted')\n .addOption(\n new Option('--format <fmt>', 'output format')\n .choices(FORMATS)\n .default('md'),\n )\n .option('--max-chars <n>', 'truncate the output', parseMaxChars);\n}\n\nexport function parseArgs(argv: readonly string[]): ParsedArgs | undefined {\n const program = parseQuietly(buildProgram(), argv.slice(1));\n if (program === undefined) {\n return undefined;\n }\n const { format, maxChars } = program.opts<{\n format: CliFormat;\n maxChars: number | undefined;\n }>();\n return { file: program.args.at(0), format, maxChars };\n}\n\n// The stream is injected rather than reading process.stdin directly so the\n// path is testable. Chunks may be Buffer (process.stdin) or string\n// (Readable.from), so both are handled.\nexport async function readHtml(\n file: string | undefined,\n stream: Readable,\n): Promise<string> {\n if (file !== undefined) {\n return readHtmlFile(file);\n }\n const chunks: string[] = [];\n for await (const chunk of stream) {\n if (typeof chunk === 'string') {\n chunks.push(chunk);\n } else {\n chunks.push(Buffer.from(chunk as Uint8Array).toString('utf8'));\n }\n }\n return chunks.join('');\n}\n\nfunction payloadText(result: CallToolResult): string {\n const first = result.content.at(0);\n return first !== undefined && 'text' in first ? first.text : '';\n}\n\nexport async function runCli(argv: readonly string[]): Promise<number> {\n if (argv[0] !== 'extract') {\n process.stderr.write(buildProgram().helpInformation());\n return 2;\n }\n\n const parsed = parseArgs(argv);\n if (parsed === undefined) {\n process.stderr.write(buildProgram().helpInformation());\n return 2;\n }\n\n try {\n const html = await readHtml(parsed.file, process.stdin);\n // json reuses the markdown pipeline; the structured object is serialized below.\n const pipelineFormat = parsed.format === 'html' ? 'html' : 'markdown';\n const result = extractArticleFromHtml({\n html,\n format: pipelineFormat,\n ...(parsed.maxChars !== undefined ? { maxChars: parsed.maxChars } : {}),\n });\n\n if (result.isError) {\n process.stderr.write(`${payloadText(result)}\\n`);\n return 1;\n }\n\n if (parsed.format === 'json') {\n process.stdout.write(\n `${JSON.stringify(result.structuredContent, null, 2)}\\n`,\n );\n } else {\n process.stdout.write(`${payloadText(result)}\\n`);\n }\n return 0;\n } catch (err) {\n process.stderr.write(`${describeError(err)}\\n`);\n return 1;\n }\n}\n"],"mappings":";;;AAkBA,IAAM,UAAgC;CAAC;CAAQ;CAAQ;AAAI;AAE3D,SAAS,cAAc,OAAuB;CAC5C,MAAM,IAAI,OAAO,KAAK;CACtB,IAAI,CAAC,OAAO,UAAU,CAAC,GACrB,MAAM,IAAI,qBAAqB,oBAAoB;CAErD,OAAO;AACT;AAEA,SAAgB,eAAwB;CACtC,OAAO,IAAI,QAAQ,qBAAqB,CAAC,CACtC,SAAS,UAAU,+BAA+B,CAAC,CACnD,UACC,IAAI,OAAO,kBAAkB,eAAe,CAAC,CAC1C,QAAQ,OAAO,CAAC,CAChB,QAAQ,IAAI,CACjB,CAAC,CACA,OAAO,mBAAmB,uBAAuB,aAAa;AACnE;AAEA,SAAgB,UAAU,MAAiD;CACzE,MAAM,UAAU,aAAa,aAAa,GAAG,KAAK,MAAM,CAAC,CAAC;CAC1D,IAAI,YAAY,KAAA,GACd;CAEF,MAAM,EAAE,QAAQ,aAAa,QAAQ,KAGlC;CACH,OAAO;EAAE,MAAM,QAAQ,KAAK,GAAG,CAAC;EAAG;EAAQ;CAAS;AACtD;AAKA,eAAsB,SACpB,MACA,QACiB;CACjB,IAAI,SAAS,KAAA,GACX,OAAO,aAAa,IAAI;CAE1B,MAAM,SAAmB,CAAC;CAC1B,WAAW,MAAM,SAAS,QACxB,IAAI,OAAO,UAAU,UACnB,OAAO,KAAK,KAAK;MAEjB,OAAO,KAAK,OAAO,KAAK,KAAmB,CAAC,CAAC,SAAS,MAAM,CAAC;CAGjE,OAAO,OAAO,KAAK,EAAE;AACvB;AAEA,SAAS,YAAY,QAAgC;CACnD,MAAM,QAAQ,OAAO,QAAQ,GAAG,CAAC;CACjC,OAAO,UAAU,KAAA,KAAa,UAAU,QAAQ,MAAM,OAAO;AAC/D;AAEA,eAAsB,OAAO,MAA0C;CACrE,IAAI,KAAK,OAAO,WAAW;EACzB,QAAQ,OAAO,MAAM,aAAa,CAAC,CAAC,gBAAgB,CAAC;EACrD,OAAO;CACT;CAEA,MAAM,SAAS,UAAU,IAAI;CAC7B,IAAI,WAAW,KAAA,GAAW;EACxB,QAAQ,OAAO,MAAM,aAAa,CAAC,CAAC,gBAAgB,CAAC;EACrD,OAAO;CACT;CAEA,IAAI;EACF,MAAM,OAAO,MAAM,SAAS,OAAO,MAAM,QAAQ,KAAK;EAEtD,MAAM,iBAAiB,OAAO,WAAW,SAAS,SAAS;EAC3D,MAAM,SAAS,uBAAuB;GACpC;GACA,QAAQ;GACR,GAAI,OAAO,aAAa,KAAA,IAAY,EAAE,UAAU,OAAO,SAAS,IAAI,CAAC;EACvE,CAAC;EAED,IAAI,OAAO,SAAS;GAClB,QAAQ,OAAO,MAAM,GAAG,YAAY,MAAM,EAAE,GAAG;GAC/C,OAAO;EACT;EAEA,IAAI,OAAO,WAAW,QACpB,QAAQ,OAAO,MACb,GAAG,KAAK,UAAU,OAAO,mBAAmB,MAAM,CAAC,EAAE,GACvD;OAEA,QAAQ,OAAO,MAAM,GAAG,YAAY,MAAM,EAAE,GAAG;EAEjD,OAAO;CACT,SAAS,KAAK;EACZ,QAAQ,OAAO,MAAM,GAAG,cAAc,GAAG,EAAE,GAAG;EAC9C,OAAO;CACT;AACF"}