@v1nvn/readability-mcp 0.14.0

package/README.md

@@ -0,0 +1,310 @@
+# readability-mcp
+
+Turn **already-rendered HTML** (captured post-JavaScript from a browser or [chrome-devtools MCP](https://github.com/anthropic/claude-code-chrome-devtools)) into clean, LLM-friendly **Markdown + metadata**, using [Mozilla Readability](https://github.com/mozilla/readability), [Turndown](https://github.com/mixmark-io/turndown), and [DOMPurify](https://github.com/cure53/DOMPurify).
+
+The key idea: **rendering and extraction are decoupled.** A real browser (chrome-devtools) owns rendering; this server only transforms HTML it reads from a file. **The server makes no outbound requests** — there is no `fetch`, no SSRF surface. Every HTML-input tool takes a `localPath` (a file on disk), never an inline string, so a full rendered page never enters the model context. The optional `baseUrl` is *origin context only*, used to absolutize relative links; it is never fetched.
+
+## Install
+
+```bash
+npm install @v1nvn/readability-mcp
+# or run on demand:
+npx @v1nvn/readability-mcp
+```
+
+Requires Node >= 22. Build from source:
+
+```bash
+git clone <repo> && cd packages/readability-mcp
+yarn install
+yarn build         # bundles to dist/index.js
+node dist/index.js # starts the stdio MCP server
+```
+
+### Docker / Smithery
+
+A `Dockerfile` (multi-stage `node:22-bookworm-slim`, runs as non-root `node`) and a `smithery.yaml` (stdio runtime) are included for container and [Smithery](https://smithery.ai) deployment:
+
+```bash
+# from the repo root — the workspace installs from root manifests
+docker build -f packages/readability-mcp/Dockerfile -t readability-mcp .
+docker run --rm -i readability-mcp            # stdio MCP server on stdin/stdout
+docker run --rm -i readability-mcp extract --format md < page.html
+```
+
+The Smithery manifest pins the `stdio` startCommand (this server ships `StdioServerTransport` only — the HTTP container runtime cannot launch it) and surfaces `READABILITY_MCP_LOG_LEVEL` as the one config knob.
+
+## The chrome-devtools handoff
+
+The motivating flow is two hops — each tool does the one thing it is best at:
+
+```js
+// 1. In the chrome-devtools MCP, grab the RENDERED document (post-JS) and write
+//    it to a file via evaluate_script's `filePath` arg (the model emits only a
+//    path, never the page bytes):
+mcp__chrome-devtools__evaluate_script({
+  function: () => document.documentElement.outerHTML,
+  filePath: "/tmp/page.html",
+});
+
+// 2. Point readability-mcp at that file.
+//    `baseUrl` is OPTIONAL context (origin for absolutizing relative links) — never fetched.
+mcp__readability__extract({ localPath: "/tmp/page.html", baseUrl: pageUrl });
+```
+
+This matters most for SPAs and JS-augmented pages, where the initial HTML is an empty `<div id="root">` and only the post-JS DOM has the content.
+
+## MCP client config
+
+Add to your MCP client config (Claude Code, Claude Desktop, etc.):
+
+```jsonc
+{
+  "mcpServers": {
+    "readability": {
+      "command": "npx",
+      "args": ["-y", "@v1nvn/readability-mcp"]
+    }
+  }
+}
+```
+
+## Tools
+
+All eleven always-on tools return MCP **structured content** (`schemaVersion` plus a tool-specific payload of `metadata` / `diagnostics` / `items` / …) validated by a zod `outputSchema`, plus a human/LLM-readable payload in `content[0].text`. A sampling-capable host also sees a twelfth — `summarize` — registered after the `initialize` handshake when the client advertises the MCP `sampling` capability. Nothing throws across the wire — failures become `{ "isError": true }` results. Every input and output field carries a description in the tool's JSON schema, so clients can introspect each option without reading these docs.
+
+**`localPath` — the only HTML input.** Every HTML-input tool takes a `localPath` pointing at a file holding the already-rendered (post-JavaScript) HTML. The server reads the bytes itself, so the page never enters the model context — the model emits only a path string. The motivating hop is chrome-devtools → readability: `evaluate_script` writes `document.documentElement.outerHTML` to a file via its `filePath` arg, then the tool reads that path. Resolved relative to the server process working directory; prefer absolute paths so the chrome-devtools capture and this read agree on location.
+
+### `extract` — primary tool
+
+Extracts the main article from rendered HTML and returns Markdown + metadata + diagnostics.
+
+| Option | Default | Description |
+| --- | --- | --- |
+| `localPath` *(required)* | — | Path to a file holding the rendered HTML (post-JS), e.g. `document.documentElement.outerHTML` written to disk by a browser/devtools capture. The server reads it so the page bytes never enter the model context. |
+| `baseUrl` | — | Optional origin. **Never fetched**; used to absolutize relative links/images. |
+| `format` | `markdown` | `markdown` \| `html` \| `text` \| `json`. `json` emits `{metadata, content, diagnostics}`. |
+| `metadataMode` | `none` | `none` \| `yaml` \| `json` — prepend a metadata block to the markdown/text payload. |
+| `extraction` | `balanced` | `balanced` \| `aggressive` \| `conservative` — maps to Readability's scorer knobs. |
+| `selectors.include` | — | Restrict extraction to a subtree: `"main"`, `"article"`, `".post"`. |
+| `selectors.exclude` | — | Strip boilerplate before Readability: `["nav", "footer", "[role=banner]"]`. |
+| `maxNodes` | — | Perf/safety cap = Readability `maxElemsToParse`. |
+| `minArticleLength` | — | Semantic alias for Readability `charThreshold`. |
+| `gfm` | `true` | Tables, strikethrough, task lists. |
+| `headingStyle` | `atx` | `atx` (`#`) \| `setext` (underlining). |
+| `codeBlockStyle` | `fenced` | `fenced` (\`\`\`) \| `indented`. |
+| `images` | `keep` | `keep` \| `drop` \| `src-only` (bare URL) \| `reference` (link-ref style). |
+| `tables` | — | `gfm` (default, native) \| `csv` \| `json` — render `<table>` elements via a rowspan/colspan-aware matrix IR. `csv`/`json` emit fenced code blocks; `gfm` re-renders native tables so headerless and span-degenerate tables round-trip consistently. When unset, tables pass through Turndown's native rule. |
+| `sanitize` | `true` | Run DOMPurify on the article HTML. |
+| `maxChars` | — | Truncate the payload at a block boundary — **never inside a fenced code block**. |
+| `wordsPerMinute` | `200` | For `readingTimeMin`. |
+| `keepClasses` | `false` | Retain all classes (default strips non-language classes). |
+| `readabilityOverrides` | — | Escape hatch — passed verbatim to `new Readability(doc, …)`. Unstable. |
+| `chunk` | — | Split the extracted markdown into token-bounded chunks (RAG/embedding-ready). `{maxTokens, overlap?, strategy?}` (strategy defaults to `semantic`) — when set, `structuredContent.chunks` is an array of `{index, text, tokenCount, headingContext}`. Only applies to `format:"markdown" \| "text"`; HTML/JSON payloads carry no markdown body to slice and leave `chunks` unset. |
+| `imageInventory` | `false` | Emit `structuredContent.images` — an array of `{src, alt, width?, height?, caption}` for every `<img>` in the extracted article (absolute resolved srcs, placeholders skipped, caption from the enclosing `<figure>`'s `<figcaption>` else `alt`). Independent of the `images` inline-rendering option. |
+| `debug` | `false` | Emit `diagnostics.trace` with per-stage `{stage, ms}` timings (`normalize`, `readability`, `sanitize`, `turndown`, `metadata`). Debug-only — `trace` is absent otherwise. |
+
+**Fallback.** If Readability's `parse()` returns no article (e.g. an app shell or image-only page), a selector cascade salvages the first usable root — `article` → `main` → `[role=main]` → largest text-dense block → `body` — and reports `diagnostics.fallbackUsed: true` with `extractedNode` naming the root that was used.
+
+**Metadata cascade.** Each metadata field is resolved by priority: **JSON-LD → OpenGraph → Twitter → `<meta>`/`<time>` → Readability → `<title>`** (first non-empty value wins). When the page carries schema.org JSON-LD, `metadata.structured` exposes the parsed primary object (Recipe/Product/Event/HowTo/Article…) with `@context` stripped and `@type` normalized, so non-article content rides on `extract` without a separate tool. Alongside the bibliographic fields, `metadata` carries `wordCount`, `readingTimeMin`, and `tokenEstimate` (with `estimator: "chars/4"` naming the heuristic) — an advisory count for context budgeting; the host re-counts before sending, so a model-specific tokenizer isn't worth the weight.
+
+### `html_to_markdown` — fragment path
+
+Converts an arbitrary HTML fragment to Markdown **without** Readability scoring (e.g. a snippet already isolated via chrome-devtools). Same Turndown + DOMPurify path; reports `fallbackUsed: true`, `extractedNode: "fragment"`. Takes `localPath` plus the same `format`, `gfm`, `headingStyle`, `codeBlockStyle`, `images`, `tables`, `sanitize`, `maxChars`, `wordsPerMinute`, `selectors`, `baseUrl`, and `debug` options as `extract`. Metadata is minimal (`baseUrl`, `wordCount`, `readingTimeMin`, and a title from the fragment's first heading).
+
+### `extract_section` — one section by selector or heading
+
+Returns just one section of a document — "give me the Authentication section" on a long doc without paying for full extraction. A thin resolver over `extract`'s `selectors.include` path, not a new extractor: selector mode passes straight through, and heading mode wraps the matched subtree in `<section data-rdrm-section-scope>` before routing through the same `selectors.include` path.
+
+| Option | Default | Description |
+| --- | --- | --- |
+| `localPath` *(required)* | — | Path to a file holding the rendered HTML (post-JS), e.g. `document.documentElement.outerHTML` written to disk by a browser/devtools capture. The server reads it so the page bytes never enter the model context. |
+| `baseUrl` | — | Optional origin. **Never fetched**; used to absolutize relative links/images. |
+| `selector` | — | CSS selector scoping extraction to one subtree; passed straight through as `selectors.include`. Provide exactly one of `selector`/`heading`. |
+| `heading` | — | Heading text selecting one section; the section spans from this heading to the next same-or-higher-level heading. Case-insensitive; first exact match wins, falling back to the first substring contain. Provide exactly one of `selector`/`heading`. |
+
+Output shape is the same as `extract` (`content`, `metadata`, `diagnostics`). Heading mode is equivalent to selector mode on the same subtree: `heading: "Authentication"` discovers the same boundary a wrapping `<section id="auth">…</section>` would expose via `selector: "#auth"`. A non-matching heading yields `{ "isError": true }` with `no heading matched: <query>`.
+
+### `extract_tables` — every table on the page
+
+Extracts **every** `<table>` on the page — a `querySelectorAll('table')` walk (page-wide by default; narrow it with `selectors.include`) in front of the same rowspan/colspan-aware matrix serializer used by the `tables` option on `extract`. Runs **no** Readability, Turndown, sanitization, or `normalizeDocument` chrome-stripping, so it captures tables outside the article body (nav, aside, footer, boilerplate) that the `tables` option on `extract` never sees — the motivating case is wiki/doc/data pages whose content is table-heavy but whose article boundary hides most of them.
+
+| Option | Default | Description |
+| --- | --- | --- |
+| `localPath` *(required)* | — | Path to a file holding the rendered HTML (post-JS), e.g. `document.documentElement.outerHTML` written to disk by a browser/devtools capture. The server reads it so the page bytes never enter the model context. |
+| `baseUrl` | — | Optional origin. **Never fetched**; carried through to `metadata.baseUrl`. |
+| `format` | `gfm` | `gfm` (default, native GFM table with a delimiter row) \| `csv` (RFC-4180-ish, quoted fields) \| `json` (array of row objects keyed by the header row). |
+| `selectors` | — | Same `include`/`exclude` shape as `extract`. Scope the walk: `include:"#shareholding"` returns only tables inside that subtree; `exclude:[".ads"]` drops matches anywhere on the page. |
+
+Output shape: `structuredContent.tables = [{index, rows, cols, markdown}]` — one entry per non-empty table in document order, where `rows`/`cols` are the matrix dimensions after rowspan/colspan resolution and `markdown` is the table rendered in the requested format. All entries' `markdown` are joined by blank lines into `content[0].text` (`"(no tables found)"` when the page has none). `metadata = {baseUrl?, format, tableCount}`. Empty `<table>` elements (no rows) are skipped, so `index` is contiguous over the emitted tables. Nested `<table>`s are emitted as their own entries in document order (the matrix walk excludes nested tables from a parent's matrix; `querySelectorAll` then returns the nested table separately).
+
+### `extract_grid` — CSS-grid / div tables (the div equivalent of extract_tables)
+
+Detects and extracts a **CSS-grid / div "table"** — for SPA pages that render data into repeating `<div>` rows instead of `<table>` (analyst-estimate tables, financial summaries, comparison grids) where `extract_tables` returns nothing. Two modes: **auto-detect** finds the container whose direct children form the largest same-shape sibling group of **≥3** rows (each row a set of **≥2** direct element-children, outside `nav`/`header`/`footer`/`aside`); **selector mode** takes explicit `rowSelector` + `cellSelector` (cells scoped to each row subtree). The detected matrix is rendered through the **same** gfm/csv/json matrix renderer as `extract_tables`. Runs **no** Readability, Turndown, sanitization, or `normalizeDocument` chrome-stripping in selector mode (auto mode strips chrome internally so an article's nav doesn't look like a 4-row grid).
+
+| Option | Default | Description |
+| --- | --- | --- |
+| `localPath` *(required)* | — | Path to a file holding the rendered HTML (post-JS), e.g. `document.documentElement.outerHTML` written to disk by a browser/devtools capture. The server reads it so the page bytes never enter the model context. |
+| `baseUrl` | — | Optional origin. **Never fetched**; carried through to `metadata.baseUrl`. |
+| `format` | `gfm` | `gfm` (default, native GFM table with a delimiter row) \| `csv` (RFC-4180-ish, quoted fields) \| `json` (array of row objects keyed by the header row — first row used as keys when non-empty, else `column_N`). |
+| `selectors` | — | Same `include`/`exclude` shape as `extract`. Scope auto-detection: `include:"#estimates"` narrows the scan to that subtree. |
+| `rowSelector` | — | CSS selector for repeating row containers. When set **with** `cellSelector`, selector mode is used (no auto-detection). Example: `'div[class*="estimate-row"]'`. |
+| `cellSelector` | — | CSS selector for cells within each row (scoped to the row subtree). Required together with `rowSelector` — both-or-neither (setting only one is rejected). Example: `'div[class*="cell"]'`. |
+
+Output shape: `structuredContent = {schemaVersion, content, grid, diagnostics, metadata}`. `grid = {rows, cols, markdown}` — the single detected grid (`rows`/`cols` 0 and `markdown` empty when nothing is detected), where `markdown` is the grid rendered in the requested format (ragged rows are padded to a dense rectangular matrix). `content[0].text` is the grid markdown, or `"(no repeating grid found)"`. `diagnostics = {detected, rowCount, colCount, containerSelector, rowTag, confidence, note}`: `detected:false` means no grid structure was found; `containerSelector`/`rowTag` name the winning cluster (or `rowSelector` in selector mode); `rowCount` counts emitted rows (a recovered header row included); `confidence` is `high` when ≥6 detected data rows, `medium` when ≥3, `low` otherwise (a recovered header is inference and does not raise it). `metadata = {baseUrl?, format, detected}`.
+
+### `extract_list` — feed/index/search pages
+
+A **second engine** for pages Readability cannot turn into one article: HN-style feeds, search-result pages, blog indexes, product grids. Strips `nav`/`header`/`footer`/`aside` + ARIA chrome roles first (the false-positive guard so an article's nav menu doesn't look like a 4-item feed), then finds the container whose direct children form a same-shape sibling cluster of **≥3** elements each carrying a navigation anchor — the cluster with the most items wins. Runs **no** Readability, Turndown, sanitization, or `normalizeDocument` chrome-stripping (the detector scores against the very chrome-bearing structure the article normalizer would discard). Returns `detected:false` on article pages.
+
+| Option | Default | Description |
+| --- | --- | --- |
+| `localPath` *(required)* | — | Path to a file holding the rendered HTML (post-JS), e.g. `document.documentElement.outerHTML` written to disk by a browser/devtools capture. The server reads it so the page bytes never enter the model context. |
+| `baseUrl` | — | Optional origin. **Never fetched**; used to absolutize item `href`s. |
+| `selectors` | — | Same `include`/`exclude` shape as `extract`. `include` picks which list the detector scores against — note the detector is comparative ("the cluster with the most items wins"), so pre-scoping to one container subverts that comparison; treat it as an "I know which list I want" escape hatch. |
+
+Output shape: `structuredContent = {schemaVersion, content, items, diagnostics, metadata}`. `items = [{title, url, snippet, score}]` in document order — `snippet` is the item's text teaser (empty when the cluster has no per-item text), `score` is the detector's internal ranking weight. `diagnostics = {detected, itemCount, containerSelector, itemTag, confidence, note}`: `detected:false` means no list structure was found (`itemCount:0`, empty `items`, and a `note` explaining why); `containerSelector`/`itemTag` name the winning cluster; `confidence` is a rough quality signal. `metadata = {baseUrl}`.
+
+### `outline` — heading pre-check
+
+Returns the document outline (`h1`–`h6` in document order with stable anchor ids) as a cheap "is this worth reading?" / "where's the section about X?" pre-check before paying for full extraction. Runs **no** Readability, Turndown, or sanitization — a pure heading walk over the normalized DOM.
+
+| Option | Default | Description |
+| --- | --- | --- |
+| `localPath` *(required)* | — | Path to a file holding the rendered HTML (post-JS), e.g. `document.documentElement.outerHTML` written to disk by a browser/devtools capture. The server reads it so the page bytes never enter the model context. |
+| `baseUrl` | — | Optional origin. **Never fetched**; carried through to `metadata.baseUrl`. |
+| `selectors` | — | Same `include`/`exclude` shape as `extract`. `include:"main"` scopes the heading walk to that subtree, dropping nav/footer headings from the outline. |
+
+Output shape: `structuredContent.outline = [{level, text, anchor}]` plus an indented-bullet TOC rendered into `content[0].text`, and `metadata = {title?, baseUrl?}` (`title` falls back from `<title>` to the first `<h1>`). Anchor precedence: the heading's own `id`, then a descendant permalink's `#fragment`, then a slug of the text (deduped `-1`, `-2`, … for generated slugs only — author ids are kept verbatim).
+
+### `extract_links` — anchor inventory for crawl/navigation
+
+Returns a structured list of anchor links — `[{text, href, rel, isExternal}]` in document order — gathered from the raw parsed DOM. Runs **no** Readability, Turndown, sanitization, or `normalizeDocument` chrome-stripping, so nav/footer/main links survive (the crawl-relevant ones). Pairs with chrome-devtools for crawl/navigation decisions: the host picks the next page without re-parsing HTML.
+
+| Option | Default | Description |
+| --- | --- | --- |
+| `localPath` *(required)* | — | Path to a file holding the rendered HTML (post-JS), e.g. `document.documentElement.outerHTML` written to disk by a browser/devtools capture. The server reads it so the page bytes never enter the model context. |
+| `baseUrl` | — | Optional origin. **Never fetched**; absolutizes relative `href`s and drives `isExternal`. |
+| `sameOriginOnly` | `false` | Drop cross-origin links; keep same-origin, relative, fragment, and non-http(s) (`mailto`/`tel`/`javascript`) links. |
+| `selectors` | — | Same `include`/`exclude` shape as `extract`. DOM-level scope (e.g. `include:"#peers"`) applied before the link walk; composes with `sameOriginOnly`'s semantic filter. |
+
+Output shape: `structuredContent.links = [{text, href, rel, isExternal}]` plus a `- [text](href)` rendering in `content[0].text`. `href` is absolutized against `baseUrl` (unchanged when `baseUrl` is absent or the pair fails to parse). `isExternal` is `true` only when `baseUrl` is provided **and** the absolutized `href` parses to a different HTTP(S) origin — relative, fragment, same-origin, `mailto:`/`tel:`/`javascript:`, and malformed hrefs are all `false`. `rel` is the raw attribute value (`"noopener noreferrer"`, `"nofollow"`, …) or `""` when absent. Anchors with no `href` are skipped; the rest are kept in document order with **no deduplication**.
+
+### `extract_metadata` — bibliographic pre-check
+
+Returns only the bibliographic metadata — `title`, `byline`, `siteName`, `lang`, `publishedTime`, `excerpt`, `canonical`, `baseUrl` — without running Readability/Turndown, as a fast pre-check for crawlers and citation. Short-circuits the pipeline before the article body is scored; resolves the same metadata cascade as `extract` (JSON-LD → OpenGraph → Twitter → `<meta>`/`<time>` → `<title>`), plus `<link rel="canonical">` → `og:url` for `canonical`. The `baseUrl` field is the origin you passed in; `canonical` is the page's declared canonical — they often differ.
+
+| Option | Default | Description |
+| --- | --- | --- |
+| `localPath` *(required)* | — | Path to a file holding the rendered HTML (post-JS), e.g. `document.documentElement.outerHTML` written to disk by a browser/devtools capture. The server reads it so the page bytes never enter the model context. |
+| `baseUrl` | — | Optional origin. **Never fetched**; carried through to `metadata.baseUrl`. |
+
+Output shape: `structuredContent.metadata = {title?, byline?, siteName?, lang?, publishedTime?, excerpt?, canonical?, baseUrl?}` plus a human-readable `key: value` rendering in `content[0].text`. Note: `wordCount`/`readingTimeMin`/`tokenEstimate` are **not** populated by this tool — they are meaningless without the extracted body.
+
+### `explain` — extraction post-mortem
+
+Post-mortem diagnostics for an `extract` call: surfaces **why** Readability picked what it picked. Runs the same normalize + Readability pipeline as `extract` (no fallback cascade, no Turndown, no DOMPurify) and reads Readability's real per-candidate `contentScore` values off the DOM expando Readability stamps during scoring. Reach for it when `extract` lands on the wrong root or strips content you expected — it shows the scored runners-up so you can tune `selectors`/`extraction`/`minArticleLength`.
+
+| Option | Default | Description |
+| --- | --- | --- |
+| `localPath` *(required)* | — | Path to a file holding the rendered HTML (post-JS), e.g. `document.documentElement.outerHTML` written to disk by a browser/devtools capture. The server reads it so the page bytes never enter the model context. |
+| `baseUrl` | — | Optional origin. **Never fetched**; used for pagination/gating detection only. |
+| `selectors` | — | Same `include`/`exclude` shape as `extract` — applied at the normalize step so the diagnosis matches what `extract` would see. |
+| `topN` | `5` | Maximum scored candidate nodes to return (highest first); 1–20. |
+
+Output shape: `structuredContent = {schemaVersion, content, chosenRoot, candidates, readerable, parseSucceeded, fallbackUsed, gating, pagination, removedNodes, snapshot}`. `chosenRoot` is Readability's raw top pick (before parent-walking/only-child post-processing); `candidates` is the ranked list (capped at `topN`) where each entry carries `{tag, id, className, selector, score, textLength}` — `score` is Readability's actual `contentScore`, not a self-rolled heuristic, and `selector` is a CSS-ish hint, **not** a unique locator (the score lives on a JS expando invisible to CSS). `removedNodes = {total, chrome, boilerplate}`. `gating`/`pagination` mirror `extract`'s diagnostics (`null` when none). `snapshot = {html, truncated}` is the post-normalize, pre-Readability HTML — "what Readability saw" — capped at 4000 chars. `parseSucceeded:false` is the signal that `extract` would have hit its fallback cascade; `fallbackUsed` is always `false` here (explain never runs the cascade).
+
+### `chunk_text` — chunk for RAG/embedding
+
+Splits already-extracted text into token-bounded chunks, each carrying `index`, `text`, `tokenCount` (chars/4, same estimator as `metadata.tokenEstimate`), and `headingContext` (the heading hierarchy path in effect at the chunk's first unit — empty string when the chunk precedes any heading). Operates on any text — pair with `extract`'s `chunk` option when you want chunks inline with the extraction.
+
+| Option | Default | Description |
+| --- | --- | --- |
+| `text` *(required)* | — | Already-extracted text to split (e.g. markdown from `extract`). No HTML parsing or Readability scoring — the input is chunked verbatim. |
+| `maxTokens` | `500` | Per-chunk token budget. No chunk exceeds this; oversized blocks are split by line, then hard-split. |
+| `overlap` | `0` | Tokens to overlap between consecutive chunks (`>=0`). The trailing overlapChars of chunk N becomes the leading context of chunk N+1. |
+| `strategy` | `semantic` | Chunking strategy. `semantic` (default) breaks on heading/section boundaries and never splits a fenced code block (an oversized code block is emitted as its own chunk that may exceed the budget — the deliberate tradeoff for keeping fences intact); `char` is the greedy char-bounded fallback that may split a code block. |
+
+Output shape: `structuredContent.chunks = [{index, text, tokenCount, headingContext}]` in order, plus a readable numbered index in `content[0].text`. Empty array when the input has no non-whitespace content.
+
+### `summarize` — host-model summarization (sampling-gated)
+
+Delegates summarization to the **host's** model via MCP `sampling/createMessage` — the server embeds no model and calls no provider directly. Only listed when the connected client advertises the `sampling` capability on `initialize` (registered after the handshake, so a non-sampling host never sees it on `tools/list`); otherwise invisible. The host picks the model and may prompt the user before each call (human-in-the-loop, per MCP). Hand it the output of `extract`/`extract_section`/`html_to_markdown`/`chunk_text` — or any markdown/text string.
+
+| Option | Default | Description |
+| --- | --- | --- |
+| `text` *(required)* | — | Markdown or text to summarize. Passed through to the host model verbatim; the server does not parse or modify it. |
+| `maxTokens` | `512` | Upper bound on the summary length in tokens, forwarded as `sampling/createMessage` `maxTokens`. The host chooses the actual length. |
+
+Output shape: a single `content[0].text` entry holding the host's summary. No `structuredContent` — the server returns whatever the host model produces. A non-text response from the host (e.g. an image) surfaces as `{ "isError": true }`.
+
+## Diagnostics
+
+`structuredContent.diagnostics` exposes: `readerable`, `extractedNode`, `fallbackUsed`, `removedNodes` (element delta vs. the document), `chromeRemoved` and `imagesResolved` (pre-conversion cleanup counts), `boilerplateRemoved` (related-posts / newsletter-signup / read-next blocks stripped before conversion, footprint-guarded so article content is never deleted), `sanitization.{scripts,iframes}` (counted across the **whole** pipeline), `pagination` (`{type:"paginated"|"infinite", nextUrl?, selector?}` — detection only; the host drives loading, this server never fetches), `gated` (`{likely, reason}` — detection only; signals a likely paywall/metered gate so the host knows the extraction may be partial — this server never fetches or authenticates), `truncated`, and `trace` (per-stage `{stage, ms}` timings — **debug-only**, emitted only when `debug:true` is passed to `extract`/`html_to_markdown`; absent otherwise). Stages are non-overlapping and ordered: `normalize`, `readability`, `sanitize`, `turndown`, `metadata` on the article path (`html_to_markdown` omits `readability`); on the fallback path a single `fallback` stage covers sanitize + turndown so the timings still sum to the pipeline's wall-clock.
+
+## Rich content
+
+- **Code-block language tags.** Before Readability scores the document, real-world code-block conventions are canonicalized to `<pre><code class="language-X">` so the language survives Readability's class stripping and Turndown emits a tagged fence. Mapped conventions: GitHub `<div class="highlight highlight-source-js">` wrappers (`-shell`, `-python`, …), React/sandpack `<pre class="sp-javascript">`, and generic `lang-X` / `brush: X`. Common language tokens are added to Readability's `classesToPreserve` so ` ```js `/` ```shell ` land in the markdown instead of a bare fence; exotic languages fall back to an untagged fence. Automatic (no option); `html_to_markdown` is unaffected (it skips Readability).
+- **Footnotes.** When an article pairs `<sup>` reference markers with a definitions list (`<ol class="footnotes">`, `<ol class="references">`, `[role="doc-endnotes"]`, or standalone `<li id="fn-…">`/`<li id="cite_note-…">`), both halves are auto-converted to Markdown footnote syntax — inline `[^N]` markers in place of the `<sup>` and an appended `[^N]: definition` block. The conversion is automatic (no option); when no footnote markup is detected, output is byte-identical to a plain turndown.
+- **Math.** KaTeX (`<span class="katex">` with an `<annotation encoding="application/x-tex">`) and MathJax (`<script type="math/tex">` / `mode=display`) are auto-converted to `$…$` (inline) or `$$…$$` (display) LaTeX before turndown runs, so raw backslashes survive unescaped and the rendered spans never leak. The conversion is automatic (no option); when the source LaTeX is absent (a broken `.katex` with no annotation, an empty MathJax script), a `[?]` placeholder is emitted in its place — never a crash.
+
+## Payload size (stdio)
+
+A full rendered SPA can be several MB as a string, and MCP tool args travel over JSON-RPC on stdio. Mitigations:
+
+- **Scoped capture (recommended)** — real pages are large (hundreds of KB of `outerHTML`), so capture only what you need rather than the whole document. Via chrome-devtools `evaluate_script` (with its `filePath` arg), write `document.head.outerHTML` (for metadata) plus a content subtree such as `document.querySelector('article')?.outerHTML || document.querySelector('main')?.outerHTML` to a file, then point `extract` at it via `localPath`. `baseUrl` absolutizes relative links within whatever HTML is passed.
+- **`selectors.include`** — scope to the article subtree, e.g. `"main"`, so only the relevant DOM is scored and serialized.
+- **`maxChars`** — cap the returned payload; truncation lands at a block boundary and never splits a fenced code block.
+- **`maxNodes`** — a hard cap on elements parsed (`Readability.maxElemsToParse`) for very large documents.
+
+## Resources (page cache)
+
+`extract({cache:true})` caches the result and exposes it as an addressable MCP Resource at `readability://page/{hash}`. Subsequent `extract` calls with the same HTML (modulo volatile bytes — see below) and the same output options hit the cache instead of re-running the pipeline. The cache is in-memory, bounded (256 entries, LRU), and TTL'd (30 min).
+
+- **`diagnostics.cache = {hit, normalizedHash, originalHash}`** appears on every cached `extract` result: `hit:true`/`false`, plus both hashes. The `normalizedHash` is what the key is built from; `originalHash` is the SHA-256 of the raw HTML. A miss where `normalizedHash` matches an existing entry but the lookup still missed points at an args-fingerprint mismatch (different `format`/`selectors`/…) rather than a genuinely different page — useful when debugging "should have hit."
+- **Normalized-hash keying.** Before hashing, the HTML is volatility-normalized: inline `<script>` blocks, CSP `<meta>` tags, per-render `nonce=` attributes, build-tool generated attribute names (`data-v-…`, `data-css-…`, `data-svelte-…`, `data-h-…`), and React/Next generated `id`s (`:R1:`, `:r1:`, `__next_…`, `reactX_…`) are stripped, and whitespace runs are collapsed. The same page re-rendered with a fresh CSP nonce or a different build hash collapses to the same key.
+- **Listing and reading.** `resources/list` enumerates current cache entries (`readability://page/{cacheKey}`, `text/markdown`); `resources/read` on a `readability://page/{hash}` URI returns the cached markdown (empty body if the entry has expired or been evicted).
+
+## CLI
+
+`readability-mcp` also runs as a one-shot CLI for extracting from a local HTML file or stdin, with no MCP server in the loop:
+
+```bash
+readability-mcp extract [file.html] [--format md|json|html] [--max-chars N]
+```
+
+- `extract` is the only subcommand; everything after it is parsed as options. With no args at all (`readability-mcp`), the stdio MCP server starts instead.
+- `file.html` is read from disk; when no file is given, HTML is read from **stdin**.
+- `--format`: `md` (default, markdown) | `json` (the `structuredContent` object, pretty-printed) | `html` (the post-pipeline HTML). Internally `json` reuses the markdown pipeline and serializes the structured object on the way out.
+- `--max-chars N` mirrors `extract`'s `maxChars` — truncate the payload at a block boundary, never inside a fenced code block.
+
+```bash
+curl -s https://example.com | readability-mcp extract --format md
+readability-mcp extract page.html --format json --max-chars 20000
+cat saved.html | readability-mcp extract
+```
+
+## Development
+
+```bash
+yarn typecheck   # tsc --noEmit
+yarn build       # vite build -> dist/index.js
+yarn lint        # eslint
+yarn test        # vitest run
+yarn test:update-goldens   # UPDATE_GOLDENS=1 vitest run
+```
+
+## Benchmark
+
+`yarn bench` prints a per-fixture metrics table (input nodes, markdown chars, token estimate, compression ratio, removed nodes, and preserved images/tables/links) plus a unified content delta against committed baselines under `test/bench/baseline/`. It also prints a **precision/recall table** of the extracted main content vs human-labeled boundaries (`test/bench/labels.ts` — one CSS selector per fixture naming its article container), with a macro-average aggregate row, and an **aggregate per-stage timing breakdown** averaged from the `debug` trace across fixtures. The bench runs in CI as a **non-blocking** job (`continue-on-error: true`), so a regression is surfaced, not gating; `bench.test.ts` additionally fails `yarn test` if the committed metrics or scores drift out of sync.
+
+```bash
+yarn bench                # print metrics + content deltas + PR/timing tables
+BENCH_UPDATE=1 yarn bench # refresh baselines (do deliberately, like UPDATE_GOLDENS)
+```
+
+Per-fixture fields: `inputNodes` (parsed element count), `markdownChars`/`tokens` (output size, chars/4), `compressionRatio` (output chars per input node), `removedNodes` (element delta across the pipeline), and `images`/`tables`/`links` (preserved content counts). PR fields: `precision` (fraction of extracted word tokens inside the labeled main content), `recall` (fraction of labeled tokens recovered), `f1` (harmonic mean), `extractedTokens`/`labeledTokens` (multiset sizes). Fixtures with no prose (the image-only `fallback` gallery) score N/A and are excluded from the aggregate.
+
+## License
+
+MIT

package/dist/assets/cli-BwKCixh6.js

@@ -0,0 +1,83 @@
+import { m as readHtmlFile, t as extractArticleFromHtml } from "./extract-BKl4PzEI.js";
+//#region src/cli.ts
+var USAGE = "Usage: readability-mcp extract [file.html] [--format md|json|html] [--max-chars N]";
+var FORMATS = [
+	"html",
+	"json",
+	"md"
+];
+function isCliFormat(value) {
+	return value !== void 0 && FORMATS.includes(value);
+}
+function parseArgs(argv) {
+	let file;
+	let format = "md";
+	let maxChars;
+	const rest = argv.slice(1);
+	for (let i = 0; i < rest.length; i++) {
+		const arg = rest[i];
+		if (arg === "--format") {
+			const value = rest.at(++i);
+			if (!isCliFormat(value)) return;
+			format = value;
+		} else if (arg === "--max-chars") {
+			const value = rest.at(++i);
+			if (value === void 0) return;
+			const n = Number(value);
+			if (!Number.isInteger(n)) return;
+			maxChars = n;
+		} else if (arg.startsWith("--")) return;
+		else if (file === void 0) file = arg;
+		else return;
+	}
+	return {
+		file,
+		format,
+		maxChars
+	};
+}
+async function readHtml(file, stream) {
+	if (file !== void 0) return readHtmlFile(file);
+	const chunks = [];
+	for await (const chunk of stream) if (typeof chunk === "string") chunks.push(chunk);
+	else chunks.push(Buffer.from(chunk).toString("utf8"));
+	return chunks.join("");
+}
+function payloadText(result) {
+	const first = result.content.at(0);
+	return first !== void 0 && "text" in first ? first.text : "";
+}
+async function runCli(argv) {
+	if (argv[0] !== "extract") {
+		process.stderr.write(`${USAGE}\n`);
+		return 2;
+	}
+	const parsed = parseArgs(argv);
+	if (parsed === void 0) {
+		process.stderr.write(`${USAGE}\n`);
+		return 2;
+	}
+	try {
+		const html = await readHtml(parsed.file, process.stdin);
+		const pipelineFormat = parsed.format === "html" ? "html" : "markdown";
+		const result = extractArticleFromHtml({
+			html,
+			format: pipelineFormat,
+			...parsed.maxChars !== void 0 ? { maxChars: parsed.maxChars } : {}
+		});
+		if (result.isError) {
+			process.stderr.write(`${payloadText(result)}\n`);
+			return 1;
+		}
+		if (parsed.format === "json") process.stdout.write(`${JSON.stringify(result.structuredContent, null, 2)}\n`);
+		else process.stdout.write(`${payloadText(result)}\n`);
+		return 0;
+	} catch (err) {
+		process.stderr.write(`${err instanceof Error ? err.message : String(err)}\n`);
+		return 1;
+	}
+}
+//#endregion
+export { runCli };
+
+//# sourceMappingURL=cli-BwKCixh6.js.map

package/dist/assets/cli-BwKCixh6.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"cli-BwKCixh6.js","names":[],"sources":["../../src/cli.ts"],"sourcesContent":["import { extractArticleFromHtml } from './tools/extract.js';\nimport { readHtmlFile } from './tools/html-source.js';\n\nimport type { CallToolResult } from '@modelcontextprotocol/sdk/types.js';\nimport type { Readable } from 'node:stream';\n\ntype CliFormat = 'html' | 'json' | 'md';\n\nexport interface ParsedArgs {\n  readonly file: string | undefined;\n  readonly format: CliFormat;\n  readonly maxChars: number | undefined;\n}\n\nconst USAGE =\n  'Usage: readability-mcp extract [file.html] [--format md|json|html] [--max-chars N]';\n\nconst FORMATS: readonly CliFormat[] = ['html', 'json', 'md'];\n\nfunction isCliFormat(value: string | undefined): value is CliFormat {\n  return value !== undefined && (FORMATS as readonly string[]).includes(value);\n}\n\n// `extract` is consumed by the caller; everything after it is parsed here.\n// Flag values are read with `.at()` (not `[]`) because the peek may advance\n// past the end for a trailing flag with no value; `.at()` surfaces that as\n// undefined where bracket indexing would not (noUncheckedIndexedAccess is off).\nexport function parseArgs(argv: readonly string[]): ParsedArgs | undefined {\n  let file: string | undefined;\n  let format: CliFormat = 'md';\n  let maxChars: number | undefined;\n\n  const rest = argv.slice(1);\n  for (let i = 0; i < rest.length; i++) {\n    const arg = rest[i];\n    if (arg === '--format') {\n      const value = rest.at(++i);\n      if (!isCliFormat(value)) {\n        return undefined;\n      }\n      format = value;\n    } else if (arg === '--max-chars') {\n      const value = rest.at(++i);\n      if (value === undefined) {\n        return undefined;\n      }\n      const n = Number(value);\n      if (!Number.isInteger(n)) {\n        return undefined;\n      }\n      maxChars = n;\n    } else if (arg.startsWith('--')) {\n      return undefined;\n    } else if (file === undefined) {\n      file = arg;\n    } else {\n      return undefined;\n    }\n  }\n\n  return { file, format, maxChars };\n}\n\n// The stream is injected rather than reading process.stdin directly so the\n// path is testable. Chunks may be Buffer (process.stdin) or string\n// (Readable.from), so both are handled.\nexport async function readHtml(\n  file: string | undefined,\n  stream: Readable,\n): Promise<string> {\n  if (file !== undefined) {\n    return readHtmlFile(file);\n  }\n  const chunks: string[] = [];\n  for await (const chunk of stream) {\n    if (typeof chunk === 'string') {\n      chunks.push(chunk);\n    } else {\n      chunks.push(Buffer.from(chunk as Uint8Array).toString('utf8'));\n    }\n  }\n  return chunks.join('');\n}\n\nfunction payloadText(result: CallToolResult): string {\n  const first = result.content.at(0);\n  return first !== undefined && 'text' in first ? first.text : '';\n}\n\nexport async function runCli(argv: readonly string[]): Promise<number> {\n  if (argv[0] !== 'extract') {\n    process.stderr.write(`${USAGE}\\n`);\n    return 2;\n  }\n\n  const parsed = parseArgs(argv);\n  if (parsed === undefined) {\n    process.stderr.write(`${USAGE}\\n`);\n    return 2;\n  }\n\n  try {\n    const html = await readHtml(parsed.file, process.stdin);\n    // json reuses the markdown pipeline; the structured object is serialized below.\n    const pipelineFormat = parsed.format === 'html' ? 'html' : 'markdown';\n    const result = extractArticleFromHtml({\n      html,\n      format: pipelineFormat,\n      ...(parsed.maxChars !== undefined ? { maxChars: parsed.maxChars } : {}),\n    });\n\n    if (result.isError) {\n      process.stderr.write(`${payloadText(result)}\\n`);\n      return 1;\n    }\n\n    if (parsed.format === 'json') {\n      process.stdout.write(\n        `${JSON.stringify(result.structuredContent, null, 2)}\\n`,\n      );\n    } else {\n      process.stdout.write(`${payloadText(result)}\\n`);\n    }\n    return 0;\n  } catch (err) {\n    process.stderr.write(\n      `${err instanceof Error ? err.message : String(err)}\\n`,\n    );\n    return 1;\n  }\n}\n"],"mappings":";;AAcA,IAAM,QACJ;AAEF,IAAM,UAAgC;CAAC;CAAQ;CAAQ;AAAI;AAE3D,SAAS,YAAY,OAA+C;CAClE,OAAO,UAAU,KAAA,KAAc,QAA8B,SAAS,KAAK;AAC7E;AAMA,SAAgB,UAAU,MAAiD;CACzE,IAAI;CACJ,IAAI,SAAoB;CACxB,IAAI;CAEJ,MAAM,OAAO,KAAK,MAAM,CAAC;CACzB,KAAK,IAAI,IAAI,GAAG,IAAI,KAAK,QAAQ,KAAK;EACpC,MAAM,MAAM,KAAK;EACjB,IAAI,QAAQ,YAAY;GACtB,MAAM,QAAQ,KAAK,GAAG,EAAE,CAAC;GACzB,IAAI,CAAC,YAAY,KAAK,GACpB;GAEF,SAAS;EACX,OAAO,IAAI,QAAQ,eAAe;GAChC,MAAM,QAAQ,KAAK,GAAG,EAAE,CAAC;GACzB,IAAI,UAAU,KAAA,GACZ;GAEF,MAAM,IAAI,OAAO,KAAK;GACtB,IAAI,CAAC,OAAO,UAAU,CAAC,GACrB;GAEF,WAAW;EACb,OAAO,IAAI,IAAI,WAAW,IAAI,GAC5B;OACK,IAAI,SAAS,KAAA,GAClB,OAAO;OAEP;CAEJ;CAEA,OAAO;EAAE;EAAM;EAAQ;CAAS;AAClC;AAKA,eAAsB,SACpB,MACA,QACiB;CACjB,IAAI,SAAS,KAAA,GACX,OAAO,aAAa,IAAI;CAE1B,MAAM,SAAmB,CAAC;CAC1B,WAAW,MAAM,SAAS,QACxB,IAAI,OAAO,UAAU,UACnB,OAAO,KAAK,KAAK;MAEjB,OAAO,KAAK,OAAO,KAAK,KAAmB,CAAC,CAAC,SAAS,MAAM,CAAC;CAGjE,OAAO,OAAO,KAAK,EAAE;AACvB;AAEA,SAAS,YAAY,QAAgC;CACnD,MAAM,QAAQ,OAAO,QAAQ,GAAG,CAAC;CACjC,OAAO,UAAU,KAAA,KAAa,UAAU,QAAQ,MAAM,OAAO;AAC/D;AAEA,eAAsB,OAAO,MAA0C;CACrE,IAAI,KAAK,OAAO,WAAW;EACzB,QAAQ,OAAO,MAAM,GAAG,MAAM,GAAG;EACjC,OAAO;CACT;CAEA,MAAM,SAAS,UAAU,IAAI;CAC7B,IAAI,WAAW,KAAA,GAAW;EACxB,QAAQ,OAAO,MAAM,GAAG,MAAM,GAAG;EACjC,OAAO;CACT;CAEA,IAAI;EACF,MAAM,OAAO,MAAM,SAAS,OAAO,MAAM,QAAQ,KAAK;EAEtD,MAAM,iBAAiB,OAAO,WAAW,SAAS,SAAS;EAC3D,MAAM,SAAS,uBAAuB;GACpC;GACA,QAAQ;GACR,GAAI,OAAO,aAAa,KAAA,IAAY,EAAE,UAAU,OAAO,SAAS,IAAI,CAAC;EACvE,CAAC;EAED,IAAI,OAAO,SAAS;GAClB,QAAQ,OAAO,MAAM,GAAG,YAAY,MAAM,EAAE,GAAG;GAC/C,OAAO;EACT;EAEA,IAAI,OAAO,WAAW,QACpB,QAAQ,OAAO,MACb,GAAG,KAAK,UAAU,OAAO,mBAAmB,MAAM,CAAC,EAAE,GACvD;OAEA,QAAQ,OAAO,MAAM,GAAG,YAAY,MAAM,EAAE,GAAG;EAEjD,OAAO;CACT,SAAS,KAAK;EACZ,QAAQ,OAAO,MACb,GAAG,eAAe,QAAQ,IAAI,UAAU,OAAO,GAAG,EAAE,GACtD;EACA,OAAO;CACT;AACF"}