@j0hanz/superfetch 2.0.0 → 2.1.0

package/README.md

@@ -1,8 +1,10 @@
 # superFetch MCP Server
 
+<!-- markdownlint-disable MD033 -->
+
 <img src="docs/logo.png" alt="SuperFetch MCP Logo" width="200">
 
-[![npm version](https://img.shields.io/npm/v/@j0hanz/superfetch.svg)](https://www.npmjs.com/package/@j0hanz/superfetch) [![Node.js](https://img.shields.io/badge/Node.js-%3E=20.12-339933?logo=nodedotjs&logoColor=white)](https://nodejs.org/) [![TypeScript](https://img.shields.io/badge/TypeScript-5.9-3178C6?logo=typescript&logoColor=white)](https://www.typescriptlang.org/)
+[![npm version](https://img.shields.io/npm/v/@j0hanz/superfetch.svg)](https://www.npmjs.com/package/@j0hanz/superfetch) [![Node.js](https://img.shields.io/badge/Node.js-%3E=20.18.1-339933?logo=nodedotjs&logoColor=white)](https://nodejs.org/) [![TypeScript](https://img.shields.io/badge/TypeScript-5.9-3178C6?logo=typescript&logoColor=white)](https://www.typescriptlang.org/)
 
 ## One-Click Install
 
@@ -12,6 +14,17 @@
 
 A [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) server that fetches web pages, extracts readable content with Mozilla Readability, and returns AI-friendly Markdown.
 
+Built for AI workflows that need _clean text_, _stable metadata_, and _safe-by-default fetching_.
+
+**You get, in one tool call:**
+
+- **Readable Markdown** (HTML → Readability → Markdown)
+- **Metadata frontmatter** for HTML (title, source, author, description, fetchedAt)
+- **Raw markdown passthrough** (GitHub/GitLab/Bitbucket/Gist URLs auto-rewritten to raw when possible)
+- **Cache + resources** for large pages (MCP `superfetch://cache/...` resources and optional download endpoint in HTTP mode)
+
+**Great for:** docs ingestion, RAG pipelines, research agents, changelog/news summarization, and “fetch this URL and cite it” workflows.
+
 [Quick Start](#quick-start) | [Tool](#available-tools) | [Resources](#resources) | [Configuration](#configuration) | [Security](#security) | [Development](#development)
 
 > **Published to [MCP Registry](https://registry.modelcontextprotocol.io/)** - Search for `io.github.j0hanz/superfetch`
@@ -23,21 +36,39 @@ A [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) server that f
 
 ## Features
 
-| Feature              | Description                                                                           |
-| -------------------- | ------------------------------------------------------------------------------------- |
-| Smart extraction     | Mozilla Readability with quality gates to strip boilerplate when it improves results  |
-| Clean Markdown       | Markdown output with optional YAML frontmatter (title + source)                       |
-| Raw content handling | Preserves raw markdown/text and rewrites GitHub/GitLab/Bitbucket blob URLs to raw     |
-| Built-in caching     | In-memory cache with TTL, max keys, and resource subscriptions                        |
-| Resilient fetching   | Redirect handling with validation, timeouts, and response size limits                 |
-| Security first       | URL validation plus SSRF/DNS/IP blocklists                                            |
-| HTTP mode            | Static token or OAuth auth, session management, rate limiting, host/origin validation |
+- **Cleaner outputs for LLMs**: Readability extraction with a quality gate to avoid making pages worse
+- **Markdown that’s easy to consume**: YAML frontmatter for HTML + consistent Markdown formatting
+- **Handles “raw content” sources**: preserves markdown/text; rewrites GitHub/GitLab/Bitbucket/Gist URLs to raw
+- **Works for both local and hosted setups**:
+  - **Stdio mode**: best for MCP clients (VS Code / Claude Desktop / Cursor)
+  - **HTTP mode**: best for self-hosting (auth, sessions, rate limiting, Host/Origin validation)
+- **Fast and resilient**: redirect validation, timeouts, and response size limits
+- **Security-first defaults**: URL validation + SSRF/DNS/IP blocklists (blocks private ranges and cloud metadata endpoints)
+
+If you’re comparing “just call `fetch()`” vs superFetch: superFetch focuses on _readability_, _consistent structure_, and _safe URL access_ for agent environments.
+
+## What it is (and isn’t)
+
+- **It is** a “URL → clean Markdown” tool designed for agent/RAG workflows.
+- **It isn’t** a headless browser: pages that require heavy client-side rendering may not extract well.
+- **It isn’t** a crawler: it fetches a single URL per call (your agent can decide what to fetch next).
+- **It’s opinionated on safety**: blocked private IP ranges and metadata endpoints are a feature, not a bug.
 
 ---
 
 ## Quick Start
 
-Add superFetch to your MCP client configuration - no installation required.
+Recommended: use **stdio mode** with your MCP client (no HTTP server).
+
+### Try it in 60 seconds
+
+1. Add the MCP server config (below)
+2. Restart your MCP client
+3. Call the `fetch-url` tool with any public URL
+
+### What the tool returns
+
+You’ll get `structuredContent` with `url`, `resolvedUrl`, optional `title`, and `markdown` (plus a `superfetch://cache/...` resource link when cache is enabled and content is large).
 
 ### Claude Desktop
 
@@ -74,6 +105,23 @@ Add to `.vscode/mcp.json` in your workspace:
 Add environment variables in your MCP client config under `env`.
 See [Configuration](#configuration) or `CONFIGURATION.md` for all available options and presets.
 
+### Example output (trimmed)
+
+```json
+{
+  "url": "https://example.com/docs",
+  "inputUrl": "https://example.com/docs",
+  "resolvedUrl": "https://example.com/docs",
+  "title": "Documentation",
+  "markdown": "---\ntitle: Documentation\nsource: https://example.com/docs\nfetchedAt: 2026-01-12T12:00:00.000Z\n---\n\n# Getting Started\n\n..."
+}
+```
+
+> **Tip (Windows):** If you encounter issues, try: `cmd /c "npx -y @j0hanz/superfetch@latest --stdio"`
+
+<details>
+<summary><strong>Other clients (Cursor, Cline, Windsurf, Codex)</strong></summary>
+
 ### Cursor
 
 1. Open Cursor Settings
@@ -92,8 +140,6 @@ See [Configuration](#configuration) or `CONFIGURATION.md` for all available opti
 }
 ```
 
-> **Tip (Windows):** If you encounter issues, try: `cmd /c "npx -y @j0hanz/superfetch@latest --stdio"`
-
 <details>
 <summary><strong>Codex IDE</strong></summary>
 
@@ -188,6 +234,35 @@ code %APPDATA%\Claude\claude_desktop_config.json
 
 </details>
 
+</details>
+
+---
+
+## Use cases
+
+### 1) Turn a docs page into “LLM-ready” Markdown
+
+- Call `fetch-url` with the docs URL
+- Feed the returned `markdown` into your summarizer / chunker
+- Use the YAML frontmatter fields (especially `source`) for citations
+
+### 2) Fetch a GitHub/GitLab/Bitbucket file as raw markdown
+
+- Pass the normal “web UI” URL to `fetch-url`
+- superFetch will rewrite it to the raw content URL when possible
+- This avoids navigation UI and reduces boilerplate
+
+### 3) Large pages: keep responses stable with cache resources
+
+- When content is large, the tool can include a `superfetch://cache/...` resource link
+- In MCP clients that support resources, you can read the full content via the resource URI
+- In HTTP mode, you can also download cached content via `/mcp/downloads/:namespace/:hash` when cache is enabled
+
+### 4) Safe-by-default web access for agents
+
+- superFetch blocks private IP ranges and common cloud metadata endpoints
+- If your agent needs internal access, this is intentionally not supported by default (see Security)
+
 ---
 
 ## Installation (Alternative)
@@ -227,7 +302,7 @@ node dist/index.js --stdio
 <details>
 <summary><strong>HTTP Mode</strong> (default)</summary>
 
-HTTP mode requires authentication. By default it binds to `127.0.0.1`. To listen on all interfaces, set `HOST=0.0.0.0` or `HOST=::` and configure OAuth (remote bindings require OAuth). Other non-loopback `HOST` values are rejected.
+HTTP mode requires authentication. By default it binds to `127.0.0.1`. Non-loopback `HOST` values require `ALLOW_REMOTE=true`. To listen on all interfaces, set `HOST=0.0.0.0` or `HOST=::`, set `ALLOW_REMOTE=true`, and configure OAuth (remote bindings require OAuth).
 
 ```bash
 API_KEY=supersecret npx -y @j0hanz/superfetch@latest
@@ -243,13 +318,15 @@ npx -y @j0hanz/superfetch@latest
 
 For multiple static tokens, set `ACCESS_TOKENS` (comma/space separated).
 
-Endpoints (auth required via `Authorization: Bearer <token>`; in static token mode, `X-API-Key` is also accepted):
+Auth is required for `/mcp` and `/mcp/downloads` via `Authorization: Bearer <token>` (static mode also accepts `X-API-Key`).
 
-- `GET /health`
-- `POST /mcp`
-- `GET /mcp` (SSE stream)
-- `DELETE /mcp`
-- `GET /mcp/downloads/:namespace/:hash`
+Endpoints:
+
+- `GET /health` (no auth; returns status, name, version, uptime)
+- `POST /mcp` (auth required)
+- `GET /mcp` (auth required; SSE stream; requires `Accept: text/event-stream`)
+- `DELETE /mcp` (auth required)
+- `GET /mcp/downloads/:namespace/:hash` (auth required)
 
 Sessions are managed via the `mcp-session-id` header (see [HTTP Mode Details](#http-mode-details)).
 
@@ -261,19 +338,20 @@ Sessions are managed via the `mcp-session-id` header (see [HTTP Mode Details](#h
 
 ### Tool Response Notes
 
-The tool returns `structuredContent` with `url`, optional `title`, and `markdown` when inline content is available. On errors, `error` is included instead of content.
+The tool returns `structuredContent` with `url`, `inputUrl`, `resolvedUrl`, optional `title`, and `markdown` when inline content is available. `resolvedUrl` may differ from `inputUrl` when the URL is rewritten to raw content (GitHub/GitLab/Bitbucket/Gist). On errors, `error` is included instead of content.
 
 The response includes:
 
 - a `text` block containing JSON of `structuredContent`
-- a `resource` block embedding markdown when inline content is available (always in stdio mode)
-- when content exceeds the inline limit and cache is enabled, a `resource_link` block pointing to `superfetch://cache/...` (inline markdown may be omitted)
+- a `resource` block embedding markdown when inline content is available (always for successful stdio responses)
+- when content exceeds the inline limit and cache is enabled, a `resource_link` block pointing to `superfetch://cache/...` (stdio mode still embeds full markdown; HTTP mode omits embedded markdown)
+- error responses set `isError: true` and return `structuredContent` with `error` and `url`
 
 ---
 
 ### `fetch-url`
 
-Fetches a webpage and converts it to clean Markdown format with optional frontmatter.
+Fetches a webpage and converts it to clean Markdown format with YAML frontmatter for HTML (raw markdown is preserved).
 
 | Parameter | Type   | Default  | Description  |
 | --------- | ------ | -------- | ------------ |
@@ -284,6 +362,8 @@ Fetches a webpage and converts it to clean Markdown format with optional frontma
 ```json
 {
   "url": "https://example.com/docs",
+  "inputUrl": "https://example.com/docs",
+  "resolvedUrl": "https://example.com/docs",
   "title": "Documentation",
   "markdown": "---\ntitle: Documentation\n---\n\n# Getting Started\n\nWelcome..."
 }
@@ -303,8 +383,8 @@ Fetches a webpage and converts it to clean Markdown format with optional frontma
 ### Large Content Handling
 
 - Inline markdown is capped at 20,000 characters (`maxInlineContentChars`).
-- **Stdio mode:** full markdown is embedded as a `resource` block.
-- **HTTP mode:** if content exceeds the inline limit and cache is enabled, the response includes a `resource_link` to `superfetch://cache/...` (no embedded markdown). If cache is disabled, the inline markdown is truncated with `...[truncated]`.
+- **Stdio mode:** full markdown is embedded as a `resource` block; if cache is enabled and content exceeds the inline limit, a `resource_link` is still included.
+- **HTTP mode:** if content exceeds the inline limit and cache is enabled, the response includes a `resource_link` to `superfetch://cache/...` and omits embedded markdown. If cache is disabled, the inline markdown is truncated with `...[truncated]`.
 - Upstream fetch size is capped at 10 MB of HTML; larger responses fail.
 
 ---
@@ -364,23 +444,26 @@ Set environment variables in your MCP client `env` or in the shell before starti
 
 ### Core Server Settings
 
-| Variable        | Default              | Description                                                   |
-| --------------- | -------------------- | ------------------------------------------------------------- |
-| `HOST`          | `127.0.0.1`          | HTTP bind address                                             |
-| `PORT`          | `3000`               | HTTP server port (1024-65535)                                 |
-| `USER_AGENT`    | `superFetch-MCP/2.0` | User-Agent header for outgoing requests                       |
-| `CACHE_ENABLED` | `true`               | Enable response caching                                       |
-| `CACHE_TTL`     | `3600`               | Cache TTL in seconds (60-86400)                               |
-| `LOG_LEVEL`     | `info`               | `debug`, `info`, `warn`, `error`                              |
-| `ALLOWED_HOSTS` | (empty)              | Additional allowed Host/Origin values (comma/space separated) |
+| Variable        | Default              | Description                                                                       |
+| --------------- | -------------------- | --------------------------------------------------------------------------------- |
+| `HOST`          | `127.0.0.1`          | HTTP bind address                                                                 |
+| `PORT`          | `3000`               | HTTP server port (1024-65535)                                                     |
+| `USER_AGENT`    | `superFetch-MCP/2.0` | User-Agent header for outgoing requests                                           |
+| `CACHE_ENABLED` | `true`               | Enable response caching                                                           |
+| `CACHE_TTL`     | `3600`               | Cache TTL in seconds (60-86400)                                                   |
+| `LOG_LEVEL`     | `info`               | Logging level. Only `debug` enables verbose logs; other values behave like `info` |
+| `ALLOW_REMOTE`  | `false`              | Allow binding to non-loopback hosts (OAuth required)                              |
+| `ALLOWED_HOSTS` | (empty)              | Additional allowed Host/Origin values (comma/space separated)                     |
+
+For HTTP server tuning (`SERVER_HEADERS_TIMEOUT_MS`, `SERVER_REQUEST_TIMEOUT_MS`, `SERVER_KEEP_ALIVE_TIMEOUT_MS`, `SERVER_SHUTDOWN_CLOSE_IDLE`, `SERVER_SHUTDOWN_CLOSE_ALL`), see `CONFIGURATION.md`.
 
 ### Auth (HTTP Mode)
 
-| Variable        | Default | Description                                                  |
-| --------------- | ------- | ------------------------------------------------------------ |
-| `AUTH_MODE`     | auto    | `static` or `oauth`. Auto-selects OAuth if any OAUTH URL set |
-| `ACCESS_TOKENS` | (empty) | Comma/space-separated static bearer tokens                   |
-| `API_KEY`       | (empty) | Adds a static bearer token and enables `X-API-Key` header    |
+| Variable        | Default | Description                                                                                                                              |
+| --------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
+| `AUTH_MODE`     | auto    | `static` or `oauth`. Auto-selects OAuth if OAUTH_ISSUER_URL, OAUTH_AUTHORIZATION_URL, OAUTH_TOKEN_URL, or OAUTH_INTROSPECTION_URL is set |
+| `ACCESS_TOKENS` | (empty) | Comma/space-separated static bearer tokens                                                                                               |
+| `API_KEY`       | (empty) | Adds a static bearer token and enables `X-API-Key` header                                                                                |
 
 Static mode requires at least one token (`ACCESS_TOKENS` or `API_KEY`).
 
@@ -415,6 +498,7 @@ Optional:
 - Inline markdown limit: 20,000 characters
 - Cache max entries: 100
 - Session TTL: 30 minutes
+- Session init timeout: 10 seconds
 - Max sessions: 200
 - Rate limit: 100 req/min per IP (60s window)
 
@@ -430,10 +514,15 @@ HTTP mode uses the MCP Streamable HTTP transport. The workflow is:
 2. The server returns `mcp-session-id` in the response headers.
 3. Use that header for subsequent `POST /mcp`, `GET /mcp`, and `DELETE /mcp` requests.
 
-If the `mcp-protocol-version` header is missing, the server defaults it to `2025-03-26`. Supported versions are `2025-03-26` and `2025-11-25`.
+If the `mcp-protocol-version` header is missing, the server assumes `2025-03-26` and rejects the request because only `2025-11-25` is supported. Clients must send `mcp-protocol-version: 2025-11-25`.
 
 `GET /mcp` and `DELETE /mcp` require `mcp-session-id`. `POST /mcp` without an `initialize` request will return 400.
 
+Additional HTTP transport notes:
+
+- `GET /mcp` requires `Accept: text/event-stream` (otherwise 406).
+- JSON-RPC batch requests are not supported (400).
+
 If the server reaches its session cap (200), it evicts the oldest session when possible; otherwise it returns a 503.
 
 Host and Origin headers are always validated. Allowed values include loopback hosts, the configured `HOST` (if not a wildcard), and any entries in `ALLOWED_HOSTS`. When binding to `0.0.0.0` or `::`, set `ALLOWED_HOSTS` to the hostnames clients will send.
@@ -484,12 +573,14 @@ Rate limiting applies to `/mcp` and `/mcp/downloads` (100 req/min per IP, 60s wi
 | `npm run build`         | Compile TypeScript                   |
 | `npm start`             | Production server                    |
 | `npm run lint`          | Run ESLint                           |
+| `npm run lint:fix`      | Auto-fix lint issues                 |
 | `npm run type-check`    | TypeScript type checking             |
 | `npm run format`        | Format with Prettier                 |
 | `npm test`              | Run Node test runner (builds dist)   |
 | `npm run test:coverage` | Run tests with experimental coverage |
 | `npm run knip`          | Find unused exports/dependencies     |
 | `npm run knip:fix`      | Auto-fix unused code                 |
+| `npm run inspector`     | Launch MCP Inspector                 |
 
 > **Note:** Tests run via `node --test` with `--experimental-transform-types` to execute `.ts` test files. Node will emit an experimental warning.
 
@@ -497,13 +588,13 @@ Rate limiting applies to `/mcp` and `/mcp/downloads` (100 req/min per IP, 60s wi
 
 | Category           | Technology                        |
 | ------------------ | --------------------------------- |
-| Runtime            | Node.js >=20.12                   |
+| Runtime            | Node.js >=20.18.1                 |
 | Language           | TypeScript 5.9                    |
-| MCP SDK            | @modelcontextprotocol/sdk ^1.25.1 |
+| MCP SDK            | @modelcontextprotocol/sdk ^1.25.2 |
 | Content Extraction | @mozilla/readability ^0.6.0       |
-| HTML Parsing       | LinkeDOM ^0.18.12                 |
-| Markdown           | Turndown ^7.2.2                   |
-| HTTP               | Express ^5.2.1, undici ^6.23.0    |
+| HTML Parsing       | linkedom ^0.18.12                 |
+| Markdown           | node-html-markdown ^2.0.0         |
+| HTTP               | Express ^5.2.1, undici ^7.18.2    |
 | Validation         | Zod ^4.3.5                        |
 
 ---
@@ -519,3 +610,5 @@ Rate limiting applies to `/mcp` and `/mcp/downloads` (100 req/min per IP, 60s wi
 7. Open a Pull Request
 
 For examples of other MCP servers, see: [github.com/modelcontextprotocol/servers](https://github.com/modelcontextprotocol/servers)
+
+<!-- markdownlint-enable MD033 -->

package/dist/cache.d.ts

@@ -0,0 +1,42 @@
+import type { Express } from 'express';
+import type { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
+export interface CacheEntry {
+    url: string;
+    title?: string;
+    content: string;
+    fetchedAt: string;
+    expiresAt: string;
+}
+export interface CachedPayload {
+    content?: string;
+    markdown?: string;
+    title?: string;
+}
+export declare function parseCachedPayload(raw: string): CachedPayload | null;
+export declare function resolveCachedPayloadContent(payload: CachedPayload): string | null;
+export interface CacheKeyParts {
+    namespace: string;
+    urlHash: string;
+}
+export declare function createCacheKey(namespace: string, url: string, vary?: Record<string, unknown> | string): string | null;
+export declare function parseCacheKey(cacheKey: string): CacheKeyParts | null;
+export declare function toResourceUri(cacheKey: string): string | null;
+interface CacheUpdateEvent {
+    cacheKey: string;
+    namespace: string;
+    urlHash: string;
+}
+type CacheUpdateListener = (event: CacheUpdateEvent) => void;
+export declare function onCacheUpdate(listener: CacheUpdateListener): () => void;
+interface CacheEntryMetadata {
+    url: string;
+    title?: string;
+}
+export declare function get(cacheKey: string | null): CacheEntry | undefined;
+export declare function set(cacheKey: string | null, content: string, metadata: CacheEntryMetadata): void;
+export declare function keys(): readonly string[];
+export declare function isEnabled(): boolean;
+export declare function registerCachedContentResource(server: McpServer): void;
+export declare function generateSafeFilename(url: string, title?: string, hashFallback?: string, extension?: string): string;
+export declare function registerDownloadRoutes(app: Express): void;
+export {};