@j0hanz/superfetch 2.5.2 → 2.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +356 -223
- package/dist/assets/logo.svg +24837 -24835
- package/dist/cache.d.ts +28 -20
- package/dist/cache.js +292 -514
- package/dist/config.d.ts +41 -7
- package/dist/config.js +298 -148
- package/dist/crypto.js +25 -12
- package/dist/dom-noise-removal.js +379 -421
- package/dist/errors.d.ts +2 -2
- package/dist/errors.js +25 -8
- package/dist/fetch.d.ts +18 -16
- package/dist/fetch.js +1132 -526
- package/dist/host-normalization.js +40 -10
- package/dist/http-native.js +628 -287
- package/dist/index.js +67 -7
- package/dist/instructions.md +44 -30
- package/dist/ip-blocklist.d.ts +8 -0
- package/dist/ip-blocklist.js +65 -0
- package/dist/json.js +14 -9
- package/dist/language-detection.d.ts +2 -11
- package/dist/language-detection.js +289 -280
- package/dist/markdown-cleanup.d.ts +0 -1
- package/dist/markdown-cleanup.js +391 -429
- package/dist/mcp-validator.js +4 -2
- package/dist/mcp.js +184 -135
- package/dist/observability.js +89 -21
- package/dist/resources.js +16 -6
- package/dist/server-tuning.d.ts +2 -0
- package/dist/server-tuning.js +25 -23
- package/dist/session.d.ts +1 -0
- package/dist/session.js +41 -33
- package/dist/tasks.d.ts +2 -0
- package/dist/tasks.js +91 -9
- package/dist/timer-utils.d.ts +5 -0
- package/dist/timer-utils.js +20 -0
- package/dist/tools.d.ts +28 -5
- package/dist/tools.js +317 -183
- package/dist/transform-types.d.ts +5 -1
- package/dist/transform.d.ts +3 -2
- package/dist/transform.js +1138 -421
- package/dist/type-guards.d.ts +1 -0
- package/dist/type-guards.js +7 -0
- package/dist/workers/transform-child.d.ts +1 -0
- package/dist/workers/transform-child.js +118 -0
- package/dist/workers/transform-worker.js +87 -78
- package/package.json +21 -13
package/README.md
CHANGED
|
@@ -4,18 +4,15 @@
|
|
|
4
4
|
|
|
5
5
|
<img src="assets/logo.svg" alt="SuperFetch MCP Logo" width="300">
|
|
6
6
|
|
|
7
|
-
[](https://www.npmjs.com/package/@j0hanz/superfetch) [](https://opensource.org/licenses/MIT) [](https://nodejs.org) [](https://www.typescriptlang.org) [](https://modelcontextprotocol.io)
|
|
8
8
|
|
|
9
|
-
[](https://insiders.vscode.dev/redirect?url=vscode%3Amcp%2Finstall%3F%7B%22name%22%3A%22superfetch%22%2C%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22%40j0hanz%2Fsuperfetch%40latest%22%2C%22--stdio%22%5D%7D) [](https://insiders.vscode.dev/redirect?url=vscode-insiders%3Amcp%2Finstall%3F%7B%22name%22%3A%22superfetch%22%2C%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22%40j0hanz%2Fsuperfetch%40latest%22%2C%22--stdio%22%5D%7D) [](https://cursor.com/install-mcp?name=superfetch&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsIkBqMGhhbnovc3VwZXJmZXRjaEBsYXRlc3QiLCItLXN0ZGlvIl19)
|
|
10
10
|
|
|
11
|
-
Fetch and convert public web
|
|
11
|
+
Fetch and convert public web content to clean Markdown both readable by humans and optimized for LLM context.
|
|
12
12
|
|
|
13
13
|
## Overview
|
|
14
14
|
|
|
15
|
-
|
|
16
|
-
primary content, and converts it into clean Markdown. It runs either as a stdio
|
|
17
|
-
MCP server (for local clients) or as a Streamable HTTP MCP server with auth,
|
|
18
|
-
cache, and SSRF protections.
|
|
15
|
+
superFetch is a [Model Context Protocol](https://modelcontextprotocol.io) (MCP) server that fetches public web pages, extracts meaningful content using Mozilla's Readability algorithm, and converts the result into clean Markdown optimized for LLM context windows. It handles noise removal, caching, SSRF protection, async task execution, and supports both stdio and Streamable HTTP transports.
|
|
19
16
|
|
|
20
17
|
## Key Features
|
|
21
18
|
|
|
@@ -25,65 +22,101 @@ cache, and SSRF protections.
|
|
|
25
22
|
- Stdio or Streamable HTTP transport with session management.
|
|
26
23
|
- SSRF protections: blocked private IP ranges and internal hostnames.
|
|
27
24
|
|
|
25
|
+
> **Note:** Content extraction quality varies depending on the HTML structure and
|
|
26
|
+
> complexity of the source page. SuperFetch works best with standard article and
|
|
27
|
+
> documentation layouts. Always verify the fetched content to ensure it meets
|
|
28
|
+
> your expectations, as some pages may require manual adjustment or alternative
|
|
29
|
+
> approaches.
|
|
30
|
+
|
|
28
31
|
## Tech Stack
|
|
29
32
|
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
33
|
+
| Component | Technology |
|
|
34
|
+
| ------------------- | ----------------------------------- |
|
|
35
|
+
| Runtime | Node.js ≥ 24 |
|
|
36
|
+
| Language | TypeScript 5.9 |
|
|
37
|
+
| MCP SDK | `@modelcontextprotocol/sdk` ^1.26.0 |
|
|
38
|
+
| Content Extraction | `@mozilla/readability` ^0.6.0 |
|
|
39
|
+
| DOM Parsing | `linkedom` ^0.18.12 |
|
|
40
|
+
| Markdown Conversion | `node-html-markdown` ^2.0.0 |
|
|
41
|
+
| Schema Validation | `zod` ^4.3.6 |
|
|
42
|
+
| Package Manager | npm |
|
|
38
43
|
|
|
39
44
|
## Architecture
|
|
40
45
|
|
|
41
|
-
|
|
46
|
+
```text
|
|
47
|
+
URL → Validate → DNS Preflight → HTTP Fetch → Decompress
|
|
48
|
+
→ Truncate HTML → Readability Extract → Noise Removal
|
|
49
|
+
→ Markdown Convert → Cleanup Pipeline → Cache → Response
|
|
50
|
+
```
|
|
42
51
|
|
|
43
|
-
1.
|
|
44
|
-
2.
|
|
45
|
-
3.
|
|
46
|
-
4.
|
|
47
|
-
5.
|
|
48
|
-
6. Convert to Markdown, inject metadata, and return via MCP.
|
|
49
|
-
7. Cache the result and expose it as a resource or download.
|
|
52
|
+
1. **URL Validation** — Normalize, block private hosts, transform raw-content URLs (GitHub, GitLab, Bitbucket)
|
|
53
|
+
2. **Fetch** — HTTP request via `undici` with redirect following, DNS preflight SSRF checks, and size limits
|
|
54
|
+
3. **Transform** — Offloaded to worker threads: parse HTML with `linkedom`, extract with Readability, remove DOM noise, convert to Markdown
|
|
55
|
+
4. **Cleanup** — Multi-pass Markdown normalization (heading promotion, spacing, skip-link removal, TypeDoc comment stripping)
|
|
56
|
+
5. **Cache + Respond** — Store result, apply inline content limits, return structured content with optional resource links
|
|
50
57
|
|
|
51
58
|
## Repository Structure
|
|
52
59
|
|
|
53
60
|
```text
|
|
54
|
-
|
|
55
|
-
├── assets/
|
|
56
|
-
|
|
57
|
-
├──
|
|
58
|
-
│ ├──
|
|
59
|
-
│
|
|
60
|
-
|
|
61
|
-
│ ├──
|
|
62
|
-
│
|
|
63
|
-
|
|
64
|
-
├──
|
|
65
|
-
├──
|
|
61
|
+
superFetch/
|
|
62
|
+
├── assets/
|
|
63
|
+
│ └── logo.svg
|
|
64
|
+
├── scripts/
|
|
65
|
+
│ ├── tasks.mjs
|
|
66
|
+
│ └── validate-fetch.mjs
|
|
67
|
+
├── src/
|
|
68
|
+
│ ├── workers/
|
|
69
|
+
│ │ ├── transform-child.ts
|
|
70
|
+
│ │ └── transform-worker.ts
|
|
71
|
+
│ ├── cache.ts
|
|
72
|
+
│ ├── config.ts
|
|
73
|
+
│ ├── crypto.ts
|
|
74
|
+
│ ├── dom-noise-removal.ts
|
|
75
|
+
│ ├── errors.ts
|
|
76
|
+
│ ├── fetch.ts
|
|
77
|
+
│ ├── host-normalization.ts
|
|
78
|
+
│ ├── http-native.ts
|
|
79
|
+
│ ├── index.ts
|
|
80
|
+
│ ├── instructions.md
|
|
81
|
+
│ ├── ip-blocklist.ts
|
|
82
|
+
│ ├── json.ts
|
|
83
|
+
│ ├── language-detection.ts
|
|
84
|
+
│ ├── markdown-cleanup.ts
|
|
85
|
+
│ ├── mcp-validator.ts
|
|
86
|
+
│ ├── mcp.ts
|
|
87
|
+
│ ├── observability.ts
|
|
88
|
+
│ ├── resources.ts
|
|
89
|
+
│ ├── server-tuning.ts
|
|
90
|
+
│ ├── session.ts
|
|
91
|
+
│ ├── tasks.ts
|
|
92
|
+
│ ├── timer-utils.ts
|
|
93
|
+
│ ├── tools.ts
|
|
94
|
+
│ ├── transform-types.ts
|
|
95
|
+
│ ├── transform.ts
|
|
96
|
+
│ └── type-guards.ts
|
|
97
|
+
├── tests/
|
|
98
|
+
│ └── *.test.ts
|
|
66
99
|
├── package.json
|
|
67
|
-
|
|
100
|
+
├── tsconfig.json
|
|
101
|
+
└── AGENTS.md
|
|
68
102
|
```
|
|
69
103
|
|
|
70
104
|
## Requirements
|
|
71
105
|
|
|
72
|
-
- Node.js
|
|
73
|
-
- npm (uses package-lock.json)
|
|
106
|
+
- **Node.js** ≥ 24
|
|
74
107
|
|
|
75
|
-
## Quickstart
|
|
108
|
+
## Quickstart
|
|
76
109
|
|
|
77
110
|
```bash
|
|
78
111
|
npx -y @j0hanz/superfetch@latest --stdio
|
|
79
112
|
```
|
|
80
113
|
|
|
81
|
-
|
|
114
|
+
Add to your MCP client configuration:
|
|
82
115
|
|
|
83
116
|
```json
|
|
84
117
|
{
|
|
85
118
|
"mcpServers": {
|
|
86
|
-
"
|
|
119
|
+
"superfetch": {
|
|
87
120
|
"command": "npx",
|
|
88
121
|
"args": ["-y", "@j0hanz/superfetch@latest", "--stdio"]
|
|
89
122
|
}
|
|
@@ -93,20 +126,22 @@ Example MCP client configuration:
|
|
|
93
126
|
|
|
94
127
|
## Installation
|
|
95
128
|
|
|
96
|
-
### NPX (
|
|
129
|
+
### NPX (Recommended)
|
|
130
|
+
|
|
131
|
+
No installation required — runs directly:
|
|
97
132
|
|
|
98
133
|
```bash
|
|
99
134
|
npx -y @j0hanz/superfetch@latest --stdio
|
|
100
135
|
```
|
|
101
136
|
|
|
102
|
-
### Global
|
|
137
|
+
### Global Install
|
|
103
138
|
|
|
104
139
|
```bash
|
|
105
140
|
npm install -g @j0hanz/superfetch
|
|
106
141
|
superfetch --stdio
|
|
107
142
|
```
|
|
108
143
|
|
|
109
|
-
### From
|
|
144
|
+
### From Source
|
|
110
145
|
|
|
111
146
|
```bash
|
|
112
147
|
git clone https://github.com/j0hanz/super-fetch-mcp-server.git
|
|
@@ -118,105 +153,110 @@ node dist/index.js --stdio
|
|
|
118
153
|
|
|
119
154
|
## Configuration
|
|
120
155
|
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
|
136
|
-
|
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
|
141
|
-
|
|
|
142
|
-
| `
|
|
143
|
-
| `
|
|
144
|
-
| `
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
|
149
|
-
|
|
|
150
|
-
| `
|
|
151
|
-
| `
|
|
152
|
-
| `
|
|
153
|
-
| `
|
|
154
|
-
| `
|
|
155
|
-
| `
|
|
156
|
-
| `
|
|
157
|
-
|
|
158
|
-
`
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
|
165
|
-
|
|
|
166
|
-
| `
|
|
167
|
-
| `
|
|
168
|
-
| `
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
|
175
|
-
|
|
|
176
|
-
| `
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
|
185
|
-
|
|
|
186
|
-
| `
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
|
194
|
-
|
|
|
195
|
-
|
|
|
196
|
-
|
|
|
197
|
-
|
|
|
198
|
-
|
|
|
199
|
-
|
|
|
200
|
-
|
|
|
201
|
-
|
|
|
156
|
+
### Runtime Modes
|
|
157
|
+
|
|
158
|
+
| Flag | Description |
|
|
159
|
+
| ----------- | ------------------------------------------- |
|
|
160
|
+
| `--stdio` | Run in stdio mode (for desktop MCP clients) |
|
|
161
|
+
| `--help` | Show usage help |
|
|
162
|
+
| `--version` | Print server version |
|
|
163
|
+
|
|
164
|
+
When no `--stdio` flag is passed, the server starts in **HTTP mode** (Streamable HTTP on port 3000 by default).
|
|
165
|
+
|
|
166
|
+
### Environment Variables
|
|
167
|
+
|
|
168
|
+
#### Core Settings
|
|
169
|
+
|
|
170
|
+
| Variable | Default | Description |
|
|
171
|
+
| ------------------ | -------------------------- | ------------------------------------------------------ |
|
|
172
|
+
| `HOST` | `127.0.0.1` | HTTP server bind address |
|
|
173
|
+
| `PORT` | `3000` | HTTP server port (1024–65535) |
|
|
174
|
+
| `LOG_LEVEL` | `info` | Log level: `debug`, `info`, `warn`, `error` |
|
|
175
|
+
| `FETCH_TIMEOUT_MS` | `15000` | HTTP fetch timeout in ms (1000–60000) |
|
|
176
|
+
| `CACHE_ENABLED` | `true` | Enable/disable in-memory content cache |
|
|
177
|
+
| `USER_AGENT` | `superFetch-MCP/{version}` | Custom User-Agent header |
|
|
178
|
+
| `ALLOW_REMOTE` | `false` | Allow remote connections in HTTP mode |
|
|
179
|
+
| `ALLOWED_HOSTS` | _(empty)_ | Comma-separated hostnames allowed to bypass block list |
|
|
180
|
+
|
|
181
|
+
#### Authentication (HTTP Mode)
|
|
182
|
+
|
|
183
|
+
| Variable | Default | Description |
|
|
184
|
+
| ------------------------- | --------- | --------------------------------------- |
|
|
185
|
+
| `ACCESS_TOKENS` | _(empty)_ | Comma-separated static bearer tokens |
|
|
186
|
+
| `API_KEY` | _(empty)_ | Single API key (added to static tokens) |
|
|
187
|
+
| `OAUTH_ISSUER_URL` | _(empty)_ | OAuth issuer URL (enables OAuth mode) |
|
|
188
|
+
| `OAUTH_AUTHORIZATION_URL` | _(empty)_ | OAuth authorization endpoint |
|
|
189
|
+
| `OAUTH_TOKEN_URL` | _(empty)_ | OAuth token endpoint |
|
|
190
|
+
| `OAUTH_INTROSPECTION_URL` | _(empty)_ | OAuth token introspection endpoint |
|
|
191
|
+
| `OAUTH_REVOCATION_URL` | _(empty)_ | OAuth token revocation endpoint |
|
|
192
|
+
| `OAUTH_REGISTRATION_URL` | _(empty)_ | OAuth dynamic client registration |
|
|
193
|
+
| `OAUTH_REQUIRED_SCOPES` | _(empty)_ | Required OAuth scopes |
|
|
194
|
+
| `OAUTH_CLIENT_ID` | _(empty)_ | OAuth client ID |
|
|
195
|
+
| `OAUTH_CLIENT_SECRET` | _(empty)_ | OAuth client secret |
|
|
196
|
+
|
|
197
|
+
#### Transform & Workers
|
|
198
|
+
|
|
199
|
+
| Variable | Default | Description |
|
|
200
|
+
| ------------------------------------------ | --------- | ----------------------------------------- |
|
|
201
|
+
| `TRANSFORM_WORKER_MODE` | `threads` | Worker mode: `threads` or `process` |
|
|
202
|
+
| `TRANSFORM_WORKER_MAX_OLD_GENERATION_MB` | _(unset)_ | V8 old generation heap limit per worker |
|
|
203
|
+
| `TRANSFORM_WORKER_MAX_YOUNG_GENERATION_MB` | _(unset)_ | V8 young generation heap limit per worker |
|
|
204
|
+
| `TRANSFORM_WORKER_CODE_RANGE_MB` | _(unset)_ | V8 code range limit per worker |
|
|
205
|
+
| `TRANSFORM_WORKER_STACK_MB` | _(unset)_ | Stack size limit per worker |
|
|
206
|
+
|
|
207
|
+
#### Content Tuning
|
|
208
|
+
|
|
209
|
+
| Variable | Default | Description |
|
|
210
|
+
| ---------------------------------- | ----------------- | ------------------------------------------------ |
|
|
211
|
+
| `SUPERFETCH_EXTRA_NOISE_TOKENS` | _(empty)_ | Additional CSS class/id tokens for noise removal |
|
|
212
|
+
| `SUPERFETCH_EXTRA_NOISE_SELECTORS` | _(empty)_ | Additional CSS selectors for noise removal |
|
|
213
|
+
| `MARKDOWN_HEADING_KEYWORDS` | _(built-in list)_ | Keywords triggering heading promotion |
|
|
214
|
+
| `SUPERFETCH_LOCALE` | _(system)_ | Locale for content processing |
|
|
215
|
+
|
|
216
|
+
#### Server Tuning
|
|
217
|
+
|
|
218
|
+
| Variable | Default | Description |
|
|
219
|
+
| ---------------------------------- | --------------- | ---------------------------------------- |
|
|
220
|
+
| `SERVER_MAX_CONNECTIONS` | `0` (unlimited) | Maximum concurrent HTTP connections |
|
|
221
|
+
| `SERVER_BLOCK_PRIVATE_CONNECTIONS` | `false` | Block connections from private IP ranges |
|
|
222
|
+
|
|
223
|
+
### Hardcoded Defaults
|
|
224
|
+
|
|
225
|
+
| Setting | Value |
|
|
226
|
+
| ------------------------ | ------------------------------- |
|
|
227
|
+
| Max HTML size | 10 MB |
|
|
228
|
+
| Max inline content chars | 0 (unlimited) |
|
|
229
|
+
| Fetch timeout | 15 s |
|
|
230
|
+
| Transform timeout | 30 s |
|
|
231
|
+
| Tool timeout | Fetch + Transform + 5 s padding |
|
|
232
|
+
| Max redirects | 5 |
|
|
233
|
+
| Cache TTL | 86400 s (24 h) |
|
|
234
|
+
| Cache max keys | 100 |
|
|
235
|
+
| Rate limit | 100 requests / 60 s |
|
|
236
|
+
| Max sessions | 200 |
|
|
237
|
+
| Session TTL | 30 min |
|
|
238
|
+
| Max URL length | 2048 chars |
|
|
239
|
+
| Worker pool max scale | 4 |
|
|
202
240
|
|
|
203
241
|
## Usage
|
|
204
242
|
|
|
205
|
-
### Stdio
|
|
243
|
+
### Stdio Mode
|
|
206
244
|
|
|
207
245
|
```bash
|
|
208
|
-
|
|
246
|
+
superfetch --stdio
|
|
209
247
|
```
|
|
210
248
|
|
|
211
|
-
|
|
249
|
+
The server communicates via JSON-RPC over stdin/stdout. All MCP clients that support stdio transport can connect directly.
|
|
250
|
+
|
|
251
|
+
### HTTP Mode
|
|
212
252
|
|
|
213
253
|
```bash
|
|
214
|
-
|
|
254
|
+
superfetch
|
|
255
|
+
# or
|
|
256
|
+
PORT=8080 HOST=0.0.0.0 ALLOW_REMOTE=true superfetch
|
|
215
257
|
```
|
|
216
258
|
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
Remote bindings require `ALLOW_REMOTE=true` and OAuth configuration.
|
|
259
|
+
The server starts a Streamable HTTP endpoint at `/mcp`. Authenticate with bearer tokens via the `ACCESS_TOKENS` or `API_KEY` environment variables.
|
|
220
260
|
|
|
221
261
|
## MCP Surface
|
|
222
262
|
|
|
@@ -224,77 +264,130 @@ Remote bindings require `ALLOW_REMOTE=true` and OAuth configuration.
|
|
|
224
264
|
|
|
225
265
|
#### `fetch-url`
|
|
226
266
|
|
|
227
|
-
Fetches a webpage and converts it to clean Markdown.
|
|
267
|
+
Fetches a webpage and converts it to clean Markdown format optimized for LLM context.
|
|
268
|
+
|
|
269
|
+
**Useful for:**
|
|
270
|
+
|
|
271
|
+
- Reading documentation, blog posts, or articles
|
|
272
|
+
- Extracting main content while removing navigation and ads
|
|
273
|
+
- Caching content to speed up repeated queries
|
|
274
|
+
|
|
275
|
+
**Limitations:**
|
|
276
|
+
|
|
277
|
+
- Does not execute complex client-side JavaScript interactions
|
|
278
|
+
|
|
279
|
+
##### Parameters
|
|
280
|
+
|
|
281
|
+
| Parameter | Type | Required | Default | Description |
|
|
282
|
+
| ------------------ | -------------- | -------- | ------- | ------------------------------------------------------------------ |
|
|
283
|
+
| `url` | `string` (URL) | Yes | — | The URL of the webpage to fetch (http/https, max 2048 chars) |
|
|
284
|
+
| `skipNoiseRemoval` | `boolean` | No | `false` | Preserve navigation, footers, and other elements normally filtered |
|
|
285
|
+
| `forceRefresh` | `boolean` | No | `false` | Bypass cache and fetch fresh content |
|
|
228
286
|
|
|
229
|
-
|
|
287
|
+
##### Returns
|
|
230
288
|
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
289
|
+
```json
|
|
290
|
+
{
|
|
291
|
+
"url": "https://example.com",
|
|
292
|
+
"resolvedUrl": "https://example.com",
|
|
293
|
+
"inputUrl": "https://example.com",
|
|
294
|
+
"title": "Example Domain",
|
|
295
|
+
"markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
|
|
296
|
+
"truncated": false
|
|
297
|
+
}
|
|
298
|
+
```
|
|
234
299
|
|
|
235
|
-
|
|
300
|
+
| Field | Type | Description |
|
|
301
|
+
| ------------- | ---------- | ----------------------------------------------- |
|
|
302
|
+
| `url` | `string` | The canonical URL (pre-raw-transform) |
|
|
303
|
+
| `inputUrl` | `string` | The original URL provided by the caller |
|
|
304
|
+
| `resolvedUrl` | `string` | The normalized/transformed URL that was fetched |
|
|
305
|
+
| `title` | `string?` | Extracted page title |
|
|
306
|
+
| `markdown` | `string?` | Extracted content in Markdown format |
|
|
307
|
+
| `truncated` | `boolean?` | Whether inline markdown was truncated |
|
|
308
|
+
| `error` | `string?` | Error message if the request failed |
|
|
309
|
+
| `statusCode` | `number?` | HTTP status code for failed requests |
|
|
310
|
+
| `details` | `object?` | Additional error details |
|
|
236
311
|
|
|
237
|
-
|
|
238
|
-
- `inputUrl` (string, optional): original input URL
|
|
239
|
-
- `resolvedUrl` (string, optional): normalized or raw-content URL
|
|
240
|
-
- `title` (string, optional): page title
|
|
241
|
-
- `markdown` (string, optional): inline markdown (may be truncated)
|
|
242
|
-
- `error` (string, optional): error message on failure
|
|
312
|
+
##### Annotations
|
|
243
313
|
|
|
244
|
-
|
|
314
|
+
| Annotation | Value |
|
|
315
|
+
| ----------------- | ------- |
|
|
316
|
+
| `readOnlyHint` | `true` |
|
|
317
|
+
| `destructiveHint` | `false` |
|
|
318
|
+
| `idempotentHint` | `true` |
|
|
319
|
+
| `openWorldHint` | `true` |
|
|
245
320
|
|
|
246
|
-
|
|
247
|
-
- Client-side JavaScript is not executed.
|
|
321
|
+
##### Async Task Execution
|
|
248
322
|
|
|
249
|
-
|
|
323
|
+
The `fetch-url` tool supports optional async task execution. Include a `task` field in the tool call to run the fetch in the background:
|
|
324
|
+
|
|
325
|
+
```json
|
|
326
|
+
{
|
|
327
|
+
"method": "tools/call",
|
|
328
|
+
"params": {
|
|
329
|
+
"name": "fetch-url",
|
|
330
|
+
"arguments": { "url": "https://example.com" },
|
|
331
|
+
"task": { "ttl": 300 }
|
|
332
|
+
}
|
|
333
|
+
}
|
|
334
|
+
```
|
|
250
335
|
|
|
251
|
-
|
|
252
|
-
- If cache is enabled and content exceeds the inline limit, the tool response
|
|
253
|
-
includes a `resource_link` block pointing to `superfetch://cache/markdown/{urlHash}`.
|
|
254
|
-
- If cache is disabled, inline markdown is truncated with `...[truncated]`.
|
|
255
|
-
- In stdio mode, the tool also embeds a `resource` block containing full
|
|
256
|
-
markdown content when available.
|
|
336
|
+
Then poll `tasks/get` until the task status is `completed` or `failed`, and retrieve the result via `tasks/result`.
|
|
257
337
|
|
|
258
338
|
### Resources
|
|
259
339
|
|
|
260
|
-
| URI
|
|
261
|
-
|
|
|
262
|
-
| `
|
|
263
|
-
| `internal://
|
|
264
|
-
| `
|
|
340
|
+
| URI Pattern | MIME Type | Description |
|
|
341
|
+
| ------------------------------------------ | ------------------ | ------------------------------------------------ |
|
|
342
|
+
| `internal://instructions` | `text/markdown` | Server instructions and usage guidance |
|
|
343
|
+
| `internal://config` | `application/json` | Current runtime configuration (secrets redacted) |
|
|
344
|
+
| `superfetch://cache/{namespace}/{urlHash}` | `text/markdown` | Cached web content snapshots (subscribable) |
|
|
345
|
+
|
|
346
|
+
The `superfetch://cache/...` resource supports subscriptions — clients receive notifications when cached content is updated.
|
|
265
347
|
|
|
266
348
|
### Tasks
|
|
267
349
|
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
|
|
350
|
+
The server declares full MCP task support:
|
|
351
|
+
|
|
352
|
+
| Endpoint | Description |
|
|
353
|
+
| -------------- | ------------------------------------ |
|
|
354
|
+
| `tasks/list` | List tasks (scoped to session/owner) |
|
|
355
|
+
| `tasks/get` | Get task status by ID |
|
|
356
|
+
| `tasks/result` | Retrieve completed task result |
|
|
357
|
+
| `tasks/cancel` | Cancel an in-flight task |
|
|
271
358
|
|
|
272
359
|
## HTTP Mode Endpoints
|
|
273
360
|
|
|
274
|
-
| Method
|
|
275
|
-
|
|
|
276
|
-
| GET | `/health`
|
|
277
|
-
| POST | `/mcp`
|
|
278
|
-
| GET | `/mcp`
|
|
279
|
-
| DELETE | `/mcp`
|
|
280
|
-
| GET | `/mcp/downloads
|
|
361
|
+
| Method | Path | Auth | Description |
|
|
362
|
+
| -------- | ----------------------------------- | ---- | ---------------------------------------- |
|
|
363
|
+
| `GET` | `/health` | No | Health check with server stats |
|
|
364
|
+
| `POST` | `/mcp` | Yes | MCP JSON-RPC (Streamable HTTP) |
|
|
365
|
+
| `GET` | `/mcp` | Yes | SSE stream for server-initiated messages |
|
|
366
|
+
| `DELETE` | `/mcp` | Yes | Terminate MCP session |
|
|
367
|
+
| `GET` | `/mcp/downloads/{namespace}/{hash}` | Yes | Download cached content |
|
|
368
|
+
|
|
369
|
+
### Session Behavior
|
|
370
|
+
|
|
371
|
+
- Sessions are created on the first `POST /mcp` request with an `initialize` message
|
|
372
|
+
- Session ID is returned in the `mcp-session-id` response header
|
|
373
|
+
- Sessions expire after 30 minutes of inactivity (max 200 concurrent)
|
|
281
374
|
|
|
282
|
-
|
|
375
|
+
### Authentication
|
|
283
376
|
|
|
284
|
-
-
|
|
285
|
-
-
|
|
377
|
+
- **Static tokens**: Set `ACCESS_TOKENS` or `API_KEY` environment variables; pass as `Authorization: Bearer <token>`
|
|
378
|
+
- **OAuth**: Configure `OAUTH_*` environment variables to enable OAuth 2.0 token introspection
|
|
286
379
|
|
|
287
380
|
## Client Configuration Examples
|
|
288
381
|
|
|
289
382
|
<details>
|
|
290
|
-
<summary
|
|
383
|
+
<summary>VS Code / VS Code Insiders</summary>
|
|
291
384
|
|
|
292
|
-
Add to
|
|
385
|
+
Add to your VS Code settings (`.vscode/mcp.json` or User Settings):
|
|
293
386
|
|
|
294
387
|
```json
|
|
295
388
|
{
|
|
296
389
|
"servers": {
|
|
297
|
-
"
|
|
390
|
+
"superfetch": {
|
|
298
391
|
"command": "npx",
|
|
299
392
|
"args": ["-y", "@j0hanz/superfetch@latest", "--stdio"]
|
|
300
393
|
}
|
|
@@ -305,14 +398,14 @@ Add to .vscode/mcp.json:
|
|
|
305
398
|
</details>
|
|
306
399
|
|
|
307
400
|
<details>
|
|
308
|
-
<summary
|
|
401
|
+
<summary>Claude Desktop</summary>
|
|
309
402
|
|
|
310
|
-
Add to claude_desktop_config.json
|
|
403
|
+
Add to `claude_desktop_config.json`:
|
|
311
404
|
|
|
312
405
|
```json
|
|
313
406
|
{
|
|
314
407
|
"mcpServers": {
|
|
315
|
-
"
|
|
408
|
+
"superfetch": {
|
|
316
409
|
"command": "npx",
|
|
317
410
|
"args": ["-y", "@j0hanz/superfetch@latest", "--stdio"]
|
|
318
411
|
}
|
|
@@ -323,12 +416,16 @@ Add to claude_desktop_config.json:
|
|
|
323
416
|
</details>
|
|
324
417
|
|
|
325
418
|
<details>
|
|
326
|
-
<summary
|
|
419
|
+
<summary>Cursor</summary>
|
|
420
|
+
|
|
421
|
+
[](https://cursor.com/install-mcp?name=superfetch&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsIkBqMGhhbnovc3VwZXJmZXRjaEBsYXRlc3QiLCItLXN0ZGlvIl19)
|
|
422
|
+
|
|
423
|
+
Or manually add to Cursor MCP settings:
|
|
327
424
|
|
|
328
425
|
```json
|
|
329
426
|
{
|
|
330
427
|
"mcpServers": {
|
|
331
|
-
"
|
|
428
|
+
"superfetch": {
|
|
332
429
|
"command": "npx",
|
|
333
430
|
"args": ["-y", "@j0hanz/superfetch@latest", "--stdio"]
|
|
334
431
|
}
|
|
@@ -339,12 +436,14 @@ Add to claude_desktop_config.json:
|
|
|
339
436
|
</details>
|
|
340
437
|
|
|
341
438
|
<details>
|
|
342
|
-
<summary
|
|
439
|
+
<summary>Windsurf</summary>
|
|
440
|
+
|
|
441
|
+
Add to your Windsurf MCP configuration:
|
|
343
442
|
|
|
344
443
|
```json
|
|
345
444
|
{
|
|
346
445
|
"mcpServers": {
|
|
347
|
-
"
|
|
446
|
+
"superfetch": {
|
|
348
447
|
"command": "npx",
|
|
349
448
|
"args": ["-y", "@j0hanz/superfetch@latest", "--stdio"]
|
|
350
449
|
}
|
|
@@ -354,58 +453,92 @@ Add to claude_desktop_config.json:
|
|
|
354
453
|
|
|
355
454
|
</details>
|
|
356
455
|
|
|
456
|
+
## Security
|
|
457
|
+
|
|
458
|
+
### SSRF Protection
|
|
459
|
+
|
|
460
|
+
superFetch blocks requests to private and internal network addresses:
|
|
461
|
+
|
|
462
|
+
- **Blocked hosts**: `localhost`, `127.0.0.0/8`, `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `169.254.0.0/16`, `100.64.0.0/10`
|
|
463
|
+
- **Blocked IPv6**: `::1`, `fc00::/7`, `fe80::/10`, IPv4-mapped private addresses (`::ffff:10.*`, etc.)
|
|
464
|
+
- **Cloud metadata**: `169.254.169.254` (AWS), `metadata.google.internal`, `metadata.azure.com`, `100.100.100.200` (Azure IMDS)
|
|
465
|
+
|
|
466
|
+
DNS preflight checks run on every redirect hop to prevent DNS rebinding attacks.
|
|
467
|
+
|
|
468
|
+
### Stdio Transport Safety
|
|
469
|
+
|
|
470
|
+
The server never writes non-protocol data to stdout. All logs and diagnostics go to stderr.
|
|
471
|
+
|
|
472
|
+
### Rate Limiting
|
|
473
|
+
|
|
474
|
+
HTTP mode enforces a rate limit of 100 requests per 60-second window per client.
|
|
475
|
+
|
|
476
|
+
### Content Safety
|
|
477
|
+
|
|
478
|
+
- HTML downloads are capped at 10 MB
|
|
479
|
+
- Worker threads run in isolation with configurable resource limits
|
|
480
|
+
- Auth tokens are stored in-memory only and compared using timing-safe equality
|
|
481
|
+
|
|
357
482
|
## Development Workflow
|
|
358
483
|
|
|
359
|
-
### Install
|
|
484
|
+
### Install Dependencies
|
|
360
485
|
|
|
361
486
|
```bash
|
|
362
487
|
npm install
|
|
363
488
|
```
|
|
364
489
|
|
|
365
|
-
|
|
366
|
-
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
|
|
|
370
|
-
|
|
|
371
|
-
|
|
|
372
|
-
|
|
|
373
|
-
|
|
|
374
|
-
|
|
|
375
|
-
|
|
|
376
|
-
|
|
|
377
|
-
|
|
|
378
|
-
|
|
|
379
|
-
|
|
|
380
|
-
| type-check | `tsc --noEmit` | Type checking. |
|
|
381
|
-
| type-check:diagnostics | `tsc --noEmit --extendedDiagnostics` | Type check diagnostics. |
|
|
382
|
-
| type-check:trace | `node -e "require('fs').rmSync('.ts-trace',{recursive:true,force:true})" && tsc --noEmit --generateTrace .ts-trace` | Generate TS trace. |
|
|
383
|
-
| lint | `eslint .` | Lint. |
|
|
384
|
-
| lint:fix | `eslint . --fix` | Lint and fix. |
|
|
385
|
-
| test | `npm run build --silent && node --test --experimental-transform-types` | Run tests (builds first). |
|
|
386
|
-
| test:coverage | `npm run build --silent && node --test --experimental-transform-types --experimental-test-coverage` | Test with coverage. |
|
|
387
|
-
| knip | `knip` | Dead code analysis. |
|
|
388
|
-
| knip:fix | `knip --fix` | Fix knip issues. |
|
|
389
|
-
| inspector | `npx @modelcontextprotocol/inspector` | MCP Inspector. |
|
|
390
|
-
| prepublishOnly | `npm run lint && npm run type-check && npm run build` | Prepublish checks. |
|
|
490
|
+
### Scripts
|
|
491
|
+
|
|
492
|
+
| Script | Command | Description |
|
|
493
|
+
| --------------- | ----------------------- | -------------------------------------------- |
|
|
494
|
+
| `dev` | `npm run dev` | TypeScript watch mode |
|
|
495
|
+
| `dev:run` | `npm run dev:run` | Run compiled output with watch + `.env` |
|
|
496
|
+
| `build` | `npm run build` | Clean, compile, copy assets, make executable |
|
|
497
|
+
| `start` | `npm start` | Run compiled server |
|
|
498
|
+
| `test` | `npm test` | Run test suite (Node.js native test runner) |
|
|
499
|
+
| `test:coverage` | `npm run test:coverage` | Run tests with coverage |
|
|
500
|
+
| `lint` | `npm run lint` | ESLint |
|
|
501
|
+
| `lint:fix` | `npm run lint:fix` | ESLint with auto-fix |
|
|
502
|
+
| `format` | `npm run format` | Prettier |
|
|
503
|
+
| `type-check` | `npm run type-check` | TypeScript type checking |
|
|
504
|
+
| `inspector` | `npm run inspector` | Build and launch MCP Inspector |
|
|
391
505
|
|
|
392
506
|
## Build and Release
|
|
393
507
|
|
|
394
|
-
|
|
395
|
-
|
|
396
|
-
|
|
397
|
-
|
|
398
|
-
|
|
508
|
+
```bash
|
|
509
|
+
npm run build # Clean → Compile → Copy Assets → chmod
|
|
510
|
+
npm run prepublishOnly # Lint → Type-Check → Build
|
|
511
|
+
npm publish # Publish to npm
|
|
512
|
+
```
|
|
513
|
+
|
|
514
|
+
The `prepare` script runs `npm run build` automatically on `npm install` from source.
|
|
399
515
|
|
|
400
516
|
## Troubleshooting
|
|
401
517
|
|
|
402
|
-
|
|
403
|
-
|
|
404
|
-
|
|
405
|
-
|
|
406
|
-
|
|
407
|
-
|
|
408
|
-
|
|
409
|
-
|
|
518
|
+
### MCP Inspector
|
|
519
|
+
|
|
520
|
+
Use the built-in inspector to test the server interactively:
|
|
521
|
+
|
|
522
|
+
```bash
|
|
523
|
+
npm run inspector
|
|
524
|
+
```
|
|
525
|
+
|
|
526
|
+
This builds the project and launches `@modelcontextprotocol/inspector` pointing to the compiled server.
|
|
527
|
+
|
|
528
|
+
### Common Issues
|
|
529
|
+
|
|
530
|
+
| Issue | Solution |
|
|
531
|
+
| ------------------------- | ------------------------------------------------------------------------------------- |
|
|
532
|
+
| `VALIDATION_ERROR` on URL | URL is blocked (private IP/localhost) or malformed. Do not retry. |
|
|
533
|
+
| `queue_full` error | Worker pool is saturated. Wait briefly, then retry or use async task mode. |
|
|
534
|
+
| Garbled output | Binary content (images, PDFs) cannot be converted. Ensure the URL serves HTML. |
|
|
535
|
+
| No output in stdio mode | Ensure `--stdio` flag is passed. Without it, the server starts in HTTP mode. |
|
|
536
|
+
| Auth errors in HTTP mode | Set `ACCESS_TOKENS` or `API_KEY` env var and pass as `Authorization: Bearer <token>`. |
|
|
537
|
+
|
|
538
|
+
### Stdout / Stderr Guidance
|
|
539
|
+
|
|
540
|
+
In stdio mode, **stdout** is reserved exclusively for MCP JSON-RPC messages. Logs and diagnostics are written to **stderr**. Never pipe stdout to a log file when using stdio transport.
|
|
541
|
+
|
|
542
|
+
## License
|
|
410
543
|
|
|
411
|
-
|
|
544
|
+
[MIT](https://opensource.org/licenses/MIT)
|