@j0hanz/fetch-url-mcp 1.13.0 → 2.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -21
- package/README.md +815 -807
- package/dist/assets/logo.svg +53 -53
- package/dist/http/auth.d.ts +7 -3
- package/dist/http/auth.d.ts.map +1 -1
- package/dist/http/auth.js +65 -37
- package/dist/http/helpers.d.ts +21 -0
- package/dist/http/helpers.d.ts.map +1 -0
- package/dist/http/helpers.js +31 -0
- package/dist/http/index.d.ts +1 -0
- package/dist/http/index.d.ts.map +1 -1
- package/dist/http/index.js +1 -0
- package/dist/http/native.d.ts +6 -24
- package/dist/http/native.d.ts.map +1 -1
- package/dist/http/native.js +141 -86
- package/dist/http/rate-limit.d.ts +1 -1
- package/dist/http/rate-limit.d.ts.map +1 -1
- package/dist/http/rate-limit.js +3 -9
- package/dist/index.js +0 -0
- package/dist/lib/config.d.ts +1 -0
- package/dist/lib/config.d.ts.map +1 -1
- package/dist/lib/config.js +1 -0
- package/dist/lib/core.d.ts +4 -3
- package/dist/lib/core.d.ts.map +1 -1
- package/dist/lib/core.js +4 -1
- package/dist/lib/error/classify.d.ts.map +1 -1
- package/dist/lib/error/classify.js +27 -6
- package/dist/lib/error/payload.d.ts +2 -2
- package/dist/lib/error/payload.d.ts.map +1 -1
- package/dist/lib/error/payload.js +2 -2
- package/dist/lib/mcp-interop.d.ts +9 -108
- package/dist/lib/mcp-interop.d.ts.map +1 -1
- package/dist/lib/mcp-interop.js +39 -240
- package/dist/lib/net/http.d.ts.map +1 -1
- package/dist/lib/net/http.js +4 -17
- package/dist/resources/index.d.ts +1 -1
- package/dist/resources/index.d.ts.map +1 -1
- package/dist/resources/index.js +67 -60
- package/dist/server.d.ts +1 -1
- package/dist/server.d.ts.map +1 -1
- package/dist/server.js +39 -34
- package/dist/tasks/adapter.d.ts +15 -0
- package/dist/tasks/adapter.d.ts.map +1 -0
- package/dist/tasks/adapter.js +138 -0
- package/dist/tasks/manager.d.ts +14 -82
- package/dist/tasks/manager.d.ts.map +1 -1
- package/dist/tasks/manager.js +48 -607
- package/dist/tasks/store.d.ts +33 -19
- package/dist/tasks/store.d.ts.map +1 -1
- package/dist/tasks/store.js +143 -80
- package/dist/tools/index.d.ts +7 -13
- package/dist/tools/index.d.ts.map +1 -1
- package/dist/tools/index.js +153 -58
- package/dist/transform/index.js +1 -1
- package/package.json +110 -108
package/README.md
CHANGED
|
@@ -1,807 +1,815 @@
|
|
|
1
|
-
# Fetch URL MCP Server
|
|
2
|
-
|
|
3
|
-
[](https://www.npmjs.com/package/%40j0hanz%2Ffetch-url-mcp) [](#contributing-and-license)
|
|
4
|
-
|
|
5
|
-
[](https://insiders.vscode.dev/redirect/mcp/install?name=fetch-url&config=%7B%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22%40j0hanz%2Ffetch-url-mcp%40latest%22%5D%7D) [](https://insiders.vscode.dev/redirect/mcp/install?name=fetch-url&config=%7B%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22%40j0hanz%2Ffetch-url-mcp%40latest%22%5D%7D&quality=insiders) [](https://vs-open.link/mcp-install?%7B%22fetch-url-mcp%22%3A%7B%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22%40j0hanz%2Ffetch-url-mcp%40latest%22%5D%7D%7D)
|
|
6
|
-
|
|
7
|
-
[](https://lmstudio.ai/install-mcp?name=fetch-url&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsIkBqMGhhbnovZmV0Y2gtdXJsLW1jcEBsYXRlc3QiXX0%3D) [](https://cursor.com/en/install-mcp?name=fetch-url&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsIkBqMGhhbnovZmV0Y2gtdXJsLW1jcEBsYXRlc3QiXX0%3D)
|
|
8
|
-
|
|
9
|
-
An MCP server that fetches web pages and converts them to clean, readable Markdown.
|
|
10
|
-
|
|
11
|
-
## Overview
|
|
12
|
-
|
|
13
|
-
This server takes a URL, fetches the page, and strips away everything you don't need — navigation, sidebars, banners, scripts — leaving just the main content as Markdown. It's perfect for feeding into LLMs, giving them the distilled essence of a page without the noise. It also recognizes GitHub, GitLab, Bitbucket, and Gist URLs and rewrites them to fetch the raw content directly.
|
|
14
|
-
|
|
15
|
-
By default it runs over stdio. Pass `--http` if you need a proper HTTP endpoint with auth, rate limiting, TLS, and session support.
|
|
16
|
-
|
|
17
|
-
## Key Features
|
|
18
|
-
|
|
19
|
-
- **HTML to Markdown** — Turns any public web page into clean, readable Markdown with metadata like `title`, `url`, `contentSize`, and `truncated`.
|
|
20
|
-
- **Smart URL handling** — Recognizes GitHub, GitLab, Bitbucket, and Gist page URLs and rewrites them to raw-content endpoints before fetching.
|
|
21
|
-
- **Task mode** — Big or slow pages can run as async MCP tasks with progress updates, instead of blocking. In HTTP mode, tasks are bound to the authenticated caller rather than a single MCP session, so they can be resumed after reconnecting
|
|
22
|
-
- **Self-documenting** — Includes an `internal://instructions` resource and a `get-help` prompt so clients know how to use it.
|
|
23
|
-
- **HTTP mode** — Optionally serves over Streamable HTTP with host/origin validation, bearer or OAuth auth, rate limiting, health checks, and TLS.
|
|
24
|
-
|
|
25
|
-
## Web Client
|
|
26
|
-
|
|
27
|
-
A browser-based client is available if you want to use the server without any MCP setup.
|
|
28
|
-
|
|
29
|
-
**[Live app](https://fetch-url-client.vercel.app)** · [Source code](https://github.com/j0hanz/fetch-url)
|
|
30
|
-
|
|
31
|
-
## Requirements
|
|
32
|
-
|
|
33
|
-
- **Node.js** >= 24
|
|
34
|
-
- **Docker** (optional) — only needed if you want to run the container image
|
|
35
|
-
|
|
36
|
-
## Quick Start
|
|
37
|
-
|
|
38
|
-
Add this to your MCP client config:
|
|
39
|
-
|
|
40
|
-
```json
|
|
41
|
-
{
|
|
42
|
-
"mcpServers": {
|
|
43
|
-
"fetch-url-mcp": {
|
|
44
|
-
"command": "npx",
|
|
45
|
-
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
46
|
-
}
|
|
47
|
-
}
|
|
48
|
-
}
|
|
49
|
-
```
|
|
50
|
-
|
|
51
|
-
## Client Configuration
|
|
52
|
-
|
|
53
|
-
<details>
|
|
54
|
-
<summary><b>Install in VS Code</b></summary>
|
|
55
|
-
|
|
56
|
-
[](https://insiders.vscode.dev/redirect/mcp/install?name=fetch-url&config=%7B%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22%40j0hanz%2Ffetch-url-mcp%40latest%22%5D%7D)
|
|
57
|
-
|
|
58
|
-
Add to `.vscode/mcp.json`:
|
|
59
|
-
|
|
60
|
-
```json
|
|
61
|
-
{
|
|
62
|
-
"servers": {
|
|
63
|
-
"fetch-url-mcp": {
|
|
64
|
-
"type": "stdio",
|
|
65
|
-
"command": "npx",
|
|
66
|
-
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
67
|
-
}
|
|
68
|
-
}
|
|
69
|
-
}
|
|
70
|
-
```
|
|
71
|
-
|
|
72
|
-
Or install via CLI:
|
|
73
|
-
|
|
74
|
-
```sh
|
|
75
|
-
code --add-mcp '{"name":"fetch-url-mcp","command":"npx","args":["-y","@j0hanz/fetch-url-mcp@latest"]}'
|
|
76
|
-
```
|
|
77
|
-
|
|
78
|
-
For more info, see [VS Code MCP docs](https://code.visualstudio.com/docs/copilot/chat/mcp-servers).
|
|
79
|
-
|
|
80
|
-
</details>
|
|
81
|
-
|
|
82
|
-
<details>
|
|
83
|
-
<summary><b>Install in VS Code Insiders</b></summary>
|
|
84
|
-
|
|
85
|
-
[](https://insiders.vscode.dev/redirect/mcp/install?name=fetch-url&config=%7B%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22%40j0hanz%2Ffetch-url-mcp%40latest%22%5D%7D&quality=insiders)
|
|
86
|
-
|
|
87
|
-
Add to `.vscode/mcp.json`:
|
|
88
|
-
|
|
89
|
-
```json
|
|
90
|
-
{
|
|
91
|
-
"servers": {
|
|
92
|
-
"fetch-url-mcp": {
|
|
93
|
-
"type": "stdio",
|
|
94
|
-
"command": "npx",
|
|
95
|
-
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
96
|
-
}
|
|
97
|
-
}
|
|
98
|
-
}
|
|
99
|
-
```
|
|
100
|
-
|
|
101
|
-
Or install via CLI:
|
|
102
|
-
|
|
103
|
-
```sh
|
|
104
|
-
code-insiders --add-mcp '{"name":"fetch-url-mcp","command":"npx","args":["-y","@j0hanz/fetch-url-mcp@latest"]}'
|
|
105
|
-
```
|
|
106
|
-
|
|
107
|
-
For more info, see [VS Code Insiders MCP docs](https://code.visualstudio.com/docs/copilot/chat/mcp-servers).
|
|
108
|
-
|
|
109
|
-
</details>
|
|
110
|
-
|
|
111
|
-
<details>
|
|
112
|
-
<summary><b>Install in Cursor</b></summary>
|
|
113
|
-
|
|
114
|
-
[](https://cursor.com/en/install-mcp?name=fetch-url&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsIkBqMGhhbnovZmV0Y2gtdXJsLW1jcEBsYXRlc3QiXX0%3D)
|
|
115
|
-
|
|
116
|
-
Add to `~/.cursor/mcp.json`:
|
|
117
|
-
|
|
118
|
-
```json
|
|
119
|
-
{
|
|
120
|
-
"mcpServers": {
|
|
121
|
-
"fetch-url-mcp": {
|
|
122
|
-
"command": "npx",
|
|
123
|
-
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
124
|
-
}
|
|
125
|
-
}
|
|
126
|
-
}
|
|
127
|
-
```
|
|
128
|
-
|
|
129
|
-
For more info, see [Cursor MCP docs](https://docs.cursor.com/context/model-context-protocol).
|
|
130
|
-
|
|
131
|
-
</details>
|
|
132
|
-
|
|
133
|
-
<details>
|
|
134
|
-
<summary><b>Install in Visual Studio</b></summary>
|
|
135
|
-
|
|
136
|
-
[](https://vs-open.link/mcp-install?%7B%22fetch-url-mcp%22%3A%7B%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22%40j0hanz%2Ffetch-url-mcp%40latest%22%5D%7D%7D)
|
|
137
|
-
|
|
138
|
-
For solution-scoped setup, add this to `.mcp.json` at the solution root:
|
|
139
|
-
|
|
140
|
-
```json
|
|
141
|
-
{
|
|
142
|
-
"servers": {
|
|
143
|
-
"fetch-url-mcp": {
|
|
144
|
-
"type": "stdio",
|
|
145
|
-
"command": "npx",
|
|
146
|
-
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
147
|
-
}
|
|
148
|
-
}
|
|
149
|
-
}
|
|
150
|
-
```
|
|
151
|
-
|
|
152
|
-
For more info, see [Visual Studio MCP docs](https://learn.microsoft.com/en-us/visualstudio/ide/mcp-servers).
|
|
153
|
-
|
|
154
|
-
</details>
|
|
155
|
-
|
|
156
|
-
<details>
|
|
157
|
-
<summary><b>Install in Goose</b></summary>
|
|
158
|
-
|
|
159
|
-
Add to `~/.config/goose/config.yaml` on macOS/Linux or `%APPDATA%\Block\goose\config\config.yaml` on Windows:
|
|
160
|
-
|
|
161
|
-
```yaml
|
|
162
|
-
extensions:
|
|
163
|
-
fetch-url-mcp:
|
|
164
|
-
name: fetch-url-mcp
|
|
165
|
-
cmd: npx
|
|
166
|
-
args: ['-y', '@j0hanz/fetch-url-mcp@latest']
|
|
167
|
-
enabled: true
|
|
168
|
-
type: stdio
|
|
169
|
-
timeout: 300
|
|
170
|
-
```
|
|
171
|
-
|
|
172
|
-
For more info, see [Goose extension docs](https://block.github.io/goose/docs/getting-started/using-extensions/).
|
|
173
|
-
|
|
174
|
-
</details>
|
|
175
|
-
|
|
176
|
-
<details>
|
|
177
|
-
<summary><b>Install in LM Studio</b></summary>
|
|
178
|
-
|
|
179
|
-
[](https://lmstudio.ai/install-mcp?name=fetch-url&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsIkBqMGhhbnovZmV0Y2gtdXJsLW1jcEBsYXRlc3QiXX0%3D)
|
|
180
|
-
|
|
181
|
-
Add to `~/.lmstudio/mcp.json` on macOS/Linux or `%USERPROFILE%/.lmstudio/mcp.json` on Windows:
|
|
182
|
-
|
|
183
|
-
```json
|
|
184
|
-
{
|
|
185
|
-
"mcpServers": {
|
|
186
|
-
"fetch-url-mcp": {
|
|
187
|
-
"command": "npx",
|
|
188
|
-
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
189
|
-
}
|
|
190
|
-
}
|
|
191
|
-
}
|
|
192
|
-
```
|
|
193
|
-
|
|
194
|
-
For more info, see [LM Studio MCP docs](https://lmstudio.ai/docs/basics/mcp).
|
|
195
|
-
|
|
196
|
-
</details>
|
|
197
|
-
|
|
198
|
-
<details>
|
|
199
|
-
<summary><b>Install in Claude Desktop</b></summary>
|
|
200
|
-
|
|
201
|
-
Add to `claude_desktop_config.json`:
|
|
202
|
-
|
|
203
|
-
```json
|
|
204
|
-
{
|
|
205
|
-
"mcpServers": {
|
|
206
|
-
"fetch-url-mcp": {
|
|
207
|
-
"command": "npx",
|
|
208
|
-
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
209
|
-
}
|
|
210
|
-
}
|
|
211
|
-
}
|
|
212
|
-
```
|
|
213
|
-
|
|
214
|
-
For more info, see [Claude Desktop MCP docs](https://modelcontextprotocol.io/quickstart/user).
|
|
215
|
-
|
|
216
|
-
</details>
|
|
217
|
-
|
|
218
|
-
<details>
|
|
219
|
-
<summary><b>Install in Claude Code</b></summary>
|
|
220
|
-
|
|
221
|
-
Use the CLI:
|
|
222
|
-
|
|
223
|
-
```sh
|
|
224
|
-
claude mcp add fetch-url-mcp -- npx -y @j0hanz/fetch-url-mcp@latest
|
|
225
|
-
```
|
|
226
|
-
|
|
227
|
-
For project-scoped config, Claude Code writes `.mcp.json` with:
|
|
228
|
-
|
|
229
|
-
```json
|
|
230
|
-
{
|
|
231
|
-
"mcpServers": {
|
|
232
|
-
"fetch-url-mcp": {
|
|
233
|
-
"command": "npx",
|
|
234
|
-
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"],
|
|
235
|
-
"env": {}
|
|
236
|
-
}
|
|
237
|
-
}
|
|
238
|
-
}
|
|
239
|
-
```
|
|
240
|
-
|
|
241
|
-
For more info, see [Claude Code MCP docs](https://docs.anthropic.com/en/docs/claude-code/mcp).
|
|
242
|
-
|
|
243
|
-
</details>
|
|
244
|
-
|
|
245
|
-
<details>
|
|
246
|
-
<summary><b>Install in Windsurf</b></summary>
|
|
247
|
-
|
|
248
|
-
Add to `~/.codeium/windsurf/mcp_config.json`:
|
|
249
|
-
|
|
250
|
-
```json
|
|
251
|
-
{
|
|
252
|
-
"mcpServers": {
|
|
253
|
-
"fetch-url-mcp": {
|
|
254
|
-
"command": "npx",
|
|
255
|
-
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
256
|
-
}
|
|
257
|
-
}
|
|
258
|
-
}
|
|
259
|
-
```
|
|
260
|
-
|
|
261
|
-
For more info, see [Windsurf MCP docs](https://docs.windsurf.com/windsurf/cascade/mcp).
|
|
262
|
-
|
|
263
|
-
</details>
|
|
264
|
-
|
|
265
|
-
<details>
|
|
266
|
-
<summary><b>Install in Amp</b></summary>
|
|
267
|
-
|
|
268
|
-
Add to `~/.config/amp/settings.json` on macOS/Linux, `%USERPROFILE%\.config\amp\settings.json` on Windows, or `.amp/settings.json` for workspace-scoped config:
|
|
269
|
-
|
|
270
|
-
```json
|
|
271
|
-
{
|
|
272
|
-
"amp.mcpServers": {
|
|
273
|
-
"fetch-url-mcp": {
|
|
274
|
-
"command": "npx",
|
|
275
|
-
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
276
|
-
}
|
|
277
|
-
}
|
|
278
|
-
}
|
|
279
|
-
```
|
|
280
|
-
|
|
281
|
-
Or install via CLI:
|
|
282
|
-
|
|
283
|
-
```sh
|
|
284
|
-
amp mcp add fetch-url-mcp -- npx -y @j0hanz/fetch-url-mcp@latest
|
|
285
|
-
```
|
|
286
|
-
|
|
287
|
-
For more info, see [Amp docs](https://ampcode.com/manual).
|
|
288
|
-
|
|
289
|
-
</details>
|
|
290
|
-
|
|
291
|
-
<details>
|
|
292
|
-
<summary><b>Install in Cline</b></summary>
|
|
293
|
-
|
|
294
|
-
Open the MCP Servers panel, choose `Configure MCP Servers`, and add this to `cline_mcp_settings.json`:
|
|
295
|
-
|
|
296
|
-
```json
|
|
297
|
-
{
|
|
298
|
-
"mcpServers": {
|
|
299
|
-
"fetch-url-mcp": {
|
|
300
|
-
"command": "npx",
|
|
301
|
-
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
302
|
-
}
|
|
303
|
-
}
|
|
304
|
-
}
|
|
305
|
-
```
|
|
306
|
-
|
|
307
|
-
For more info, see [Cline MCP docs](https://docs.cline.bot/mcp/configuring-mcp-servers).
|
|
308
|
-
|
|
309
|
-
</details>
|
|
310
|
-
|
|
311
|
-
<details>
|
|
312
|
-
<summary><b>Install in Codex CLI</b></summary>
|
|
313
|
-
|
|
314
|
-
Use the CLI:
|
|
315
|
-
|
|
316
|
-
```sh
|
|
317
|
-
codex mcp add fetch-url-mcp -- npx -y @j0hanz/fetch-url-mcp@latest
|
|
318
|
-
```
|
|
319
|
-
|
|
320
|
-
Or add this to `~/.codex/config.toml` or project-scoped `.codex/config.toml`:
|
|
321
|
-
|
|
322
|
-
```toml
|
|
323
|
-
[mcp_servers.fetch-url-mcp]
|
|
324
|
-
command = "npx"
|
|
325
|
-
args = ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
326
|
-
```
|
|
327
|
-
|
|
328
|
-
For more info, see [Codex MCP docs](https://developers.openai.com/codex/mcp/).
|
|
329
|
-
|
|
330
|
-
</details>
|
|
331
|
-
|
|
332
|
-
<details>
|
|
333
|
-
<summary><b>Install in GitHub Copilot</b></summary>
|
|
334
|
-
|
|
335
|
-
Add to `.vscode/mcp.json`:
|
|
336
|
-
|
|
337
|
-
```json
|
|
338
|
-
{
|
|
339
|
-
"servers": {
|
|
340
|
-
"fetch-url-mcp": {
|
|
341
|
-
"type": "stdio",
|
|
342
|
-
"command": "npx",
|
|
343
|
-
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
344
|
-
}
|
|
345
|
-
}
|
|
346
|
-
}
|
|
347
|
-
```
|
|
348
|
-
|
|
349
|
-
For more info, see [GitHub Copilot MCP docs](https://code.visualstudio.com/docs/copilot/chat/mcp-servers).
|
|
350
|
-
|
|
351
|
-
</details>
|
|
352
|
-
|
|
353
|
-
<details>
|
|
354
|
-
<summary><b>Install in Warp</b></summary>
|
|
355
|
-
|
|
356
|
-
Open `Personal > MCP Servers` in Warp, choose `+ Add`, and either add a CLI server with:
|
|
357
|
-
|
|
358
|
-
- `command`: `npx`
|
|
359
|
-
- `args`: `["-y", "@j0hanz/fetch-url-mcp@latest"]`
|
|
360
|
-
|
|
361
|
-
Or paste this JSON snippet when using Warp's multi-server import flow:
|
|
362
|
-
|
|
363
|
-
```json
|
|
364
|
-
{
|
|
365
|
-
"mcpServers": {
|
|
366
|
-
"fetch-url-mcp": {
|
|
367
|
-
"command": "npx",
|
|
368
|
-
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
369
|
-
}
|
|
370
|
-
}
|
|
371
|
-
}
|
|
372
|
-
```
|
|
373
|
-
|
|
374
|
-
For more info, see [Warp MCP docs](https://docs.warp.dev/features/warp-ai/mcp).
|
|
375
|
-
|
|
376
|
-
</details>
|
|
377
|
-
|
|
378
|
-
<details>
|
|
379
|
-
<summary><b>Install in Kiro</b></summary>
|
|
380
|
-
|
|
381
|
-
Use Kiro's MCP Servers panel or the `Add to Kiro` install flow. Kiro stores workspace-scoped MCP config in `.kiro/settings/mcp.json` and user-scoped config in `~/.kiro/settings/mcp.json`.
|
|
382
|
-
|
|
383
|
-
For this server, use:
|
|
384
|
-
|
|
385
|
-
- `command`: `npx`
|
|
386
|
-
- `args`: `["-y", "@j0hanz/fetch-url-mcp@latest"]`
|
|
387
|
-
|
|
388
|
-
For more info, see [Kiro MCP docs](https://kiro.dev/blog/unlock-your-development-productivity-with-kiro-and-mcp/).
|
|
389
|
-
|
|
390
|
-
</details>
|
|
391
|
-
|
|
392
|
-
<details>
|
|
393
|
-
<summary><b>Install in Gemini CLI</b></summary>
|
|
394
|
-
|
|
395
|
-
Add to `~/.gemini/settings.json`:
|
|
396
|
-
|
|
397
|
-
```json
|
|
398
|
-
{
|
|
399
|
-
"mcpServers": {
|
|
400
|
-
"fetch-url-mcp": {
|
|
401
|
-
"command": "npx",
|
|
402
|
-
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
403
|
-
}
|
|
404
|
-
}
|
|
405
|
-
}
|
|
406
|
-
```
|
|
407
|
-
|
|
408
|
-
For more info, see [Gemini CLI MCP docs](https://google-gemini.github.io/gemini-cli/docs/tools/mcp-server.html).
|
|
409
|
-
|
|
410
|
-
</details>
|
|
411
|
-
|
|
412
|
-
<details>
|
|
413
|
-
<summary><b>Install in Zed</b></summary>
|
|
414
|
-
|
|
415
|
-
Add to `~/.config/zed/settings.json`:
|
|
416
|
-
|
|
417
|
-
```json
|
|
418
|
-
{
|
|
419
|
-
"context_servers": {
|
|
420
|
-
"fetch-url-mcp": {
|
|
421
|
-
"command": "npx",
|
|
422
|
-
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"],
|
|
423
|
-
"env": {}
|
|
424
|
-
}
|
|
425
|
-
}
|
|
426
|
-
}
|
|
427
|
-
```
|
|
428
|
-
|
|
429
|
-
For more info, see [Zed MCP docs](https://zed.dev/docs/ai/mcp).
|
|
430
|
-
|
|
431
|
-
</details>
|
|
432
|
-
|
|
433
|
-
<details>
|
|
434
|
-
<summary><b>Install in Augment</b></summary>
|
|
435
|
-
|
|
436
|
-
Use the Augment Settings panel and either add the server manually or choose `Import from JSON`:
|
|
437
|
-
|
|
438
|
-
```json
|
|
439
|
-
{
|
|
440
|
-
"mcpServers": {
|
|
441
|
-
"fetch-url-mcp": {
|
|
442
|
-
"command": "npx",
|
|
443
|
-
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
444
|
-
}
|
|
445
|
-
}
|
|
446
|
-
}
|
|
447
|
-
```
|
|
448
|
-
|
|
449
|
-
For more info, see [Augment MCP docs](https://docs.augmentcode.com/setup-augment/mcp).
|
|
450
|
-
|
|
451
|
-
</details>
|
|
452
|
-
|
|
453
|
-
<details>
|
|
454
|
-
<summary><b>Install in Roo Code</b></summary>
|
|
455
|
-
|
|
456
|
-
Use Roo Code's MCP Servers UI or marketplace flow.
|
|
457
|
-
|
|
458
|
-
For this server, use:
|
|
459
|
-
|
|
460
|
-
- `command`: `npx`
|
|
461
|
-
- `args`: `["-y", "@j0hanz/fetch-url-mcp@latest"]`
|
|
462
|
-
|
|
463
|
-
For more info, see [Roo Code docs](https://docs.roocode.com/).
|
|
464
|
-
|
|
465
|
-
</details>
|
|
466
|
-
|
|
467
|
-
<details>
|
|
468
|
-
<summary><b>Install in Kilo Code</b></summary>
|
|
469
|
-
|
|
470
|
-
Use Kilo Code's MCP Servers UI or marketplace flow.
|
|
471
|
-
|
|
472
|
-
For this server, use:
|
|
473
|
-
|
|
474
|
-
- `command`: `npx`
|
|
475
|
-
- `args`: `["-y", "@j0hanz/fetch-url-mcp@latest"]`
|
|
476
|
-
|
|
477
|
-
For more info, see [Kilo Code docs](https://kilocode.ai/docs).
|
|
478
|
-
|
|
479
|
-
</details>
|
|
480
|
-
|
|
481
|
-
## Use Cases
|
|
482
|
-
|
|
483
|
-
- **Documentation for LLMs** — Grab a docs page, blog post, or reference article as Markdown and pass it straight into a context window.
|
|
484
|
-
- **Repository content** — Hand it a GitHub, GitLab, or Bitbucket URL and it resolves the raw content endpoint. Works with Gists too.
|
|
485
|
-
- **Slow or large pages** — Task mode lets big fetches run in the background while sending monotonic progress updates back to the client, while `tasks/get` exposes the latest `statusMessage`, `
|
|
486
|
-
|
|
487
|
-
## Architecture
|
|
488
|
-
|
|
489
|
-
```text
|
|
490
|
-
[MCP Client]
|
|
491
|
-
├─ stdio -> `src/index.ts` -> `startStdioServer()` -> `createMcpServer()`
|
|
492
|
-
└─ HTTP (`--http`) -> `src/index.ts` -> `startHttpServer()` -> HTTP dispatcher
|
|
493
|
-
├─ `GET /health`
|
|
494
|
-
├─ `GET /.well-known/oauth-protected-resource`
|
|
495
|
-
├─ `GET /.well-known/oauth-protected-resource/mcp`
|
|
496
|
-
└─ `POST|GET|DELETE /mcp`
|
|
497
|
-
|
|
498
|
-
`createMcpServer()`
|
|
499
|
-
├─ registers tool: `fetch-url`
|
|
500
|
-
├─ registers prompt: `get-help`
|
|
501
|
-
├─ registers resources:
|
|
502
|
-
│ - `internal://instructions`
|
|
503
|
-
├─ enables capabilities: logging, resources, prompts, tasks
|
|
504
|
-
└─ installs task handlers, log-level handling, and shutdown cleanup
|
|
505
|
-
|
|
506
|
-
`fetch-url` execution
|
|
507
|
-
├─ validate input with `fetchUrlInputSchema`
|
|
508
|
-
├─ normalize URL and block local/private targets unless allowed
|
|
509
|
-
├─ rewrite supported code-host URLs to raw endpoints when possible
|
|
510
|
-
├─ fetch content via the shared pipeline
|
|
511
|
-
├─ transform HTML into Markdown in the transform worker path
|
|
512
|
-
└─ validate `structuredContent` with `fetchUrlOutputSchema`
|
|
513
|
-
```
|
|
514
|
-
|
|
515
|
-
### Request Lifecycle
|
|
516
|
-
|
|
517
|
-
```text
|
|
518
|
-
[Client] -- initialize {protocolVersion, capabilities} --> [Server]
|
|
519
|
-
[Server] -- {protocolVersion, capabilities, serverInfo} --> [Client]
|
|
520
|
-
[Client] -- notifications/initialized --> [Server]
|
|
521
|
-
[Client] -- tools/call {name, arguments} --> [Server]
|
|
522
|
-
[Server] -- {content: [{type, text}], structuredContent?, isError?} --> [Client]
|
|
523
|
-
```
|
|
524
|
-
|
|
525
|
-
## MCP Surface
|
|
526
|
-
|
|
527
|
-
### Tools
|
|
528
|
-
|
|
529
|
-
#### `fetch-url`
|
|
530
|
-
|
|
531
|
-
Takes a URL and returns Markdown. Read-only
|
|
532
|
-
|
|
533
|
-
| Parameter | Type | Required | Description |
|
|
534
|
-
| --------- | -------- | -------- | --------------------------- |
|
|
535
|
-
| `url` | `string` | yes | Target URL. Max 2048 chars. |
|
|
536
|
-
|
|
537
|
-
You get text content back by default. If output validation passes, the response also includes `structuredContent` with typed fields: `url`, `resolvedUrl`, `finalUrl`, `title`, `metadata`, `markdown`, `fetchedAt`, `contentSize`, and `truncated`. A `true` value for `truncated` means the content hit a server-side size limit.
|
|
538
|
-
|
|
539
|
-
To opt into progress updates, include `_meta.progressToken` in the tool call. The token may be a string or number
|
|
540
|
-
|
|
541
|
-
To run the tool in task mode, include `
|
|
542
|
-
|
|
543
|
-
```json
|
|
544
|
-
{
|
|
545
|
-
"method": "tools/call",
|
|
546
|
-
"params": {
|
|
547
|
-
"name": "fetch-url",
|
|
548
|
-
"arguments": {
|
|
549
|
-
"url": "https://example.com/docs"
|
|
550
|
-
},
|
|
551
|
-
"
|
|
552
|
-
"
|
|
553
|
-
|
|
554
|
-
|
|
555
|
-
|
|
556
|
-
|
|
557
|
-
|
|
558
|
-
|
|
559
|
-
|
|
560
|
-
|
|
561
|
-
|
|
562
|
-
|
|
563
|
-
|
|
564
|
-
|
|
565
|
-
|
|
566
|
-
|
|
567
|
-
|
|
568
|
-
|
|
569
|
-
|
|
570
|
-
|
|
571
|
-
|
|
572
|
-
|
|
573
|
-
|
|
574
|
-
|
|
|
575
|
-
|
|
576
|
-
|
|
577
|
-
|
|
578
|
-
|
|
579
|
-
|
|
580
|
-
|
|
|
581
|
-
|
|
582
|
-
|
|
583
|
-
|
|
584
|
-
|
|
|
585
|
-
|
|
|
586
|
-
|
|
|
587
|
-
|
|
|
588
|
-
|
|
589
|
-
|
|
590
|
-
|
|
591
|
-
|
|
|
592
|
-
|
|
|
593
|
-
|
|
594
|
-
|
|
595
|
-
|
|
596
|
-
|
|
|
597
|
-
|
|
598
|
-
|
|
599
|
-
|
|
600
|
-
|
|
601
|
-
|
|
602
|
-
|
|
603
|
-
|
|
604
|
-
|
|
605
|
-
|
|
606
|
-
|
|
607
|
-
|
|
608
|
-
|
|
609
|
-
|
|
610
|
-
|
|
611
|
-
|
|
612
|
-
|
|
613
|
-
|
|
|
614
|
-
|
|
|
615
|
-
| `
|
|
616
|
-
| `
|
|
617
|
-
| `
|
|
618
|
-
| `
|
|
619
|
-
| `
|
|
620
|
-
| `
|
|
621
|
-
|
|
622
|
-
|
|
623
|
-
|
|
624
|
-
|
|
|
625
|
-
|
|
|
626
|
-
| `
|
|
627
|
-
|
|
628
|
-
|
|
629
|
-
|
|
630
|
-
|
|
|
631
|
-
|
|
|
632
|
-
| `
|
|
633
|
-
| `
|
|
634
|
-
| `
|
|
635
|
-
| `
|
|
636
|
-
| `
|
|
637
|
-
|
|
638
|
-
|
|
639
|
-
|
|
640
|
-
|
|
|
641
|
-
|
|
|
642
|
-
| `
|
|
643
|
-
|
|
644
|
-
|
|
645
|
-
|
|
646
|
-
|
|
647
|
-
|
|
648
|
-
|
|
|
649
|
-
|
|
|
650
|
-
| `
|
|
651
|
-
|
|
652
|
-
|
|
653
|
-
|
|
654
|
-
|
|
655
|
-
|
|
656
|
-
|
|
|
657
|
-
|
|
|
658
|
-
| `
|
|
659
|
-
|
|
660
|
-
###
|
|
661
|
-
|
|
662
|
-
| Variable
|
|
663
|
-
|
|
|
664
|
-
| `
|
|
665
|
-
|
|
666
|
-
|
|
667
|
-
|
|
668
|
-
|
|
669
|
-
|
|
670
|
-
|
|
671
|
-
|
|
|
672
|
-
|
|
|
673
|
-
| `
|
|
674
|
-
|
|
675
|
-
|
|
676
|
-
|
|
677
|
-
|
|
|
678
|
-
|
|
|
679
|
-
|
|
680
|
-
|
|
681
|
-
|
|
682
|
-
|
|
|
683
|
-
|
|
|
684
|
-
| `
|
|
685
|
-
|
|
686
|
-
|
|
687
|
-
|
|
688
|
-
|
|
689
|
-
|
|
690
|
-
|
|
691
|
-
|
|
|
692
|
-
|
|
|
693
|
-
| `
|
|
694
|
-
|
|
695
|
-
|
|
696
|
-
|
|
697
|
-
|
|
698
|
-
|
|
|
699
|
-
|
|
|
700
|
-
| `
|
|
701
|
-
|
|
702
|
-
|
|
703
|
-
|
|
704
|
-
|
|
|
705
|
-
|
|
|
706
|
-
|
|
707
|
-
|
|
708
|
-
|
|
709
|
-
|
|
|
710
|
-
|
|
|
711
|
-
|
|
|
712
|
-
|
|
713
|
-
|
|
714
|
-
|
|
715
|
-
|
|
|
716
|
-
|
|
|
717
|
-
|
|
|
718
|
-
|
|
719
|
-
|
|
720
|
-
|
|
721
|
-
|
|
722
|
-
|
|
723
|
-
|
|
|
724
|
-
|
|
725
|
-
|
|
726
|
-
|
|
727
|
-
|
|
728
|
-
|
|
729
|
-
|
|
|
730
|
-
|
|
|
731
|
-
| `npm run
|
|
732
|
-
| `npm run
|
|
733
|
-
| `npm run
|
|
734
|
-
| `npm
|
|
735
|
-
|
|
736
|
-
|
|
737
|
-
|
|
738
|
-
|
|
739
|
-
|
|
|
740
|
-
|
|
|
741
|
-
|
|
742
|
-
|
|
743
|
-
|
|
744
|
-
|
|
745
|
-
|
|
|
746
|
-
|
|
|
747
|
-
| `
|
|
748
|
-
| `
|
|
749
|
-
| `
|
|
750
|
-
| `
|
|
751
|
-
| `
|
|
752
|
-
| `
|
|
753
|
-
| `
|
|
754
|
-
| `
|
|
755
|
-
| `
|
|
756
|
-
| `
|
|
757
|
-
| `
|
|
758
|
-
| `
|
|
759
|
-
| `
|
|
760
|
-
| `
|
|
761
|
-
| `
|
|
762
|
-
| `
|
|
763
|
-
| `
|
|
764
|
-
|
|
765
|
-
|
|
766
|
-
|
|
767
|
-
|
|
768
|
-
|
|
769
|
-
|
|
770
|
-
|
|
771
|
-
|
|
772
|
-
|
|
773
|
-
|
|
774
|
-
|
|
775
|
-
|
|
776
|
-
|
|
777
|
-
|
|
778
|
-
|
|
779
|
-
|
|
780
|
-
|
|
781
|
-
|
|
782
|
-
|
|
|
783
|
-
|
|
|
784
|
-
|
|
|
785
|
-
|
|
786
|
-
|
|
787
|
-
|
|
788
|
-
|
|
|
789
|
-
|
|
|
790
|
-
|
|
|
791
|
-
|
|
|
792
|
-
|
|
793
|
-
|
|
794
|
-
|
|
795
|
-
|
|
|
796
|
-
|
|
797
|
-
|
|
798
|
-
|
|
799
|
-
|
|
800
|
-
|
|
801
|
-
|
|
802
|
-
|
|
803
|
-
|
|
804
|
-
|
|
805
|
-
## License
|
|
806
|
-
|
|
807
|
-
|
|
1
|
+
# Fetch URL MCP Server
|
|
2
|
+
|
|
3
|
+
[](https://www.npmjs.com/package/%40j0hanz%2Ffetch-url-mcp) [](#contributing-and-license)
|
|
4
|
+
|
|
5
|
+
[](https://insiders.vscode.dev/redirect/mcp/install?name=fetch-url&config=%7B%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22%40j0hanz%2Ffetch-url-mcp%40latest%22%5D%7D) [](https://insiders.vscode.dev/redirect/mcp/install?name=fetch-url&config=%7B%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22%40j0hanz%2Ffetch-url-mcp%40latest%22%5D%7D&quality=insiders) [](https://vs-open.link/mcp-install?%7B%22fetch-url-mcp%22%3A%7B%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22%40j0hanz%2Ffetch-url-mcp%40latest%22%5D%7D%7D)
|
|
6
|
+
|
|
7
|
+
[](https://lmstudio.ai/install-mcp?name=fetch-url&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsIkBqMGhhbnovZmV0Y2gtdXJsLW1jcEBsYXRlc3QiXX0%3D) [](https://cursor.com/en/install-mcp?name=fetch-url&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsIkBqMGhhbnovZmV0Y2gtdXJsLW1jcEBsYXRlc3QiXX0%3D)
|
|
8
|
+
|
|
9
|
+
An MCP server that fetches web pages and converts them to clean, readable Markdown.
|
|
10
|
+
|
|
11
|
+
## Overview
|
|
12
|
+
|
|
13
|
+
This server takes a URL, fetches the page, and strips away everything you don't need — navigation, sidebars, banners, scripts — leaving just the main content as Markdown. It's perfect for feeding into LLMs, giving them the distilled essence of a page without the noise. It also recognizes GitHub, GitLab, Bitbucket, and Gist URLs and rewrites them to fetch the raw content directly.
|
|
14
|
+
|
|
15
|
+
By default it runs over stdio. Pass `--http` if you need a proper HTTP endpoint with auth, rate limiting, TLS, and session support.
|
|
16
|
+
|
|
17
|
+
## Key Features
|
|
18
|
+
|
|
19
|
+
- **HTML to Markdown** — Turns any public web page into clean, readable Markdown with metadata like `title`, `url`, `contentSize`, and `truncated`.
|
|
20
|
+
- **Smart URL handling** — Recognizes GitHub, GitLab, Bitbucket, and Gist page URLs and rewrites them to raw-content endpoints before fetching.
|
|
21
|
+
- **Task mode** — Big or slow pages can run as async MCP tasks with progress updates, instead of blocking. In HTTP mode, tasks are bound to the authenticated caller rather than a single MCP session, so they can be resumed after reconnecting as the same authenticated subject. Polling task state exposes task summaries; numeric progress remains out-of-band via `notifications/progress`.
|
|
22
|
+
- **Self-documenting** — Includes an `internal://instructions` resource and a `get-help` prompt so clients know how to use it.
|
|
23
|
+
- **HTTP mode** — Optionally serves over Streamable HTTP with host/origin validation, bearer or OAuth auth, rate limiting, health checks, and TLS.
|
|
24
|
+
|
|
25
|
+
## Web Client
|
|
26
|
+
|
|
27
|
+
A browser-based client is available if you want to use the server without any MCP setup.
|
|
28
|
+
|
|
29
|
+
**[Live app](https://fetch-url-client.vercel.app)** · [Source code](https://github.com/j0hanz/fetch-url)
|
|
30
|
+
|
|
31
|
+
## Requirements
|
|
32
|
+
|
|
33
|
+
- **Node.js** >= 24
|
|
34
|
+
- **Docker** (optional) — only needed if you want to run the container image
|
|
35
|
+
|
|
36
|
+
## Quick Start
|
|
37
|
+
|
|
38
|
+
Add this to your MCP client config:
|
|
39
|
+
|
|
40
|
+
```json
|
|
41
|
+
{
|
|
42
|
+
"mcpServers": {
|
|
43
|
+
"fetch-url-mcp": {
|
|
44
|
+
"command": "npx",
|
|
45
|
+
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
46
|
+
}
|
|
47
|
+
}
|
|
48
|
+
}
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
## Client Configuration
|
|
52
|
+
|
|
53
|
+
<details>
|
|
54
|
+
<summary><b>Install in VS Code</b></summary>
|
|
55
|
+
|
|
56
|
+
[](https://insiders.vscode.dev/redirect/mcp/install?name=fetch-url&config=%7B%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22%40j0hanz%2Ffetch-url-mcp%40latest%22%5D%7D)
|
|
57
|
+
|
|
58
|
+
Add to `.vscode/mcp.json`:
|
|
59
|
+
|
|
60
|
+
```json
|
|
61
|
+
{
|
|
62
|
+
"servers": {
|
|
63
|
+
"fetch-url-mcp": {
|
|
64
|
+
"type": "stdio",
|
|
65
|
+
"command": "npx",
|
|
66
|
+
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
67
|
+
}
|
|
68
|
+
}
|
|
69
|
+
}
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
Or install via CLI:
|
|
73
|
+
|
|
74
|
+
```sh
|
|
75
|
+
code --add-mcp '{"name":"fetch-url-mcp","command":"npx","args":["-y","@j0hanz/fetch-url-mcp@latest"]}'
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
For more info, see [VS Code MCP docs](https://code.visualstudio.com/docs/copilot/chat/mcp-servers).
|
|
79
|
+
|
|
80
|
+
</details>
|
|
81
|
+
|
|
82
|
+
<details>
|
|
83
|
+
<summary><b>Install in VS Code Insiders</b></summary>
|
|
84
|
+
|
|
85
|
+
[](https://insiders.vscode.dev/redirect/mcp/install?name=fetch-url&config=%7B%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22%40j0hanz%2Ffetch-url-mcp%40latest%22%5D%7D&quality=insiders)
|
|
86
|
+
|
|
87
|
+
Add to `.vscode/mcp.json`:
|
|
88
|
+
|
|
89
|
+
```json
|
|
90
|
+
{
|
|
91
|
+
"servers": {
|
|
92
|
+
"fetch-url-mcp": {
|
|
93
|
+
"type": "stdio",
|
|
94
|
+
"command": "npx",
|
|
95
|
+
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
96
|
+
}
|
|
97
|
+
}
|
|
98
|
+
}
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
Or install via CLI:
|
|
102
|
+
|
|
103
|
+
```sh
|
|
104
|
+
code-insiders --add-mcp '{"name":"fetch-url-mcp","command":"npx","args":["-y","@j0hanz/fetch-url-mcp@latest"]}'
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
For more info, see [VS Code Insiders MCP docs](https://code.visualstudio.com/docs/copilot/chat/mcp-servers).
|
|
108
|
+
|
|
109
|
+
</details>
|
|
110
|
+
|
|
111
|
+
<details>
|
|
112
|
+
<summary><b>Install in Cursor</b></summary>
|
|
113
|
+
|
|
114
|
+
[](https://cursor.com/en/install-mcp?name=fetch-url&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsIkBqMGhhbnovZmV0Y2gtdXJsLW1jcEBsYXRlc3QiXX0%3D)
|
|
115
|
+
|
|
116
|
+
Add to `~/.cursor/mcp.json`:
|
|
117
|
+
|
|
118
|
+
```json
|
|
119
|
+
{
|
|
120
|
+
"mcpServers": {
|
|
121
|
+
"fetch-url-mcp": {
|
|
122
|
+
"command": "npx",
|
|
123
|
+
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
124
|
+
}
|
|
125
|
+
}
|
|
126
|
+
}
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
For more info, see [Cursor MCP docs](https://docs.cursor.com/context/model-context-protocol).
|
|
130
|
+
|
|
131
|
+
</details>
|
|
132
|
+
|
|
133
|
+
<details>
|
|
134
|
+
<summary><b>Install in Visual Studio</b></summary>
|
|
135
|
+
|
|
136
|
+
[](https://vs-open.link/mcp-install?%7B%22fetch-url-mcp%22%3A%7B%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22%40j0hanz%2Ffetch-url-mcp%40latest%22%5D%7D%7D)
|
|
137
|
+
|
|
138
|
+
For solution-scoped setup, add this to `.mcp.json` at the solution root:
|
|
139
|
+
|
|
140
|
+
```json
|
|
141
|
+
{
|
|
142
|
+
"servers": {
|
|
143
|
+
"fetch-url-mcp": {
|
|
144
|
+
"type": "stdio",
|
|
145
|
+
"command": "npx",
|
|
146
|
+
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
147
|
+
}
|
|
148
|
+
}
|
|
149
|
+
}
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
For more info, see [Visual Studio MCP docs](https://learn.microsoft.com/en-us/visualstudio/ide/mcp-servers).
|
|
153
|
+
|
|
154
|
+
</details>
|
|
155
|
+
|
|
156
|
+
<details>
|
|
157
|
+
<summary><b>Install in Goose</b></summary>
|
|
158
|
+
|
|
159
|
+
Add to `~/.config/goose/config.yaml` on macOS/Linux or `%APPDATA%\Block\goose\config\config.yaml` on Windows:
|
|
160
|
+
|
|
161
|
+
```yaml
|
|
162
|
+
extensions:
|
|
163
|
+
fetch-url-mcp:
|
|
164
|
+
name: fetch-url-mcp
|
|
165
|
+
cmd: npx
|
|
166
|
+
args: ['-y', '@j0hanz/fetch-url-mcp@latest']
|
|
167
|
+
enabled: true
|
|
168
|
+
type: stdio
|
|
169
|
+
timeout: 300
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
For more info, see [Goose extension docs](https://block.github.io/goose/docs/getting-started/using-extensions/).
|
|
173
|
+
|
|
174
|
+
</details>
|
|
175
|
+
|
|
176
|
+
<details>
|
|
177
|
+
<summary><b>Install in LM Studio</b></summary>
|
|
178
|
+
|
|
179
|
+
[](https://lmstudio.ai/install-mcp?name=fetch-url&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsIkBqMGhhbnovZmV0Y2gtdXJsLW1jcEBsYXRlc3QiXX0%3D)
|
|
180
|
+
|
|
181
|
+
Add to `~/.lmstudio/mcp.json` on macOS/Linux or `%USERPROFILE%/.lmstudio/mcp.json` on Windows:
|
|
182
|
+
|
|
183
|
+
```json
|
|
184
|
+
{
|
|
185
|
+
"mcpServers": {
|
|
186
|
+
"fetch-url-mcp": {
|
|
187
|
+
"command": "npx",
|
|
188
|
+
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
189
|
+
}
|
|
190
|
+
}
|
|
191
|
+
}
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
For more info, see [LM Studio MCP docs](https://lmstudio.ai/docs/basics/mcp).
|
|
195
|
+
|
|
196
|
+
</details>
|
|
197
|
+
|
|
198
|
+
<details>
|
|
199
|
+
<summary><b>Install in Claude Desktop</b></summary>
|
|
200
|
+
|
|
201
|
+
Add to `claude_desktop_config.json`:
|
|
202
|
+
|
|
203
|
+
```json
|
|
204
|
+
{
|
|
205
|
+
"mcpServers": {
|
|
206
|
+
"fetch-url-mcp": {
|
|
207
|
+
"command": "npx",
|
|
208
|
+
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
209
|
+
}
|
|
210
|
+
}
|
|
211
|
+
}
|
|
212
|
+
```
|
|
213
|
+
|
|
214
|
+
For more info, see [Claude Desktop MCP docs](https://modelcontextprotocol.io/quickstart/user).
|
|
215
|
+
|
|
216
|
+
</details>
|
|
217
|
+
|
|
218
|
+
<details>
|
|
219
|
+
<summary><b>Install in Claude Code</b></summary>
|
|
220
|
+
|
|
221
|
+
Use the CLI:
|
|
222
|
+
|
|
223
|
+
```sh
|
|
224
|
+
claude mcp add fetch-url-mcp -- npx -y @j0hanz/fetch-url-mcp@latest
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
For project-scoped config, Claude Code writes `.mcp.json` with:
|
|
228
|
+
|
|
229
|
+
```json
|
|
230
|
+
{
|
|
231
|
+
"mcpServers": {
|
|
232
|
+
"fetch-url-mcp": {
|
|
233
|
+
"command": "npx",
|
|
234
|
+
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"],
|
|
235
|
+
"env": {}
|
|
236
|
+
}
|
|
237
|
+
}
|
|
238
|
+
}
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+
For more info, see [Claude Code MCP docs](https://docs.anthropic.com/en/docs/claude-code/mcp).
|
|
242
|
+
|
|
243
|
+
</details>
|
|
244
|
+
|
|
245
|
+
<details>
|
|
246
|
+
<summary><b>Install in Windsurf</b></summary>
|
|
247
|
+
|
|
248
|
+
Add to `~/.codeium/windsurf/mcp_config.json`:
|
|
249
|
+
|
|
250
|
+
```json
|
|
251
|
+
{
|
|
252
|
+
"mcpServers": {
|
|
253
|
+
"fetch-url-mcp": {
|
|
254
|
+
"command": "npx",
|
|
255
|
+
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
256
|
+
}
|
|
257
|
+
}
|
|
258
|
+
}
|
|
259
|
+
```
|
|
260
|
+
|
|
261
|
+
For more info, see [Windsurf MCP docs](https://docs.windsurf.com/windsurf/cascade/mcp).
|
|
262
|
+
|
|
263
|
+
</details>
|
|
264
|
+
|
|
265
|
+
<details>
|
|
266
|
+
<summary><b>Install in Amp</b></summary>
|
|
267
|
+
|
|
268
|
+
Add to `~/.config/amp/settings.json` on macOS/Linux, `%USERPROFILE%\.config\amp\settings.json` on Windows, or `.amp/settings.json` for workspace-scoped config:
|
|
269
|
+
|
|
270
|
+
```json
|
|
271
|
+
{
|
|
272
|
+
"amp.mcpServers": {
|
|
273
|
+
"fetch-url-mcp": {
|
|
274
|
+
"command": "npx",
|
|
275
|
+
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
276
|
+
}
|
|
277
|
+
}
|
|
278
|
+
}
|
|
279
|
+
```
|
|
280
|
+
|
|
281
|
+
Or install via CLI:
|
|
282
|
+
|
|
283
|
+
```sh
|
|
284
|
+
amp mcp add fetch-url-mcp -- npx -y @j0hanz/fetch-url-mcp@latest
|
|
285
|
+
```
|
|
286
|
+
|
|
287
|
+
For more info, see [Amp docs](https://ampcode.com/manual).
|
|
288
|
+
|
|
289
|
+
</details>
|
|
290
|
+
|
|
291
|
+
<details>
|
|
292
|
+
<summary><b>Install in Cline</b></summary>
|
|
293
|
+
|
|
294
|
+
Open the MCP Servers panel, choose `Configure MCP Servers`, and add this to `cline_mcp_settings.json`:
|
|
295
|
+
|
|
296
|
+
```json
|
|
297
|
+
{
|
|
298
|
+
"mcpServers": {
|
|
299
|
+
"fetch-url-mcp": {
|
|
300
|
+
"command": "npx",
|
|
301
|
+
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
302
|
+
}
|
|
303
|
+
}
|
|
304
|
+
}
|
|
305
|
+
```
|
|
306
|
+
|
|
307
|
+
For more info, see [Cline MCP docs](https://docs.cline.bot/mcp/configuring-mcp-servers).
|
|
308
|
+
|
|
309
|
+
</details>
|
|
310
|
+
|
|
311
|
+
<details>
|
|
312
|
+
<summary><b>Install in Codex CLI</b></summary>
|
|
313
|
+
|
|
314
|
+
Use the CLI:
|
|
315
|
+
|
|
316
|
+
```sh
|
|
317
|
+
codex mcp add fetch-url-mcp -- npx -y @j0hanz/fetch-url-mcp@latest
|
|
318
|
+
```
|
|
319
|
+
|
|
320
|
+
Or add this to `~/.codex/config.toml` or project-scoped `.codex/config.toml`:
|
|
321
|
+
|
|
322
|
+
```toml
|
|
323
|
+
[mcp_servers.fetch-url-mcp]
|
|
324
|
+
command = "npx"
|
|
325
|
+
args = ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
326
|
+
```
|
|
327
|
+
|
|
328
|
+
For more info, see [Codex MCP docs](https://developers.openai.com/codex/mcp/).
|
|
329
|
+
|
|
330
|
+
</details>
|
|
331
|
+
|
|
332
|
+
<details>
|
|
333
|
+
<summary><b>Install in GitHub Copilot</b></summary>
|
|
334
|
+
|
|
335
|
+
Add to `.vscode/mcp.json`:
|
|
336
|
+
|
|
337
|
+
```json
|
|
338
|
+
{
|
|
339
|
+
"servers": {
|
|
340
|
+
"fetch-url-mcp": {
|
|
341
|
+
"type": "stdio",
|
|
342
|
+
"command": "npx",
|
|
343
|
+
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
344
|
+
}
|
|
345
|
+
}
|
|
346
|
+
}
|
|
347
|
+
```
|
|
348
|
+
|
|
349
|
+
For more info, see [GitHub Copilot MCP docs](https://code.visualstudio.com/docs/copilot/chat/mcp-servers).
|
|
350
|
+
|
|
351
|
+
</details>
|
|
352
|
+
|
|
353
|
+
<details>
|
|
354
|
+
<summary><b>Install in Warp</b></summary>
|
|
355
|
+
|
|
356
|
+
Open `Personal > MCP Servers` in Warp, choose `+ Add`, and either add a CLI server with:
|
|
357
|
+
|
|
358
|
+
- `command`: `npx`
|
|
359
|
+
- `args`: `["-y", "@j0hanz/fetch-url-mcp@latest"]`
|
|
360
|
+
|
|
361
|
+
Or paste this JSON snippet when using Warp's multi-server import flow:
|
|
362
|
+
|
|
363
|
+
```json
|
|
364
|
+
{
|
|
365
|
+
"mcpServers": {
|
|
366
|
+
"fetch-url-mcp": {
|
|
367
|
+
"command": "npx",
|
|
368
|
+
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
369
|
+
}
|
|
370
|
+
}
|
|
371
|
+
}
|
|
372
|
+
```
|
|
373
|
+
|
|
374
|
+
For more info, see [Warp MCP docs](https://docs.warp.dev/features/warp-ai/mcp).
|
|
375
|
+
|
|
376
|
+
</details>
|
|
377
|
+
|
|
378
|
+
<details>
|
|
379
|
+
<summary><b>Install in Kiro</b></summary>
|
|
380
|
+
|
|
381
|
+
Use Kiro's MCP Servers panel or the `Add to Kiro` install flow. Kiro stores workspace-scoped MCP config in `.kiro/settings/mcp.json` and user-scoped config in `~/.kiro/settings/mcp.json`.
|
|
382
|
+
|
|
383
|
+
For this server, use:
|
|
384
|
+
|
|
385
|
+
- `command`: `npx`
|
|
386
|
+
- `args`: `["-y", "@j0hanz/fetch-url-mcp@latest"]`
|
|
387
|
+
|
|
388
|
+
For more info, see [Kiro MCP docs](https://kiro.dev/blog/unlock-your-development-productivity-with-kiro-and-mcp/).
|
|
389
|
+
|
|
390
|
+
</details>
|
|
391
|
+
|
|
392
|
+
<details>
|
|
393
|
+
<summary><b>Install in Gemini CLI</b></summary>
|
|
394
|
+
|
|
395
|
+
Add to `~/.gemini/settings.json`:
|
|
396
|
+
|
|
397
|
+
```json
|
|
398
|
+
{
|
|
399
|
+
"mcpServers": {
|
|
400
|
+
"fetch-url-mcp": {
|
|
401
|
+
"command": "npx",
|
|
402
|
+
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
403
|
+
}
|
|
404
|
+
}
|
|
405
|
+
}
|
|
406
|
+
```
|
|
407
|
+
|
|
408
|
+
For more info, see [Gemini CLI MCP docs](https://google-gemini.github.io/gemini-cli/docs/tools/mcp-server.html).
|
|
409
|
+
|
|
410
|
+
</details>
|
|
411
|
+
|
|
412
|
+
<details>
|
|
413
|
+
<summary><b>Install in Zed</b></summary>
|
|
414
|
+
|
|
415
|
+
Add to `~/.config/zed/settings.json`:
|
|
416
|
+
|
|
417
|
+
```json
|
|
418
|
+
{
|
|
419
|
+
"context_servers": {
|
|
420
|
+
"fetch-url-mcp": {
|
|
421
|
+
"command": "npx",
|
|
422
|
+
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"],
|
|
423
|
+
"env": {}
|
|
424
|
+
}
|
|
425
|
+
}
|
|
426
|
+
}
|
|
427
|
+
```
|
|
428
|
+
|
|
429
|
+
For more info, see [Zed MCP docs](https://zed.dev/docs/ai/mcp).
|
|
430
|
+
|
|
431
|
+
</details>
|
|
432
|
+
|
|
433
|
+
<details>
|
|
434
|
+
<summary><b>Install in Augment</b></summary>
|
|
435
|
+
|
|
436
|
+
Use the Augment Settings panel and either add the server manually or choose `Import from JSON`:
|
|
437
|
+
|
|
438
|
+
```json
|
|
439
|
+
{
|
|
440
|
+
"mcpServers": {
|
|
441
|
+
"fetch-url-mcp": {
|
|
442
|
+
"command": "npx",
|
|
443
|
+
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
|
|
444
|
+
}
|
|
445
|
+
}
|
|
446
|
+
}
|
|
447
|
+
```
|
|
448
|
+
|
|
449
|
+
For more info, see [Augment MCP docs](https://docs.augmentcode.com/setup-augment/mcp).
|
|
450
|
+
|
|
451
|
+
</details>
|
|
452
|
+
|
|
453
|
+
<details>
|
|
454
|
+
<summary><b>Install in Roo Code</b></summary>
|
|
455
|
+
|
|
456
|
+
Use Roo Code's MCP Servers UI or marketplace flow.
|
|
457
|
+
|
|
458
|
+
For this server, use:
|
|
459
|
+
|
|
460
|
+
- `command`: `npx`
|
|
461
|
+
- `args`: `["-y", "@j0hanz/fetch-url-mcp@latest"]`
|
|
462
|
+
|
|
463
|
+
For more info, see [Roo Code docs](https://docs.roocode.com/).
|
|
464
|
+
|
|
465
|
+
</details>
|
|
466
|
+
|
|
467
|
+
<details>
|
|
468
|
+
<summary><b>Install in Kilo Code</b></summary>
|
|
469
|
+
|
|
470
|
+
Use Kilo Code's MCP Servers UI or marketplace flow.
|
|
471
|
+
|
|
472
|
+
For this server, use:
|
|
473
|
+
|
|
474
|
+
- `command`: `npx`
|
|
475
|
+
- `args`: `["-y", "@j0hanz/fetch-url-mcp@latest"]`
|
|
476
|
+
|
|
477
|
+
For more info, see [Kilo Code docs](https://kilocode.ai/docs).
|
|
478
|
+
|
|
479
|
+
</details>
|
|
480
|
+
|
|
481
|
+
## Use Cases
|
|
482
|
+
|
|
483
|
+
- **Documentation for LLMs** — Grab a docs page, blog post, or reference article as Markdown and pass it straight into a context window.
|
|
484
|
+
- **Repository content** — Hand it a GitHub, GitLab, or Bitbucket URL and it resolves the raw content endpoint. Works with Gists too.
|
|
485
|
+
- **Slow or large pages** — Task mode lets big fetches run in the background while sending monotonic progress updates back to the client, while `tasks/get` exposes the latest task summary fields such as `statusMessage`, `createdAt`, `lastUpdatedAt`, `ttl`, and `pollInterval`.
|
|
486
|
+
|
|
487
|
+
## Architecture
|
|
488
|
+
|
|
489
|
+
```text
|
|
490
|
+
[MCP Client]
|
|
491
|
+
├─ stdio -> `src/index.ts` -> `startStdioServer()` -> `createMcpServer()`
|
|
492
|
+
└─ HTTP (`--http`) -> `src/index.ts` -> `startHttpServer()` -> HTTP dispatcher
|
|
493
|
+
├─ `GET /health`
|
|
494
|
+
├─ `GET /.well-known/oauth-protected-resource`
|
|
495
|
+
├─ `GET /.well-known/oauth-protected-resource/mcp`
|
|
496
|
+
└─ `POST|GET|DELETE /mcp`
|
|
497
|
+
|
|
498
|
+
`createMcpServer()`
|
|
499
|
+
├─ registers tool: `fetch-url`
|
|
500
|
+
├─ registers prompt: `get-help`
|
|
501
|
+
├─ registers resources:
|
|
502
|
+
│ - `internal://instructions`
|
|
503
|
+
├─ enables capabilities: logging, resources, prompts, tasks
|
|
504
|
+
└─ installs task handlers, log-level handling, and shutdown cleanup
|
|
505
|
+
|
|
506
|
+
`fetch-url` execution
|
|
507
|
+
├─ validate input with `fetchUrlInputSchema`
|
|
508
|
+
├─ normalize URL and block local/private targets unless allowed
|
|
509
|
+
├─ rewrite supported code-host URLs to raw endpoints when possible
|
|
510
|
+
├─ fetch content via the shared pipeline
|
|
511
|
+
├─ transform HTML into Markdown in the transform worker path
|
|
512
|
+
└─ validate `structuredContent` with `fetchUrlOutputSchema`
|
|
513
|
+
```
|
|
514
|
+
|
|
515
|
+
### Request Lifecycle
|
|
516
|
+
|
|
517
|
+
```text
|
|
518
|
+
[Client] -- initialize {protocolVersion, capabilities} --> [Server]
|
|
519
|
+
[Server] -- {protocolVersion, capabilities, serverInfo} --> [Client]
|
|
520
|
+
[Client] -- notifications/initialized --> [Server]
|
|
521
|
+
[Client] -- tools/call {name, arguments} --> [Server]
|
|
522
|
+
[Server] -- {content: [{type, text}], structuredContent?, isError?} --> [Client]
|
|
523
|
+
```
|
|
524
|
+
|
|
525
|
+
## MCP Surface
|
|
526
|
+
|
|
527
|
+
### Tools
|
|
528
|
+
|
|
529
|
+
#### `fetch-url`
|
|
530
|
+
|
|
531
|
+
Takes a URL and returns Markdown. Read-only, with no JavaScript execution. Supports running as a background MCP task for large or slow pages. In task mode, `tasks/get` and `tasks/list` expose task summaries including `status`, `statusMessage`, `createdAt`, `lastUpdatedAt`, `ttl`, and `pollInterval`; numeric progress remains out-of-band via `notifications/progress`.
|
|
532
|
+
|
|
533
|
+
| Parameter | Type | Required | Description |
|
|
534
|
+
| --------- | -------- | -------- | --------------------------- |
|
|
535
|
+
| `url` | `string` | yes | Target URL. Max 2048 chars. |
|
|
536
|
+
|
|
537
|
+
You get text content back by default. If output validation passes, the response also includes `structuredContent` with typed fields: `url`, `resolvedUrl`, `finalUrl`, `title`, `metadata`, `markdown`, `fetchedAt`, `contentSize`, and `truncated`. A `true` value for `truncated` means the content hit a server-side size limit.
|
|
538
|
+
|
|
539
|
+
To opt into progress updates, include `_meta.progressToken` in the tool call. The token may be a string or number, and the server may emit monotonic `notifications/progress` updates while the fetch runs.
|
|
540
|
+
|
|
541
|
+
To run the tool in task mode, include `params.task = { ttl?: <ms> }`. `tasks/result` waits until the task reaches a terminal status and then returns the stored output or a terminal error payload for cancelled or failed tasks. Task summaries and final results include `_meta["io.modelcontextprotocol/related-task"] = { "taskId": "<server-task-id>" }`.
|
|
542
|
+
|
|
543
|
+
```json
|
|
544
|
+
{
|
|
545
|
+
"method": "tools/call",
|
|
546
|
+
"params": {
|
|
547
|
+
"name": "fetch-url",
|
|
548
|
+
"arguments": {
|
|
549
|
+
"url": "https://example.com/docs"
|
|
550
|
+
},
|
|
551
|
+
"task": {
|
|
552
|
+
"ttl": 300000,
|
|
553
|
+
"pollInterval": 1000
|
|
554
|
+
},
|
|
555
|
+
"_meta": {
|
|
556
|
+
"progressToken": 7
|
|
557
|
+
}
|
|
558
|
+
}
|
|
559
|
+
}
|
|
560
|
+
```
|
|
561
|
+
|
|
562
|
+
```text
|
|
563
|
+
1. [Client] -- tools/call {name: "fetch-url", arguments} --> [Server]
|
|
564
|
+
2. [Server] -- dispatch("fetch-url") --> [src/tools/fetch-url.ts]
|
|
565
|
+
3. [Handler] -- validate(fetchUrlInputSchema) --> normalize / fetch / transform
|
|
566
|
+
4. [Handler] -- validate(fetchUrlOutputSchema) --> assemble content + structuredContent
|
|
567
|
+
5. [Server] -- result or tool error --> [Client]
|
|
568
|
+
```
|
|
569
|
+
|
|
570
|
+
### Resources
|
|
571
|
+
|
|
572
|
+
| Resource | URI | MIME Type | Description |
|
|
573
|
+
| ---------------------------- | ------------------------- | --------------- | -------------------------------------------- |
|
|
574
|
+
| `fetch-url-mcp-instructions` | `internal://instructions` | `text/markdown` | Guidance for using the Fetch URL MCP server. |
|
|
575
|
+
|
|
576
|
+
### Prompts
|
|
577
|
+
|
|
578
|
+
| Prompt | Arguments | Description |
|
|
579
|
+
| ---------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
580
|
+
| `get-help` | `topic?` | Return Fetch URL server instructions: workflows, task mode, and error handling. Optional values: `capabilities`, `workflows`, `constraints`, `errors`. |
|
|
581
|
+
|
|
582
|
+
## MCP Capabilities
|
|
583
|
+
|
|
584
|
+
| Capability | Status | Notes |
|
|
585
|
+
| ---------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------- |
|
|
586
|
+
| completions | confirmed | Advertised in `createServerCapabilities()`. |
|
|
587
|
+
| logging | confirmed | Advertised in `createServerCapabilities()`. |
|
|
588
|
+
| resources | confirmed | Static instruction resource is registered during server startup. Subscription and list-changed support are not advertised. |
|
|
589
|
+
| prompts | confirmed | `get-help` is registered during server startup. |
|
|
590
|
+
| tasks | confirmed | Advertised in `createServerCapabilities()` and backed by registered task handlers plus optional tool task support. |
|
|
591
|
+
| progress notifications | confirmed | Opt-in via `_meta.progressToken`. Tool execution reports monotonic `notifications/progress` updates during fetch and transform stages. |
|
|
592
|
+
| task status updates | confirmed | `notifications/tasks/status` is emitted when task status changes and `TASKS_STATUS_NOTIFICATIONS=true`. |
|
|
593
|
+
|
|
594
|
+
### Tool Annotations
|
|
595
|
+
|
|
596
|
+
| Annotation | Value |
|
|
597
|
+
| ----------------- | ------- |
|
|
598
|
+
| `readOnlyHint` | `true` |
|
|
599
|
+
| `destructiveHint` | `false` |
|
|
600
|
+
| `idempotentHint` | `true` |
|
|
601
|
+
| `openWorldHint` | `true` |
|
|
602
|
+
|
|
603
|
+
### Structured Output
|
|
604
|
+
|
|
605
|
+
The tool declares an `outputSchema` and includes `structuredContent` in the response when validation passes. Clients that support structured output get typed data directly; the rest use the text fallback.
|
|
606
|
+
|
|
607
|
+
## Configuration
|
|
608
|
+
|
|
609
|
+
All configuration is through environment variables. For basic stdio usage, nothing needs to be set.
|
|
610
|
+
|
|
611
|
+
### HTTP Server
|
|
612
|
+
|
|
613
|
+
| Variable | Default | Notes |
|
|
614
|
+
| ------------------------------------- | ----------- | ---------------------------------------------------------------------- |
|
|
615
|
+
| `HOST` | `127.0.0.1` | Bind address. Non-loopback bindings also require `ALLOW_REMOTE=true`. |
|
|
616
|
+
| `PORT` | `3000` | Listening port for `--http`. |
|
|
617
|
+
| `ALLOW_REMOTE` | `false` | Must be enabled to bind to a non-loopback interface. |
|
|
618
|
+
| `ALLOWED_HOSTS` | empty | Additional allowed `Host` and `Origin` values. |
|
|
619
|
+
| `SERVER_MAX_CONNECTIONS` | `0` | Optional connection cap. |
|
|
620
|
+
| `SERVER_TRUST_PROXY` | `false` | Trust `X-Forwarded-For` / `Forwarded` for client IP resolution. |
|
|
621
|
+
| `SERVER_HEADERS_TIMEOUT_MS` | unset | Optional Node server tuning. |
|
|
622
|
+
| `SERVER_REQUEST_TIMEOUT_MS` | unset | Optional Node server tuning. |
|
|
623
|
+
| `SERVER_KEEP_ALIVE_TIMEOUT_MS` | unset | Optional keep-alive tuning. |
|
|
624
|
+
| `SERVER_KEEP_ALIVE_TIMEOUT_BUFFER_MS` | unset | Optional keep-alive tuning buffer. |
|
|
625
|
+
| `SERVER_MAX_HEADERS_COUNT` | unset | Optional header count limit. |
|
|
626
|
+
| `SERVER_BLOCK_PRIVATE_CONNECTIONS` | `false` | Enables inbound private-network protections when not trusting a proxy. |
|
|
627
|
+
|
|
628
|
+
### Authentication & OAuth
|
|
629
|
+
|
|
630
|
+
| Variable | Default | Notes |
|
|
631
|
+
| ------------------------- | ------- | ----------------------------------------------------------- |
|
|
632
|
+
| `ACCESS_TOKENS` | unset | Comma- or space-separated static bearer tokens. |
|
|
633
|
+
| `API_KEY` | unset | Alternate static token source for header auth. |
|
|
634
|
+
| `OAUTH_ISSUER_URL` | unset | Enables OAuth mode when combined with the other OAuth URLs. |
|
|
635
|
+
| `OAUTH_AUTHORIZATION_URL` | unset | Optional explicit authorization endpoint. |
|
|
636
|
+
| `OAUTH_TOKEN_URL` | unset | Optional explicit token endpoint. |
|
|
637
|
+
| `OAUTH_REVOCATION_URL` | unset | Optional OAuth revocation endpoint. |
|
|
638
|
+
| `OAUTH_REGISTRATION_URL` | unset | Optional OAuth dynamic client registration endpoint. |
|
|
639
|
+
| `OAUTH_INTROSPECTION_URL` | unset | Required for OAuth token introspection. |
|
|
640
|
+
| `OAUTH_REQUIRED_SCOPES` | empty | Required scopes enforced after auth. |
|
|
641
|
+
| `OAUTH_CLIENT_ID` | unset | Optional introspection client ID. |
|
|
642
|
+
| `OAUTH_CLIENT_SECRET` | unset | Optional introspection client secret. |
|
|
643
|
+
|
|
644
|
+
### TLS
|
|
645
|
+
|
|
646
|
+
| Variable | Default | Notes |
|
|
647
|
+
| ---------------------- | ------- | ----------------------------------------------------------- |
|
|
648
|
+
| `SERVER_TLS_KEY_FILE` | unset | Enable HTTPS when set together with `SERVER_TLS_CERT_FILE`. |
|
|
649
|
+
| `SERVER_TLS_CERT_FILE` | unset | TLS certificate path. |
|
|
650
|
+
| `SERVER_TLS_CA_FILE` | unset | Optional custom CA bundle. |
|
|
651
|
+
|
|
652
|
+
### Fetching
|
|
653
|
+
|
|
654
|
+
| Variable | Default | Notes |
|
|
655
|
+
| ------------------- | ------------------------- | -------------------------------------------------- |
|
|
656
|
+
| `ALLOW_LOCAL_FETCH` | `false` | Allows loopback and private-network fetch targets. |
|
|
657
|
+
| `FETCH_TIMEOUT_MS` | `15000` | Network fetch timeout in milliseconds. |
|
|
658
|
+
| `USER_AGENT` | `fetch-url-mcp/<version>` | Override the outbound user agent string. |
|
|
659
|
+
|
|
660
|
+
### Tool Output
|
|
661
|
+
|
|
662
|
+
| Variable | Default | Notes |
|
|
663
|
+
| -------------------------- | ------- | ---------------------------------------------- |
|
|
664
|
+
| `MAX_INLINE_CONTENT_CHARS` | `0` | `0` means no explicit inline truncation limit. |
|
|
665
|
+
|
|
666
|
+
### Tasks
|
|
667
|
+
|
|
668
|
+
| Variable | Default | Notes |
|
|
669
|
+
| ---------------------------- | ------- | ------------------------------------------------------------------------------------ |
|
|
670
|
+
| `TASKS_MAX_TOTAL` | `5000` | Total retained task capacity, including completed/cancelled tasks until they expire. |
|
|
671
|
+
| `TASKS_MAX_PER_OWNER` | `1000` | Per-owner retained task cap, clamped to the total cap. |
|
|
672
|
+
| `TASKS_STATUS_NOTIFICATIONS` | `false` | Enables status notifications for tasks. |
|
|
673
|
+
| `TASKS_REQUIRE_INTERCEPTION` | `true` | Requires interception for task-capable tool execution. |
|
|
674
|
+
|
|
675
|
+
### Transform Workers
|
|
676
|
+
|
|
677
|
+
| Variable | Default | Notes |
|
|
678
|
+
| ------------------------------------------ | --------- | ------------------------------------- |
|
|
679
|
+
| `TRANSFORM_CANCEL_ACK_TIMEOUT_MS` | `200` | Cancellation acknowledgement timeout. |
|
|
680
|
+
| `TRANSFORM_WORKER_MODE` | `threads` | Worker execution mode. |
|
|
681
|
+
| `TRANSFORM_WORKER_MAX_OLD_GENERATION_MB` | unset | Optional worker memory limit. |
|
|
682
|
+
| `TRANSFORM_WORKER_MAX_YOUNG_GENERATION_MB` | unset | Optional worker memory limit. |
|
|
683
|
+
| `TRANSFORM_WORKER_CODE_RANGE_MB` | unset | Optional worker memory limit. |
|
|
684
|
+
| `TRANSFORM_WORKER_STACK_MB` | unset | Optional worker stack size. |
|
|
685
|
+
|
|
686
|
+
### Content Cleanup
|
|
687
|
+
|
|
688
|
+
| Variable | Default | Notes |
|
|
689
|
+
| ------------------------------------- | -------------- | ------------------------------------------ |
|
|
690
|
+
| `FETCH_URL_MCP_EXTRA_NOISE_TOKENS` | empty | Extra noise-removal tokens. |
|
|
691
|
+
| `FETCH_URL_MCP_EXTRA_NOISE_SELECTORS` | empty | Extra DOM selectors for noise removal. |
|
|
692
|
+
| `FETCH_URL_MCP_LOCALE` | system default | Locale override for extraction heuristics. |
|
|
693
|
+
| `MARKDOWN_HEADING_KEYWORDS` | built-in list | Override heading keywords used by cleanup. |
|
|
694
|
+
|
|
695
|
+
### Logging
|
|
696
|
+
|
|
697
|
+
| Variable | Default | Notes |
|
|
698
|
+
| ------------ | ------- | ------------------------------------ |
|
|
699
|
+
| `LOG_LEVEL` | `info` | `debug`, `info`, `warn`, or `error`. |
|
|
700
|
+
| `LOG_FORMAT` | `text` | Set to `json` for structured logs. |
|
|
701
|
+
|
|
702
|
+
## HTTP Endpoints
|
|
703
|
+
|
|
704
|
+
| Method | Path | Auth | Purpose |
|
|
705
|
+
| -------- | ------------------------------------------- | ------------------------------------------ | ------------------------------------------------------- |
|
|
706
|
+
| `GET` | `/health` | no, unless `?verbose=1` on a remote server | Basic health response, with optional diagnostics. |
|
|
707
|
+
| `GET` | `/.well-known/oauth-protected-resource` | no | OAuth protected-resource metadata. |
|
|
708
|
+
| `GET` | `/.well-known/oauth-protected-resource/mcp` | no | OAuth protected-resource metadata for the MCP endpoint. |
|
|
709
|
+
| `POST` | `/mcp` | yes | Session initialization and JSON-RPC requests. |
|
|
710
|
+
| `GET` | `/mcp` | yes | Session-bound server-to-client stream handling. |
|
|
711
|
+
| `DELETE` | `/mcp` | yes | Session shutdown. |
|
|
712
|
+
|
|
713
|
+
## Security
|
|
714
|
+
|
|
715
|
+
| Control | Status | Notes |
|
|
716
|
+
| -------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
717
|
+
| Host and origin validation | implemented | HTTP requests are rejected unless `Host` and `Origin` match the allowlist built from loopback, the configured host, and `ALLOWED_HOSTS`. |
|
|
718
|
+
| Authentication | implemented | HTTP mode supports static bearer tokens locally or OAuth token introspection; remote bindings require OAuth. |
|
|
719
|
+
| Protocol version checks | implemented | Session-bound MCP HTTP requests validate `MCP-Protocol-Version` and pin it to the negotiated session version. |
|
|
720
|
+
| Rate limiting | implemented | Requests pass through the HTTP rate limiter before route dispatch. Enable `SERVER_TRUST_PROXY=true` behind a trusted reverse proxy so limits apply to the forwarded client IP instead of the proxy hop. |
|
|
721
|
+
| Outbound SSRF protections | implemented | Local/private IPs, metadata endpoints, and `.local`/`.internal` hosts are blocked unless `ALLOW_LOCAL_FETCH=true`. |
|
|
722
|
+
| TLS | optional | HTTPS is enabled when both TLS key and certificate files are configured. |
|
|
723
|
+
| Stdio logging safety | implemented | Server logs are written to stderr, not stdout, so stdio MCP traffic stays clean. |
|
|
724
|
+
|
|
725
|
+
## Development
|
|
726
|
+
|
|
727
|
+
### Essential Commands
|
|
728
|
+
|
|
729
|
+
| Command | Description |
|
|
730
|
+
| -------------------- | ------------------------------------------------- |
|
|
731
|
+
| `npm run build` | Clean, compile TypeScript, copy assets. |
|
|
732
|
+
| `npm run dev` | Watch mode TypeScript compilation. |
|
|
733
|
+
| `npm run dev:run` | Run the server with `--watch` and `.env` support. |
|
|
734
|
+
| `npm start` | Start the compiled server. |
|
|
735
|
+
| `npm test` | Run the full test suite. |
|
|
736
|
+
| `npm run lint` | Lint with ESLint. |
|
|
737
|
+
| `npm run lint:fix` | Auto-fix lint issues. |
|
|
738
|
+
| `npm run type-check` | Type-check source and tests. |
|
|
739
|
+
| `npm run format` | Format with Prettier. |
|
|
740
|
+
| `npm run inspector` | Build and launch MCP Inspector. |
|
|
741
|
+
|
|
742
|
+
<details>
|
|
743
|
+
<summary><b>All npm scripts</b></summary>
|
|
744
|
+
|
|
745
|
+
| Script | Command |
|
|
746
|
+
| ------------------------ | ------------------------------------------------------------------------------------------------------------------- |
|
|
747
|
+
| `clean` | `node scripts/tasks.mjs clean` |
|
|
748
|
+
| `build` | `node scripts/tasks.mjs build` |
|
|
749
|
+
| `copy:assets` | `node scripts/tasks.mjs copy:assets` |
|
|
750
|
+
| `prepare` | `npm run build` |
|
|
751
|
+
| `dev` | `tsc --watch --preserveWatchOutput` |
|
|
752
|
+
| `dev:run` | `node --env-file=.env --watch dist/index.js` |
|
|
753
|
+
| `start` | `node dist/index.js` |
|
|
754
|
+
| `format` | `prettier --write .` |
|
|
755
|
+
| `type-check` | `node scripts/tasks.mjs type-check` |
|
|
756
|
+
| `type-check:src` | `node node_modules/typescript/bin/tsc -p tsconfig.json --noEmit` |
|
|
757
|
+
| `type-check:tests` | `node node_modules/typescript/bin/tsc -p tsconfig.test.json --noEmit` |
|
|
758
|
+
| `type-check:diagnostics` | `tsc --noEmit --extendedDiagnostics` |
|
|
759
|
+
| `type-check:trace` | `node -e "require('fs').rmSync('.ts-trace',{recursive:true,force:true})" && tsc --noEmit --generateTrace .ts-trace` |
|
|
760
|
+
| `lint` | `eslint .` |
|
|
761
|
+
| `lint:tests` | `eslint src/__tests__` |
|
|
762
|
+
| `lint:fix` | `eslint . --fix` |
|
|
763
|
+
| `test` | `node scripts/tasks.mjs test` |
|
|
764
|
+
| `test:fast` | `node --test --import tsx/esm src/__tests__/**/*.test.ts node-tests/**/*.test.ts` |
|
|
765
|
+
| `test:coverage` | `node scripts/tasks.mjs test --coverage` |
|
|
766
|
+
| `knip` | `knip` |
|
|
767
|
+
| `knip:fix` | `knip --fix` |
|
|
768
|
+
| `inspector` | `npm run build && npx -y @modelcontextprotocol/inspector node dist/index.js --stdio` |
|
|
769
|
+
| `prepublishOnly` | `npm run lint && npm run type-check && npm run build` |
|
|
770
|
+
|
|
771
|
+
</details>
|
|
772
|
+
|
|
773
|
+
## Build and Release
|
|
774
|
+
|
|
775
|
+
- `npm run prepublishOnly` runs lint, type-check, and build as a single release gate.
|
|
776
|
+
- CI workflows are under `.github/workflows/`.
|
|
777
|
+
- `Dockerfile` and `docker-compose.yml` are included for containerized runs.
|
|
778
|
+
- Published on npm as [`@j0hanz/fetch-url-mcp`](https://www.npmjs.com/package/@j0hanz/fetch-url-mcp).
|
|
779
|
+
|
|
780
|
+
## Troubleshooting
|
|
781
|
+
|
|
782
|
+
| Symptom | Likely Cause | Fix |
|
|
783
|
+
| --------------------------------------------------- | ----------------------------------- | ----------------------------------------------------------------------------------- |
|
|
784
|
+
| Server output mixes with MCP traffic on stdio | Logs going to stdout | Ensure all logging writes to stderr; the server does this by default. |
|
|
785
|
+
| HTTP mode returns `403` | Host/origin mismatch | Add the domain to `ALLOWED_HOSTS` or verify loopback bindings. |
|
|
786
|
+
| HTTP mode rate limits every request from your proxy | `SERVER_TRUST_PROXY` not enabled | Enable `SERVER_TRUST_PROXY=true` when the server is behind a trusted reverse proxy. |
|
|
787
|
+
| HTTP mode returns `401` | Missing or invalid token | Set `ACCESS_TOKENS` or configure OAuth env vars for remote bindings. |
|
|
788
|
+
| Fetch returns private-IP error | SSRF protections blocked the target | Set `ALLOW_LOCAL_FETCH=true` if the target is intentionally local. |
|
|
789
|
+
| `truncated: true` in response | Content exceeded inline limits | Increase `MAX_INLINE_CONTENT_CHARS` or accept truncated output. |
|
|
790
|
+
| Transform timeout or worker crash | Large or complex HTML | Tune `TRANSFORM_WORKER_MAX_OLD_GENERATION_MB` or increase `FETCH_TIMEOUT_MS`. |
|
|
791
|
+
| Client config not working | Wrong config format for the client | Check the matching `<details>` block above — config keys vary by client. |
|
|
792
|
+
|
|
793
|
+
## Credits
|
|
794
|
+
|
|
795
|
+
| Dependency | Registry |
|
|
796
|
+
| ------------------------------------------------------------------------------------------ | -------- |
|
|
797
|
+
| [@modelcontextprotocol/server](https://www.npmjs.com/package/@modelcontextprotocol/server) | npm |
|
|
798
|
+
| [@modelcontextprotocol/node](https://www.npmjs.com/package/@modelcontextprotocol/node) | npm |
|
|
799
|
+
| [@mozilla/readability](https://www.npmjs.com/package/@mozilla/readability) | npm |
|
|
800
|
+
| [linkedom](https://www.npmjs.com/package/linkedom) | npm |
|
|
801
|
+
| [node-html-markdown](https://www.npmjs.com/package/node-html-markdown) | npm |
|
|
802
|
+
| [undici](https://www.npmjs.com/package/undici) | npm |
|
|
803
|
+
| [zod](https://www.npmjs.com/package/zod) | npm |
|
|
804
|
+
|
|
805
|
+
## Contributing and License
|
|
806
|
+
|
|
807
|
+
Pull requests welcome. Please make sure these pass before submitting:
|
|
808
|
+
|
|
809
|
+
1. `npm run lint` and `npm run type-check`
|
|
810
|
+
2. `npm test`
|
|
811
|
+
3. `npm run format`
|
|
812
|
+
|
|
813
|
+
## License
|
|
814
|
+
|
|
815
|
+
MIT License. See [LICENSE](LICENSE) for details.
|