@fnnm/fetch 0.1.0

package/CLAUDE.md

@@ -0,0 +1,110 @@
+# CLAUDE.md
+
+Guidance for Claude Code working in this repo.
+
+## Project Overview
+
+`@fnnm/fetch` is a Model Context Protocol (MCP) server that fetches URLs and converts HTML/PDF to Markdown for LLMs. TypeScript/Node port of the original Python `fetch`. Communicates over stdio (JSON-RPC).
+
+**Security:** the server can reach local/internal IP addresses. Deploy with caution.
+
+## Commands
+
+```bash
+npm run build     # tsc → dist/
+npm start         # node dist/index.js
+npm run dev       # ts-node src/index.ts
+npm test          # vitest
+docker build -t mcp/fetch .
+```
+
+No CLI flags. Configuration is environment-only:
+- `HTTPS_PROXY` / `HTTP_PROXY` (plus lowercase variants) — outbound proxy, wired through undici `ProxyAgent`.
+
+## Architecture
+
+```
+src/
+├── index.ts              # entry → runServer()
+├── server.ts             # createServer(): McpServer "fetch" v0.1.0; registers tool + prompt
+├── tools/
+│   ├── fetch.ts          # FetchSchema (zod), handleFetch()
+│   └── fetch.test.ts
+├── utils/
+│   ├── convert.ts        # HTML→MD (jsdom+readability+turndown), PDF→text (pdf-parse)
+│   ├── convert.test.ts
+│   ├── robots.ts         # robots.txt check, dual user-agents
+│   └── robots.test.ts
+└── types.d.ts
+```
+
+### Entry flow
+1. `index.ts` → `runServer()`
+2. `server.ts:runServer()` → `createServer()` builds `McpServer`, connects `StdioServerTransport`.
+
+### Dual-mode operation (tool vs prompt)
+
+| Mode | Trigger | robots.txt | User-Agent | max_length |
+|------|---------|------------|------------|------------|
+| Autonomous | `fetch` tool | checked | `ModelContextProtocol/1.0 (Autonomous; ...)` | caller-supplied (default 5000) |
+| Manual | `fetch` prompt | bypassed | `ModelContextProtocol/1.0 (User-Specified; ...)` | 1,000,000 |
+
+Hardcoded in `server.ts`: the tool calls `handleFetch(args, false)`; the prompt calls `handleFetch({ url, max_length: 1_000_000, start_index: 0, raw: false }, true)`.
+
+### `handleFetch` (`tools/fetch.ts`)
+- 30s `AbortController` timeout around native `fetch`.
+- Proxy: reads `HTTPS_PROXY`/`HTTP_PROXY` env, builds undici `ProxyAgent`, attaches as `dispatcher`.
+- Content routing:
+  - PDF (`application/pdf` header or `.pdf` URL) → `extractContentFromPdf`
+  - HTML (`text/html` or empty content-type) and `raw=false` → `extractContentFromHtml`
+  - Otherwise → raw text with `Content type ... cannot be simplified to markdown` prefix
+- Pagination: `start_index` / `max_length` slice; if truncated, appends a hint carrying the next `start_index`.
+- Errors raised as `McpError(ErrorCode.InternalError, ...)`.
+
+### Content conversion (`utils/convert.ts`)
+- HTML: `JSDOM` → `@mozilla/readability` → `turndown` (atx headings, fenced code). Failure returns a sentinel `<error>...</error>` string rather than throwing.
+- PDF: `pdf-parse`, loaded via `createRequire` (CommonJS interop).
+
+### robots.txt (`utils/robots.ts`)
+- Fetches `/robots.txt` with the chosen UA.
+- `401`/`403` → deny (McpError). Other `4xx` → allow (treat as no robots). Network error → McpError.
+- `robotsParser.isAllowed(...) === false` → deny.
+
+## Dependencies
+
+| Package | Use |
+|---------|-----|
+| `@modelcontextprotocol/sdk` | MCP server, stdio transport, types |
+| `zod` | input schema + validation |
+| `undici` | `ProxyAgent` for proxied fetch |
+| `jsdom`, `@mozilla/readability`, `turndown` | HTML → Markdown |
+| `pdf-parse` | PDF → text |
+| `robots-parser` | robots.txt evaluation |
+| `vitest` | tests |
+
+## Notable differences from the Python original
+
+- No `--user-agent`, `--ignore-robots-txt`, `--proxy-url` CLI flags. Proxy is env-only; user-agent is fixed.
+- No PDF TTLCache (PDF is re-parsed on every call).
+- No `verify_ssl` toggle (native fetch; Node TLS settings apply).
+- Registered MCP server name is `fetch`.
+
+## Running in an MCP client
+
+Build first (`npm run build`), then point the client at the built entry:
+
+```json
+{
+  "mcpServers": {
+    "fetch": { "command": "node", "args": ["/abs/path/to/dist/index.js"] }
+  }
+}
+```
+
+`package.json` declares no `bin` field, so `npx @fnnm/fetch` will not work until one is added and the package is published.
+
+## Extending
+
+- **New content type:** add detection in `handleFetch` (`tools/fetch.ts`), add an extractor in `utils/convert.ts`, return `(text, "")` on success or fall through to the raw prefix path.
+- **Robots behavior:** edit `checkMayAutonomouslyFetchUrl` (`utils/robots.ts`) or the `ignoreRobotsTxt` argument passed in `server.ts`.
+- **New tool/prompt:** define a zod schema, register via `server.tool(...)` / `server.prompt(...)` in `createServer()`.

package/Dockerfile

@@ -0,0 +1,31 @@
+# Generated by MCP Server Migration
+FROM node:18-alpine AS builder
+
+WORKDIR /app
+
+# Install dependencies
+COPY package.json package-lock.json ./
+RUN npm ci
+
+# Copy source
+COPY tsconfig.json ./
+COPY src ./src
+
+# Build
+RUN npm run build
+
+# Production image
+FROM node:18-alpine AS release
+
+WORKDIR /app
+
+# Copy built assets and production deps
+COPY --from=builder /app/dist ./dist
+COPY --from=builder /app/package.json ./package.json
+COPY --from=builder /app/package-lock.json ./package-lock.json
+
+# Install only production dependencies
+RUN npm ci --omit=dev
+
+# Entry point
+ENTRYPOINT ["node", "dist/index.js"]

package/LICENSE

@@ -0,0 +1,7 @@
+Copyright (c) 2024 Anthropic, PBC.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

package/README.md

@@ -0,0 +1,160 @@
+# Fetch MCP Server
+
+A Model Context Protocol (MCP) server that fetches URLs and converts HTML/PDF to Markdown for LLMs. TypeScript/Node port of the original `mcp-server-fetch2`, communicating over stdio.
+
+> [!CAUTION]
+> This server can access local/internal IP addresses and may represent a security risk. Exercise caution to ensure it does not expose sensitive data.
+
+The fetch tool truncates the response; use the `start_index` argument to page through content in chunks until the model finds what it needs.
+
+### Available Tools
+
+- `fetch` — Fetches a URL and extracts its contents as markdown (HTML) or text (PDF).
+    - `url` (string, required): URL to fetch
+    - `max_length` (integer, optional): Maximum characters to return (default: 5000)
+    - `start_index` (integer, optional): Start content from this character index (default: 0)
+    - `raw` (boolean, optional): Return raw content without markdown conversion (default: false)
+
+### Prompts
+
+- **fetch** — Fetch a URL and extract its contents as markdown.
+  - Arguments:
+    - `url` (string, required): URL to fetch
+
+## Installation
+
+Requires Node.js 18+.
+
+```bash
+git clone https://github.com/fxcl/fetch.git
+cd fetch
+npm install
+npm run build
+```
+
+The server runs as `node dist/index.js`. `package.json` declares a `bin` field, so once published `npx @fnnm/fetch` will also work.
+
+## Configuration
+
+### Configure for Claude.app / Claude Desktop
+
+Add to your Claude settings:
+
+<details>
+<summary>Using node (built from source)</summary>
+
+```json
+{
+  "mcpServers": {
+    "fetch": {
+      "command": "node",
+      "args": ["/abs/path/to/fetch/dist/index.js"]
+    }
+  }
+}
+```
+</details>
+
+<details>
+<summary>Using docker</summary>
+
+```json
+{
+  "mcpServers": {
+    "fetch": {
+      "command": "docker",
+      "args": ["run", "-i", "--rm", "mcp/fetch"]
+    }
+  }
+}
+```
+</details>
+
+### Configure for VS Code
+
+Add the following to `.vscode/mcp.json` in your workspace, or to your User Settings (JSON) via `Ctrl+Shift+P` → `Preferences: Open User Settings (JSON)`. The `mcp` key is required when using `mcp.json`.
+
+<details>
+<summary>Using node</summary>
+
+```json
+{
+  "mcp": {
+    "servers": {
+      "fetch": {
+        "command": "node",
+        "args": ["/abs/path/to/fetch/dist/index.js"]
+      }
+    }
+  }
+}
+```
+</details>
+
+<details>
+<summary>Using Docker</summary>
+
+```json
+{
+  "mcp": {
+    "servers": {
+      "fetch": {
+        "command": "docker",
+        "args": ["run", "-i", "--rm", "mcp/fetch"]
+      }
+    }
+  }
+}
+```
+</details>
+
+### Behavior — robots.txt
+
+The server obeys a site's robots.txt when the request comes from the model (via the `fetch` tool), but bypasses it when the request is user-initiated (via the `fetch` prompt). This behavior is fixed in the implementation; there is no CLI flag to toggle it.
+
+### Behavior — User-agent
+
+The server sends one of two fixed user-agents depending on whether the request is model-initiated (tool) or user-initiated (prompt):
+
+```
+ModelContextProtocol/1.0 (Autonomous; +https://github.com/modelcontextprotocol/servers)
+```
+```
+ModelContextProtocol/1.0 (User-Specified; +https://github.com/modelcontextprotocol/servers)
+```
+
+The user-agent is not configurable.
+
+### Behavior — Proxy
+
+The server honors the `HTTPS_PROXY` / `HTTP_PROXY` environment variables (lowercase variants too), wired through an undici `ProxyAgent`. To set a proxy in `mcpServers`, add an `env` block:
+
+```json
+{
+  "mcpServers": {
+    "fetch": {
+      "command": "node",
+      "args": ["/abs/path/to/fetch/dist/index.js"],
+      "env": {
+        "HTTPS_PROXY": "http://user:pass@proxy.example.com:8080"
+      }
+    }
+  }
+}
+```
+
+## Debugging
+
+Use the MCP inspector:
+
+```bash
+npx @modelcontextprotocol/inspector node dist/index.js
+```
+
+## Contributing
+
+Pull requests welcome. For examples of other MCP servers and implementation patterns, see https://github.com/modelcontextprotocol/servers.
+
+## License
+
+MIT. See [LICENSE](LICENSE).

package/dist/index.js

@@ -0,0 +1,6 @@
+#!/usr/bin/env node
+import { runServer } from './server.js';
+runServer().catch((error) => {
+    console.error("Fatal error running server:", error);
+    process.exit(1);
+});

package/dist/manual-test.js

@@ -0,0 +1,28 @@
+import { handleFetch } from './tools/fetch.js';
+async function testFetch(url, raw = false) {
+    console.log(`\n--- Fetching ${url} (Raw: ${raw}) ---`);
+    try {
+        const result = await handleFetch({
+            url,
+            max_length: 5000,
+            start_index: 0,
+            raw
+        }, true); // ignoreRobotsTxt=true for manual testing to ensure we get content
+        if (result.isError) {
+            console.error("Error result:", result);
+        }
+        else {
+            console.log("Success! Content preview:");
+            console.log(result.content[0].text.substring(0, 500) + "...");
+        }
+    }
+    catch (error) {
+        console.error("Exception:", error.message);
+    }
+}
+async function run() {
+    await testFetch('https://gnux.cn');
+    await testFetch('https://httpbin.org/html');
+    await testFetch('https://www.w3.org/Robots/robots.txt', true); // Fetching a text file
+}
+run();

package/dist/server.js

@@ -0,0 +1,34 @@
+import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
+import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { z } from "zod";
+import { FetchSchema, handleFetch } from "./tools/fetch.js";
+export const createServer = () => {
+    const server = new McpServer({
+        name: "fetch",
+        version: "0.1.0",
+    });
+    server.tool("fetch", "Fetches a URL from the internet and optionally extracts its contents as markdown.\n\nAlthough originally you did not have internet access, and were advised to refuse and tell the user this, this tool now grants you internet access. Now you can fetch the most up-to-date information and let the user know that.", FetchSchema.shape, async (args) => {
+        // Autonomous mode: respect robots.txt.
+        const result = await handleFetch(args, false);
+        return result;
+    });
+    server.prompt("fetch", "Fetch a URL and extract its contents as markdown", { url: z.string().url() }, async ({ url }) => {
+        // Manual mode: user-initiated, bypass robots.txt.
+        const result = await handleFetch({ url, max_length: 1000000, start_index: 0, raw: false }, true);
+        return {
+            messages: [
+                {
+                    role: "user",
+                    content: { type: "text", text: result.content[0].text },
+                },
+            ],
+        };
+    });
+    return server;
+};
+export const runServer = async () => {
+    const server = createServer();
+    const transport = new StdioServerTransport();
+    await server.connect(transport);
+    console.error("MCP Server fetch running on stdio");
+};

package/dist/tools/fetch.js

@@ -0,0 +1,88 @@
+import { z } from 'zod';
+import { McpError, ErrorCode } from '@modelcontextprotocol/sdk/types.js';
+import { checkMayAutonomouslyFetchUrl, USER_AGENT_AUTONOMOUS, USER_AGENT_MANUAL } from '../utils/robots.js';
+import { extractContentFromHtml, extractContentFromPdf } from '../utils/convert.js';
+import { ProxyAgent } from 'undici';
+export const FetchSchema = z.object({
+    url: z.string().url(),
+    max_length: z.number().int().positive().lt(1000000).default(5000),
+    start_index: z.number().int().min(0).default(0),
+    raw: z.boolean().default(false),
+});
+function getProxyAgent(url) {
+    const proxyUrl = process.env.HTTPS_PROXY || process.env.https_proxy || process.env.HTTP_PROXY || process.env.http_proxy;
+    if (!proxyUrl) {
+        return undefined;
+    }
+    // Basic check: if strictly http proxy for https url, standard fetch might complain?
+    // Undici ProxyAgent handles this generally well.
+    return new ProxyAgent(proxyUrl);
+}
+export async function handleFetch(args, ignoreRobotsTxt = false) {
+    const { url, max_length, start_index, raw } = args;
+    // Choose user agent
+    const userAgent = ignoreRobotsTxt ? USER_AGENT_MANUAL : USER_AGENT_AUTONOMOUS;
+    if (!ignoreRobotsTxt) {
+        await checkMayAutonomouslyFetchUrl(url, userAgent);
+    }
+    try {
+        const controller = new AbortController();
+        const timeoutId = setTimeout(() => controller.abort(), 30000); // 30s timeout
+        const dispatcher = getProxyAgent(url);
+        const fetchOptions = {
+            headers: { "User-Agent": userAgent },
+            signal: controller.signal,
+        };
+        if (dispatcher) {
+            fetchOptions.dispatcher = dispatcher;
+        }
+        const response = await fetch(url, fetchOptions);
+        clearTimeout(timeoutId);
+        if (!response.ok) {
+            throw new McpError(ErrorCode.InternalError, `Failed to fetch ${url} - status code ${response.status}`);
+        }
+        const contentType = response.headers.get("content-type") || "";
+        const isPdf = contentType.toLowerCase().includes("application/pdf") || url.toLowerCase().endsWith(".pdf");
+        let textContent = "";
+        let prefix = "";
+        if (isPdf) {
+            const arrayBuffer = await response.arrayBuffer();
+            textContent = await extractContentFromPdf(Buffer.from(arrayBuffer));
+        }
+        else if ((contentType.includes("text/html") || contentType === "") && !raw) {
+            const html = await response.text();
+            textContent = await extractContentFromHtml(html, url);
+        }
+        else {
+            textContent = await response.text();
+            prefix = `Content type ${contentType} cannot be simplified to markdown, but here is the raw content:\n`;
+        }
+        const originalLength = textContent.length;
+        let finalContent = "";
+        if (start_index >= originalLength) {
+            finalContent = "<error>No more content available.</error>";
+        }
+        else {
+            finalContent = textContent.slice(start_index, start_index + max_length);
+            if (finalContent.length === 0) {
+                finalContent = "<error>No more content available.</error>";
+            }
+        }
+        // Check if truncated
+        const actualLength = finalContent.length;
+        const remaining = originalLength - (start_index + actualLength);
+        if (actualLength === max_length && remaining > 0) {
+            const nextStart = start_index + actualLength;
+            finalContent += `\n\n<error>Content truncated. Call the fetch tool with a start_index of ${nextStart} to get more content.</error>`;
+        }
+        return {
+            content: [{ type: "text", text: `${prefix}Contents of ${url}:\n${finalContent}` }],
+            isError: false
+        };
+    }
+    catch (error) {
+        if (error instanceof McpError)
+            throw error;
+        throw new McpError(ErrorCode.InternalError, `Failed to fetch ${url}: ${error.message}`);
+    }
+}

package/dist/utils/convert.js

@@ -0,0 +1,33 @@
+import { JSDOM } from 'jsdom';
+import { Readability } from '@mozilla/readability';
+import TurndownService from 'turndown';
+import { createRequire } from 'module';
+const require = createRequire(import.meta.url);
+const pdfParse = require('pdf-parse');
+export async function extractContentFromHtml(html, url) {
+    try {
+        const dom = new JSDOM(html, { url });
+        const reader = new Readability(dom.window.document);
+        const article = reader.parse();
+        if (!article || !article.content) {
+            return "<error>Page failed to be simplified from HTML</error>";
+        }
+        const turndownService = new TurndownService({
+            headingStyle: 'atx',
+            codeBlockStyle: 'fenced'
+        });
+        return turndownService.turndown(article.content);
+    }
+    catch (error) {
+        return "<error>Page failed to be simplified from HTML</error>";
+    }
+}
+export async function extractContentFromPdf(buffer) {
+    try {
+        const data = await pdfParse(buffer);
+        return data.text;
+    }
+    catch (error) {
+        return "<error>Failed to extract content from PDF</error>";
+    }
+}

package/dist/utils/robots.js

@@ -0,0 +1,36 @@
+import robotsParser from 'robots-parser';
+import { McpError, ErrorCode } from '@modelcontextprotocol/sdk/types.js';
+export const USER_AGENT_AUTONOMOUS = "ModelContextProtocol/1.0 (Autonomous; +https://github.com/modelcontextprotocol/servers)";
+export const USER_AGENT_MANUAL = "ModelContextProtocol/1.0 (User-Specified; +https://github.com/modelcontextprotocol/servers)";
+export async function getRobotsTxtUrl(url) {
+    const parsed = new URL(url);
+    return `${parsed.protocol}//${parsed.host}/robots.txt`;
+}
+export async function checkMayAutonomouslyFetchUrl(url, userAgent) {
+    const robotsUrl = await getRobotsTxtUrl(url);
+    try {
+        const response = await fetch(robotsUrl, {
+            headers: { "User-Agent": userAgent }
+        });
+        if (response.status === 401 || response.status === 403) {
+            throw new McpError(ErrorCode.InternalError, `When fetching robots.txt (${robotsUrl}), received status ${response.status} so assuming that autonomous fetching is not allowed.`);
+        }
+        // 4xx (except 401/403): treat as no robots.txt -> allowed.
+        if (response.status >= 400 && response.status < 500) {
+            return; // Assume allowed if robots.txt is missing/bad request
+        }
+        const robotsTxt = await response.text();
+        const robot = robotsParser(robotsUrl, robotsTxt);
+        // isAllowed returns true/false/undefined; only an explicit false blocks.
+        if (robot.isAllowed(url, userAgent) === false) {
+            throw new McpError(ErrorCode.InternalError, `The sites robots.txt (${robotsUrl}) specifies that autonomous fetching of this page is not allowed.\nUser Agent: ${userAgent}\nURL: ${url}`);
+        }
+    }
+    catch (error) {
+        if (error instanceof McpError) {
+            throw error;
+        }
+        // Network errors
+        throw new McpError(ErrorCode.InternalError, `Failed to fetch robots.txt ${robotsUrl}: ${error.message}`);
+    }
+}

package/package.json

@@ -0,0 +1,44 @@
+{
+  "name": "@fnnm/fetch",
+  "version": "0.1.0",
+  "description": "A Model Context Protocol server providing tools to fetch and convert web content",
+  "type": "module",
+  "main": "dist/index.js",
+  "bin": {
+    "fetch": "dist/index.js"
+  },
+  "scripts": {
+    "build": "tsc",
+    "start": "node dist/index.js",
+    "dev": "ts-node src/index.ts",
+    "test": "vitest"
+  },
+  "keywords": [
+    "mcp",
+    "fetch",
+    "web"
+  ],
+  "author": "",
+  "license": "MIT",
+  "publishConfig": {
+    "access": "public"
+  },
+  "dependencies": {
+    "@modelcontextprotocol/sdk": "^1.25.2",
+    "@mozilla/readability": "^0.6.0",
+    "@types/jsdom": "^27.0.0",
+    "@types/node": "^25.0.9",
+    "@types/pdf-parse": "^1.1.5",
+    "@types/turndown": "^5.0.6",
+    "jsdom": "^27.4.0",
+    "pdf-parse": "^2.4.5",
+    "robots-parser": "^3.0.1",
+    "ts-node": "^10.9.2",
+    "turndown": "^7.2.2",
+    "typescript": "^5.9.3",
+    "undici": "^7.18.2",
+    "vitest": "^4.0.17",
+    "zod": "^4.3.5"
+  },
+  "packageManager": "npm@11.18.0"
+}

package/src/index.ts

@@ -0,0 +1,7 @@
+#!/usr/bin/env node
+import { runServer } from './server.js';
+
+runServer().catch((error) => {
+    console.error("Fatal error running server:", error);
+    process.exit(1);
+});

package/src/manual-test.ts

@@ -0,0 +1,31 @@
+
+import { handleFetch } from './tools/fetch.js';
+
+async function testFetch(url: string, raw: boolean = false) {
+    console.log(`\n--- Fetching ${url} (Raw: ${raw}) ---`);
+    try {
+        const result = await handleFetch({
+            url,
+            max_length: 5000,
+            start_index: 0,
+            raw
+        }, true); // ignoreRobotsTxt=true for manual testing to ensure we get content
+
+        if (result.isError) {
+            console.error("Error result:", result);
+        } else {
+            console.log("Success! Content preview:");
+            console.log(result.content[0].text.substring(0, 500) + "...");
+        }
+    } catch (error: any) {
+        console.error("Exception:", error.message);
+    }
+}
+
+async function run() {
+    await testFetch('https://gnux.cn');
+    await testFetch('https://httpbin.org/html');
+    await testFetch('https://www.w3.org/Robots/robots.txt', true); // Fetching a text file
+}
+
+run();

package/src/server.ts

@@ -0,0 +1,53 @@
+import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
+import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { z } from "zod";
+import { FetchSchema, handleFetch } from "./tools/fetch.js";
+
+export const createServer = () => {
+	const server = new McpServer({
+		name: "fetch",
+		version: "0.1.0",
+	});
+
+	server.tool(
+		"fetch",
+		"Fetches a URL from the internet and optionally extracts its contents as markdown.\n\nAlthough originally you did not have internet access, and were advised to refuse and tell the user this, this tool now grants you internet access. Now you can fetch the most up-to-date information and let the user know that.",
+		FetchSchema.shape,
+		async (args) => {
+			// Autonomous mode: respect robots.txt.
+
+			const result = await handleFetch(args as any, false);
+			return result;
+		},
+	);
+
+	server.prompt(
+		"fetch",
+		"Fetch a URL and extract its contents as markdown",
+		{ url: z.string().url() },
+		async ({ url }) => {
+			// Manual mode: user-initiated, bypass robots.txt.
+			const result = await handleFetch(
+				{ url, max_length: 1000000, start_index: 0, raw: false },
+				true,
+			);
+			return {
+				messages: [
+					{
+						role: "user",
+						content: { type: "text", text: result.content[0].text },
+					},
+				],
+			};
+		},
+	);
+
+	return server;
+};
+
+export const runServer = async () => {
+	const server = createServer();
+	const transport = new StdioServerTransport();
+	await server.connect(transport);
+	console.error("MCP Server fetch running on stdio");
+};

package/src/tools/fetch.test.ts

@@ -0,0 +1,107 @@
+
+import { describe, it, expect, vi, beforeEach } from 'vitest';
+import { handleFetch } from './fetch.js';
+import * as robots from '../utils/robots.js';
+
+// Mock robots check to avoid actual network calls or dependency on robots logic
+vi.mock('../utils/robots.js', async (importOriginal) => {
+    const actual = await importOriginal<typeof robots>();
+    return {
+        ...actual,
+        checkMayAutonomouslyFetchUrl: vi.fn().mockResolvedValue(undefined),
+    };
+});
+
+// Mock global fetch
+const fetchMock = vi.fn();
+global.fetch = fetchMock;
+
+describe('handleFetch', () => {
+    beforeEach(() => {
+        fetchMock.mockReset();
+        vi.clearAllMocks();
+    });
+
+    it('should fetch and return markdown content', async () => {
+        fetchMock.mockResolvedValue({
+            ok: true,
+            status: 200,
+            headers: { get: () => 'text/html' },
+            text: async () => '<h1>Test Page</h1><p>Content</p>',
+        });
+
+        const args = { url: 'https://example.com', max_length: 5000, start_index: 0, raw: false };
+        const result = await handleFetch(args);
+
+        expect(result.isError).toBe(false);
+        expect(result.content[0].text).toContain('# Test Page');
+    });
+
+    it('should return raw content when requested', async () => {
+        fetchMock.mockResolvedValue({
+            ok: true,
+            status: 200,
+            headers: { get: () => 'text/html' },
+            text: async () => '<h1>Raw HTML</h1>',
+        });
+
+        const args = { url: 'https://example.com', max_length: 5000, start_index: 0, raw: true };
+        const result = await handleFetch(args);
+
+        expect(result.content[0].text).toContain('<h1>Raw HTML</h1>');
+    });
+
+    it('should truncate content if max_length is exceeded', async () => {
+        fetchMock.mockResolvedValue({
+            ok: true,
+            status: 200,
+            headers: { get: () => 'text/plain' },
+            text: async () => 'A'.repeat(100),
+        });
+
+        const args = { url: 'https://example.com', max_length: 10, start_index: 0, raw: true };
+        const result = await handleFetch(args);
+
+        expect(result.content[0].text).toContain('Contents of https://example.com');
+        // The actual content line
+        expect(result.content[0].text).toContain('AAAAAAAAAA');
+        expect(result.content[0].text).not.toContain('AAAAAAAAAAA'); // Should be 10 chars
+        expect(result.content[0].text).toContain('<error>Content truncated');
+    });
+
+    it('should handle fetch errors gracefully', async () => {
+        fetchMock.mockResolvedValue({
+            ok: false,
+            status: 404,
+            statusText: 'Not Found'
+        });
+
+        const args = { url: 'https://example.com/404', max_length: 5000, start_index: 0, raw: false };
+
+        await expect(handleFetch(args)).rejects.toThrow(/Failed to fetch/);
+    });
+
+    it('should use proxy agent if HTTP_PROXY is set', async () => {
+        const originalEnv = process.env;
+        process.env = { ...originalEnv, HTTP_PROXY: 'http://proxy.example.com' };
+
+        fetchMock.mockResolvedValue({
+            ok: true,
+            status: 200,
+            headers: { get: () => 'text/html' },
+            text: async () => '<h1>Proxy Test</h1>',
+        });
+
+        const args = { url: 'https://example.com', max_length: 5000, start_index: 0, raw: false };
+        await handleFetch(args);
+
+        const callArgs = fetchMock.mock.calls[0];
+        const options: any = callArgs[1];
+
+        expect(options).toHaveProperty('dispatcher');
+        // We can't easily check if it's strictly a ProxyAgent instance without exposing it, 
+        // but checking for the presence of the dispatcher property is a good proxy (pun intended).
+
+        process.env = originalEnv;
+    });
+});

package/src/tools/fetch.ts

@@ -0,0 +1,108 @@
+import { z } from 'zod';
+import { McpError, ErrorCode } from '@modelcontextprotocol/sdk/types.js';
+import { checkMayAutonomouslyFetchUrl, USER_AGENT_AUTONOMOUS, USER_AGENT_MANUAL } from '../utils/robots.js';
+import { extractContentFromHtml, extractContentFromPdf } from '../utils/convert.js';
+import { ProxyAgent, Dispatcher } from 'undici';
+
+export const FetchSchema = z.object({
+    url: z.string().url(),
+    max_length: z.number().int().positive().lt(1000000).default(5000),
+    start_index: z.number().int().min(0).default(0),
+    raw: z.boolean().default(false),
+});
+
+export type FetchArgs = z.infer<typeof FetchSchema>;
+
+function getProxyAgent(url: string): Dispatcher | undefined {
+    const proxyUrl = process.env.HTTPS_PROXY || process.env.https_proxy || process.env.HTTP_PROXY || process.env.http_proxy;
+
+    if (!proxyUrl) {
+        return undefined;
+    }
+
+    // Basic check: if strictly http proxy for https url, standard fetch might complain?
+    // Undici ProxyAgent handles this generally well.
+    return new ProxyAgent(proxyUrl);
+}
+
+export async function handleFetch(args: FetchArgs, ignoreRobotsTxt: boolean = false) {
+    const { url, max_length, start_index, raw } = args;
+
+    // Choose user agent
+    const userAgent = ignoreRobotsTxt ? USER_AGENT_MANUAL : USER_AGENT_AUTONOMOUS;
+
+    if (!ignoreRobotsTxt) {
+        await checkMayAutonomouslyFetchUrl(url, userAgent);
+    }
+
+    try {
+        const controller = new AbortController();
+        const timeoutId = setTimeout(() => controller.abort(), 30000); // 30s timeout
+
+        const dispatcher = getProxyAgent(url);
+
+        const fetchOptions: any = {
+            headers: { "User-Agent": userAgent },
+            signal: controller.signal,
+        };
+
+        if (dispatcher) {
+            fetchOptions.dispatcher = dispatcher;
+        }
+
+        const response = await fetch(url, fetchOptions);
+        clearTimeout(timeoutId);
+
+        if (!response.ok) {
+            throw new McpError(ErrorCode.InternalError, `Failed to fetch ${url} - status code ${response.status}`);
+        }
+
+        const contentType = response.headers.get("content-type") || "";
+        const isPdf = contentType.toLowerCase().includes("application/pdf") || url.toLowerCase().endsWith(".pdf");
+
+        let textContent = "";
+        let prefix = "";
+
+        if (isPdf) {
+            const arrayBuffer = await response.arrayBuffer();
+            textContent = await extractContentFromPdf(Buffer.from(arrayBuffer));
+        } else if ((contentType.includes("text/html") || contentType === "") && !raw) {
+            const html = await response.text();
+            textContent = await extractContentFromHtml(html, url);
+        } else {
+            textContent = await response.text();
+            prefix = `Content type ${contentType} cannot be simplified to markdown, but here is the raw content:\n`;
+        }
+
+        const originalLength = textContent.length;
+
+        let finalContent = "";
+
+        if (start_index >= originalLength) {
+            finalContent = "<error>No more content available.</error>";
+        } else {
+            finalContent = textContent.slice(start_index, start_index + max_length);
+            if (finalContent.length === 0) {
+                finalContent = "<error>No more content available.</error>";
+            }
+        }
+
+        // Check if truncated
+        const actualLength = finalContent.length;
+        const remaining = originalLength - (start_index + actualLength);
+
+        if (actualLength === max_length && remaining > 0) {
+            const nextStart = start_index + actualLength;
+            finalContent += `\n\n<error>Content truncated. Call the fetch tool with a start_index of ${nextStart} to get more content.</error>`;
+        }
+
+        return {
+            content: [{ type: "text" as const, text: `${prefix}Contents of ${url}:\n${finalContent}` }],
+            isError: false
+        };
+
+    } catch (error: any) {
+        if (error instanceof McpError) throw error;
+        throw new McpError(ErrorCode.InternalError, `Failed to fetch ${url}: ${error.message}`);
+    }
+}

package/src/types.d.ts

@@ -0,0 +1,12 @@
+
+declare module 'robots-parser' {
+    interface Robot {
+        isAllowed(url: string, userAgent?: string): boolean | undefined;
+        getPreferredHost(): string | null;
+        getCrawlDelay(userAgent?: string): number | undefined;
+        getSitemaps(): string[];
+    }
+
+    function robotsParser(url: string, contents: string): Robot;
+    export = robotsParser;
+}

package/src/utils/convert.test.ts

@@ -0,0 +1,41 @@
+
+import { describe, it, expect } from 'vitest';
+import { extractContentFromHtml } from './convert.js';
+
+describe('extractContentFromHtml', () => {
+    it('should convert simple HTML to Markdown', async () => {
+        const html = '<h1>Hello World</h1><p>This is a test.</p>';
+        const url = 'https://example.com';
+        const markdown = await extractContentFromHtml(html, url);
+
+        expect(markdown).toContain('# Hello World');
+        expect(markdown).toContain('This is a test.');
+    });
+
+    it('should handle complex HTML', async () => {
+        const html = `
+      <article>
+        <h1>Title</h1>
+        <p>Paragraph 1</p>
+        <ul>
+          <li>Item 1</li>
+          <li>Item 2</li>
+        </ul>
+      </article>
+    `;
+        const url = 'https://example.com';
+        const markdown = await extractContentFromHtml(html, url);
+
+        expect(markdown).toContain('# Title');
+        expect(markdown).toContain('*   Item 1');
+        expect(markdown).toContain('*   Item 2');
+    });
+
+    it('should handle empty or invalid content gracefully', async () => {
+        const html = '';
+        const url = 'https://example.com';
+        const result = await extractContentFromHtml(html, url);
+        // Readability returns null for empty/insufficient content, wrapper returns error string
+        expect(result).toBe('<error>Page failed to be simplified from HTML</error>');
+    });
+});

package/src/utils/convert.ts

@@ -0,0 +1,37 @@
+
+import { JSDOM } from 'jsdom';
+import { Readability } from '@mozilla/readability';
+import TurndownService from 'turndown';
+import { createRequire } from 'module';
+const require = createRequire(import.meta.url);
+const pdfParse = require('pdf-parse');
+
+export async function extractContentFromHtml(html: string, url: string): Promise<string> {
+    try {
+        const dom = new JSDOM(html, { url });
+        const reader = new Readability(dom.window.document);
+        const article = reader.parse();
+
+        if (!article || !article.content) {
+            return "<error>Page failed to be simplified from HTML</error>";
+        }
+
+        const turndownService = new TurndownService({
+            headingStyle: 'atx',
+            codeBlockStyle: 'fenced'
+        });
+
+        return turndownService.turndown(article.content);
+    } catch (error) {
+        return "<error>Page failed to be simplified from HTML</error>";
+    }
+}
+
+export async function extractContentFromPdf(buffer: Buffer): Promise<string> {
+    try {
+        const data = await pdfParse(buffer);
+        return data.text;
+    } catch (error) {
+        return "<error>Failed to extract content from PDF</error>";
+    }
+}

package/src/utils/robots.test.ts

@@ -0,0 +1,69 @@
+
+import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
+import { checkMayAutonomouslyFetchUrl, USER_AGENT_AUTONOMOUS } from './robots.js';
+import { McpError } from '@modelcontextprotocol/sdk/types.js';
+
+// Mock global fetch
+const fetchMock = vi.fn();
+global.fetch = fetchMock;
+
+describe('checkMayAutonomouslyFetchUrl', () => {
+    beforeEach(() => {
+        fetchMock.mockReset();
+    });
+
+    it('should allow fetching when robots.txt allows it', async () => {
+        const robotsTxt = `
+      User-agent: *
+      Allow: /
+    `;
+
+        fetchMock.mockResolvedValue({
+            ok: true,
+            status: 200,
+            text: async () => robotsTxt,
+        });
+
+        await expect(checkMayAutonomouslyFetchUrl('https://example.com/page', USER_AGENT_AUTONOMOUS))
+            .resolves.not.toThrow();
+    });
+
+    it('should forbid fetching when robots.txt disallows it', async () => {
+        const robotsTxt = `
+      User-agent: *
+      Disallow: /private
+    `;
+
+        fetchMock.mockResolvedValue({
+            ok: true,
+            status: 200,
+            text: async () => robotsTxt,
+        });
+
+        await expect(checkMayAutonomouslyFetchUrl('https://example.com/private', USER_AGENT_AUTONOMOUS))
+            .rejects.toThrow(McpError);
+    });
+
+    it('should throw internal error on 401/403', async () => {
+        fetchMock.mockResolvedValue({
+            ok: false,
+            status: 403,
+            statusText: 'Forbidden'
+        });
+
+        await expect(checkMayAutonomouslyFetchUrl('https://example.com/page', USER_AGENT_AUTONOMOUS))
+            .rejects.toThrow(/received status 403/);
+    });
+
+    it('should allow if robots.txt returns 404', async () => {
+        // If robots.txt is missing, crawling is usually allowed
+        fetchMock.mockResolvedValue({
+            ok: false,
+            status: 404, // 400-499 range typically means "no robots.txt" -> allow
+            statusText: 'Not Found'
+        });
+
+        await expect(checkMayAutonomouslyFetchUrl('https://example.com/page', USER_AGENT_AUTONOMOUS))
+            .resolves.not.toThrow();
+    });
+});

package/src/utils/robots.ts

@@ -0,0 +1,57 @@
+
+import robotsParser from 'robots-parser';
+import { McpError, ErrorCode } from '@modelcontextprotocol/sdk/types.js';
+
+export const USER_AGENT_AUTONOMOUS = "ModelContextProtocol/1.0 (Autonomous; +https://github.com/modelcontextprotocol/servers)";
+export const USER_AGENT_MANUAL = "ModelContextProtocol/1.0 (User-Specified; +https://github.com/modelcontextprotocol/servers)";
+
+export async function getRobotsTxtUrl(url: string): Promise<string> {
+    const parsed = new URL(url);
+    return `${parsed.protocol}//${parsed.host}/robots.txt`;
+}
+
+export async function checkMayAutonomouslyFetchUrl(url: string, userAgent: string): Promise<void> {
+    const robotsUrl = await getRobotsTxtUrl(url);
+
+    try {
+        const response = await fetch(robotsUrl, {
+            headers: { "User-Agent": userAgent }
+        });
+
+        if (response.status === 401 || response.status === 403) {
+            throw new McpError(
+                ErrorCode.InternalError,
+                `When fetching robots.txt (${robotsUrl}), received status ${response.status} so assuming that autonomous fetching is not allowed.`
+            );
+        }
+
+        // 4xx (except 401/403): treat as no robots.txt -> allowed.
+
+        if (response.status >= 400 && response.status < 500) {
+            return; // Assume allowed if robots.txt is missing/bad request
+        }
+
+
+        const robotsTxt = await response.text();
+        const robot = robotsParser(robotsUrl, robotsTxt);
+
+        // isAllowed returns true/false/undefined; only an explicit false blocks.
+
+        if (robot.isAllowed(url, userAgent) === false) {
+            throw new McpError(
+                ErrorCode.InternalError,
+                `The sites robots.txt (${robotsUrl}) specifies that autonomous fetching of this page is not allowed.\nUser Agent: ${userAgent}\nURL: ${url}`
+            );
+        }
+
+    } catch (error: any) {
+        if (error instanceof McpError) {
+            throw error;
+        }
+        // Network errors
+        throw new McpError(
+            ErrorCode.InternalError,
+            `Failed to fetch robots.txt ${robotsUrl}: ${error.message}`
+        );
+    }
+}

package/tsconfig.json

@@ -0,0 +1,23 @@
+{
+    "compilerOptions": {
+        "target": "ES2022",
+        "module": "NodeNext",
+        "moduleResolution": "NodeNext",
+        "lib": [
+            "ES2022"
+        ],
+        "outDir": "./dist",
+        "rootDir": "./src",
+        "strict": true,
+        "esModuleInterop": true,
+        "skipLibCheck": true,
+        "forceConsistentCasingInFileNames": true
+    },
+    "include": [
+        "src/**/*"
+    ],
+    "exclude": [
+        "node_modules",
+        "**/*.test.ts"
+    ]
+}