nectr-mcp 0.2.0 → 0.3.0

package/README.md

@@ -1,13 +1,10 @@
 # nectr-mcp
 
-An [MCP](https://modelcontextprotocol.io) server for [nectr](https://nectr.ch), the markdown library for your LLMs.
+An [MCP](https://modelcontextprotocol.io) server for [nectr](https://nectr.ch), which shows how AI reads a web page.
 
-It lets a model **search** your library, **read** any saved page as clean markdown, and **save** new pages. You don't need to paste links into the chat.
+It gives a model one tool, `scan_page`. Given a public page's address, it returns the page as clean markdown and what an AI agent gets from it: the size in words and estimated tokens, which AI crawlers the site's robots.txt allows, linked PDFs and whether they hold text, and the text that only appears after JavaScript runs.
 
-- **Search:** a model can't guess your permalinks, but it can call `search_library("pricing page")`.
-- **Private pages:** pages you haven't shared have no working public link, but your key can still read them.
-- **Saving:** "save these docs and keep them current" becomes one step, and nectr re-crawls them on your plan's schedule.
-- **Freshness:** every read says when the page was last fetched.
+> **Coming from 0.2?** The library tools (`search_library`, `get_page`, `convert_url`) are gone. Your existing `nct_` key works with `scan_page`.
 
 ## Setup
 
@@ -28,7 +25,7 @@ It lets a model **search** your library, **read** any saved page as clean markdo
 }
 ```
 
-3. Restart the client. The four tools below should appear.
+3. Restart the client. The `scan_page` tool should appear.
 
 Node.js 20 or newer is required.
 
@@ -37,29 +34,33 @@ Node.js 20 or newer is required.
 | `NECTR_API_KEY` | yes | — | Your `nct_` API key |
 | `NECTR_URL` | no | `https://nectr.ch` | The nectr site to talk to, for a self-hosted instance |
 
-## Tools
+## The tool
 
-| Tool | What it does | Writes? |
-|---|---|---|
-| `search_library(query?, limit?)` | Finds saved pages by title or URL. Returns ids, URLs, freshness and a two-sentence summary per page, but **not** the bodies. Leave out `query` to list the most recently refreshed pages. | no |
-| `get_page(id \| url, max_chars?)` | Returns the clean markdown of one page, with when it was last fetched. Long pages are cut off at 40,000 characters by default; ask for more with `max_chars`. | no |
-| `convert_url(url, force_refresh?)` | Saves a web page, PDF or DOCX to your library. Counts against your monthly conversion quota unless the page is already saved. | yes |
+`scan_page(url, markdown?, max_chars?)` fetches the page twice, with a browser and as plain HTTP, the way AI crawlers do, and returns:
 
-There is deliberately **no delete**. A model that could delete your library by misreading a request isn't worth the convenience.
+| Section | What it says |
+|---|---|
+| Size | Words and estimated tokens of the whole page as markdown, against the raw HTML an agent receives, and the words left without JavaScript |
+| robots.txt | Whether GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and CCBot may fetch the page |
+| Linked PDFs | Up to 3 PDFs on the same site: pages, words, and whether there's a text layer at all |
+| Only with JavaScript | The share of words an agent misses without JavaScript, and the first 10 of those blocks |
+| Markdown | The page as markdown, last. `markdown` picks `"full"` (after JavaScript runs, the default), `"without_javascript"` or `"none"`. It's cut at 40,000 characters unless you raise `max_chars` |
+
+The tool is read-only: nectr stores nothing from a scan. nectr honours the site's robots.txt for itself, so a page whose robots.txt keeps nectr out isn't fetched.
 
-Search returns summaries rather than bodies so that looking around your library doesn't fill the model's context. It reads a full page only when it chooses to.
+The full response, field by field, is in the [scan API documentation](https://nectr.ch/docs#scan).
 
 ## Limits and errors
 
-The server uses the same developer API as any other script, so your plan's limits apply unchanged. Errors are worded so a model knows whether to retry:
+The server uses the same developer API as any other script, so the API's limits apply: 10 scans a minute from one IP address. Errors are worded so a model knows whether to retry:
 
 | nectr says | The model is told |
 |---|---|
 | 401: the key is invalid or revoked | Don't retry; you need a new key |
-| 402: over the monthly quota | Don't retry until the quota resets |
-| 422: the page can't be read (for example, it's behind a login) | nectr's reason, as given |
+| 403: robots.txt keeps nectr out, or scans aren't open to the account | nectr's reason; don't retry |
+| 422: the page can't be scanned (not public, not found, a PDF) | nectr's reason, as given |
 | 429: rate limited | How many seconds to wait |
-| 503: the converter is busy | Retry in a few seconds |
+| 503: nectr is busy | Retry in a few seconds |
 
 ## Security
 
@@ -67,7 +68,7 @@ The server uses the same developer API as any other script, so your plan's limit
 
 - Create a dedicated key for each client, so you can revoke one without breaking the others.
 - If a key leaks, delete it under **Settings → API keys** in nectr. It stops working immediately.
-- This server only ever reads and adds pages. **The key itself is broader:** it has the same access to the nectr API as your account, deletion included, so treat it like a password.
+- This server only ever scans. **The key itself is broader:** it has the same access to the nectr API as your account, so treat it like a password.
 
 ## Development
 
@@ -94,12 +95,11 @@ The [MCP Inspector](https://github.com/modelcontextprotocol/inspector) calls the
 $k = "nct_your_key"
 
 npx @modelcontextprotocol/inspector --cli node dist/index.js -e NECTR_API_KEY=$k --method tools/list
-npx @modelcontextprotocol/inspector --cli node dist/index.js -e NECTR_API_KEY=$k --method tools/call --tool-name search_library
-npx @modelcontextprotocol/inspector --cli node dist/index.js -e NECTR_API_KEY=$k --method tools/call --tool-name search_library --tool-arg query=pricing
-npx @modelcontextprotocol/inspector --cli node dist/index.js -e NECTR_API_KEY=$k --method tools/call --tool-name get_page --tool-arg id=PAGE-ID
+npx @modelcontextprotocol/inspector --cli node dist/index.js -e NECTR_API_KEY=$k --method tools/call --tool-name scan_page --tool-arg url=https://example.com/
+npx @modelcontextprotocol/inspector --cli node dist/index.js -e NECTR_API_KEY=$k --method tools/call --tool-name scan_page --tool-arg url=https://example.com/ --tool-arg markdown=none
 ```
 
-Add `-e NECTR_URL=http://localhost:8000` to test against a local nectr backend. Leave out `--cli` for the web interface; the argument order is the same.
+Add `-e NECTR_URL=http://localhost:8000` to test against a local nectr backend. There, scans are open to admins only unless the backend runs with `SCANS_ENABLED=true`; other accounts get a 403. Leave out `--cli` for the web interface; the argument order is the same.
 
 The Inspector's own dependencies need a newer Node than this server does. If it fails with **"Cannot find native binding"** after you upgrade Node, delete its cached copy from the npx cache (`%LocalAppData%\npm-cache\_npx` on Windows, `~/.npm/_npx` elsewhere) and run it again.
 

package/dist/client.js

@@ -1,7 +1,7 @@
 /**
  * A thin client over the public nectr API (`/api/v1`), authenticated with an
- * `nct_` API key. Every endpoint used here is the same one the developer API
- * documents; this package adds no new server-side surface.
+ * `nct_` API key. It uses the scan endpoint the developer API documents
+ * (nectr.ch/docs#scan); this package adds no new server-side surface.
  */
 /** An HTTP error from nectr, carrying what a caller needs to explain it. */
 export class NectrApiError extends Error {
@@ -25,8 +25,10 @@ export class NectrUnreachableError extends Error {
         this.name = "NectrUnreachableError";
     }
 }
-const USER_AGENT = "nectr-mcp/0.2.0";
-const TIMEOUT_MS = 60_000; // a real crawl can take a while; reads are fast
+const USER_AGENT = "nectr-mcp/0.3.0";
+// A scan fetches the page twice (a browser and plain HTTP) and checks linked
+// PDFs, so it can take a while.
+const TIMEOUT_MS = 90_000;
 export class NectrClient {
     apiKey;
     siteUrl;
@@ -38,20 +40,8 @@ export class NectrClient {
         this.fetchImpl = fetchImpl;
         this.api = `${siteUrl}/api/v1`;
     }
-    search(query, limit) {
-        const params = new URLSearchParams({ page: "1", limit: String(limit) });
-        if (query)
-            params.set("search", query);
-        return this.request("GET", `/conversions?${params}`);
-    }
-    getPage(id) {
-        return this.request("GET", `/conversions/${encodeURIComponent(id)}`);
-    }
-    convert(url, forceRefresh) {
-        return this.request("POST", "/conversions", { url, force_refresh: forceRefresh });
-    }
-    me() {
-        return this.request("GET", "/auth/me");
+    scan(url) {
+        return this.request("POST", "/scans", { url });
     }
     async request(method, path, body) {
         let res;
@@ -121,18 +111,15 @@ export function describeError(err) {
     switch (err.status) {
         case 401:
             return "The nectr API key was rejected — it is invalid or has been revoked. Do not retry; the user needs to create a new key in nectr under Settings → API keys.";
-        case 402:
-            return `Over the nectr plan's quota: ${err.detail} Do not retry until the quota resets or the plan is upgraded.`;
         case 403:
-            return `nectr refused this request: ${err.detail}`;
-        case 404:
-            return "Not found in this nectr library. It may have been deleted, or the id is wrong.";
+            // A robots.txt that keeps nectr out, or scans closed to this account.
+            return `nectr will not scan this page: ${err.detail} Do not retry.`;
         case 422:
-            return `nectr could not convert this page: ${err.detail}`;
+            return `nectr could not scan this page: ${err.detail}`;
         case 429:
             return `Rate limited by nectr. Wait ${wait ?? 60} seconds before trying again.`;
         case 503:
-            return `nectr is busy converting other pages. Retry in ${wait ?? 5} seconds.`;
+            return `nectr is busy scanning other pages. Retry in ${wait ?? 5} seconds.`;
         default:
             return err.status >= 500
                 ? `nectr had a server error (${err.status}). Try again later.`

package/dist/server.js

@@ -1,6 +1,6 @@
 import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
 import { registerTools } from "./tools.js";
-export const VERSION = "0.2.0";
+export const VERSION = "0.3.0";
 export function createServer(client) {
     const server = new McpServer({ name: "nectr", version: VERSION });
     registerTools(server, client);

package/dist/tools.js

@@ -1,108 +1,104 @@
 import { z } from "zod";
 import { describeError } from "./client.js";
-/** Default cap on a page body returned to a model: roughly 10k tokens. */
+/** Default cap on the markdown returned to a model: roughly 10k tokens. */
 export const DEFAULT_MAX_CHARS = 40_000;
-const SEARCH_MAX = 25;
-const plural = (n, word) => `${n} ${word}${n === 1 ? "" : "s"}`;
+/** How many "only with JavaScript" blocks to list; nectr returns up to 50. */
+export const JS_BLOCKS_SHOWN = 10;
+export const MARKDOWN_CHOICES = ["full", "without_javascript", "none"];
+const plural = (n, word) => `${n.toLocaleString("en")} ${word}${n === 1 ? "" : "s"}`;
 const text = (t) => ({ content: [{ type: "text", text: t }] });
 const failure = (err) => ({
     content: [{ type: "text", text: describeError(err) }],
     isError: true,
 });
-function titleOf(p) {
-    return p.custom_title || p.title || p.url;
+function size(scan) {
+    return [
+        "## Size",
+        `- The whole page as markdown (after JavaScript runs): ${plural(scan.rendered_words, "word")}, ≈ ${plural(scan.markdown_tokens, "token")}.`,
+        `- The raw HTML an agent receives when it fetches the URL: ≈ ${plural(scan.html_tokens, "token")}.`,
+        `- Without JavaScript: ${plural(scan.plain_words, "word")} as markdown.`,
+    ];
 }
-function statusNote(status) {
-    if (status === "completed")
-        return "";
-    if (status === "unreadable")
-        return " · UNREADABLE (login wall or placeholder; no content)";
-    return ` · ${status.toUpperCase()} (no content)`;
-}
-/** Compare URLs the way a person would: scheme/host case, fragment and trailing slash ignored. */
-export function sameUrl(a, b) {
-    const norm = (u) => {
-        try {
-            const p = new URL(u.trim());
-            p.hash = "";
-            return `${p.protocol}//${p.host}${p.pathname.replace(/\/+$/, "")}${p.search}`.toLowerCase();
-        }
-        catch {
-            return u.trim().replace(/\/+$/, "").toLowerCase();
-        }
-    };
-    return norm(a) === norm(b);
-}
-// ── Handlers (exported for tests) ───────────────────────────────────────────
-export async function searchLibrary(client, args) {
-    const query = args.query?.trim() || undefined;
-    const limit = Math.min(Math.max(args.limit ?? 10, 1), SEARCH_MAX);
-    try {
-        const list = await client.search(query, limit);
-        if (list.items.length === 0) {
-            return text(query
-                ? `No saved pages match "${query}". Use convert_url to add one.`
-                : "The library is empty. Use convert_url to add a page.");
-        }
-        const heading = query
-            ? `${list.items.length} of ${plural(list.total, "saved page")} matching "${query}":`
-            : `${list.items.length} most recently refreshed of ${plural(list.total, "saved page")}:`;
-        // Summaries and ids only. Bodies are fetched deliberately with get_page:
-        // returning ten full pages here would flood the context on every search.
-        const rows = list.items.map((p, i) => [
-            `${i + 1}. ${titleOf(p)}`,
-            `   id: ${p.id}`,
-            `   url: ${p.url}`,
-            `   fetched: ${p.fetched_at} · ${plural(p.word_count, "word")}${statusNote(p.status)}`,
-            ...(p.summary ? [`   summary: ${p.summary}`] : []),
-        ].join("\n"));
-        return text(`${heading}\n\n${rows.join("\n\n")}\n\nRead one with get_page(id).`);
+function robots(scan) {
+    if (!scan.robots_found) {
+        return ["## robots.txt", "The site has no robots.txt, so every crawler may fetch this page."];
     }
-    catch (err) {
-        return failure(err);
+    return [
+        "## robots.txt",
+        ...scan.robots.map((r) => `- ${r.agent}: ${r.allowed ? "allowed" : "blocked"}`),
+    ];
+}
+function pdfs(scan) {
+    if (scan.pdfs.length === 0 && scan.pdfs_not_checked === 0)
+        return [];
+    const lines = scan.pdfs.map((p) => {
+        if (p.has_text === null)
+            return `- ${p.url}: not checked${p.note ? ` (${p.note})` : ""}`;
+        const facts = [
+            p.pages !== null ? plural(p.pages, "page") : null,
+            p.words !== null ? `${plural(p.words, "word")} in the first pages` : null,
+            p.has_text ? "has text" : "no text layer (a scanned image; AI reads nothing)",
+        ].filter(Boolean);
+        return `- ${p.url}: ${facts.join(", ")}`;
+    });
+    if (scan.pdfs_not_checked > 0) {
+        lines.push(`- ${plural(scan.pdfs_not_checked, "more linked PDF")} not checked (past the first 3, or on another site).`);
     }
+    return ["## Linked PDFs", ...lines];
 }
-export async function getPage(client, args) {
-    const maxChars = Math.max(args.max_chars ?? DEFAULT_MAX_CHARS, 1_000);
-    try {
-        let id = args.id?.trim();
-        if (!id) {
-            const url = args.url?.trim();
-            if (!url)
-                return failure(new Error("Pass either id or url."));
-            // By id rather than by public link: a private page has no working
-            // public link, but its owner's key can still read it by id.
-            const term = url.replace(/^https?:\/\//i, "").replace(/\/+$/, "");
-            const list = await client.search(term, SEARCH_MAX);
-            const hit = list.items.find((p) => sameUrl(p.url, url));
-            if (!hit) {
-                return text(`No saved page for ${url}. Save it first with convert_url.`);
-            }
-            id = hit.id;
-        }
-        const page = await client.getPage(id);
-        if (page.status !== "completed") {
-            return text(`"${titleOf(page)}" (${page.url}) has no readable content${statusNote(page.status)}.`);
-        }
-        const body = page.markdown_content;
-        const meta = `id: ${page.id} · url: ${page.url} · fetched: ${page.fetched_at} · ${page.word_count} words`;
-        if (body.length <= maxChars)
-            return text(`${meta}\n\n${body}`);
-        return text(`${meta}\n\n${body.slice(0, maxChars)}\n\n[Truncated: showing ${maxChars} of ${body.length} characters. ` +
-            `Call get_page again with a larger max_chars for the rest.]`);
+function onlyWithJavaScript(scan) {
+    const blocks = scan.only_with_javascript;
+    if (blocks.length === 0) {
+        return ["## Only with JavaScript", "Nothing: an agent that doesn't run JavaScript gets the same text."];
     }
-    catch (err) {
-        return failure(err);
+    const share = Math.round(scan.missing_word_share * 100);
+    const shown = blocks.slice(0, JS_BLOCKS_SHOWN).map((b) => `- ${b.replace(/\s+/g, " ").trim()}`);
+    const more = blocks.length > JS_BLOCKS_SHOWN ? [`- …and ${blocks.length - JS_BLOCKS_SHOWN} more.`] : [];
+    return [
+        "## Only with JavaScript",
+        `${share}% of the page's words appear only after JavaScript runs, in ${plural(blocks.length, "block")}. ` +
+            "An agent that doesn't run JavaScript misses them:",
+        ...shown,
+        ...more,
+    ];
+}
+function markdown(scan, choice, maxChars) {
+    if (choice === "none")
+        return [];
+    if (!scan.markdown_included) {
+        return ["## Markdown", "nectr didn't include the markdown with this scan."];
     }
+    const full = choice === "full";
+    const body = (full ? scan.rendered_markdown : scan.plain_markdown) ?? "";
+    const heading = full
+        ? "## The page as markdown (after JavaScript runs)"
+        : "## The page as markdown, without JavaScript";
+    if (body.length <= maxChars)
+        return [heading, body];
+    return [
+        heading,
+        body.slice(0, maxChars),
+        `[Truncated: showing ${maxChars.toLocaleString("en")} of ${body.length.toLocaleString("en")} characters. ` +
+            "Call scan_page again with a larger max_chars for the rest.]",
+    ];
 }
-export async function convertUrl(client, args) {
+// ── Handler (exported for tests) ────────────────────────────────────────────
+export async function scanPage(client, args) {
+    const choice = args.markdown ?? "full";
+    const maxChars = Math.max(args.max_chars ?? DEFAULT_MAX_CHARS, 1_000);
     try {
-        const saved = await client.convert(args.url, args.force_refresh ?? false);
-        const how = saved.cached ? "Already in the library (served the saved copy)" : "Saved";
-        return text(`${how}: ${titleOf(saved)}\n` +
-            `id: ${saved.id} · ${saved.word_count} words · fetched: ${saved.fetched_at}` +
-            (saved.summary ? `\nsummary: ${saved.summary}` : "") +
-            `\n\nnectr keeps this page current on the plan's refresh schedule. Read it with get_page("${saved.id}").`);
+        const scan = await client.scan(args.url);
+        // Short findings first, the long markdown last: a body of 40,000
+        // characters at the top would push the findings out of view.
+        const sections = [
+            [`# ${scan.title || scan.url}`, `url: ${scan.url} · scanned: ${scan.scanned_at}`],
+            size(scan),
+            robots(scan),
+            pdfs(scan),
+            onlyWithJavaScript(scan),
+            markdown(scan, choice, maxChars),
+        ].filter((s) => s.length > 0);
+        return text(sections.map((s) => s.join("\n")).join("\n\n"));
     }
     catch (err) {
         return failure(err);
@@ -110,44 +106,26 @@ export async function convertUrl(client, args) {
 }
 // ── Registration ────────────────────────────────────────────────────────────
 export function registerTools(server, client) {
-    server.registerTool("search_library", {
-        title: "Search the nectr library",
-        description: "Search the user's saved nectr pages by title or URL. Returns ids, URLs, freshness and a " +
-            "short summary per page, not the page bodies. Omit query to list the most recently refreshed pages.",
-        inputSchema: {
-            query: z.string().max(500).optional().describe("Words from the title or URL. Omit to list recent pages."),
-            limit: z.number().int().min(1).max(SEARCH_MAX).optional().describe("Max results (default 10)."),
-        },
-        annotations: { readOnlyHint: true, openWorldHint: false },
-    }, (args) => searchLibrary(client, args));
-    server.registerTool("get_page", {
-        title: "Read a saved page",
-        description: "Get the clean markdown of one saved nectr page, by id (from search_library) or by its " +
-            "original URL. Includes when it was last fetched. Works for private pages too.",
+    server.registerTool("scan_page", {
+        title: "Scan a page the way AI reads it",
+        description: "Fetch one public web page with nectr and return how AI agents see it: the page as clean markdown, " +
+            "its size in words and estimated tokens (against the raw HTML), which AI crawlers its robots.txt " +
+            "allows, linked PDFs and whether they have text, and the text that only appears after JavaScript " +
+            "runs. Nothing is stored. nectr honours the site's robots.txt for itself.",
         inputSchema: {
-            id: z.string().optional().describe("Page id from search_library or convert_url."),
-            url: z.string().optional().describe("The page's original URL, if the id is not known."),
+            url: z.url().describe("The http(s) address of a public web page."),
+            markdown: z
+                .enum(MARKDOWN_CHOICES)
+                .optional()
+                .describe('Which markdown to include: "full" (after JavaScript runs, the default), ' +
+                '"without_javascript" (what an agent that runs no JavaScript gets) or "none".'),
             max_chars: z
                 .number()
                 .int()
                 .min(1000)
                 .optional()
-                .describe(`Truncate the body after this many characters (default ${DEFAULT_MAX_CHARS}).`),
-        },
-        annotations: { readOnlyHint: true, openWorldHint: false },
-    }, (args) => getPage(client, args));
-    server.registerTool("convert_url", {
-        title: "Save a page to nectr",
-        description: "Convert a web page, PDF or DOCX to clean markdown and save it to the user's nectr library, " +
-            "where it is re-crawled automatically to stay current. Returns the new page's id and summary. " +
-            "Counts against the user's monthly conversion quota unless the page is already saved.",
-        inputSchema: {
-            url: z.url().describe("The http(s) URL to save."),
-            force_refresh: z
-                .boolean()
-                .optional()
-                .describe("Re-crawl now even if the page is already saved (counts against the quota)."),
+                .describe(`Truncate the markdown after this many characters (default ${DEFAULT_MAX_CHARS}).`),
         },
-        annotations: { readOnlyHint: false, destructiveHint: false, idempotentHint: true, openWorldHint: true },
-    }, (args) => convertUrl(client, args));
+        annotations: { readOnlyHint: true, idempotentHint: true, openWorldHint: true },
+    }, (args) => scanPage(client, args));
 }

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "nectr-mcp",
-  "version": "0.2.0",
-  "description": "MCP server for nectr: search, read and add to your nectr markdown library from any MCP client.",
+  "version": "0.3.0",
+  "description": "MCP server for nectr: scan any public page the way AI reads it, from any MCP client.",
   "type": "module",
   "bin": {
     "nectr-mcp": "dist/index.js"
@@ -34,7 +34,9 @@
     "model-context-protocol",
     "nectr",
     "markdown",
-    "llm"
+    "llm",
+    "ai-crawlers",
+    "robots.txt"
   ],
   "repository": {
     "type": "git",