nectr-mcp 0.3.0 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -2,9 +2,14 @@
2
2
 
3
3
  An [MCP](https://modelcontextprotocol.io) server for [nectr](https://nectr.ch), which shows how AI reads a web page.
4
4
 
5
- It gives a model one tool, `scan_page`. Given a public page's address, it returns the page as clean markdown and what an AI agent gets from it: the size in words and estimated tokens, which AI crawlers the site's robots.txt allows, linked PDFs and whether they hold text, and the text that only appears after JavaScript runs.
5
+ It gives a model two tools:
6
6
 
7
- > **Coming from 0.2?** The library tools (`search_library`, `get_page`, `convert_url`) are gone. Your existing `nct_` key works with `scan_page`.
7
+ - **`scan_page`**: given a public page's address, it returns what an AI agent gets from it: the size in words and estimated tokens, which AI crawlers the site's robots.txt allows, linked PDFs and whether they hold text, and the text that only appears after JavaScript runs. For pages on your own verified sites, it also returns the page as clean markdown.
8
+ - **`list_sites`**: lists your sites in nectr and their pages' URLs, so you can see which pages you'll get markdown for.
9
+
10
+ > **Coming from 0.3?** `scan_page` now includes the markdown only for pages on your own verified sites. Any other page still gets the full scan, without the markdown.
11
+ >
12
+ > **Coming from 0.2?** The library tools (`search_library`, `get_page`, `convert_url`) are gone. Your existing `nct_` key works with these tools.
8
13
 
9
14
  ## Setup
10
15
 
@@ -25,16 +30,18 @@ It gives a model one tool, `scan_page`. Given a public page's address, it return
25
30
  }
26
31
  ```
27
32
 
28
- 3. Restart the client. The `scan_page` tool should appear.
33
+ 3. Restart the client. The `scan_page` and `list_sites` tools should appear.
29
34
 
30
35
  Node.js 20 or newer is required.
31
36
 
32
37
  | Variable | Required | Default | Purpose |
33
38
  |---|---|---|---|
34
39
  | `NECTR_API_KEY` | yes | — | Your `nct_` API key |
35
- | `NECTR_URL` | no | `https://nectr.ch` | The nectr site to talk to, for a self-hosted instance |
40
+ | `NECTR_URL` | no | `https://nectr.ch` | The nectr site to talk to, for a self-hosted instance. Use its exact https address: the server won't follow a redirect, because the key would be dropped on the way |
41
+
42
+ ## The tools
36
43
 
37
- ## The tool
44
+ ### scan_page
38
45
 
39
46
  `scan_page(url, markdown?, max_chars?)` fetches the page twice, with a browser and as plain HTTP, the way AI crawlers do, and returns:
40
47
 
@@ -44,12 +51,16 @@ Node.js 20 or newer is required.
44
51
  | robots.txt | Whether GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and CCBot may fetch the page |
45
52
  | Linked PDFs | Up to 3 PDFs on the same site: pages, words, and whether there's a text layer at all |
46
53
  | Only with JavaScript | The share of words an agent misses without JavaScript, and the first 10 of those blocks |
47
- | Markdown | The page as markdown, last. `markdown` picks `"full"` (after JavaScript runs, the default), `"without_javascript"` or `"none"`. It's cut at 40,000 characters unless you raise `max_chars` |
54
+ | Markdown | The page as markdown, last, and only for a page on one of your verified sites; otherwise the tool says it was left out and points to `list_sites`. `markdown` picks `"full"` (after JavaScript runs, the default), `"without_javascript"` or `"none"`. It's cut at 40,000 characters unless you raise `max_chars` |
48
55
 
49
- The tool is read-only: nectr stores nothing from a scan. nectr honours the site's robots.txt for itself, so a page whose robots.txt keeps nectr out isn't fetched.
56
+ Any public page can be scanned. The tool is read-only: nectr stores nothing from a scan. nectr honours the site's robots.txt for itself, so a page whose robots.txt keeps nectr out isn't fetched.
50
57
 
51
58
  The full response, field by field, is in the [scan API documentation](https://nectr.ch/docs#scan).
52
59
 
60
+ ### list_sites
61
+
62
+ `list_sites()` takes no arguments. For each of your sites it gives the domain, whether it's verified (only verified sites get markdown from `scan_page`), how many pages nectr has copied as markdown, the site's llms.txt address, and the URL of each page (up to 100 per site) with its title and word count. A page nectr couldn't copy shows its status and the reason.
63
+
53
64
  ## Limits and errors
54
65
 
55
66
  The server uses the same developer API as any other script, so the API's limits apply: 10 scans a minute from one IP address. Errors are worded so a model knows whether to retry:
@@ -61,6 +72,7 @@ The server uses the same developer API as any other script, so the API's limits
61
72
  | 422: the page can't be scanned (not public, not found, a PDF) | nectr's reason, as given |
62
73
  | 429: rate limited | How many seconds to wait |
63
74
  | 503: nectr is busy | Retry in a few seconds |
75
+ | A redirect (for example `NECTR_URL` set to `http://…`) | Don't retry; set `NECTR_URL` to nectr's https address |
64
76
 
65
77
  ## Security
66
78
 
@@ -68,7 +80,7 @@ The server uses the same developer API as any other script, so the API's limits
68
80
 
69
81
  - Create a dedicated key for each client, so you can revoke one without breaking the others.
70
82
  - If a key leaks, delete it under **Settings → API keys** in nectr. It stops working immediately.
71
- - This server only ever scans. **The key itself is broader:** it has the same access to the nectr API as your account, so treat it like a password.
83
+ - This server only ever scans pages and reads your list of sites. **The key itself is broader:** it has the same access to the nectr API as your account, so treat it like a password.
72
84
 
73
85
  ## Development
74
86
 
package/dist/client.js CHANGED
@@ -1,7 +1,7 @@
1
1
  /**
2
2
  * A thin client over the public nectr API (`/api/v1`), authenticated with an
3
- * `nct_` API key. It uses the scan endpoint the developer API documents
4
- * (nectr.ch/docs#scan); this package adds no new server-side surface.
3
+ * `nct_` API key. It uses the scan endpoint (nectr.ch/docs#scan) and the key
4
+ * holder's sites and their pages; this package adds no new server-side surface.
5
5
  */
6
6
  /** An HTTP error from nectr, carrying what a caller needs to explain it. */
7
7
  export class NectrApiError extends Error {
@@ -25,7 +25,23 @@ export class NectrUnreachableError extends Error {
25
25
  this.name = "NectrUnreachableError";
26
26
  }
27
27
  }
28
- const USER_AGENT = "nectr-mcp/0.3.0";
28
+ /**
29
+ * nectr answered with a redirect, which the API never should. Following it is
30
+ * worse than failing: fetch drops the Authorization header on a redirect to
31
+ * another origin (http → https included), so the request would go on without
32
+ * the key and quietly get a keyless answer.
33
+ */
34
+ export class NectrRedirectError extends Error {
35
+ siteUrl;
36
+ location;
37
+ constructor(siteUrl, location) {
38
+ super(`nectr at ${siteUrl} redirected${location ? ` to ${location}` : ""}`);
39
+ this.siteUrl = siteUrl;
40
+ this.location = location;
41
+ this.name = "NectrRedirectError";
42
+ }
43
+ }
44
+ const USER_AGENT = "nectr-mcp/0.4.0";
29
45
  // A scan fetches the page twice (a browser and plain HTTP) and checks linked
30
46
  // PDFs, so it can take a while.
31
47
  const TIMEOUT_MS = 90_000;
@@ -43,6 +59,12 @@ export class NectrClient {
43
59
  scan(url) {
44
60
  return this.request("POST", "/scans", { url });
45
61
  }
62
+ sites() {
63
+ return this.request("GET", "/sites");
64
+ }
65
+ pages(siteId) {
66
+ return this.request("GET", `/sites/${encodeURIComponent(siteId)}/pages`);
67
+ }
46
68
  async request(method, path, body) {
47
69
  let res;
48
70
  try {
@@ -55,12 +77,16 @@ export class NectrClient {
55
77
  ...(body === undefined ? {} : { "Content-Type": "application/json" }),
56
78
  },
57
79
  body: body === undefined ? undefined : JSON.stringify(body),
80
+ redirect: "manual",
58
81
  signal: AbortSignal.timeout(TIMEOUT_MS),
59
82
  });
60
83
  }
61
84
  catch (err) {
62
85
  throw new NectrUnreachableError(this.siteUrl, err);
63
86
  }
87
+ if (res.status >= 300 && res.status < 400) {
88
+ throw new NectrRedirectError(this.siteUrl, res.headers.get("location"));
89
+ }
64
90
  if (!res.ok) {
65
91
  throw new NectrApiError(res.status, await readDetail(res), parseRetryAfter(res));
66
92
  }
@@ -104,6 +130,10 @@ export function describeError(err) {
104
130
  if (err instanceof NectrUnreachableError) {
105
131
  return `Could not reach nectr at ${err.siteUrl}. Check the network or NECTR_URL; retrying immediately is unlikely to help.`;
106
132
  }
133
+ if (err instanceof NectrRedirectError) {
134
+ const to = err.location ? ` to ${err.location}` : "";
135
+ return `nectr at ${err.siteUrl} redirected${to}, so the request was not sent on. Do not retry; the user needs to set NECTR_URL to nectr's https address, or remove it to use https://nectr.ch.`;
136
+ }
107
137
  if (!(err instanceof NectrApiError)) {
108
138
  return `Unexpected error: ${err instanceof Error ? err.message : String(err)}`;
109
139
  }
package/dist/server.js CHANGED
@@ -1,6 +1,6 @@
1
1
  import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
2
2
  import { registerTools } from "./tools.js";
3
- export const VERSION = "0.3.0";
3
+ export const VERSION = "0.4.0";
4
4
  export function createServer(client) {
5
5
  const server = new McpServer({ name: "nectr", version: VERSION });
6
6
  registerTools(server, client);
package/dist/tools.js CHANGED
@@ -4,6 +4,8 @@ import { describeError } from "./client.js";
4
4
  export const DEFAULT_MAX_CHARS = 40_000;
5
5
  /** How many "only with JavaScript" blocks to list; nectr returns up to 50. */
6
6
  export const JS_BLOCKS_SHOWN = 10;
7
+ /** How many pages of one site list_sites shows. */
8
+ export const PAGES_SHOWN = 100;
7
9
  export const MARKDOWN_CHOICES = ["full", "without_javascript", "none"];
8
10
  const plural = (n, word) => `${n.toLocaleString("en")} ${word}${n === 1 ? "" : "s"}`;
9
11
  const text = (t) => ({ content: [{ type: "text", text: t }] });
@@ -66,7 +68,11 @@ function markdown(scan, choice, maxChars) {
66
68
  if (choice === "none")
67
69
  return [];
68
70
  if (!scan.markdown_included) {
69
- return ["## Markdown", "nectr didn't include the markdown with this scan."];
71
+ return [
72
+ "## Markdown",
73
+ "Not included: nectr only returns the markdown for pages on the user's own verified sites, " +
74
+ "and this page isn't on one. Call list_sites to see which sites and pages are the user's.",
75
+ ];
70
76
  }
71
77
  const full = choice === "full";
72
78
  const body = (full ? scan.rendered_markdown : scan.plain_markdown) ?? "";
@@ -104,6 +110,44 @@ export async function scanPage(client, args) {
104
110
  return failure(err);
105
111
  }
106
112
  }
113
+ function page(p) {
114
+ const facts = [
115
+ p.title,
116
+ p.words !== null ? plural(p.words, "word") : null,
117
+ p.status === "copied" ? null : `${p.status}${p.reason ? ` (${p.reason})` : ""}`,
118
+ ].filter(Boolean);
119
+ return `- ${p.url}${facts.length > 0 ? `: ${facts.join(", ")}` : ""}`;
120
+ }
121
+ function site(s, pages) {
122
+ const state = !s.verified
123
+ ? "not verified, so scan_page won't include markdown for its pages"
124
+ : s.paused_at
125
+ ? "verified, paused"
126
+ : "verified";
127
+ const shown = pages.slice(0, PAGES_SHOWN).map(page);
128
+ const more = pages.length > PAGES_SHOWN ? [`- …and ${pages.length - PAGES_SHOWN} more.`] : [];
129
+ return [
130
+ `## ${s.domain}`,
131
+ `${state} · ${plural(pages.length, "page")}, ${s.pages_copied.toLocaleString("en")} copied as markdown` +
132
+ (s.llms_txt_url ? ` · llms.txt: ${s.llms_txt_url}` : ""),
133
+ ...(pages.length === 0 ? ["No pages listed yet."] : [...shown, ...more]),
134
+ ];
135
+ }
136
+ // ── Handlers (exported for tests) ───────────────────────────────────────────
137
+ export async function listSites(client) {
138
+ try {
139
+ const sites = await client.sites();
140
+ if (sites.length === 0) {
141
+ return text("The user has no sites in nectr, so scan_page won't include markdown for any page.");
142
+ }
143
+ const pages = await Promise.all(sites.map((s) => client.pages(s.id)));
144
+ const sections = sites.map((s, i) => site(s, pages[i]));
145
+ return text([`# The user's sites in nectr`, ...sections.map((s) => s.join("\n"))].join("\n\n"));
146
+ }
147
+ catch (err) {
148
+ return failure(err);
149
+ }
150
+ }
107
151
  // ── Registration ────────────────────────────────────────────────────────────
108
152
  export function registerTools(server, client) {
109
153
  server.registerTool("scan_page", {
@@ -111,7 +155,8 @@ export function registerTools(server, client) {
111
155
  description: "Fetch one public web page with nectr and return how AI agents see it: the page as clean markdown, " +
112
156
  "its size in words and estimated tokens (against the raw HTML), which AI crawlers its robots.txt " +
113
157
  "allows, linked PDFs and whether they have text, and the text that only appears after JavaScript " +
114
- "runs. Nothing is stored. nectr honours the site's robots.txt for itself.",
158
+ "runs. Any public page can be scanned, but the markdown is only included for pages on the user's " +
159
+ "own verified sites (see list_sites). Nothing is stored. nectr honours the site's robots.txt for itself.",
115
160
  inputSchema: {
116
161
  url: z.url().describe("The http(s) address of a public web page."),
117
162
  markdown: z
@@ -128,4 +173,11 @@ export function registerTools(server, client) {
128
173
  },
129
174
  annotations: { readOnlyHint: true, idempotentHint: true, openWorldHint: true },
130
175
  }, (args) => scanPage(client, args));
176
+ server.registerTool("list_sites", {
177
+ title: "List the user's sites and pages",
178
+ description: "List the user's sites in nectr and the URLs of their pages, with whether each site is verified. " +
179
+ "scan_page includes the markdown only for pages on these verified sites.",
180
+ inputSchema: {},
181
+ annotations: { readOnlyHint: true, idempotentHint: true, openWorldHint: false },
182
+ }, () => listSites(client));
131
183
  }
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "nectr-mcp",
3
- "version": "0.3.0",
3
+ "version": "0.4.0",
4
4
  "description": "MCP server for nectr: scan any public page the way AI reads it, from any MCP client.",
5
5
  "type": "module",
6
6
  "bin": {
@@ -21,13 +21,13 @@
21
21
  "prepublishOnly": "npm run typecheck && npm test && npm run build"
22
22
  },
23
23
  "dependencies": {
24
- "@modelcontextprotocol/sdk": "^1.30.0",
25
- "zod": "^4.6.5"
24
+ "@modelcontextprotocol/sdk": "^1.32.2",
25
+ "zod": "^4.6.7"
26
26
  },
27
27
  "devDependencies": {
28
28
  "@types/node": "^24.0.0",
29
29
  "typescript": "^7.0.2",
30
- "vitest": "^5.0.1"
30
+ "vitest": "^5.0.5"
31
31
  },
32
32
  "keywords": [
33
33
  "mcp",