nectr-mcp 0.3.1 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +20 -8
- package/dist/client.js +33 -3
- package/dist/server.js +1 -1
- package/dist/tools.js +54 -2
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -2,9 +2,14 @@
|
|
|
2
2
|
|
|
3
3
|
An [MCP](https://modelcontextprotocol.io) server for [nectr](https://nectr.ch), which shows how AI reads a web page.
|
|
4
4
|
|
|
5
|
-
It gives a model
|
|
5
|
+
It gives a model two tools:
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
- **`scan_page`**: given a public page's address, it returns what an AI agent gets from it: the size in words and estimated tokens, which AI crawlers the site's robots.txt allows, linked PDFs and whether they hold text, and the text that only appears after JavaScript runs. For pages on your own verified sites, it also returns the page as clean markdown.
|
|
8
|
+
- **`list_sites`**: lists your sites in nectr and their pages' URLs, so you can see which pages you'll get markdown for.
|
|
9
|
+
|
|
10
|
+
> **Coming from 0.3?** `scan_page` now includes the markdown only for pages on your own verified sites. Any other page still gets the full scan, without the markdown.
|
|
11
|
+
>
|
|
12
|
+
> **Coming from 0.2?** The library tools (`search_library`, `get_page`, `convert_url`) are gone. Your existing `nct_` key works with these tools.
|
|
8
13
|
|
|
9
14
|
## Setup
|
|
10
15
|
|
|
@@ -25,16 +30,18 @@ It gives a model one tool, `scan_page`. Given a public page's address, it return
|
|
|
25
30
|
}
|
|
26
31
|
```
|
|
27
32
|
|
|
28
|
-
3. Restart the client. The `scan_page`
|
|
33
|
+
3. Restart the client. The `scan_page` and `list_sites` tools should appear.
|
|
29
34
|
|
|
30
35
|
Node.js 20 or newer is required.
|
|
31
36
|
|
|
32
37
|
| Variable | Required | Default | Purpose |
|
|
33
38
|
|---|---|---|---|
|
|
34
39
|
| `NECTR_API_KEY` | yes | — | Your `nct_` API key |
|
|
35
|
-
| `NECTR_URL` | no | `https://nectr.ch` | The nectr site to talk to, for a self-hosted instance |
|
|
40
|
+
| `NECTR_URL` | no | `https://nectr.ch` | The nectr site to talk to, for a self-hosted instance. Use its exact https address: the server won't follow a redirect, because the key would be dropped on the way |
|
|
41
|
+
|
|
42
|
+
## The tools
|
|
36
43
|
|
|
37
|
-
|
|
44
|
+
### scan_page
|
|
38
45
|
|
|
39
46
|
`scan_page(url, markdown?, max_chars?)` fetches the page twice, with a browser and as plain HTTP, the way AI crawlers do, and returns:
|
|
40
47
|
|
|
@@ -44,12 +51,16 @@ Node.js 20 or newer is required.
|
|
|
44
51
|
| robots.txt | Whether GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and CCBot may fetch the page |
|
|
45
52
|
| Linked PDFs | Up to 3 PDFs on the same site: pages, words, and whether there's a text layer at all |
|
|
46
53
|
| Only with JavaScript | The share of words an agent misses without JavaScript, and the first 10 of those blocks |
|
|
47
|
-
| Markdown | The page as markdown, last
|
|
54
|
+
| Markdown | The page as markdown, last, and only for a page on one of your verified sites; otherwise the tool says it was left out and points to `list_sites`. `markdown` picks `"full"` (after JavaScript runs, the default), `"without_javascript"` or `"none"`. It's cut at 40,000 characters unless you raise `max_chars` |
|
|
48
55
|
|
|
49
|
-
The tool is read-only: nectr stores nothing from a scan. nectr honours the site's robots.txt for itself, so a page whose robots.txt keeps nectr out isn't fetched.
|
|
56
|
+
Any public page can be scanned. The tool is read-only: nectr stores nothing from a scan. nectr honours the site's robots.txt for itself, so a page whose robots.txt keeps nectr out isn't fetched.
|
|
50
57
|
|
|
51
58
|
The full response, field by field, is in the [scan API documentation](https://nectr.ch/docs#scan).
|
|
52
59
|
|
|
60
|
+
### list_sites
|
|
61
|
+
|
|
62
|
+
`list_sites()` takes no arguments. For each of your sites it gives the domain, whether it's verified (only verified sites get markdown from `scan_page`), how many pages nectr has copied as markdown, the site's llms.txt address, and the URL of each page (up to 100 per site) with its title and word count. A page nectr couldn't copy shows its status and the reason.
|
|
63
|
+
|
|
53
64
|
## Limits and errors
|
|
54
65
|
|
|
55
66
|
The server uses the same developer API as any other script, so the API's limits apply: 10 scans a minute from one IP address. Errors are worded so a model knows whether to retry:
|
|
@@ -61,6 +72,7 @@ The server uses the same developer API as any other script, so the API's limits
|
|
|
61
72
|
| 422: the page can't be scanned (not public, not found, a PDF) | nectr's reason, as given |
|
|
62
73
|
| 429: rate limited | How many seconds to wait |
|
|
63
74
|
| 503: nectr is busy | Retry in a few seconds |
|
|
75
|
+
| A redirect (for example `NECTR_URL` set to `http://…`) | Don't retry; set `NECTR_URL` to nectr's https address |
|
|
64
76
|
|
|
65
77
|
## Security
|
|
66
78
|
|
|
@@ -68,7 +80,7 @@ The server uses the same developer API as any other script, so the API's limits
|
|
|
68
80
|
|
|
69
81
|
- Create a dedicated key for each client, so you can revoke one without breaking the others.
|
|
70
82
|
- If a key leaks, delete it under **Settings → API keys** in nectr. It stops working immediately.
|
|
71
|
-
- This server only ever scans. **The key itself is broader:** it has the same access to the nectr API as your account, so treat it like a password.
|
|
83
|
+
- This server only ever scans pages and reads your list of sites. **The key itself is broader:** it has the same access to the nectr API as your account, so treat it like a password.
|
|
72
84
|
|
|
73
85
|
## Development
|
|
74
86
|
|
package/dist/client.js
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
/**
|
|
2
2
|
* A thin client over the public nectr API (`/api/v1`), authenticated with an
|
|
3
|
-
* `nct_` API key. It uses the scan endpoint
|
|
4
|
-
*
|
|
3
|
+
* `nct_` API key. It uses the scan endpoint (nectr.ch/docs#scan) and the key
|
|
4
|
+
* holder's sites and their pages; this package adds no new server-side surface.
|
|
5
5
|
*/
|
|
6
6
|
/** An HTTP error from nectr, carrying what a caller needs to explain it. */
|
|
7
7
|
export class NectrApiError extends Error {
|
|
@@ -25,7 +25,23 @@ export class NectrUnreachableError extends Error {
|
|
|
25
25
|
this.name = "NectrUnreachableError";
|
|
26
26
|
}
|
|
27
27
|
}
|
|
28
|
-
|
|
28
|
+
/**
|
|
29
|
+
* nectr answered with a redirect, which the API never should. Following it is
|
|
30
|
+
* worse than failing: fetch drops the Authorization header on a redirect to
|
|
31
|
+
* another origin (http → https included), so the request would go on without
|
|
32
|
+
* the key and quietly get a keyless answer.
|
|
33
|
+
*/
|
|
34
|
+
export class NectrRedirectError extends Error {
|
|
35
|
+
siteUrl;
|
|
36
|
+
location;
|
|
37
|
+
constructor(siteUrl, location) {
|
|
38
|
+
super(`nectr at ${siteUrl} redirected${location ? ` to ${location}` : ""}`);
|
|
39
|
+
this.siteUrl = siteUrl;
|
|
40
|
+
this.location = location;
|
|
41
|
+
this.name = "NectrRedirectError";
|
|
42
|
+
}
|
|
43
|
+
}
|
|
44
|
+
const USER_AGENT = "nectr-mcp/0.4.0";
|
|
29
45
|
// A scan fetches the page twice (a browser and plain HTTP) and checks linked
|
|
30
46
|
// PDFs, so it can take a while.
|
|
31
47
|
const TIMEOUT_MS = 90_000;
|
|
@@ -43,6 +59,12 @@ export class NectrClient {
|
|
|
43
59
|
scan(url) {
|
|
44
60
|
return this.request("POST", "/scans", { url });
|
|
45
61
|
}
|
|
62
|
+
sites() {
|
|
63
|
+
return this.request("GET", "/sites");
|
|
64
|
+
}
|
|
65
|
+
pages(siteId) {
|
|
66
|
+
return this.request("GET", `/sites/${encodeURIComponent(siteId)}/pages`);
|
|
67
|
+
}
|
|
46
68
|
async request(method, path, body) {
|
|
47
69
|
let res;
|
|
48
70
|
try {
|
|
@@ -55,12 +77,16 @@ export class NectrClient {
|
|
|
55
77
|
...(body === undefined ? {} : { "Content-Type": "application/json" }),
|
|
56
78
|
},
|
|
57
79
|
body: body === undefined ? undefined : JSON.stringify(body),
|
|
80
|
+
redirect: "manual",
|
|
58
81
|
signal: AbortSignal.timeout(TIMEOUT_MS),
|
|
59
82
|
});
|
|
60
83
|
}
|
|
61
84
|
catch (err) {
|
|
62
85
|
throw new NectrUnreachableError(this.siteUrl, err);
|
|
63
86
|
}
|
|
87
|
+
if (res.status >= 300 && res.status < 400) {
|
|
88
|
+
throw new NectrRedirectError(this.siteUrl, res.headers.get("location"));
|
|
89
|
+
}
|
|
64
90
|
if (!res.ok) {
|
|
65
91
|
throw new NectrApiError(res.status, await readDetail(res), parseRetryAfter(res));
|
|
66
92
|
}
|
|
@@ -104,6 +130,10 @@ export function describeError(err) {
|
|
|
104
130
|
if (err instanceof NectrUnreachableError) {
|
|
105
131
|
return `Could not reach nectr at ${err.siteUrl}. Check the network or NECTR_URL; retrying immediately is unlikely to help.`;
|
|
106
132
|
}
|
|
133
|
+
if (err instanceof NectrRedirectError) {
|
|
134
|
+
const to = err.location ? ` to ${err.location}` : "";
|
|
135
|
+
return `nectr at ${err.siteUrl} redirected${to}, so the request was not sent on. Do not retry; the user needs to set NECTR_URL to nectr's https address, or remove it to use https://nectr.ch.`;
|
|
136
|
+
}
|
|
107
137
|
if (!(err instanceof NectrApiError)) {
|
|
108
138
|
return `Unexpected error: ${err instanceof Error ? err.message : String(err)}`;
|
|
109
139
|
}
|
package/dist/server.js
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
|
|
2
2
|
import { registerTools } from "./tools.js";
|
|
3
|
-
export const VERSION = "0.
|
|
3
|
+
export const VERSION = "0.4.0";
|
|
4
4
|
export function createServer(client) {
|
|
5
5
|
const server = new McpServer({ name: "nectr", version: VERSION });
|
|
6
6
|
registerTools(server, client);
|
package/dist/tools.js
CHANGED
|
@@ -4,6 +4,8 @@ import { describeError } from "./client.js";
|
|
|
4
4
|
export const DEFAULT_MAX_CHARS = 40_000;
|
|
5
5
|
/** How many "only with JavaScript" blocks to list; nectr returns up to 50. */
|
|
6
6
|
export const JS_BLOCKS_SHOWN = 10;
|
|
7
|
+
/** How many pages of one site list_sites shows. */
|
|
8
|
+
export const PAGES_SHOWN = 100;
|
|
7
9
|
export const MARKDOWN_CHOICES = ["full", "without_javascript", "none"];
|
|
8
10
|
const plural = (n, word) => `${n.toLocaleString("en")} ${word}${n === 1 ? "" : "s"}`;
|
|
9
11
|
const text = (t) => ({ content: [{ type: "text", text: t }] });
|
|
@@ -66,7 +68,11 @@ function markdown(scan, choice, maxChars) {
|
|
|
66
68
|
if (choice === "none")
|
|
67
69
|
return [];
|
|
68
70
|
if (!scan.markdown_included) {
|
|
69
|
-
return [
|
|
71
|
+
return [
|
|
72
|
+
"## Markdown",
|
|
73
|
+
"Not included: nectr only returns the markdown for pages on the user's own verified sites, " +
|
|
74
|
+
"and this page isn't on one. Call list_sites to see which sites and pages are the user's.",
|
|
75
|
+
];
|
|
70
76
|
}
|
|
71
77
|
const full = choice === "full";
|
|
72
78
|
const body = (full ? scan.rendered_markdown : scan.plain_markdown) ?? "";
|
|
@@ -104,6 +110,44 @@ export async function scanPage(client, args) {
|
|
|
104
110
|
return failure(err);
|
|
105
111
|
}
|
|
106
112
|
}
|
|
113
|
+
function page(p) {
|
|
114
|
+
const facts = [
|
|
115
|
+
p.title,
|
|
116
|
+
p.words !== null ? plural(p.words, "word") : null,
|
|
117
|
+
p.status === "copied" ? null : `${p.status}${p.reason ? ` (${p.reason})` : ""}`,
|
|
118
|
+
].filter(Boolean);
|
|
119
|
+
return `- ${p.url}${facts.length > 0 ? `: ${facts.join(", ")}` : ""}`;
|
|
120
|
+
}
|
|
121
|
+
function site(s, pages) {
|
|
122
|
+
const state = !s.verified
|
|
123
|
+
? "not verified, so scan_page won't include markdown for its pages"
|
|
124
|
+
: s.paused_at
|
|
125
|
+
? "verified, paused"
|
|
126
|
+
: "verified";
|
|
127
|
+
const shown = pages.slice(0, PAGES_SHOWN).map(page);
|
|
128
|
+
const more = pages.length > PAGES_SHOWN ? [`- …and ${pages.length - PAGES_SHOWN} more.`] : [];
|
|
129
|
+
return [
|
|
130
|
+
`## ${s.domain}`,
|
|
131
|
+
`${state} · ${plural(pages.length, "page")}, ${s.pages_copied.toLocaleString("en")} copied as markdown` +
|
|
132
|
+
(s.llms_txt_url ? ` · llms.txt: ${s.llms_txt_url}` : ""),
|
|
133
|
+
...(pages.length === 0 ? ["No pages listed yet."] : [...shown, ...more]),
|
|
134
|
+
];
|
|
135
|
+
}
|
|
136
|
+
// ── Handlers (exported for tests) ───────────────────────────────────────────
|
|
137
|
+
export async function listSites(client) {
|
|
138
|
+
try {
|
|
139
|
+
const sites = await client.sites();
|
|
140
|
+
if (sites.length === 0) {
|
|
141
|
+
return text("The user has no sites in nectr, so scan_page won't include markdown for any page.");
|
|
142
|
+
}
|
|
143
|
+
const pages = await Promise.all(sites.map((s) => client.pages(s.id)));
|
|
144
|
+
const sections = sites.map((s, i) => site(s, pages[i]));
|
|
145
|
+
return text([`# The user's sites in nectr`, ...sections.map((s) => s.join("\n"))].join("\n\n"));
|
|
146
|
+
}
|
|
147
|
+
catch (err) {
|
|
148
|
+
return failure(err);
|
|
149
|
+
}
|
|
150
|
+
}
|
|
107
151
|
// ── Registration ────────────────────────────────────────────────────────────
|
|
108
152
|
export function registerTools(server, client) {
|
|
109
153
|
server.registerTool("scan_page", {
|
|
@@ -111,7 +155,8 @@ export function registerTools(server, client) {
|
|
|
111
155
|
description: "Fetch one public web page with nectr and return how AI agents see it: the page as clean markdown, " +
|
|
112
156
|
"its size in words and estimated tokens (against the raw HTML), which AI crawlers its robots.txt " +
|
|
113
157
|
"allows, linked PDFs and whether they have text, and the text that only appears after JavaScript " +
|
|
114
|
-
"runs.
|
|
158
|
+
"runs. Any public page can be scanned, but the markdown is only included for pages on the user's " +
|
|
159
|
+
"own verified sites (see list_sites). Nothing is stored. nectr honours the site's robots.txt for itself.",
|
|
115
160
|
inputSchema: {
|
|
116
161
|
url: z.url().describe("The http(s) address of a public web page."),
|
|
117
162
|
markdown: z
|
|
@@ -128,4 +173,11 @@ export function registerTools(server, client) {
|
|
|
128
173
|
},
|
|
129
174
|
annotations: { readOnlyHint: true, idempotentHint: true, openWorldHint: true },
|
|
130
175
|
}, (args) => scanPage(client, args));
|
|
176
|
+
server.registerTool("list_sites", {
|
|
177
|
+
title: "List the user's sites and pages",
|
|
178
|
+
description: "List the user's sites in nectr and the URLs of their pages, with whether each site is verified. " +
|
|
179
|
+
"scan_page includes the markdown only for pages on these verified sites.",
|
|
180
|
+
inputSchema: {},
|
|
181
|
+
annotations: { readOnlyHint: true, idempotentHint: true, openWorldHint: false },
|
|
182
|
+
}, () => listSites(client));
|
|
131
183
|
}
|