nectr-mcp 0.4.1 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +53 -3
- package/dist/client.js +29 -2
- package/dist/server.js +1 -1
- package/dist/tools.js +124 -10
- package/package.json +2 -1
package/README.md
CHANGED
|
@@ -4,9 +4,11 @@ An [MCP](https://modelcontextprotocol.io) server for [nectr](https://nectr.ch),
|
|
|
4
4
|
|
|
5
5
|
It gives a model two tools:
|
|
6
6
|
|
|
7
|
-
- **`scan_page`**: given a public page's address, it returns what an AI agent gets from it: the size in words and estimated tokens, which AI crawlers the site's robots.txt allows, linked PDFs and whether they hold text, and the text that only appears after JavaScript runs. For pages on your own verified sites, it also returns the page as clean markdown.
|
|
8
|
-
- **`list_sites`**: lists your sites in nectr and their pages' URLs, so you can see which pages you'll get markdown for.
|
|
7
|
+
- **`scan_page`**: given a public page's address, it returns what an AI agent gets from it: the size in words and estimated tokens, which AI crawlers the site's robots.txt allows, linked PDFs and whether they hold text, its data tables and the structured data it declares (JSON-LD, microdata, schema.org RDFa), and the text that only appears after JavaScript runs. For pages on your own verified sites, it also returns the page as clean markdown, the tables' rows and the structured data itself.
|
|
8
|
+
- **`list_sites`**: lists your sites in nectr and their pages' URLs, so you can see which pages you'll get markdown for, and the datasets you've published from each site with their addresses.
|
|
9
9
|
|
|
10
|
+
> **New in 0.5:** `scan_page` reports a page's data tables and the structured data it declares, and `list_sites` shows the datasets you've published from each site.
|
|
11
|
+
>
|
|
10
12
|
> **Coming from 0.3?** `scan_page` now includes the markdown only for pages on your own verified sites. Any other page still gets the full scan, without the markdown.
|
|
11
13
|
>
|
|
12
14
|
> **Coming from 0.2?** The library tools (`search_library`, `get_page`, `convert_url`) are gone. Your existing `nct_` key works with these tools.
|
|
@@ -50,8 +52,11 @@ Node.js 20 or newer is required.
|
|
|
50
52
|
| Size | Words and estimated tokens of the whole page as markdown, against the raw HTML an agent receives, and the words left without JavaScript |
|
|
51
53
|
| robots.txt | Whether GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and CCBot may fetch the page |
|
|
52
54
|
| Linked PDFs | Up to 3 PDFs on the same site: pages, words, and whether there's a text layer at all |
|
|
55
|
+
| Tables | Each data table's title, columns and row count (the first 20). The rows are in the markdown, for a page on one of your verified sites. Left out when the page has none |
|
|
56
|
+
| Structured data | What the page declares about itself, experimental on nectr's side: each item's schema.org types and where it was declared (JSON-LD, microdata or RDFa, the first 20), and its definition lists. "None" when it declares nothing |
|
|
53
57
|
| Only with JavaScript | The share of words an agent misses without JavaScript, and the first 10 of those blocks. For a page that isn't on your verified sites, nectr sends only the first 5, shortened, so the tool doesn't count them |
|
|
54
58
|
| Markdown | The page as markdown, last, and only for a page on one of your verified sites; otherwise the tool says it was left out and points to `list_sites`. `markdown` picks `"full"` (after JavaScript runs, the default), `"without_javascript"` or `"none"`. It's cut at 40,000 characters unless you raise `max_chars` |
|
|
59
|
+
| Structured data as JSON | For a page on one of your verified sites, after the markdown: the structured data itself, compact, cut at `max_chars` like the markdown |
|
|
55
60
|
|
|
56
61
|
Any public page can be scanned. The tool is read-only: nectr stores nothing from a scan. nectr honours the site's robots.txt for itself, so a page whose robots.txt keeps nectr out isn't fetched.
|
|
57
62
|
|
|
@@ -61,6 +66,8 @@ The full response, field by field, is in the [scan API documentation](https://ne
|
|
|
61
66
|
|
|
62
67
|
`list_sites()` takes no arguments. For each of your sites it gives the domain, whether it's verified (only verified sites get markdown from `scan_page`), how many pages nectr has copied as markdown, the site's llms.txt address, and the URL of each page (up to 100 per site) with its title and word count. A page nectr couldn't copy shows its status and the reason.
|
|
63
68
|
|
|
69
|
+
Before the pages come the datasets you've published from the site with [Publish](https://nectr.ch/docs#publish) (up to 50 per site): each one's name, kind (table, list, structured data or page content), title and source page, its JSON address and, for tables and lists, its CSV address, then the site's catalogue. A dataset that wasn't on its page at the last fetch says so, and from when it's deleted if it stays missing. Publishing and renaming datasets stay in nectr's dashboard: the tool only reads them. When nectr doesn't answer for the datasets, the site is listed without them.
|
|
70
|
+
|
|
64
71
|
## Limits and errors
|
|
65
72
|
|
|
66
73
|
The server uses the same developer API as any other script, so the API's limits apply: 10 scans a minute from one IP address. Errors are worded so a model knows whether to retry:
|
|
@@ -73,6 +80,8 @@ The server uses the same developer API as any other script, so the API's limits
|
|
|
73
80
|
| 429: rate limited | How many seconds to wait |
|
|
74
81
|
| 503: nectr is busy | Retry in a few seconds |
|
|
75
82
|
| A redirect (for example `NECTR_URL` set to `http://…`) | Don't retry; set `NECTR_URL` to nectr's https address |
|
|
83
|
+
| No answer within 90 seconds (a page with large linked PDFs can take that long) | nectr was reached but slow; trying once more may work |
|
|
84
|
+
| No answer at all (network, DNS, TLS) | nectr couldn't be reached; check the network or `NECTR_URL` |
|
|
76
85
|
|
|
77
86
|
## Security
|
|
78
87
|
|
|
@@ -80,7 +89,7 @@ The server uses the same developer API as any other script, so the API's limits
|
|
|
80
89
|
|
|
81
90
|
- Create a dedicated key for each client, so you can revoke one without breaking the others.
|
|
82
91
|
- If a key leaks, delete it under **Settings → API keys** in nectr. It stops working immediately.
|
|
83
|
-
- This server only ever scans pages and reads your
|
|
92
|
+
- This server only ever scans pages and reads your sites, their pages and their published datasets. **The key itself is broader:** it has the same access to the nectr API as your account, so treat it like a password.
|
|
84
93
|
|
|
85
94
|
## Development
|
|
86
95
|
|
|
@@ -117,10 +126,51 @@ The Inspector's own dependencies need a newer Node than this server does. If it
|
|
|
117
126
|
|
|
118
127
|
The quickest sanity check needs no key at all: `npx -y nectr-mcp` should print "NECTR_API_KEY is not set" and exit.
|
|
119
128
|
|
|
129
|
+
### Checks before publishing
|
|
130
|
+
|
|
131
|
+
`npm run smoke` starts the built server the way an MCP client does and runs the release checks against nectr with your key: the tools and their read-only marks, `list_sites`, a page on one of your verified sites (markdown, tables, structured data as JSON, `markdown=none`, `max_chars`), a page that declares nothing and a page that isn't yours. Each prints PASS, SKIP (with the reason) or FAIL, and the script exits 1 if anything failed. It runs four or five scans.
|
|
132
|
+
|
|
133
|
+
```bash
|
|
134
|
+
npm run build
|
|
135
|
+
NECTR_API_KEY=nct_... npm run smoke
|
|
136
|
+
# optional: NECTR_SMOKE_PAGE=<a page of yours with a table and JSON-LD>, NECTR_SMOKE_FOREIGN=<someone else's page>, NECTR_URL=…
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
It stops after `list_sites` when the key is rejected: nectr runs a scan sent with a rejected key as an anonymous one, so the scan checks would test nothing.
|
|
140
|
+
|
|
141
|
+
### Trying it in Claude
|
|
142
|
+
|
|
143
|
+
To try a local build in a real conversation before publishing, point the client at `dist/index.js` instead of `npx`, and rebuild after each change.
|
|
144
|
+
|
|
145
|
+
**Claude Desktop** (Settings → Developer → Edit Config, or `~/Library/Application Support/Claude/claude_desktop_config.json` on macOS). Desktop doesn't start from your shell, so give `node`'s full path (`which node`; with nvm it changes when you switch versions):
|
|
146
|
+
|
|
147
|
+
```json
|
|
148
|
+
{
|
|
149
|
+
"mcpServers": {
|
|
150
|
+
"nectr-dev": {
|
|
151
|
+
"command": "/full/path/to/node",
|
|
152
|
+
"args": ["/full/path/to/nectr-mcp/dist/index.js"],
|
|
153
|
+
"env": { "NECTR_API_KEY": "nct_your_key_here" }
|
|
154
|
+
}
|
|
155
|
+
}
|
|
156
|
+
}
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
Quit Desktop completely and open it again; `nectr-dev` then shows under the tools button. Ask, for example, "Which of my nectr sites have published datasets?" or "Scan https://example.com/ with nectr and tell me what it declares", and check that the answer quotes the tool. Its logs are in `~/Library/Logs/Claude/mcp-server-nectr-dev.log`.
|
|
160
|
+
|
|
161
|
+
**Claude Code**, from this folder:
|
|
162
|
+
|
|
163
|
+
```bash
|
|
164
|
+
claude mcp add nectr-dev -e NECTR_API_KEY=nct_... -- node "$PWD/dist/index.js"
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
Then `/mcp` in a session lists it. `claude mcp remove nectr-dev` takes it out again.
|
|
168
|
+
|
|
120
169
|
### Publishing
|
|
121
170
|
|
|
122
171
|
```bash
|
|
123
172
|
npm version patch # or minor / major; also bump VERSION in src/server.ts and USER_AGENT in src/client.ts
|
|
173
|
+
npm run build && NECTR_API_KEY=nct_... npm run smoke # every check PASS or SKIP
|
|
124
174
|
npm publish # prepublishOnly runs typecheck, tests and build first
|
|
125
175
|
```
|
|
126
176
|
|
package/dist/client.js
CHANGED
|
@@ -1,7 +1,8 @@
|
|
|
1
1
|
/**
|
|
2
2
|
* A thin client over the public nectr API (`/api/v1`), authenticated with an
|
|
3
3
|
* `nct_` API key. It uses the scan endpoint (nectr.ch/docs#scan) and the key
|
|
4
|
-
* holder's sites
|
|
4
|
+
* holder's sites, their pages and their published datasets, all read-only;
|
|
5
|
+
* this package adds no new server-side surface.
|
|
5
6
|
*/
|
|
6
7
|
/** An HTTP error from nectr, carrying what a caller needs to explain it. */
|
|
7
8
|
export class NectrApiError extends Error {
|
|
@@ -25,6 +26,21 @@ export class NectrUnreachableError extends Error {
|
|
|
25
26
|
this.name = "NectrUnreachableError";
|
|
26
27
|
}
|
|
27
28
|
}
|
|
29
|
+
/**
|
|
30
|
+
* nectr was reached but didn't answer in time. Not the same as unreachable:
|
|
31
|
+
* a scan of a page with large linked PDFs can take this long, and trying once
|
|
32
|
+
* more may work.
|
|
33
|
+
*/
|
|
34
|
+
export class NectrTimeoutError extends Error {
|
|
35
|
+
siteUrl;
|
|
36
|
+
seconds;
|
|
37
|
+
constructor(siteUrl, seconds) {
|
|
38
|
+
super(`nectr at ${siteUrl} didn't answer within ${seconds} seconds`);
|
|
39
|
+
this.siteUrl = siteUrl;
|
|
40
|
+
this.seconds = seconds;
|
|
41
|
+
this.name = "NectrTimeoutError";
|
|
42
|
+
}
|
|
43
|
+
}
|
|
28
44
|
/**
|
|
29
45
|
* nectr answered with a redirect, which the API never should. Following it is
|
|
30
46
|
* worse than failing: fetch drops the Authorization header on a redirect to
|
|
@@ -41,7 +57,7 @@ export class NectrRedirectError extends Error {
|
|
|
41
57
|
this.name = "NectrRedirectError";
|
|
42
58
|
}
|
|
43
59
|
}
|
|
44
|
-
const USER_AGENT = "nectr-mcp/0.
|
|
60
|
+
const USER_AGENT = "nectr-mcp/0.5.0";
|
|
45
61
|
// A scan fetches the page twice (a browser and plain HTTP) and checks linked
|
|
46
62
|
// PDFs, so it can take a while.
|
|
47
63
|
const TIMEOUT_MS = 90_000;
|
|
@@ -65,6 +81,9 @@ export class NectrClient {
|
|
|
65
81
|
pages(siteId) {
|
|
66
82
|
return this.request("GET", `/sites/${encodeURIComponent(siteId)}/pages`);
|
|
67
83
|
}
|
|
84
|
+
datasets(siteId) {
|
|
85
|
+
return this.request("GET", `/sites/${encodeURIComponent(siteId)}/datasets`);
|
|
86
|
+
}
|
|
68
87
|
async request(method, path, body) {
|
|
69
88
|
let res;
|
|
70
89
|
try {
|
|
@@ -82,6 +101,10 @@ export class NectrClient {
|
|
|
82
101
|
});
|
|
83
102
|
}
|
|
84
103
|
catch (err) {
|
|
104
|
+
// AbortSignal.timeout aborts with a DOMException named "TimeoutError".
|
|
105
|
+
if (err instanceof Error && err.name === "TimeoutError") {
|
|
106
|
+
throw new NectrTimeoutError(this.siteUrl, TIMEOUT_MS / 1000);
|
|
107
|
+
}
|
|
85
108
|
throw new NectrUnreachableError(this.siteUrl, err);
|
|
86
109
|
}
|
|
87
110
|
if (res.status >= 300 && res.status < 400) {
|
|
@@ -130,6 +153,10 @@ export function describeError(err) {
|
|
|
130
153
|
if (err instanceof NectrUnreachableError) {
|
|
131
154
|
return `Could not reach nectr at ${err.siteUrl}. Check the network or NECTR_URL; retrying immediately is unlikely to help.`;
|
|
132
155
|
}
|
|
156
|
+
if (err instanceof NectrTimeoutError) {
|
|
157
|
+
return (`nectr at ${err.siteUrl} didn't answer within ${err.seconds} seconds, so the request was abandoned. ` +
|
|
158
|
+
"nectr was reached; a page with large linked PDFs can take this long to scan. Trying once more may work.");
|
|
159
|
+
}
|
|
133
160
|
if (err instanceof NectrRedirectError) {
|
|
134
161
|
const to = err.location ? ` to ${err.location}` : "";
|
|
135
162
|
return `nectr at ${err.siteUrl} redirected${to}, so the request was not sent on. Do not retry; the user needs to set NECTR_URL to nectr's https address, or remove it to use https://nectr.ch.`;
|
package/dist/server.js
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
|
|
2
2
|
import { registerTools } from "./tools.js";
|
|
3
|
-
export const VERSION = "0.
|
|
3
|
+
export const VERSION = "0.5.0";
|
|
4
4
|
export function createServer(client) {
|
|
5
5
|
const server = new McpServer({ name: "nectr", version: VERSION });
|
|
6
6
|
registerTools(server, client);
|
package/dist/tools.js
CHANGED
|
@@ -6,6 +6,11 @@ export const DEFAULT_MAX_CHARS = 40_000;
|
|
|
6
6
|
export const JS_BLOCKS_SHOWN = 10;
|
|
7
7
|
/** How many pages of one site list_sites shows. */
|
|
8
8
|
export const PAGES_SHOWN = 100;
|
|
9
|
+
/** How many tables, and how many structured data items, scan_page lists; nectr returns up to 50 and 100. */
|
|
10
|
+
export const TABLES_SHOWN = 20;
|
|
11
|
+
export const ITEMS_SHOWN = 20;
|
|
12
|
+
/** How many published datasets list_sites shows per site; nectr allows up to 200. */
|
|
13
|
+
export const DATASETS_SHOWN = 50;
|
|
9
14
|
export const MARKDOWN_CHOICES = ["full", "without_javascript", "none"];
|
|
10
15
|
const plural = (n, word) => `${n.toLocaleString("en")} ${word}${n === 1 ? "" : "s"}`;
|
|
11
16
|
const text = (t) => ({ content: [{ type: "text", text: t }] });
|
|
@@ -48,6 +53,74 @@ function pdfs(scan) {
|
|
|
48
53
|
}
|
|
49
54
|
return ["## Linked PDFs", ...lines];
|
|
50
55
|
}
|
|
56
|
+
const SOURCES = { "json-ld": "JSON-LD", microdata: "microdata", rdfa: "RDFa" };
|
|
57
|
+
const NOT_THE_OWNERS = "only for pages on the user's own verified sites (see list_sites)";
|
|
58
|
+
function tables(scan, choice) {
|
|
59
|
+
// Absent from an older nectr; empty when the page has none.
|
|
60
|
+
if (!scan.tables || scan.tables.length === 0)
|
|
61
|
+
return [];
|
|
62
|
+
const shown = scan.tables.slice(0, TABLES_SHOWN).map((t, i) => {
|
|
63
|
+
const columns = `${plural(t.columns.length, "column")} (${t.columns.join(", ")})`;
|
|
64
|
+
return `- ${t.title || `Table ${i + 1}`}: ${columns}, ${plural(t.row_count, "row")}`;
|
|
65
|
+
});
|
|
66
|
+
const more = scan.tables.length > TABLES_SHOWN ? [`- …and ${scan.tables.length - TABLES_SHOWN} more.`] : [];
|
|
67
|
+
// The owner's rows are already in the markdown, as markdown tables:
|
|
68
|
+
// repeating them would double the output.
|
|
69
|
+
const rows = !scan.markdown_included
|
|
70
|
+
? `Their rows are ${NOT_THE_OWNERS}.`
|
|
71
|
+
: choice === "full"
|
|
72
|
+
? "Their rows are in the markdown below."
|
|
73
|
+
: 'For their rows, call scan_page with markdown "full".';
|
|
74
|
+
return ["## Tables", `${plural(scan.tables.length, "data table")} on the page, with JavaScript:`, ...shown, ...more, rows];
|
|
75
|
+
}
|
|
76
|
+
function structuredData(scan) {
|
|
77
|
+
const data = scan.structured_data;
|
|
78
|
+
if (!data)
|
|
79
|
+
return []; // an older nectr
|
|
80
|
+
const skipped = data.skipped_json_ld > 0
|
|
81
|
+
? [`nectr left out ${plural(data.skipped_json_ld, "JSON-LD block")}: not valid JSON, or too large.`]
|
|
82
|
+
: [];
|
|
83
|
+
if (data.items.length === 0 && data.definition_lists.length === 0) {
|
|
84
|
+
return [
|
|
85
|
+
"## Structured data",
|
|
86
|
+
"None: the page declares no JSON-LD, microdata or schema.org RDFa, and has no definition lists.",
|
|
87
|
+
...skipped,
|
|
88
|
+
];
|
|
89
|
+
}
|
|
90
|
+
const items = data.items
|
|
91
|
+
.slice(0, ITEMS_SHOWN)
|
|
92
|
+
.map((item) => `- ${item.types.join(", ") || "No type"} (${SOURCES[item.source] ?? item.source})`);
|
|
93
|
+
const more = data.items.length > ITEMS_SHOWN ? [`- …and ${data.items.length - ITEMS_SHOWN} more.`] : [];
|
|
94
|
+
const lists = data.definition_lists.length > 0
|
|
95
|
+
? [`- ${plural(data.definition_lists.length, "definition list")} (<dl>), ${plural(data.definition_lists.reduce((n, d) => n + d.term_count, 0), "term")} in all`]
|
|
96
|
+
: [];
|
|
97
|
+
const where = scan.markdown_included ? "The data itself is at the end, as JSON." : `The data itself is ${NOT_THE_OWNERS}.`;
|
|
98
|
+
return [
|
|
99
|
+
"## Structured data",
|
|
100
|
+
"What the page declares about itself, in page order (nectr marks this experimental):",
|
|
101
|
+
...items,
|
|
102
|
+
...more,
|
|
103
|
+
...lists,
|
|
104
|
+
...skipped,
|
|
105
|
+
where,
|
|
106
|
+
];
|
|
107
|
+
}
|
|
108
|
+
/** For the owner: the structured data itself, compact, last like the markdown. */
|
|
109
|
+
function structuredDataJson(scan, maxChars) {
|
|
110
|
+
const data = scan.structured_data;
|
|
111
|
+
if (!data || !scan.markdown_included || (data.items.length === 0 && data.definition_lists.length === 0))
|
|
112
|
+
return [];
|
|
113
|
+
const body = JSON.stringify({ items: data.items, definition_lists: data.definition_lists });
|
|
114
|
+
const heading = "## Structured data as JSON";
|
|
115
|
+
if (body.length <= maxChars)
|
|
116
|
+
return [heading, body];
|
|
117
|
+
return [
|
|
118
|
+
heading,
|
|
119
|
+
body.slice(0, maxChars),
|
|
120
|
+
`[Truncated: showing ${maxChars.toLocaleString("en")} of ${body.length.toLocaleString("en")} characters. ` +
|
|
121
|
+
"Call scan_page again with a larger max_chars for the rest.]",
|
|
122
|
+
];
|
|
123
|
+
}
|
|
51
124
|
function onlyWithJavaScript(scan) {
|
|
52
125
|
const blocks = scan.only_with_javascript;
|
|
53
126
|
if (blocks.length === 0) {
|
|
@@ -106,8 +179,11 @@ export async function scanPage(client, args) {
|
|
|
106
179
|
size(scan),
|
|
107
180
|
robots(scan),
|
|
108
181
|
pdfs(scan),
|
|
182
|
+
tables(scan, choice),
|
|
183
|
+
structuredData(scan),
|
|
109
184
|
onlyWithJavaScript(scan),
|
|
110
185
|
markdown(scan, choice, maxChars),
|
|
186
|
+
structuredDataJson(scan, maxChars),
|
|
111
187
|
].filter((s) => s.length > 0);
|
|
112
188
|
return text(sections.map((s) => s.join("\n")).join("\n\n"));
|
|
113
189
|
}
|
|
@@ -123,7 +199,35 @@ function page(p) {
|
|
|
123
199
|
].filter(Boolean);
|
|
124
200
|
return `- ${p.url}${facts.length > 0 ? `: ${facts.join(", ")}` : ""}`;
|
|
125
201
|
}
|
|
126
|
-
|
|
202
|
+
const KINDS = {
|
|
203
|
+
table: "table",
|
|
204
|
+
list: "list",
|
|
205
|
+
structured: "structured data",
|
|
206
|
+
content: "page content",
|
|
207
|
+
};
|
|
208
|
+
function dataset(d) {
|
|
209
|
+
const what = `${KINDS[d.kind] ?? d.kind}${d.title ? ` "${d.title}"` : ""} from ${d.page_url}`;
|
|
210
|
+
const where = d.found
|
|
211
|
+
? `JSON ${d.json_url}${d.csv_url ? ` · CSV ${d.csv_url}` : ""}`
|
|
212
|
+
: "not on its page at the last fetch, so its addresses answer not found" +
|
|
213
|
+
(d.deleted_after ? `; deleted if still missing at a fetch from ${d.deleted_after.slice(0, 10)}` : "");
|
|
214
|
+
return `- ${d.name} (${what}): ${where}`;
|
|
215
|
+
}
|
|
216
|
+
/** The site's published datasets, or nothing when it has none or nectr didn't send them. */
|
|
217
|
+
function datasets(s, list) {
|
|
218
|
+
if (!list || list.length === 0)
|
|
219
|
+
return [];
|
|
220
|
+
const shown = list.slice(0, DATASETS_SHOWN).map(dataset);
|
|
221
|
+
const more = list.length > DATASETS_SHOWN ? [`- …and ${list.length - DATASETS_SHOWN} more.`] : [];
|
|
222
|
+
// The catalogue sits beside the datasets: …/s/<domain>/datasets.json.
|
|
223
|
+
const found = list.find((d) => d.found);
|
|
224
|
+
const catalogue = found
|
|
225
|
+
? [`All of the site's datasets that are on their pages, as JSON: ${found.json_url.replace(/\/datasets\/[^/]+\.json$/, "/datasets.json")}`]
|
|
226
|
+
: [];
|
|
227
|
+
const paused = s.paused_at ? ["The site is paused, so none of them is served until it's resumed."] : [];
|
|
228
|
+
return [`Published datasets (${list.length}):`, ...shown, ...more, ...catalogue, ...paused];
|
|
229
|
+
}
|
|
230
|
+
function site(s, pages, published) {
|
|
127
231
|
const state = !s.verified
|
|
128
232
|
? "not verified, so scan_page won't include markdown for its pages"
|
|
129
233
|
: s.paused_at
|
|
@@ -135,7 +239,9 @@ function site(s, pages) {
|
|
|
135
239
|
`## ${s.domain}`,
|
|
136
240
|
`${state} · ${plural(pages.length, "page")}, ${s.pages_copied.toLocaleString("en")} copied as markdown` +
|
|
137
241
|
(s.llms_txt_url ? ` · llms.txt: ${s.llms_txt_url}` : ""),
|
|
138
|
-
|
|
242
|
+
// Before the pages, which can run to a hundred lines.
|
|
243
|
+
...datasets(s, published),
|
|
244
|
+
...(pages.length === 0 ? ["No pages listed yet."] : ["Pages:", ...shown, ...more]),
|
|
139
245
|
];
|
|
140
246
|
}
|
|
141
247
|
// ── Handlers (exported for tests) ───────────────────────────────────────────
|
|
@@ -145,8 +251,13 @@ export async function listSites(client) {
|
|
|
145
251
|
if (sites.length === 0) {
|
|
146
252
|
return text("The user has no sites in nectr, so scan_page won't include markdown for any page.");
|
|
147
253
|
}
|
|
148
|
-
const pages = await Promise.all(
|
|
149
|
-
|
|
254
|
+
const [pages, published] = await Promise.all([
|
|
255
|
+
Promise.all(sites.map((s) => client.pages(s.id))),
|
|
256
|
+
// Datasets are extra: a nectr without Publish, or one that refuses the
|
|
257
|
+
// request, leaves them out rather than failing the whole list.
|
|
258
|
+
Promise.all(sites.map((s) => client.datasets(s.id).catch(() => null))),
|
|
259
|
+
]);
|
|
260
|
+
const sections = sites.map((s, i) => site(s, pages[i], published[i]));
|
|
150
261
|
return text([`# The user's sites in nectr`, ...sections.map((s) => s.join("\n"))].join("\n\n"));
|
|
151
262
|
}
|
|
152
263
|
catch (err) {
|
|
@@ -159,9 +270,11 @@ export function registerTools(server, client) {
|
|
|
159
270
|
title: "Scan a page the way AI reads it",
|
|
160
271
|
description: "Fetch one public web page with nectr and return how AI agents see it: the page as clean markdown, " +
|
|
161
272
|
"its size in words and estimated tokens (against the raw HTML), which AI crawlers its robots.txt " +
|
|
162
|
-
"allows, linked PDFs and whether they have text, and the
|
|
163
|
-
"
|
|
164
|
-
"
|
|
273
|
+
"allows, linked PDFs and whether they have text, its data tables and the structured data it declares " +
|
|
274
|
+
"(JSON-LD, microdata, schema.org RDFa), and the text that only appears after JavaScript runs. Any " +
|
|
275
|
+
"public page can be scanned, but the markdown, the tables' rows and the structured data itself are " +
|
|
276
|
+
"only included for pages on the user's own verified sites (see list_sites). Nothing is stored. nectr " +
|
|
277
|
+
"honours the site's robots.txt for itself.",
|
|
165
278
|
inputSchema: {
|
|
166
279
|
url: z.url().describe("The http(s) address of a public web page."),
|
|
167
280
|
markdown: z
|
|
@@ -174,14 +287,15 @@ export function registerTools(server, client) {
|
|
|
174
287
|
.int()
|
|
175
288
|
.min(1000)
|
|
176
289
|
.optional()
|
|
177
|
-
.describe(`Truncate the markdown after this many characters (default ${DEFAULT_MAX_CHARS}).`),
|
|
290
|
+
.describe(`Truncate the markdown, and the structured data's JSON, each after this many characters (default ${DEFAULT_MAX_CHARS}).`),
|
|
178
291
|
},
|
|
179
292
|
annotations: { readOnlyHint: true, idempotentHint: true, openWorldHint: true },
|
|
180
293
|
}, (args) => scanPage(client, args));
|
|
181
294
|
server.registerTool("list_sites", {
|
|
182
295
|
title: "List the user's sites and pages",
|
|
183
|
-
description: "List the user's sites in nectr and the URLs of their pages, with whether each site is verified
|
|
184
|
-
"
|
|
296
|
+
description: "List the user's sites in nectr and the URLs of their pages, with whether each site is verified, and " +
|
|
297
|
+
"the datasets the user has published from each site (tables, lists, structured data or whole pages, " +
|
|
298
|
+
"with their JSON and CSV addresses). scan_page includes the markdown only for pages on these verified sites.",
|
|
185
299
|
inputSchema: {},
|
|
186
300
|
annotations: { readOnlyHint: true, idempotentHint: true, openWorldHint: false },
|
|
187
301
|
}, () => listSites(client));
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "nectr-mcp",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.5.0",
|
|
4
4
|
"description": "MCP server for nectr: scan any public page the way AI reads it, from any MCP client.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -18,6 +18,7 @@
|
|
|
18
18
|
"build": "tsc -p tsconfig.build.json",
|
|
19
19
|
"typecheck": "tsc --noEmit",
|
|
20
20
|
"test": "vitest run",
|
|
21
|
+
"smoke": "node scripts/smoke.mjs",
|
|
21
22
|
"prepublishOnly": "npm run typecheck && npm test && npm run build"
|
|
22
23
|
},
|
|
23
24
|
"dependencies": {
|