@333eco/corpus 2.4.0 → 2.4.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +38 -11
- package/dist/corpus.json +1 -1
- package/package.json +1 -1
- package/src/base-tools.mjs +3 -3
- package/src/read-tools.mjs +43 -34
- package/src/search.mjs +95 -7
- package/src/server.mjs +11 -70
package/README.md
CHANGED
|
@@ -83,6 +83,18 @@ verbatim. **A zero result was recording the matcher's limits while being read as
|
|
|
83
83
|
⭐ On a zero result the response now names **which terms appear in no document at all**
|
|
84
84
|
(`absent_terms`), so a dead end says something instead of nothing.
|
|
85
85
|
|
|
86
|
+
⭐ **Diacritics are optional.** `metta` finds what `mettā` finds, and `Tonle Sap` reaches the
|
|
87
|
+
text that writes `Tonlé Sap` — before 2.4.1 a plain-ASCII query found 6 documents where the
|
|
88
|
+
marked one found 20. ⛔ **Fold to match, serve verbatim:** the fold touches only what is
|
|
89
|
+
compared, and every excerpt carries the corpus's own spelling, because a Pāli or Sanskrit
|
|
90
|
+
spelling is a tradition marker rather than a typo. Only Latin combining diacritics are folded;
|
|
91
|
+
Khmer and Burmese vowel signs are marks too, and are left exactly as written.
|
|
92
|
+
|
|
93
|
+
⭐ **Every result says it is an excerpt.** The text ends with a line naming how many of the
|
|
94
|
+
matches it shows and the `read_documents` call that reads those documents in full. It sits in
|
|
95
|
+
the result rather than only in the server's instructions, because a result reaches the model in
|
|
96
|
+
every client, at the moment it decides whether to keep reading.
|
|
97
|
+
|
|
86
98
|
### Reading a whole shelf
|
|
87
99
|
|
|
88
100
|
⭐ **Search answers from excerpts, and a model answers from what it opened.** A frontier
|
|
@@ -101,8 +113,15 @@ corpus already fits one — but a way to read the shelf.
|
|
|
101
113
|
- **Paged by size.** `max_bytes` defaults to 90,000 (under the 25,000-token point at which
|
|
102
114
|
Claude Code moves a tool result into a file) and may go to 400,000. **A document is never
|
|
103
115
|
split** — half a document verifies nothing — so one larger than the page arrives alone and
|
|
104
|
-
says
|
|
105
|
-
rather than serving page 2 of another corpus.
|
|
116
|
+
says so in its `[PAGE]` line. That last block names the cursor for the next page; a cursor from a
|
|
117
|
+
different corpus version is refused rather than serving page 2 of another corpus.
|
|
118
|
+
- **`list_documents` says how to read what it listed:** `read_with` carries the same filters as
|
|
119
|
+
`read_documents` arguments.
|
|
120
|
+
|
|
121
|
+
⚠️ **The server can point; it cannot make a model read.** Its instructions say *for a broad
|
|
122
|
+
question, read the relevant shelf in full* — deliberately conditional, because a whole shelf read
|
|
123
|
+
to answer a one-fact question is slower, costlier and often worse. Clients differ in whether they
|
|
124
|
+
show server instructions at all, which is why the pointers also live in the results.
|
|
106
125
|
|
|
107
126
|
⭐ **Every document ends with an `[END OF DOCUMENT — <slug> · <n> words]` line**, by every
|
|
108
127
|
route. The server never cuts a document, but a client may, and a cut document reads as a
|
|
@@ -176,15 +195,23 @@ hands every consumer an obligation nobody can discharge, so **the build fails**
|
|
|
176
195
|
rather than serving it — the same reasoning as the licence gate: a property, not a
|
|
177
196
|
rule someone has to remember.
|
|
178
197
|
|
|
179
|
-
**
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
198
|
+
**Documents come as text; everything else is structured.** `get_document` and
|
|
199
|
+
`read_documents` return **`content` only** — each document behind its provenance header
|
|
200
|
+
and ending at its `[END OF DOCUMENT]` line. The other tools return `structuredContent`
|
|
201
|
+
(the typed object) plus a readable or compact `content`, and anything a reader needs —
|
|
202
|
+
search's `reading` line included — is in the structured part.
|
|
203
|
+
|
|
204
|
+
⛔⛔ **Why the document tools carry no `structuredContent` (2.4.2).** From 2.0.0 to 2.4.1
|
|
205
|
+
they split by role: text in `content`, the envelope *without the body* in
|
|
206
|
+
`structuredContent`. A Claude client shown a result that carries `structuredContent` hands its
|
|
207
|
+
model **only that part** — so for eleven days **no document text reached any Claude-based
|
|
208
|
+
reader**, while every check here passed, because every check read the server's output
|
|
209
|
+
directly rather than through a client. The MCP spec assumes the two parts carry the same
|
|
210
|
+
information; a server that splits them is at the mercy of whichever part a client picks. The
|
|
211
|
+
document is the payload and its provenance already rides in the header, so the document tools
|
|
212
|
+
now send the text and nothing else — and `check-parity` fails if either ever carries
|
|
213
|
+
`structuredContent` again. `outputSchema` is deliberately not declared: a schema binds the server
|
|
214
|
+
on every future change.
|
|
188
215
|
|
|
189
216
|
**Text is returned verbatim and is never summarised.** Not a stylistic
|
|
190
217
|
preference — a summary cannot be hash-verified, so summarising at the server
|
package/dist/corpus.json
CHANGED
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@333eco/corpus",
|
|
3
|
-
"version": "2.4.
|
|
3
|
+
"version": "2.4.2",
|
|
4
4
|
"description": "An MCP server for an open-licensed corpus, served with verifiable provenance — every document carries its sha256, DOI and OpenTimestamps proof so a consuming agent can check its own citation.",
|
|
5
5
|
"license": "CC0-1.0",
|
|
6
6
|
"author": "Thon Ly",
|
package/src/base-tools.mjs
CHANGED
|
@@ -27,9 +27,9 @@ export const BASE_TOOLS = [
|
|
|
27
27
|
title: "Search the corpus",
|
|
28
28
|
annotations: { ...READ_ONLY, title: "Search the corpus" },
|
|
29
29
|
description:
|
|
30
|
-
"Full-text search across the open-licensed corpus. Returns matching documents with
|
|
31
|
-
"and a short excerpt around each match — not the full text;
|
|
32
|
-
"be
|
|
30
|
+
"Full-text search across the open-licensed corpus; diacritics are optional. Returns matching documents with " +
|
|
31
|
+
"a provenance envelope and a short excerpt around each match — not the full text; read_documents or " +
|
|
32
|
+
"get_document return that. Every result can be verified via its sha256, DOI and OpenTimestamps proof.",
|
|
33
33
|
inputSchema: {
|
|
34
34
|
type: "object",
|
|
35
35
|
properties: {
|
package/src/read-tools.mjs
CHANGED
|
@@ -43,7 +43,7 @@ export const READ_TOOLS = [
|
|
|
43
43
|
// facets are described once, on list_documents.
|
|
44
44
|
description:
|
|
45
45
|
"Whole documents, never summarised: those matching the list_documents filters, or the given slugs. " +
|
|
46
|
-
"One content block each
|
|
46
|
+
"One content block each; the last block names the cursor for the next page.",
|
|
47
47
|
inputSchema: {
|
|
48
48
|
type: "object",
|
|
49
49
|
properties: {
|
|
@@ -65,9 +65,27 @@ export const READ_TOOLS = [
|
|
|
65
65
|
|
|
66
66
|
export const READ_TOOL_NAMES = READ_TOOLS.map((t) => t.name);
|
|
67
67
|
|
|
68
|
+
// ⚠️ CONDITIONAL ON PURPOSE (2.4.1). "Prefer read_documents" unqualified would send a model to
|
|
69
|
+
// read a 260,000-token shelf to answer a one-fact question — slower, costlier and often worse.
|
|
70
|
+
// ⛔ It names shelves by their facets (list_documents' own vocabulary), never by what the
|
|
71
|
+
// documents say: describing the API, not the subject matter.
|
|
68
72
|
export const READ_INSTRUCTIONS =
|
|
69
|
-
"
|
|
70
|
-
"
|
|
73
|
+
" For a broad question, read the relevant shelf in full: choose it with list_documents, read it with " +
|
|
74
|
+
"read_documents. Search returns excerpts, suited to finding one passage. Every document ends with an " +
|
|
75
|
+
"[END OF DOCUMENT] line; if it is missing, the text was cut short.";
|
|
76
|
+
|
|
77
|
+
/**
|
|
78
|
+
* list_documents' pointer to reading what it just listed: the same filters, as arguments
|
|
79
|
+
* read_documents accepts. Only the facets actually given, so the pointer reads back exactly.
|
|
80
|
+
*/
|
|
81
|
+
export const readWith = (args, count) =>
|
|
82
|
+
count
|
|
83
|
+
? {
|
|
84
|
+
tool: "read_documents",
|
|
85
|
+
arguments: Object.fromEntries(["genre", "category", "voice", "licence"].filter((k) => args?.[k]).map((k) => [k, args[k]])),
|
|
86
|
+
note: "Reads every document listed here in full, paged by size."
|
|
87
|
+
}
|
|
88
|
+
: null;
|
|
71
89
|
|
|
72
90
|
// ⚠️ The cursor carries the INDEX VERSION as well as the offset. Pages are offsets
|
|
73
91
|
// into a list, and a list read across a redeploy is two lists: without the version
|
|
@@ -92,9 +110,9 @@ const decodeCursor = (cursor, version) => {
|
|
|
92
110
|
|
|
93
111
|
/**
|
|
94
112
|
* read_documents. `ctx` is what differs between the surfaces:
|
|
95
|
-
* { documents, bySlug,
|
|
113
|
+
* { documents, bySlug, version }.
|
|
96
114
|
*/
|
|
97
|
-
export const readDocuments = ({ documents, bySlug,
|
|
115
|
+
export const readDocuments = ({ documents, bySlug, version }, args) => {
|
|
98
116
|
const hasFilter = ["genre", "category", "voice", "licence"].some((k) => args?.[k]);
|
|
99
117
|
let matched;
|
|
100
118
|
if (args?.slugs !== undefined) {
|
|
@@ -131,43 +149,34 @@ export const readDocuments = ({ documents, bySlug, envelope, version }, args) =>
|
|
|
131
149
|
const next = nextOffset < matched.length ? encodeCursor(version, nextOffset) : undefined;
|
|
132
150
|
|
|
133
151
|
if (!matched.length) {
|
|
134
|
-
return {
|
|
135
|
-
content: [{ type: "text", text: "no documents match these filters. Call list_documents to see the facets and their counts." }],
|
|
136
|
-
structuredContent: { matched: 0, returned: 0, offset: 0, page_bytes: 0, max_bytes: budget, documents: [] }
|
|
137
|
-
};
|
|
152
|
+
return { content: [{ type: "text", text: "no documents match these filters. Call list_documents to see the facets and their counts." }] };
|
|
138
153
|
}
|
|
139
154
|
|
|
155
|
+
// ⛔⛔ TEXT ONLY — NO structuredContent (2.4.2, founder-ruled 2026-09-13). A Claude client shown a
|
|
156
|
+
// result that carries structuredContent hands its model ONLY that part; the envelope-without-body
|
|
157
|
+
// shape this tool shipped with in 2.4.0 therefore delivered no document to any Claude-based reader
|
|
158
|
+
// (§the-text-never-arrived). The document IS the payload, and the provenance already rides in each
|
|
159
|
+
// document's header, so the structured copy bought validation for programs at the price of the
|
|
160
|
+
// text for models. Everything a reader needs to keep going is in the [PAGE] line.
|
|
140
161
|
const content = page.map((p) => ({ type: "text", text: p.text }));
|
|
141
|
-
|
|
142
|
-
// order to keep going, or to know it has everything.
|
|
162
|
+
const over = page.length === 1 && used > budget ? " This document is larger than max_bytes and was returned whole rather than split." : "";
|
|
143
163
|
content.push({
|
|
144
164
|
type: "text",
|
|
145
165
|
text: next
|
|
146
|
-
? `[PAGE — documents ${start + 1}–${nextOffset} of ${matched.length}
|
|
147
|
-
: `[PAGE — documents ${start + 1}–${nextOffset} of ${matched.length}
|
|
166
|
+
? `[PAGE — documents ${start + 1}–${nextOffset} of ${matched.length}.${over} ${matched.length - nextOffset} more: call read_documents again with the same arguments and cursor "${next}".]`
|
|
167
|
+
: `[PAGE — documents ${start + 1}–${nextOffset} of ${matched.length}.${over} That is every matching document.]`
|
|
148
168
|
});
|
|
169
|
+
return { content };
|
|
170
|
+
};
|
|
149
171
|
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
...(page.length === 1 && used > budget ? { over_budget: "one document larger than max_bytes, returned whole rather than split" } : {}),
|
|
159
|
-
...(next ? { next_cursor: next } : {}),
|
|
160
|
-
documents: page.map((p) => ({
|
|
161
|
-
...envelope(p.d),
|
|
162
|
-
genre: p.d.genre,
|
|
163
|
-
category: p.d.category,
|
|
164
|
-
voice: p.d.voice,
|
|
165
|
-
bytes: p.d.bytes,
|
|
166
|
-
words: p.d.words,
|
|
167
|
-
text_in: "content, one block per document in this order, each prefixed by its provenance header"
|
|
168
|
-
}))
|
|
169
|
-
}
|
|
170
|
-
};
|
|
172
|
+
/**
|
|
173
|
+
* get_document — shared by both surfaces since 2.4.2, TEXT ONLY for the same reason as read_documents:
|
|
174
|
+
* the envelope-without-body in structuredContent was the only part a Claude client showed its model.
|
|
175
|
+
*/
|
|
176
|
+
export const getDocument = ({ bySlug }, args) => {
|
|
177
|
+
const d = bySlug.get(String(args?.slug ?? ""));
|
|
178
|
+
if (!d) throw new Error(`no document with slug "${args?.slug}". Call list_documents to see what is available.`);
|
|
179
|
+
return { content: [{ type: "text", text: documentText(d) }] };
|
|
171
180
|
};
|
|
172
181
|
|
|
173
182
|
/**
|
package/src/search.mjs
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
// How `search_corpus` decides a document matches.
|
|
2
2
|
//
|
|
3
|
-
// ⭐⭐ SHARED, AND IT HAS TO BE
|
|
4
|
-
//
|
|
5
|
-
//
|
|
3
|
+
// ⭐⭐ SHARED, AND IT HAS TO BE — matching and, since 2.4.1, the whole search_corpus
|
|
4
|
+
// tool (see searchCorpus below). Matching is ranking, tokenising and window-finding, and
|
|
5
|
+
// if the stdio server and the worker
|
|
6
6
|
// ever disagreed about which documents answer a query, the corpus would have two
|
|
7
7
|
// opinions about itself. That is the same failure `sync.mjs` exists to prevent,
|
|
8
8
|
// arriving through the search path instead of the data path.
|
|
@@ -33,15 +33,49 @@
|
|
|
33
33
|
// any document containing "b" and "heart" separately — precision traded away for
|
|
34
34
|
// nothing, in the corpus where those marks matter most.
|
|
35
35
|
|
|
36
|
+
import { structuredWithText } from "./results.mjs";
|
|
37
|
+
|
|
36
38
|
const TOKEN = /[\p{L}\p{N}]+(?:['’-][\p{L}\p{N}]+)*/gu;
|
|
37
39
|
|
|
40
|
+
// ⭐⭐ DIACRITICS ARE OPTIONAL IN A QUERY (2.4.1). A model types `metta`; the corpus writes
|
|
41
|
+
// `mettā`. Without a fold that search found 6 documents against 20, and `Tonle Sap` found none
|
|
42
|
+
// — a reader typing plain ASCII, which is most of them, silently missed most of the Pāli
|
|
43
|
+
// material. ⛔ FOLD TO MATCH, SERVE VERBATIM: the spelling is a tradition marker (Pāli for a
|
|
44
|
+
// Theravāda citation, Sanskrit for a Hindu one), so the fold touches only what is COMPARED —
|
|
45
|
+
// every excerpt is cut from the original text.
|
|
46
|
+
// ⚠️ LENGTH-PRESERVING, character by character, so a position in the folded text is the same
|
|
47
|
+
// position in the original and the excerpt window lands where the match is. Only Latin
|
|
48
|
+
// combining diacritics (U+0300–U+036F) are removed, and only where the base letter survives
|
|
49
|
+
// as one character: Khmer and Burmese vowel signs are marks too, and stripping them would
|
|
50
|
+
// make Khmer text match what it does not say.
|
|
51
|
+
const LATIN_MARKS = /[\u0300-\u036f]/g;
|
|
52
|
+
const foldChar = (c) => {
|
|
53
|
+
const f = c.normalize("NFD").replace(LATIN_MARKS, "");
|
|
54
|
+
return f.length === c.length ? f : c;
|
|
55
|
+
};
|
|
56
|
+
export const fold = (s) => String(s ?? "").replace(/[^\x00-\x7f]/gu, foldChar);
|
|
57
|
+
|
|
58
|
+
// The folded, lowercased body, computed once per document text rather than on every search.
|
|
59
|
+
// Keyed by the string itself: the corpus holds one string per document for the life of the
|
|
60
|
+
// process, so the cache is bounded by the corpus.
|
|
61
|
+
const HAY = new Map();
|
|
62
|
+
const hayOf = (body) => {
|
|
63
|
+
let h = HAY.get(body);
|
|
64
|
+
if (h === undefined) {
|
|
65
|
+
h = fold(body.toLowerCase());
|
|
66
|
+
HAY.set(body, h);
|
|
67
|
+
}
|
|
68
|
+
return h;
|
|
69
|
+
};
|
|
70
|
+
const foldQuery = (query) => fold(String(query ?? "").normalize("NFC").toLowerCase());
|
|
71
|
+
|
|
38
72
|
// Dropped only when something survives the dropping. These are words present in
|
|
39
73
|
// essentially every document, so they never narrow the result set — but they do
|
|
40
74
|
// wreck the tightest-window calculation, because an "and" is always close by.
|
|
41
75
|
const STOP = new Set(["a", "an", "and", "as", "at", "be", "by", "for", "from", "in", "is", "it", "of", "on", "or", "the", "to", "with"]);
|
|
42
76
|
|
|
43
77
|
export const terms = (query) => {
|
|
44
|
-
const all =
|
|
78
|
+
const all = foldQuery(query).match(TOKEN) ?? [];
|
|
45
79
|
const kept = all.filter((t) => !STOP.has(t));
|
|
46
80
|
return kept.length ? kept : all;
|
|
47
81
|
};
|
|
@@ -138,8 +172,8 @@ const tightestWindow = (lists) => {
|
|
|
138
172
|
// term match's excerpt may NOT contain the literal query, and a caller reading
|
|
139
173
|
// an excerpt deserves to know which question it answered.
|
|
140
174
|
export const match = (body, query, span = 320) => {
|
|
141
|
-
const hay = body
|
|
142
|
-
const phrase =
|
|
175
|
+
const hay = hayOf(body);
|
|
176
|
+
const phrase = foldQuery(query).trim();
|
|
143
177
|
|
|
144
178
|
if (phrase) {
|
|
145
179
|
const i = hay.indexOf(phrase);
|
|
@@ -175,6 +209,60 @@ export const rank = (hits, limit) =>
|
|
|
175
209
|
export const absentTerms = (documents, query) => {
|
|
176
210
|
const ts = [...new Set(terms(query))];
|
|
177
211
|
if (ts.length === 0) return [];
|
|
178
|
-
const hays = documents.map((d) => d.text
|
|
212
|
+
const hays = documents.map((d) => hayOf(d.text));
|
|
179
213
|
return ts.filter((t) => !hays.some((h) => h.includes(t)));
|
|
180
214
|
};
|
|
215
|
+
|
|
216
|
+
/* ---------------------------------------------------------- search_corpus ---
|
|
217
|
+
⭐ THE WHOLE TOOL, SHARED (2.4.1). It lived twice, once in each surface's callTool, and
|
|
218
|
+
its guidance line was about to be written twice too — the shape in which the envelope
|
|
219
|
+
drifted. One function, imported by both. */
|
|
220
|
+
|
|
221
|
+
// A search returns EXCERPTS, and a model that reads only excerpts answers from them. So the
|
|
222
|
+
// text a model reads says so at the moment it decides what to do next — in the result, where
|
|
223
|
+
// every client delivers it, rather than only in server instructions a client may drop.
|
|
224
|
+
const readingLine = (results, matches) => {
|
|
225
|
+
const slugs = results.map((r) => r.slug);
|
|
226
|
+
return (
|
|
227
|
+
`[Showing ${results.length} of ${matches} matching document${matches === 1 ? "" : "s"}, as excerpts. ` +
|
|
228
|
+
`To answer from the full texts, call read_documents with slugs ${JSON.stringify(slugs)}` +
|
|
229
|
+
(matches > results.length ? `; raise limit to see more of the ${matches} matches` : "") +
|
|
230
|
+
".]"
|
|
231
|
+
);
|
|
232
|
+
};
|
|
233
|
+
|
|
234
|
+
export const searchCorpus = ({ documents, envelope }, args) => {
|
|
235
|
+
const q = String(args?.query ?? "");
|
|
236
|
+
if (!q) throw new Error("query is required");
|
|
237
|
+
const pool = documents.filter((d) => !args?.genre || d.genre === args.genre);
|
|
238
|
+
const hits = pool.map((d) => ({ d, m: match(d.text, q) })).filter((h) => h.m !== null);
|
|
239
|
+
const results = rank(hits, Number(args?.limit ?? 10)).map((h) => ({
|
|
240
|
+
...envelope(h.d),
|
|
241
|
+
genre: h.d.genre,
|
|
242
|
+
excerpt: h.m.excerpt,
|
|
243
|
+
// ⭐ HOW it matched, not just that it did: a "terms" excerpt need not
|
|
244
|
+
// contain the literal query, and a caller reading it should know
|
|
245
|
+
// which question the excerpt is answering.
|
|
246
|
+
match: h.m.mode
|
|
247
|
+
}));
|
|
248
|
+
// ⚠️ `matches` is the TOTAL found, not the number returned — it used to be
|
|
249
|
+
// capped at `limit`, which made "10 matches" and "at least 10 matches"
|
|
250
|
+
// indistinguishable. `returned` carries the capped count.
|
|
251
|
+
const absent = hits.length ? [] : absentTerms(pool, q);
|
|
252
|
+
const readable = results.length
|
|
253
|
+
? results.map((h) => `${h.slug} — ${h.title}${h.match === "terms" ? " [all terms, not the phrase]" : ""}\n ${h.excerpt}`).join("\n\n") +
|
|
254
|
+
"\n\n" + readingLine(results, hits.length)
|
|
255
|
+
: absent.length
|
|
256
|
+
? `no document matches "${q}". No document contains: ${absent.join(", ")}.`
|
|
257
|
+
: `no document matches "${q}" — every term appears somewhere, but no single document holds them all. Try fewer terms.`;
|
|
258
|
+
return structuredWithText(readable, {
|
|
259
|
+
query: q,
|
|
260
|
+
matches: hits.length,
|
|
261
|
+
returned: results.length,
|
|
262
|
+
// ⛔ 2.4.2: the reading line was in `content` only, and a Claude client shows its model only
|
|
263
|
+
// structuredContent — so the 2.4.1 nudge reached no Claude reader. It rides here too.
|
|
264
|
+
...(results.length ? { reading: readingLine(results, hits.length) } : {}),
|
|
265
|
+
...(absent.length ? { absent_terms: absent } : {}),
|
|
266
|
+
results: results
|
|
267
|
+
});
|
|
268
|
+
};
|
package/src/server.mjs
CHANGED
|
@@ -35,14 +35,13 @@ import { dirname, join, resolve } from "node:path";
|
|
|
35
35
|
import { fileURLToPath } from "node:url";
|
|
36
36
|
import { createInterface } from "node:readline";
|
|
37
37
|
import { RESOURCE_TEMPLATES, listResources, readResource, completeArgument } from "./resources.mjs";
|
|
38
|
-
import { structured
|
|
39
|
-
import { documentText } from "./resources.mjs";
|
|
38
|
+
import { structured } from "./results.mjs";
|
|
40
39
|
import { envelope } from "./envelope.mjs";
|
|
41
40
|
import { PROMPTS, getPrompt } from "./prompts.mjs";
|
|
42
41
|
import { BASE_TOOLS, withFacets } from "./base-tools.mjs";
|
|
43
|
-
import { READ_TOOLS, READ_TOOL_NAMES, READ_INSTRUCTIONS, filterDocuments, readDocuments, completenessOf, manifestProblems } from "./read-tools.mjs";
|
|
42
|
+
import { READ_TOOLS, READ_TOOL_NAMES, READ_INSTRUCTIONS, filterDocuments, readDocuments, getDocument, readWith, completenessOf, manifestProblems } from "./read-tools.mjs";
|
|
44
43
|
import { PROGRAM_TOOLS, PROGRAM_TOOL_NAMES, PROGRAM_INSTRUCTIONS, callProgramTool } from "./program-tools.mjs";
|
|
45
|
-
import {
|
|
44
|
+
import { searchCorpus } from "./search.mjs";
|
|
46
45
|
|
|
47
46
|
const HERE = dirname(fileURLToPath(import.meta.url));
|
|
48
47
|
const INDEX = resolve(HERE, "..", "dist", "corpus.json");
|
|
@@ -121,77 +120,17 @@ const text = (value) => ({ content: [{ type: "text", text: typeof value === "str
|
|
|
121
120
|
|
|
122
121
|
const callTool = (name, args) => {
|
|
123
122
|
if (name === "search_corpus") {
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
const hits = pool.map((d) => ({ d, m: match(d.text, q) })).filter((h) => h.m !== null);
|
|
128
|
-
const results = rank(hits, Number(args?.limit ?? 10)).map((h) => ({
|
|
129
|
-
...envelope(h.d),
|
|
130
|
-
genre: h.d.genre,
|
|
131
|
-
excerpt: h.m.excerpt,
|
|
132
|
-
// ⭐ HOW it matched, not just that it did: a "terms" excerpt need not
|
|
133
|
-
// contain the literal query, and a caller reading it should know
|
|
134
|
-
// which question the excerpt is answering.
|
|
135
|
-
match: h.m.mode
|
|
136
|
-
}));
|
|
137
|
-
// ⚠️ `matches` is the TOTAL found, not the number returned — it used to be
|
|
138
|
-
// capped at `limit`, which made "10 matches" and "at least 10 matches"
|
|
139
|
-
// indistinguishable. `returned` carries the capped count.
|
|
140
|
-
const absent = hits.length ? [] : absentTerms(pool, q);
|
|
141
|
-
const readable = results.length
|
|
142
|
-
? results.map((h) => `${h.slug} — ${h.title}${h.match === "terms" ? " [all terms, not the phrase]" : ""}\n ${h.excerpt}`).join("\n\n")
|
|
143
|
-
: absent.length
|
|
144
|
-
? `no document matches "${q}". No document contains: ${absent.join(", ")}.`
|
|
145
|
-
: `no document matches "${q}" — every term appears somewhere, but no single document holds them all. Try fewer terms.`;
|
|
146
|
-
return structuredWithText(readable, {
|
|
147
|
-
query: q,
|
|
148
|
-
matches: hits.length,
|
|
149
|
-
returned: results.length,
|
|
150
|
-
...(absent.length ? { absent_terms: absent } : {}),
|
|
151
|
-
results: results
|
|
152
|
-
});
|
|
123
|
+
// Shared with the other surface (search.mjs): the answer, the excerpts and the
|
|
124
|
+
// reading line cannot differ between them.
|
|
125
|
+
return searchCorpus({ documents: corpus.documents, envelope }, args);
|
|
153
126
|
}
|
|
154
127
|
|
|
155
128
|
if (name === "get_document") {
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
// ⭐ The document goes to `content` WITH ITS PROVENANCE HEADER — the same
|
|
159
|
-
// bytes resources/read returns, so the two doors into a document agree —
|
|
160
|
-
// and the envelope goes to `structuredContent` WITHOUT the body. Split by
|
|
161
|
-
// role; nothing is sent twice.
|
|
162
|
-
return structuredWithText(documentText(d), {
|
|
163
|
-
...envelope(d),
|
|
164
|
-
genre: d.genre,
|
|
165
|
-
bytes: d.bytes,
|
|
166
|
-
words: d.words,
|
|
167
|
-
// ⚠️ category and voice were in list_documents and NOT here, so a
|
|
168
|
-
// caller who fetched ONE document could not see the facets it had
|
|
169
|
-
// just filtered on — the discovery call and the read call disagreed
|
|
170
|
-
// about what a document is. Both are cheap strings; the reason the
|
|
171
|
-
// body is excluded (it is in `content`, never sent twice) does not
|
|
172
|
-
// apply to them.
|
|
173
|
-
category: d.category,
|
|
174
|
-
voice: d.voice,
|
|
175
|
-
repo: d.repo,
|
|
176
|
-
path: d.path,
|
|
177
|
-
metadata_convention: d.metadata_convention,
|
|
178
|
-
// ⚠️ PRESENT ONLY WHERE THEY MEAN SOMETHING, and dropping them was a
|
|
179
|
-
// real bug: `editorial` is what tells a consumer that the inline
|
|
180
|
-
// [VERBATIM]/[SCAFFOLD] markers in `text` are a convention rather
|
|
181
|
-
// than noise, and `segments` is the same split structurally. The
|
|
182
|
-
// markers alone are the load-bearing half — they survive quotation —
|
|
183
|
-
// but shipping them with nothing that explains them made the
|
|
184
|
-
// convention look like an artefact of bad extraction.
|
|
185
|
-
...(d.editorial ? { editorial: d.editorial } : {}),
|
|
186
|
-
...(d.segments ? { segments: d.segments } : {}),
|
|
187
|
-
// ⚠️ Deliberately absent: the body is in `content`. Carrying it here
|
|
188
|
-
// too is the duplication this shape exists to avoid.
|
|
189
|
-
text_in: "content[0].text, prefixed by the provenance header and ending at its [END OF DOCUMENT] line"
|
|
190
|
-
});
|
|
129
|
+
// Shared (read-tools.mjs), and TEXT ONLY since 2.4.2 — see §the-text-never-arrived there.
|
|
130
|
+
return getDocument({ bySlug: bySlug }, args);
|
|
191
131
|
}
|
|
192
|
-
|
|
193
132
|
if (name === "read_documents") {
|
|
194
|
-
return readDocuments({ documents: corpus.documents, bySlug,
|
|
133
|
+
return readDocuments({ documents: corpus.documents, bySlug, version: String(corpus.package_version ?? "") }, args);
|
|
195
134
|
}
|
|
196
135
|
|
|
197
136
|
if (name === "list_documents") {
|
|
@@ -233,6 +172,8 @@ const callTool = (name, args) => {
|
|
|
233
172
|
...(corpus.voices ? { voices: corpus.voices } : {}),
|
|
234
173
|
// A87: the manifest's counts and reasons beside the listing they account for.
|
|
235
174
|
...(corpus.manifest ? { completeness: completenessOf(corpus) } : {}),
|
|
175
|
+
// 2.4.1: what to call to read this listing in full — the same filters, as arguments.
|
|
176
|
+
...(readWith(args, list.length) ? { read_with: readWith(args, list.length) } : {}),
|
|
236
177
|
documents: list
|
|
237
178
|
});
|
|
238
179
|
}
|