@333eco/corpus 2.4.0 → 2.4.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -83,6 +83,18 @@ verbatim. **A zero result was recording the matcher's limits while being read as
83
83
  ⭐ On a zero result the response now names **which terms appear in no document at all**
84
84
  (`absent_terms`), so a dead end says something instead of nothing.
85
85
 
86
+ ⭐ **Diacritics are optional.** `metta` finds what `mettā` finds, and `Tonle Sap` reaches the
87
+ text that writes `Tonlé Sap` — before 2.4.1 a plain-ASCII query found 6 documents where the
88
+ marked one found 20. ⛔ **Fold to match, serve verbatim:** the fold touches only what is
89
+ compared, and every excerpt carries the corpus's own spelling, because a Pāli or Sanskrit
90
+ spelling is a tradition marker rather than a typo. Only Latin combining diacritics are folded;
91
+ Khmer and Burmese vowel signs are marks too, and are left exactly as written.
92
+
93
+ ⭐ **Every result says it is an excerpt.** The text ends with a line naming how many of the
94
+ matches it shows and the `read_documents` call that reads those documents in full. It sits in
95
+ the result rather than only in the server's instructions, because a result reaches the model in
96
+ every client, at the moment it decides whether to keep reading.
97
+
86
98
  ### Reading a whole shelf
87
99
 
88
100
  ⭐ **Search answers from excerpts, and a model answers from what it opened.** A frontier
@@ -101,8 +113,15 @@ corpus already fits one — but a way to read the shelf.
101
113
  - **Paged by size.** `max_bytes` defaults to 90,000 (under the 25,000-token point at which
102
114
  Claude Code moves a tool result into a file) and may go to 400,000. **A document is never
103
115
  split** — half a document verifies nothing — so one larger than the page arrives alone and
104
- says `over_budget`. Follow `next_cursor`; a cursor from a different corpus version is refused
105
- rather than serving page 2 of another corpus.
116
+ says so in its `[PAGE]` line. That last block names the cursor for the next page; a cursor from a
117
+ different corpus version is refused rather than serving page 2 of another corpus.
118
+ - **`list_documents` says how to read what it listed:** `read_with` carries the same filters as
119
+ `read_documents` arguments.
120
+
121
+ ⚠️ **The server can point; it cannot make a model read.** Its instructions say *for a broad
122
+ question, read the relevant shelf in full* — deliberately conditional, because a whole shelf read
123
+ to answer a one-fact question is slower, costlier and often worse. Clients differ in whether they
124
+ show server instructions at all, which is why the pointers also live in the results.
106
125
 
107
126
  ⭐ **Every document ends with an `[END OF DOCUMENT — <slug> · <n> words]` line**, by every
108
127
  route. The server never cuts a document, but a client may, and a cut document reads as a
@@ -176,15 +195,23 @@ hands every consumer an obligation nobody can discharge, so **the build fails**
176
195
  rather than serving it — the same reasoning as the licence gate: a property, not a
177
196
  rule someone has to remember.
178
197
 
179
- **Results are structured.** Every tool returns `structuredContent` — the typed
180
- object — and uses `content` for the human form: the document text with its
181
- provenance header for `get_document`, readable excerpts for `search_corpus`, a
182
- compact serialisation for the rest. ⛔ **Nothing is sent twice.** The spec's
183
- back-compat advice is to serialise the JSON into `content` as well; measured, that
184
- doubles every response (1.76×–2.00×), so this server splits by role instead —
185
- `content` is read, `structuredContent` is validated, and the two carry different
186
- things. `outputSchema` is deliberately not declared yet: a schema binds the server
187
- on every future change, and the envelope is still moving.
198
+ **Documents come as text; everything else is structured.** `get_document` and
199
+ `read_documents` return **`content` only** — each document behind its provenance header
200
+ and ending at its `[END OF DOCUMENT]` line. The other tools return `structuredContent`
201
+ (the typed object) plus a readable or compact `content`, and anything a reader needs —
202
+ search's `reading` line included — is in the structured part.
203
+
204
+ ⛔⛔ **Why the document tools carry no `structuredContent` (2.4.2).** From 2.0.0 to 2.4.1
205
+ they split by role: text in `content`, the envelope *without the body* in
206
+ `structuredContent`. A Claude client shown a result that carries `structuredContent` hands its
207
+ model **only that part** — so for eleven days **no document text reached any Claude-based
208
+ reader**, while every check here passed, because every check read the server's output
209
+ directly rather than through a client. The MCP spec assumes the two parts carry the same
210
+ information; a server that splits them is at the mercy of whichever part a client picks. The
211
+ document is the payload and its provenance already rides in the header, so the document tools
212
+ now send the text and nothing else — and `check-parity` fails if either ever carries
213
+ `structuredContent` again. `outputSchema` is deliberately not declared: a schema binds the server
214
+ on every future change.
188
215
 
189
216
  **Text is returned verbatim and is never summarised.** Not a stylistic
190
217
  preference — a summary cannot be hash-verified, so summarising at the server
package/dist/corpus.json CHANGED
@@ -1,5 +1,5 @@
1
1
  {
2
- "package_version": "2.4.0",
2
+ "package_version": "2.4.2",
3
3
  "generated_from": [
4
4
  "H3/publications",
5
5
  "TH/publications",
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@333eco/corpus",
3
- "version": "2.4.0",
3
+ "version": "2.4.2",
4
4
  "description": "An MCP server for an open-licensed corpus, served with verifiable provenance — every document carries its sha256, DOI and OpenTimestamps proof so a consuming agent can check its own citation.",
5
5
  "license": "CC0-1.0",
6
6
  "author": "Thon Ly",
@@ -27,9 +27,9 @@ export const BASE_TOOLS = [
27
27
  title: "Search the corpus",
28
28
  annotations: { ...READ_ONLY, title: "Search the corpus" },
29
29
  description:
30
- "Full-text search across the open-licensed corpus. Returns matching documents with a provenance envelope " +
31
- "and a short excerpt around each match — not the full text; call get_document for that. Every result can " +
32
- "be independently verified via its sha256, DOI and OpenTimestamps proof.",
30
+ "Full-text search across the open-licensed corpus; diacritics are optional. Returns matching documents with " +
31
+ "a provenance envelope and a short excerpt around each match — not the full text; read_documents or " +
32
+ "get_document return that. Every result can be verified via its sha256, DOI and OpenTimestamps proof.",
33
33
  inputSchema: {
34
34
  type: "object",
35
35
  properties: {
@@ -43,7 +43,7 @@ export const READ_TOOLS = [
43
43
  // facets are described once, on list_documents.
44
44
  description:
45
45
  "Whole documents, never summarised: those matching the list_documents filters, or the given slugs. " +
46
- "One content block each. Paged: follow next_cursor.",
46
+ "One content block each; the last block names the cursor for the next page.",
47
47
  inputSchema: {
48
48
  type: "object",
49
49
  properties: {
@@ -65,9 +65,27 @@ export const READ_TOOLS = [
65
65
 
66
66
  export const READ_TOOL_NAMES = READ_TOOLS.map((t) => t.name);
67
67
 
68
+ // ⚠️ CONDITIONAL ON PURPOSE (2.4.1). "Prefer read_documents" unqualified would send a model to
69
+ // read a 260,000-token shelf to answer a one-fact question — slower, costlier and often worse.
70
+ // ⛔ It names shelves by their facets (list_documents' own vocabulary), never by what the
71
+ // documents say: describing the API, not the subject matter.
68
72
  export const READ_INSTRUCTIONS =
69
- " Search returns excerpts; to answer broadly, read whole documents with read_documents. Every document ends " +
70
- "with an [END OF DOCUMENT] line — if it is missing, the text was cut short.";
73
+ " For a broad question, read the relevant shelf in full: choose it with list_documents, read it with " +
74
+ "read_documents. Search returns excerpts, suited to finding one passage. Every document ends with an " +
75
+ "[END OF DOCUMENT] line; if it is missing, the text was cut short.";
76
+
77
+ /**
78
+ * list_documents' pointer to reading what it just listed: the same filters, as arguments
79
+ * read_documents accepts. Only the facets actually given, so the pointer reads back exactly.
80
+ */
81
+ export const readWith = (args, count) =>
82
+ count
83
+ ? {
84
+ tool: "read_documents",
85
+ arguments: Object.fromEntries(["genre", "category", "voice", "licence"].filter((k) => args?.[k]).map((k) => [k, args[k]])),
86
+ note: "Reads every document listed here in full, paged by size."
87
+ }
88
+ : null;
71
89
 
72
90
  // ⚠️ The cursor carries the INDEX VERSION as well as the offset. Pages are offsets
73
91
  // into a list, and a list read across a redeploy is two lists: without the version
@@ -92,9 +110,9 @@ const decodeCursor = (cursor, version) => {
92
110
 
93
111
  /**
94
112
  * read_documents. `ctx` is what differs between the surfaces:
95
- * { documents, bySlug, envelope, version }.
113
+ * { documents, bySlug, version }.
96
114
  */
97
- export const readDocuments = ({ documents, bySlug, envelope, version }, args) => {
115
+ export const readDocuments = ({ documents, bySlug, version }, args) => {
98
116
  const hasFilter = ["genre", "category", "voice", "licence"].some((k) => args?.[k]);
99
117
  let matched;
100
118
  if (args?.slugs !== undefined) {
@@ -131,43 +149,34 @@ export const readDocuments = ({ documents, bySlug, envelope, version }, args) =>
131
149
  const next = nextOffset < matched.length ? encodeCursor(version, nextOffset) : undefined;
132
150
 
133
151
  if (!matched.length) {
134
- return {
135
- content: [{ type: "text", text: "no documents match these filters. Call list_documents to see the facets and their counts." }],
136
- structuredContent: { matched: 0, returned: 0, offset: 0, page_bytes: 0, max_bytes: budget, documents: [] }
137
- };
152
+ return { content: [{ type: "text", text: "no documents match these filters. Call list_documents to see the facets and their counts." }] };
138
153
  }
139
154
 
155
+ // ⛔⛔ TEXT ONLY — NO structuredContent (2.4.2, founder-ruled 2026-09-13). A Claude client shown a
156
+ // result that carries structuredContent hands its model ONLY that part; the envelope-without-body
157
+ // shape this tool shipped with in 2.4.0 therefore delivered no document to any Claude-based reader
158
+ // (§the-text-never-arrived). The document IS the payload, and the provenance already rides in each
159
+ // document's header, so the structured copy bought validation for programs at the price of the
160
+ // text for models. Everything a reader needs to keep going is in the [PAGE] line.
140
161
  const content = page.map((p) => ({ type: "text", text: p.text }));
141
- // The one block that is not a document: what a reader of `content` alone needs in
142
- // order to keep going, or to know it has everything.
162
+ const over = page.length === 1 && used > budget ? " This document is larger than max_bytes and was returned whole rather than split." : "";
143
163
  content.push({
144
164
  type: "text",
145
165
  text: next
146
- ? `[PAGE — documents ${start + 1}–${nextOffset} of ${matched.length}. ${matched.length - nextOffset} more: call read_documents again with the same arguments and cursor "${next}".]`
147
- : `[PAGE — documents ${start + 1}–${nextOffset} of ${matched.length}. That is every matching document.]`
166
+ ? `[PAGE — documents ${start + 1}–${nextOffset} of ${matched.length}.${over} ${matched.length - nextOffset} more: call read_documents again with the same arguments and cursor "${next}".]`
167
+ : `[PAGE — documents ${start + 1}–${nextOffset} of ${matched.length}.${over} That is every matching document.]`
148
168
  });
169
+ return { content };
170
+ };
149
171
 
150
- return {
151
- content,
152
- structuredContent: {
153
- matched: matched.length,
154
- returned: page.length,
155
- offset: start,
156
- page_bytes: used,
157
- max_bytes: budget,
158
- ...(page.length === 1 && used > budget ? { over_budget: "one document larger than max_bytes, returned whole rather than split" } : {}),
159
- ...(next ? { next_cursor: next } : {}),
160
- documents: page.map((p) => ({
161
- ...envelope(p.d),
162
- genre: p.d.genre,
163
- category: p.d.category,
164
- voice: p.d.voice,
165
- bytes: p.d.bytes,
166
- words: p.d.words,
167
- text_in: "content, one block per document in this order, each prefixed by its provenance header"
168
- }))
169
- }
170
- };
172
+ /**
173
+ * get_document — shared by both surfaces since 2.4.2, TEXT ONLY for the same reason as read_documents:
174
+ * the envelope-without-body in structuredContent was the only part a Claude client showed its model.
175
+ */
176
+ export const getDocument = ({ bySlug }, args) => {
177
+ const d = bySlug.get(String(args?.slug ?? ""));
178
+ if (!d) throw new Error(`no document with slug "${args?.slug}". Call list_documents to see what is available.`);
179
+ return { content: [{ type: "text", text: documentText(d) }] };
171
180
  };
172
181
 
173
182
  /**
package/src/search.mjs CHANGED
@@ -1,8 +1,8 @@
1
1
  // How `search_corpus` decides a document matches.
2
2
  //
3
- // ⭐⭐ SHARED, AND IT HAS TO BE. The two surfaces keep their own `callTool` on
4
- // purpose — twenty lines each, diffable by eye. Matching is not that: it is
5
- // ranking, tokenising and window-finding, and if the stdio server and the worker
3
+ // ⭐⭐ SHARED, AND IT HAS TO BE — matching and, since 2.4.1, the whole search_corpus
4
+ // tool (see searchCorpus below). Matching is ranking, tokenising and window-finding, and
5
+ // if the stdio server and the worker
6
6
  // ever disagreed about which documents answer a query, the corpus would have two
7
7
  // opinions about itself. That is the same failure `sync.mjs` exists to prevent,
8
8
  // arriving through the search path instead of the data path.
@@ -33,15 +33,49 @@
33
33
  // any document containing "b" and "heart" separately — precision traded away for
34
34
  // nothing, in the corpus where those marks matter most.
35
35
 
36
+ import { structuredWithText } from "./results.mjs";
37
+
36
38
  const TOKEN = /[\p{L}\p{N}]+(?:['’-][\p{L}\p{N}]+)*/gu;
37
39
 
40
+ // ⭐⭐ DIACRITICS ARE OPTIONAL IN A QUERY (2.4.1). A model types `metta`; the corpus writes
41
+ // `mettā`. Without a fold that search found 6 documents against 20, and `Tonle Sap` found none
42
+ // — a reader typing plain ASCII, which is most of them, silently missed most of the Pāli
43
+ // material. ⛔ FOLD TO MATCH, SERVE VERBATIM: the spelling is a tradition marker (Pāli for a
44
+ // Theravāda citation, Sanskrit for a Hindu one), so the fold touches only what is COMPARED —
45
+ // every excerpt is cut from the original text.
46
+ // ⚠️ LENGTH-PRESERVING, character by character, so a position in the folded text is the same
47
+ // position in the original and the excerpt window lands where the match is. Only Latin
48
+ // combining diacritics (U+0300–U+036F) are removed, and only where the base letter survives
49
+ // as one character: Khmer and Burmese vowel signs are marks too, and stripping them would
50
+ // make Khmer text match what it does not say.
51
+ const LATIN_MARKS = /[\u0300-\u036f]/g;
52
+ const foldChar = (c) => {
53
+ const f = c.normalize("NFD").replace(LATIN_MARKS, "");
54
+ return f.length === c.length ? f : c;
55
+ };
56
+ export const fold = (s) => String(s ?? "").replace(/[^\x00-\x7f]/gu, foldChar);
57
+
58
+ // The folded, lowercased body, computed once per document text rather than on every search.
59
+ // Keyed by the string itself: the corpus holds one string per document for the life of the
60
+ // process, so the cache is bounded by the corpus.
61
+ const HAY = new Map();
62
+ const hayOf = (body) => {
63
+ let h = HAY.get(body);
64
+ if (h === undefined) {
65
+ h = fold(body.toLowerCase());
66
+ HAY.set(body, h);
67
+ }
68
+ return h;
69
+ };
70
+ const foldQuery = (query) => fold(String(query ?? "").normalize("NFC").toLowerCase());
71
+
38
72
  // Dropped only when something survives the dropping. These are words present in
39
73
  // essentially every document, so they never narrow the result set — but they do
40
74
  // wreck the tightest-window calculation, because an "and" is always close by.
41
75
  const STOP = new Set(["a", "an", "and", "as", "at", "be", "by", "for", "from", "in", "is", "it", "of", "on", "or", "the", "to", "with"]);
42
76
 
43
77
  export const terms = (query) => {
44
- const all = String(query ?? "").toLowerCase().match(TOKEN) ?? [];
78
+ const all = foldQuery(query).match(TOKEN) ?? [];
45
79
  const kept = all.filter((t) => !STOP.has(t));
46
80
  return kept.length ? kept : all;
47
81
  };
@@ -138,8 +172,8 @@ const tightestWindow = (lists) => {
138
172
  // term match's excerpt may NOT contain the literal query, and a caller reading
139
173
  // an excerpt deserves to know which question it answered.
140
174
  export const match = (body, query, span = 320) => {
141
- const hay = body.toLowerCase();
142
- const phrase = String(query ?? "").toLowerCase().trim();
175
+ const hay = hayOf(body);
176
+ const phrase = foldQuery(query).trim();
143
177
 
144
178
  if (phrase) {
145
179
  const i = hay.indexOf(phrase);
@@ -175,6 +209,60 @@ export const rank = (hits, limit) =>
175
209
  export const absentTerms = (documents, query) => {
176
210
  const ts = [...new Set(terms(query))];
177
211
  if (ts.length === 0) return [];
178
- const hays = documents.map((d) => d.text.toLowerCase());
212
+ const hays = documents.map((d) => hayOf(d.text));
179
213
  return ts.filter((t) => !hays.some((h) => h.includes(t)));
180
214
  };
215
+
216
+ /* ---------------------------------------------------------- search_corpus ---
217
+ ⭐ THE WHOLE TOOL, SHARED (2.4.1). It lived twice, once in each surface's callTool, and
218
+ its guidance line was about to be written twice too — the shape in which the envelope
219
+ drifted. One function, imported by both. */
220
+
221
+ // A search returns EXCERPTS, and a model that reads only excerpts answers from them. So the
222
+ // text a model reads says so at the moment it decides what to do next — in the result, where
223
+ // every client delivers it, rather than only in server instructions a client may drop.
224
+ const readingLine = (results, matches) => {
225
+ const slugs = results.map((r) => r.slug);
226
+ return (
227
+ `[Showing ${results.length} of ${matches} matching document${matches === 1 ? "" : "s"}, as excerpts. ` +
228
+ `To answer from the full texts, call read_documents with slugs ${JSON.stringify(slugs)}` +
229
+ (matches > results.length ? `; raise limit to see more of the ${matches} matches` : "") +
230
+ ".]"
231
+ );
232
+ };
233
+
234
+ export const searchCorpus = ({ documents, envelope }, args) => {
235
+ const q = String(args?.query ?? "");
236
+ if (!q) throw new Error("query is required");
237
+ const pool = documents.filter((d) => !args?.genre || d.genre === args.genre);
238
+ const hits = pool.map((d) => ({ d, m: match(d.text, q) })).filter((h) => h.m !== null);
239
+ const results = rank(hits, Number(args?.limit ?? 10)).map((h) => ({
240
+ ...envelope(h.d),
241
+ genre: h.d.genre,
242
+ excerpt: h.m.excerpt,
243
+ // ⭐ HOW it matched, not just that it did: a "terms" excerpt need not
244
+ // contain the literal query, and a caller reading it should know
245
+ // which question the excerpt is answering.
246
+ match: h.m.mode
247
+ }));
248
+ // ⚠️ `matches` is the TOTAL found, not the number returned — it used to be
249
+ // capped at `limit`, which made "10 matches" and "at least 10 matches"
250
+ // indistinguishable. `returned` carries the capped count.
251
+ const absent = hits.length ? [] : absentTerms(pool, q);
252
+ const readable = results.length
253
+ ? results.map((h) => `${h.slug} — ${h.title}${h.match === "terms" ? " [all terms, not the phrase]" : ""}\n ${h.excerpt}`).join("\n\n") +
254
+ "\n\n" + readingLine(results, hits.length)
255
+ : absent.length
256
+ ? `no document matches "${q}". No document contains: ${absent.join(", ")}.`
257
+ : `no document matches "${q}" — every term appears somewhere, but no single document holds them all. Try fewer terms.`;
258
+ return structuredWithText(readable, {
259
+ query: q,
260
+ matches: hits.length,
261
+ returned: results.length,
262
+ // ⛔ 2.4.2: the reading line was in `content` only, and a Claude client shows its model only
263
+ // structuredContent — so the 2.4.1 nudge reached no Claude reader. It rides here too.
264
+ ...(results.length ? { reading: readingLine(results, hits.length) } : {}),
265
+ ...(absent.length ? { absent_terms: absent } : {}),
266
+ results: results
267
+ });
268
+ };
package/src/server.mjs CHANGED
@@ -35,14 +35,13 @@ import { dirname, join, resolve } from "node:path";
35
35
  import { fileURLToPath } from "node:url";
36
36
  import { createInterface } from "node:readline";
37
37
  import { RESOURCE_TEMPLATES, listResources, readResource, completeArgument } from "./resources.mjs";
38
- import { structured, structuredWithText } from "./results.mjs";
39
- import { documentText } from "./resources.mjs";
38
+ import { structured } from "./results.mjs";
40
39
  import { envelope } from "./envelope.mjs";
41
40
  import { PROMPTS, getPrompt } from "./prompts.mjs";
42
41
  import { BASE_TOOLS, withFacets } from "./base-tools.mjs";
43
- import { READ_TOOLS, READ_TOOL_NAMES, READ_INSTRUCTIONS, filterDocuments, readDocuments, completenessOf, manifestProblems } from "./read-tools.mjs";
42
+ import { READ_TOOLS, READ_TOOL_NAMES, READ_INSTRUCTIONS, filterDocuments, readDocuments, getDocument, readWith, completenessOf, manifestProblems } from "./read-tools.mjs";
44
43
  import { PROGRAM_TOOLS, PROGRAM_TOOL_NAMES, PROGRAM_INSTRUCTIONS, callProgramTool } from "./program-tools.mjs";
45
- import { match, rank, absentTerms } from "./search.mjs";
44
+ import { searchCorpus } from "./search.mjs";
46
45
 
47
46
  const HERE = dirname(fileURLToPath(import.meta.url));
48
47
  const INDEX = resolve(HERE, "..", "dist", "corpus.json");
@@ -121,77 +120,17 @@ const text = (value) => ({ content: [{ type: "text", text: typeof value === "str
121
120
 
122
121
  const callTool = (name, args) => {
123
122
  if (name === "search_corpus") {
124
- const q = String(args?.query ?? "");
125
- if (!q) throw new Error("query is required");
126
- const pool = corpus.documents.filter((d) => !args?.genre || d.genre === args.genre);
127
- const hits = pool.map((d) => ({ d, m: match(d.text, q) })).filter((h) => h.m !== null);
128
- const results = rank(hits, Number(args?.limit ?? 10)).map((h) => ({
129
- ...envelope(h.d),
130
- genre: h.d.genre,
131
- excerpt: h.m.excerpt,
132
- // ⭐ HOW it matched, not just that it did: a "terms" excerpt need not
133
- // contain the literal query, and a caller reading it should know
134
- // which question the excerpt is answering.
135
- match: h.m.mode
136
- }));
137
- // ⚠️ `matches` is the TOTAL found, not the number returned — it used to be
138
- // capped at `limit`, which made "10 matches" and "at least 10 matches"
139
- // indistinguishable. `returned` carries the capped count.
140
- const absent = hits.length ? [] : absentTerms(pool, q);
141
- const readable = results.length
142
- ? results.map((h) => `${h.slug} — ${h.title}${h.match === "terms" ? " [all terms, not the phrase]" : ""}\n ${h.excerpt}`).join("\n\n")
143
- : absent.length
144
- ? `no document matches "${q}". No document contains: ${absent.join(", ")}.`
145
- : `no document matches "${q}" — every term appears somewhere, but no single document holds them all. Try fewer terms.`;
146
- return structuredWithText(readable, {
147
- query: q,
148
- matches: hits.length,
149
- returned: results.length,
150
- ...(absent.length ? { absent_terms: absent } : {}),
151
- results: results
152
- });
123
+ // Shared with the other surface (search.mjs): the answer, the excerpts and the
124
+ // reading line cannot differ between them.
125
+ return searchCorpus({ documents: corpus.documents, envelope }, args);
153
126
  }
154
127
 
155
128
  if (name === "get_document") {
156
- const d = bySlug.get(String(args?.slug ?? ""));
157
- if (!d) throw new Error(`no document with slug "${args?.slug}". Call list_documents to see what is available.`);
158
- // ⭐ The document goes to `content` WITH ITS PROVENANCE HEADER — the same
159
- // bytes resources/read returns, so the two doors into a document agree —
160
- // and the envelope goes to `structuredContent` WITHOUT the body. Split by
161
- // role; nothing is sent twice.
162
- return structuredWithText(documentText(d), {
163
- ...envelope(d),
164
- genre: d.genre,
165
- bytes: d.bytes,
166
- words: d.words,
167
- // ⚠️ category and voice were in list_documents and NOT here, so a
168
- // caller who fetched ONE document could not see the facets it had
169
- // just filtered on — the discovery call and the read call disagreed
170
- // about what a document is. Both are cheap strings; the reason the
171
- // body is excluded (it is in `content`, never sent twice) does not
172
- // apply to them.
173
- category: d.category,
174
- voice: d.voice,
175
- repo: d.repo,
176
- path: d.path,
177
- metadata_convention: d.metadata_convention,
178
- // ⚠️ PRESENT ONLY WHERE THEY MEAN SOMETHING, and dropping them was a
179
- // real bug: `editorial` is what tells a consumer that the inline
180
- // [VERBATIM]/[SCAFFOLD] markers in `text` are a convention rather
181
- // than noise, and `segments` is the same split structurally. The
182
- // markers alone are the load-bearing half — they survive quotation —
183
- // but shipping them with nothing that explains them made the
184
- // convention look like an artefact of bad extraction.
185
- ...(d.editorial ? { editorial: d.editorial } : {}),
186
- ...(d.segments ? { segments: d.segments } : {}),
187
- // ⚠️ Deliberately absent: the body is in `content`. Carrying it here
188
- // too is the duplication this shape exists to avoid.
189
- text_in: "content[0].text, prefixed by the provenance header and ending at its [END OF DOCUMENT] line"
190
- });
129
+ // Shared (read-tools.mjs), and TEXT ONLY since 2.4.2 — see §the-text-never-arrived there.
130
+ return getDocument({ bySlug: bySlug }, args);
191
131
  }
192
-
193
132
  if (name === "read_documents") {
194
- return readDocuments({ documents: corpus.documents, bySlug, envelope, version: String(corpus.package_version ?? "") }, args);
133
+ return readDocuments({ documents: corpus.documents, bySlug, version: String(corpus.package_version ?? "") }, args);
195
134
  }
196
135
 
197
136
  if (name === "list_documents") {
@@ -233,6 +172,8 @@ const callTool = (name, args) => {
233
172
  ...(corpus.voices ? { voices: corpus.voices } : {}),
234
173
  // A87: the manifest's counts and reasons beside the listing they account for.
235
174
  ...(corpus.manifest ? { completeness: completenessOf(corpus) } : {}),
175
+ // 2.4.1: what to call to read this listing in full — the same filters, as arguments.
176
+ ...(readWith(args, list.length) ? { read_with: readWith(args, list.length) } : {}),
236
177
  documents: list
237
178
  });
238
179
  }