@333eco/corpus 2.4.1 → 2.4.2

package/README.md

@@ -113,8 +113,8 @@ corpus already fits one — but a way to read the shelf.
 - **Paged by size.** `max_bytes` defaults to 90,000 (under the 25,000-token point at which
   Claude Code moves a tool result into a file) and may go to 400,000. **A document is never
   split** — half a document verifies nothing — so one larger than the page arrives alone and
-  says `over_budget`. Follow `next_cursor`; a cursor from a different corpus version is refused
-  rather than serving page 2 of another corpus.
+  says so in its `[PAGE]` line. That last block names the cursor for the next page; a cursor from a
+  different corpus version is refused rather than serving page 2 of another corpus.
 - **`list_documents` says how to read what it listed:** `read_with` carries the same filters as
   `read_documents` arguments.
 
@@ -195,15 +195,23 @@ hands every consumer an obligation nobody can discharge, so **the build fails**
 rather than serving it — the same reasoning as the licence gate: a property, not a
 rule someone has to remember.
 
-**Results are structured.** Every tool returns `structuredContent` — the typed
-object — and uses `content` for the human form: the document text with its
-provenance header for `get_document`, readable excerpts for `search_corpus`, a
-compact serialisation for the rest. ⛔ **Nothing is sent twice.** The spec's
-back-compat advice is to serialise the JSON into `content` as well; measured, that
-doubles every response (1.76×–2.00×), so this server splits by role instead —
-`content` is read, `structuredContent` is validated, and the two carry different
-things. `outputSchema` is deliberately not declared yet: a schema binds the server
-on every future change, and the envelope is still moving.
+**Documents come as text; everything else is structured.** `get_document` and
+`read_documents` return **`content` only** — each document behind its provenance header
+and ending at its `[END OF DOCUMENT]` line. The other tools return `structuredContent`
+(the typed object) plus a readable or compact `content`, and anything a reader needs —
+search's `reading` line included — is in the structured part.
+
+⛔⛔ **Why the document tools carry no `structuredContent` (2.4.2).** From 2.0.0 to 2.4.1
+they split by role: text in `content`, the envelope *without the body* in
+`structuredContent`. A Claude client shown a result that carries `structuredContent` hands its
+model **only that part** — so for eleven days **no document text reached any Claude-based
+reader**, while every check here passed, because every check read the server's output
+directly rather than through a client. The MCP spec assumes the two parts carry the same
+information; a server that splits them is at the mercy of whichever part a client picks. The
+document is the payload and its provenance already rides in the header, so the document tools
+now send the text and nothing else — and `check-parity` fails if either ever carries
+`structuredContent` again. `outputSchema` is deliberately not declared: a schema binds the server
+on every future change.
 
 **Text is returned verbatim and is never summarised.** Not a stylistic
 preference — a summary cannot be hash-verified, so summarising at the server

package/dist/corpus.json

@@ -1,5 +1,5 @@
 {
-  "package_version": "2.4.1",
+  "package_version": "2.4.2",
   "generated_from": [
     "H3/publications",
     "TH/publications",

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@333eco/corpus",
-    "version": "2.4.1",
+    "version": "2.4.2",
     "description": "An MCP server for an open-licensed corpus, served with verifiable provenance — every document carries its sha256, DOI and OpenTimestamps proof so a consuming agent can check its own citation.",
     "license": "CC0-1.0",
     "author": "Thon Ly",

package/src/read-tools.mjs

@@ -43,7 +43,7 @@ export const READ_TOOLS = [
         // facets are described once, on list_documents.
         description:
             "Whole documents, never summarised: those matching the list_documents filters, or the given slugs. " +
-            "One content block each. Paged: follow next_cursor.",
+            "One content block each; the last block names the cursor for the next page.",
         inputSchema: {
             type: "object",
             properties: {
@@ -110,9 +110,9 @@ const decodeCursor = (cursor, version) => {
 
 /**
  * read_documents. `ctx` is what differs between the surfaces:
- * { documents, bySlug, envelope, version }.
+ * { documents, bySlug, version }.
  */
-export const readDocuments = ({ documents, bySlug, envelope, version }, args) => {
+export const readDocuments = ({ documents, bySlug, version }, args) => {
     const hasFilter = ["genre", "category", "voice", "licence"].some((k) => args?.[k]);
     let matched;
     if (args?.slugs !== undefined) {
@@ -149,43 +149,34 @@ export const readDocuments = ({ documents, bySlug, envelope, version }, args) =>
     const next = nextOffset < matched.length ? encodeCursor(version, nextOffset) : undefined;
 
     if (!matched.length) {
-        return {
-            content: [{ type: "text", text: "no documents match these filters. Call list_documents to see the facets and their counts." }],
-            structuredContent: { matched: 0, returned: 0, offset: 0, page_bytes: 0, max_bytes: budget, documents: [] }
-        };
+        return { content: [{ type: "text", text: "no documents match these filters. Call list_documents to see the facets and their counts." }] };
     }
 
+    // ⛔⛔ TEXT ONLY — NO structuredContent (2.4.2, founder-ruled 2026-09-13). A Claude client shown a
+    // result that carries structuredContent hands its model ONLY that part; the envelope-without-body
+    // shape this tool shipped with in 2.4.0 therefore delivered no document to any Claude-based reader
+    // (§the-text-never-arrived). The document IS the payload, and the provenance already rides in each
+    // document's header, so the structured copy bought validation for programs at the price of the
+    // text for models. Everything a reader needs to keep going is in the [PAGE] line.
     const content = page.map((p) => ({ type: "text", text: p.text }));
-    // The one block that is not a document: what a reader of `content` alone needs in
-    // order to keep going, or to know it has everything.
+    const over = page.length === 1 && used > budget ? " This document is larger than max_bytes and was returned whole rather than split." : "";
     content.push({
         type: "text",
         text: next
-            ? `[PAGE — documents ${start + 1}–${nextOffset} of ${matched.length}. ${matched.length - nextOffset} more: call read_documents again with the same arguments and cursor "${next}".]`
-            : `[PAGE — documents ${start + 1}–${nextOffset} of ${matched.length}. That is every matching document.]`
+            ? `[PAGE — documents ${start + 1}–${nextOffset} of ${matched.length}.${over} ${matched.length - nextOffset} more: call read_documents again with the same arguments and cursor "${next}".]`
+            : `[PAGE — documents ${start + 1}–${nextOffset} of ${matched.length}.${over} That is every matching document.]`
     });
+    return { content };
+};
 
-    return {
-        content,
-        structuredContent: {
-            matched: matched.length,
-            returned: page.length,
-            offset: start,
-            page_bytes: used,
-            max_bytes: budget,
-            ...(page.length === 1 && used > budget ? { over_budget: "one document larger than max_bytes, returned whole rather than split" } : {}),
-            ...(next ? { next_cursor: next } : {}),
-            documents: page.map((p) => ({
-                ...envelope(p.d),
-                genre: p.d.genre,
-                category: p.d.category,
-                voice: p.d.voice,
-                bytes: p.d.bytes,
-                words: p.d.words,
-                text_in: "content, one block per document in this order, each prefixed by its provenance header"
-            }))
-        }
-    };
+/**
+ * get_document — shared by both surfaces since 2.4.2, TEXT ONLY for the same reason as read_documents:
+ * the envelope-without-body in structuredContent was the only part a Claude client showed its model.
+ */
+export const getDocument = ({ bySlug }, args) => {
+    const d = bySlug.get(String(args?.slug ?? ""));
+    if (!d) throw new Error(`no document with slug "${args?.slug}". Call list_documents to see what is available.`);
+    return { content: [{ type: "text", text: documentText(d) }] };
 };
 
 /**

package/src/search.mjs

@@ -259,6 +259,9 @@ export const searchCorpus = ({ documents, envelope }, args) => {
         query: q,
         matches: hits.length,
         returned: results.length,
+        // ⛔ 2.4.2: the reading line was in `content` only, and a Claude client shows its model only
+        // structuredContent — so the 2.4.1 nudge reached no Claude reader. It rides here too.
+        ...(results.length ? { reading: readingLine(results, hits.length) } : {}),
         ...(absent.length ? { absent_terms: absent } : {}),
         results: results
     });

package/src/server.mjs

@@ -35,12 +35,11 @@ import { dirname, join, resolve } from "node:path";
 import { fileURLToPath } from "node:url";
 import { createInterface } from "node:readline";
 import { RESOURCE_TEMPLATES, listResources, readResource, completeArgument } from "./resources.mjs";
-import { structured, structuredWithText } from "./results.mjs";
-import { documentText } from "./resources.mjs";
+import { structured } from "./results.mjs";
 import { envelope } from "./envelope.mjs";
 import { PROMPTS, getPrompt } from "./prompts.mjs";
 import { BASE_TOOLS, withFacets } from "./base-tools.mjs";
-import { READ_TOOLS, READ_TOOL_NAMES, READ_INSTRUCTIONS, filterDocuments, readDocuments, readWith, completenessOf, manifestProblems } from "./read-tools.mjs";
+import { READ_TOOLS, READ_TOOL_NAMES, READ_INSTRUCTIONS, filterDocuments, readDocuments, getDocument, readWith, completenessOf, manifestProblems } from "./read-tools.mjs";
 import { PROGRAM_TOOLS, PROGRAM_TOOL_NAMES, PROGRAM_INSTRUCTIONS, callProgramTool } from "./program-tools.mjs";
 import { searchCorpus } from "./search.mjs";
 
@@ -127,45 +126,11 @@ const callTool = (name, args) => {
     }
 
     if (name === "get_document") {
-        const d = bySlug.get(String(args?.slug ?? ""));
-        if (!d) throw new Error(`no document with slug "${args?.slug}". Call list_documents to see what is available.`);
-        // ⭐ The document goes to `content` WITH ITS PROVENANCE HEADER — the same
-        // bytes resources/read returns, so the two doors into a document agree —
-        // and the envelope goes to `structuredContent` WITHOUT the body. Split by
-        // role; nothing is sent twice.
-        return structuredWithText(documentText(d), {
-            ...envelope(d),
-            genre: d.genre,
-            bytes: d.bytes,
-            words: d.words,
-            // ⚠️ category and voice were in list_documents and NOT here, so a
-            // caller who fetched ONE document could not see the facets it had
-            // just filtered on — the discovery call and the read call disagreed
-            // about what a document is. Both are cheap strings; the reason the
-            // body is excluded (it is in `content`, never sent twice) does not
-            // apply to them.
-            category: d.category,
-            voice: d.voice,
-            repo: d.repo,
-            path: d.path,
-            metadata_convention: d.metadata_convention,
-            // ⚠️ PRESENT ONLY WHERE THEY MEAN SOMETHING, and dropping them was a
-            // real bug: `editorial` is what tells a consumer that the inline
-            // [VERBATIM]/[SCAFFOLD] markers in `text` are a convention rather
-            // than noise, and `segments` is the same split structurally. The
-            // markers alone are the load-bearing half — they survive quotation —
-            // but shipping them with nothing that explains them made the
-            // convention look like an artefact of bad extraction.
-            ...(d.editorial ? { editorial: d.editorial } : {}),
-            ...(d.segments ? { segments: d.segments } : {}),
-            // ⚠️ Deliberately absent: the body is in `content`. Carrying it here
-            // too is the duplication this shape exists to avoid.
-            text_in: "content[0].text, prefixed by the provenance header and ending at its [END OF DOCUMENT] line"
-        });
+        // Shared (read-tools.mjs), and TEXT ONLY since 2.4.2 — see §the-text-never-arrived there.
+        return getDocument({ bySlug: bySlug }, args);
     }
-
     if (name === "read_documents") {
-        return readDocuments({ documents: corpus.documents, bySlug, envelope, version: String(corpus.package_version ?? "") }, args);
+        return readDocuments({ documents: corpus.documents, bySlug, version: String(corpus.package_version ?? "") }, args);
     }
 
     if (name === "list_documents") {