@panaversity/ksor 0.0.14 → 0.0.16

package/CHANGELOG.md

@@ -1,5 +1,177 @@
 # @panaversity/ksor
 
+## 0.0.16
+
+### Patch Changes
+
+- 144aba8: Ingest says when a document's ordering key is one this record does not read
+
+  A record's reading order comes from the governed `order:` key alone. A corpus
+  arriving from Docusaurus, Hugo or Jekyll carries its own — `sidebar_position`,
+  `weight`, `nav_order` — and ksor ignored them in silence, falling back to file
+  name. That is a WRONG order, not a missing one, and it is the order served to
+  `llms.txt`, the rendered sidebar and the MCP `outline` alike.
+
+  Found on a real 81-document book where 73 files declared `sidebar_position`. Its
+  second chapter came out ninth; its preface came out eleventh. Nothing said why.
+
+  ```
+  plain-tree: 73 document(s) declare `sidebar_position`, which this record does not
+  read — reading order fell back to file name (about.md, how-to-sell.md,
+  thesis.md, and 70 more). Rename it to `order:` to keep the intended sequence.
+  ```
+
+  It reports on the same channel the adapter already uses for skipped files, where
+  the principle was already written down: a skip is reported, never silent. A
+  document that declares BOTH keys says nothing — `order:` wins, so nothing fell
+  back, and a warning there would only teach the reader to ignore the channel.
+
+- 2e9c987: Ingest says what the navigation rule now is, not what it used to be
+
+  0.0.15 changed how a section is judged to be navigation — shape rather than
+  length — and left every sentence describing it behind. So a fresh `ksor ingest`
+  reported:
+
+  ```
+  not searchable: 1 of 5 chunk(s) (20%) are shorter than the navigation threshold
+  ```
+
+  There is no navigation threshold any more, and the page in question was not
+  short: it was an index of links, which is exactly what the rule now catches. The
+  remedy was wrong in the same way — "lengthen these sections" is no longer how a
+  page becomes searchable, and padding a link list would not have made it one.
+
+  ```
+  not searchable: 1 of 5 chunk(s) (20%) read as navigation rather than content
+  FOUND ONLY BY NAME: knowledge/index — no searchable chunk at all; a page of
+  links reads as navigation; give it prose of its own, or reach it by slug
+  ```
+
+  Found by running the published artifact rather than by reading the diff. The
+  same stale description was corrected in the three other places it had been
+  copied to.
+
+- 1e26c07: A YAML list in frontmatter no longer costs the document its title
+
+  The frontmatter reader emptied a document's ENTIRE metadata whenever a top-level
+  value opened with `[ { | > & * !`. One `authors: ["…"]` line beside the title,
+  and the title went with it — along with `order:` and `sor_id:`.
+
+  Found on a real 81-document book, where four chapters were served under names
+  derived from their filenames:
+
+  | served as                    | declared                                                        |
+  | ---------------------------- | --------------------------------------------------------------- |
+  | `Preface Agent Native`       | `Preface: The Right Side of the Line`                           |
+  | `System Of Context`          | `The System of Context: Connecting the Records to Real Work`    |
+  | `Designing The Vertical Sor` | `Designing the Vertical System of Record from First Principles` |
+
+  Titles reach the site, `llms.txt` and the MCP `outline`, so this was wrong on
+  every surface at once, and silently.
+
+  The reader is documented as PyYAML-compatible and empties the map only where
+  PyYAML raises. PyYAML does not raise on a flow sequence — it parses it. Two
+  different things were being conflated:
+
+  - **invalid** — an unquoted `a: b: c`, a trailing `:`. PyYAML raises; the map is
+    still emptied, unchanged.
+  - **valid but not modelled here** — a flow sequence or mapping, a block scalar,
+    an anchor. PyYAML parses these. Only the KEY is beyond the reader now; the
+    document survives.
+
+  **One identity change to know about.** A document that declares `sor_id:`
+  _alongside_ such a value previously had that override silently dropped, so its
+  stable_id fell back to the path. The override now stands, on both surfaces
+  together — so re-ingesting changes the stable_id of exactly those documents, and
+  any takedown row keyed on the old path-derived id must be re-pointed. The site
+  and the kernel change in step, which is the property `stable-id-conformance`
+  exists to hold.
+
+  One governance guard gets quieter and no weaker: ingest used to REFUSE a
+  document declaring `visibility:` beside a flow list, because the map was emptied
+  and the tier silently defaulted. The cause is gone, so it ingests with the right
+  visibility; the refusal still stands for frontmatter PyYAML genuinely rejects.
+
+- d4334c7: A quiz no longer swallows the explanation that precedes it
+
+  The previous release moved navigation from a length test to a shape test, so a
+  short fact stopped being mistaken for a link list. The rule that decides whether
+  a whole section is _a widget_ — a quiz, a slide embed — was left on the old
+  threshold: under 250 characters of teaching before the widget, and the entire
+  section was labelled `assessment` or `embed`, neither of which any search
+  returns.
+
+  So a section carrying a complete 180-character explanation followed by a
+  knowledge check lost the explanation too. Same defect as the last one, one path
+  over.
+
+  Both paths now ask the same question: is what comes BEFORE the widget actually
+  navigation-shaped? A heading with only a quiz under it is still a quiz. A link
+  list before a quiz is still a quiz. An explanation before a quiz is an
+  explanation, and stays searchable.
+
+  Found by ingesting a real 81-document curriculum corpus, where 610 chunks landed
+  as `assessment` and 186 as `embed` — together 79% of everything unsearchable in
+  that record.
+
+  `CHUNK_POLICY` moves to v7 (persisted provenance; the labels it names changed),
+  and `NAV_MAX_CHARS` is deleted — nothing reads it now. **Re-run `ksor ingest` to
+  pick this up**; unchanged content is not re-embedded.
+
+## 0.0.15
+
+### Patch Changes
+
+- 5b076e6: Short documents reach search again — navigation is a shape, not a length
+
+  A record could be fully ingested, report "embedded 16, failed 0", and still be
+  unable to answer questions it plainly contained. Sections were classified as
+  navigation by LENGTH — anything under 250 characters — and navigation is
+  excluded from every retrieval arm. On a handbook that inverts the intent,
+  because a handbook's most valuable statements are its shortest.
+
+  Walked on 0.0.14 with three ordinary policy statements — a refund window, an
+  escalation path, a badge rule, 200-300 characters each. Three of four chunks
+  were unsearchable, and:
+
+  > **Q.** "how long does a buyer have to send something back"
+  > **A.** the scaffold's placeholder page — against a record stating _thirty days_
+
+  The answer was in the corpus, correctly ingested, readable by slug, and
+  unreachable by search.
+
+  Navigation is now decided by shape: a section is navigation when link lines are
+  most of it, or when what remains after them is too short to answer anything —
+  the same floor the serving predicate already applies. Length is no longer
+  consulted, so a 180-character link list is navigation and a 51-character fact is
+  not, which is the ordering length had backwards.
+
+  Measured on an authored handbook gold set with real embeddings, paired: short
+  substantive facts went **0/9 to 9/9 at rank 1**, the long-prose control held at
+  **4/4**, and the link-list page was returned **0** times. That last number is the
+  one that matters — admitting everything would have improved the first two and
+  made the product worse.
+
+  **To pick this up, re-run `ksor ingest`.** Chunks are re-classified on every
+  build and unchanged content is not re-embedded, so the upgrade costs a build,
+  not an embedding bill. `CHUNK_POLICY` moves to v6 because it is persisted
+  provenance and the behaviour it labels has changed.
+
+- 5763e8b: Internal: a pool test that raced Postgres, and a comment that had it backwards
+
+  No adopter-visible behaviour changes.
+
+  `idle.db.test.ts` sampled `pg_stat_activity` immediately after a previous test's
+  `pool.end()`. Those are two different clocks — `end()` resolves when the client
+  socket closes, while the row disappears only once the server-side backend
+  actually exits — so the suite was order-coupled through the database and went
+  red in CI on a branch that changed nothing but a document. Each test now waits
+  for a quiet database before it starts, and states that it does.
+
+  The comment added in the previous release explaining the `env.example` guard fix
+  described the rename backwards: the TEMPLATE holds `env.example` and
+  `materialize.ts` maps it to `.env.example` on emit, not the other way round.
+
 ## 0.0.14
 
 ### Patch Changes

package/dist/cli.mjs

@@ -16,7 +16,7 @@ import { bodyLimit } from "hono/body-limit";
 import { execFileSync, spawnSync } from "node:child_process";
 import { parseArgs } from "node:util";
 import { readFile, readdir, stat } from "node:fs/promises";
-//#region ../content-gateway/dist/main-VL9lXhmT.mjs
+//#region ../content-gateway/dist/main-BFWvQ6vY.mjs
 /**
 * A connection could not be ESTABLISHED in time — retryable.
 *
@@ -430,7 +430,7 @@ const EMBED_DIM$1 = 1536;
 const EMBED_TASK_DOCUMENT$1 = "RETRIEVAL_DOCUMENT";
 const EMBED_TASK_QUERY$1 = "RETRIEVAL_QUERY";
 /** bump ⇒ provenance (v5: CommonMark fences). All char limits count CODE POINTS (Python len parity). */
-const CHUNK_POLICY$1 = "heading-aware-1500-content-only-v5";
+const CHUNK_POLICY$1 = "heading-aware-1500-content-only-v7";
 /**
 * The kernel's view of `instance.md` (adapted from oracle SC/instance.py
 * under decision 11 — an adaptation, not a port: ksor has ONE instance file
@@ -4452,7 +4452,7 @@ async function withPgRetry(op, options = {}) {
 	throw lastError;
 }
 //#endregion
-//#region ../content/dist/commands-BRK58wVc.mjs
+//#region ../content/dist/commands-B3-NrLiw.mjs
 /**
 * EVAL-LOCKED constants, quarried verbatim from the oracle
 * (sor-agentfactory @ b554f91, config.py) — changing any of these is a
@@ -4467,7 +4467,7 @@ const EMBED_DIM = 1536;
 const EMBED_TASK_DOCUMENT = "RETRIEVAL_DOCUMENT";
 const EMBED_TASK_QUERY = "RETRIEVAL_QUERY";
 /** bump ⇒ provenance (v5: CommonMark fences). All char limits count CODE POINTS (Python len parity). */
-const CHUNK_POLICY = "heading-aware-1500-content-only-v5";
+const CHUNK_POLICY = "heading-aware-1500-content-only-v7";
 const MAX_CHARS = 1500;
 /** < Gemini's 2048-token embed input. */
 const HARD_MAX_CHARS = 4e3;
@@ -7004,6 +7004,17 @@ function buildManifestFromTree(root, options) {
 	const files = [];
 	const sources = /* @__PURE__ */ new Map();
 	const skipped = [];
+	/** foreign ordering key -> the documents that declare it and no `order:`. */
+	const foreignOrder = /* @__PURE__ */ new Map();
+	const noteForeignOrder = (meta, path) => {
+		if (meta["order"] !== void 0 && meta["order"] !== null) return;
+		for (const key of FOREIGN_ORDER_KEYS) {
+			if (meta[key] === void 0 || meta[key] === null) continue;
+			const seen = foreignOrder.get(key) ?? [];
+			seen.push(path);
+			foreignOrder.set(key, seen);
+		}
+	};
 	const fullPath = (relSegs, name) => `${rootPath}/${[...relSegs, name].join("/")}`;
 	const addFile = (nodeSid, fileSegs) => {
 		const rel = fileSegs.join("/");
@@ -7028,8 +7039,10 @@ function buildManifestFromTree(root, options) {
 				continue;
 			}
 			if (INDEX_NAMES.includes(f.name)) continue;
+			const fileMeta = frontmatterMeta(f.text);
+			noteForeignOrder(fileMeta, fullPath(relSegs, f.name));
 			ordered.push({
-				order: orderValue(frontmatterMeta(f.text)["order"]),
+				order: orderValue(fileMeta["order"]),
 				tie: tieKey(f.name),
 				entry: f
 			});
@@ -7041,6 +7054,7 @@ function buildManifestFromTree(root, options) {
 			}
 			const index = indexOf(d, fullPath(relSegs, d.name));
 			const dirMeta = index === null ? {} : frontmatterMeta(index.text);
+			if (index !== null) noteForeignOrder(dirMeta, fullPath(relSegs, `${d.name}/${index.name}`));
 			ordered.push({
 				order: orderValue(dirMeta["order"]),
 				tie: tieKey(d.name),
@@ -7100,6 +7114,12 @@ function buildManifestFromTree(root, options) {
 	}
 	walk(root, [], null);
 	for (const s of skipped) onSkip(`plain-tree: skipped ${s}`);
+	for (const [key, paths] of foreignOrder) {
+		const rel = paths.map((x) => x.startsWith(`${rootPath}/`) ? x.slice(rootPath.length + 1) : x);
+		const shown = rel.slice(0, 3).join(", ");
+		const more = rel.length - Math.min(3, rel.length);
+		onSkip(`plain-tree: ${rel.length} document(s) declare \`${key}\`, which this record does not read — reading order fell back to file name (${shown}${more > 0 ? `, and ${more} more` : ""}). Rename it to \`order:\` to keep the intended sequence.`);
+	}
 	if (files.length === 0) throw new ManifestError(`plain-tree root ${rootPath} contains no Markdown`);
 	const manifest = {
 		format: 1,
@@ -7186,6 +7206,22 @@ function codePointCompare(a, b) {
 	}
 	return as.length - bs.length;
 }
+/**
+* Ordering keys OTHER ecosystems read, which this record does not.
+*
+* Reading order here is the governed `order:` key alone (decision 9 retired the
+* predecessor's Docusaurus keys; the MCP door had been reading them). But a
+* corpus arriving from Docusaurus, Hugo or Jekyll carries its own, and ignoring
+* one silently produces a WRONG order rather than a missing one — filename
+* order, served to `llms.txt`, the sidebar and the `outline` tool alike. Found
+* on a real 81-document book where 73 files declared `sidebar_position` (#74).
+*/
+const FOREIGN_ORDER_KEYS = [
+	"sidebar_position",
+	"position",
+	"weight",
+	"nav_order"
+];
 /** Re-exported so every reader of a document agrees where its frontmatter ENDS. */
 const FRONTMATTER$1 = /^\uFEFF?---\r?\n([\s\S]*?)\r?\n---[ \t]*\r?\n?/;
 const YAML_BOOLS = {
@@ -7209,8 +7245,12 @@ const YAML_BOOLS = {
 	OFF: false
 };
 /**
-* Minimal PyYAML-compatible frontmatter reader for the FOUR scalar keys this
-* adapter consumes (`title`, `position`, `sidebar_position`, `sor_id`) — the
+* Minimal PyYAML-compatible frontmatter reader. It parses every top-level
+* scalar; the adapter consumes `title`, `order` and `sor_id`, and reads the rest
+* only to WARN about them (see FOREIGN_ORDER_KEYS). The wording here named
+* `position` and `sidebar_position` until now, which is what this adapter read
+* before ordering became one governed key — the keys it names are the ones it
+* stopped reading. The
 * kernel discards every other frontmatter key at build time (taxonomy comes
 * from the manifest), so a YAML dependency would buy nothing (guard rule 5).
 * Scope, deliberately narrow pending a shared markdown module: top-level
@@ -7268,11 +7308,11 @@ function scalarValue(raw) {
 		ok: true,
 		value: Number.parseFloat(plain.replaceAll("_", ""))
 	};
-	if (/:[ \t]/.test(plain) || plain.endsWith(":")) return {
-		ok: false,
+	if (/^[|>&*!{[]/.test(plain)) return {
+		ok: true,
 		value: null
 	};
-	if (/^[|>&*!{[]/.test(plain)) return {
+	if (/:[ \t]/.test(plain) || plain.endsWith(":")) return {
 		ok: false,
 		value: null
 	};
@@ -7882,23 +7922,78 @@ function cleanBody(rawBody) {
 function teachingBody(content) {
 	return pyStrip(pySplitLines(content, false).filter((ln) => !HEADING.test(ln)).join("\n"));
 }
-function classify(content, headingPath) {
-	if (JSX_ASSESS.test(content)) return "assessment";
-	const leaf = headingPath.length > 0 ? headingPath[headingPath.length - 1] : "";
-	if (JSX_EMBED.test(content) || content.includes("docs.google.com/presentation") || leaf.includes("Teaching Aid")) return "embed";
-	if (cpLen(teachingBody(content)) < 250) return "nav";
-	return "prose";
+/**
+* A line that is navigation rather than prose: strip its list marker and what
+* remains is nothing but links.
+*
+* Deliberately narrow. A line with prose AROUND a link ("Claim within thirty
+* days; see the [expenses page](x) to file.") is prose, because the sentence is
+* the content and the link is incidental.
+*/
+const NAV_LINE = /^(?:[-*+]\s+|\d+[.)]\s+)?(?:\[[^\]]*\]\([^)]*\)|<https?:\/\/[^>]*>|https?:\/\/\S+)(?:[\s,;·|>—–-]*(?:\[[^\]]*\]\([^)]*\)|<https?:\/\/[^>]*>|https?:\/\/\S+))*[\s.,;:]*$/;
+/**
+* Is this segment NAVIGATION — a thing that points at content rather than
+* being content?
+*
+* The oracle answered this with length: under 250 code points meant nav.
+* On the curriculum corpus it was tuned against, that proxy holds — a short
+* segment there really is a link list. On a handbook it inverts, because a
+* handbook's most valuable statements are its shortest ("Six months, with a
+* written review at three and six"), and `nav` is excluded from search. Issue
+* #55, walked live on 0.0.14: three of four chunks in an ordinary policy
+* record were unsearchable, and a question the record plainly answered was
+* served the scaffold's placeholder instead.
+*
+* So the question is asked about SHAPE, which is what "navigation" always
+* meant. A segment is nav when link lines are most of it, or when what is left
+* after them is too little to answer anything (MIN_CONTENT_CHARS — the same
+* floor the serving predicate applies, so this never labels `prose` something
+* search would refuse to return anyway).
+*
+* Length is no longer consulted. A 180-character link list is nav and a
+* 51-character fact is prose, which is the ordering length got backwards.
+*/
+function isNavShaped(content) {
+	const lines = pySplitLines(teachingBody(content), false).map((ln) => pyStrip(ln)).filter((ln) => ln !== "");
+	if (lines.length === 0) return true;
+	if (lines.filter((ln) => NAV_LINE.test(ln)).length * 2 > lines.length) return true;
+	return cpLen(lines.filter((ln) => !NAV_LINE.test(ln)).join(" ")) < 24;
 }
-/** A segment DOMINATED by a line-leading widget (with < NAV_MAX_CHARS of
-* teaching body before it) labels EVERY fragment — a char-sliced widget must
-* not leak as prose. */
-function segmentMarkerType(span) {
+/**
+* Does a line-leading widget DOMINATE this span?
+*
+* The widget regexes match an opening tag only, so the tag's position is where
+* teaching stops and markup begins. The question is therefore about what comes
+* BEFORE it: if that is navigation-shaped, the span is the widget; if it is real
+* explanation, the widget is a minority of a teaching passage.
+*
+* This used to be a length test — 250 characters of teaching body before the
+* widget and the whole span became `assessment`, which no retrieval arm returns.
+* #55 moved navigation from length to shape and left this path behind, so a
+* section carrying 180 characters of real explanation before a `<Quiz>` lost the
+* explanation with it (issue #75).
+*/
+function dominantWidget(span) {
 	for (const [re, label] of [[JSX_ASSESS, "assessment"], [JSX_EMBED, "embed"]]) {
 		const m = re.exec(span);
-		if (m !== null && cpLen(teachingBody(span.slice(0, m.index))) < 250) return label;
+		if (m !== null && isNavShaped(span.slice(0, m.index))) return label;
 	}
 	return null;
 }
+function classify(content, headingPath) {
+	const widget = dominantWidget(content);
+	if (widget !== null) return widget;
+	const leaf = headingPath.length > 0 ? headingPath[headingPath.length - 1] : "";
+	if (content.includes("docs.google.com/presentation") || leaf.includes("Teaching Aid")) return "embed";
+	if (isNavShaped(content)) return "nav";
+	return "prose";
+}
+/** A segment dominated by a line-leading widget labels EVERY fragment — a
+* char-sliced widget must not leak as prose. Same question as `classify`, asked
+* of the whole segment rather than one piece of it. */
+function segmentMarkerType(span) {
+	return dominantWidget(span);
+}
 /** Walk lines; headings count only OUTSIDE fences; every line lands in exactly
 * one segment (byte-exact). H1 records a title but never enters the path. */
 function segmentText(text) {
@@ -8006,7 +8101,7 @@ function chunkText(text, maxChars = MAX_CHARS) {
 			} else prefix += seg.text;
 			continue;
 		}
-		const segIsNav = cpLen(teachingBody(seg.text)) < 250;
+		const segIsNav = isNavShaped(seg.text);
 		const segMarker = segmentMarkerType(seg.text);
 		for (const piece of subsplit(seg.text, maxChars)) {
 			let sourceType;
@@ -8919,11 +9014,11 @@ async function ingestCommand(args) {
 	process.stdout.write(`ingest: generation ${report.generation} — ${report.nodes} nodes, ${report.chunks} chunks; embedded ${report.embedded}, carried ${report.carried}, failed ${report.failed}\n`);
 	if (report.unsearchable > 0) {
 		const pct = Math.round(report.unsearchable / Math.max(report.chunks, 1) * 100);
-		process.stdout.write(`  not searchable: ${report.unsearchable} of ${report.chunks} chunk(s) (${pct}%) are shorter than the navigation threshold — stored and readable, but no search returns them\n`);
+		process.stdout.write(`  not searchable: ${report.unsearchable} of ${report.chunks} chunk(s) (${pct}%) read as navigation rather than content — stored and readable, but no search returns them\n`);
 		if (report.unsearchableSources.length > 0) {
 			const named = report.unsearchableSources.slice(0, 10).join(", ");
 			const more = report.unsearchableSources.length - Math.min(10, report.unsearchableSources.length);
-			process.stdout.write(`  FOUND ONLY BY NAME: ${named}${more > 0 ? `, and ${more} more` : ""} — no searchable chunk at all; lengthen these sections or read them by slug
+			process.stdout.write(`  FOUND ONLY BY NAME: ${named}${more > 0 ? `, and ${more} more` : ""} — no searchable chunk at all — a page of links reads as navigation; give it prose of its own, or reach it by slug
 `);
 		}
 	}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@panaversity/ksor",
-  "version": "0.0.14",
+  "version": "0.0.16",
   "description": "Knowledge System of Record — compile governed markdown into a static site for people and an MCP server for AI agents, with citations and measured abstention.",
   "keywords": [
     "abstention",

package/templates/scaffold/system/site/lib/denial-rule.ts

@@ -136,8 +136,16 @@ function readScalar(raw: string): ScalarRead {
   // override — which is what the kernel does. Kept in step with `scalarValue`
   // in ingest/adapters/plain-tree.ts and bound to it by
   // `stable-id-conformance.test.ts`.
+  // VALID YAML this reader does not model: a flow sequence or mapping, a block
+  // scalar, an anchor/alias/tag. PyYAML parses every one — the DOCUMENT is fine
+  // and only this KEY is beyond the reader, so it must not empty the map.
+  // Checked BEFORE the ": " test, because a flow mapping legitimately contains
+  // one (`meta: {a: 1}`).
+  // `typed` rather than `refused`: the key exists but is not a string, so this
+  // map (which holds strings) omits it and no override is taken — exactly what
+  // the kernel now does with `value: null` (issue #78).
+  if (/^[|>&*!{[]/.test(plain)) return { kind: "typed", value: "" };
   if (/:[ \t]/.test(plain) || plain.endsWith(":")) return { kind: "refused", value: "" };
-  if (/^[|>&*!{[]/.test(plain)) return { kind: "refused", value: "" };
   if (YAML_TYPED.test(plain)) return { kind: "typed", value: "" };
   return { kind: "string", value: plain };
 }