@panaversity/ksor 0.0.17 → 0.0.19

package/CHANGELOG.md

@@ -1,5 +1,101 @@
 # @panaversity/ksor
 
+## 0.0.19
+
+### Patch Changes
+
+- aa4bdce: Record why the vector index is unused, and what fixing it would cost
+
+  Diagnosis only — no serving behaviour changes. Answers are unaffected, and were
+  already correct: the query plans a sequential scan, and a sequential scan is
+  EXACT. What grows with the corpus is the work, not the error.
+
+  The cause recorded until now — a window function, then joins and predicates
+  Postgres cannot estimate — was incomplete. Testing each clause on its own shows
+  a cost mispricing underneath: a full sequential pass over 20,000 chunks,
+  including 20,000 1536-dimension distance computations, is priced at 1904 for
+  work that takes ~130 ms, while the HNSW scan's startup cost alone is 2137.
+
+  A restructured arm reaches 36 ms against 648 ms — but only with `ef_search` at
+  pgvector's default, which is the setting where the index missed the true nearest
+  neighbour for 1 query in 100 on a bed with real cluster structure, dropping the
+  top-1 similarity by 0.99. Against this record's ~0.01 abstention separation,
+  that flips an abstention: the corpus holds the answer and the door says it does
+  not. The speed and the approximation cannot be separated, so taking them is an
+  owner decision rather than a tuning change.
+
+  Both the current plan and the fix path are now pinned by tests, so neither can
+  drift unnoticed.
+
+- b9f3d00: A citation pin no longer outlives a restriction
+
+  A snapshot token pins a generation so a citation keeps resolving to the same
+  bytes. It was also deciding the _audience_ question — evaluating `visibility` on
+  the pinned row — so a document restricted after the token was issued kept reading
+  in full for the token's life, to a caller the record had just closed it to.
+
+  Three routes refused it and one served it, on the same surface, in the same
+  second: `outline` omitted it, `search` filtered it, an unpinned `read` refused it,
+  and `read` with a pre-flip token returned the whole document.
+
+  The generation pointers are why the obvious guard missed it. A flip sets
+  `rollback_generation` to the generation just superseded, so a pre-flip pin is
+  exactly the rollback pointer — servable by design, and the check that narrows a
+  pin to {active, rollback} passed it.
+
+  **Governance is now read from the record as it stands.** A pin still decides
+  which generation's content is served; it no longer decides whether the caller may
+  have it. A document the record no longer contains cannot be resurrected by one
+  either. Unpinned reads are unaffected — with nothing pinned, the two generations
+  are the same one and the check is an identity.
+
+  The cost is deliberate: a citation can stop resolving within the token's 30
+  minutes when the record restricts what it points at. That is what "the record
+  changed" should look like. The alternative is a window in which a withdrawal is
+  not a withdrawal.
+
+## 0.0.18
+
+### Patch Changes
+
+- ea049fd: A takedown can no longer stop applying without saying so
+
+  Two ways a recorded withdrawal quietly stopped covering what it was recorded to
+  cover. Both were found by attacking the door before exposing it publicly, and
+  both were reproduced end to end against a real database.
+
+  **A denial matched nothing after the document moved.** `takedown_denylist`
+  records a `stable_id`, and the serving predicate matches those rows against the
+  documents in the generation being served — so an id that no longer exists denies
+  nothing. The default stable_id is derived from the file's path, which means an
+  ordinary rename or move of a withdrawn document was enough: search, read,
+  outline and the site all served it again, with no error anywhere. Adding an
+  `index.md` beside a withdrawn section did the same, by changing the section's id.
+
+  Serving now refuses in that state, and so does the ingest that would create it —
+  the same check at both ends, so a generation where a withdrawal has stopped
+  applying cannot be published _or_ served:
+
+  ```
+  2 takedown(s) match no document in generation 7: knowledge/legal/notice.md, …
+    why: … an id that no longer exists denies NOTHING — so a withdrawn document
+    that was renamed, moved, or had an index.md added beside it is served again
+    fix: point the denial at where the document lives now, or retire it
+    deliberately — never guess which one, because the tool cannot tell a rename
+    from a deletion
+  ```
+
+  Refusing rather than re-pointing automatically is the whole point: a tool that
+  guessed would eventually guess that a withdrawn document had been deleted when
+  it had been renamed.
+
+  **A withdrawn section did not cover its own directory.** When a section has no
+  `index.md` and its documents all live one level further down, it had no file to
+  name its own directory, so only the subdirectory was exported to the site. A
+  document written directly under the withdrawn section published to `/docs` and
+  `llms.txt` in the window before the next ingest. The section's directory is now
+  derived from its own identity, which for an index-less section is its path.
+
 ## 0.0.17
 
 ### Patch Changes

package/dist/cli.mjs

@@ -15,7 +15,7 @@ import { bodyLimit } from "hono/body-limit";
 import { execFileSync, spawnSync } from "node:child_process";
 import { parseArgs } from "node:util";
 import { readFile, readdir, stat } from "node:fs/promises";
-//#region ../content-gateway/dist/main-BtKmcm72.mjs
+//#region ../content-gateway/dist/main-DDeyGVjK.mjs
 /**
 * A connection could not be ESTABLISHED in time — retryable.
 *
@@ -1830,7 +1830,8 @@ async function assertGovernanceServable$1(pool, instance, targetGeneration) {
 		if (generation === 0) return {
 			generation,
 			builtAt: null,
-			restricted: 0
+			restricted: 0,
+			orphaned: []
 		};
 		return {
 			generation,
@@ -1839,13 +1840,27 @@ async function assertGovernanceServable$1(pool, instance, targetGeneration) {
 				instance.corpusId,
 				generation
 			])).rows[0]?.schema_version ?? null,
-			restricted: declaresModel ? 0 : Number((await client.query("SELECT count(*)::int AS n FROM content_nodes WHERE tenant_id = $1 AND generation = $2 AND visibility IS NOT NULL", [instance.tenantId, generation])).rows[0].n)
+			restricted: declaresModel ? 0 : Number((await client.query("SELECT count(*)::int AS n FROM content_nodes WHERE tenant_id = $1 AND generation = $2 AND visibility IS NOT NULL", [instance.tenantId, generation])).rows[0].n),
+			orphaned: (await client.query("SELECT d.stable_id FROM takedown_denylist d WHERE d.tenant_id = $1 AND d.corpus_id = $2   AND NOT EXISTS (SELECT 1 FROM content_nodes n                    WHERE n.tenant_id = d.tenant_id AND n.corpus_id = d.corpus_id                      AND n.generation = $3 AND n.stable_id = d.stable_id) ORDER BY d.stable_id", [
+				instance.tenantId,
+				instance.corpusId,
+				generation
+			])).rows.map((r) => r.stable_id)
 		};
 	});
 	if (state.generation === 0) return;
 	if (declaresModel && (state.builtAt === null || compareSchemaVersion$1(state.builtAt, "2.2") < 0)) throw new GovernanceGateError$1(`generation ${state.generation} was built against schema ${state.builtAt ?? "(before 2.4, which is when a generation started recording this)"}, older than 2.2 — the version that put visibility on the node row\n  why: instance.md declares an audience model, but the documents in this generation carry no visibility at all. Every one of them would be served at default_visibility — the WIDEST tier — including any document whose frontmatter restricts it
   fix: rebuild the record so its governance reaches the database:
     ksor ingest --instance instance.md --knowledge knowledge --flip`);
+	if (state.orphaned.length > 0) {
+		const named = state.orphaned.slice(0, 5).join(", ");
+		const more = state.orphaned.length - Math.min(5, state.orphaned.length);
+		throw new GovernanceGateError$1(`${state.orphaned.length} takedown(s) match no document in generation ${state.generation}: ${named}${more > 0 ? `, and ${more} more` : ""}\n  why: a denial is recorded against a stable_id, and the serving predicate matches it against the documents in this generation. An id that no longer exists denies NOTHING — so a withdrawn document that was renamed, moved, or had an index.md added beside it is served again by search, read, outline and the site, with no error anywhere. The denial is meant to be immune to reorganization; this is the state where it is not
+  fix: point the denial at where the document lives now, or retire it deliberately — never guess which one, because the tool cannot tell a rename from a deletion:
+    ksor takedown --instance instance.md --stable-id <the new id> --reason <why> --actor <who>
+    ksor takedown --instance instance.md --revoke <the old id> --actor <who>
+  (ksor takedown --list shows what is recorded)`);
+	}
 	if (!declaresModel && state.restricted > 0) throw new GovernanceGateError$1(`${state.restricted} document(s) in generation ${state.generation} declare visibility:, but instance.md declares no audiences:
   why: an author restricted those documents and nothing would enforce it — this door would serve them in full to every caller, and the frontmatter key saying otherwise would be the only trace. The site refuses to BUILD in this exact state (ksor-visibility-without-audiences); the door must not serve in it
   fix: declare the model in instance.md (audiences: least-restricted first, plus default_visibility:), or remove the visibility: keys and re-ingest`);
@@ -2038,6 +2053,28 @@ g AS (
         (SELECT active_generation FROM corpora
           WHERE tenant_id = $1 AND corpus_id = $2)
     ) AS gen
+),
+-- The generation the record is on RIGHT NOW, which decides governance even when
+-- content is served from a pinned one.
+--
+-- A snapshot pin exists so a citation keeps resolving to the same bytes. It used
+-- to decide the audience question too, by evaluating visibility on the pinned
+-- row — so a document restricted after the token was issued kept reading in full
+-- for the token's life, while outline, search and an unpinned read all
+-- refused it in the same second. servableGenerations could not catch it: a
+-- flip sets rollback_generation to the generation just superseded, so a pre-flip
+-- pin IS the rollback pointer and is servable by design (issue #87).
+--
+-- Pins yield. A citation may stop resolving within the token's life, which is
+-- what "the record changed" should look like — the alternative is a window in
+-- which a withdrawal is not a withdrawal, and decision 19 says a surface that
+-- refuses must refuse everywhere, which includes its own fourth route.
+--
+-- When nothing is pinned this is the SAME generation as g, so the join is an
+-- identity and no unpinned read changes behaviour.
+live AS (
+    SELECT active_generation AS gen FROM corpora
+     WHERE tenant_id = $1 AND corpus_id = $2
 )`;
 /** Candidates by LEAF slug ($4), each with its full root path (for suffix disambiguation). */
 const NODE_BY_SLUG_SQL = `
@@ -2067,7 +2104,12 @@ SELECT n.node_id, n.slug, n.title, n.stable_id, n.path, n.generation, n.permalin
 FROM tree n
 JOIN content_nodes self ON self.node_id = n.node_id AND self.tenant_id = $1
                        AND self.generation = n.generation
-WHERE n.slug = $4 AND ${DENY$1} AND ${audienceAllowed$1("self")}
+-- INNER join, so a document the record no longer contains cannot be
+-- resurrected by a pin either: no live row, no read.
+JOIN live ON TRUE
+JOIN content_nodes now ON now.tenant_id = $1 AND now.generation = live.gen
+                      AND now.stable_id = self.stable_id
+WHERE n.slug = $4 AND ${DENY$1} AND ${audienceAllowed$1("now")}
 ORDER BY n.path`;
 const ALIAS_SQL = `
 WITH ${GEN}
@@ -2084,8 +2126,11 @@ const NODE_BY_STABLE_ID_SQL = `
 WITH RECURSIVE ${GEN}, ${DENIED_CTE$1}
 SELECT n.node_id, n.slug, n.title, n.stable_id, n.stable_id::text AS path, n.generation, n.permalink
 FROM content_nodes n JOIN g ON n.generation = g.gen
+JOIN live ON TRUE
+JOIN content_nodes now ON now.tenant_id = $1 AND now.generation = live.gen
+                      AND now.stable_id = n.stable_id
 WHERE n.tenant_id = $1 AND n.stable_id = $4 AND n.status = 'published' AND ${DENY$1}
-  AND ${AUDIENCE_ALLOWED$1}`;
+  AND ${audienceAllowed$1("now")}`;
 const DOCUMENT_CHUNKS_SQL = `
 WITH ${GEN}
 SELECT c.ordinal, COALESCE(c.heading_path_text, ''), c.content
@@ -4501,7 +4546,7 @@ async function withPgRetry(op, options = {}) {
 	throw lastError;
 }
 //#endregion
-//#region ../content/dist/commands-CXqK2c2f.mjs
+//#region ../content/dist/commands-DcPJJlNb.mjs
 /**
 * EVAL-LOCKED constants, quarried verbatim from the oracle
 * (sor-agentfactory @ b554f91, config.py) — changing any of these is a
@@ -6425,7 +6470,8 @@ async function assertGovernanceServable(pool, instance, targetGeneration) {
 		if (generation === 0) return {
 			generation,
 			builtAt: null,
-			restricted: 0
+			restricted: 0,
+			orphaned: []
 		};
 		return {
 			generation,
@@ -6434,13 +6480,27 @@ async function assertGovernanceServable(pool, instance, targetGeneration) {
 				instance.corpusId,
 				generation
 			])).rows[0]?.schema_version ?? null,
-			restricted: declaresModel ? 0 : Number((await client.query("SELECT count(*)::int AS n FROM content_nodes WHERE tenant_id = $1 AND generation = $2 AND visibility IS NOT NULL", [instance.tenantId, generation])).rows[0].n)
+			restricted: declaresModel ? 0 : Number((await client.query("SELECT count(*)::int AS n FROM content_nodes WHERE tenant_id = $1 AND generation = $2 AND visibility IS NOT NULL", [instance.tenantId, generation])).rows[0].n),
+			orphaned: (await client.query("SELECT d.stable_id FROM takedown_denylist d WHERE d.tenant_id = $1 AND d.corpus_id = $2   AND NOT EXISTS (SELECT 1 FROM content_nodes n                    WHERE n.tenant_id = d.tenant_id AND n.corpus_id = d.corpus_id                      AND n.generation = $3 AND n.stable_id = d.stable_id) ORDER BY d.stable_id", [
+				instance.tenantId,
+				instance.corpusId,
+				generation
+			])).rows.map((r) => r.stable_id)
 		};
 	});
 	if (state.generation === 0) return;
 	if (declaresModel && (state.builtAt === null || compareSchemaVersion(state.builtAt, "2.2") < 0)) throw new GovernanceGateError(`generation ${state.generation} was built against schema ${state.builtAt ?? "(before 2.4, which is when a generation started recording this)"}, older than 2.2 — the version that put visibility on the node row\n  why: instance.md declares an audience model, but the documents in this generation carry no visibility at all. Every one of them would be served at default_visibility — the WIDEST tier — including any document whose frontmatter restricts it
   fix: rebuild the record so its governance reaches the database:
     ksor ingest --instance instance.md --knowledge knowledge --flip`);
+	if (state.orphaned.length > 0) {
+		const named = state.orphaned.slice(0, 5).join(", ");
+		const more = state.orphaned.length - Math.min(5, state.orphaned.length);
+		throw new GovernanceGateError(`${state.orphaned.length} takedown(s) match no document in generation ${state.generation}: ${named}${more > 0 ? `, and ${more} more` : ""}\n  why: a denial is recorded against a stable_id, and the serving predicate matches it against the documents in this generation. An id that no longer exists denies NOTHING — so a withdrawn document that was renamed, moved, or had an index.md added beside it is served again by search, read, outline and the site, with no error anywhere. The denial is meant to be immune to reorganization; this is the state where it is not
+  fix: point the denial at where the document lives now, or retire it deliberately — never guess which one, because the tool cannot tell a rename from a deletion:
+    ksor takedown --instance instance.md --stable-id <the new id> --reason <why> --actor <who>
+    ksor takedown --instance instance.md --revoke <the old id> --actor <who>
+  (ksor takedown --list shows what is recorded)`);
+	}
 	if (!declaresModel && state.restricted > 0) throw new GovernanceGateError(`${state.restricted} document(s) in generation ${state.generation} declare visibility:, but instance.md declares no audiences:
   why: an author restricted those documents and nothing would enforce it — this door would serve them in full to every caller, and the frontmatter key saying otherwise would be the only trace. The site refuses to BUILD in this exact state (ksor-visibility-without-audiences); the door must not serve in it
   fix: declare the model in instance.md (audiences: least-restricted first, plus default_visibility:), or remove the visibility: keys and re-ingest`);
@@ -6858,7 +6918,10 @@ async function deniedStableIds(pool, instance) {
 *
 * The seed's OWN file counts when the seed has children, and only then — see
 * the SQL comment: a container's index.md names its directory, a leaf's file
-* names its parent's.
+* names its parent's. An index-less container has no such file, so the one it
+* would have had is synthesized from its (path-derived) "#section" id — without
+* it, a section whose descendants all live one level down contributed only the
+* subdirectory and left its own level publishable (issue #86).
 */
 async function deniedSubtreeDirs(pool, instance) {
 	const paths = await runRead(pool, instance.tenantId, async (client) => {
@@ -6866,7 +6929,7 @@ async function deniedSubtreeDirs(pool, instance) {
          SELECT active_generation AS g FROM corpora WHERE tenant_id = $1 AND corpus_id = $2
        ),
        seed AS (
-         SELECT n.node_id
+         SELECT n.node_id, n.stable_id
            FROM takedown_denylist d
            JOIN content_nodes n ON n.tenant_id = d.tenant_id AND n.stable_id = d.stable_id
            JOIN gen ON n.generation = gen.g
@@ -6904,7 +6967,28 @@ async function deniedSubtreeDirs(pool, instance) {
                  WHERE NOT EXISTS (SELECT 1 FROM content_nodes kid
                                     JOIN gen ON kid.generation = gen.g
                                    WHERE kid.tenant_id = $1 AND kid.parent_id = s2.node_id)
-              )`, [instance.tenantId, instance.corpusId])).rows.map((r) => String(r.origin_path));
+              )
+        UNION
+        -- An INDEX-LESS container has no file at all, so the join above drops
+        -- it however many descendants it has, and a section whose files all
+        -- live one level down contributed only the SUBdirectory — leaving a
+        -- document written directly under the withdrawn section publishable
+        -- (issue #86). The round-10 "seed counts when it has children" rule was
+        -- right and could not fire here, because there was nothing to count.
+        --
+        -- So the index.md it WOULD have had is synthesized, and the container
+        -- then names its directory exactly as an index-bearing one does. This is
+        -- not the stable_id prefix matching decision 14 rejects: that fails
+        -- because a sor_id: override decouples an id from its path, and a
+        -- "#section" id is generated from the path with no frontmatter in
+        -- reach: there is no index file to carry an override
+        -- (adapters/plain-tree.ts:214-219).
+        SELECT substring(sd.stable_id from '^(.*)#section$') || '/index.md'
+          FROM seed sd
+         WHERE sd.stable_id LIKE '%#section'
+           AND EXISTS (SELECT 1 FROM content_nodes kid
+                        JOIN gen ON kid.generation = gen.g
+                       WHERE kid.tenant_id = $1 AND kid.parent_id = sd.node_id)`, [instance.tenantId, instance.corpusId])).rows.map((r) => String(r.origin_path));
 	});
 	const dirs = /* @__PURE__ */ new Set();
 	for (const raw of paths) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@panaversity/ksor",
-  "version": "0.0.17",
+  "version": "0.0.19",
   "description": "Knowledge System of Record — compile governed markdown into a static site for people and an MCP server for AI agents, with citations and measured abstention.",
   "keywords": [
     "abstention",