@panaversity/ksor 0.0.12 → 0.0.14

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,178 @@
1
1
  # @panaversity/ksor
2
2
 
3
+ ## 0.0.14
4
+
5
+ ### Patch Changes
6
+
7
+ - a0d98b0: Cut dead weight, and repair two guards that had quietly stopped guarding
8
+
9
+ A sweep across every package, with each candidate handed to a second reviewer
10
+ whose job was to prove it still alive. Net −154 lines. Nothing an adopter can
11
+ observe changes; two things that were supposed to fail no longer stay silent.
12
+
13
+ **The two repairs.** A guard asserting that no scaffolded document describes
14
+ serving as publishing — a claim this repo has had to correct four times — ran
15
+ `readFileSync` inside a `try` whose `catch` returned quietly, and one of its five
16
+ filenames was `.env.example` while the scaffold emits `env.example`. So the row
17
+ covering the file that actually carries the serving variables had never executed.
18
+ The name is fixed and a missing file now fails instead of passing. Separately,
19
+ two doc-blocks described a stdio transport in the present tense; there is no
20
+ stdio door in the product, and the suite claiming to drive one drives HTTP.
21
+
22
+ **The removals.** A 134-line live-walk script pinned to `@panaversity/ksor@0.0.4`
23
+ that nothing referenced. `AuthConfig.jwksUrl`, computed and stored but never read
24
+ — its live twin is `explicitJwksUrl`; the boot-time validation of
25
+ `KSOR_JWKS_URL` stays exactly where it was. An `allowedAudiences.length > 0 &&`
26
+ operand that no path can reach as false, and whose false side would have skipped
27
+ the audience allowlist entirely. A `PoolTimeoutError` message parameter no caller
28
+ passed, which was also the one input where two retry classifiers disagreed —
29
+ removing it closes that. Two `instanceof X || instanceof Error` disjuncts where
30
+ `X extends Error`, so the first could never decide anything. One unused icon
31
+ export in the workbench shell.
32
+
33
+ **Left alone deliberately.** `SearchScope.kinds` is genuinely dead, but removing
34
+ it renumbers positional parameters across three SQL statements, two of which
35
+ derive a shared CTE by string substitution, and the test that would catch a wrong
36
+ renumber is gated on a database. That is a change to make on its own, with the
37
+ gate watching — not alongside a release.
38
+
39
+ - ce1595b: Ingest names the real reason it could not record a commit
40
+
41
+ Every first ingest of a freshly scaffolded project printed "knowledge/ is not in
42
+ a git repository". That is false: `ksor init` runs `git init`, so the repository
43
+ exists — it simply has no commit yet, and `rev-parse HEAD` fails with "unknown
44
+ revision" rather than because nothing is there. The reader was sent to `git
45
+ init`, which they had already run, in the one message that decides whether an
46
+ answer can be traced back to a reviewed commit.
47
+
48
+ Three different states were collapsing into that one sentence, and each has a
49
+ different next command:
50
+
51
+ ```
52
+ knowledge/ is in a git repository with no commits yet …
53
+ fix: commit the record (git add knowledge && git commit) and re-run
54
+
55
+ knowledge/ is not in a git repository …
56
+ fix: git init, commit the record, and re-run
57
+
58
+ git is not installed …
59
+ fix: install git, or pass --source-commit <sha> if the record is versioned elsewhere
60
+ ```
61
+
62
+ Verified on a real scaffold: the fresh case prints the first, and committing the
63
+ record turns the next ingest's `source:` line into an actual SHA.
64
+
65
+ - 474dedc: Internal: the env-contract drift test scans only the checkout's source
66
+
67
+ No adopter-visible behaviour changes. The test that guarantees every
68
+ adopter-settable environment variable is named in the scaffold's `env.example`
69
+ walked `packages/` with a `statSync` per entry, and descended into the fake npm
70
+ install another suite roots inside `packages/ksor`. That cost two ways: the
71
+ copied template sources were scanned twice, and an entry deleted between the
72
+ `readdir` and the `statSync` crashed the whole run — which is what took CI red
73
+ on run 32526491721, on an `llms.txt` being cleaned up concurrently.
74
+
75
+ The walk now takes each entry's type from the readdir snapshot itself, so a
76
+ vanishing entry cannot crash it, and it skips transient install trees, so its
77
+ input no longer depends on whether another suite is mid-run. The `REPO_ONLY`
78
+ exemption list was deleted as dead: it named seven variables that no scanned
79
+ file can contain, because the walk excludes test files in the first place. The
80
+ honesty check that is supposed to catch stale exemptions now covers every
81
+ exemption list, which is what its name always claimed.
82
+
83
+ ## 0.0.13
84
+
85
+ ### Patch Changes
86
+
87
+ - 8c5013b: A provider outage is never reported as "the record does not cover this"
88
+
89
+ When the embedding provider is down, the vector arm does not run — so an empty
90
+ result says nothing about coverage. It says we could not look. That distinction
91
+ was fixed once for records with a calibrated floor, and the condition was the
92
+ bug: it left the case out that matters most.
93
+
94
+ An **uncalibrated** record is the default state of every fresh scaffold. There
95
+ the emptiness came from the keyword arm, which abstains when it returns no rows
96
+ — and it returns nothing for almost every natural-language question, because
97
+ `websearch_to_tsquery` ANDs its terms (measured 12 of 12 on real questions). So
98
+ during any outage an uncalibrated record answered every question with
99
+ `abstained: true`, while the tool description instructs the agent to state that
100
+ as fact and never fall back on its own knowledge.
101
+
102
+ It reached this release because the existing test asked a question the keyword
103
+ arm could answer, so the degraded path served real hits and looked correct. Ask
104
+ the way a person asks and it did not. That case is now covered.
105
+
106
+ Found live against the published 0.0.12 with an invalid key — the same state a
107
+ rejected CI key had produced that morning, which is how likely this is.
108
+
109
+ **`ksor calibrate` also stops blessing a floor on far-domain evidence alone.**
110
+ The built-in out-of-corpus probes are all far-domain — dinner, taxes, boiling an
111
+ egg — and a shipped set cannot be scope-adjacent, because adjacency depends on a
112
+ corpus it has never seen. Far-domain probes score low against anything, so the
113
+ margin comes out inflated. Measured on one record, changing only the probe set:
114
+ built-ins reported "separable, margin 0.072" and recommended a floor; eight
115
+ scope-adjacent near-misses reported "NOT separable, margin -0.030" — and that
116
+ floor then answered six of the eight live, with citations. The tool already knew
117
+ to say "widen the probe set", but said it only on the not-separable branch, which
118
+ is when it is least needed. It now says it whenever the built-ins are used.
119
+
120
+ - 692d296: `ksor ingest` says how much of the record no search will return
121
+
122
+ A chunk shorter than the navigation threshold is stored, embedded and readable —
123
+ and excluded from every retrieval arm by the serving predicate. That rule exists
124
+ for a good reason: a "See also: [a] [b] [c]" block should never be a search hit.
125
+ But it decides by LENGTH ALONE, so a short _substantive_ paragraph is caught by
126
+ it too — and a policy handbook is made of short substantive statements.
127
+
128
+ Measured on a realistic five-document operations handbook with real embeddings:
129
+ **10 of 16 chunks unsearchable, and one entire document that `outline` lists and
130
+ `read` returns in full but `search` can never find.** A complete policy
131
+ statement — "Probation: six months, with a written review at three and six" —
132
+ is 191 characters, so the record treats it as navigation. The ingest line
133
+ reported a cheerful `16 chunks; embedded 16` and said nothing.
134
+
135
+ It says it now:
136
+
137
+ ```
138
+ ingest: generation 1 — 2 nodes, 4 chunks; embedded 4, carried 0, failed 0
139
+ not searchable: 3 of 4 chunk(s) (75%) are shorter than the navigation
140
+ threshold — stored and readable, but no search returns them
141
+ FOUND ONLY BY NAME: knowledge/onboarding:prose — no searchable chunk at all
142
+ ```
143
+
144
+ This does **not** change the threshold, and nothing that was searchable stops
145
+ being so. Where that line belongs needs a gold-set measurement, which is issue
146
+ #55. What is fixed here is the silence — because the silence is what let a
147
+ record ship most of itself unfindable, and told its owner everything was fine.
148
+
149
+ The count is computed with the serving predicate's own admission test, and a db
150
+ test compares it against what the SQL actually admits: a report the database
151
+ disagrees with would be worse than none.
152
+
153
+ - c4976dd: Two defects introduced in 0.0.12, found by verifying the published package live
154
+
155
+ **The discovery document became invalid exactly when a record became real.** The
156
+ MCP registry schema caps `ServerDetail.description` at **100 characters**
157
+ (2025-12-11). 0.0.12 started generating that description from the record's own
158
+ prose — a real improvement over the hard-coded sentence it replaced — and capped
159
+ it at 300. The unfilled placeholder is 88 characters and validates; a described
160
+ record's title plus scope sentence is routinely 150-350 and does not. So
161
+ `/.well-known/mcp/server.json` passed validation until the owner did the thing
162
+ the scaffold asks for, then silently stopped, with nothing in the build to say
163
+ so. It is now assembled inside the schema's budget and trimmed at a word
164
+ boundary rather than mid-word.
165
+
166
+ **The boot report reassured the operator in the one configuration that needs a
167
+ warning.** `KSOR_ALLOW_PUBLIC_UNAUTHENTICATED=1` permits an unauthenticated
168
+ public bind, and the auth line kept printing `DISABLED — 0.0.0.0 only, and a
169
+ public bind will refuse to boot` — false on both counts, at the moment the whole
170
+ record is being served to anyone who can reach the port. It now says that,
171
+ naming the variable responsible.
172
+
173
+ Both shipped in 0.0.12 and both were mine; the aligned boot report and the
174
+ self-describing discovery document are otherwise unchanged.
175
+
3
176
  ## 0.0.12
4
177
 
5
178
  ### Patch Changes
package/dist/cli.mjs CHANGED
@@ -16,7 +16,7 @@ import { bodyLimit } from "hono/body-limit";
16
16
  import { execFileSync, spawnSync } from "node:child_process";
17
17
  import { parseArgs } from "node:util";
18
18
  import { readFile, readdir, stat } from "node:fs/promises";
19
- //#region ../content-gateway/dist/main-CqOtG-q1.mjs
19
+ //#region ../content-gateway/dist/main-VL9lXhmT.mjs
20
20
  /**
21
21
  * A connection could not be ESTABLISHED in time — retryable.
22
22
  *
@@ -42,8 +42,8 @@ var ConnectTimeoutError$1 = class extends Error {
42
42
  * a thundering herd aimed at the component already drowning.
43
43
  */
44
44
  var PoolTimeoutError$1 = class extends Error {
45
- constructor(detail = "the configured checkout bound") {
46
- super(`pool checkout timed out (${detail}) — the pool is saturated; shedding this request is the recovery path, retrying it is not`);
45
+ constructor() {
46
+ super("pool checkout timed out (the configured checkout bound) — the pool is saturated; shedding this request is the recovery path, retrying it is not");
47
47
  this.name = "PoolTimeoutError";
48
48
  }
49
49
  };
@@ -1114,11 +1114,24 @@ g AS (
1114
1114
  WHERE tenant_id = $1 AND corpus_id = $2)
1115
1115
  ) AS gen
1116
1116
  )`;
1117
- const ARM_WHERE$1 = `
1117
+ /**
1118
+ * The arm predicate, built for the parameter numbering of the query that uses
1119
+ * it — a FUNCTION, not a string the caller renumbers afterwards.
1120
+ *
1121
+ * It used to be derived with `ARM_WHERE.replaceAll("$5", "$4")`, which works
1122
+ * only while the predicate happens to contain exactly one placeholder and no
1123
+ * other text matching it. Adding any second parameter to the predicate breaks
1124
+ * every derived query silently — and when it breaks, the failure arrives as a
1125
+ * driver error that the serving layer correctly reduces to "content store
1126
+ * temporarily unavailable", which tells you nothing about the cause. Found by
1127
+ * tripping over it while trying a fix for issue #59; taking the number as an
1128
+ * argument makes the coupling visible instead of textual.
1129
+ */
1130
+ const armWhere$1 = (kindsParam) => `
1118
1131
  c.tenant_id = $1 AND c.generation = g.gen
1119
1132
  AND c.embedding_status = 'embedded' AND ${SERVABLE$1}
1120
1133
  AND n.status = 'published'
1121
- AND ($5::text[] IS NULL OR n.kind = ANY($5::text[]))
1134
+ AND (${kindsParam}::text[] IS NULL OR n.kind = ANY(${kindsParam}::text[]))
1122
1135
  AND ${DENY$1}
1123
1136
  AND ${AUDIENCE_ALLOWED$1}`;
1124
1137
  const JOINS$1 = `
@@ -1127,6 +1140,15 @@ const JOINS$1 = `
1127
1140
  JOIN sources s ON s.source_id = c.source_id AND s.tenant_id = c.tenant_id
1128
1141
  AND s.generation = c.generation
1129
1142
  JOIN content_nodes n ON n.node_id = s.node_id AND n.tenant_id = s.tenant_id`;
1143
+ /**
1144
+ * Exported ONLY so a test can EXPLAIN the real thing.
1145
+ *
1146
+ * The index regression this fixes was announced as fixed once before, and came
1147
+ * back through a different clause, because nothing ever asserted the RESULT —
1148
+ * that the plan opens `idx_chunks_hnsw`. A timing threshold would be flaky and
1149
+ * would not have caught it either; the plan shape is the property that matters
1150
+ * (issue #59).
1151
+ */
1130
1152
  const HYBRID_SQL$1 = `
1131
1153
  WITH RECURSIVE ${GEN_CTE$1}, ${DENIED_CTE$1},
1132
1154
  -- The top-k is taken by a PLAIN \`ORDER BY <distance> LIMIT\`, and the rank
@@ -1147,7 +1169,7 @@ WITH RECURSIVE ${GEN_CTE$1}, ${DENIED_CTE$1},
1147
1169
  FROM (
1148
1170
  SELECT c.chunk_id, g.gen, (c.embedding <=> $3::vector) AS dist
1149
1171
  ${JOINS$1}
1150
- WHERE ${ARM_WHERE$1}
1172
+ WHERE ${armWhere$1("$5")}
1151
1173
  ORDER BY c.embedding <=> $3::vector, c.chunk_id
1152
1174
  LIMIT $6
1153
1175
  ) ranked),
@@ -1156,7 +1178,7 @@ WITH RECURSIVE ${GEN_CTE$1}, ${DENIED_CTE$1},
1156
1178
  row_number() OVER (ORDER BY ts_rank_cd(c.search_tsv,
1157
1179
  websearch_to_tsquery($9::regconfig, $4)) DESC, c.chunk_id) AS r
1158
1180
  ${JOINS$1}
1159
- WHERE ${ARM_WHERE$1}
1181
+ WHERE ${armWhere$1("$5")}
1160
1182
  AND c.search_tsv @@ websearch_to_tsquery($9::regconfig, $4)
1161
1183
  ORDER BY r LIMIT $6),
1162
1184
  fused AS (
@@ -1179,10 +1201,10 @@ WITH RECURSIVE ${GEN_CTE$1.replace("$8", "$6")}, ${DENIED_CTE$1}
1179
1201
  ts_rank_cd(c.search_tsv, websearch_to_tsquery($7::regconfig, $3)) AS score,
1180
1202
  g.gen, n.permalink
1181
1203
  ${JOINS$1}
1182
- WHERE ${ARM_WHERE$1.replaceAll("$5", "$4")}
1204
+ WHERE ${armWhere$1("$4")}
1183
1205
  AND c.search_tsv @@ websearch_to_tsquery($7::regconfig, $3)
1184
1206
  ORDER BY score DESC, c.chunk_id LIMIT $5`;
1185
- `${GEN_CTE$1.replace("$8", "$5")}${DENIED_CTE$1}${JOINS$1}${ARM_WHERE$1.replaceAll("$5", "$4")}`;
1207
+ `${GEN_CTE$1.replace("$8", "$5")}${DENIED_CTE$1}${JOINS$1}${armWhere$1("$4")}`;
1186
1208
  const HIT_COLUMNS = 9;
1187
1209
  /** Serialize a query vector as a pgvector literal. */
1188
1210
  function vectorLiteral$1(vector) {
@@ -2604,7 +2626,7 @@ async function search(ctx, query, k = 10) {
2604
2626
  const r = await client.query("SELECT active_generation FROM corpora WHERE tenant_id = $1 AND corpus_id = $2", [inst.tenantId, inst.corpusId]);
2605
2627
  return Number(r.rows[0]?.active_generation ?? 0) === 0;
2606
2628
  }, audienceScope(ctx));
2607
- const reason = embedFailed && inst.abstain.vectorFloor !== null ? "unavailable" : unpublished ? "unpublished" : "abstained";
2629
+ const reason = embedFailed ? "unavailable" : unpublished ? "unpublished" : "abstained";
2608
2630
  return {
2609
2631
  ok: false,
2610
2632
  abstained: reason === "abstained",
@@ -3057,7 +3079,7 @@ function buildServer(ctx, version) {
3057
3079
  structuredContent: result
3058
3080
  };
3059
3081
  } catch (error) {
3060
- if (error instanceof EmptyQueryError || error instanceof Error) return {
3082
+ if (error instanceof Error) return {
3061
3083
  content: [{
3062
3084
  type: "text",
3063
3085
  text: `Error: ${error.message}`
@@ -3166,7 +3188,7 @@ function toolError(error) {
3166
3188
  return {
3167
3189
  content: [{
3168
3190
  type: "text",
3169
- text: `Error: ${error instanceof UnknownSlug || error instanceof Error ? error.message : String(error)}`
3191
+ text: `Error: ${error instanceof Error ? error.message : String(error)}`
3170
3192
  }],
3171
3193
  isError: true
3172
3194
  };
@@ -3344,12 +3366,10 @@ function configFromEnv(env) {
3344
3366
  const allowedAudiences = (env.KSOR_JWT_ALLOWED_AUDIENCES ?? "").split(",").map((a) => a.trim()).filter((a) => a !== "");
3345
3367
  const issuer = (env.KSOR_SSO_ISSUER ?? "").trim() || null;
3346
3368
  const explicit = (env.KSOR_JWKS_URL ?? "").trim();
3347
- const jwksUrl = explicit || `${ssoUrl}/api/auth/jwks`;
3348
- assertHttpUrl("KSOR_JWKS_URL", jwksUrl, true);
3369
+ assertHttpUrl("KSOR_JWKS_URL", explicit || `${ssoUrl}/api/auth/jwks`, true);
3349
3370
  return {
3350
3371
  ssoUrl,
3351
3372
  resourceUrl,
3352
- jwksUrl,
3353
3373
  explicitJwksUrl: explicit === "" ? null : explicit,
3354
3374
  allowedAudiences,
3355
3375
  issuer,
@@ -3470,7 +3490,7 @@ function createVerify(config, deps, jwksOf) {
3470
3490
  cause: err
3471
3491
  });
3472
3492
  }
3473
- if (config.allowedAudiences.length > 0 && !audOk(claims.aud, config.allowedAudiences)) {
3493
+ if (!audOk(claims.aud, config.allowedAudiences)) {
3474
3494
  reject(key);
3475
3495
  throw new TokenVerifyError(`token aud ${JSON.stringify(claims.aud ?? null)} not in allowlist ${JSON.stringify(config.allowedAudiences)}`, { transient: false });
3476
3496
  }
@@ -3595,9 +3615,10 @@ function withoutSdkResponseModeWarning(body) {
3595
3615
  * it survivable (`buildAuth` refuses a non-loopback bind without auth, so the
3596
3616
  * only way to read this line is on a host that cannot be reached from outside).
3597
3617
  */
3598
- function authPosture(mode, host) {
3599
- if (mode === "disabled") return `DISABLED — ${host} only, and a public bind will refuse to boot`;
3600
- return "bearer tokens, verified against the record's authorization server";
3618
+ function authPosture(mode, host, publicUnauthenticated) {
3619
+ if (mode !== "disabled") return "bearer tokens, verified against the record's authorization server";
3620
+ if (publicUnauthenticated) return `UNAUTHENTICATED and bound to ${host} — KSOR_ALLOW_PUBLIC_UNAUTHENTICATED=1 is set, so the whole record is served to anyone who can reach this port`;
3621
+ return `DISABLED — ${host} only, and a public bind will refuse to boot`;
3601
3622
  }
3602
3623
  /**
3603
3624
  * What the record will refuse. `null` means no floor is declared and the gate
@@ -3625,8 +3646,8 @@ const UNDESCRIBED_RECORD = "instance.md is still the scaffold template — agent
3625
3646
  * Composition (oracle main.py's boot order, adapted): instance → DSN via
3626
3647
  * the declared env NAME → provider → pool → space guard → service context.
3627
3648
  * Auth is built by the door that needs it (http.ts) — BEFORE the pool
3628
- * serves anything; stdio is the local loopback-equivalent door and runs
3629
- * with auth off by construction.
3649
+ * serves anything; a loopback bind is the local-equivalent door and is the
3650
+ * only posture that may run with auth explicitly disabled.
3630
3651
  */
3631
3652
  async function compose(instancePath, version) {
3632
3653
  let instanceText;
@@ -4021,7 +4042,7 @@ async function runHttp(composition) {
4021
4042
  s.once("error", reject);
4022
4043
  });
4023
4044
  if (recordIsUndescribed(instance.instructions)) console.error(bootLine("identity", UNDESCRIBED_RECORD));
4024
- console.error(bootLine("auth", authPosture(auth.mode, bind.host)));
4045
+ console.error(bootLine("auth", authPosture(auth.mode, bind.host, process.env["KSOR_ALLOW_PUBLIC_UNAUTHENTICATED"] === "1" && !loopback)));
4025
4046
  for (const line of keyLines) console.error(line);
4026
4047
  console.error(bootLine("abstain", abstainPosture(instance.abstain.vectorFloor)));
4027
4048
  console.error(bootLine("serving", `http://${bind.host}:${bind.port}/mcp`));
@@ -4126,8 +4147,8 @@ var ConnectTimeoutError = class extends Error {
4126
4147
  * a thundering herd aimed at the component already drowning.
4127
4148
  */
4128
4149
  var PoolTimeoutError = class extends Error {
4129
- constructor(detail = "the configured checkout bound") {
4130
- super(`pool checkout timed out (${detail}) — the pool is saturated; shedding this request is the recovery path, retrying it is not`);
4150
+ constructor() {
4151
+ super("pool checkout timed out (the configured checkout bound) — the pool is saturated; shedding this request is the recovery path, retrying it is not");
4131
4152
  this.name = "PoolTimeoutError";
4132
4153
  }
4133
4154
  };
@@ -4431,7 +4452,7 @@ async function withPgRetry(op, options = {}) {
4431
4452
  throw lastError;
4432
4453
  }
4433
4454
  //#endregion
4434
- //#region ../content/dist/commands-Cysnkk_R.mjs
4455
+ //#region ../content/dist/commands-BRK58wVc.mjs
4435
4456
  /**
4436
4457
  * EVAL-LOCKED constants, quarried verbatim from the oracle
4437
4458
  * (sor-agentfactory @ b554f91, config.py) — changing any of these is a
@@ -5193,11 +5214,24 @@ g AS (
5193
5214
  WHERE tenant_id = $1 AND corpus_id = $2)
5194
5215
  ) AS gen
5195
5216
  )`;
5196
- const ARM_WHERE = `
5217
+ /**
5218
+ * The arm predicate, built for the parameter numbering of the query that uses
5219
+ * it — a FUNCTION, not a string the caller renumbers afterwards.
5220
+ *
5221
+ * It used to be derived with `ARM_WHERE.replaceAll("$5", "$4")`, which works
5222
+ * only while the predicate happens to contain exactly one placeholder and no
5223
+ * other text matching it. Adding any second parameter to the predicate breaks
5224
+ * every derived query silently — and when it breaks, the failure arrives as a
5225
+ * driver error that the serving layer correctly reduces to "content store
5226
+ * temporarily unavailable", which tells you nothing about the cause. Found by
5227
+ * tripping over it while trying a fix for issue #59; taking the number as an
5228
+ * argument makes the coupling visible instead of textual.
5229
+ */
5230
+ const armWhere = (kindsParam) => `
5197
5231
  c.tenant_id = $1 AND c.generation = g.gen
5198
5232
  AND c.embedding_status = 'embedded' AND ${SERVABLE}
5199
5233
  AND n.status = 'published'
5200
- AND ($5::text[] IS NULL OR n.kind = ANY($5::text[]))
5234
+ AND (${kindsParam}::text[] IS NULL OR n.kind = ANY(${kindsParam}::text[]))
5201
5235
  AND ${DENY}
5202
5236
  AND ${AUDIENCE_ALLOWED}`;
5203
5237
  const JOINS = `
@@ -5206,14 +5240,14 @@ const JOINS = `
5206
5240
  JOIN sources s ON s.source_id = c.source_id AND s.tenant_id = c.tenant_id
5207
5241
  AND s.generation = c.generation
5208
5242
  JOIN content_nodes n ON n.node_id = s.node_id AND n.tenant_id = s.tenant_id`;
5209
- `${GEN_CTE}${DENIED_CTE}${JOINS}${ARM_WHERE}${JOINS}${ARM_WHERE}`;
5210
- `${GEN_CTE.replace("$8", "$6")}${DENIED_CTE}${JOINS}${ARM_WHERE.replaceAll("$5", "$4")}`;
5243
+ `${GEN_CTE}${DENIED_CTE}${JOINS}${armWhere("$5")}${JOINS}${armWhere("$5")}`;
5244
+ `${GEN_CTE.replace("$8", "$6")}${DENIED_CTE}${JOINS}${armWhere("$4")}`;
5211
5245
  /** The calibrator's standalone top-1 signal (the read path gets it free from HYBRID_SQL). */
5212
5246
  const TOP_ONE_SQL = `
5213
5247
  WITH RECURSIVE ${GEN_CTE.replace("$8", "$5")}, ${DENIED_CTE}
5214
5248
  SELECT 1 - (c.embedding <=> $3::vector) AS score
5215
5249
  ${JOINS}
5216
- WHERE ${ARM_WHERE.replaceAll("$5", "$4")}
5250
+ WHERE ${armWhere("$4")}
5217
5251
  ORDER BY c.embedding <=> $3::vector, c.chunk_id
5218
5252
  LIMIT 1`;
5219
5253
  /** Serialize a query vector as a pgvector literal. */
@@ -5795,6 +5829,29 @@ const BUILT_IN_OOC = [
5795
5829
  * measurement had an easier question set than production would.
5796
5830
  */
5797
5831
  const SYNTHESIZED_CAVEAT = "CAVEAT: synthesized queries are written FROM the passages they are then scored against, so they share vocabulary a reader's question will not. This door measures an UPPER BOUND on separation — treat the floor below as provisional until it has been checked against questions the corpus did not write (--queries-file), and re-run if real questions score under it.";
5832
+ /**
5833
+ * What a separability verdict is worth when the probes came from the binary.
5834
+ *
5835
+ * Every entry in BUILT_IN_OOC is far-domain — dinner, taxes, football, boiling
5836
+ * an egg. Those score low against ANY corpus, so max-OOC comes out artificially
5837
+ * low and the margin is inflated from that end, exactly as synthesized in-corpus
5838
+ * queries inflate it from the other. Measured on one record, changing ONLY the
5839
+ * probe set: built-ins reported "separable, margin 0.072" and recommended a
5840
+ * floor; eight scope-adjacent near-misses on the same corpus and the same
5841
+ * in-corpus questions reported "NOT separable, margin -0.030". The recommended
5842
+ * floor then answered six of those eight near-misses live, with citations
5843
+ * (2026-08-21).
5844
+ *
5845
+ * The tool already knows this — the not-separable branch tells the operator to
5846
+ * "widen the probe set (scope-adjacent near-misses, not only far-domain
5847
+ * questions)". It said so only AFTER weak probes had failed to bless a floor,
5848
+ * which is the one case where the advice is least needed.
5849
+ *
5850
+ * A shipped set cannot be scope-adjacent, because adjacency depends on a corpus
5851
+ * the binary has never seen. So this is stated whenever built-ins are used, on
5852
+ * BOTH branches, rather than pretending a better default exists.
5853
+ */
5854
+ const BUILT_IN_OOC_CAVEAT = "CAVEAT: the out-of-corpus probes are the BUILT-IN set, which is entirely far-domain — a shipped set cannot be scope-adjacent, because adjacency depends on a corpus it has never seen. Far-domain probes score low against anything, so this margin is an OVER-estimate and a floor it blesses may still answer near-misses just outside your scope. Re-run with --ooc-file naming questions a reader might plausibly ask that this record does NOT cover, and trust that verdict over this one.";
5798
5855
  const QUERIES_FILE_CAVEAT = "CAVEAT: --queries-file floors are measured on human/gold-derived queries — section-weighted eval targets, NOT per-node passage samples — so this floor's low tail is a different distribution than the synthesized door's; record 'door: queries-file' beside the number and never compare the two doors' floors as interchangeable.";
5799
5856
  /**
5800
5857
  * The report dict, assembled from every scored query. `in_corpus_queries` /
@@ -5836,6 +5893,7 @@ function buildReport(detail, meta, targetPrecision = .95, now = /* @__PURE__ */
5836
5893
  paste_why,
5837
5894
  margin: pythonRound(marginOf(points), 4),
5838
5895
  separable,
5896
+ ooc_source: meta.oocSource,
5839
5897
  target: rec.target,
5840
5898
  measured_at: now.toISOString().slice(0, 10),
5841
5899
  low_tail,
@@ -5856,6 +5914,7 @@ function renderReport(report) {
5856
5914
  const gen = report.generation === null ? "unknown (no generation pinned)" : String(report.generation);
5857
5915
  lines.push(`\nmeasured on generation ${gen} (${how}), model ${report.model}, door: ${report.door}`);
5858
5916
  lines.push(report.door === "queries-file" ? QUERIES_FILE_CAVEAT : SYNTHESIZED_CAVEAT);
5917
+ if (report.ooc_source === "built-in") lines.push(BUILT_IN_OOC_CAVEAT);
5859
5918
  lines.push(`AURC = ${pythonFloatRepr(report.aurc)} (lower = better separation)`);
5860
5919
  lines.push(`separation margin: ${pythonFormatFixed(report.margin, 3)} (over ${report.in_corpus_queries} in-corpus / ${report.ooc_probes} out-of-corpus probes)`);
5861
5920
  if (z) lines.push(`zero-FA floor (never refuse a real question): ${pythonFormatFixed(z.floor, 3)} -> coverage ${pythonFormatFixed(z.coverage, 3)}, ooc leak ${pythonFormatFixed(z.risk, 3)}`);
@@ -6110,13 +6169,15 @@ async function runCalibration(pool, options) {
6110
6169
  }
6111
6170
  inQueries = normalizeQueries(synthesized);
6112
6171
  }
6172
+ const oocSource = options.oocProbes === void 0 ? "built-in" : "provided";
6113
6173
  const ooc = normalizeQueries(options.oocProbes ?? BUILT_IN_OOC);
6114
6174
  return buildReport([...await scoreQueries(pool, scope, options.provider, inQueries, true), ...await scoreQueries(pool, scope, options.provider, ooc, false)], {
6115
6175
  generation: counted.measured,
6116
6176
  pinned: generation !== null,
6117
6177
  model: options.provider.modelId,
6118
6178
  dim: options.provider.dim,
6119
- door
6179
+ door,
6180
+ oocSource
6120
6181
  }, options.targetPrecision ?? .95);
6121
6182
  }
6122
6183
  /**
@@ -8062,6 +8123,15 @@ async function drain(pending, io) {
8062
8123
  */
8063
8124
  const CHUNK_INSERT_PREFIX = "INSERT INTO chunks (tenant_id, generation, source_id, ordinal, content, chunk_hash, heading_path, heading_path_text, anchor, labels, embedding_status) VALUES ";
8064
8125
  const CHUNK_PARAMS = 10;
8126
+ /**
8127
+ * Dense character count — whitespace removed, exactly as the serving
8128
+ * predicate's `length(regexp_replace(c.content, '\s', '', 'g'))` computes it.
8129
+ * Written here rather than approximated so the ingest report and the SQL admit
8130
+ * the same chunks.
8131
+ */
8132
+ function denseLength(content) {
8133
+ return content.replace(/\s/g, "").length;
8134
+ }
8065
8135
  /** 500 rows × 10 params stays far under Postgres's 65535 bind-parameter cap. */
8066
8136
  const CHUNK_ROWS_PER_STATEMENT = 500;
8067
8137
  /**
@@ -8106,6 +8176,8 @@ async function buildStructure(client, opts) {
8106
8176
  const treeRoot = resolve(opts.treeRoot);
8107
8177
  let nSources = 0;
8108
8178
  let nChunks = 0;
8179
+ let nUnsearchable = 0;
8180
+ const unsearchableSources = [];
8109
8181
  const chunkRows = [];
8110
8182
  for (const f of manifest.files) {
8111
8183
  const src = opts.files.get(f.path);
@@ -8129,6 +8201,7 @@ async function buildStructure(client, opts) {
8129
8201
  manifest.source_commit
8130
8202
  ]);
8131
8203
  nSources += 1;
8204
+ let sourceServable = 0;
8132
8205
  for (const chunk of chunkText(body)) {
8133
8206
  chunkRows.push([
8134
8207
  tenantId,
@@ -8143,7 +8216,10 @@ async function buildStructure(client, opts) {
8143
8216
  JSON.stringify({ source_type: chunk.sourceType })
8144
8217
  ]);
8145
8218
  nChunks += 1;
8219
+ if (chunk.sourceType === "prose" && denseLength(chunk.content) >= 24) sourceServable += 1;
8220
+ else nUnsearchable += 1;
8146
8221
  }
8222
+ if (sourceServable === 0 && nChunks > 0) unsearchableSources.push(sid);
8147
8223
  }
8148
8224
  for (let i = 0; i < chunkRows.length; i += CHUNK_ROWS_PER_STATEMENT) {
8149
8225
  const slice = chunkRows.slice(i, i + CHUNK_ROWS_PER_STATEMENT);
@@ -8182,7 +8258,9 @@ async function buildStructure(client, opts) {
8182
8258
  sources: nSources,
8183
8259
  chunks: nChunks,
8184
8260
  carried,
8185
- pending: health.pending
8261
+ pending: health.pending,
8262
+ unsearchable: nUnsearchable,
8263
+ unsearchableSources
8186
8264
  };
8187
8265
  }
8188
8266
  /**
@@ -8365,6 +8443,8 @@ async function buildGeneration(pool, instance, options) {
8365
8443
  ready: true,
8366
8444
  centroids: 0,
8367
8445
  flipped: false,
8446
+ unsearchable: 0,
8447
+ unsearchableSources: [],
8368
8448
  refusal: null,
8369
8449
  health: {
8370
8450
  ok: true,
@@ -8469,6 +8549,8 @@ async function buildGeneration(pool, instance, options) {
8469
8549
  ready: fin.ready,
8470
8550
  centroids: fin.centroids,
8471
8551
  flipped: fin.flipped,
8552
+ unsearchable: stats.unsearchable,
8553
+ unsearchableSources: stats.unsearchableSources,
8472
8554
  refusal: fin.refusal,
8473
8555
  health: fin.health,
8474
8556
  unchanged: false
@@ -8608,19 +8690,46 @@ function composeProvider(instance) {
8608
8690
  return fail$1(REFUSED, `instance embedding.provider: ${exc instanceof Error ? exc.message : String(exc)}`);
8609
8691
  }
8610
8692
  }
8611
- /**
8612
- * The commit the corpus was ingested from, resolved from git when the tree is
8613
- * in a repository.
8614
- *
8615
- * `--source-commit` has always existed and the golden path never passed it, so
8616
- * EVERY generation an adopter produced recorded the literal string
8617
- * "unspecified" — product principle 6 requires a build to record the exact
8618
- * corpus that produced it, and a placeholder records nothing (review
8619
- * 2026-08-20). Resolved here rather than in the scaffold script so it is right
8620
- * however the verb is invoked. A tree that is not a repository, or a git that
8621
- * is not installed, still records the honest sentinel rather than failing an
8622
- * ingest over provenance metadata.
8623
- */
8693
+ function provenanceGap(knowledgeDir) {
8694
+ if (knowledgeDir === void 0) return "not-asked";
8695
+ const run = (args) => {
8696
+ try {
8697
+ return {
8698
+ ok: true,
8699
+ out: execFileSync("git", [
8700
+ "-C",
8701
+ knowledgeDir,
8702
+ ...args
8703
+ ], {
8704
+ encoding: "utf8",
8705
+ stdio: [
8706
+ "ignore",
8707
+ "pipe",
8708
+ "ignore"
8709
+ ]
8710
+ }).trim()
8711
+ };
8712
+ } catch {
8713
+ return {
8714
+ ok: false,
8715
+ out: ""
8716
+ };
8717
+ }
8718
+ };
8719
+ if (!run(["--version"]).ok && !run(["rev-parse", "--git-dir"]).ok) return "no-git";
8720
+ if (!run(["rev-parse", "--git-dir"]).ok) return "no-repo";
8721
+ return "no-commit";
8722
+ }
8723
+ /** The remedy for each, because the reader's next command differs. */
8724
+ function provenanceNotice(gap) {
8725
+ const why = "so this generation cannot be traced back to a reviewed commit";
8726
+ switch (gap) {
8727
+ case "no-commit": return `source: unspecified — knowledge/ is in a git repository with no commits yet, ${why}.\n fix: commit the record (git add knowledge && git commit) and re-run`;
8728
+ case "no-repo": return `source: unspecified — knowledge/ is not in a git repository, ${why}.\n fix: git init, commit the record, and re-run`;
8729
+ case "no-git": return `source: unspecified — git is not installed, ${why}.\n fix: install git, or pass --source-commit <sha> if the record is versioned elsewhere`;
8730
+ case "not-asked": return `source: unspecified — no knowledge directory was given, ${why}.`;
8731
+ }
8732
+ }
8624
8733
  function detectSourceCommit(knowledgeDir) {
8625
8734
  if (knowledgeDir === void 0) return "unspecified";
8626
8735
  try {
@@ -8806,8 +8915,18 @@ async function ingestCommand(args) {
8806
8915
  process.stdout.write(`ingest: unchanged — generation ${report.generation} already serves this corpus\n`);
8807
8916
  return 0;
8808
8917
  }
8809
- process.stdout.write(sourceCommit === "unspecified" ? "source: unspecified — knowledge/ is not in a git repository, so this generation cannot be traced back to a reviewed commit\n" : `source: ${sourceCommit}\n`);
8918
+ process.stdout.write(sourceCommit === "unspecified" ? provenanceNotice(provenanceGap(values.knowledge)) + "\n" : `source: ${sourceCommit}\n`);
8810
8919
  process.stdout.write(`ingest: generation ${report.generation} — ${report.nodes} nodes, ${report.chunks} chunks; embedded ${report.embedded}, carried ${report.carried}, failed ${report.failed}\n`);
8920
+ if (report.unsearchable > 0) {
8921
+ const pct = Math.round(report.unsearchable / Math.max(report.chunks, 1) * 100);
8922
+ process.stdout.write(` not searchable: ${report.unsearchable} of ${report.chunks} chunk(s) (${pct}%) are shorter than the navigation threshold — stored and readable, but no search returns them\n`);
8923
+ if (report.unsearchableSources.length > 0) {
8924
+ const named = report.unsearchableSources.slice(0, 10).join(", ");
8925
+ const more = report.unsearchableSources.length - Math.min(10, report.unsearchableSources.length);
8926
+ process.stdout.write(` FOUND ONLY BY NAME: ${named}${more > 0 ? `, and ${more} more` : ""} — no searchable chunk at all; lengthen these sections or read them by slug
8927
+ `);
8928
+ }
8929
+ }
8811
8930
  if (report.refusal !== null) return fail$1(REFUSED, report.refusal);
8812
8931
  const governance = await withPool(dsn, (pool) => assertGovernanceServable(pool, instance, report.generation).then(() => null, (error) => error instanceof Error ? error.message : String(error)));
8813
8932
  if (governance !== null) return fail$1(REFUSED, `generation ${report.generation} was built and NOT activated — no surface could serve it\n ${governance.split("\n").join("\n ")}\n note: generation ${report.generation} is left behind, un-activated; \`ksor gc\` reaps it once the grace window passes. The previously active generation still serves.`);
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@panaversity/ksor",
3
- "version": "0.0.12",
3
+ "version": "0.0.14",
4
4
  "description": "Knowledge System of Record — compile governed markdown into a static site for people and an MCP server for AI agents, with citations and measured abstention.",
5
5
  "keywords": [
6
6
  "abstention",
@@ -89,12 +89,38 @@ function readInstanceScope(): string | null {
89
89
  for (const para of afterHeading.split(/\n[ \t]*\n/)) {
90
90
  const one = para.trim().replace(/\s+/g, " ");
91
91
  if (one === "" || one.startsWith("#") || one.startsWith("-") || one.startsWith(">")) continue;
92
- const sentence = /^(.+?[.!?])(\s|$)/.exec(one)?.[1] ?? one;
93
- return sentence.length > 300 ? `${sentence.slice(0, 297)}...` : sentence;
92
+ return sentence(one);
94
93
  }
95
94
  return null;
96
95
  }
97
96
 
97
+ /**
98
+ * The MCP registry schema caps `ServerDetail.description` at **100 characters**
99
+ * (`ServerDetail.description.maxLength`, 2025-12-11). Over it, the document a
100
+ * validating client reads is invalid — and the failure was shaped exactly
101
+ * wrong: the unfilled placeholder is 88 characters and validates, so the
102
+ * document became invalid the moment an owner did the thing the scaffold asks
103
+ * for and wrote a real scope sentence. Silent, and only for real records.
104
+ *
105
+ * A hard truncation would publish a sentence cut mid-word, so this trims at a
106
+ * word boundary and marks it, and the whole description is assembled inside the
107
+ * budget rather than clipped after the fact.
108
+ */
109
+ const DESCRIPTION_MAX = 100;
110
+
111
+ function sentence(one: string): string {
112
+ const first = /^(.+?[.!?])(\s|$)/.exec(one)?.[1] ?? one;
113
+ return first;
114
+ }
115
+
116
+ /** Fit `text` inside `max`, breaking on a word rather than mid-word. */
117
+ function fit(text: string, max: number): string {
118
+ if (text.length <= max) return text;
119
+ const cut = text.slice(0, max - 1);
120
+ const at = cut.lastIndexOf(" ");
121
+ return `${(at > max / 2 ? cut.slice(0, at) : cut).replace(/[,;:.\s]+$/, "")}\u2026`;
122
+ }
123
+
98
124
  /** null until the owner has written one — never a guess. */
99
125
  export const appScope: string | null = readInstanceScope();
100
126
 
@@ -103,9 +129,11 @@ export const appScope: string | null = readInstanceScope();
103
129
  * registry document and anything else that needs one cannot drift apart.
104
130
  */
105
131
  export function recordDescription(): string {
106
- return appScope === null
107
- ? `${appTitle} — its owner has not yet described what this record covers.`
108
- : `${appTitle} — ${appScope}`;
132
+ const whole =
133
+ appScope === null
134
+ ? `${appTitle} — its owner has not yet described what this record covers.`
135
+ : `${appTitle} — ${appScope}`;
136
+ return fit(whole, DESCRIPTION_MAX);
109
137
  }
110
138
 
111
139
  /**