@panaversity/ksor 0.0.12 → 0.0.13

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,98 @@
1
1
  # @panaversity/ksor
2
2
 
3
+ ## 0.0.13
4
+
5
+ ### Patch Changes
6
+
7
+ - 8c5013b: A provider outage is never reported as "the record does not cover this"
8
+
9
+ When the embedding provider is down, the vector arm does not run — so an empty
10
+ result says nothing about coverage. It says we could not look. That distinction
11
+ was fixed once for records with a calibrated floor, and the condition was the
12
+ bug: it left the case out that matters most.
13
+
14
+ An **uncalibrated** record is the default state of every fresh scaffold. There
15
+ the emptiness came from the keyword arm, which abstains when it returns no rows
16
+ — and it returns nothing for almost every natural-language question, because
17
+ `websearch_to_tsquery` ANDs its terms (measured 12 of 12 on real questions). So
18
+ during any outage an uncalibrated record answered every question with
19
+ `abstained: true`, while the tool description instructs the agent to state that
20
+ as fact and never fall back on its own knowledge.
21
+
22
+ It reached this release because the existing test asked a question the keyword
23
+ arm could answer, so the degraded path served real hits and looked correct. Ask
24
+ the way a person asks and it did not. That case is now covered.
25
+
26
+ Found live against the published 0.0.12 with an invalid key — the same state a
27
+ rejected CI key had produced that morning, which is how likely this is.
28
+
29
+ **`ksor calibrate` also stops blessing a floor on far-domain evidence alone.**
30
+ The built-in out-of-corpus probes are all far-domain — dinner, taxes, boiling an
31
+ egg — and a shipped set cannot be scope-adjacent, because adjacency depends on a
32
+ corpus it has never seen. Far-domain probes score low against anything, so the
33
+ margin comes out inflated. Measured on one record, changing only the probe set:
34
+ built-ins reported "separable, margin 0.072" and recommended a floor; eight
35
+ scope-adjacent near-misses reported "NOT separable, margin -0.030" — and that
36
+ floor then answered six of the eight live, with citations. The tool already knew
37
+ to say "widen the probe set", but said it only on the not-separable branch, which
38
+ is when it is least needed. It now says it whenever the built-ins are used.
39
+
40
+ - 692d296: `ksor ingest` says how much of the record no search will return
41
+
42
+ A chunk shorter than the navigation threshold is stored, embedded and readable —
43
+ and excluded from every retrieval arm by the serving predicate. That rule exists
44
+ for a good reason: a "See also: [a] [b] [c]" block should never be a search hit.
45
+ But it decides by LENGTH ALONE, so a short _substantive_ paragraph is caught by
46
+ it too — and a policy handbook is made of short substantive statements.
47
+
48
+ Measured on a realistic five-document operations handbook with real embeddings:
49
+ **10 of 16 chunks unsearchable, and one entire document that `outline` lists and
50
+ `read` returns in full but `search` can never find.** A complete policy
51
+ statement — "Probation: six months, with a written review at three and six" —
52
+ is 191 characters, so the record treats it as navigation. The ingest line
53
+ reported a cheerful `16 chunks; embedded 16` and said nothing.
54
+
55
+ It says it now:
56
+
57
+ ```
58
+ ingest: generation 1 — 2 nodes, 4 chunks; embedded 4, carried 0, failed 0
59
+ not searchable: 3 of 4 chunk(s) (75%) are shorter than the navigation
60
+ threshold — stored and readable, but no search returns them
61
+ FOUND ONLY BY NAME: knowledge/onboarding:prose — no searchable chunk at all
62
+ ```
63
+
64
+ This does **not** change the threshold, and nothing that was searchable stops
65
+ being so. Where that line belongs needs a gold-set measurement, which is issue
66
+ #55. What is fixed here is the silence — because the silence is what let a
67
+ record ship most of itself unfindable, and told its owner everything was fine.
68
+
69
+ The count is computed with the serving predicate's own admission test, and a db
70
+ test compares it against what the SQL actually admits: a report the database
71
+ disagrees with would be worse than none.
72
+
73
+ - c4976dd: Two defects introduced in 0.0.12, found by verifying the published package live
74
+
75
+ **The discovery document became invalid exactly when a record became real.** The
76
+ MCP registry schema caps `ServerDetail.description` at **100 characters**
77
+ (2025-12-11). 0.0.12 started generating that description from the record's own
78
+ prose — a real improvement over the hard-coded sentence it replaced — and capped
79
+ it at 300. The unfilled placeholder is 88 characters and validates; a described
80
+ record's title plus scope sentence is routinely 150-350 and does not. So
81
+ `/.well-known/mcp/server.json` passed validation until the owner did the thing
82
+ the scaffold asks for, then silently stopped, with nothing in the build to say
83
+ so. It is now assembled inside the schema's budget and trimmed at a word
84
+ boundary rather than mid-word.
85
+
86
+ **The boot report reassured the operator in the one configuration that needs a
87
+ warning.** `KSOR_ALLOW_PUBLIC_UNAUTHENTICATED=1` permits an unauthenticated
88
+ public bind, and the auth line kept printing `DISABLED — 0.0.0.0 only, and a
89
+ public bind will refuse to boot` — false on both counts, at the moment the whole
90
+ record is being served to anyone who can reach the port. It now says that,
91
+ naming the variable responsible.
92
+
93
+ Both shipped in 0.0.12 and both were mine; the aligned boot report and the
94
+ self-describing discovery document are otherwise unchanged.
95
+
3
96
  ## 0.0.12
4
97
 
5
98
  ### Patch Changes
package/dist/cli.mjs CHANGED
@@ -16,7 +16,7 @@ import { bodyLimit } from "hono/body-limit";
16
16
  import { execFileSync, spawnSync } from "node:child_process";
17
17
  import { parseArgs } from "node:util";
18
18
  import { readFile, readdir, stat } from "node:fs/promises";
19
- //#region ../content-gateway/dist/main-CqOtG-q1.mjs
19
+ //#region ../content-gateway/dist/main-B1G1VdMp.mjs
20
20
  /**
21
21
  * A connection could not be ESTABLISHED in time — retryable.
22
22
  *
@@ -1114,11 +1114,24 @@ g AS (
1114
1114
  WHERE tenant_id = $1 AND corpus_id = $2)
1115
1115
  ) AS gen
1116
1116
  )`;
1117
- const ARM_WHERE$1 = `
1117
+ /**
1118
+ * The arm predicate, built for the parameter numbering of the query that uses
1119
+ * it — a FUNCTION, not a string the caller renumbers afterwards.
1120
+ *
1121
+ * It used to be derived with `ARM_WHERE.replaceAll("$5", "$4")`, which works
1122
+ * only while the predicate happens to contain exactly one placeholder and no
1123
+ * other text matching it. Adding any second parameter to the predicate breaks
1124
+ * every derived query silently — and when it breaks, the failure arrives as a
1125
+ * driver error that the serving layer correctly reduces to "content store
1126
+ * temporarily unavailable", which tells you nothing about the cause. Found by
1127
+ * tripping over it while trying a fix for issue #59; taking the number as an
1128
+ * argument makes the coupling visible instead of textual.
1129
+ */
1130
+ const armWhere$1 = (kindsParam) => `
1118
1131
  c.tenant_id = $1 AND c.generation = g.gen
1119
1132
  AND c.embedding_status = 'embedded' AND ${SERVABLE$1}
1120
1133
  AND n.status = 'published'
1121
- AND ($5::text[] IS NULL OR n.kind = ANY($5::text[]))
1134
+ AND (${kindsParam}::text[] IS NULL OR n.kind = ANY(${kindsParam}::text[]))
1122
1135
  AND ${DENY$1}
1123
1136
  AND ${AUDIENCE_ALLOWED$1}`;
1124
1137
  const JOINS$1 = `
@@ -1127,6 +1140,15 @@ const JOINS$1 = `
1127
1140
  JOIN sources s ON s.source_id = c.source_id AND s.tenant_id = c.tenant_id
1128
1141
  AND s.generation = c.generation
1129
1142
  JOIN content_nodes n ON n.node_id = s.node_id AND n.tenant_id = s.tenant_id`;
1143
+ /**
1144
+ * Exported ONLY so a test can EXPLAIN the real thing.
1145
+ *
1146
+ * The index regression this fixes was announced as fixed once before, and came
1147
+ * back through a different clause, because nothing ever asserted the RESULT —
1148
+ * that the plan opens `idx_chunks_hnsw`. A timing threshold would be flaky and
1149
+ * would not have caught it either; the plan shape is the property that matters
1150
+ * (issue #59).
1151
+ */
1130
1152
  const HYBRID_SQL$1 = `
1131
1153
  WITH RECURSIVE ${GEN_CTE$1}, ${DENIED_CTE$1},
1132
1154
  -- The top-k is taken by a PLAIN \`ORDER BY <distance> LIMIT\`, and the rank
@@ -1147,7 +1169,7 @@ WITH RECURSIVE ${GEN_CTE$1}, ${DENIED_CTE$1},
1147
1169
  FROM (
1148
1170
  SELECT c.chunk_id, g.gen, (c.embedding <=> $3::vector) AS dist
1149
1171
  ${JOINS$1}
1150
- WHERE ${ARM_WHERE$1}
1172
+ WHERE ${armWhere$1("$5")}
1151
1173
  ORDER BY c.embedding <=> $3::vector, c.chunk_id
1152
1174
  LIMIT $6
1153
1175
  ) ranked),
@@ -1156,7 +1178,7 @@ WITH RECURSIVE ${GEN_CTE$1}, ${DENIED_CTE$1},
1156
1178
  row_number() OVER (ORDER BY ts_rank_cd(c.search_tsv,
1157
1179
  websearch_to_tsquery($9::regconfig, $4)) DESC, c.chunk_id) AS r
1158
1180
  ${JOINS$1}
1159
- WHERE ${ARM_WHERE$1}
1181
+ WHERE ${armWhere$1("$5")}
1160
1182
  AND c.search_tsv @@ websearch_to_tsquery($9::regconfig, $4)
1161
1183
  ORDER BY r LIMIT $6),
1162
1184
  fused AS (
@@ -1179,10 +1201,10 @@ WITH RECURSIVE ${GEN_CTE$1.replace("$8", "$6")}, ${DENIED_CTE$1}
1179
1201
  ts_rank_cd(c.search_tsv, websearch_to_tsquery($7::regconfig, $3)) AS score,
1180
1202
  g.gen, n.permalink
1181
1203
  ${JOINS$1}
1182
- WHERE ${ARM_WHERE$1.replaceAll("$5", "$4")}
1204
+ WHERE ${armWhere$1("$4")}
1183
1205
  AND c.search_tsv @@ websearch_to_tsquery($7::regconfig, $3)
1184
1206
  ORDER BY score DESC, c.chunk_id LIMIT $5`;
1185
- `${GEN_CTE$1.replace("$8", "$5")}${DENIED_CTE$1}${JOINS$1}${ARM_WHERE$1.replaceAll("$5", "$4")}`;
1207
+ `${GEN_CTE$1.replace("$8", "$5")}${DENIED_CTE$1}${JOINS$1}${armWhere$1("$4")}`;
1186
1208
  const HIT_COLUMNS = 9;
1187
1209
  /** Serialize a query vector as a pgvector literal. */
1188
1210
  function vectorLiteral$1(vector) {
@@ -2604,7 +2626,7 @@ async function search(ctx, query, k = 10) {
2604
2626
  const r = await client.query("SELECT active_generation FROM corpora WHERE tenant_id = $1 AND corpus_id = $2", [inst.tenantId, inst.corpusId]);
2605
2627
  return Number(r.rows[0]?.active_generation ?? 0) === 0;
2606
2628
  }, audienceScope(ctx));
2607
- const reason = embedFailed && inst.abstain.vectorFloor !== null ? "unavailable" : unpublished ? "unpublished" : "abstained";
2629
+ const reason = embedFailed ? "unavailable" : unpublished ? "unpublished" : "abstained";
2608
2630
  return {
2609
2631
  ok: false,
2610
2632
  abstained: reason === "abstained",
@@ -3595,9 +3617,10 @@ function withoutSdkResponseModeWarning(body) {
3595
3617
  * it survivable (`buildAuth` refuses a non-loopback bind without auth, so the
3596
3618
  * only way to read this line is on a host that cannot be reached from outside).
3597
3619
  */
3598
- function authPosture(mode, host) {
3599
- if (mode === "disabled") return `DISABLED — ${host} only, and a public bind will refuse to boot`;
3600
- return "bearer tokens, verified against the record's authorization server";
3620
+ function authPosture(mode, host, publicUnauthenticated) {
3621
+ if (mode !== "disabled") return "bearer tokens, verified against the record's authorization server";
3622
+ if (publicUnauthenticated) return `UNAUTHENTICATED and bound to ${host} — KSOR_ALLOW_PUBLIC_UNAUTHENTICATED=1 is set, so the whole record is served to anyone who can reach this port`;
3623
+ return `DISABLED — ${host} only, and a public bind will refuse to boot`;
3601
3624
  }
3602
3625
  /**
3603
3626
  * What the record will refuse. `null` means no floor is declared and the gate
@@ -4021,7 +4044,7 @@ async function runHttp(composition) {
4021
4044
  s.once("error", reject);
4022
4045
  });
4023
4046
  if (recordIsUndescribed(instance.instructions)) console.error(bootLine("identity", UNDESCRIBED_RECORD));
4024
- console.error(bootLine("auth", authPosture(auth.mode, bind.host)));
4047
+ console.error(bootLine("auth", authPosture(auth.mode, bind.host, process.env["KSOR_ALLOW_PUBLIC_UNAUTHENTICATED"] === "1" && !loopback)));
4025
4048
  for (const line of keyLines) console.error(line);
4026
4049
  console.error(bootLine("abstain", abstainPosture(instance.abstain.vectorFloor)));
4027
4050
  console.error(bootLine("serving", `http://${bind.host}:${bind.port}/mcp`));
@@ -4431,7 +4454,7 @@ async function withPgRetry(op, options = {}) {
4431
4454
  throw lastError;
4432
4455
  }
4433
4456
  //#endregion
4434
- //#region ../content/dist/commands-Cysnkk_R.mjs
4457
+ //#region ../content/dist/commands-LOfF_iLT.mjs
4435
4458
  /**
4436
4459
  * EVAL-LOCKED constants, quarried verbatim from the oracle
4437
4460
  * (sor-agentfactory @ b554f91, config.py) — changing any of these is a
@@ -5193,11 +5216,24 @@ g AS (
5193
5216
  WHERE tenant_id = $1 AND corpus_id = $2)
5194
5217
  ) AS gen
5195
5218
  )`;
5196
- const ARM_WHERE = `
5219
+ /**
5220
+ * The arm predicate, built for the parameter numbering of the query that uses
5221
+ * it — a FUNCTION, not a string the caller renumbers afterwards.
5222
+ *
5223
+ * It used to be derived with `ARM_WHERE.replaceAll("$5", "$4")`, which works
5224
+ * only while the predicate happens to contain exactly one placeholder and no
5225
+ * other text matching it. Adding any second parameter to the predicate breaks
5226
+ * every derived query silently — and when it breaks, the failure arrives as a
5227
+ * driver error that the serving layer correctly reduces to "content store
5228
+ * temporarily unavailable", which tells you nothing about the cause. Found by
5229
+ * tripping over it while trying a fix for issue #59; taking the number as an
5230
+ * argument makes the coupling visible instead of textual.
5231
+ */
5232
+ const armWhere = (kindsParam) => `
5197
5233
  c.tenant_id = $1 AND c.generation = g.gen
5198
5234
  AND c.embedding_status = 'embedded' AND ${SERVABLE}
5199
5235
  AND n.status = 'published'
5200
- AND ($5::text[] IS NULL OR n.kind = ANY($5::text[]))
5236
+ AND (${kindsParam}::text[] IS NULL OR n.kind = ANY(${kindsParam}::text[]))
5201
5237
  AND ${DENY}
5202
5238
  AND ${AUDIENCE_ALLOWED}`;
5203
5239
  const JOINS = `
@@ -5206,14 +5242,14 @@ const JOINS = `
5206
5242
  JOIN sources s ON s.source_id = c.source_id AND s.tenant_id = c.tenant_id
5207
5243
  AND s.generation = c.generation
5208
5244
  JOIN content_nodes n ON n.node_id = s.node_id AND n.tenant_id = s.tenant_id`;
5209
- `${GEN_CTE}${DENIED_CTE}${JOINS}${ARM_WHERE}${JOINS}${ARM_WHERE}`;
5210
- `${GEN_CTE.replace("$8", "$6")}${DENIED_CTE}${JOINS}${ARM_WHERE.replaceAll("$5", "$4")}`;
5245
+ `${GEN_CTE}${DENIED_CTE}${JOINS}${armWhere("$5")}${JOINS}${armWhere("$5")}`;
5246
+ `${GEN_CTE.replace("$8", "$6")}${DENIED_CTE}${JOINS}${armWhere("$4")}`;
5211
5247
  /** The calibrator's standalone top-1 signal (the read path gets it free from HYBRID_SQL). */
5212
5248
  const TOP_ONE_SQL = `
5213
5249
  WITH RECURSIVE ${GEN_CTE.replace("$8", "$5")}, ${DENIED_CTE}
5214
5250
  SELECT 1 - (c.embedding <=> $3::vector) AS score
5215
5251
  ${JOINS}
5216
- WHERE ${ARM_WHERE.replaceAll("$5", "$4")}
5252
+ WHERE ${armWhere("$4")}
5217
5253
  ORDER BY c.embedding <=> $3::vector, c.chunk_id
5218
5254
  LIMIT 1`;
5219
5255
  /** Serialize a query vector as a pgvector literal. */
@@ -5795,6 +5831,29 @@ const BUILT_IN_OOC = [
5795
5831
  * measurement had an easier question set than production would.
5796
5832
  */
5797
5833
  const SYNTHESIZED_CAVEAT = "CAVEAT: synthesized queries are written FROM the passages they are then scored against, so they share vocabulary a reader's question will not. This door measures an UPPER BOUND on separation — treat the floor below as provisional until it has been checked against questions the corpus did not write (--queries-file), and re-run if real questions score under it.";
5834
+ /**
5835
+ * What a separability verdict is worth when the probes came from the binary.
5836
+ *
5837
+ * Every entry in BUILT_IN_OOC is far-domain — dinner, taxes, football, boiling
5838
+ * an egg. Those score low against ANY corpus, so max-OOC comes out artificially
5839
+ * low and the margin is inflated from that end, exactly as synthesized in-corpus
5840
+ * queries inflate it from the other. Measured on one record, changing ONLY the
5841
+ * probe set: built-ins reported "separable, margin 0.072" and recommended a
5842
+ * floor; eight scope-adjacent near-misses on the same corpus and the same
5843
+ * in-corpus questions reported "NOT separable, margin -0.030". The recommended
5844
+ * floor then answered six of those eight near-misses live, with citations
5845
+ * (2026-08-21).
5846
+ *
5847
+ * The tool already knows this — the not-separable branch tells the operator to
5848
+ * "widen the probe set (scope-adjacent near-misses, not only far-domain
5849
+ * questions)". It said so only AFTER weak probes had failed to bless a floor,
5850
+ * which is the one case where the advice is least needed.
5851
+ *
5852
+ * A shipped set cannot be scope-adjacent, because adjacency depends on a corpus
5853
+ * the binary has never seen. So this is stated whenever built-ins are used, on
5854
+ * BOTH branches, rather than pretending a better default exists.
5855
+ */
5856
+ const BUILT_IN_OOC_CAVEAT = "CAVEAT: the out-of-corpus probes are the BUILT-IN set, which is entirely far-domain — a shipped set cannot be scope-adjacent, because adjacency depends on a corpus it has never seen. Far-domain probes score low against anything, so this margin is an OVER-estimate and a floor it blesses may still answer near-misses just outside your scope. Re-run with --ooc-file naming questions a reader might plausibly ask that this record does NOT cover, and trust that verdict over this one.";
5798
5857
  const QUERIES_FILE_CAVEAT = "CAVEAT: --queries-file floors are measured on human/gold-derived queries — section-weighted eval targets, NOT per-node passage samples — so this floor's low tail is a different distribution than the synthesized door's; record 'door: queries-file' beside the number and never compare the two doors' floors as interchangeable.";
5799
5858
  /**
5800
5859
  * The report dict, assembled from every scored query. `in_corpus_queries` /
@@ -5836,6 +5895,7 @@ function buildReport(detail, meta, targetPrecision = .95, now = /* @__PURE__ */
5836
5895
  paste_why,
5837
5896
  margin: pythonRound(marginOf(points), 4),
5838
5897
  separable,
5898
+ ooc_source: meta.oocSource,
5839
5899
  target: rec.target,
5840
5900
  measured_at: now.toISOString().slice(0, 10),
5841
5901
  low_tail,
@@ -5856,6 +5916,7 @@ function renderReport(report) {
5856
5916
  const gen = report.generation === null ? "unknown (no generation pinned)" : String(report.generation);
5857
5917
  lines.push(`\nmeasured on generation ${gen} (${how}), model ${report.model}, door: ${report.door}`);
5858
5918
  lines.push(report.door === "queries-file" ? QUERIES_FILE_CAVEAT : SYNTHESIZED_CAVEAT);
5919
+ if (report.ooc_source === "built-in") lines.push(BUILT_IN_OOC_CAVEAT);
5859
5920
  lines.push(`AURC = ${pythonFloatRepr(report.aurc)} (lower = better separation)`);
5860
5921
  lines.push(`separation margin: ${pythonFormatFixed(report.margin, 3)} (over ${report.in_corpus_queries} in-corpus / ${report.ooc_probes} out-of-corpus probes)`);
5861
5922
  if (z) lines.push(`zero-FA floor (never refuse a real question): ${pythonFormatFixed(z.floor, 3)} -> coverage ${pythonFormatFixed(z.coverage, 3)}, ooc leak ${pythonFormatFixed(z.risk, 3)}`);
@@ -6110,13 +6171,15 @@ async function runCalibration(pool, options) {
6110
6171
  }
6111
6172
  inQueries = normalizeQueries(synthesized);
6112
6173
  }
6174
+ const oocSource = options.oocProbes === void 0 ? "built-in" : "provided";
6113
6175
  const ooc = normalizeQueries(options.oocProbes ?? BUILT_IN_OOC);
6114
6176
  return buildReport([...await scoreQueries(pool, scope, options.provider, inQueries, true), ...await scoreQueries(pool, scope, options.provider, ooc, false)], {
6115
6177
  generation: counted.measured,
6116
6178
  pinned: generation !== null,
6117
6179
  model: options.provider.modelId,
6118
6180
  dim: options.provider.dim,
6119
- door
6181
+ door,
6182
+ oocSource
6120
6183
  }, options.targetPrecision ?? .95);
6121
6184
  }
6122
6185
  /**
@@ -8062,6 +8125,15 @@ async function drain(pending, io) {
8062
8125
  */
8063
8126
  const CHUNK_INSERT_PREFIX = "INSERT INTO chunks (tenant_id, generation, source_id, ordinal, content, chunk_hash, heading_path, heading_path_text, anchor, labels, embedding_status) VALUES ";
8064
8127
  const CHUNK_PARAMS = 10;
8128
+ /**
8129
+ * Dense character count — whitespace removed, exactly as the serving
8130
+ * predicate's `length(regexp_replace(c.content, '\s', '', 'g'))` computes it.
8131
+ * Written here rather than approximated so the ingest report and the SQL admit
8132
+ * the same chunks.
8133
+ */
8134
+ function denseLength(content) {
8135
+ return content.replace(/\s/g, "").length;
8136
+ }
8065
8137
  /** 500 rows × 10 params stays far under Postgres's 65535 bind-parameter cap. */
8066
8138
  const CHUNK_ROWS_PER_STATEMENT = 500;
8067
8139
  /**
@@ -8106,6 +8178,8 @@ async function buildStructure(client, opts) {
8106
8178
  const treeRoot = resolve(opts.treeRoot);
8107
8179
  let nSources = 0;
8108
8180
  let nChunks = 0;
8181
+ let nUnsearchable = 0;
8182
+ const unsearchableSources = [];
8109
8183
  const chunkRows = [];
8110
8184
  for (const f of manifest.files) {
8111
8185
  const src = opts.files.get(f.path);
@@ -8129,6 +8203,7 @@ async function buildStructure(client, opts) {
8129
8203
  manifest.source_commit
8130
8204
  ]);
8131
8205
  nSources += 1;
8206
+ let sourceServable = 0;
8132
8207
  for (const chunk of chunkText(body)) {
8133
8208
  chunkRows.push([
8134
8209
  tenantId,
@@ -8143,7 +8218,10 @@ async function buildStructure(client, opts) {
8143
8218
  JSON.stringify({ source_type: chunk.sourceType })
8144
8219
  ]);
8145
8220
  nChunks += 1;
8221
+ if (chunk.sourceType === "prose" && denseLength(chunk.content) >= 24) sourceServable += 1;
8222
+ else nUnsearchable += 1;
8146
8223
  }
8224
+ if (sourceServable === 0 && nChunks > 0) unsearchableSources.push(sid);
8147
8225
  }
8148
8226
  for (let i = 0; i < chunkRows.length; i += CHUNK_ROWS_PER_STATEMENT) {
8149
8227
  const slice = chunkRows.slice(i, i + CHUNK_ROWS_PER_STATEMENT);
@@ -8182,7 +8260,9 @@ async function buildStructure(client, opts) {
8182
8260
  sources: nSources,
8183
8261
  chunks: nChunks,
8184
8262
  carried,
8185
- pending: health.pending
8263
+ pending: health.pending,
8264
+ unsearchable: nUnsearchable,
8265
+ unsearchableSources
8186
8266
  };
8187
8267
  }
8188
8268
  /**
@@ -8365,6 +8445,8 @@ async function buildGeneration(pool, instance, options) {
8365
8445
  ready: true,
8366
8446
  centroids: 0,
8367
8447
  flipped: false,
8448
+ unsearchable: 0,
8449
+ unsearchableSources: [],
8368
8450
  refusal: null,
8369
8451
  health: {
8370
8452
  ok: true,
@@ -8469,6 +8551,8 @@ async function buildGeneration(pool, instance, options) {
8469
8551
  ready: fin.ready,
8470
8552
  centroids: fin.centroids,
8471
8553
  flipped: fin.flipped,
8554
+ unsearchable: stats.unsearchable,
8555
+ unsearchableSources: stats.unsearchableSources,
8472
8556
  refusal: fin.refusal,
8473
8557
  health: fin.health,
8474
8558
  unchanged: false
@@ -8808,6 +8892,16 @@ async function ingestCommand(args) {
8808
8892
  }
8809
8893
  process.stdout.write(sourceCommit === "unspecified" ? "source: unspecified — knowledge/ is not in a git repository, so this generation cannot be traced back to a reviewed commit\n" : `source: ${sourceCommit}\n`);
8810
8894
  process.stdout.write(`ingest: generation ${report.generation} — ${report.nodes} nodes, ${report.chunks} chunks; embedded ${report.embedded}, carried ${report.carried}, failed ${report.failed}\n`);
8895
+ if (report.unsearchable > 0) {
8896
+ const pct = Math.round(report.unsearchable / Math.max(report.chunks, 1) * 100);
8897
+ process.stdout.write(` not searchable: ${report.unsearchable} of ${report.chunks} chunk(s) (${pct}%) are shorter than the navigation threshold — stored and readable, but no search returns them\n`);
8898
+ if (report.unsearchableSources.length > 0) {
8899
+ const named = report.unsearchableSources.slice(0, 10).join(", ");
8900
+ const more = report.unsearchableSources.length - Math.min(10, report.unsearchableSources.length);
8901
+ process.stdout.write(` FOUND ONLY BY NAME: ${named}${more > 0 ? `, and ${more} more` : ""} — no searchable chunk at all; lengthen these sections or read them by slug
8902
+ `);
8903
+ }
8904
+ }
8811
8905
  if (report.refusal !== null) return fail$1(REFUSED, report.refusal);
8812
8906
  const governance = await withPool(dsn, (pool) => assertGovernanceServable(pool, instance, report.generation).then(() => null, (error) => error instanceof Error ? error.message : String(error)));
8813
8907
  if (governance !== null) return fail$1(REFUSED, `generation ${report.generation} was built and NOT activated — no surface could serve it\n ${governance.split("\n").join("\n ")}\n note: generation ${report.generation} is left behind, un-activated; \`ksor gc\` reaps it once the grace window passes. The previously active generation still serves.`);
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@panaversity/ksor",
3
- "version": "0.0.12",
3
+ "version": "0.0.13",
4
4
  "description": "Knowledge System of Record — compile governed markdown into a static site for people and an MCP server for AI agents, with citations and measured abstention.",
5
5
  "keywords": [
6
6
  "abstention",
@@ -89,12 +89,38 @@ function readInstanceScope(): string | null {
89
89
  for (const para of afterHeading.split(/\n[ \t]*\n/)) {
90
90
  const one = para.trim().replace(/\s+/g, " ");
91
91
  if (one === "" || one.startsWith("#") || one.startsWith("-") || one.startsWith(">")) continue;
92
- const sentence = /^(.+?[.!?])(\s|$)/.exec(one)?.[1] ?? one;
93
- return sentence.length > 300 ? `${sentence.slice(0, 297)}...` : sentence;
92
+ return sentence(one);
94
93
  }
95
94
  return null;
96
95
  }
97
96
 
97
+ /**
98
+ * The MCP registry schema caps `ServerDetail.description` at **100 characters**
99
+ * (`ServerDetail.description.maxLength`, 2025-12-11). Over it, the document a
100
+ * validating client reads is invalid — and the failure was shaped exactly
101
+ * wrong: the unfilled placeholder is 88 characters and validates, so the
102
+ * document became invalid the moment an owner did the thing the scaffold asks
103
+ * for and wrote a real scope sentence. Silent, and only for real records.
104
+ *
105
+ * A hard truncation would publish a sentence cut mid-word, so this trims at a
106
+ * word boundary and marks it, and the whole description is assembled inside the
107
+ * budget rather than clipped after the fact.
108
+ */
109
+ const DESCRIPTION_MAX = 100;
110
+
111
+ function sentence(one: string): string {
112
+ const first = /^(.+?[.!?])(\s|$)/.exec(one)?.[1] ?? one;
113
+ return first;
114
+ }
115
+
116
+ /** Fit `text` inside `max`, breaking on a word rather than mid-word. */
117
+ function fit(text: string, max: number): string {
118
+ if (text.length <= max) return text;
119
+ const cut = text.slice(0, max - 1);
120
+ const at = cut.lastIndexOf(" ");
121
+ return `${(at > max / 2 ? cut.slice(0, at) : cut).replace(/[,;:.\s]+$/, "")}\u2026`;
122
+ }
123
+
98
124
  /** null until the owner has written one — never a guess. */
99
125
  export const appScope: string | null = readInstanceScope();
100
126
 
@@ -103,9 +129,11 @@ export const appScope: string | null = readInstanceScope();
103
129
  * registry document and anything else that needs one cannot drift apart.
104
130
  */
105
131
  export function recordDescription(): string {
106
- return appScope === null
107
- ? `${appTitle} — its owner has not yet described what this record covers.`
108
- : `${appTitle} — ${appScope}`;
132
+ const whole =
133
+ appScope === null
134
+ ? `${appTitle} — its owner has not yet described what this record covers.`
135
+ : `${appTitle} — ${appScope}`;
136
+ return fit(whole, DESCRIPTION_MAX);
109
137
  }
110
138
 
111
139
  /**