klypix-mcp 1.95.0 → 1.97.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -235,13 +235,23 @@ not a filing convention.
235
235
  session's brief. That is brief priority, not a retrieval-ranking boost.
236
236
  - **You can ask what the project believed then.** `brain_ask` with `as_of: 2026-03-01` reweights
237
237
  ranking by card lifecycle dates, so corrections made later do not leak backwards.
238
- - **Retrieval is local.** Lexical by default. If the optional on-device model is installed,
239
- `brain_ask` and `search_all_brains` use BGE semantic ranking with lexical help for exact
240
- identifiers, paths, and versions — still entirely on your machine. The previous cross-encoder
241
- is available for experiments with `KLYPIX_RERANK=1`, but is off by default because it reduced
242
- precision and added latency on the frozen human-paraphrase evaluation. Without the embedding
243
- model, retrieval degrades cleanly to lexical. `npx klypix-mcp install` deliberately does not
244
- install that model, so a fresh install is lexical.
238
+ - **Retrieval is local.** With the on-device model ("search by meaning"), `brain_ask` and
239
+ `search_all_brains` use BGE semantic ranking with lexical help for exact identifiers, paths, and
240
+ versions — still entirely on your machine. Without it, retrieval is keyword-only. The previous
241
+ cross-encoder is available for experiments with `KLYPIX_RERANK=1`, but is off by default because
242
+ it reduced precision and added latency on the frozen human-paraphrase evaluation.
243
+ - **Search by meaning is installed for you (1.97).** `npx klypix-mcp install` run by a person in a
244
+ terminal installs it: about 150 MB to download once, about 110 MB on disk after the files Node
245
+ never loads are removed. Skip it with `--no-semantic` (or `KLYPIX_SEMANTIC=0`). Installs with no
246
+ terminal — CI, scripts, an AI agent running the install — get it only with `--semantic`. The
247
+ Claude plugin never downloads it. Measured on the frozen 102-question set: the right card in the
248
+ top 5 for 54 questions with it, 19 without.
249
+ - **Preparing a brain never makes a question wait.** Each card is turned into a meaning vector once
250
+ (about 45 ms a card on a laptop CPU, one card at a time, ~305 MB peak), newest cards first, by a
251
+ low-priority background job that saves as it goes. A question on a brain that is not ready yet
252
+ answers right away from what is ready plus keywords, and says how far along the preparation is
253
+ ("Getting this brain ready for search by meaning: 900 of 3,670 cards so far, about 5 min left").
254
+ After that, only new or edited cards are prepared.
245
255
 
246
256
  ### Bounded semantic-memory runtime
247
257
 
@@ -1143,6 +1153,15 @@ independently validated.
1143
1153
  0.43, top-1 33%. The stratum that matters most is the honest one: **paraphrase questions that share
1144
1154
  no words with their card reach recall@5 44%**; status, temporal and multi-hop questions sit at
1145
1155
  81–93% because they still share vocabulary. Lexical-only scores 0% on the same set.
1156
+ - **The caller's words (`brain_ask`, 1.96).** A question can end with one more line,
1157
+ `Also: <words a card answering it would likely use>`; the calling agent is itself a model, so it
1158
+ can guess the card's wording at no extra cost. The words join the meaning query only. Measured
1159
+ 2026-10-11 on the same brain grown to 3,829 cards (102 of the frozen questions still answerable),
1160
+ with words written by Gemini 2.5 Flash as a stand-in for the calling agent: recall@5 54% → 59%
1161
+ (8 found, 3 lost), recall@8 61% → 68%, recall@20 68% → 76%, MRR 0.375 → 0.424. With no local
1162
+ model (a fresh install, the Claude plugin) the words count as extra keywords at half weight:
1163
+ recall@5 19% → 24% (5 found, none lost), recall@10 25% → 31%. Without the line, ranking is
1164
+ unchanged byte for byte.
1146
1165
  - **Second brain.** The same ranker on a different project's brain (1,601 cards, 19 hand-written
1147
1166
  paraphrases, first cross-brain run): recall@5 **42%** (CI 23–64), top-1 11%, MRR 0.25. Paraphrase
1148
1167
  recall transfers between brains; the ranker is not tuned to the brain it was built on. It is
@@ -12,6 +12,10 @@
12
12
  // npx klypix-mcp install --codex-hooks # optional prompt/file awareness; Codex asks for trust
13
13
  // npx klypix-mcp install --allow-untagged # acknowledge deploying an UNTAGGED source checkout
14
14
  // # (dev deploy; also KLYPIX_MCP_ALLOW_UNTAGGED=1)
15
+ // npx klypix-mcp install --semantic # also install search by meaning (~150 MB download, once)
16
+ // npx klypix-mcp install --no-semantic # never (also KLYPIX_SEMANTIC=0)
17
+ // A person running install in a terminal gets search by meaning by default;
18
+ // scripted, CI and agent-run installs (no terminal) only with --semantic.
15
19
  //
16
20
  // Never-throws-silently: it's an explicit CLI, so it reports what it did and exits
17
21
  // non-zero on a real failure. Never wires a broken settings.json (refuse + restore).
@@ -35,6 +39,7 @@ import { brainInstallDecision, deploySourceDecision } from '../src/install-versi
35
39
  import { acquireInstallLockSync, releaseInstallLockSync } from '../src/install-lock.mjs';
36
40
  import { collectRepoState } from '../src/repo-state.mjs';
37
41
  import { runSetup, renderBrief } from '../src/setup.mjs';
42
+ import { SEMANTIC_RUNTIME_MANIFEST, dirBytes, pruneSemanticRuntime, semanticInstallDecision } from '../src/semantic-install.mjs';
38
43
 
39
44
  const __dirname = path.dirname(fileURLToPath(import.meta.url));
40
45
  const PKG_ROOT = path.resolve(__dirname, '..');
@@ -60,6 +65,10 @@ const SETUP_JSON = process.argv.includes('--json');
60
65
  // this is source authority (knowingly deploy an untagged working tree).
61
66
  const ALLOW_UNTAGGED = process.argv.includes('--allow-untagged')
62
67
  || process.env.KLYPIX_MCP_ALLOW_UNTAGGED === '1';
68
+ // Search by meaning (1.97.0). Measured 2026-10-11 on the real 102-question
69
+ // set: the right card in the top 5 for 54 questions with it, 19 without.
70
+ const SEMANTIC_FLAG = process.argv.includes('--semantic');
71
+ const NO_SEMANTIC = process.argv.includes('--no-semantic') || process.env.KLYPIX_SEMANTIC === '0';
63
72
 
64
73
  const HOME = os.homedir();
65
74
  const CLAUDE_DIR = path.join(HOME, '.claude');
@@ -143,6 +152,52 @@ function wireCodex() {
143
152
  };
144
153
  }
145
154
 
155
+ // ── Search by meaning: the optional on-device runtime (src/semantic-install.mjs) ──
156
+ function installSemanticRuntimeIfWanted() {
157
+ const semanticDir = path.join(BRAIN_DIR, 'semantic');
158
+ const transformers = path.join(semanticDir, 'node_modules', '@huggingface', 'transformers');
159
+ const decision = semanticInstallDecision({
160
+ optOut: NO_SEMANTIC,
161
+ flag: SEMANTIC_FLAG,
162
+ runtimeOnly: RUNTIME_ONLY,
163
+ plugin: String(process.env.KLYPIX_PLUGIN ?? '').trim() === '1',
164
+ present: exists(transformers),
165
+ ci: Boolean(process.env.CI),
166
+ tty: Boolean(process.stdout.isTTY),
167
+ });
168
+ if (decision !== 'install') return { decision };
169
+ console.log('• Installing search by meaning (about 150 MB to download, once; skip with --no-semantic)…');
170
+ const started = Date.now();
171
+ try {
172
+ fs.mkdirSync(semanticDir, { recursive: true });
173
+ fs.writeFileSync(path.join(semanticDir, 'package.json'), `${JSON.stringify(SEMANTIC_RUNTIME_MANIFEST, null, 2)}\n`);
174
+ const npmArgs = ['install', '--prefix', semanticDir, '--omit=dev', '--no-audit', '--no-fund', '--loglevel=error'];
175
+ if (process.platform === 'win32') execFileSync('cmd.exe', ['/d', '/s', '/c', 'npm.cmd', ...npmArgs], { stdio: ['ignore', 'ignore', 'pipe'], timeout: 600_000, windowsHide: true });
176
+ else execFileSync('npm', npmArgs, { stdio: ['ignore', 'ignore', 'pipe'], timeout: 600_000 });
177
+ if (!exists(transformers)) throw new Error('npm finished but the runtime is missing');
178
+ let freed = 0;
179
+ try { freed = pruneSemanticRuntime(semanticDir); } catch { /* pruning is an optimisation */ }
180
+ return { decision, ok: true, seconds: Math.round((Date.now() - started) / 1000), sizeMb: Math.round(dirBytes(path.join(semanticDir, 'node_modules')) / 1048576), freedMb: Math.round(freed / 1048576) };
181
+ } catch (error) {
182
+ const why = String(error?.stderr || error?.message || error).split('\n').map(s => s.trim()).filter(Boolean).slice(-1)[0] || 'npm failed';
183
+ return { decision, ok: false, error: why.slice(0, 200) };
184
+ }
185
+ }
186
+
187
+ function reportSemantic(step) {
188
+ const turnOn = 'npx klypix-mcp install --semantic';
189
+ if (!step) return;
190
+ if (step.decision === 'install' && step.ok) {
191
+ console.log(`✓ search by meaning installed in ${step.seconds}s (${step.sizeMb} MB on disk). Your project brains are being prepared in the background, newest cards first: questions work right away and get better as it finishes.`);
192
+ } else if (step.decision === 'install') {
193
+ console.error(`⚠ search by meaning could not be installed (${step.error}). Keyword search still works. Try again with: ${turnOn}`);
194
+ } else if (step.decision === 'present') {
195
+ console.log('✓ search by meaning: on');
196
+ } else if (step.decision === 'skip-noninteractive' || step.decision === 'skip-ci') {
197
+ console.log(`• search by meaning is off. Turn it on with: ${turnOn} (about 150 MB, once; agents then find the right card about 3x as often)`);
198
+ }
199
+ }
200
+
146
201
  function reportCodex(result) {
147
202
  if (result.ok) {
148
203
  const mcp = result.mcp.action === 'skipped-no-project-brain'
@@ -594,6 +649,9 @@ try {
594
649
  const notWired = ['SessionStart', 'UserPromptSubmit', 'Stop', 'PostToolUse', 'PreToolUse'].filter(e => !wiredFor(e));
595
650
 
596
651
  releaseInstallLockSync(installLock);
652
+ // Search by meaning: installed AFTER the brain itself is committed, so a
653
+ // slow or failed download never touches a working keyword-only install.
654
+ const semanticStep = installSemanticRuntimeIfWanted();
597
655
  // Users who already enabled the optional local semantic runtime should not
598
656
  // pay a multi-minute first question after a model/cache contract upgrade.
599
657
  // Migrate registered brains once in a detached process after the atomic
@@ -617,6 +675,7 @@ try {
617
675
  }
618
676
  if (!RUNTIME_ONLY) reportCodex(codex);
619
677
  console.log(`✓ installed klypix brain v${VERSION} → ${BRAIN_DIR} (${n} scripts, ${deps} dep packages)`);
678
+ reportSemantic(semanticStep);
620
679
  if (RUNTIME_ONLY) console.log('✓ runtime-only update: host settings and project files were preserved');
621
680
  else if (!notWired.length) console.log('✓ wired 5 hooks: SessionStart · UserPromptSubmit (--prompt) · Stop (--capture) · PostToolUse (--live) · PreToolUse (--guard) → settings.json');
622
681
  else console.error(`⚠ readiness: ${notWired.length} hook(s) did NOT take (${notWired.join(', ')}) — the brain will read but not capture/sync. Re-run \`npx klypix-mcp install --force\` or check ${SETTINGS}.`);
@@ -1,7 +1,11 @@
1
1
  #!/usr/bin/env node
2
- // One detached, best-effort cache migration after a compatible core update.
3
- // It runs only when the optional semantic runtime already exists, processes
4
- // registered brains sequentially, and never changes brain content.
2
+ // One detached, best-effort preparation of brains for search by meaning: after
3
+ // an install or a compatible core update (every registered brain), or started
4
+ // by a question on a brain that is not ready yet (--brain <file> --only: that
5
+ // brain alone). It runs at below-normal priority, reports progress to the
6
+ // status file while it works (brainPreparationStatus reads it), refreshes its
7
+ // lock so a long run is never taken for a dead one, and never changes brain
8
+ // content.
5
9
  import fs from 'fs';
6
10
  import os from 'os';
7
11
  import path from 'path';
@@ -9,6 +13,7 @@ import { parseKlypix } from '../src/klypix-format.mjs';
9
13
  import { readRegisteredProjectBrains } from '../src/mcp-auto-update.mjs';
10
14
  import {
11
15
  EMBEDDING_CACHE_KEY,
16
+ cachedVectorsForBrain,
12
17
  disposeSemanticModels,
13
18
  getEmbedderForUse,
14
19
  vectorsForBrain,
@@ -22,6 +27,7 @@ const brainDir = path.resolve(arg('brain-dir', path.join(os.homedir(), '.claude'
22
27
  const lockFile = path.join(brainDir, '.semantic-update-warm.lock');
23
28
  const statusFile = path.join(brainDir, '.semantic-update-warm.json');
24
29
  const explicitBrain = arg('brain');
30
+ const onlyExplicit = process.argv.includes('--only');
25
31
  const startedAt = Date.now();
26
32
  let lock = null;
27
33
 
@@ -31,6 +37,9 @@ const atomicJson = (file, value) => {
31
37
  fs.renameSync(tmp, file);
32
38
  };
33
39
 
40
+ // A background job: never compete with the person's own work for the CPU.
41
+ try { os.setPriority(os.constants.priority.PRIORITY_BELOW_NORMAL); } catch { /* best-effort */ }
42
+
34
43
  try {
35
44
  fs.mkdirSync(brainDir, { recursive: true });
36
45
  try {
@@ -45,13 +54,17 @@ try {
45
54
  }
46
55
  fs.writeFileSync(lock, JSON.stringify({ pid: process.pid, startedAt, modelKey: EMBEDDING_CACHE_KEY }));
47
56
 
48
- const registered = readRegisteredProjectBrains(brainDir).map(item => item.brainPath);
57
+ const registered = onlyExplicit && explicitBrain ? [] : readRegisteredProjectBrains(brainDir).map(item => item.brainPath);
49
58
  const candidates = [...new Set([explicitBrain ? path.resolve(explicitBrain) : null, ...registered].filter(Boolean))]
50
59
  .filter(file => {
51
60
  try { return fs.statSync(file).isFile(); } catch { return false; }
52
61
  });
53
62
  if (!candidates.length) {
54
- atomicJson(statusFile, { protocol: 1, result: 'no-registered-brains', modelKey: EMBEDDING_CACHE_KEY, completedAt: new Date().toISOString() });
63
+ // Nothing to prepare yet, but fetch the model now (~33 MB, once) so the
64
+ // first question on a new brain does not pay for the download.
65
+ let model = 'unavailable';
66
+ try { model = (await getEmbedderForUse(() => {}, 5 * 60_000)) ? 'downloaded' : 'unavailable'; } catch { /* the first question fetches it */ }
67
+ atomicJson(statusFile, { protocol: 1, result: 'no-registered-brains', model, modelKey: EMBEDDING_CACHE_KEY, completedAt: new Date().toISOString() });
55
68
  } else {
56
69
  const pipe = await getEmbedderForUse(() => {}, 5 * 60_000);
57
70
  if (!pipe) {
@@ -61,7 +74,18 @@ try {
61
74
  for (const file of candidates) {
62
75
  try {
63
76
  const { struct } = await parseKlypix(fs.readFileSync(file));
64
- const vectors = await vectorsForBrain(pipe, file, struct.cards);
77
+ const brainStarted = new Date().toISOString();
78
+ let lastReport = 0;
79
+ const report = ({ done, total }, force = false) => {
80
+ if (!force && Date.now() - lastReport < 3_000) return;
81
+ lastReport = Date.now();
82
+ try { atomicJson(statusFile, { protocol: 1, result: 'preparing', modelKey: EMBEDDING_CACHE_KEY, brain: file, done, total, startedAt: brainStarted, updatedAt: new Date().toISOString() }); } catch { /* advisory */ }
83
+ try { const now = new Date(); fs.utimesSync(lockFile, now, now); } catch { /* advisory */ }
84
+ };
85
+ let alreadyReady = 0;
86
+ try { alreadyReady = cachedVectorsForBrain(file, struct.cards).size; } catch { /* starts from zero */ }
87
+ report({ done: alreadyReady, total: struct.cards.filter(c => c.type !== 'container' && String(c.text || '').trim()).length }, true);
88
+ const vectors = await vectorsForBrain(pipe, file, struct.cards, { maxEmbed: Infinity, onProgress: report });
65
89
  warmed++;
66
90
  cards += vectors.size;
67
91
  } catch { failed++; }
@@ -508,7 +508,7 @@ server.registerTool('search_all_brains', {
508
508
 
509
509
  server.registerTool('brain_ask', {
510
510
  title: 'Ask the project brain a question (whole-brain, correction-aware answer)',
511
- description: 'Answer a natural-language question from the WHOLE project brain — "what did we decide about X?", "where did the auth work land?", "why did we drop Y?". Ranks every card lexically, blended with on-device semantic similarity ONLY when the optional local model is installed (a fresh install is lexical; it degrades cleanly). INCLUDES superseded/archived history (flagged, so you can see how a decision changed), and attaches each stale card\'s live CORRECTION so the answer reflects the current truth, not an outdated card. Returns a synthesis-ready context (full cards + provenance + lifecycle) for you to turn into a direct, cited answer — it does not itself write prose. Prefer this over search_canvases when the user asks a QUESTION (not a keyword lookup). Optional as_of (YYYY-MM-DD) answers "what was true then". Defaults to the project brain ("brain").',
511
+ description: 'Answer a natural-language question from the WHOLE project brain — "what did we decide about X?", "where did the auth work land?", "why did we drop Y?". Ranks every card by meaning with the on-device model (installed by `npx klypix-mcp install` run in a terminal, or with `--semantic`), with keywords for exact names; without the model it is keyword-only, and while a brain is still being prepared the answer says how far along it is. INCLUDES superseded/archived history (flagged, so you can see how a decision changed), and attaches each stale card\'s live CORRECTION so the answer reflects the current truth, not an outdated card. Returns a synthesis-ready context (full cards + provenance + lifecycle) for you to turn into a direct, cited answer — it does not itself write prose. Prefer this over search_canvases when the user asks a QUESTION (not a keyword lookup). Optional as_of (YYYY-MM-DD) answers "what was true then". Cards are often worded differently from the question, so end the question with one more line, `Also: <8–14 words a card answering it would likely use: synonyms, feature, file or tool names>`; they widen the search while the question still leads (measured: right card in the top 5 for 60 of 102 test questions instead of 55; 24 instead of 19 when no local model is installed). Defaults to the project brain ("brain").',
512
512
  inputSchema: {
513
513
  question: z.string().describe('The natural-language question to answer from the brain.'),
514
514
  canvas: z.string().optional().describe('Brain canvas filename/path. Defaults to the project brain ("brain").'),
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "klypix-mcp",
3
- "version": "1.95.0",
3
+ "version": "1.97.0",
4
4
  "mcpName": "io.github.dahshanlabs/klypix-mcp",
5
5
  "description": "Active state management for multi-agent coding: a shared, versioned project brain over MCP.",
6
6
  "type": "module",
@@ -86,7 +86,7 @@
86
86
  "bench": "node bin/klypix-mcp.mjs bench",
87
87
  "test:bench": "node test/bench.mjs",
88
88
  "pretest": "node test/publish-workflow.mjs",
89
- "test": "node test/publish-verdict.mjs && node test/npx-owned-names.mjs && node test/project-graph.mjs && node test/project-map-cli.mjs && node test/mcp-auto-update.mjs && node test/mcp-supervisor.mjs && node test/runtime-inspector.mjs && node test/codex-hooks.mjs && node test/request-identity.mjs && node test/session-identity-core.mjs && node test/agent-presence.mjs && node test/message-delivery-v3.mjs && node test/claude-message-delivery-v3.mjs && node test/session-mailbox.mjs && node test/session-reopen.mjs && node test/lane-write-retry.mjs && node test/result-reconcile.mjs && node test/evidence-publication-gate.mjs && node test/release-evidence-cli.mjs && node test/intent-guard.mjs && node test/git-capture-install.mjs && node test/brain-history.mjs && node test/brain-graveyard.mjs && node test/archived-visibility.mjs && node test/finding-routing.mjs && node test/finding-routing-hook.mjs && node test/presence-relay.mjs && node test/install-version.mjs && node test/install-rename-backoff.mjs && node test/project-binding-rebind.mjs && node test/context-gateway.mjs && node test/repo-state.mjs && node test/released-tag-guard.mjs && node test/conformance.mjs && node test/brain-doctor.mjs && node test/version-currency.mjs && node test/ship-capture.mjs && node test/capture-gap.mjs && node test/lane-message.mjs && node test/brain-quality.mjs && node test/brain-connect-orphans.mjs && node test/orphan-gardener.mjs && node test/brief-and-recall.mjs && node test/guard-cards.mjs && node test/layout-cluster.mjs && node test/brain-ask.mjs && node test/retrieval-fusion.mjs && node test/eval-retrieval.mjs && node test/field-report-2026-07-04.mjs && node test/autoprop.mjs && node test/overlay-recency-2026-07-12.mjs && node test/brain-challenge.mjs && node test/brain-lens.mjs && node test/brain-kind.mjs && node test/rule-drafts.mjs && node test/claim-engine.mjs && node test/partial-notes.mjs && node test/lifecycle-prefix.mjs && node test/close-link-safety.mjs && node test/resolve-ledger.mjs && node test/plan-fulfillment.mjs && node test/arrange-receipts.mjs && node test/skill-staleness.mjs && node test/canvas-view.mjs && node test/status-completeness.mjs && node test/semantic-security.mjs && node test/semantic-gate.mjs && node test/memory-runtime.mjs && node test/semantic-cache.mjs && node test/semantic-hash-parity.mjs && node test/enrichment.mjs && node test/provenance.mjs && node test/confirm-trail.mjs && node test/hook-fallback.mjs && node test/eval-hook-lane.mjs && node test/hook-unified-lane.mjs && node test/decay-status.mjs && node test/decay-hook.mjs && node test/marker-suffix-grammar.mjs && node test/evidence-anchors.mjs && node test/brain-evidence.mjs && node test/presence-visibility.mjs && node test/undeclared-active.mjs && node test/presence-liveness.mjs && node test/observed-scope.mjs && node test/release-lease.mjs && node test/release-reconcile.mjs && node test/release-ancestry.mjs && node test/release-claim-join.mjs && node test/release-claims.mjs && node test/release-handshake.mjs && node test/completion-guard.mjs && node test/merge-brains.mjs && node test/merge-readings.mjs && node test/revival-map.mjs && node test/merge-scale.mjs && node test/concurrent-writes.mjs && node test/lock-interop.mjs && node test/capture-write-failure.mjs && node test/a2a-smoke.mjs && node test/one-command-setup.mjs && node test/cli-args.mjs && node test/format-guard.mjs && node test/canvas-groups.mjs && node test/git-tools.mjs && node test/link-compat.mjs && node test/uninstall.mjs && node test/current-guidance.mjs && node test/status-shape.mjs && node test/status-hook.mjs && node test/read-canvas-saved-readings.mjs && node test/document-readings.mjs && node test/open-lease.mjs && node test/agent-card-style.mjs && node test/tool-schema-freeze.mjs && node test/app-lease-vectors.mjs && node test/app-tools-stdio.mjs && node test/card-contents-files.mjs && node test/codex-app-table.mjs && node test/plugin-mode.mjs && node test/app-bridge-vectors.mjs && node test/app-bridge-client.mjs && node test/app-tools-static.mjs && node test/app-tools-stdio-live.mjs",
89
+ "test": "node test/publish-verdict.mjs && node test/npx-owned-names.mjs && node test/project-graph.mjs && node test/project-map-cli.mjs && node test/mcp-auto-update.mjs && node test/mcp-supervisor.mjs && node test/runtime-inspector.mjs && node test/codex-hooks.mjs && node test/request-identity.mjs && node test/session-identity-core.mjs && node test/agent-presence.mjs && node test/message-delivery-v3.mjs && node test/claude-message-delivery-v3.mjs && node test/session-mailbox.mjs && node test/session-reopen.mjs && node test/lane-write-retry.mjs && node test/result-reconcile.mjs && node test/evidence-publication-gate.mjs && node test/release-evidence-cli.mjs && node test/intent-guard.mjs && node test/git-capture-install.mjs && node test/brain-history.mjs && node test/brain-graveyard.mjs && node test/archived-visibility.mjs && node test/finding-routing.mjs && node test/finding-routing-hook.mjs && node test/presence-relay.mjs && node test/install-version.mjs && node test/install-rename-backoff.mjs && node test/project-binding-rebind.mjs && node test/context-gateway.mjs && node test/repo-state.mjs && node test/released-tag-guard.mjs && node test/conformance.mjs && node test/brain-doctor.mjs && node test/version-currency.mjs && node test/ship-capture.mjs && node test/capture-gap.mjs && node test/lane-message.mjs && node test/brain-quality.mjs && node test/brain-connect-orphans.mjs && node test/orphan-gardener.mjs && node test/brief-and-recall.mjs && node test/guard-cards.mjs && node test/layout-cluster.mjs && node test/brain-ask.mjs && node test/brain-ask-words.mjs && node test/retrieval-fusion.mjs && node test/eval-retrieval.mjs && node test/field-report-2026-07-04.mjs && node test/autoprop.mjs && node test/overlay-recency-2026-07-12.mjs && node test/brain-challenge.mjs && node test/brain-lens.mjs && node test/brain-kind.mjs && node test/rule-drafts.mjs && node test/claim-engine.mjs && node test/partial-notes.mjs && node test/lifecycle-prefix.mjs && node test/close-link-safety.mjs && node test/resolve-ledger.mjs && node test/plan-fulfillment.mjs && node test/arrange-receipts.mjs && node test/skill-staleness.mjs && node test/canvas-view.mjs && node test/status-completeness.mjs && node test/semantic-security.mjs && node test/semantic-gate.mjs && node test/memory-runtime.mjs && node test/semantic-cache.mjs && node test/semantic-preparation.mjs && node test/semantic-hash-parity.mjs && node test/enrichment.mjs && node test/provenance.mjs && node test/confirm-trail.mjs && node test/hook-fallback.mjs && node test/eval-hook-lane.mjs && node test/hook-unified-lane.mjs && node test/decay-status.mjs && node test/decay-hook.mjs && node test/marker-suffix-grammar.mjs && node test/evidence-anchors.mjs && node test/brain-evidence.mjs && node test/presence-visibility.mjs && node test/undeclared-active.mjs && node test/presence-liveness.mjs && node test/observed-scope.mjs && node test/release-lease.mjs && node test/release-reconcile.mjs && node test/release-ancestry.mjs && node test/release-claim-join.mjs && node test/release-claims.mjs && node test/release-handshake.mjs && node test/completion-guard.mjs && node test/merge-brains.mjs && node test/merge-readings.mjs && node test/revival-map.mjs && node test/merge-scale.mjs && node test/concurrent-writes.mjs && node test/lock-interop.mjs && node test/capture-write-failure.mjs && node test/a2a-smoke.mjs && node test/one-command-setup.mjs && node test/cli-args.mjs && node test/format-guard.mjs && node test/canvas-groups.mjs && node test/git-tools.mjs && node test/link-compat.mjs && node test/uninstall.mjs && node test/current-guidance.mjs && node test/status-shape.mjs && node test/status-hook.mjs && node test/read-canvas-saved-readings.mjs && node test/document-readings.mjs && node test/open-lease.mjs && node test/agent-card-style.mjs && node test/tool-schema-freeze.mjs && node test/app-lease-vectors.mjs && node test/app-tools-stdio.mjs && node test/card-contents-files.mjs && node test/codex-app-table.mjs && node test/plugin-mode.mjs && node test/app-bridge-vectors.mjs && node test/app-bridge-client.mjs && node test/app-tools-static.mjs && node test/app-tools-stdio-live.mjs",
90
90
  "test:memory": "node test/memory-runtime.mjs",
91
91
  "test:memory:soak": "node --expose-gc test/memory-soak.mjs",
92
92
  "runtime": "node bin/klypix-runtime.mjs",
@@ -30,7 +30,7 @@ import {
30
30
  findUnrecordedMigrations, captureIntoBrain, tidyBrain, noteToCaptureInput,
31
31
  selectGardenCandidates, applyGarden, detectContradictions,
32
32
  findDuplicatePartialNotes, collapseDuplicatePartialNotes,
33
- rankForQuestion, questionContextToMarkdown, findLegacyShipCards,
33
+ rankForQuestion, splitAskWords, questionContextToMarkdown, findLegacyShipCards,
34
34
  challengeBrain, challengeContextToMarkdown, buildRenderSpec, structToBrief,
35
35
  brainLensData, lensToMarkdown, deathDateOfCard,
36
36
  statusContextToMarkdown, findFulfillmentCandidates, findStaleOpenCards, releaseFulfilledOpens, openStatusSummary, perAreaTableToMarkdown,
@@ -56,7 +56,7 @@ import {
56
56
  dot, embedTexts, getEmbedder, getEmbedderForUse, withRerankerForUse,
57
57
  rerankHits, semanticFallbackNotice, semanticMemorySnapshot,
58
58
  semanticRuntimeInstalled, shouldPrewarmSemantic, vectorsForBrain,
59
- cachedVectorsForBrain,
59
+ cachedVectorsForBrain, startBrainPreparation, preparationNotice,
60
60
  } from './semantic-memory.mjs';
61
61
 
62
62
  // The MCP and A2A faces prewarm through this protocol-neutral module. Bounded
@@ -503,7 +503,7 @@ export async function opSearchAllBrains({ vault, query, as_of, log = () => {} })
503
503
  let struct;
504
504
  try { ({ struct } = await parseKlypix(fs.readFileSync(b.path))); } catch { continue; }
505
505
  let vecs = null;
506
- if (qv) { try { vecs = await vectorsForBrain(pipe, b.path, struct.cards); } catch { /* lexical for this brain */ } }
506
+ if (qv) { try { const r = await questionVectors(pipe, b.path, struct.cards); vecs = r.usable ? r.vecs : null; } catch { /* lexical for this brain */ } }
507
507
  for (const c of struct.cards) {
508
508
  if (c.type === 'container') continue;
509
509
  const t = String(c.text || '').toLowerCase();
@@ -765,8 +765,25 @@ export async function opBrainTaskContext({
765
765
  };
766
766
  }
767
767
 
768
+ // Card meanings for a question, never waiting on a preparation (1.97.0): at
769
+ // most a few seconds of embedding inline; whatever is still missing is handed
770
+ // to the background preparation (startBrainPreparation) and the answer says
771
+ // so (preparationNotice). Meaning ranking is used once at least
772
+ // MEANING_READY_SHARE of the cards are ready — below that the keyword path
773
+ // covers every card, which a mostly-unprepared meaning ranking would not.
774
+ const MEANING_READY_SHARE = 0.5;
775
+ async function questionVectors(pipe, file, cards) {
776
+ const stats = {};
777
+ const vecs = await vectorsForBrain(pipe, file, cards, { stats });
778
+ if (stats.pending > 0) startBrainPreparation(file);
779
+ const usable = Boolean(vecs && vecs.size) && (!stats.wanted || stats.ready / stats.wanted >= MEANING_READY_SHARE);
780
+ return { vecs, stats, usable };
781
+ }
782
+
768
783
  export async function opBrainAsk({ vault, canvas, question, as_of, k = 10, log = () => {} }) {
769
- const q = String(question || '').trim();
784
+ // A last line `Also: <words>` is the caller's guess at the card's wording; it
785
+ // joins the meaning query only (splitAskWords in klypix-format.mjs).
786
+ const { question: q, words } = splitAskWords(question);
770
787
  if (!q) return err('brain_ask needs a question.');
771
788
  const t = brainTarget(vault, canvas);
772
789
  if (t.ambiguous) return ambiguousBrainErr(t.ambiguous);
@@ -779,13 +796,14 @@ export async function opBrainAsk({ vault, canvas, question, as_of, k = 10, log =
779
796
  // Semantic blend (best-effort, time-bounded): embed the question + the brain's
780
797
  // cards on-device, hand rankForQuestion a Map<cardId, cosine>. A missing/warming
781
798
  // model degrades cleanly to pure lexical.
782
- let semantic = null, mode = 'lexical', cardVecs = null;
799
+ let semantic = null, mode = 'lexical', cardVecs = null, prep = null;
783
800
  try {
784
801
  const pipe = await getEmbedderForUse(log, 20_000);
785
802
  if (pipe) {
786
- const [qv] = await embedTexts(pipe, [q], { kind: 'query' });
787
- const vecs = await vectorsForBrain(pipe, t.file, struct.cards);
788
- if (qv && vecs && vecs.size) { semantic = new Map(); for (const [id, v] of vecs) semantic.set(id, dot(qv, v)); cardVecs = vecs; mode = 'semantic+lexical (on-device)'; }
803
+ const [qv] = await embedTexts(pipe, [words ? `${q}\n${words}` : q], { kind: 'query' });
804
+ const { vecs, stats, usable } = await questionVectors(pipe, t.file, struct.cards);
805
+ prep = stats;
806
+ if (qv && usable) { semantic = new Map(); for (const [id, v] of vecs) semantic.set(id, dot(qv, v)); cardVecs = vecs; mode = `semantic+lexical (on-device)${words ? ' + your Also words' : ''}`; }
789
807
  }
790
808
  } catch { semantic = null; cardVecs = null; }
791
809
  // Parity with search_all_brains (8445e9c): never say "warming — retry" to a
@@ -799,6 +817,8 @@ export async function opBrainAsk({ vault, canvas, question, as_of, k = 10, log =
799
817
  if (!semantic) {
800
818
  fallbackNotice = semanticFallbackNotice(false);
801
819
  if (fallbackNotice) mode = fallbackNotice.startsWith('LEXICAL ONLY') ? 'lexical-only (no semantic runtime)' : 'lexical (semantic warming)';
820
+ if (prep?.pending > 0) mode = 'lexical (preparing search by meaning)';
821
+ if (words) mode += ' + your Also words';
802
822
  }
803
823
  const kk = Math.max(1, Math.min(20, k || 10));
804
824
  const timeTravel = asOfTs != null;
@@ -811,7 +831,7 @@ export async function opBrainAsk({ vault, canvas, question, as_of, k = 10, log =
811
831
  // vectors that rank the question, reused; null degrades pairing to its
812
832
  // lexical anchor/coverage fallback (works on every host, embeddings or not).
813
833
  const pairSim = cardVecs ? (a, b) => { const va = cardVecs.get(a), vb = cardVecs.get(b); return va && vb ? dot(va, vb) : null; } : null;
814
- const result = rankForQuestion(struct, q, { semantic, k: wantRerank ? Math.max(kk, 50) : kk, as_of: timeTravel ? as_of : null, pairSim });
834
+ const result = rankForQuestion(struct, q, { semantic, k: wantRerank ? Math.max(kk, 50) : kk, as_of: timeTravel ? as_of : null, pairSim, words });
815
835
  if (wantRerank && result.hits.length > 1) {
816
836
  try {
817
837
  const reranked = await withRerankerForUse(log, 8_000, rr => rerankHits(rr, q, result.hits));
@@ -846,7 +866,11 @@ export async function opBrainAsk({ vault, canvas, question, as_of, k = 10, log =
846
866
  mode += ' + status-mode';
847
867
  } catch { statusMd = ''; }
848
868
  }
849
- const noticeMd = fallbackNotice ? `> ${fallbackNotice}\n\n` : '';
869
+ // While the brain is still being prepared, that is the one thing to say: the
870
+ // runtime IS installed, so the "warming — retry" fallback line would only
871
+ // repeat it less precisely.
872
+ const prepNotice = preparationNotice(t.file, prep, { usedMeaning: Boolean(semantic) });
873
+ const noticeMd = prepNotice ? `> ${prepNotice}\n\n` : (fallbackNotice ? `> ${fallbackNotice}\n\n` : '');
850
874
  const evidenceCache = new Map();
851
875
  const evidenceForCard = (card) => formatCardEvidence(inspectCardEvidence(card, { projectRoot: path.dirname(t.file), cache: evidenceCache }));
852
876
  return { blocks: [text(stamp + noticeMd + statusMd + questionContextToMarkdown(q, result, { mode, as_of: timeTravel ? as_of : null, evidenceForCard }))] };
@@ -871,8 +895,8 @@ export async function opBrainChallenge({ vault, canvas, claim, k = 8, via, log =
871
895
  const pipe = await getEmbedderForUse(log, 20_000);
872
896
  if (pipe) {
873
897
  const [qv] = await embedTexts(pipe, [q], { kind: 'query' });
874
- const vecs = await vectorsForBrain(pipe, t.file, struct.cards);
875
- if (qv && vecs && vecs.size) { semantic = new Map(); for (const [id, v] of vecs) semantic.set(id, dot(qv, v)); mode = 'semantic+lexical (on-device)'; }
898
+ const { vecs, usable } = await questionVectors(pipe, t.file, struct.cards);
899
+ if (qv && usable) { semantic = new Map(); for (const [id, v] of vecs) semantic.set(id, dot(qv, v)); mode = 'semantic+lexical (on-device)'; }
876
900
  }
877
901
  } catch { semantic = null; }
878
902
  const result = challengeBrain(struct, q, { semantic, k: Math.max(1, Math.min(20, k || 8)) });
@@ -1160,7 +1184,11 @@ export async function opBrainReconcile({ vault, canvas, root, mode = 'all', ref
1160
1184
  if (mode === 'plans' && cold && installed) {
1161
1185
  try {
1162
1186
  const pipe = await getEmbedderForUse(log, 20_000);
1163
- if (pipe) vecs = await vectorsForBrain(pipe, file, struct.cards);
1187
+ if (pipe) {
1188
+ const stats = {};
1189
+ vecs = await vectorsForBrain(pipe, file, struct.cards, { stats });
1190
+ if (stats.pending > 0) startBrainPreparation(file);
1191
+ }
1164
1192
  } catch (e) { embedErr = e; }
1165
1193
  }
1166
1194
  let pairSim = null, how;
@@ -1696,7 +1724,9 @@ export async function opBrainConnect({ vault, canvas, apply = false, max = 24, t
1696
1724
  try { pipe = await getEmbedderForUse(log, 20_000); } catch { /* saturated/unavailable → structural */ }
1697
1725
  if (pipe) {
1698
1726
  try {
1699
- const vecs = await vectorsForBrain(pipe, file, struct.cards);
1727
+ const stats = {};
1728
+ const vecs = await vectorsForBrain(pipe, file, struct.cards, { stats });
1729
+ if (stats.pending > 0) startBrainPreparation(file);
1700
1730
  const items = live.filter(c => vecs.get(c.id));
1701
1731
  const sources = normalizedScope === 'orphans' ? items.filter(c => orphanIds.has(c.id)) : items;
1702
1732
  for (const a of sources) {
@@ -4042,14 +4042,51 @@ export const deathDateOfCard = (card) => {
4042
4042
  const m = /(?:↩︎ superseded|↩ superseded|✅|⤵ consolidated) (\d{4}-\d{2}-\d{2})/.exec(String(c.text || ''));
4043
4043
  return m ? Date.parse(m[1]) : null;
4044
4044
  };
4045
- export function rankForQuestion(struct, question, { semantic = null, k = 10, as_of = null, now = Date.now(), recentDays = 30, pairSim = null } = {}) {
4045
+ // THE CALLER'S WORDS (1.96.0). A question is worded the asker's way; the card
4046
+ // that answers it was worded the day it was written ("magnification controls"
4047
+ // vs a card that says "zoom"). The calling agent is itself a model, so brain_ask
4048
+ // invites it to add one last line `Also: <words a card answering this would
4049
+ // likely use>`. Those words join the MEANING query only (embedded together with
4050
+ // the question, one vector); the keyword anchor test, status shape and the
4051
+ // rendered question all stay the question's own. Measured 2026-10-11 on the
4052
+ // real 3,829-card brain, frozen v2 set (102 answerable), production embedder +
4053
+ // warm enrichment-aware vectors, model-written words: recall@5 55 → 60
4054
+ // (+8 −3), @8 62 → 69, @20 69 → 78, MRR .375 → .424. Fusing a separate words
4055
+ // vector by rank instead did less (+5 −2 at equal weight). Carried inside
4056
+ // `question` because no existing tool's input schema may change (see
4057
+ // test/tool-schema-freeze.mjs) — a new property would force every host to
4058
+ // reconnect; a line in the text works the moment the worker hot-swaps.
4059
+ export const ASK_WORDS_MAX = 240;
4060
+ // Keyword-only mode (no meaning model), same brain and set, words as extra
4061
+ // tokens, through splitAskWords: recall@5 19 → 24 of 102 at half weight with
4062
+ // none lost (@10 26 → 32); full weight lost a question the question alone
4063
+ // found. Measured 2026-10-11.
4064
+ export const ASK_WORDS_LEXICAL_WEIGHT = 0.5;
4065
+ export function splitAskWords(question) {
4066
+ const whole = String(question || '').trim();
4067
+ const lines = whole.split(/\r?\n/);
4068
+ const last = lines.length > 1 ? /^\s*also\s*:\s*(.*)$/i.exec(lines[lines.length - 1]) : null;
4069
+ const rest = last ? lines.slice(0, -1).join('\n').trim() : '';
4070
+ // No words line, or nothing above it: the text is the question, untouched.
4071
+ if (!last || !rest) return { question: whole, words: '' };
4072
+ const words = last[1].replace(/\s+/g, ' ').trim().slice(0, ASK_WORDS_MAX).trim();
4073
+ // An empty `Also:` line is dropped; words equal to the question add nothing.
4074
+ return { question: rest, words: words.toLowerCase() === rest.toLowerCase() ? '' : words };
4075
+ }
4076
+
4077
+ export function rankForQuestion(struct, question, { semantic = null, k = 10, as_of = null, now = Date.now(), recentDays = 30, pairSim = null, words = '' } = {}) {
4046
4078
  // Status-shaped questions score by their CONTENT tokens only — "remaining"
4047
4079
  // must never lexically select the stale cards that say "remaining:".
4048
4080
  const { content: tokens, statusShaped, strong: statusStrong, areaFamilies = [] } = splitQueryTokens(question);
4081
+ // The caller's Also words (splitAskWords). With the meaning model they ride
4082
+ // the query vector (opBrainAsk); WITHOUT it — a fresh install, the Claude
4083
+ // plugin — they are extra keyword tokens at ASK_WORDS_LEXICAL_WEIGHT, after
4084
+ // the question's own, so the question still leads.
4085
+ const extra = words && !semantic ? splitQueryTokens(words).content.filter(t => !tokens.includes(t)) : [];
4049
4086
  // Area scope (1.85.0): this function HAS the struct, so it resolves the
4050
4087
  // prompt's static families to exact area titles for the status renderer.
4051
4088
  const areas = statusStrong ? areaHintsFromPrompt(struct, question) : null;
4052
- if (!struct || !Array.isArray(struct.cards) || (!tokens.length && !semantic)) return { hits: [], total: 0, tokens, statusShaped, statusStrong, areaFamilies, areas };
4089
+ if (!struct || !Array.isArray(struct.cards) || (!tokens.length && !extra.length && !semantic)) return { hits: [], total: 0, tokens, statusShaped, statusStrong, areaFamilies, areas };
4053
4090
  const isArchived = (c) => /^archive$/i.test(c.area || '');
4054
4091
  const asOfTs = as_of ? Date.parse(as_of) : null;
4055
4092
  const timeTravel = asOfTs != null && Number.isFinite(asOfTs);
@@ -4087,6 +4124,10 @@ export function rankForQuestion(struct, question, { semantic = null, k = 10, as_
4087
4124
  else if (wordMatches(tagWords, tagStems, tok)) lex += 3;
4088
4125
  else if (wordMatches(bodyW, bodyStems, tok)) lex += lenNorm;
4089
4126
  }
4127
+ for (const tok of extra) {
4128
+ if (wordMatches(titleW, titleStems, tok) || wordMatches(tagWords, tagStems, tok)) lex += 3 * ASK_WORDS_LEXICAL_WEIGHT;
4129
+ else if (wordMatches(bodyW, bodyStems, tok)) lex += lenNorm * ASK_WORDS_LEXICAL_WEIGHT;
4130
+ }
4090
4131
  const sem = semantic ? (semantic.get(c.id) ?? null) : null;
4091
4132
  if (!semantic && lex <= 0) continue;
4092
4133
  if (semantic && lex <= 0 && sem == null) continue;
@@ -0,0 +1,84 @@
1
+ // Search by meaning: what the installer installs, who gets it, and what it
2
+ // trims afterwards. Pure helpers, kept out of bin/klypix-install.mjs because
3
+ // that file runs the install the moment it is imported.
4
+ //
5
+ // Measured 2026-10-11 on a clean npm cache: ~150 MB download, ~20 s on a fast
6
+ // line. Pruning what Node never loads (onnxruntime-web, other platforms' and
7
+ // architectures' native binaries) leaves ~110 MB on disk instead of ~390 MB,
8
+ // with the same answers and memory (load + 50 questions checked on the pruned
9
+ // tree). The gain, on the real 102-question set: the right card in the top 5
10
+ // for 54 questions with it, 19 without.
11
+ import fs from 'fs';
12
+ import path from 'path';
13
+
14
+ // Same pins as the desktop app's electron/semanticInstaller.ts: an owned npm
15
+ // tree, so a vulnerable sharp can never be resolved independently.
16
+ export const SEMANTIC_RUNTIME_MANIFEST = Object.freeze({
17
+ name: 'klypix-semantic-runtime',
18
+ private: true,
19
+ version: '1.0.0',
20
+ dependencies: { '@huggingface/transformers': '4.2.0', sharp: '0.35.5' },
21
+ overrides: {
22
+ '@huggingface/transformers': { sharp: '$sharp' },
23
+ 'onnxruntime-node': { 'adm-zip': '0.6.1' },
24
+ 'onnxruntime-web': { protobufjs: '7.6.5' },
25
+ },
26
+ });
27
+
28
+ // Who gets it. A person at a terminal gets it unless they opt out; anything
29
+ // without a terminal (CI, scripts, an agent running the install, the test
30
+ // suite) only when it asks with --semantic. Plugin mode never: the Claude
31
+ // plugin runs only its pinned package and downloads no models.
32
+ export function semanticInstallDecision({ optOut, flag, runtimeOnly, plugin, present, ci, tty }) {
33
+ if (plugin) return 'skip-plugin';
34
+ if (optOut) return 'skip-optout';
35
+ if (present) return 'present';
36
+ if (flag) return 'install';
37
+ if (runtimeOnly) return 'skip-runtime-only';
38
+ if (ci) return 'skip-ci';
39
+ if (!tty) return 'skip-noninteractive';
40
+ return 'install';
41
+ }
42
+
43
+ export function dirBytes(dir) {
44
+ let total = 0;
45
+ let entries = [];
46
+ try { entries = fs.readdirSync(dir, { withFileTypes: true }); } catch { return 0; }
47
+ for (const entry of entries) {
48
+ const p = path.join(dir, entry.name);
49
+ if (entry.isDirectory()) total += dirBytes(p);
50
+ else { try { total += fs.statSync(p).size; } catch { /* gone */ } }
51
+ }
52
+ return total;
53
+ }
54
+
55
+ // Remove what Node never loads. Touches only <semanticDir>/node_modules, and
56
+ // never removes the running platform's binary for the running architecture
57
+ // (a tree without it is left alone rather than pruned to nothing).
58
+ export function pruneSemanticRuntime(semanticDir, platform = process.platform, arch = process.arch) {
59
+ const nm = path.join(semanticDir, 'node_modules');
60
+ let freed = 0;
61
+ const drop = (p) => {
62
+ try { fs.statSync(p); } catch { return; }
63
+ freed += dirBytes(p);
64
+ fs.rmSync(p, { recursive: true, force: true });
65
+ };
66
+ drop(path.join(nm, 'onnxruntime-web'));
67
+ const binRoot = path.join(nm, 'onnxruntime-node', 'bin');
68
+ let napis = [];
69
+ try { napis = fs.readdirSync(binRoot); } catch { /* none */ }
70
+ for (const napi of napis) {
71
+ let platforms = [];
72
+ try { platforms = fs.readdirSync(path.join(binRoot, napi)); } catch { continue; }
73
+ if (!platforms.includes(platform)) continue;
74
+ for (const plat of platforms) {
75
+ const platDir = path.join(binRoot, napi, plat);
76
+ if (plat !== platform) { drop(platDir); continue; }
77
+ let arches = [];
78
+ try { arches = fs.readdirSync(platDir); } catch { continue; }
79
+ if (!arches.includes(arch)) continue;
80
+ for (const a of arches) if (a !== arch) drop(path.join(platDir, a));
81
+ }
82
+ }
83
+ return freed;
84
+ }
@@ -10,6 +10,8 @@ import path from 'path';
10
10
  import crypto from 'crypto';
11
11
  import { enrichmentTextFor, readEnrichment } from './enrichment.mjs';
12
12
  import { createRequire } from 'module';
13
+ import { spawn } from 'child_process';
14
+ import { fileURLToPath } from 'url';
13
15
  import * as autoUpdateLib from './mcp-auto-update.mjs';
14
16
 
15
17
  // Plugin mode (KLYPIX_PLUGIN=1) never loads the on-device semantic runtime:
@@ -42,7 +44,13 @@ const intEnv = (name, fallback, min, max) => {
42
44
  const n = Number(process.env[name]);
43
45
  return Number.isFinite(n) ? Math.max(min, Math.min(max, Math.trunc(n))) : fallback;
44
46
  };
45
- const EMBED_BATCH_SIZE = intEnv('KLYPIX_EMBED_BATCH_SIZE', 16, 1, 64);
47
+ // One card per inference call. A batch pads every card to its longest member
48
+ // (cards run up to 1,500 characters), so batching wastes work on a brain's
49
+ // mixed lengths. Measured 2026-10-11, real bge-small q8, 400 real cards on a
50
+ // laptop CPU: batch 16 = 96 ms a card, 1,176 MB peak; 8 = 79 ms, 720 MB;
51
+ // 4 = 63–70 ms, ~505 MB; 1 = 41–49 ms, ~305 MB. The fastest is also the lightest.
52
+ export const EMBED_BATCH_DEFAULT = 1;
53
+ const EMBED_BATCH_SIZE = intEnv('KLYPIX_EMBED_BATCH_SIZE', EMBED_BATCH_DEFAULT, 1, 64);
46
54
  const RERANK_BATCH_SIZE = intEnv('KLYPIX_RERANK_BATCH_SIZE', 4, 1, 32);
47
55
  const MAX_INFERENCE_QUEUE = intEnv('KLYPIX_SEMANTIC_MAX_QUEUE', 16, 1, 256);
48
56
  const idleRaw = process.env.KLYPIX_SEMANTIC_IDLE_MS;
@@ -351,13 +359,12 @@ export function semanticFallbackNotice(hadSemantic) {
351
359
  if (semanticRuntimeInstalled()) {
352
360
  return 'lexical only — the on-device semantic model is still warming or the queue is saturated; retry shortly for semantic ranking';
353
361
  }
354
- // Deliberately names no install command: there is no `--semantic` flag, and
355
- // bin/klypix-install.mjs skips @huggingface/transformers on purpose ("optional,
356
- // huge"). Inventing a command here would ship a wrong instruction to every
357
- // MCP host. State the fact and the location; let the caller decide.
358
- return 'LEXICAL ONLY — @huggingface/transformers is not present, so these are keyword matches with no semantic ranking. '
359
- + 'Measured on a real 2,377-card brain: recall@5 0% lexical vs ~30% with the runtime. Retrying will not help. '
360
- + `Semantic ranking activates once the runtime resolves, or is placed at ${path.join(PB_DIR, 'semantic', 'node_modules', '@huggingface', 'transformers')}.`;
362
+ // Since 1.97.0 there IS an install command (bin/klypix-install.mjs --semantic,
363
+ // which a person at a terminal gets by default), so the notice names it: an
364
+ // agent reading this can offer it to the person instead of guessing.
365
+ return 'LEXICAL ONLY — search by meaning is not installed, so these are keyword matches. '
366
+ + 'Measured on a real 3,829-card brain: the right card in the top 5 for 19 of 102 questions with keywords, 54 with search by meaning. Retrying will not help. '
367
+ + 'Turn it on with: npx klypix-mcp install --semantic (about 150 MB to download, once).';
361
368
  }
362
369
 
363
370
  export function getEmbedder(log = () => {}) {
@@ -500,11 +507,11 @@ function cacheCandidates(brainPath) {
500
507
 
501
508
  const wait = (ms) => new Promise(resolve => setTimeout(resolve, ms));
502
509
 
503
- async function withCrossProcessCacheLock(brainPath, fn) {
510
+ async function withCrossProcessCacheLock(brainPath, fn, timeoutMs = CACHE_LOCK_TIMEOUT_MS) {
504
511
  const file = cacheCandidates(brainPath)[0].file;
505
512
  const lockFile = `${file}.lock`;
506
513
  fs.mkdirSync(EMB_DIR, { recursive: true });
507
- const deadline = Date.now() + CACHE_LOCK_TIMEOUT_MS;
514
+ const deadline = Date.now() + timeoutMs;
508
515
  let handle = null;
509
516
  let waited = false;
510
517
  while (!handle) {
@@ -527,7 +534,10 @@ async function withCrossProcessCacheLock(brainPath, fn) {
527
534
  await wait(75);
528
535
  }
529
536
  }
530
- try { return await fn(); }
537
+ // A long holder (the background preparation) refreshes the lock so it is
538
+ // never mistaken for an abandoned one at CACHE_LOCK_STALE_MS.
539
+ const touch = () => { try { const now = new Date(); fs.utimesSync(lockFile, now, now); } catch { /* best-effort */ } };
540
+ try { return await fn(touch); }
531
541
  finally {
532
542
  try { fs.closeSync(handle); } catch { /* */ }
533
543
  try { fs.unlinkSync(lockFile); } catch { /* stale cleanup handles leftovers */ }
@@ -658,7 +668,23 @@ function readCache(brainPath, accept) {
658
668
  };
659
669
  }
660
670
 
661
- async function vectorsForBrainUnlocked(pipe, brainPath, cards) {
671
+ // PREPARING A BRAIN NEVER HOLDS A QUESTION (1.97.0). Embedding every card of a
672
+ // fresh brain takes ~110 ms a card on a laptop CPU (measured 2026-10-11: 100
673
+ // cards 11 s, 3,670 cards 6.8 min), and the old writer did it inside the first
674
+ // question and saved nothing until the end — the agent's call waited minutes
675
+ // and an interrupted run lost everything. Now an interactive caller embeds at
676
+ // most INTERACTIVE_EMBED_BUDGET missing cards (a few seconds) and gets back
677
+ // what is ready; the rest is prepared by the detached, low-priority warm
678
+ // process (bin/klypix-semantic-warm.mjs, maxEmbed: Infinity), which saves as
679
+ // it goes so progress survives an interruption and readers see it.
680
+ export const INTERACTIVE_EMBED_BUDGET = 32;
681
+ const INTERACTIVE_LOCK_WAIT_MS = 3_000;
682
+ const PERSIST_EVERY_MS = 15_000;
683
+ const EMBED_CHUNK = 64;
684
+
685
+ async function vectorsForBrainUnlocked(pipe, brainPath, cards, opts = {}) {
686
+ const maxEmbed = Number.isFinite(opts.maxEmbed) ? Math.max(0, opts.maxEmbed) : (opts.maxEmbed === Infinity ? Infinity : INTERACTIVE_EMBED_BUDGET);
687
+ const stats = opts.stats && typeof opts.stats === 'object' ? opts.stats : {};
662
688
  const want = cards.filter((card) => card.type !== 'container' && (card.text || '').trim());
663
689
  // Question enrichment (1.77): the prompt/intent that produced a card is
664
690
  // appended to its EMBED input — the asker's own vocabulary bridging the
@@ -692,26 +718,51 @@ ${extra}` : base;
692
718
  if (entry.t !== textHashes.get(card.id)) { entry.t = textHashes.get(card.id); repaired++; }
693
719
  }
694
720
  if (repaired) dirty = true;
695
- const missing = want.filter((card) => cache.cards[card.id]?.h !== desiredHashes.get(card.id));
696
- if (missing.length) {
697
- let vectors;
698
- try { vectors = await embedTexts(pipe, missing.map((card) => embedInputFor(card))); }
699
- catch (error) {
700
- // The repair above needs no model, so it lands even when the embed does
701
- // not. Everything the merge refused goes back beside it: those vectors
702
- // are still on disk and a reader may still accept them through `t`.
703
- // The persisted file is a superset snapshot: it can contain refused
704
- // entries and entries for deleted cards. Readers reject them through
705
- // the acceptance rule, and the next write that is dirty for another
706
- // reason trims them.
707
- if (repaired) await persistCache(file, { ...cache, cards: { ...loaded.held, ...cache.cards } });
708
- throw error;
721
+ // Newest first: the cards people ask about most are ready soonest.
722
+ const missing = want
723
+ .filter((card) => cache.cards[card.id]?.h !== desiredHashes.get(card.id))
724
+ .sort((a, b) => (b.createdAt || 0) - (a.createdAt || 0));
725
+ const toEmbed = maxEmbed === Infinity ? missing : missing.slice(0, maxEmbed);
726
+ stats.wanted = want.length;
727
+ stats.embedded = 0;
728
+ stats.pending = missing.length - toEmbed.length;
729
+ if (toEmbed.length) {
730
+ let lastPersist = Date.now();
731
+ for (let offset = 0; offset < toEmbed.length; offset += EMBED_CHUNK) {
732
+ const chunk = toEmbed.slice(offset, offset + EMBED_CHUNK);
733
+ let vectors;
734
+ try { vectors = await embedTexts(pipe, chunk.map((card) => embedInputFor(card))); }
735
+ catch (error) {
736
+ // The repair above needs no model, so it lands even when the embed does
737
+ // not, and so do the chunks already embedded. Everything the merge
738
+ // refused goes back beside them: those vectors are still on disk and a
739
+ // reader may still accept them through `t`. The persisted file is a
740
+ // superset snapshot: it can contain refused entries and entries for
741
+ // deleted cards. Readers reject them through the acceptance rule, and
742
+ // the next write that is dirty for another reason trims them.
743
+ if (repaired || stats.embedded) await persistCache(file, { ...cache, cards: { ...loaded.held, ...cache.cards } });
744
+ stats.pending = missing.length - stats.embedded;
745
+ throw error;
746
+ }
747
+ chunk.forEach((card, index) => {
748
+ cache.cards[card.id] = { h: desiredHashes.get(card.id), t: textHashes.get(card.id), v: vectors[index] };
749
+ });
750
+ stats.embedded += chunk.length;
751
+ dirty = true;
752
+ if (typeof opts.touch === 'function') opts.touch();
753
+ const more = offset + EMBED_CHUNK < toEmbed.length;
754
+ // Save as it goes, so a long preparation survives an interruption and
755
+ // readers (cachedVectorsForBrain) see the progress.
756
+ if (more && Date.now() - lastPersist >= PERSIST_EVERY_MS) {
757
+ await persistCache(file, { ...cache, cards: { ...loaded.held, ...cache.cards } });
758
+ lastPersist = Date.now();
759
+ }
760
+ if (typeof opts.onProgress === 'function') {
761
+ try { opts.onProgress({ done: want.length - (missing.length - stats.embedded), total: want.length }); } catch { /* progress is advisory */ }
762
+ }
709
763
  }
710
- missing.forEach((card, index) => {
711
- cache.cards[card.id] = { h: desiredHashes.get(card.id), t: textHashes.get(card.id), v: vectors[index] };
712
- });
713
- dirty = true;
714
764
  }
765
+ stats.ready = want.length - stats.pending;
715
766
  const live = new Set(want.map((card) => card.id));
716
767
  for (const id of Object.keys(cache.cards)) {
717
768
  if (!live.has(id)) { delete cache.cards[id]; dirty = true; }
@@ -725,6 +776,72 @@ ${extra}` : base;
725
776
  return map;
726
777
  }
727
778
 
779
+ // ── Background preparation: start it, read how far it got ────────────────────
780
+ // startBrainPreparation spawns the detached, low-priority warm process for ONE
781
+ // brain (--only: a question never makes this machine prepare every registered
782
+ // project). At most once a minute per brain from this process; the warm
783
+ // process's own lock makes a second start a no-op. The warm writes its
784
+ // progress to WARM_STATUS_FILE while it runs; brainPreparationStatus reads it
785
+ // so an answer can say how far along the preparation is.
786
+ const WARM_STATUS_FILE = path.join(PB_DIR, '.semantic-update-warm.json');
787
+ const WARM_STALE_MS = 2 * 60_000;
788
+ const lastPreparationStart = new Map();
789
+
790
+ export function startBrainPreparation(brainPath, { spawnFn = spawn, now = Date.now() } = {}) {
791
+ if (semanticDisabledByPlugin()) return false;
792
+ const key = canonicalBrainKey(brainPath);
793
+ if (now - (lastPreparationStart.get(key) || 0) < 60_000) return false;
794
+ const here = path.dirname(fileURLToPath(import.meta.url));
795
+ const script = [path.join(here, 'klypix-semantic-warm.mjs'), path.join(here, '..', 'bin', 'klypix-semantic-warm.mjs')]
796
+ .find((file) => fs.existsSync(file));
797
+ if (!script) return false;
798
+ lastPreparationStart.set(key, now);
799
+ try {
800
+ const child = spawnFn(process.execPath, [script, '--brain-dir', PB_DIR, '--brain', path.resolve(brainPath), '--only'], {
801
+ detached: true,
802
+ stdio: 'ignore',
803
+ windowsHide: true,
804
+ env: { ...process.env },
805
+ });
806
+ child.unref?.();
807
+ return true;
808
+ } catch { return false; }
809
+ }
810
+
811
+ export function brainPreparationStatus(brainPath, { now = Date.now(), statusFile = WARM_STATUS_FILE } = {}) {
812
+ try {
813
+ const status = JSON.parse(fs.readFileSync(statusFile, 'utf8'));
814
+ if (status?.result !== 'preparing' || typeof status.brain !== 'string') return null;
815
+ if (canonicalBrainKey(status.brain) !== canonicalBrainKey(brainPath)) return null;
816
+ const updated = Date.parse(status.updatedAt || '');
817
+ if (!Number.isFinite(updated) || now - updated > WARM_STALE_MS) return null; // the run died
818
+ const done = Number(status.done) || 0, total = Number(status.total) || 0;
819
+ const started = Date.parse(status.startedAt || '');
820
+ const ratePerMs = done > 0 && Number.isFinite(started) && updated > started ? done / (updated - started) : null;
821
+ return { done, total, remainingMs: ratePerMs && total > done ? Math.round((total - done) / ratePerMs) : null };
822
+ } catch { return null; }
823
+ }
824
+
825
+ // One sentence an answer carries while a brain is being prepared. `stats`
826
+ // comes from vectorsForBrain; the live progress, when the warm reports it,
827
+ // is fresher.
828
+ export function preparationNotice(brainPath, stats, { usedMeaning = false, now = Date.now() } = {}) {
829
+ if (!stats || !(stats.pending > 0)) return null;
830
+ const live = brainPreparationStatus(brainPath, { now });
831
+ const total = live?.total || stats.wanted || 0;
832
+ const done = live ? live.done : (stats.ready || 0);
833
+ const minutes = live?.remainingMs != null
834
+ ? Math.max(1, Math.round(live.remainingMs / 60_000))
835
+ // ~45 ms a card measured one at a time; ~0.1 s leaves room for a busy
836
+ // machine and the low-priority run until the live rate is known.
837
+ : Math.max(1, Math.round(((total - done) * 0.1) / 60));
838
+ const n = (x) => Number(x).toLocaleString('en-US');
839
+ return `Getting this brain ready for search by meaning: ${n(done)} of ${n(total)} cards so far, about ${minutes} min left. `
840
+ + (usedMeaning
841
+ ? 'This answer used the cards that are ready plus keywords; it gets better as the rest finish.'
842
+ : 'This answer used keyword search; ask again when it is done for better matches.');
843
+ }
844
+
728
845
  // READ-ONLY card vectors from the warm cache: never loads the model, never
729
846
  // embeds a missing card, never writes. For fast paths (brain_sync task context)
730
847
  // that may USE card↔card similarity when it is already paid for — plan ↔ 🏁
@@ -771,16 +888,32 @@ export function cachedVectorsForBrain(brainPath, cards) {
771
888
  return map;
772
889
  }
773
890
 
774
- export async function vectorsForBrain(pipe, brainPath, cards) {
775
- if (!BOUNDED) return vectorsForBrainUnlocked(pipe, brainPath, cards);
891
+ // opts.maxEmbed: how many missing cards this call may embed — the interactive
892
+ // default is INTERACTIVE_EMBED_BUDGET; the background warm passes Infinity.
893
+ // opts.stats receives { wanted, ready, embedded, pending }; opts.onProgress
894
+ // gets { done, total } after each saved chunk.
895
+ export async function vectorsForBrain(pipe, brainPath, cards, opts = {}) {
896
+ if (!BOUNDED) return vectorsForBrainUnlocked(pipe, brainPath, cards, opts);
776
897
  const key = canonicalBrainKey(brainPath);
777
898
  const previous = vectorBuildTails.get(key) || Promise.resolve();
899
+ // A question never waits behind a preparation: it gets INTERACTIVE_LOCK_WAIT_MS,
900
+ // then reads what the preparation has saved so far (below).
901
+ const interactive = opts.maxEmbed !== Infinity;
778
902
  const run = previous.catch(() => {}).then(() => withCrossProcessCacheLock(
779
903
  brainPath,
780
- () => vectorsForBrainUnlocked(pipe, brainPath, cards),
904
+ (touch) => vectorsForBrainUnlocked(pipe, brainPath, cards, { ...opts, touch }),
905
+ interactive ? INTERACTIVE_LOCK_WAIT_MS : CACHE_LOCK_TIMEOUT_MS,
781
906
  ));
782
907
  vectorBuildTails.set(key, run);
783
908
  try { return await run; }
909
+ catch (error) {
910
+ if (!interactive || error?.code !== 'KLYPIX_SEMANTIC_CACHE_BUSY') throw error;
911
+ // The background preparation holds the lock: answer from what it has saved.
912
+ const map = cachedVectorsForBrain(brainPath, cards);
913
+ const wanted = cards.filter((card) => card && card.type !== 'container' && typeof card.text === 'string' && card.text.trim()).length;
914
+ if (opts.stats && typeof opts.stats === 'object') Object.assign(opts.stats, { wanted, ready: map.size, embedded: 0, pending: Math.max(0, wanted - map.size), busy: true });
915
+ return map;
916
+ }
784
917
  finally { if (vectorBuildTails.get(key) === run) vectorBuildTails.delete(key); }
785
918
  }
786
919