@theglitchking/semantic-memory 1.0.0

package/docs/smart-middle-activation.md

@@ -0,0 +1,97 @@
+---
+title: Smart-middle corpus activation (preview, ships in 2.0.0b)
+tier: standard
+domains:
+  - architecture
+  - configuration
+status: draft
+last_updated: 2026-05-07
+version: 1.0.0
+audience:
+  - developers
+purpose: Document the first-run heuristic semantic-memory uses to auto-detect which corpora to enable. Designed to give zero-friction onboarding for solo users and full-stack visibility for teams.
+---
+
+# Smart-middle activation
+
+> **Status: planned for 2.0.0b.** This document describes the auto-detection heuristic that runs the first time `semantic-memory` is invoked in a project (i.e., when `<project-root>/.semantic/corpora.json` doesn't exist). The implementation lands with the multi-corpus refactor.
+
+## The three approaches considered
+
+When this design was being scoped, three default-corpora strategies were on the table:
+
+1. **All-on**: enable all six corpora (vault, code, plans, docs, research, project-map) by default. Heavy footprint; lots of empty `no_index` returns for projects that don't have all six markers.
+2. **Vault-only**: enable just `vault` by default; require explicit opt-in for everything else. Lightest footprint; requires manual config for any non-trivial project.
+3. **Smart-middle**: scan the project for known markers (`.planning/`, `src/`, etc.) and auto-enable corresponding corpora. Friction-free onboarding without paying for empty corpora.
+
+The user picked smart-middle. This doc specifies the resolution.
+
+## Marker → corpus mapping
+
+| Marker (file or directory) | Activates | Default `root` | Default `chunker` |
+|---|---|---|---|
+| `.claude/.vault/` | `vault` | `./.claude/.vault` | `markdown` |
+| `.planning/.meta/workspace.json` with `mode: "lg"` | `plans` | `./.planning` | `markdown` |
+| `.planning/` (without lg-mode workspace.json) | (skipped — sm-mode plans aren't multi-corpus material) | — | — |
+| `.documentation/` | `docs` | `./.documentation` | `markdown` |
+| `.research/` | `research` | `./.research` | `markdown` |
+| `.babel-fish/01-vocabulary.md` | `project-map` (with glossary extractor enabled) | `./.babel-fish` | `markdown` |
+| Detected source dir with TS/JS files | `code` | `<detected>` | `tree-sitter-ts` |
+
+### Source-dir detection for `code`
+
+The `code` corpus is the only one that requires content sniffing rather than a single marker. The detection logic:
+
+1. If `package.json` exists at project root: read `main` / `module` / `exports` to find canonical source paths
+2. If `tsconfig.json` exists: read `compilerOptions.rootDir` and `include`
+3. If `src/` directory exists at project root with at least one `.ts` / `.tsx` / `.js` / `.jsx` file: use `src/`
+4. If multiple candidates: pick the one with the most TS/JS files; tie-break by depth (shallower wins)
+5. If none found: skip the `code` corpus (don't activate)
+
+When activated, the default `glob` is `**/*.{ts,tsx,js,jsx}` (Python deferred to a future minor; see Phase 3.2.0 in the meta-plan).
+
+## Decision banner
+
+After scanning, semantic-memory prints a decision banner to stderr explaining what it activated and why:
+
+```
+[semantic-memory] First-run corpus activation:
+[semantic-memory]   ✓ vault         from .claude/.vault/  (markdown chunker)
+[semantic-memory]   ✓ plans         from .planning/        (markdown chunker, lg-mode workspace detected)
+[semantic-memory]   ✓ code          from src/              (tree-sitter-ts chunker, 47 files)
+[semantic-memory]   - docs          (no .documentation/ found)
+[semantic-memory]   - research      (no .research/ found)
+[semantic-memory]   - project-map   (no .babel-fish/01-vocabulary.md found)
+[semantic-memory]
+[semantic-memory] Generated .semantic/corpora.json. Edit to override.
+```
+
+Banner intent: NO MAGIC. The user sees what got turned on, what didn't, and why. Editing `corpora.json` directly is always the override path.
+
+## Override and force-rerun
+
+After the first run, `corpora.json` is the source of truth. To re-run smart-middle activation (after adding new markers, for example), delete `<project-root>/.semantic/corpora.json` and re-invoke semantic-memory.
+
+A `--re-detect` CLI flag may be added in a later minor for the case where the user wants to merge new markers into an existing `corpora.json` without losing manual customizations. (TBD — design open.)
+
+## Why this design
+
+- **Solo devs win**: a fresh project with just `.claude/.vault/` gets the legacy semantic-sidekick experience. No new config to learn.
+- **Teams win**: a project with `.planning/` (lg mode) + `src/` + `.babel-fish/` gets all three corpora wired automatically — no need to read a multi-page config doc to get value out of the new architecture.
+- **Banner is the trust mechanism**: users see what got auto-activated and can disable corpora they don't want by editing `corpora.json`. The decision is never invisible.
+- **Backwards compatibility is free**: projects without ANY of the markers fall back to legacy single-vault behavior (which is what semantic-sidekick 0.2.5 does today). No `corpora.json` is even written in that case.
+
+## Inter-corpus interactions (when multiple corpora are active)
+
+- **Unified knowledge graph**: nodes from all active corpora share one graph. A `related_docs:` edge in a vault note can resolve to a code symbol in the `code` corpus.
+- **Cross-corpus search**: `search_all` queries all enabled corpora and merges results.
+- **Drift detection**: registered iff `code` + at least one doc-bearing corpus (vault, plans, docs) are both active.
+- **Translation**: registered iff `project-map` corpus is active AND its `glossary.json` was extracted successfully.
+
+These behaviors are detailed in `docs/corpora-json.md` (the schema reference).
+
+## See also
+
+- `docs/corpora-json.md` — `corpora.json` schema reference
+- `docs/regression-snapshot.md` — the gate that protects existing single-vault behavior during the refactor
+- `~/workspace/the-glitch-kingdom/persistent-planning/.planning/layered-planning-with-mcp-and-hewtd-frontmatter/task_plan.md` — meta-plan; smart-middle activation is decision C3

package/hooks/hooks.json

@@ -0,0 +1,39 @@
+{
+  "description": "semantic-sidekick activation + mode hooks. SessionStart reconciles .mcp.json + resets mode + preloads vault context. UserPromptSubmit injects vault search results (mode-aware, fingerprinted). Stop emits mode-specific transition-capture prompts (research→synthesis, outage→postmortem, vault-first→generic capture).",
+  "hooks": {
+    "SessionStart": [
+      {
+        "hooks": [
+          {
+            "type": "command",
+            "command": "node \"${CLAUDE_PLUGIN_ROOT}/hooks/session-start.js\""
+          },
+          {
+            "type": "command",
+            "command": "node \"${CLAUDE_PLUGIN_ROOT}/hooks/vault-context.js\""
+          }
+        ]
+      }
+    ],
+    "UserPromptSubmit": [
+      {
+        "hooks": [
+          {
+            "type": "command",
+            "command": "node \"${CLAUDE_PLUGIN_ROOT}/hooks/vault-context.js\""
+          }
+        ]
+      }
+    ],
+    "Stop": [
+      {
+        "hooks": [
+          {
+            "type": "command",
+            "command": "node \"${CLAUDE_PLUGIN_ROOT}/hooks/vault-context.js\""
+          }
+        ]
+      }
+    ]
+  }
+}

package/hooks/reconcile.js

@@ -0,0 +1,171 @@
+// Plugin-specific SessionStart reconciliation for semantic-sidekick.
+//
+// Responsibilities (all conservative — the user's config always wins):
+//   1. Ensure <project>/.claude/.vault exists.
+//   2. If .mcp.json is missing a "semantic-vault" entry AND the package is
+//      installed locally, add one in the node-against-node_modules form.
+//      Existing entries (any shape) are preserved — if you want to rewrite
+//      a stale entry, run `npx --no @theglitchking/semantic-sidekick
+//      normalize-config`.
+//   3. If hit-em-with-the-docs is enabled AND ./.documentation exists,
+//      conditionally add a read-only "semantic-sidekick" entry pointed at
+//      ./.documentation (same preserve-existing rule). Remove it ONLY if
+//      the existing entry matches a shape we recognize as ours.
+//
+// Any write is preceded by a backup to .mcp.json.bak. Never writes the
+// `npx @latest` form — that shape is fragile (npx cache corruption causes
+// ERR_MODULE_NOT_FOUND) and was the root cause of the 0.10.0 bugfix.
+
+import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
+import { homedir } from "node:os";
+import { dirname, join, relative } from "node:path";
+
+const PKG = "@theglitchking/semantic-sidekick";
+const VAULT_KEY = "semantic-vault";
+const DOCS_KEY = "semantic-sidekick";
+
+function readJson(path, fallback = null) {
+  try {
+    const raw = readFileSync(path, "utf8");
+    return raw.trim() ? JSON.parse(raw) : fallback;
+  } catch {
+    return fallback;
+  }
+}
+
+function hewtdEnabled(projectRoot) {
+  // Check user-level AND project-level settings. Plugin enablement is per-scope,
+  // and hewtd might be enabled at either. Project-scope was previously ignored,
+  // which caused the docs MCP to get silently removed by the reconcile pass on
+  // every SessionStart for users who'd enabled hewtd only per-project.
+  const candidates = [
+    join(homedir(), ".claude", "settings.json"),
+    join(projectRoot, ".claude", "settings.json"),
+    join(projectRoot, ".claude", "settings.local.json"),
+  ];
+  for (const path of candidates) {
+    const settings = readJson(path);
+    const enabled = settings?.enabledPlugins || {};
+    for (const key of Object.keys(enabled)) {
+      if (key.startsWith("hit-em-with-the-docs@") && enabled[key] === true) {
+        return true;
+      }
+    }
+  }
+  return false;
+}
+
+/** Absolute path to the locally-installed bin script, or null. */
+export function findLocalBin(projectRoot) {
+  const p = join(projectRoot, "node_modules", "@theglitchking", "semantic-sidekick", "bin", "semantic-sidekick");
+  return existsSync(p) ? p : null;
+}
+
+/** Returns the arg form that goes into .mcp.json for this project. */
+function binArg(projectRoot) {
+  const abs = findLocalBin(projectRoot);
+  if (!abs) return null;
+  const rel = relative(projectRoot, abs);
+  // Always use ./-prefixed relative form so it's machine-portable within the repo.
+  return rel.startsWith("..") ? abs : `./${rel}`;
+}
+
+function buildEntry(binArgStr, notesPath, extraArgs = []) {
+  return {
+    type: "stdio",
+    command: "node",
+    args: [binArgStr, "--notes", notesPath, ...extraArgs],
+  };
+}
+
+/**
+ * Detect an entry as "one of ours" so we can safely remove it when the
+ * condition that created it no longer holds. Matches both shapes we've ever
+ * written: the legacy npx-form and the new node-against-node_modules form.
+ */
+function isOurEntry(entry, notesPath) {
+  if (!entry || typeof entry !== "object") return false;
+  const cmd = entry.command;
+  const args = Array.isArray(entry.args) ? entry.args : [];
+  const argStr = args.map((a) => String(a)).join(" ");
+
+  // Legacy npx form we shipped in 0.8.0 and prior SessionStart hooks.
+  if (cmd === "npx" && argStr.includes(PKG) && argStr.includes(notesPath)) return true;
+  // Current node-against-node_modules form (0.10.0+).
+  if (
+    cmd === "node" &&
+    args.some((a) => typeof a === "string" && a.includes("node_modules/@theglitchking/semantic-sidekick")) &&
+    argStr.includes(notesPath)
+  ) {
+    return true;
+  }
+  return false;
+}
+
+function backup(mcpPath) {
+  try {
+    if (existsSync(mcpPath)) {
+      writeFileSync(join(dirname(mcpPath), ".mcp.json.bak"), readFileSync(mcpPath, "utf8"));
+    }
+  } catch {}
+}
+
+export function reconcile(projectRoot) {
+  const vaultDir = join(projectRoot, ".claude", ".vault");
+  const docsDir = join(projectRoot, ".documentation");
+  const mcpPath = join(projectRoot, ".mcp.json");
+
+  try { mkdirSync(vaultDir, { recursive: true }); } catch {}
+
+  const docsWired = hewtdEnabled(projectRoot) && existsSync(docsDir);
+
+  let data = { mcpServers: {} };
+  if (existsSync(mcpPath)) {
+    const parsed = readJson(mcpPath, null);
+    if (parsed === null) {
+      let raw = "";
+      try { raw = readFileSync(mcpPath, "utf8"); } catch {}
+      if (raw.trim()) {
+        process.stderr.write(`semantic-sidekick hook: could not parse ${mcpPath}; leaving untouched\n`);
+        return;
+      }
+    } else if (parsed && typeof parsed === "object") {
+      data = parsed;
+      if (!data.mcpServers || typeof data.mcpServers !== "object") {
+        data.mcpServers = {};
+      }
+    }
+  }
+
+  const before = JSON.stringify(data);
+
+  const bin = binArg(projectRoot);
+
+  // VAULT entry: create if missing AND we have a local install. Never
+  // overwrite an existing entry (even if it looks "wrong" — the user
+  // controls their .mcp.json).
+  if (!data.mcpServers[VAULT_KEY]) {
+    if (bin) {
+      data.mcpServers[VAULT_KEY] = buildEntry(bin, "./.claude/.vault");
+    }
+  }
+
+  // DOCS entry: same preserve-existing rule. Conditional on hewtd +
+  // .documentation. When removing, only remove shapes we recognize as ours.
+  if (docsWired) {
+    if (!data.mcpServers[DOCS_KEY]) {
+      if (bin) {
+        data.mcpServers[DOCS_KEY] = buildEntry(bin, "./.documentation", ["--read-only"]);
+      }
+    }
+  } else if (data.mcpServers[DOCS_KEY] && isOurEntry(data.mcpServers[DOCS_KEY], ".documentation")) {
+    delete data.mcpServers[DOCS_KEY];
+  }
+
+  const after = JSON.stringify(data);
+  if (after === before) return;
+
+  backup(mcpPath);
+  mkdirSync(dirname(mcpPath), { recursive: true });
+  writeFileSync(mcpPath, JSON.stringify(data, null, 2) + "\n");
+}

package/hooks/session-start.js

@@ -0,0 +1,81 @@
+#!/usr/bin/env node
+// semantic-sidekick SessionStart hook.
+//
+// Critical path (zero external deps): run reconcile.js to wire .mcp.json +
+// create .claude/.vault. This ALWAYS runs, even if the plugin is installed
+// in a location where @theglitchking/claude-plugin-runtime isn't resolvable
+// (which happens when Claude Code extracts the plugin tarball without running
+// npm install afterward — the default for `/plugin install`).
+//
+// Optional: if claude-plugin-runtime IS resolvable, delegate to
+// runSessionStart which also does policy-gated auto-update checks. Otherwise
+// just emit the minimum-valid SessionStart response.
+//
+// Either way, this hook MUST NOT throw — a broken SessionStart hook is worse
+// than an idle one.
+
+import { reconcile } from "./reconcile.js";
+
+async function main() {
+  const projectRoot = process.env.CLAUDE_PROJECT_DIR || process.cwd();
+
+  // 1. Reconcile — no-deps, always runs.
+  try {
+    process.chdir(projectRoot);
+  } catch {
+    // If we can't chdir, emit an empty response and bail. Don't let this kill
+    // the session.
+    emit();
+    return;
+  }
+
+  try {
+    await reconcile(projectRoot);
+  } catch (err) {
+    process.stderr.write(`[semantic-sidekick] reconcile failed: ${err?.message || err}\n`);
+  }
+
+  // 2. Optional runtime-backed update check. Skip silently if the runtime
+  //    isn't installed in this plugin's extraction dir (the common case when
+  //    installed via /plugin install without a subsequent npm install).
+  try {
+    const mod = await import("@theglitchking/claude-plugin-runtime");
+    if (typeof mod.runSessionStart === "function") {
+      // runSessionStart emits its own response. It also re-runs reconcile —
+      // that's idempotent (reconcile writes only what's missing), so the
+      // double-invocation is harmless. We return early to avoid double-emit.
+      await mod.runSessionStart({
+        packageName: "@theglitchking/semantic-sidekick",
+        pluginName: "semantic-sidekick",
+        configFile: "semantic-sidekick.json",
+        reconcile,
+      });
+      return;
+    }
+  } catch (err) {
+    // ERR_MODULE_NOT_FOUND is the expected case when the runtime isn't
+    // resolvable. Any other error gets logged but doesn't block the session.
+    const msg = String(err?.message || err);
+    if (!msg.includes("Cannot find package") && !msg.includes("ERR_MODULE_NOT_FOUND")) {
+      process.stderr.write(`[semantic-sidekick] runtime import failed: ${msg}\n`);
+    }
+  }
+
+  // Fallback emit — minimum-valid SessionStart response with empty context.
+  // vault-context.js (the other SessionStart hook) provides the actual preload.
+  emit();
+}
+
+function emit(additionalContext) {
+  const out = { hookSpecificOutput: { hookEventName: "SessionStart" } };
+  if (additionalContext && String(additionalContext).trim()) {
+    out.hookSpecificOutput.additionalContext = String(additionalContext);
+  }
+  process.stdout.write(JSON.stringify(out) + "\n");
+}
+
+main().catch((err) => {
+  process.stderr.write(`[semantic-sidekick] session-start.js fatal: ${err?.message || err}\n`);
+  // Still emit — never block the session.
+  emit();
+});