@inetafrica/open-claudia 2.4.0 → 2.5.0

package/CHANGELOG.md

@@ -1,5 +1,29 @@
 # Changelog
 
+## v2.5.0
+- **Context packs: skills become living topic documents (skills + memory merged).** Each pack is `~/.open-claudia/packs/<dir>/PACK.md` with four sections — Stance (how to think about the topic: user preferences, hard rules), Procedure (verified how-to, what skills used to be), State (where work stands now; replaced wholesale as things move), Journal (dated one-line log of past sessions, capped at 30 entries).
+  - **Pre-turn router** (`core/packs.js` + `system-prompt.js`): the incoming message is FTS5-matched against all packs (porter-stemmed, field-weighted — name/description/tags hits count double, threshold keeps a single incidental body word from dragging a pack in; BM25 was unusable because its IDF collapses on small corpora). Matched packs are injected into the per-turn context block, once per (channel, session, pack-version) so re-injection only happens when a pack actually changed. Rides the user message, not the system prompt, to preserve the prompt-cache prefix.
+  - **Post-turn reviewer** (`core/pack-review.js`): after every substantial turn (>400 chars) a haiku subagent reviews the exchange and returns a strict-JSON decision — update a pack (journal line + State rewrite), create one for a new durable topic, or nothing. The model returns JSON only; all file writes are applied by bot code, so the reviewer needs no tools or permissions. Active by default (most working turns produce at least a journal line), `PACK_REVIEW=off` to disable, `PACK_REVIEW_MODEL` to change model. Every applied mutation is announced in chat (one line) — no silent learning.
+  - **In-turn writes announced**: `Write`/`Edit` aimed at a `PACK.md` triggers `Updating pack: X` / `New pack: X` chat lines, same as skills did.
+  - **`open-claudia pack list|show|match|migrate|remove|reindex`**: inspect packs, debug the router (`match` shows scores), and `pack migrate` folds existing `~/.claude/skills` into packs (body → Procedure section) with originals backed up to `~/.open-claudia/backup/skills-pre-packs/` — run it once after upgrading.
+  - `spawnSubagent` learned `model` and `systemPrompt` overrides (used by the reviewer; also available to `open-claudia agent`).
+- **Entity memory (phase 2): contextual notes on people, places, projects, orgs and systems.** Each entity is a short file at `~/.open-claudia/entities/<slug>.md` — frontmatter (name, type, aliases, description) plus Notes (current truth, replaced wholesale) and Log (dated one-line observations, capped at 40).
+  - **Same router, second store** (`core/entities.js`): incoming messages are FTS5-matched against entities with the pack scoring scheme, except the strong fields are name/aliases — a single mention of "Emmanuel" injects the Emmanuel note, but one stray body word can't. Injected as a `## Known entities` block beside packs, deduped per (channel, session, entity-version).
+  - **Same reviewer, one call**: the post-turn haiku reviewer now also returns an `entities` array — it creates or updates entity notes (merge semantics: aliases union, Notes replace, Log appends) in the same pass that maintains packs, at no extra subagent cost. Entity changes are announced in chat like pack changes.
+  - **`open-claudia entity list|show|match|note|remove|reindex`**: inspect the store, debug the router, or jot a note manually (`entity note Emmanuel "owns inet-central org cleanup" --type person`).
+  - Direct `Write`/`Edit` to an entity file announces `Updating entity: X` / `New entity: X`, same as packs.
+  - This completes phase 2 of the packs system; "dream" consolidation (pack merging + skill trees + entity linking) is next.
+
+## v2.4.2
+- Fix replies losing everything written before tool calls. The backend's final `result` event carries only the LAST text segment of a turn, but the stream parser assigned it over the accumulated `assistantText` — so a turn shaped "long explanation → tool calls → short closing line" delivered only the closing line, which read as nonsense without its context. `evt.result` is now a fallback used only when nothing was accumulated (some Cursor turns). Fixed in both `runClaude` and the auxiliary runner.
+
+## v2.4.1
+- Fix Telegram messages occasionally arriving as raw HTML (`<b>`, `<code>` shown literally). Root cause: the Markdown→HTML pass could inject `<i>` tags *inside* model-authored `<code>` spans (two snake_case identifiers on one line pair their underscores as italics), producing unbalanced HTML that Telegram rejects — and the parse-failure fallback then resent the converted body verbatim, tags and all.
+  - `<code>`/`<pre>` spans are now stashed whole (content included) before Markdown conversion, so nothing can be injected inside them.
+  - The italic `_..._` rule only fires at word boundaries, so snake_case identifiers anywhere in the text can no longer pair up.
+  - Model-authored entities (`&lt;`, `&amp;`, `&#…;`) are preserved instead of double-escaped — `<code>&lt;name&gt;</code>` now renders as `<name>` instead of `&lt;name&gt;`.
+  - Last-resort fallback (send + edit) now strips tags and decodes entities (`htmlToPlain`) so a rejected message degrades to clean plain text, never raw markup.
+
 ## v2.4.0
 - **FTS5 transcript index (cross-session recall).** Project transcripts are now indexed in SQLite FTS5 via Node's built-in `node:sqlite` (no new dependency), turning "did we discuss X last week?" into a ~50ms ranked lookup instead of a linear grep over JSONL.
   - `core/transcript-index.js`: WAL-mode DB at `~/.open-claudia/transcripts/index.db` (0600), FTS5 table with porter tokenizer, per-file byte offsets for idempotent incremental indexing. Partial trailing lines wait for the next pass; a replaced/truncated transcript drops its stale rows and reindexes. Fail-soft: without `node:sqlite` every call no-ops and transcript-window remains the path.

package/bin/cli.js

@@ -274,6 +274,16 @@ switch (command) {
     break;
   }
 
+  case "pack": {
+    require("./pack").run(args.slice(1));
+    break;
+  }
+
+  case "entity": {
+    require("./entity").run(args.slice(1));
+    break;
+  }
+
   case "agent": {
     require("./agent").run(args.slice(1));
     break;
@@ -320,6 +330,8 @@ Memory tools:
                                              (alias: ts; --all for every project; --help for options)
   open-claudia transcript-window <pattern>   Search project transcript, show hits with context
                                              (alias: tw; --help for options)
+  open-claudia pack list|show|match|migrate  Context packs: living topic docs (skills + memory)
+  open-claudia entity list|show|match|note   Entity notes: people/places/projects memory
 
 Background work (only inside an active bot-spawned task):
   open-claudia schedule-wakeup <when> "<prompt>"   One-shot future wake-up; resumes session

package/bin/entity.js

@@ -0,0 +1,77 @@
+// CLI: inspect and manage entity notes (people/places/projects memory).
+//   open-claudia entity list
+//   open-claudia entity show <slug-or-name>
+//   open-claudia entity match "<text>"            — debug the router
+//   open-claudia entity note <name> "<text>" [--type person|place|project|org|system]
+//   open-claudia entity remove <slug-or-name>
+//   open-claudia entity reindex
+
+const fs = require("fs");
+const path = require("path");
+const entities = require("../core/entities");
+
+function run(args) {
+  const cmd = (args[0] || "list").toLowerCase();
+  const rest = args.slice(1);
+
+  switch (cmd) {
+    case "list": {
+      const all = entities.listEntities();
+      if (all.length === 0) return console.log(`No entities yet (${entities.ENTITIES_DIR}).`);
+      for (const e of all) {
+        const seen = e.last_seen ? ` last-seen ${e.last_seen.slice(0, 10)}` : "";
+        const aka = e.aliases.length ? ` aka ${e.aliases.join(", ")}` : "";
+        console.log(`${e.slug} (${e.type})${aka}${seen}\n    ${e.description}`);
+      }
+      break;
+    }
+
+    case "show": {
+      const e = entities.findEntity(rest.join(" "));
+      if (!e) { console.error(`No entity: ${rest.join(" ")}`); process.exitCode = 1; return; }
+      console.log(fs.readFileSync(path.join(entities.ENTITIES_DIR, e.slug + ".md"), "utf-8"));
+      break;
+    }
+
+    case "match": {
+      const text = rest.join(" ");
+      if (!text) { console.error('Usage: entity match "<text>"'); process.exitCode = 1; return; }
+      const hits = entities.matchEntities(text, { limit: 5 });
+      if (hits.length === 0) return console.log("No matching entities.");
+      for (const h of hits) console.log(`${h.score.toFixed(2)}  ${h.slug} — ${h.name}`);
+      break;
+    }
+
+    case "note": {
+      const typeIdx = rest.indexOf("--type");
+      let type = null;
+      if (typeIdx !== -1) {
+        type = rest[typeIdx + 1] || null;
+        rest.splice(typeIdx, 2);
+      }
+      const [name, ...noteParts] = rest;
+      const note = noteParts.join(" ");
+      if (!name || !note) { console.error('Usage: entity note <name> "<text>" [--type <type>]'); process.exitCode = 1; return; }
+      const { entity, created } = entities.upsertEntity({ name, type, log: note });
+      console.log(`${created ? "Created" : "Updated"} entity ${entity.slug} (${entity.type}).`);
+      break;
+    }
+
+    case "remove": {
+      const removed = entities.removeEntity(rest.join(" "));
+      if (!removed) { console.error(`No entity: ${rest.join(" ")}`); process.exitCode = 1; return; }
+      console.log(`Removed entity ${removed.slug}.`);
+      break;
+    }
+
+    case "reindex": {
+      console.log(entities.reindex() ? "Reindexed." : "Index unavailable (node:sqlite missing?).");
+      break;
+    }
+
+    default:
+      console.log("Usage: open-claudia entity [list|show <slug>|match \"<text>\"|note <name> \"<text>\"|remove <slug>|reindex]");
+  }
+}
+
+module.exports = { run };

package/bin/pack.js

@@ -0,0 +1,100 @@
+// CLI: inspect and manage context packs (living skills+memory docs).
+//   open-claudia pack list
+//   open-claudia pack show <dir>
+//   open-claudia pack match "<text>"        — debug the router
+//   open-claudia pack migrate               — fold ~/.claude/skills into packs
+//   open-claudia pack remove <dir>
+//   open-claudia pack reindex
+
+const fs = require("fs");
+const path = require("path");
+const packs = require("../core/packs");
+
+function run(args) {
+  const cmd = (args[0] || "list").toLowerCase();
+  const rest = args.slice(1);
+
+  switch (cmd) {
+    case "list": {
+      const all = packs.listPacks();
+      if (all.length === 0) return console.log(`No packs yet (${packs.PACKS_DIR}).`);
+      for (const p of all) {
+        const used = p.last_used ? ` last-used ${p.last_used.slice(0, 10)}` : "";
+        console.log(`${p.dir} — ${p.name}${p.tags.length ? ` [${p.tags.join(", ")}]` : ""}${used}\n    ${p.description}`);
+      }
+      break;
+    }
+
+    case "show": {
+      const p = packs.findPack(rest[0]);
+      if (!p) { console.error(`No pack: ${rest[0]}`); process.exitCode = 1; return; }
+      console.log(fs.readFileSync(path.join(packs.PACKS_DIR, p.dir, "PACK.md"), "utf-8"));
+      break;
+    }
+
+    case "match": {
+      const text = rest.join(" ");
+      if (!text) { console.error('Usage: pack match "<text>"'); process.exitCode = 1; return; }
+      const hits = packs.matchPacks(text, { limit: 5 });
+      if (hits.length === 0) return console.log("No matching packs.");
+      for (const h of hits) console.log(`${h.score.toFixed(2)}  ${h.dir} — ${h.name}`);
+      break;
+    }
+
+    case "remove": {
+      const removed = packs.removePack(rest[0]);
+      if (!removed) { console.error(`No pack: ${rest[0]}`); process.exitCode = 1; return; }
+      console.log(`Removed pack ${removed.dir}.`);
+      break;
+    }
+
+    case "reindex": {
+      console.log(packs.reindex() ? "Reindexed." : "Index unavailable (node:sqlite missing?).");
+      break;
+    }
+
+    case "migrate": {
+      const skills = require("../core/skills");
+      const backupDir = path.join(require("../config-dir"), "backup", "skills-pre-packs");
+      const all = skills.listSkills();
+      if (all.length === 0) return console.log("No skills to migrate.");
+      fs.mkdirSync(backupDir, { recursive: true });
+      let migrated = 0, skipped = 0;
+      for (const s of all) {
+        const dir = packs.slugify(s.dir);
+        if (packs.readPack(dir)) {
+          console.log(`skip ${s.dir} (pack exists)`);
+          skipped++;
+          continue;
+        }
+        const content = fs.readFileSync(s.file, "utf-8");
+        const body = content.replace(/^---\n[\s\S]*?\n---\n?/, "").trim();
+        packs.createPack({
+          dir,
+          name: s.name,
+          description: s.description,
+          tags: [],
+          procedure: body,
+          journal: "Migrated from ~/.claude/skills.",
+        });
+        // Fold: move the original skill out of ~/.claude/skills so the
+        // pack is the single source of truth. Backup kept, never deleted.
+        const skillDir = path.join(skills.SKILLS_DIR, s.dir);
+        const dest = path.join(backupDir, s.dir);
+        fs.rmSync(dest, { recursive: true, force: true });
+        fs.cpSync(skillDir, dest, { recursive: true });
+        fs.rmSync(skillDir, { recursive: true, force: true });
+        console.log(`migrated ${s.dir} -> pack ${dir}`);
+        migrated++;
+      }
+      packs.reindex();
+      console.log(`Done: ${migrated} migrated, ${skipped} skipped. Originals backed up at ${backupDir}.`);
+      break;
+    }
+
+    default:
+      console.log("Usage: open-claudia pack [list|show <dir>|match \"<text>\"|migrate|remove <dir>|reindex]");
+  }
+}
+
+module.exports = { run };

package/channels/telegram/adapter.js

@@ -10,7 +10,7 @@ const TelegramBot = require("node-telegram-bot-api");
 const { TEMP_DIR, FILES_DIR, CONFIG_DIR } = require("../../core/config");
 const { canonicalForChannel } = require("../../core/identity");
 const { portableToInlineKeyboard } = require("../types");
-const { telegramHtml } = require("./format");
+const { telegramHtml, htmlToPlain } = require("./format");
 
 class TelegramAdapter {
   constructor({ id = "telegram", token, ownerChatId, chatIds }) {
@@ -205,7 +205,7 @@ class TelegramAdapter {
 
   async send(channelId, text, opts = {}) {
     const o = {};
-    const body = opts.parseMode === "HTML" ? telegramHtml(text) : text;
+    let body = opts.parseMode === "HTML" ? telegramHtml(text) : text;
     if (opts.parseMode) o.parse_mode = opts.parseMode;
     const kb = this._normalizeKeyboard(opts.keyboard);
     if (kb) o.reply_markup = kb;
@@ -229,7 +229,9 @@ class TelegramAdapter {
           continue;
         }
         if (opts.parseMode && o.parse_mode) {
+          console.error("Send: HTML rejected, falling back to plain:", errMsg);
           delete o.parse_mode;
+          body = htmlToPlain(body);
           continue;
         }
         console.error("Send error:", errMsg);
@@ -255,7 +257,7 @@ class TelegramAdapter {
       if (errMsg.includes("message is not modified")) return;
       if (opts.parseMode && o.parse_mode) {
         delete o.parse_mode;
-        try { await this.bot.editMessageText(text, o); return; } catch (e2) {}
+        try { await this.bot.editMessageText(htmlToPlain(body), o); return; } catch (e2) {}
       }
       if (!errMsg.includes("message to edit not found")) console.error("Edit error:", errMsg);
     }

package/channels/telegram/format.js

@@ -14,6 +14,15 @@ function escapeHtml(text) {
     .replace(/>/g, ">");
 }
 
+// Like escapeHtml, but keeps entities the model already wrote (< & {)
+// instead of double-escaping them into visible "<" text.
+function escapeHtmlPreservingEntities(text) {
+  return String(text || "")
+    .replace(/&(?!(?:amp|lt|gt|quot|#\d+|#x[0-9a-fA-F]+);)/g, "&")
+    .replace(/</g, "&lt;")
+    .replace(/>/g, "&gt;");
+}
+
 function stripMarkdown(text) {
   return String(text || "")
     .replace(/[*_`~]/g, "")
@@ -48,6 +57,13 @@ function stashAllowedHtmlTags(text) {
   };
   let out = text;
 
+  // Stash whole model-authored <code>/<pre> spans, content included, so
+  // Markdown conversion can never inject tags inside them (e.g. two
+  // snake_case identifiers on one line becoming a stray <i> pair).
+  out = out.replace(/&lt;(code|pre)&gt;([\s\S]*?)&lt;\/\1&gt;/gi, (_, tag, inner) => {
+    return stash(`<${tag.toLowerCase()}>${inner}</${tag.toLowerCase()}>`);
+  });
+
   // Hide model-authored Telegram HTML tags from Markdown conversion. This keeps
   // underscores in hrefs such as wan2_2_video from becoming bogus <i> tags.
   for (const tag of ["b", "strong", "i", "em", "u", "s", "strike", "del", "code", "pre", "blockquote"]) {
@@ -83,8 +99,9 @@ function convertMarkdownToHtml(text) {
   out = out.replace(/\*\*([^*\n][\s\S]*?[^*\n])\*\*/g, "<b>$1</b>");
   out = out.replace(/(?<!\*)\*([^*\n]+)\*(?!\*)/g, "<b>$1</b>");
 
-  // Italic. Keep conservative to avoid mangling snake_case identifiers.
-  out = out.replace(/(?<!_)_([^_\n]+)_(?!_)/g, "<i>$1</i>");
+  // Italic. Only fire when the underscores sit at word boundaries, so
+  // snake_case identifiers (skill_manage ... skill_manage) never pair up.
+  out = out.replace(/(?<![A-Za-z0-9_])_([^_\n]+)_(?![A-Za-z0-9_])/g, "<i>$1</i>");
 
   // Strikethrough and Telegram spoiler syntax.
   out = out.replace(/~~([^~\n]+)~~/g, "<s>$1</s>");
@@ -95,7 +112,7 @@ function convertMarkdownToHtml(text) {
 
 function telegramHtml(text) {
   const { text: withoutCode, blocks } = stashCode(text);
-  let out = escapeHtml(withoutCode);
+  let out = escapeHtmlPreservingEntities(withoutCode);
   const { text: withoutHtmlTags, tags } = stashAllowedHtmlTags(out);
   out = convertMarkdownToHtml(withoutHtmlTags);
 
@@ -110,4 +127,16 @@ function telegramHtml(text) {
   return out;
 }
 
-module.exports = { escapeHtml, stripMarkdown, telegramHtml };
+// Last-resort plain text for when Telegram rejects the HTML: drop tags,
+// decode entities, so the user never sees raw <b>/<code> markup.
+function htmlToPlain(html) {
+  return String(html || "")
+    .replace(/<\/?[a-zA-Z][^>]*>/g, "")
+    .replace(/&lt;/g, "<")
+    .replace(/&gt;/g, ">")
+    .replace(/&quot;/g, '"')
+    .replace(/&#(\d+);/g, (_, n) => String.fromCodePoint(Number(n)))
+    .replace(/&amp;/g, "&");
+}
+
+module.exports = { escapeHtml, stripMarkdown, telegramHtml, htmlToPlain };

package/core/entities.js

@@ -0,0 +1,329 @@
+// Entity memory: short living notes on the people, places, projects,
+// orgs and systems that come up in conversation (a Honcho-style store,
+// our own version). Each entity is ~/.open-claudia/entities/<slug>.md
+// with frontmatter and two sections:
+//   ## Notes — current truth about the entity (replaces wholesale)
+//   ## Log   — dated one-line observations, newest last
+// The pre-turn router injects matched entities alongside context packs;
+// the post-turn reviewer extracts/updates them after each turn.
+
+const fs = require("fs");
+const path = require("path");
+
+let DatabaseSync = null;
+try { ({ DatabaseSync } = require("node:sqlite")); } catch (e) { /* old node — matching disabled */ }
+
+const CONFIG_DIR = require("../config-dir");
+const ENTITIES_DIR = process.env.ENTITIES_DIR ? path.resolve(process.env.ENTITIES_DIR) : path.join(CONFIG_DIR, "entities");
+
+const TYPES = ["person", "place", "project", "org", "system", "thing"];
+const MAX_LOG_ENTRIES = 40;
+
+function ensureDir() {
+  fs.mkdirSync(ENTITIES_DIR, { recursive: true, mode: 0o700 });
+}
+
+function slugify(name) {
+  return String(name || "").toLowerCase().trim()
+    .replace(/[^a-z0-9]+/g, "-").replace(/^-+|-+$/g, "").slice(0, 60);
+}
+
+function normalizeType(type) {
+  const t = String(type || "").toLowerCase().trim();
+  return TYPES.includes(t) ? t : "thing";
+}
+
+function parseFrontmatter(content) {
+  const m = String(content || "").match(/^---\n([\s\S]*?)\n---/);
+  if (!m) return { fm: {}, body: String(content || "") };
+  const fm = {};
+  for (const line of m[1].split("\n")) {
+    const kv = line.match(/^([a-z_]+):\s*(.*)$/);
+    if (kv) fm[kv[1]] = kv[2].trim().replace(/^["']|["']$/g, "");
+  }
+  return { fm, body: content.slice(m[0].length).replace(/^\n+/, "") };
+}
+
+function parseSections(body) {
+  const out = { Notes: "", Log: "" };
+  let current = null;
+  for (const line of String(body || "").split("\n")) {
+    const h = line.match(/^##\s+(Notes|Log)\s*$/i);
+    if (h) {
+      current = h[1].toLowerCase() === "notes" ? "Notes" : "Log";
+      continue;
+    }
+    if (current) out[current] += line + "\n";
+  }
+  out.Notes = out.Notes.trim();
+  out.Log = out.Log.trim();
+  return out;
+}
+
+function serialize(ent) {
+  return [
+    "---",
+    `name: ${ent.name}`,
+    `type: ${ent.type}`,
+    `aliases: ${(ent.aliases || []).join(", ")}`,
+    `description: ${ent.description || ""}`,
+    `created: ${ent.created || new Date().toISOString()}`,
+    `updated: ${ent.updated || new Date().toISOString()}`,
+    `last_seen: ${ent.last_seen || ""}`,
+    "---",
+    "",
+    `## Notes`,
+    "",
+    (ent.sections?.Notes || "").trim(),
+    "",
+    `## Log`,
+    "",
+    (ent.sections?.Log || "").trim(),
+    "",
+  ].join("\n");
+}
+
+function entityFile(slug) {
+  return path.join(ENTITIES_DIR, slug + ".md");
+}
+
+function readEntity(slug) {
+  let content;
+  try { content = fs.readFileSync(entityFile(slug), "utf-8"); }
+  catch (e) { return null; }
+  const { fm, body } = parseFrontmatter(content);
+  return {
+    slug,
+    name: fm.name || slug,
+    type: normalizeType(fm.type),
+    aliases: (fm.aliases || "").split(",").map((a) => a.trim()).filter(Boolean),
+    description: fm.description || "",
+    created: fm.created || "",
+    updated: fm.updated || "",
+    last_seen: fm.last_seen || "",
+    sections: parseSections(body),
+  };
+}
+
+function listEntities() {
+  let names;
+  try { names = fs.readdirSync(ENTITIES_DIR); } catch (e) { return []; }
+  const out = [];
+  for (const f of names) {
+    if (!f.endsWith(".md") || f.startsWith(".")) continue;
+    const e = readEntity(f.slice(0, -3));
+    if (e) out.push(e);
+  }
+  out.sort((a, b) => a.slug.localeCompare(b.slug));
+  return out;
+}
+
+function findEntity(nameOrSlug) {
+  const needle = String(nameOrSlug || "").trim().toLowerCase();
+  if (!needle) return null;
+  const direct = readEntity(slugify(needle));
+  if (direct) return direct;
+  return listEntities().find((e) =>
+    e.slug === needle ||
+    e.name.toLowerCase() === needle ||
+    e.aliases.some((a) => a.toLowerCase() === needle)
+  ) || null;
+}
+
+function writeEntity(ent) {
+  ensureDir();
+  fs.writeFileSync(entityFile(ent.slug), serialize(ent), { mode: 0o600 });
+  markIndexDirty();
+  return ent.slug;
+}
+
+function today() {
+  return new Date().toISOString().slice(0, 10);
+}
+
+// Create-or-merge. Notes replace (current truth); log lines append
+// (capped); aliases union; description/type update when supplied.
+function upsertEntity({ name, type, aliases, description, notes, log } = {}) {
+  const cleanName = String(name || "").trim();
+  if (!cleanName) throw new Error("entity needs a name");
+  const existing = findEntity(cleanName);
+  const ent = existing || {
+    slug: slugify(cleanName),
+    name: cleanName,
+    type: normalizeType(type),
+    aliases: [],
+    description: "",
+    created: new Date().toISOString(),
+    sections: { Notes: "", Log: "" },
+  };
+  if (!ent.slug) throw new Error("entity needs a name");
+  ent.updated = new Date().toISOString();
+  if (type) ent.type = normalizeType(type);
+  if (description) ent.description = String(description).trim();
+  for (const a of [].concat(aliases || [])) {
+    const alias = String(a).trim();
+    if (alias && alias.toLowerCase() !== ent.name.toLowerCase() &&
+        !ent.aliases.some((x) => x.toLowerCase() === alias.toLowerCase())) {
+      ent.aliases.push(alias);
+    }
+  }
+  ent.aliases = ent.aliases.slice(0, 8);
+  if (typeof notes === "string" && notes.trim()) ent.sections.Notes = notes.trim();
+  if (typeof log === "string" && log.trim()) {
+    const entries = ent.sections.Log.split("\n").filter((l) => l.trim());
+    entries.push(`- [${today()}] ${log.trim().replace(/\n+/g, " ")}`);
+    ent.sections.Log = entries.slice(-MAX_LOG_ENTRIES).join("\n");
+  }
+  writeEntity(ent);
+  return { entity: ent, created: !existing };
+}
+
+function removeEntity(nameOrSlug) {
+  const ent = findEntity(nameOrSlug);
+  if (!ent) return null;
+  fs.rmSync(entityFile(ent.slug), { force: true });
+  markIndexDirty();
+  return ent;
+}
+
+function touchSeen(slugs) {
+  const now = new Date().toISOString();
+  for (const slug of [].concat(slugs || [])) {
+    const ent = readEntity(slug);
+    if (!ent) continue;
+    ent.last_seen = now;
+    try { writeEntity(ent); } catch (e) {}
+  }
+}
+
+// Recognise a Write/Edit aimed at an entity file (for chat announcements).
+function entityNameFromPath(filePath) {
+  if (!filePath) return null;
+  const resolved = path.resolve(String(filePath));
+  const rel = path.relative(ENTITIES_DIR, resolved);
+  if (rel.startsWith("..") || path.isAbsolute(rel)) return null;
+  if (rel.includes(path.sep) || !rel.endsWith(".md")) return null;
+  return rel.slice(0, -3);
+}
+
+// ---------------------------------------------------------------------------
+// FTS matching — same field-weighted per-term scoring as packs (BM25 is
+// unusable on a small corpus): a hit on name/aliases counts 2, anything
+// else 1. Threshold 2 means a single name mention is enough to inject,
+// but one stray body word is not.
+
+let _db = null;
+let _indexedAt = 0;
+let _dirty = true;
+
+function markIndexDirty() { _dirty = true; }
+
+function openDb() {
+  if (!DatabaseSync) return null;
+  if (_db) return _db;
+  try {
+    ensureDir();
+    const dbPath = path.join(ENTITIES_DIR, "index.db");
+    const db = new DatabaseSync(dbPath);
+    try { fs.chmodSync(dbPath, 0o600); } catch (e) {}
+    db.exec("PRAGMA journal_mode=WAL");
+    db.exec("PRAGMA busy_timeout=3000");
+    db.exec(`CREATE VIRTUAL TABLE IF NOT EXISTS entities USING fts5(
+      name, aliases, description, content, slug UNINDEXED,
+      tokenize='porter unicode61'
+    )`);
+    _db = db;
+    return db;
+  } catch (e) {
+    return null;
+  }
+}
+
+function latestMtime() {
+  let latest = 0;
+  let names = [];
+  try { names = fs.readdirSync(ENTITIES_DIR); } catch (e) { return 0; }
+  for (const f of names) {
+    if (!f.endsWith(".md")) continue;
+    try {
+      const m = fs.statSync(path.join(ENTITIES_DIR, f)).mtimeMs;
+      if (m > latest) latest = m;
+    } catch (e) {}
+  }
+  return latest;
+}
+
+function reindex() {
+  const db = openDb();
+  if (!db) return false;
+  const all = listEntities();
+  db.exec("BEGIN");
+  try {
+    db.exec("DELETE FROM entities");
+    const insert = db.prepare("INSERT INTO entities (name, aliases, description, content, slug) VALUES (?, ?, ?, ?, ?)");
+    for (const e of all) {
+      insert.run(e.name, e.aliases.join(" "), e.description, [e.sections.Notes, e.sections.Log].join("\n"), e.slug);
+    }
+    db.exec("COMMIT");
+  } catch (e) {
+    try { db.exec("ROLLBACK"); } catch (e2) {}
+    return false;
+  }
+  _dirty = false;
+  _indexedAt = Date.now();
+  return true;
+}
+
+function ensureIndex() {
+  if (_dirty || latestMtime() > _indexedAt) reindex();
+}
+
+const STOPWORDS = new Set(("a an and are as at be but by can did do for from had has have how i if in is it its " +
+  "me my no not of on or our so that the their then there this to up us was we what when where which who why " +
+  "will with you your yes yeah ok okay please thanks just like dont don't im i'm its it's").split(" "));
+
+function queryTerms(text) {
+  const seen = new Set();
+  const terms = [];
+  for (const raw of String(text || "").toLowerCase().split(/[^a-z0-9_.-]+/)) {
+    const t = raw.replace(/^[.-]+|[.-]+$/g, "");
+    if (t.length < 3 || STOPWORDS.has(t) || seen.has(t)) continue;
+    seen.add(t);
+    terms.push(t);
+    if (terms.length >= 40) break;
+  }
+  return terms;
+}
+
+function matchEntities(text, { limit = 4, threshold = null } = {}) {
+  const db = openDb();
+  if (!db) return [];
+  ensureIndex();
+  const terms = queryTerms(text);
+  if (terms.length === 0) return [];
+  const minScore = threshold ?? Number(process.env.ENTITY_MATCH_THRESHOLD || 2);
+  const scores = new Map();
+  let stmt;
+  try { stmt = db.prepare("SELECT slug FROM entities WHERE entities MATCH ?"); } catch (e) { return []; }
+  for (const t of terms) {
+    const quoted = `"${t.replace(/"/g, '""')}"`;
+    let strong = [];
+    let any = [];
+    try { strong = stmt.all(`{name aliases} : ${quoted}`); } catch (e) {}
+    try { any = stmt.all(quoted); } catch (e) {}
+    const strongSet = new Set(strong.map((r) => r.slug));
+    for (const r of any) scores.set(r.slug, (scores.get(r.slug) || 0) + (strongSet.has(r.slug) ? 2 : 1));
+  }
+  const names = new Map(listEntities().map((e) => [e.slug, e.name]));
+  return [...scores.entries()]
+    .filter(([, s]) => s >= minScore)
+    .sort((a, b) => b[1] - a[1])
+    .slice(0, Math.max(1, Math.min(10, limit)))
+    .map(([slug, score]) => ({ slug, name: names.get(slug) || slug, score }));
+}
+
+module.exports = {
+  ENTITIES_DIR, TYPES, slugify,
+  listEntities, findEntity, readEntity, writeEntity, upsertEntity, removeEntity,
+  touchSeen, entityNameFromPath, matchEntities, reindex, markIndexDirty,
+};