@lh8ppl/claude-memory-kit 0.2.4 → 0.3.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (43) hide show
  1. package/README.md +16 -10
  2. package/bin/cmk-capture-prompt.mjs +21 -1
  3. package/package.json +2 -1
  4. package/src/audit-log.mjs +1 -0
  5. package/src/auto-drain.mjs +17 -1
  6. package/src/auto-extract.mjs +72 -16
  7. package/src/auto-persona.mjs +86 -1
  8. package/src/capture-prompt.mjs +34 -1
  9. package/src/capture-turn.mjs +64 -6
  10. package/src/config-core.mjs +161 -0
  11. package/src/conflict-queue.mjs +20 -3
  12. package/src/content-hash.mjs +30 -0
  13. package/src/doctor.mjs +62 -3
  14. package/src/forget.mjs +13 -0
  15. package/src/frontmatter.mjs +4 -1
  16. package/src/import-anthropic-memory.mjs +25 -1
  17. package/src/import-claude-md.mjs +333 -0
  18. package/src/index-db.mjs +39 -0
  19. package/src/index-rebuild.mjs +48 -4
  20. package/src/index.mjs +10 -0
  21. package/src/inject-context.mjs +179 -7
  22. package/src/install.mjs +180 -1
  23. package/src/mcp-server.mjs +63 -8
  24. package/src/memory-health.mjs +229 -0
  25. package/src/memory-write.mjs +32 -10
  26. package/src/merge-facts.mjs +12 -0
  27. package/src/native-binding.mjs +142 -0
  28. package/src/poison-guard.mjs +55 -0
  29. package/src/provenance.mjs +4 -0
  30. package/src/remember-core.mjs +53 -8
  31. package/src/repair.mjs +20 -3
  32. package/src/result-shapes.mjs +1 -1
  33. package/src/scratchpad.mjs +5 -3
  34. package/src/search.mjs +96 -9
  35. package/src/semantic-backend.mjs +599 -0
  36. package/src/settings-hooks.mjs +4 -1
  37. package/src/subcommands.mjs +359 -42
  38. package/src/transcript-index.mjs +165 -0
  39. package/src/turn-tools.mjs +179 -0
  40. package/src/write-fact.mjs +34 -3
  41. package/template/.claude/skills/memory-search/SKILL.md +86 -0
  42. package/template/.gitattributes.fragment +16 -0
  43. package/template/CLAUDE.md.template +3 -1
package/README.md CHANGED
@@ -5,13 +5,15 @@
5
5
  ## What it does
6
6
 
7
7
  - **Cross-project persona — the wedge (v0.2)** — when you state how you work *everywhere* ("always use uv, never pip", "from now on run the linter before committing"), the per-turn auto-extract promotes it into your **user tier** (`~/.claude-memory-kit/`) **that turn**. So a brand-new project **cold-opens already knowing your style** — layered structure, your tooling, your testing discipline — with no hand-curation and no waiting. Carry it between your own machines with `cmk persona export`/`import`, or pin a single fact across projects with `cmk lessons promote`.
8
- - **Frozen snapshot at session start** — MEMORY.md + USER.md + SOUL.md + INDEX.md + today's session log inject once at the first tool call, so Claude sees your context every session without you re-telling it.
8
+ - **Frozen snapshot at session start** — MEMORY.md + USER.md + SOUL.md + INDEX.md + today's session log inject once at the first tool call, so Claude sees your context every session without you re-telling it. The snapshot opens with an **authority instruction** ("when injected memory contradicts your assumptions, injected memory wins"), so the agent leads with its memory instead of re-deriving answers from the code.
9
9
  - **Auto-extract on every assistant turn** — a background `claude --print` subagent reads each turn and saves durable facts to memory. Durable project knowledge (setup/config, conventions, workflows, tool quirks) becomes a **rich Why/How fact file** (structured + searchable); lighter signals stay terse `MEMORY.md` bullets. Runs automatically, so the rich tier survives even when the model uses Claude Code's built-in memory instead. No manual writes needed.
10
+ - **Claude knows WHEN to recall** — the auto-invoked `memory-search` skill fires on "what did we decide about X" / "have we seen this error before" and searches the deep archive in a forked side-context, returning a curated citation-backed summary. Read-only by contract.
10
11
  - **Explicit capture when you want it** — say "remember this" / "from now on" / "we decided" / "forget X" (the `memory-write` skill), or run `cmk remember "<fact>"`. Both dedup, screen for secrets, abstract machine paths to `~`, and write silently. For backtick/quote-heavy rich facts, capture them shell-safe as JSON: `cmk remember --from-file fact.json` (or `--json` from stdin) — content never touches the shell.
11
- - **Search + MCP — Claude runs every memory op for you, in conversation** — `cmk search "<term>"` (keyword/hybrid over facts + scratchpads). `cmk install` registers the kit's **MCP server**, so Claude can do the whole memory surface as tools without you ever typing `cmk`: capture (`mk_remember`, rich Why/How too), recall (`mk_search` / `mk_get` / `mk_timeline` / `mk_cite`), adjust trust (`mk_trust`), promote a fact across projects (`mk_lessons_promote`), forget (`mk_forget` — previews first, then deletes on confirm), and clear the review/conflict queues (`mk_queue_list` / `mk_queue_resolve`). The tools are allow-listed on install, so they run prompt-free.
12
+ - **Search + MCP — Claude runs every memory op for you, in conversation** — `cmk search "<term>"` (keyword over facts + scratchpads; with the optional local embedder, **semantic + hybrid recall**: ask in your own words and get the fact even with zero keyword overlap — measured R@5 0.941 / paraphrase 1.000 on the kit's benchmark, no API calls). `cmk install` registers the kit's **MCP server**, so Claude can do the whole memory surface as tools without you ever typing `cmk`: capture (`mk_remember`, rich Why/How too), recall (`mk_search` / `mk_get` / `mk_timeline` / `mk_cite`), adjust trust (`mk_trust`), promote a fact across projects (`mk_lessons_promote`), forget (`mk_forget` — previews first, then deletes on confirm), and clear the review/conflict queues (`mk_queue_list` / `mk_queue_resolve`). The tools are allow-listed on install, so they run prompt-free.
12
13
  - **Bounded by compression** — session → daily → weekly Haiku rollups (cron or lazy-on-read) keep the snapshot small as history grows. The session-buffer rollup self-heals at session start too, so memory stays bounded even if you never cleanly close the window.
14
+ - **Don't start empty — import the rules you already own** — `cmk import-claude-md` parses an existing `CLAUDE.md` / `.cursorrules` / `AGENTS.md` into typed, searchable facts through the same safe write path (secret screening, sanitization, dedup), with provenance back to source file + line. `--dry-run` previews first.
13
15
  - **Per-project, in-repo** — `context/` lives inside your project and travels with `git clone`. Each project keeps its own memory.
14
- - **9 health checks** — `cmk doctor` validates install, hook wiring, distill freshness, INDEX consistency, cron registration, and stale locks.
16
+ - **8 health checks** — `cmk doctor` validates hook wiring, distill freshness, transcript firing, INDEX consistency, cron registration, native-memory coexistence, stale locks, and native-binding health (npm 12 readiness) — each failure with a repair command.
15
17
 
16
18
  ## Install — pick ONE route
17
19
 
@@ -22,11 +24,14 @@ Each route is complete on its own. **Don't run both** — they wire the same hoo
22
24
  ```bash
23
25
  npm install -g @lh8ppl/claude-memory-kit
24
26
  cd ~/my-project
25
- cmk install # scaffolds context/ + the memory-write skill AND wires the lifecycle hooks into .claude/settings.json
27
+ cmk install # scaffolds context/ + the memory-write + memory-search skills AND wires the lifecycle hooks into .claude/settings.json
28
+ cmk install --with-semantic # (optional) local semantic recall — one-time ~260 MB, search defaults to hybrid
29
+ cmk register-crons # (optional) scheduled background compression — otherwise self-heals lazily
30
+ cmk import-claude-md --yes # (optional) seed memory from an existing CLAUDE.md / .cursorrules (--dry-run previews)
26
31
  cmk doctor # verify, then restart Claude Code
27
32
  ```
28
33
 
29
- `cmk install` is a complete entry point: it scaffolds `context/`, drops the `memory-write` skill into `.claude/skills/` (committed — travels with `git clone`), and writes the 5 lifecycle hooks (PATH-resolved, cross-OS) into the project's `.claude/settings.json`. It also **registers the kit's MCP server** in `.mcp.json` and allow-lists its tools (`mcp__cmk__*`) in `.claude/settings.json`, so Claude can drive memory as tools with no per-call prompt. No separate `/plugin` step needed. Use `cmk install --no-hooks` to skip the hooks + MCP wiring (scaffold-only).
34
+ `cmk install` is a complete entry point: it scaffolds `context/`, drops the `memory-write` + `memory-search` skills into `.claude/skills/` (committed — travels with `git clone`), and writes the 5 lifecycle hooks (PATH-resolved, cross-OS) into the project's `.claude/settings.json`. It also **registers the kit's MCP server** in `.mcp.json` and allow-lists its tools (`mcp__cmk__*`) in `.claude/settings.json`, so Claude can drive memory as tools with no per-call prompt, and writes a `.gitattributes` block pinning committed memory to LF (so a Windows clone can't mangle line endings — your memory stays readable cross-platform). No separate `/plugin` step needed. Use `cmk install --no-hooks` to skip the hooks + MCP wiring (scaffold-only).
30
35
 
31
36
  > Installing the package globally adds the `cmk` CLI **and** the installer. It's the `cmk install` *subcommand* that wires the hooks — not the bare `npm install`.
32
37
 
@@ -39,7 +44,7 @@ Inside Claude Code:
39
44
  /plugin install claude-memory-kit
40
45
  ```
41
46
 
42
- Then say *"bootstrap the memory system"* to scaffold this project's `context/`. The plugin bundles the hooks + the `bootstrap` and `memory-write` skills, so it's complete without the npm CLI (add the CLI later only if you want `cmk search` / `cmk doctor` / cron).
47
+ Then say *"bootstrap the memory system"* to scaffold this project's `context/`. The plugin bundles the hooks + the `bootstrap`, `memory-write`, and `memory-search` skills, so it's complete without the npm CLI (add the CLI later only if you want `cmk search` / `cmk doctor` / cron).
43
48
 
44
49
  ## CLI
45
50
 
@@ -47,10 +52,10 @@ Most-used commands (full list via `cmk --help`):
47
52
 
48
53
  | Command | Purpose |
49
54
  | --- | --- |
50
- | `cmk install` | Scaffold `context/` + the `memory-write` skill + `.gitignore` + CLAUDE.md block + wire hooks (`--no-hooks` for scaffold-only) |
51
- | `cmk doctor` | Run HC-1..HC-9 health checks, surface repair commands |
55
+ | `cmk install` | Scaffold `context/` + the `memory-write`/`memory-search` skills + `.gitignore` + CLAUDE.md block + wire hooks (`--no-hooks` for scaffold-only) |
56
+ | `cmk doctor` | Run HC-1..HC-8 health checks, surface repair commands |
52
57
  | `cmk repair --hooks` / `--locks` / `--index` / `--all` | Idempotent self-repair |
53
- | `cmk search "<query>" [--mode keyword\|semantic\|hybrid]` | Search accumulated memory (keyword default) |
58
+ | `cmk search "<query>" [--mode keyword\|semantic\|hybrid] [--scope facts\|transcripts]` | Search memory — by meaning with the embedder (hybrid default after `--with-semantic`); `--scope transcripts` = the raw session record |
54
59
  | `cmk get <id…>` / `cmk timeline <id>` / `cmk cite <id>` / `cmk recent-activity` | Read the index back — full fact bodies + provenance, sequential context around an observation, a canonical citation link, recent changes (the CLI side of the `mk_*` MCP read tools) |
55
60
  | `cmk roll --scope now\|today\|recent` | Manually trigger a compression pipeline |
56
61
  | `cmk register-crons [--dry-run] [--unregister]` | Register daily + weekly jobs with cron / launchd / Task Scheduler |
@@ -60,12 +65,13 @@ Most-used commands (full list via `cmk --help`):
60
65
  | `cmk persona generate` | Run cross-project persona synthesis on demand (instead of waiting for the weekly pass) |
61
66
  | `cmk persona export <file>` / `import <file>` | Carry your cross-project persona (the user tier) to another of **your** machines — export to one portable bundle, import on the other (overwrites with backup + rollback). The persona stays private (never committed to a project) |
62
67
  | `cmk import-anthropic-memory [--dry-run] [--yes]` | Merge bullets from Anthropic's native auto-memory into MEMORY.md |
68
+ | `cmk import-claude-md [file] [--dry-run] [--yes]` | Onboard from the rules you already own — parse an existing `CLAUDE.md` / `.cursorrules` / `AGENTS.md` into typed facts through the safe write path (Poison_Guard + sanitization + dedup) |
63
69
 
64
70
  ## Requirements
65
71
 
66
72
  - Node.js ≥ 20
67
73
  - Claude Code (for the hook-driven auto-memory loop)
68
- - Optional: Python 3.12+ for Layer 5b semantic search (deferred to a later release; keyword search ships today)
74
+ - Optional: `cmk install --with-semantic` for semantic/hybrid recall (installs the local `@huggingface/transformers` embedder, ~260 MB once — no API, no Python)
69
75
 
70
76
  ## Three-tier model
71
77
 
@@ -25,9 +25,10 @@ const modulePath = join(__dirname, '..', 'src', 'capture-prompt.mjs');
25
25
 
26
26
  let readHookStdin;
27
27
  let capturePrompt;
28
+ let buildMemoryHint;
28
29
  try {
29
30
  ({ readHookStdin } = await import(pathToFileURL(readHookStdinPath).href));
30
- ({ capturePrompt } = await import(pathToFileURL(modulePath).href));
31
+ ({ capturePrompt, buildMemoryHint } = await import(pathToFileURL(modulePath).href));
31
32
  } catch (err) {
32
33
  process.stderr.write(
33
34
  `cmk-capture-prompt: failed to load modules: ${err?.message ?? err}\n`,
@@ -61,5 +62,24 @@ try {
61
62
  );
62
63
  }
63
64
 
65
+ // Task 75.2 — emit the "memory available" recall nudge as additionalContext
66
+ // (the MODEL-facing UserPromptSubmit field per Anthropic's hooks doc;
67
+ // systemMessage is user-display). Best-effort: a hint failure must never
68
+ // break the capture protocol.
69
+ try {
70
+ const hint = buildMemoryHint({ projectRoot: process.cwd(), prompt: payload?.prompt });
71
+ if (hint) {
72
+ process.stdout.write(
73
+ JSON.stringify({
74
+ continue: true,
75
+ hookSpecificOutput: { hookEventName: 'UserPromptSubmit', additionalContext: hint },
76
+ }),
77
+ );
78
+ process.exit(0);
79
+ }
80
+ } catch (err) {
81
+ process.stderr.write(`cmk-capture-prompt: hint failed: ${err?.message ?? err}\n`);
82
+ }
83
+
64
84
  emitContinue();
65
85
  process.exit(0);
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@lh8ppl/claude-memory-kit",
3
- "version": "0.2.4",
3
+ "version": "0.3.1",
4
4
  "description": "cmk — the CLI for claude-memory-kit. Per-project, in-repo memory system for Claude Code.",
5
5
  "type": "module",
6
6
  "bin": {
@@ -35,6 +35,7 @@
35
35
  "chokidar": "^5.0.0",
36
36
  "commander": "^15.0.0",
37
37
  "js-yaml": "^4.1.0",
38
+ "sqlite-vec": "^0.1.9",
38
39
  "zod": "^4.4.3"
39
40
  },
40
41
  "engines": {
package/src/audit-log.mjs CHANGED
@@ -33,6 +33,7 @@ export const REASON_CODES = Object.freeze({
33
33
  FACT_CREATED: 'fact-created', // writeFact: a new fact file was written (Task 123.A — the default create audit; callers emitting a richer code opt out via audit:false)
34
34
  DUPLICATE: 'duplicate', // writeFact: same path + same id
35
35
  DUPLICATE_ELSEWHERE: 'duplicate-elsewhere', // writeFact: different path + same id
36
+ INDEX_REBUILD_FAILED: 'index-rebuild-failed', // writeFact: the fact landed on disk but the best-effort INDEX.md rebuild threw (e.g. a detached auto-extract child killed mid-rebuild). Surfaces what was previously a SILENTLY swallowed catch (D-152) so a lagging committed INDEX is diagnosable; the next reindex/cmk reindex self-heals.
36
37
  USER_REQUESTED: 'user-requested', // forget: user-initiated tombstone
37
38
  CURATED_MERGE: 'curated-merge', // mergeFacts: explicit merge of A + B → C
38
39
  SCRATCHPAD_APPEND: 'scratchpad-append', // scratchpad: appendScratchpadBullet (Task 12)
@@ -22,6 +22,7 @@
22
22
 
23
23
  import { resolveReviewQueue } from './review-queue.mjs';
24
24
  import { resolveConflictQueue } from './conflict-queue.mjs';
25
+ import { resolvePersonaReviewQueue } from './auto-persona.mjs';
25
26
  import { mergeFacts } from './merge-facts.mjs';
26
27
 
27
28
  // Stateless optimistic resolvers (no per-entry judgement — that's the point).
@@ -55,5 +56,20 @@ export async function autoDrainQueues({ tier = 'P', projectRoot, userDir, scratc
55
56
  mergeFn: mergeFacts, // never invoked under KEEP_OLD; wired for correctness
56
57
  });
57
58
 
58
- return { review, conflict };
59
+ // Persona-review queue (D-154): the medium-confidence cross-project persona
60
+ // candidates that were ROUTED here with the promise of an auto-drain that was
61
+ // never implemented — so they stranded (the v0.3.1 cold-open found the user's
62
+ // architecture philosophy stuck here). Drain it optimistically like the review
63
+ // queue (sync; userDir-scoped so it runs regardless of `tier`). Best-effort: a
64
+ // persona-drain hiccup must not fail the project-tier review/conflict drain.
65
+ let persona = { promoted: 0, drained: 0, queuePath: null };
66
+ if (userDir) {
67
+ try {
68
+ persona = resolvePersonaReviewQueue({ userDir });
69
+ } catch {
70
+ // best-effort; the queue file survives for the next pass
71
+ }
72
+ }
73
+
74
+ return { review, conflict, persona };
59
75
  }
@@ -48,8 +48,8 @@ import {
48
48
  appendFileSync,
49
49
  } from 'node:fs';
50
50
  import { join, dirname } from 'node:path';
51
- import { createHash } from 'node:crypto';
52
51
  import { generateId } from '@lh8ppl/cmk-canonicalize';
52
+ import { hashContent } from './content-hash.mjs';
53
53
  import { memoryWrite } from './memory-write.mjs';
54
54
  import { writeFact } from './write-fact.mjs';
55
55
  import { buildRichFactBody, slugifyFact } from './rich-fact.mjs';
@@ -196,7 +196,14 @@ function extractRetainSegments(text) {
196
196
 
197
197
  // --- Dedup context --------------------------------------------------
198
198
 
199
- function readLastEntryFromNowMd(projectRoot) {
199
+ // Task 132 (D-122): exported for capture-turn, which snapshots the dedup
200
+ // context BEFORE appending the current turn to now.md and passes it here
201
+ // inside the turn file. Reading now.md from THIS module (after the append)
202
+ // was the self-poisoning bug: the "last entry" was the very turn being
203
+ // extracted, so Haiku was told "do not re-emit facts already here" about
204
+ // its own input → nothing_durable on every organic turn (since Task 87;
205
+ // live A/B repro 2026-06-11, cut-gate8).
206
+ export function readLastEntryFromNowMd(projectRoot) {
200
207
  const nowMd = join(projectRoot, ...NOW_MD_RELATIVE);
201
208
  if (!existsSync(nowMd)) return '';
202
209
  let body;
@@ -221,29 +228,38 @@ function readLastEntryFromNowMd(projectRoot) {
221
228
  // --- Turn-file parser (bi-turn) -------------------------------------
222
229
 
223
230
  // Parse the temp-file format Task 21's capture-turn writes:
231
+ // DEDUP_CONTEXT: ← Task 132: optional; the last now.md entry
232
+ // <previous entry> as it stood BEFORE the current turn was
233
+ // appended (capture-turn snapshots it)
224
234
  // USER_TURN:
225
235
  // <user body>
226
236
  //
227
237
  // ASSISTANT_TURN:
228
238
  // <assistant body>
229
- // Either section may be empty. If no USER_TURN: / ASSISTANT_TURN:
239
+ // Any section may be empty. If no USER_TURN: / ASSISTANT_TURN:
230
240
  // markers are present, fall back to "the whole file is the assistant
231
241
  // turn" so old-format temp files (pre-2026-05-26) still work — useful
232
242
  // when running auto-extract against a turn buffer that pre-dates this
233
- // amendment (unlikely after the rollout, but defensive).
243
+ // amendment (unlikely after the rollout, but defensive). A missing
244
+ // DEDUP_CONTEXT marker means NO dedup section — never a now.md re-read
245
+ // (that re-read was the Task 132 self-poisoning bug).
234
246
  const USER_TURN_RE = /^[ \t]*USER_TURN:\s*\n([\s\S]*?)(?=^[ \t]*ASSISTANT_TURN:|\Z)/m;
235
247
  const ASSISTANT_TURN_RE = /^[ \t]*ASSISTANT_TURN:\s*\n([\s\S]*)$/m;
248
+ const DEDUP_CONTEXT_RE =
249
+ /^[ \t]*DEDUP_CONTEXT:\s*\n([\s\S]*?)(?=^[ \t]*USER_TURN:|^[ \t]*ASSISTANT_TURN:)/m;
236
250
 
237
251
  function parseTurnFile(rawTurn) {
252
+ const dedupMatch = rawTurn.match(DEDUP_CONTEXT_RE);
238
253
  const userMatch = rawTurn.match(USER_TURN_RE);
239
254
  const assistantMatch = rawTurn.match(ASSISTANT_TURN_RE);
240
255
  if (!userMatch && !assistantMatch) {
241
256
  // Old-format / unlabeled — treat whole content as assistant.
242
- return { userTurn: '', assistantTurn: rawTurn.trim() };
257
+ return { userTurn: '', assistantTurn: rawTurn.trim(), dedupContext: '' };
243
258
  }
244
259
  return {
245
260
  userTurn: (userMatch?.[1] ?? '').trim(),
246
261
  assistantTurn: (assistantMatch?.[1] ?? '').trim(),
262
+ dedupContext: (dedupMatch?.[1] ?? '').trim(),
247
263
  };
248
264
  }
249
265
 
@@ -441,10 +457,11 @@ function parseRichFactBlock(blockLines) {
441
457
  // Exported for direct unit-testing (cli-rich-fact.test.js) — the BEGIN_FACT
442
458
  // format is the extraction prompt's contract, pinned independently of a live
443
459
  // Haiku call.
444
- export function parseRichFacts(haikuOutput) {
460
+ export function parseRichFacts(haikuOutput, { onClipped } = {}) {
445
461
  if (!haikuOutput || typeof haikuOutput !== 'string') return [];
446
462
  const lines = haikuOutput.split('\n');
447
463
  const facts = [];
464
+ let clipped = 0;
448
465
  let i = 0;
449
466
  while (i < lines.length) {
450
467
  if (lines[i].trim().toUpperCase() !== 'BEGIN_FACT') {
@@ -455,19 +472,36 @@ export function parseRichFacts(haikuOutput) {
455
472
  // don't let it swallow the following block), or end-of-output.
456
473
  i++;
457
474
  const blockLines = [];
475
+ let terminated = false;
458
476
  while (i < lines.length) {
459
477
  const marker = lines[i].trim().toUpperCase();
460
478
  if (marker === 'END_FACT') {
479
+ terminated = true;
461
480
  i++;
462
481
  break;
463
482
  }
464
- if (marker === 'BEGIN_FACT') break; // close here; leave i for the outer loop
483
+ if (marker === 'BEGIN_FACT') {
484
+ // Implicit close by the next block — the body up to here is whole.
485
+ terminated = true;
486
+ break;
487
+ }
465
488
  blockLines.push(lines[i]);
466
489
  i++;
467
490
  }
491
+ // Task 136 (D-124): a block that ran into END-OF-OUTPUT without any
492
+ // terminator is the signature of the compressor's maxOutputBytes slice
493
+ // cutting Haiku's reply mid-fact. Writing it would persist a corrupted
494
+ // stub (cut-gate9's P-BaTM3L42: body "The `clau"). Drop it — losing the
495
+ // clipped fact beats storing a mangled one; the count reaches
496
+ // extract.log via onClipped for observability.
497
+ if (!terminated) {
498
+ clipped++;
499
+ continue;
500
+ }
468
501
  const fact = parseRichFactBlock(blockLines);
469
502
  if (fact) facts.push(fact);
470
503
  }
504
+ if (clipped > 0 && typeof onClipped === 'function') onClipped(clipped);
471
505
  return facts;
472
506
  }
473
507
 
@@ -629,10 +663,9 @@ function routeRichFact({ candidate, projectRoot, ts }) {
629
663
  sourceFile: 'auto-extract',
630
664
  sourceLine: 1,
631
665
  // Content fingerprint for the provenance field — NOT a security context.
632
- // Matches the kit's sha1-of-content convention (write-fact.mjs caller in
633
- // subcommands.runRememberRich, memory-write.mjs); writeFact dedups by the
634
- // content-addressed id, this is just source_sha1. // NOSONAR
635
- sourceSha1: createHash('sha1').update(body).digest('hex'), // NOSONAR
666
+ // Routes through the shared hashContent (SHA-256, D-149); writeFact dedups
667
+ // by the content-addressed id, this is just source_sha1 metadata.
668
+ sourceSha1: hashContent(body),
636
669
  createdAt: ts,
637
670
  projectRoot,
638
671
  });
@@ -710,6 +743,14 @@ export async function runAutoExtract({
710
743
  duration_ms: Date.now() - t0,
711
744
  };
712
745
  const logPath = writeExtractLogEntry({ projectRoot, ts, entry });
746
+ // Task 132: this early return bypasses the lock-held finally that
747
+ // normally unlinks the turn file — clean it up here or every
748
+ // concurrent rejection leaks one .extract-*.tmp (cut-gate8 finding).
749
+ try {
750
+ if (turnFile && existsSync(turnFile)) unlinkSync(turnFile);
751
+ } catch {
752
+ // best-effort; the sweepStaleTurnFiles janitor catches stragglers
753
+ }
713
754
  return {
714
755
  action: 'concurrent',
715
756
  error_category: ERROR_CATEGORIES.CONCURRENT_RUN,
@@ -764,10 +805,12 @@ export async function runAutoExtract({
764
805
  // override.
765
806
  const retainSegments = extractRetainSegments(rawTurn);
766
807
  const sanitized = stripNoiseTags(rawTurn);
767
- const { userTurn, assistantTurn } = parseTurnFile(sanitized);
808
+ const { userTurn, assistantTurn, dedupContext } = parseTurnFile(sanitized);
768
809
 
769
- // 3. Build prompt with dedup context (last `## ` entry from now.md).
770
- const dedupContext = readLastEntryFromNowMd(projectRoot);
810
+ // 3. Dedup context comes from the TURN FILE (Task 132) capture-turn
811
+ // snapshotted the last now.md entry BEFORE appending the current
812
+ // turn. Re-reading now.md here would see the current turn as
813
+ // "already captured" and suppress every extraction (D-122).
771
814
  const instructions = buildExtractionInstructions();
772
815
  const promptBody = buildExtractionPrompt({
773
816
  userTurn,
@@ -788,7 +831,12 @@ export async function runAutoExtract({
788
831
  haikuResult = await haikuBackend.compress({
789
832
  input: promptBody,
790
833
  instructions,
791
- maxOutputBytes: 2000,
834
+ // Task 136 (D-124): 8192, was 2000. A dense turn legitimately yields
835
+ // 3-4 rich facts (~700-900 bytes each) + terse/persona lines — the old
836
+ // cap clipped the 3rd fact mid-word and a corrupted stub reached disk
837
+ // (cut-gate9). The parser now also DROPS clipped trailing blocks; the
838
+ // raised budget makes the drop rare instead of routine.
839
+ maxOutputBytes: 8192,
792
840
  preserveCitationIds: false,
793
841
  // 90s, not 25s: the real `claude --print` extraction (full turn +
794
842
  // instructions) consistently exceeded a 25s ceiling on a live machine
@@ -854,7 +902,12 @@ export async function runAutoExtract({
854
902
  // SAME Haiku output may carry BEGIN_FACT blocks (durable project KNOWLEDGE)
855
903
  // alongside the terse TRUST_ lines; route them to the fact store via
856
904
  // writeFact (richer + searchable). No second LLM call — same outputText.
857
- const richFacts = parseRichFacts(haikuResult.outputText);
905
+ let clippedFactsDropped = 0;
906
+ const richFacts = parseRichFacts(haikuResult.outputText, {
907
+ onClipped: (n) => {
908
+ clippedFactsDropped = n;
909
+ },
910
+ });
858
911
  // XOR safety net: the prompt asks Haiku to emit a fact as EITHER a rich
859
912
  // block OR a terse line, never both. If it does both for the same fact, the
860
913
  // rich block wins — drop any terse candidate whose canonical id matches a
@@ -1069,6 +1122,9 @@ export async function runAutoExtract({
1069
1122
  ...baseEntry,
1070
1123
  ...personaLogFields,
1071
1124
  rich_facts_written: richFactsWritten,
1125
+ // Task 136: only present when the output cap clipped a trailing fact —
1126
+ // the signal to consider raising the budget further.
1127
+ ...(clippedFactsDropped > 0 ? { clipped_facts_dropped: clippedFactsDropped } : {}),
1072
1128
  success: true,
1073
1129
  observation_count,
1074
1130
  duration_ms: Date.now() - t0,
@@ -38,7 +38,7 @@
38
38
  // promotion primitive), audit-log, result-shapes, cooldown, compressor.
39
39
  // Per design §16.16 + §6.2 (conflict) + §6.8 (auto-drain) + §8.3 + tasks.md 45.
40
40
 
41
- import { readFileSync, appendFileSync, mkdirSync, existsSync, readdirSync } from 'node:fs';
41
+ import { readFileSync, writeFileSync, appendFileSync, mkdirSync, existsSync, readdirSync } from 'node:fs';
42
42
  import { join, dirname } from 'node:path';
43
43
  import { generateId } from '@lh8ppl/cmk-canonicalize';
44
44
  import { ERROR_CATEGORIES, errorResult } from './result-shapes.mjs';
@@ -425,6 +425,91 @@ export function appendPersonaReviewQueue({ userDir, entries, now }) {
425
425
  return queuePath;
426
426
  }
427
427
 
428
+ // Parse persona-review.md back into candidate objects. The queue lines are
429
+ // - (U-XXXXXXXX) [TARGET § SECTION] <text>
430
+ // <!-- target: TARGET, section: SECTION, confidence: C, reason: ..., ... -->
431
+ // The HTML comment is authoritative for target/section/confidence (the bracket
432
+ // prefix is human-readable redundancy); fall back to the bracket if absent.
433
+ // ReDoS-safe: NEGATED character classes (not lazy `.+?...+?` pairs) so the regex
434
+ // is linear — each group matches "anything but the delimiter that ends it", which
435
+ // cannot backtrack across that delimiter (the canonicalize stripTrailingPunct
436
+ // lesson — Task 140 / D-143 — applied at write).
437
+ const PERSONA_QUEUE_LINE_RE = /^- \([UPL]-[^)]+\)\s+\[([^§\]]+)§([^\]]+)\]\s+(\S.*)$/;
438
+ // No `\s*` sits adjacent to a `[^,]+` capture: `\s*` and `[^,]+` both match a
439
+ // space, and that overlap is the super-linear-backtracking ambiguity Sonar
440
+ // flags. Each value is captured by `[^,]+` (which absorbs leading/trailing
441
+ // space — we `.trim()` below), with the `,` and label as fixed delimiters.
442
+ const PERSONA_QUEUE_META_RE = /target:([^,]+),\s*section:([^,]+),\s*confidence:\s*(\w+)/;
443
+ export function parsePersonaReviewQueue(text) {
444
+ const lines = (text ?? '').split(/\r?\n/);
445
+ const candidates = [];
446
+ for (let i = 0; i < lines.length; i++) {
447
+ const m = PERSONA_QUEUE_LINE_RE.exec(lines[i].trim());
448
+ if (!m) continue;
449
+ let [, target, section, body] = m;
450
+ let confidence = 'medium';
451
+ const meta = PERSONA_QUEUE_META_RE.exec(lines[i + 1] ?? '');
452
+ if (meta) {
453
+ target = meta[1].trim();
454
+ section = meta[2].trim();
455
+ confidence = meta[3].trim().toLowerCase();
456
+ }
457
+ candidates.push({ target: target.trim(), section: section.trim(), confidence, text: body.trim() });
458
+ }
459
+ return candidates;
460
+ }
461
+
462
+ /**
463
+ * Auto-drain the persona-review queue (the down-payment for Task 151 / D-154).
464
+ *
465
+ * The medium-confidence persona candidates were ROUTED to persona-review.md with
466
+ * the documented promise that "the daily/weekly auto-drain acts on them" — but
467
+ * that drain was never implemented, so they STRANDED (the v0.3.1 cold-open found
468
+ * the user's architecture philosophy stuck here, never reaching the persona).
469
+ * This makes the promise real: the same optimistic auto-promote the review queue
470
+ * already gets (D-6) — trust the synthesis, mistakes self-correct via `cmk forget`
471
+ * (the post-hoc-reversibility model every surveyed memory system uses instead of
472
+ * a pre-promotion human gate). NOT a manual command: runs inside autoDrainQueues
473
+ * on the daily/weekly maintenance passes. The full recurrence-scored redesign is
474
+ * Task 151 (v0.4); this just stops the stranding.
475
+ *
476
+ * @returns {{promoted: number, drained: number, queuePath: string|null}}
477
+ */
478
+ export function resolvePersonaReviewQueue({ userDir, now, settings } = {}) {
479
+ const userTierRoot = resolveTierRoot({ tier: 'U', userDir });
480
+ const queuePath = join(userTierRoot, 'queues', 'persona-review.md');
481
+ let text;
482
+ try {
483
+ text = readFileSync(queuePath, 'utf8');
484
+ } catch {
485
+ return { promoted: 0, drained: 0, queuePath: null }; // no queue → nothing to drain
486
+ }
487
+ const candidates = parsePersonaReviewQueue(text);
488
+ if (candidates.length === 0) return { promoted: 0, drained: 0, queuePath };
489
+
490
+ // Re-feed through the SAME promote path the synthesis uses (home-path sanitize
491
+ // + Poison_Guard + dedup + audit all inherited). OPTIMISTIC AUTO-DRAIN: these
492
+ // candidates already SURVIVED a synthesis pass without being superseded; the
493
+ // drain IS the decision to promote them (the field-standard "auto-promote then
494
+ // post-hoc revert via cmk forget" posture — see the persona-promotion research
495
+ // note). So force confidence:'high' to clear promoteCandidatesToUserTier's
496
+ // confidence gate — otherwise they'd re-queue forever (the gate that stranded
497
+ // them in the first place). The full recurrence-scored model is Task 151 (v0.4).
498
+ const promotable = candidates.map((c) => ({ ...c, confidence: 'high' }));
499
+ const r = promoteCandidatesToUserTier({ candidates: promotable, userDir, now, settings });
500
+ const promoted = r.promoted?.length ?? 0;
501
+
502
+ // Clear the queue — the candidates are now resolved (promoted or de-duped into
503
+ // existing persona). Leave a tombstone header so the file isn't silently empty.
504
+ const ts = now ?? new Date().toISOString().replace(/\.\d{3}Z$/, 'Z');
505
+ writeFileSync(
506
+ queuePath,
507
+ `<!-- persona-review queue — auto-drained ${ts}: ${candidates.length} candidate(s) promoted to the persona. -->\n`,
508
+ 'utf8',
509
+ );
510
+ return { promoted, drained: candidates.length, queuePath };
511
+ }
512
+
428
513
  export function promoteCandidatesToUserTier({ candidates, userDir, now, settings, trust = 'medium', source = 'persona-synthesis' }) {
429
514
  // `trust`/`source` default to the AUTO-persona posture (medium, system-derived
430
515
  // — 45.6). The EXPLICIT path (`cmk lessons promote`) passes trust:'high' +
@@ -24,7 +24,7 @@
24
24
  // One heading per turn so downstream tools can scan by ## markers
25
25
  // (matches claude-remember's compaction strategy).
26
26
 
27
- import { existsSync, mkdirSync, appendFileSync } from 'node:fs';
27
+ import { existsSync, mkdirSync, appendFileSync, readFileSync } from 'node:fs';
28
28
  import { join } from 'node:path';
29
29
  import { sanitizePrivacyTags } from './privacy.mjs';
30
30
 
@@ -35,6 +35,39 @@ function dateFromIso(iso) {
35
35
  return String(iso).slice(0, 10);
36
36
  }
37
37
 
38
+ // Task 75.2 — the per-prompt "memory available" recall nudge (memsearch's
39
+ // UserPromptSubmit hint, D-115's 75.2 half). The SessionStart snapshot +
40
+ // its authority preamble cover the session OPEN; this keeps the agent
41
+ // aware MID-session (after the snapshot scrolls into history) that a deep,
42
+ // searchable archive exists behind the bounded snapshot. Conditions keep
43
+ // it noise-free: substantive prompts only (≥10 chars — "ok"/"go" never pay
44
+ // the hint; memsearch's heuristic) and only when there IS an archive to
45
+ // recall from (a granular INDEX.md). One line — the per-prompt token cost
46
+ // stays negligible, and it rides the EXISTING hook (no extra spawn).
47
+ const HINT_MIN_PROMPT_CHARS = 10;
48
+
49
+ export function buildMemoryHint({ projectRoot, prompt } = {}) {
50
+ if (typeof prompt !== 'string' || prompt.trim().length < HINT_MIN_PROMPT_CHARS) {
51
+ return null;
52
+ }
53
+ try {
54
+ const indexPath = join(projectRoot, 'context', 'memory', 'INDEX.md');
55
+ if (!existsSync(indexPath)) return null;
56
+ // `cmk install` scaffolds INDEX.md on every project, so existence alone
57
+ // is always true post-install (skill-review finding). Require at least
58
+ // one real entry — a fresh, empty project must not advertise recorded
59
+ // memory it does not have. Entry lines start "- (" (the reindex format).
60
+ if (!readFileSync(indexPath, 'utf8').includes('\n- (')) return null;
61
+ } catch {
62
+ return null;
63
+ }
64
+ return (
65
+ '[claude-memory-kit] Recorded memory available beyond the session snapshot — ' +
66
+ 'use the memory-search skill when the answer may already be recorded (prior decisions, history, conventions, ' +
67
+ 'project structure/architecture, where things live). Recall it; do not re-read the code to reconstruct it.'
68
+ );
69
+ }
70
+
38
71
  export function capturePrompt({ payload, projectRoot, now } = {}) {
39
72
  if (!payload || typeof payload !== 'object') {
40
73
  return { action: 'noop', reason: 'no-payload' };
@@ -26,14 +26,22 @@
26
26
  // <projectRoot>/context/transcripts/.extract-<ts>.tmp
27
27
  // so the detached child can read it without sharing stdin.
28
28
  //
29
- // Both-turns temp-file shape (design §6.4 amendment, 2026-05-26):
30
- // The temp file now contains BOTH the prior user prompt AND the
31
- // just-captured assistant turn, separated by literal markers:
29
+ // Both-turns temp-file shape (design §6.4 amendment, 2026-05-26;
30
+ // DEDUP_CONTEXT added by Task 132 / D-122, 2026-06-11):
31
+ // The temp file contains the dedup snapshot, the prior user prompt,
32
+ // AND the just-captured assistant turn, separated by literal markers:
33
+ // DEDUP_CONTEXT:
34
+ // <the last now.md entry BEFORE this turn was appended — may be empty>
35
+ //
32
36
  // USER_TURN:
33
37
  // <user body>
34
38
  //
35
39
  // ASSISTANT_TURN:
36
40
  // <assistant body>
41
+ // The dedup snapshot MUST be taken before appendConversationToNowMd
42
+ // runs: auto-extract used to re-read now.md after the append and saw
43
+ // the current turn as "already captured" → suppressed every organic
44
+ // extraction (the D-122 self-poisoning bug, found by cut-gate8).
37
45
  // This lets auto-extract identify candidate-origin (user-stated vs
38
46
  // assistant-inferred) and apply the demotion rule from design §6.4
39
47
  // (assistant-origin facts demote one trust level so user review is
@@ -55,6 +63,8 @@ import {
55
63
  import { join } from 'node:path';
56
64
  import { spawn } from 'node:child_process';
57
65
  import { sanitizePrivacyTags } from './privacy.mjs';
66
+ import { extractTurnToolActivity, readTranscriptTail } from './turn-tools.mjs';
67
+ import { readLastEntryFromNowMd } from './auto-extract.mjs';
58
68
 
59
69
  function dateFromIso(iso) {
60
70
  return String(iso).slice(0, 10);
@@ -226,8 +236,11 @@ function appendConversationToNowMd({ projectRoot, ts, userTurn, assistantTurn })
226
236
  // capture-prompt sanitized when writing it) and the assistant body
227
237
  // is the now-sanitized argument. Markers are literal-prefix lines so
228
238
  // auto-extract's parser can split cleanly.
229
- function assembleBothTurnsBody({ userTurn, assistantTurn }) {
239
+ function assembleBothTurnsBody({ userTurn, assistantTurn, dedupContext = '' }) {
230
240
  return [
241
+ 'DEDUP_CONTEXT:',
242
+ dedupContext,
243
+ '',
231
244
  'USER_TURN:',
232
245
  userTurn,
233
246
  '',
@@ -299,9 +312,34 @@ export function captureTurn({
299
312
  mkdirSync(transcriptsDir, { recursive: true });
300
313
  }
301
314
  const sanitized = sanitizePrivacyTags(turnText);
315
+
316
+ // Task 104.1 (D-117) — enrich the assistant entry with the turn's TOOL
317
+ // ACTIVITY, read from Anthropic's live session JSONL (the Stop payload's
318
+ // transcript_path). The payload itself carries only the assistant TEXT;
319
+ // the JSONL is the only record of tool calls/results — and it expires
320
+ // (~30 days, machine-local), so this is the moment to extract the current
321
+ // turn into the kit's own durable format (the L3 raw tier, design §19).
322
+ // Best-effort by contract: a missing path, unreadable file, or shifted
323
+ // format degrades to a text-only entry, never a capture failure. The
324
+ // block is privacy-sanitized like everything else that reaches disk.
325
+ // The now.md buffer + the auto-extract turn file deliberately stay
326
+ // TEXT-ONLY (tool noise would bloat the compressor/extractor inputs).
327
+ let toolsSection = '';
328
+ try {
329
+ if (typeof payload?.transcript_path === 'string' && payload.transcript_path !== '') {
330
+ const tail = readTranscriptTail(payload.transcript_path);
331
+ const activity = tail ? extractTurnToolActivity(tail) : null;
332
+ if (activity) {
333
+ toolsSection = `\n**Tools:**\n\n${sanitizePrivacyTags(activity)}\n`;
334
+ }
335
+ }
336
+ } catch {
337
+ // enrichment is best-effort; the text entry below is the durable record
338
+ }
339
+
302
340
  appendFileSync(
303
341
  transcriptPath,
304
- `## ${ts} — assistant\n\n${sanitized}\n\n`,
342
+ `## ${ts} — assistant\n\n${sanitized}\n${toolsSection}\n`,
305
343
  'utf8',
306
344
  );
307
345
 
@@ -315,6 +353,26 @@ export function captureTurn({
315
353
  // `sanitized` text we just appended above.
316
354
  const userTurn = readLastUserTurnFromTranscript(transcriptPath);
317
355
 
356
+ // Task 132 (D-122): snapshot the dedup context BEFORE the now.md append
357
+ // below — after the append, "the last now.md entry" IS the current turn,
358
+ // and feeding that to auto-extract as "do not re-emit facts already
359
+ // here" suppressed every organic extraction. Best-effort: an unreadable
360
+ // now.md just means no dedup section this turn.
361
+ let dedupContext = '';
362
+ try {
363
+ // Skill-review I1: neutralize line-start section markers INSIDE the
364
+ // snapshot — conversation text can legitimately contain "USER_TURN:"
365
+ // (this repo's own sessions discuss the turn-file format), and the
366
+ // parser anchors on the first line-start marker it sees. The dedup is
367
+ // advisory context for Haiku; a cosmetic "· " prefix is harmless.
368
+ dedupContext = readLastEntryFromNowMd(projectRoot).replace(
369
+ /^([ ]*)(DEDUP_CONTEXT:|USER_TURN:|ASSISTANT_TURN:)/gm,
370
+ '$1· $2',
371
+ );
372
+ } catch {
373
+ // no dedup context — extraction still runs, worst case re-emits a dup
374
+ }
375
+
318
376
  // Task 87: buffer the conversation into now.md so the SessionEnd compressor
319
377
  // summarizes the DIALOGUE, not observe-edit's filename log. Best-effort.
320
378
  appendConversationToNowMd({ projectRoot, ts, userTurn, assistantTurn: sanitized });
@@ -327,7 +385,7 @@ export function captureTurn({
327
385
  try {
328
386
  writeFileSync(
329
387
  turnFile,
330
- assembleBothTurnsBody({ userTurn, assistantTurn: sanitized }),
388
+ assembleBothTurnsBody({ userTurn, assistantTurn: sanitized, dedupContext }),
331
389
  'utf8',
332
390
  );
333
391
  } catch (err) {