@inetafrica/open-claudia 2.6.57 → 2.6.59

package/CHANGELOG.md

@@ -1,5 +1,13 @@
 # Changelog
 
+## v2.6.58
+- **Tiered recall gate — stop paying 15 seconds for "thanks".** Measured over 700+ live turns, the old discoverer pre-gate returned true whenever *any* seed matched, so it gated just 0.4% of turns — a bare "ok" with an incidental keyword hit still paid the full seed → graph → walker pass (p50 ~15s). A new `recallTier()` classifies each turn into three tiers: **skip** (pleasantries/acks up to 4 words *regardless of seeds*, emoji-only, or ≤2 words with no seeds — no recall at all), **seeds** (short command-like turns ≤4 words — user-origin keyword seeds inject directly with no graph expansion, no walker, no excerpts; the classic-engine baseline at ~ms cost), and **full** (substantive turns — the whole pipeline). The tier is logged per turn and the metrics summary now tracks the seeds-only share alongside the gated share.
+- **Walker candidate cap.** The latency deep-dive showed the walk itself dominates, not process spawn: 15–40 candidates × 600-char excerpts = 10–25KB prompts to the haiku judge on every substantive turn. Full-tier candidates are now priority-ordered — user-origin seeds by match score, then context seeds, then graph-activated nodes by activation — and capped at `RECALL_WALKER_MAX_CANDIDATES` (default 14), so the judge reads the strongest few instead of everything the graph touched.
+- **Episodic memory — recall now remembers how it did things last time.** The discoverer gains a transcript hop: on full-tier turns it queries the project-transcript FTS index (up to `RECALL_EPISODE_LIMIT`, default 3; `RECALL_EPISODES=off` disables) and offers the hits to the walker as `episode:` candidates with their matched snippets. The walker keeps one only if that past exchange holds a decision, method, or outcome directly reusable for the current message — and the fail-open path (walker off/failed) *drops* episodes rather than injecting unjudged snippets. Kept episodes render as a "Past work that looks related" block with the snippet plus a ready-to-run `open-claudia transcript-search … --all` pointer for the full context, and surface in the `/recall` banner as 📓 lines.
+- **Real cost + phase telemetry for recall.** The warm walker's stream-json `total_cost_usd` is cumulative per session; it now tracks the running total and reports the per-walk *delta*, which flows into `recall-metrics.jsonl` per turn (`costUsd`) instead of the flat 0 logged before. Each turn also records `seedMs` / `expandMs` / `walkerMs` and which walker path ran (`warm`/`cold`/`off`/`failed`), so the p50 can be decomposed instead of guessed at.
+- **Weak-use signal from the post-turn reviewer.** Precision metrics only counted explicit 📖 opens as "used". Now, when the reviewer decides a turn's work updated pack X or entity Y, that adjudication is logged as use (`source: "reviewer"`) and co-updated nodes get a Hebbian co-work reinforcement — so packs that quietly shape work without being re-opened stop looking like noise.
+- **Journal dedupe.** The reviewer could append near-identical Journal lines turn after turn (observed: 28 same-day duplicates on one busy pack). `updatePack` now token-Jaccard-compares a new journal line (dates stripped) against the last 5 entries and skips appends at ≥0.75 similarity, flagging `journalDeduped` instead. Adds `test-journal-dedupe.js`.
+
 ## v2.6.57
 - **Move the inactivity watchdog into the live runner.** Deep review found v2.6.56 put the watchdog in the legacy `bot-agent.js` monolith, but the LaunchAgent runs `bot.js`, which delegates heavy turns through `core/runner.js`. That meant the installed live path still only had the 6-hour hard timeout and could still wedge on a silent child. This release ports the watchdog to `core/runner.js`, updates activity on child stdout/stderr, terminates the process tree after `OC_TURN_IDLE_MS` of silence, suppresses the confusing partial/no-output final message, clears stuck busy state with a failsafe, and drains any queued follow-up messages instead of stranding them. Adds a static regression test that fails if the watchdog only exists in `bot-agent.js` again.
 

package/README.md

@@ -267,7 +267,7 @@ Open Claudia layers three memory systems on top of the backend's native sessions
 
 **Recall engines** — how packs and entities get matched and surfaced is pluggable per chat via `/engine` (or the `RECALL_ENGINE` env default). **classic** (the default) is keyword FTS plus a relevance judge with headline injection — stable and unchanged. **discoverer** (opt-in) adds a typed-edge graph over the same corpus (`parent`/`governed-by`/`related` edges with weights in `recall-graph.db`) and runs: a pre-gate that skips recall on trivial turns → FTS seeding → spreading activation across the graph (1–2 hops — auto-pulls cross-cutting concerns the query never named) → a walker that reads each candidate and returns the genuinely-relevant set with one-line why-bullets (fail-open to keyword seeds, so it never recalls worse than classic). Edges form structurally from pack `parent` frontmatter and `[[links]]`, and strengthen via Hebbian co-use when the agent opens packs together (📖); weights decay over time. Inspect with `open-claudia recall-stats` and `open-claudia recall graph [--sync]`, or flip on `/recall` to watch — per turn — which packs/entities surfaced and why, right in the chat. Switch back any time with `/engine classic`.
 
-**Dream consolidation** — while the per-turn reviewer takes quick notes, *dream* is the slow overnight pass (default 4am, on a high-tier model — opus by default, set `DREAM_TIER` or `DREAM_MODEL`): it merges packs that drifted into the same topic, builds parent/sub pack trees with umbrella summaries, tightens descriptions and tags so the router matches with less noise, dedupes entities, cross-links notes, and tends the recall graph (structural sync, weight decay, orphan prune). Anything merged away is backed up under `~/.open-claudia/backup/dream-<stamp>/` first, and every dream that changes something reports in chat. Configure with `DREAM_CRON` / `DREAM_MODEL`, disable with `DREAM=off`.
+**Dream consolidation** — while the per-turn reviewer takes quick notes, *dream* is the slow overnight pass (default 4am, on a high-tier model — opus by default, set `DREAM_TIER` or `DREAM_MODEL`): it merges packs that drifted into the same topic, builds parent/sub pack trees with umbrella summaries, tightens descriptions and tags so the router matches with less noise, dedupes entities, cross-links notes, and tends the recall graph (structural sync, weight decay, orphan prune). The pass is *evidence-grounded*: per-turn recall telemetry (what surfaced, what got kept, what the agent actually opened) feeds the prompt, so archive/merge calls rest on usage numbers rather than vibes. Most nights run as cheap *deltas* — full pack bodies only for what changed since the last dream plus graph neighbours and pre-computed merge candidates; a full-corpus sweep runs every `DREAM_FULL_SWEEP_DAYS` (30). Deterministic phases run even if the model call fails: journal backfill dedupe, weakening of graph nodes that keep arriving but never get kept, co-rescue edge reinforcement, episodic index tending, and a memory-health report (rescue rate, noise, latency, spend) in the morning chat summary. Dream may also tune one bounded recall knob per night (`recall-tuning.json`) — every change is checked against the next window's health and auto-rolled-back if rescue drops or noise rises; env pins always win. Anything merged away is backed up under `~/.open-claudia/backup/dream-<stamp>/` first, and every dream that changes something reports in chat. Configure with `DREAM_CRON` / `DREAM_MODEL`, disable with `DREAM=off`.
 
 **Personality** — your `soul.md` holds identity and hard rules; `~/.open-claudia/persona.md` holds the voice on top — tone, quirks, emoji habits. It feeds into the system prompt and the dream pass may evolve it gently (bounded, backed up, announced). Edit it directly any time.
 
@@ -279,6 +279,7 @@ open-claudia entity list|show <slug>|match "<text>"|note <name> "<text>"|remove
 open-claudia dream [--dry-run]               # run the consolidation pass now
 open-claudia recall-stats                    # discoverer-engine metrics summary
 open-claudia recall graph [--sync]           # recall-graph node/edge stats; --sync rebuilds structural edges
+open-claudia kpi [--json|--html|--knobs]     # harness KPIs by version+model; --html writes a chart report, --knobs shows dream-tuned knob state
 open-claudia transcript-search "<query>"     # alias: ts; --all for every project
 open-claudia transcript-window "<pattern>"   # alias: tw; hits with surrounding turns
 ```
@@ -436,6 +437,11 @@ All stored in `~/.open-claudia/`:
 | `MEMORY_RECALL_MAX_CHARS` | No | Hard cap for auto-injected pack/entity memory per turn (default `9000`, `off` disables auto recall injection) |
 | `RECALL_ENGINE` | No | Default recall engine when a chat hasn't set one via `/engine` (`classic` or `discoverer`, default `classic`) |
 | `RECALL_GRAPH_DB` / `RECALL_METRICS` | No | Override the discoverer graph DB path; `off` on metrics disables per-turn recall logging |
+| `RECALL_WALKER_MAX_CANDIDATES` | No | Cap on candidates sent to the discoverer's walker judge per turn (default `14`; dream-tunable 8–20, env pins it) |
+| `RECALL_EPISODES` / `RECALL_EPISODE_LIMIT` | No | `off` disables episodic transcript candidates in recall; limit caps transcript hits per turn (default `3`; dream-tunable 1–5) |
+| `RECALL_SEEDS_TIER_MAX_WORDS` | No | Turns at or under this word count take the fast seeds-only recall path (default `4`; dream-tunable 2–6) |
+| `RECALL_DECAY_HALF_LIFE_DAYS` | No | Half-life for recall-graph edge decay during dream tending (default `60`; dream-tunable 10–90) |
+| `RECALL_CASCADE` / `RECALL_CASCADE_SHADOW_RATE` | No | `off` disables the deterministic keep/drop cascade before the walker judge; shadow rate is the fraction of cascade verdicts also judged to audit agreement (default `0.1`) |
 | `PROJECT_TRANSCRIPTS` | No | Enable redacted project transcripts (default `true`) |
 | `TRANSCRIPT_MAX_ENTRY_CHARS` | No | Max chars per transcript entry (default `12000`) |
 | `TRANSCRIPTS_DIR` / `PACKS_DIR` / `ENTITIES_DIR` | No | Override storage directories |
@@ -446,6 +452,7 @@ All stored in `~/.open-claudia/`:
 | `DREAM_CRON` | No | Schedule for the dream pass (default `0 4 * * *`) |
 | `DREAM_MODEL` | No | Explicit model override for the dream pass (otherwise picked from `DREAM_TIER`) |
 | `DREAM_TIER` | No | Model tier for the dream pass: `low` (haiku) / `medium` (sonnet) / `high` (opus, default) |
+| `DREAM_FULL_SWEEP_DAYS` | No | Days between full-corpus dream sweeps; other nights are cheap deltas over recently-touched packs (default `30`) |
 | `PERSONA_FILE` | No | Override the persona file location |
 | `WEB_UI` / `WEB_PORT` / `WEB_PASSWORD` | No | Web UI toggle, port, and password |
 | `WHISPER_CLI` / `WHISPER_MODEL` | No | whisper.cpp binary and model for voice notes |

package/bin/cli.js

@@ -318,6 +318,10 @@ switch (command) {
     require("./recall").run(["stats", ...args.slice(1)]);
     break;
   }
+  case "kpi": {
+    require("./kpi").run(args.slice(1));
+    break;
+  }
   case "recall": {
     require("./recall").run(args.slice(1));
     break;

package/bin/kpi.js

@@ -0,0 +1,55 @@
+// open-claudia kpi [--json] [--html [--out <path>]] [--knobs]
+//
+// Compare harness performance across versions and models: rescue rate, noise,
+// latency by tier, cost/turn, cascade + walker health — aggregated from the
+// version-stamped recall telemetry, plus the nightly dream KPI snapshots.
+
+const path = require("path");
+
+const HELP = `
+Harness KPIs by version.
+
+  open-claudia kpi                 Version-vs-version table
+  open-claudia kpi --json          Raw aggregates
+  open-claudia kpi --html          Write a self-contained HTML report (charts), print its path
+  open-claudia kpi --html --out p  Write the report to a specific path
+  open-claudia kpi --knobs         Current recall-knob values + dream tuning history
+
+Data sources: recall-metrics.jsonl (per-turn, version+model stamped) and
+recall-kpi.jsonl (one snapshot per dream night).
+`;
+
+function run(args) {
+  if (args.includes("--help") || args[0] === "help") { console.log(HELP); return; }
+  const kpi = require(path.join(__dirname, "..", "core", "recall", "kpi"));
+
+  if (args.includes("--knobs")) {
+    const tuning = require(path.join(__dirname, "..", "core", "recall", "tuning"));
+    console.log("Recall knobs (env pin > dream tuning > default)");
+    console.log("───────────────────────────────────────────────");
+    for (const k of tuning.knobTable()) {
+      console.log(`${k.name.padEnd(22)} ${String(k.current).padEnd(6)}${k.pinned ? " [pinned by env]" : k.current !== k.default ? " [dream-tuned]" : ""}`);
+    }
+    const hist = tuning.history();
+    if (hist.length) {
+      console.log("\nTuning history");
+      for (const h of hist.slice(-10)) console.log(`  ${h.ts.slice(0, 10)} ${h.knob}: ${h.from} → ${h.to} — ${h.reason}`);
+    }
+    return;
+  }
+
+  if (args.includes("--html")) {
+    const i = args.indexOf("--out");
+    const out = i !== -1 ? args[i + 1] : undefined;
+    console.log(kpi.writeHtmlReport(out));
+    return;
+  }
+
+  const rows = kpi.aggregate();
+  if (args.includes("--json")) { console.log(JSON.stringify({ versions: rows, snapshots: kpi.trend() }, null, 2)); return; }
+  console.log(kpi.renderTable(rows));
+  const snaps = kpi.trend();
+  if (snaps.length) console.log(`\n${snaps.length} nightly snapshot(s) in recall-kpi.jsonl — \`open-claudia kpi --html\` for charts.`);
+}
+
+module.exports = { run, HELP };