@lorekit/cli 1.30.2 → 1.31.0

package/README.md

@@ -47,7 +47,7 @@ Linux, and Windows (npm creates the `lorekit` shim on every platform).
 Sets up the full memory loop — the same three parts as the Claude plugin,
 without needing a marketplace:
 
-1. **Skills** (`lorekit-memory` + `lorekit-setup`) — the model-invoked authoring judgment and the self-improvement loop authoring counterpart.
+1. **Skills** (`lorekit-memory` + `lorekit-setup` + `lorekit-groom`) — the model-invoked runtime read/write loop, the self-improvement loop authoring counterpart, and the store-grooming maintenance counterpart.
 2. **MCP server** (`lorekit`) — the connection to your lessons, merged into the
    MCP config (preserving any other servers).
 3. **Hooks** — the *deterministic* layer: lessons injected on every
@@ -751,7 +751,7 @@ also returns their headroom against the plan's memory cap.
 
 ## What the skills do
 
-`install` scaffolds two skills:
+`install` scaffolds three skills:
 
 The **`lorekit-memory`** skill teaches an agent to:
 
@@ -770,6 +770,14 @@ to wire a self-improvement loop into one of *your own* skills or workflows — t
 two-tier model, the lesson bucket convention, and the entrenchment guards. See
 its `SKILL.md` and `rules/self-improvement-loops.md`.
 
+The **`lorekit-groom`** skill is the maintenance counterpart: it teaches an agent
+to run a grooming pass over an accumulated store — survey (`stats` / `scopes`),
+lint, dedupe & merge near-duplicates, set expiry (TTL) on time-bound lessons, and
+prune or archive obsolete ones. It always analyses read-only and proposes a plan
+before mutating (archive is preferred over hard-delete), because the store is
+shared and a merge or delete is permanent for every agent. See its `SKILL.md`,
+`rules/grooming-pass.md`, and `references/merge-and-expiry.md`.
+
 The **skills** are model-invoked (the agent chooses to use them). For a
 **deterministic** guarantee — lessons injected on every session start, a nudge
 on every tool failure — use the framework plugins in [`plugins/`](../../plugins/),

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@lorekit/cli",
-  "version": "1.30.2",
+  "version": "1.31.0",
   "description": "Install the LoreKit shared-memory skill and run health checks for the LoreKit MCP server.",
   "license": "MIT",
   "repository": {

package/skill/lorekit-groom/SKILL.md

@@ -0,0 +1,176 @@
+---
+name: lorekit-groom
+description: >
+  Grooms and maintains an accumulated LoreKit memory store — the cleanup
+  counterpart to lorekit-memory (which reads and records single lessons). It
+  finds and merges near-duplicate lessons, lints out low-quality ones, sets
+  expiry (TTL) on time-bound lessons, and prunes or archives obsolete ones,
+  ALWAYS analysing read-only and proposing a plan before any change (nothing
+  is deleted or merged without confirmation, and archive is preferred over
+  hard-delete). Reach for this whenever a lesson/memory store has grown noisy,
+  large, or stale and needs tidying — not for reading or writing one lesson.
+  Triggers on "groom my memories", "clean up my lessons", "dedupe the memory
+  store", "merge duplicate lessons", "too many memories", "my lorekit is a
+  mess", "consolidate lessons", "add expiry dates to old lessons", "prune stale
+  branch lessons", "lint my memories", "/lorekit-groom".
+user-invocable: true
+argument-hint: '[scope-hint]'
+license: MIT
+metadata:
+  author: mthines
+  version: '1.0.0'
+  workflow_type: shared-memory-grooming-and-consolidation
+  tags:
+    - lorekit
+    - persistent-memory
+    - shared-memory
+    - mcp
+    - lessons
+    - grooming
+    - dedupe
+    - lint
+    - maintenance
+---
+
+# LoreKit Groom
+
+Keep a shared memory store healthy as it grows.
+
+A memory store that is never tended slowly rots: the same lesson gets written
+three times in slightly different words, a stub with an empty value sits next to
+a real one, a branch-specific gotcha outlives the branch by a year. Noise like
+that is not harmless — it drowns the signal every future read depends on, so the
+next agent scrolls past ten near-identical lines instead of acting on one clear
+one.
+
+This skill runs a **grooming pass** over the lessons for a scope: survey → lint →
+dedupe & merge → set expiry → prune → verify. It is the maintenance counterpart
+to the other two LoreKit skills:
+
+| Skill | Job |
+|-------|-----|
+| `lorekit-memory` | Runtime: read scoped lessons on a task, write one on failure |
+| `lorekit-setup` | Authoring: wire a self-improvement loop into a host |
+| **`lorekit-groom`** | **Maintenance: consolidate and clean an accumulated store** |
+
+---
+
+## The one rule that governs everything: propose, then confirm
+
+Grooming edits a **shared, persistent** store — a lesson you merge or delete is
+gone for every agent on every machine, not just this session. So the entire pass
+is read-only until a human says otherwise:
+
+1. **Analyse read-only.** Every survey step below is a read command. Run them
+   first and build the full picture before touching anything.
+2. **Propose a concrete plan.** Show exactly what you intend to do — which
+   lessons merge into which, what gets an expiry, what gets archived — with the
+   actual keys and values, grouped so a human can scan it in one screen.
+3. **Only mutate after confirmation.** Wait for the go-ahead, then apply. Prefer
+   **archive** (reversible) over hard-delete; reserve permanent deletion for
+   genuine junk and only with explicit sign-off.
+
+Never batch-delete on a similarity score alone — the dedupe heuristic is a
+word-overlap guess, not a semantic judge (see the toolbox note below). It finds
+*candidates for a human's eye*, and merging is a judgement call about meaning.
+
+If you are asked to "just clean it up" without oversight, still surface the plan
+once and get a single confirmation for the batch — that is the floor, not a
+step to skip.
+
+---
+
+## The grooming pass
+
+Follow [rules/grooming-pass.md](./rules/grooming-pass.md) for the full playbook.
+The short version is six phases; the first four are pure analysis.
+
+1. **Survey** — size the problem. `lorekit stats` (counts per scope + store) and
+   `lorekit scopes` (store-wide inventory: every scope with its lesson count)
+   show where the mass is. Pick the noisiest scope to start.
+2. **Lint** — `lorekit lint --json` flags structurally bad lessons (empty /
+   whitespace / suspiciously short / untrimmed values, empty keys, malformed
+   scopes). These are the cheapest wins: fix or drop them first.
+3. **Dedupe** — `lorekit dedupe --json` clusters near-duplicate lessons. Start at
+   a high `--threshold` (e.g. `0.85`) for confident duplicates, then lower it to
+   surface looser paraphrases. Use `lorekit show <scope::key>` to read each
+   cluster member's full value before deciding.
+4. **Plan** — turn the findings into a proposed set of merges, expiries, and
+   removals. This is where propose-then-confirm happens.
+5. **Apply** (after confirmation) — merge, expire, and prune. See the toolbox.
+6. **Verify** — re-run `lorekit lint` (should exit 0) and `lorekit dedupe` (no
+   clusters at your threshold) and `lorekit stats` (a lower count). Green means
+   the pass landed.
+
+---
+
+## The toolbox
+
+Grooming spans two surfaces. **Analysis is the CLI** — its read commands
+(`lint`, `dedupe`, `stats`, `scopes`, `show`, `list`, `search`, `tree`, `diff`)
+survey the store far more richly than the MCP tools do, across both the offline
+and remote stores at once. **Mutation is the `memory.*` MCP tools** — and
+crucially, *removal* is MCP-only: there is no `lorekit delete` command, so
+archiving and deleting always go through `memory.delete` / `memory.archive`.
+
+| Job | How | Surface |
+|-----|-----|---------|
+| Count lessons per scope/store | `lorekit stats [--scope <s>]` | CLI (read) |
+| Inventory every scope + lesson count | `lorekit scopes` | CLI (read) |
+| Find low-quality lessons | `lorekit lint --json` | CLI (read) |
+| Find near-duplicate clusters | `lorekit dedupe --json [--threshold <n>]` | CLI (read) |
+| Read one lesson in full | `lorekit show <scope::key> [--json]` | CLI (read) |
+| Compare offline vs remote | `lorekit diff` | CLI (read) |
+| Write the merged/consolidated lesson | `memory.write` (or `lorekit write`) | MCP / CLI |
+| Set / clear an expiry | `memory.write { ttl_days }` / `{ clear_ttl: true }` | MCP / CLI (`--ttl-days` / `--clear-ttl`) |
+| Archive a lesson (reversible) | `memory.archive` (or `memory.delete`) | **MCP only** |
+| Hard-delete a lesson (permanent) | `memory.delete { force: true }` | **MCP only** |
+
+> **`dedupe` is a heuristic, not a semantic judge.** It clusters on Jaccard
+> word-token overlap, so it can both miss reworded duplicates *and* group
+> coincidental ones. Treat every cluster as a candidate to read and decide on,
+> never as an instruction to merge.
+
+> **If the `memory.*` MCP tools are not connected**, analysis still works
+> (the CLI reads and `lorekit write --ttl-days` sets expiries), but you cannot
+> archive or delete. Say so once, complete everything you can, and hand the
+> removal list to the human rather than silently skipping it.
+
+Merging, expiry tiers, and the archive-vs-delete call each have real judgement
+in them — the details, with examples, live in
+[references/merge-and-expiry.md](./references/merge-and-expiry.md). Read it
+before your first merge or TTL decision in a pass.
+
+---
+
+## Scope in one line
+
+Lessons are partitioned by a canonical scope string (`::` is the only separator):
+
+```text
+global                                universal principles
+project::{name}                       monorepo-wide
+repo::{owner}/{repo}                   this repository's codebase
+branch::{owner}/{repo}::{branch}       short-lived, this branch only
+```
+
+Scope drives two grooming decisions: **branch-scoped lessons are the prime
+candidates for expiry** (a branch is short-lived, its lessons usually are too),
+and **a merge lands in the narrowest scope that still covers all its members**.
+Both are spelled out in the reference file.
+
+---
+
+## Setup
+
+Grooming uses the same install as the rest of LoreKit:
+
+```bash
+npx @lorekit/cli install
+npx @lorekit/cli doctor
+```
+
+`doctor` confirms the store backend (`remote`, `local`, or `off`) and token
+permission. Removal needs a token with **write** permission (`lk_rw_*` or
+`lk_wo_*`); a read-only token can survey and propose but not apply. If a mutate
+call fails with an authorization error, report it and stop — do not retry.

package/skill/lorekit-groom/references/merge-and-expiry.md

@@ -0,0 +1,123 @@
+# Merge, expiry, and removal — the judgement calls
+
+The mechanical parts of grooming (survey, lint, dedupe) are commands. The parts
+that need thought are: how to synthesise a merged lesson, where it should live,
+how long a lesson should last, and whether a stale lesson should be archived or
+deleted. This file is the reference for those four decisions. Read it before the
+first merge or TTL choice in a pass.
+
+## Table of contents
+
+1. [Synthesising a merged lesson](#1-synthesising-a-merged-lesson)
+2. [Choosing the merged lesson's scope](#2-choosing-the-merged-lessons-scope)
+3. [Choosing the key](#3-choosing-the-key)
+4. [Expiry tiers (TTL)](#4-expiry-tiers-ttl)
+5. [Archive vs. hard-delete](#5-archive-vs-hard-delete)
+
+---
+
+## 1. Synthesising a merged lesson
+
+A merge is not a concatenation. Three lessons that say the same thing become
+**one lesson that says it best** — otherwise you have replaced three near-copies
+with one long lumpy one and gained nothing.
+
+- **Keep the observation, drop the repetition.** Write the single clearest
+  statement of the shared point. If two members add genuinely different detail
+  (one names the failing command, another names the fix), fold both facts into
+  one lesson; if they only reword each other, keep the sharpest phrasing.
+- **Stay an observation, not a rule.** LoreKit lessons are advisory ("migrations
+  that skip an explicit transaction have left the schema half-applied here"), not
+  commandments ("ALWAYS wrap migrations"). Merging is a chance to soften a member
+  that drifted into a rigid MUST.
+- **Preserve the union of tags and the richest provenance.** Carry every source's
+  tags onto the merged lesson so it stays as findable as the originals. Keep the
+  most complete origin (repo / branch / commit / PR) among the members.
+- **Prefer updating a survivor over minting a new key.** If one member is already
+  well-named and well-scoped, write the merged value onto *its* `scope::key`
+  (an in-place update) and delete the others. That keeps any external links to
+  that key alive. Mint a fresh key only when no member's key fits the merged
+  meaning.
+
+**Example**
+
+```text
+Sources:
+  - "wrap every migration in a transaction"
+  - "schema changes need a transaction or they half-apply on error"
+  - "migrations must be atomic"
+Merged value:
+  "Migrations here half-apply on error unless wrapped in an explicit
+   transaction — a failed step otherwise leaves the schema partly changed."
+```
+
+The merged version keeps the *why* (half-apply on error) that only one source
+had, and states it as an observation.
+
+## 2. Choosing the merged lesson's scope
+
+Merge into the **narrowest scope that still correctly covers every member**.
+
+- All members share a scope → merge stays in that scope.
+- Members span scopes (e.g. two `branch::…` and one `repo::…` saying the same
+  thing) → the lesson is really about the broader scope. Merge up to the
+  narrowest scope that is still true for all of them — usually `repo::` when a
+  branch lesson turned out to be a repo-wide truth. Broadening scope is itself a
+  useful grooming outcome: a durable lesson trapped on a dead branch is nearly
+  invisible.
+- Never merge *down* into a scope narrower than some members — that hides the
+  lesson from contexts where it applies.
+
+Precedence check: bare `lorekit tree` shows which scope's lesson wins when keys
+collide across scopes. Run it without `--scope` — that flag narrows the
+resolution to a single scope, which by construction cannot show a collision.
+Use it to confirm a merged lesson will actually surface where you intend.
+
+## 3. Choosing the key
+
+- Reuse a good survivor's key when one exists (see §1 — keeps links alive).
+- Otherwise write a short, specific, hyphenated key that names the *observation*,
+  not the symptom: `db-migrations-need-explicit-tx`, not `migration-bug` or
+  `note-3`. A key is how a future agent recognises the lesson at a glance in a
+  list, so vague keys are their own kind of noise.
+
+## 4. Expiry tiers (TTL)
+
+`ttl_days` (1–365) auto-expires a lesson that many days after the write, after
+which it is hidden from reads. Set it via `memory.write { ttl_days }` or
+`lorekit write --ttl-days <n>`; clear it with `{ clear_ttl: true }` /
+`--clear-ttl` to make a lesson permanent again.
+
+Match the TTL to how long the lesson's *truth* lasts, not to how old it is:
+
+| Lesson kind | Guidance |
+|-------------|----------|
+| `branch::` — branch-specific gotcha | Short. The branch is short-lived; 14–30 days usually outlasts it. Prime expiry candidates. |
+| Version- or dependency-pinned ("works around the bug in lib X 2.3") | Medium (30–90d), or archive once the pin is gone. |
+| Repo- or project-wide architectural truth | Usually **permanent** — no TTL. Expiring durable knowledge is how a store forgets the things worth keeping. |
+| `global` principle | Permanent. |
+
+The instinct to worry about is expiring something durable. When unsure whether a
+lesson's truth is time-bound, leave it permanent and flag it in the plan rather
+than quietly attaching a TTL — a lesson that silently vanishes is worse than one
+that lingers.
+
+## 5. Archive vs. hard-delete
+
+Two removal paths, and the default is the reversible one.
+
+- **Archive** (`memory.archive`, or `memory.delete` without `force`) —
+  soft-removes: hidden from reads but restorable. This is the default for
+  anything that is *stale* rather than *junk*: superseded workarounds, lessons
+  about a since-deleted subsystem, a merge's source lessons. If a future run
+  might ever want it back, archive.
+- **Hard-delete** (`memory.delete { force: true }`) — permanent, no recovery.
+  Reserve it for genuine junk: lint casualties with no meaning (empty values,
+  placeholder keys), test detritus, accidental writes. Only with explicit
+  human sign-off in the plan.
+
+When a lesson came from another agent or teammate (check its tags / source),
+lean further toward archive — deleting someone else's recorded knowledge outright
+is a heavier call than tidying your own.
+
+Rule of thumb: **if you would hesitate to lose it forever, archive it.**

package/skill/lorekit-groom/rules/grooming-pass.md

@@ -0,0 +1,148 @@
+# The grooming pass — playbook
+
+A single, repeatable pass over the lessons for a scope. The first four phases are
+read-only; nothing is mutated until a human has seen and approved the plan.
+
+Work **one scope at a time**. A whole-store pass in a single breath produces a
+wall of proposed changes no one can review; a scope at a time keeps each
+confirmation small enough to actually read.
+
+---
+
+## Phase 1 — Survey (read-only)
+
+Size the problem before touching it, so you groom where the mass actually is
+rather than wherever you happened to look first.
+
+```bash
+lorekit stats            # counts per scope and per store — where the mass is
+lorekit scopes           # every scope in the store + its lesson count
+```
+
+`scopes` is the one command that is store-wide rather than cwd-scoped, so it is
+how you notice a `branch::…` scope you had forgotten about, or a scope with
+hundreds of lessons that dwarfs the rest. Neither command reports a
+last-activity date — the inventory is counts only — so judge staleness from the
+lessons themselves once you narrow in (`lorekit list --scope <scope>` tags each
+one with its `updated` date). Read both, then pick the noisiest scope as the
+target for this pass and narrow to it with `--scope <scope>` from here on.
+
+## Phase 2 — Lint (read-only)
+
+```bash
+lorekit lint --json --scope <scope>
+```
+
+Findings are structural, not semantic — each names its rule:
+
+- **empty-value / short-value / untrimmed-value** — the lesson carries little or
+  no signal, or has stray leading/trailing whitespace.
+- **empty-key** — no key to address it by.
+- **malformed-scope** — the scope string is invalid.
+
+These are the cheapest wins and the least controversial, so clear them first.
+For each: either **fix it in place** (rewrite a too-short value into a real
+observation, trim whitespace — a `memory.write` to the same `scope`+`key`
+updates in place) or, if the lesson is genuinely empty of meaning, **queue it for
+removal** in the plan. `lint` exits non-zero while findings remain, which also
+makes it a clean CI gate — a passing `lint` is your Phase 6 proof.
+
+## Phase 3 — Dedupe (read-only)
+
+```bash
+lorekit dedupe --json --scope <scope> --threshold 0.85
+```
+
+Each cluster is a set of lessons whose values overlap heavily by word tokens.
+Start high (`0.85`) to see the confident duplicates, then re-run lower (`0.75`,
+`0.7`) to surface looser paraphrases — but the lower you go, the more the
+clusters are coincidental overlaps rather than true duplicates, so read more
+carefully.
+
+For every cluster you intend to act on, read the members in full first:
+
+```bash
+lorekit show <scope::key> --json
+```
+
+You are deciding whether these lessons *mean* the same thing, which the score
+cannot tell you. Only clusters that survive that read become merges in the plan.
+
+## Phase 4 — Build and propose the plan
+
+Turn Phases 2–3 into one concrete, scannable proposal. Group it by action and
+show the real content, because a human approving a shared-store change needs to
+see what actually changes, not a count:
+
+```text
+MERGE (3 → 1) in repo::acme/api
+  keep/new key: db-migrations-need-explicit-tx
+  merged from:
+    - repo::acme/api::migrations-wrap-in-transaction   "wrap every migration in a tx…"
+    - repo::acme/api::tx-around-schema-changes          "schema changes need a transaction…"
+    - repo::acme/api::migration-atomicity               "migrations must be atomic…"
+  proposed value: "<the synthesised lesson>"
+
+EXPIRE (set TTL)
+    - branch::acme/api::feat-x::stub-endpoint-shape  → ttl_days: 14  (branch-scoped, short-lived)
+
+ARCHIVE (reversible)
+    - repo::acme/api::old-node-14-workaround         (node 14 dropped; superseded)
+
+DELETE (permanent — junk only)
+    - repo::acme/api::asdf                            (empty value, no key meaning)
+```
+
+Then stop and get confirmation. See
+[references/merge-and-expiry.md](../references/merge-and-expiry.md) for how to
+synthesise a merged value, choose its scope and key, pick a TTL tier, and decide
+archive vs delete — those are the judgement calls, and getting them right is the
+whole point of doing this by hand rather than by script.
+
+## Phase 5 — Apply (only after confirmation)
+
+Apply in an order that never loses information, because the store is the only
+record — there is no undo for a hard delete:
+
+1. **Fixes and merges first — write before you remove.** Write the corrected or
+   merged lesson (`memory.write`, or `lorekit write`). Confirm it landed
+   (`lorekit show <scope::key>`) *before* removing any source lesson, so a
+   failure mid-way leaves the originals intact rather than a gap.
+2. **Expiries.** `memory.write { scope, key, ttl_days }` on the lessons that get
+   a TTL, or `{ clear_ttl: true }` to make one permanent again. Same key updates
+   in place — no new lesson is created.
+3. **Removals last.** Archive (`memory.archive`, reversible) for anything a
+   future run might still want; `memory.delete { force: true }` (permanent) only
+   for the junk explicitly approved for deletion. Remove the merge sources here,
+   once their replacement is confirmed written.
+
+Preserve `tags` and provenance when you rewrite a lesson — carry the union of the
+sources' tags onto a merge so the consolidated lesson stays as findable as the
+originals were.
+
+## Phase 6 — Verify (read-only)
+
+```bash
+lorekit lint --scope <scope>              # expect exit 0 — clean
+lorekit dedupe --scope <scope> --threshold 0.85   # expect no clusters
+lorekit stats --scope <scope>             # expect a lower count than Phase 1
+```
+
+Report the before/after: lessons removed, clusters merged, expiries set, and the
+new count. Then either move to the next scope or close out the pass.
+
+---
+
+## Guardrails
+
+- **Read before you remove.** Never delete a merge source until its replacement
+  is written and confirmed.
+- **Archive beats delete.** Prefer the reversible path unless the lesson is
+  provably junk (a lint casualty) and the human approved permanent removal.
+- **The score is a hint, not a verdict.** A dedupe cluster is a candidate to
+  read, not a merge to execute.
+- **One scope, one confirmation.** Keep each proposed batch small enough to
+  actually review.
+- **Someone else's lesson deserves more caution.** A lesson tagged from another
+  agent or a teammate is not automatically yours to delete — when in doubt,
+  archive rather than hard-delete, and flag it in the plan.

package/src/config.mjs

@@ -12,8 +12,10 @@ export const SKILL_SOURCE = path.join(PKG_ROOT, 'skill', SKILL_NAME);
 // Every skill the CLI ships. `lorekit-memory` is the operational read/write
 // loop (kept first as the primary — `SKILL_NAME`/`SKILL_SOURCE` above alias it
 // for back-compat); `lorekit-setup` is the authoring skill that wires a
-// self-improvement loop into a host. install/uninstall/doctor iterate this list.
-export const SKILLS = ['lorekit-memory', 'lorekit-setup'].map((name) => ({
+// self-improvement loop into a host; `lorekit-groom` is the maintenance skill
+// that dedupes/lints/merges/expires an accumulated store. install/uninstall/
+// doctor iterate this list.
+export const SKILLS = ['lorekit-memory', 'lorekit-setup', 'lorekit-groom'].map((name) => ({
   name,
   source: path.join(PKG_ROOT, 'skill', name),
 }));

package/src/mcp.mjs

@@ -154,6 +154,12 @@ export async function restFetch(baseUrl, token, path, { method = 'GET', body, ti
       ...(token ? { authorization: `Bearer ${token}` } : {}),
       ...(traceparent ? { traceparent } : {}),
       ...(correlationId ? { 'x-lorekit-correlation-id': correlationId } : {}),
+      // Name the calling surface so usage analytics can tell a CLI read from a
+      // dashboard one. Not cosmetic: `GET /memories/read-activity` EXCLUDES the
+      // `dashboard` client (a human browsing lore is not consuming it), so a
+      // caller that wants its reads counted has to be attributable. Constant,
+      // never env-driven — this binary is always the CLI.
+      'x-lorekit-client': 'cli',
     };
     const res = await fetch(url, {
       method,