superlocalmemory 4.1.24 → 4.2.2

package/.claude-plugin/marketplace.json

@@ -9,7 +9,7 @@
       "author": {
         "name": "Qualixar"
       },
-      "description": "Local-first agent memory + reversible context compression and KV cache, as an MCP server. Serves the 56-tool full profile by default; set SLM_MCP_PROFILE=code for the 38-tool code profile.",
+      "description": "Local-first agent memory + reversible context compression and KV cache, as an MCP server, with optional pictures, PDFs and bot messaging. Serves the 58-tool full profile by default; set SLM_MCP_PROFILE=code for the 39-tool code profile.",
       "homepage": "https://github.com/qualixar/superlocalmemory",
       "keywords": [
         "memory",
@@ -21,7 +21,7 @@
       "license": "AGPL-3.0-or-later",
       "name": "superlocalmemory",
       "source": "./plugin",
-      "version": "4.1.24"
+      "version": "4.2.2"
     },
     {
       "author": {
@@ -39,7 +39,7 @@
       "license": "AGPL-3.0-or-later",
       "name": "superlocalmemory-codex",
       "source": "./codex-plugin",
-      "version": "4.1.24"
+      "version": "4.2.2"
     }
   ]
 }

package/CHANGELOG.md

@@ -5,6 +5,393 @@ All notable changes to SuperLocalMemory will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [4.2.2] — Recall never refuses answers it has after a model change, and "What am I working on?" answers from your work
+
+A fix release. Upgrading never moves or deletes memory. Text stays on the model it uses today.
+
+### Fixes
+
+- **Moving text to the newer memory engine could make recall refuse answers it had.**
+  Recall decides "this is in your memory" with a check fitted on one text model
+  (nomic). After "Upgrade memory engine" (the Settings card, or `slm embedder upgrade`)
+  moved text to EmbeddingGemma 2, that check scored correct memories far
+  below its bar, and recall said "That isn't in SuperLocalMemory" to questions it
+  had answered before. The check is now chosen by the text model in use, and
+  SuperLocalMemory no longer moves text to a model the check was not fitted for,
+  whichever way it is asked (the Settings card, the embedding picker, the config file,
+  the CLI or MCP); only `slm embedder switch --force` does. Measured on the release harness (a synthetic store): with the old check, EmbeddingGemma 2 refused all 203 real questions. A check fitted for it found every real answer on questions it had not seen, but also answered questions with no answer (it refused 69% of made-up questions; nomic refuses 98.5%), so it is not used: text stays on nomic, and Gemma keeps serving pictures and documents.
+- **Two models, one memory.** Text is searched with the text model (nomic); if you
+  turn on pictures and documents, EmbeddingGemma 2 reads those, and one recall returns
+  both. The picture model unloads after 30 minutes unused. "Upgrade memory engine" is
+  no longer offered: the Settings card now appears only for a store whose text already
+  moved, the npm installer no longer offers it, and `slm embedder upgrade` is hidden.
+- **Install, three ways, the same result.** The README now has a prompt to paste into
+  any agent (Claude Code, Codex, Cursor, …), and the npm and pip commands. The agent
+  setup steps ask the person one question, whether to turn on pictures and documents,
+  and `npm install -g superlocalmemory --media` turns them on from the start.
+- **If your text already moved,** the Settings card, `slm doctor` (new check
+  "Answer check fit") and the MCP `health` tool say so and how to go back: Roll back on the card, or
+  `slm embedder rollback`. Your previous vectors are kept, so it takes minutes, and
+  the rollback message now says that instead of promising a full re-index.
+- **A model switch kept its hub baselines.** Each memory carries a baseline that
+  stops a few "close to everything" memories from looking like answers. A switch
+  renamed the vector columns, a trigger followed the old column, and the clean-up
+  afterwards deleted every baseline (27,236 to 0 on one store). The trigger is now
+  re-bound on every switch, a store an earlier switch damaged is repaired by itself,
+  and the new model's baselines are measured before the switch finishes.
+- **`slm embedder calibrate` works on a real install.** It stopped on two settings
+  that only its own tests had (`config.profile_id`, `RetrievalEngine._channels`). It
+  is now tested against a real engine, and a test checks every setting it reads
+  exists.
+- **"What am I working on?" answers from your work.** The newest rows on a busy
+  store were the hooks' own bookkeeping ("[project] session ended …",
+  "File changed: …", "files: …") and the prompts you typed to an agent, so they
+  won a question about the present. Bookkeeping is now recognised in all its shapes,
+  flagged when it is saved, and the rows already stored are flagged once on upgrade
+  (nothing deleted; they stay listable). Your captured prompts stay searchable but
+  rank as requests: below what you and your agents saved, and never as the answer
+  to "what am I working on?". Ask "what did I ask for…" and they come back first.
+
+### Upgrading
+
+- The store's schema version moves to 62 (a data repair, no new table). Going back
+  to 4.2.1 needs `slm db prepare-downgrade` first, as for earlier releases.
+
+## [4.2.1] — Each person sees only their own workspace, and moving to the newer memory engine finishes
+
+A fix release. Upgrading never moves or deletes memory.
+
+### Security
+
+- **Recall Lab, Memories search and memory chat stay in the signed-in person's
+  workspace.** These three dashboard requests checked that a signed-in person could
+  read their workspace, then searched the machine's active workspace instead. With
+  more than one workspace, someone signed in to one could read another's memories.
+  They now search only the workspace that was checked, and a test signs in two people
+  in two workspaces and fails if either sees the other's memories. Saved views, the
+  REST `/recall` route, MCP and the CLI were not affected.
+
+### Fixes
+
+- **Moving to the newer memory engine no longer stops at a long memory.** A memory
+  over 8,000 characters made the EmbeddingGemma 2 worker refuse its batch, and the
+  whole re-index stopped. It is now embedded from its start (the model reads about
+  that much anyway), the same way the previous engine handled it.
+- **Hermes plugin keeps working after an SLM upgrade.** It refused every `/slm`
+  command unless SLM was exactly the release it shipped with. It now works with that
+  release and every newer one of the same major version, and its message says how to
+  move a pinned plugin ([Hermes](docs/hermes.md#upgrade)).
+- **Recall Lab:** clicking a result opens the whole memory, not the shortened text.
+- **Dashboard:** a source opened from "Ask your memory" opens that memory in the
+  Memories tab. Settings no longer comes up blank after moving between panes.
+- **Choose folder:** while remote access is on, folders cannot be connected; the page
+  now says so next to the button before anyone clicks, instead of after a click below
+  the fold.
+- **Sign-in:** when every person must sign in (company mode), a signed-out visitor
+  gets one sign-in screen instead of panes that never load, and a signed-in person
+  sees who they are, their role and workspace, with Sign out. On a personal computer
+  nothing changes: the owner never signs in. Governance says so plainly, and the first
+  person you add starts as an admin. There is no default account or password.
+- **README:** the product film follows the twelve superpowers; the architecture
+  diagram is redrawn.
+
+## [4.2.0] — Recall that says "That isn't in SuperLocalMemory", memory that looks after itself, pictures and documents
+
+This release changes what recall does when memory cannot answer, adds a Memory
+health service that improves older memories without touching what you wrote,
+and lets SLM remember pictures and documents. It also closes a long list of
+erasure, access-control and backup gaps. Upgrading never moves or deletes
+memory, and an update that changes the store takes a restore point first.
+
+### Many agents at once
+
+- **A question waits for its full answer instead of getting a keyword guess.** When
+  several agents ask at once, an agent's recall, `slm recall`, the Recall Lab and
+  saved views now wait their turn (up to about twelve seconds) and run the full
+  pipeline. If every slot is still busy, they get a plain "SuperLocalMemory is
+  answering other questions right now. Ask again in about N seconds" (HTTP 503 with
+  `Retry-After`, `code: RECALL_BUSY` over MCP, exit code 75 from the CLI), never a
+  stale keyword answer presented as recall. Hooks that run before a prompt keep their
+  three-second limit, so typing is never held up. Measured on a 27,000-memory store
+  with twelve agents asking at once: twelve full answers in 2.4 to 5.4 seconds, where
+  before two of the twelve fell back to keywords.
+- **The same question asked twice at once runs once.** Identical recalls in flight
+  share one run, and a complete answer is reused for a few seconds while nothing is
+  written. Any write (remember, update, delete, forget, a scope change) drops it at
+  once, so an agent never reads an answer from before its own write.
+- **Background work steps aside for questions.** Local-model extraction and summaries
+  in Mode B now wait while a recall is running (never more than twenty seconds), so a
+  long generation no longer sits in front of every recall.
+- **The dashboard reads once for everyone.** Memory health, totals and the compliance
+  pane are computed once and shared for a few seconds between every open dashboard;
+  a repeat look answers in milliseconds. The audit trail is opened once, with an
+  index for the workspace it is read for.
+- **"What am I working on" answers with your newest memories.** A present-tense
+  question no longer faces the word-match test (its answer shares no word with it),
+  so it stops saying "That isn't in SuperLocalMemory."
+- **Dashboard:** a "please wait" bar with the mascot while SLM starts or warms up;
+  loading and error messages carry the mascot; the folder dialog comes to the front
+  instead of opening behind other windows; Settings rows wrap instead of squeezing a
+  label to one letter per line.
+
+### Recall quality
+
+- **Recall says so when nothing supports the question.** Every recall now comes back
+  in one of three tiers. *Supported*: at least one memory is strong enough evidence.
+  *Closest memories*: nothing supports it, but the question is a real one and a few
+  memories are close, so you see up to three, labelled, under "I don't have this
+  exactly — here are the closest memories." *Not in memory*: a nonsense question, or
+  nothing close, returns no results and the sentence "That isn't in
+  SuperLocalMemory." Before this, a nonsense question could still return the nearest
+  memories. Each memory is scored on several signals together (meaning, keywords,
+  names, rare terms, how the question is phrased, and how often the memory matches
+  everything); pinned memories are always returned. The command line and the
+  dashboard show the closest memories by default; over MCP and the REST API an agent
+  gets them when it asks (`include_related`). It is far better at rejecting nonsense
+  than at spotting an on-topic question memory has no answer to, which is what the
+  answer check is for ([Recall](docs/recall.md#when-memory-cannot-answer)).
+- **`slm recall --show-weak`** (the `show_weak` option on the MCP recall tool, the
+  REST API and the dashboard's "Show closest matches") returns the closest weak
+  matches when nothing supports the question, each marked weak, with the same
+  sentence on top.
+- **A recall that runs before SLM is ready says so.** For a few minutes after a start
+  the models, probes and memory graph are still loading. Recall then shows the
+  keyword finds it has (up to five) under "SuperLocalMemory is still warming up —
+  results may be incomplete", and never says the memory is empty. The response
+  carries `warming_up` and `status_note` for agents.
+- **Layered recall.** `depth` 0 returns ids, the tier and a one-line preview; 1 (the
+  default) the full facts; 2 adds the source memory, up to three neighbours from the
+  same session, up to three facts about the same entities and the correction that
+  replaced it. `max_tokens` is a hard cap that is never exceeded, a new `expand`
+  tool drills into one result, and every injected context block is capped too.
+  `slm recall --depth 2 --max-tokens 1500`.
+- **A recall that matches by keyword alone is no longer written off**, and the
+  keyword fallback serves only memories that cover the words you asked with.
+  Questions are also rewritten into base forms and known aliases for the keyword
+  search; the check that decides whether a memory answers always reads your
+  original words, so a rewrite can never make nonsense answer.
+- **The on-device answer check (Laya) is installed by default where it can run.** On
+  an Apple Silicon Mac, setup installs it (about 1.1 GB) without asking in unattended
+  runs, and the first command after an upgrade offers it the same way: it asks in a
+  terminal and starts a background install without one. It is independent of the
+  operating mode, so Mode A and Mode B both run it entirely on your computer. Opt out
+  with `slm setup --no-laya` or `SLM_SKIP_LAYA=1`; answering no once is remembered. It is
+  never installed on other hardware, over an install you already have, or when you chose
+  Off or the online check. The check has its own time budget, so a slow check can no
+  longer eat the recall ceiling. The online option (Jev) stays off until you choose it
+  and tick its consent.
+- **Local checks are calibrated in the background**, with visible progress and a
+  cancel button, and a new calibration is rejected if it does worse than the
+  current one on held-out questions.
+- **Recall stays fast while you write.** A save no longer forces the whole graph
+  to rebuild on the next recall: the cached graph follows writes, deletes and
+  erasures, and full rebuilds run in the background. On a 26,000-fact store with
+  a write every 2 seconds, p95 recall fell from 4.3 s to 0.94 s and peak memory
+  from up to 1.7 GB to about 260 MB. Large stores get new recall indexes, and
+  recall waits are bounded all the way through the HTTP layer.
+- **Entity matching is exact.** A name matches exactly, then after
+  normalisation, then with one typo in one word. It never merges different
+  numbers or versions, never chains near-miss names into an unrelated person or
+  project, and timestamps, hashes, paths, status words and sentence fragments
+  are no longer saved as people or projects. `slm db repair-entities` previews,
+  applies (after a backup, with an undo record) and undoes the clean-up of links
+  left by the older rules.
+- **Saving is checked against what you wrote.** Every statement SLM extracts is
+  checked against its source before it is stored: no added names, numbers or
+  dates, no flipped negation. The original note is always kept. Quoted speech is
+  never rewritten, relative dates ("last Tuesday") resolve against the memory's
+  own date, and saving the same text twice adds no row.
+- **The learning loop learns from real outcomes.** One query id runs from recall
+  to the reward, and each outcome settles exactly one recall, across restarts.
+  `slm outcome success|partial|failure <fact_id>...` tells SLM whether the
+  memories a recall gave you helped; MCP, the REST API, the dashboard and the
+  command share one writer. Health shows "Not demonstrated" until an outcome has
+  settled.
+- **Upgrade memory engine** (preview, opt-in): `slm embedder upgrade` or the
+  Settings card re-reads memories with the one-model engine in the background,
+  with recall working throughout and one-click rollback.
+- **Quantized retrieval is used when configured**, the status says when a tier is
+  degraded, and damaged or older sketches are repaired by `slm quantize` instead
+  of hiding the memory from search.
+
+### Memory health
+
+- **A Memory health card** in the dashboard shows what SLM can improve in older
+  memories and what it is doing about it. A background worker runs only when the
+  computer is idle and **only adds**: it fills in missing observation and event
+  dates, names of people and tools, tags, a project and a kind on older memories,
+  and never edits the text you saved or replaces a value you set. It stops when you
+  are working and when memory is tight ([Memory health](docs/memory-health.md)).
+- **Stubs are marked, not hidden.** A memory that is only a label and an id
+  (for example a session receipt) carries a **stub** badge in lists and counts.
+  Only recall leaves stubs out of its answers.
+- **Anything that deletes or moves something waits for your explicit Fix.**
+  Repairs that remove junk entity links, re-file hidden memories or repair failed
+  saves are offered as a count with a **Fix** button, and every run that changed
+  something has an **Undo** that restores exactly what it changed (anything you
+  edited since is kept).
+- **`slm enrich`** is the command-line side: `status`, `pause`, `resume`,
+  `undo`, and `fix entity-links|refile-hidden|repair-saves` (the last has no
+  undo, so it takes a safety copy first and says so). `--include-undone`
+  applies items from a repair you undid earlier.
+- **One safety copy per store**, taken before the first automatic change and
+  reused by every later run.
+- **`slm doctor`** reports the same state, a **Failed saves** row (it warns when
+  saves were given up on after repeated attempts and fails at 50 or more), and
+  `slm doctor --refile-hidden` lists memories that were moved out of sight
+  although their retention score says to keep them; `--yes` moves exactly those
+  back.
+
+### Privacy and erasure
+
+- **Erasing a person leaves no readable copy.** The bytes of the data folder are
+  checked after an erasure and everything they found is closed: queued saves,
+  the encrypted request journal, the recall queue, caches, reversible
+  compression originals, free pages inside the side databases, event previews,
+  the vector index itself, the Lance history, the graph's entity nodes, an erased
+  picture's backup mirror and the prompt text two logs kept.
+- **Erasure says exactly what it did.** `slm gdpr erase --json` reports
+  `success: false` for an incomplete erasure, the dashboard lists what was
+  removed and what remains, and the stored receipt is written after every step.
+  A backup keeps its erasure obligation until it is deleted or re-read clean, and
+  a restore replays targeted erasures as well as profile ones.
+- **Exporting your data works on a real store.** Export writes plain JSON,
+  leaves embeddings out unless you ask (`--include-embeddings`), streams to
+  disk, and reports failures instead of "Export started". The full export now
+  carries the audit records about you, pictures, documents, folders and jobs,
+  and lists what it leaves out under `export_manifest`.
+- **Personal data is redacted on every save path** when redaction is on,
+  including metadata, tags, keys and events, for pictures as well as text.
+- **Mode A says what it is:** "Memories are saved and recalled without a
+  language model — nothing leaves this device." The server serves that one
+  sentence and the dashboard shows it, and it stays true when the on-device
+  answer check is running.
+
+### Access control
+
+- **Company mode checks a role on every change and every data route.** Every
+  state-changing route asks for a role before the request body is read, so a
+  refusal is 401 or 403, never 422 or 500; with login required, every page of
+  data needs a signed-in user. While users are enrolled, the install token or an
+  API key alone no longer administers the workspace; use an admin account, or
+  `slm team` on the SLM computer (`slm team status`, `slm team policy
+  --require-login on|off`, so the owner can never be locked out).
+- **Skill approvals and maintenance are scoped to the workspace.** Skill
+  evolution keeps each workspace's skills, backups and quarantine apart; an
+  approval applies to the workspace it was made in, and installation-wide
+  settings need authority over every workspace.
+- **Roles and profiles apply to every picture, document, folder and feature
+  route.** Reading a file by path, connecting a folder and turning features on
+  need the owner or an admin, and SLM's own data folder can never be named.
+- **A connected web app cannot see or change what it may not read.** It cannot
+  update, delete, pin, replace or give feedback on a memory by id when it may not
+  see that memory; it is answered as if the memory did not exist. Pictures saved
+  before this release are held back from web apps until they are re-checked in
+  the background.
+- With `SLM_REQUIRE_CREDENTIALS=1` the dashboard key is no longer handed to other
+  programs on the computer; the dashboard asks for it once (`slm token show`).
+
+### Backup and restore
+
+- **A restore is all or nothing.** While it runs, the store is held so no daemon
+  can start and write part-way; everything the restore may change is copied
+  first, and a failure puts those files back. Only if that put-back also fails
+  does `slm backup restore-safety-copy` bring back the safety copy taken after
+  the service stopped. The dashboard and the command line say in plain words
+  what happened and what to run next.
+- **Backups are complete and honest.** Local backups copy picture and PDF
+  originals (never the upload-link database), a backup just taken is no longer
+  "overdue", and the dashboard export is one archive with every store, the
+  originals and a manifest.
+- **A damaged keyword index is repaired for good.** On SQLite 3.44.0 to 3.46.0,
+  `slm db repair --apply` turns off the setting that damages the index, rebuilds
+  it, checks it again and says plainly if it is still damaged. Your memories are
+  kept.
+
+### Dashboard
+
+- **Documents & Images** pane with progress, a drop zone, **Find a picture**,
+  connected folders and a **Choose folder** button that uses the computer's own
+  picker; **Bot messages** tab listing every bot by name with mute, rename and
+  retire; a **What's new** card that knows what is already on.
+- **Connected apps shows which apps are connected right now**: live MCP
+  processes by host with a count, a flag on apps still running older SLM code,
+  and a reconnect. Connect, doctor, repair and restart are buttons.
+- **Search says why it found nothing.** The Memories search shows the tier's sentence
+  ("That isn't in SuperLocalMemory.", the closest memories, or the warming-up note)
+  instead of an empty list, and has a **Show closest matches** switch.
+- **Answer Check** tab, Settings → Answer check with a test for each option and
+  **Calibrate on my store**, and the Memory health card described above.
+- Memory guards read real memory use on macOS (physical footprint); the
+  resident size under-reported up to 15 times on a busy Mac.
+
+### Pictures and documents
+
+- **Save screenshots, photos and PDFs and find them by describing them.** A
+  picture model runs on your computer (EmbeddingGemma 2), OCR reads the text
+  inside pictures, and each PDF page is indexed by its picture and its text. No
+  generative model looks at your files. Off by default; turn it on from the npm
+  installer or upgrader, the dashboard, or `slm media enable`. Pictures live in
+  their own database, so your existing memories and embeddings are never
+  changed. It needs a computer with 16 GB of memory and downloads about 1.5 GB the
+  first time; limits are in `docs/pictures-and-documents.md`.
+- `slm media status|disable|gc|repair`; `slm sources add` mirrors **folders and
+  Obsidian vaults** read-only (properties, links and canvases kept) and re-reads a
+  file as soon as it changes.
+- **Recall finds pictures and pages** beside text memories, shows them to local AI
+  apps, and counts a picture as evidence only above the model's own similarity
+  floor. The generated picture caption stays in the picture's metadata and never
+  becomes searchable words or facts.
+- **Erasing a memory erases its picture**, and a replaced picture or PDF is
+  erased for good at the end of its grace period.
+- Pictures and documents work on Linux 64-bit ARM, are refused with a plain
+  reason on computers under 16 GB, and the picture tools are stopped before they
+  pass their memory cap. The picture model embeds in small passes; its peak fell
+  from about 12 GB to under 4 GB on large batches.
+
+### Bots and web apps
+
+- **Bots that talk to each other.** Web apps connected through Web access
+  (ChatGPT, Claude on the web, Grok Bot, Muse, Composio) can join SLM-Mesh beside
+  the agents on your computer: see peers, send a named bot a message, wait for a
+  reply and read shared state. Each app needs two yeses: a box on the approval
+  page and a switch on the **Web access** row in Connected apps (or `slm remote
+  keys allow <key> mesh`). Messages reach a web app at least once, marked as a
+  repeat if it did not acknowledge them.
+- **Web apps save real files.** A connected app can ask for a one-time upload
+  link: you open it, pick the file, and it streams to your computer through the
+  connection gateway, which keeps no copy. A link belongs to the app that asked
+  for it, works once and expires in 10 minutes. ChatGPT can hand over an
+  attached file directly once its file hosts are on your allowed list.
+- **Frameworks and editors.** Google ADK, CrewAI and Semantic Kernel run through
+  an isolated engine bridge instead of failing to install next to SLM; Continue,
+  OpenCode, Antigravity, VS Code and Gemini CLI setup write each host's current
+  schema; and Claude Desktop on Windows is configured in `%APPDATA%\Claude`.
+- Every plugin, skill and setup text now teaches this release: a new `slm-media`
+  skill, and the mesh, web access, governance and bot skills cover upload links,
+  `mesh_wait` and the Connected apps switches. Hermes registers all 16 skills
+  and its four advisor tools load again.
+
+### Reliability
+
+- **Windows reads and writes SLM's own files as UTF-8**: the dashboard, the
+  answer-check settings and the backup reader no longer depend on the Windows
+  code page, so text outside it round-trips.
+- Services and ports belong to one install: `slm dashboard`, Health, Settings and
+  HTTP setup use the configured port, strict credential settings survive
+  auto-start, and a second install gets its own service name. A daemon started
+  with `python -m` publishes its ready record, so `stop` no longer waits 90
+  seconds.
+- `--sync` saves return after enrichment completes, the Python API shares the
+  daemon's embedder, failed session opens exit non-zero, and a queued embedding
+  upgrade says why it waits. A model refused for lack of memory says so instead
+  of "no vector", and an embedding pause explains itself in the model status.
+- `npm install` says plainly when the Python it found cannot work (an Intel Mac,
+  32-bit Windows, Windows on ARM, 32-bit Linux, Node under Rosetta) and how to
+  install a supported one; pip installs on 64-bit ARM Linux.
+- `slm doctor` makes at most one short health probe of the daemon, never starts
+  it, and prints a full report within a hard deadline instead of hanging.
+
 ## [4.1.24] — Connecting ChatGPT, Grok Bot, Muse and dots works the first time
 
 ### Added