memtrails 0.3.0 → 0.4.1

package/CHANGELOG.md

@@ -3,6 +3,62 @@
 All notable changes are listed here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and
 versions follow [Semantic Versioning](https://semver.org/).
 
+## [Unreleased]
+
+## [0.4.1] — 2026-11-05
+
+- Native Python (`pip install memtrails`, asyncio) and Rust (`cargo add memtrails`, tokio) versions, with the same
+  results and the same files as this library: held to shared answers (`conformance/`), and a memory folder written by
+  any of the three is searched the same by the others (`interop/`).
+- Fixes: search no longer shows a short record's own summary again as the passage that matched (`match`, and after
+  " … " in `searchText`); a match is now only ever a passage beyond the start of what is shown.
+- Fixes: the downloaded Qdrant binary and embedding model are only run or loaded from folders that are this user's
+  alone. The cache folder (and the model's folder) is made private when it is created or found open to others, and
+  refused with a clear error when it belongs to someone else or is a link. `fromFiles` no longer reads through a note
+  that was replaced by a link after the folder was listed, and `fromJsonl` and `fromMbox` skip a line longer than
+  64 MiB (with the reason) instead of holding it in memory.
+- Fixes: a rebuild that is stopped (a close, a newer rebuild) returns only once the batches it was writing are written,
+  so none of them lands in the index after it.
+- Fixes: `close()` while memory's own Qdrant is being restarted (after it died) stops that restart and waits for the
+  Qdrant it began; one could come up after the close and stay running.
+- Fixes: an add no longer runs beside a rebuild (a record could be logged but never indexed, then skipped as unchanged
+  when added again); `close()` waits for the adds and searches in flight, and every call after it is refused with
+  "memory is closed".
+- `LlmRequest.signal` (optional): aborted when no one will read the answer any more; `openAiAsk` stops its call and its
+  retries. Asks made side by side (consolidation's windows, cues and reconcile batches, gathering's parts, sleep's
+  relate batches, profiles) stop the others when one fails. Consolidation that fails or is stopped starts no further
+  extraction and returns only once the ones being read are over; before, they ran on, paid for, after it returned.
+
+## [0.4.0] — 2026-11-04
+
+- Prompt examples are invented now. The extraction and gathering prompts had a few example sentences taken from LoCoMo
+  conversations, one of them (in `gather`'s notes rule) from a held-out conversation. Their effect on the published
+  results, measured on the judged answers: leaving out every held-out question that mentions any of those words or
+  people (48 questions) gives 81.1 against Mem0's 76.6 (+111/−55), against 80.9 and 76.5 on all 1,307. The question
+  the example came from was answered wrong by both systems.
+- The search index layout is 7: `kinds` now selects records by their kind on every point and learned items by theirs,
+  so an existing index is rebuilt from the logs on the next open.
+- Fixes: a crash in the middle of a write no longer loses the records added after it; consolidation no longer
+  overwrites records changed or removed while the model answers; a title cut through an emoji no longer fails a write;
+  a thread that never goes quiet is consolidated; imported history is related and kept like new knowledge (`learnedAt`);
+  a malformed `from`/`to` is an error; `gather` and `sleep` report `paused` when the model did not read.
+- `bench/audited.ts`: judged LoCoMo answers without the questions whose golden answer an independent audit
+  ([locomo-audit](https://github.com/dial481/locomo-audit)) found wrong, dropped for every system alike. As shipped,
+  84.8 against Mem0's 79.9 (+109/−49).
+- `gather`: a memory a small model keeps by quoting its whole line (`[m3] (2023-06-03) …`) now counts; it read as nothing
+  kept before.
+
+- Security: the secret filter catches common key formats it missed (GITHUB_TOKEN=, AWS_SECRET_ACCESS_KEY=, JSON-style
+  secrets, GitLab, Hugging Face, npm, SendGrid, Telegram, the whole password of a URL); crafted e-mail and HTML can no
+  longer make the importers or the filter take quadratic time; downloads are verified on disk in a temp folder of their
+  own under a lock, with a stall timeout, and the Qdrant binary runs only while it is this user's alone; Qdrant gets a
+  minimal environment and must prove it is ours before it is given the key; an OpenAI key is sent to OpenAI only; the
+  memory folder is private; Slack channel folders cannot lead out of the export.
+- One process at a time in a memory folder (a lock); consolidate, sleep, rebuild and remove run one at a time; `close()`
+  stops the running job and waits for it; an open that fails stops what it started; `remove()` takes the record's kept
+  episode out of the long-term log too; `MEMTRAILS_EMBED_URL` and `MEMTRAILS_RERANK_URL` are used.
+- The release is built and tested in a job without publishing rights; actions are pinned to commits.
+
 ## [0.3.0] — 2026-11-02
 
 - `gather(q, { notes: true })`: the same model call also writes what the kept memories say that answers the question,

package/README.md

@@ -17,7 +17,7 @@ import { openMemory } from 'memtrails';
 const memory = await openMemory({ home: './memory' });
 
 await memory.add([
-  { id: 'msg-1', kind: 'email', author: 'Ana', title: 'Gateway certificate expired', at: '2026-04-02',
+  { id: 'msg-1', kind: 'email', author: 'Ana', title: 'Proxy certificate expired', at: '2026-04-02',
     text: 'The TLS certificate expired on the 1st. Renewal stopped because the cron job was removed in March.' },
   { id: 'c9f1e2', kind: 'commit', author: 'Ion', title: 'Restore certificate renewal cron', at: '2026-04-03',
     text: 'Files: deploy/cron/renew.sh' },
@@ -41,7 +41,8 @@ await memory.close();
   against 50.3% for plain search, and most of a cross-encoder reranker's gain, in about 3 ms on one CPU thread.
 - **Measured, honestly.** On all 1,307 held-out LoCoMo questions with 6,000 characters to read, memtrails alone on a
   CPU scores 77.4 (strict judge) against Mem0's 76.5 — a tie, reported as one; with `gather` reading what search
-  found and noting what it says, 80.9 — ahead beyond chance. [Benchmarks](docs/BENCHMARKS.md).
+  found and noting what it says, 80.9 — ahead beyond chance; 84.8 against 79.9 without the questions whose official
+  answer an independent audit found wrong. [Benchmarks](docs/BENCHMARKS.md).
 - **Explorable.** A concept graph links what records share — people, places, systems, names — so an agent can walk
   from what it found to what a search for the words does not find.
 - **Built for agents.** `memory.tools()` gives `memory_search`, `memory_open` and `memory_explore` with zod and JSON
@@ -57,6 +58,17 @@ Node.js 22 or newer, on macOS or Linux (x64 or arm64). Windows has the pieces it
 (218 MB, from Hugging Face) and the Qdrant binary (~30 MB, from Qdrant's GitHub release) into `~/.cache/memtrails`,
 each checked against a pinned SHA-256. Set `MEMTRAILS_CACHE` to put them elsewhere.
 
+### Python and Rust
+
+The same library, written natively in each language, fully async, with the same results and the same files: a memory
+folder written by one opens in the others (checked on every change by [`interop/`](https://github.com/cmaftuleac/memtrails/tree/main/interop) and held to the shared
+answers in [`conformance/`](https://github.com/cmaftuleac/memtrails/tree/main/conformance)).
+
+```bash
+pip install memtrails          # Python 3.10–3.13, asyncio — see python/README.md
+cargo add memtrails            # Rust 1.88+, tokio — see rust/README.md
+```
+
 ## Adding data
 
 A **record** is anything with text. Only `id` and `text` are required.
@@ -67,9 +79,9 @@ await memory.add({
   source: 'work-mailbox',          // part of the identity
   kind: 'email',                   // any word: email, message, note, memo, article, commit, ticket, …
   collection: 'ops',               // search can be scoped to collections
-  thread: 'gateway-cert',          // records that belong together
+  thread: 'proxy-cert',          // records that belong together
   author: 'Ana Popescu',
-  title: 'Gateway certificate expired',
+  title: 'Proxy certificate expired',
   text: '…',
   at: '2026-04-02T09:14:00Z',      // ms, ISO string or Date
   url: 'https://mail.example/…',
@@ -111,7 +123,7 @@ await memory.search({ query: 'who renewed the certificate?', limit: 8 });
 await memory.search({ query: 'standup notes', collections: ['ops'], kinds: ['message'], from: '2026-04', to: '2026-04' });
 
 // The same results as a reader is shown them: one line a memory, best first or oldest first.
-await memory.searchText({ query: 'what happened with the gateway?', order: 'time' });
+await memory.searchText({ query: 'what happened with the proxy?', order: 'time' });
 
 // The best sentences of what was found, for a small context window.
 await memory.room({ query: 'what made the renewal stop?', chars: 6000 });
@@ -120,7 +132,7 @@ await memory.room({ query: 'what made the renewal stop?', chars: 6000 });
 await memory.open(hits[0].id);
 
 // What the concept graph reaches from a topic or from items, that search did not find.
-await memory.explore({ topic: 'the gateway' });
+await memory.explore({ topic: 'the proxy' });
 ```
 
 With dates, what happened within them comes first and the rest follows: a question that names a time is rarely only
@@ -141,7 +153,7 @@ const memory = await openMemory({
 });
 await memory.consolidate();   // after adding
 await memory.sleep();         // once a day
-await memory.gather({ query: 'every doctor I saw this year' });   // a set, for counting and listing questions
+await memory.gather({ query: 'every plumber we called this year' });   // a set, for counting and listing questions
 ```
 
 Any provider works: pass `llm: { baseUrl, model, apiKey }` for an OpenAI-compatible server (a hosted model, vLLM,
@@ -174,7 +186,7 @@ for (const t of tools) server.tool(t.name, t.description, t.inputSchema, t.handl
 | Environment | |
 |---|---|
 | `MEMTRAILS_CACHE` | Where downloads live (default `~/.cache/memtrails`). |
-| `MEMTRAILS_MODEL`, `MEMTRAILS_LLM_API_KEY` or `OPENAI_API_KEY`, `MEMTRAILS_LLM_BASE_URL` | The language model when `llm` is not given. |
+| `MEMTRAILS_MODEL`, `MEMTRAILS_LLM_API_KEY` or `OPENAI_API_KEY`, `MEMTRAILS_LLM_BASE_URL` | The language model when `llm` is not given. `OPENAI_API_KEY` is sent only to OpenAI's own address. |
 | `MEMTRAILS_EMBED_URL`, `MEMTRAILS_RERANK_URL` (+ `_KEY`, `_MODEL`) | Servers, when not given in options. |
 
 ## How it works
@@ -191,17 +203,21 @@ the [API reference](docs/API.md), and [what was tried and why](docs/RESEARCH.md)
 |---|---|---|---|
 | LoCoMo, all 1,307 held-out questions, 6,000 characters, strict judge, `gather` at 40 with notes | **80.9** | 76.5 | — |
 | the same, Mem0's own lenient judge | **91.9** | 87.8 | — |
+| the same, strict, without the 80 questions whose golden answer is wrong³ | **84.8** | 79.9 | — |
 | the same, `gather` without notes, strict / lenient | 79.5 / 91.4 | 76.5 / 87.8 | — |
 | the same with `search` alone, no model when searching, strict / lenient | 77.4 / 89.3 | 76.5 / 87.8 | — |
 | LoCoMo, 1,307 held-out questions, 28,000 characters, one search¹ | 79.9 | 80.6 | 42.1 |
 | LoCoMo, the same with an agent using each system's own tools¹ | **79.0** | 77.1 | 47.0 |
+| LongMemEval-S, 60-question sample, 6,000 characters, strict judge, `gather` at 40 with notes (a tie) | 90.0 | 88.3 | — |
 | LongMemEval-S retrieval, strict all sessions @5 (467)² | **96.15** | — | — |
 
 The same reader and judges for every system. With `gather` (a model reads what search found, keeps what the answer
 needs and notes what it says) memtrails leads Mem0 beyond chance: +113/−55 questions strict, +78/−25 lenient (both
 p < 0.001). With `search` alone,
 the small-room and one-search comparisons are ties; the agent lead is
-small and was judged once. ¹ An earlier build (2026-10-27). ² gbrain-evals publishes 95.96 on the same count: a tie. [Full results, protocol and limitations](docs/BENCHMARKS.md).
+small and was judged once. ¹ An earlier build (2026-10-27). ² gbrain-evals publishes 95.96 on the same count: a tie. ³ As found by an
+independent audit of LoCoMo ([locomo-audit](https://github.com/dial481/locomo-audit)), dropped for both systems;
++109/−49 (p < 0.001), `bench/audited.ts` reproduces it. [Full results, protocol and limitations](docs/BENCHMARKS.md).
 
 ## Limitations
 

package/dist/core/consolidate.d.ts

@@ -7,7 +7,7 @@ export declare const SAME_EPISODE = 0.97;
  *  days are two observations, and a list question needs both. */
 export declare const SAME_OBSERVATION = 0.97;
 export declare const RELATED = 0.8;
-/** A conversation is consolidated once it has been quiet this long — or its entries are a day old. */
+/** A conversation is consolidated once it has been quiet this long — or its oldest waiting entry is a day old. */
 export declare const SETTLED_MS: number;
 /** Extraction calls in flight: they are independent, and one at a time made a backfill take days (~86 s a window). */
 export declare const EXTRACT_CONCURRENCY = 8;
@@ -34,7 +34,7 @@ export interface ConsolidateDeps extends WriterDeps {
     /** What a window's reading may keep, checked before anything is reconciled or written (e.g. facts only
      *  from the person's own words and results the agent confirmed, each with where it came from). An answer it cannot
      *  give leaves the window for the next run, as an extraction that was not answered does. */
-    guard?: (win: StmEntry[], x: Extraction) => Promise<Extraction | {
+    guard?: (win: StmEntry[], x: Extraction, signal?: AbortSignal) => Promise<Extraction | {
         unavailable: string;
     } | {
         busy: true;
@@ -57,6 +57,12 @@ export interface ConsolidationReport {
  *  them), and how many go in one ask. */
 export declare const CUE_BACKLOG = 40;
 export type Relation = 'same' | 'updates' | 'contradicts' | 'related' | 'different';
+export type Decision = {
+    relation: Relation;
+    title: string;
+    summary: string;
+    body: string;
+};
 /** Links an item keeps to others about the same subject (exploration walks them), newest last. */
 export declare const MAX_LINKS = 24;
 /** Pairs asked about together: at most 8, and at most RELATE_CHARS of their texts — a long pair on its own. */
@@ -65,6 +71,16 @@ export declare function relateBatches<T extends {
     it: MemoryItem;
     known: MemoryItem;
 }>(pairs: T[]): T[][];
+/** When an item was written to long-term memory. `ts` is its source's own time — for history imported today, long ago —
+ *  so "learned since the last sleep" and "unused for half a year" read this. An item written before the field existed
+ *  has only `ts`. */
+export declare const learnedAt: (m: {
+    ts: number;
+    learnedAt?: number;
+}) => number;
+export declare const stamped: (it: MemoryItem, at: number) => MemoryItem;
+/** The number of a label the model wrote back: `p3`, or the line it labels quoted whole (`[p3] NEW …`) — 0 for none. */
+export declare function labelNo(id: string, letter: 'p' | 'i'): number;
 export declare class Consolidator {
     private readonly d;
     private readonly vecs;
@@ -128,17 +144,16 @@ export declare class Consolidator {
         summary: string;
         body: string;
     } | null): Promise<void>;
-    /** Two items about the same subject that say different things: each lists the other (memory_explore walks these). */
+    /** Two items about the same subject that say different things: each lists the other (memory_explore walks these) —
+     *  the newest MAX_LINKS, so an item on a busy subject does not grow its list, and its payload, without end. */
     link(a: MemoryItem, b: MemoryItem): Promise<void>;
     /** Superseded: out of the index and of recall, kept in the log with what replaced it. */
     retire(known: MemoryItem, by: string): Promise<void>;
+    /** The model's decision for each pair, by its label (`p1`, …) — or why it did not answer. */
     relate(batch: Array<{
         it: MemoryItem;
         known: MemoryItem;
-    }>): Promise<Map<string, {
-        relation: Relation;
-        title: string;
-        summary: string;
-        body: string;
-    }> | null>;
+    }>, signal?: AbortSignal): Promise<Map<string, Decision> | {
+        paused: string;
+    }>;
 }

package/dist/core/consolidate.js

@@ -15,7 +15,7 @@
  * Then the window's entries are marked done and leave the short-term index: what mattered of them is long-term now.
  */
 import { redactString } from './redact.js';
-import { defaultAsk } from '../llm.js';
+import { allOrAbort, defaultAsk } from '../llm.js';
 import { episodeBody } from './episodes.js';
 import { docText } from './embedder.js';
 import { cleanConcepts, cleanCues, extractWindow, observeWindow, windowsOf } from './extract.js';
@@ -29,7 +29,7 @@ export const SAME_EPISODE = 0.97;
  *  days are two observations, and a list question needs both. */
 export const SAME_OBSERVATION = 0.97;
 export const RELATED = 0.80;
-/** A conversation is consolidated once it has been quiet this long — or its entries are a day old. */
+/** A conversation is consolidated once it has been quiet this long — or its oldest waiting entry is a day old. */
 export const SETTLED_MS = 60 * 60_000;
 /** Extraction calls in flight: they are independent, and one at a time made a backfill take days (~86 s a window). */
 export const EXTRACT_CONCURRENCY = 8;
@@ -116,32 +116,41 @@ export function relateBatches(pairs) {
         out.push(cur);
     return out;
 }
+/** When an item was written to long-term memory. `ts` is its source's own time — for history imported today, long ago —
+ *  so "learned since the last sleep" and "unused for half a year" read this. An item written before the field existed
+ *  has only `ts`. */
+export const learnedAt = (m) => m.learnedAt ?? m.ts;
+export const stamped = (it, at) => ({ ...it, learnedAt: at });
+/** The number of a label the model wrote back: `p3`, or the line it labels quoted whole (`[p3] NEW …`) — 0 for none. */
+export function labelNo(id, letter) {
+    return Number(new RegExp(`\\b${letter}(\\d+)`).exec(id)?.[1] ?? 0);
+}
 /** One remembered episode or candidate as a long-term item (not yet reconciled). */
 function itemOf(c, src, now, sources = [src]) {
     const project = c.scope === 'project' ? src.project : c.scope;
     // The files its requests changed, from their recorded steps — by code, not the model: paths are what summaries lose.
     const files = [...new Set(sources.flatMap((e) => e.files))];
-    return {
+    return stamped({
         uid: `mem:${fingerprint(c.kind, c.scope, project, c.title, c.summary)}`, kind: c.kind, scope: c.scope, project: src.project,
         engine: src.engine, session: src.session, cwd: src.cwd, where: src.title, ts: src.ts || now, firstTs: src.ts || now,
         title: c.title, summary: c.summary, body: c.body, files, appliesWhen: c.appliesWhen, confidence: c.confidence,
         importance: c.importance, status: 'active', provenance: c.provenance, seen: 1,
         fingerprint: fingerprint(c.kind, c.title, c.summary, c.body), cues: c.cues, concepts: c.concepts,
         ...(c.origin ? { origin: c.origin } : {}),
-    };
+    }, now);
 }
-function episodeItem(e, key) {
-    return {
+function episodeItem(e, key, now) {
+    return stamped({
         uid: `ep:${e.uid}`, kind: 'episode', scope: 'project', project: e.project, engine: e.engine, session: e.session, cwd: e.cwd,
         where: e.title, ts: e.ts, firstTs: e.ts, title: key.title || e.title, summary: key.summary, body: episodeBody(e), files: e.files,
         appliesWhen: '', confidence: 'observed', importance: e.files.length ? 3 : 2, status: 'active', provenance: [e.uid], seen: 1,
         fingerprint: fingerprint(e.request, e.reply, e.files), cues: key.cues ?? [], concepts: key.concepts ?? [],
         ...(key.outcome ? { outcome: key.outcome } : {}), ...(key.origin ? { origin: key.origin } : {}),
-    };
+    }, now);
 }
 /** An observation as a long-term item: its text is its key, its date beside when it was said. */
-function observationItem(o, src) {
-    return {
+function observationItem(o, src, now) {
+    return stamped({
         uid: `obs:${fingerprint(o.text, o.eventAt, o.provenance[0] ?? src.uid)}`, kind: 'observation', scope: 'person', project: src.project,
         engine: src.engine, session: src.session, cwd: src.cwd, where: src.title, ts: src.ts, firstTs: src.ts,
         title: cut(o.text, 100), summary: o.text, body: '', files: [], appliesWhen: '', confidence: 'stated', importance: 2,
@@ -149,7 +158,7 @@ function observationItem(o, src) {
         cues: o.cues ?? [], concepts: o.concepts, ...(o.eventAt ? { eventAt: o.eventAt } : {}),
         ...(o.startAt ? { startAt: o.startAt } : {}), ...(o.endAt ? { endAt: o.endAt } : {}), phase: o.phase, ...(o.about ? { about: o.about } : {}),
         ...(o.origin ? { origin: o.origin } : {}),
-    };
+    }, now);
 }
 function cosine(a, b) {
     let ab = 0;
@@ -225,13 +234,16 @@ export class Consolidator {
         for (const [uid, m] of this.d.stm.entries()) {
             if (m.done || m.synthetic)
                 continue;
-            const s = by.get(m.session) ?? { uids: [], last: 0 };
+            const s = by.get(m.session) ?? { uids: [], first: Infinity, last: 0 };
             s.uids.push(uid);
+            s.first = Math.min(s.first, m.ts);
             s.last = Math.max(s.last, m.ts);
             by.set(m.session, s);
         }
-        return [...by.entries()].map(([session, s]) => ({ session, ...s }))
-            .filter((s) => now - s.last >= SETTLED_MS || s.uids.every((u) => now - (this.d.stm.meta(u)?.ts ?? now) >= DAY_MS))
+        // Quiet for an hour, or waiting a day since its oldest unread entry: a thread that never goes quiet (a message every
+        // half hour) is still read.
+        return [...by.entries()].filter(([, s]) => now - s.last >= SETTLED_MS || now - s.first >= DAY_MS)
+            .map(([session, { uids, last }]) => ({ session, uids, last }))
             .sort((a, b) => b.last - a.last);
     }
     /** Consolidate up to `maxWindows` windows. Stops early when the model pauses or the worker is stopped.
@@ -250,6 +262,8 @@ export class Consolidator {
         }
         const limit = Math.max(1, this.d.concurrency ?? EXTRACT_CONCURRENCY);
         const extracting = [];
+        /** Aborted once the lanes end — failed, paused, stopped or done: what is still being read is read for no one. */
+        const stop = new AbortController();
         // A pool: the next window starts when ANY one is read, not when the oldest is — one slow answer (measured: an
         // answer that ran on for 19,000 lines) held every other place idle while the windows after it waited their turn.
         // They are still reconciled in order.
@@ -260,17 +274,18 @@ export class Consolidator {
             while (!stopped && reading < limit && next < wins.length && !this.d.aborted()) {
                 const win = wins[next];
                 const head = win[0];
-                const p = this.read(win, { cwd: head.cwd, engine: head.engine, title: head.title });
+                const p = this.read(win, { cwd: head.cwd, engine: head.engine, title: head.title }, stop.signal);
                 reading++;
-                // One left behind by a pause or a stop is not an unhandled rejection; the awaited one still throws.
-                p.then(() => { reading--; launch(); }, () => { reading--; });
+                // One left behind by a pause or a stop is not an unhandled rejection; the awaited one still throws. One that
+                // throws starts no other: the run throws when it is reached, and no one would read their answers.
+                p.then(() => { reading--; launch(); }, () => { reading--; stopped = true; });
                 extracting[next++] = p;
             }
         };
         launch();
         const together = Math.max(1, Math.min(limit, this.d.reconcileConcurrency ?? 1));
         /** Window i, once read: reconciled, observed, marked done — or 'stop' when the run pauses. */
-        const settle = async (i) => {
+        const settle = async (i, signal) => {
             const x = await extracting[i].catch((err) => { stopped = true; throw err; });
             const win = wins[i];
             const head = win[0];
@@ -304,17 +319,23 @@ export class Consolidator {
             const learned = new Set(x.candidates.flatMap((c) => c.provenance));
             const kept = x.episodes.filter((k) => byUid.has(k.uid) && !learned.has(k.uid));
             const fresh = [
-                ...kept.map((k) => episodeItem(byUid.get(k.uid), k)),
+                ...kept.map((k) => episodeItem(byUid.get(k.uid), k, now)),
                 ...x.candidates.map((c) => itemOf(c, byUid.get(c.provenance.at(-1) ?? '') ?? head, now, c.provenance.flatMap((u) => byUid.get(u) ?? []))),
             ];
-            await this.reconcile(fresh, r);
-            await this.observe(x.observations.map((o) => observationItem(o, byUid.get(o.provenance.at(-1) ?? '') ?? head)), r);
+            await this.reconcile(fresh, r, signal);
+            await this.observe(x.observations.map((o) => observationItem(o, byUid.get(o.provenance.at(-1) ?? '') ?? head, now)), r);
             r.dropped += x.dropped;
             r.episodes += kept.length;
             // Read: marked done in the queue. The requests themselves stay searchable — the raw record is the truth and the
             // memories learned from them are keys to it; dropping them lost every fact the extraction did not keep (measured on
-            // LongMemEval: 47 of 60 misses were facts said in passing inside a request).
-            this.d.stm.put(win.map((e) => ({ ...e, done: true })));
+            // LongMemEval: 47 of 60 misses were facts said in passing inside a request). Each as it is now, not as it was read
+            // before the model's long answer: one removed meanwhile stays removed, one changed meanwhile is read again.
+            this.d.stm.put(win.flatMap((e) => {
+                if (this.d.stm.meta(e.uid)?.fp !== e.fingerprint)
+                    return [];
+                const cur = this.d.stm.get(e.uid);
+                return cur ? [{ ...cur, done: true }] : [];
+            }));
             r.windows++;
             // One at a time, each window's keys are dropped after it; windows at once share the cache until the run ends.
             if (together === 1)
@@ -323,19 +344,30 @@ export class Consolidator {
         };
         let at = 0;
         let halt = false;
-        const lane = async () => {
-            while (!halt && at < wins.length && extracting[at] && !this.d.aborted()) {
+        const lane = async (signal) => {
+            while (!halt && at < wins.length && extracting[at] && !this.d.aborted() && !signal.aborted) {
+                // No window is started after a failure: the run throws, and no one would read its answer.
                 try {
-                    if (await settle(at++) === 'stop')
+                    if (await settle(at++, signal) === 'stop')
                         halt = true;
                 }
                 catch (err) {
                     halt = true;
+                    stopped = true;
                     throw err;
                 }
             }
         };
-        await Promise.all(Array.from({ length: together }, lane));
+        try {
+            await allOrAbort(Array.from({ length: together }, () => lane), stop.signal);
+        }
+        finally {
+            // What is still being read once the lanes end is aborted, and over before the run goes on: no paid ask outlives
+            // it, and none starts after it (`stopped`).
+            stopped = true;
+            stop.abort();
+            await Promise.allSettled(extracting.filter((p) => p !== undefined));
+        }
         if (halt && r.paused !== undefined)
             return r;
         this.vecs.clear();
@@ -350,22 +382,27 @@ export class Consolidator {
         return r;
     }
     /** One window's reading, and what the guard (when there is one) lets it keep. */
-    async read(win, meta) {
-        const x = await this.extract(win, meta);
-        return this.d.guard && 'candidates' in x ? this.d.guard(win, x) : x;
+    async read(win, meta, signal) {
+        const x = await this.extract(win, meta, signal);
+        if (!this.d.guard || !('candidates' in x))
+            return x;
+        // Read for no one once the run stopped: the guard (it may ask a model too) is not asked.
+        if (signal?.aborted)
+            return { unavailable: 'the run stopped' };
+        return this.d.guard(win, x, signal);
     }
     /** One window's extraction: the one ask — and, when observations are read apart, the window's small windows too, each
      *  with the one before it as context. Any part not answered leaves the whole window for the next run. */
-    async extract(win, meta) {
+    async extract(win, meta, signal) {
         const chars = this.d.observeChars;
         const known = this.d.resolve ? await this.knownFor(win) : undefined;
         if (!chars)
-            return extractWindow(win, meta, { ask: this.d.ask, known });
+            return extractWindow(win, meta, { ask: this.d.ask, known, ...(signal ? { signal } : {}) });
         const subs = windowsOf(win, chars);
-        const [x, ...seen] = await Promise.all([
-            extractWindow(win, meta, { ask: this.d.ask, observations: false, known }),
-            ...subs.map((sub, k) => observeWindow(sub, k ? subs[k - 1] : [], meta, { ask: this.d.ask, chars, known })),
-        ]);
+        const [x, ...seen] = await allOrAbort([
+            (s) => extractWindow(win, meta, { ask: this.d.ask, observations: false, known, signal: s }),
+            ...subs.map((sub, k) => (s) => observeWindow(sub, k ? subs[k - 1] : [], meta, { ask: this.d.ask, chars, known, signal: s })),
+        ], signal);
         if (!('candidates' in x))
             return x;
         const failed = seen.find((o) => !('observations' in o));
@@ -427,7 +464,8 @@ export class Consolidator {
         for (let i = 0; i < batches.length; i += limit) {
             if (this.d.aborted())
                 break;
-            const answers = await Promise.all(batches.slice(i, i + limit).map((b) => this.cueBatch(b)));
+            // One that throws stops the others (allOrAbort).
+            const answers = await allOrAbort(batches.slice(i, i + limit).map((b) => (signal) => this.cueBatch(b, signal)));
             const paused = answers.find((a) => 'paused' in a);
             const got = answers.flatMap((a) => ('items' in a ? a.items : []));
             if (got.length) {
@@ -443,11 +481,12 @@ export class Consolidator {
         }
         return { cued };
     }
-    async cueBatch(items) {
+    async cueBatch(items, signal) {
         const prompt = items.map((it, n) => `[i${n + 1}] (${it.kind}) ${it.title}\n${it.summary}${it.appliesWhen ? `\nNeeded when: ${it.appliesWhen}` : ''}`).join('\n\n');
         const a = await (this.d.ask ?? defaultAsk)({
             purpose: 'memory', system: CUE_SYSTEM, prompt: redactString(prompt), schema: CUE_SCHEMA,
             name: 'memory_cues', maxTokens: 550 * items.length + 200, timeoutMs: 120_000, consequence: 'so these memories wait for their cues',
+            ...(signal ? { signal } : {}),
         });
         if ('unavailable' in a)
             return { paused: a.unavailable };
@@ -455,7 +494,7 @@ export class Consolidator {
             return { paused: 'the fast model is busy' };
         const out = [];
         for (const x of a.answer.items ?? []) {
-            const it = typeof x.id === 'string' ? items[Number(x.id.slice(1)) - 1] : undefined;
+            const it = typeof x.id === 'string' ? items[labelNo(x.id, 'i') - 1] : undefined;
             const cues = it?.cues?.length ? it.cues : cleanCues(x.cues);
             const concepts = it?.concepts?.length ? it.concepts : cleanConcepts(x.concepts);
             if (it && (cues.length || concepts.length))
@@ -464,7 +503,7 @@ export class Consolidator {
         return { items: out };
     }
     /** Each item against what long-term memory holds — written as it goes, so a later item sees an earlier one. */
-    async reconcile(items, r) {
+    async reconcile(items, r, signal) {
         const ask = [];
         // Every key of the window in one embedding call (the cache serves each below and again when it is written).
         await this.embed(items.map((it) => docText(it.title, it.summary)), embedBudget(items.length));
@@ -495,10 +534,14 @@ export class Consolidator {
         }
         // The pairs are fixed before any is asked about: every batch asked at once, the answers applied in order.
         const batches = relateBatches(ask);
-        const answers = await Promise.all(batches.map((b) => this.relate(b)));
+        // One that throws stops the others (allOrAbort); a run stopped meanwhile (a lane beside this one failed) applies none.
+        const answers = await allOrAbort(batches.map((b) => (s) => this.relate(b, s)), signal);
+        if (signal?.aborted)
+            throw new Error('consolidation stopped');
         for (const [b, batch] of batches.entries()) {
+            const got = answers[b];
             for (const [n, { it, known }] of batch.entries()) {
-                const d = answers[b]?.get(`p${n + 1}`);
+                const d = got instanceof Map ? got.get(`p${n + 1}`) : undefined;
                 if (!d || d.relation === 'different') {
                     await this.add(it);
                     r.added++;
@@ -605,16 +648,18 @@ export class Consolidator {
             files: [...new Set([...known.files, ...it.files])],
             confidence: known.confidence === 'verified' || it.confidence === 'verified' ? 'verified' : known.confidence,
             cues: [...new Set([...(known.cues ?? []), ...(it.cues ?? [])])],
-            related: [...new Set([...(known.related ?? []), ...(it.related ?? [])])].filter((u) => u !== known.uid),
+            related: [...new Set([...(known.related ?? []), ...(it.related ?? [])])].filter((u) => u !== known.uid).slice(-MAX_LINKS),
             concepts: [...new Map([...(known.concepts ?? []), ...(it.concepts ?? [])].map((c) => [conceptKey(c), c])).values()],
             absorbed: [...new Set([...(known.absorbed ?? []), ...kept.map((k) => k.uid)])],
         };
         merged.fingerprint = fingerprint(merged.kind, merged.title, merged.summary, merged.body);
+        // Changed now: what sleep relates next.
+        merged.learnedAt = Math.max(learnedAt(known), learnedAt(it));
         if (text)
             await writeLtm(this.writer, [merged]);
         else {
             await this.d.client.setPayload([known.uid], { ts: merged.ts, importance: merged.importance });
-            if ((merged.related?.length ?? 0) > (known.related?.length ?? 0))
+            if (merged.related?.join('\n') !== (known.related ?? []).join('\n'))
                 await this.d.client.setPayload([known.uid], { related: merged.related }, true);
             if ((merged.concepts?.length ?? 0) > (known.concepts?.length ?? 0))
                 await writeConcepts(this.writer, [merged]);
@@ -624,9 +669,10 @@ export class Consolidator {
         }
         this.d.ltm.put([...kept, merged]);
     }
-    /** Two items about the same subject that say different things: each lists the other (memory_explore walks these). */
+    /** Two items about the same subject that say different things: each lists the other (memory_explore walks these) —
+     *  the newest MAX_LINKS, so an item on a busy subject does not grow its list, and its payload, without end. */
     async link(a, b) {
-        const add = (x, y) => ({ ...x, related: [...new Set([...(x.related ?? []), y.uid])] });
+        const add = (x, y) => ({ ...x, related: [...(x.related ?? []).filter((u) => u !== y.uid), y.uid].slice(-MAX_LINKS) });
         const [one, two] = [add(a, b), add(b, a)];
         this.d.ltm.put([one, two]);
         await this.d.client.setPayload([one.uid], { related: one.related }, true);
@@ -638,7 +684,8 @@ export class Consolidator {
         await writeLtm(this.writer, [old]);
         this.d.ltm.put([old]);
     }
-    async relate(batch) {
+    /** The model's decision for each pair, by its label (`p1`, …) — or why it did not answer. */
+    async relate(batch, signal) {
         // Whole texts: a merge the model writes from a shortened body loses the rest of it.
         const prompt = batch.map(({ it, known }, n) => `[p${n + 1}]\nNEW (${it.kind}): ${it.title}\n${it.summary}\n${it.body}\n\nKNOWN (${known.kind}): ${known.title}\n${known.summary}\n${known.body}`).join('\n\n');
         // Room for every merged text in full (≈ 2.5 characters a token, the two sides together), with the decisions.
@@ -646,14 +693,19 @@ export class Consolidator {
         const a = await (this.d.ask ?? defaultAsk)({
             purpose: 'memory', system: RELATE_SYSTEM, prompt: redactString(prompt), schema: RELATE_SCHEMA,
             name: 'memory_reconcile', maxTokens: 900 * batch.length + 200 + room, timeoutMs: 300_000, consequence: 'so these memories are kept apart for now',
+            ...(signal ? { signal } : {}),
         });
+        if ('unavailable' in a)
+            return { paused: a.unavailable };
         if (!('answer' in a))
-            return null;
+            return { paused: 'the fast model is busy' };
         const out = new Map();
         for (const p of a.answer.pairs ?? []) {
-            if (typeof p.id !== 'string' || !RELATIONS.includes(String(p.relation)))
+            // A small model may quote the label's line rather than the label: the number is looked for.
+            const n = typeof p.id === 'string' ? labelNo(p.id, 'p') : 0;
+            if (!n || !RELATIONS.includes(String(p.relation)))
                 continue;
-            out.set(p.id, { relation: p.relation, title: (p.title ?? '').trim(), summary: (p.summary ?? '').trim(), body: (p.body ?? '').trim() });
+            out.set(`p${n}`, { relation: p.relation, title: (p.title ?? '').trim(), summary: (p.summary ?? '').trim(), body: (p.body ?? '').trim() });
         }
         return out;
     }

package/dist/core/episodes.d.ts

@@ -22,7 +22,7 @@ export declare const isConversation: (e: {
     engine?: string;
 }) => boolean;
 /** A record's body: what its key summarises, its chunks are cut from, and opening it shows — whole, never shortened. */
-export declare function episodeBody(e: Pick<ReadableRequest, 'request' | 'reply' | 'files'> & Partial<Pick<ReadableRequest, 'said' | 'steps' | 'engine'>>): string;
+export declare function episodeBody(e: Pick<ReadableRequest, 'request' | 'reply' | 'files'> & Partial<Pick<ReadableRequest, 'said' | 'engine'>>): string;
 /** For each day a request was made, the days a time said relatively that day means — yesterday, the week before, the
  *  last weekend, each of the six days before by name, this week, last month, last year (UTC as the requests' times) —
  *  so the model reads "last week" or "on Friday" off a line instead of counting. "Last week" is the seven days before,

package/dist/core/episodes.js

@@ -9,10 +9,7 @@ export const isConversation = (e) => e.engine === 'conversation';
 export function episodeBody(e) {
     if (e.engine !== undefined && !isConversation(e))
         return e.reply.trim() ? `${e.request}\n${e.reply}` : e.request;
-    const parts = [`Asked: ${e.request}`];
-    if (e.steps?.length)
-        parts.push(`Steps:\n${e.steps.join('\n')}`);
-    parts.push(`Done: ${e.said?.trim() ? e.said : e.reply}`);
+    const parts = [`Asked: ${e.request}`, `Done: ${e.said?.trim() ? e.said : e.reply}`];
     if (e.files.length)
         parts.push(`Files: ${e.files.join(', ')}`);
     return parts.join('\n');
@@ -41,10 +38,7 @@ export function renderForExtraction(e, n) {
     // Whole: a record too long for one window is read in parts and its parts joined (extract.ts), never shortened.
     if (e.engine !== undefined && !isConversation(e))
         return [`## Record ${n} · ${when} · ${e.engine}`, e.request, e.reply].filter(Boolean).join('\n');
-    const who = e.typed === 'human' ? 'typed by the person' : 'sent to the agent';
-    const parts = [`## Request ${n} · ${when} · ${who}`, e.request];
-    if (e.steps.length)
-        parts.push(`Steps:\n${e.steps.join('\n')}`);
+    const parts = [`## Request ${n} · ${when} · typed by the person`, e.request];
     if (e.said)
         parts.push(`Agent said:\n${e.said}`);
     if (e.files.length)