memtrails 0.3.0 → 0.4.0

package/CHANGELOG.md

@@ -3,6 +3,38 @@
 All notable changes are listed here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and
 versions follow [Semantic Versioning](https://semver.org/).
 
+## [Unreleased]
+
+## [0.4.0] — 2026-11-04
+
+- Prompt examples are invented now. The extraction and gathering prompts had a few example sentences taken from LoCoMo
+  conversations, one of them (in `gather`'s notes rule) from a held-out conversation. Their effect on the published
+  results, measured on the judged answers: leaving out every held-out question that mentions any of those words or
+  people (48 questions) gives 81.1 against Mem0's 76.6 (+111/−55), against 80.9 and 76.5 on all 1,307. The question
+  the example came from was answered wrong by both systems.
+- The search index layout is 7: `kinds` now selects records by their kind on every point and learned items by theirs,
+  so an existing index is rebuilt from the logs on the next open.
+- Fixes: a crash in the middle of a write no longer loses the records added after it; consolidation no longer
+  overwrites records changed or removed while the model answers; a title cut through an emoji no longer fails a write;
+  a thread that never goes quiet is consolidated; imported history is related and kept like new knowledge (`learnedAt`);
+  a malformed `from`/`to` is an error; `gather` and `sleep` report `paused` when the model did not read.
+- `bench/audited.ts`: judged LoCoMo answers without the questions whose golden answer an independent audit
+  ([locomo-audit](https://github.com/dial481/locomo-audit)) found wrong, dropped for every system alike. As shipped,
+  84.8 against Mem0's 79.9 (+109/−49).
+- `gather`: a memory a small model keeps by quoting its whole line (`[m3] (2023-06-03) …`) now counts; it read as nothing
+  kept before.
+
+- Security: the secret filter catches common key formats it missed (GITHUB_TOKEN=, AWS_SECRET_ACCESS_KEY=, JSON-style
+  secrets, GitLab, Hugging Face, npm, SendGrid, Telegram, the whole password of a URL); crafted e-mail and HTML can no
+  longer make the importers or the filter take quadratic time; downloads are verified on disk in a temp folder of their
+  own under a lock, with a stall timeout, and the Qdrant binary runs only while it is this user's alone; Qdrant gets a
+  minimal environment and must prove it is ours before it is given the key; an OpenAI key is sent to OpenAI only; the
+  memory folder is private; Slack channel folders cannot lead out of the export.
+- One process at a time in a memory folder (a lock); consolidate, sleep, rebuild and remove run one at a time; `close()`
+  stops the running job and waits for it; an open that fails stops what it started; `remove()` takes the record's kept
+  episode out of the long-term log too; `MEMTRAILS_EMBED_URL` and `MEMTRAILS_RERANK_URL` are used.
+- The release is built and tested in a job without publishing rights; actions are pinned to commits.
+
 ## [0.3.0] — 2026-11-02
 
 - `gather(q, { notes: true })`: the same model call also writes what the kept memories say that answers the question,

package/README.md

@@ -17,7 +17,7 @@ import { openMemory } from 'memtrails';
 const memory = await openMemory({ home: './memory' });
 
 await memory.add([
-  { id: 'msg-1', kind: 'email', author: 'Ana', title: 'Gateway certificate expired', at: '2026-04-02',
+  { id: 'msg-1', kind: 'email', author: 'Ana', title: 'Proxy certificate expired', at: '2026-04-02',
     text: 'The TLS certificate expired on the 1st. Renewal stopped because the cron job was removed in March.' },
   { id: 'c9f1e2', kind: 'commit', author: 'Ion', title: 'Restore certificate renewal cron', at: '2026-04-03',
     text: 'Files: deploy/cron/renew.sh' },
@@ -41,7 +41,8 @@ await memory.close();
   against 50.3% for plain search, and most of a cross-encoder reranker's gain, in about 3 ms on one CPU thread.
 - **Measured, honestly.** On all 1,307 held-out LoCoMo questions with 6,000 characters to read, memtrails alone on a
   CPU scores 77.4 (strict judge) against Mem0's 76.5 — a tie, reported as one; with `gather` reading what search
-  found and noting what it says, 80.9 — ahead beyond chance. [Benchmarks](docs/BENCHMARKS.md).
+  found and noting what it says, 80.9 — ahead beyond chance; 84.8 against 79.9 without the questions whose official
+  answer an independent audit found wrong. [Benchmarks](docs/BENCHMARKS.md).
 - **Explorable.** A concept graph links what records share — people, places, systems, names — so an agent can walk
   from what it found to what a search for the words does not find.
 - **Built for agents.** `memory.tools()` gives `memory_search`, `memory_open` and `memory_explore` with zod and JSON
@@ -67,9 +68,9 @@ await memory.add({
   source: 'work-mailbox',          // part of the identity
   kind: 'email',                   // any word: email, message, note, memo, article, commit, ticket, …
   collection: 'ops',               // search can be scoped to collections
-  thread: 'gateway-cert',          // records that belong together
+  thread: 'proxy-cert',          // records that belong together
   author: 'Ana Popescu',
-  title: 'Gateway certificate expired',
+  title: 'Proxy certificate expired',
   text: '…',
   at: '2026-04-02T09:14:00Z',      // ms, ISO string or Date
   url: 'https://mail.example/…',
@@ -111,7 +112,7 @@ await memory.search({ query: 'who renewed the certificate?', limit: 8 });
 await memory.search({ query: 'standup notes', collections: ['ops'], kinds: ['message'], from: '2026-04', to: '2026-04' });
 
 // The same results as a reader is shown them: one line a memory, best first or oldest first.
-await memory.searchText({ query: 'what happened with the gateway?', order: 'time' });
+await memory.searchText({ query: 'what happened with the proxy?', order: 'time' });
 
 // The best sentences of what was found, for a small context window.
 await memory.room({ query: 'what made the renewal stop?', chars: 6000 });
@@ -120,7 +121,7 @@ await memory.room({ query: 'what made the renewal stop?', chars: 6000 });
 await memory.open(hits[0].id);
 
 // What the concept graph reaches from a topic or from items, that search did not find.
-await memory.explore({ topic: 'the gateway' });
+await memory.explore({ topic: 'the proxy' });
 ```
 
 With dates, what happened within them comes first and the rest follows: a question that names a time is rarely only
@@ -141,7 +142,7 @@ const memory = await openMemory({
 });
 await memory.consolidate();   // after adding
 await memory.sleep();         // once a day
-await memory.gather({ query: 'every doctor I saw this year' });   // a set, for counting and listing questions
+await memory.gather({ query: 'every plumber we called this year' });   // a set, for counting and listing questions
 ```
 
 Any provider works: pass `llm: { baseUrl, model, apiKey }` for an OpenAI-compatible server (a hosted model, vLLM,
@@ -174,7 +175,7 @@ for (const t of tools) server.tool(t.name, t.description, t.inputSchema, t.handl
 | Environment | |
 |---|---|
 | `MEMTRAILS_CACHE` | Where downloads live (default `~/.cache/memtrails`). |
-| `MEMTRAILS_MODEL`, `MEMTRAILS_LLM_API_KEY` or `OPENAI_API_KEY`, `MEMTRAILS_LLM_BASE_URL` | The language model when `llm` is not given. |
+| `MEMTRAILS_MODEL`, `MEMTRAILS_LLM_API_KEY` or `OPENAI_API_KEY`, `MEMTRAILS_LLM_BASE_URL` | The language model when `llm` is not given. `OPENAI_API_KEY` is sent only to OpenAI's own address. |
 | `MEMTRAILS_EMBED_URL`, `MEMTRAILS_RERANK_URL` (+ `_KEY`, `_MODEL`) | Servers, when not given in options. |
 
 ## How it works
@@ -191,17 +192,21 @@ the [API reference](docs/API.md), and [what was tried and why](docs/RESEARCH.md)
 |---|---|---|---|
 | LoCoMo, all 1,307 held-out questions, 6,000 characters, strict judge, `gather` at 40 with notes | **80.9** | 76.5 | — |
 | the same, Mem0's own lenient judge | **91.9** | 87.8 | — |
+| the same, strict, without the 80 questions whose golden answer is wrong³ | **84.8** | 79.9 | — |
 | the same, `gather` without notes, strict / lenient | 79.5 / 91.4 | 76.5 / 87.8 | — |
 | the same with `search` alone, no model when searching, strict / lenient | 77.4 / 89.3 | 76.5 / 87.8 | — |
 | LoCoMo, 1,307 held-out questions, 28,000 characters, one search¹ | 79.9 | 80.6 | 42.1 |
 | LoCoMo, the same with an agent using each system's own tools¹ | **79.0** | 77.1 | 47.0 |
+| LongMemEval-S, 60-question sample, 6,000 characters, strict judge, `gather` at 40 with notes (a tie) | 90.0 | 88.3 | — |
 | LongMemEval-S retrieval, strict all sessions @5 (467)² | **96.15** | — | — |
 
 The same reader and judges for every system. With `gather` (a model reads what search found, keeps what the answer
 needs and notes what it says) memtrails leads Mem0 beyond chance: +113/−55 questions strict, +78/−25 lenient (both
 p < 0.001). With `search` alone,
 the small-room and one-search comparisons are ties; the agent lead is
-small and was judged once. ¹ An earlier build (2026-10-27). ² gbrain-evals publishes 95.96 on the same count: a tie. [Full results, protocol and limitations](docs/BENCHMARKS.md).
+small and was judged once. ¹ An earlier build (2026-10-27). ² gbrain-evals publishes 95.96 on the same count: a tie. ³ As found by an
+independent audit of LoCoMo ([locomo-audit](https://github.com/dial481/locomo-audit)), dropped for both systems;
++109/−49 (p < 0.001), `bench/audited.ts` reproduces it. [Full results, protocol and limitations](docs/BENCHMARKS.md).
 
 ## Limitations
 

package/dist/core/consolidate.d.ts

@@ -7,7 +7,7 @@ export declare const SAME_EPISODE = 0.97;
  *  days are two observations, and a list question needs both. */
 export declare const SAME_OBSERVATION = 0.97;
 export declare const RELATED = 0.8;
-/** A conversation is consolidated once it has been quiet this long — or its entries are a day old. */
+/** A conversation is consolidated once it has been quiet this long — or its oldest waiting entry is a day old. */
 export declare const SETTLED_MS: number;
 /** Extraction calls in flight: they are independent, and one at a time made a backfill take days (~86 s a window). */
 export declare const EXTRACT_CONCURRENCY = 8;
@@ -57,6 +57,12 @@ export interface ConsolidationReport {
  *  them), and how many go in one ask. */
 export declare const CUE_BACKLOG = 40;
 export type Relation = 'same' | 'updates' | 'contradicts' | 'related' | 'different';
+export type Decision = {
+    relation: Relation;
+    title: string;
+    summary: string;
+    body: string;
+};
 /** Links an item keeps to others about the same subject (exploration walks them), newest last. */
 export declare const MAX_LINKS = 24;
 /** Pairs asked about together: at most 8, and at most RELATE_CHARS of their texts — a long pair on its own. */
@@ -65,6 +71,16 @@ export declare function relateBatches<T extends {
     it: MemoryItem;
     known: MemoryItem;
 }>(pairs: T[]): T[][];
+/** When an item was written to long-term memory. `ts` is its source's own time — for history imported today, long ago —
+ *  so "learned since the last sleep" and "unused for half a year" read this. An item written before the field existed
+ *  has only `ts`. */
+export declare const learnedAt: (m: {
+    ts: number;
+    learnedAt?: number;
+}) => number;
+export declare const stamped: (it: MemoryItem, at: number) => MemoryItem;
+/** The number of a label the model wrote back: `p3`, or the line it labels quoted whole (`[p3] NEW …`) — 0 for none. */
+export declare function labelNo(id: string, letter: 'p' | 'i'): number;
 export declare class Consolidator {
     private readonly d;
     private readonly vecs;
@@ -128,17 +144,16 @@ export declare class Consolidator {
         summary: string;
         body: string;
     } | null): Promise<void>;
-    /** Two items about the same subject that say different things: each lists the other (memory_explore walks these). */
+    /** Two items about the same subject that say different things: each lists the other (memory_explore walks these) —
+     *  the newest MAX_LINKS, so an item on a busy subject does not grow its list, and its payload, without end. */
     link(a: MemoryItem, b: MemoryItem): Promise<void>;
     /** Superseded: out of the index and of recall, kept in the log with what replaced it. */
     retire(known: MemoryItem, by: string): Promise<void>;
+    /** The model's decision for each pair, by its label (`p1`, …) — or why it did not answer. */
     relate(batch: Array<{
         it: MemoryItem;
         known: MemoryItem;
-    }>): Promise<Map<string, {
-        relation: Relation;
-        title: string;
-        summary: string;
-        body: string;
-    }> | null>;
+    }>): Promise<Map<string, Decision> | {
+        paused: string;
+    }>;
 }

package/dist/core/consolidate.js

@@ -29,7 +29,7 @@ export const SAME_EPISODE = 0.97;
  *  days are two observations, and a list question needs both. */
 export const SAME_OBSERVATION = 0.97;
 export const RELATED = 0.80;
-/** A conversation is consolidated once it has been quiet this long — or its entries are a day old. */
+/** A conversation is consolidated once it has been quiet this long — or its oldest waiting entry is a day old. */
 export const SETTLED_MS = 60 * 60_000;
 /** Extraction calls in flight: they are independent, and one at a time made a backfill take days (~86 s a window). */
 export const EXTRACT_CONCURRENCY = 8;
@@ -116,32 +116,41 @@ export function relateBatches(pairs) {
         out.push(cur);
     return out;
 }
+/** When an item was written to long-term memory. `ts` is its source's own time — for history imported today, long ago —
+ *  so "learned since the last sleep" and "unused for half a year" read this. An item written before the field existed
+ *  has only `ts`. */
+export const learnedAt = (m) => m.learnedAt ?? m.ts;
+export const stamped = (it, at) => ({ ...it, learnedAt: at });
+/** The number of a label the model wrote back: `p3`, or the line it labels quoted whole (`[p3] NEW …`) — 0 for none. */
+export function labelNo(id, letter) {
+    return Number(new RegExp(`\\b${letter}(\\d+)`).exec(id)?.[1] ?? 0);
+}
 /** One remembered episode or candidate as a long-term item (not yet reconciled). */
 function itemOf(c, src, now, sources = [src]) {
     const project = c.scope === 'project' ? src.project : c.scope;
     // The files its requests changed, from their recorded steps — by code, not the model: paths are what summaries lose.
     const files = [...new Set(sources.flatMap((e) => e.files))];
-    return {
+    return stamped({
         uid: `mem:${fingerprint(c.kind, c.scope, project, c.title, c.summary)}`, kind: c.kind, scope: c.scope, project: src.project,
         engine: src.engine, session: src.session, cwd: src.cwd, where: src.title, ts: src.ts || now, firstTs: src.ts || now,
         title: c.title, summary: c.summary, body: c.body, files, appliesWhen: c.appliesWhen, confidence: c.confidence,
         importance: c.importance, status: 'active', provenance: c.provenance, seen: 1,
         fingerprint: fingerprint(c.kind, c.title, c.summary, c.body), cues: c.cues, concepts: c.concepts,
         ...(c.origin ? { origin: c.origin } : {}),
-    };
+    }, now);
 }
-function episodeItem(e, key) {
-    return {
+function episodeItem(e, key, now) {
+    return stamped({
         uid: `ep:${e.uid}`, kind: 'episode', scope: 'project', project: e.project, engine: e.engine, session: e.session, cwd: e.cwd,
         where: e.title, ts: e.ts, firstTs: e.ts, title: key.title || e.title, summary: key.summary, body: episodeBody(e), files: e.files,
         appliesWhen: '', confidence: 'observed', importance: e.files.length ? 3 : 2, status: 'active', provenance: [e.uid], seen: 1,
         fingerprint: fingerprint(e.request, e.reply, e.files), cues: key.cues ?? [], concepts: key.concepts ?? [],
         ...(key.outcome ? { outcome: key.outcome } : {}), ...(key.origin ? { origin: key.origin } : {}),
-    };
+    }, now);
 }
 /** An observation as a long-term item: its text is its key, its date beside when it was said. */
-function observationItem(o, src) {
-    return {
+function observationItem(o, src, now) {
+    return stamped({
         uid: `obs:${fingerprint(o.text, o.eventAt, o.provenance[0] ?? src.uid)}`, kind: 'observation', scope: 'person', project: src.project,
         engine: src.engine, session: src.session, cwd: src.cwd, where: src.title, ts: src.ts, firstTs: src.ts,
         title: cut(o.text, 100), summary: o.text, body: '', files: [], appliesWhen: '', confidence: 'stated', importance: 2,
@@ -149,7 +158,7 @@ function observationItem(o, src) {
         cues: o.cues ?? [], concepts: o.concepts, ...(o.eventAt ? { eventAt: o.eventAt } : {}),
         ...(o.startAt ? { startAt: o.startAt } : {}), ...(o.endAt ? { endAt: o.endAt } : {}), phase: o.phase, ...(o.about ? { about: o.about } : {}),
         ...(o.origin ? { origin: o.origin } : {}),
-    };
+    }, now);
 }
 function cosine(a, b) {
     let ab = 0;
@@ -225,13 +234,16 @@ export class Consolidator {
         for (const [uid, m] of this.d.stm.entries()) {
             if (m.done || m.synthetic)
                 continue;
-            const s = by.get(m.session) ?? { uids: [], last: 0 };
+            const s = by.get(m.session) ?? { uids: [], first: Infinity, last: 0 };
             s.uids.push(uid);
+            s.first = Math.min(s.first, m.ts);
             s.last = Math.max(s.last, m.ts);
             by.set(m.session, s);
         }
-        return [...by.entries()].map(([session, s]) => ({ session, ...s }))
-            .filter((s) => now - s.last >= SETTLED_MS || s.uids.every((u) => now - (this.d.stm.meta(u)?.ts ?? now) >= DAY_MS))
+        // Quiet for an hour, or waiting a day since its oldest unread entry: a thread that never goes quiet (a message every
+        // half hour) is still read.
+        return [...by.entries()].filter(([, s]) => now - s.last >= SETTLED_MS || now - s.first >= DAY_MS)
+            .map(([session, { uids, last }]) => ({ session, uids, last }))
             .sort((a, b) => b.last - a.last);
     }
     /** Consolidate up to `maxWindows` windows. Stops early when the model pauses or the worker is stopped.
@@ -304,17 +316,23 @@ export class Consolidator {
             const learned = new Set(x.candidates.flatMap((c) => c.provenance));
             const kept = x.episodes.filter((k) => byUid.has(k.uid) && !learned.has(k.uid));
             const fresh = [
-                ...kept.map((k) => episodeItem(byUid.get(k.uid), k)),
+                ...kept.map((k) => episodeItem(byUid.get(k.uid), k, now)),
                 ...x.candidates.map((c) => itemOf(c, byUid.get(c.provenance.at(-1) ?? '') ?? head, now, c.provenance.flatMap((u) => byUid.get(u) ?? []))),
             ];
             await this.reconcile(fresh, r);
-            await this.observe(x.observations.map((o) => observationItem(o, byUid.get(o.provenance.at(-1) ?? '') ?? head)), r);
+            await this.observe(x.observations.map((o) => observationItem(o, byUid.get(o.provenance.at(-1) ?? '') ?? head, now)), r);
             r.dropped += x.dropped;
             r.episodes += kept.length;
             // Read: marked done in the queue. The requests themselves stay searchable — the raw record is the truth and the
             // memories learned from them are keys to it; dropping them lost every fact the extraction did not keep (measured on
-            // LongMemEval: 47 of 60 misses were facts said in passing inside a request).
-            this.d.stm.put(win.map((e) => ({ ...e, done: true })));
+            // LongMemEval: 47 of 60 misses were facts said in passing inside a request). Each as it is now, not as it was read
+            // before the model's long answer: one removed meanwhile stays removed, one changed meanwhile is read again.
+            this.d.stm.put(win.flatMap((e) => {
+                if (this.d.stm.meta(e.uid)?.fp !== e.fingerprint)
+                    return [];
+                const cur = this.d.stm.get(e.uid);
+                return cur ? [{ ...cur, done: true }] : [];
+            }));
             r.windows++;
             // One at a time, each window's keys are dropped after it; windows at once share the cache until the run ends.
             if (together === 1)
@@ -455,7 +473,7 @@ export class Consolidator {
             return { paused: 'the fast model is busy' };
         const out = [];
         for (const x of a.answer.items ?? []) {
-            const it = typeof x.id === 'string' ? items[Number(x.id.slice(1)) - 1] : undefined;
+            const it = typeof x.id === 'string' ? items[labelNo(x.id, 'i') - 1] : undefined;
             const cues = it?.cues?.length ? it.cues : cleanCues(x.cues);
             const concepts = it?.concepts?.length ? it.concepts : cleanConcepts(x.concepts);
             if (it && (cues.length || concepts.length))
@@ -497,8 +515,9 @@ export class Consolidator {
         const batches = relateBatches(ask);
         const answers = await Promise.all(batches.map((b) => this.relate(b)));
         for (const [b, batch] of batches.entries()) {
+            const got = answers[b];
             for (const [n, { it, known }] of batch.entries()) {
-                const d = answers[b]?.get(`p${n + 1}`);
+                const d = got instanceof Map ? got.get(`p${n + 1}`) : undefined;
                 if (!d || d.relation === 'different') {
                     await this.add(it);
                     r.added++;
@@ -605,16 +624,18 @@ export class Consolidator {
             files: [...new Set([...known.files, ...it.files])],
             confidence: known.confidence === 'verified' || it.confidence === 'verified' ? 'verified' : known.confidence,
             cues: [...new Set([...(known.cues ?? []), ...(it.cues ?? [])])],
-            related: [...new Set([...(known.related ?? []), ...(it.related ?? [])])].filter((u) => u !== known.uid),
+            related: [...new Set([...(known.related ?? []), ...(it.related ?? [])])].filter((u) => u !== known.uid).slice(-MAX_LINKS),
             concepts: [...new Map([...(known.concepts ?? []), ...(it.concepts ?? [])].map((c) => [conceptKey(c), c])).values()],
             absorbed: [...new Set([...(known.absorbed ?? []), ...kept.map((k) => k.uid)])],
         };
         merged.fingerprint = fingerprint(merged.kind, merged.title, merged.summary, merged.body);
+        // Changed now: what sleep relates next.
+        merged.learnedAt = Math.max(learnedAt(known), learnedAt(it));
         if (text)
             await writeLtm(this.writer, [merged]);
         else {
             await this.d.client.setPayload([known.uid], { ts: merged.ts, importance: merged.importance });
-            if ((merged.related?.length ?? 0) > (known.related?.length ?? 0))
+            if (merged.related?.join('\n') !== (known.related ?? []).join('\n'))
                 await this.d.client.setPayload([known.uid], { related: merged.related }, true);
             if ((merged.concepts?.length ?? 0) > (known.concepts?.length ?? 0))
                 await writeConcepts(this.writer, [merged]);
@@ -624,9 +645,10 @@ export class Consolidator {
         }
         this.d.ltm.put([...kept, merged]);
     }
-    /** Two items about the same subject that say different things: each lists the other (memory_explore walks these). */
+    /** Two items about the same subject that say different things: each lists the other (memory_explore walks these) —
+     *  the newest MAX_LINKS, so an item on a busy subject does not grow its list, and its payload, without end. */
     async link(a, b) {
-        const add = (x, y) => ({ ...x, related: [...new Set([...(x.related ?? []), y.uid])] });
+        const add = (x, y) => ({ ...x, related: [...(x.related ?? []).filter((u) => u !== y.uid), y.uid].slice(-MAX_LINKS) });
         const [one, two] = [add(a, b), add(b, a)];
         this.d.ltm.put([one, two]);
         await this.d.client.setPayload([one.uid], { related: one.related }, true);
@@ -638,6 +660,7 @@ export class Consolidator {
         await writeLtm(this.writer, [old]);
         this.d.ltm.put([old]);
     }
+    /** The model's decision for each pair, by its label (`p1`, …) — or why it did not answer. */
     async relate(batch) {
         // Whole texts: a merge the model writes from a shortened body loses the rest of it.
         const prompt = batch.map(({ it, known }, n) => `[p${n + 1}]\nNEW (${it.kind}): ${it.title}\n${it.summary}\n${it.body}\n\nKNOWN (${known.kind}): ${known.title}\n${known.summary}\n${known.body}`).join('\n\n');
@@ -647,13 +670,17 @@ export class Consolidator {
             purpose: 'memory', system: RELATE_SYSTEM, prompt: redactString(prompt), schema: RELATE_SCHEMA,
             name: 'memory_reconcile', maxTokens: 900 * batch.length + 200 + room, timeoutMs: 300_000, consequence: 'so these memories are kept apart for now',
         });
+        if ('unavailable' in a)
+            return { paused: a.unavailable };
         if (!('answer' in a))
-            return null;
+            return { paused: 'the fast model is busy' };
         const out = new Map();
         for (const p of a.answer.pairs ?? []) {
-            if (typeof p.id !== 'string' || !RELATIONS.includes(String(p.relation)))
+            // A small model may quote the label's line rather than the label: the number is looked for.
+            const n = typeof p.id === 'string' ? labelNo(p.id, 'p') : 0;
+            if (!n || !RELATIONS.includes(String(p.relation)))
                 continue;
-            out.set(p.id, { relation: p.relation, title: (p.title ?? '').trim(), summary: (p.summary ?? '').trim(), body: (p.body ?? '').trim() });
+            out.set(`p${n}`, { relation: p.relation, title: (p.title ?? '').trim(), summary: (p.summary ?? '').trim(), body: (p.body ?? '').trim() });
         }
         return out;
     }

package/dist/core/episodes.d.ts

@@ -22,7 +22,7 @@ export declare const isConversation: (e: {
     engine?: string;
 }) => boolean;
 /** A record's body: what its key summarises, its chunks are cut from, and opening it shows — whole, never shortened. */
-export declare function episodeBody(e: Pick<ReadableRequest, 'request' | 'reply' | 'files'> & Partial<Pick<ReadableRequest, 'said' | 'steps' | 'engine'>>): string;
+export declare function episodeBody(e: Pick<ReadableRequest, 'request' | 'reply' | 'files'> & Partial<Pick<ReadableRequest, 'said' | 'engine'>>): string;
 /** For each day a request was made, the days a time said relatively that day means — yesterday, the week before, the
  *  last weekend, each of the six days before by name, this week, last month, last year (UTC as the requests' times) —
  *  so the model reads "last week" or "on Friday" off a line instead of counting. "Last week" is the seven days before,

package/dist/core/episodes.js

@@ -9,10 +9,7 @@ export const isConversation = (e) => e.engine === 'conversation';
 export function episodeBody(e) {
     if (e.engine !== undefined && !isConversation(e))
         return e.reply.trim() ? `${e.request}\n${e.reply}` : e.request;
-    const parts = [`Asked: ${e.request}`];
-    if (e.steps?.length)
-        parts.push(`Steps:\n${e.steps.join('\n')}`);
-    parts.push(`Done: ${e.said?.trim() ? e.said : e.reply}`);
+    const parts = [`Asked: ${e.request}`, `Done: ${e.said?.trim() ? e.said : e.reply}`];
     if (e.files.length)
         parts.push(`Files: ${e.files.join(', ')}`);
     return parts.join('\n');
@@ -41,10 +38,7 @@ export function renderForExtraction(e, n) {
     // Whole: a record too long for one window is read in parts and its parts joined (extract.ts), never shortened.
     if (e.engine !== undefined && !isConversation(e))
         return [`## Record ${n} · ${when} · ${e.engine}`, e.request, e.reply].filter(Boolean).join('\n');
-    const who = e.typed === 'human' ? 'typed by the person' : 'sent to the agent';
-    const parts = [`## Request ${n} · ${when} · ${who}`, e.request];
-    if (e.steps.length)
-        parts.push(`Steps:\n${e.steps.join('\n')}`);
+    const parts = [`## Request ${n} · ${when} · typed by the person`, e.request];
     if (e.said)
         parts.push(`Agent said:\n${e.said}`);
     if (e.files.length)

package/dist/core/extract.js

@@ -39,15 +39,15 @@ what was decided, done, built, changed, fixed, deployed or found, and when. One
 several in one, none left out to keep the list short. Facts said in passing inside another request count; a small or
 one-off event counts; when in doubt, keep it. In a conversation between people, the replies are another person's words:
 what they tell about themselves counts as much. For each:
-- text: 10 to 60 words that stand alone — the event or fact itself, as told ("Melanie took her family camping in the
-  mountains"), never "On <date>, X said that …" (the date has its own field). Name who and what (names, never "he",
+- text: 10 to 60 words that stand alone — the event or fact itself, as told ("Rosa took her nephews kayaking on Lake
+  Bled"), never "On <date>, X said that …" (the date has its own field). Name who and what (names, never "he",
   "she", "it", "they"); keep every proper noun, title, number and quoted phrase exactly. A time said relatively stays
   as said, with the date it means, read off the Calendar line of the day it was said: "last week (the week before 26 June
   2023)", "yesterday (7 May 2023)", "on Friday (12 April 2024)". When the part names something only by description
-  ("the book you recommended", "my home country", "the kids") and KNOWN FROM OTHER CONVERSATIONS names it, write the name too ("Becoming Nicole",
-  "Sweden", "her three children") — never a fact from KNOWN that this part does not tell. What the person or a speaker
+  ("the book you recommended", "my home country", "the kids") and KNOWN FROM OTHER CONVERSATIONS names it, write the name too ("The Overstory",
+  "Portugal", "her three children") — never a fact from KNOWN that this part does not tell. What the person or a speaker
   says about their own life is true as told: no commentary on what the conversation did or did not confirm. A plan is
-  written as a plan ("Melanie planned to go swimming with the kids"). When it replaces something said before, say both
+  written as a plan ("Rosa planned to take the kids to the aquarium"). When it replaces something said before, say both
   (now X — before, Y).
 - about: who or what it is about, named as the conversation names them ("the person" for the one whose requests these
   are, unless the conversation names them).

package/dist/core/gather.d.ts

@@ -10,6 +10,9 @@ export interface Gathered {
     /** With `notes`: what the kept memories say that answers the question, a few short lines (empty otherwise). Show them
      *  before the memories, as notes on them: on LoCoMo this took 79.5 to 80.9 (docs/BENCHMARKS.md). */
     notes: string[];
+    /** Set when the model did not read some candidates (none configured, unreachable, busy): why. They are all kept, by
+     *  rank — the result is then a wide search, not a gathered set. */
+    paused?: string;
 }
 /** `from`/`to`: the time the question asks about ("on 10 April 2022", "in October 2023"). What happened then is searched
  *  for besides everything else and read first — not instead: a question also names dates it only counts from ("as of

package/dist/core/gather.js

@@ -1,7 +1,7 @@
 /**
  * Gathering: a SET of memories for a question, not a top-k.
  *
- * A question that counts, lists, totals or compares ("how many doctors did I see", "what did I spend on luxury items")
+ * A question that counts, lists, totals or compares ("how many plumbers did we call", "what did I spend on camera gear")
  * needs every instance, and one ranked below the cut is a wrong answer however good the rest are — measured on
  * LongMemEval's multi-session questions: every fact our answers missed was in the store, as a dated observation, and not
  * in the room. Mem0 and Graphiti hand the reader a fixed number by similarity; here the fast model reads them:
@@ -34,7 +34,7 @@ cancelled, sold, replaced, ended, a new amount) or dates it. When unsure, keep i
 wrong, an extra one does not.
 
 In "searches", write up to ${MORE} short searches for instances the question could count that none of these memories
-shows — other kinds, other words for the same thing (for "luxury items": "designer handbag", "jewelry", "watch") —
+shows — other kinds, other words for the same thing (for "camera gear": "lens", "tripod", "memory card") —
 in the language of the question. Leave it empty when nothing is missing. Never translate.`;
 const SCHEMA = {
     type: 'object', additionalProperties: false, required: ['keep', 'searches'],
@@ -48,7 +48,7 @@ const SCHEMA = {
 const NOTES_RULE = `
 
 In "notes", write what the memories you kept say that answers the question, in a few short lines: every instance with
-its date as the memories give it ("Books Tim has read: Harry Potter (2023-08-02), The Name of the Wind (2023-08-11)"),
+its date as the memories give it ("Concerts Lena went to: a jazz trio at the Blue Room (2024-03-02), the orchestra's Mahler night (2024-04-19)"),
 names, numbers and words as written. Only what they state, as plain statements: never write that something is not said,
 not given, unclear or unknown — leave it out instead, since other memories may hold it — and never a guess. Leave notes
 empty when the memories you kept say nothing about it. In the language of the question.`;
@@ -78,16 +78,18 @@ async function pick(ask, question, list, notes = false) {
             name: 'memory_gather', maxTokens: 40 * part.length + (notes ? 1400 : 600), timeoutMs: 120_000, consequence: 'so these memories are kept unread',
         });
         if (!('answer' in a))
-            return { keep: part.map((h) => h.id), searches: [], notes: [] };
+            return { keep: part.map((h) => h.id), searches: [], notes: [], paused: 'unavailable' in a ? a.unavailable : 'the fast model is busy' };
         const got = a.answer;
         const ids = Array.isArray(got.keep) ? got.keep.filter((x) => typeof x === 'string') : [];
-        const keep = ids.map((x) => part[Number(x.replace(/^m/, '')) - 1]?.id).filter((x) => !!x);
+        // A small model can quote the line it means ("[m3] (2023-06-03) …"), so the label is looked for, not assumed to be all of it.
+        const keep = ids.map((x) => part[Number(/\bm(\d+)/.exec(x)?.[1] ?? 0) - 1]?.id).filter((x) => !!x);
         const searches = Array.isArray(got.searches) ? got.searches.filter((x) => typeof x === 'string' && !!x.trim()) : [];
         const said = notes && Array.isArray(got.notes) ? got.notes.filter((x) => typeof x === 'string' && !!x.trim()).map((x) => redactString(x.trim())) : [];
-        return { keep, searches, notes: said };
+        return { keep, searches, notes: said, paused: undefined };
     }));
+    const paused = answers.find((a) => a.paused !== undefined)?.paused;
     return { keep: new Set(answers.flatMap((a) => a.keep)), searches: [...new Set(answers.flatMap((a) => a.searches.map((s) => s.trim())))],
-        notes: answers.flatMap((a) => a.notes) };
+        notes: answers.flatMap((a) => a.notes), ...(paused !== undefined ? { paused } : {}) };
 }
 /** `from`/`to`: the time the question asks about ("on 10 April 2022", "in October 2023"). What happened then is searched
  *  for besides everything else and read first — not instead: a question also names dates it only counts from ("as of
@@ -114,5 +116,7 @@ export async function memoryGather(d, q, o) {
     const all = [...first, ...more];
     const kept = all.filter((h) => one.keep.has(h.id) || two.keep.has(h.id));
     const keptIds = new Set(kept.map((h) => h.id));
-    return { found: [...kept, ...all.filter((h) => !keptIds.has(h.id))], kept: kept.map((h) => h.id), searches, notes: [...one.notes, ...two.notes] };
+    const paused = one.paused ?? two.paused;
+    return { found: [...kept, ...all.filter((h) => !keptIds.has(h.id))], kept: kept.map((h) => h.id), searches, notes: [...one.notes, ...two.notes],
+        ...(paused !== undefined ? { paused } : {}) };
 }

package/dist/core/model.d.ts

@@ -1,3 +1,4 @@
+import { type DownloadOptions } from './qdrant-binary.js';
 export declare const MODEL_REPO = "onnx-community/embeddinggemma-300m-ONNX";
 export declare const MODEL_REVISION = "5090578d9565bb06545b4552f76e6bc2c93e4a66";
 export declare const MODEL_FILES: ReadonlyArray<{
@@ -6,11 +7,14 @@ export declare const MODEL_FILES: ReadonlyArray<{
     sha256: string;
 }>;
 export declare function defaultModelDir(): string;
+/** Every file in place at its pinned size, and the marker written after the last of them was verified. */
 export declare function modelReady(dir?: string): boolean;
-/** The model in `dir` (fetched and checked unless it is already there); the directory, ready for `modelDir`. */
-export declare function ensureModel(opts?: {
+/**
+ * The model in `dir` (fetched and checked unless it is already there); the directory, ready for `modelDir`. Each file
+ * is fetched to a temp of its own, checked on disk and renamed into place; one run at a time per `dir` (withLock), and
+ * the marker is taken away first, so a set left half-replaced is never taken as ready.
+ */
+export declare function ensureModel(opts?: DownloadOptions & {
     dir?: string;
     log?: (line: string) => void;
-    fetchImpl?: typeof fetch;
-    signal?: AbortSignal;
 }): Promise<string>;