memtrails 0.0.0-stage → 0.1.0

package/CHANGELOG.md

@@ -0,0 +1,21 @@
+# Changelog
+
+All notable changes are listed here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and
+versions follow [Semantic Versioning](https://semver.org/).
+
+## [0.1.0] — 2026-10-31
+
+First public release.
+
+- `openMemory`: records of any kind and conversations, searchable at once with no server and no language model.
+- Local embedding (EmbeddingGemma-300m, 4-bit ONNX, fetched and verified on first use) or any OpenAI-compatible
+  embeddings server; a Qdrant process of its own (pinned, verified) or your Qdrant server.
+- Search: dense keys, chunks and cues plus BM25 keywords, fused by score at the item level; dates first when asked;
+  the best 30 reordered word by word (MaxSim on one-bit token vectors), or by any rerank server.
+- `room`: the best sentences of what was found, for a small context.
+- `open`, `explore` (concept graph, Personalized PageRank), `remove`, `rebuild`, `stats`.
+- Optional, with any language model: `consolidate` (typed items, dated observations, episodes, cues, concepts),
+  `sleep` (merge, supersede, link, archive, profiles), `gather` (sets for counting and listing questions).
+- Agent tools: `memory_search`, `memory_open`, `memory_explore` with zod and JSON schemas.
+- Importers: mbox and .eml, Slack exports, Git history, folders of Markdown, text and HTML, JSONL.
+- Benchmarks and research notes: docs/BENCHMARKS.md, docs/RESEARCH.md.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Corneliu Maftuleac
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,218 @@
-# Temporary Holding Version
+# memtrails
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+**Retrieval and exploration for any text — e-mail, chat, notes, memos, documents, news, commit messages, conversations.**
+
+[![CI](https://github.com/cmaftuleac/memtrails/actions/workflows/ci.yml/badge.svg)](https://github.com/cmaftuleac/memtrails/actions/workflows/ci.yml)
+[![npm](https://img.shields.io/npm/v/memtrails.svg)](https://www.npmjs.com/package/memtrails)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+memtrails keeps what you give it and finds it again. Everything you add is searchable the moment it is added, on your
+own machine, with no server and no language model: a local embedding model, a keyword channel, and a word-by-word
+reordering that needs no reranker. In any language. Give it a language model too, and it consolidates what it holds
+into typed knowledge, dated events and profiles of people and things — and an agent can explore it.
+
+```ts
+import { openMemory } from 'memtrails';
+
+const memory = await openMemory({ home: './memory' });
+
+await memory.add([
+  { id: 'msg-1', kind: 'email', author: 'Ana', title: 'Gateway certificate expired', at: '2026-04-02',
+    text: 'The TLS certificate expired on the 1st. Renewal stopped because the cron job was removed in March.' },
+  { id: 'c9f1e2', kind: 'commit', author: 'Ion', title: 'Restore certificate renewal cron', at: '2026-04-03',
+    text: 'Files: deploy/cron/renew.sh' },
+]);
+
+const hits = await memory.search({ query: 'why did the certificate stop renewing?' });
+// [{ id, kind: 'email', title, summary, match, ts, record: { id: 'msg-1', author: 'Ana' }, score }, …]
+
+await memory.close();
+```
+
+## Why memtrails
+
+- **Searchable at once.** No ingestion pipeline to wait for and no model call per record. Add a mailbox and search it.
+- **On your machine.** The embedding model (EmbeddingGemma-300m, 4-bit, ~220 MB) and the index (Qdrant) run locally
+  and are fetched on first use. Nothing leaves the machine unless you turn on a language model.
+- **Any language.** Multilingual embeddings, keyword folding for any script, no word lists. A question in Romanian
+  finds an English memory and the other way round.
+- **Little to read, much found.** Search reorders its best 30 word by word (MaxSim over the model's own token vectors,
+  stored at one bit a number): on 2,011 benchmark questions the answer reached the reader in 53.1% of top-5 lists
+  against 50.3% for plain search, and most of a cross-encoder reranker's gain, in about 3 ms on one CPU thread.
+- **Measured, honestly.** On all 1,307 held-out LoCoMo questions with 6,000 characters to read, memtrails alone on a
+  CPU scores 77.2 (strict judge) against Mem0's 76.5 — a tie, reported as one. [Benchmarks](docs/BENCHMARKS.md).
+- **Explorable.** A concept graph links what records share — people, places, systems, names — so an agent can walk
+  from what it found to what a search for the words does not find.
+- **Built for agents.** `memory.tools()` gives `memory_search`, `memory_open` and `memory_explore` with zod and JSON
+  schemas, ready for an MCP server or any function-calling API.
+
+## Install
+
+```bash
+npm install memtrails
+```
+
+Node.js 22 or newer, on macOS or Linux (x64 or arm64). Windows has the pieces it needs but is not tested yet. The first `openMemory` downloads the embedding model
+(218 MB, from Hugging Face) and the Qdrant binary (~30 MB, from Qdrant's GitHub release) into `~/.cache/memtrails`,
+each checked against a pinned SHA-256. Set `MEMTRAILS_CACHE` to put them elsewhere.
+
+## Adding data
+
+A **record** is anything with text. Only `id` and `text` are required.
+
+```ts
+await memory.add({
+  id: '<CAF8w@mail.example>',      // unique within its source; adding the same id again updates it
+  source: 'work-mailbox',          // part of the identity
+  kind: 'email',                   // any word: email, message, note, memo, article, commit, ticket, …
+  collection: 'ops',               // search can be scoped to collections
+  thread: 'gateway-cert',          // records that belong together
+  author: 'Ana Popescu',
+  title: 'Gateway certificate expired',
+  text: '…',
+  at: '2026-04-02T09:14:00Z',      // ms, ISO string or Date
+  url: 'https://mail.example/…',
+  metadata: { to: 'ops@example.com' },
+});
+```
+
+Conversations (a person and an assistant, or any two parties) are added as turns; each time the person speaks becomes
+one record, with what was answered:
+
+```ts
+await memory.addConversation({ id: 'chat-42', turns: [
+  { role: 'user', text: 'Where do we have lunch on Friday?' },
+  { role: 'assistant', text: 'At the Italian place near the office, at one.' },
+] });
+```
+
+### Importers
+
+Streaming importers for common sources, in `memtrails/importers`:
+
+```ts
+import { fromMbox, fromSlackExport, fromGit, fromFiles, fromJsonl } from 'memtrails/importers';
+
+await memory.import(fromMbox('./archive.mbox'));                 // e-mail (mbox / .eml)
+await memory.import(fromSlackExport('./slack-export'));          // a Slack workspace export
+await memory.import(fromGit('./my-repo', { since: '2025-01-01' })); // commit messages and changed files
+await memory.import(fromFiles('./notes'));                       // Markdown, text and HTML files
+await memory.import(fromJsonl('./records.jsonl'));               // one record per line
+```
+
+`memory.import` takes any async iterable of records, so a database cursor or an API client works the same way.
+Unchanged records cost nothing; `memory.remove([...ids])` forgets.
+
+## Searching
+
+```ts
+await memory.search({ query: 'who renewed the certificate?', limit: 8 });
+await memory.search({ query: 'standup notes', collections: ['ops'], kinds: ['message'], from: '2026-04', to: '2026-04' });
+
+// The same results as a reader is shown them: one line a memory, best first or oldest first.
+await memory.searchText({ query: 'what happened with the gateway?', order: 'time' });
+
+// The best sentences of what was found, for a small context window.
+await memory.room({ query: 'what made the renewal stop?', chars: 6000 });
+
+// One item whole — for something learned, with the records it was learned from.
+await memory.open(hits[0].id);
+
+// What the concept graph reaches from a topic or from items, that search did not find.
+await memory.explore({ topic: 'the gateway' });
+```
+
+With dates, what happened within them comes first and the rest follows: a question that names a time is rarely only
+about that time.
+
+## Consolidation and sleep (optional, with a language model)
+
+Records are kept as they are, for good. With a language model, memtrails also reads threads in windows and keeps what
+matters as typed items — facts, decisions, procedures, pitfalls, preferences, dated observations ("Ana renewed the
+certificate on 2026-04-03"), kept episodes — reconciled against what it already knows, so nothing is stored twice. A
+nightly `sleep` merges near-duplicates, retires what was superseded, links related items and writes a profile of each
+person or thing that many items mention.
+
+```ts
+const memory = await openMemory({
+  home: './memory',
+  llm: { model: 'gpt-6-luna', reasoningEffort: 'low', apiKey: process.env.OPENAI_API_KEY },   // or any OpenAI-compatible endpoint: baseUrl
+});
+await memory.consolidate();   // after adding
+await memory.sleep();         // once a day
+await memory.gather({ query: 'every doctor I saw this year' });   // a set, for counting and listing questions
+```
+
+Any provider works: pass `llm: { baseUrl, model, apiKey }` for an OpenAI-compatible server (a hosted model, vLLM,
+Ollama, …), or your own function `(request) => Promise<answer>`. Every call asks for strict structured output, and
+`onUsage` reports each call's tokens. On the LoCoMo benchmark, consolidating and sleeping a conversation of ~700
+messages cost about $0.07 with GPT-6 Luna (the model every benchmark here used).
+
+## For agents
+
+```ts
+const tools = memory.tools();
+// [{ name: 'memory_search', description, inputSchema (zod), jsonSchema, handler }, memory_open, memory_explore]
+
+// With the MCP TypeScript SDK:
+for (const t of tools) server.tool(t.name, t.description, t.inputSchema, t.handler);
+```
+
+## Configuration
+
+| Option | Default | |
+|---|---|---|
+| `home` | — | Where this memory lives: its logs, index and word vectors. |
+| `embedding.modelDir` | fetched into the cache | A directory with EmbeddingGemma-300m ONNX (`onnx/model_q4.onnx`, tokenizer). |
+| `embedding.server` | — | `{ url, apiKey?, model? }`: an OpenAI-compatible embeddings server serving EmbeddingGemma (text-embeddings-inference, Ollama, …). |
+| `embedding.threads` | 2 | CPU threads for the local model. |
+| `rerank` | `'words'` | `'words'`, a rerank server `{ url, apiKey?, model? }` (any `/rerank` API), your own function, or `false`. |
+| `llm` | from the environment | `{ model, apiKey?, baseUrl?, reasoningEffort?, onUsage? }` or your own `Ask` function. |
+| `qdrant` | a process of its own | `{ url, apiKey?, collection? }` for your Qdrant server, or `{ binary }`. |
+
+| Environment | |
+|---|---|
+| `MEMTRAILS_CACHE` | Where downloads live (default `~/.cache/memtrails`). |
+| `MEMTRAILS_MODEL`, `MEMTRAILS_LLM_API_KEY` or `OPENAI_API_KEY`, `MEMTRAILS_LLM_BASE_URL` | The language model when `llm` is not given. |
+| `MEMTRAILS_EMBED_URL`, `MEMTRAILS_RERANK_URL` (+ `_KEY`, `_MODEL`) | Servers, when not given in options. |
+
+## How it works
+
+Every record goes to an append-only log (the truth) and to a Qdrant index (derived, rebuilt from the logs whenever it
+is lost or out of date): a key vector of its start, its whole text in ~96-word chunks, and a BM25 keyword vector.
+A search runs every channel at once, fuses them by score at the level of items (dense 0.7, keywords 0.3), and reorders
+the best 30 word by word. Consolidation, sleep and the concept graph sit on top. See [Architecture](docs/ARCHITECTURE.md),
+the [API reference](docs/API.md), and [what was tried and why](docs/RESEARCH.md).
+
+## Benchmarks
+
+| | memtrails | Mem0 | Graphiti |
+|---|---|---|---|
+| LoCoMo, all 1,307 held-out questions, 6,000 characters, strict judge | **77.2** (no server) | 76.5 | — |
+| the same, Mem0's own lenient judge | **89.3** | 87.8 | — |
+| LoCoMo, 1,307 held-out questions, 28,000 characters, one search¹ | 79.9 | 80.6 | 42.1 |
+| LoCoMo, the same with an agent using each system's own tools¹ | **79.0** | 77.1 | 47.0 |
+| LongMemEval-S retrieval, strict all sessions @5 (467)² | **96.15** | — | — |
+
+The same reader and judges for every system. The small-room and one-search comparisons are ties; the agent lead is
+small and was judged once. ¹ An earlier build (2026-10-27). ² gbrain-evals publishes 95.96 on the same count: a tie. [Full results, protocol and limitations](docs/BENCHMARKS.md).
+
+## Limitations
+
+- English benchmarks dominate the evidence. Other languages were measured with translated questions over English memories, not memories written in other languages.
+- The word-by-word reorder was trained on nothing; a small per-person table taught on a person's own use did better in tests but is not in this release.
+- Questions whose answer is only implied — relevant through outside knowledge, with no word in common — are rarely found by any retrieval system, this one included.
+- Consolidation's prompts were tuned on conversations; on other records they are reasonable but unmeasured.
+- The index stores 768-dimensional EmbeddingGemma vectors; another embedding model needs its own index and is not measured.
+
+## Privacy
+
+Records stay in `home`. Nothing is sent anywhere unless you configure a language model, an embedding server or a
+rerank server; then what they need is sent to them. Secrets (keys, tokens, credentials in URLs, private keys) are
+scrubbed from what is sent to a language model and from what consolidation keeps.
+
+## Licence
+
+MIT © Corneliu Maftuleac. Third parties: EmbeddingGemma is Google's, under the
+[Gemma Terms of Use](https://ai.google.dev/gemma/terms), downloaded from Hugging Face, not shipped; Qdrant is Apache-2.0,
+downloaded from its release; LoCoMo (CC BY-NC 4.0) and LongMemEval are linked, never redistributed.

package/dist/core/consolidate.d.ts

@@ -0,0 +1,144 @@
+import { type Ask, type Extraction } from './extract.js';
+import { type LtmStore, type MemoryItem, type StmEntry, type StmStore } from './store.js';
+import { type WriterDeps } from './writer.js';
+export declare const SAME = 0.93;
+export declare const SAME_EPISODE = 0.97;
+/** An observation is the same only when it says the same thing (this close) about the same date: two dinners on two
+ *  days are two observations, and a list question needs both. */
+export declare const SAME_OBSERVATION = 0.97;
+export declare const RELATED = 0.8;
+/** A conversation is consolidated once it has been quiet this long — or its entries are a day old. */
+export declare const SETTLED_MS: number;
+/** Extraction calls in flight: they are independent, and one at a time made a backfill take days (~86 s a window). */
+export declare const EXTRACT_CONCURRENCY = 8;
+export interface ConsolidateDeps extends WriterDeps {
+    stm: StmStore;
+    ltm: LtmStore;
+    log: (line: string) => void;
+    aborted: () => boolean;
+    ask?: Ask;
+    now?: () => number;
+    /** Windows the fast model extracts at once (EXTRACT_CONCURRENCY when not given). */
+    concurrency?: number;
+    /** Windows reconciled at once (1, in order, when not given). A backfill sets more: a window may then not see what one
+     *  reconciled beside it wrote — the duplicates that leaves are the sleep pass's to merge — and an item two windows
+     *  touch at once is changed under its lock, from what it is by then. */
+    reconcileConcurrency?: number;
+    /** A window's size in characters (WINDOW_CHARS when not given). */
+    windowChars?: number;
+    /** Observations read apart, in windows of this many characters (observeWindow); absent: in the window's one ask. */
+    observeChars?: number;
+    /** Each window read with what other conversations say that it may refer to (knownFor): "the book you recommended"
+     *  written as the book's name. */
+    resolve?: boolean;
+    /** What a window's reading may keep, checked before anything is reconciled or written (e.g. facts only
+     *  from the person's own words and results the agent confirmed, each with where it came from). An answer it cannot
+     *  give leaves the window for the next run, as an extraction that was not answered does. */
+    guard?: (win: StmEntry[], x: Extraction) => Promise<Extraction | {
+        unavailable: string;
+    } | {
+        busy: true;
+    }>;
+}
+export interface ConsolidationReport {
+    windows: number;
+    added: number;
+    merged: number;
+    updated: number;
+    superseded: number;
+    linked: number;
+    dropped: number;
+    episodes: number;
+    observed: number;
+    cued: number;
+    paused?: string;
+}
+/** Items given their cues and concepts after a run's windows (the backlog: saved notes, what was learned before
+ *  them), and how many go in one ask. */
+export declare const CUE_BACKLOG = 40;
+export type Relation = 'same' | 'updates' | 'contradicts' | 'related' | 'different';
+/** Links an item keeps to others about the same subject (exploration walks them), newest last. */
+export declare const MAX_LINKS = 24;
+/** Pairs asked about together: at most 8, and at most RELATE_CHARS of their texts — a long pair on its own. */
+export declare const RELATE_CHARS = 60000;
+export declare function relateBatches<T extends {
+    it: MemoryItem;
+    known: MemoryItem;
+}>(pairs: T[]): T[][];
+export declare class Consolidator {
+    private readonly d;
+    private readonly vecs;
+    /** Windows whose extraction failed, by their first request: how many times in a row. */
+    private readonly failed;
+    /** Windows read in smaller parts from now on, by their first request: at most this many requests a part. */
+    private readonly narrowed;
+    /** An item changed by one window at a time (windows reconciled at once): the last holder's turn, by uid. */
+    private readonly locks;
+    constructor(d: ConsolidateDeps);
+    /** Runs with the items locked — taken in uid order, so two windows never wait on each other (sleep's answers too). */
+    locked<T>(uids: string[], run: () => Promise<T>): Promise<T>;
+    /** The item as it is now, or null when it stopped being active (merged or superseded by a window beside this one). */
+    private latest;
+    /** Embed with a per-run cache: the key is embedded once for reconcile and reused when it is written. */
+    private readonly embed;
+    private get writer();
+    /** Conversations with requests waiting, settled, newest first. */
+    pending(now?: number): Array<{
+        session: string;
+        uids: string[];
+        last: number;
+    }>;
+    /** Consolidate up to `maxWindows` windows. Stops early when the model pauses or the worker is stopped.
+     *  The fast model extracts several windows at once (they are independent); each is reconciled in order, newest
+     *  conversation first, so a later item still sees what an earlier one wrote. */
+    run(maxWindows?: number): Promise<ConsolidationReport>;
+    /** One window's reading, and what the guard (when there is one) lets it keep. */
+    private read;
+    /** One window's extraction: the one ask — and, when observations are read apart, the window's small windows too, each
+     *  with the one before it as context. Any part not answered leaves the whole window for the next run. */
+    private extract;
+    /** What other conversations say that this window may refer to: a few of its requests searched for (memory_search —
+     *  the raw requests of every conversation, captured before any is consolidated, and what was learned), the best
+     *  passages outside this conversation, dated. The order conversations are consolidated in does not matter: the raw
+     *  record holds them all. Nothing found, or a search that fails, reads the window without it. */
+    private knownFor;
+    /** A window as it is read now: in parts of at most the size its earlier failures narrowed it to, each part narrowed
+     *  again by its own failures. */
+    private narrow;
+    /** Encoding's backlog: active items without cues or concepts — saved notes (they come in curated, without the
+     *  consolidation call) and what was learned before either existed — get them, newest first, a few an ask, several
+     *  asks at once. Cues an item already has stay. */
+    fillCues(max?: number): Promise<{
+        cued: number;
+        paused?: string;
+    }>;
+    private cueBatch;
+    /** Each item against what long-term memory holds — written as it goes, so a later item sees an earlier one. */
+    private reconcile;
+    /** Observations are only ever added: one is the same as a known one only when it says the same (SAME_OBSERVATION)
+     *  about the same date — then the known one is told again (provenance, `seen`). Never asked about, never merged into
+     *  another text, never superseded: an older one stays, dated, beside the newer. */
+    private observe;
+    private add;
+    /** The known item absorbs the new one: provenance, `seen`, the newest time, the higher importance — and, when the
+     *  model merged the texts, the merged key (embedded again). Nothing of either is lost: the new item's own text, and the
+     *  known one's before a rewrite, stay in the log as `merged` records the item lists (`absorbed`). */
+    merge(known: MemoryItem, it: MemoryItem, text: {
+        title: string;
+        summary: string;
+        body: string;
+    } | null): Promise<void>;
+    /** Two items about the same subject that say different things: each lists the other (memory_explore walks these). */
+    link(a: MemoryItem, b: MemoryItem): Promise<void>;
+    /** Superseded: out of the index and of recall, kept in the log with what replaced it. */
+    retire(known: MemoryItem, by: string): Promise<void>;
+    relate(batch: Array<{
+        it: MemoryItem;
+        known: MemoryItem;
+    }>): Promise<Map<string, {
+        relation: Relation;
+        title: string;
+        summary: string;
+        body: string;
+    }> | null>;
+}