@openanonymity/nanomem 0.1.0

package/README.md

@@ -0,0 +1,194 @@
+# @openanonymity/nanomem
+
+**Personal memory you own, in files you can actually read.**
+
+`nanomem` turns chats, notes, and exports into a markdown memory system that an LLM can update and retrieve as facts evolve over time. The result stays inspectable, portable, and user-owned instead of disappearing into hidden vector state.
+
+## Why it exists
+
+`nanomem` is for building memory that can last beyond a single chat session, model, or tool.
+
+It turns raw conversations, notes, and exports into a memory system that can:
+
+- compress repeated interactions into stable knowledge
+- keep up with changing facts over time
+- preserve history without cluttering current context
+- stay inspectable and user-controlled
+
+Retrieval is only one part of memory. `nanomem` is built for the maintenance layer too: updating facts, resolving conflicts, and preserving history over time.
+
+## Features
+
+- **User-owned memory.** Keep memory in markdown files you can inspect, edit, version, and move across tools.
+- **Evolving memory state.** Keep facts current as they change over time instead of treating memory as an append-only log.
+- **Compaction and cleanup.** Collapse repeated signals into stable knowledge and move stale memory into history.
+- **Conflict-aware updates.** Resolve outdated or contradictory facts using recency, source, and confidence.
+- **Import your existing history.** Start from ChatGPT exports, [OA Chat](https://chat.openanonymity.ai) exports, transcripts, message arrays, markdown notes, or whole markdown directories.
+- **Flexible storage.** Run on local files, IndexedDB, in-memory storage, or a custom backend.
+- **Built to plug in.** Use it from the CLI, as a library, or as a memory layer for other agents.
+
+## Quick start
+
+Install:
+
+```bash
+npm install -g @openanonymity/nanomem
+```
+
+Set up once:
+
+```bash
+nanomem login
+```
+
+This walks you through provider, model, API key, and where to store your memory. Config is saved to `~/.config/nanomem/config.json`. Filesystem memory lives in `~/nanomem/` by default.
+
+Import history or notes:
+
+```bash
+nanomem import conversations.json
+nanomem import my-notes.md
+nanomem import ./notes/
+```
+
+Retrieve memory later:
+
+```bash
+nanomem retrieve "what are my hobbies?"
+nanomem retrieve "what are my hobbies?" --render
+```
+
+Compact and clean up memory:
+
+```bash
+nanomem compact
+```
+
+Scripted setup also works:
+
+```bash
+nanomem login --provider openai --api-key sk-... --model gpt-5.4-mini
+nanomem login --provider anthropic --api-key sk-ant-... --model claude-sonnet-4-6 --path ~/project/memory
+```
+
+Supported providers include OpenAI, Anthropic, Tinfoil, OpenRouter, and OpenAI-compatible endpoints via `--base-url`.
+
+## How it works
+
+```text
+conversation / notes / exports
+            |
+            v
+      memory import / ingest
+            |
+            |  LLM extraction with tool calls
+            |  create / append / update / archive / delete
+            v
+   markdown memory filesystem
+            |
+            |  memory retrieve
+            |  file selection + bullet-level scoring
+            v
+      assembled memory context
+            |
+            v
+       memory compact
+       dedup + temporal cleanup + history preservation
+```
+
+The core engine has three parts:
+
+- **Ingestion.** Extract durable facts from conversations or documents and organize them into topic files.
+- **Retrieval.** Navigate the memory filesystem and assemble relevant context for a query.
+- **Compaction.** Deduplicate repeated facts, keep current memory concise, and move stale or superseded facts into history.
+
+## Memory format
+
+Memory is stored as markdown with structured metadata:
+
+```md
+# Memory: Work
+
+## Working
+### Current context
+- Preparing for a product launch next month | topic=work | tier=working | status=active | source=user_statement | confidence=high | updated_at=2026-04-07 | review_at=2026-04-20
+
+## Long-Term
+### Stable facts
+- Leads the backend team at Acme | topic=work | tier=long_term | status=active | source=user_statement | confidence=high | updated_at=2026-04-07
+
+## History
+### No longer current
+- Previously lived in New York | topic=personal | tier=history | status=superseded | source=user_statement | confidence=high | updated_at=2024-06-01
+```
+
+That structure is what lets the system do more than retrieval: it can keep track of source, confidence, recency, temporary context, and historical state.
+
+## Using it in code
+
+```js
+import { createMemoryBank } from '@openanonymity/nanomem';
+
+const memory = createMemoryBank({
+  llm: { apiKey: 'sk-...', model: 'gpt-5.4-mini' },
+  storage: 'filesystem',
+  storagePath: './memory'
+});
+
+await memory.init();
+
+await memory.ingest([
+  { role: 'user', content: 'I just moved to Seattle.' },
+  { role: 'assistant', content: 'Noted.' }
+]);
+
+const result = await memory.retrieve('Where do I live now?');
+await memory.compact();
+```
+
+## Common commands
+
+```bash
+nanomem import <file|dir|->
+nanomem retrieve <query> [--context <file>]
+nanomem tree
+nanomem compact
+nanomem export --format zip
+nanomem status
+```
+
+For terminal use, `--render` will format markdown-heavy output like `read` and `retrieve` into a more readable ANSI-rendered view while leaving `--json` and piped output unchanged.
+
+## Import formats
+
+`nanomem import` supports:
+
+- ChatGPT exports
+- [OA Chat](https://chat.openanonymity.ai) exports
+- markdown notes
+- recursive markdown directory imports
+- JSON message arrays
+- plain text `User:` / `Assistant:` transcripts
+
+Import can operate in both conversation-oriented and document-oriented modes, depending on the source or explicit flags.
+
+```bash
+nanomem import conversations.json              # conversation mode
+nanomem import ./notes/                        # document mode (auto for directories)
+nanomem import my-notes.md --format markdown   # document mode (explicit)
+```
+
+## Storage backends
+
+- `filesystem` for local markdown folders
+- `indexeddb` for browser storage
+- `ram` for testing or ephemeral usage
+- custom backend objects for your own storage layer
+
+## Learn more
+
+Internals: [docs/memory-system.md](./docs/memory-system.md)
+
+## License
+
+MIT

package/package.json

@@ -0,0 +1,85 @@
+{
+    "name": "@openanonymity/nanomem",
+    "version": "0.1.0",
+    "description": "LLM-driven personal memory with agentic retrieval, extraction, and compaction",
+    "type": "module",
+    "bin": {
+        "nanomem": "src/cli.js"
+    },
+    "engines": {
+        "node": ">=18"
+    },
+    "repository": {
+        "type": "git",
+        "url": "git+https://github.com/openanonymity/nanomem.git"
+    },
+    "bugs": {
+        "url": "https://github.com/openanonymity/nanomem/issues"
+    },
+    "scripts": {
+        "prepublishOnly": "npm run build:types",
+        "build:types": "tsc",
+        "prepack": "tsc"
+    },
+    "exports": {
+        ".": {
+            "types": "./types/index.d.ts",
+            "default": "./src/index.js"
+        },
+        "./bullets": {
+            "types": "./types/bullets/index.d.ts",
+            "default": "./src/bullets/index.js"
+        },
+        "./backends": {
+            "types": "./types/backends/BaseStorage.d.ts",
+            "default": "./src/backends/BaseStorage.js"
+        },
+        "./utils": {
+            "types": "./types/utils/portability.d.ts",
+            "default": "./src/utils/portability.js"
+        },
+        "./imports": {
+            "types": "./types/imports/index.d.ts",
+            "default": "./src/imports/index.js"
+        },
+        "./llm": {
+            "types": "./types/llm/openai.d.ts",
+            "default": "./src/llm/openai.js"
+        },
+        "./llm/openai": {
+            "types": "./types/llm/openai.d.ts",
+            "default": "./src/llm/openai.js"
+        },
+        "./llm/anthropic": {
+            "types": "./types/llm/anthropic.d.ts",
+            "default": "./src/llm/anthropic.js"
+        }
+    },
+    "types": "./types/index.d.ts",
+    "main": "./src/index.js",
+    "files": [
+        "src/",
+        "types/"
+    ],
+    "keywords": [
+        "memory",
+        "llm",
+        "ai",
+        "personal-memory",
+        "retrieval",
+        "extraction"
+    ],
+    "license": "MIT",
+    "devDependencies": {
+        "@types/node": "^25.5.2",
+        "typescript": "^6.0.2"
+    },
+    "dependencies": {
+        "@pierre/diffs": "^1.1.12"
+    },
+    "directories": {
+        "doc": "docs"
+    },
+    "author": "openanonymity",
+    "homepage": "https://github.com/openanonymity/nanomem#readme"
+}

package/src/backends/BaseStorage.js

@@ -0,0 +1,177 @@
+/**
+ * BaseStorage — Abstract storage backend interface.
+ *
+ * Subclasses must implement:
+ *   init()                            → void
+ *   _readRaw(path)                    → string|null
+ *   _writeRaw(path, content, meta)    → void
+ *   delete(path)                      → void
+ *   exists(path)                      → boolean
+ *   rebuildTree()                    → void         (regenerate _tree.md)
+ *   exportAll()                       → [{path, content, ...}]
+ *
+ * BaseStorage provides (NOT abstract — do not override):
+ *   read(path)           → _readRaw
+ *   write(path, content) → metadata generation + _writeRaw + rebuildTree
+ *
+ * BaseStorage also provides default implementations for:
+ *   search(query)   → [{path, snippet}]
+ *   ls(dirPath)     → {files: string[], dirs: string[]}
+ *   getTree()      → string
+ */
+/** @import { ExportRecord, ListResult, SearchResult, StorageMetadata } from '../types.js' */
+import { parseBullets, extractTitles, countBullets } from '../bullets/index.js';
+
+export class BaseStorage {
+
+    // ─── Abstract (backends must implement) ─────────────────────
+
+    /** @returns {Promise<void>} */ async init() { throw new Error('BaseStorage.init() not implemented'); }
+    /** @returns {Promise<string | null>} */ async _readRaw(_path) { throw new Error('BaseStorage._readRaw() not implemented'); }
+    async _writeRaw(_path, _content, _meta) { throw new Error('BaseStorage._writeRaw() not implemented'); }
+    /** @param {string} _path @returns {Promise<void>} */ async delete(_path) { throw new Error('BaseStorage.delete() not implemented'); }
+    /** @param {string} _path @returns {Promise<boolean>} */ async exists(_path) { throw new Error('BaseStorage.exists() not implemented'); }
+    /** @returns {Promise<void>} */ async rebuildTree() { throw new Error('BaseStorage.rebuildTree() not implemented'); }
+    /** @returns {Promise<ExportRecord[]>} */ async exportAll() { throw new Error('BaseStorage.exportAll() not implemented'); }
+    /** @returns {Promise<void>} */ async clear() { throw new Error('BaseStorage.clear() not implemented'); }
+
+    // ─── Provided: read/write ───────────────────────────────────
+
+    /**
+     * @param {string} path
+     * @returns {Promise<string | null>}
+     */
+    async read(path) {
+        return this._readRaw(path);
+    }
+
+    /**
+     * @param {string} path
+     * @param {string} content
+     * @returns {Promise<void>}
+     */
+    async write(path, content) {
+        if (this._isInternalPath(path)) {
+            await this._writeRaw(path, String(content || ''), {});
+            return;
+        }
+        const str = String(content || '');
+        const meta = {
+            oneLiner: this._generateOneLiner(str),
+            itemCount: countBullets(str),
+            titles: extractTitles(str),
+        };
+        await this._writeRaw(path, str, meta);
+        await this.rebuildTree();
+    }
+
+    // ─── Shared: getTree, search, ls ───────────────────────────
+
+    /** @returns {Promise<string | null>} */
+    async getTree() {
+        return this.read('_tree.md');
+    }
+
+    /**
+     * @param {string} query
+     * @returns {Promise<SearchResult[]>}
+     */
+    async search(query) {
+        if (!query?.trim()) return [];
+        const lowerQuery = query.toLowerCase();
+        const all = await this.exportAll();
+        const results = [];
+
+        for (const rec of all) {
+            if (this._isInternalPath(rec.path)) continue;
+            const content = rec.content || '';
+            if (!content.toLowerCase().includes(lowerQuery)) continue;
+
+            // Return all lines that contain the query
+            const matchingLines = content.split('\n')
+                .filter(line => line.toLowerCase().includes(lowerQuery))
+                .map(line => line.trim())
+                .filter(Boolean);
+
+            results.push({
+                path: rec.path,
+                lines: matchingLines,
+            });
+        }
+
+        return results;
+    }
+
+    /**
+     * @param {string} [dirPath]
+     * @returns {Promise<ListResult>}
+     */
+    async ls(dirPath) {
+        const allPaths = await this._listAllPaths();
+        const prefix = dirPath ? dirPath + '/' : '';
+        const files = [];
+        const dirSet = new Set();
+
+        for (const filePath of allPaths) {
+            if (this._isInternalPath(filePath)) continue;
+            if (prefix && !filePath.startsWith(prefix)) continue;
+            const relative = filePath.slice(prefix.length);
+            if (!relative.includes('/')) {
+                files.push(filePath);
+            } else {
+                dirSet.add(relative.split('/')[0]);
+            }
+        }
+
+        return { files, dirs: [...dirSet] };
+    }
+
+    // ─── Shared helpers ──────────────────────────────────────────
+
+    _isInternalPath(path) {
+        return path === '_tree.md';
+    }
+
+    /** Override for efficient path listing. Default uses exportAll(). */
+    async _listAllPaths() {
+        const all = await this.exportAll();
+        return all.map(r => r.path);
+    }
+
+    _parentPath(filePath) {
+        const lastSlash = filePath.lastIndexOf('/');
+        return lastSlash === -1 ? '' : filePath.slice(0, lastSlash);
+    }
+
+    /** Generate a one-line summary of file content for the index. */
+    _generateOneLiner(content) {
+        if (!content) return '';
+
+        const bullets = parseBullets(content);
+        if (bullets.length > 0) {
+            const factTexts = bullets
+                .filter(b => b.section !== 'archive')
+                .slice(0, 4)
+                .map(b => b.text.trim())
+                .filter(Boolean);
+            if (factTexts.length > 0) {
+                const joined = factTexts.join('; ');
+                return joined.length > 120 ? joined.slice(0, 117) + '...' : joined;
+            }
+        }
+
+        const titles = extractTitles(content);
+        if (titles.length > 0) {
+            const joined = titles.join('; ');
+            return joined.length > 120 ? joined.slice(0, 117) + '...' : joined;
+        }
+
+        const lines = content.split('\n');
+        for (const line of lines) {
+            const trimmed = line.trim();
+            if (!trimmed || trimmed.startsWith('#')) continue;
+            return trimmed.length > 120 ? trimmed.slice(0, 117) + '...' : trimmed;
+        }
+        return content.slice(0, 120);
+    }
+}

package/src/backends/filesystem.js

@@ -0,0 +1,177 @@
+/**
+ * FileSystemStorage — Node.js filesystem storage backend.
+ *
+ * Stores memory files as .md files on disk. Uses fs/promises (Node 18+).
+ */
+/** @import { ExportRecord, StorageMetadata } from '../types.js' */
+import { readdir, readFile, writeFile, unlink, mkdir, rm, stat } from 'node:fs/promises';
+import { join, dirname } from 'node:path';
+import { BaseStorage } from './BaseStorage.js';
+import { countBullets } from '../bullets/index.js';
+import { buildTree, createBootstrapRecords } from './schema.js';
+
+class FileSystemStorage extends BaseStorage {
+    constructor(rootDir) {
+        super();
+        if (!rootDir) throw new Error('FileSystemStorage requires a rootDir');
+        this._root = rootDir;
+        this._initialized = false;
+    }
+
+    async init() {
+        if (this._initialized) return;
+        this._initialized = true;
+
+        await mkdir(this._root, { recursive: true });
+
+        const entries = await this._walkFiles();
+        if (entries.length === 0) {
+            const seeds = createBootstrapRecords(Date.now());
+            for (const seed of seeds) {
+                await this._writeRaw(seed.path, seed.content || '');
+            }
+        }
+
+    }
+
+    async _readRaw(path) {
+        await this.init();
+        try {
+            return await readFile(this._resolve(path), 'utf-8');
+        } catch (err) {
+            if (err && typeof err === 'object' && 'code' in err && err.code === 'ENOENT') return null;
+            throw err;
+        }
+    }
+
+    async _writeRaw(path, content, _meta = {}) {
+        const fullPath = this._resolve(path);
+        await mkdir(dirname(fullPath), { recursive: true });
+        await writeFile(fullPath, String(content || ''), 'utf-8');
+    }
+
+    /**
+     * @param {string} path
+     * @returns {Promise<void>}
+     */
+    async delete(path) {
+        if (this._isInternalPath(path)) return;
+        await this.init();
+        try {
+            await unlink(this._resolve(path));
+        } catch (err) {
+            if (!(err && typeof err === 'object' && 'code' in err && err.code === 'ENOENT')) throw err;
+        }
+        await this.rebuildTree();
+    }
+
+    async clear() {
+        await rm(this._root, { recursive: true, force: true });
+        this._initialized = false;
+        await this.init();
+    }
+
+    /**
+     * @param {string} path
+     * @returns {Promise<boolean>}
+     */
+    async exists(path) {
+        await this.init();
+        try {
+            await stat(this._resolve(path));
+            return true;
+        } catch {
+            return false;
+        }
+    }
+
+    async rebuildTree() {
+        await this.init();
+        const allFiles = await this._walkFiles();
+        const files = allFiles.filter((f) => !this._isInternalPath(f)).sort();
+        const fileRecords = [];
+
+        for (const filePath of files) {
+            const content = await this.read(filePath);
+            let updatedAt = Date.now();
+            try {
+                const s = await stat(this._resolve(filePath));
+                updatedAt = s.mtimeMs;
+            } catch { /* skip */ }
+            fileRecords.push({
+                path: filePath,
+                itemCount: countBullets(content || ''),
+                oneLiner: this._generateOneLiner(content || ''),
+                updatedAt,
+            });
+        }
+
+        await this._writeRaw('_tree.md', buildTree(fileRecords));
+    }
+
+    /** @returns {Promise<ExportRecord[]>} */
+    async exportAll() {
+        await this.init();
+        const allFiles = await this._walkFiles();
+        const records = [];
+
+        for (const filePath of allFiles) {
+            const content = await this.read(filePath) || '';
+            let updatedAt = Date.now();
+            try {
+                const s = await stat(this._resolve(filePath));
+                updatedAt = s.mtimeMs;
+            } catch { /* skip */ }
+            records.push({
+                path: filePath,
+                content,
+                oneLiner: this._generateOneLiner(content),
+                itemCount: countBullets(content),
+                updatedAt,
+            });
+        }
+
+        return records;
+    }
+
+    async _listAllPaths() {
+        await this.init();
+        return (await this._walkFiles()).filter((p) => !this._isInternalPath(p));
+    }
+
+    // ─── Internal helpers ────────────────────────────────────────
+
+    _resolve(path) {
+        return join(this._root, path);
+    }
+
+    async _ensureDir(dirPath) {
+        await mkdir(join(this._root, dirPath), { recursive: true });
+    }
+
+    async _walkFiles(dir = '') {
+        const results = [];
+        const fullDir = dir ? join(this._root, dir) : this._root;
+
+        let entries;
+        try {
+            entries = await readdir(fullDir, { withFileTypes: true });
+        } catch {
+            return results;
+        }
+
+        for (const entry of entries) {
+            if (entry.name.startsWith('.')) continue;
+            const relPath = dir ? `${dir}/${entry.name}` : entry.name;
+            if (entry.isDirectory()) {
+                results.push(...await this._walkFiles(relPath));
+            } else if (entry.name.endsWith('.md')) {
+                results.push(relPath);
+            }
+        }
+
+        return results;
+    }
+}
+
+export { FileSystemStorage };