@kisev/memomatic 1.0.0 → 12.0.1

package/README.md

@@ -4,15 +4,27 @@
 
 `@kisev/memomatic` is a personal learning memory for agents following the OpenClaw
 memory architecture: a tiered Markdown corpus you can read as plain files, a
-rebuildable SQLite index with FTS5 and optional local embeddings, a `sessions`
-extraction pass, and a nightly `dream` consolidation sweep.
+rebuildable SQLite index with FTS5, optional local embeddings and an optional
+reranker, a `sessions` extraction pass, and a nightly `dream` consolidation sweep.
 
 ## Install
 
 ```bash
+# Registry version
+npm view --prefer-online @kisev/memomatic@latest version
+
+# Install
 npm install --global @kisev/memomatic
+
+# Installed package and active CLI
+npm list --global @kisev/memomatic --depth=0
+memomatic --version
 ```
 
+Tags can move between preview and installation; the last commands show the
+installed package and the CLI resolved from PATH. Use `@dev` in both the view
+and install commands for the development channel.
+
 The corpus lives under `$XDG_STATE_HOME/memomatic/` (`MEMORY.md`, `USER.md`,
 daily notes, `DREAMS.md`); rules live in `$XDG_CONFIG_HOME/memomatic/MEMORY_RULES.md`.
 Requires Node.js 22.13+. Connect `memomatic mcp-serve` as a local stdio MCP server
@@ -30,8 +42,9 @@ Nightly runs are scheduled by the systemd user units in `assets/systemd/`
 `memomatic-dream.service`/`.timer` for consolidation).
 
 Configure `sessions.model` (or keep a legacy `dream.model`) and an OpenCode
-provider before expecting session extraction; without a model, the session
-watermark is preserved. Follow the
+V2 provider before expecting session extraction; without a model, the session
+watermark is preserved. Session extraction and its model server require
+OpenCode `2.x`; a pre-V2 session database is not parsed. Follow the
 [setup, first-entry, and scheduling guide](../../docs/how-to/memomatic.md).
 
 ## CLI, Sessions, and Dream
@@ -65,7 +78,7 @@ when memomatic is absent.
 
 Every entry can carry a `source` annotation. Visibility derives from it:
 `team-*`, `gitlab`, and `spec-manage` entries may be quoted in team-facing
-artifacts; every other source (`people-journal`, `stopit`,
+artifacts; every other source (`people-journal`, `handoff`,
 `mattermost-triage`, `task-*`, `docs-*`, `user`) is personal-only. The label
 is exposed in search responses.
 

package/README.ru.md

@@ -4,15 +4,28 @@
 
 `@kisev/memomatic` — персональная обучающая память для агентов по архитектуре
 памяти OpenClaw: ярусный Markdown-корпус, который можно читать как обычные файлы,
-пересобираемый SQLite-индекс с FTS5 и опциональными локальными эмбеддингами,
-проход извлечения `sessions` и ночной проход консолидации `dream`.
+пересобираемый SQLite-индекс с FTS5, опциональными локальными эмбеддингами и
+опциональным реранкером, проход извлечения `sessions` и ночной проход
+консолидации `dream`.
 
 ## Установка
 
 ```bash
+# Registry version
+npm view --prefer-online @kisev/memomatic@latest version
+
+# Install
 npm install --global @kisev/memomatic
+
+# Installed package and active CLI
+npm list --global @kisev/memomatic --depth=0
+memomatic --version
 ```
 
+Тег может сдвинуться между просмотром и установкой; последние команды показывают
+установленный пакет и CLI из PATH. Для dev-канала укажите `@dev` и в просмотре,
+и в установке.
+
 Корпус живёт в `$XDG_STATE_HOME/memomatic/` (`MEMORY.md`, `USER.md`, ежедневные
 записи, `DREAMS.md`); правила — в `$XDG_CONFIG_HOME/memomatic/MEMORY_RULES.md`.
 Нужен Node.js 22.13+. Подключите `memomatic mcp-serve` как локальный MCP-сервер stdio
@@ -30,8 +43,9 @@ memomatic dream      # консолидация: inbox + продвижение
 `memomatic-dream.service`/`.timer` для консолидации).
 
 Перед извлечением сессий настройте `sessions.model` (или оставьте старый
-`dream.model`) и провайдера OpenCode; без модели отметка обработанных сессий
-сохраняется. См.
+`dream.model`) и провайдера OpenCode V2; без модели отметка обработанных сессий
+сохраняется. Извлечение сессий и его сервер модели требуют OpenCode `2.x`;
+база сессий до V2 не разбирается. См.
 [настройку, первую запись и расписание](../../docs/ru/how-to/memomatic.md).
 
 ## CLI, Sessions и Dream
@@ -65,7 +79,7 @@ inbox, бэклог сессий, кандидатов на продвижени
 
 Каждая запись может нести аннотацию `source`. Из неё выводится видимость:
 записи `team-*`, `gitlab` и `spec-manage` можно цитировать в командных
-артефактах; остальные источники (`people-journal`, `stopit`,
+артефактах; остальные источники (`people-journal`, `handoff`,
 `mattermost-triage`, `task-*`, `docs-*`, `user`) — personal-only. Метка
 возвращается в поиске.
 

package/dist/executor.js

@@ -1,8 +1,9 @@
 import { spawn } from "node:child_process";
-import { mkdtemp, rm } from "node:fs/promises";
+import { mkdtemp, rm, writeFile } from "node:fs/promises";
 import { tmpdir } from "node:os";
 import { join } from "node:path";
 import { randomBytes } from "node:crypto";
+import { isolatedProviderConfig } from "./opencode-config.js";
 export function extractJson(response) {
     const fenced = /```(?:json)?\s*([\s\S]*?)```/i.exec(response);
     for (const candidate of [fenced?.[1], response]) {
@@ -49,30 +50,34 @@ export class OpenCodeExecutor {
             if (!["http:", "https:"].includes(url.protocol) || url.username || url.password)
                 throw new Error("invalid OpenCode URL");
             this.url = url.href.replace(/\/$/, "");
-            if (process.env.OPENCODE_SERVER_PASSWORD)
-                this.authorization = `Basic ${Buffer.from(`${process.env.OPENCODE_SERVER_USERNAME ?? "opencode"}:${process.env.OPENCODE_SERVER_PASSWORD}`).toString("base64")}`;
+            const password = process.env.OPENCODE_PASSWORD ?? process.env.OPENCODE_SERVER_PASSWORD;
+            if (password)
+                this.authorization = `Basic ${Buffer.from(`opencode:${password}`).toString("base64")}`;
             return;
         }
-        let inherited = {};
-        if (process.env.OPENCODE_CONFIG_CONTENT) {
-            try {
-                const value = JSON.parse(process.env.OPENCODE_CONFIG_CONTENT);
-                if (!value || typeof value !== "object" || Array.isArray(value))
-                    throw new Error();
-                inherited = value;
-            }
-            catch {
-                throw new Error("OpenCode inline configuration must be a JSON object");
-            }
-        }
+        const inherited = await isolatedProviderConfig();
         this.directory = await mkdtemp(join(tmpdir(), "memomatic-opencode-"));
+        const configFile = join(this.directory, "opencode.json");
+        await writeFile(configFile, JSON.stringify({
+            ...inherited,
+            plugins: ["-opencode.skill", "-opencode.config.skill", "-opencode.tool.skill"],
+            permissions: [{ action: "*", resource: "*", effect: "deny" }],
+            agents: {
+                memomatic: {
+                    mode: "primary",
+                    description: "Bounded memory extraction",
+                    system: "Return only the requested JSON. Do not use tools.",
+                    permissions: [{ action: "*", resource: "*", effect: "deny" }],
+                },
+            },
+        }), { mode: 0o600 });
         const password = randomBytes(24).toString("hex");
         this.authorization = `Basic ${Buffer.from(`opencode:${password}`).toString("base64")}`;
         this.options.observe?.({
             phase: "opencode.start",
             message: "Starting one isolated OpenCode server for this run",
         });
-        const child = spawn(this.options.binary ?? "opencode", ["serve", "--hostname", "127.0.0.1", "--port", "0", "--pure"], {
+        const child = spawn(this.options.binary ?? "opencode", ["serve", "--hostname", "127.0.0.1", "--port", "0"], {
             cwd: this.directory,
             detached: process.platform !== "win32",
             stdio: ["ignore", "pipe", "pipe"],
@@ -80,25 +85,11 @@ export class OpenCodeExecutor {
                 ...process.env,
                 OPENCODE_SERVER_USERNAME: "opencode",
                 OPENCODE_SERVER_PASSWORD: password,
-                OPENCODE_DISABLE_PROJECT_CONFIG: "1",
-                OPENCODE_DISABLE_EXTERNAL_SKILLS: "1",
-                OPENCODE_DISABLE_CLAUDE_CODE_SKILLS: "1",
-                OPENCODE_DISABLE_DEFAULT_PLUGINS: "1",
-                OPENCODE_CONFIG_CONTENT: JSON.stringify({
-                    ...inherited,
-                    permission: "deny",
-                    tools: { "*": false },
-                    agent: {
-                        ...inherited.agent,
-                        memomatic: {
-                            mode: "primary",
-                            description: "Bounded memory extraction",
-                            prompt: "Return only the requested JSON. Do not use tools.",
-                            permission: "deny",
-                            tools: { "*": false },
-                        },
-                    },
-                }),
+                OPENCODE_CONFIG_PROJECT_DISABLE: "1",
+                OPENCODE_CONFIG_DIR: this.directory,
+                OPENCODE_CONFIG: configFile,
+                OPENCODE_CONFIG_CONTENT: "{}",
+                OPENCODE_PASSWORD: password,
             },
         });
         this.child = child;
@@ -141,9 +132,9 @@ export class OpenCodeExecutor {
         child.stderr?.resume();
         this.options.observe?.({ phase: "opencode.ready", message: "OpenCode server ready" });
     }
-    async request(path, body, signal) {
+    async request(path, body, signal, method = "POST") {
         const response = await fetch(`${this.url}${path}`, {
-            method: "POST",
+            method,
             headers: {
                 "content-type": "application/json",
                 ...(this.authorization ? { authorization: this.authorization } : {}),
@@ -152,8 +143,8 @@ export class OpenCodeExecutor {
             signal,
         });
         if (!response.ok)
-            throw new Error(`OpenCode ${path.includes("message") ? "model request" : "session request"} failed: HTTP ${response.status}`);
-        return (await response.json());
+            throw new Error(`OpenCode ${path.includes("generate") ? "model request" : "session request"} failed: HTTP ${response.status}`);
+        return response.status === 204 ? {} : (await response.json());
     }
     async complete(request) {
         this.options.signal?.throwIfAborted();
@@ -174,56 +165,43 @@ export class OpenCodeExecutor {
             }), 15000);
             heartbeat.unref();
             try {
-                const session = await this.request("/session", {
-                    title: "[memomatic-internal] memory extraction",
-                    permission: [{ permission: "*", pattern: "*", action: "deny" }],
-                }, signal);
-                if (typeof session.id !== "string")
-                    throw new Error("OpenCode session response is malformed");
-                id = session.id;
                 const slash = this.options.model?.indexOf("/") ?? -1;
                 if (this.options.model && slash < 1)
                     throw new Error("model must use provider/model format");
-                this.usage.calls++;
-                this.usage.charactersSent += request.system.length + request.prompt.length;
-                this.options.observe?.({
-                    phase: "model.request",
-                    message: "Sending prepared fragment to OpenCode",
-                    characters: request.prompt.length,
-                    attempt: attempt + 1,
-                });
-                const response = await this.request(`/session/${encodeURIComponent(id)}/message`, {
+                const session = await this.request("/api/session", {
+                    title: "[memomatic-internal] memory extraction",
                     agent: this.options.url ? "build" : "memomatic",
-                    system: request.system,
-                    tools: { "*": false },
+                    ...(this.directory ? { location: { directory: this.directory } } : {}),
+                    permissions: [{ action: "*", resource: "*", effect: "deny" }],
                     ...(this.options.model
                         ? {
                             model: {
                                 providerID: this.options.model.slice(0, slash),
-                                modelID: this.options.model.slice(slash + 1),
+                                id: this.options.model.slice(slash + 1),
+                                ...(this.options.variant ? { variant: this.options.variant } : {}),
                             },
                         }
                         : {}),
-                    ...(this.options.variant ? { variant: this.options.variant } : {}),
-                    parts: [{ type: "text", text: `[memomatic-internal] ${request.prompt}` }],
                 }, signal);
-                const info = response.info;
-                if (info?.error)
-                    throw new Error("OpenCode model returned an error; inspect the configured provider");
-                if (info?.tokens) {
-                    this.usage.usageAvailable = true;
-                    this.usage.inputTokens += info.tokens.input ?? 0;
-                    this.usage.outputTokens += info.tokens.output ?? 0;
-                    this.usage.cacheReadTokens += info.tokens.cache?.read ?? 0;
-                    this.usage.cacheWriteTokens += info.tokens.cache?.write ?? 0;
-                }
-                const parts = response.parts;
-                if (!Array.isArray(parts))
+                const data = session.data;
+                if (typeof data?.id !== "string")
+                    throw new Error("OpenCode session response is malformed");
+                id = data.id;
+                await this.request(`/api/experimental/session/${encodeURIComponent(id)}/instructions/entries/memomatic`, { value: request.system }, signal, "PUT");
+                this.usage.calls++;
+                this.usage.charactersSent += request.system.length + request.prompt.length;
+                this.options.observe?.({
+                    phase: "model.request",
+                    message: "Sending prepared fragment to OpenCode",
+                    characters: request.prompt.length,
+                    attempt: attempt + 1,
+                });
+                const response = await this.request(`/api/session/${encodeURIComponent(id)}/generate`, { prompt: `[memomatic-internal] ${request.prompt}` }, signal);
+                const generated = response.data;
+                if (typeof generated?.text !== "string")
                     throw new Error("OpenCode model response is malformed");
-                const parsed = extractJson(parts
-                    .filter((part) => part.type === "text")
-                    .map((part) => part.text ?? "")
-                    .join("\n"));
+                // V2 transient generation returns text only, not exact token usage.
+                const parsed = extractJson(generated.text);
                 this.options.observe?.({
                     phase: "model.done",
                     message: "Model response received",
@@ -234,7 +212,7 @@ export class OpenCodeExecutor {
             }
             catch (error) {
                 if (id)
-                    await this.request(`/session/${encodeURIComponent(id)}/abort`, {}, AbortSignal.timeout(3000)).catch(() => undefined);
+                    await this.request(`/api/session/${encodeURIComponent(id)}/interrupt?resume=false`, {}, AbortSignal.timeout(3000)).catch(() => undefined);
                 if (this.options.signal?.aborted)
                     throw this.options.signal.reason;
                 if (attempt === (this.options.retries ?? 1))

package/dist/ingest.js

@@ -4,5 +4,12 @@ export function opencodeDatabasePath() {
     const base = process.env.XDG_DATA_HOME ?? join(homedir(), ".local/share");
     if (!isAbsolute(base))
         throw new Error("XDG_DATA_HOME must be an absolute path");
+    if (process.env.OPENCODE_DB) {
+        if (process.env.OPENCODE_DB === ":memory:")
+            throw new Error("An in-memory OpenCode database cannot be ingested from another process");
+        return isAbsolute(process.env.OPENCODE_DB)
+            ? process.env.OPENCODE_DB
+            : join(base, "opencode", process.env.OPENCODE_DB);
+    }
     return join(base, "opencode", "opencode.db");
 }

package/dist/ingestion.js

@@ -21,24 +21,20 @@ export function planIngestion(file, store, options) {
     const db = new DatabaseSync(file, { readOnly: true });
     try {
         db.exec("BEGIN");
-        const hasParts = Boolean(db.prepare("SELECT 1 FROM sqlite_master WHERE name='part' AND type='table'").get());
-        const hasUpdated = db
-            .prepare("PRAGMA table_info(session)")
-            .all()
-            .some((row) => row.name === "time_updated");
-        const fastRevision = hasParts &&
-            hasUpdated &&
-            ["message", "part"].every((table) => db
+        const required = {
+            session_v2: ["id", "title", "directory", "time_created", "time_updated"],
+            session_message: ["id", "session_id", "type", "seq", "data", "time_created", "time_updated"],
+        };
+        for (const [table, fields] of Object.entries(required)) {
+            const columns = new Set(db
                 .prepare(`PRAGMA table_info(${table})`)
                 .all()
-                .some((row) => row.name === "time_updated"));
-        const partHasSession = hasParts &&
-            db
-                .prepare("PRAGMA table_info(part)")
-                .all()
-                .some((row) => row.name === "session_id");
+                .map((row) => row.name));
+            if (!fields.every((field) => columns.has(field)))
+                throw new Error("OpenCode V2 session projections are unavailable; start V2 to migrate history before ingestion");
+        }
         const sessions = db
-            .prepare(`SELECT id,title,directory,time_created,${hasUpdated ? "time_updated" : "time_created"} AS revision FROM session WHERE ${hasUpdated ? "time_updated" : "time_created"} < ? ORDER BY time_created,id`)
+            .prepare("SELECT id,COALESCE(title,'') AS title,directory,time_created,time_updated AS revision FROM session_v2 WHERE time_updated < ? ORDER BY time_created,id")
             .all(options.before);
         for (const [position, session] of sessions.entries()) {
             if (session.title.startsWith("[memomatic-internal]")) {
@@ -47,66 +43,40 @@ export function planIngestion(file, store, options) {
             }
             const sealKey = `ingest-session-v2:${digest(session.id)}`;
             let seal;
-            if (fastRevision) {
-                const messages = db
-                    .prepare("SELECT COUNT(*) AS count,MAX(time_updated) AS updated FROM message WHERE session_id=?")
-                    .get(session.id);
-                const parts = db
-                    .prepare(`SELECT COUNT(*) AS count,MAX(time_updated) AS updated FROM part WHERE ${partHasSession ? "session_id=?" : "message_id IN (SELECT id FROM message WHERE session_id=?)"}`)
-                    .get(session.id);
-                if (Math.max(Number(messages.updated), Number(parts.updated)) >= options.before) {
+            {
+                const revisions = db
+                    .prepare("SELECT id,seq,time_updated FROM session_message WHERE session_id=? ORDER BY seq,id")
+                    .all(session.id);
+                if (revisions.some((message) => Number(message.time_updated) >= options.before)) {
                     result.skipped++;
                     continue;
                 }
-                const revision = digest(JSON.stringify([session.revision, session.title, session.directory, messages, parts]));
+                const revision = digest(JSON.stringify([session.revision, session.title, session.directory, revisions]));
                 if (store.getMeta(sealKey) === revision) {
                     result.cachedSessions++;
                     continue;
                 }
                 seal = [sealKey, revision];
             }
-            const rows = hasParts
-                ? db
-                    .prepare(`SELECT m.id,m.data AS message,p.data AS part FROM message m JOIN part p ON p.message_id=m.id
-                WHERE ${partHasSession ? "p" : "m"}.session_id=?
-                AND CASE WHEN json_valid(p.data) THEN json_extract(p.data,'$.type') END='text'
-                AND CASE WHEN json_valid(m.data) THEN json_extract(m.data,'$.role') END IN ('user','assistant')
-                ORDER BY m.time_created,m.id,p.time_created,p.id`)
-                    .all(session.id)
-                : db
-                    .prepare("SELECT rowid AS id,data FROM message WHERE session_id=? ORDER BY time_created,rowid")
-                    .all(session.id).flatMap((row) => {
-                    try {
-                        const message = JSON.parse(row.data);
-                        return (message.parts ?? []).map((part) => ({
-                            id: String(row.id),
-                            message: row.data,
-                            part: JSON.stringify(part),
-                        }));
-                    }
-                    catch {
-                        return [];
-                    }
-                });
+            const rows = db
+                .prepare(`
+        SELECT id,type AS role,json_extract(data,'$.text') AS text,seq,0 AS part_index
+        FROM session_message WHERE session_id=? AND type='user'
+          AND CASE WHEN json_valid(data) THEN json_type(data,'$.text') END='text'
+        UNION ALL
+        SELECT m.id,m.type AS role,json_extract(p.value,'$.text') AS text,m.seq,CAST(p.key AS INTEGER) AS part_index
+        FROM session_message m JOIN json_each(CASE WHEN json_valid(m.data) THEN json_extract(m.data,'$.content') ELSE '[]' END) p
+        WHERE m.session_id=? AND m.type='assistant'
+          AND CASE WHEN json_valid(p.value) THEN json_extract(p.value,'$.type') END='text'
+          AND CASE WHEN json_valid(p.value) THEN json_type(p.value,'$.text') END='text'
+        ORDER BY seq,part_index
+      `)
+                .all(session.id, session.id);
             const messages = new Map();
             for (const row of rows) {
-                try {
-                    const message = JSON.parse(row.message);
-                    const part = JSON.parse(row.part);
-                    if (!["user", "assistant"].includes(message.role) ||
-                        part.type !== "text" ||
-                        typeof part.text !== "string")
-                        continue;
-                    const item = messages.get(row.id) ?? {
-                        role: message.role,
-                        texts: [],
-                    };
-                    item.texts.push(part.text);
-                    messages.set(row.id, item);
-                }
-                catch {
-                    result.skipped++;
-                }
+                const item = messages.get(row.id) ?? { role: row.role, texts: [] };
+                item.texts.push(row.text);
+                messages.set(row.id, item);
             }
             const firstUser = [...messages.values()].find((message) => message.role === "user");
             if (firstUser?.texts.join("\n").trimStart().startsWith("[memomatic-internal]")) {

package/dist/opencode-config.d.ts

@@ -0,0 +1 @@
+export declare function isolatedProviderConfig(): Promise<Record<string, unknown>>;

package/dist/opencode-config.js

@@ -0,0 +1,84 @@
+import { readFile } from "node:fs/promises";
+import { homedir } from "node:os";
+import { dirname, isAbsolute, join, resolve } from "node:path";
+import { parse } from "jsonc-parser";
+const PROVIDER_FIELDS = [
+    "providers",
+    "provider",
+    "model",
+    "enabled_providers",
+    "disabled_providers",
+    "experimental",
+    "enterprise",
+];
+function record(value) {
+    return Boolean(value && typeof value === "object" && !Array.isArray(value));
+}
+function merge(left, right, key = "") {
+    if (key === "policies" && Array.isArray(left) && Array.isArray(right))
+        return [...left, ...right];
+    if (!record(left) || !record(right))
+        return right;
+    return Object.fromEntries([...new Set([...Object.keys(left), ...Object.keys(right)])].map((key) => [
+        key,
+        key in right ? merge(left[key], right[key], key) : left[key],
+    ]));
+}
+function anchorFiles(value, directory) {
+    if (typeof value === "string")
+        return value.replace(/\{file:([^}]+)\}/g, (original, path) => isAbsolute(path) || path.startsWith("~/") ? original : `{file:${resolve(directory, path)}}`);
+    if (Array.isArray(value))
+        return value.map((entry) => anchorFiles(entry, directory));
+    if (record(value))
+        return Object.fromEntries(Object.entries(value).map(([key, entry]) => [key, anchorFiles(entry, directory)]));
+    return value;
+}
+export async function isolatedProviderConfig() {
+    const root = process.env.OPENCODE_CONFIG_DIR ??
+        join(process.env.XDG_CONFIG_HOME ?? join(homedir(), ".config"), "opencode");
+    const paths = [
+        join(root, "opencode.json"),
+        join(root, "opencode.jsonc"),
+        ...(process.env.OPENCODE_CONFIG ? [resolve(process.env.OPENCODE_CONFIG)] : []),
+    ];
+    let result = {};
+    const include = (value, directory) => {
+        if (!record(value))
+            throw new Error("OpenCode configuration must be a JSON object");
+        const selected = Object.fromEntries(PROVIDER_FIELDS.filter((key) => key in value).map((key) => [
+            key,
+            anchorFiles(value[key], directory),
+        ]));
+        if (record(selected.experimental))
+            selected.experimental = { policies: selected.experimental.policies ?? [] };
+        result = merge(result, selected);
+    };
+    for (const path of [...new Set(paths)]) {
+        let text;
+        try {
+            text = await readFile(path, "utf8");
+        }
+        catch (error) {
+            if (error.code === "ENOENT" &&
+                path !== (process.env.OPENCODE_CONFIG ? resolve(process.env.OPENCODE_CONFIG) : undefined))
+                continue;
+            throw new Error("Cannot read OpenCode provider configuration");
+        }
+        const errors = [];
+        const value = parse(text, errors, { allowTrailingComma: true });
+        if (errors.length)
+            throw new Error("OpenCode provider configuration is not valid JSON/JSONC");
+        include(value, dirname(path));
+    }
+    if (process.env.OPENCODE_CONFIG_CONTENT) {
+        let value;
+        try {
+            value = JSON.parse(process.env.OPENCODE_CONFIG_CONTENT);
+        }
+        catch {
+            throw new Error("OpenCode inline configuration must be a JSON object");
+        }
+        include(value, process.cwd());
+    }
+    return result;
+}

package/dist/search.d.ts

@@ -14,7 +14,11 @@ export type SearchHit = {
         exact: boolean;
         recency: number;
         importance: number;
-        reason: "lexical" | "semantic" | "both";
+        reason: "lexical" | "semantic" | "both" | "rerank";
+        rerank?: {
+            candidates: number;
+            threshold: number;
+        };
     };
 };
 export declare function corpusFiles(paths: MemomaticPaths): Promise<string[]>;

package/dist/search.js

@@ -3,7 +3,7 @@ import { join } from "node:path";
 import { cosineSimilarity, stableIdFor } from "./store.js";
 import { entryKind, parseCorpusEntries } from "./corpus.js";
 import { entryImportance, entryObservedAt } from "./entries.js";
-import { embedTexts, embeddingFingerprint } from "./settings.js";
+import { embedTexts, embeddingFingerprint, rerankTexts } from "./settings.js";
 import { check } from "./operations.js";
 export async function corpusFiles(paths) {
     const files = [];
@@ -73,9 +73,10 @@ export async function reindex(paths, settings, store, options = {}) {
         completed: 0,
         cached: entries.length - missing.length,
     });
-    for (let start = 0; start < missing.length; start += 32) {
+    const batchSize = Math.min(32, settings.embedding?.maxBatchTexts ?? 32);
+    for (let start = 0; start < missing.length; start += batchSize) {
         check(options);
-        const batch = missing.slice(start, start + 32);
+        const batch = missing.slice(start, start + batchSize);
         const computed = await embedTexts(settings, batch.map((item) => item.entry.text), "document", options.signal);
         if (vectors && computed)
             batch.forEach((item, index) => {
@@ -85,7 +86,7 @@ export async function reindex(paths, settings, store, options = {}) {
             phase: "index.embeddings",
             message: "Embedding batch complete",
             total: missing.length,
-            completed: Math.min(start + 32, missing.length),
+            completed: Math.min(start + batchSize, missing.length),
         });
     }
     check(options);
@@ -117,40 +118,21 @@ export function queryTokens(query) {
             .match(/[\p{L}\p{N}]+/gu) ?? []),
     ].filter((token) => token.length >= 2);
 }
-export async function search(store, query, settings, options = {}) {
-    if (!query.trim())
-        throw new Error("query is required");
-    const tokens = queryTokens(query);
-    const keyword = store.ftsSearch(query, 200);
-    const vector = new Map();
-    const storedVectors = store.vectors();
-    if (settings.embedding &&
-        store.allEntries().length &&
-        store.getMeta("embeddingFingerprint") !== embeddingFingerprint(settings))
-        throw new Error("embedding index is missing or belongs to another model; run memomatic index");
-    const queryVector = (await embedTexts(settings, [query], "query"))?.[0];
-    if (queryVector?.length) {
-        for (const row of storedVectors) {
-            if (row.data.length !== queryVector.length)
-                throw new Error("embedding dimensions changed; run memomatic index");
-            const similarity = cosineSimilarity(queryVector, row.data);
-            if (similarity > 0.05)
-                vector.set(row.stableId, similarity);
-        }
-    }
-    const candidates = new Set([...keyword.keys(), ...vector.keys()]);
-    const entries = store
-        .allEntries()
-        .filter((item) => options.project === undefined || item.project === options.project || item.kind === "user");
-    const byId = new Map(entries.map((entry) => [entry.stableId, entry]));
+function lexicalCoverage(entry, tokens, keyword) {
+    const words = queryTokens(entry.text);
+    const matchedTokens = tokens.filter((token) => words.some((word) => word.startsWith(token)));
+    return {
+        matchedTokens,
+        lexical: keyword.has(entry.stableId) && tokens.length ? matchedTokens.length / tokens.length : 0,
+    };
+}
+function gatedHits(tokens, keyword, vector, byId, settings) {
     const hits = [];
-    for (const stableId of candidates) {
+    for (const stableId of new Set([...keyword.keys(), ...vector.keys()])) {
         const entry = byId.get(stableId);
         if (!entry)
             continue;
-        const words = queryTokens(entry.text);
-        const matchedTokens = tokens.filter((token) => words.some((word) => word.startsWith(token)));
-        const lexical = keyword.has(stableId) && tokens.length ? matchedTokens.length / tokens.length : 0;
+        const { matchedTokens, lexical } = lexicalCoverage(entry, tokens, keyword);
         const semantic = vector.get(stableId) ?? null;
         const lexicalAccepted = lexical >= settings.search.minScore && lexical >= 0.5;
         const semanticAccepted = semantic !== null &&
@@ -164,6 +146,7 @@ export async function search(store, query, settings, options = {}) {
         const score = relevance * recency * (0.85 + 0.15 * importance);
         if (score < settings.search.minScore)
             continue;
+        const words = queryTokens(entry.text);
         hits.push({
             entry,
             score,
@@ -180,10 +163,104 @@ export async function search(store, query, settings, options = {}) {
             },
         });
     }
-    hits.sort((left, right) => Number(right.explanation.exact) - Number(left.explanation.exact) ||
+    return hits.sort((left, right) => Number(right.explanation.exact) - Number(left.explanation.exact) ||
         right.score - left.score ||
         (right.explanation.semantic ?? 0) - (left.explanation.semantic ?? 0) ||
         left.entry.stableId.localeCompare(right.entry.stableId));
+}
+/**
+ * Reranker path: collect a bounded wide candidate set from lexical and vector
+ * search before the strict relevance gates, apply the project filter, and let
+ * the reranker decide final relevance and order without giving literal
+ * matches an unconditional priority.
+ */
+async function rerankedHits(query, tokens, keyword, vector, byId, settings) {
+    const reranker = settings.reranker;
+    // The project filter narrows the pool before top-N selection, so hidden
+    // entries cannot consume the candidate budget.
+    const visible = (scores) => {
+        const filtered = new Map();
+        for (const [stableId, score] of scores)
+            if (byId.has(stableId))
+                filtered.set(stableId, score);
+        return filtered;
+    };
+    const widest = (scores) => [...scores.entries()]
+        .sort((left, right) => right[1] - left[1] || left[0].localeCompare(right[0]))
+        .slice(0, reranker.candidates)
+        .flatMap(([stableId]) => {
+        const entry = byId.get(stableId);
+        return entry ? [entry] : [];
+    });
+    const candidates = [];
+    const seen = new Set();
+    for (const entry of [...widest(visible(keyword)), ...widest(visible(vector))]) {
+        if (seen.has(entry.stableId))
+            continue;
+        seen.add(entry.stableId);
+        candidates.push(entry);
+    }
+    if (!candidates.length)
+        return [];
+    // Reranker documents stay raw entry text without embedding prefixes.
+    const ranked = await rerankTexts(settings, query, candidates.map((entry) => entry.text));
+    const hits = [];
+    for (const { index, score } of ranked) {
+        if (score < reranker.minScore)
+            continue;
+        const entry = candidates[index];
+        const { matchedTokens, lexical } = lexicalCoverage(entry, tokens, keyword);
+        const semantic = vector.get(entry.stableId) ?? null;
+        const recency = recencyMultiplier(entry, settings.search.halfLifeDays);
+        const importance = Math.max(0, Math.min(1, (entry.importance - 1) / 9));
+        const words = queryTokens(entry.text);
+        hits.push({
+            entry,
+            score,
+            snippet: entry.text.slice(0, 240),
+            explanation: {
+                matchedTokens,
+                lexical,
+                semantic,
+                relevance: score,
+                recency,
+                importance,
+                exact: tokens.length > 0 && tokens.every((token) => words.includes(token)),
+                reason: "rerank",
+                rerank: { candidates: candidates.length, threshold: reranker.minScore },
+            },
+        });
+    }
+    return hits;
+}
+export async function search(store, query, settings, options = {}) {
+    if (!query.trim())
+        throw new Error("query is required");
+    const tokens = queryTokens(query);
+    const keyword = store.ftsSearch(query, settings.reranker ? Math.max(200, settings.reranker.candidates) : 200);
+    const vector = new Map();
+    const storedVectors = store.vectors();
+    if (settings.embedding &&
+        store.allEntries().length &&
+        store.getMeta("embeddingFingerprint") !== embeddingFingerprint(settings))
+        throw new Error("embedding index is missing or belongs to another model; run memomatic index");
+    const queryVector = (await embedTexts(settings, [query], "query"))?.[0];
+    if (queryVector?.length) {
+        for (const row of storedVectors) {
+            if (row.data.length !== queryVector.length)
+                throw new Error("embedding dimensions changed; run memomatic index");
+            const similarity = cosineSimilarity(queryVector, row.data);
+            if (similarity > 0.05)
+                vector.set(row.stableId, similarity);
+        }
+    }
+    const entries = store
+        .allEntries()
+        .filter((item) => options.project === undefined || item.project === options.project || item.kind === "user");
+    const byId = new Map(entries.map((entry) => [entry.stableId, entry]));
+    const hits = settings.reranker
+        ? await rerankedHits(query, tokens, keyword, vector, byId, settings)
+        : gatedHits(tokens, keyword, vector, byId, settings);
     const limited = hits.slice(0, settings.search.maxResults);
     if (options.markSurfaced !== false && limited.length)
         store.markSurfaced(limited.map((hit) => hit.entry.stableId), query);

package/dist/service.d.ts

@@ -45,7 +45,11 @@ export declare function searchMemoryResults(context: MemomaticContext, query: st
         exact: boolean;
         recency: number;
         importance: number;
-        reason: "lexical" | "semantic" | "both";
+        reason: "lexical" | "semantic" | "both" | "rerank";
+        rerank?: {
+            candidates: number;
+            threshold: number;
+        };
     } | undefined;
     source: string | null;
     visibility: import("./visibility.js").EntryVisibility;

package/dist/settings.d.ts

@@ -1,9 +1,28 @@
+export type EmbeddingSettings = {
+    url: string;
+    model: string;
+    /** Explicit query instruction; null keeps the built-in Qwen3 auto-detection. */
+    queryInstruction: string | null;
+    documentPrefix: string;
+    queryPrefix: string;
+    bearerEnv: string | null;
+    timeoutMs: number;
+    maxBatchTexts: number;
+    maxTextChars: number;
+};
+export type RerankerSettings = {
+    url: string;
+    model: string;
+    bearerEnv: string | null;
+    timeoutMs: number;
+    candidates: number;
+    minScore: number;
+    maxDocuments: number;
+    maxChars: number;
+};
 export type MemomaticSettings = {
-    embedding: {
-        url: string;
-        model: string;
-        queryInstruction?: string;
-    } | null;
+    embedding: EmbeddingSettings | null;
+    reranker: RerankerSettings | null;
     sessions: {
         model: string | null;
         variant: string | null;
@@ -39,10 +58,43 @@ export type MemomaticSettings = {
         days: number;
     };
 };
+export declare const DEFAULT_MAX_TEXT_CHARS = 8000;
+export declare const RERANKER_TEMPLATE_ALLOWANCE_CHARS = 1024;
+export declare const defaultEmbedding: (url: string, model: string) => EmbeddingSettings;
+export declare const defaultReranker: (url: string, model: string) => RerankerSettings;
 export declare const defaultSettings: () => MemomaticSettings;
 export declare function normalizeSettings(raw: unknown): MemomaticSettings;
 export declare function loadSettings(settingsFile: string): Promise<MemomaticSettings>;
-export declare function embedTexts(settings: MemomaticSettings, texts: string[], purpose?: "document" | "query", signal?: AbortSignal): Promise<Float32Array[] | null>;
+/** Splits long text at paragraph, line or space boundaries within the budget. */
+export declare function chunkText(text: string, maxChars: number): string[];
 export declare const MEMORY_QUERY_INSTRUCTION = "Given a search query, retrieve relevant personal memory entries that answer the query.";
-export declare function embeddingQuery(settings: MemomaticSettings, query: string): string;
+export declare function embeddingQueryText(embedding: EmbeddingSettings, query: string): string;
+/** Headroom a formatted query chunk needs beyond its raw text. */
+export declare function embeddingQueryOverhead(embedding: EmbeddingSettings): number;
+export declare function embeddingDocumentText(embedding: EmbeddingSettings, text: string): string;
+/** Headroom a formatted document chunk needs beyond its raw text. */
+export declare function embeddingDocumentOverhead(embedding: EmbeddingSettings): number;
+/**
+ * Embeds texts through an OpenAI-compatible `embeddings` endpoint. Raw inputs
+ * are split with chunkText first and every chunk is formatted afterwards, so
+ * each document chunk carries the document prefix and each query chunk the
+ * query prefix plus the Qwen instruction, all inside maxTextChars. Chunks are
+ * embedded chunk-wise and mean-pooled back onto one vector per input text.
+ * Request character budgets are a conservative proxy, not exact tokenization.
+ */
+export declare function embedTexts(settings: MemomaticSettings, texts: string[], purpose?: "document" | "query", signal?: AbortSignal): Promise<Float32Array[] | null>;
+export type RerankHit = {
+    index: number;
+    score: number;
+};
+/**
+ * Ranks documents against a query through a `/rerank`-style endpoint
+ * (`{model, query, documents, top_n}` request, `results[].index` and
+ * `results[].relevance_score` response). Documents are chunked so the query,
+ * one document chunk and the server template stay inside maxChars; a query
+ * too large to leave room for a chunk fails before the request. Responses
+ * must contain exactly top_n results, and chunk scores aggregate to one best
+ * score per input document.
+ */
+export declare function rerankTexts(settings: MemomaticSettings, query: string, documents: string[], signal?: AbortSignal): Promise<RerankHit[]>;
 export declare function embeddingFingerprint(settings: MemomaticSettings): string;

package/dist/settings.js

@@ -1,7 +1,36 @@
 import { readFile } from "node:fs/promises";
 import { createHash } from "node:crypto";
+// Conservative character proxy for an input budget of roughly 4096 tokens:
+// dense scripts need about two characters per token, so 8000 characters stay
+// inside the example budget without claiming exact tokenization. Long texts
+// are split at chunkText instead of measured with a tokenizer.
+export const DEFAULT_MAX_TEXT_CHARS = 8000;
+// Headroom for the server-side rerank template wrapped around query and document.
+export const RERANKER_TEMPLATE_ALLOWANCE_CHARS = 1024;
+export const defaultEmbedding = (url, model) => ({
+    url,
+    model,
+    queryInstruction: null,
+    documentPrefix: "",
+    queryPrefix: "",
+    bearerEnv: null,
+    timeoutMs: 30_000,
+    maxBatchTexts: 32,
+    maxTextChars: DEFAULT_MAX_TEXT_CHARS,
+});
+export const defaultReranker = (url, model) => ({
+    url,
+    model,
+    bearerEnv: null,
+    timeoutMs: 30_000,
+    candidates: 64,
+    minScore: 0,
+    maxDocuments: 64,
+    maxChars: DEFAULT_MAX_TEXT_CHARS,
+});
 export const defaultSettings = () => ({
     embedding: null,
+    reranker: null,
     sessions: {
         model: null,
         variant: null,
@@ -50,10 +79,56 @@ const SESSIONS_KEYS = [
     "idleMs",
     "opencodeUrl",
 ];
+function httpUrl(value, section) {
+    if (typeof value !== "string")
+        throw new Error(`${section}.url must be a string`);
+    let parsed;
+    try {
+        parsed = new URL(value);
+    }
+    catch {
+        throw new Error(`${section}.url must be an absolute http(s) URL`);
+    }
+    if (parsed.protocol !== "http:" && parsed.protocol !== "https:")
+        throw new Error(`${section}.url must use http or https`);
+    return value;
+}
+function bearerEnvName(value, section) {
+    if (value === null || value === undefined)
+        return null;
+    if (typeof value !== "string" || !/^[A-Za-z_][A-Za-z0-9_]*$/.test(value))
+        throw new Error(`${section}.bearerEnv must name an environment variable`);
+    return value;
+}
+function boundedInt(value, section, key, min, max) {
+    if (!Number.isSafeInteger(value) || value < min || value > max)
+        throw new Error(`invalid ${section}.${key}; expected an integer between ${min} and ${max}`);
+    return value;
+}
+function optionalText(value, section, key) {
+    if (typeof value !== "string")
+        throw new Error(`${section}.${key} must be a string`);
+    return value;
+}
+function serviceEndpoint(raw, section, defaults, validate) {
+    if (raw === null || raw === undefined)
+        return null;
+    if (typeof raw !== "object")
+        throw new Error(`invalid ${section} settings`);
+    const url = httpUrl(raw.url, section);
+    if (typeof raw.model !== "string" || !raw.model.trim())
+        throw new Error(`${section}.model must be a non-empty string`);
+    const merged = defaults(url, raw.model);
+    for (const [key, value] of Object.entries(raw)) {
+        if (key in merged && value !== null && value !== undefined)
+            merged[key] = value;
+    }
+    validate(merged);
+    return merged;
+}
 export function normalizeSettings(raw) {
     const defaults = defaultSettings();
     const source = (raw ?? {});
-    const embedding = source.embedding;
     const search = mergeSection(defaults.search, source.search);
     const dream = mergeSection(defaults.dream, source.dream);
     // Legacy single-model configs put extraction knobs under `dream`; adopt them
@@ -106,16 +181,32 @@ export function normalizeSettings(raw) {
         search.halfLifeDays <= 0 ||
         ![search.minScore, search.minSemanticScore].every((value) => Number.isFinite(value) && value >= 0 && value <= 1))
         throw new Error("invalid search settings");
+    const embedding = serviceEndpoint((source.embedding ?? null), "embedding", (url, model) => defaultEmbedding(url, model), (merged) => {
+        if (merged.queryInstruction !== null && typeof merged.queryInstruction !== "string")
+            throw new Error("embedding.queryInstruction must be a string or null");
+        merged.queryInstruction =
+            typeof merged.queryInstruction === "string" ? merged.queryInstruction : null;
+        merged.documentPrefix = optionalText(merged.documentPrefix, "embedding", "documentPrefix");
+        merged.queryPrefix = optionalText(merged.queryPrefix, "embedding", "queryPrefix");
+        merged.bearerEnv = bearerEnvName(merged.bearerEnv, "embedding");
+        merged.timeoutMs = boundedInt(merged.timeoutMs, "embedding", "timeoutMs", 1, Number.MAX_SAFE_INTEGER);
+        merged.maxBatchTexts = boundedInt(merged.maxBatchTexts, "embedding", "maxBatchTexts", 1, 512);
+        merged.maxTextChars = boundedInt(merged.maxTextChars, "embedding", "maxTextChars", 256, 1_000_000);
+    });
+    const reranker = serviceEndpoint((source.reranker ?? null), "reranker", (url, model) => defaultReranker(url, model), (merged) => {
+        merged.bearerEnv = bearerEnvName(merged.bearerEnv, "reranker");
+        merged.timeoutMs = boundedInt(merged.timeoutMs, "reranker", "timeoutMs", 1, Number.MAX_SAFE_INTEGER);
+        merged.candidates = boundedInt(merged.candidates, "reranker", "candidates", 1, 1000);
+        // Scores are ranking signals whose scale depends on the model, so the
+        // threshold is any finite non-negative number, not a probability bound.
+        if (!Number.isFinite(merged.minScore) || merged.minScore < 0)
+            throw new Error("invalid reranker.minScore; expected a finite number of at least 0");
+        merged.maxDocuments = boundedInt(merged.maxDocuments, "reranker", "maxDocuments", 1, 512);
+        merged.maxChars = boundedInt(merged.maxChars, "reranker", "maxChars", 256, 1_000_000);
+    });
     return {
-        embedding: embedding && typeof embedding.url === "string" && typeof embedding.model === "string"
-            ? {
-                url: embedding.url,
-                model: embedding.model,
-                ...(typeof embedding.queryInstruction === "string"
-                    ? { queryInstruction: embedding.queryInstruction }
-                    : {}),
-            }
-            : null,
+        embedding,
+        reranker,
         sessions,
         dream,
         search,
@@ -132,25 +223,92 @@ export async function loadSettings(settingsFile) {
         throw error;
     }
 }
-export async function embedTexts(settings, texts, purpose = "document", signal) {
-    if (!settings.embedding || !texts.length)
-        return null;
-    const response = await fetch(settings.embedding.url, {
-        body: JSON.stringify({
-            input: texts.map((text) => (purpose === "query" ? embeddingQuery(settings, text) : text)),
-            model: settings.embedding.model,
-        }),
-        headers: { "content-type": "application/json" },
+/** Splits long text at paragraph, line or space boundaries within the budget. */
+export function chunkText(text, maxChars) {
+    if (text.length <= maxChars)
+        return [text];
+    const chunks = [];
+    let start = 0;
+    while (start < text.length) {
+        let end = Math.min(start + maxChars, text.length);
+        if (end < text.length) {
+            const window = text.slice(start, end);
+            const boundary = Math.max(window.lastIndexOf("\n\n"), window.lastIndexOf("\n"), window.lastIndexOf(" "));
+            if (boundary >= Math.floor(maxChars / 2))
+                end = start + boundary + 1;
+        }
+        const chunk = text.slice(start, end).trim();
+        if (chunk)
+            chunks.push(chunk);
+        start = end;
+    }
+    return chunks.length ? chunks : [text.slice(0, maxChars)];
+}
+function bearerToken(bearerEnv, service) {
+    if (!bearerEnv)
+        return undefined;
+    const token = process.env[bearerEnv];
+    if (!token)
+        throw new Error(`${service} bearerEnv "${bearerEnv}" is not set`);
+    // A control character makes the Authorization header value invalid; the
+    // fetch-level rejection echoes the token back, so reject it here where the
+    // diagnostic stays free of the secret.
+    if (/[\u0000-\u001f\u007f]/.test(token))
+        throw new Error(`${service} bearerEnv "${bearerEnv}" contains a control character; expected a single-line token`);
+    return token;
+}
+async function postJson(url, body, token, timeoutMs, signal, service) {
+    const response = await fetch(url, {
+        body: JSON.stringify(body),
+        headers: {
+            "content-type": "application/json",
+            ...(token ? { authorization: `Bearer ${token}` } : {}),
+        },
         method: "POST",
         signal: signal
-            ? AbortSignal.any([signal, AbortSignal.timeout(30_000)])
-            : AbortSignal.timeout(30_000),
+            ? AbortSignal.any([signal, AbortSignal.timeout(timeoutMs)])
+            : AbortSignal.timeout(timeoutMs),
+    }).catch((error) => {
+        const failure = error;
+        if (failure.name === "TimeoutError")
+            throw new Error(`${service} request timed out after ${timeoutMs}ms`);
+        if (failure.name === "AbortError")
+            throw new Error(`${service} request was aborted`);
+        throw new Error(`${service} request failed: ${failure.message}`);
     });
     if (!response.ok)
-        throw new Error(`embedding request failed: ${response.status} ${await response.text()}`);
-    const payload = (await response.json());
-    const list = payload.data ?? [];
-    if (list.length !== texts.length)
+        throw new Error(`${service} request failed: ${response.status} ${(await response.text()).slice(0, 2000)}`);
+    try {
+        return await response.json();
+    }
+    catch {
+        throw new Error(`${service} response is not valid JSON`);
+    }
+}
+export const MEMORY_QUERY_INSTRUCTION = "Given a search query, retrieve relevant personal memory entries that answer the query.";
+function queryInstructionFor(embedding) {
+    return (embedding.queryInstruction ??
+        (/qwen3[-_]?embedding/i.test(embedding.model) ? MEMORY_QUERY_INSTRUCTION : ""));
+}
+export function embeddingQueryText(embedding, query) {
+    const instruction = queryInstructionFor(embedding);
+    return `${embedding.queryPrefix}${instruction ? `Instruct: ${instruction}\nQuery:${query}` : query}`;
+}
+/** Headroom a formatted query chunk needs beyond its raw text. */
+export function embeddingQueryOverhead(embedding) {
+    const instruction = queryInstructionFor(embedding);
+    return (embedding.queryPrefix.length + (instruction ? `Instruct: ${instruction}\nQuery:`.length : 0));
+}
+export function embeddingDocumentText(embedding, text) {
+    return `${embedding.documentPrefix}${text}`;
+}
+/** Headroom a formatted document chunk needs beyond its raw text. */
+export function embeddingDocumentOverhead(embedding) {
+    return embedding.documentPrefix.length;
+}
+function parseEmbeddingPayload(payload, count) {
+    const list = (payload.data ?? []);
+    if (!Array.isArray(list) || list.length !== count)
         throw new Error("embedding response is incomplete");
     const byIndex = new Map();
     for (const item of list) {
@@ -160,21 +318,145 @@ export async function embedTexts(settings, texts, purpose = "document", signal)
             !item.embedding.some((value) => value !== 0))
             throw new Error("embedding response is malformed");
         const index = item.index ?? byIndex.size;
-        if (!Number.isInteger(index) || index < 0 || index >= texts.length || byIndex.has(index))
+        if (!Number.isInteger(index) || index < 0 || index >= count || byIndex.has(index))
             throw new Error("embedding response has invalid indices");
         byIndex.set(index, item.embedding);
     }
-    const vectors = texts.map((_, index) => Float32Array.from(byIndex.get(index)));
+    const vectors = Array.from({ length: count }, (_, index) => Float32Array.from(byIndex.get(index)));
     if (!vectors.every((vector) => vector.length === vectors[0].length && vector.every(Number.isFinite)))
         throw new Error("embedding response has incompatible vectors");
     return vectors;
 }
-export const MEMORY_QUERY_INSTRUCTION = "Given a search query, retrieve relevant personal memory entries that answer the query.";
-export function embeddingQuery(settings, query) {
+function meanPool(vectors) {
+    if (vectors.length === 1)
+        return vectors[0];
+    const pooled = new Float32Array(vectors[0].length);
+    for (const vector of vectors) {
+        if (vector.length !== pooled.length)
+            throw new Error("embedding response has incompatible vectors");
+        for (let index = 0; index < vector.length; index += 1)
+            pooled[index] += vector[index];
+    }
+    for (let index = 0; index < pooled.length; index += 1)
+        pooled[index] /= vectors.length;
+    return pooled;
+}
+/**
+ * Embeds texts through an OpenAI-compatible `embeddings` endpoint. Raw inputs
+ * are split with chunkText first and every chunk is formatted afterwards, so
+ * each document chunk carries the document prefix and each query chunk the
+ * query prefix plus the Qwen instruction, all inside maxTextChars. Chunks are
+ * embedded chunk-wise and mean-pooled back onto one vector per input text.
+ * Request character budgets are a conservative proxy, not exact tokenization.
+ */
+export async function embedTexts(settings, texts, purpose = "document", signal) {
+    if (!settings.embedding || !texts.length)
+        return null;
     const embedding = settings.embedding;
-    const instruction = embedding?.queryInstruction ??
-        (/qwen3[-_]?embedding/i.test(embedding?.model ?? "") ? MEMORY_QUERY_INSTRUCTION : "");
-    return instruction ? `Instruct: ${instruction}\nQuery:${query}` : query;
+    const token = bearerToken(embedding.bearerEnv, "embedding");
+    const format = purpose === "query"
+        ? (text) => embeddingQueryText(embedding, text)
+        : (text) => embeddingDocumentText(embedding, text);
+    const overhead = purpose === "query" ? embeddingQueryOverhead(embedding) : embeddingDocumentOverhead(embedding);
+    const chunkBudget = embedding.maxTextChars - overhead;
+    if (chunkBudget < 1)
+        throw new Error(`embedding ${purpose} formatting needs ${overhead} characters of headroom and leaves no room inside maxTextChars=${embedding.maxTextChars}`);
+    // Split the raw text with formatting headroom reserved, then format each
+    // chunk so prefixes and instructions cover every piece, not just the first.
+    const chunkLists = texts.map((text) => chunkText(text, chunkBudget).map(format));
+    const chunks = chunkLists.flat();
+    const vectors = new Array(chunks.length);
+    for (let start = 0; start < chunks.length; start += embedding.maxBatchTexts) {
+        const batch = chunks.slice(start, start + embedding.maxBatchTexts);
+        const payload = await postJson(embedding.url, { input: batch, model: embedding.model, encoding_format: "float" }, token, embedding.timeoutMs, signal, "embedding");
+        parseEmbeddingPayload(payload, batch.length).forEach((vector, index) => (vectors[start + index] = vector));
+    }
+    const pooled = [];
+    let offset = 0;
+    for (const list of chunkLists) {
+        pooled.push(meanPool(vectors.slice(offset, offset + list.length)));
+        offset += list.length;
+    }
+    return pooled;
+}
+function parseRerankResults(payload, count, offset) {
+    const results = payload.results;
+    if (!Array.isArray(results))
+        throw new Error("reranker response is missing results");
+    // top_n equals the number of sent documents, so a complete answer ranks
+    // every candidate; an empty or shortened response is a failure, not an
+    // empty success.
+    if (results.length !== count)
+        throw new Error("reranker response is incomplete");
+    const seen = new Set();
+    const hits = [];
+    for (const item of results) {
+        const record = item;
+        const index = record?.index;
+        const score = record?.relevance_score;
+        if (!Number.isInteger(index) || index < 0 || index >= count)
+            throw new Error("reranker response has invalid indices");
+        if (typeof score !== "number" || !Number.isFinite(score))
+            throw new Error("reranker response is malformed");
+        if (seen.has(index))
+            throw new Error("reranker response has invalid indices");
+        seen.add(index);
+        hits.push({ index: offset + index, score });
+    }
+    return hits;
+}
+/**
+ * Ranks documents against a query through a `/rerank`-style endpoint
+ * (`{model, query, documents, top_n}` request, `results[].index` and
+ * `results[].relevance_score` response). Documents are chunked so the query,
+ * one document chunk and the server template stay inside maxChars; a query
+ * too large to leave room for a chunk fails before the request. Responses
+ * must contain exactly top_n results, and chunk scores aggregate to one best
+ * score per input document.
+ */
+export async function rerankTexts(settings, query, documents, signal) {
+    const reranker = settings.reranker;
+    if (!reranker || !documents.length)
+        return [];
+    const token = bearerToken(reranker.bearerEnv, "reranker");
+    // Query, one document chunk and the template allowance share maxChars. A
+    // small remainder is honored as-is; inflating it would push the request
+    // past the model input budget.
+    const remaining = reranker.maxChars - RERANKER_TEMPLATE_ALLOWANCE_CHARS - query.length;
+    if (remaining < 1)
+        throw new Error(`reranker query of ${query.length} characters leaves no document room inside maxChars=${reranker.maxChars}`);
+    const chunkLists = documents.map((document) => chunkText(document, remaining));
+    const offsets = [];
+    let total = 0;
+    for (const list of chunkLists) {
+        offsets.push(total);
+        total += list.length;
+    }
+    const scores = new Map();
+    const chunks = chunkLists.flat();
+    for (let start = 0; start < total; start += reranker.maxDocuments) {
+        const batch = chunks.slice(start, start + reranker.maxDocuments);
+        const payload = await postJson(reranker.url, {
+            model: reranker.model,
+            query,
+            documents: batch,
+            top_n: batch.length,
+        }, token, reranker.timeoutMs, signal, "reranker");
+        for (const { index, score } of parseRerankResults(payload, batch.length, start))
+            scores.set(index, Math.max(scores.get(index) ?? Number.NEGATIVE_INFINITY, score));
+    }
+    const hits = [];
+    chunkLists.forEach((list, index) => {
+        let best = null;
+        for (let offset = 0; offset < list.length; offset += 1) {
+            const score = scores.get(offsets[index] + offset);
+            if (score !== undefined && (best === null || score > best))
+                best = score;
+        }
+        if (best !== null)
+            hits.push({ index, score: best });
+    });
+    return hits.sort((left, right) => right.score - left.score || left.index - right.index);
 }
 export function embeddingFingerprint(settings) {
     return createHash("sha256")
@@ -182,7 +464,15 @@ export function embeddingFingerprint(settings) {
         ? {
             url: settings.embedding.url,
             model: settings.embedding.model,
-            documentFormat: "raw-v1",
+            document: {
+                prefix: settings.embedding.documentPrefix,
+                maxChars: settings.embedding.maxTextChars,
+                pooling: "mean-v1",
+                // Chunking now happens before formatting, so every chunk
+                // carries the prefix/instruction; bumping this version
+                // invalidates indexes built by the previous algorithm.
+                formatting: "per-chunk-v2",
+            },
         }
         : null))
         .digest("hex");

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@kisev/memomatic",
-  "version": "1.0.0",
+  "version": "12.0.1",
   "description": "Personal learning memory for agents: tiered Markdown corpus, rebuildable index, dream sweep.",
   "homepage": "https://github.com/kisev/skills#readme",
   "bugs": {
@@ -30,12 +30,13 @@
   },
   "scripts": {
     "build": "node scripts/clean-dist.mjs && tsc --project tsconfig.json",
-    "test": "node --test test/",
+    "test": "node --test test/*.test.mjs",
     "pack:check": "node test/pack-allowlist.mjs"
   },
   "dependencies": {
-    "@kisev/safe-fs": "^1.0.0",
-    "commander": "15.0.0"
+    "@kisev/safe-fs": "^12.0.1",
+    "commander": "15.0.0",
+    "jsonc-parser": "3.3.1"
   },
   "devDependencies": {
     "@types/node": "22.15.30",