@staix/agent-mem 0.9.1 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +1 -1
- package/README.md +17 -5
- package/package.json +1 -1
- package/src/cli.ts +22 -9
- package/src/commands-table.ts +5 -4
- package/src/index.ts +1 -0
- package/src/ingest.ts +98 -7
- package/src/mcp-memory.ts +47 -35
- package/src/mcp.ts +1 -1
- package/src/memory/candidates.ts +44 -1
- package/src/memory/capture.ts +24 -6
- package/src/memory/commands.ts +23 -4
- package/src/memory/consolidate.ts +9 -6
- package/src/memory/consolidation-clusters.ts +12 -7
- package/src/memory/eval-fixture.ts +7 -3
- package/src/memory/eval.ts +17 -8
- package/src/memory/external-import.ts +8 -1
- package/src/memory/platform.ts +8 -8
- package/src/memory/policy.ts +62 -3
- package/src/memory/project-import.ts +31 -14
- package/src/memory/search-fts.ts +68 -3
- package/src/memory/search-score.ts +66 -0
- package/src/memory/store.ts +40 -13
- package/src/session-brief-worker.ts +5 -1
- package/src/setup-agy.ts +9 -0
- package/src/setup-jcode.ts +7 -0
- package/src/setup-json.ts +151 -0
- package/src/setup-owned.ts +40 -19
- package/src/setup.ts +147 -34
package/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Curated memory with append-only JSONL logs, a derived SQLite index, short- and long-term tiers, a typed graph, goals with leases, and git sync between machines. Extracted from [agent-hub](https://github.com/STAIxBWLB/agent-hub), with compatible storage and entry IDs.
|
|
4
4
|
|
|
5
|
-
Version: 0.
|
|
5
|
+
Version: 0.11.0. Requires Bun 1.4.2 or later; no runtime dependencies.
|
|
6
6
|
|
|
7
7
|
## Quick start
|
|
8
8
|
|
|
@@ -15,7 +15,7 @@ amem setup all
|
|
|
15
15
|
amem setup claude --apply
|
|
16
16
|
```
|
|
17
17
|
|
|
18
|
-
Replace `claude` with `codex`, `kimi` or `
|
|
18
|
+
Replace `claude` with `codex`, `kimi`, `pi`, `qwen` or `jcode` for that client. Setup shows its changes and asks for confirmation. Complete the client's own hook-trust review when prompted.
|
|
19
19
|
|
|
20
20
|
Agents submit pending memory candidates through MCP. Inspect them after a session and accept or reject each one:
|
|
21
21
|
|
|
@@ -26,12 +26,18 @@ amem review accept CANDIDATE_ID # or: amem review reject CANDIDATE_ID
|
|
|
26
26
|
|
|
27
27
|
Replace `CANDIDATE_ID` with an id from the list. Acceptance is person-only and applies the configured privacy policy.
|
|
28
28
|
|
|
29
|
-
Claude's registered hooks capture events during the session. The configured Codex/Kimi SessionEnd hooks and Pi shutdown extension ingest transcript tails incrementally at session end. If those integrations are unavailable, run `amem ingest codex` (or `kimi` / `pi`) after each session, then repeat the review loop. The [setup section](#setup--plugin) describes the registration and native trust steps.
|
|
29
|
+
Claude's registered hooks capture events during the session. The configured Codex/Kimi/Qwen SessionEnd hooks and Pi shutdown extension ingest transcript tails incrementally at session end. If those integrations are unavailable, run `amem ingest codex` (or `kimi` / `pi` / `qwen`) after each session, then repeat the review loop. The [setup section](#setup--plugin) describes the registration and native trust steps.
|
|
30
30
|
|
|
31
31
|
From a checkout, install development dependencies with `bun install` and run `bun src/cli.ts --help`.
|
|
32
32
|
|
|
33
33
|
## Setup / plugin
|
|
34
34
|
|
|
35
|
+
`amem setup agy` registers MCP only in the current git repository's `.agents/mcp_config.json`, with an explicit absolute repository cwd and a private receipt per repository. `setup all` excludes agy; run it separately in each repository. Setup refuses tracked config files and JSONC, and warns when the file is not ignored. Restart Antigravity after applying, use one workspace per session, and ensure no global, plugin or running server is also named `amem`. No global configuration, ingest command or hook is installed. Undo removes `.agents` only when setup created it and it is still empty; pre-existing or nonempty directories remain. Older receipts still restore their configuration without directory cleanup. See the [Antigravity contract and capture limitation](docs/antigravity.md).
|
|
36
|
+
|
|
37
|
+
`amem setup jcode` registers MCP only, using Jcode v0.99.1's global `JCODE_HOME/mcp.json` (default `~/.jcode/mcp.json`). An explicitly set `JCODE_HOME` must be absolute for amem setup: empty, relative and literal `~` values are refused. Unset uses the default, and absolute values retain their literal spelling without trimming. It owns only `servers.amem`, sets `shared: false` for a workspace-bound session, and preserves unrelated configuration bytes. Jcode has no native MCP add/remove command in this release, so setup edits the owned entry directly. No Jcode ingest command or capture hook is installed: saved messages lack per-message origin while the session cwd can change. Launch Jcode with `JCODE_DISABLE_CLAUDE_MCP=1` and without a same-name project override, which could otherwise replace the private binding. Use a new workspace-bound session when changing projects. See the [Jcode contract and capture limitation](docs/jcode.md).
|
|
38
|
+
|
|
39
|
+
`amem setup qwen` previews a user-scope MCP registration and a bounded SessionEnd ingest hook in the same `QWEN_HOME/settings.json` (default `~/.qwen/settings.json`). `--apply` and `--undo` follow the same confirmed, private-receipt contract as the other targets. Unrelated settings, servers and hooks are preserved. Qwen runs this hook on normal interactive exit; use `amem ingest qwen` when the session ends without that hook. Native-client operation is source-verified; the installed-package tests use stub registration CLIs.
|
|
40
|
+
|
|
35
41
|
Preview the exact configuration files, native registration commands and private undo receipts before changing anything:
|
|
36
42
|
|
|
37
43
|
```sh
|
|
@@ -203,7 +209,11 @@ Project identity (#15 D2) keeps `memoryProjectKey`: a worktree, a submodule and
|
|
|
203
209
|
|
|
204
210
|
`store.search(query, filter)` searches the curated tier and the current project's capture shard with SQLite FTS5 and BM25 plus tier/recency weighting: natural multi-word queries match partial term sets (the substring filter required every word), and a curated row outranks the same text in the capture tier. Prefix terms match inflections and Korean particles in either direction by retaining the original query term and adding a common-particle-stripped alternative. The FTS copy of the curated rows lives in its own derived `search.db` (`index.db`'s table set is a frozen contract and gains nothing); it is written from the same apply path as the index, and a prune, crash or rebuild is detected and resynced, so the incremental FTS copy equals a full resync. Where FTS5 is unavailable the engine falls back to the substring filter and reports `engine: "substring"`; capture recall needs FTS5 and is skipped in the fallback. The library entry exports `CaptureStore`, `captureSearch`, `readCaptureConfig`, `writeCaptureConfig`, `SearchIndex`, `fts5Available`, the shard helpers (`resolveAlias`, `appendAlias`, `listAliases`, `shardKeyHash`), the catalog (`rebuildCatalog`, `readCatalog`, `scanCatalog`) and the capacity advisor (`capacityReport`).
|
|
205
211
|
|
|
206
|
-
`amem
|
|
212
|
+
`amem search <words...>` ranks the current project's current curated notes with `store.search` (FTS5/BM25 plus tier and recency weights). Natural multi-word queries can match partial term sets: `amem search lighthouse beacon` can return a note containing only one of those terms. Add `--filter` to retain the previous behavior for current-project curated notes: every query word must appear as a substring, and results are newest first. `--history`, `--as-of` and an empty query retain the previous filter/newest-first path. Repository-scope notes remain a separately screened append to these results. `--all-projects` keeps its existing ranked path and cannot be combined with `--filter`. JSON output retains the existing note fields.
|
|
213
|
+
|
|
214
|
+
`amem eval [--set ko|<file>] [--k 1,5,10] [--json]` measures recall with no model and no network (agent-mem#40 F13): it replays a question set against ranked `store.search` and reports recall@k, hit@k and MRR. The ordinary curated project `amem search` path now uses that ranked search; `--filter` selects the legacy substring behavior. Without `--set` it scores the built-in synthetic set (26 invented harbour entries, long and short tier, written a day apart, and 32 questions, shipped in `src/memory/eval-set.json`), which is seeded into a throwaway store, so it never reads or writes your memory home and gives the same numbers on every machine. A set of your own either carries its own `entries` (self-contained, named by `key`) or, without entries, names real entry ids (or a prefix of 8 or more hex characters) and is scored against this project's store; each name is resolved like `amem show` does (a promoted short-term id finds its long-term copy, and a superseded id stands for the entry that replaced it, since search returns only live entries), and a prefix that matches more than one live entry is refused. `mem:<id>` as a brief prints it is accepted. Questions whose expected entry is not in the store score 0 in `all` and are left out of `answerable`, and `answerable` is scored over the entries the store holds, so a ranking problem and a coverage problem stay apart (`n/a` when no question is answerable). `--set ko` scores the second built-in set instead: 37 invented Korean entries and 38 questions (particle variants, spaced and unspaced compounds, mixed Korean-English terms, Latin abbreviations, short and long tier), with its baseline recorded in `docs/contracts.md` (agent-mem#62). Set format: `docs/contracts.md#recall-evaluation-f13`.
|
|
215
|
+
|
|
216
|
+
External retrieval measurements for MCP, library search and a plain BM25 baseline are recorded in [LongMemEval S/M](docs/bench/longmemeval.md). The full datasets stay outside the repository; CI runs only synthetic harness fixtures.
|
|
207
217
|
|
|
208
218
|
The R7 capacity advisor reports this machine's load and proposes capture bounds: `amem capacity [--json]` shows capture size and growth, disk, memory, the rolling `metrics.jsonl` readings and the proposal with its binding constraint; `amem capacity apply [--cap-mb N] [--retention-days N]` saves a preview, and only a matching later `--confirm` writes the home default config. Applying is person-only (agent markers refused, a terminal required); an agent may read the proposal through the library, never apply it. Under the 2026-10-17 owner decision (item 2) the proposal is the largest cap and retention that fit, and it may be above the current values: the cap is the disk-share bound itself (default 2% of free space, `--disk-share`), and retention is the share divided by the measured daily growth, up to a 365-day ceiling (a `ponytail:` constant, a person setting is the upgrade path). A hook p95 over 150 ms or hook RSS over 100 MB halves the proposed cap, and search RSS over 200 MB halves the retention. The reason names every binding constraint: `disk-share retention` or `retention ceiling`, `disk-share cap`, `R3 search RSS`, `R2 hook cost`. A proposal is never written by the advisor, and `apply` still takes `--cap-mb` and `--retention-days` to override it. Benchmark samples stay in the measuring script's synthetic home; only real hook and search invocations feed the advisor's rolling metrics.
|
|
209
219
|
|
|
@@ -282,7 +292,7 @@ Successful native project identities have a private, bounded derived cache (128
|
|
|
282
292
|
|
|
283
293
|
`amem doctor` prints one read-only report of this project's memory: duplicate and stale notes, pending candidates, capture freshness and hook registration for each CLI (a CLI with no hook is `manual`), the SessionStart brief's token cost, whether the index equals a rebuild of the logs, reader leases and the policy in effect. `amem doctor --json` is a stable machine-readable form (`amem.doctor/v1`). It writes nothing into the memory home (every check runs on a temporary copy of the logs) and creates nothing in a home you never used.
|
|
284
294
|
|
|
285
|
-
Intake runs at the end of each Codex, Kimi and
|
|
295
|
+
Intake runs at the end of each Codex, Kimi, Pi and Qwen session once `amem setup <cli> --apply` has registered it. Sync is off unless you ask for it: amem registers no sync hook and runs nothing between sessions. To sync on your own schedule, call `amem sync --if-due` from your shell (for example an EXIT trap in `~/.zshrc`; it needs a terminal). It syncs at most hourly, backs off 15 minutes doubling to a day after failures, stops after 30 seconds, and does nothing without a remote set by `amem sync --remote <private git url>`. `--force` skips the schedule and the backoff.
|
|
286
296
|
|
|
287
297
|
Split-home sync durably holds verified foreign-generation prunes with a "held, not applied" warning while unrelated sync continues; targets can remain visible, and full cross-generation removal remains a separate ceiling (see [the sync contract](docs/contracts.md#per-shard-private-sync-15-t3)).
|
|
288
298
|
|
|
@@ -298,6 +308,8 @@ amem ingest pi --since 2026-10-01T00:00:00Z --json
|
|
|
298
308
|
|
|
299
309
|
Discovery uses `CODEX_HOME` or `~/.codex` (`sessions/` and `archived_sessions/`), `KIMI_CODE_HOME` or `~/.kimi-code` (`sessions/**/wire.jsonl`), and `PI_CODING_AGENT_DIR` or `~/.pi/agent` (`sessions/`). It does not follow directory symlinks. A native record must carry a recognized schema/time and an observed absolute working-directory binding that resolves to this canonical project, including reviewed aliases. Unknown working directories, unsupported Kimi wire versions, missing timestamps and unavailable source roots are reported rather than guessed as an empty successful history. Foreign/private/secret rows are not imported. Full native envelopes and extracted capture inputs both pass current policy.
|
|
300
310
|
|
|
311
|
+
Qwen Code uses `QWEN_RUNTIME_DIR`, then `QWEN_HOME`, then `~/.qwen`, with bounded discovery of `projects/*/chats/*.jsonl`. Each row proves its own project through `cwd`; a sanitized directory name is not an identity. The reader supports the verified 0.26.1 native envelope, an internal format rather than a public schema, and excludes thought/media content. For `advanced.runtimeOutputDir`, export the matching `QWEN_RUNTIME_DIR`; amem does not reproduce merged Qwen settings or read `.env`. See the [Qwen capture contract](docs/qwen-code.md) for source pins, cursor upgrade behavior and limitations.
|
|
312
|
+
|
|
301
313
|
Per-source cursor databases and a fair rotation cursor live in the capture shard's private `ingest/` directory. Filenames and source identities are hashes; cursor contents retain offsets, anchor hashes, scope/protocol/cutoff and ownership metadata, not raw transcript paths or text. Capture receipts make replay idempotent across a crash between append and cursor acknowledgement. Repeating the command resumes its bounded tour instead of repeatedly favoring the first files. Broadening `--since` rechecks earlier history while existing capture receipts prevent duplicates. The JSON receipt reports `files`, `read`, `imported`, `skipped`, `spooled`, `remaining`, `errors` and an optional bounded `reason`; unavailable coverage is not silently represented by zero imported rows.
|
|
302
314
|
|
|
303
315
|
The default invocation reads at most 4 MiB and attempts 32 sources; trusted library options can raise those limits to 16 MiB and 128 sources. It examines at most 4,096 records, bounds a record at 256 KiB, and caps discovery at 4,096 directory entries/depth 8. Re-run a receipt with `remaining: true` after inspecting its reason; ingest creates no polling timer or watcher. Oversized records, including an unbroken record longer than the 4 MiB default window, are counted once and discarded through their next newline using a durable byte offset/skip flag. Later valid records remain reachable without retaining oversized content; an unfinished oversized tail stays pending until its newline arrives. The cursor assumes append-only native files and verifies file identity plus the last 1,024 acknowledged bytes. Historical rewrites outside that anchor are a declared ceiling, not full-file verification. These readers do not establish account-backed native interoperability, P5 cutover or P6 live-week acceptance.
|
package/package.json
CHANGED
package/src/cli.ts
CHANGED
|
@@ -43,6 +43,7 @@ import { appendGitattributes, appendRepoLine, boundRepoRoot, planGitattributes,
|
|
|
43
43
|
import type { Visibility } from "./memory/sync.ts";
|
|
44
44
|
import { applySplit, previewSplit } from "./memory/capacity-split.ts";
|
|
45
45
|
import { evalText, parseKs } from "./memory/eval.ts";
|
|
46
|
+
import { unrankedScoreDetails } from "./memory/search-score.ts";
|
|
46
47
|
import { builtinSet, evalSeeded, evalStore, FIXTURE_PATH, readEvalSet } from "./memory/eval-fixture.ts";
|
|
47
48
|
import { markShown, pendingSummary } from "./memory/candidate-pending.ts";
|
|
48
49
|
import { buildNote, FAIL_TEMPLATE, hasFailLabel, noteAdd, parseFailTemplate, type FailFields } from "./memory/note-add.ts";
|
|
@@ -168,9 +169,9 @@ export async function runCli(argv: readonly string[], context: CliContext = {}):
|
|
|
168
169
|
const parsed = parseCommand(argv);
|
|
169
170
|
if (parsed.options["--help"] === true) return commandHelp();
|
|
170
171
|
const kind = parsed.positionals[0];
|
|
171
|
-
if (parsed.positionals.length !== 1 || !["codex", "kimi", "pi"].includes(kind ?? "")) throw new Error("usage: amem ingest codex|kimi|pi [--since ISO] [--json]");
|
|
172
|
+
if (parsed.positionals.length !== 1 || !["codex", "kimi", "pi", "qwen"].includes(kind ?? "")) throw new Error("usage: amem ingest codex|kimi|pi|qwen [--since ISO] [--json]");
|
|
172
173
|
const { ingestTranscript } = await import("./ingest.ts");
|
|
173
|
-
return JSON.stringify(await ingestTranscript(kind as "codex" | "kimi" | "pi", {
|
|
174
|
+
return JSON.stringify(await ingestTranscript(kind as "codex" | "kimi" | "pi" | "qwen", {
|
|
174
175
|
cwd: context.cwd ?? process.cwd(), base: memoryHome(context.env ?? process.env),
|
|
175
176
|
...(typeof parsed.options["--since"] === "string" ? { since: parsed.options["--since"] as string } : {}),
|
|
176
177
|
}));
|
|
@@ -195,10 +196,10 @@ export async function runCli(argv: readonly string[], context: CliContext = {}):
|
|
|
195
196
|
}
|
|
196
197
|
const { definition, options, rawArgs: args } = parsed;
|
|
197
198
|
if (parsed.path[0] === "setup") {
|
|
198
|
-
if (parsed.positionals.length !== 1 || options["--apply"] === true && options["--undo"] === true) throw new Error("usage: amem setup claude|codex|kimi|pi|all [--apply|--undo]");
|
|
199
|
+
if (parsed.positionals.length !== 1 || options["--apply"] === true && options["--undo"] === true) throw new Error("usage: amem setup claude|codex|kimi|pi|qwen|jcode|agy|all [--apply|--undo]");
|
|
199
200
|
const { runSetup } = await import("./setup.ts");
|
|
200
201
|
return runSetup(parsed.positionals[0]!, options["--apply"] === true ? "apply" : options["--undo"] === true ? "undo" : "preview", {
|
|
201
|
-
env: context.env ?? process.env, stdinTTY: context.stdinTTY ?? process.stdin.isTTY,
|
|
202
|
+
env: context.env ?? process.env, stdinTTY: context.stdinTTY ?? process.stdin.isTTY, cwd: context.cwd ?? process.cwd(),
|
|
202
203
|
...(context.setupConfirm ? { confirm: context.setupConfirm } : {}),
|
|
203
204
|
});
|
|
204
205
|
}
|
|
@@ -573,7 +574,7 @@ export async function runCli(argv: readonly string[], context: CliContext = {}):
|
|
|
573
574
|
try {
|
|
574
575
|
const result = await importProject(store, { from: options["--from"], project: key, confirm: options["--confirm"] === true,
|
|
575
576
|
allowShared: options["--allow-shared"] === true, policy: note => policy.screenStandalone(note) });
|
|
576
|
-
const counts = `sources ${JSON.stringify(result.provenance.source)}, peers ${JSON.stringify(result.provenance.peer)}, scopes ${JSON.stringify(result.provenance.scope)}`;
|
|
577
|
+
const counts = `sources ${JSON.stringify(result.provenance.source)}, peers ${JSON.stringify(result.provenance.peer)}, scopes ${JSON.stringify(result.provenance.scope)}, machines ${JSON.stringify(result.provenance.machine)}`;
|
|
577
578
|
const summary = result.applied ? `imported ${result.added} of ${result.read} curated notes into ${result.project} with their original ids`
|
|
578
579
|
: `preview: ${result.wouldAdd} of ${result.read} curated notes can be imported into ${result.project} with their original ids (re-run with --confirm to write)`;
|
|
579
580
|
return options["--json"] ? JSON.stringify(result, null, 2)
|
|
@@ -589,6 +590,8 @@ export async function runCli(argv: readonly string[], context: CliContext = {}):
|
|
|
589
590
|
return options["--json"] ? JSON.stringify(written, null, 2) : `${op === "forget" ? "forgot" : op === "archive" ? "archived" : "unarchived"} ${written.key}${requested !== written.key ? ` (alias ${requested})` : ""}`;
|
|
590
591
|
}
|
|
591
592
|
if (sub === "search" && options["--all-projects"] === true) {
|
|
593
|
+
if (options["--explain"] === true) throw new Error("--explain applies to current-project ranked search, not --all-projects");
|
|
594
|
+
if (options["--filter"] === true) throw new Error("--filter applies to project-local search, not --all-projects");
|
|
592
595
|
const words = parsed.positionals.join(" ");
|
|
593
596
|
if (!words.trim()) throw new Error("usage: amem search --all-projects <words> [--limit N] [--include-archived] [--json]");
|
|
594
597
|
const result = searchAllProjects(memoryHome(), words, { ...(options["--limit"] !== undefined ? { limit: Number(options["--limit"]) } : {}), includeArchived: options["--include-archived"] === true });
|
|
@@ -597,6 +600,11 @@ export async function runCli(argv: readonly string[], context: CliContext = {}):
|
|
|
597
600
|
return [`crossed projects: searched ${result.projects.length} (${result.projects.join(", ")})`, ...(result.hits.length ? result.hits.map(one) : ["(nothing)"])].join("\n");
|
|
598
601
|
}
|
|
599
602
|
if (sub === "search" && options["--include-archived"] === true) throw new Error("--include-archived only applies with --all-projects");
|
|
603
|
+
if (sub === "search" && options["--explain"] === true) {
|
|
604
|
+
if (options["--filter"] === true) throw new Error("--explain cannot be combined with --filter");
|
|
605
|
+
if (options["--history"] === true || options["--as-of"] !== undefined) throw new Error("--explain cannot be combined with --history or --as-of");
|
|
606
|
+
if (!parsed.positionals.join(" ").trim()) throw new Error("--explain requires a non-empty search query");
|
|
607
|
+
}
|
|
600
608
|
if (sub === "alias") {
|
|
601
609
|
if (parsed.path[1] === "add") {
|
|
602
610
|
const [from, to] = parsed.positionals;
|
|
@@ -685,10 +693,12 @@ export async function runCli(argv: readonly string[], context: CliContext = {}):
|
|
|
685
693
|
const kind = options["--kind"], since = options["--since"], limit = options["--limit"] !== undefined ? Number(options["--limit"]) : 50;
|
|
686
694
|
const rows = read.views.filter(row => options["--tier"] !== "short" && (kind === undefined || row.kind === kind) && (typeof since !== "string" || Date.parse(row.at) >= Date.parse(since))
|
|
687
695
|
&& words.every(word => `${row.title ?? ""}\n${row.text}\n${row.about.join(" ")}`.toLowerCase().includes(word))).slice(0, limit);
|
|
688
|
-
|
|
696
|
+
const repoDetails = explainSearch ? rows.map(row => ({ ...row, scoreDetails: unrankedScoreDetails("repo", null) })) : rows;
|
|
697
|
+
if (options["--json"]) { try { const base = JSON.parse(output) as unknown[]; return JSON.stringify([...base, ...repoDetails], null, 2); } catch { return output; } }
|
|
689
698
|
const line = (row: RepoView) => `repo:${row.id.slice(0, 8)} ${row.at.slice(0, 16).replace("T", " ")} repo ${row.kind.padEnd(8)} ${`${row.title ? `${row.title}: ` : ""}${row.text}`.replace(/\s+/g, " ").slice(0, 100)}`;
|
|
690
699
|
const note = read.state === "unsafe" ? [`(repo scope unavailable: ${read.reason})`] : read.skipped.malformed + read.skipped.mismatched + read.skipped.withheld ? [`(${read.skipped.malformed + read.skipped.mismatched} repo lines skipped as invalid, ${read.skipped.withheld} withheld by the screen)`] : [];
|
|
691
|
-
|
|
700
|
+
const lines = explainSearch ? rows.flatMap(row => [line(row), `score: ${JSON.stringify(unrankedScoreDetails("repo", null))}`]) : rows.map(line);
|
|
701
|
+
return [output, ...(rows.length ? ["repo scope (.amem/log.jsonl):", ...lines] : []), ...note].join("\n");
|
|
692
702
|
};
|
|
693
703
|
const crosses = definition.policyFree === true;
|
|
694
704
|
const cwd = context.cwd ?? process.cwd();
|
|
@@ -731,11 +741,14 @@ export async function runCli(argv: readonly string[], context: CliContext = {}):
|
|
|
731
741
|
if (args[index] === "--capture-days") { captureArgs.add(index); captureArgs.add(index + 1); }
|
|
732
742
|
else if (args[index]?.startsWith("--capture-days=")) captureArgs.add(index);
|
|
733
743
|
}
|
|
734
|
-
|
|
744
|
+
// --filter and --explain select standalone CLI behavior; hosts keep their existing raw command arguments.
|
|
745
|
+
const legacySearch = options["--filter"] === true;
|
|
746
|
+
const explainSearch = sub === "search" && options["--explain"] === true;
|
|
747
|
+
const syncArgs = args.filter((arg, index) => arg !== "--if-due" && arg !== "--force" && !captureArgs.has(index) && (!legacySearch || arg !== "--filter") && (!explainSearch || arg !== "--explain"));
|
|
735
748
|
let output: string;
|
|
736
749
|
const stop = ifDue ? stopSignal(context.syncDeadlineMs ?? SYNC_IF_DUE_DEADLINE_MS, process) : undefined;
|
|
737
750
|
try {
|
|
738
|
-
output = await withRepoRows(await rawMemoryCommand(syncArgs, cwd, { person: true, project: boundProject(), open, readable: notes => notes, stale: createRecallStalenessProbe({ cwd: recallRoot ?? cwd, rootConfirmed, policy: () => configured }), outputReady: notes => outputReady(notes.map(view => view.id)), importPolicy: () => text => configured.declinesPersonal(text),
|
|
751
|
+
output = await withRepoRows(await rawMemoryCommand(syncArgs, cwd, { person: true, rankedSearch: !legacySearch, ...(explainSearch ? { explainSearch: true } : {}), project: boundProject(), open, readable: notes => notes, stale: createRecallStalenessProbe({ cwd: recallRoot ?? cwd, rootConfirmed, policy: () => configured }), outputReady: notes => outputReady(notes.map(view => view.id)), importPolicy: () => text => configured.declinesPersonal(text),
|
|
739
752
|
sync: { screen: row => configured.screenStandalone(row), ...(stop ? { signal: stop.signal } : {}) }, // the repository-narrowed policy binds what sync publishes too
|
|
740
753
|
onBridgeStatus: status => { receipt.bridgeStatus = status; } }));
|
|
741
754
|
} catch (error) {
|
package/src/commands-table.ts
CHANGED
|
@@ -25,20 +25,21 @@ export const GLOBAL_OPTIONS = { "--help": flag("Show help") };
|
|
|
25
25
|
export const ROOT_OPTIONS = { "--version": flag("Show version"), ...json };
|
|
26
26
|
export const COMMANDS: Record<string, CommandDefinition> = {
|
|
27
27
|
consolidate: { description: "Queue deterministic project-local consolidation candidates; curated notes change only after person review", mutates: true, options: { ...json, "--project": text("The current project's key; another project is refused (run from its directory)") } },
|
|
28
|
-
setup: { description: "Preview reversible MCP and
|
|
28
|
+
setup: { description: "Preview reversible MCP setup and supported capture hooks for native coding CLIs; apply and undo are person-only", options: { "--apply": flag("Apply after terminal confirmation"), "--undo": flag("Restore original configuration after confirmation") }, args: [{ name: "target", values: ["claude", "codex", "kimi", "pi", "qwen", "jcode", "agy", "all"] }] },
|
|
29
29
|
mcp: { description: "Run the project-bound MCP stdio server for this session" },
|
|
30
30
|
hook: { description: "Run a fail-open, context-only Claude Code hook", args: [{ name: "event", values: [...HOOK_EVENTS] }] },
|
|
31
|
-
ingest: { description: "Incrementally capture native session transcripts without a watcher", options: { ...json, "--since": text("ISO date or time") }, args: [{ name: "kind", values: ["codex", "kimi", "pi"] }] },
|
|
31
|
+
ingest: { description: "Incrementally capture native session transcripts without a watcher", options: { ...json, "--since": text("ISO date or time") }, args: [{ name: "kind", values: ["codex", "kimi", "pi", "qwen"] }] },
|
|
32
32
|
"vault-candidates": { description: "List screened durable curated vault candidates (per project, or across projects)", options: { ...json, "--limit": text("Maximum candidates per project (1..100)"),
|
|
33
33
|
"--all-projects": flag("List every active project's candidates; the reply says it crossed projects"), "--project": text("List one named project's candidates") } },
|
|
34
34
|
list: { description: "List memory", options: { ...filters, ...history } },
|
|
35
|
-
search: { description: "
|
|
35
|
+
search: { description: "Rank current curated project notes; history/as-of and empty queries retain newest-first filtering", options: { ...filters, ...history,
|
|
36
|
+
"--filter": flag("Current-project curated notes: match every word as a substring, newest first; not with --all-projects"), "--explain": flag("Include the score factors for ranked project-local results"), "--all-projects": flag("Search every active project; the reply says it crossed projects"),
|
|
36
37
|
"--include-archived": flag("With --all-projects, also read archived projects") }, args: [{ name: "words", variadic: true }] },
|
|
37
38
|
brief: { description: "Read intent/path memory: fails first, then decisions and contracts; untrusted data within a UTF-8 byte cap", options: { ...json,
|
|
38
39
|
"--for": text("Explicit intent"), "--paths": text("Repeatable named path"), "--symbols": text("Repeatable named symbol"), "--bytes": text("Complete brief-body byte cap (1..8192, default 2048)") } },
|
|
39
40
|
projects: { description: "List the project shards of this memory home (#15 D3)", options: { ...json, "--all": flag("Include forgotten projects (archived ones are always listed)") } },
|
|
40
41
|
usage: { description: "Totals and the largest project shards", options: { ...json, "--limit": text("Largest shards to list (1..100)"), "--all": flag("Include forgotten projects") } },
|
|
41
|
-
eval: { description: "Replay a local question set against
|
|
42
|
+
eval: { description: "Replay a local question set against ranked store.search and report recall@k and MRR (no model, no network; the built-in synthetic set without --set, `--set ko` for the built-in Korean one)", options: { ...json,
|
|
42
43
|
"--set": { description: "`ko` (the built-in Korean set) or a question set file (amem.eval/v1); with entries it is self-contained, without them its ids name entries of this project", value: "path" }, "--k": text("Cutoffs, comma separated (default 1,5,10; each 1..100)") } },
|
|
43
44
|
gc: { description: "Apply each shard's retention and cap across shards in bounded batches (preview without --confirm)", mutates: "confirm", options: {
|
|
44
45
|
...json, "--confirm": flag("Evict now; without it, preview only"), "--batches": text("evict() windows per shard in this run (1..1000, default 1)") } },
|
package/src/index.ts
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
export * from "./memory/store.ts";
|
|
2
|
+
export type { ScoreFactors } from "./memory/search-score.ts";
|
|
2
3
|
export * from "./memory/capture.ts";
|
|
3
4
|
export { fts5Available, ftsTerms, SearchIndex, captureSearch } from "./memory/search-fts.ts";
|
|
4
5
|
export type { CuratedDoc, RankedRef } from "./memory/search-fts.ts";
|
package/src/ingest.ts
CHANGED
|
@@ -2,7 +2,7 @@ import { Database } from "bun:sqlite";
|
|
|
2
2
|
import { createHash, randomUUID } from "node:crypto";
|
|
3
3
|
import { closeSync, chmodSync, constants, fchmodSync, fstatSync, lstatSync, mkdirSync, openSync, opendirSync, readSync, type Stats } from "node:fs";
|
|
4
4
|
import { homedir } from "node:os";
|
|
5
|
-
import { dirname, isAbsolute, join } from "node:path";
|
|
5
|
+
import { dirname, isAbsolute, join, resolve } from "node:path";
|
|
6
6
|
import { CaptureStore, type CaptureInput } from "./memory/capture.ts";
|
|
7
7
|
import { memoryHome } from "./memory/home.ts";
|
|
8
8
|
import { canonicalPath, processLiveness, processSignature, realPath } from "./memory/platform.ts";
|
|
@@ -11,7 +11,7 @@ import { configuredPolicyOptions } from "./memory/config.ts";
|
|
|
11
11
|
import { memoryProjectKey } from "./memory/project.ts";
|
|
12
12
|
import { effectiveAliases } from "./memory/shard.ts";
|
|
13
13
|
|
|
14
|
-
export type IngestKind = "codex" | "kimi" | "pi";
|
|
14
|
+
export type IngestKind = "codex" | "kimi" | "pi" | "qwen";
|
|
15
15
|
export interface IngestOptions {
|
|
16
16
|
cwd: string; base?: string;
|
|
17
17
|
/** Trusted host/test seam. The ordinary CLI discovers native roots instead. */
|
|
@@ -28,6 +28,8 @@ interface Position { identity: string; offset: number; anchor: string; scope: Sc
|
|
|
28
28
|
interface Cursor extends Position { token: string | null; pid: number | null; signature: string | null }
|
|
29
29
|
const MAX_BYTES = 16 * 1024 * 1024, DEFAULT_BYTES = 4 * 1024 * 1024, MAX_FILES = 128, DEFAULT_FILES = 32;
|
|
30
30
|
const MAX_CURSOR_BYTES = 128 * 1024;
|
|
31
|
+
// Bump after extending the verified Qwen decoder: replay skipped versions with existing capture receipts deduplicating accepted rows.
|
|
32
|
+
const QWEN_CONTRACT = "qwen-0.26.1-v1";
|
|
31
33
|
const utf8 = new TextDecoder("utf-8", { fatal: true });
|
|
32
34
|
const MAX_RECORD = 256 * 1024, MAX_RECORDS = 4096, MAX_DISCOVERY = 4096, MAX_DEPTH = 8, ANCHOR_BYTES = 1024;
|
|
33
35
|
const hash = (value: string | Uint8Array) => createHash("sha256").update(value).digest("hex");
|
|
@@ -115,6 +117,7 @@ function lease(file: string): { position: Position; save(value: Position): void;
|
|
|
115
117
|
|
|
116
118
|
/** Native roots only. Discovery never follows a directory symlink or enters an unbounded tree. */
|
|
117
119
|
function discover(kind: IngestKind): { files: string[]; bounded: boolean; unavailable: boolean } {
|
|
120
|
+
if (kind === "qwen") return discoverQwen();
|
|
118
121
|
const root = kind === "codex" ? process.env.CODEX_HOME?.trim() || join(homedir(), ".codex")
|
|
119
122
|
: kind === "kimi" ? process.env.KIMI_CODE_HOME?.trim() || join(homedir(), ".kimi-code")
|
|
120
123
|
: process.env.PI_CODING_AGENT_DIR?.trim() || join(homedir(), ".pi", "agent");
|
|
@@ -139,6 +142,38 @@ function discover(kind: IngestKind): { files: string[]; bounded: boolean; unavai
|
|
|
139
142
|
return { files: files.sort(), bounded, unavailable: rootsRead === 0 };
|
|
140
143
|
}
|
|
141
144
|
|
|
145
|
+
/** Only the native projects/<one directory>/chats/*.jsonl layout, never settings or account files. */
|
|
146
|
+
function discoverQwen(): { files: string[]; bounded: boolean; unavailable: boolean } {
|
|
147
|
+
const given = process.env.QWEN_RUNTIME_DIR || process.env.QWEN_HOME || join(homedir(), ".qwen");
|
|
148
|
+
const expanded = given === "~" ? homedir() : given.startsWith("~/") || given.startsWith("~\\") ? join(homedir(), ...given.slice(2).split(/[/\\]+/)) : given;
|
|
149
|
+
const root = resolve(expanded);
|
|
150
|
+
const files: string[] = []; let visited = 0, bounded = false;
|
|
151
|
+
const open = (folder: string) => {
|
|
152
|
+
const stat = lstatSync(folder);
|
|
153
|
+
if (!stat.isDirectory() || stat.isSymbolicLink()) throw Error("native directory unavailable");
|
|
154
|
+
return opendirSync(folder);
|
|
155
|
+
};
|
|
156
|
+
let projects: ReturnType<typeof opendirSync>;
|
|
157
|
+
try { if (lstatSync(root).isSymbolicLink()) throw Error("native directory unavailable"); projects = open(join(root, "projects")); }
|
|
158
|
+
catch { return { files, bounded, unavailable: true }; }
|
|
159
|
+
try {
|
|
160
|
+
for (let project = projects.readSync(); project; project = projects.readSync()) {
|
|
161
|
+
if (++visited > MAX_DISCOVERY) { bounded = true; break; }
|
|
162
|
+
if (!project.isDirectory() || project.isSymbolicLink()) continue;
|
|
163
|
+
let chats: ReturnType<typeof opendirSync>;
|
|
164
|
+
try { chats = open(join(root, "projects", project.name, "chats")); } catch { continue; }
|
|
165
|
+
try {
|
|
166
|
+
for (let item = chats.readSync(); item; item = chats.readSync()) {
|
|
167
|
+
if (++visited > MAX_DISCOVERY) { bounded = true; break; }
|
|
168
|
+
if (item.isFile() && !item.isSymbolicLink() && item.name.endsWith(".jsonl")) files.push(join(root, "projects", project.name, "chats", item.name));
|
|
169
|
+
}
|
|
170
|
+
} finally { chats.closeSync(); }
|
|
171
|
+
if (bounded) break;
|
|
172
|
+
}
|
|
173
|
+
} finally { projects.closeSync(); }
|
|
174
|
+
return { files: files.sort(), bounded, unavailable: false };
|
|
175
|
+
}
|
|
176
|
+
|
|
142
177
|
function timeOf(record: Record<string, any>, kind: IngestKind): string | undefined {
|
|
143
178
|
const value = kind === "kimi" ? record.time : record.timestamp;
|
|
144
179
|
const millis = typeof value === "number" && kind === "kimi" && Number.isFinite(value) && value >= 100_000_000_000 ? value
|
|
@@ -151,7 +186,55 @@ function contentText(content: unknown): string {
|
|
|
151
186
|
if (!Array.isArray(content)) return "";
|
|
152
187
|
return content.filter(item => object(item) && ["text", "input_text", "output_text"].includes(item.type) && typeof item.text === "string").map(item => item.text).join("\n");
|
|
153
188
|
}
|
|
189
|
+
const qwenUuid = (value: unknown): boolean => typeof value === "string" && /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(value);
|
|
190
|
+
const qwenSession = (value: unknown): boolean => typeof value === "string" && /^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}(?:-agent-[a-zA-Z0-9_.-]+)?$/i.test(value);
|
|
191
|
+
function qwenPart(part: unknown): boolean {
|
|
192
|
+
if (!object(part)) return false;
|
|
193
|
+
if (part.thought === true) return true;
|
|
194
|
+
if (Object.keys(part).some(key => !["text", "thought", "thoughtSignature", "functionCall", "functionResponse"].includes(key))) return false;
|
|
195
|
+
if (part.thought !== undefined && typeof part.thought !== "boolean" || part.thoughtSignature !== undefined && typeof part.thoughtSignature !== "string") return false;
|
|
196
|
+
const forms = ["text", "functionCall", "functionResponse"].filter(key => part[key] !== undefined);
|
|
197
|
+
if (forms.length !== 1) return false;
|
|
198
|
+
if (forms[0] === "text") return typeof part.text === "string";
|
|
199
|
+
const value = part[forms[0]!];
|
|
200
|
+
return object(value) && typeof value.name === "string" && value.name.length > 0
|
|
201
|
+
&& (value.id === undefined || typeof value.id === "string")
|
|
202
|
+
&& (forms[0] === "functionCall" ? value.args === undefined || object(value.args)
|
|
203
|
+
: object(value.response) && (value.parts === undefined || Array.isArray(value.parts)));
|
|
204
|
+
}
|
|
205
|
+
function qwenRecord(raw: Record<string, any>): boolean {
|
|
206
|
+
// ponytail: CLI version is not a schema tag. Support only the 0.26.1 writer shape verified at
|
|
207
|
+
// QwenLM/qwen-code 1ce173117196ab9a9f77b5d19ac148cbabe25dfd; extend with native fixtures, not version guessing.
|
|
208
|
+
if (raw.version !== "0.26.1" || !qwenUuid(raw.uuid) || !qwenSession(raw.sessionId)
|
|
209
|
+
|| raw.parentUuid !== null && !qwenUuid(raw.parentUuid) || !["user", "assistant", "tool_result", "system"].includes(raw.type)
|
|
210
|
+
|| typeof raw.cwd !== "string" || !isAbsolute(raw.cwd) || !timeOf(raw, "qwen")) return false;
|
|
211
|
+
if (raw.type === "system" || raw.message === undefined) return true;
|
|
212
|
+
return object(raw.message) && raw.message.role === (raw.type === "assistant" ? "model" : "user")
|
|
213
|
+
&& Array.isArray(raw.message.parts) && raw.message.parts.every(qwenPart);
|
|
214
|
+
}
|
|
215
|
+
function extractQwen(raw: Record<string, any>, project: string): CaptureInput | undefined {
|
|
216
|
+
if (raw.type === "system" || !object(raw.message)) return undefined;
|
|
217
|
+
const excerpts: string[] = [], tools: string[] = [], events: string[] = [];
|
|
218
|
+
for (const part of raw.message.parts) {
|
|
219
|
+
if (part.thought === true) continue;
|
|
220
|
+
if (typeof part.text === "string") { if (part.text) { excerpts.push(part.text); events.push("message"); } }
|
|
221
|
+
else if (object(part.functionCall)) {
|
|
222
|
+
const call = part.functionCall;
|
|
223
|
+
excerpts.push(JSON.stringify({ functionCall: { name: call.name, ...(call.id !== undefined ? { id: call.id } : {}), ...(call.args !== undefined ? { args: call.args } : {}) } }));
|
|
224
|
+
tools.push(call.name); events.push("function_call");
|
|
225
|
+
} else if (object(part.functionResponse)) {
|
|
226
|
+
const response = part.functionResponse;
|
|
227
|
+
// FunctionResponse.parts contains media, not textual tool output. Never stringify it into capture.
|
|
228
|
+
excerpts.push(JSON.stringify({ functionResponse: { name: response.name, ...(response.id !== undefined ? { id: response.id } : {}), response: response.response } }));
|
|
229
|
+
tools.push(response.name); events.push("function_call_output");
|
|
230
|
+
}
|
|
231
|
+
}
|
|
232
|
+
if (!excerpts.length) return undefined;
|
|
233
|
+
return { project, agent: "qwen", event: events.every(event => event === events[0]) ? events[0]! : "message", excerpt: excerpts.join("\n").slice(0, 2000),
|
|
234
|
+
...(tools.length && tools.every(tool => tool === tools[0]) ? { tool: tools[0]!.slice(0, 64) } : {}) };
|
|
235
|
+
}
|
|
154
236
|
function extract(kind: IngestKind, raw: Record<string, any>, project: string): CaptureInput | undefined {
|
|
237
|
+
if (kind === "qwen") return extractQwen(raw, project);
|
|
155
238
|
const message = kind === "codex" && raw.type === "response_item" ? raw.payload
|
|
156
239
|
: kind === "pi" && raw.type === "message" ? raw.message
|
|
157
240
|
: kind === "kimi" && raw.type === "context.append_message" ? raw.message : undefined;
|
|
@@ -171,7 +254,10 @@ function extract(kind: IngestKind, raw: Record<string, any>, project: string): C
|
|
|
171
254
|
/**
|
|
172
255
|
* Incremental deterministic native-message ingestion; no watcher, account, model or network operation.
|
|
173
256
|
* Native source contracts: Codex rust-v0.163.0 protocol RolloutItem; Pi session-manager v3;
|
|
174
|
-
* Kimi b5e28fd6 human/persist/v2/wire.ts (protocol 1.5) and sessionIndexSource.recoverCwd
|
|
257
|
+
* Kimi b5e28fd6 human/persist/v2/wire.ts (protocol 1.5) and sessionIndexSource.recoverCwd;
|
|
258
|
+
* Qwen 1ce173117196ab9a9f77b5d19ac148cbabe25dfd ChatRecord and config/storage.ts.
|
|
259
|
+
* Qwen discovery uses exported environment roots, not merged settings or .env files. For a configured
|
|
260
|
+
* advanced.runtimeOutputDir, export its resolved path as QWEN_RUNTIME_DIR or pass a trusted native file.
|
|
175
261
|
* ponytail: discovery covers at most 4096 directory entries and depth 8. A larger native tree reports
|
|
176
262
|
* remaining/source_bound; a future native index cursor can extend coverage without a resident watcher.
|
|
177
263
|
* Within that bound a durable hash-only rotation admits at most maxFiles source attempts per call, including
|
|
@@ -185,7 +271,7 @@ export async function ingestTranscript(kind: IngestKind, options: IngestOptions)
|
|
|
185
271
|
const result: IngestReceipt = { files: 0, read: 0, imported: 0, skipped: 0, spooled: 0, remaining: false, errors: 0 };
|
|
186
272
|
const fail = (reason: NonNullable<IngestReceipt["reason"]>, remaining = false) => { result.errors++; result.reason ??= reason; result.remaining ||= remaining; };
|
|
187
273
|
const bytes = options.maxBytes ?? DEFAULT_BYTES, count = options.maxFiles ?? DEFAULT_FILES;
|
|
188
|
-
if (!["codex", "kimi", "pi"].includes(kind) || !Number.isSafeInteger(bytes) || bytes < 1 || bytes > MAX_BYTES || !Number.isSafeInteger(count) || count < 1 || count > MAX_FILES
|
|
274
|
+
if (!["codex", "kimi", "pi", "qwen"].includes(kind) || !Number.isSafeInteger(bytes) || bytes < 1 || bytes > MAX_BYTES || !Number.isSafeInteger(count) || count < 1 || count > MAX_FILES
|
|
189
275
|
|| options.files !== undefined && (!Array.isArray(options.files) || options.files.length > MAX_FILES || options.files.some(file => typeof file !== "string" || !isAbsolute(file) || file.includes("\0")))
|
|
190
276
|
|| options.since !== undefined && !Number.isFinite(Date.parse(options.since))) { fail("source_bound"); return result; }
|
|
191
277
|
const since = options.since === undefined ? undefined : new Date(options.since).toISOString();
|
|
@@ -238,7 +324,9 @@ export async function ingestTranscript(kind: IngestKind, options: IngestOptions)
|
|
|
238
324
|
if (!cursor) { fail("cursor_busy", true); continue; }
|
|
239
325
|
let position = cursor.position;
|
|
240
326
|
const cutoff = since ?? "", broader = position.cutoff !== "" && (cutoff === "" || cutoff < position.cutoff);
|
|
241
|
-
if (position.identity !== identity || position.offset > initial.size || position.anchor !== anchor(fd, position.offset) || broader
|
|
327
|
+
if (position.identity !== identity || position.offset > initial.size || position.anchor !== anchor(fd, position.offset) || broader
|
|
328
|
+
|| kind === "qwen" && position.protocol !== QWEN_CONTRACT) position = { ...blank(), identity };
|
|
329
|
+
if (kind === "qwen") position.protocol = QWEN_CONTRACT;
|
|
242
330
|
position.cutoff = cutoff;
|
|
243
331
|
if (!position.anchor) position.anchor = anchor(fd, position.offset);
|
|
244
332
|
cursor.save(position); result.files++;
|
|
@@ -299,12 +387,15 @@ export async function ingestTranscript(kind: IngestKind, options: IngestOptions)
|
|
|
299
387
|
: kind === "pi" && raw.type === "session" ? raw : undefined;
|
|
300
388
|
const permitted = await policy.screen(raw);
|
|
301
389
|
guard(); metadataGuard();
|
|
302
|
-
|
|
390
|
+
const known = kind !== "qwen" || qwenRecord(raw);
|
|
391
|
+
if (kind === "qwen") position.scope = permitted && known ? scopeOf(raw.cwd) : "unknown";
|
|
392
|
+
else if (context !== undefined) position.scope = permitted && object(context) ? scopeOf(context.cwd) : "unknown";
|
|
303
393
|
else if (raw.cwd !== undefined) position.scope = permitted ? scopeOf(raw.cwd) : "unknown";
|
|
304
394
|
if (kind === "kimi" && raw.type === "metadata") {
|
|
305
395
|
position.protocol = typeof raw.protocol_version === "string" && ["1.0", "1.1", "1.2", "1.3", "1.4", "1.5"].includes(raw.protocol_version) ? raw.protocol_version : "unknown";
|
|
306
396
|
}
|
|
307
|
-
|
|
397
|
+
if (!known) fail("schema_or_time");
|
|
398
|
+
const input = known ? extract(kind, raw, project) : undefined;
|
|
308
399
|
if (input) {
|
|
309
400
|
const at = timeOf(raw, kind);
|
|
310
401
|
if (!at || position.scope === "unknown" || kind === "kimi" && !/^1\.[0-5]$/.test(position.protocol)) { fail("schema_or_time"); result.skipped++; }
|
package/src/mcp-memory.ts
CHANGED
|
@@ -10,7 +10,7 @@ import { join } from "node:path";
|
|
|
10
10
|
import type { McpHandlers, McpOptions } from "./mcp.ts";
|
|
11
11
|
import { CandidateStore, acceptedCorrectionText, candidateTargetRevision, type CandidateInput } from "./memory/candidates.ts";
|
|
12
12
|
import { capacityReport } from "./memory/capacity.ts";
|
|
13
|
-
import { CaptureStore } from "./memory/capture.ts";
|
|
13
|
+
import { CaptureStore, type CaptureRow } from "./memory/capture.ts";
|
|
14
14
|
import { hashJSON, readSelector } from "./memory/generation.ts";
|
|
15
15
|
import { memoryHome } from "./memory/home.ts";
|
|
16
16
|
import { canonicalPath, realPath } from "./memory/platform.ts";
|
|
@@ -18,7 +18,7 @@ import { createStandalonePolicy, type StandalonePolicy, type StandalonePolicyOpt
|
|
|
18
18
|
import { configuredPolicyOptions } from "./memory/config.ts";
|
|
19
19
|
import { memoryProjectKey } from "./memory/project.ts";
|
|
20
20
|
import { registerReaderLease } from "./memory/reader-lease.ts";
|
|
21
|
-
import { type ReadRef, type ReadWindow, SearchIndex } from "./memory/search-fts.ts";
|
|
21
|
+
import { type ReadRef, type ReadWindow, SearchIndex, ScreenedSearchIndex } from "./memory/search-fts.ts";
|
|
22
22
|
import { aliasedKeys, readShardedSelection, resolveAlias } from "./memory/shard.ts";
|
|
23
23
|
import { projectState } from "./memory/lifecycle.ts";
|
|
24
24
|
import { boundRepoRoot, readRepoScope, repoReadPolicy, type RepoView } from "./memory/repo-scope.ts";
|
|
@@ -28,9 +28,10 @@ import { WorkListStore, type WorkListFields } from "./memory/work-lists.ts";
|
|
|
28
28
|
|
|
29
29
|
// Graph retains its complete-corpus ceiling; search/timeline/show page matching bodies instead.
|
|
30
30
|
export const MCP_SCAN_LIMIT = 5000;
|
|
31
|
-
// ponytail:
|
|
31
|
+
// ponytail: evidence still ranks its newest 5000 permitted candidates; upgrade to the screened search stream if evidence needs global ranking.
|
|
32
32
|
const SEARCH_CANDIDATES = 5000;
|
|
33
33
|
const READ_PAGE = 256;
|
|
34
|
+
type SearchValue = { source: "curated"; evidence: true; view: View } | { source: "repo"; evidence: false; view: RepoView } | { source: "capture"; evidence: false; capture: CaptureRow };
|
|
34
35
|
export interface MemoryMcpOptions { cwd: string; base?: string; project?: string; policy?: StandalonePolicyOptions; allowed?: () => boolean; policyVersion?: () => string; assertCurrent?: () => void }
|
|
35
36
|
export interface MemoryMcp { tools: McpHandlers; close(): void; onEmitted: NonNullable<McpOptions["onEmitted"]> }
|
|
36
37
|
const unavailable = (): never => { throw new Error("memory operation unavailable"); };
|
|
@@ -109,19 +110,25 @@ export function createMemoryMcp(options: MemoryMcpOptions): MemoryMcp {
|
|
|
109
110
|
if (rows.length > MCP_SCAN_LIMIT) unavailable();
|
|
110
111
|
return screen(rows.filter(row => row.project === project && recallViewIsLive(row, store.readNow())));
|
|
111
112
|
};
|
|
112
|
-
const
|
|
113
|
-
options: Omit<ReadWindow, "project" | "limit" | "after">,
|
|
114
|
-
|
|
115
|
-
|
|
113
|
+
const permittedRows = async function* <T>(read: (window: ReadWindow) => { rows: T[]; next?: ReadRef; engine?: "fts5" | "substring" },
|
|
114
|
+
options: Omit<ReadWindow, "project" | "limit" | "after">, matches: (row: T, engine?: "fts5" | "substring") => boolean = () => true): AsyncGenerator<T> {
|
|
115
|
+
let after: ReadWindow["after"];
|
|
116
|
+
for (;;) {
|
|
116
117
|
guard();
|
|
117
118
|
const page = read({ ...options, project, limit: READ_PAGE, after });
|
|
118
|
-
for (const row of page.rows)
|
|
119
|
-
|
|
119
|
+
for (const row of page.rows) {
|
|
120
|
+
const allowed = await currentPolicy().screen(row); guard();
|
|
121
|
+
if (allowed && matches(row, page.engine)) yield row;
|
|
120
122
|
}
|
|
121
123
|
guard();
|
|
122
124
|
if (!page.next) break;
|
|
123
125
|
after = { position: page.next.position, id: page.next.id };
|
|
124
126
|
}
|
|
127
|
+
};
|
|
128
|
+
const readPermitted = async <T>(read: (window: ReadWindow) => { rows: T[]; next?: ReadRef; engine?: "fts5" | "substring" },
|
|
129
|
+
options: Omit<ReadWindow, "project" | "limit" | "after">, cap: number, matches: (row: T, engine?: "fts5" | "substring") => boolean = () => true): Promise<T[]> => {
|
|
130
|
+
const result: T[] = [];
|
|
131
|
+
for await (const row of permittedRows(read, options, matches)) { result.push(row); if (result.length === cap) break; }
|
|
125
132
|
return result;
|
|
126
133
|
};
|
|
127
134
|
const anchorAt = async (id: string): Promise<string | undefined> => {
|
|
@@ -180,41 +187,46 @@ export function createMemoryMcp(options: MemoryMcpOptions): MemoryMcp {
|
|
|
180
187
|
} finally { index.close(); }
|
|
181
188
|
},
|
|
182
189
|
async amem_search(args) {
|
|
183
|
-
bound(args, ["query", "limit", "since", "includeCapture"]); const query = textOf(args, "query")!, limit = limitOf(args), since = sinceOf(args);
|
|
190
|
+
bound(args, ["query", "limit", "since", "includeCapture", "explain"]); const query = textOf(args, "query")!, limit = limitOf(args), since = sinceOf(args);
|
|
184
191
|
if (args.includeCapture !== undefined && typeof args.includeCapture !== "boolean") unavailable();
|
|
192
|
+
if (args.explain !== undefined && typeof args.explain !== "boolean") unavailable();
|
|
185
193
|
if (!await currentPolicy().screen({ query })) unavailable();
|
|
186
|
-
|
|
194
|
+
// ponytail: scan all matches into private disposable disk state, keeping one body page in JS.
|
|
195
|
+
// Work scales with matches; a policy-bound cache is the upgrade if this reaches the unchanged watchdog.
|
|
196
|
+
const index = new ScreenedSearchIndex<SearchValue>();
|
|
197
|
+
const releaseIndex = () => index.close(); closers.push(releaseIndex);
|
|
187
198
|
try {
|
|
188
199
|
const words = query.toLowerCase().split(/\s+/).filter(Boolean);
|
|
189
200
|
const captureState: { engine?: "fts5" | "substring" } = {};
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
201
|
+
let batch: { body: string; ref: string; tier: string; at: string; value: SearchValue; expiresAt?: string | null }[] = [];
|
|
202
|
+
const append = (value: SearchValue) => {
|
|
203
|
+
const row = value.source === "capture" ? value.capture : value.view;
|
|
204
|
+
batch.push(value.source === "capture"
|
|
205
|
+
? { value, ref: `capture:${value.capture.id}`, at: row.at, tier: "capture", body: `${value.capture.excerpt}\n${value.capture.paths.join(" ")}\n${value.capture.event}` }
|
|
206
|
+
: { value, ref: value.source === "repo" ? `repo:${value.view.id}` : value.view.id, at: row.at, tier: value.source === "repo" ? "long" : value.view.tier, body: `${value.view.title ?? ""}\n${value.view.text}\n${value.view.about.join(" ")}`, expiresAt: value.source === "curated" ? value.view.expiresAt : null });
|
|
207
|
+
if (batch.length === READ_PAGE) { index.append(batch); batch = []; }
|
|
208
|
+
};
|
|
209
|
+
if (args.includeCapture !== false) {
|
|
210
|
+
for await (const row of permittedRows(window => {
|
|
211
|
+
const page = capture.readPage({ ...window, query, ...(captureState.engine === "substring" ? { match: "substring" as const } : {}) });
|
|
212
|
+
captureState.engine ??= page.engine; return page;
|
|
213
|
+
}, { since, order: "id" }, (row, engine) => index.available && engine === "fts5" || words.every(word => `${row.excerpt}\n${row.paths.join(" ")}\n${row.event}`.toLowerCase().includes(word)))) {
|
|
214
|
+
append({ source: "capture", evidence: false, capture: row });
|
|
215
|
+
}
|
|
216
|
+
}
|
|
196
217
|
const useFts = index.available && captureState.engine !== "substring";
|
|
197
|
-
const
|
|
198
|
-
view => recallViewIsLive(view, store.readNow()) && (useFts || words.every(word => `${view.title ?? ""}\n${view.text}\n${view.about.join(" ")}`.toLowerCase().includes(word))));
|
|
218
|
+
for await (const view of permittedRows(window => store.readPage({ ...window, ...(useFts ? { query } : {}) }), { since, order: "id" },
|
|
219
|
+
view => recallViewIsLive(view, store.readNow()) && (useFts || words.every(word => `${view.title ?? ""}\n${view.text}\n${view.about.join(" ")}`.toLowerCase().includes(word))))) append({ source: "curated", evidence: true, view });
|
|
199
220
|
const repoNotes = (await repoRows()).filter(view => (!since || Date.parse(view.at) >= Date.parse(since)) && (useFts || words.every(word => `${view.title ?? ""}\n${view.text}\n${view.about.join(" ")}`.toLowerCase().includes(word))));
|
|
200
|
-
const
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
}
|
|
207
|
-
const ranks = new Map((useFts ? index.query(query, MCP_SCAN_LIMIT * 2, project) : []).map(row => [row.ref, row.rank]));
|
|
208
|
-
const selected = hits.flatMap(hit => {
|
|
209
|
-
const rank = ranks.get(hit.ref);
|
|
210
|
-
if (useFts ? rank === undefined : !words.every(word => hit.body.toLowerCase().includes(word))) return [];
|
|
211
|
-
const weight = hit.tier === "long" ? 1 : hit.tier === "capture" ? 0.5 : 0.9;
|
|
212
|
-
const score = -(rank ?? -1) * weight / (1 + Math.max(0, Date.now() - Date.parse(hit.at)) / (30 * 86400_000));
|
|
213
|
-
return [{ source: hit.source, evidence: hit.evidence, ...(hit.source === "capture" ? { capture: hit.capture } : { view: hit.view }), score }];
|
|
214
|
-
}).sort((a, b) => b.score - a.score).slice(0, limit);
|
|
221
|
+
for (const view of repoNotes) append({ source: "repo", evidence: false, view });
|
|
222
|
+
if (batch.length) index.append(batch);
|
|
223
|
+
const selected = index.best(query, limit, store.readNow(), useFts, args.explain === true).map(({ value, score, scoreDetails }) => {
|
|
224
|
+
const hit = { ...value, score };
|
|
225
|
+
return scoreDetails ? { ...hit, scoreDetails: { source: value.source, ...scoreDetails } } : hit;
|
|
226
|
+
});
|
|
215
227
|
const probe = stale(), result = { engine: useFts ? "fts5" : "substring", hits: selected.map(hit => hit.source === "curated" && "view" in hit && hit.view ? { ...hit, view: decorate(hit.view as View, probe) } : hit) };
|
|
216
228
|
guard(); return observed(result, selected.flatMap(hit => hit.source === "curated" && "view" in hit && hit.view ? [hit.view.id] : []), selected.map(hit => "view" in hit ? hit.view : hit.capture));
|
|
217
|
-
} finally {
|
|
229
|
+
} finally { const position = closers.indexOf(releaseIndex); if (position >= 0) closers.splice(position, 1); releaseIndex(); }
|
|
218
230
|
},
|
|
219
231
|
async amem_show(args) {
|
|
220
232
|
bound(args, ["id"]); guard(); const id = textOf(args, "id")!;
|
package/src/mcp.ts
CHANGED
|
@@ -9,7 +9,7 @@ const tool = <Name extends string>(name: Name, description: string, inputSchema:
|
|
|
9
9
|
|
|
10
10
|
/** Storage scope comes from startup, never tool arguments; candidate proposals remain person-reviewed. */
|
|
11
11
|
export const MCP_TOOLS = [
|
|
12
|
-
tool("amem_search", "Search screened memory in this project. Capture results are recall, not curated evidence.", object({ query: string(1000), limit, since: string(40), includeCapture: { type: "boolean" } }, ["query"])),
|
|
12
|
+
tool("amem_search", "Search screened memory in this project. Capture results are recall, not curated evidence.", object({ query: string(1000), limit, since: string(40), includeCapture: { type: "boolean" }, explain: { type: "boolean" } }, ["query"])),
|
|
13
13
|
tool("amem_evidence", "Return a bounded bundle of cited, live curated entries for a question. Untrusted memory data only: no answer or model call. Capture, repo-only notes, pending candidates and model answers are not evidence.", object({ question: string(1000), limit: { type: "integer", minimum: 1, maximum: 20 }, maxBytes: { type: "integer", minimum: 256, maximum: 8192 } }, ["question"])),
|
|
14
14
|
tool("amem_show", "Show a screened memory entry in this project by its exact id or unique prefix.", object({ id: string(80) }, ["id"])),
|
|
15
15
|
tool("amem_timeline", "Read screened memory near an entry or time in this project.", object({ id: string(80), since: string(40), limit })),
|