@staix/agent-mem 0.8.0 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +1 -1
- package/README.md +19 -7
- package/package.json +1 -1
- package/src/cli.ts +23 -12
- package/src/commands-table.ts +7 -6
- package/src/ingest.ts +98 -7
- package/src/mcp-memory.ts +42 -34
- package/src/memory/capture.ts +25 -7
- package/src/memory/commands.ts +13 -4
- package/src/memory/eval.ts +2 -2
- package/src/memory/import.ts +91 -0
- package/src/memory/platform.ts +5 -1
- package/src/memory/policy.ts +88 -9
- package/src/memory/search-fts.ts +59 -3
- package/src/memory/shard-sync-format.ts +3 -4
- package/src/memory/shard-sync.ts +135 -17
- package/src/session-brief-worker.ts +5 -1
- package/src/setup-agy.ts +9 -0
- package/src/setup-jcode.ts +7 -0
- package/src/setup-json.ts +151 -0
- package/src/setup-owned.ts +40 -19
- package/src/setup.ts +147 -34
package/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Curated memory with append-only JSONL logs, a derived SQLite index, short- and long-term tiers, a typed graph, goals with leases, and git sync between machines. Extracted from [agent-hub](https://github.com/STAIxBWLB/agent-hub), with compatible storage and entry IDs.
|
|
4
4
|
|
|
5
|
-
Version: 0.
|
|
5
|
+
Version: 0.10.0. Requires Bun 1.4.2 or later; no runtime dependencies.
|
|
6
6
|
|
|
7
7
|
## Quick start
|
|
8
8
|
|
|
@@ -15,7 +15,7 @@ amem setup all
|
|
|
15
15
|
amem setup claude --apply
|
|
16
16
|
```
|
|
17
17
|
|
|
18
|
-
Replace `claude` with `codex`, `kimi` or `
|
|
18
|
+
Replace `claude` with `codex`, `kimi`, `pi`, `qwen` or `jcode` for that client. Setup shows its changes and asks for confirmation. Complete the client's own hook-trust review when prompted.
|
|
19
19
|
|
|
20
20
|
Agents submit pending memory candidates through MCP. Inspect them after a session and accept or reject each one:
|
|
21
21
|
|
|
@@ -26,12 +26,18 @@ amem review accept CANDIDATE_ID # or: amem review reject CANDIDATE_ID
|
|
|
26
26
|
|
|
27
27
|
Replace `CANDIDATE_ID` with an id from the list. Acceptance is person-only and applies the configured privacy policy.
|
|
28
28
|
|
|
29
|
-
Claude's registered hooks capture events during the session. The configured Codex/Kimi SessionEnd hooks and Pi shutdown extension ingest transcript tails incrementally at session end. If those integrations are unavailable, run `amem ingest codex` (or `kimi` / `pi`) after each session, then repeat the review loop. The [setup section](#setup--plugin) describes the registration and native trust steps.
|
|
29
|
+
Claude's registered hooks capture events during the session. The configured Codex/Kimi/Qwen SessionEnd hooks and Pi shutdown extension ingest transcript tails incrementally at session end. If those integrations are unavailable, run `amem ingest codex` (or `kimi` / `pi` / `qwen`) after each session, then repeat the review loop. The [setup section](#setup--plugin) describes the registration and native trust steps.
|
|
30
30
|
|
|
31
31
|
From a checkout, install development dependencies with `bun install` and run `bun src/cli.ts --help`.
|
|
32
32
|
|
|
33
33
|
## Setup / plugin
|
|
34
34
|
|
|
35
|
+
`amem setup agy` registers MCP only in the current git repository's `.agents/mcp_config.json`, with an explicit absolute repository cwd and a private receipt per repository. `setup all` excludes agy; run it separately in each repository. Setup refuses tracked config files and JSONC, and warns when the file is not ignored. Restart Antigravity after applying, use one workspace per session, and ensure no global, plugin or running server is also named `amem`. No global configuration, ingest command or hook is installed. Undo removes `.agents` only when setup created it and it is still empty; pre-existing or nonempty directories remain. Older receipts still restore their configuration without directory cleanup. See the [Antigravity contract and capture limitation](docs/antigravity.md).
|
|
36
|
+
|
|
37
|
+
`amem setup jcode` registers MCP only, using Jcode v0.99.1's global `JCODE_HOME/mcp.json` (default `~/.jcode/mcp.json`). An explicitly set `JCODE_HOME` must be absolute for amem setup: empty, relative and literal `~` values are refused. Unset uses the default, and absolute values retain their literal spelling without trimming. It owns only `servers.amem`, sets `shared: false` for a workspace-bound session, and preserves unrelated configuration bytes. Jcode has no native MCP add/remove command in this release, so setup edits the owned entry directly. No Jcode ingest command or capture hook is installed: saved messages lack per-message origin while the session cwd can change. Launch Jcode with `JCODE_DISABLE_CLAUDE_MCP=1` and without a same-name project override, which could otherwise replace the private binding. Use a new workspace-bound session when changing projects. See the [Jcode contract and capture limitation](docs/jcode.md).
|
|
38
|
+
|
|
39
|
+
`amem setup qwen` previews a user-scope MCP registration and a bounded SessionEnd ingest hook in the same `QWEN_HOME/settings.json` (default `~/.qwen/settings.json`). `--apply` and `--undo` follow the same confirmed, private-receipt contract as the other targets. Unrelated settings, servers and hooks are preserved. Qwen runs this hook on normal interactive exit; use `amem ingest qwen` when the session ends without that hook. Native-client operation is source-verified; the installed-package tests use stub registration CLIs.
|
|
40
|
+
|
|
35
41
|
Preview the exact configuration files, native registration commands and private undo receipts before changing anything:
|
|
36
42
|
|
|
37
43
|
```sh
|
|
@@ -203,7 +209,9 @@ Project identity (#15 D2) keeps `memoryProjectKey`: a worktree, a submodule and
|
|
|
203
209
|
|
|
204
210
|
`store.search(query, filter)` searches the curated tier and the current project's capture shard with SQLite FTS5 and BM25 plus tier/recency weighting: natural multi-word queries match partial term sets (the substring filter required every word), and a curated row outranks the same text in the capture tier. Prefix terms match inflections and Korean particles in either direction by retaining the original query term and adding a common-particle-stripped alternative. The FTS copy of the curated rows lives in its own derived `search.db` (`index.db`'s table set is a frozen contract and gains nothing); it is written from the same apply path as the index, and a prune, crash or rebuild is detected and resynced, so the incremental FTS copy equals a full resync. Where FTS5 is unavailable the engine falls back to the substring filter and reports `engine: "substring"`; capture recall needs FTS5 and is skipped in the fallback. The library entry exports `CaptureStore`, `captureSearch`, `readCaptureConfig`, `writeCaptureConfig`, `SearchIndex`, `fts5Available`, the shard helpers (`resolveAlias`, `appendAlias`, `listAliases`, `shardKeyHash`), the catalog (`rebuildCatalog`, `readCatalog`, `scanCatalog`) and the capacity advisor (`capacityReport`).
|
|
205
211
|
|
|
206
|
-
`amem
|
|
212
|
+
`amem search <words...>` ranks the current project's current curated notes with `store.search` (FTS5/BM25 plus tier and recency weights). Natural multi-word queries can match partial term sets: `amem search lighthouse beacon` can return a note containing only one of those terms. Add `--filter` to retain the previous behavior for current-project curated notes: every query word must appear as a substring, and results are newest first. `--history`, `--as-of` and an empty query retain the previous filter/newest-first path. Repository-scope notes remain a separately screened append to these results. `--all-projects` keeps its existing ranked path and cannot be combined with `--filter`. JSON output retains the existing note fields.
|
|
213
|
+
|
|
214
|
+
`amem eval [--set ko|<file>] [--k 1,5,10] [--json]` measures recall with no model and no network (agent-mem#40 F13): it replays a question set against ranked `store.search` and reports recall@k, hit@k and MRR. The ordinary curated project `amem search` path now uses that ranked search; `--filter` selects the legacy substring behavior. Without `--set` it scores the built-in synthetic set (26 invented harbour entries, long and short tier, written a day apart, and 32 questions, shipped in `src/memory/eval-set.json`), which is seeded into a throwaway store, so it never reads or writes your memory home and gives the same numbers on every machine. A set of your own either carries its own `entries` (self-contained, named by `key`) or, without entries, names real entry ids (or a prefix of 8 or more hex characters) and is scored against this project's store; each name is resolved like `amem show` does (a promoted short-term id finds its long-term copy, and a superseded id stands for the entry that replaced it, since search returns only live entries), and a prefix that matches more than one live entry is refused. `mem:<id>` as a brief prints it is accepted. Questions whose expected entry is not in the store score 0 in `all` and are left out of `answerable`, and `answerable` is scored over the entries the store holds, so a ranking problem and a coverage problem stay apart (`n/a` when no question is answerable). `--set ko` scores the second built-in set instead: 37 invented Korean entries and 38 questions (particle variants, spaced and unspaced compounds, mixed Korean-English terms, Latin abbreviations, short and long tier), with its baseline recorded in `docs/contracts.md` (agent-mem#62). Set format: `docs/contracts.md#recall-evaluation-f13`.
|
|
207
215
|
|
|
208
216
|
The R7 capacity advisor reports this machine's load and proposes capture bounds: `amem capacity [--json]` shows capture size and growth, disk, memory, the rolling `metrics.jsonl` readings and the proposal with its binding constraint; `amem capacity apply [--cap-mb N] [--retention-days N]` saves a preview, and only a matching later `--confirm` writes the home default config. Applying is person-only (agent markers refused, a terminal required); an agent may read the proposal through the library, never apply it. Under the 2026-10-17 owner decision (item 2) the proposal is the largest cap and retention that fit, and it may be above the current values: the cap is the disk-share bound itself (default 2% of free space, `--disk-share`), and retention is the share divided by the measured daily growth, up to a 365-day ceiling (a `ponytail:` constant, a person setting is the upgrade path). A hook p95 over 150 ms or hook RSS over 100 MB halves the proposed cap, and search RSS over 200 MB halves the retention. The reason names every binding constraint: `disk-share retention` or `retention ceiling`, `disk-share cap`, `R3 search RSS`, `R2 hook cost`. A proposal is never written by the advisor, and `apply` still takes `--cap-mb` and `--retention-days` to override it. Benchmark samples stay in the measuring script's synthetic home; only real hook and search invocations feed the advisor's rolling metrics.
|
|
209
217
|
|
|
@@ -282,7 +290,9 @@ Successful native project identities have a private, bounded derived cache (128
|
|
|
282
290
|
|
|
283
291
|
`amem doctor` prints one read-only report of this project's memory: duplicate and stale notes, pending candidates, capture freshness and hook registration for each CLI (a CLI with no hook is `manual`), the SessionStart brief's token cost, whether the index equals a rebuild of the logs, reader leases and the policy in effect. `amem doctor --json` is a stable machine-readable form (`amem.doctor/v1`). It writes nothing into the memory home (every check runs on a temporary copy of the logs) and creates nothing in a home you never used.
|
|
284
292
|
|
|
285
|
-
Intake runs at the end of each Codex, Kimi and
|
|
293
|
+
Intake runs at the end of each Codex, Kimi, Pi and Qwen session once `amem setup <cli> --apply` has registered it. Sync is off unless you ask for it: amem registers no sync hook and runs nothing between sessions. To sync on your own schedule, call `amem sync --if-due` from your shell (for example an EXIT trap in `~/.zshrc`; it needs a terminal). It syncs at most hourly, backs off 15 minutes doubling to a day after failures, stops after 30 seconds, and does nothing without a remote set by `amem sync --remote <private git url>`. `--force` skips the schedule and the backoff.
|
|
294
|
+
|
|
295
|
+
Split-home sync durably holds verified foreign-generation prunes with a "held, not applied" warning while unrelated sync continues; targets can remain visible, and full cross-generation removal remains a separate ceiling (see [the sync contract](docs/contracts.md#per-shard-private-sync-15-t3)).
|
|
286
296
|
|
|
287
297
|
## Native transcript intake
|
|
288
298
|
|
|
@@ -296,6 +306,8 @@ amem ingest pi --since 2026-10-01T00:00:00Z --json
|
|
|
296
306
|
|
|
297
307
|
Discovery uses `CODEX_HOME` or `~/.codex` (`sessions/` and `archived_sessions/`), `KIMI_CODE_HOME` or `~/.kimi-code` (`sessions/**/wire.jsonl`), and `PI_CODING_AGENT_DIR` or `~/.pi/agent` (`sessions/`). It does not follow directory symlinks. A native record must carry a recognized schema/time and an observed absolute working-directory binding that resolves to this canonical project, including reviewed aliases. Unknown working directories, unsupported Kimi wire versions, missing timestamps and unavailable source roots are reported rather than guessed as an empty successful history. Foreign/private/secret rows are not imported. Full native envelopes and extracted capture inputs both pass current policy.
|
|
298
308
|
|
|
309
|
+
Qwen Code uses `QWEN_RUNTIME_DIR`, then `QWEN_HOME`, then `~/.qwen`, with bounded discovery of `projects/*/chats/*.jsonl`. Each row proves its own project through `cwd`; a sanitized directory name is not an identity. The reader supports the verified 0.26.1 native envelope, an internal format rather than a public schema, and excludes thought/media content. For `advanced.runtimeOutputDir`, export the matching `QWEN_RUNTIME_DIR`; amem does not reproduce merged Qwen settings or read `.env`. See the [Qwen capture contract](docs/qwen-code.md) for source pins, cursor upgrade behavior and limitations.
|
|
310
|
+
|
|
299
311
|
Per-source cursor databases and a fair rotation cursor live in the capture shard's private `ingest/` directory. Filenames and source identities are hashes; cursor contents retain offsets, anchor hashes, scope/protocol/cutoff and ownership metadata, not raw transcript paths or text. Capture receipts make replay idempotent across a crash between append and cursor acknowledgement. Repeating the command resumes its bounded tour instead of repeatedly favoring the first files. Broadening `--since` rechecks earlier history while existing capture receipts prevent duplicates. The JSON receipt reports `files`, `read`, `imported`, `skipped`, `spooled`, `remaining`, `errors` and an optional bounded `reason`; unavailable coverage is not silently represented by zero imported rows.
|
|
300
312
|
|
|
301
313
|
The default invocation reads at most 4 MiB and attempts 32 sources; trusted library options can raise those limits to 16 MiB and 128 sources. It examines at most 4,096 records, bounds a record at 256 KiB, and caps discovery at 4,096 directory entries/depth 8. Re-run a receipt with `remaining: true` after inspecting its reason; ingest creates no polling timer or watcher. Oversized records, including an unbroken record longer than the 4 MiB default window, are counted once and discarded through their next newline using a durable byte offset/skip flag. Later valid records remain reachable without retaining oversized content; an unfinished oversized tail stays pending until its newline arrives. The cursor assumes append-only native files and verifies file identity plus the last 1,024 acknowledged bytes. Historical rewrites outside that anchor are a declared ceiling, not full-file verification. These readers do not establish account-backed native interoperability, P5 cutover or P6 live-week acceptance.
|
|
@@ -367,7 +379,7 @@ Real migration remains blocked until the separately reviewed generation-capable
|
|
|
367
379
|
|
|
368
380
|
## Curated shard migration (#15 T2)
|
|
369
381
|
|
|
370
|
-
`amem migrate --shards` moves the curated single home into `projects/<key-hash>/` and `user/` through a resumable generation journal. The original single-home log bytes are never rewritten. A populated selected generation is supported and keeps its generation ID. Each shard keeps private legacy inputs in read-only `legacy/`, public-main inputs in read-only `legacy-main/`, and generation-born rows in its own `log/` and `local/`. It has its own `index.db`, `search.db` and mutation lock; original IDs, hashes, placement and own/local/public provenance are preserved. The source generation bytes stay unchanged. Immutable prune proofs are validated before effects; a multi-shard prune is translated into one deterministic row per shard, with a receipt naming the original row. Existing capture and `work/` directories are not migration inputs and are left untouched. `catalog.db` remains derived and rebuildable, now including curated entry counts and last use. Split curated stores sync the opened canonical project and user overlay to separate private-remote refs (`amem/p/<key-hash>` and `amem/u`) with `amem sync --policy-free`; unopened projects and machine data are not fetched. Original IDs and attribution survive independent generations.
|
|
382
|
+
`amem migrate --shards` moves the curated single home into `projects/<key-hash>/` and `user/` through a resumable generation journal. The original single-home log bytes are never rewritten. A populated selected generation is supported and keeps its generation ID. Each shard keeps private legacy inputs in read-only `legacy/`, public-main inputs in read-only `legacy-main/`, and generation-born rows in its own `log/` and `local/`. It has its own `index.db`, `search.db` and mutation lock; original IDs, hashes, placement and own/local/public provenance are preserved. The source generation bytes stay unchanged. Immutable prune proofs are validated before effects; a multi-shard prune is translated into one deterministic row per shard, with a receipt naming the original row. Existing capture and `work/` directories are not migration inputs and are left untouched. `catalog.db` remains derived and rebuildable, now including curated entry counts and last use. Split curated stores sync the opened canonical project and user overlay to separate private-remote refs (`amem/p/<key-hash>` and `amem/u`) with `amem sync --policy-free`; unopened projects and machine data are not fetched. Original IDs and attribution survive independent generations. See `docs/contracts.md`, "Per-shard private sync", for prune transport and application boundaries.
|
|
371
383
|
|
|
372
384
|
Before splitting a live home, every installed host must open it through `openShardStore({ project })` or the explicit user shard. Plain `new MemoryStore()` refuses any split marker, including an interrupted split, so it cannot silently keep writing the old layout. No host capability registry currently exists; migration preflight output states that limitation. Do not release T2 for agent-hub use until its memory factory is integrated with `openShardStore` and that host version is installed. Existing unannounced older builds remain the documented unchanged-source ceiling.
|
|
373
385
|
|
|
@@ -413,7 +425,7 @@ The frozen nine-tool contract for dotfiles#255 and maru#458:
|
|
|
413
425
|
|
|
414
426
|
Search/show/timeline label capture as recall with `evidence: false`; capture never enters the curated graph. Policy screens complete rows before selection, limits or graph construction. The standalone secret screen, private flags, PII patterns and deny paths compose with a host's stricter policy through `createMemoryMcp({cwd, base, policy})`. Screening errors fail closed. `base` is the exact memory root; historical library `home` options still name its parent. Scope and profile changes refuse further access instead of silently reopening another store. Graph reads retain the 5,000-row curated ceiling; work-list reads stream the complete ordered history without an operation-count ceiling, applying policy before latest-value reduction. The working state scales with the current result, not historical payloads. The graph ceiling is an operational bound, not an exposed count of unscreened data. Graphs cover curated-note relationships, not goal/suggestion tables.
|
|
415
427
|
|
|
416
|
-
`WorkListStore` writes `projects/<key-hash>/work/` with its own append log, derived index and lock. Scalar fields reduce by latest value, null clears, list status `done` closes, and item status is `todo|doing|done|dropped`. Work lists are outside generation selection and capture eviction. `amem import claude-mem` additionally imports `work_state_entries` only through persisted person-reviewed D2 aliases, preserving source DB bytes and stable replay receipts; JSON keeps observation fields and adds `work` counts/cursor.
|
|
428
|
+
`WorkListStore` writes `projects/<key-hash>/work/` with its own append log, derived index and lock. Scalar fields reduce by latest value, null clears, list status `done` closes, and item status is `todo|doing|done|dropped`. Work lists are outside generation selection and capture eviction. `amem import claude-mem` additionally imports `work_state_entries` only through persisted person-reviewed D2 aliases, preserving source DB bytes and stable replay receipts; JSON keeps observation fields and adds `work` counts/cursor. `--capture-days N` also imports observations and session summaries from the last N days into machine-local capture under the same aliases, with read/added/spooled/skipped counts, stable replay receipts and the configured capture cap and retention. Widening N after an earlier import requires `--restart` to revisit rows below the saved cursor. Chroma vectors are never read.
|
|
417
429
|
|
|
418
430
|
`amem vault-candidates --json` and the MCP read tool select durable long-term notes under current policy, excluding task/review/PR/CI progress and already-linked entries. The deterministic status filter covers English status phrases and can miss paraphrased or other-language status text; candidates require person review. No vault is opened, indexed or written. Library `recordVaultLink` records an exact-entry hash and relative Markdown path in a generation-only local append op; it survives index rebuild and shard migration, stays machine-local, and does not change shared-v1 schemas or sync semantics. There is no MCP link writer.
|
|
419
431
|
|
package/package.json
CHANGED
package/src/cli.ts
CHANGED
|
@@ -11,7 +11,7 @@ import { serveMcp, writeConfirmed } from "./mcp.ts";
|
|
|
11
11
|
import { createStandalonePolicy } from "./memory/policy.ts";
|
|
12
12
|
import { configuredPolicy, configuredPolicyOptions, loadAmemConfig, repoConfigRoot } from "./memory/config.ts";
|
|
13
13
|
import { vaultCandidates } from "./memory/vault.ts";
|
|
14
|
-
import { importClaudeMemWorkState } from "./memory/import.ts";
|
|
14
|
+
import { importClaudeMemCapture, importClaudeMemWorkState } from "./memory/import.ts";
|
|
15
15
|
import { closeSync, constants, existsSync, fstatSync, lstatSync, openSync, readFileSync, readSync, rmSync, writeFileSync } from "node:fs";
|
|
16
16
|
import { join } from "node:path";
|
|
17
17
|
import { MemoryStore, looksSecret, machineId, openShardStore, type StoreOptions } from "./memory/store.ts";
|
|
@@ -168,9 +168,9 @@ export async function runCli(argv: readonly string[], context: CliContext = {}):
|
|
|
168
168
|
const parsed = parseCommand(argv);
|
|
169
169
|
if (parsed.options["--help"] === true) return commandHelp();
|
|
170
170
|
const kind = parsed.positionals[0];
|
|
171
|
-
if (parsed.positionals.length !== 1 || !["codex", "kimi", "pi"].includes(kind ?? "")) throw new Error("usage: amem ingest codex|kimi|pi [--since ISO] [--json]");
|
|
171
|
+
if (parsed.positionals.length !== 1 || !["codex", "kimi", "pi", "qwen"].includes(kind ?? "")) throw new Error("usage: amem ingest codex|kimi|pi|qwen [--since ISO] [--json]");
|
|
172
172
|
const { ingestTranscript } = await import("./ingest.ts");
|
|
173
|
-
return JSON.stringify(await ingestTranscript(kind as "codex" | "kimi" | "pi", {
|
|
173
|
+
return JSON.stringify(await ingestTranscript(kind as "codex" | "kimi" | "pi" | "qwen", {
|
|
174
174
|
cwd: context.cwd ?? process.cwd(), base: memoryHome(context.env ?? process.env),
|
|
175
175
|
...(typeof parsed.options["--since"] === "string" ? { since: parsed.options["--since"] as string } : {}),
|
|
176
176
|
}));
|
|
@@ -195,10 +195,10 @@ export async function runCli(argv: readonly string[], context: CliContext = {}):
|
|
|
195
195
|
}
|
|
196
196
|
const { definition, options, rawArgs: args } = parsed;
|
|
197
197
|
if (parsed.path[0] === "setup") {
|
|
198
|
-
if (parsed.positionals.length !== 1 || options["--apply"] === true && options["--undo"] === true) throw new Error("usage: amem setup claude|codex|kimi|pi|all [--apply|--undo]");
|
|
198
|
+
if (parsed.positionals.length !== 1 || options["--apply"] === true && options["--undo"] === true) throw new Error("usage: amem setup claude|codex|kimi|pi|qwen|jcode|agy|all [--apply|--undo]");
|
|
199
199
|
const { runSetup } = await import("./setup.ts");
|
|
200
200
|
return runSetup(parsed.positionals[0]!, options["--apply"] === true ? "apply" : options["--undo"] === true ? "undo" : "preview", {
|
|
201
|
-
env: context.env ?? process.env, stdinTTY: context.stdinTTY ?? process.stdin.isTTY,
|
|
201
|
+
env: context.env ?? process.env, stdinTTY: context.stdinTTY ?? process.stdin.isTTY, cwd: context.cwd ?? process.cwd(),
|
|
202
202
|
...(context.setupConfirm ? { confirm: context.setupConfirm } : {}),
|
|
203
203
|
});
|
|
204
204
|
}
|
|
@@ -549,7 +549,7 @@ export async function runCli(argv: readonly string[], context: CliContext = {}):
|
|
|
549
549
|
return options["--json"] ? JSON.stringify(result, null, 2) : `${result.dryRun ? "preview (nothing evicted; add --confirm):\n" : ""}${gcText(result)}`;
|
|
550
550
|
}
|
|
551
551
|
if (sub === "import" && ["claude-auto", "codex"].includes(parsed.path[1] ?? "")) {
|
|
552
|
-
for (const flag of ["--db", "--limit", "--restart"]) if (options[flag] !== undefined) throw new Error(`${flag} only applies to import claude-mem`);
|
|
552
|
+
for (const flag of ["--db", "--limit", "--restart", "--capture-days"]) if (options[flag] !== undefined) throw new Error(`${flag} only applies to import claude-mem`);
|
|
553
553
|
const source = parsed.path[1] as ExternalMemorySource, policy = policyOf();
|
|
554
554
|
const result = await importExternalMemory({ base: memoryHome(), project: boundProject(), source, cwd: policyCwd, env: context.env ?? process.env,
|
|
555
555
|
...(typeof options["--from"] === "string" ? { from: options["--from"] } : {}), policy: value => policy.screenStandalone(value) });
|
|
@@ -589,6 +589,7 @@ export async function runCli(argv: readonly string[], context: CliContext = {}):
|
|
|
589
589
|
return options["--json"] ? JSON.stringify(written, null, 2) : `${op === "forget" ? "forgot" : op === "archive" ? "archived" : "unarchived"} ${written.key}${requested !== written.key ? ` (alias ${requested})` : ""}`;
|
|
590
590
|
}
|
|
591
591
|
if (sub === "search" && options["--all-projects"] === true) {
|
|
592
|
+
if (options["--filter"] === true) throw new Error("--filter applies to project-local search, not --all-projects");
|
|
592
593
|
const words = parsed.positionals.join(" ");
|
|
593
594
|
if (!words.trim()) throw new Error("usage: amem search --all-projects <words> [--limit N] [--include-archived] [--json]");
|
|
594
595
|
const result = searchAllProjects(memoryHome(), words, { ...(options["--limit"] !== undefined ? { limit: Number(options["--limit"]) } : {}), includeArchived: options["--include-archived"] === true });
|
|
@@ -724,11 +725,20 @@ export async function runCli(argv: readonly string[], context: CliContext = {}):
|
|
|
724
725
|
}
|
|
725
726
|
// Only a real attempt is stamped: a local refusal (lock held, no remote, policy, usage) is not, and a future stamp is no history (src/memory/sync-cadence.ts).
|
|
726
727
|
const stampSync = (error: string | undefined, cause?: StopCause) => { if (syncOutcome(error, cause) !== "local") recordSyncAttempt(syncHome, error === undefined, Date.now()); };
|
|
727
|
-
const
|
|
728
|
+
const captureImport = sub === "import" && parsed.path[1] === "claude-mem", captureDays = options["--capture-days"] === undefined ? undefined : Number(options["--capture-days"]);
|
|
729
|
+
if (captureImport && captureDays !== undefined && (!Number.isSafeInteger(captureDays) || captureDays <= 0)) throw new Error("--capture-days takes a positive integer");
|
|
730
|
+
const captureArgs = new Set<number>();
|
|
731
|
+
if (captureImport) for (let index = 0; index < args.length; index++) {
|
|
732
|
+
if (args[index] === "--capture-days") { captureArgs.add(index); captureArgs.add(index + 1); }
|
|
733
|
+
else if (args[index]?.startsWith("--capture-days=")) captureArgs.add(index);
|
|
734
|
+
}
|
|
735
|
+
// --filter selects the standalone CLI mode; hosts keep their existing raw command arguments.
|
|
736
|
+
const legacySearch = options["--filter"] === true;
|
|
737
|
+
const syncArgs = args.filter((arg, index) => arg !== "--if-due" && arg !== "--force" && !captureArgs.has(index) && (!legacySearch || arg !== "--filter"));
|
|
728
738
|
let output: string;
|
|
729
739
|
const stop = ifDue ? stopSignal(context.syncDeadlineMs ?? SYNC_IF_DUE_DEADLINE_MS, process) : undefined;
|
|
730
740
|
try {
|
|
731
|
-
output = await withRepoRows(await rawMemoryCommand(syncArgs, cwd, { person: true, project: boundProject(), open, readable: notes => notes, stale: createRecallStalenessProbe({ cwd: recallRoot ?? cwd, rootConfirmed, policy: () => configured }), outputReady: notes => outputReady(notes.map(view => view.id)), importPolicy: () => text => configured.declinesPersonal(text),
|
|
741
|
+
output = await withRepoRows(await rawMemoryCommand(syncArgs, cwd, { person: true, rankedSearch: !legacySearch, project: boundProject(), open, readable: notes => notes, stale: createRecallStalenessProbe({ cwd: recallRoot ?? cwd, rootConfirmed, policy: () => configured }), outputReady: notes => outputReady(notes.map(view => view.id)), importPolicy: () => text => configured.declinesPersonal(text),
|
|
732
742
|
sync: { screen: row => configured.screenStandalone(row), ...(stop ? { signal: stop.signal } : {}) }, // the repository-narrowed policy binds what sync publishes too
|
|
733
743
|
onBridgeStatus: status => { receipt.bridgeStatus = status; } }));
|
|
734
744
|
} catch (error) {
|
|
@@ -747,10 +757,11 @@ export async function runCli(argv: readonly string[], context: CliContext = {}):
|
|
|
747
757
|
if (options["--json"]) { try { const result = JSON.parse(output) as { ok?: unknown; error?: unknown }; if (result.ok === false) failure = typeof result.error === "string" ? result.error : "sync failed"; } catch { /* not a sync result */ } }
|
|
748
758
|
stampSync(failure, stop?.cause());
|
|
749
759
|
}
|
|
750
|
-
if (
|
|
751
|
-
const policy = configured;
|
|
752
|
-
const work = await importClaudeMemWorkState({ base: memoryHome(), project: boundProject(), policy: row => policy.screen(row), ...
|
|
753
|
-
|
|
760
|
+
if (captureImport) {
|
|
761
|
+
const policy = configured, db = options["--db"] ? { db: String(options["--db"]) } : {}, limit = options["--limit"] ? { limit: Number(options["--limit"]) } : {}, restart = options["--restart"] === true;
|
|
762
|
+
const work = await importClaudeMemWorkState({ base: memoryHome(), project: boundProject(), policy: row => policy.screen(row), ...db, ...limit, restart });
|
|
763
|
+
const capture = captureDays === undefined ? undefined : await importClaudeMemCapture({ base: memoryHome(), project: boundProject(), cwd, days: captureDays, ...db, ...limit, restart, policy: { host: policy } });
|
|
764
|
+
return options["--json"] ? JSON.stringify({ ...JSON.parse(output), work, ...(capture ? { capture } : {}) }, null, 2) : `${output}\nwork lists: ${JSON.stringify(work)}${capture ? `\ncapture: ${JSON.stringify(capture)}` : ""}`;
|
|
754
765
|
}
|
|
755
766
|
return !options["--json"] && receipt.bridgeStatus
|
|
756
767
|
? `${output}\n(legacy malformed lines skipped: ${receipt.bridgeStatus.skippedRows})` : output;
|
package/src/commands-table.ts
CHANGED
|
@@ -25,20 +25,21 @@ export const GLOBAL_OPTIONS = { "--help": flag("Show help") };
|
|
|
25
25
|
export const ROOT_OPTIONS = { "--version": flag("Show version"), ...json };
|
|
26
26
|
export const COMMANDS: Record<string, CommandDefinition> = {
|
|
27
27
|
consolidate: { description: "Queue deterministic project-local consolidation candidates; curated notes change only after person review", mutates: true, options: { ...json, "--project": text("The current project's key; another project is refused (run from its directory)") } },
|
|
28
|
-
setup: { description: "Preview reversible MCP and
|
|
28
|
+
setup: { description: "Preview reversible MCP setup and supported capture hooks for native coding CLIs; apply and undo are person-only", options: { "--apply": flag("Apply after terminal confirmation"), "--undo": flag("Restore original configuration after confirmation") }, args: [{ name: "target", values: ["claude", "codex", "kimi", "pi", "qwen", "jcode", "agy", "all"] }] },
|
|
29
29
|
mcp: { description: "Run the project-bound MCP stdio server for this session" },
|
|
30
30
|
hook: { description: "Run a fail-open, context-only Claude Code hook", args: [{ name: "event", values: [...HOOK_EVENTS] }] },
|
|
31
|
-
ingest: { description: "Incrementally capture native session transcripts without a watcher", options: { ...json, "--since": text("ISO date or time") }, args: [{ name: "kind", values: ["codex", "kimi", "pi"] }] },
|
|
31
|
+
ingest: { description: "Incrementally capture native session transcripts without a watcher", options: { ...json, "--since": text("ISO date or time") }, args: [{ name: "kind", values: ["codex", "kimi", "pi", "qwen"] }] },
|
|
32
32
|
"vault-candidates": { description: "List screened durable curated vault candidates (per project, or across projects)", options: { ...json, "--limit": text("Maximum candidates per project (1..100)"),
|
|
33
33
|
"--all-projects": flag("List every active project's candidates; the reply says it crossed projects"), "--project": text("List one named project's candidates") } },
|
|
34
34
|
list: { description: "List memory", options: { ...filters, ...history } },
|
|
35
|
-
search: { description: "
|
|
35
|
+
search: { description: "Rank current curated project notes; history/as-of and empty queries retain newest-first filtering", options: { ...filters, ...history,
|
|
36
|
+
"--filter": flag("Current-project curated notes: match every word as a substring, newest first; not with --all-projects"), "--all-projects": flag("Search every active project; the reply says it crossed projects"),
|
|
36
37
|
"--include-archived": flag("With --all-projects, also read archived projects") }, args: [{ name: "words", variadic: true }] },
|
|
37
38
|
brief: { description: "Read intent/path memory: fails first, then decisions and contracts; untrusted data within a UTF-8 byte cap", options: { ...json,
|
|
38
39
|
"--for": text("Explicit intent"), "--paths": text("Repeatable named path"), "--symbols": text("Repeatable named symbol"), "--bytes": text("Complete brief-body byte cap (1..8192, default 2048)") } },
|
|
39
40
|
projects: { description: "List the project shards of this memory home (#15 D3)", options: { ...json, "--all": flag("Include forgotten projects (archived ones are always listed)") } },
|
|
40
41
|
usage: { description: "Totals and the largest project shards", options: { ...json, "--limit": text("Largest shards to list (1..100)"), "--all": flag("Include forgotten projects") } },
|
|
41
|
-
eval: { description: "Replay a local question set against
|
|
42
|
+
eval: { description: "Replay a local question set against ranked store.search and report recall@k and MRR (no model, no network; the built-in synthetic set without --set, `--set ko` for the built-in Korean one)", options: { ...json,
|
|
42
43
|
"--set": { description: "`ko` (the built-in Korean set) or a question set file (amem.eval/v1); with entries it is self-contained, without them its ids name entries of this project", value: "path" }, "--k": text("Cutoffs, comma separated (default 1,5,10; each 1..100)") } },
|
|
43
44
|
gc: { description: "Apply each shard's retention and cap across shards in bounded batches (preview without --confirm)", mutates: "confirm", options: {
|
|
44
45
|
...json, "--confirm": flag("Evict now; without it, preview only"), "--batches": text("evict() windows per shard in this run (1..1000, default 1)") } },
|
|
@@ -76,9 +77,9 @@ export const COMMANDS: Record<string, CommandDefinition> = {
|
|
|
76
77
|
sync: { description: "Sync a private remote (--if-due: only when the schedule and the failure backoff allow it; --force ignores both)", options: { ...json, ...policy, "--remote": text("Private git URL"),
|
|
77
78
|
"--if-due": flag("Sync only when due (hourly after a success, 15 minutes doubling to a day after failures); stops after 30 seconds"), "--force": flag("With --if-due: ignore the schedule and the backoff") }, mutates: true, policyFree: true },
|
|
78
79
|
machine: { description: "Show or label this machine", options: { ...json, "--label": text("Machine label") }, mutates: "label" },
|
|
79
|
-
import: { description: "Import external memory", options: { ...json, ...policy, "--db": { description: "SQLite database file", value: "path" }, "--limit": text("Maximum observations"), "--restart": flag("Restart import cursor") }, mutates: true, policyFree: true,
|
|
80
|
+
import: { description: "Import external memory", options: { ...json, ...policy, "--db": { description: "SQLite database file", value: "path" }, "--limit": text("Maximum observations"), "--restart": flag("Restart import cursor"), "--capture-days": text("Import recent claude-mem observations and summaries into capture") }, mutates: true, policyFree: true,
|
|
80
81
|
commands: {
|
|
81
|
-
"claude-mem": { description: "Import claude-mem observations", options: { "--db": { description: "SQLite database file", value: "path" }, "--limit": text("Maximum observations"), "--restart": flag("Restart import cursor") } },
|
|
82
|
+
"claude-mem": { description: "Import claude-mem observations", options: { "--db": { description: "SQLite database file", value: "path" }, "--limit": text("Maximum observations"), "--restart": flag("Restart import cursor"), "--capture-days": text("Import recent observations and summaries into capture") } },
|
|
82
83
|
"claude-auto": { description: "Import Claude Code auto memory as review candidates; CLAUDE.md and AGENTS.md stay excluded", options: { "--from": { description: "Claude projects directory (default ~/.claude/projects)", value: "path" } } },
|
|
83
84
|
codex: { description: "Import Codex memories as review candidates; originals stay unchanged", options: { "--from": { description: "Codex memories directory (default ~/.codex/memories)", value: "path" } } },
|
|
84
85
|
} },
|
package/src/ingest.ts
CHANGED
|
@@ -2,7 +2,7 @@ import { Database } from "bun:sqlite";
|
|
|
2
2
|
import { createHash, randomUUID } from "node:crypto";
|
|
3
3
|
import { closeSync, chmodSync, constants, fchmodSync, fstatSync, lstatSync, mkdirSync, openSync, opendirSync, readSync, type Stats } from "node:fs";
|
|
4
4
|
import { homedir } from "node:os";
|
|
5
|
-
import { dirname, isAbsolute, join } from "node:path";
|
|
5
|
+
import { dirname, isAbsolute, join, resolve } from "node:path";
|
|
6
6
|
import { CaptureStore, type CaptureInput } from "./memory/capture.ts";
|
|
7
7
|
import { memoryHome } from "./memory/home.ts";
|
|
8
8
|
import { canonicalPath, processLiveness, processSignature, realPath } from "./memory/platform.ts";
|
|
@@ -11,7 +11,7 @@ import { configuredPolicyOptions } from "./memory/config.ts";
|
|
|
11
11
|
import { memoryProjectKey } from "./memory/project.ts";
|
|
12
12
|
import { effectiveAliases } from "./memory/shard.ts";
|
|
13
13
|
|
|
14
|
-
export type IngestKind = "codex" | "kimi" | "pi";
|
|
14
|
+
export type IngestKind = "codex" | "kimi" | "pi" | "qwen";
|
|
15
15
|
export interface IngestOptions {
|
|
16
16
|
cwd: string; base?: string;
|
|
17
17
|
/** Trusted host/test seam. The ordinary CLI discovers native roots instead. */
|
|
@@ -28,6 +28,8 @@ interface Position { identity: string; offset: number; anchor: string; scope: Sc
|
|
|
28
28
|
interface Cursor extends Position { token: string | null; pid: number | null; signature: string | null }
|
|
29
29
|
const MAX_BYTES = 16 * 1024 * 1024, DEFAULT_BYTES = 4 * 1024 * 1024, MAX_FILES = 128, DEFAULT_FILES = 32;
|
|
30
30
|
const MAX_CURSOR_BYTES = 128 * 1024;
|
|
31
|
+
// Bump after extending the verified Qwen decoder: replay skipped versions with existing capture receipts deduplicating accepted rows.
|
|
32
|
+
const QWEN_CONTRACT = "qwen-0.26.1-v1";
|
|
31
33
|
const utf8 = new TextDecoder("utf-8", { fatal: true });
|
|
32
34
|
const MAX_RECORD = 256 * 1024, MAX_RECORDS = 4096, MAX_DISCOVERY = 4096, MAX_DEPTH = 8, ANCHOR_BYTES = 1024;
|
|
33
35
|
const hash = (value: string | Uint8Array) => createHash("sha256").update(value).digest("hex");
|
|
@@ -115,6 +117,7 @@ function lease(file: string): { position: Position; save(value: Position): void;
|
|
|
115
117
|
|
|
116
118
|
/** Native roots only. Discovery never follows a directory symlink or enters an unbounded tree. */
|
|
117
119
|
function discover(kind: IngestKind): { files: string[]; bounded: boolean; unavailable: boolean } {
|
|
120
|
+
if (kind === "qwen") return discoverQwen();
|
|
118
121
|
const root = kind === "codex" ? process.env.CODEX_HOME?.trim() || join(homedir(), ".codex")
|
|
119
122
|
: kind === "kimi" ? process.env.KIMI_CODE_HOME?.trim() || join(homedir(), ".kimi-code")
|
|
120
123
|
: process.env.PI_CODING_AGENT_DIR?.trim() || join(homedir(), ".pi", "agent");
|
|
@@ -139,6 +142,38 @@ function discover(kind: IngestKind): { files: string[]; bounded: boolean; unavai
|
|
|
139
142
|
return { files: files.sort(), bounded, unavailable: rootsRead === 0 };
|
|
140
143
|
}
|
|
141
144
|
|
|
145
|
+
/** Only the native projects/<one directory>/chats/*.jsonl layout, never settings or account files. */
|
|
146
|
+
function discoverQwen(): { files: string[]; bounded: boolean; unavailable: boolean } {
|
|
147
|
+
const given = process.env.QWEN_RUNTIME_DIR || process.env.QWEN_HOME || join(homedir(), ".qwen");
|
|
148
|
+
const expanded = given === "~" ? homedir() : given.startsWith("~/") || given.startsWith("~\\") ? join(homedir(), ...given.slice(2).split(/[/\\]+/)) : given;
|
|
149
|
+
const root = resolve(expanded);
|
|
150
|
+
const files: string[] = []; let visited = 0, bounded = false;
|
|
151
|
+
const open = (folder: string) => {
|
|
152
|
+
const stat = lstatSync(folder);
|
|
153
|
+
if (!stat.isDirectory() || stat.isSymbolicLink()) throw Error("native directory unavailable");
|
|
154
|
+
return opendirSync(folder);
|
|
155
|
+
};
|
|
156
|
+
let projects: ReturnType<typeof opendirSync>;
|
|
157
|
+
try { if (lstatSync(root).isSymbolicLink()) throw Error("native directory unavailable"); projects = open(join(root, "projects")); }
|
|
158
|
+
catch { return { files, bounded, unavailable: true }; }
|
|
159
|
+
try {
|
|
160
|
+
for (let project = projects.readSync(); project; project = projects.readSync()) {
|
|
161
|
+
if (++visited > MAX_DISCOVERY) { bounded = true; break; }
|
|
162
|
+
if (!project.isDirectory() || project.isSymbolicLink()) continue;
|
|
163
|
+
let chats: ReturnType<typeof opendirSync>;
|
|
164
|
+
try { chats = open(join(root, "projects", project.name, "chats")); } catch { continue; }
|
|
165
|
+
try {
|
|
166
|
+
for (let item = chats.readSync(); item; item = chats.readSync()) {
|
|
167
|
+
if (++visited > MAX_DISCOVERY) { bounded = true; break; }
|
|
168
|
+
if (item.isFile() && !item.isSymbolicLink() && item.name.endsWith(".jsonl")) files.push(join(root, "projects", project.name, "chats", item.name));
|
|
169
|
+
}
|
|
170
|
+
} finally { chats.closeSync(); }
|
|
171
|
+
if (bounded) break;
|
|
172
|
+
}
|
|
173
|
+
} finally { projects.closeSync(); }
|
|
174
|
+
return { files: files.sort(), bounded, unavailable: false };
|
|
175
|
+
}
|
|
176
|
+
|
|
142
177
|
function timeOf(record: Record<string, any>, kind: IngestKind): string | undefined {
|
|
143
178
|
const value = kind === "kimi" ? record.time : record.timestamp;
|
|
144
179
|
const millis = typeof value === "number" && kind === "kimi" && Number.isFinite(value) && value >= 100_000_000_000 ? value
|
|
@@ -151,7 +186,55 @@ function contentText(content: unknown): string {
|
|
|
151
186
|
if (!Array.isArray(content)) return "";
|
|
152
187
|
return content.filter(item => object(item) && ["text", "input_text", "output_text"].includes(item.type) && typeof item.text === "string").map(item => item.text).join("\n");
|
|
153
188
|
}
|
|
189
|
+
const qwenUuid = (value: unknown): boolean => typeof value === "string" && /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(value);
|
|
190
|
+
const qwenSession = (value: unknown): boolean => typeof value === "string" && /^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}(?:-agent-[a-zA-Z0-9_.-]+)?$/i.test(value);
|
|
191
|
+
function qwenPart(part: unknown): boolean {
|
|
192
|
+
if (!object(part)) return false;
|
|
193
|
+
if (part.thought === true) return true;
|
|
194
|
+
if (Object.keys(part).some(key => !["text", "thought", "thoughtSignature", "functionCall", "functionResponse"].includes(key))) return false;
|
|
195
|
+
if (part.thought !== undefined && typeof part.thought !== "boolean" || part.thoughtSignature !== undefined && typeof part.thoughtSignature !== "string") return false;
|
|
196
|
+
const forms = ["text", "functionCall", "functionResponse"].filter(key => part[key] !== undefined);
|
|
197
|
+
if (forms.length !== 1) return false;
|
|
198
|
+
if (forms[0] === "text") return typeof part.text === "string";
|
|
199
|
+
const value = part[forms[0]!];
|
|
200
|
+
return object(value) && typeof value.name === "string" && value.name.length > 0
|
|
201
|
+
&& (value.id === undefined || typeof value.id === "string")
|
|
202
|
+
&& (forms[0] === "functionCall" ? value.args === undefined || object(value.args)
|
|
203
|
+
: object(value.response) && (value.parts === undefined || Array.isArray(value.parts)));
|
|
204
|
+
}
|
|
205
|
+
function qwenRecord(raw: Record<string, any>): boolean {
|
|
206
|
+
// ponytail: CLI version is not a schema tag. Support only the 0.26.1 writer shape verified at
|
|
207
|
+
// QwenLM/qwen-code 1ce173117196ab9a9f77b5d19ac148cbabe25dfd; extend with native fixtures, not version guessing.
|
|
208
|
+
if (raw.version !== "0.26.1" || !qwenUuid(raw.uuid) || !qwenSession(raw.sessionId)
|
|
209
|
+
|| raw.parentUuid !== null && !qwenUuid(raw.parentUuid) || !["user", "assistant", "tool_result", "system"].includes(raw.type)
|
|
210
|
+
|| typeof raw.cwd !== "string" || !isAbsolute(raw.cwd) || !timeOf(raw, "qwen")) return false;
|
|
211
|
+
if (raw.type === "system" || raw.message === undefined) return true;
|
|
212
|
+
return object(raw.message) && raw.message.role === (raw.type === "assistant" ? "model" : "user")
|
|
213
|
+
&& Array.isArray(raw.message.parts) && raw.message.parts.every(qwenPart);
|
|
214
|
+
}
|
|
215
|
+
function extractQwen(raw: Record<string, any>, project: string): CaptureInput | undefined {
|
|
216
|
+
if (raw.type === "system" || !object(raw.message)) return undefined;
|
|
217
|
+
const excerpts: string[] = [], tools: string[] = [], events: string[] = [];
|
|
218
|
+
for (const part of raw.message.parts) {
|
|
219
|
+
if (part.thought === true) continue;
|
|
220
|
+
if (typeof part.text === "string") { if (part.text) { excerpts.push(part.text); events.push("message"); } }
|
|
221
|
+
else if (object(part.functionCall)) {
|
|
222
|
+
const call = part.functionCall;
|
|
223
|
+
excerpts.push(JSON.stringify({ functionCall: { name: call.name, ...(call.id !== undefined ? { id: call.id } : {}), ...(call.args !== undefined ? { args: call.args } : {}) } }));
|
|
224
|
+
tools.push(call.name); events.push("function_call");
|
|
225
|
+
} else if (object(part.functionResponse)) {
|
|
226
|
+
const response = part.functionResponse;
|
|
227
|
+
// FunctionResponse.parts contains media, not textual tool output. Never stringify it into capture.
|
|
228
|
+
excerpts.push(JSON.stringify({ functionResponse: { name: response.name, ...(response.id !== undefined ? { id: response.id } : {}), response: response.response } }));
|
|
229
|
+
tools.push(response.name); events.push("function_call_output");
|
|
230
|
+
}
|
|
231
|
+
}
|
|
232
|
+
if (!excerpts.length) return undefined;
|
|
233
|
+
return { project, agent: "qwen", event: events.every(event => event === events[0]) ? events[0]! : "message", excerpt: excerpts.join("\n").slice(0, 2000),
|
|
234
|
+
...(tools.length && tools.every(tool => tool === tools[0]) ? { tool: tools[0]!.slice(0, 64) } : {}) };
|
|
235
|
+
}
|
|
154
236
|
function extract(kind: IngestKind, raw: Record<string, any>, project: string): CaptureInput | undefined {
|
|
237
|
+
if (kind === "qwen") return extractQwen(raw, project);
|
|
155
238
|
const message = kind === "codex" && raw.type === "response_item" ? raw.payload
|
|
156
239
|
: kind === "pi" && raw.type === "message" ? raw.message
|
|
157
240
|
: kind === "kimi" && raw.type === "context.append_message" ? raw.message : undefined;
|
|
@@ -171,7 +254,10 @@ function extract(kind: IngestKind, raw: Record<string, any>, project: string): C
|
|
|
171
254
|
/**
|
|
172
255
|
* Incremental deterministic native-message ingestion; no watcher, account, model or network operation.
|
|
173
256
|
* Native source contracts: Codex rust-v0.163.0 protocol RolloutItem; Pi session-manager v3;
|
|
174
|
-
* Kimi b5e28fd6 human/persist/v2/wire.ts (protocol 1.5) and sessionIndexSource.recoverCwd
|
|
257
|
+
* Kimi b5e28fd6 human/persist/v2/wire.ts (protocol 1.5) and sessionIndexSource.recoverCwd;
|
|
258
|
+
* Qwen 1ce173117196ab9a9f77b5d19ac148cbabe25dfd ChatRecord and config/storage.ts.
|
|
259
|
+
* Qwen discovery uses exported environment roots, not merged settings or .env files. For a configured
|
|
260
|
+
* advanced.runtimeOutputDir, export its resolved path as QWEN_RUNTIME_DIR or pass a trusted native file.
|
|
175
261
|
* ponytail: discovery covers at most 4096 directory entries and depth 8. A larger native tree reports
|
|
176
262
|
* remaining/source_bound; a future native index cursor can extend coverage without a resident watcher.
|
|
177
263
|
* Within that bound a durable hash-only rotation admits at most maxFiles source attempts per call, including
|
|
@@ -185,7 +271,7 @@ export async function ingestTranscript(kind: IngestKind, options: IngestOptions)
|
|
|
185
271
|
const result: IngestReceipt = { files: 0, read: 0, imported: 0, skipped: 0, spooled: 0, remaining: false, errors: 0 };
|
|
186
272
|
const fail = (reason: NonNullable<IngestReceipt["reason"]>, remaining = false) => { result.errors++; result.reason ??= reason; result.remaining ||= remaining; };
|
|
187
273
|
const bytes = options.maxBytes ?? DEFAULT_BYTES, count = options.maxFiles ?? DEFAULT_FILES;
|
|
188
|
-
if (!["codex", "kimi", "pi"].includes(kind) || !Number.isSafeInteger(bytes) || bytes < 1 || bytes > MAX_BYTES || !Number.isSafeInteger(count) || count < 1 || count > MAX_FILES
|
|
274
|
+
if (!["codex", "kimi", "pi", "qwen"].includes(kind) || !Number.isSafeInteger(bytes) || bytes < 1 || bytes > MAX_BYTES || !Number.isSafeInteger(count) || count < 1 || count > MAX_FILES
|
|
189
275
|
|| options.files !== undefined && (!Array.isArray(options.files) || options.files.length > MAX_FILES || options.files.some(file => typeof file !== "string" || !isAbsolute(file) || file.includes("\0")))
|
|
190
276
|
|| options.since !== undefined && !Number.isFinite(Date.parse(options.since))) { fail("source_bound"); return result; }
|
|
191
277
|
const since = options.since === undefined ? undefined : new Date(options.since).toISOString();
|
|
@@ -238,7 +324,9 @@ export async function ingestTranscript(kind: IngestKind, options: IngestOptions)
|
|
|
238
324
|
if (!cursor) { fail("cursor_busy", true); continue; }
|
|
239
325
|
let position = cursor.position;
|
|
240
326
|
const cutoff = since ?? "", broader = position.cutoff !== "" && (cutoff === "" || cutoff < position.cutoff);
|
|
241
|
-
if (position.identity !== identity || position.offset > initial.size || position.anchor !== anchor(fd, position.offset) || broader
|
|
327
|
+
if (position.identity !== identity || position.offset > initial.size || position.anchor !== anchor(fd, position.offset) || broader
|
|
328
|
+
|| kind === "qwen" && position.protocol !== QWEN_CONTRACT) position = { ...blank(), identity };
|
|
329
|
+
if (kind === "qwen") position.protocol = QWEN_CONTRACT;
|
|
242
330
|
position.cutoff = cutoff;
|
|
243
331
|
if (!position.anchor) position.anchor = anchor(fd, position.offset);
|
|
244
332
|
cursor.save(position); result.files++;
|
|
@@ -299,12 +387,15 @@ export async function ingestTranscript(kind: IngestKind, options: IngestOptions)
|
|
|
299
387
|
: kind === "pi" && raw.type === "session" ? raw : undefined;
|
|
300
388
|
const permitted = await policy.screen(raw);
|
|
301
389
|
guard(); metadataGuard();
|
|
302
|
-
|
|
390
|
+
const known = kind !== "qwen" || qwenRecord(raw);
|
|
391
|
+
if (kind === "qwen") position.scope = permitted && known ? scopeOf(raw.cwd) : "unknown";
|
|
392
|
+
else if (context !== undefined) position.scope = permitted && object(context) ? scopeOf(context.cwd) : "unknown";
|
|
303
393
|
else if (raw.cwd !== undefined) position.scope = permitted ? scopeOf(raw.cwd) : "unknown";
|
|
304
394
|
if (kind === "kimi" && raw.type === "metadata") {
|
|
305
395
|
position.protocol = typeof raw.protocol_version === "string" && ["1.0", "1.1", "1.2", "1.3", "1.4", "1.5"].includes(raw.protocol_version) ? raw.protocol_version : "unknown";
|
|
306
396
|
}
|
|
307
|
-
|
|
397
|
+
if (!known) fail("schema_or_time");
|
|
398
|
+
const input = known ? extract(kind, raw, project) : undefined;
|
|
308
399
|
if (input) {
|
|
309
400
|
const at = timeOf(raw, kind);
|
|
310
401
|
if (!at || position.scope === "unknown" || kind === "kimi" && !/^1\.[0-5]$/.test(position.protocol)) { fail("schema_or_time"); result.skipped++; }
|
package/src/mcp-memory.ts
CHANGED
|
@@ -10,7 +10,7 @@ import { join } from "node:path";
|
|
|
10
10
|
import type { McpHandlers, McpOptions } from "./mcp.ts";
|
|
11
11
|
import { CandidateStore, acceptedCorrectionText, candidateTargetRevision, type CandidateInput } from "./memory/candidates.ts";
|
|
12
12
|
import { capacityReport } from "./memory/capacity.ts";
|
|
13
|
-
import { CaptureStore } from "./memory/capture.ts";
|
|
13
|
+
import { CaptureStore, type CaptureRow } from "./memory/capture.ts";
|
|
14
14
|
import { hashJSON, readSelector } from "./memory/generation.ts";
|
|
15
15
|
import { memoryHome } from "./memory/home.ts";
|
|
16
16
|
import { canonicalPath, realPath } from "./memory/platform.ts";
|
|
@@ -18,7 +18,7 @@ import { createStandalonePolicy, type StandalonePolicy, type StandalonePolicyOpt
|
|
|
18
18
|
import { configuredPolicyOptions } from "./memory/config.ts";
|
|
19
19
|
import { memoryProjectKey } from "./memory/project.ts";
|
|
20
20
|
import { registerReaderLease } from "./memory/reader-lease.ts";
|
|
21
|
-
import { type ReadRef, type ReadWindow, SearchIndex } from "./memory/search-fts.ts";
|
|
21
|
+
import { type ReadRef, type ReadWindow, SearchIndex, ScreenedSearchIndex } from "./memory/search-fts.ts";
|
|
22
22
|
import { aliasedKeys, readShardedSelection, resolveAlias } from "./memory/shard.ts";
|
|
23
23
|
import { projectState } from "./memory/lifecycle.ts";
|
|
24
24
|
import { boundRepoRoot, readRepoScope, repoReadPolicy, type RepoView } from "./memory/repo-scope.ts";
|
|
@@ -28,9 +28,10 @@ import { WorkListStore, type WorkListFields } from "./memory/work-lists.ts";
|
|
|
28
28
|
|
|
29
29
|
// Graph retains its complete-corpus ceiling; search/timeline/show page matching bodies instead.
|
|
30
30
|
export const MCP_SCAN_LIMIT = 5000;
|
|
31
|
-
// ponytail:
|
|
31
|
+
// ponytail: evidence still ranks its newest 5000 permitted candidates; upgrade to the screened search stream if evidence needs global ranking.
|
|
32
32
|
const SEARCH_CANDIDATES = 5000;
|
|
33
33
|
const READ_PAGE = 256;
|
|
34
|
+
type SearchValue = { source: "curated"; evidence: true; view: View } | { source: "repo"; evidence: false; view: RepoView } | { source: "capture"; evidence: false; capture: CaptureRow };
|
|
34
35
|
export interface MemoryMcpOptions { cwd: string; base?: string; project?: string; policy?: StandalonePolicyOptions; allowed?: () => boolean; policyVersion?: () => string; assertCurrent?: () => void }
|
|
35
36
|
export interface MemoryMcp { tools: McpHandlers; close(): void; onEmitted: NonNullable<McpOptions["onEmitted"]> }
|
|
36
37
|
const unavailable = (): never => { throw new Error("memory operation unavailable"); };
|
|
@@ -109,19 +110,25 @@ export function createMemoryMcp(options: MemoryMcpOptions): MemoryMcp {
|
|
|
109
110
|
if (rows.length > MCP_SCAN_LIMIT) unavailable();
|
|
110
111
|
return screen(rows.filter(row => row.project === project && recallViewIsLive(row, store.readNow())));
|
|
111
112
|
};
|
|
112
|
-
const
|
|
113
|
-
options: Omit<ReadWindow, "project" | "limit" | "after">,
|
|
114
|
-
|
|
115
|
-
|
|
113
|
+
const permittedRows = async function* <T>(read: (window: ReadWindow) => { rows: T[]; next?: ReadRef; engine?: "fts5" | "substring" },
|
|
114
|
+
options: Omit<ReadWindow, "project" | "limit" | "after">, matches: (row: T, engine?: "fts5" | "substring") => boolean = () => true): AsyncGenerator<T> {
|
|
115
|
+
let after: ReadWindow["after"];
|
|
116
|
+
for (;;) {
|
|
116
117
|
guard();
|
|
117
118
|
const page = read({ ...options, project, limit: READ_PAGE, after });
|
|
118
|
-
for (const row of page.rows)
|
|
119
|
-
|
|
119
|
+
for (const row of page.rows) {
|
|
120
|
+
const allowed = await currentPolicy().screen(row); guard();
|
|
121
|
+
if (allowed && matches(row, page.engine)) yield row;
|
|
120
122
|
}
|
|
121
123
|
guard();
|
|
122
124
|
if (!page.next) break;
|
|
123
125
|
after = { position: page.next.position, id: page.next.id };
|
|
124
126
|
}
|
|
127
|
+
};
|
|
128
|
+
const readPermitted = async <T>(read: (window: ReadWindow) => { rows: T[]; next?: ReadRef; engine?: "fts5" | "substring" },
|
|
129
|
+
options: Omit<ReadWindow, "project" | "limit" | "after">, cap: number, matches: (row: T, engine?: "fts5" | "substring") => boolean = () => true): Promise<T[]> => {
|
|
130
|
+
const result: T[] = [];
|
|
131
|
+
for await (const row of permittedRows(read, options, matches)) { result.push(row); if (result.length === cap) break; }
|
|
125
132
|
return result;
|
|
126
133
|
};
|
|
127
134
|
const anchorAt = async (id: string): Promise<string | undefined> => {
|
|
@@ -183,38 +190,39 @@ export function createMemoryMcp(options: MemoryMcpOptions): MemoryMcp {
|
|
|
183
190
|
bound(args, ["query", "limit", "since", "includeCapture"]); const query = textOf(args, "query")!, limit = limitOf(args), since = sinceOf(args);
|
|
184
191
|
if (args.includeCapture !== undefined && typeof args.includeCapture !== "boolean") unavailable();
|
|
185
192
|
if (!await currentPolicy().screen({ query })) unavailable();
|
|
186
|
-
|
|
193
|
+
// ponytail: scan all matches into private disposable disk state, keeping one body page in JS.
|
|
194
|
+
// Work scales with matches; a policy-bound cache is the upgrade if this reaches the unchanged watchdog.
|
|
195
|
+
const index = new ScreenedSearchIndex<SearchValue>();
|
|
196
|
+
const releaseIndex = () => index.close(); closers.push(releaseIndex);
|
|
187
197
|
try {
|
|
188
198
|
const words = query.toLowerCase().split(/\s+/).filter(Boolean);
|
|
189
199
|
const captureState: { engine?: "fts5" | "substring" } = {};
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
200
|
+
let batch: { body: string; ref: string; tier: string; at: string; value: SearchValue; expiresAt?: string | null }[] = [];
|
|
201
|
+
const append = (value: SearchValue) => {
|
|
202
|
+
const row = value.source === "capture" ? value.capture : value.view;
|
|
203
|
+
batch.push(value.source === "capture"
|
|
204
|
+
? { value, ref: `capture:${value.capture.id}`, at: row.at, tier: "capture", body: `${value.capture.excerpt}\n${value.capture.paths.join(" ")}\n${value.capture.event}` }
|
|
205
|
+
: { value, ref: value.source === "repo" ? `repo:${value.view.id}` : value.view.id, at: row.at, tier: value.source === "repo" ? "long" : value.view.tier, body: `${value.view.title ?? ""}\n${value.view.text}\n${value.view.about.join(" ")}`, expiresAt: value.source === "curated" ? value.view.expiresAt : null });
|
|
206
|
+
if (batch.length === READ_PAGE) { index.append(batch); batch = []; }
|
|
207
|
+
};
|
|
208
|
+
if (args.includeCapture !== false) {
|
|
209
|
+
for await (const row of permittedRows(window => {
|
|
210
|
+
const page = capture.readPage({ ...window, query, ...(captureState.engine === "substring" ? { match: "substring" as const } : {}) });
|
|
211
|
+
captureState.engine ??= page.engine; return page;
|
|
212
|
+
}, { since, order: "id" }, (row, engine) => index.available && engine === "fts5" || words.every(word => `${row.excerpt}\n${row.paths.join(" ")}\n${row.event}`.toLowerCase().includes(word)))) {
|
|
213
|
+
append({ source: "capture", evidence: false, capture: row });
|
|
214
|
+
}
|
|
215
|
+
}
|
|
196
216
|
const useFts = index.available && captureState.engine !== "substring";
|
|
197
|
-
const
|
|
198
|
-
view => recallViewIsLive(view, store.readNow()) && (useFts || words.every(word => `${view.title ?? ""}\n${view.text}\n${view.about.join(" ")}`.toLowerCase().includes(word))));
|
|
217
|
+
for await (const view of permittedRows(window => store.readPage({ ...window, ...(useFts ? { query } : {}) }), { since, order: "id" },
|
|
218
|
+
view => recallViewIsLive(view, store.readNow()) && (useFts || words.every(word => `${view.title ?? ""}\n${view.text}\n${view.about.join(" ")}`.toLowerCase().includes(word))))) append({ source: "curated", evidence: true, view });
|
|
199
219
|
const repoNotes = (await repoRows()).filter(view => (!since || Date.parse(view.at) >= Date.parse(since)) && (useFts || words.every(word => `${view.title ?? ""}\n${view.text}\n${view.about.join(" ")}`.toLowerCase().includes(word))));
|
|
200
|
-
const
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
if (useFts) {
|
|
204
|
-
index.resync(() => hits.map(hit => ({ id: hit.ref, at: hit.at, project, tier: hit.tier, scope: "project", body: hit.body })));
|
|
205
|
-
if (index.needsResync(hits.length)) unavailable();
|
|
206
|
-
}
|
|
207
|
-
const ranks = new Map((useFts ? index.query(query, MCP_SCAN_LIMIT * 2, project) : []).map(row => [row.ref, row.rank]));
|
|
208
|
-
const selected = hits.flatMap(hit => {
|
|
209
|
-
const rank = ranks.get(hit.ref);
|
|
210
|
-
if (useFts ? rank === undefined : !words.every(word => hit.body.toLowerCase().includes(word))) return [];
|
|
211
|
-
const weight = hit.tier === "long" ? 1 : hit.tier === "capture" ? 0.5 : 0.9;
|
|
212
|
-
const score = -(rank ?? -1) * weight / (1 + Math.max(0, Date.now() - Date.parse(hit.at)) / (30 * 86400_000));
|
|
213
|
-
return [{ source: hit.source, evidence: hit.evidence, ...(hit.source === "capture" ? { capture: hit.capture } : { view: hit.view }), score }];
|
|
214
|
-
}).sort((a, b) => b.score - a.score).slice(0, limit);
|
|
220
|
+
for (const view of repoNotes) append({ source: "repo", evidence: false, view });
|
|
221
|
+
if (batch.length) index.append(batch);
|
|
222
|
+
const selected = index.best(query, limit, store.readNow(), useFts).map(({ value, score }) => ({ ...value, score }));
|
|
215
223
|
const probe = stale(), result = { engine: useFts ? "fts5" : "substring", hits: selected.map(hit => hit.source === "curated" && "view" in hit && hit.view ? { ...hit, view: decorate(hit.view as View, probe) } : hit) };
|
|
216
224
|
guard(); return observed(result, selected.flatMap(hit => hit.source === "curated" && "view" in hit && hit.view ? [hit.view.id] : []), selected.map(hit => "view" in hit ? hit.view : hit.capture));
|
|
217
|
-
} finally {
|
|
225
|
+
} finally { const position = closers.indexOf(releaseIndex); if (position >= 0) closers.splice(position, 1); releaseIndex(); }
|
|
218
226
|
},
|
|
219
227
|
async amem_show(args) {
|
|
220
228
|
bound(args, ["id"]); guard(); const id = textOf(args, "id")!;
|