@staix/agent-mem 0.4.0 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -2,7 +2,7 @@
2
2
 
3
3
  Curated memory with append-only JSONL logs, a derived SQLite index, short- and long-term tiers, a typed graph, goals with leases, and git sync between machines. Extracted from [agent-hub](https://github.com/STAIxBWLB/agent-hub), with compatible storage and entry IDs.
4
4
 
5
- Version: 0.4.0. Requires Bun 1.4.2 or later; no runtime dependencies.
5
+ Version: 0.5.0. Requires Bun 1.4.2 or later; no runtime dependencies.
6
6
 
7
7
  ## Install
8
8
 
@@ -135,7 +135,7 @@ The JSONL segments are authoritative. `index.db` is derived and can be rebuilt.
135
135
 
136
136
  From issue #13 (phase P1), the package carries a second, machine-local tier beside the curated store, born sharded (#15 T1): one capture shard per project key at `projects/<key-hash>/machine/<machine-id>/` under the memory home, with its own locks so two projects never contend. The capture tier records deterministic tool-event rows (project, agent, event, tool, paths, exit status and a bounded, secret-screened excerpt) in the shard's `capture.db`. It is never synced, never read as curated evidence, and never writes a curated log, so its maintenance can never evict notes, goals or work lists. Capture is bounded by a soft size cap and a retention window (defaults 512 MB per shard, 90 days), resolved at every invocation from the library options, then the shard's `config.json`, then the home default `capture-config.json`; eviction removes the oldest capture rows first in bounded batches. The 2026-10-17 owner decision keeps the P1 cap per shard; the machine-wide total is #15 T5. On lock contention a writer appends to an append-only spool and exits fast; the next uncontended writer replays the spool idempotently. The capture path makes no network or model call. The pre-T1 machine-wide `capture/` (an unreleased layout of this branch) is migrated per project on open and removed once empty.
137
137
 
138
- Project identity (#15 D2) keeps `memoryProjectKey`: a worktree, a submodule and the main checkout of one repository share its key; two unrelated same-named repositories without a remote get distinct path-hash keys, flagged `local` in the shard's `project.json` and the catalog. A no-remote key hashes the main working root (using the actual top-level for a separate git dir), so keys written from the repository root keep their pre-#15 value. Known limit: a linked worktree of a no-remote separate-git-dir repository gets its own key, because git does not record that repository's main working root (probe: `alpha` and `alpha/sub` share `alpha@<hash>`, `alpha-wt` gets `alpha-wt@<hash>`); the key stays as is for root-key compatibility, and a reviewed alias reconciles its capture. Keys written by older builds from a subdirectory or worktree still used that invocation's cwd hash; a person-reviewed alias maps such a key to its canonical one for capture, work lists and the CLI and MCP project binding. Curated rows written under the old key keep it and are not remapped, so the CLI and MCP stop showing that key's project-scoped notes, goals and candidates (the library still reads them by the raw key; shared rows stay visible). An alias can be retargeted but not removed, so add one only after checking the old key's private notes; curated row reconciliation is a later spec change. `amem alias add <from-key> <to-key>` appends an op to `aliases.jsonl` (latest wins, never chained, never automatic), `amem alias list` shows the ops. `catalog.db` in the memory home (#15 D3's storage part) is derived only from the shards and the alias log and rebuilds to the same rows; the D3 management commands are #15 T5 and not built.
138
+ Project identity (#15 D2) keeps `memoryProjectKey`: a worktree, a submodule and the main checkout of one repository share its key; two unrelated same-named repositories without a remote get distinct path-hash keys, flagged `local` in the shard's `project.json` and the catalog. A no-remote key hashes the main working root (using the actual top-level for a separate git dir), so keys written from the repository root keep their pre-#15 value. Known limit: a linked worktree of a no-remote separate-git-dir repository gets its own key, because git does not record that repository's main working root (probe: `alpha` and `alpha/sub` share `alpha@<hash>`, `alpha-wt` gets `alpha-wt@<hash>`); the key stays as is for root-key compatibility, and a reviewed alias reconciles its capture. Keys written by older builds from a subdirectory or worktree still used that invocation's cwd hash; a person-reviewed alias maps such a key to its canonical one for capture, work lists and the CLI and MCP project binding. Curated rows written under the old key keep their key in the log (nothing is rewritten), and the CLI and MCP read them under the canonical project: the project filter of notes, goals, candidates, vault candidates and work lists covers the canonical key and every key aliased to it, and presents those rows under the canonical key (#27). The library `MemoryStore` still reads by the raw key unless a host passes `reconcile`, and shared rows stay visible as before. Writes still go to the key's own log and shard, and a settle or forget of an aliased row lands in the shard that holds it. `amem alias add <from-key> <to-key>` appends an op to `aliases.jsonl` (latest wins, never chained, never automatic), `amem alias remove <from-key>` appends a removal (the old key reads under itself again, the history stays), `amem alias list` shows the ops including removals. Both are person-only and need a terminal. An amem or hub older than this change ignores a removal and keeps the old binding until it is upgraded, so upgrade every host that shares the home first. A correction accepted on an aliased row lives under the canonical key, so after a removal the old key shows the superseded note again (see `docs/contracts.md`). `catalog.db` in the memory home (#15 D3's storage part) is derived only from the shards and the alias log and rebuilds to the same rows; the D3 management commands are #15 T5 and not built.
139
139
 
140
140
  `store.search(query, filter)` searches the curated tier and the current project's capture shard with SQLite FTS5 and BM25 plus tier/recency weighting: natural multi-word queries match partial term sets (the substring filter required every word), and a curated row outranks the same text in the capture tier. Prefix terms match inflections and Korean particles in either direction by retaining the original query term and adding a common-particle-stripped alternative. The FTS copy of the curated rows lives in its own derived `search.db` (`index.db`'s table set is a frozen contract and gains nothing); it is written from the same apply path as the index, and a prune, crash or rebuild is detected and resynced, so the incremental FTS copy equals a full resync. Where FTS5 is unavailable the engine falls back to the substring filter and reports `engine: "substring"`; capture recall needs FTS5 and is skipped in the fallback. The library entry exports `CaptureStore`, `captureSearch`, `readCaptureConfig`, `writeCaptureConfig`, `SearchIndex`, `fts5Available`, the shard helpers (`resolveAlias`, `appendAlias`, `listAliases`, `shardKeyHash`), the catalog (`rebuildCatalog`, `readCatalog`, `scanCatalog`) and the capacity advisor (`capacityReport`).
141
141
 
@@ -238,6 +238,8 @@ A native shard crash tail is left to its original writer's recovery. Migration c
238
238
 
239
239
  Run `amem mcp` as a stdio child of the agent CLI. It binds the working directory, canonical project key, exact memory home and selected profile once, accepts no project/home overrides in tool arguments, and closes all stores on stdin EOF. It has no listener, daemon, watcher or model call. Ordinary person commands retain their agent-marker refusal and mutation TTY checks; `mcp` is the sole agent entry point.
240
240
 
241
+ MCP search, timeline and show remain available above 5,000 rows. Reads page exact-project candidates and screen them before result selection. Search ranks the newest 5,000 permitted matching candidates per tier; `since` narrows that window. Timeline can anchor an older row directly. Capture MCP indexing is lazy on queried reads, never hook construction or append. Index contention uses a call-pinned substring fallback; later searches retry. See [MCP candidate reads](docs/contracts.md#mcp-candidate-reads-13-task-209) for the bounded-window and fallback contract.
242
+
241
243
  The frozen nine-tool contract for dotfiles#255 and maru#458:
242
244
 
243
245
  | Tool | Purpose |
@@ -252,7 +254,7 @@ The frozen nine-tool contract for dotfiles#255 and maru#458:
252
254
  | `amem_vault_candidates` | Read-only screened durable curated notes |
253
255
  | `amem_capacity` | Numeric machine capacity proposal; no apply capability |
254
256
 
255
- Search/show/timeline label capture as recall with `evidence: false`; capture never enters the curated graph. Policy screens complete rows before selection, limits or graph construction. The standalone secret screen, private flags, PII patterns and deny paths compose with a host's stricter policy through `createMemoryMcp({cwd, base, policy})`. Screening errors fail closed. `base` is the exact memory root; historical library `home` options still name its parent. Scope and profile changes refuse further access instead of silently reopening another store. Reads refuse above 5,000 rows per curated/capture tier; work-list reads use the same 5,000-entry scan ceiling. Count-first admission refuses an oversized work-list corpus before loading or parsing its fields, rather than returning an incomplete screened result. These are operational ceilings, not exposed counts of unscreened data. Graphs cover curated-note relationships, not goal/suggestion tables.
257
+ Search/show/timeline label capture as recall with `evidence: false`; capture never enters the curated graph. Policy screens complete rows before selection, limits or graph construction. The standalone secret screen, private flags, PII patterns and deny paths compose with a host's stricter policy through `createMemoryMcp({cwd, base, policy})`. Screening errors fail closed. `base` is the exact memory root; historical library `home` options still name its parent. Scope and profile changes refuse further access instead of silently reopening another store. Graph reads retain the 5,000-row curated ceiling; work-list reads stream the complete ordered history without an operation-count ceiling, applying policy before latest-value reduction. The working state scales with the current result, not historical payloads. These are operational ceilings, not exposed counts of unscreened data. Graphs cover curated-note relationships, not goal/suggestion tables.
256
258
 
257
259
  `WorkListStore` writes `projects/<key-hash>/work/` with its own append log, derived index and lock. Scalar fields reduce by latest value, null clears, list status `done` closes, and item status is `todo|doing|done|dropped`. Work lists are outside generation selection and capture eviction. `amem import claude-mem` additionally imports `work_state_entries` only through persisted person-reviewed D2 aliases, preserving source DB bytes and stable replay receipts; JSON keeps observation fields and adds `work` counts/cursor.
258
260
 
@@ -262,7 +264,7 @@ Do not record vault links on a home shared with a host using published agent-mem
262
264
 
263
265
  General memory candidates are separate from curated evidence. `amem review accept <id> --kind memory --policy-free` remains person-only and rechecks the current standalone privacy policy before the accepted note append. Machine capacity reports numeric aggregate use across shards, as R7 requires; it does not return foreign project names or text.
264
266
 
265
- `bun scripts/measure-mcp-budget.ts` exercises an actual stdio session with 200 curated notes, 2,000 capture rows and 2,000 work-list operations reduced into 100 items. It checks the R3 idle RSS limit of 100 MB, peak RSS limit of 200 MB across searches and work-list reads, and child reaping after EOF. CI runs the harness on macOS and Linux. All harness data is synthetic; live P6 measurements remain separate.
267
+ `bun scripts/measure-mcp-budget.ts` exercises an actual stdio session with 5,001 curated notes, 5,001 capture rows and 10,001 work-list operations reduced into 100 items. It checks the R3 idle RSS limit of 100 MB, peak RSS limit of 200 MB across searches, latest/anchored timelines and work-list reads, and child reaping after EOF. CI runs the harness on macOS and Linux. All harness data is synthetic; live P6 measurements remain separate.
266
268
 
267
269
  ## Development
268
270
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@staix/agent-mem",
3
- "version": "0.4.0",
3
+ "version": "0.5.0",
4
4
  "description": "Curated, shared memory for coding agents: append-only logs, a derived index, tiers and a typed graph.",
5
5
  "license": "MIT",
6
6
  "type": "module",
package/src/cli.ts CHANGED
@@ -9,7 +9,7 @@ import { MemoryStore, looksSecret, machineId, openShardStore, type StoreOptions
9
9
  import { rawMemoryCommand, rawGoalCommand } from "./memory/commands.ts";
10
10
  import { memoryProjectKey } from "./memory/project.ts";
11
11
  import { CandidateStore, type CandidateFilter } from "./memory/candidates.ts";
12
- import { appendAlias, listAliases, readShardedSelection, resolveAlias } from "./memory/shard.ts";
12
+ import { aliasedKeys, appendAlias, appendUnalias, listAliases, readShardedSelection, resolveAlias } from "./memory/shard.ts";
13
13
  import { capacityReport, capacityText } from "./memory/capacity.ts";
14
14
  import { readCaptureConfigFile, writeCaptureConfigFile, type CaptureConfig } from "./memory/capture.ts";
15
15
 
@@ -70,12 +70,15 @@ function shardStatePresent(): boolean {
70
70
  catch (error) { if ((error as NodeJS.ErrnoException).code === "ENOENT") return false; throw error; }
71
71
  }
72
72
 
73
- function curatedStore(project: string, opts: StoreOptions): MemoryStore {
73
+ /** `reconciled` (#27): the project filter also covers every key aliased to `project`; prune and other planners keep the exact key. */
74
+ function curatedStore(project: string, opts: StoreOptions, reconciled = true): MemoryStore {
75
+ const aliases = reconciled ? aliasedKeys(memoryHome(), project) : [];
76
+ const withAliases: StoreOptions = aliases.length ? { ...opts, reconcile: { canonical: project, aliases } } : opts;
74
77
  if (shardStatePresent()) {
75
78
  readShardedSelection(memoryHome());
76
- return openShardStore({ ...opts, project });
79
+ return openShardStore({ ...withAliases, project });
77
80
  }
78
- return new MemoryStore(opts);
81
+ return new MemoryStore(withAliases);
79
82
  }
80
83
 
81
84
  export async function runCli(argv: readonly string[], context: CliContext = {}): Promise<string> {
@@ -136,7 +139,7 @@ export async function runCli(argv: readonly string[], context: CliContext = {}):
136
139
  }
137
140
  const store = options["--user"] === true
138
141
  ? openShardStore({ memoryDir: memoryHome(), user: true, deferPromotions: true })
139
- : curatedStore(project, { deferPromotions: true });
142
+ : curatedStore(project, { deferPromotions: true }, false);
140
143
  try {
141
144
  const auth = { project, actor: "person" as const, authorize: () => true };
142
145
  return JSON.stringify(applying ? applyPrune(store.prunePort(), input, auth) : planPrune(store.prunePort(), { cutoff: String(options["--cutoff"]) }, auth), null, 2);
@@ -148,7 +151,7 @@ export async function runCli(argv: readonly string[], context: CliContext = {}):
148
151
  if (sub === "list" && options["--kind"] !== "memory" && resolveProfile(memoryHome()).kind === "generation") throw new Error("candidate review is not yet integrated with this generation; select --kind memory for the standalone memory candidate surface");
149
152
  const id = parsed.positionals[0];
150
153
  if (sub !== "list" && (!id || !/^[a-f0-9]{64}$/.test(id))) throw new Error("review accept/reject requires a full 64-character candidate id");
151
- const project = boundProject(), candidates = new CandidateStore();
154
+ const project = boundProject(), aliases = aliasedKeys(memoryHome(), project), candidates = new CandidateStore(aliases.length ? { reconcile: { canonical: project, aliases } } : {});
152
155
  try {
153
156
  if (sub === "list") {
154
157
  const rows = candidates.list({ project, ...(options["--status"] ? { status: options["--status"] as CandidateFilter["status"] } : {}),
@@ -219,8 +222,15 @@ export async function runCli(argv: readonly string[], context: CliContext = {}):
219
222
  const op = appendAlias(memoryHome(), { alias: from, key: to, actor: "person", machine: machineId(memoryHome()).id });
220
223
  return options["--json"] ? JSON.stringify(op, null, 2) : `alias ${op.alias} -> ${op.key}`;
221
224
  }
225
+ if (parsed.path[1] === "remove") {
226
+ // #27 option 2: person only (ordinary CLI refusal of agent markers above, a TTY through `mutates`), append only, the latest op wins.
227
+ const [from] = parsed.positionals;
228
+ if (!from || parsed.positionals.length !== 1) throw new Error("usage: amem alias remove <alias-key>");
229
+ const op = appendUnalias(memoryHome(), { alias: from, actor: "person", machine: machineId(memoryHome()).id });
230
+ return options["--json"] ? JSON.stringify(op, null, 2) : `alias ${op.alias} removed`;
231
+ }
222
232
  const ops = listAliases(memoryHome());
223
- return options["--json"] ? JSON.stringify(ops, null, 2) : ops.length ? ops.map((op) => `${op.at} ${op.alias} -> ${op.key} (${op.actor})`).join("\n") : "no aliases recorded";
233
+ return options["--json"] ? JSON.stringify(ops, null, 2) : ops.length ? ops.map((op) => op.op === "alias" ? `${op.at} ${op.alias} -> ${op.key} (${op.actor})` : `${op.at} ${op.alias} removed (${op.actor})`).join("\n") : "no aliases recorded";
224
234
  }
225
235
  // ponytail: board task links need the host's authenticated task boundary; add a host bridge before supporting them here.
226
236
  if (goal && sub === "link") throw new Error("goal link needs host integration; use ahub goal link");
@@ -75,6 +75,7 @@ export const COMMANDS: Record<string, CommandDefinition> = {
75
75
  alias: { description: "Person-reviewed project key aliases (#15 D2)", options: json, defaultCommand: "list", commands: {
76
76
  list: { description: "List alias ops" },
77
77
  add: { description: "Map one key onto its canonical key (append-only op)", args: [{ name: "from-key" }, { name: "to-key" }], mutates: true },
78
+ remove: { description: "Remove an alias so the key resolves to itself again (append-only op, the latest wins)", args: [{ name: "alias-key" }], mutates: true },
78
79
  } },
79
80
  completion: { description: "Print static shell completion", args: [{ name: "shell", values: ["zsh", "bash", "fish"] }] },
80
81
  help: { description: "Show help" }, version: { description: "Show version" },
package/src/mcp-memory.ts CHANGED
@@ -10,14 +10,17 @@ import { canonicalPath, realPath } from "./memory/platform.ts";
10
10
  import { createStandalonePolicy, type StandalonePolicyOptions } from "./memory/policy.ts";
11
11
  import { memoryProjectKey } from "./memory/project.ts";
12
12
  import { registerReaderLease } from "./memory/reader-lease.ts";
13
- import { SearchIndex } from "./memory/search-fts.ts";
14
- import { readShardedSelection, resolveAlias } from "./memory/shard.ts";
13
+ import { type ReadRef, type ReadWindow, SearchIndex } from "./memory/search-fts.ts";
14
+ import { aliasedKeys, readShardedSelection, resolveAlias } from "./memory/shard.ts";
15
15
  import { MemoryStore, openShardStore, type Graph, type View } from "./memory/store.ts";
16
16
  import { vaultCandidates } from "./memory/vault.ts";
17
17
  import { WorkListStore, type WorkListFields } from "./memory/work-lists.ts";
18
18
 
19
- // ponytail: 5000 rows per tier per operation; above this ceiling refuse rather than select on an incomplete privacy-filtered corpus. Upgrade to screened streaming pagination.
19
+ // Graph retains its complete-corpus ceiling; search/timeline/show page matching bodies instead.
20
20
  export const MCP_SCAN_LIMIT = 5000;
21
+ // ponytail: rank the newest 5000 permitted matching candidates per tier; use since to narrow or upgrade to a policy-bound persistent index for global ranking.
22
+ const SEARCH_CANDIDATES = 5000;
23
+ const READ_PAGE = 256;
21
24
  export interface MemoryMcpOptions { cwd: string; base?: string; project?: string; policy?: StandalonePolicyOptions }
22
25
  export interface MemoryMcp { tools: McpHandlers; close(): void }
23
26
  const unavailable = (): never => { throw new Error("memory operation unavailable"); };
@@ -27,7 +30,7 @@ const textOf = (args: Record<string, unknown>, key: string, optional = false): s
27
30
  const value = args[key]; if (value === undefined && optional) return undefined;
28
31
  return typeof value === "string" && value.trim() && !value.includes("\0") ? value : unavailable();
29
32
  };
30
- const sinceOf = (args: Record<string, unknown>): string | undefined => { const since = textOf(args, "since", true); if (since !== undefined && !Number.isFinite(Date.parse(since))) unavailable(); return since; };
33
+ const sinceOf = (args: Record<string, unknown>): string | undefined => { const since = textOf(args, "since", true); if (since !== undefined && !Number.isFinite(Date.parse(since))) unavailable(); return since === undefined ? undefined : new Date(since).toISOString(); };
31
34
 
32
35
  /** All identities are resolved once. Per-call arguments cannot change projects, homes, profiles or policy. */
33
36
  export function createMemoryMcp(options: MemoryMcpOptions): MemoryMcp {
@@ -40,15 +43,17 @@ export function createMemoryMcp(options: MemoryMcpOptions): MemoryMcp {
40
43
  const selection = readSelector(base).revision;
41
44
  const sharded = !!lstatSync(join(base, "shards.json"), { throwIfNoEntry: false });
42
45
  const shardSelection = sharded ? JSON.stringify(readShardedSelection(base)) : undefined;
43
- const store = sharded ? openShardStore({ memoryDir: base, project, deferPromotions: true }) : new MemoryStore({ memoryDir: base, deferPromotions: true });
46
+ // #27: the project filter covers the canonical key and every key aliased to it; the keys are fixed with the project, so a later alias change fails the guard below.
47
+ const aliases = aliasedKeys(base, project), reconcile = aliases.length ? { canonical: project, aliases } : undefined;
48
+ const store = sharded ? openShardStore({ memoryDir: base, project, deferPromotions: true, ...(reconcile ? { reconcile } : {}) }) : new MemoryStore({ memoryDir: base, deferPromotions: true, ...(reconcile ? { reconcile } : {}) });
44
49
  closers.push(() => store.close());
45
50
  const capture = new CaptureStore({ memoryDir: base, project }); closers.push(() => capture.close());
46
- const candidates = new CandidateStore({ memoryDir: base }); closers.push(() => candidates.close());
47
- const work = new WorkListStore({ base, project, policy: async row => { const permitted = await policy.screen(row); guard(); return permitted; } }); closers.push(() => work.close());
51
+ const candidates = new CandidateStore({ memoryDir: base, ...(reconcile ? { reconcile } : {}) }); closers.push(() => candidates.close());
52
+ const work = new WorkListStore({ base, project, aliases, policy: async row => { const permitted = await policy.screen(row); guard(); return permitted; } }); closers.push(() => work.close());
48
53
  const guard = () => {
49
54
  if (closed) unavailable();
50
55
  release.verify();
51
- if (resolveAlias(base, requested) !== project || readSelector(base).revision !== selection) unavailable();
56
+ if (resolveAlias(base, requested) !== project || aliasedKeys(base, project).join("\n") !== aliases.join("\n") || readSelector(base).revision !== selection) unavailable();
52
57
  const split = !!lstatSync(join(base, "shards.json"), { throwIfNoEntry: false });
53
58
  if (split !== sharded || split && JSON.stringify(readShardedSelection(base)) !== shardSelection) unavailable();
54
59
  };
@@ -63,46 +68,84 @@ export function createMemoryMcp(options: MemoryMcpOptions): MemoryMcp {
63
68
  if (rows.length > MCP_SCAN_LIMIT) unavailable();
64
69
  return screen(rows.filter(row => row.project === project));
65
70
  };
66
- const captures = async () => screen(capture.rows({ project, limit: MCP_SCAN_LIMIT + 1 }));
67
- const find = (rows: View[], id: string): View | null => { const found = rows.filter(row => row.id === id || row.id.startsWith(id.replace(/^mem:/, ""))); return found.length === 1 ? found[0]! : null; };
71
+ const readPermitted = async <T>(read: (window: ReadWindow) => { rows: T[]; next?: ReadRef; engine?: "fts5" | "substring" },
72
+ options: Omit<ReadWindow, "project" | "limit" | "after">, cap: number, matches: (row: T, engine?: "fts5" | "substring") => boolean = () => true): Promise<T[]> => {
73
+ const result: T[] = []; let after: ReadWindow["after"];
74
+ while (result.length < cap) {
75
+ guard();
76
+ const page = read({ ...options, project, limit: READ_PAGE, after });
77
+ for (const row of page.rows) if (await policy.screen(row) && matches(row, page.engine)) {
78
+ result.push(row); if (result.length === cap) break;
79
+ }
80
+ guard();
81
+ if (!page.next) break;
82
+ after = { position: page.next.position, id: page.next.id };
83
+ }
84
+ return result;
85
+ };
86
+ const anchorAt = async (id: string): Promise<string | undefined> => {
87
+ if (/^capture:[1-9][0-9]*$/.test(id)) {
88
+ return (await readPermitted(window => capture.readPage(window), { id: id.slice(8) }, 1))[0]?.at;
89
+ }
90
+ const rows = await readPermitted(window => store.readPage({ ...window, includeSuperseded: true }), { prefix: id.replace(/^mem:/, "") }, 2);
91
+ return rows.length === 1 ? rows[0]!.at : undefined;
92
+ };
68
93
  const tools: McpHandlers = {
69
94
  async amem_search(args) {
70
95
  bound(args, ["query", "limit", "since", "includeCapture"]); const query = textOf(args, "query")!, limit = limitOf(args), since = sinceOf(args);
71
96
  if (args.includeCapture !== undefined && typeof args.includeCapture !== "boolean") unavailable();
72
97
  if (!await policy.screen({ query })) unavailable();
73
- const notes = await curated(), recalls = args.includeCapture === false ? [] : await captures();
74
- const hits = [...notes.map(view => ({ source: "curated" as const, evidence: true, view, ref: view.id, at: view.at, tier: view.tier, body: `${view.title ?? ""}\n${view.text}\n${view.about.join(" ")}` })),
75
- ...recalls.map(row => ({ source: "capture" as const, evidence: false, capture: row, ref: `capture:${row.id}`, at: row.at, tier: "capture", body: `${row.excerpt}\n${row.paths.join(" ")}\n${row.event}` }))].filter(hit => !since || Date.parse(hit.at) >= Date.parse(since));
76
98
  const index = new SearchIndex(":memory:");
77
99
  try {
78
- index.resync(() => hits.map(hit => ({ id: hit.ref, at: hit.at, project, tier: hit.tier, scope: "project", body: hit.body })));
79
- if (index.available && index.needsResync(hits.length)) unavailable();
80
- const ranks = new Map(index.query(query, MCP_SCAN_LIMIT * 2, project).map(row => [row.ref, row.rank]));
81
100
  const words = query.toLowerCase().split(/\s+/).filter(Boolean);
101
+ const captureState: { engine?: "fts5" | "substring" } = {};
102
+ const recalls = args.includeCapture === false ? [] : await readPermitted(window => {
103
+ const page = capture.readPage({ ...window, query, ...(captureState.engine === "substring" ? { match: "substring" as const } : {}) });
104
+ captureState.engine ??= page.engine;
105
+ return page;
106
+ }, { since }, SEARCH_CANDIDATES,
107
+ (row, engine) => index.available && engine === "fts5" || words.every(word => `${row.excerpt}\n${row.paths.join(" ")}\n${row.event}`.toLowerCase().includes(word)));
108
+ const useFts = index.available && captureState.engine !== "substring";
109
+ const notes = await readPermitted(window => store.readPage({ ...window, ...(useFts ? { query } : {}) }), { since }, SEARCH_CANDIDATES,
110
+ view => useFts || words.every(word => `${view.title ?? ""}\n${view.text}\n${view.about.join(" ")}`.toLowerCase().includes(word)));
111
+ const hits = [...notes.map(view => ({ source: "curated" as const, evidence: true, view, ref: view.id, at: view.at, tier: view.tier, body: `${view.title ?? ""}\n${view.text}\n${view.about.join(" ")}` })),
112
+ ...recalls.map(row => ({ source: "capture" as const, evidence: false, capture: row, ref: `capture:${row.id}`, at: row.at, tier: "capture", body: `${row.excerpt}\n${row.paths.join(" ")}\n${row.event}` }))].filter(hit => !since || Date.parse(hit.at) >= Date.parse(since));
113
+ if (useFts) {
114
+ index.resync(() => hits.map(hit => ({ id: hit.ref, at: hit.at, project, tier: hit.tier, scope: "project", body: hit.body })));
115
+ if (index.needsResync(hits.length)) unavailable();
116
+ }
117
+ const ranks = new Map((useFts ? index.query(query, MCP_SCAN_LIMIT * 2, project) : []).map(row => [row.ref, row.rank]));
82
118
  const selected = hits.flatMap(hit => {
83
119
  const rank = ranks.get(hit.ref);
84
- if (index.available ? rank === undefined : !words.every(word => hit.body.toLowerCase().includes(word))) return [];
120
+ if (useFts ? rank === undefined : !words.every(word => hit.body.toLowerCase().includes(word))) return [];
85
121
  const weight = hit.tier === "long" ? 1 : hit.tier === "capture" ? 0.5 : 0.9;
86
122
  const score = -(rank ?? -1) * weight / (1 + Math.max(0, Date.now() - Date.parse(hit.at)) / (30 * 86400_000));
87
123
  return [{ source: hit.source, evidence: hit.evidence, ...(hit.source === "curated" ? { view: hit.view } : { capture: hit.capture }), score }];
88
124
  }).sort((a, b) => b.score - a.score).slice(0, limit);
89
- guard(); return { engine: index.available ? "fts5" : "substring", hits: selected };
125
+ guard(); return { engine: useFts ? "fts5" : "substring", hits: selected };
90
126
  } finally { index.close(); }
91
127
  },
92
128
  async amem_show(args) {
93
129
  bound(args, ["id"]); guard(); const id = textOf(args, "id")!;
94
- if (/^capture:[1-9][0-9]*$/.test(id)) { const row = (await captures()).find(row => `capture:${row.id}` === id); guard(); return row ? { source: "capture", evidence: false, capture: row } : null; }
95
- const result = find(await curated(true), id); guard(); return result;
130
+ if (/^capture:[1-9][0-9]*$/.test(id)) {
131
+ const row = (await readPermitted(window => capture.readPage(window), { id: id.slice(8) }, 1))[0];
132
+ guard(); return row ? { source: "capture", evidence: false, capture: row } : null;
133
+ }
134
+ const rows = await readPermitted(window => store.readPage({ ...window, includeSuperseded: true }), { prefix: id.replace(/^mem:/, "") }, 2);
135
+ guard(); return rows.length === 1 ? rows[0]! : null;
96
136
  },
97
137
  async amem_timeline(args) {
98
- bound(args, ["id", "since", "limit"]); const rows = await curated(true), recalls = await captures(), id = textOf(args, "id", true), since = sinceOf(args), limit = limitOf(args);
99
- const entries = [...rows.map(row => ({ at: row.at, value: row as unknown })), ...recalls.map(row => ({ at: row.at, value: { source: "capture", evidence: false, capture: row } as unknown }))];
100
- let selected = entries.filter(row => !since || Date.parse(row.at) >= Date.parse(since));
101
- if (id) {
102
- const anchor = id.startsWith("capture:") ? recalls.find(row => `capture:${row.id}` === id) : find(rows, id);
103
- if (!anchor) return []; selected = selected.sort((a, b) => Math.abs(Date.parse(a.at) - Date.parse(anchor.at)) - Math.abs(Date.parse(b.at) - Date.parse(anchor.at)));
104
- } else selected.sort((a, b) => b.at.localeCompare(a.at));
105
- guard(); return selected.slice(0, limit).map(row => row.value);
138
+ bound(args, ["id", "since", "limit"]);
139
+ const id = textOf(args, "id", true), since = sinceOf(args), limit = limitOf(args);
140
+ guard();
141
+ const anchor = id ? await anchorAt(id) : undefined;
142
+ if (id && !anchor) { guard(); return []; }
143
+ const rows = await readPermitted(window => store.readPage({ ...window, includeSuperseded: true }), { since, anchor }, limit);
144
+ const recalls = await readPermitted(window => capture.readPage(window), { since, anchor }, limit);
145
+ const entries = [...rows.map(row => ({ at: row.at, ref: row.id, value: row as unknown })),
146
+ ...recalls.map(row => ({ at: row.at, ref: `capture:${row.id}`, value: { source: "capture", evidence: false, capture: row } as unknown }))];
147
+ entries.sort((a, b) => (anchor ? Math.abs(Date.parse(a.at) - Date.parse(anchor)) - Math.abs(Date.parse(b.at) - Date.parse(anchor)) : Date.parse(b.at) - Date.parse(a.at)) || (a.ref < b.ref ? -1 : a.ref > b.ref ? 1 : 0));
148
+ guard(); return entries.slice(0, limit).map(row => row.value);
106
149
  },
107
150
  async amem_graph(args) {
108
151
  bound(args, ["start", "hops", "limit"]); const start = textOf(args, "start")!, cap = limitOf(args), hops = args.hops ?? 1;
@@ -76,7 +76,11 @@ export class CandidateStore {
76
76
  readonly path: string;
77
77
  private readonly db: Database;
78
78
  private readonly now: () => number;
79
- constructor(options: { home?: string; memoryDir?: string; now?: () => number } = {}) {
79
+ /** #27: the keys aliased to the canonical project; its candidates read, accept and reject under the canonical key. Absent in the library default. */
80
+ private readonly reconcile?: { canonical: string; aliases: readonly string[] };
81
+ private scope(project: string): string { return JSON.stringify(this.reconcile && project === this.reconcile.canonical ? [project, ...this.reconcile.aliases] : [project]); }
82
+ constructor(options: { home?: string; memoryDir?: string; now?: () => number; reconcile?: { canonical: string; aliases: readonly string[] } } = {}) {
83
+ if (options.reconcile?.aliases.length) this.reconcile = { canonical: options.reconcile.canonical, aliases: [...options.reconcile.aliases] };
80
84
  if (options.memoryDir !== undefined && options.home !== undefined) throw new Error("choose a home or an exact memory directory");
81
85
  this.dir = options.memoryDir ?? (options.home === undefined ? memoryHome() : join(options.home, "memory"));
82
86
  this.now = options.now ?? Date.now;
@@ -112,7 +116,7 @@ export class CandidateStore {
112
116
  get(id: string, project: string): Candidate | undefined {
113
117
  if (!candidateId(id) || !text(project, 400)) throw new Error("invalid candidate identity");
114
118
  this.prune();
115
- const row = this.db.query("SELECT * FROM candidates WHERE id = ? AND project = ?").get(id, project) as Parameters<CandidateStore["decode"]>[0] | null;
119
+ const row = this.db.query("SELECT * FROM candidates WHERE id = ? AND project IN (SELECT value FROM json_each(?))").get(id, this.scope(project)) as Parameters<CandidateStore["decode"]>[0] | null;
116
120
  return row ? this.decode(row) : undefined;
117
121
  }
118
122
  list(filter: CandidateFilter): Candidate[] {
@@ -120,7 +124,7 @@ export class CandidateStore {
120
124
  const limit = filter.limit ?? CANDIDATE_RETENTION.count;
121
125
  if (!Number.isSafeInteger(limit) || limit < 1 || limit > CANDIDATE_RETENTION.count) throw new Error("candidate limit must be 1..500");
122
126
  this.prune();
123
- const where = ["project = ?"], bind: (string | number)[] = [filter.project];
127
+ const where = ["project IN (SELECT value FROM json_each(?))"], bind: (string | number)[] = [this.scope(filter.project)];
124
128
  if (filter.status === undefined) where.push("status IN ('pending', 'accepting')"); else if (filter.status !== "all") { where.push("status = ?"); bind.push(filter.status); }
125
129
  if (filter.kind) { where.push("kind = ?"); bind.push(filter.kind); }
126
130
  return (this.db.query(`SELECT * FROM candidates WHERE ${where.join(" AND ")} ORDER BY at DESC, rowid DESC LIMIT ?`).all(...bind, limit) as Parameters<CandidateStore["decode"]>[0][]).map(row => this.decode(row));
@@ -137,7 +141,7 @@ export class CandidateStore {
137
141
  counters(project: string, entry?: string): Counter[] {
138
142
  if (!text(project, 400) || entry !== undefined && !entryId(entry)) throw new Error("invalid counter filter");
139
143
  this.prune();
140
- const rows = this.db.query("SELECT * FROM counters WHERE project = ? ORDER BY at DESC, rowid DESC").all(project) as { id: string; payload: string; at: number; rendered: number | null; delivered: number | null; cited: number | null }[];
144
+ const rows = this.db.query("SELECT * FROM counters WHERE project IN (SELECT value FROM json_each(?)) ORDER BY at DESC, rowid DESC").all(this.scope(project)) as { id: string; payload: string; at: number; rendered: number | null; delivered: number | null; cited: number | null }[];
141
145
  return rows.map(row => ({ id: row.id, key: JSON.parse(row.payload) as CounterKey, createdAt: new Date(row.at).toISOString(), rendered: row.rendered, delivered: row.delivered, cited: row.cited })).filter(row => !entry || row.key.entryId === entry);
142
146
  }
143
147
  reject(id: string, context: { project: string; actor: "person" | "agent"; verdict: CandidateVerdict }): void {
@@ -146,7 +150,7 @@ export class CandidateStore {
146
150
  const candidate = this.get(id, context.project);
147
151
  if (!candidate) throw new Error("candidate unavailable");
148
152
  if (candidate.status === "accepted" || candidate.status === "accepting") throw new Error("candidate acceptance already started; inspect its ordinary note before any further action");
149
- this.db.query("UPDATE candidates SET status = 'rejected' WHERE id = ? AND project = ? AND status = 'pending'").run(id, context.project);
153
+ this.db.query("UPDATE candidates SET status = 'rejected' WHERE id = ? AND project IN (SELECT value FROM json_each(?)) AND status = 'pending'").run(id, this.scope(context.project));
150
154
  }).immediate();
151
155
  }
152
156
  accept(id: string, context: CandidateAcceptance): CandidateAcceptanceResult {
@@ -185,7 +189,7 @@ export class CandidateStore {
185
189
  if (!current) throw new Error("candidate unavailable");
186
190
  if (current.status === "accepted") return current.acceptedNoteId!;
187
191
  if (current.status !== "pending") throw new Error("candidate is rejected or acceptance was interrupted; no automatic replay");
188
- this.db.query("UPDATE candidates SET status = 'accepting' WHERE id = ? AND project = ?").run(id, context.project);
192
+ this.db.query("UPDATE candidates SET status = 'accepting' WHERE id = ? AND project IN (SELECT value FROM json_each(?))").run(id, this.scope(context.project));
189
193
  return undefined;
190
194
  }).immediate();
191
195
  if (started) return { candidateId: id, noteId: started, alreadyAccepted: true };
@@ -193,7 +197,7 @@ export class CandidateStore {
193
197
  // automatic replay; add explicit person-audited recovery before offering background acceptance retries.
194
198
  const noteId = append();
195
199
  if (data.kind === "correction" && !context.store.supersede(target!.id, noteId, "person", context.project)) throw new Error("candidate replacement refused; inspect the accepted note before recovery");
196
- this.db.query("UPDATE candidates SET status = 'accepted', note_id = ? WHERE id = ? AND project = ?").run(noteId, id, context.project);
200
+ this.db.query("UPDATE candidates SET status = 'accepted', note_id = ? WHERE id = ? AND project IN (SELECT value FROM json_each(?))").run(noteId, id, this.scope(context.project));
197
201
  return { candidateId: id, noteId, alreadyAccepted: false };
198
202
  });
199
203
  }
@@ -1,13 +1,13 @@
1
1
  import { Database } from "bun:sqlite";
2
2
  import { createHash, randomBytes } from "node:crypto";
3
- import { appendFileSync, chmodSync, existsSync, linkSync, mkdirSync, readFileSync, renameSync, rmSync, statSync, writeFileSync } from "node:fs";
3
+ import { chmodSync, closeSync, existsSync, mkdirSync, openSync, opendirSync, readFileSync, readSync, renameSync, rmSync, statSync, writeFileSync } from "node:fs";
4
4
  import { join } from "node:path";
5
5
  import { looksSecret } from "./screen.ts";
6
6
  import { isDenied } from "./platform.ts";
7
7
  import { memoryBase } from "./home.ts";
8
8
  import { captureShardDir, ensureShardDescriptor, resolveAlias } from "./shard.ts";
9
9
  import { machineId } from "./store.ts";
10
- import { fts5Available } from "./search-fts.ts";
10
+ import { readWindowSql, readRef, type ReadWindow, type ReadRef, ftsTerms, fts5Available } from "./search-fts.ts";
11
11
 
12
12
  /**
13
13
  * #13 design A, tier 2, born sharded (#15 T1): machine-local capture, one shard per project key. A shard lives at
@@ -25,10 +25,10 @@ import { fts5Available } from "./search-fts.ts";
25
25
  * ponytail: the cap counts row payload bytes, not the sqlite file's page slack or WAL; a VACUUM blocks writers and
26
26
  * stays out of the hook path, so file size can exceed the cap until pages are reused.
27
27
  *
28
- * R2 contention: a writer that cannot take its shard's lock within `busyTimeoutMs` appends the row to the
29
- * append-only `spool.jsonl` and exits instead of blocking the agent. The next uncontended writer (or an explicit
30
- * `drain()`) replays the spool; replay is idempotent per spooled line, so a crash between insert and truncation
31
- * never doubles a row.
28
+ * R2 contention: a writer that cannot take its shard's lock within `busyTimeoutMs` closes a private segment
29
+ * before publishing it atomically in `spool/`. The next uncontended writer (or explicit `drain()`) replays it
30
+ * under a unique claim name. Segments are immutable; a claim's byte offset advances only after the rows commit.
31
+ * Replay is idempotent per spooled line, so a crash between insert and acknowledgement never doubles a row.
32
32
  *
33
33
  * Migration: the pre-T1 machine-wide `capture/` (this branch never shipped) is drained per project on open —
34
34
  * each project key's rows move into its shard with their original timestamps, and the legacy store is removed
@@ -50,6 +50,13 @@ export const CAPTURE_DRAIN_MAX_LINES = 1000;
50
50
  const ROW_OVERHEAD_BYTES = 96;
51
51
  const SPOOL = "spool.jsonl";
52
52
  const DRAINING = "spool.jsonl.draining";
53
+ const SPOOL_DIR = "spool";
54
+ const SPOOL_MAX_FILES = 32;
55
+ const SPOOL_MAX_BYTES = 256 * 1024;
56
+ const SPOOL_MAX_MS = 25;
57
+ const segmentName = /^spool-[0-9a-f-]+\.jsonl(?:\.offset-(\d+)(-skip)?)?(?:\.draining-[0-9a-f]+)?$/;
58
+ let spoolSequence = 0;
59
+ const spoolToken = () => `${Date.now().toString(16).padStart(13, "0")}-${process.pid.toString(16).padStart(8, "0")}-${(spoolSequence++).toString(16).padStart(8, "0")}-${randomBytes(12).toString("hex")}`;
53
60
 
54
61
  export interface CaptureInput {
55
62
  project: string; agent: string; event: string;
@@ -182,7 +189,7 @@ export class CaptureStore {
182
189
  db.run("CREATE TABLE IF NOT EXISTS meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)");
183
190
  this.ensureFts();
184
191
  } catch (error) {
185
- if (!lockError(error)) throw error;
192
+ if (!lockError(error)) { db.close(); throw error; }
186
193
  this.contended = true; // R2: the hook spools and exits instead of waiting out another writer
187
194
  }
188
195
  this.migrated = this.contended ? 0 : this.migrateLegacy();
@@ -239,7 +246,13 @@ export class CaptureStore {
239
246
 
240
247
  /** A line moved from the legacy spool, byte for byte: the shard's drain normalizes whatever format it holds. */
241
248
  private spoolRaw(line: string): void {
242
- appendFileSync(join(this.dir, SPOOL), line + "\n", { mode: 0o600 });
249
+ const directory = join(this.dir, SPOOL_DIR), writing = join(directory, ".writing");
250
+ mkdirSync(writing, { recursive: true, mode: 0o700 });
251
+ const name = `spool-${spoolToken()}.jsonl`, temporary = join(writing, name);
252
+ try {
253
+ writeFileSync(temporary, line + "\n", { mode: 0o600, flag: "wx" }); // closes before readers can see it
254
+ renameSync(temporary, join(directory, name));
255
+ } finally { rmSync(temporary, { force: true }); }
243
256
  }
244
257
 
245
258
  /** External-content FTS5 over the excerpts, kept in sync by triggers, so capture search needs no catch-up pass. */
@@ -254,6 +267,29 @@ export class CaptureStore {
254
267
  if (!had && (this.db.query("SELECT COUNT(*) AS n FROM rows").get() as { n: number }).n > 0) this.db.run("INSERT INTO fts (fts) VALUES ('rebuild')");
255
268
  }
256
269
 
270
+ private mcpReady = false;
271
+ /** One-time MCP read cost only: hooks neither create this derived index nor scan history to fill it. */
272
+ private ensureMcpFts(): boolean {
273
+ if (!fts5Available()) return false;
274
+ if (this.mcpReady) return true;
275
+ try {
276
+ this.db.transaction(() => {
277
+ const hadMcp = !!this.db.query("SELECT 1 FROM sqlite_master WHERE name = 'mcp_fts' AND type = 'table'").get();
278
+ const body = (row: string) => `${row}.excerpt || char(10) || COALESCE((SELECT group_concat(value, ' ') FROM json_each(${row}.paths)), '') || char(10) || ${row}.event`;
279
+ this.db.run("CREATE VIRTUAL TABLE IF NOT EXISTS mcp_fts USING fts5(body)");
280
+ this.db.run(`CREATE TRIGGER IF NOT EXISTS rows_mcp_ai AFTER INSERT ON rows BEGIN INSERT INTO mcp_fts(rowid, body) VALUES (new.id, ${body("new")}); END`);
281
+ this.db.run("CREATE TRIGGER IF NOT EXISTS rows_mcp_ad AFTER DELETE ON rows BEGIN DELETE FROM mcp_fts WHERE rowid = old.id; END");
282
+ this.db.run(`CREATE TRIGGER IF NOT EXISTS rows_mcp_au AFTER UPDATE ON rows BEGIN DELETE FROM mcp_fts WHERE rowid = old.id; INSERT INTO mcp_fts(rowid, body) VALUES (new.id, ${body("new")}); END`);
283
+ if (!hadMcp) this.db.run(`INSERT INTO mcp_fts(rowid, body) SELECT rows.id, ${body("rows")} FROM rows`);
284
+ }).immediate();
285
+ this.mcpReady = true;
286
+ return true;
287
+ } catch {
288
+ // Missing/unavailable derived search is not a capture read failure. Retry on the next independent read.
289
+ return false;
290
+ }
291
+ }
292
+
257
293
  private bytes(): number {
258
294
  return Number((this.db.query("SELECT value FROM meta WHERE key = 'bytes'").get() as { value: string } | null)?.value ?? "0");
259
295
  }
@@ -306,35 +342,79 @@ export class CaptureStore {
306
342
  }
307
343
 
308
344
  private spool(row: NormalizedCapture): void {
309
- appendFileSync(join(this.dir, SPOOL), JSON.stringify({ ...row, nonce: randomBytes(4).toString("hex") }) + "\n", { mode: 0o600 });
345
+ this.spoolRaw(JSON.stringify({ ...row, nonce: randomBytes(12).toString("hex") }));
310
346
  }
311
347
 
312
348
  /**
313
- * Replay the spool, finishing a leftover draining file from a crash first. Idempotent per spooled line and bounded
314
- * per invocation; malformed or incomplete lines are skipped and counted instead of poisoning every later append.
315
- * ponytail: the shared spool/draining names cannot fully close multi-process replacement and already-open-writer
316
- * races. Per-writer spool files with unique drain names are the upgrade path if that hard guarantee is required.
349
+ * Immutable segments with unique claims: a stale drainer can remove only its own old name. Recovery may
350
+ * steal a live claim; both readers can replay it safely because row ids dedupe and neither mutates its inode.
351
+ * Offset acknowledgement is another rename, after commit, so a crash always leaves replayable evidence.
352
+ * ponytail: unpublished files left in .writing by a killed writer are not replayed (they may be partial).
353
+ * Quiescent maintenance may remove them; durable recovery of an interrupted append needs a journal/fsync.
354
+ * Legacy fixed names are accepted for upgrade, but old-version writers must stop before upgrading: their
355
+ * already-open shared inode cannot be made immutable by a newer reader.
317
356
  */
318
357
  drain(): { drained: number; skipped: number } {
319
- let budget = CAPTURE_DRAIN_MAX_LINES, drained = 0, skipped = 0;
358
+ let budget = CAPTURE_DRAIN_MAX_LINES, byteBudget = SPOOL_MAX_BYTES, drained = 0, skipped = 0;
320
359
  if (this.contended) return { drained, skipped };
360
+ const started = performance.now();
361
+ const directory = join(this.dir, SPOOL_DIR);
362
+ mkdirSync(directory, { recursive: true, mode: 0o700 });
363
+ const candidates: { file: string; name: string; offset: number; skip: boolean }[] = [];
364
+ // Fixed-name imports are only compatibility inputs; all acknowledgements use unique names thereafter.
365
+ for (const name of [DRAINING, SPOOL]) {
366
+ if (existsSync(join(this.dir, name))) candidates.push({ file: join(this.dir, name), name: `spool-${spoolToken()}.jsonl`, offset: 0, skip: false });
367
+ }
368
+ const entries = opendirSync(directory);
369
+ try {
370
+ for (let scanned = 0; scanned < 256 && candidates.length < SPOOL_MAX_FILES; scanned++) {
371
+ const entry = entries.readSync();
372
+ if (!entry) break;
373
+ const match = segmentName.exec(entry.name);
374
+ if (entry.isFile() && match) candidates.push({ file: join(directory, entry.name), name: entry.name.split(".offset-")[0]!.split(".draining-")[0]!,
375
+ offset: Number(match[1] ?? 0), skip: !!match[2] });
376
+ }
377
+ } finally { entries.closeSync(); }
321
378
  const validRow = (row: Partial<NormalizedCapture>): row is NormalizedCapture =>
322
379
  typeof row.at === "string" && row.at.length > 0 && typeof row.project === "string" && row.project.length > 0
323
380
  && typeof row.agent === "string" && row.agent.length > 0 && typeof row.event === "string" && row.event.length > 0
324
381
  && (row.tool === null || typeof row.tool === "string") && Array.isArray(row.paths) && row.paths.every((p) => typeof p === "string")
325
382
  && (row.status === null || typeof row.status === "string") && typeof row.excerpt === "string"
326
383
  && typeof row.bytes === "number" && Number.isFinite(row.bytes) && row.bytes >= 0;
327
- for (const renameFirst of [false, true]) {
328
- const file = join(this.dir, DRAINING);
329
- if (renameFirst && existsSync(join(this.dir, SPOOL))) {
330
- if (budget <= 0) break;
331
- // link+remove, not rename: a draining file another process owns is never clobbered at claim time.
332
- try { linkSync(join(this.dir, SPOOL), file); rmSync(join(this.dir, SPOOL)); } catch { continue; }
384
+ candidates.sort((a, b) => a.name.localeCompare(b.name));
385
+ for (const source of candidates) {
386
+ if (budget <= 0 || byteBudget <= 0 || performance.now() - started >= SPOOL_MAX_MS) break;
387
+ const file = join(directory, `${source.name}.offset-${source.offset}${source.skip ? "-skip" : ""}.draining-${randomBytes(12).toString("hex")}`);
388
+ try { renameSync(source.file, file); } catch { continue; } // another drainer claimed this exact name
389
+ let bytes: Buffer, size: number;
390
+ try {
391
+ const fd = openSync(file, "r");
392
+ try {
393
+ size = statSync(file).size;
394
+ bytes = Buffer.alloc(Math.min(byteBudget, Math.max(0, size - source.offset)));
395
+ bytes = bytes.subarray(0, readSync(fd, bytes, 0, bytes.length, source.offset));
396
+ } finally { closeSync(fd); }
397
+ } catch { continue; } // a recovery reader may already have moved our unique claim
398
+ byteBudget -= bytes.length;
399
+ let consumed = 0, skipping = source.skip;
400
+ const batch: string[] = [];
401
+ while (consumed < bytes.length && batch.length < budget) {
402
+ const newline = bytes.indexOf(10, consumed);
403
+ if (newline < 0) {
404
+ if (source.offset + bytes.length < size) {
405
+ // A short window may only be the allowance left after another file, not an oversized row.
406
+ if (consumed === 0 && bytes.length === SPOOL_MAX_BYTES) { consumed = bytes.length; skipping = true; }
407
+ break;
408
+ }
409
+ if (!skipping) batch.push(bytes.subarray(consumed).toString("utf8"));
410
+ else skipped++;
411
+ consumed = bytes.length; skipping = false; break;
412
+ }
413
+ const line = bytes.subarray(consumed, newline).toString("utf8");
414
+ consumed = newline + 1;
415
+ if (skipping) { skipped++; skipping = false; }
416
+ else if (line) batch.push(line);
333
417
  }
334
- if (!existsSync(file)) continue;
335
- const lines = readFileSync(file, "utf8").split("\n").filter(Boolean);
336
- if (!lines.length) { rmSync(file, { force: true }); continue; }
337
- const batch = lines.slice(0, Math.max(0, budget));
338
418
  try {
339
419
  this.db.transaction(() => {
340
420
  for (const line of batch) {
@@ -356,8 +436,12 @@ export class CaptureStore {
356
436
  break; // still contended: the spool stays for the next invocation
357
437
  }
358
438
  budget -= batch.length;
359
- if (batch.length === lines.length) rmSync(file, { force: true });
360
- else { writeFileSync(file, lines.slice(batch.length).join("\n") + "\n", { mode: 0o600 }); break; }
439
+ const next = source.offset + consumed;
440
+ if (next >= size) rmSync(file, { force: true });
441
+ else {
442
+ // Never rewrite a shared inode. Only the owner of this exact claim can acknowledge its prefix.
443
+ try { renameSync(file, join(directory, `${source.name}.offset-${next}${skipping ? "-skip" : ""}`)); } catch { /* recovered elsewhere */ }
444
+ }
361
445
  }
362
446
  return { drained, skipped };
363
447
  }
@@ -424,6 +508,19 @@ export class CaptureStore {
424
508
  return rows.map(toRow);
425
509
  }
426
510
 
511
+ /** Exact-project, bounded body pages; MCP screens before filling its public candidate window. */
512
+ readPage(window: ReadWindow & { query?: string; match?: "substring" }): { rows: CaptureRow[]; next?: ReadRef; engine?: "fts5" | "substring" } {
513
+ const { columns, where, bind } = readWindowSql(window, "id");
514
+ const engine = window.query === undefined ? undefined : window.match !== "substring" && this.ensureMcpFts() ? "fts5" : "substring";
515
+ if (window.query !== undefined && engine === "fts5") {
516
+ const match = ftsTerms(window.query); if (!match) return { rows: [], engine };
517
+ where.push("id IN (SELECT rowid FROM mcp_fts WHERE mcp_fts MATCH ?)"); bind.push(match);
518
+ }
519
+ const rows = this.db.query(`SELECT *, ${columns} FROM rows WHERE ${where.join(" AND ")} ORDER BY candidate_position, candidate_id LIMIT ?`)
520
+ .all(...bind, window.limit) as (RawRow & { candidate_id: string; candidate_position: number })[];
521
+ return { rows: rows.map(toRow), ...(engine ? { engine } : {}), ...(rows.length ? { next: readRef(rows[rows.length - 1]!) } : {}) };
522
+ }
523
+
427
524
  stats(): { rows: number; bytes: number; capMb: number; retentionDays: number; fileBytes: number } {
428
525
  let fileBytes = 0;
429
526
  try { fileBytes = statSync(join(this.dir, "capture.db")).size; } catch { /* not created yet */ }