@staix/agent-mem 0.4.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +6 -4
- package/package.json +1 -1
- package/src/cli.ts +17 -7
- package/src/commands-table.ts +1 -0
- package/src/mcp-memory.ts +71 -28
- package/src/memory/candidates.ts +11 -7
- package/src/memory/capture.ts +123 -26
- package/src/memory/catalog.ts +3 -4
- package/src/memory/import.ts +2 -3
- package/src/memory/search-fts.ts +44 -3
- package/src/memory/shard.ts +44 -10
- package/src/memory/store.ts +104 -33
- package/src/memory/work-lists.ts +138 -50
package/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Curated memory with append-only JSONL logs, a derived SQLite index, short- and long-term tiers, a typed graph, goals with leases, and git sync between machines. Extracted from [agent-hub](https://github.com/STAIxBWLB/agent-hub), with compatible storage and entry IDs.
|
|
4
4
|
|
|
5
|
-
Version: 0.
|
|
5
|
+
Version: 0.5.0. Requires Bun 1.4.2 or later; no runtime dependencies.
|
|
6
6
|
|
|
7
7
|
## Install
|
|
8
8
|
|
|
@@ -135,7 +135,7 @@ The JSONL segments are authoritative. `index.db` is derived and can be rebuilt.
|
|
|
135
135
|
|
|
136
136
|
From issue #13 (phase P1), the package carries a second, machine-local tier beside the curated store, born sharded (#15 T1): one capture shard per project key at `projects/<key-hash>/machine/<machine-id>/` under the memory home, with its own locks so two projects never contend. The capture tier records deterministic tool-event rows (project, agent, event, tool, paths, exit status and a bounded, secret-screened excerpt) in the shard's `capture.db`. It is never synced, never read as curated evidence, and never writes a curated log, so its maintenance can never evict notes, goals or work lists. Capture is bounded by a soft size cap and a retention window (defaults 512 MB per shard, 90 days), resolved at every invocation from the library options, then the shard's `config.json`, then the home default `capture-config.json`; eviction removes the oldest capture rows first in bounded batches. The 2026-10-17 owner decision keeps the P1 cap per shard; the machine-wide total is #15 T5. On lock contention a writer appends to an append-only spool and exits fast; the next uncontended writer replays the spool idempotently. The capture path makes no network or model call. The pre-T1 machine-wide `capture/` (an unreleased layout of this branch) is migrated per project on open and removed once empty.
|
|
137
137
|
|
|
138
|
-
Project identity (#15 D2) keeps `memoryProjectKey`: a worktree, a submodule and the main checkout of one repository share its key; two unrelated same-named repositories without a remote get distinct path-hash keys, flagged `local` in the shard's `project.json` and the catalog. A no-remote key hashes the main working root (using the actual top-level for a separate git dir), so keys written from the repository root keep their pre-#15 value. Known limit: a linked worktree of a no-remote separate-git-dir repository gets its own key, because git does not record that repository's main working root (probe: `alpha` and `alpha/sub` share `alpha@<hash>`, `alpha-wt` gets `alpha-wt@<hash>`); the key stays as is for root-key compatibility, and a reviewed alias reconciles its capture. Keys written by older builds from a subdirectory or worktree still used that invocation's cwd hash; a person-reviewed alias maps such a key to its canonical one for capture, work lists and the CLI and MCP project binding. Curated rows written under the old key keep
|
|
138
|
+
Project identity (#15 D2) keeps `memoryProjectKey`: a worktree, a submodule and the main checkout of one repository share its key; two unrelated same-named repositories without a remote get distinct path-hash keys, flagged `local` in the shard's `project.json` and the catalog. A no-remote key hashes the main working root (using the actual top-level for a separate git dir), so keys written from the repository root keep their pre-#15 value. Known limit: a linked worktree of a no-remote separate-git-dir repository gets its own key, because git does not record that repository's main working root (probe: `alpha` and `alpha/sub` share `alpha@<hash>`, `alpha-wt` gets `alpha-wt@<hash>`); the key stays as is for root-key compatibility, and a reviewed alias reconciles its capture. Keys written by older builds from a subdirectory or worktree still used that invocation's cwd hash; a person-reviewed alias maps such a key to its canonical one for capture, work lists and the CLI and MCP project binding. Curated rows written under the old key keep their key in the log (nothing is rewritten), and the CLI and MCP read them under the canonical project: the project filter of notes, goals, candidates, vault candidates and work lists covers the canonical key and every key aliased to it, and presents those rows under the canonical key (#27). The library `MemoryStore` still reads by the raw key unless a host passes `reconcile`, and shared rows stay visible as before. Writes still go to the key's own log and shard, and a settle or forget of an aliased row lands in the shard that holds it. `amem alias add <from-key> <to-key>` appends an op to `aliases.jsonl` (latest wins, never chained, never automatic), `amem alias remove <from-key>` appends a removal (the old key reads under itself again, the history stays), `amem alias list` shows the ops including removals. Both are person-only and need a terminal. An amem or hub older than this change ignores a removal and keeps the old binding until it is upgraded, so upgrade every host that shares the home first. A correction accepted on an aliased row lives under the canonical key, so after a removal the old key shows the superseded note again (see `docs/contracts.md`). `catalog.db` in the memory home (#15 D3's storage part) is derived only from the shards and the alias log and rebuilds to the same rows; the D3 management commands are #15 T5 and not built.
|
|
139
139
|
|
|
140
140
|
`store.search(query, filter)` searches the curated tier and the current project's capture shard with SQLite FTS5 and BM25 plus tier/recency weighting: natural multi-word queries match partial term sets (the substring filter required every word), and a curated row outranks the same text in the capture tier. Prefix terms match inflections and Korean particles in either direction by retaining the original query term and adding a common-particle-stripped alternative. The FTS copy of the curated rows lives in its own derived `search.db` (`index.db`'s table set is a frozen contract and gains nothing); it is written from the same apply path as the index, and a prune, crash or rebuild is detected and resynced, so the incremental FTS copy equals a full resync. Where FTS5 is unavailable the engine falls back to the substring filter and reports `engine: "substring"`; capture recall needs FTS5 and is skipped in the fallback. The library entry exports `CaptureStore`, `captureSearch`, `readCaptureConfig`, `writeCaptureConfig`, `SearchIndex`, `fts5Available`, the shard helpers (`resolveAlias`, `appendAlias`, `listAliases`, `shardKeyHash`), the catalog (`rebuildCatalog`, `readCatalog`, `scanCatalog`) and the capacity advisor (`capacityReport`).
|
|
141
141
|
|
|
@@ -238,6 +238,8 @@ A native shard crash tail is left to its original writer's recovery. Migration c
|
|
|
238
238
|
|
|
239
239
|
Run `amem mcp` as a stdio child of the agent CLI. It binds the working directory, canonical project key, exact memory home and selected profile once, accepts no project/home overrides in tool arguments, and closes all stores on stdin EOF. It has no listener, daemon, watcher or model call. Ordinary person commands retain their agent-marker refusal and mutation TTY checks; `mcp` is the sole agent entry point.
|
|
240
240
|
|
|
241
|
+
MCP search, timeline and show remain available above 5,000 rows. Reads page exact-project candidates and screen them before result selection. Search ranks the newest 5,000 permitted matching candidates per tier; `since` narrows that window. Timeline can anchor an older row directly. Capture MCP indexing is lazy on queried reads, never hook construction or append. Index contention uses a call-pinned substring fallback; later searches retry. See [MCP candidate reads](docs/contracts.md#mcp-candidate-reads-13-task-209) for the bounded-window and fallback contract.
|
|
242
|
+
|
|
241
243
|
The frozen nine-tool contract for dotfiles#255 and maru#458:
|
|
242
244
|
|
|
243
245
|
| Tool | Purpose |
|
|
@@ -252,7 +254,7 @@ The frozen nine-tool contract for dotfiles#255 and maru#458:
|
|
|
252
254
|
| `amem_vault_candidates` | Read-only screened durable curated notes |
|
|
253
255
|
| `amem_capacity` | Numeric machine capacity proposal; no apply capability |
|
|
254
256
|
|
|
255
|
-
Search/show/timeline label capture as recall with `evidence: false`; capture never enters the curated graph. Policy screens complete rows before selection, limits or graph construction. The standalone secret screen, private flags, PII patterns and deny paths compose with a host's stricter policy through `createMemoryMcp({cwd, base, policy})`. Screening errors fail closed. `base` is the exact memory root; historical library `home` options still name its parent. Scope and profile changes refuse further access instead of silently reopening another store.
|
|
257
|
+
Search/show/timeline label capture as recall with `evidence: false`; capture never enters the curated graph. Policy screens complete rows before selection, limits or graph construction. The standalone secret screen, private flags, PII patterns and deny paths compose with a host's stricter policy through `createMemoryMcp({cwd, base, policy})`. Screening errors fail closed. `base` is the exact memory root; historical library `home` options still name its parent. Scope and profile changes refuse further access instead of silently reopening another store. Graph reads retain the 5,000-row curated ceiling; work-list reads stream the complete ordered history without an operation-count ceiling, applying policy before latest-value reduction. The working state scales with the current result, not historical payloads. These are operational ceilings, not exposed counts of unscreened data. Graphs cover curated-note relationships, not goal/suggestion tables.
|
|
256
258
|
|
|
257
259
|
`WorkListStore` writes `projects/<key-hash>/work/` with its own append log, derived index and lock. Scalar fields reduce by latest value, null clears, list status `done` closes, and item status is `todo|doing|done|dropped`. Work lists are outside generation selection and capture eviction. `amem import claude-mem` additionally imports `work_state_entries` only through persisted person-reviewed D2 aliases, preserving source DB bytes and stable replay receipts; JSON keeps observation fields and adds `work` counts/cursor.
|
|
258
260
|
|
|
@@ -262,7 +264,7 @@ Do not record vault links on a home shared with a host using published agent-mem
|
|
|
262
264
|
|
|
263
265
|
General memory candidates are separate from curated evidence. `amem review accept <id> --kind memory --policy-free` remains person-only and rechecks the current standalone privacy policy before the accepted note append. Machine capacity reports numeric aggregate use across shards, as R7 requires; it does not return foreign project names or text.
|
|
264
266
|
|
|
265
|
-
`bun scripts/measure-mcp-budget.ts` exercises an actual stdio session with
|
|
267
|
+
`bun scripts/measure-mcp-budget.ts` exercises an actual stdio session with 5,001 curated notes, 5,001 capture rows and 10,001 work-list operations reduced into 100 items. It checks the R3 idle RSS limit of 100 MB, peak RSS limit of 200 MB across searches, latest/anchored timelines and work-list reads, and child reaping after EOF. CI runs the harness on macOS and Linux. All harness data is synthetic; live P6 measurements remain separate.
|
|
266
268
|
|
|
267
269
|
## Development
|
|
268
270
|
|
package/package.json
CHANGED
package/src/cli.ts
CHANGED
|
@@ -9,7 +9,7 @@ import { MemoryStore, looksSecret, machineId, openShardStore, type StoreOptions
|
|
|
9
9
|
import { rawMemoryCommand, rawGoalCommand } from "./memory/commands.ts";
|
|
10
10
|
import { memoryProjectKey } from "./memory/project.ts";
|
|
11
11
|
import { CandidateStore, type CandidateFilter } from "./memory/candidates.ts";
|
|
12
|
-
import { appendAlias, listAliases, readShardedSelection, resolveAlias } from "./memory/shard.ts";
|
|
12
|
+
import { aliasedKeys, appendAlias, appendUnalias, listAliases, readShardedSelection, resolveAlias } from "./memory/shard.ts";
|
|
13
13
|
import { capacityReport, capacityText } from "./memory/capacity.ts";
|
|
14
14
|
import { readCaptureConfigFile, writeCaptureConfigFile, type CaptureConfig } from "./memory/capture.ts";
|
|
15
15
|
|
|
@@ -70,12 +70,15 @@ function shardStatePresent(): boolean {
|
|
|
70
70
|
catch (error) { if ((error as NodeJS.ErrnoException).code === "ENOENT") return false; throw error; }
|
|
71
71
|
}
|
|
72
72
|
|
|
73
|
-
|
|
73
|
+
/** `reconciled` (#27): the project filter also covers every key aliased to `project`; prune and other planners keep the exact key. */
|
|
74
|
+
function curatedStore(project: string, opts: StoreOptions, reconciled = true): MemoryStore {
|
|
75
|
+
const aliases = reconciled ? aliasedKeys(memoryHome(), project) : [];
|
|
76
|
+
const withAliases: StoreOptions = aliases.length ? { ...opts, reconcile: { canonical: project, aliases } } : opts;
|
|
74
77
|
if (shardStatePresent()) {
|
|
75
78
|
readShardedSelection(memoryHome());
|
|
76
|
-
return openShardStore({ ...
|
|
79
|
+
return openShardStore({ ...withAliases, project });
|
|
77
80
|
}
|
|
78
|
-
return new MemoryStore(
|
|
81
|
+
return new MemoryStore(withAliases);
|
|
79
82
|
}
|
|
80
83
|
|
|
81
84
|
export async function runCli(argv: readonly string[], context: CliContext = {}): Promise<string> {
|
|
@@ -136,7 +139,7 @@ export async function runCli(argv: readonly string[], context: CliContext = {}):
|
|
|
136
139
|
}
|
|
137
140
|
const store = options["--user"] === true
|
|
138
141
|
? openShardStore({ memoryDir: memoryHome(), user: true, deferPromotions: true })
|
|
139
|
-
: curatedStore(project, { deferPromotions: true });
|
|
142
|
+
: curatedStore(project, { deferPromotions: true }, false);
|
|
140
143
|
try {
|
|
141
144
|
const auth = { project, actor: "person" as const, authorize: () => true };
|
|
142
145
|
return JSON.stringify(applying ? applyPrune(store.prunePort(), input, auth) : planPrune(store.prunePort(), { cutoff: String(options["--cutoff"]) }, auth), null, 2);
|
|
@@ -148,7 +151,7 @@ export async function runCli(argv: readonly string[], context: CliContext = {}):
|
|
|
148
151
|
if (sub === "list" && options["--kind"] !== "memory" && resolveProfile(memoryHome()).kind === "generation") throw new Error("candidate review is not yet integrated with this generation; select --kind memory for the standalone memory candidate surface");
|
|
149
152
|
const id = parsed.positionals[0];
|
|
150
153
|
if (sub !== "list" && (!id || !/^[a-f0-9]{64}$/.test(id))) throw new Error("review accept/reject requires a full 64-character candidate id");
|
|
151
|
-
const project = boundProject(), candidates = new CandidateStore();
|
|
154
|
+
const project = boundProject(), aliases = aliasedKeys(memoryHome(), project), candidates = new CandidateStore(aliases.length ? { reconcile: { canonical: project, aliases } } : {});
|
|
152
155
|
try {
|
|
153
156
|
if (sub === "list") {
|
|
154
157
|
const rows = candidates.list({ project, ...(options["--status"] ? { status: options["--status"] as CandidateFilter["status"] } : {}),
|
|
@@ -219,8 +222,15 @@ export async function runCli(argv: readonly string[], context: CliContext = {}):
|
|
|
219
222
|
const op = appendAlias(memoryHome(), { alias: from, key: to, actor: "person", machine: machineId(memoryHome()).id });
|
|
220
223
|
return options["--json"] ? JSON.stringify(op, null, 2) : `alias ${op.alias} -> ${op.key}`;
|
|
221
224
|
}
|
|
225
|
+
if (parsed.path[1] === "remove") {
|
|
226
|
+
// #27 option 2: person only (ordinary CLI refusal of agent markers above, a TTY through `mutates`), append only, the latest op wins.
|
|
227
|
+
const [from] = parsed.positionals;
|
|
228
|
+
if (!from || parsed.positionals.length !== 1) throw new Error("usage: amem alias remove <alias-key>");
|
|
229
|
+
const op = appendUnalias(memoryHome(), { alias: from, actor: "person", machine: machineId(memoryHome()).id });
|
|
230
|
+
return options["--json"] ? JSON.stringify(op, null, 2) : `alias ${op.alias} removed`;
|
|
231
|
+
}
|
|
222
232
|
const ops = listAliases(memoryHome());
|
|
223
|
-
return options["--json"] ? JSON.stringify(ops, null, 2) : ops.length ? ops.map((op) => `${op.at} ${op.alias} -> ${op.key} (${op.actor})`).join("\n") : "no aliases recorded";
|
|
233
|
+
return options["--json"] ? JSON.stringify(ops, null, 2) : ops.length ? ops.map((op) => op.op === "alias" ? `${op.at} ${op.alias} -> ${op.key} (${op.actor})` : `${op.at} ${op.alias} removed (${op.actor})`).join("\n") : "no aliases recorded";
|
|
224
234
|
}
|
|
225
235
|
// ponytail: board task links need the host's authenticated task boundary; add a host bridge before supporting them here.
|
|
226
236
|
if (goal && sub === "link") throw new Error("goal link needs host integration; use ahub goal link");
|
package/src/commands-table.ts
CHANGED
|
@@ -75,6 +75,7 @@ export const COMMANDS: Record<string, CommandDefinition> = {
|
|
|
75
75
|
alias: { description: "Person-reviewed project key aliases (#15 D2)", options: json, defaultCommand: "list", commands: {
|
|
76
76
|
list: { description: "List alias ops" },
|
|
77
77
|
add: { description: "Map one key onto its canonical key (append-only op)", args: [{ name: "from-key" }, { name: "to-key" }], mutates: true },
|
|
78
|
+
remove: { description: "Remove an alias so the key resolves to itself again (append-only op, the latest wins)", args: [{ name: "alias-key" }], mutates: true },
|
|
78
79
|
} },
|
|
79
80
|
completion: { description: "Print static shell completion", args: [{ name: "shell", values: ["zsh", "bash", "fish"] }] },
|
|
80
81
|
help: { description: "Show help" }, version: { description: "Show version" },
|
package/src/mcp-memory.ts
CHANGED
|
@@ -10,14 +10,17 @@ import { canonicalPath, realPath } from "./memory/platform.ts";
|
|
|
10
10
|
import { createStandalonePolicy, type StandalonePolicyOptions } from "./memory/policy.ts";
|
|
11
11
|
import { memoryProjectKey } from "./memory/project.ts";
|
|
12
12
|
import { registerReaderLease } from "./memory/reader-lease.ts";
|
|
13
|
-
import { SearchIndex } from "./memory/search-fts.ts";
|
|
14
|
-
import { readShardedSelection, resolveAlias } from "./memory/shard.ts";
|
|
13
|
+
import { type ReadRef, type ReadWindow, SearchIndex } from "./memory/search-fts.ts";
|
|
14
|
+
import { aliasedKeys, readShardedSelection, resolveAlias } from "./memory/shard.ts";
|
|
15
15
|
import { MemoryStore, openShardStore, type Graph, type View } from "./memory/store.ts";
|
|
16
16
|
import { vaultCandidates } from "./memory/vault.ts";
|
|
17
17
|
import { WorkListStore, type WorkListFields } from "./memory/work-lists.ts";
|
|
18
18
|
|
|
19
|
-
//
|
|
19
|
+
// Graph retains its complete-corpus ceiling; search/timeline/show page matching bodies instead.
|
|
20
20
|
export const MCP_SCAN_LIMIT = 5000;
|
|
21
|
+
// ponytail: rank the newest 5000 permitted matching candidates per tier; use since to narrow or upgrade to a policy-bound persistent index for global ranking.
|
|
22
|
+
const SEARCH_CANDIDATES = 5000;
|
|
23
|
+
const READ_PAGE = 256;
|
|
21
24
|
export interface MemoryMcpOptions { cwd: string; base?: string; project?: string; policy?: StandalonePolicyOptions }
|
|
22
25
|
export interface MemoryMcp { tools: McpHandlers; close(): void }
|
|
23
26
|
const unavailable = (): never => { throw new Error("memory operation unavailable"); };
|
|
@@ -27,7 +30,7 @@ const textOf = (args: Record<string, unknown>, key: string, optional = false): s
|
|
|
27
30
|
const value = args[key]; if (value === undefined && optional) return undefined;
|
|
28
31
|
return typeof value === "string" && value.trim() && !value.includes("\0") ? value : unavailable();
|
|
29
32
|
};
|
|
30
|
-
const sinceOf = (args: Record<string, unknown>): string | undefined => { const since = textOf(args, "since", true); if (since !== undefined && !Number.isFinite(Date.parse(since))) unavailable(); return since; };
|
|
33
|
+
const sinceOf = (args: Record<string, unknown>): string | undefined => { const since = textOf(args, "since", true); if (since !== undefined && !Number.isFinite(Date.parse(since))) unavailable(); return since === undefined ? undefined : new Date(since).toISOString(); };
|
|
31
34
|
|
|
32
35
|
/** All identities are resolved once. Per-call arguments cannot change projects, homes, profiles or policy. */
|
|
33
36
|
export function createMemoryMcp(options: MemoryMcpOptions): MemoryMcp {
|
|
@@ -40,15 +43,17 @@ export function createMemoryMcp(options: MemoryMcpOptions): MemoryMcp {
|
|
|
40
43
|
const selection = readSelector(base).revision;
|
|
41
44
|
const sharded = !!lstatSync(join(base, "shards.json"), { throwIfNoEntry: false });
|
|
42
45
|
const shardSelection = sharded ? JSON.stringify(readShardedSelection(base)) : undefined;
|
|
43
|
-
|
|
46
|
+
// #27: the project filter covers the canonical key and every key aliased to it; the keys are fixed with the project, so a later alias change fails the guard below.
|
|
47
|
+
const aliases = aliasedKeys(base, project), reconcile = aliases.length ? { canonical: project, aliases } : undefined;
|
|
48
|
+
const store = sharded ? openShardStore({ memoryDir: base, project, deferPromotions: true, ...(reconcile ? { reconcile } : {}) }) : new MemoryStore({ memoryDir: base, deferPromotions: true, ...(reconcile ? { reconcile } : {}) });
|
|
44
49
|
closers.push(() => store.close());
|
|
45
50
|
const capture = new CaptureStore({ memoryDir: base, project }); closers.push(() => capture.close());
|
|
46
|
-
const candidates = new CandidateStore({ memoryDir: base }); closers.push(() => candidates.close());
|
|
47
|
-
const work = new WorkListStore({ base, project, policy: async row => { const permitted = await policy.screen(row); guard(); return permitted; } }); closers.push(() => work.close());
|
|
51
|
+
const candidates = new CandidateStore({ memoryDir: base, ...(reconcile ? { reconcile } : {}) }); closers.push(() => candidates.close());
|
|
52
|
+
const work = new WorkListStore({ base, project, aliases, policy: async row => { const permitted = await policy.screen(row); guard(); return permitted; } }); closers.push(() => work.close());
|
|
48
53
|
const guard = () => {
|
|
49
54
|
if (closed) unavailable();
|
|
50
55
|
release.verify();
|
|
51
|
-
if (resolveAlias(base, requested) !== project || readSelector(base).revision !== selection) unavailable();
|
|
56
|
+
if (resolveAlias(base, requested) !== project || aliasedKeys(base, project).join("\n") !== aliases.join("\n") || readSelector(base).revision !== selection) unavailable();
|
|
52
57
|
const split = !!lstatSync(join(base, "shards.json"), { throwIfNoEntry: false });
|
|
53
58
|
if (split !== sharded || split && JSON.stringify(readShardedSelection(base)) !== shardSelection) unavailable();
|
|
54
59
|
};
|
|
@@ -63,46 +68,84 @@ export function createMemoryMcp(options: MemoryMcpOptions): MemoryMcp {
|
|
|
63
68
|
if (rows.length > MCP_SCAN_LIMIT) unavailable();
|
|
64
69
|
return screen(rows.filter(row => row.project === project));
|
|
65
70
|
};
|
|
66
|
-
const
|
|
67
|
-
|
|
71
|
+
const readPermitted = async <T>(read: (window: ReadWindow) => { rows: T[]; next?: ReadRef; engine?: "fts5" | "substring" },
|
|
72
|
+
options: Omit<ReadWindow, "project" | "limit" | "after">, cap: number, matches: (row: T, engine?: "fts5" | "substring") => boolean = () => true): Promise<T[]> => {
|
|
73
|
+
const result: T[] = []; let after: ReadWindow["after"];
|
|
74
|
+
while (result.length < cap) {
|
|
75
|
+
guard();
|
|
76
|
+
const page = read({ ...options, project, limit: READ_PAGE, after });
|
|
77
|
+
for (const row of page.rows) if (await policy.screen(row) && matches(row, page.engine)) {
|
|
78
|
+
result.push(row); if (result.length === cap) break;
|
|
79
|
+
}
|
|
80
|
+
guard();
|
|
81
|
+
if (!page.next) break;
|
|
82
|
+
after = { position: page.next.position, id: page.next.id };
|
|
83
|
+
}
|
|
84
|
+
return result;
|
|
85
|
+
};
|
|
86
|
+
const anchorAt = async (id: string): Promise<string | undefined> => {
|
|
87
|
+
if (/^capture:[1-9][0-9]*$/.test(id)) {
|
|
88
|
+
return (await readPermitted(window => capture.readPage(window), { id: id.slice(8) }, 1))[0]?.at;
|
|
89
|
+
}
|
|
90
|
+
const rows = await readPermitted(window => store.readPage({ ...window, includeSuperseded: true }), { prefix: id.replace(/^mem:/, "") }, 2);
|
|
91
|
+
return rows.length === 1 ? rows[0]!.at : undefined;
|
|
92
|
+
};
|
|
68
93
|
const tools: McpHandlers = {
|
|
69
94
|
async amem_search(args) {
|
|
70
95
|
bound(args, ["query", "limit", "since", "includeCapture"]); const query = textOf(args, "query")!, limit = limitOf(args), since = sinceOf(args);
|
|
71
96
|
if (args.includeCapture !== undefined && typeof args.includeCapture !== "boolean") unavailable();
|
|
72
97
|
if (!await policy.screen({ query })) unavailable();
|
|
73
|
-
const notes = await curated(), recalls = args.includeCapture === false ? [] : await captures();
|
|
74
|
-
const hits = [...notes.map(view => ({ source: "curated" as const, evidence: true, view, ref: view.id, at: view.at, tier: view.tier, body: `${view.title ?? ""}\n${view.text}\n${view.about.join(" ")}` })),
|
|
75
|
-
...recalls.map(row => ({ source: "capture" as const, evidence: false, capture: row, ref: `capture:${row.id}`, at: row.at, tier: "capture", body: `${row.excerpt}\n${row.paths.join(" ")}\n${row.event}` }))].filter(hit => !since || Date.parse(hit.at) >= Date.parse(since));
|
|
76
98
|
const index = new SearchIndex(":memory:");
|
|
77
99
|
try {
|
|
78
|
-
index.resync(() => hits.map(hit => ({ id: hit.ref, at: hit.at, project, tier: hit.tier, scope: "project", body: hit.body })));
|
|
79
|
-
if (index.available && index.needsResync(hits.length)) unavailable();
|
|
80
|
-
const ranks = new Map(index.query(query, MCP_SCAN_LIMIT * 2, project).map(row => [row.ref, row.rank]));
|
|
81
100
|
const words = query.toLowerCase().split(/\s+/).filter(Boolean);
|
|
101
|
+
const captureState: { engine?: "fts5" | "substring" } = {};
|
|
102
|
+
const recalls = args.includeCapture === false ? [] : await readPermitted(window => {
|
|
103
|
+
const page = capture.readPage({ ...window, query, ...(captureState.engine === "substring" ? { match: "substring" as const } : {}) });
|
|
104
|
+
captureState.engine ??= page.engine;
|
|
105
|
+
return page;
|
|
106
|
+
}, { since }, SEARCH_CANDIDATES,
|
|
107
|
+
(row, engine) => index.available && engine === "fts5" || words.every(word => `${row.excerpt}\n${row.paths.join(" ")}\n${row.event}`.toLowerCase().includes(word)));
|
|
108
|
+
const useFts = index.available && captureState.engine !== "substring";
|
|
109
|
+
const notes = await readPermitted(window => store.readPage({ ...window, ...(useFts ? { query } : {}) }), { since }, SEARCH_CANDIDATES,
|
|
110
|
+
view => useFts || words.every(word => `${view.title ?? ""}\n${view.text}\n${view.about.join(" ")}`.toLowerCase().includes(word)));
|
|
111
|
+
const hits = [...notes.map(view => ({ source: "curated" as const, evidence: true, view, ref: view.id, at: view.at, tier: view.tier, body: `${view.title ?? ""}\n${view.text}\n${view.about.join(" ")}` })),
|
|
112
|
+
...recalls.map(row => ({ source: "capture" as const, evidence: false, capture: row, ref: `capture:${row.id}`, at: row.at, tier: "capture", body: `${row.excerpt}\n${row.paths.join(" ")}\n${row.event}` }))].filter(hit => !since || Date.parse(hit.at) >= Date.parse(since));
|
|
113
|
+
if (useFts) {
|
|
114
|
+
index.resync(() => hits.map(hit => ({ id: hit.ref, at: hit.at, project, tier: hit.tier, scope: "project", body: hit.body })));
|
|
115
|
+
if (index.needsResync(hits.length)) unavailable();
|
|
116
|
+
}
|
|
117
|
+
const ranks = new Map((useFts ? index.query(query, MCP_SCAN_LIMIT * 2, project) : []).map(row => [row.ref, row.rank]));
|
|
82
118
|
const selected = hits.flatMap(hit => {
|
|
83
119
|
const rank = ranks.get(hit.ref);
|
|
84
|
-
if (
|
|
120
|
+
if (useFts ? rank === undefined : !words.every(word => hit.body.toLowerCase().includes(word))) return [];
|
|
85
121
|
const weight = hit.tier === "long" ? 1 : hit.tier === "capture" ? 0.5 : 0.9;
|
|
86
122
|
const score = -(rank ?? -1) * weight / (1 + Math.max(0, Date.now() - Date.parse(hit.at)) / (30 * 86400_000));
|
|
87
123
|
return [{ source: hit.source, evidence: hit.evidence, ...(hit.source === "curated" ? { view: hit.view } : { capture: hit.capture }), score }];
|
|
88
124
|
}).sort((a, b) => b.score - a.score).slice(0, limit);
|
|
89
|
-
guard(); return { engine:
|
|
125
|
+
guard(); return { engine: useFts ? "fts5" : "substring", hits: selected };
|
|
90
126
|
} finally { index.close(); }
|
|
91
127
|
},
|
|
92
128
|
async amem_show(args) {
|
|
93
129
|
bound(args, ["id"]); guard(); const id = textOf(args, "id")!;
|
|
94
|
-
if (/^capture:[1-9][0-9]*$/.test(id)) {
|
|
95
|
-
|
|
130
|
+
if (/^capture:[1-9][0-9]*$/.test(id)) {
|
|
131
|
+
const row = (await readPermitted(window => capture.readPage(window), { id: id.slice(8) }, 1))[0];
|
|
132
|
+
guard(); return row ? { source: "capture", evidence: false, capture: row } : null;
|
|
133
|
+
}
|
|
134
|
+
const rows = await readPermitted(window => store.readPage({ ...window, includeSuperseded: true }), { prefix: id.replace(/^mem:/, "") }, 2);
|
|
135
|
+
guard(); return rows.length === 1 ? rows[0]! : null;
|
|
96
136
|
},
|
|
97
137
|
async amem_timeline(args) {
|
|
98
|
-
bound(args, ["id", "since", "limit"]);
|
|
99
|
-
const
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
138
|
+
bound(args, ["id", "since", "limit"]);
|
|
139
|
+
const id = textOf(args, "id", true), since = sinceOf(args), limit = limitOf(args);
|
|
140
|
+
guard();
|
|
141
|
+
const anchor = id ? await anchorAt(id) : undefined;
|
|
142
|
+
if (id && !anchor) { guard(); return []; }
|
|
143
|
+
const rows = await readPermitted(window => store.readPage({ ...window, includeSuperseded: true }), { since, anchor }, limit);
|
|
144
|
+
const recalls = await readPermitted(window => capture.readPage(window), { since, anchor }, limit);
|
|
145
|
+
const entries = [...rows.map(row => ({ at: row.at, ref: row.id, value: row as unknown })),
|
|
146
|
+
...recalls.map(row => ({ at: row.at, ref: `capture:${row.id}`, value: { source: "capture", evidence: false, capture: row } as unknown }))];
|
|
147
|
+
entries.sort((a, b) => (anchor ? Math.abs(Date.parse(a.at) - Date.parse(anchor)) - Math.abs(Date.parse(b.at) - Date.parse(anchor)) : Date.parse(b.at) - Date.parse(a.at)) || (a.ref < b.ref ? -1 : a.ref > b.ref ? 1 : 0));
|
|
148
|
+
guard(); return entries.slice(0, limit).map(row => row.value);
|
|
106
149
|
},
|
|
107
150
|
async amem_graph(args) {
|
|
108
151
|
bound(args, ["start", "hops", "limit"]); const start = textOf(args, "start")!, cap = limitOf(args), hops = args.hops ?? 1;
|
package/src/memory/candidates.ts
CHANGED
|
@@ -76,7 +76,11 @@ export class CandidateStore {
|
|
|
76
76
|
readonly path: string;
|
|
77
77
|
private readonly db: Database;
|
|
78
78
|
private readonly now: () => number;
|
|
79
|
-
|
|
79
|
+
/** #27: the keys aliased to the canonical project; its candidates read, accept and reject under the canonical key. Absent in the library default. */
|
|
80
|
+
private readonly reconcile?: { canonical: string; aliases: readonly string[] };
|
|
81
|
+
private scope(project: string): string { return JSON.stringify(this.reconcile && project === this.reconcile.canonical ? [project, ...this.reconcile.aliases] : [project]); }
|
|
82
|
+
constructor(options: { home?: string; memoryDir?: string; now?: () => number; reconcile?: { canonical: string; aliases: readonly string[] } } = {}) {
|
|
83
|
+
if (options.reconcile?.aliases.length) this.reconcile = { canonical: options.reconcile.canonical, aliases: [...options.reconcile.aliases] };
|
|
80
84
|
if (options.memoryDir !== undefined && options.home !== undefined) throw new Error("choose a home or an exact memory directory");
|
|
81
85
|
this.dir = options.memoryDir ?? (options.home === undefined ? memoryHome() : join(options.home, "memory"));
|
|
82
86
|
this.now = options.now ?? Date.now;
|
|
@@ -112,7 +116,7 @@ export class CandidateStore {
|
|
|
112
116
|
get(id: string, project: string): Candidate | undefined {
|
|
113
117
|
if (!candidateId(id) || !text(project, 400)) throw new Error("invalid candidate identity");
|
|
114
118
|
this.prune();
|
|
115
|
-
const row = this.db.query("SELECT * FROM candidates WHERE id = ? AND project
|
|
119
|
+
const row = this.db.query("SELECT * FROM candidates WHERE id = ? AND project IN (SELECT value FROM json_each(?))").get(id, this.scope(project)) as Parameters<CandidateStore["decode"]>[0] | null;
|
|
116
120
|
return row ? this.decode(row) : undefined;
|
|
117
121
|
}
|
|
118
122
|
list(filter: CandidateFilter): Candidate[] {
|
|
@@ -120,7 +124,7 @@ export class CandidateStore {
|
|
|
120
124
|
const limit = filter.limit ?? CANDIDATE_RETENTION.count;
|
|
121
125
|
if (!Number.isSafeInteger(limit) || limit < 1 || limit > CANDIDATE_RETENTION.count) throw new Error("candidate limit must be 1..500");
|
|
122
126
|
this.prune();
|
|
123
|
-
const where = ["project
|
|
127
|
+
const where = ["project IN (SELECT value FROM json_each(?))"], bind: (string | number)[] = [this.scope(filter.project)];
|
|
124
128
|
if (filter.status === undefined) where.push("status IN ('pending', 'accepting')"); else if (filter.status !== "all") { where.push("status = ?"); bind.push(filter.status); }
|
|
125
129
|
if (filter.kind) { where.push("kind = ?"); bind.push(filter.kind); }
|
|
126
130
|
return (this.db.query(`SELECT * FROM candidates WHERE ${where.join(" AND ")} ORDER BY at DESC, rowid DESC LIMIT ?`).all(...bind, limit) as Parameters<CandidateStore["decode"]>[0][]).map(row => this.decode(row));
|
|
@@ -137,7 +141,7 @@ export class CandidateStore {
|
|
|
137
141
|
counters(project: string, entry?: string): Counter[] {
|
|
138
142
|
if (!text(project, 400) || entry !== undefined && !entryId(entry)) throw new Error("invalid counter filter");
|
|
139
143
|
this.prune();
|
|
140
|
-
const rows = this.db.query("SELECT * FROM counters WHERE project
|
|
144
|
+
const rows = this.db.query("SELECT * FROM counters WHERE project IN (SELECT value FROM json_each(?)) ORDER BY at DESC, rowid DESC").all(this.scope(project)) as { id: string; payload: string; at: number; rendered: number | null; delivered: number | null; cited: number | null }[];
|
|
141
145
|
return rows.map(row => ({ id: row.id, key: JSON.parse(row.payload) as CounterKey, createdAt: new Date(row.at).toISOString(), rendered: row.rendered, delivered: row.delivered, cited: row.cited })).filter(row => !entry || row.key.entryId === entry);
|
|
142
146
|
}
|
|
143
147
|
reject(id: string, context: { project: string; actor: "person" | "agent"; verdict: CandidateVerdict }): void {
|
|
@@ -146,7 +150,7 @@ export class CandidateStore {
|
|
|
146
150
|
const candidate = this.get(id, context.project);
|
|
147
151
|
if (!candidate) throw new Error("candidate unavailable");
|
|
148
152
|
if (candidate.status === "accepted" || candidate.status === "accepting") throw new Error("candidate acceptance already started; inspect its ordinary note before any further action");
|
|
149
|
-
this.db.query("UPDATE candidates SET status = 'rejected' WHERE id = ? AND project
|
|
153
|
+
this.db.query("UPDATE candidates SET status = 'rejected' WHERE id = ? AND project IN (SELECT value FROM json_each(?)) AND status = 'pending'").run(id, this.scope(context.project));
|
|
150
154
|
}).immediate();
|
|
151
155
|
}
|
|
152
156
|
accept(id: string, context: CandidateAcceptance): CandidateAcceptanceResult {
|
|
@@ -185,7 +189,7 @@ export class CandidateStore {
|
|
|
185
189
|
if (!current) throw new Error("candidate unavailable");
|
|
186
190
|
if (current.status === "accepted") return current.acceptedNoteId!;
|
|
187
191
|
if (current.status !== "pending") throw new Error("candidate is rejected or acceptance was interrupted; no automatic replay");
|
|
188
|
-
this.db.query("UPDATE candidates SET status = 'accepting' WHERE id = ? AND project
|
|
192
|
+
this.db.query("UPDATE candidates SET status = 'accepting' WHERE id = ? AND project IN (SELECT value FROM json_each(?))").run(id, this.scope(context.project));
|
|
189
193
|
return undefined;
|
|
190
194
|
}).immediate();
|
|
191
195
|
if (started) return { candidateId: id, noteId: started, alreadyAccepted: true };
|
|
@@ -193,7 +197,7 @@ export class CandidateStore {
|
|
|
193
197
|
// automatic replay; add explicit person-audited recovery before offering background acceptance retries.
|
|
194
198
|
const noteId = append();
|
|
195
199
|
if (data.kind === "correction" && !context.store.supersede(target!.id, noteId, "person", context.project)) throw new Error("candidate replacement refused; inspect the accepted note before recovery");
|
|
196
|
-
this.db.query("UPDATE candidates SET status = 'accepted', note_id = ? WHERE id = ? AND project
|
|
200
|
+
this.db.query("UPDATE candidates SET status = 'accepted', note_id = ? WHERE id = ? AND project IN (SELECT value FROM json_each(?))").run(noteId, id, this.scope(context.project));
|
|
197
201
|
return { candidateId: id, noteId, alreadyAccepted: false };
|
|
198
202
|
});
|
|
199
203
|
}
|
package/src/memory/capture.ts
CHANGED
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
import { Database } from "bun:sqlite";
|
|
2
2
|
import { createHash, randomBytes } from "node:crypto";
|
|
3
|
-
import {
|
|
3
|
+
import { chmodSync, closeSync, existsSync, mkdirSync, openSync, opendirSync, readFileSync, readSync, renameSync, rmSync, statSync, writeFileSync } from "node:fs";
|
|
4
4
|
import { join } from "node:path";
|
|
5
5
|
import { looksSecret } from "./screen.ts";
|
|
6
6
|
import { isDenied } from "./platform.ts";
|
|
7
7
|
import { memoryBase } from "./home.ts";
|
|
8
8
|
import { captureShardDir, ensureShardDescriptor, resolveAlias } from "./shard.ts";
|
|
9
9
|
import { machineId } from "./store.ts";
|
|
10
|
-
import { fts5Available } from "./search-fts.ts";
|
|
10
|
+
import { readWindowSql, readRef, type ReadWindow, type ReadRef, ftsTerms, fts5Available } from "./search-fts.ts";
|
|
11
11
|
|
|
12
12
|
/**
|
|
13
13
|
* #13 design A, tier 2, born sharded (#15 T1): machine-local capture, one shard per project key. A shard lives at
|
|
@@ -25,10 +25,10 @@ import { fts5Available } from "./search-fts.ts";
|
|
|
25
25
|
* ponytail: the cap counts row payload bytes, not the sqlite file's page slack or WAL; a VACUUM blocks writers and
|
|
26
26
|
* stays out of the hook path, so file size can exceed the cap until pages are reused.
|
|
27
27
|
*
|
|
28
|
-
* R2 contention: a writer that cannot take its shard's lock within `busyTimeoutMs`
|
|
29
|
-
*
|
|
30
|
-
*
|
|
31
|
-
* never doubles a row.
|
|
28
|
+
* R2 contention: a writer that cannot take its shard's lock within `busyTimeoutMs` closes a private segment
|
|
29
|
+
* before publishing it atomically in `spool/`. The next uncontended writer (or explicit `drain()`) replays it
|
|
30
|
+
* under a unique claim name. Segments are immutable; a claim's byte offset advances only after the rows commit.
|
|
31
|
+
* Replay is idempotent per spooled line, so a crash between insert and acknowledgement never doubles a row.
|
|
32
32
|
*
|
|
33
33
|
* Migration: the pre-T1 machine-wide `capture/` (this branch never shipped) is drained per project on open —
|
|
34
34
|
* each project key's rows move into its shard with their original timestamps, and the legacy store is removed
|
|
@@ -50,6 +50,13 @@ export const CAPTURE_DRAIN_MAX_LINES = 1000;
|
|
|
50
50
|
const ROW_OVERHEAD_BYTES = 96;
|
|
51
51
|
const SPOOL = "spool.jsonl";
|
|
52
52
|
const DRAINING = "spool.jsonl.draining";
|
|
53
|
+
const SPOOL_DIR = "spool";
|
|
54
|
+
const SPOOL_MAX_FILES = 32;
|
|
55
|
+
const SPOOL_MAX_BYTES = 256 * 1024;
|
|
56
|
+
const SPOOL_MAX_MS = 25;
|
|
57
|
+
const segmentName = /^spool-[0-9a-f-]+\.jsonl(?:\.offset-(\d+)(-skip)?)?(?:\.draining-[0-9a-f]+)?$/;
|
|
58
|
+
let spoolSequence = 0;
|
|
59
|
+
const spoolToken = () => `${Date.now().toString(16).padStart(13, "0")}-${process.pid.toString(16).padStart(8, "0")}-${(spoolSequence++).toString(16).padStart(8, "0")}-${randomBytes(12).toString("hex")}`;
|
|
53
60
|
|
|
54
61
|
export interface CaptureInput {
|
|
55
62
|
project: string; agent: string; event: string;
|
|
@@ -182,7 +189,7 @@ export class CaptureStore {
|
|
|
182
189
|
db.run("CREATE TABLE IF NOT EXISTS meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)");
|
|
183
190
|
this.ensureFts();
|
|
184
191
|
} catch (error) {
|
|
185
|
-
if (!lockError(error)) throw error;
|
|
192
|
+
if (!lockError(error)) { db.close(); throw error; }
|
|
186
193
|
this.contended = true; // R2: the hook spools and exits instead of waiting out another writer
|
|
187
194
|
}
|
|
188
195
|
this.migrated = this.contended ? 0 : this.migrateLegacy();
|
|
@@ -239,7 +246,13 @@ export class CaptureStore {
|
|
|
239
246
|
|
|
240
247
|
/** A line moved from the legacy spool, byte for byte: the shard's drain normalizes whatever format it holds. */
|
|
241
248
|
private spoolRaw(line: string): void {
|
|
242
|
-
|
|
249
|
+
const directory = join(this.dir, SPOOL_DIR), writing = join(directory, ".writing");
|
|
250
|
+
mkdirSync(writing, { recursive: true, mode: 0o700 });
|
|
251
|
+
const name = `spool-${spoolToken()}.jsonl`, temporary = join(writing, name);
|
|
252
|
+
try {
|
|
253
|
+
writeFileSync(temporary, line + "\n", { mode: 0o600, flag: "wx" }); // closes before readers can see it
|
|
254
|
+
renameSync(temporary, join(directory, name));
|
|
255
|
+
} finally { rmSync(temporary, { force: true }); }
|
|
243
256
|
}
|
|
244
257
|
|
|
245
258
|
/** External-content FTS5 over the excerpts, kept in sync by triggers, so capture search needs no catch-up pass. */
|
|
@@ -254,6 +267,29 @@ export class CaptureStore {
|
|
|
254
267
|
if (!had && (this.db.query("SELECT COUNT(*) AS n FROM rows").get() as { n: number }).n > 0) this.db.run("INSERT INTO fts (fts) VALUES ('rebuild')");
|
|
255
268
|
}
|
|
256
269
|
|
|
270
|
+
private mcpReady = false;
|
|
271
|
+
/** One-time MCP read cost only: hooks neither create this derived index nor scan history to fill it. */
|
|
272
|
+
private ensureMcpFts(): boolean {
|
|
273
|
+
if (!fts5Available()) return false;
|
|
274
|
+
if (this.mcpReady) return true;
|
|
275
|
+
try {
|
|
276
|
+
this.db.transaction(() => {
|
|
277
|
+
const hadMcp = !!this.db.query("SELECT 1 FROM sqlite_master WHERE name = 'mcp_fts' AND type = 'table'").get();
|
|
278
|
+
const body = (row: string) => `${row}.excerpt || char(10) || COALESCE((SELECT group_concat(value, ' ') FROM json_each(${row}.paths)), '') || char(10) || ${row}.event`;
|
|
279
|
+
this.db.run("CREATE VIRTUAL TABLE IF NOT EXISTS mcp_fts USING fts5(body)");
|
|
280
|
+
this.db.run(`CREATE TRIGGER IF NOT EXISTS rows_mcp_ai AFTER INSERT ON rows BEGIN INSERT INTO mcp_fts(rowid, body) VALUES (new.id, ${body("new")}); END`);
|
|
281
|
+
this.db.run("CREATE TRIGGER IF NOT EXISTS rows_mcp_ad AFTER DELETE ON rows BEGIN DELETE FROM mcp_fts WHERE rowid = old.id; END");
|
|
282
|
+
this.db.run(`CREATE TRIGGER IF NOT EXISTS rows_mcp_au AFTER UPDATE ON rows BEGIN DELETE FROM mcp_fts WHERE rowid = old.id; INSERT INTO mcp_fts(rowid, body) VALUES (new.id, ${body("new")}); END`);
|
|
283
|
+
if (!hadMcp) this.db.run(`INSERT INTO mcp_fts(rowid, body) SELECT rows.id, ${body("rows")} FROM rows`);
|
|
284
|
+
}).immediate();
|
|
285
|
+
this.mcpReady = true;
|
|
286
|
+
return true;
|
|
287
|
+
} catch {
|
|
288
|
+
// Missing/unavailable derived search is not a capture read failure. Retry on the next independent read.
|
|
289
|
+
return false;
|
|
290
|
+
}
|
|
291
|
+
}
|
|
292
|
+
|
|
257
293
|
private bytes(): number {
|
|
258
294
|
return Number((this.db.query("SELECT value FROM meta WHERE key = 'bytes'").get() as { value: string } | null)?.value ?? "0");
|
|
259
295
|
}
|
|
@@ -306,35 +342,79 @@ export class CaptureStore {
|
|
|
306
342
|
}
|
|
307
343
|
|
|
308
344
|
private spool(row: NormalizedCapture): void {
|
|
309
|
-
|
|
345
|
+
this.spoolRaw(JSON.stringify({ ...row, nonce: randomBytes(12).toString("hex") }));
|
|
310
346
|
}
|
|
311
347
|
|
|
312
348
|
/**
|
|
313
|
-
*
|
|
314
|
-
*
|
|
315
|
-
*
|
|
316
|
-
*
|
|
349
|
+
* Immutable segments with unique claims: a stale drainer can remove only its own old name. Recovery may
|
|
350
|
+
* steal a live claim; both readers can replay it safely because row ids dedupe and neither mutates its inode.
|
|
351
|
+
* Offset acknowledgement is another rename, after commit, so a crash always leaves replayable evidence.
|
|
352
|
+
* ponytail: unpublished files left in .writing by a killed writer are not replayed (they may be partial).
|
|
353
|
+
* Quiescent maintenance may remove them; durable recovery of an interrupted append needs a journal/fsync.
|
|
354
|
+
* Legacy fixed names are accepted for upgrade, but old-version writers must stop before upgrading: their
|
|
355
|
+
* already-open shared inode cannot be made immutable by a newer reader.
|
|
317
356
|
*/
|
|
318
357
|
drain(): { drained: number; skipped: number } {
|
|
319
|
-
let budget = CAPTURE_DRAIN_MAX_LINES, drained = 0, skipped = 0;
|
|
358
|
+
let budget = CAPTURE_DRAIN_MAX_LINES, byteBudget = SPOOL_MAX_BYTES, drained = 0, skipped = 0;
|
|
320
359
|
if (this.contended) return { drained, skipped };
|
|
360
|
+
const started = performance.now();
|
|
361
|
+
const directory = join(this.dir, SPOOL_DIR);
|
|
362
|
+
mkdirSync(directory, { recursive: true, mode: 0o700 });
|
|
363
|
+
const candidates: { file: string; name: string; offset: number; skip: boolean }[] = [];
|
|
364
|
+
// Fixed-name imports are only compatibility inputs; all acknowledgements use unique names thereafter.
|
|
365
|
+
for (const name of [DRAINING, SPOOL]) {
|
|
366
|
+
if (existsSync(join(this.dir, name))) candidates.push({ file: join(this.dir, name), name: `spool-${spoolToken()}.jsonl`, offset: 0, skip: false });
|
|
367
|
+
}
|
|
368
|
+
const entries = opendirSync(directory);
|
|
369
|
+
try {
|
|
370
|
+
for (let scanned = 0; scanned < 256 && candidates.length < SPOOL_MAX_FILES; scanned++) {
|
|
371
|
+
const entry = entries.readSync();
|
|
372
|
+
if (!entry) break;
|
|
373
|
+
const match = segmentName.exec(entry.name);
|
|
374
|
+
if (entry.isFile() && match) candidates.push({ file: join(directory, entry.name), name: entry.name.split(".offset-")[0]!.split(".draining-")[0]!,
|
|
375
|
+
offset: Number(match[1] ?? 0), skip: !!match[2] });
|
|
376
|
+
}
|
|
377
|
+
} finally { entries.closeSync(); }
|
|
321
378
|
const validRow = (row: Partial<NormalizedCapture>): row is NormalizedCapture =>
|
|
322
379
|
typeof row.at === "string" && row.at.length > 0 && typeof row.project === "string" && row.project.length > 0
|
|
323
380
|
&& typeof row.agent === "string" && row.agent.length > 0 && typeof row.event === "string" && row.event.length > 0
|
|
324
381
|
&& (row.tool === null || typeof row.tool === "string") && Array.isArray(row.paths) && row.paths.every((p) => typeof p === "string")
|
|
325
382
|
&& (row.status === null || typeof row.status === "string") && typeof row.excerpt === "string"
|
|
326
383
|
&& typeof row.bytes === "number" && Number.isFinite(row.bytes) && row.bytes >= 0;
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
if (
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
|
|
384
|
+
candidates.sort((a, b) => a.name.localeCompare(b.name));
|
|
385
|
+
for (const source of candidates) {
|
|
386
|
+
if (budget <= 0 || byteBudget <= 0 || performance.now() - started >= SPOOL_MAX_MS) break;
|
|
387
|
+
const file = join(directory, `${source.name}.offset-${source.offset}${source.skip ? "-skip" : ""}.draining-${randomBytes(12).toString("hex")}`);
|
|
388
|
+
try { renameSync(source.file, file); } catch { continue; } // another drainer claimed this exact name
|
|
389
|
+
let bytes: Buffer, size: number;
|
|
390
|
+
try {
|
|
391
|
+
const fd = openSync(file, "r");
|
|
392
|
+
try {
|
|
393
|
+
size = statSync(file).size;
|
|
394
|
+
bytes = Buffer.alloc(Math.min(byteBudget, Math.max(0, size - source.offset)));
|
|
395
|
+
bytes = bytes.subarray(0, readSync(fd, bytes, 0, bytes.length, source.offset));
|
|
396
|
+
} finally { closeSync(fd); }
|
|
397
|
+
} catch { continue; } // a recovery reader may already have moved our unique claim
|
|
398
|
+
byteBudget -= bytes.length;
|
|
399
|
+
let consumed = 0, skipping = source.skip;
|
|
400
|
+
const batch: string[] = [];
|
|
401
|
+
while (consumed < bytes.length && batch.length < budget) {
|
|
402
|
+
const newline = bytes.indexOf(10, consumed);
|
|
403
|
+
if (newline < 0) {
|
|
404
|
+
if (source.offset + bytes.length < size) {
|
|
405
|
+
// A short window may only be the allowance left after another file, not an oversized row.
|
|
406
|
+
if (consumed === 0 && bytes.length === SPOOL_MAX_BYTES) { consumed = bytes.length; skipping = true; }
|
|
407
|
+
break;
|
|
408
|
+
}
|
|
409
|
+
if (!skipping) batch.push(bytes.subarray(consumed).toString("utf8"));
|
|
410
|
+
else skipped++;
|
|
411
|
+
consumed = bytes.length; skipping = false; break;
|
|
412
|
+
}
|
|
413
|
+
const line = bytes.subarray(consumed, newline).toString("utf8");
|
|
414
|
+
consumed = newline + 1;
|
|
415
|
+
if (skipping) { skipped++; skipping = false; }
|
|
416
|
+
else if (line) batch.push(line);
|
|
333
417
|
}
|
|
334
|
-
if (!existsSync(file)) continue;
|
|
335
|
-
const lines = readFileSync(file, "utf8").split("\n").filter(Boolean);
|
|
336
|
-
if (!lines.length) { rmSync(file, { force: true }); continue; }
|
|
337
|
-
const batch = lines.slice(0, Math.max(0, budget));
|
|
338
418
|
try {
|
|
339
419
|
this.db.transaction(() => {
|
|
340
420
|
for (const line of batch) {
|
|
@@ -356,8 +436,12 @@ export class CaptureStore {
|
|
|
356
436
|
break; // still contended: the spool stays for the next invocation
|
|
357
437
|
}
|
|
358
438
|
budget -= batch.length;
|
|
359
|
-
|
|
360
|
-
|
|
439
|
+
const next = source.offset + consumed;
|
|
440
|
+
if (next >= size) rmSync(file, { force: true });
|
|
441
|
+
else {
|
|
442
|
+
// Never rewrite a shared inode. Only the owner of this exact claim can acknowledge its prefix.
|
|
443
|
+
try { renameSync(file, join(directory, `${source.name}.offset-${next}${skipping ? "-skip" : ""}`)); } catch { /* recovered elsewhere */ }
|
|
444
|
+
}
|
|
361
445
|
}
|
|
362
446
|
return { drained, skipped };
|
|
363
447
|
}
|
|
@@ -424,6 +508,19 @@ export class CaptureStore {
|
|
|
424
508
|
return rows.map(toRow);
|
|
425
509
|
}
|
|
426
510
|
|
|
511
|
+
/** Exact-project, bounded body pages; MCP screens before filling its public candidate window. */
|
|
512
|
+
readPage(window: ReadWindow & { query?: string; match?: "substring" }): { rows: CaptureRow[]; next?: ReadRef; engine?: "fts5" | "substring" } {
|
|
513
|
+
const { columns, where, bind } = readWindowSql(window, "id");
|
|
514
|
+
const engine = window.query === undefined ? undefined : window.match !== "substring" && this.ensureMcpFts() ? "fts5" : "substring";
|
|
515
|
+
if (window.query !== undefined && engine === "fts5") {
|
|
516
|
+
const match = ftsTerms(window.query); if (!match) return { rows: [], engine };
|
|
517
|
+
where.push("id IN (SELECT rowid FROM mcp_fts WHERE mcp_fts MATCH ?)"); bind.push(match);
|
|
518
|
+
}
|
|
519
|
+
const rows = this.db.query(`SELECT *, ${columns} FROM rows WHERE ${where.join(" AND ")} ORDER BY candidate_position, candidate_id LIMIT ?`)
|
|
520
|
+
.all(...bind, window.limit) as (RawRow & { candidate_id: string; candidate_position: number })[];
|
|
521
|
+
return { rows: rows.map(toRow), ...(engine ? { engine } : {}), ...(rows.length ? { next: readRef(rows[rows.length - 1]!) } : {}) };
|
|
522
|
+
}
|
|
523
|
+
|
|
427
524
|
stats(): { rows: number; bytes: number; capMb: number; retentionDays: number; fileBytes: number } {
|
|
428
525
|
let fileBytes = 0;
|
|
429
526
|
try { fileBytes = statSync(join(this.dir, "capture.db")).size; } catch { /* not created yet */ }
|