djournal 0.8.0 → 0.9.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -26,13 +26,17 @@ Default to Overview when unclear.
26
26
 
27
27
  ## Discover cheaply
28
28
 
29
- 1. Resolve the journal root according to `STATE.md`. Enumerate
30
- `<journal-root>/work/*/work.md`; use slug, title, description, status,
31
- visibility, and metadata to rank candidate work items.
32
- 2. For legacy work items, use folder names without rejecting them.
33
- 3. Read entry frontmatter under `journal/`, `_research/`, `docs/`, and
34
- `decisions/` for candidate work items. Build indexes by ID, type, timestamp,
35
- title, summary, and links. Infer legacy values best-effort.
29
+ 1. Run `djournal recall search "<query>" --json` from the project. Add `--work`,
30
+ `--type`, `--status`, or `--visibility` only when the request supplies that
31
+ constraint. Use the ranked paths, IDs, matched fields, summaries, and links
32
+ to choose canonical files for deep reads. The cache is a disposable local
33
+ projection; never cite it instead of the Markdown source.
34
+ 2. If the command is unavailable, fails, or returns a recoverable cache warning
35
+ without results, resolve the journal root according to `STATE.md` and use the
36
+ canonical fallback: enumerate `<journal-root>/work/*/work.md`, rank work-item
37
+ metadata, then read entry frontmatter under `journal/`, `_research/`, `docs/`,
38
+ and `decisions/`. Infer legacy metadata best-effort.
39
+ 3. For legacy work items, use folder names without rejecting them.
36
40
  4. If no candidate matches, say so and list plausible work items. If more than
37
41
  three match, use their plan/latest summaries to narrow before deep reads.
38
42
 
package/README.md CHANGED
@@ -248,6 +248,28 @@ djournal work bind 2026-08-14-01-example --session "$DJOURNAL_SESSION_ID"
248
248
  If several work items are active and no session binding exists, the session-start
249
249
  hook asks the agent to prompt for a choice instead of guessing from recency.
250
250
 
251
+ ## Fast recall
252
+
253
+ Use recall directly as a search CLI:
254
+
255
+ ```bash
256
+ djournal recall search "billing webhook retries"
257
+ djournal recall search "billing webhook retries" --type decision,research --limit 5 --json
258
+ ```
259
+
260
+ Or install a harness and ask its agent naturally: “Recall why we chose the
261
+ current billing webhook retry strategy.” The bundled `recall` skill calls the
262
+ same CLI, selects the best journal paths, reads the canonical Markdown, follows
263
+ relevant links, and returns an evidence-backed answer.
264
+
265
+ Recall uses a private, per-project MiniSearch snapshot to rank relevant work
266
+ items and entry paths before reading canonical Markdown. Warm queries reuse the
267
+ snapshot; additions, edits, deletes, and work-metadata changes refresh it on the
268
+ next search. Use `--no-cache` for a one-call in-memory index. Cache corruption or
269
+ write denial is recoverable and never triggers a share, sync, commit, or push.
270
+ See [Fast recall](docs/recall.md) for standalone CLI examples, agent behavior,
271
+ filters, and cache lifecycle details.
272
+
251
273
  At session start, djournal reads a user-local npm release cache and reports a
252
274
  known CLI or project-asset update every session until it is installed. Stale
253
275
  release data is refreshed in the background at most once every 24 hours. Run
@@ -274,6 +296,7 @@ djournal is licensed under the [Apache License 2.0](LICENSE).
274
296
  ## Documentation
275
297
 
276
298
  - [Installation and repository layouts](docs/installation.md)
299
+ - [Fast recall: CLI and agent usage](docs/recall.md)
277
300
  - [Remote Git sync setup](docs/remote-sync.md)
278
301
  - [Read-only team journal access](docs/read-only-team-journal.md)
279
302
  - [Architecture and data model](docs/architecture.md)
package/bin/journal.js CHANGED
@@ -19,6 +19,7 @@ const {
19
19
  uninstall,
20
20
  upgrade,
21
21
  } = require("../lib/installer/index.js");
22
+ const { buildRecallIndex, searchRecall } = require("../lib/recall/index.js");
22
23
 
23
24
  const sourceRoot = path.resolve(__dirname, "..");
24
25
 
@@ -35,6 +36,8 @@ function usage() {
35
36
  journal sync [--target DIR] [--work SLUG] [--dry-run] [--json]
36
37
  journal work list [--target DIR] [--active] [--json]
37
38
  journal work bind SLUG [--target DIR] [--session ID] [--dry-run] [--json]
39
+ journal recall index [--target DIR] [--rebuild] [--no-cache] [--json]
40
+ journal recall search QUERY [--target DIR] [--work SLUG] [--type LIST] [--kind LIST] [--status LIST] [--visibility LIST] [--limit N] [--no-cache] [--json]
38
41
  journal update check [--target DIR] [--json]
39
42
 
40
43
  Options:
@@ -44,6 +47,13 @@ Options:
44
47
  --work SLUG Select a journal work item instead of active state
45
48
  --session ID Bind a journal work item to a specific session
46
49
  --active Show only lifecycle-active work items for work list
50
+ --type LIST Filter recall entries by comma-separated entry types
51
+ --kind LIST Filter recall results by comma-separated work,entry kinds
52
+ --status LIST Filter recall results by work-item status
53
+ --visibility LIST Filter recall results by visibility
54
+ --limit N Limit recall results per kind (1-100)
55
+ --rebuild Rebuild the persistent recall cache from scratch
56
+ --no-cache Search with an ephemeral in-memory index
47
57
  --harness LIST Comma-separated codex,claude-code,pi selection
48
58
  --all Select all work items for share, or every harness
49
59
  --instructions-only Install core instructions without harness hooks
@@ -53,7 +63,7 @@ Options:
53
63
  function parseArgs(argv) {
54
64
  const args = [...argv];
55
65
  const command = args.shift();
56
- if (!command || !["install", "upgrade", "uninstall", "status", "doctor", "config", "share", "pull", "sync", "update", "work"].includes(command)) {
66
+ if (!command || !["install", "upgrade", "uninstall", "status", "doctor", "config", "share", "pull", "sync", "update", "work", "recall"].includes(command)) {
57
67
  throw new InstallerError(usage(), "USAGE");
58
68
  }
59
69
  const options = { command, harnesses: [] };
@@ -67,6 +77,13 @@ function parseArgs(argv) {
67
77
  if (!args.length || args[0].startsWith("-")) throw new InstallerError("work bind requires a work slug", "USAGE");
68
78
  options.work = args.shift();
69
79
  }
80
+ } else if (command === "recall") {
81
+ options.subcommand = args.shift();
82
+ if (!["index", "search"].includes(options.subcommand)) throw new InstallerError("recall requires the index or search subcommand", "USAGE");
83
+ if (options.subcommand === "search") {
84
+ if (!args.length || args[0].startsWith("-")) throw new InstallerError("recall search requires a query", "USAGE");
85
+ options.query = args.shift();
86
+ }
70
87
  }
71
88
  while (args.length) {
72
89
  const arg = args.shift();
@@ -82,6 +99,26 @@ function parseArgs(argv) {
82
99
  if (!args.length) throw new InstallerError("--session requires a value", "USAGE");
83
100
  options.session = args.shift();
84
101
  } else if (arg.startsWith("--session=")) options.session = arg.slice(10);
102
+ else if (arg === "--type") {
103
+ if (!args.length) throw new InstallerError("--type requires a value", "USAGE");
104
+ options.type = args.shift();
105
+ } else if (arg.startsWith("--type=")) options.type = arg.slice(7);
106
+ else if (arg === "--kind") {
107
+ if (!args.length) throw new InstallerError("--kind requires a value", "USAGE");
108
+ options.kind = args.shift();
109
+ } else if (arg.startsWith("--kind=")) options.kind = arg.slice(7);
110
+ else if (arg === "--status") {
111
+ if (!args.length) throw new InstallerError("--status requires a value", "USAGE");
112
+ options.status = args.shift();
113
+ } else if (arg.startsWith("--status=")) options.status = arg.slice(9);
114
+ else if (arg === "--visibility") {
115
+ if (!args.length) throw new InstallerError("--visibility requires a value", "USAGE");
116
+ options.visibility = args.shift();
117
+ } else if (arg.startsWith("--visibility=")) options.visibility = arg.slice(13);
118
+ else if (arg === "--limit") {
119
+ if (!args.length) throw new InstallerError("--limit requires a value", "USAGE");
120
+ options.limit = Number(args.shift());
121
+ } else if (arg.startsWith("--limit=")) options.limit = Number(arg.slice(8));
85
122
  else if (arg === "--harness") {
86
123
  if (!args.length) throw new InstallerError("--harness requires a value", "USAGE");
87
124
  options.harnesses.push(...args.shift().split(",").filter(Boolean));
@@ -93,6 +130,8 @@ function parseArgs(argv) {
93
130
  else if (arg === "--auto") options.auto = true;
94
131
  else if (arg === "--yes") options.yes = true;
95
132
  else if (arg === "--json") options.json = true;
133
+ else if (arg === "--rebuild" && command === "recall") options.rebuild = true;
134
+ else if (arg === "--no-cache" && command === "recall") options.noCache = true;
96
135
  else if (arg === "--passive" && command === "update") options.passive = true;
97
136
  else if (arg === "--refresh-cache" && command === "update") options.refreshCache = true;
98
137
  else if (arg === "--help" || arg === "-h") throw new InstallerError(usage(), "HELP");
@@ -111,6 +150,17 @@ function parseArgs(argv) {
111
150
  if (options.active && !(command === "work" && options.subcommand === "list")) {
112
151
  throw new InstallerError("--active is only supported for work list", "USAGE");
113
152
  }
153
+ if (command === "recall") {
154
+ if (typeof options.limit !== "undefined" && (!Number.isInteger(options.limit) || options.limit < 1 || options.limit > 100)) {
155
+ throw new InstallerError("--limit must be an integer from 1 to 100", "USAGE");
156
+ }
157
+ if (options.rebuild && options.subcommand !== "index") throw new InstallerError("--rebuild is only supported for recall index", "USAGE");
158
+ if (options.subcommand === "index" && ["work", "type", "kind", "status", "visibility", "limit"].some((key) => typeof options[key] !== "undefined")) {
159
+ throw new InstallerError("recall filters are only supported for recall search", "USAGE");
160
+ }
161
+ } else if (["type", "kind", "status", "visibility", "limit", "rebuild", "noCache"].some((key) => typeof options[key] !== "undefined")) {
162
+ throw new InstallerError("recall filters are only supported for recall commands", "USAGE");
163
+ }
114
164
  options.target = path.resolve(options.target || process.cwd());
115
165
  options.sourceRoot = sourceRoot;
116
166
  options.interactive = !options.yes && process.stdin.isTTY && process.stdout.isTTY;
@@ -144,6 +194,21 @@ function print(value, json) {
144
194
  process.stdout.write(`bound ${value.work} to session ${value.sessionHash}\n`);
145
195
  return;
146
196
  }
197
+ if (value.action === "recall-index") {
198
+ process.stdout.write(`recall index: ${value.cache.status} (${value.cache.mode})\n`);
199
+ process.stdout.write(`documents: ${value.cache.documentCount}\n`);
200
+ process.stdout.write(`elapsed: ${value.cache.totalMs.toFixed(1)} ms\n`);
201
+ for (const item of value.warnings || []) process.stdout.write(`warning: ${item.code}${item.path ? ` ${item.path}` : ""}${item.detail ? ` - ${item.detail}` : ""}\n`);
202
+ return;
203
+ }
204
+ if (value.action === "recall-search") {
205
+ process.stdout.write(`recall search: ${value.cache.status} (${value.cache.mode}), ${value.cache.totalMs.toFixed(1)} ms\n`);
206
+ for (const item of [...(value.workItems || []), ...(value.entries || [])]) {
207
+ process.stdout.write(`${item.score.toFixed(2).padStart(8)} ${item.path} ${item.title}\n`);
208
+ }
209
+ for (const item of value.warnings || []) process.stdout.write(`warning: ${item.code}${item.path ? ` ${item.path}` : ""}${item.detail ? ` - ${item.detail}` : ""}\n`);
210
+ return;
211
+ }
147
212
  if (value.action) process.stdout.write(`${value.action}: ${value.target}\n`);
148
213
  else if (typeof value.installed === "boolean") process.stdout.write(value.installed ? `installed: ${value.target}\n` : `not installed: ${value.target}\n`);
149
214
  else process.stdout.write(`${value.ok ? "ok" : "failed"}: ${value.target}\n`);
@@ -155,6 +220,10 @@ function print(value, json) {
155
220
  if (value.latestVersion) process.stdout.write(`latest version: ${value.latestVersion}\n`);
156
221
  if (value.updateAvailable) process.stdout.write("update available: yes\n");
157
222
  if (value.projectUpdateAvailable) process.stdout.write("project upgrade available: yes\n");
223
+ if (value.recallCache) {
224
+ const cache = value.recallCache;
225
+ process.stdout.write(`recall cache: ${cache.valid ? `${cache.documentCount} documents, ${cache.indexVersion}` : (cache.exists ? "invalid; rebuilds on next recall" : "not built")}\n`);
226
+ }
158
227
  if (value.workItems) {
159
228
  for (const item of value.workItems) {
160
229
  process.stdout.write(`${item.changed ? "shared" : "unchanged"} ${item.work}\n`);
@@ -217,6 +286,8 @@ async function main() {
217
286
  else if (options.command === "sync") result = sync(options);
218
287
  else if (options.command === "work" && options.subcommand === "list") result = listWork(options);
219
288
  else if (options.command === "work" && options.subcommand === "bind") result = bindWork(options);
289
+ else if (options.command === "recall" && options.subcommand === "index") result = buildRecallIndex(options);
290
+ else if (options.command === "recall" && options.subcommand === "search") result = searchRecall(options.query, options);
220
291
  else if (options.command === "update") {
221
292
  result = await checkUpdates(options);
222
293
  if (result.refreshNeeded) result.backgroundRefreshStarted = launchUpdateRefresh(options);
@@ -225,7 +296,7 @@ async function main() {
225
296
  print(result, options.json);
226
297
  if (result.ok === false || result.installed === false || result.clean === false || result.conflicts?.length) process.exitCode = 2;
227
298
  } catch (error) {
228
- const code = error instanceof InstallerError ? error.code : "UNEXPECTED";
299
+ const code = error instanceof InstallerError ? error.code : (error?.code || "UNEXPECTED");
229
300
  if (process.argv.includes("--json")) {
230
301
  process.stderr.write(`${JSON.stringify({ ok: false, code, message: error.message }, null, 2)}\n`);
231
302
  } else process.stderr.write(`${code}: ${error.message}\n`);
@@ -11,6 +11,7 @@ The canonical journal lives in the user's djournal home, keyed by project:
11
11
  ~/.djournal/projects/<project-key>/
12
12
  config.json
13
13
  sessions/ # local operational session bindings
14
+ cache/recall/ # disposable private recall index
14
15
  .journal/
15
16
  state.json
16
17
  work/<work-item>/
@@ -27,6 +28,29 @@ The installed project contains a marker that points back to that store:
27
28
  .djournal.json
28
29
  ```
29
30
 
31
+ ## Recall index
32
+
33
+ `djournal recall search` uses a serialized MiniSearch inverted index at
34
+ `<project-store>/cache/recall/snapshot.json`. It indexes work metadata, entry
35
+ metadata, headings, and body terms, while storing only the metadata needed to
36
+ return explainable ranked paths. The complete Markdown body is not a stored
37
+ result field, and every final answer still reads the selected canonical files.
38
+
39
+ The snapshot is schema-, index-option-, and project-versioned. Each search compares a
40
+ manifest of source paths, sizes, and modification times; changed work metadata
41
+ reindexes its entries, and additions, edits, renames, and deletions are applied
42
+ incrementally. Missing, incompatible, oversized, or malformed snapshots rebuild
43
+ from Markdown. Writes use a private directory, a short-lived lock, and atomic
44
+ rename. If persistence is unavailable, the same query runs with an ephemeral
45
+ in-memory index.
46
+
47
+ The cache is outside `.journal/`, is never shared, and may be removed at any
48
+ time. `djournal recall index --rebuild` recreates it explicitly, while
49
+ `djournal recall search "query" --no-cache` bypasses it for one invocation.
50
+ Pull and sync do not block on prewarming: the next search performs authoritative
51
+ reconciliation. This keeps publication latency predictable while preserving a
52
+ warm snapshot whenever prior recall has already built one.
53
+
30
54
  When sync is enabled, a repository-local `.journal/` is a projection containing
31
55
  the shared work items copied out of the global store.
32
56
 
@@ -22,6 +22,14 @@ This still writes local files under the user's djournal home. "Read-only" means
22
22
  read-only with respect to the organization remote, not a filesystem mode that
23
23
  prevents local cache or canonical-store hydration.
24
24
 
25
+ Recall also writes a private derived index under
26
+ `~/.djournal/projects/<project-key>/cache/recall/`. It is not placed in the Git
27
+ projection and is never published. Every CLI invocation loads the serialized
28
+ index, checks source-file metadata, and incrementally refreshes only stale
29
+ documents before searching; it does not keep a daemon or a process permanently
30
+ in memory. If the local store is not writable, recall builds the index in memory
31
+ for that invocation and returns results without attempting a remote write.
32
+
25
33
  ## Setup for a Non-Technical Reader
26
34
 
27
35
  Ask the organization admin for the team journal repository URL and read-only Git
@@ -37,6 +45,8 @@ djournal config sync.mode standalone
37
45
  djournal config sync.path .
38
46
  djournal config sync.auto false
39
47
  djournal pull
48
+ djournal recall index
49
+ djournal recall search "onboarding decisions"
40
50
  djournal status
41
51
  ```
42
52
 
@@ -60,6 +70,12 @@ djournal sync
60
70
  If read-only Git permissions are configured correctly, accidental pushes should
61
71
  fail at the remote even if someone runs a publication command.
62
72
 
73
+ To avoid persistent local cache writes as well, add `--no-cache` to a recall
74
+ search. To clear a persistent cache, delete only the project store's
75
+ `cache/recall/` directory or run `djournal recall index --rebuild`; canonical
76
+ journal history is unaffected. A malformed or incompatible cache is ignored and
77
+ rebuilt automatically.
78
+
63
79
  ## Copy/Paste Agent Prompt
64
80
 
65
81
  Use this prompt after installation when asking an LLM agent to connect the local
package/docs/recall.md ADDED
@@ -0,0 +1,102 @@
1
+ # Fast Recall
2
+
3
+ Recall finds the journal files most likely to answer a question before anyone
4
+ spends time opening and summarizing them. It can be used directly as a small
5
+ search CLI, or indirectly through an agent that has djournal's `recall` skill.
6
+ Both paths use the same local index and keep Markdown as the source of truth.
7
+
8
+ ## Use it as a standalone CLI
9
+
10
+ From an installed project, search in plain-text mode:
11
+
12
+ ```bash
13
+ djournal recall search "why did we choose webhook retries"
14
+ ```
15
+
16
+ The output ranks matching work items and entries by score and prints their
17
+ canonical `.journal/...` paths. Use JSON when another script or tool will consume
18
+ the results:
19
+
20
+ ```bash
21
+ djournal recall search "webhook retries" --type decision,research --limit 5 --json
22
+ ```
23
+
24
+ Available filters narrow results without changing the journal:
25
+
26
+ ```bash
27
+ djournal recall search "checkout" \
28
+ --work 2026-08-14-01-checkout \
29
+ --kind entry \
30
+ --status active \
31
+ --visibility team_shared
32
+ ```
33
+
34
+ The first search builds the private index automatically. Prebuilding is useful
35
+ after importing a large journal, and rebuilding is available for diagnostics:
36
+
37
+ ```bash
38
+ djournal recall index
39
+ djournal recall index --rebuild
40
+ ```
41
+
42
+ For a strictly no-write invocation, use an ephemeral index:
43
+
44
+ ```bash
45
+ djournal recall search "onboarding decisions" --no-cache --json
46
+ ```
47
+
48
+ CLI recall returns evidence candidates, not a generated answer. A human or
49
+ calling program decides which returned Markdown files to read next.
50
+
51
+ ## Use it through an agent
52
+
53
+ Install djournal for the agent harness you use:
54
+
55
+ ```bash
56
+ djournal install --harness codex
57
+ # or: --harness claude-code
58
+ # or: --harness pi
59
+ ```
60
+
61
+ Then ask naturally:
62
+
63
+ ```text
64
+ Recall why we chose the current billing webhook retry strategy.
65
+ ```
66
+
67
+ The `recall` skill classifies the question, calls the same CLI with `--json`,
68
+ uses the ranked paths to select a small evidence set, and reads those canonical
69
+ Markdown files before answering. It follows relevant entry links and starts
70
+ from the latest spine entry when the question needs chronology or current
71
+ status. Its final response includes the journal sources that support the answer.
72
+
73
+ If the CLI is unavailable or the cache cannot produce candidates, the skill
74
+ falls back to filesystem discovery. Recall remains read-only in either path: it
75
+ does not create entries, share work, sync, commit, or push.
76
+
77
+ ## What happens on each call
78
+
79
+ There is no daemon. Each invocation loads the serialized MiniSearch snapshot,
80
+ checks source-file metadata, refreshes changed documents, searches in memory,
81
+ and exits. Warm calls avoid reparsing unchanged Markdown; additions, edits,
82
+ renames, and deletions are reconciled before results are returned.
83
+
84
+ The persistent snapshot lives at:
85
+
86
+ ```text
87
+ ~/.djournal/projects/<project-key>/cache/recall/snapshot.json
88
+ ```
89
+
90
+ It sits outside `.journal/`, so it is not canonical history and is never shared.
91
+ If it is missing, incompatible, malformed, or unwritable, recall rebuilds it or
92
+ continues in memory for the current call. Removing `cache/recall/` is safe.
93
+
94
+ Inspect cache health with:
95
+
96
+ ```bash
97
+ djournal status
98
+ djournal doctor
99
+ ```
100
+
101
+ For the storage and freshness model, see [Architecture and data model](architecture.md).
102
+ For remote readers, see [Read-only team journal access](read-only-team-journal.md).
@@ -476,7 +476,7 @@ function copyRecord(sourceRoot, target, relative, oldRecord, harness) {
476
476
  const absolute = resolveWithin(target, relative);
477
477
  const current = readOptional(absolute);
478
478
  if (oldRecord) {
479
- if (current === null || sha256(current) !== oldRecord.installedHash) {
479
+ if (current === null || (sha256(current) !== oldRecord.installedHash && !current.equals(source))) {
480
480
  throw new InstallerError(`modified installed asset: ${relative}`, "ASSET_CONFLICT");
481
481
  }
482
482
  } else if (current !== null && !current.equals(source)) {
@@ -1212,7 +1212,16 @@ function doctor(options) {
1212
1212
  detail: `${assetStatus}; Pi project trust required`,
1213
1213
  });
1214
1214
  }
1215
- return { ok: checks.every((check) => check.ok), target, detected, checks };
1215
+ const { inspectSnapshot } = require("../recall/snapshot.js");
1216
+ const recallCache = inspectSnapshot(projectContext(target, options), options);
1217
+ checks.push({
1218
+ name: "recall cache",
1219
+ ok: true,
1220
+ detail: recallCache.valid
1221
+ ? `${recallCache.documentCount} documents; ${recallCache.indexVersion}; ${recallCache.builtAt}`
1222
+ : (recallCache.exists ? "invalid but recoverable on next recall" : "not built; created on first recall"),
1223
+ });
1224
+ return { ok: checks.every((check) => check.ok), target, detected, recallCache, checks };
1216
1225
  }
1217
1226
 
1218
1227
  async function install(options) {
@@ -1361,7 +1370,10 @@ function recordStatus(target, record) {
1361
1370
  function status(options) {
1362
1371
  const target = checkTarget(options.target || process.cwd());
1363
1372
  const manifest = loadManifest(target);
1364
- const config = readProjectConfig(projectContext(target, options));
1373
+ const context = projectContext(target, options);
1374
+ const config = readProjectConfig(context);
1375
+ const { inspectSnapshot } = require("../recall/snapshot.js");
1376
+ const recallCache = inspectSnapshot(context, options);
1365
1377
  const cache = readUpdateCache(options);
1366
1378
  const updateState = deriveUpdateState({
1367
1379
  cliVersion: packageVersion(packageSourceRoot(options)),
@@ -1369,7 +1381,7 @@ function status(options) {
1369
1381
  cache,
1370
1382
  });
1371
1383
  const updateChecksEnabled = passiveChecksEnabled(config, options.env || process.env, { allowTest: options.allowTest });
1372
- if (!manifest) return { installed: false, target, ...updateState, updateChecksEnabled };
1384
+ if (!manifest) return { installed: false, target, ...updateState, updateChecksEnabled, recallCache };
1373
1385
  const files = manifest.files.map((record) => ({ path: record.path, mode: record.mode, status: recordStatus(target, record) }));
1374
1386
  return {
1375
1387
  installed: true,
@@ -1377,6 +1389,7 @@ function status(options) {
1377
1389
  toolVersion: manifest.toolVersion,
1378
1390
  ...updateState,
1379
1391
  updateChecksEnabled,
1392
+ recallCache,
1380
1393
  harnesses: manifest.harnesses,
1381
1394
  files,
1382
1395
  clean: files.every((file) => file.status === "clean"),
@@ -0,0 +1,327 @@
1
+ "use strict";
2
+
3
+ const crypto = require("node:crypto");
4
+ const fs = require("node:fs");
5
+ const path = require("node:path");
6
+ const YAML = require("yaml");
7
+
8
+ const ENTRY_DIRECTORIES = ["journal", "_research", "docs", "decisions"];
9
+ const MAX_MARKDOWN_BYTES = 16 * 1024 * 1024;
10
+
11
+ class RecallCorpusError extends Error {
12
+ constructor(message, code = "RECALL_CORPUS_ERROR") {
13
+ super(message);
14
+ this.name = "RecallCorpusError";
15
+ this.code = code;
16
+ }
17
+ }
18
+
19
+ function sha256(value) {
20
+ return crypto.createHash("sha256").update(value).digest("hex");
21
+ }
22
+
23
+ function posix(relative) {
24
+ return relative.split(path.sep).join("/");
25
+ }
26
+
27
+ function fingerprint(stat) {
28
+ return { size: stat.size, mtimeMs: stat.mtimeMs };
29
+ }
30
+
31
+ function sameFingerprint(left, right) {
32
+ return Boolean(left && right && left.size === right.size && left.mtimeMs === right.mtimeMs);
33
+ }
34
+
35
+ function assertWithin(root, candidate) {
36
+ const boundary = path.resolve(root);
37
+ const resolved = path.resolve(candidate);
38
+ if (resolved !== boundary && !resolved.startsWith(`${boundary}${path.sep}`)) {
39
+ throw new RecallCorpusError(`journal path escapes canonical root: ${candidate}`, "UNSAFE_PATH");
40
+ }
41
+ return resolved;
42
+ }
43
+
44
+ function listMarkdown(directory, journalRoot, workSlug, area, warnings, result) {
45
+ let entries;
46
+ try { entries = fs.readdirSync(directory, { withFileTypes: true }).sort((a, b) => a.name.localeCompare(b.name)); }
47
+ catch (error) {
48
+ if (error.code === "ENOENT") return;
49
+ throw error;
50
+ }
51
+ for (const entry of entries) {
52
+ const absolute = assertWithin(journalRoot, path.join(directory, entry.name));
53
+ if (entry.isSymbolicLink()) {
54
+ warnings.push({ code: "SYMLINK_IGNORED", path: posix(path.relative(journalRoot, absolute)) });
55
+ continue;
56
+ }
57
+ if (entry.isDirectory()) {
58
+ listMarkdown(absolute, journalRoot, workSlug, area, warnings, result);
59
+ continue;
60
+ }
61
+ if (!entry.isFile() || path.extname(entry.name).toLowerCase() !== ".md") continue;
62
+ const stat = fs.lstatSync(absolute);
63
+ if (!stat.isFile() || stat.isSymbolicLink()) continue;
64
+ result.push({
65
+ absolute,
66
+ relative: posix(path.relative(journalRoot, absolute)),
67
+ workSlug,
68
+ area,
69
+ kind: "entry",
70
+ ...fingerprint(stat),
71
+ });
72
+ }
73
+ }
74
+
75
+ function discoverSourceFiles(context) {
76
+ const journalRoot = path.resolve(context.journalRoot);
77
+ const workRoot = path.join(journalRoot, "work");
78
+ const warnings = [];
79
+ const files = [];
80
+ let workDirectories = [];
81
+ try { workDirectories = fs.readdirSync(workRoot, { withFileTypes: true }).sort((a, b) => a.name.localeCompare(b.name)); }
82
+ catch (error) { if (error.code !== "ENOENT") throw error; }
83
+
84
+ for (const entry of workDirectories) {
85
+ if (!entry.isDirectory() || entry.isSymbolicLink()) {
86
+ if (entry.isSymbolicLink()) warnings.push({ code: "SYMLINK_IGNORED", path: `work/${entry.name}` });
87
+ continue;
88
+ }
89
+ const workDirectory = assertWithin(journalRoot, path.join(workRoot, entry.name));
90
+ const workFile = path.join(workDirectory, "work.md");
91
+ try {
92
+ const stat = fs.lstatSync(workFile);
93
+ if (stat.isFile() && !stat.isSymbolicLink()) {
94
+ files.push({
95
+ absolute: workFile,
96
+ relative: posix(path.relative(journalRoot, workFile)),
97
+ workSlug: entry.name,
98
+ area: "work",
99
+ kind: "work",
100
+ ...fingerprint(stat),
101
+ });
102
+ }
103
+ } catch (error) { if (error.code !== "ENOENT") throw error; }
104
+ for (const area of ENTRY_DIRECTORIES) {
105
+ listMarkdown(path.join(workDirectory, area), journalRoot, entry.name, area, warnings, files);
106
+ }
107
+ }
108
+ return { files: files.sort((a, b) => a.relative.localeCompare(b.relative)), warnings };
109
+ }
110
+
111
+ function stableRead(descriptor) {
112
+ for (let attempt = 0; attempt < 2; attempt += 1) {
113
+ const before = fs.lstatSync(descriptor.absolute);
114
+ if (!before.isFile() || before.isSymbolicLink()) {
115
+ throw new RecallCorpusError(`source is not a regular file: ${descriptor.relative}`, "UNSAFE_PATH");
116
+ }
117
+ if (before.size > MAX_MARKDOWN_BYTES) {
118
+ throw new RecallCorpusError(`journal file exceeds ${MAX_MARKDOWN_BYTES} bytes: ${descriptor.relative}`, "RECALL_FILE_TOO_LARGE");
119
+ }
120
+ const buffer = fs.readFileSync(descriptor.absolute);
121
+ const after = fs.lstatSync(descriptor.absolute);
122
+ if (sameFingerprint(fingerprint(before), fingerprint(after))) {
123
+ return { buffer, stat: fingerprint(after), sha256: sha256(buffer) };
124
+ }
125
+ }
126
+ throw new RecallCorpusError(`journal file changed while indexing: ${descriptor.relative}`, "RECALL_SOURCE_CHANGED");
127
+ }
128
+
129
+ function parseMarkdown(buffer, relative, warnings) {
130
+ const text = buffer.toString("utf8");
131
+ if (!text.startsWith("---\n") && !text.startsWith("---\r\n")) return { metadata: {}, body: text };
132
+ const match = text.match(/^---\r?\n([\s\S]*?)\r?\n---(?:\r?\n|$)/);
133
+ if (!match) {
134
+ warnings.push({ code: "MALFORMED_FRONTMATTER", path: relative, detail: "unterminated" });
135
+ return { metadata: {}, body: text };
136
+ }
137
+ try {
138
+ const metadata = YAML.parse(match[1]) || {};
139
+ return { metadata: typeof metadata === "object" && !Array.isArray(metadata) ? metadata : {}, body: text.slice(match[0].length) };
140
+ } catch (error) {
141
+ warnings.push({ code: "MALFORMED_FRONTMATTER", path: relative, detail: error.message.split("\n")[0] });
142
+ return { metadata: {}, body: text.slice(match[0].length) };
143
+ }
144
+ }
145
+
146
+ function firstHeading(body, fallback) {
147
+ const match = body.match(/^#\s+(.+)$/m);
148
+ return match ? match[1].trim() : fallback;
149
+ }
150
+
151
+ function headings(body) {
152
+ return [...body.matchAll(/^#{1,6}\s+(.+)$/gm)].map((match) => match[1].trim()).join("\n");
153
+ }
154
+
155
+ function legacyTitle(relative) {
156
+ return path.basename(relative, path.extname(relative))
157
+ .replace(/^\d{4}-\d{2}-\d{2}-\d{2}-/, "")
158
+ .replace(/[-_]+/g, " ")
159
+ .replace(/\b\w/g, (letter) => letter.toUpperCase());
160
+ }
161
+
162
+ function scalar(value, fallback = "") {
163
+ return typeof value === "string" || typeof value === "number" ? String(value) : fallback;
164
+ }
165
+
166
+ function normalizeLinks(value) {
167
+ if (!Array.isArray(value)) return [];
168
+ return value.filter((link) => link && typeof link === "object").map((link) => ({
169
+ id: scalar(link.id),
170
+ fromEntryId: scalar(link.fromEntryId),
171
+ toEntryId: scalar(link.toEntryId),
172
+ relation: scalar(link.relation),
173
+ createdAt: scalar(link.createdAt),
174
+ targetPath: scalar(link.targetPath),
175
+ }));
176
+ }
177
+
178
+ function inferredEntryType(area, relative) {
179
+ if (area === "decisions") return "decision";
180
+ if (area === "docs") return "doc";
181
+ if (area === "_research") return "research";
182
+ const name = path.basename(relative).toLowerCase();
183
+ if (name.includes("plan")) return "plan";
184
+ if (name.includes("status")) return "status";
185
+ return "manual";
186
+ }
187
+
188
+ function rawId(kind, metadata, relative) {
189
+ const value = scalar(metadata.id);
190
+ return value || `path_${sha256(relative).slice(0, 24)}`;
191
+ }
192
+
193
+ function logicalPath(relative) {
194
+ return `.journal/${posix(relative)}`;
195
+ }
196
+
197
+ function readWorkContexts(files, warnings) {
198
+ const result = new Map();
199
+ for (const descriptor of files.filter((file) => file.kind === "work")) {
200
+ const read = stableRead(descriptor);
201
+ const parsed = parseMarkdown(read.buffer, descriptor.relative, warnings);
202
+ const metadata = parsed.metadata;
203
+ result.set(descriptor.workSlug, {
204
+ descriptor,
205
+ read,
206
+ metadata,
207
+ body: parsed.body,
208
+ workItemId: scalar(metadata.id),
209
+ title: scalar(metadata.title, descriptor.workSlug),
210
+ description: scalar(metadata.description),
211
+ status: scalar(metadata.status, "unknown"),
212
+ visibility: scalar(metadata.visibility, "local_only"),
213
+ });
214
+ }
215
+ return result;
216
+ }
217
+
218
+ function documentFromDescriptor(descriptor, workContexts, warnings, existingRead) {
219
+ const work = workContexts.get(descriptor.workSlug) || {
220
+ workItemId: "",
221
+ title: descriptor.workSlug,
222
+ description: "",
223
+ status: "unknown",
224
+ visibility: "local_only",
225
+ };
226
+ const read = existingRead || stableRead(descriptor);
227
+ const parsed = descriptor.kind === "work" && work.descriptor?.relative === descriptor.relative
228
+ ? { metadata: work.metadata, body: work.body }
229
+ : parseMarkdown(read.buffer, descriptor.relative, warnings);
230
+ const metadata = parsed.metadata;
231
+ const fallbackTitle = legacyTitle(descriptor.relative);
232
+ const itemId = rawId(descriptor.kind, metadata, descriptor.relative);
233
+ const kind = descriptor.kind;
234
+ const entryType = kind === "entry" ? scalar(metadata.entryType, inferredEntryType(descriptor.area, descriptor.relative)) : "work";
235
+ return {
236
+ document: {
237
+ id: `${kind}:${itemId}`,
238
+ rawId: scalar(metadata.id, itemId),
239
+ kind,
240
+ path: logicalPath(descriptor.relative),
241
+ sourceRelative: descriptor.relative,
242
+ workSlug: descriptor.workSlug,
243
+ workItemId: kind === "work" ? scalar(metadata.id, work.workItemId) : scalar(metadata.workItemId, work.workItemId),
244
+ workTitle: kind === "work" ? scalar(metadata.title, work.title) : work.title,
245
+ workDescription: kind === "work" ? scalar(metadata.description, work.description) : work.description,
246
+ title: scalar(metadata.title, firstHeading(parsed.body, fallbackTitle)),
247
+ summary: scalar(metadata.summary, kind === "work" ? scalar(metadata.description) : ""),
248
+ headings: headings(parsed.body),
249
+ body: parsed.body,
250
+ entryType,
251
+ status: work.status,
252
+ visibility: work.visibility,
253
+ createdAt: scalar(metadata.createdAt),
254
+ updatedAt: scalar(metadata.updatedAt),
255
+ links: normalizeLinks(metadata.links),
256
+ incomingLinks: [],
257
+ },
258
+ manifest: {
259
+ size: read.stat.size,
260
+ mtimeMs: read.stat.mtimeMs,
261
+ sha256: read.sha256,
262
+ documentId: `${kind}:${itemId}`,
263
+ workSlug: descriptor.workSlug,
264
+ kind,
265
+ },
266
+ };
267
+ }
268
+
269
+ function normalizeDuplicateIds(records, warnings) {
270
+ const seen = new Map();
271
+ for (const record of records) {
272
+ const prior = seen.get(record.document.id);
273
+ if (!prior) {
274
+ seen.set(record.document.id, record.document.path);
275
+ continue;
276
+ }
277
+ warnings.push({ code: "DUPLICATE_DOCUMENT_ID", path: record.document.path, detail: prior });
278
+ const replacement = `${record.document.kind}:path_${sha256(record.document.sourceRelative).slice(0, 24)}`;
279
+ record.document.id = replacement;
280
+ record.manifest.documentId = replacement;
281
+ }
282
+ }
283
+
284
+ function deriveIncomingLinks(records, warnings) {
285
+ const byRawId = new Map();
286
+ for (const record of records.filter((item) => item.document.kind === "entry")) {
287
+ if (!byRawId.has(record.document.rawId)) byRawId.set(record.document.rawId, record.document);
288
+ }
289
+ for (const record of records) {
290
+ for (const link of record.document.links) {
291
+ const target = byRawId.get(link.toEntryId);
292
+ if (!target) {
293
+ if (link.toEntryId) warnings.push({ code: "UNRESOLVED_LINK", path: record.document.path, detail: link.toEntryId });
294
+ continue;
295
+ }
296
+ target.incomingLinks.push({
297
+ fromEntryId: record.document.rawId,
298
+ relation: link.relation,
299
+ sourcePath: record.document.path,
300
+ });
301
+ }
302
+ }
303
+ }
304
+
305
+ function buildCorpus(context, discovery = discoverSourceFiles(context)) {
306
+ const warnings = [...discovery.warnings];
307
+ const workContexts = readWorkContexts(discovery.files, warnings);
308
+ const workReads = new Map([...workContexts.values()].map((work) => [work.descriptor.relative, work.read]));
309
+ const records = discovery.files.map((descriptor) => documentFromDescriptor(descriptor, workContexts, warnings, workReads.get(descriptor.relative)));
310
+ normalizeDuplicateIds(records, warnings);
311
+ deriveIncomingLinks(records, warnings);
312
+ return { records, files: discovery.files, warnings };
313
+ }
314
+
315
+ module.exports = {
316
+ ENTRY_DIRECTORIES,
317
+ MAX_MARKDOWN_BYTES,
318
+ RecallCorpusError,
319
+ buildCorpus,
320
+ documentFromDescriptor,
321
+ discoverSourceFiles,
322
+ fingerprint,
323
+ readWorkContexts,
324
+ sameFingerprint,
325
+ sha256,
326
+ stableRead,
327
+ };
@@ -0,0 +1,56 @@
1
+ "use strict";
2
+
3
+ const { performance } = require("node:perf_hooks");
4
+ const { projectContext } = require("../installer/index.js");
5
+ const { searchIndex } = require("./search.js");
6
+ const { ensureSnapshot } = require("./snapshot.js");
7
+
8
+ function contextFor(options = {}) {
9
+ const target = options.target || process.cwd();
10
+ return options.context || projectContext(target, options);
11
+ }
12
+
13
+ function cacheSummary(ensured, timings = {}) {
14
+ return {
15
+ mode: ensured.mode,
16
+ status: ensured.status,
17
+ documentCount: Object.keys(ensured.snapshot.documents || {}).length,
18
+ builtAt: ensured.snapshot.builtAt,
19
+ ...timings,
20
+ };
21
+ }
22
+
23
+ function buildRecallIndex(options = {}) {
24
+ const started = performance.now();
25
+ const context = contextFor(options);
26
+ const ensured = ensureSnapshot(context, options);
27
+ return {
28
+ action: "recall-index",
29
+ target: context.target,
30
+ projectKey: ensured.snapshot.projectKey,
31
+ cache: cacheSummary(ensured, { totalMs: performance.now() - started }),
32
+ warnings: ensured.warnings,
33
+ };
34
+ }
35
+
36
+ function searchRecall(query, options = {}) {
37
+ const started = performance.now();
38
+ const context = contextFor(options);
39
+ const ensureStarted = performance.now();
40
+ const ensured = ensureSnapshot(context, options);
41
+ const ensureMs = performance.now() - ensureStarted;
42
+ const searchStarted = performance.now();
43
+ const results = searchIndex(ensured.index, query, { ...options, incomingLinks: ensured.snapshot.incomingLinks });
44
+ const searchMs = performance.now() - searchStarted;
45
+ return {
46
+ action: "recall-search",
47
+ target: context.target,
48
+ projectKey: ensured.snapshot.projectKey,
49
+ query,
50
+ cache: cacheSummary(ensured, { ensureMs, searchMs, totalMs: performance.now() - started }),
51
+ ...results,
52
+ warnings: ensured.warnings,
53
+ };
54
+ }
55
+
56
+ module.exports = { buildRecallIndex, searchRecall };
@@ -0,0 +1,93 @@
1
+ "use strict";
2
+
3
+ const MiniSearch = require("minisearch");
4
+
5
+ const INDEX_VERSION = "minisearch-v1";
6
+ const INDEX_OPTIONS = Object.freeze({
7
+ idField: "id",
8
+ fields: ["path", "workTitle", "workDescription", "title", "summary", "headings", "body"],
9
+ storeFields: [
10
+ "rawId", "kind", "path", "workSlug", "workItemId", "workTitle", "title", "summary",
11
+ "entryType", "status", "visibility", "createdAt", "updatedAt", "links", "incomingLinks",
12
+ ],
13
+ });
14
+
15
+ function createSearchIndex(documents = []) {
16
+ const index = new MiniSearch(INDEX_OPTIONS);
17
+ if (documents.length) index.addAll(documents);
18
+ return index;
19
+ }
20
+
21
+ function loadSearchIndex(value) {
22
+ return MiniSearch.loadJSON(JSON.stringify(value), INDEX_OPTIONS);
23
+ }
24
+
25
+ function serializeSearchIndex(index) {
26
+ return JSON.parse(JSON.stringify(index));
27
+ }
28
+
29
+ function list(value) {
30
+ if (!value) return [];
31
+ return Array.isArray(value) ? value : String(value).split(",").map((item) => item.trim()).filter(Boolean);
32
+ }
33
+
34
+ function resultShape(result, incomingLinks) {
35
+ return {
36
+ id: result.rawId,
37
+ documentId: result.id,
38
+ path: result.path,
39
+ workSlug: result.workSlug,
40
+ workItemId: result.workItemId,
41
+ title: result.title,
42
+ summary: result.summary,
43
+ entryType: result.entryType,
44
+ status: result.status,
45
+ visibility: result.visibility,
46
+ createdAt: result.createdAt,
47
+ updatedAt: result.updatedAt,
48
+ score: result.score,
49
+ terms: result.terms,
50
+ match: result.match,
51
+ links: result.links || [],
52
+ incomingLinks: incomingLinks?.[result.rawId] || [],
53
+ };
54
+ }
55
+
56
+ function searchIndex(index, query, options = {}) {
57
+ if (typeof query !== "string" || !query.trim()) {
58
+ const error = new Error("recall search requires a non-empty query");
59
+ error.code = "USAGE";
60
+ throw error;
61
+ }
62
+ const types = new Set(list(options.types || options.type));
63
+ const kinds = new Set(list(options.kinds || options.kind));
64
+ const statuses = new Set(list(options.status));
65
+ const visibilities = new Set(list(options.visibility));
66
+ const limit = Number.isInteger(options.limit) && options.limit > 0 ? Math.min(options.limit, 100) : 10;
67
+ const results = index.search(query.trim(), {
68
+ boost: { title: 7, workTitle: 5, summary: 4, path: 3, workDescription: 2, headings: 2, body: 1 },
69
+ prefix: (term) => term.length >= 3,
70
+ fuzzy: (term) => term.length >= 5 ? 0.2 : false,
71
+ filter: (result) => {
72
+ if (options.work && result.workSlug !== options.work) return false;
73
+ if (types.size && !types.has(result.entryType)) return false;
74
+ if (kinds.size && !kinds.has(result.kind)) return false;
75
+ if (statuses.size && !statuses.has(result.status)) return false;
76
+ if (visibilities.size && !visibilities.has(result.visibility)) return false;
77
+ return true;
78
+ },
79
+ });
80
+ return {
81
+ workItems: results.filter((result) => result.kind === "work").slice(0, limit).map((result) => resultShape(result, options.incomingLinks)),
82
+ entries: results.filter((result) => result.kind === "entry").slice(0, limit).map((result) => resultShape(result, options.incomingLinks)),
83
+ };
84
+ }
85
+
86
+ module.exports = {
87
+ INDEX_OPTIONS,
88
+ INDEX_VERSION,
89
+ createSearchIndex,
90
+ loadSearchIndex,
91
+ searchIndex,
92
+ serializeSearchIndex,
93
+ };
@@ -0,0 +1,304 @@
1
+ "use strict";
2
+
3
+ const crypto = require("node:crypto");
4
+ const fs = require("node:fs");
5
+ const path = require("node:path");
6
+ const {
7
+ buildCorpus,
8
+ discoverSourceFiles,
9
+ documentFromDescriptor,
10
+ readWorkContexts,
11
+ sameFingerprint,
12
+ sha256,
13
+ } = require("./corpus.js");
14
+ const {
15
+ INDEX_VERSION,
16
+ createSearchIndex,
17
+ loadSearchIndex,
18
+ serializeSearchIndex,
19
+ } = require("./search.js");
20
+
21
+ const SNAPSHOT_SCHEMA_VERSION = 1;
22
+ const MAX_SNAPSHOT_BYTES = 256 * 1024 * 1024;
23
+ const LOCK_STALE_MS = 30_000;
24
+
25
+ class RecallSnapshotError extends Error {
26
+ constructor(message, code = "RECALL_SNAPSHOT_ERROR") {
27
+ super(message);
28
+ this.name = "RecallSnapshotError";
29
+ this.code = code;
30
+ }
31
+ }
32
+
33
+ function projectIdentity(context) {
34
+ return context.projectKey || `local-${sha256(path.resolve(context.root || context.target)).slice(0, 16)}`;
35
+ }
36
+
37
+ function cachePaths(context, options = {}) {
38
+ const directory = path.resolve(options.cacheDir || path.join(context.root || context.target, "cache", "recall"));
39
+ return { directory, snapshot: path.join(directory, "snapshot.json"), lock: path.join(directory, "refresh.lock") };
40
+ }
41
+
42
+ function warning(code, detail, file) {
43
+ return { code, ...(file ? { path: file } : {}), ...(detail ? { detail } : {}) };
44
+ }
45
+
46
+ function validManifest(value) {
47
+ return value && typeof value === "object" && !Array.isArray(value) && Object.values(value).every((item) => (
48
+ item && typeof item === "object" && Number.isFinite(item.size) && Number.isFinite(item.mtimeMs) &&
49
+ typeof item.sha256 === "string" && typeof item.documentId === "string"
50
+ ));
51
+ }
52
+
53
+ function validateSnapshot(value, context) {
54
+ if (!value || typeof value !== "object" || Array.isArray(value)) return false;
55
+ if (value.schemaVersion !== SNAPSHOT_SCHEMA_VERSION || value.indexVersion !== INDEX_VERSION) return false;
56
+ if (value.projectKey !== projectIdentity(context) || typeof value.builtAt !== "string") return false;
57
+ if (!validManifest(value.files) || !value.documents || typeof value.documents !== "object" || !value.index) return false;
58
+ return true;
59
+ }
60
+
61
+ function loadSnapshot(context, options = {}) {
62
+ const file = cachePaths(context, options).snapshot;
63
+ try {
64
+ const stat = fs.statSync(file);
65
+ if (!stat.isFile() || stat.size > MAX_SNAPSHOT_BYTES) {
66
+ return { snapshot: null, warnings: [warning("RECALL_CACHE_INVALID", "snapshot is not a bounded regular file")] };
67
+ }
68
+ const value = JSON.parse(fs.readFileSync(file, "utf8"));
69
+ if (!validateSnapshot(value, context)) {
70
+ return { snapshot: null, warnings: [warning("RECALL_CACHE_INCOMPATIBLE", "snapshot schema, index version, or project identity differs")] };
71
+ }
72
+ return { snapshot: value, warnings: [] };
73
+ } catch (error) {
74
+ if (error.code === "ENOENT") return { snapshot: null, warnings: [] };
75
+ return { snapshot: null, warnings: [warning("RECALL_CACHE_INVALID", error.message)] };
76
+ }
77
+ }
78
+
79
+ function acquireLock(paths, options = {}) {
80
+ fs.mkdirSync(paths.directory, { recursive: true, mode: 0o700 });
81
+ try { fs.chmodSync(paths.directory, 0o700); } catch {}
82
+ try {
83
+ const descriptor = fs.openSync(paths.lock, "wx", 0o600);
84
+ fs.writeFileSync(descriptor, `${JSON.stringify({ pid: process.pid, createdAt: new Date().toISOString() })}\n`);
85
+ fs.closeSync(descriptor);
86
+ return () => { try { fs.unlinkSync(paths.lock); } catch {} };
87
+ } catch (error) {
88
+ if (error.code !== "EEXIST") throw error;
89
+ try {
90
+ const stat = fs.statSync(paths.lock);
91
+ const now = Number(options.now ?? Date.now());
92
+ if (Number.isFinite(now) && now - stat.mtimeMs > (options.lockStaleMs || LOCK_STALE_MS)) {
93
+ fs.unlinkSync(paths.lock);
94
+ return acquireLock(paths, options);
95
+ }
96
+ } catch (retryError) {
97
+ if (retryError.code === "ENOENT") return acquireLock(paths, options);
98
+ }
99
+ return null;
100
+ }
101
+ }
102
+
103
+ function inspectSnapshot(context, options = {}) {
104
+ const paths = cachePaths(context, options);
105
+ const loaded = loadSnapshot(context, options);
106
+ const exists = fs.existsSync(paths.snapshot);
107
+ let persistenceAvailable = false;
108
+ try {
109
+ let probe = paths.directory;
110
+ while (!fs.existsSync(probe) && path.dirname(probe) !== probe) probe = path.dirname(probe);
111
+ fs.accessSync(probe, fs.constants.W_OK);
112
+ persistenceAvailable = true;
113
+ } catch {}
114
+ return {
115
+ path: paths.snapshot,
116
+ exists,
117
+ valid: Boolean(loaded.snapshot),
118
+ persistenceAvailable,
119
+ schemaVersion: loaded.snapshot?.schemaVersion,
120
+ indexVersion: loaded.snapshot?.indexVersion,
121
+ documentCount: loaded.snapshot ? Object.keys(loaded.snapshot.documents).length : 0,
122
+ builtAt: loaded.snapshot?.builtAt,
123
+ warnings: loaded.warnings,
124
+ };
125
+ }
126
+
127
+ function writeSnapshot(context, snapshot, options = {}) {
128
+ const paths = cachePaths(context, options);
129
+ let release;
130
+ try {
131
+ release = acquireLock(paths, options);
132
+ if (!release) return { persisted: false, warning: warning("RECALL_CACHE_LOCKED", "another process is refreshing the cache") };
133
+ const temporary = path.join(paths.directory, `.snapshot.${process.pid}.${crypto.randomBytes(4).toString("hex")}.tmp`);
134
+ try {
135
+ fs.writeFileSync(temporary, `${JSON.stringify(snapshot)}\n`, { mode: 0o600 });
136
+ fs.renameSync(temporary, paths.snapshot);
137
+ } catch (error) {
138
+ try { fs.unlinkSync(temporary); } catch {}
139
+ throw error;
140
+ }
141
+ return { persisted: true };
142
+ } catch (error) {
143
+ return { persisted: false, warning: warning("RECALL_CACHE_UNWRITABLE", error.message) };
144
+ } finally {
145
+ if (release) release();
146
+ }
147
+ }
148
+
149
+ function storedDocument(document) {
150
+ const { body, headings, workDescription, sourceRelative, ...stored } = document;
151
+ return stored;
152
+ }
153
+
154
+ function incomingLinks(documents) {
155
+ const incoming = {};
156
+ for (const document of Object.values(documents)) {
157
+ for (const link of document.links || []) {
158
+ if (!link.toEntryId) continue;
159
+ if (!incoming[link.toEntryId]) incoming[link.toEntryId] = [];
160
+ incoming[link.toEntryId].push({
161
+ fromEntryId: document.rawId,
162
+ relation: link.relation,
163
+ sourcePath: document.path,
164
+ });
165
+ }
166
+ }
167
+ return incoming;
168
+ }
169
+
170
+ function snapshotFromRecords(context, records, index) {
171
+ const files = {};
172
+ const documents = {};
173
+ for (const record of records) {
174
+ files[record.document.sourceRelative] = record.manifest;
175
+ documents[record.document.id] = storedDocument(record.document);
176
+ }
177
+ return {
178
+ schemaVersion: SNAPSHOT_SCHEMA_VERSION,
179
+ indexVersion: INDEX_VERSION,
180
+ projectKey: projectIdentity(context),
181
+ builtAt: new Date().toISOString(),
182
+ files,
183
+ documents,
184
+ incomingLinks: incomingLinks(documents),
185
+ index: serializeSearchIndex(index),
186
+ };
187
+ }
188
+
189
+ function fullBuild(context, discovery) {
190
+ const corpus = buildCorpus(context, discovery);
191
+ const index = createSearchIndex(corpus.records.map((record) => record.document));
192
+ return { index, snapshot: snapshotFromRecords(context, corpus.records, index), warnings: corpus.warnings };
193
+ }
194
+
195
+ function changedPaths(snapshot, discovery, force) {
196
+ const current = new Map(discovery.files.map((file) => [file.relative, file]));
197
+ const changed = new Set();
198
+ const removed = [];
199
+ if (force) for (const relative of current.keys()) changed.add(relative);
200
+ else {
201
+ for (const descriptor of discovery.files) {
202
+ if (!sameFingerprint(snapshot.files[descriptor.relative], descriptor)) changed.add(descriptor.relative);
203
+ }
204
+ for (const relative of Object.keys(snapshot.files)) if (!current.has(relative)) removed.push(relative);
205
+ }
206
+ const affectedWork = new Set();
207
+ for (const relative of changed) if (current.get(relative)?.kind === "work") affectedWork.add(current.get(relative).workSlug);
208
+ for (const relative of removed) if (snapshot.files[relative]?.kind === "work") affectedWork.add(snapshot.files[relative].workSlug);
209
+ for (const descriptor of discovery.files) if (affectedWork.has(descriptor.workSlug)) changed.add(descriptor.relative);
210
+ return { changed, removed, current };
211
+ }
212
+
213
+ function incrementalBuild(context, prior, discovery) {
214
+ const warnings = [...discovery.warnings];
215
+ const index = loadSearchIndex(prior.index);
216
+ const files = { ...prior.files };
217
+ const documents = { ...prior.documents };
218
+ const changes = changedPaths(prior, discovery, false);
219
+ if (!changes.changed.size && !changes.removed.length) return { index, snapshot: prior, warnings, changed: false };
220
+
221
+ for (const relative of changes.removed) {
222
+ const old = files[relative];
223
+ if (old?.documentId && index.has(old.documentId)) index.discard(old.documentId);
224
+ if (old?.documentId) delete documents[old.documentId];
225
+ delete files[relative];
226
+ }
227
+
228
+ const workContexts = readWorkContexts(discovery.files, warnings);
229
+ for (const relative of [...changes.changed].sort()) {
230
+ const descriptor = changes.current.get(relative);
231
+ if (!descriptor) continue;
232
+ const old = files[relative];
233
+ if (old?.documentId && index.has(old.documentId)) index.discard(old.documentId);
234
+ if (old?.documentId) delete documents[old.documentId];
235
+ const record = documentFromDescriptor(descriptor, workContexts, warnings);
236
+ if (index.has(record.document.id)) {
237
+ warnings.push(warning("DUPLICATE_DOCUMENT_ID", record.document.id, record.document.path));
238
+ record.document.id = `${record.document.kind}:path_${sha256(record.document.sourceRelative).slice(0, 24)}`;
239
+ record.manifest.documentId = record.document.id;
240
+ }
241
+ index.add(record.document);
242
+ files[relative] = record.manifest;
243
+ documents[record.document.id] = storedDocument(record.document);
244
+ }
245
+
246
+ const snapshot = {
247
+ schemaVersion: SNAPSHOT_SCHEMA_VERSION,
248
+ indexVersion: INDEX_VERSION,
249
+ projectKey: projectIdentity(context),
250
+ builtAt: new Date().toISOString(),
251
+ files,
252
+ documents,
253
+ incomingLinks: incomingLinks(documents),
254
+ index: serializeSearchIndex(index),
255
+ };
256
+ return { index, snapshot, warnings, changed: true };
257
+ }
258
+
259
+ function ensureSnapshot(context, options = {}) {
260
+ const discovery = discoverSourceFiles(context);
261
+ const loaded = options.noCache || options.rebuild ? { snapshot: null, warnings: [] } : loadSnapshot(context, options);
262
+ let built;
263
+ let status;
264
+ if (!loaded.snapshot || options.rebuild || options.noCache) {
265
+ built = fullBuild(context, discovery);
266
+ status = "rebuilt";
267
+ } else {
268
+ try {
269
+ built = incrementalBuild(context, loaded.snapshot, discovery);
270
+ status = built.changed ? "refreshed" : "hit";
271
+ } catch (error) {
272
+ loaded.warnings.push(warning("RECALL_CACHE_REBUILD", error.message));
273
+ built = fullBuild(context, discovery);
274
+ status = "rebuilt";
275
+ }
276
+ }
277
+
278
+ const warnings = [...loaded.warnings, ...built.warnings];
279
+ let mode = "persistent";
280
+ if (options.noCache) mode = "memory";
281
+ else if (status !== "hit") {
282
+ const written = writeSnapshot(context, built.snapshot, options);
283
+ if (!written.persisted) {
284
+ mode = "memory";
285
+ if (written.warning) warnings.push(written.warning);
286
+ }
287
+ }
288
+ return { index: built.index, snapshot: built.snapshot, mode, status, warnings };
289
+ }
290
+
291
+ module.exports = {
292
+ LOCK_STALE_MS,
293
+ MAX_SNAPSHOT_BYTES,
294
+ RecallSnapshotError,
295
+ SNAPSHOT_SCHEMA_VERSION,
296
+ cachePaths,
297
+ ensureSnapshot,
298
+ incomingLinks,
299
+ inspectSnapshot,
300
+ loadSnapshot,
301
+ projectIdentity,
302
+ validateSnapshot,
303
+ writeSnapshot,
304
+ };
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "djournal",
3
- "version": "0.8.0",
3
+ "version": "0.9.1",
4
4
  "description": "Durable project memory for coding agents, stored as linked Markdown",
5
5
  "type": "commonjs",
6
6
  "bin": {
@@ -24,9 +24,10 @@
24
24
  "scripts": {
25
25
  "check:package": "node scripts/check-package.cjs",
26
26
  "check:pr-title": "node scripts/check-pr-title.js",
27
+ "benchmark:recall": "node scripts/benchmark-recall.cjs",
27
28
  "link:global": "npm link",
28
29
  "release": "semantic-release",
29
- "test": "node tests/update-checker.test.js && node tests/journal-hook.test.js && node tests/pi-adapter.test.js && node tests/installer/installer.test.js && node tests/release-policy.test.js && node tests/skill-instructions.test.js && npm run check:package",
30
+ "test": "node tests/update-checker.test.js && node tests/journal-hook.test.js && node tests/pi-adapter.test.js && node tests/recall.test.js && node tests/installer/installer.test.js && node tests/release-policy.test.js && node tests/skill-instructions.test.js && npm run check:package",
30
31
  "prepublishOnly": "npm test"
31
32
  },
32
33
  "repository": {
@@ -56,6 +57,8 @@
56
57
  "semantic-release": "25.0.3"
57
58
  },
58
59
  "dependencies": {
59
- "semver": "7.8.5"
60
+ "minisearch": "7.2.0",
61
+ "semver": "7.8.5",
62
+ "yaml": "2.9.0"
60
63
  }
61
64
  }