@howaboua/pi-agent-comms 0.0.1 → 0.0.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,16 @@
1
1
  # @howaboua/pi-agent-comms
2
2
 
3
+ ## 0.0.2
4
+
5
+ - Notes lookup warnings now distinguish unavailable remote notes from local checkpoints.
6
+ - Notes guidance calls for fewer, shorter updates, using task notes and/or checkpoints with History links instead of copied investigation or harness detail.
7
+ - Notes guidance keeps no-action report updates to their ID and outcome when checkpointing is needed.
8
+ - Retired session records are removed automatically after Pi exits.
9
+ - Board guidance skips report posts without a new peer-relevant change.
10
+ - Board guidance keeps distinct topics in focused threads instead of catch-alls.
11
+ - Communication guidance distinguishes non-waking notes and board posts from direct messages, regardless of messaging tool.
12
+ - Notes and Board settings now open on matching Enabled controls.
13
+
3
14
  ## 0.0.1
4
15
 
5
16
  - Requires Node.js 22.19 or newer.
package/README.md CHANGED
@@ -37,7 +37,7 @@ Existing notes, history, settings, model roles, board archives, subscriptions, v
37
37
  - Tab or Shift-Tab switches between Notes and Board. `/board settings` remains an alias for the Board tab.
38
38
  - `/roles` opens Notes' model roles subview, also available through **Model roles** in Notes.
39
39
 
40
- Notes defaults to **All models**. Its **Active** setting also offers **Whitelist** and **Off**. Board defaults to on and retains `/board on|off [session|folder|global]` and `/board status`. Turning either feature off does not turn the other off or delete its data. Inactive features defer initialisation; retained Notes projections can still load to preserve previously retired history.
40
+ Both tabs open on **Enabled**. Notes offers **Off**, **On** (all models, the default) and **Whitelist**. Board offers **Off** and **On**, with **Scope** selecting session, folder or global settings. Board also retains `/board on|off [session|folder|global]` and `/board status`. Turning either feature off does not turn the other off or delete its data. Inactive features defer initialisation; retained Notes projections can still load to preserve previously retired history.
41
41
 
42
42
  ## Guides
43
43
 
package/docs/BOARD.md CHANGED
@@ -27,7 +27,7 @@ Reload Pi after installation. Ask the agent to post a note to the board, then ru
27
27
  ## Board settings
28
28
 
29
29
  - `/board` opens the Board tab. `/notes` opens the Notes tab. Tab or Shift-Tab switches between them.
30
- - `/board settings` is an alias for the Board tab. It offers enablement and scope, **Open viewer**, consolidation, automatic summaries and the global summary model: Luna or Custom.
30
+ - `/board settings` is an alias for the Board tab. **Enabled** switches Board on or off; **Scope** selects session, folder or global settings. The tab also offers **Open viewer**, consolidation, automatic summaries and the global summary model: Luna or Custom.
31
31
  - `/board status` shows the current board and enablement.
32
32
  - `/board on` enables the board for this session.
33
33
  - `/board off` disables it without deleting history.
package/docs/NOTES.md CHANGED
@@ -44,6 +44,18 @@ Idle rollover is off by default. Choose 25m, 55m, 2h, 6h or 12h in `/notes`. Hou
44
44
 
45
45
  ## Management
46
46
 
47
+ ### Session pruning
48
+
49
+ Open **Session auto-pruning** in `/notes` to set this session's retention limits. **Run now** previews and confirms irreversible removal of retired payloads. Automatic pruning runs after a new window's first completed turn. Saved SQLite notes and current context remain intact. Settings follow the session on resume. New and forked sessions start with pruning off.
50
+
51
+ Live pruning keeps every history entry ID so Pi can still navigate its in-memory tree safely. With retention enabled, Notes automatically removes retired empty records after Pi exits. A separate helper waits for the actual process to end before changing the saved tree. Reloading, forking and switching sessions do not trigger this cleanup. No command or additional confirmation is needed. No backup is kept. Memory savings apply when the pruned session next opens.
52
+
53
+ Physical cleanup currently requires Linux and an available `node` executable. Other platforms retain live payload pruning. Cleanup skips another registered local Notes process using the same saved session, known open-file holders and files that change during preparation. A simultaneous reopen is checked against the saved generation. If cleanup won that race, the new Pi process stops and asks you to reopen it rather than continue with a stale tree. Failures are reported when the session next opens. Tree branches, labels, compaction boundaries and supported Notes archive references are retained. Other content and saved notes are unchanged.
54
+
55
+ Coordination covers local processes running this Notes release under the same OS user. Keep session storage free of concurrent writers from other machines, other OS users, older Notes versions and external JSONL editors while a Notes-enabled process is closing. Unrelated sessions do not block cleanup.
56
+
57
+ Update Observatory before applying if you use its historical index. Older versions may block the rewritten file. Current versions reconcile removed records without discarding indexed statistics.
58
+
47
59
  ### Model roles
48
60
 
49
61
  `/roles` opens the read-only model roles subview of Notes. You can also choose
@@ -89,7 +101,7 @@ recovery keep their existing Notes settings.
89
101
 
90
102
  `/notes` opens the settings panel in interactive Pi. It shows continuity status, storage counts and remote capabilities. Changes save immediately.
91
103
 
92
- - **Active** selects **Off**, **All models** (the default) or **Whitelist**. Whitelist enables Notes only for exact model IDs in the global settings file, regardless of provider. Off disables Notes independently of Board. Switching models applies the choice immediately; other models keep their installed compaction behaviour. Inactive Notes defers initialisation, but retained context projections can still load to preserve previously retired history.
104
+ - **Enabled** selects **Off**, **On** (all models, the default) or **Whitelist**. Whitelist enables Notes only for exact model IDs in the global settings file, regardless of provider. Off disables Notes independently of Board. Switching models applies the choice immediately; other models keep their installed compaction behaviour. Inactive Notes defers initialisation, but retained context projections can still load to preserve previously retired history.
93
105
  - **Normal compaction** uses the installed Pi or PCC compaction flow for `/compact` and `new_context`. Off keeps notes-only rollover, the default.
94
106
  - **New window requests** controls the agent's `new_context` tool independently of the compaction method. **off** disables the tool. **pi-limits** uses Pi's eligibility check and continues in the current window when there is too little history. **unrestricted**, the default, accepts requests at any size. With normal compaction on and nothing to summarise, unrestricted requests open a new window identity without discarding the existing context. Manual `/compact` and idle rollover retain their separate policies.
95
107
  - **Idle rollover** selects the interval or disables it.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@howaboua/pi-agent-comms",
3
- "version": "0.0.1",
3
+ "version": "0.0.2",
4
4
  "description": "Session notes, context continuity and agent message boards for Pi",
5
5
  "type": "module",
6
6
  "license": "MIT",
@@ -55,7 +55,7 @@
55
55
  "unicode-case-folding": "1.1.2"
56
56
  },
57
57
  "devDependencies": {
58
- "@howaboua/pi-codex-conversion": "4.0.10",
58
+ "@howaboua/pi-codex-conversion": "5.0.0",
59
59
  "@types/better-sqlite3": "9.6.0"
60
60
  },
61
61
  "peerDependencies": {
@@ -63,7 +63,7 @@
63
63
  "@earendil-works/pi-agent-core": ">=1.0.4",
64
64
  "@earendil-works/pi-ai": ">=1.1.0",
65
65
  "@earendil-works/pi-tui": ">=1.1.0",
66
- "@howaboua/pi-codex-conversion": ">=4.0.10"
66
+ "@howaboua/pi-codex-conversion": ">=5.0.0"
67
67
  },
68
68
  "peerDependenciesMeta": {
69
69
  "@howaboua/pi-codex-conversion": {
@@ -17,8 +17,8 @@ export function boardBriefing(
17
17
  state,
18
18
  member ? "Follow your assigned scope." : "",
19
19
  gpt
20
- ? "Channels=workstreams; threads=topics. Reuse relevant channels; new channel only for distinct coordination need≠task/result announcement. Read≠subscribe; subscribe to threads→replies, channels→new threads."
21
- : "Channels group workstreams, threads group topics. Reuse a relevant channel. Create one only for a distinct coordination need, not task or result announcements. Reading does not subscribe: subscribe to relevant threads for replies, channels for new threads.",
20
+ ? "Channels=workstreams; threads=focused topics, not catch-alls. Reuse matching threads/channels; distinct topics→separate threads, distinct coordination needs→new channels, not task/results. Read≠subscribe; threads→replies, channels→new threads."
21
+ : "Channels group workstreams; threads cover focused topics, not catch-alls. Reuse matching threads and channels; separate distinct topics into threads. Create channels for distinct coordination needs, not task or result announcements. Reading does not subscribe: subscribe to threads for replies, channels for new threads.",
22
22
  ]
23
23
  .filter(Boolean)
24
24
  .join(" ");
@@ -33,7 +33,7 @@ export async function openBoardSettings(
33
33
  return;
34
34
  }
35
35
  let scope: BoardScope = "session";
36
- let selected = "scope";
36
+ let selected = "enabled";
37
37
  const sessionId = ctx.sessionManager.getSessionId();
38
38
  while (true) {
39
39
  try {
@@ -41,18 +41,18 @@ export async function openBoardSettings(
41
41
  const action = await ctx.ui.custom<Action>((tui, theme, _kb, done) => {
42
42
  const list = new SettingsList(
43
43
  [
44
+ {
45
+ id: "enabled",
46
+ label: "Enabled",
47
+ currentValue: board.enabled(ctx) ? "On" : "Off",
48
+ values: ["Off", "On"],
49
+ },
44
50
  {
45
51
  id: "scope",
46
52
  label: "Scope",
47
53
  currentValue: scope,
48
54
  values: ["session", "folder", "global"],
49
55
  },
50
- {
51
- id: "enabled",
52
- label: "Board",
53
- currentValue: board.enabled(ctx) ? "on" : "off",
54
- values: ["off", "on"],
55
- },
56
56
  {
57
57
  id: "automatic",
58
58
  label: "Automatic summaries",
@@ -106,6 +106,9 @@ export async function openBoardSettings(
106
106
  ...wrap(
107
107
  `${tabs ? "Tab switch tab · " : ""}↑/↓ select · Enter/Space change · Esc close`,
108
108
  ).map((line) => theme.fg("dim", line)),
109
+ ...wrap("Settings saved immediately").map((line) =>
110
+ theme.fg("dim", line),
111
+ ),
109
112
  rule,
110
113
  ].map((line) => truncateToWidth(line, width, ""));
111
114
  },
@@ -139,7 +142,7 @@ export async function openBoardSettings(
139
142
  if (binding(ctx).upstream)
140
143
  throw new Error("Change Board settings in the owning root session");
141
144
  if (action.id === "enabled")
142
- await board.setSetting(ctx, scope, action.value === "on");
145
+ await board.setSetting(ctx, scope, action.value === "On");
143
146
  else if (action.id === "automatic")
144
147
  board.setAutomaticSummaries(ctx, action.value === "on");
145
148
  else if (action.id === "model") {
package/src/guidance.ts CHANGED
@@ -18,38 +18,40 @@ export interface CommsFeature extends FeatureSettings {
18
18
  function communicationGuidance(
19
19
  notes: boolean,
20
20
  board: boolean,
21
- agents: boolean,
22
21
  gpt = false,
23
22
  ): string {
24
23
  return [
25
24
  board
26
25
  ? gpt
27
- ? "Board→peer findings/discovery, ownership/decisions/blockers affecting peers. Self-contained detail welcome; direct discovery≠parent relay. Existing thread replies; known shared peers→link≠repeat. No ack/status spam/duplicate completion reports."
28
- : "Board: share findings, discoveries, ownership, decisions and blockers that change peers' work. Self-contained detail lets peers read directly, without a parent relay. Reply in existing threads. Link rather than repeat for peers with known Board access. Skip acknowledgements, routine status and duplicate completion reports."
26
+ ? "Board→new peer-work findings/decisions/ownership/blockers only; no change→no post. Peers read directly; no parent relay. No idle wake; active subscribers may be interrupted."
27
+ : "Board: post only findings, decisions, ownership changes or blockers that change peers' work. Peers read posts directly; do not relay them through the parent. Posts do not wake idle agents but may interrupt active subscribers."
29
28
  : "",
30
29
  notes
31
30
  ? [
31
+ gpt
32
+ ? "Notes writes don't wake peers."
33
+ : "Notes writes do not wake peers.",
32
34
  board
33
35
  ? gpt
34
- ? "Notes→current work/progress/recovery + Board references."
35
- : "Notes: current work, progress and recovery, with Board references."
36
+ ? "Notes→recovery; link Board evidence."
37
+ : "Notes holds recovery checkpoints; link Board evidence."
36
38
  : "",
37
39
  gpt
38
- ? "Keep Notes checkpoints. Known recipient access→brief outcome/blocker + path; recipient READS checkpoint≠request full report."
39
- : "Keep checkpoints in Notes. When the recipient is known to have access, return a brief outcome or blocker plus its path. The recipient reads the checkpoint for details rather than asking for a full report.",
40
+ ? "Known recipient Notes access→brief outcome/blocker + checkpoint path; recipient reads checkpoint instead of requesting its contents."
41
+ : "With known recipient Notes access, return a brief outcome or blocker plus the checkpoint path; the recipient reads the checkpoint instead of requesting its contents.",
40
42
  ]
41
43
  .filter(Boolean)
42
44
  .join(" ")
43
45
  : "",
44
46
  notes || board
45
47
  ? gpt
46
- ? `${notes && board ? "Notes/Board" : notes ? "Notes" : "Board"} available here≠recipient access. Unknown/unshared access→self-contained actions taken/findings/unresolved state, not bare references.`
47
- : `${notes && board ? "Notes and Board" : notes ? "Notes" : "Board"} availability here does not establish recipient access. If access is unknown or unshared, send a self-contained account of what you did, found and left unresolved, not just private references.`
48
+ ? `${notes && board ? "Notes/Board" : notes ? "Notes" : "Board"} available here≠recipient access. Known access→link≠repeat; otherwise self-contained actions/findings/unresolved state.`
49
+ : `${notes && board ? "Notes and Board" : notes ? "Notes" : "Board"} availability here does not establish recipient access. Link rather than repeat when access is known; otherwise give a self-contained account of actions, findings and unresolved state.`
48
50
  : "",
49
- agents && notes && board
51
+ notes || board
50
52
  ? gpt
51
- ? "agents→assignments/urgent action≠default peer knowledge bus."
52
- : "Use agents for assignments and urgent action, not as the default peer knowledge relay."
53
+ ? "Direct messages→assignments or urgent action, not status or courtesy updates."
54
+ : "Use direct messages for assignments or urgent action, not status or courtesy updates."
53
55
  : "",
54
56
  ]
55
57
  .filter(Boolean)
@@ -73,11 +75,9 @@ export function registerCommunicationGuidance(
73
75
  (ctx) => Boolean(notes?.enabled(ctx) || board?.enabled(ctx)),
74
76
  async (ctx, windowId, messages) => {
75
77
  // Feature owners supply enablement independently of provider declarations.
76
- // Shepherdr keeps agents active, including as a nested Code/Notebook tool.
77
78
  const capabilities = [
78
79
  notes?.enabled(ctx) ?? false,
79
80
  board?.enabled(ctx) ?? false,
80
- pi.getActiveTools().includes("agents"),
81
81
  Boolean(notes?.gptPromptMode() || board?.gptPromptMode()),
82
82
  ] as const;
83
83
  const routing = communicationGuidance(...capabilities);
@@ -21,7 +21,7 @@
21
21
  - Return initial and rebound identity from before_agent_start, never send it during startup. Pi inserts its initial system entry ahead of returned hook messages; startup sends otherwise put per-session identity before the shared cache prefix.
22
22
  - Automatic rollover uses the successor window message as its custom kickoff. That entry is not yet in the branch during before_agent_start; adopt its identity instead of restoring the outgoing window or returning another boundary.
23
23
  - Compaction handoff reads are transient branch contributions anchored to their persisted summary. Rechain only the projected branch, never append contributions or persist their IDs as Pi's first-kept boundary. Compare projected entry identity, not branch length.
24
- - Session pruning preserves JSONL order/count/IDs and unchanged raw lines; Observatory rebases derived source ranges using header `notesPruning` generations, never a tail receipt alone. Its final tail-copy/rename must contain no await. Released Pi's already-flushed same-path writes are append-only; re-trace SDK rewrite paths when changing this boundary. Do not mutate its live history or coordinate foreign JSONL writers.
24
+ - Live session pruning preserves JSONL order/count/IDs. Sparse cleanup runs only after the owning Pi process exits, never inside shutdown or on reload/fork. Retain roots, leaves, branchpoints, physical last and supported structured reference targets; skip live registered owners. FD checks do not prove an idle manager has closed. Observatory rebases derived ranges using new header `notesPruning` generations, never tail receipts alone. Final validation/rename must contain no await. Never mutate Pi's live history or coordinate foreign JSONL writers.
25
25
  - Pi persists incoming messages after before_agent_start, agent_start and message_end hooks, before returned startup messages. History assigns the kickoff prefix only to the first untrimmed window, never later or identity-only boundaries. Label saved-note continuation prompts in context once their entries exist. Schedule pruning from Notes' admitted window identity, then verify its persisted marker after settlement.
26
26
  - Pruning must not turn removed delivered instruction carriers into pending-message redelivery. Coalesce only proven complete state snapshots per branch leaf; unknown custom protocols can contain deltas. Keep current Notes revisions, monitor snapshots, Board routes and notebook epochs, not historical model-facing text disguised as state.
27
27
  - Role handoffs prepare the outgoing provider before selecting the successor model and starting the normal window kickoff. Keep checkpoint and compaction gates, rechecking session/window after async work. Role guidance exposes IDs/descriptions only. In-run compaction restores its persisted carrier before the summary without a new turn or prefix edit.
@@ -7,6 +7,11 @@ import type {
7
7
  import type { CommsFeature } from "../guidance.js";
8
8
  import type { OpenSettings } from "../settings-navigation.js";
9
9
  import type { NotesRuntime } from "./index.js";
10
+ import {
11
+ admitSessionCleanup,
12
+ previousCleanupFailure,
13
+ } from "./session-pruning-exit.mjs";
14
+ import { cleanupStatus } from "./session-pruning-state.mjs";
10
15
  import {
11
16
  legacyNotesConfigured,
12
17
  loadNotesSettings,
@@ -53,6 +58,43 @@ export function createNotesFeature(
53
58
  let store: NotesStore | undefined;
54
59
  let storage: Promise<NotesStore> | undefined;
55
60
  let closed = false;
61
+ let staleOpening = false;
62
+ let releaseCleanupOwner: (() => void) | undefined;
63
+ const admit = async (ctx: ExtensionContext) => {
64
+ if (releaseCleanupOwner) return !staleOpening;
65
+ let reason: string | undefined;
66
+ try {
67
+ const claim = await admitSessionCleanup(
68
+ ctx.sessionManager.getSessionFile(),
69
+ ctx.sessionManager.getSessionId(),
70
+ ctx.sessionManager.getHeader(),
71
+ );
72
+ releaseCleanupOwner = claim?.release;
73
+ if (claim?.stale)
74
+ reason =
75
+ "This session was cleaned while Pi was opening it. Restart Pi to load its saved tree safely.";
76
+ } catch (error) {
77
+ reason = `Pi could not safely open this session: ${String(error)}. Resolve the problem and restart Pi.`;
78
+ }
79
+ if (!reason) return true;
80
+ staleOpening = true;
81
+ console.error(reason);
82
+ ctx.ui.notify(reason, "error");
83
+ if (ctx.mode === "rpc") {
84
+ // RPC honours startup shutdown after attaching input. Returning lets it
85
+ // reach that check before accepting normal commands.
86
+ ctx.shutdown();
87
+ return false;
88
+ }
89
+ if (ctx.mode === "tui") ctx.shutdown();
90
+ else {
91
+ // Released print/json mode installs graceful SIGTERM handling before
92
+ // session_start, but does not bind the documented ctx.shutdown action.
93
+ process.kill(process.pid, "SIGTERM");
94
+ }
95
+ // Do not finish binding or start an initial prompt/tree while exiting.
96
+ await new Promise<never>(() => {});
97
+ };
56
98
  const ensureStore = () =>
57
99
  (storage ??= (async () => {
58
100
  const { NotesStore } = await import("./store.js");
@@ -110,6 +152,22 @@ export function createNotesFeature(
110
152
  await open(ctx);
111
153
  });
112
154
  pi.on("session_start", async (_event, ctx) => {
155
+ if (!(await admit(ctx))) return;
156
+ const file = ctx.sessionManager.getSessionFile();
157
+ if (file) {
158
+ try {
159
+ const error = previousCleanupFailure(
160
+ cleanupStatus(file, dirname(notesSettingsPath())),
161
+ ctx.sessionManager.getSessionId(),
162
+ );
163
+ if (error) ctx.ui.notify(error, "warning");
164
+ } catch (error) {
165
+ ctx.ui.notify(
166
+ `Previous session cleanup status could not be read: ${String(error)}. Check Notes storage permissions and retry.`,
167
+ "warning",
168
+ );
169
+ }
170
+ }
113
171
  try {
114
172
  settings = loadNotesSettings(ctx);
115
173
  writeSettings(notesSettingsPath(), settings);
@@ -126,12 +184,20 @@ export function createNotesFeature(
126
184
  );
127
185
  }
128
186
  });
187
+ pi.on("agent_settled", async (_event, ctx) => {
188
+ // A brand-new session has no JSONL file until its first persisted turn.
189
+ await admit(ctx);
190
+ });
129
191
  pi.on("model_select", async (_event, ctx) => {
130
192
  if (!runtime && settings && modelIsActive(settings, ctx.model))
131
193
  await ensure(ctx);
132
194
  });
133
- pi.on("session_shutdown", () => {
195
+ pi.on("session_shutdown", (event) => {
134
196
  closed = true;
197
+ if (event.reason !== "quit") {
198
+ releaseCleanupOwner?.();
199
+ releaseCleanupOwner = undefined;
200
+ }
135
201
  store?.close();
136
202
  });
137
203
  pi.registerCommand("notes", {
@@ -1,6 +1,7 @@
1
1
  import { open } from "node:fs/promises";
2
+ import { StringDecoder } from "node:string_decoder";
2
3
 
3
- /** @typedef {{file:string; sessionId:string; generation:string; excluded:Set<string>}} PrunedHistory */
4
+ /** @typedef {{file:string; sessionId:string; generation:string; size:number; surviving:Set<string>; excluded:Set<string>}} PrunedHistory */
4
5
 
5
6
  /** The live SessionManager still contains old text until restart. Read the committed
6
7
  * source so later indexing, settings toggles and /reload cannot resurrect its terms.
@@ -8,11 +9,11 @@ import { open } from "node:fs/promises";
8
9
  export function createHistoryPruning() {
9
10
  /** @type {PrunedHistory | undefined} */
10
11
  let cached;
11
- const empty = new Set();
12
+ const empty = () => false;
12
13
  /** @param {import("./sqlite.mjs").Database} db
13
14
  * @param {string} sessionId
14
15
  * @param {string | undefined} file
15
- * @returns {Promise<ReadonlySet<string>>}
16
+ * @returns {Promise<(id:string) => boolean>}
16
17
  */
17
18
  return async (db, sessionId, file) => {
18
19
  if (!file) return empty;
@@ -25,92 +26,125 @@ export function createHistoryPruning() {
25
26
  return empty;
26
27
  throw error;
27
28
  }
28
- let stream;
29
29
  try {
30
30
  const { size } = await input.stat();
31
31
  if (!size) return empty;
32
- stream = input.createReadStream({
33
- start: 0,
34
- end: size - 1,
35
- autoClose: false,
36
- encoding: "utf8",
37
- });
38
- /** @type {PrunedHistory | undefined} */
39
- let source;
40
- let pending = "";
41
- for await (const chunk of stream) {
42
- pending += chunk;
43
- let start = 0;
44
- let end;
45
- while ((end = pending.indexOf("\n", start)) >= 0) {
46
- const entry = JSON.parse(pending.slice(start, end));
47
- start = end + 1;
48
- if (!source) {
49
- if (entry.type !== "session" || entry.id !== sessionId)
50
- throw new Error("History source identity changed");
51
- const marker = entry.notesPruning;
52
- if (marker === undefined) return empty;
53
- if (
54
- marker?.version !== 1 ||
55
- marker.sessionId !== sessionId ||
56
- typeof marker.generation !== "string"
57
- )
58
- throw new Error("Invalid history pruning marker");
59
- if (
60
- cached?.file === file &&
61
- cached.sessionId === sessionId &&
62
- cached.generation === marker.generation
63
- )
64
- return cached.excluded;
65
- source = {
66
- file,
67
- sessionId,
68
- generation: marker.generation,
69
- excluded: new Set(),
70
- };
71
- continue;
32
+ let header;
33
+ for await (const entry of records(input, 0, size)) {
34
+ header = entry;
35
+ break;
36
+ }
37
+ if (header?.type !== "session" || header.id !== sessionId)
38
+ throw new Error("History source identity changed");
39
+ const marker = header.notesPruning;
40
+ if (marker === undefined) return empty;
41
+ if (
42
+ marker?.version !== 1 ||
43
+ marker.sessionId !== sessionId ||
44
+ typeof marker.generation !== "string"
45
+ )
46
+ throw new Error("Invalid history pruning marker");
47
+ const previous =
48
+ cached?.file === file &&
49
+ cached.sessionId === sessionId &&
50
+ cached.generation === marker.generation &&
51
+ cached.size <= size
52
+ ? cached
53
+ : undefined;
54
+ const incremental = previous !== undefined;
55
+ if (previous?.size === size) return exclusion(previous);
56
+ /** @type {PrunedHistory} */
57
+ const source = previous ?? {
58
+ file,
59
+ sessionId,
60
+ generation: marker.generation,
61
+ size: 0,
62
+ surviving: new Set(),
63
+ excluded: new Set(),
64
+ };
65
+ // A failed suffix scan invalidates mutated cache sets. Ordinary appends
66
+ // scan only their suffix, not the full session or every indexed document.
67
+ cached = undefined;
68
+ for await (const entry of records(input, source.size, size)) {
69
+ if (!incremental && entry.type === "session") continue;
70
+ if (typeof entry.id !== "string" || source.surviving.has(entry.id))
71
+ throw new Error("Invalid history source entry identity");
72
+ source.surviving.add(entry.id);
73
+ const message = entry.type === "message" ? entry.message : undefined;
74
+ if (
75
+ (entry.type === "custom" &&
76
+ entry.customType === "notes-compaction:pruned:v1") ||
77
+ (["user", "assistant"].includes(message?.role) &&
78
+ !visibleText(message.content))
79
+ )
80
+ source.excluded.add(entry.id);
81
+ }
82
+ source.size = size;
83
+ const excluded = exclusion(source);
84
+ if (!incremental) {
85
+ const removeTerms = db.prepare(
86
+ "DELETE FROM history_terms WHERE rowid = ?",
87
+ );
88
+ const removeDocument = db.prepare(
89
+ "DELETE FROM history_documents WHERE id = ?",
90
+ );
91
+ /** @type {import("./sqlite.mjs").Statement<[string], {id:number; item_id:string}>} */
92
+ const documents = db.prepare(
93
+ "SELECT id, item_id FROM history_documents WHERE session_id = ?",
94
+ );
95
+ db.transaction(() => {
96
+ for (const row of documents.all(sessionId)) {
97
+ if (!excluded(row.item_id)) continue;
98
+ removeTerms.run(row.id);
99
+ removeDocument.run(row.id);
72
100
  }
73
- const message = entry.type === "message" ? entry.message : undefined;
74
- if (
75
- (entry.type === "custom" &&
76
- entry.customType === "notes-compaction:pruned:v1") ||
77
- (["user", "assistant"].includes(message?.role) &&
78
- !visibleText(message.content))
79
- )
80
- source.excluded.add(entry.id);
81
- }
82
- pending = pending.slice(start);
101
+ }).immediate();
83
102
  }
84
- // Only complete lines from the opened inode count. A concurrent append's
85
- // incomplete suffix is new history, never an old pruned conversation.
86
- if (!source) throw new Error("History source header is missing");
87
- const excluded = source.excluded;
88
- const removeTerms = db.prepare(
89
- "DELETE FROM history_terms WHERE rowid = ?",
90
- );
91
- const removeDocument = db.prepare(
92
- "DELETE FROM history_documents WHERE id = ?",
93
- );
94
- /** @type {import("./sqlite.mjs").Statement<[string], {id:number; item_id:string}>} */
95
- const documents = db.prepare(
96
- "SELECT id, item_id FROM history_documents WHERE session_id = ?",
97
- );
98
- db.transaction(() => {
99
- for (const row of documents.all(sessionId)) {
100
- if (!excluded.has(row.item_id)) continue;
101
- removeTerms.run(row.id);
102
- removeDocument.run(row.id);
103
- }
104
- }).immediate();
105
103
  cached = source;
106
104
  return excluded;
107
105
  } finally {
108
- stream?.destroy();
109
106
  await input.close();
110
107
  }
111
108
  };
112
109
  }
113
110
 
111
+ /** @param {import("node:fs/promises").FileHandle} input @param {number} start @param {number} size */
112
+ async function* records(input, start, size) {
113
+ const buffer = Buffer.alloc(64 * 1024);
114
+ const decoder = new StringDecoder("utf8");
115
+ let pending = "";
116
+ for (let position = start; position < size; ) {
117
+ const { bytesRead } = await input.read(
118
+ buffer,
119
+ 0,
120
+ Math.min(buffer.length, size - position),
121
+ position,
122
+ );
123
+ if (!bytesRead)
124
+ throw new Error("History source changed during reconciliation");
125
+ position += bytesRead;
126
+ pending += decoder.write(buffer.subarray(0, bytesRead));
127
+ let offset = 0;
128
+ let end;
129
+ while ((end = pending.indexOf("\n", offset)) >= 0) {
130
+ yield JSON.parse(pending.slice(offset, end));
131
+ offset = end + 1;
132
+ }
133
+ pending = pending.slice(offset);
134
+ }
135
+ pending += decoder.end();
136
+ if (pending.length)
137
+ throw new Error(
138
+ "History source has an incomplete record; retry after the turn completes",
139
+ );
140
+ }
141
+
142
+ /** @param {PrunedHistory} source */
143
+ function exclusion(source) {
144
+ /** @param {string} id */
145
+ return (id) => !source.surviving.has(id) || source.excluded.has(id);
146
+ }
147
+
114
148
  /** @param {unknown} content */
115
149
  function visibleText(content) {
116
150
  return typeof content === "string"
@@ -97,7 +97,7 @@ async function index(request) {
97
97
  }
98
98
  }
99
99
  for (const item of request.items) {
100
- if (excluded.has(item.item_id)) continue;
100
+ if (excluded(item.item_id)) continue;
101
101
  const fingerprint = createHash("sha256")
102
102
  .update(item.content)
103
103
  .digest("hex");
@@ -79,7 +79,7 @@ export async function lookupNotes(
79
79
  );
80
80
  else if (lookupFailed)
81
81
  warnings.push(
82
- "Some notes could not be retrieved. Available notes are shown; retry the lookup to check for missing results.",
82
+ "Remote notes could not be checked. Local results are shown; retry to check for remote-only notes.",
83
83
  );
84
84
  if (responses.length && canProject && replacementsInScope)
85
85
  warnings.push(