@howaboua/pi-notes-compaction 0.0.16 → 0.0.17
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +9 -0
- package/README.md +13 -1
- package/package.json +3 -3
- package/src/guidance.ts +14 -14
- package/src/notes/AGENTS.md +1 -1
- package/src/notes/feature.ts +67 -1
- package/src/notes/history-search-pruning.mjs +110 -76
- package/src/notes/history-search-worker.mjs +1 -1
- package/src/notes/notes-lookup.ts +1 -1
- package/src/notes/session-pruning-cleanup.mjs +163 -0
- package/src/notes/session-pruning-exit.mjs +312 -0
- package/src/notes/session-pruning-screen.ts +1 -1
- package/src/notes/session-pruning-sparse.mjs +133 -0
- package/src/notes/session-pruning-state.mjs +123 -0
- package/src/notes/session-pruning.ts +34 -1
- package/src/notes/settings-controls.ts +13 -17
- package/src/notes/settings-screen.ts +2 -6
- package/src/notes/window-message.ts +6 -18
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,14 @@
|
|
|
1
1
|
# @howaboua/pi-notes-compaction
|
|
2
2
|
|
|
3
|
+
## 0.0.17
|
|
4
|
+
|
|
5
|
+
- Notes lookup warnings now distinguish unavailable remote notes from local checkpoints.
|
|
6
|
+
- Notes guidance calls for fewer, shorter updates, using task notes and/or checkpoints with History links instead of copied investigation or harness detail.
|
|
7
|
+
- Notes guidance keeps no-action report updates to their ID and outcome when checkpointing is needed.
|
|
8
|
+
- Retired session records are removed automatically after Pi exits.
|
|
9
|
+
- Notes guidance distinguishes non-waking writes from direct messages, regardless of messaging tool.
|
|
10
|
+
- Notes settings now open on Enabled, with Off, On and Whitelist choices.
|
|
11
|
+
|
|
3
12
|
## 0.0.16
|
|
4
13
|
|
|
5
14
|
- Requires Node.js 22.19 or newer.
|
package/README.md
CHANGED
|
@@ -84,11 +84,23 @@ unsupported reasoning levels. Provider capabilities and context limits still
|
|
|
84
84
|
apply, and switching does not guarantee cache reuse. Context retirement and
|
|
85
85
|
recovery keep their existing Notes settings.
|
|
86
86
|
|
|
87
|
+
### Session pruning
|
|
88
|
+
|
|
89
|
+
Open **Session auto-pruning** in `/notes` to set this session's retention limits. **Run now** previews and confirms irreversible removal of retired payloads. Automatic pruning runs after a new window's first completed turn. Saved SQLite notes and current context remain intact. Settings follow the session on resume. New and forked sessions start with pruning off.
|
|
90
|
+
|
|
91
|
+
Live pruning keeps every history entry ID so Pi can still navigate its in-memory tree safely. With retention enabled, Notes automatically removes retired empty records after Pi exits. A separate helper waits for the actual process to end before changing the saved tree. Reloading, forking and switching sessions do not trigger this cleanup. No command or additional confirmation is needed. No backup is kept. Memory savings apply when the pruned session next opens.
|
|
92
|
+
|
|
93
|
+
Physical cleanup currently requires Linux and an available `node` executable. Other platforms retain live payload pruning. Cleanup skips another registered local Notes process using the same saved session, known open-file holders and files that change during preparation. A simultaneous reopen is checked against the saved generation. If cleanup won that race, the new Pi process stops and asks you to reopen it rather than continue with a stale tree. Failures are reported when the session next opens. Tree branches, labels, compaction boundaries and supported Notes archive references are retained. Other content and saved notes are unchanged.
|
|
94
|
+
|
|
95
|
+
Coordination covers local processes running this Notes release under the same OS user. Keep session storage free of concurrent writers from other machines, other OS users, older Notes versions and external JSONL editors while a Notes-enabled process is closing. Unrelated sessions do not block cleanup.
|
|
96
|
+
|
|
97
|
+
Update Observatory before applying if you use its historical index. Older versions may block the rewritten file. Current versions reconcile removed records without discarding indexed statistics.
|
|
98
|
+
|
|
87
99
|
### Notes settings
|
|
88
100
|
|
|
89
101
|
`/notes` opens the settings panel in interactive Pi. It shows continuity status, storage counts and remote capabilities. Changes save immediately.
|
|
90
102
|
|
|
91
|
-
- **
|
|
103
|
+
- **Enabled** selects **Off**, **On** (all models, the default) or **Whitelist**. Whitelist enables the extension only for exact model IDs in the global settings file, regardless of provider. Switching models applies the choice immediately; other models keep their installed compaction behaviour. Off disables Notes without deleting saved data.
|
|
92
104
|
- **Normal compaction** uses the installed Pi or PCC compaction flow for `/compact` and `new_context`. Off keeps notes-only rollover, the default.
|
|
93
105
|
- **New window requests** controls the agent's `new_context` tool independently of the compaction method. **off** disables the tool. **pi-limits** uses Pi's eligibility check and continues in the current window when there is too little history. **unrestricted**, the default, accepts requests at any size. With normal compaction on and nothing to summarise, unrestricted requests open a new window identity without discarding the existing context. Manual `/compact` and idle rollover retain their separate policies.
|
|
94
106
|
- **Idle rollover** selects the interval or disables it.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@howaboua/pi-notes-compaction",
|
|
3
|
-
"version": "0.0.
|
|
3
|
+
"version": "0.0.17",
|
|
4
4
|
"publishConfig": {
|
|
5
5
|
"access": "public"
|
|
6
6
|
},
|
|
@@ -41,7 +41,7 @@
|
|
|
41
41
|
"better-sqlite3": "13.0.3"
|
|
42
42
|
},
|
|
43
43
|
"devDependencies": {
|
|
44
|
-
"@howaboua/pi-codex-conversion": "
|
|
44
|
+
"@howaboua/pi-codex-conversion": "5.0.0",
|
|
45
45
|
"@types/better-sqlite3": "9.6.0"
|
|
46
46
|
},
|
|
47
47
|
"peerDependencies": {
|
|
@@ -49,7 +49,7 @@
|
|
|
49
49
|
"@earendil-works/pi-agent-core": ">=1.0.4",
|
|
50
50
|
"@earendil-works/pi-ai": ">=1.0.4",
|
|
51
51
|
"@earendil-works/pi-tui": ">=1.0.4",
|
|
52
|
-
"@howaboua/pi-codex-conversion": ">=
|
|
52
|
+
"@howaboua/pi-codex-conversion": ">=5.0.0"
|
|
53
53
|
},
|
|
54
54
|
"peerDependenciesMeta": {
|
|
55
55
|
"@howaboua/pi-codex-conversion": {
|
package/src/guidance.ts
CHANGED
|
@@ -18,38 +18,40 @@ export interface CommsFeature extends FeatureSettings {
|
|
|
18
18
|
function communicationGuidance(
|
|
19
19
|
notes: boolean,
|
|
20
20
|
board: boolean,
|
|
21
|
-
agents: boolean,
|
|
22
21
|
gpt = false,
|
|
23
22
|
): string {
|
|
24
23
|
return [
|
|
25
24
|
board
|
|
26
25
|
? gpt
|
|
27
|
-
? "Board→peer findings/
|
|
28
|
-
: "Board:
|
|
26
|
+
? "Board→new peer-work findings/decisions/ownership/blockers only; no change→no post. Peers read directly; no parent relay. No idle wake; active subscribers may be interrupted."
|
|
27
|
+
: "Board: post only findings, decisions, ownership changes or blockers that change peers' work. Peers read posts directly; do not relay them through the parent. Posts do not wake idle agents but may interrupt active subscribers."
|
|
29
28
|
: "",
|
|
30
29
|
notes
|
|
31
30
|
? [
|
|
31
|
+
gpt
|
|
32
|
+
? "Notes writes don't wake peers."
|
|
33
|
+
: "Notes writes do not wake peers.",
|
|
32
34
|
board
|
|
33
35
|
? gpt
|
|
34
|
-
? "Notes→
|
|
35
|
-
: "Notes
|
|
36
|
+
? "Notes→recovery; link Board evidence."
|
|
37
|
+
: "Notes holds recovery checkpoints; link Board evidence."
|
|
36
38
|
: "",
|
|
37
39
|
gpt
|
|
38
|
-
? "
|
|
39
|
-
: "
|
|
40
|
+
? "Known recipient Notes access→brief outcome/blocker + checkpoint path; recipient reads checkpoint instead of requesting its contents."
|
|
41
|
+
: "With known recipient Notes access, return a brief outcome or blocker plus the checkpoint path; the recipient reads the checkpoint instead of requesting its contents.",
|
|
40
42
|
]
|
|
41
43
|
.filter(Boolean)
|
|
42
44
|
.join(" ")
|
|
43
45
|
: "",
|
|
44
46
|
notes || board
|
|
45
47
|
? gpt
|
|
46
|
-
? `${notes && board ? "Notes/Board" : notes ? "Notes" : "Board"} available here≠recipient access.
|
|
47
|
-
: `${notes && board ? "Notes and Board" : notes ? "Notes" : "Board"} availability here does not establish recipient access.
|
|
48
|
+
? `${notes && board ? "Notes/Board" : notes ? "Notes" : "Board"} available here≠recipient access. Known access→link≠repeat; otherwise self-contained actions/findings/unresolved state.`
|
|
49
|
+
: `${notes && board ? "Notes and Board" : notes ? "Notes" : "Board"} availability here does not establish recipient access. Link rather than repeat when access is known; otherwise give a self-contained account of actions, findings and unresolved state.`
|
|
48
50
|
: "",
|
|
49
|
-
|
|
51
|
+
notes || board
|
|
50
52
|
? gpt
|
|
51
|
-
? "
|
|
52
|
-
: "Use
|
|
53
|
+
? "Direct messages→assignments or urgent action, not status or courtesy updates."
|
|
54
|
+
: "Use direct messages for assignments or urgent action, not status or courtesy updates."
|
|
53
55
|
: "",
|
|
54
56
|
]
|
|
55
57
|
.filter(Boolean)
|
|
@@ -73,11 +75,9 @@ export function registerCommunicationGuidance(
|
|
|
73
75
|
(ctx) => Boolean(notes?.enabled(ctx) || board?.enabled(ctx)),
|
|
74
76
|
async (ctx, windowId, messages) => {
|
|
75
77
|
// Feature owners supply enablement independently of provider declarations.
|
|
76
|
-
// Shepherdr keeps agents active, including as a nested Code/Notebook tool.
|
|
77
78
|
const capabilities = [
|
|
78
79
|
notes?.enabled(ctx) ?? false,
|
|
79
80
|
board?.enabled(ctx) ?? false,
|
|
80
|
-
pi.getActiveTools().includes("agents"),
|
|
81
81
|
Boolean(notes?.gptPromptMode() || board?.gptPromptMode()),
|
|
82
82
|
] as const;
|
|
83
83
|
const routing = communicationGuidance(...capabilities);
|
package/src/notes/AGENTS.md
CHANGED
|
@@ -21,7 +21,7 @@
|
|
|
21
21
|
- Return initial and rebound identity from before_agent_start, never send it during startup. Pi inserts its initial system entry ahead of returned hook messages; startup sends otherwise put per-session identity before the shared cache prefix.
|
|
22
22
|
- Automatic rollover uses the successor window message as its custom kickoff. That entry is not yet in the branch during before_agent_start; adopt its identity instead of restoring the outgoing window or returning another boundary.
|
|
23
23
|
- Compaction handoff reads are transient branch contributions anchored to their persisted summary. Rechain only the projected branch, never append contributions or persist their IDs as Pi's first-kept boundary. Compare projected entry identity, not branch length.
|
|
24
|
-
-
|
|
24
|
+
- Live session pruning preserves JSONL order/count/IDs. Sparse cleanup runs only after the owning Pi process exits, never inside shutdown or on reload/fork. Retain roots, leaves, branchpoints, physical last and supported structured reference targets; skip live registered owners. FD checks do not prove an idle manager has closed. Observatory rebases derived ranges using new header `notesPruning` generations, never tail receipts alone. Final validation/rename must contain no await. Never mutate Pi's live history or coordinate foreign JSONL writers.
|
|
25
25
|
- Pi persists incoming messages after before_agent_start, agent_start and message_end hooks, before returned startup messages. History assigns the kickoff prefix only to the first untrimmed window, never later or identity-only boundaries. Label saved-note continuation prompts in context once their entries exist. Schedule pruning from Notes' admitted window identity, then verify its persisted marker after settlement.
|
|
26
26
|
- Pruning must not turn removed delivered instruction carriers into pending-message redelivery. Coalesce only proven complete state snapshots per branch leaf; unknown custom protocols can contain deltas. Keep current Notes revisions, monitor snapshots, Board routes and notebook epochs, not historical model-facing text disguised as state.
|
|
27
27
|
- Role handoffs prepare the outgoing provider before selecting the successor model and starting the normal window kickoff. Keep checkpoint and compaction gates, rechecking session/window after async work. Role guidance exposes IDs/descriptions only. In-run compaction restores its persisted carrier before the summary without a new turn or prefix edit.
|
package/src/notes/feature.ts
CHANGED
|
@@ -7,6 +7,11 @@ import type {
|
|
|
7
7
|
import type { CommsFeature } from "../guidance.js";
|
|
8
8
|
import type { OpenSettings } from "../settings-navigation.js";
|
|
9
9
|
import type { NotesRuntime } from "./index.js";
|
|
10
|
+
import {
|
|
11
|
+
admitSessionCleanup,
|
|
12
|
+
previousCleanupFailure,
|
|
13
|
+
} from "./session-pruning-exit.mjs";
|
|
14
|
+
import { cleanupStatus } from "./session-pruning-state.mjs";
|
|
10
15
|
import {
|
|
11
16
|
legacyNotesConfigured,
|
|
12
17
|
loadNotesSettings,
|
|
@@ -53,6 +58,43 @@ export function createNotesFeature(
|
|
|
53
58
|
let store: NotesStore | undefined;
|
|
54
59
|
let storage: Promise<NotesStore> | undefined;
|
|
55
60
|
let closed = false;
|
|
61
|
+
let staleOpening = false;
|
|
62
|
+
let releaseCleanupOwner: (() => void) | undefined;
|
|
63
|
+
const admit = async (ctx: ExtensionContext) => {
|
|
64
|
+
if (releaseCleanupOwner) return !staleOpening;
|
|
65
|
+
let reason: string | undefined;
|
|
66
|
+
try {
|
|
67
|
+
const claim = await admitSessionCleanup(
|
|
68
|
+
ctx.sessionManager.getSessionFile(),
|
|
69
|
+
ctx.sessionManager.getSessionId(),
|
|
70
|
+
ctx.sessionManager.getHeader(),
|
|
71
|
+
);
|
|
72
|
+
releaseCleanupOwner = claim?.release;
|
|
73
|
+
if (claim?.stale)
|
|
74
|
+
reason =
|
|
75
|
+
"This session was cleaned while Pi was opening it. Restart Pi to load its saved tree safely.";
|
|
76
|
+
} catch (error) {
|
|
77
|
+
reason = `Pi could not safely open this session: ${String(error)}. Resolve the problem and restart Pi.`;
|
|
78
|
+
}
|
|
79
|
+
if (!reason) return true;
|
|
80
|
+
staleOpening = true;
|
|
81
|
+
console.error(reason);
|
|
82
|
+
ctx.ui.notify(reason, "error");
|
|
83
|
+
if (ctx.mode === "rpc") {
|
|
84
|
+
// RPC honours startup shutdown after attaching input. Returning lets it
|
|
85
|
+
// reach that check before accepting normal commands.
|
|
86
|
+
ctx.shutdown();
|
|
87
|
+
return false;
|
|
88
|
+
}
|
|
89
|
+
if (ctx.mode === "tui") ctx.shutdown();
|
|
90
|
+
else {
|
|
91
|
+
// Released print/json mode installs graceful SIGTERM handling before
|
|
92
|
+
// session_start, but does not bind the documented ctx.shutdown action.
|
|
93
|
+
process.kill(process.pid, "SIGTERM");
|
|
94
|
+
}
|
|
95
|
+
// Do not finish binding or start an initial prompt/tree while exiting.
|
|
96
|
+
await new Promise<never>(() => {});
|
|
97
|
+
};
|
|
56
98
|
const ensureStore = () =>
|
|
57
99
|
(storage ??= (async () => {
|
|
58
100
|
const { NotesStore } = await import("./store.js");
|
|
@@ -110,6 +152,22 @@ export function createNotesFeature(
|
|
|
110
152
|
await open(ctx);
|
|
111
153
|
});
|
|
112
154
|
pi.on("session_start", async (_event, ctx) => {
|
|
155
|
+
if (!(await admit(ctx))) return;
|
|
156
|
+
const file = ctx.sessionManager.getSessionFile();
|
|
157
|
+
if (file) {
|
|
158
|
+
try {
|
|
159
|
+
const error = previousCleanupFailure(
|
|
160
|
+
cleanupStatus(file, dirname(notesSettingsPath())),
|
|
161
|
+
ctx.sessionManager.getSessionId(),
|
|
162
|
+
);
|
|
163
|
+
if (error) ctx.ui.notify(error, "warning");
|
|
164
|
+
} catch (error) {
|
|
165
|
+
ctx.ui.notify(
|
|
166
|
+
`Previous session cleanup status could not be read: ${String(error)}. Check Notes storage permissions and retry.`,
|
|
167
|
+
"warning",
|
|
168
|
+
);
|
|
169
|
+
}
|
|
170
|
+
}
|
|
113
171
|
try {
|
|
114
172
|
settings = loadNotesSettings(ctx);
|
|
115
173
|
writeSettings(notesSettingsPath(), settings);
|
|
@@ -126,12 +184,20 @@ export function createNotesFeature(
|
|
|
126
184
|
);
|
|
127
185
|
}
|
|
128
186
|
});
|
|
187
|
+
pi.on("agent_settled", async (_event, ctx) => {
|
|
188
|
+
// A brand-new session has no JSONL file until its first persisted turn.
|
|
189
|
+
await admit(ctx);
|
|
190
|
+
});
|
|
129
191
|
pi.on("model_select", async (_event, ctx) => {
|
|
130
192
|
if (!runtime && settings && modelIsActive(settings, ctx.model))
|
|
131
193
|
await ensure(ctx);
|
|
132
194
|
});
|
|
133
|
-
pi.on("session_shutdown", () => {
|
|
195
|
+
pi.on("session_shutdown", (event) => {
|
|
134
196
|
closed = true;
|
|
197
|
+
if (event.reason !== "quit") {
|
|
198
|
+
releaseCleanupOwner?.();
|
|
199
|
+
releaseCleanupOwner = undefined;
|
|
200
|
+
}
|
|
135
201
|
store?.close();
|
|
136
202
|
});
|
|
137
203
|
pi.registerCommand("notes", {
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
import { open } from "node:fs/promises";
|
|
2
|
+
import { StringDecoder } from "node:string_decoder";
|
|
2
3
|
|
|
3
|
-
/** @typedef {{file:string; sessionId:string; generation:string; excluded:Set<string>}} PrunedHistory */
|
|
4
|
+
/** @typedef {{file:string; sessionId:string; generation:string; size:number; surviving:Set<string>; excluded:Set<string>}} PrunedHistory */
|
|
4
5
|
|
|
5
6
|
/** The live SessionManager still contains old text until restart. Read the committed
|
|
6
7
|
* source so later indexing, settings toggles and /reload cannot resurrect its terms.
|
|
@@ -8,11 +9,11 @@ import { open } from "node:fs/promises";
|
|
|
8
9
|
export function createHistoryPruning() {
|
|
9
10
|
/** @type {PrunedHistory | undefined} */
|
|
10
11
|
let cached;
|
|
11
|
-
const empty =
|
|
12
|
+
const empty = () => false;
|
|
12
13
|
/** @param {import("./sqlite.mjs").Database} db
|
|
13
14
|
* @param {string} sessionId
|
|
14
15
|
* @param {string | undefined} file
|
|
15
|
-
* @returns {Promise<
|
|
16
|
+
* @returns {Promise<(id:string) => boolean>}
|
|
16
17
|
*/
|
|
17
18
|
return async (db, sessionId, file) => {
|
|
18
19
|
if (!file) return empty;
|
|
@@ -25,92 +26,125 @@ export function createHistoryPruning() {
|
|
|
25
26
|
return empty;
|
|
26
27
|
throw error;
|
|
27
28
|
}
|
|
28
|
-
let stream;
|
|
29
29
|
try {
|
|
30
30
|
const { size } = await input.stat();
|
|
31
31
|
if (!size) return empty;
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
32
|
+
let header;
|
|
33
|
+
for await (const entry of records(input, 0, size)) {
|
|
34
|
+
header = entry;
|
|
35
|
+
break;
|
|
36
|
+
}
|
|
37
|
+
if (header?.type !== "session" || header.id !== sessionId)
|
|
38
|
+
throw new Error("History source identity changed");
|
|
39
|
+
const marker = header.notesPruning;
|
|
40
|
+
if (marker === undefined) return empty;
|
|
41
|
+
if (
|
|
42
|
+
marker?.version !== 1 ||
|
|
43
|
+
marker.sessionId !== sessionId ||
|
|
44
|
+
typeof marker.generation !== "string"
|
|
45
|
+
)
|
|
46
|
+
throw new Error("Invalid history pruning marker");
|
|
47
|
+
const previous =
|
|
48
|
+
cached?.file === file &&
|
|
49
|
+
cached.sessionId === sessionId &&
|
|
50
|
+
cached.generation === marker.generation &&
|
|
51
|
+
cached.size <= size
|
|
52
|
+
? cached
|
|
53
|
+
: undefined;
|
|
54
|
+
const incremental = previous !== undefined;
|
|
55
|
+
if (previous?.size === size) return exclusion(previous);
|
|
56
|
+
/** @type {PrunedHistory} */
|
|
57
|
+
const source = previous ?? {
|
|
58
|
+
file,
|
|
59
|
+
sessionId,
|
|
60
|
+
generation: marker.generation,
|
|
61
|
+
size: 0,
|
|
62
|
+
surviving: new Set(),
|
|
63
|
+
excluded: new Set(),
|
|
64
|
+
};
|
|
65
|
+
// A failed suffix scan invalidates mutated cache sets. Ordinary appends
|
|
66
|
+
// scan only their suffix, not the full session or every indexed document.
|
|
67
|
+
cached = undefined;
|
|
68
|
+
for await (const entry of records(input, source.size, size)) {
|
|
69
|
+
if (!incremental && entry.type === "session") continue;
|
|
70
|
+
if (typeof entry.id !== "string" || source.surviving.has(entry.id))
|
|
71
|
+
throw new Error("Invalid history source entry identity");
|
|
72
|
+
source.surviving.add(entry.id);
|
|
73
|
+
const message = entry.type === "message" ? entry.message : undefined;
|
|
74
|
+
if (
|
|
75
|
+
(entry.type === "custom" &&
|
|
76
|
+
entry.customType === "notes-compaction:pruned:v1") ||
|
|
77
|
+
(["user", "assistant"].includes(message?.role) &&
|
|
78
|
+
!visibleText(message.content))
|
|
79
|
+
)
|
|
80
|
+
source.excluded.add(entry.id);
|
|
81
|
+
}
|
|
82
|
+
source.size = size;
|
|
83
|
+
const excluded = exclusion(source);
|
|
84
|
+
if (!incremental) {
|
|
85
|
+
const removeTerms = db.prepare(
|
|
86
|
+
"DELETE FROM history_terms WHERE rowid = ?",
|
|
87
|
+
);
|
|
88
|
+
const removeDocument = db.prepare(
|
|
89
|
+
"DELETE FROM history_documents WHERE id = ?",
|
|
90
|
+
);
|
|
91
|
+
/** @type {import("./sqlite.mjs").Statement<[string], {id:number; item_id:string}>} */
|
|
92
|
+
const documents = db.prepare(
|
|
93
|
+
"SELECT id, item_id FROM history_documents WHERE session_id = ?",
|
|
94
|
+
);
|
|
95
|
+
db.transaction(() => {
|
|
96
|
+
for (const row of documents.all(sessionId)) {
|
|
97
|
+
if (!excluded(row.item_id)) continue;
|
|
98
|
+
removeTerms.run(row.id);
|
|
99
|
+
removeDocument.run(row.id);
|
|
72
100
|
}
|
|
73
|
-
|
|
74
|
-
if (
|
|
75
|
-
(entry.type === "custom" &&
|
|
76
|
-
entry.customType === "notes-compaction:pruned:v1") ||
|
|
77
|
-
(["user", "assistant"].includes(message?.role) &&
|
|
78
|
-
!visibleText(message.content))
|
|
79
|
-
)
|
|
80
|
-
source.excluded.add(entry.id);
|
|
81
|
-
}
|
|
82
|
-
pending = pending.slice(start);
|
|
101
|
+
}).immediate();
|
|
83
102
|
}
|
|
84
|
-
// Only complete lines from the opened inode count. A concurrent append's
|
|
85
|
-
// incomplete suffix is new history, never an old pruned conversation.
|
|
86
|
-
if (!source) throw new Error("History source header is missing");
|
|
87
|
-
const excluded = source.excluded;
|
|
88
|
-
const removeTerms = db.prepare(
|
|
89
|
-
"DELETE FROM history_terms WHERE rowid = ?",
|
|
90
|
-
);
|
|
91
|
-
const removeDocument = db.prepare(
|
|
92
|
-
"DELETE FROM history_documents WHERE id = ?",
|
|
93
|
-
);
|
|
94
|
-
/** @type {import("./sqlite.mjs").Statement<[string], {id:number; item_id:string}>} */
|
|
95
|
-
const documents = db.prepare(
|
|
96
|
-
"SELECT id, item_id FROM history_documents WHERE session_id = ?",
|
|
97
|
-
);
|
|
98
|
-
db.transaction(() => {
|
|
99
|
-
for (const row of documents.all(sessionId)) {
|
|
100
|
-
if (!excluded.has(row.item_id)) continue;
|
|
101
|
-
removeTerms.run(row.id);
|
|
102
|
-
removeDocument.run(row.id);
|
|
103
|
-
}
|
|
104
|
-
}).immediate();
|
|
105
103
|
cached = source;
|
|
106
104
|
return excluded;
|
|
107
105
|
} finally {
|
|
108
|
-
stream?.destroy();
|
|
109
106
|
await input.close();
|
|
110
107
|
}
|
|
111
108
|
};
|
|
112
109
|
}
|
|
113
110
|
|
|
111
|
+
/** @param {import("node:fs/promises").FileHandle} input @param {number} start @param {number} size */
|
|
112
|
+
async function* records(input, start, size) {
|
|
113
|
+
const buffer = Buffer.alloc(64 * 1024);
|
|
114
|
+
const decoder = new StringDecoder("utf8");
|
|
115
|
+
let pending = "";
|
|
116
|
+
for (let position = start; position < size; ) {
|
|
117
|
+
const { bytesRead } = await input.read(
|
|
118
|
+
buffer,
|
|
119
|
+
0,
|
|
120
|
+
Math.min(buffer.length, size - position),
|
|
121
|
+
position,
|
|
122
|
+
);
|
|
123
|
+
if (!bytesRead)
|
|
124
|
+
throw new Error("History source changed during reconciliation");
|
|
125
|
+
position += bytesRead;
|
|
126
|
+
pending += decoder.write(buffer.subarray(0, bytesRead));
|
|
127
|
+
let offset = 0;
|
|
128
|
+
let end;
|
|
129
|
+
while ((end = pending.indexOf("\n", offset)) >= 0) {
|
|
130
|
+
yield JSON.parse(pending.slice(offset, end));
|
|
131
|
+
offset = end + 1;
|
|
132
|
+
}
|
|
133
|
+
pending = pending.slice(offset);
|
|
134
|
+
}
|
|
135
|
+
pending += decoder.end();
|
|
136
|
+
if (pending.length)
|
|
137
|
+
throw new Error(
|
|
138
|
+
"History source has an incomplete record; retry after the turn completes",
|
|
139
|
+
);
|
|
140
|
+
}
|
|
141
|
+
|
|
142
|
+
/** @param {PrunedHistory} source */
|
|
143
|
+
function exclusion(source) {
|
|
144
|
+
/** @param {string} id */
|
|
145
|
+
return (id) => !source.surviving.has(id) || source.excluded.has(id);
|
|
146
|
+
}
|
|
147
|
+
|
|
114
148
|
/** @param {unknown} content */
|
|
115
149
|
function visibleText(content) {
|
|
116
150
|
return typeof content === "string"
|
|
@@ -97,7 +97,7 @@ async function index(request) {
|
|
|
97
97
|
}
|
|
98
98
|
}
|
|
99
99
|
for (const item of request.items) {
|
|
100
|
-
if (excluded
|
|
100
|
+
if (excluded(item.item_id)) continue;
|
|
101
101
|
const fingerprint = createHash("sha256")
|
|
102
102
|
.update(item.content)
|
|
103
103
|
.digest("hex");
|
|
@@ -79,7 +79,7 @@ export async function lookupNotes(
|
|
|
79
79
|
);
|
|
80
80
|
else if (lookupFailed)
|
|
81
81
|
warnings.push(
|
|
82
|
-
"
|
|
82
|
+
"Remote notes could not be checked. Local results are shown; retry to check for remote-only notes.",
|
|
83
83
|
);
|
|
84
84
|
if (responses.length && canProject && replacementsInScope)
|
|
85
85
|
warnings.push(
|
|
@@ -0,0 +1,163 @@
|
|
|
1
|
+
import { randomUUID } from "node:crypto";
|
|
2
|
+
import {
|
|
3
|
+
closeSync,
|
|
4
|
+
fsyncSync,
|
|
5
|
+
lstatSync,
|
|
6
|
+
openSync,
|
|
7
|
+
readdirSync,
|
|
8
|
+
renameSync,
|
|
9
|
+
statSync,
|
|
10
|
+
unlinkSync,
|
|
11
|
+
} from "node:fs";
|
|
12
|
+
import { open } from "node:fs/promises";
|
|
13
|
+
import { resolve } from "node:path";
|
|
14
|
+
import {
|
|
15
|
+
planSparsePruning,
|
|
16
|
+
scanSparseSource,
|
|
17
|
+
} from "./session-pruning-sparse.mjs";
|
|
18
|
+
import { cleanupPaths, tryCleanupLock } from "./session-pruning-state.mjs";
|
|
19
|
+
|
|
20
|
+
/** @param {string} path @param {import("node:fs").Stats} source */
|
|
21
|
+
function checkSource(path, source) {
|
|
22
|
+
const current = lstatSync(path);
|
|
23
|
+
if (
|
|
24
|
+
!current.isFile() ||
|
|
25
|
+
current.nlink !== 1 ||
|
|
26
|
+
current.dev !== source.dev ||
|
|
27
|
+
current.ino !== source.ino ||
|
|
28
|
+
current.size !== source.size ||
|
|
29
|
+
current.mtimeMs !== source.mtimeMs ||
|
|
30
|
+
current.ctimeMs !== source.ctimeMs
|
|
31
|
+
)
|
|
32
|
+
throw new Error(
|
|
33
|
+
"Session changed during cleanup. Nothing was replaced. Cleanup will retry on the next normal Pi exit.",
|
|
34
|
+
);
|
|
35
|
+
}
|
|
36
|
+
|
|
37
|
+
/** Open descriptors supplement the registered session owners, not replace them.
|
|
38
|
+
* @param {import("node:fs").Stats} source
|
|
39
|
+
*/
|
|
40
|
+
function checkHolders(source) {
|
|
41
|
+
if (process.platform !== "linux")
|
|
42
|
+
throw new Error(
|
|
43
|
+
"Automatic session cleanup is unavailable on this platform. Retired payloads remain pruned.",
|
|
44
|
+
);
|
|
45
|
+
const holders = new Set();
|
|
46
|
+
for (const pid of readdirSync("/proc")) {
|
|
47
|
+
if (!/^\d+$/.test(pid) || Number(pid) === process.pid) continue;
|
|
48
|
+
const dir = `/proc/${pid}`;
|
|
49
|
+
try {
|
|
50
|
+
if (statSync(dir).uid !== process.getuid?.()) continue;
|
|
51
|
+
for (const fd of readdirSync(`${dir}/fd`)) {
|
|
52
|
+
try {
|
|
53
|
+
const file = statSync(`${dir}/fd/${fd}`);
|
|
54
|
+
if (file.dev === source.dev && file.ino === source.ino)
|
|
55
|
+
holders.add(pid);
|
|
56
|
+
} catch (error) {
|
|
57
|
+
if (!vanished(error)) throw error;
|
|
58
|
+
}
|
|
59
|
+
}
|
|
60
|
+
} catch (error) {
|
|
61
|
+
if (
|
|
62
|
+
error instanceof Error &&
|
|
63
|
+
"code" in error &&
|
|
64
|
+
(error.code === "EACCES" || error.code === "EPERM")
|
|
65
|
+
)
|
|
66
|
+
continue;
|
|
67
|
+
else if (!vanished(error))
|
|
68
|
+
throw new Error(
|
|
69
|
+
`Cannot inspect open holders for PID ${pid}: ${String(error)}`,
|
|
70
|
+
);
|
|
71
|
+
}
|
|
72
|
+
}
|
|
73
|
+
if (holders.size)
|
|
74
|
+
throw new Error(
|
|
75
|
+
`Session record cleanup was skipped because process(es) ${[...holders].join(", ")} still hold it open. It will retry on the next normal Pi exit.`,
|
|
76
|
+
);
|
|
77
|
+
}
|
|
78
|
+
|
|
79
|
+
/** @param {unknown} error */
|
|
80
|
+
function vanished(error) {
|
|
81
|
+
return (
|
|
82
|
+
error instanceof Error &&
|
|
83
|
+
"code" in error &&
|
|
84
|
+
(error.code === "ENOENT" || error.code === "ESRCH")
|
|
85
|
+
);
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
/** Called only by the after-exit worker.
|
|
89
|
+
* @param {string} file @param {string} sessionId @param {()=>void} checkOwners
|
|
90
|
+
*/
|
|
91
|
+
export async function cleanSparseSession(file, sessionId, checkOwners) {
|
|
92
|
+
const path = resolve(file);
|
|
93
|
+
const source = lstatSync(path);
|
|
94
|
+
if (!source.isFile() || source.nlink !== 1)
|
|
95
|
+
throw new Error(
|
|
96
|
+
"Session record cleanup requires a regular session file without symbolic or hard links. Retired payloads remain pruned.",
|
|
97
|
+
);
|
|
98
|
+
checkOwners();
|
|
99
|
+
checkHolders(source);
|
|
100
|
+
const lock = cleanupPaths(path).lock;
|
|
101
|
+
let releaseLock;
|
|
102
|
+
const temp = `${path}.notes-prune-${randomUUID()}.tmp`;
|
|
103
|
+
/** @type {Awaited<ReturnType<typeof open>>|undefined} */
|
|
104
|
+
let output;
|
|
105
|
+
try {
|
|
106
|
+
releaseLock = tryCleanupLock(path);
|
|
107
|
+
if (!releaseLock)
|
|
108
|
+
throw new Error(
|
|
109
|
+
`Another session cleanup may be running. Only after verifying that none is active, remove the stale lock ${JSON.stringify(lock)}. Cleanup will retry on the next normal Pi exit. Do not remove an active cleanup's lock.`,
|
|
110
|
+
);
|
|
111
|
+
checkOwners();
|
|
112
|
+
const plan = await planSparsePruning(path, source.size);
|
|
113
|
+
if (plan.header.id !== sessionId)
|
|
114
|
+
throw new Error("Session identity changed. Cleanup was skipped.");
|
|
115
|
+
checkSource(path, source);
|
|
116
|
+
if (!plan.removed.size)
|
|
117
|
+
return { removed: 0, before: source.size, after: source.size };
|
|
118
|
+
output = await open(temp, "wx", source.mode & 0o777);
|
|
119
|
+
const target = output;
|
|
120
|
+
const generation = randomUUID();
|
|
121
|
+
await scanSparseSource(path, source.size, async (entry, raw) => {
|
|
122
|
+
if (plan.removed.has(entry.id)) return;
|
|
123
|
+
const parentId = plan.parents.get(entry.id) ?? null;
|
|
124
|
+
const replacement =
|
|
125
|
+
entry.type === "session"
|
|
126
|
+
? {
|
|
127
|
+
...entry,
|
|
128
|
+
notesPruning: { version: 1, sessionId: entry.id, generation },
|
|
129
|
+
}
|
|
130
|
+
: entry.parentId === parentId
|
|
131
|
+
? entry
|
|
132
|
+
: { ...entry, parentId };
|
|
133
|
+
await target.writeFile(
|
|
134
|
+
`${replacement === entry ? raw : JSON.stringify(replacement)}\n`,
|
|
135
|
+
);
|
|
136
|
+
});
|
|
137
|
+
await output.sync();
|
|
138
|
+
await output.close();
|
|
139
|
+
output = undefined;
|
|
140
|
+
// No await from final holder/source validation through atomic replacement.
|
|
141
|
+
checkOwners();
|
|
142
|
+
checkHolders(source);
|
|
143
|
+
checkSource(path, source);
|
|
144
|
+
const after = statSync(temp).size;
|
|
145
|
+
renameSync(temp, path);
|
|
146
|
+
const directory = openSync(resolve(path, ".."), "r");
|
|
147
|
+
try {
|
|
148
|
+
fsyncSync(directory);
|
|
149
|
+
} finally {
|
|
150
|
+
closeSync(directory);
|
|
151
|
+
}
|
|
152
|
+
return { removed: plan.removed.size, before: source.size, after };
|
|
153
|
+
} finally {
|
|
154
|
+
await output?.close();
|
|
155
|
+
try {
|
|
156
|
+
unlinkSync(temp);
|
|
157
|
+
} catch (error) {
|
|
158
|
+
if (!vanished(error)) throw error;
|
|
159
|
+
} finally {
|
|
160
|
+
releaseLock?.();
|
|
161
|
+
}
|
|
162
|
+
}
|
|
163
|
+
}
|