pi-braid 0.43.6 → 0.43.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +83 -0
- package/TOOLS.md +1 -1
- package/build-info.json +4 -4
- package/dist/extensions/core/prompt-builder/tool-guidelines.js +1 -1
- package/dist/extensions/core/skills-tool/applicability.d.ts +134 -0
- package/dist/extensions/core/skills-tool/applicability.js +161 -0
- package/dist/extensions/core/skills-tool/auto-inject.d.ts +31 -0
- package/dist/extensions/core/skills-tool/auto-inject.js +320 -1
- package/dist/extensions/core/skills-tool/index.js +83 -4
- package/dist/extensions/core/skills-tool/loading-guidance.d.ts +4 -1
- package/dist/extensions/core/skills-tool/loading-guidance.js +6 -2
- package/dist/extensions/core/skills-tool/session-records.d.ts +395 -8
- package/dist/extensions/core/skills-tool/session-records.js +645 -5
- package/dist/extensions/core/skills-tool/skill-families.d.ts +60 -0
- package/dist/extensions/core/skills-tool/skill-families.js +105 -0
- package/dist/extensions/enforcement/handlers/analysis/dreamer-handler.js +5 -1
- package/dist/extensions/enforcement/handlers/register-tool-call.js +5 -2
- package/dist/extensions/enforcement/verification-judge.d.ts +7 -0
- package/dist/extensions/enforcement/verification-judge.js +28 -21
- package/dist/extensions/memory/session-recall/ask.d.ts +95 -0
- package/dist/extensions/memory/session-recall/ask.js +376 -0
- package/dist/extensions/memory/session-recall/corpus-index.d.ts +200 -0
- package/dist/extensions/memory/session-recall/corpus-index.js +480 -0
- package/dist/extensions/memory/session-recall/index.js +173 -21
- package/dist/extensions/memory/session-recall/search.d.ts +29 -0
- package/dist/extensions/memory/session-recall/search.js +42 -9
- package/dist/lib/ai/payload-hygiene.d.ts +3 -0
- package/dist/lib/ai/payload-hygiene.js +34 -2
- package/dist/lib/config/braid-paths.d.ts +4 -0
- package/dist/lib/config/braid-paths.js +6 -0
- package/dist/lib/config/braid-settings.d.ts +22 -0
- package/dist/lib/config/braid-settings.js +15 -0
- package/dist/lib/detection/tool-loop-detector.d.ts +95 -46
- package/dist/lib/detection/tool-loop-detector.js +288 -180
- package/dist/lib/reasoning/dreamer.d.ts +2 -1
- package/dist/lib/reasoning/dreamer.js +12 -7
- package/docs/LOCAL_DEV.md +1 -1
- package/docs/SETUP.md +1 -0
- package/package.json +1 -1
- package/src/extensions/core/AGENTS.md +3 -1
- package/src/extensions/core/prompt-builder/tool-guidelines.ts +1 -1
- package/src/extensions/core/skills-tool/applicability.ts +220 -0
- package/src/extensions/core/skills-tool/auto-inject.ts +391 -0
- package/src/extensions/core/skills-tool/index.ts +96 -3
- package/src/extensions/core/skills-tool/loading-guidance.ts +6 -2
- package/src/extensions/core/skills-tool/session-records.ts +932 -5
- package/src/extensions/core/skills-tool/skill-families.ts +112 -0
- package/src/extensions/enforcement/handlers/analysis/dreamer-handler.ts +5 -1
- package/src/extensions/enforcement/handlers/register-tool-call.ts +5 -2
- package/src/extensions/enforcement/verification-judge.ts +32 -25
- package/src/extensions/memory/AGENTS.md +2 -2
- package/src/extensions/memory/session-recall/AGENTS.md +92 -18
- package/src/extensions/memory/session-recall/ask.ts +435 -0
- package/src/extensions/memory/session-recall/corpus-index.ts +609 -0
- package/src/extensions/memory/session-recall/index.ts +202 -21
- package/src/extensions/memory/session-recall/search.ts +61 -10
- package/src/lib/ai/payload-hygiene.ts +47 -3
- package/src/lib/config/braid-paths.ts +7 -0
- package/src/lib/config/braid-settings.ts +42 -3
- package/src/lib/detection/AGENTS.md +8 -7
- package/src/lib/detection/tool-loop-detector.ts +339 -195
- package/src/lib/reasoning/dreamer.ts +13 -6
|
@@ -0,0 +1,112 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* The tool families a skill declares it governs.
|
|
3
|
+
*
|
|
4
|
+
* A trigger phrase says WHEN to open a skill file; nothing in the file says
|
|
5
|
+
* what the file's advice is then used FOR. `governs:` closes that: the tool
|
|
6
|
+
* families whose calls the body is written to direct. A family is the tool
|
|
7
|
+
* name's first underscore-delimited segment (`search_documents` → `search`),
|
|
8
|
+
* the same token `loading-guidance.ts` pairs with a skill name in the
|
|
9
|
+
* mechanical trigger, so "before the first call in the family" in the prompt
|
|
10
|
+
* and "a call in a declared family" in the record cannot mean two things.
|
|
11
|
+
*
|
|
12
|
+
* The declaration is authored data in the skill's own frontmatter — pi's
|
|
13
|
+
* parser already returns it, and the pack's authoring contract already admits
|
|
14
|
+
* any YAML-valid extra key. No new file format and no new field parser is
|
|
15
|
+
* needed, which is why this module only reads.
|
|
16
|
+
*
|
|
17
|
+
* Most skills govern nothing: advice about prose, a palette or a checklist
|
|
18
|
+
* directs no tool, and a file without the key is read as declaring no family.
|
|
19
|
+
* The absence is a fact, not a gap — an obligation that names no family is
|
|
20
|
+
* satisfied by the read alone.
|
|
21
|
+
*
|
|
22
|
+
* A declared family may not be the detector's own tool (`read`): a call to it
|
|
23
|
+
* IS the read leg, so counting it as an applied call would let an obligation
|
|
24
|
+
* satisfy itself. The parser drops such an entry; the shipped pack's contract
|
|
25
|
+
* refuses it by name so an author sees the problem before a release.
|
|
26
|
+
*
|
|
27
|
+
* Nothing here names a skill, a package or a path. Reads are memoized by file
|
|
28
|
+
* mtime + size, so a turn pays one stat per skill file and a parse only when
|
|
29
|
+
* a file changed.
|
|
30
|
+
*/
|
|
31
|
+
|
|
32
|
+
import { readFileSync, statSync } from "node:fs";
|
|
33
|
+
import { parseFrontmatter } from "@earendil-works/pi-coding-agent";
|
|
34
|
+
|
|
35
|
+
/**
|
|
36
|
+
* Max families read from one declaration. A body that directs more than four
|
|
37
|
+
* tool families is not one skill's job, and the record's per-session rows are
|
|
38
|
+
* bounded by the same number. The pack's contract mirrors this cap.
|
|
39
|
+
*/
|
|
40
|
+
export const MAX_GOVERNED_FAMILIES = 4;
|
|
41
|
+
|
|
42
|
+
/** The token shape a declaration must use: a tool name's first segment. */
|
|
43
|
+
export const GOVERNED_FAMILY_SHAPE = /^[a-z][a-z0-9]*$/;
|
|
44
|
+
|
|
45
|
+
/**
|
|
46
|
+
* The tool whose call is the read leg itself. Never a declarable family: see
|
|
47
|
+
* the module header.
|
|
48
|
+
*/
|
|
49
|
+
export const DETECTOR_TOOL = "read";
|
|
50
|
+
|
|
51
|
+
/** The family a tool name belongs to: its first underscore-delimited segment. */
|
|
52
|
+
export function toolFamilyToken(toolName: string): string {
|
|
53
|
+
return (toolName ?? "").split("_")[0].toLowerCase();
|
|
54
|
+
}
|
|
55
|
+
|
|
56
|
+
/**
|
|
57
|
+
* Normalize a frontmatter `governs` value into family tokens. Accepts a YAML
|
|
58
|
+
* list or a comma-separated string, lowercases, drops malformed, detector, and
|
|
59
|
+
* surplus entries, and dedupes in authored order. Pure.
|
|
60
|
+
*/
|
|
61
|
+
export function parseGovernedFamilies(value: unknown): string[] {
|
|
62
|
+
const raw = Array.isArray(value) ? value : typeof value === "string" ? value.split(",") : [];
|
|
63
|
+
const families: string[] = [];
|
|
64
|
+
for (const item of raw) {
|
|
65
|
+
if (typeof item !== "string") continue;
|
|
66
|
+
const family = item.trim().toLowerCase();
|
|
67
|
+
if (!GOVERNED_FAMILY_SHAPE.test(family) || family === DETECTOR_TOOL) continue;
|
|
68
|
+
if (families.includes(family)) continue;
|
|
69
|
+
families.push(family);
|
|
70
|
+
if (families.length >= MAX_GOVERNED_FAMILIES) break;
|
|
71
|
+
}
|
|
72
|
+
return families;
|
|
73
|
+
}
|
|
74
|
+
|
|
75
|
+
interface FamilyCacheEntry {
|
|
76
|
+
mtimeMs: number;
|
|
77
|
+
size: number;
|
|
78
|
+
families: string[];
|
|
79
|
+
}
|
|
80
|
+
|
|
81
|
+
const cache = new Map<string, FamilyCacheEntry>();
|
|
82
|
+
|
|
83
|
+
/**
|
|
84
|
+
* Read the families a skill file declares. Missing or unreadable file, or
|
|
85
|
+
* malformed frontmatter, yields an empty list (a declaration that cannot be
|
|
86
|
+
* read is not a declaration). Memoized: an unchanged file never re-reads.
|
|
87
|
+
*/
|
|
88
|
+
export function readGovernedFamilies(filePath: string): string[] {
|
|
89
|
+
let st;
|
|
90
|
+
try {
|
|
91
|
+
st = statSync(filePath);
|
|
92
|
+
} catch {
|
|
93
|
+
cache.delete(filePath);
|
|
94
|
+
return [];
|
|
95
|
+
}
|
|
96
|
+
const cached = cache.get(filePath);
|
|
97
|
+
if (cached && cached.mtimeMs === st.mtimeMs && cached.size === st.size) return cached.families;
|
|
98
|
+
let families: string[] = [];
|
|
99
|
+
try {
|
|
100
|
+
const { frontmatter } = parseFrontmatter(readFileSync(filePath, "utf-8"));
|
|
101
|
+
families = parseGovernedFamilies((frontmatter as Record<string, unknown>).governs);
|
|
102
|
+
} catch {
|
|
103
|
+
// Unreadable or unparseable: no declared family, and no throw on a turn.
|
|
104
|
+
}
|
|
105
|
+
cache.set(filePath, { mtimeMs: st.mtimeMs, size: st.size, families });
|
|
106
|
+
return families;
|
|
107
|
+
}
|
|
108
|
+
|
|
109
|
+
/** Test seam: forget every memoized read. */
|
|
110
|
+
export function clearGovernedFamilyCache(): void {
|
|
111
|
+
cache.clear();
|
|
112
|
+
}
|
|
@@ -9,6 +9,7 @@ import { emitSignal } from "../../../../lib/reasoning/belief-policy-engine.js";
|
|
|
9
9
|
import { dream } from "../../../../lib/reasoning/dreamer.js";
|
|
10
10
|
import { recordAdvice } from "../../../../lib/reasoning/advice-tracker.js";
|
|
11
11
|
import { tryClaim } from "../../../../lib/enforcement/turn-dedup.js";
|
|
12
|
+
import { getToolLoopState } from "../../../../lib/detection/tool-loop-detector.js";
|
|
12
13
|
import { getState } from "../../state.js";
|
|
13
14
|
|
|
14
15
|
const log = createLogger("enforcement:dreamer");
|
|
@@ -25,7 +26,10 @@ export async function dreamerHandler(pi: any, _event: any, ctx: ExtensionContext
|
|
|
25
26
|
}
|
|
26
27
|
}
|
|
27
28
|
|
|
28
|
-
|
|
29
|
+
// The loop path reads the detector's own recorded calls (tool + argument
|
|
30
|
+
// identity); gs.toolLog is names only and cannot distinguish one call
|
|
31
|
+
// repeated from many different calls under one name.
|
|
32
|
+
const dreamAdvice = dream(ctx, gs.toolLog, historicalSequences, getToolLoopState(ctx).history);
|
|
29
33
|
if (!dreamAdvice) return;
|
|
30
34
|
|
|
31
35
|
emitSignal(ctx, "stability", "predicted-decline", "info", false, {
|
|
@@ -46,9 +46,12 @@ export function registerToolCallHandler(pi: ExtensionAPI): void {
|
|
|
46
46
|
|
|
47
47
|
recordToolCall(toolName, ctx);
|
|
48
48
|
|
|
49
|
-
// Record tool call for tool-loop-detector (per-turn tool sets)
|
|
49
|
+
// Record tool call for tool-loop-detector (per-turn tool sets). The
|
|
50
|
+
// arguments ride along: a loop is the same call repeating, and the tool
|
|
51
|
+
// name alone cannot say whether sixteen `bash` calls are sixteen commands
|
|
52
|
+
// or one command sixteen times.
|
|
50
53
|
const loopState = getToolLoopState(ctx);
|
|
51
|
-
recordLoopToolCall(loopState, toolName);
|
|
54
|
+
recordLoopToolCall(loopState, toolName, input);
|
|
52
55
|
recordToolInput(toolName, input, getState(ctx));
|
|
53
56
|
|
|
54
57
|
if (toolName === "bash") {
|
|
@@ -126,6 +126,13 @@ export interface JudgeContext {
|
|
|
126
126
|
|
|
127
127
|
/**
|
|
128
128
|
* Build the verification analysis prompt. Exported for tests.
|
|
129
|
+
*
|
|
130
|
+
* Order is load-bearing: the ground-truth sections (git history, the
|
|
131
|
+
* disk-checked deliverables manifest, the code-changes flag) come BEFORE the
|
|
132
|
+
* bounded ones (task evidence, tool log, reasoning, response). The bounded
|
|
133
|
+
* sections are the ones a bound can shorten: a head kept under a size cap, a
|
|
134
|
+
* tail cut by a redaction. A reader that meets the verdict-bearing sections
|
|
135
|
+
* first cannot have them removed by whatever truncated the evidence.
|
|
129
136
|
*/
|
|
130
137
|
export function buildVerificationPrompt(context: JudgeContext): string {
|
|
131
138
|
const sections: string[] = [];
|
|
@@ -138,31 +145,6 @@ export function buildVerificationPrompt(context: JudgeContext): string {
|
|
|
138
145
|
sections.push(context.toolsUsed.length > 0 ? context.toolsUsed.join(", ") : "(none)");
|
|
139
146
|
sections.push("");
|
|
140
147
|
|
|
141
|
-
// Task-wide evidence (since last user message, session-store
|
|
142
|
-
// backed) comes BEFORE the current-window tool log — it is the surviving
|
|
143
|
-
// view when turn breaks cut toolLog down to the tail turn.
|
|
144
|
-
if (context.taskEvidence && context.taskEvidence.length > 0) {
|
|
145
|
-
sections.push("## TASK EVIDENCE (since last user message, disk/session-surviving)");
|
|
146
|
-
sections.push(context.taskEvidence.join("\n"));
|
|
147
|
-
sections.push("");
|
|
148
|
-
}
|
|
149
|
-
|
|
150
|
-
if (context.toolLog.length > 0) {
|
|
151
|
-
sections.push("## TOOL EXECUTION LOG");
|
|
152
|
-
sections.push(context.toolLog.slice(-50).join("\n")); // Last 50 entries (matches turnHistory cap)
|
|
153
|
-
sections.push("");
|
|
154
|
-
}
|
|
155
|
-
|
|
156
|
-
sections.push(`## CODE CHANGES: ${context.hasCodeChanges ? "YES" : "NO"}`);
|
|
157
|
-
// (Verification evidence comes from the TOOL EXECUTION LOG above, not a boolean)
|
|
158
|
-
sections.push("");
|
|
159
|
-
|
|
160
|
-
if (context.thinking) {
|
|
161
|
-
sections.push("## AGENT REASONING (thinking blocks)");
|
|
162
|
-
sections.push(context.thinking);
|
|
163
|
-
sections.push("");
|
|
164
|
-
}
|
|
165
|
-
|
|
166
148
|
if (context.gitHistory) {
|
|
167
149
|
sections.push("## GIT HISTORY (recent commits)");
|
|
168
150
|
sections.push(context.gitHistory);
|
|
@@ -188,6 +170,31 @@ export function buildVerificationPrompt(context: JudgeContext): string {
|
|
|
188
170
|
sections.push("");
|
|
189
171
|
}
|
|
190
172
|
|
|
173
|
+
sections.push(`## CODE CHANGES: ${context.hasCodeChanges ? "YES" : "NO"}`);
|
|
174
|
+
// (Verification evidence comes from the TOOL EXECUTION LOG, not a boolean)
|
|
175
|
+
sections.push("");
|
|
176
|
+
|
|
177
|
+
// Task-wide evidence (since last user message, session-store backed) comes
|
|
178
|
+
// BEFORE the current-window tool log — it is the surviving view when turn
|
|
179
|
+
// breaks cut toolLog down to the tail turn.
|
|
180
|
+
if (context.taskEvidence && context.taskEvidence.length > 0) {
|
|
181
|
+
sections.push("## TASK EVIDENCE (since last user message, disk/session-surviving)");
|
|
182
|
+
sections.push(context.taskEvidence.join("\n"));
|
|
183
|
+
sections.push("");
|
|
184
|
+
}
|
|
185
|
+
|
|
186
|
+
if (context.toolLog.length > 0) {
|
|
187
|
+
sections.push("## TOOL EXECUTION LOG");
|
|
188
|
+
sections.push(context.toolLog.slice(-50).join("\n")); // Last 50 entries (matches turnHistory cap)
|
|
189
|
+
sections.push("");
|
|
190
|
+
}
|
|
191
|
+
|
|
192
|
+
if (context.thinking) {
|
|
193
|
+
sections.push("## AGENT REASONING (thinking blocks)");
|
|
194
|
+
sections.push(context.thinking);
|
|
195
|
+
sections.push("");
|
|
196
|
+
}
|
|
197
|
+
|
|
191
198
|
if (context.lastResponse) {
|
|
192
199
|
sections.push("## AGENT RESPONSE (visible text)");
|
|
193
200
|
sections.push(context.lastResponse);
|
|
@@ -117,8 +117,8 @@ The memory extensions expose 4 LLM-callable tools. The unified `memory` tool cov
|
|
|
117
117
|
| -------------- | ------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
118
118
|
| `memory_store` | memory-agent-bridge | semantic (unified) | Save insights/patterns for cross-session recall |
|
|
119
119
|
| `memory` | memory-agent-bridge | all three | Unified access: `recall_search` (semantic search, also cross-searches episodes), `recall_list`, `recall_get`, `episodes_search` (case-based reasoning), `episodes_list`, `episodes_add`, `heuristics_list`, `heuristics_add`, `heuristics_evolve` |
|
|
120
|
-
| `session` | session-recall | session log | Search past sessions (action=search) or query one (action=query)
|
|
120
|
+
| `session` | session-recall | session log | Search past sessions (action=search), ask what the owner said earlier with citations (action=ask), or query one (action=query) |
|
|
121
121
|
|
|
122
|
-
**Rule of thumb:** start with `memory` action=recall_search for "have I seen this before?". Use `memory` action=episodes_search for "how did a similar task go?". Use `memory` action=heuristics_list when investigating behavior tuning. Use `session` for cross-session history.
|
|
122
|
+
**Rule of thumb:** start with `memory` action=recall_search for "have I seen this before?". Use `memory` action=episodes_search for "how did a similar task go?". Use `memory` action=heuristics_list when investigating behavior tuning. Use `session` for cross-session history, and `session` action=ask when the question is "what did the owner say or decide about X?" - it reads the indexed user turns and cites each hit.
|
|
123
123
|
|
|
124
124
|
The data-layer functions (`addEpisode`, `addPrinciple`, `recordToolCall`, etc.) live in `three-memory-stores/episodes.ts`, `heuristics.ts`, `tool-reliability.ts` and are imported by many extensions (e.g. `finalize.ts` adds episodes on task completion). These are infrastructure and should not be moved out of the directory.
|
|
@@ -2,26 +2,43 @@
|
|
|
2
2
|
|
|
3
3
|
# Session Recall Extension
|
|
4
4
|
|
|
5
|
-
Search and query past sessions. The agent can call this to find
|
|
6
|
-
work without leaving the current
|
|
5
|
+
Search, ask over, and query past sessions. The agent can call this to find
|
|
6
|
+
prior work - or to find what the owner said - without leaving the current
|
|
7
|
+
session.
|
|
7
8
|
|
|
8
9
|
## Architecture
|
|
9
10
|
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
11
|
+
`index.ts` is the extension entry point and the tool surface. Helpers are
|
|
12
|
+
co-located by role, not extracted further:
|
|
13
|
+
|
|
14
|
+
| File | Owns |
|
|
15
|
+
| ---------------------- | ------------------------------------------------------------------------- |
|
|
16
|
+
| `search.ts` | live file discovery, the rg/grep/node backends, line hits, project labels |
|
|
17
|
+
| `rank.ts` | the four-signal file ranking used by `search` |
|
|
18
|
+
| `query.ts` | one session's transcript for the LLM paths |
|
|
19
|
+
| `corpus-index.ts` | the user-turn index: extraction, incremental build, fail-closed load |
|
|
20
|
+
| `ask.ts` | BM25 retrieval over that index and the cited answer's rendering |
|
|
21
|
+
| `analyze.ts` | multi-session LLM analysis |
|
|
22
|
+
| `context-aggregate.ts` | the `context` action's aggregation |
|
|
23
|
+
| `stats.ts` | session-file statistics |
|
|
13
24
|
|
|
14
25
|
## Tools exposed
|
|
15
26
|
|
|
16
27
|
| Tool | Actions | Purpose |
|
|
17
28
|
| --------- | ------- | -------------------------------------------------------------------- |
|
|
18
29
|
| `session` | search | Full-text search across session history |
|
|
30
|
+
| `session` | ask | BM25 over the user-turn index, answered with citations |
|
|
19
31
|
| `session` | query | Structured query (by date, by tool, by tag) |
|
|
20
32
|
| `session` | analyze | Multi-session LLM analysis over a session list |
|
|
21
33
|
| `session` | batch | Pattern aggregation across sessions (by project/month) |
|
|
22
34
|
| `session` | stats | Session-file statistics (count, size, by project/month) |
|
|
23
35
|
| `session` | context | Cross-session §8 context-adoption aggregate (loss/coverage/adoption) |
|
|
24
36
|
|
|
37
|
+
`search`, `batch` and `ask` never return the calling session's own transcript
|
|
38
|
+
(`search.ts: excludeSessionPath`), and none of them reports a bare match count
|
|
39
|
+
as its result: `search` and `batch` render the matched lines with their line
|
|
40
|
+
numbers, ranked and capped; `ask` renders cited turns.
|
|
41
|
+
|
|
25
42
|
## `context` action
|
|
26
43
|
|
|
27
44
|
Repeatable cross-session aggregation of the §8 context-management counters -
|
|
@@ -43,6 +60,58 @@ Signals are matched to the durable session-file markers:
|
|
|
43
60
|
pure aggregation lives in `context-aggregate.ts` (testable against fixture
|
|
44
61
|
session files); `search.ts` shares the file-discovery helpers.
|
|
45
62
|
|
|
63
|
+
## The corpus index and `ask`
|
|
64
|
+
|
|
65
|
+
The transcript corpus holds gigabytes of agent reasoning and tool output, and
|
|
66
|
+
under 10 MB of the owner's own words. `ask` reads that fraction as a
|
|
67
|
+
population: a derived store of one row per user turn, written by
|
|
68
|
+
`scripts/build-corpus-index.mjs` and queried by `askCorpusIndex`.
|
|
69
|
+
|
|
70
|
+
- **Location**: `<braid root>/corpus-index/`, one shard per project
|
|
71
|
+
(`corpusIndexDir()` in `lib/config/braid-paths.ts`; the shard's file-name
|
|
72
|
+
stem is `shardSlug`, one rule for the writer and the reader). Derived data:
|
|
73
|
+
deleting the directory costs a rebuild, nothing else.
|
|
74
|
+
- **Row**: `{session, session_id, project, ts, line, text}`, plus the shard's
|
|
75
|
+
first line, a manifest carrying format version, project, build time, row
|
|
76
|
+
count, text bytes and the `(path, mtime, size, turns)` of every source file.
|
|
77
|
+
- **Content**: `message` entries with role `user` and text. Tool results
|
|
78
|
+
arrive as user-role messages with no text block and drop out with the text
|
|
79
|
+
filter; assistant turns, `custom_message` injections and durable `custom`
|
|
80
|
+
entries never enter a row, so no key, tool argument or environment value is
|
|
81
|
+
retrievable from the index.
|
|
82
|
+
- **Compaction archives are not indexed**: an `archives/*.jsonl` envelope is a
|
|
83
|
+
second copy of messages the session body carries (`writeOriginalsArchive`
|
|
84
|
+
stores the pre-compaction originals), so indexing both would count the same
|
|
85
|
+
words twice.
|
|
86
|
+
- **Build**: one pass over the session files on a full build; later runs re-parse
|
|
87
|
+
only the files whose `(mtime, size)` moved. A shard that cannot be read is
|
|
88
|
+
rebuilt, never merged - derived data has one authority.
|
|
89
|
+
- **Fail closed**: `loadCorpusIndex` returns `{ok: false, reason}` naming the
|
|
90
|
+
shard and the defect when a manifest does not parse, the format is unknown,
|
|
91
|
+
a row does not parse, or the row count disagrees with the manifest. `ask`
|
|
92
|
+
then says it cannot answer and prints the build command; it never answers
|
|
93
|
+
from half a store.
|
|
94
|
+
- **Scoring**: BM25 (`k1 = 1.2`, `b = 0.75`) over one document per turn, with a
|
|
95
|
+
fixed English function-word list and a symmetric suffix fold, so `renders`
|
|
96
|
+
reaches `rendered` and a hyphen separates words. No embeddings, no vector
|
|
97
|
+
store: at this size the lexical model is cheap, testable and re-derivable by
|
|
98
|
+
eye. What would earn one is a measured recall@3 below the bar on questions
|
|
99
|
+
whose answering passage is in the index.
|
|
100
|
+
- **Freshness**: an answer states the population it read and how many session
|
|
101
|
+
files moved since the build (`corpusIndexFreshness`), so a stale shard is
|
|
102
|
+
visible rather than silent.
|
|
103
|
+
|
|
104
|
+
Rebuild after a day of sessions:
|
|
105
|
+
|
|
106
|
+
```bash
|
|
107
|
+
npm run build && node packages/core/scripts/build-corpus-index.mjs
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
Recall over a pre-registered question set is measured on the live corpus, not in
|
|
111
|
+
the suite: `test/manual/ask-corpus-recall.mjs --questions <file.json>` scores
|
|
112
|
+
whether the answering turn appears in the top 3 and prints the rank of every
|
|
113
|
+
expected turn, including the misses.
|
|
114
|
+
|
|
46
115
|
## What it indexes
|
|
47
116
|
|
|
48
117
|
- Session title (if set)
|
|
@@ -78,22 +147,27 @@ full-corpus backend invocation and an N-term query N.
|
|
|
78
147
|
|
|
79
148
|
## Storage
|
|
80
149
|
|
|
81
|
-
Sessions are stored as JSONL
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
150
|
+
Sessions are stored as JSONL under pi's session root (one encoded-cwd directory
|
|
151
|
+
per project). `search` and `batch` scan those files live with the rg/grep/node
|
|
152
|
+
backends; `ask` reads the precomputed corpus index described above. The
|
|
153
|
+
extension writes no session file - the index is the one artifact it writes, and
|
|
154
|
+
it is written only by the build script.
|
|
85
155
|
|
|
86
156
|
## Performance
|
|
87
157
|
|
|
88
|
-
The corpus measured
|
|
89
|
-
single-term search is one full-corpus pass plus the bounded
|
|
90
|
-
on the dev machine against that corpus, down from ~10.6 s
|
|
91
|
-
cost a second full pass. A multi-term query is one pass per
|
|
92
|
-
two, down from ~19.3 s). `--max-filesize` caps any one file at
|
|
93
|
-
runaway session cannot dominate a scan; the
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
the
|
|
158
|
+
The corpus measured when the search cost work landed was ~3,250 session files /
|
|
159
|
+
2.3 GB. A single-term search is one full-corpus pass plus the bounded
|
|
160
|
+
refinements: ~4.5 s on the dev machine against that corpus, down from ~10.6 s
|
|
161
|
+
when the role signal cost a second full pass. A multi-term query is one pass per
|
|
162
|
+
term (~8.9 s for two, down from ~19.3 s). `--max-filesize` caps any one file at
|
|
163
|
+
32 MB, so a runaway session cannot dominate a scan; the files above the cap are
|
|
164
|
+
not searched.
|
|
165
|
+
|
|
166
|
+
The corpus measured for the corpus index is 3,832 session files / 3.8 GB. A full
|
|
167
|
+
index build parses every file in 20.6 s and writes 20,477 turns (9.66 MB) across
|
|
168
|
+
341 project shards; an incremental build re-parsing today's changed sessions
|
|
169
|
+
reports 1.0 s of work. An `ask` pays 0.2 s to load the shards, 0.2 s for the
|
|
170
|
+
freshness walk, and ~0.5 s to tokenize and score the population.
|
|
97
171
|
|
|
98
172
|
## Local Contracts
|
|
99
173
|
|