llm-wiki-kit 0.2.9 → 0.2.10
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +4 -4
- package/docs/concepts.md +4 -3
- package/docs/manual.md +19 -3
- package/docs/operations.md +21 -2
- package/package.json +1 -1
- package/src/cli.js +4 -2
- package/src/consolidate.js +62 -29
- package/src/maintenance.js +97 -2
- package/src/templates.js +13 -4
- package/src/wiki-lint.js +47 -9
- package/src/wiki-search.js +84 -19
- package/src/wiki-visibility.js +76 -0
package/README.md
CHANGED
|
@@ -126,13 +126,13 @@ Installed npm runtimes also perform a cached update notice check from hooks whil
|
|
|
126
126
|
|
|
127
127
|
`llm-wiki post-update --workspace <project>` reapplies the current runtime's hook entries and safe managed template updates without running `npm install -g`. Use `post-update --all --workspace <search-root>` to reapply templates across discovered project roots.
|
|
128
128
|
|
|
129
|
-
`llm-wiki context "<query>"` prints the full debug view of the layered context sources used by hooks. Hook injection may render those sources as functional compact context for Codex and Claude, but this CLI stays verbose so maintainers can inspect retrieval, snippets, memory, index, and
|
|
129
|
+
`llm-wiki context "<query>"` prints the full debug view of the layered context sources used by hooks. Hook injection may render those sources as functional compact context for Codex and Claude, but this CLI stays verbose so maintainers can inspect retrieval, snippets, memory, index, expansion behavior, and context budget metadata. Daily use should rely on hook injection. By default, episodic `wiki/queries/`, `wiki/context/`, and `session-log` pages are excluded from search unless they were promoted with `memory_type: semantic` or `procedural` and `importance >= 4`; use `--include-episodic` only when debugging old automatic records. Archived or superseded pages are hidden unless `--include-archived` is requested, while stale pages remain searchable with lower score.
|
|
130
130
|
|
|
131
|
-
`llm-wiki lint` checks wiki health and detects outdated managed rules from older kit versions. Agents may use it before/after meaningful wiki maintenance.
|
|
131
|
+
`llm-wiki lint` checks wiki health and detects outdated managed rules from older kit versions. It also warns when `memory.md` is near budget, wiki page count nears the search cap, hidden episodic/context pages accumulate, or stale/archived pages lack supersession/link discoverability. Agents may use it before/after meaningful wiki maintenance.
|
|
132
132
|
|
|
133
|
-
`llm-wiki consolidate` refreshes only generated marker blocks in `wiki/memory.md` and `wiki/index.md`. It is an agent maintenance helper, not a command users should run after every turn.
|
|
133
|
+
`llm-wiki consolidate` refreshes only generated marker blocks in `wiki/memory.md` and `wiki/index.md`. Generated maps keep durable non-archived pages, hide default episodic records, skip stale/archived/superseded pages, and report those counts in dry-run output. It is an agent maintenance helper, not a command users should run after every turn.
|
|
134
134
|
|
|
135
|
-
`llm-wiki maintenance` prints the pending queue from `llm-wiki/outputs/maintenance/queue.md`. Hooks create only selective candidates; the active agent should merge reusable items into existing durable wiki pages and mark queue items `done` or `skipped` without delaying unrelated user answers.
|
|
135
|
+
`llm-wiki maintenance` prints the pending queue and review due status from `llm-wiki/outputs/maintenance/queue.md`. Hooks create only selective candidates; the active agent should merge reusable items into existing durable wiki pages and mark queue items `done` or `skipped` without delaying unrelated user answers. Periodic maintenance is a soft agent-side reminder, not a user command loop.
|
|
136
136
|
|
|
137
137
|
`llm-wiki projects --workspace /apps` lists project roots that already have `llm-wiki-kit` state or an older `llm-wiki/wiki/index.md`, and shows the update commands to run. `llm-wiki update --workspace /apps` updates the global runtime once, then reapplies managed templates across every known or discovered project root under `/apps`.
|
|
138
138
|
|
package/docs/concepts.md
CHANGED
|
@@ -39,7 +39,8 @@ The maintenance loop is intentionally layered:
|
|
|
39
39
|
|
|
40
40
|
- `memory.md`: short hot index for current durable facts.
|
|
41
41
|
- `index.md`: broad navigation map.
|
|
42
|
-
- MiniSearch + wikilinks: retrieval over durable `wiki/**/*.md`, with episodic `wiki/queries
|
|
42
|
+
- MiniSearch + wikilinks: retrieval over durable `wiki/**/*.md`, with episodic `wiki/queries/`, `wiki/context/`, and `session-log` pages hidden by default unless promoted or `--include-episodic` is requested; archived/superseded pages stay preserved but hidden unless `--include-archived` is requested.
|
|
43
43
|
- `outputs/maintenance/queue.md`: selective reminders for explicit durable requests that need review, plus stale turn recovery.
|
|
44
|
-
- `lint`: finds broken links, stale pages, duplicates, metadata gaps, secret-like content,
|
|
45
|
-
- `
|
|
44
|
+
- `lint`: finds broken links, stale pages, duplicates, metadata gaps, secret-like content, outdated managed rules, memory/page-count budget pressure, hidden episodic growth, and stale/archived discoverability gaps.
|
|
45
|
+
- `maintenance`: reports `reviewDue` only when periodic thresholds are met; hook reminders are soft and limited to session start/instructions loaded or maintenance-related prompts.
|
|
46
|
+
- `consolidate`: agent helper that refreshes generated blocks in `memory.md` and `index.md` while preserving handwritten notes, keeping default query/context/session pages out of the durable generated maps, and skipping stale/archived/superseded pages.
|
package/docs/manual.md
CHANGED
|
@@ -271,6 +271,7 @@ llm-wiki context "auth architecture" --workspace /path/to/project --json
|
|
|
271
271
|
llm-wiki context "auth architecture" --workspace /path/to/project --limit 8
|
|
272
272
|
llm-wiki context "auth architecture" --workspace /path/to/project --no-expand
|
|
273
273
|
llm-wiki context "auth architecture" --workspace /path/to/project --include-episodic
|
|
274
|
+
llm-wiki context "auth architecture" --workspace /path/to/project --include-archived
|
|
274
275
|
```
|
|
275
276
|
|
|
276
277
|
읽는 자료:
|
|
@@ -281,7 +282,7 @@ llm-wiki context "auth architecture" --workspace /path/to/project --include-epis
|
|
|
281
282
|
- MiniSearch 또는 substring fallback 결과
|
|
282
283
|
- strong match의 one-hop wikilink neighbors
|
|
283
284
|
|
|
284
|
-
기본 검색은 episodic `wiki/queries
|
|
285
|
+
기본 검색은 durable semantic/procedural page를 우선하고, episodic `wiki/queries/`, `wiki/context/`, `session-log` 계열은 숨긴다. 해당 page가 `memory_type: semantic|procedural`이고 `importance >= 4`이면 promoted durable page로 취급한다. `status: archived` 또는 `superseded_by`가 있는 page도 기본 검색에서 제외하며, 필요한 조사 때만 `--include-archived`로 복구한다. `status: stale` page는 보존하고 검색할 수 있지만 score를 낮춘다. JSON 출력에는 memory/index/hit/snippet budget metadata가 포함된다.
|
|
285
286
|
|
|
286
287
|
### `llm-wiki lint`
|
|
287
288
|
|
|
@@ -302,6 +303,10 @@ llm-wiki lint --workspace /path/to/project
|
|
|
302
303
|
- secret-like content
|
|
303
304
|
- duplicate aliases/titles
|
|
304
305
|
- stale pages and orphan candidates
|
|
306
|
+
- `memory.md` near-budget/oversized 상태
|
|
307
|
+
- wiki page count가 search cap에 근접했는지
|
|
308
|
+
- default-hidden episodic/context/session page 성장
|
|
309
|
+
- stale/archived page의 supersession/link discoverability
|
|
305
310
|
- outdated managed rules/templates
|
|
306
311
|
- stale or oversized maintenance queue
|
|
307
312
|
|
|
@@ -321,7 +326,7 @@ llm-wiki consolidate --workspace /path/to/project --dry-run
|
|
|
321
326
|
- `wiki/memory.md`의 generated memory map
|
|
322
327
|
- `wiki/index.md`의 generated page map
|
|
323
328
|
|
|
324
|
-
handwritten content는 보존한다. malformed marker block은 덮어쓰지 않고 건너뛴다. 기본 `query`, `context`, `session-log` page는 explicit durable metadata가 없으면 generated map에서 제외한다.
|
|
329
|
+
handwritten content는 보존한다. malformed marker block은 덮어쓰지 않고 건너뛴다. generated map에는 durable non-archived page만 넣는다. `stale`, `archived`, `superseded_by` page는 보존하지만 generated map에서는 제외한다. 기본 `query`, `context`, `session-log` page는 explicit durable metadata가 없으면 generated map에서 제외한다. `--dry-run` 결과는 indexed durable pages, hidden episodic pages, archived/stale/superseded skipped counts를 보고한다.
|
|
325
330
|
|
|
326
331
|
### `llm-wiki maintenance`
|
|
327
332
|
|
|
@@ -332,7 +337,18 @@ llm-wiki maintenance --workspace /path/to/project
|
|
|
332
337
|
llm-wiki maintenance --workspace /path/to/project --json
|
|
333
338
|
```
|
|
334
339
|
|
|
335
|
-
이 명령은 page merge를 자동 수행하지 않는다. active agent가 pending item을 읽고, 가장 가까운 durable wiki 문서에 병합한 뒤 queue item을 `done` 또는 `skipped`로 표시한다.
|
|
340
|
+
이 명령은 page merge를 자동 수행하지 않는다. 정기 maintenance는 agent-side task이며 사용자가 매 turn 실행할 필요가 없다. active agent가 pending item을 읽고, 가장 가까운 durable wiki 문서에 병합한 뒤 queue item을 `done` 또는 `skipped`로 표시한다.
|
|
341
|
+
|
|
342
|
+
JSON 출력에는 `reviewDue`, `reviewReasons`, `pendingCount`, `stalePendingCount`, `health`, `recommendedCommands`가 포함된다. Review due 조건은 마지막 review 후 14일 경과, pending 5개 이상, stale/result-missing pending item, lint warning/error, `memory.md` near-budget 이상, wiki page count가 search cap의 80% 이상인 경우다. Hook은 due 상태를 `SessionStart`/`InstructionsLoaded`에서만 짧게 보여주고, `UserPromptSubmit`에서는 사용자가 wiki/maintenance/정리 관련 질문을 했을 때만 보여준다. 현재 답변을 중단하거나 정리를 강제하지 않는다.
|
|
343
|
+
|
|
344
|
+
Agent checklist:
|
|
345
|
+
|
|
346
|
+
```bash
|
|
347
|
+
llm-wiki lint --workspace /path/to/project
|
|
348
|
+
llm-wiki maintenance --workspace /path/to/project
|
|
349
|
+
llm-wiki consolidate --workspace /path/to/project --dry-run
|
|
350
|
+
llm-wiki consolidate --workspace /path/to/project
|
|
351
|
+
```
|
|
336
352
|
|
|
337
353
|
### `llm-wiki uninstall`
|
|
338
354
|
|
package/docs/operations.md
CHANGED
|
@@ -157,7 +157,7 @@ Pre-compact preservation defaults to `LLM_WIKI_KIT_PRECOMPACT_ENFORCEMENT=limite
|
|
|
157
157
|
- one-hop wikilink neighbors for the strongest matches
|
|
158
158
|
- redacted output fields for query text, memory/index/log excerpts, hit paths, titles, snippets, matched terms, and link expansion metadata
|
|
159
159
|
|
|
160
|
-
Default context search
|
|
160
|
+
Default context search prioritizes durable semantic/procedural pages. It hides episodic `wiki/queries/`, `wiki/context/`, and `session-log` pages unless they were explicitly promoted with `memory_type: semantic` or `procedural` and `importance >= 4`. It also hides `status: archived` and `superseded_by` pages unless `--include-archived` is requested. `status: stale` pages remain searchable but receive a lower score. JSON output includes memory/index/hit/snippet budget metadata so maintainers can see how much context each layer consumed. Use `--include-episodic` only when debugging historical automatic query pages:
|
|
161
161
|
|
|
162
162
|
Use it only when you want to inspect what the next agent turn should see:
|
|
163
163
|
|
|
@@ -166,6 +166,7 @@ llm-wiki context "auth architecture" --workspace /path/to/project
|
|
|
166
166
|
llm-wiki context "auth architecture" --workspace /path/to/project --json
|
|
167
167
|
llm-wiki context "auth architecture" --workspace /path/to/project --limit 8 --no-expand
|
|
168
168
|
llm-wiki context "auth architecture" --workspace /path/to/project --include-episodic
|
|
169
|
+
llm-wiki context "auth architecture" --workspace /path/to/project --include-archived
|
|
169
170
|
```
|
|
170
171
|
|
|
171
172
|
`llm-wiki lint` checks Markdown wiki health without modifying files:
|
|
@@ -179,6 +180,10 @@ llm-wiki context "auth architecture" --workspace /path/to/project --include-epis
|
|
|
179
180
|
- secret-like content patterns such as tokens, password assignments, bearer credentials, and private keys
|
|
180
181
|
- duplicate aliases or titles
|
|
181
182
|
- stale pages and orphan candidates
|
|
183
|
+
- `memory.md` near-budget or oversized state
|
|
184
|
+
- wiki page count near the default search cap
|
|
185
|
+
- default-hidden episodic/context/session page growth
|
|
186
|
+
- stale/archived pages without supersession metadata or links
|
|
182
187
|
- outdated managed rules/templates from earlier `llm-wiki-kit` versions
|
|
183
188
|
- stale or oversized maintenance queues
|
|
184
189
|
|
|
@@ -188,7 +193,10 @@ Broken links, invalid source IDs, and secret-like content are errors and return
|
|
|
188
193
|
|
|
189
194
|
- refreshes the generated block inside `wiki/memory.md`
|
|
190
195
|
- refreshes the generated block inside `wiki/index.md`
|
|
196
|
+
- keeps only durable non-archived pages in generated maps
|
|
197
|
+
- skips `stale`, `archived`, and `superseded_by` pages from generated maps while preserving them on disk
|
|
191
198
|
- excludes default `query`, `context`, and `session-log` pages from generated maps unless they are explicitly durable (`memory_type: semantic` or `procedural`, `importance >= 4`)
|
|
199
|
+
- reports indexed durable pages, hidden episodic pages, and archived/stale/superseded skipped counts, including in `--dry-run`
|
|
192
200
|
- skips malformed generated marker blocks instead of overwriting them
|
|
193
201
|
- preserves handwritten content outside marker blocks
|
|
194
202
|
- appends a log entry when files change
|
|
@@ -196,7 +204,18 @@ Broken links, invalid source IDs, and secret-like content are errors and return
|
|
|
196
204
|
|
|
197
205
|
Agents may run `consolidate` after meaningful wiki growth. Users should not need to run it after every turn.
|
|
198
206
|
|
|
199
|
-
`llm-wiki maintenance --workspace <project>` prints queue counts and the first pending items. It does not merge wiki pages by itself; the active agent should review pending items, update the closest existing durable wiki document, then mark the queue item `done` or `skipped`.
|
|
207
|
+
`llm-wiki maintenance --workspace <project>` prints queue counts, review due status, and the first pending items. It does not merge wiki pages by itself; the active agent should review pending items, update the closest existing durable wiki document, then mark the queue item `done` or `skipped`. Periodic maintenance is an agent-side task, not something users need to run after every turn.
|
|
208
|
+
|
|
209
|
+
`llm-wiki maintenance --workspace <project> --json` includes `reviewDue`, `reviewReasons`, `pendingCount`, `stalePendingCount`, `health`, and `recommendedCommands`. Review is due when the last review is older than 14 days, pending queue size reaches 5, stale or result-missing pending items exist, lint has warnings/errors, `memory.md` is near budget, or wiki page count reaches 80% of the search cap. Hook reminders are soft: `SessionStart`/`InstructionsLoaded` may show a short due note, while `UserPromptSubmit` shows it only for wiki/maintenance/cleanup-related prompts. The reminder never blocks the current answer.
|
|
210
|
+
|
|
211
|
+
Recommended agent checklist:
|
|
212
|
+
|
|
213
|
+
```bash
|
|
214
|
+
llm-wiki lint --workspace /path/to/project
|
|
215
|
+
llm-wiki maintenance --workspace /path/to/project
|
|
216
|
+
llm-wiki consolidate --workspace /path/to/project --dry-run
|
|
217
|
+
llm-wiki consolidate --workspace /path/to/project
|
|
218
|
+
```
|
|
200
219
|
|
|
201
220
|
When a new runtime sees an older project, `SessionStart`/`InstructionsLoaded` automatically reapplies safe managed template updates. Files that are clearly generated by older kit versions are refreshed. Files that look user-edited are preserved and surfaced to the active agent as cleanup context instead of being overwritten. Set `LLM_WIKI_KIT_AUTO_PROJECT_UPDATE=0` only while diagnosing automatic template refresh behavior.
|
|
202
221
|
|
package/package.json
CHANGED
package/src/cli.js
CHANGED
|
@@ -70,6 +70,8 @@ function parseOptions(args) {
|
|
|
70
70
|
options.expand = false;
|
|
71
71
|
} else if (arg === '--include-episodic') {
|
|
72
72
|
options.includeEpisodic = true;
|
|
73
|
+
} else if (arg === '--include-archived') {
|
|
74
|
+
options.includeArchived = true;
|
|
73
75
|
} else if (arg === '--replace-hooks') {
|
|
74
76
|
options.replaceHooks = true;
|
|
75
77
|
} else if (arg === '--all') {
|
|
@@ -116,7 +118,7 @@ Usage:
|
|
|
116
118
|
llm-wiki hook claude <EventName>
|
|
117
119
|
llm-wiki bootstrap --workspace <project>
|
|
118
120
|
llm-wiki migrate --workspace <project>
|
|
119
|
-
llm-wiki context "<query>" --workspace <project> [--limit 5] [--no-expand] [--include-episodic]
|
|
121
|
+
llm-wiki context "<query>" --workspace <project> [--limit 5] [--no-expand] [--include-episodic] [--include-archived]
|
|
120
122
|
llm-wiki lint --workspace <project>
|
|
121
123
|
llm-wiki consolidate --workspace <project> [--dry-run]
|
|
122
124
|
llm-wiki maintenance --workspace <project> [--json]
|
|
@@ -234,7 +236,7 @@ Usage:
|
|
|
234
236
|
|
|
235
237
|
if (command === 'maintenance') {
|
|
236
238
|
const projectRoot = resolve(options.workspace || process.cwd());
|
|
237
|
-
printJsonOrText(await maintenanceSummary(projectRoot, options), options, formatMaintenanceResult);
|
|
239
|
+
printJsonOrText(await maintenanceSummary(projectRoot, { ...options, includeLint: true }), options, formatMaintenanceResult);
|
|
238
240
|
return;
|
|
239
241
|
}
|
|
240
242
|
|
package/src/consolidate.js
CHANGED
|
@@ -5,6 +5,14 @@ import { hasSecretLikeText, isSensitivePath, normalizeForStorage } from './redac
|
|
|
5
5
|
import { indexPage, memoryPage } from './templates.js';
|
|
6
6
|
import { collectWikiPages } from './wiki-model.js';
|
|
7
7
|
import { runLint } from './wiki-lint.js';
|
|
8
|
+
import {
|
|
9
|
+
isArchivedPage,
|
|
10
|
+
isEpisodicLayerPage,
|
|
11
|
+
isPromotedDurablePage,
|
|
12
|
+
isStalePage,
|
|
13
|
+
isSupersededPage,
|
|
14
|
+
pageImportance,
|
|
15
|
+
} from './wiki-visibility.js';
|
|
8
16
|
|
|
9
17
|
export const MEMORY_START = '<!-- llm-wiki-kit:memory-start -->';
|
|
10
18
|
export const MEMORY_END = '<!-- llm-wiki-kit:memory-end -->';
|
|
@@ -37,37 +45,53 @@ function pageReference(page) {
|
|
|
37
45
|
return `[[${pageTarget(page)}|${page.title}]]`;
|
|
38
46
|
}
|
|
39
47
|
|
|
40
|
-
function importance(page) {
|
|
41
|
-
const value = Number(page.frontmatter.importance || 0);
|
|
42
|
-
return Number.isFinite(value) ? value : 0;
|
|
43
|
-
}
|
|
44
|
-
|
|
45
48
|
function isPromotableEpisodicPage(page) {
|
|
46
|
-
return
|
|
49
|
+
return isPromotedDurablePage(page);
|
|
47
50
|
}
|
|
48
51
|
|
|
49
|
-
function
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
) {
|
|
61
|
-
|
|
52
|
+
function classifyGeneratedPages(pages) {
|
|
53
|
+
const stats = {
|
|
54
|
+
durablePages: 0,
|
|
55
|
+
memoryPages: 0,
|
|
56
|
+
hiddenEpisodicPages: 0,
|
|
57
|
+
archivedSkippedPages: 0,
|
|
58
|
+
staleSkippedPages: 0,
|
|
59
|
+
supersededSkippedPages: 0,
|
|
60
|
+
sensitiveSkippedPages: 0,
|
|
61
|
+
};
|
|
62
|
+
const candidates = [];
|
|
63
|
+
for (const page of pages) {
|
|
64
|
+
if (['wiki/index.md', 'wiki/log.md', 'wiki/memory.md'].includes(page.rel)) continue;
|
|
65
|
+
if (isSensitivePath(page.rel) || isSensitivePath(page.title) || hasSecretLikeText(`${page.rel}\n${page.title}\n${page.content}`)) {
|
|
66
|
+
stats.sensitiveSkippedPages += 1;
|
|
67
|
+
continue;
|
|
68
|
+
}
|
|
69
|
+
if (isArchivedPage(page)) {
|
|
70
|
+
stats.archivedSkippedPages += 1;
|
|
71
|
+
continue;
|
|
72
|
+
}
|
|
73
|
+
if (isSupersededPage(page)) {
|
|
74
|
+
stats.supersededSkippedPages += 1;
|
|
75
|
+
continue;
|
|
76
|
+
}
|
|
77
|
+
if (isStalePage(page)) {
|
|
78
|
+
stats.staleSkippedPages += 1;
|
|
79
|
+
continue;
|
|
80
|
+
}
|
|
81
|
+
if (isEpisodicLayerPage(page) && !isPromotableEpisodicPage(page)) {
|
|
82
|
+
stats.hiddenEpisodicPages += 1;
|
|
83
|
+
continue;
|
|
84
|
+
}
|
|
85
|
+
candidates.push(page);
|
|
62
86
|
}
|
|
63
|
-
|
|
87
|
+
stats.durablePages = candidates.length;
|
|
88
|
+
return { candidates, stats };
|
|
64
89
|
}
|
|
65
90
|
|
|
66
91
|
function sortedPages(pages) {
|
|
67
92
|
return pages
|
|
68
|
-
.filter(isGeneratedBlockCandidate)
|
|
69
93
|
.sort((a, b) => {
|
|
70
|
-
const importanceDiff =
|
|
94
|
+
const importanceDiff = pageImportance(b) - pageImportance(a);
|
|
71
95
|
if (importanceDiff !== 0) return importanceDiff;
|
|
72
96
|
const typeDiff = String(a.type || '').localeCompare(String(b.type || ''));
|
|
73
97
|
if (typeDiff !== 0) return typeDiff;
|
|
@@ -75,12 +99,16 @@ function sortedPages(pages) {
|
|
|
75
99
|
});
|
|
76
100
|
}
|
|
77
101
|
|
|
78
|
-
function buildGeneratedMemoryBlock(pages) {
|
|
79
|
-
const
|
|
102
|
+
function buildGeneratedMemoryBlock(pages, stats) {
|
|
103
|
+
const allCandidates = sortedPages(pages);
|
|
104
|
+
const candidates = allCandidates.slice(0, 25);
|
|
105
|
+
stats.memoryPages = candidates.length;
|
|
80
106
|
const lines = [
|
|
81
107
|
'## Generated Memory Map',
|
|
82
108
|
'',
|
|
83
|
-
`-
|
|
109
|
+
`- Durable pages indexed: ${allCandidates.length}`,
|
|
110
|
+
`- Hidden episodic pages: ${stats.hiddenEpisodicPages}`,
|
|
111
|
+
`- Skipped archived/stale/superseded pages: ${stats.archivedSkippedPages}/${stats.staleSkippedPages}/${stats.supersededSkippedPages}`,
|
|
84
112
|
];
|
|
85
113
|
if (candidates.length === 0) {
|
|
86
114
|
lines.push('- No durable pages found yet.');
|
|
@@ -91,7 +119,7 @@ function buildGeneratedMemoryBlock(pages) {
|
|
|
91
119
|
const metadata = [
|
|
92
120
|
page.type || 'unknown',
|
|
93
121
|
page.memoryType ? `memory:${page.memoryType}` : '',
|
|
94
|
-
|
|
122
|
+
pageImportance(page) > 0 ? `importance:${pageImportance(page)}` : '',
|
|
95
123
|
page.confidence ? `confidence:${page.confidence}` : '',
|
|
96
124
|
].filter(Boolean).join(', ');
|
|
97
125
|
lines.push(`- ${pageReference(page)}${metadata ? ` - ${metadata}` : ''}`);
|
|
@@ -111,7 +139,7 @@ function buildGeneratedIndexBlock(pages) {
|
|
|
111
139
|
const lines = [
|
|
112
140
|
'## Generated Page Map',
|
|
113
141
|
'',
|
|
114
|
-
`-
|
|
142
|
+
`- Durable pages indexed: ${candidates.length}`,
|
|
115
143
|
];
|
|
116
144
|
if (groups.size === 0) {
|
|
117
145
|
lines.push('- No durable pages found yet.');
|
|
@@ -142,6 +170,7 @@ async function updateMarkedFile(projectRoot, rel, initialContent, startMarker, e
|
|
|
142
170
|
export async function runConsolidate(projectRoot, options = {}) {
|
|
143
171
|
const lintResult = await runLint(projectRoot, { maxFiles: options.maxFiles || 1000 });
|
|
144
172
|
const pages = await collectWikiPages(projectRoot, { maxFiles: options.maxFiles || 1000 });
|
|
173
|
+
const { candidates, stats } = classifyGeneratedPages(pages);
|
|
145
174
|
const changed = [];
|
|
146
175
|
const skipped = [];
|
|
147
176
|
|
|
@@ -151,7 +180,7 @@ export async function runConsolidate(projectRoot, options = {}) {
|
|
|
151
180
|
memoryPage(),
|
|
152
181
|
MEMORY_START,
|
|
153
182
|
MEMORY_END,
|
|
154
|
-
buildGeneratedMemoryBlock(
|
|
183
|
+
buildGeneratedMemoryBlock(candidates, stats),
|
|
155
184
|
options,
|
|
156
185
|
);
|
|
157
186
|
if (memoryChange?.changed) changed.push(memoryChange.changed);
|
|
@@ -163,7 +192,7 @@ export async function runConsolidate(projectRoot, options = {}) {
|
|
|
163
192
|
indexPage(),
|
|
164
193
|
INDEX_START,
|
|
165
194
|
INDEX_END,
|
|
166
|
-
buildGeneratedIndexBlock(
|
|
195
|
+
buildGeneratedIndexBlock(candidates),
|
|
167
196
|
options,
|
|
168
197
|
);
|
|
169
198
|
if (indexChange?.changed) changed.push(indexChange.changed);
|
|
@@ -182,6 +211,7 @@ export async function runConsolidate(projectRoot, options = {}) {
|
|
|
182
211
|
dryRun: Boolean(options.dryRun),
|
|
183
212
|
changed,
|
|
184
213
|
skipped,
|
|
214
|
+
counts: stats,
|
|
185
215
|
lint: {
|
|
186
216
|
ok: finalLintResult.ok,
|
|
187
217
|
errorCount: finalLintResult.errorCount,
|
|
@@ -197,6 +227,9 @@ export function formatConsolidateResult(result) {
|
|
|
197
227
|
`- dry run: ${result.dryRun ? 'yes' : 'no'}`,
|
|
198
228
|
`- changed: ${result.changed.length ? result.changed.map((item) => item.path).join(', ') : 'none'}`,
|
|
199
229
|
`- skipped: ${result.skipped?.length ? result.skipped.map((item) => `${item.path} (${item.reason})`).join(', ') : 'none'}`,
|
|
230
|
+
`- durable pages indexed: ${result.counts?.durablePages ?? 0}`,
|
|
231
|
+
`- hidden episodic pages: ${result.counts?.hiddenEpisodicPages ?? 0}`,
|
|
232
|
+
`- skipped archived/stale/superseded: ${result.counts?.archivedSkippedPages ?? 0}/${result.counts?.staleSkippedPages ?? 0}/${result.counts?.supersededSkippedPages ?? 0}`,
|
|
200
233
|
`- lint errors: ${result.lint.errorCount}`,
|
|
201
234
|
`- lint warnings: ${result.lint.warningCount}`,
|
|
202
235
|
].join('\n');
|
package/src/maintenance.js
CHANGED
|
@@ -4,11 +4,15 @@ import { appendText, exists, kitDataDir, readJson, readText, sha256, writeTextIf
|
|
|
4
4
|
import { classifyTurn, isMaintenanceRelatedQuery } from './capture-policy.js';
|
|
5
5
|
import { redactText, summarizeForStorage } from './redaction.js';
|
|
6
6
|
import { buildEntryFromTurnState, hasRecoverableTurnState } from './state.js';
|
|
7
|
+
import { collectWikiPages, DEFAULT_MAX_WIKI_FILES, MEMORY_BYTE_LIMIT } from './wiki-model.js';
|
|
7
8
|
|
|
8
9
|
export const MAINTENANCE_QUEUE_REL = 'llm-wiki/outputs/maintenance/queue.md';
|
|
9
10
|
const DEFAULT_STALE_TURN_MS = 10 * 60 * 1000;
|
|
10
11
|
const DEFAULT_STALE_PENDING_DAYS = 7;
|
|
11
12
|
const DEFAULT_PENDING_LIMIT = 20;
|
|
13
|
+
const DEFAULT_REVIEW_PENDING_LIMIT = 5;
|
|
14
|
+
const DEFAULT_REVIEW_INTERVAL_DAYS = 14;
|
|
15
|
+
const MEMORY_NEAR_BUDGET_BYTES = 20 * 1024;
|
|
12
16
|
|
|
13
17
|
function queuePath(projectRoot) {
|
|
14
18
|
return join(projectRoot, MAINTENANCE_QUEUE_REL);
|
|
@@ -125,12 +129,26 @@ export async function readMaintenanceQueue(projectRoot) {
|
|
|
125
129
|
export function summarizeMaintenanceQueue(queue, options = {}) {
|
|
126
130
|
const staleDays = options.staleDays ?? DEFAULT_STALE_PENDING_DAYS;
|
|
127
131
|
const pendingLimit = options.pendingLimit ?? DEFAULT_PENDING_LIMIT;
|
|
132
|
+
const reviewPendingLimit = options.reviewPendingLimit ?? DEFAULT_REVIEW_PENDING_LIMIT;
|
|
133
|
+
const reviewIntervalDays = options.reviewIntervalDays ?? DEFAULT_REVIEW_INTERVAL_DAYS;
|
|
128
134
|
const pending = queue.items.filter((item) => item.status === 'pending');
|
|
129
135
|
const staleCutoff = Date.now() - staleDays * 24 * 60 * 60 * 1000;
|
|
130
136
|
const stalePending = pending.filter((item) => {
|
|
131
137
|
const time = Date.parse(item.created_at || item.last_seen_at || '');
|
|
132
138
|
return Number.isFinite(time) && time < staleCutoff;
|
|
133
139
|
});
|
|
140
|
+
const reviewItems = queue.items.filter((item) => item.status === 'done' || item.status === 'skipped');
|
|
141
|
+
const reviewTimes = reviewItems
|
|
142
|
+
.map((item) => Date.parse(item.last_seen_at || item.created_at || ''))
|
|
143
|
+
.filter(Number.isFinite);
|
|
144
|
+
const lastReviewMs = reviewTimes.length > 0 ? Math.max(...reviewTimes) : null;
|
|
145
|
+
const reviewReasons = [];
|
|
146
|
+
if (pending.length >= reviewPendingLimit) reviewReasons.push(`pending queue has ${pending.length} items (threshold ${reviewPendingLimit})`);
|
|
147
|
+
if (stalePending.length > 0) reviewReasons.push(`${stalePending.length} pending item(s) older than ${staleDays} days`);
|
|
148
|
+
if (pending.some((item) => item.result_missing)) reviewReasons.push('pending recovered turn state needs review');
|
|
149
|
+
if (lastReviewMs && Date.now() - lastReviewMs >= reviewIntervalDays * 24 * 60 * 60 * 1000) {
|
|
150
|
+
reviewReasons.push(`last maintenance review is older than ${reviewIntervalDays} days`);
|
|
151
|
+
}
|
|
134
152
|
return {
|
|
135
153
|
path: queue.path,
|
|
136
154
|
exists: queue.exists,
|
|
@@ -142,14 +160,79 @@ export function summarizeMaintenanceQueue(queue, options = {}) {
|
|
|
142
160
|
doneCount: queue.items.filter((item) => item.status === 'done').length,
|
|
143
161
|
skippedCount: queue.items.filter((item) => item.status === 'skipped').length,
|
|
144
162
|
stalePending,
|
|
163
|
+
stalePendingCount: stalePending.length,
|
|
145
164
|
stalePendingDays: staleDays,
|
|
146
165
|
pendingLimit,
|
|
166
|
+
reviewPendingLimit,
|
|
167
|
+
reviewIntervalDays,
|
|
168
|
+
lastReviewAt: lastReviewMs ? new Date(lastReviewMs).toISOString() : null,
|
|
169
|
+
reviewDue: reviewReasons.length > 0,
|
|
170
|
+
reviewReasons,
|
|
147
171
|
tooManyPending: pending.length > pendingLimit,
|
|
148
172
|
};
|
|
149
173
|
}
|
|
150
174
|
|
|
151
175
|
export async function maintenanceSummary(projectRoot, options = {}) {
|
|
152
|
-
|
|
176
|
+
const summary = summarizeMaintenanceQueue(await readMaintenanceQueue(projectRoot), options);
|
|
177
|
+
const health = await maintenanceHealthSignals(projectRoot, options);
|
|
178
|
+
const reviewReasons = [...summary.reviewReasons];
|
|
179
|
+
if (health.memoryNearBudget) reviewReasons.push(`memory.md is near budget (${health.memoryBytes} bytes)`);
|
|
180
|
+
if (health.pageCountNearSearchCap) reviewReasons.push(`wiki page count ${health.pageCount} is near search cap ${health.searchCap}`);
|
|
181
|
+
if (health.lint && health.lint.issueCount > 0) {
|
|
182
|
+
reviewReasons.push(`lint has ${health.lint.errorCount} error(s) and ${health.lint.warningCount} warning(s)`);
|
|
183
|
+
}
|
|
184
|
+
return {
|
|
185
|
+
...summary,
|
|
186
|
+
reviewDue: reviewReasons.length > 0,
|
|
187
|
+
reviewReasons,
|
|
188
|
+
health,
|
|
189
|
+
recommendedCommands: recommendedMaintenanceCommands(projectRoot),
|
|
190
|
+
};
|
|
191
|
+
}
|
|
192
|
+
|
|
193
|
+
function recommendedMaintenanceCommands(projectRoot) {
|
|
194
|
+
return [
|
|
195
|
+
`llm-wiki lint --workspace ${projectRoot}`,
|
|
196
|
+
`llm-wiki maintenance --workspace ${projectRoot}`,
|
|
197
|
+
`llm-wiki consolidate --workspace ${projectRoot} --dry-run`,
|
|
198
|
+
`llm-wiki consolidate --workspace ${projectRoot}`,
|
|
199
|
+
];
|
|
200
|
+
}
|
|
201
|
+
|
|
202
|
+
async function maintenanceHealthSignals(projectRoot, options = {}) {
|
|
203
|
+
const memoryText = await readText(join(projectRoot, 'llm-wiki', 'wiki', 'memory.md'), '');
|
|
204
|
+
const memoryBytes = Buffer.byteLength(memoryText, 'utf8');
|
|
205
|
+
let pageCount = 0;
|
|
206
|
+
const searchCap = options.searchCap || DEFAULT_MAX_WIKI_FILES;
|
|
207
|
+
try {
|
|
208
|
+
pageCount = (await collectWikiPages(projectRoot, { maxFiles: searchCap })).length;
|
|
209
|
+
} catch {
|
|
210
|
+
pageCount = 0;
|
|
211
|
+
}
|
|
212
|
+
let lint = null;
|
|
213
|
+
if (options.includeLint && !options.skipLint) {
|
|
214
|
+
try {
|
|
215
|
+
const module = await import('./wiki-lint.js');
|
|
216
|
+
const result = await module.runLint(projectRoot, { maxFiles: options.maxFiles || 1000, skipMaintenance: true });
|
|
217
|
+
lint = {
|
|
218
|
+
ok: result.ok,
|
|
219
|
+
issueCount: result.issueCount,
|
|
220
|
+
errorCount: result.errorCount,
|
|
221
|
+
warningCount: result.warningCount,
|
|
222
|
+
};
|
|
223
|
+
} catch {
|
|
224
|
+
lint = null;
|
|
225
|
+
}
|
|
226
|
+
}
|
|
227
|
+
return {
|
|
228
|
+
memoryBytes,
|
|
229
|
+
memoryNearBudget: memoryBytes >= MEMORY_NEAR_BUDGET_BYTES,
|
|
230
|
+
memoryOversized: memoryBytes > MEMORY_BYTE_LIMIT,
|
|
231
|
+
pageCount,
|
|
232
|
+
searchCap,
|
|
233
|
+
pageCountNearSearchCap: pageCount >= Math.floor(searchCap * 0.8),
|
|
234
|
+
lint,
|
|
235
|
+
};
|
|
153
236
|
}
|
|
154
237
|
|
|
155
238
|
export async function appendMaintenanceItem(projectRoot, item) {
|
|
@@ -225,11 +308,11 @@ export async function recoverStaleTurnStates(projectRoot, options = {}) {
|
|
|
225
308
|
}
|
|
226
309
|
|
|
227
310
|
export function formatMaintenanceContext(summary, options = {}) {
|
|
311
|
+
if (!summary.reviewDue) return '';
|
|
228
312
|
const eventName = options.eventName || '';
|
|
229
313
|
const defaultLimit = eventName === 'SessionStart' || eventName === 'InstructionsLoaded' ? 1 : 5;
|
|
230
314
|
const limit = options.limit || defaultLimit;
|
|
231
315
|
let pending = summary.pending.slice(0, limit);
|
|
232
|
-
if (pending.length === 0) return '';
|
|
233
316
|
|
|
234
317
|
if (eventName === 'UserPromptSubmit') {
|
|
235
318
|
if (!isMaintenanceRelatedQuery(options.query || '', summary.pending)) return '';
|
|
@@ -242,6 +325,7 @@ export function formatMaintenanceContext(summary, options = {}) {
|
|
|
242
325
|
eventName === 'UserPromptSubmit'
|
|
243
326
|
? 'LLM Wiki maintenance status:'
|
|
244
327
|
: 'LLM Wiki maintenance status:',
|
|
328
|
+
`- review due: yes (${(summary.reviewReasons || []).slice(0, 2).join('; ') || 'periodic review threshold met'}).`,
|
|
245
329
|
`- pending review items: ${summary.pendingCount}. 현재 요청이 우선이며, 관련 있을 때만 durable wiki 정리에 사용한다.`,
|
|
246
330
|
];
|
|
247
331
|
for (const item of pending) {
|
|
@@ -258,14 +342,25 @@ export function formatMaintenanceResult(summary) {
|
|
|
258
342
|
'llm-wiki maintenance',
|
|
259
343
|
`- queue: ${summary.path}`,
|
|
260
344
|
`- pending: ${summary.pendingCount}`,
|
|
345
|
+
`- stale pending: ${summary.stalePendingCount || 0}`,
|
|
261
346
|
`- done: ${summary.doneCount}`,
|
|
262
347
|
`- skipped: ${summary.skippedCount}`,
|
|
348
|
+
`- review due: ${summary.reviewDue ? 'yes' : 'no'}`,
|
|
263
349
|
];
|
|
350
|
+
if ((summary.reviewReasons || []).length > 0) {
|
|
351
|
+
lines.push(`- review reasons: ${summary.reviewReasons.join('; ')}`);
|
|
352
|
+
}
|
|
264
353
|
if (summary.pending.length > 0) {
|
|
265
354
|
lines.push('', 'Pending:');
|
|
266
355
|
for (const item of summary.pending.slice(0, 10)) {
|
|
267
356
|
lines.push(`- ${item.topic || item.id}: ${item.source} -> ${item.suggested_target}`);
|
|
268
357
|
}
|
|
269
358
|
}
|
|
359
|
+
if ((summary.recommendedCommands || []).length > 0) {
|
|
360
|
+
lines.push('', 'Recommended commands:');
|
|
361
|
+
for (const command of summary.recommendedCommands) {
|
|
362
|
+
lines.push(`- ${command}`);
|
|
363
|
+
}
|
|
364
|
+
}
|
|
270
365
|
return lines.join('\n');
|
|
271
366
|
}
|
package/src/templates.js
CHANGED
|
@@ -15,7 +15,7 @@ This repository uses llm-wiki-kit as a hook-first living Markdown wiki for Codex
|
|
|
15
15
|
- \`llm-wiki/wiki/memory.md\`는 짧은 핵심 기억이다. 긴 설명 대신 현재 상태와 중요한 문서 링크만 유지한다.
|
|
16
16
|
- hook이 주입한 context를 참고하되, 현재 사용자 답변을 먼저 처리한다. 수동 확인이나 정리에는 \`llm-wiki context\`, \`llm-wiki lint\`, \`llm-wiki consolidate\`를 agent 보조 도구로 사용한다.
|
|
17
17
|
- hook은 redacted raw envelope와 필요한 live Q&A를 안전하게 남긴다. \`wiki/queries\`/\`wiki/decisions\` 자동 승격은 기본값이 아니며, durable 지식은 중요도와 동의 흐름에 따라 agent가 기존 정식 wiki 문서에 합친다.
|
|
18
|
-
- hook은 종료/시작 경계에서 정말 필요한 정리 후보를 \`llm-wiki/outputs/maintenance/queue.md\`에 남길 수 있다. pending 항목은 현재 응답을 지연시키지 않는 범위에서 agent가 병합하거나 done/skipped로 표시한다.
|
|
18
|
+
- hook은 종료/시작 경계에서 정말 필요한 정리 후보를 \`llm-wiki/outputs/maintenance/queue.md\`에 남길 수 있다. 정기 maintenance는 agent-side soft reminder이며, pending 항목은 현재 응답을 지연시키지 않는 범위에서 agent가 병합하거나 done/skipped로 표시한다.
|
|
19
19
|
- 새 문서를 만들기 전에 기존 wiki 문서를 먼저 찾아 갱신한다. 반복해서 쓸 사실은 \`outputs/questions/\`에만 두지 말고 적절한 wiki 문서에 합친다.
|
|
20
20
|
- 일회성 작업 기록은 필요할 때 \`llm-wiki/outputs/questions/\`에 보존하고, 재사용 가능한 사실/지식은 승인된 경우 \`wiki/architecture/\`, \`wiki/debugging/\`, \`wiki/decisions/\`, \`wiki/concepts/\`, \`procedures/\`에 반영한다.
|
|
21
21
|
- 검증 명령, 근거 파일, 불확실한 점을 함께 남긴다. 추론은 추론이라고 표시하고, 모순은 지우지 말고 Open Questions 또는 Contradictions에 남긴다.
|
|
@@ -49,7 +49,8 @@ Codex와 Claude Code를 평소처럼 사용하는 동안 living Markdown LLM Wik
|
|
|
49
49
|
- 오래 남길 내용이 생기면 새 문서부터 만들지 말고 기존 \`wiki/\` 문서를 먼저 찾아 갱신한다.
|
|
50
50
|
- 단순 답변, 상태 확인, 일회성 대화는 과도하게 \`wiki/queries\`나 maintenance로 승격하지 않는다.
|
|
51
51
|
- 반복해서 쓸 지식은 중요도와 동의 흐름에 따라 \`wiki/architecture/\`, \`wiki/debugging/\`, \`wiki/decisions/\`, \`wiki/concepts/\`, \`procedures/\`에 합친다.
|
|
52
|
-
- hook이 만든 \`outputs/maintenance/queue.md\` pending 항목은 현재 요청과 관련 있을 때 확인하고, 기존 정식 wiki 문서에 병합한 뒤 done 또는 skipped로 표시한다.
|
|
52
|
+
- hook이 만든 \`outputs/maintenance/queue.md\` pending 항목은 현재 요청과 관련 있거나 review due 안내가 있을 때 확인하고, 기존 정식 wiki 문서에 병합한 뒤 done 또는 skipped로 표시한다.
|
|
53
|
+
- 정기 maintenance는 자동 수정이 아니라 agent review다. \`SessionStart\`/\`InstructionsLoaded\`에서만 짧게 안내되고, \`UserPromptSubmit\`에서는 사용자가 wiki/maintenance/정리 관련 질문을 한 경우에만 안내된다.
|
|
53
54
|
- \`wiki/memory.md\`는 짧게 유지한다. 긴 설명 대신 현재 상태와 중요한 문서 링크를 둔다.
|
|
54
55
|
- 모순은 덮어쓰지 말고 \`Contradictions\` 또는 \`Open Questions\`에 보존한다.
|
|
55
56
|
- 인증값, token, password, private key, \`.env\` 원문은 wiki에 저장하지 않는다.
|
|
@@ -182,7 +183,7 @@ export function procedure(name) {
|
|
|
182
183
|
3. 관련 \`wiki/\` 문서를 최소한으로 읽는다.
|
|
183
184
|
4. 정확한 근거가 필요할 때만 raw source를 확인한다.
|
|
184
185
|
5. 검증된 사실과 추론을 분리한다.
|
|
185
|
-
6. 수동 확인이 필요할 때만 \`llm-wiki context "<query>"\`를 쓴다. 과거 episodic query까지 봐야 할 때는 \`--include-episodic\`을 붙인다.
|
|
186
|
+
6. 수동 확인이 필요할 때만 \`llm-wiki context "<query>"\`를 쓴다. 과거 episodic query/context까지 봐야 할 때는 \`--include-episodic\`을 붙이고, archived/superseded page까지 봐야 할 때만 \`--include-archived\`를 붙인다.
|
|
186
187
|
7. 일회성 답변은 durable wiki로 승격하지 않는다. 필요한 작업 turn만 \`outputs/questions/\`에 남긴다.
|
|
187
188
|
8. 반복해서 쓸 사실/지식은 중요도와 동의 흐름에 따라 \`wiki/architecture/\`, \`wiki/debugging/\`, \`wiki/decisions/\`, \`wiki/concepts/\`, \`procedures/\`에 합친다.
|
|
188
189
|
`,
|
|
@@ -190,7 +191,15 @@ export function procedure(name) {
|
|
|
190
191
|
|
|
191
192
|
\`llm-wiki lint --workspace <project>\`는 agent가 필요할 때 쓰는 wiki 건강 점검 도구다. 사용자가 매번 실행해야 하는 명령이 아니다.
|
|
192
193
|
|
|
193
|
-
점검 대상: stale page, orphan page, broken wiki/Markdown link, unsafe source id, secret-like content, missing source, duplicate concept/title, unsupported claim, unresolved contradiction, outdated managed rule.
|
|
194
|
+
점검 대상: stale page, orphan page, broken wiki/Markdown link, unsafe source id, secret-like content, missing source, duplicate concept/title, unsupported claim, unresolved contradiction, outdated managed rule, memory budget, page count growth, hidden episodic/context growth, stale/archived discoverability, maintenance review due.
|
|
195
|
+
|
|
196
|
+
정기 maintenance는 사용자가 매 turn 실행할 필요가 없는 agent-side task다. 필요할 때 agent가 다음 순서로 확인한다:
|
|
197
|
+
|
|
198
|
+
1. \`llm-wiki lint --workspace <project>\`
|
|
199
|
+
2. \`llm-wiki maintenance --workspace <project>\`
|
|
200
|
+
3. pending item을 기존 durable page에 병합한 뒤 \`done\` 또는 \`skipped\`로 표시한다.
|
|
201
|
+
4. \`llm-wiki consolidate --workspace <project> --dry-run\`
|
|
202
|
+
5. 필요할 때 \`llm-wiki consolidate --workspace <project>\`
|
|
194
203
|
|
|
195
204
|
자동 수정은 확실히 kit가 관리하는 영역에만 적용한다. 사용자 편집 가능성이 있는 문서는 덮어쓰지 말고 다음 작업 context에 정리 필요성을 올린다.
|
|
196
205
|
`,
|
package/src/wiki-lint.js
CHANGED
|
@@ -8,10 +8,19 @@ import {
|
|
|
8
8
|
buildAliasMap,
|
|
9
9
|
buildWikiGraph,
|
|
10
10
|
collectWikiPages,
|
|
11
|
+
DEFAULT_MAX_WIKI_FILES,
|
|
12
|
+
MEMORY_BYTE_LIMIT,
|
|
11
13
|
normalizeTarget,
|
|
12
14
|
resolveWikiLink,
|
|
13
15
|
wikiRoot,
|
|
14
16
|
} from './wiki-model.js';
|
|
17
|
+
import {
|
|
18
|
+
hasSupersession,
|
|
19
|
+
isArchivedPage,
|
|
20
|
+
isEpisodicLayerPage,
|
|
21
|
+
isPromotedDurablePage,
|
|
22
|
+
isStalePage,
|
|
23
|
+
} from './wiki-visibility.js';
|
|
15
24
|
|
|
16
25
|
const VALID_TYPES = new Set([
|
|
17
26
|
'source',
|
|
@@ -29,6 +38,9 @@ const VALID_STATUS = new Set(['draft', 'reviewed', 'stale', 'archived']);
|
|
|
29
38
|
const VALID_CONFIDENCE = new Set(['high', 'medium', 'low']);
|
|
30
39
|
const VALID_MEMORY_TYPES = new Set(['semantic', 'episodic', 'procedural']);
|
|
31
40
|
const CORE_PAGES = new Set(['wiki/index.md', 'wiki/log.md', 'wiki/memory.md']);
|
|
41
|
+
const MEMORY_NEAR_BUDGET_BYTES = 20 * 1024;
|
|
42
|
+
const HIDDEN_PAGE_WARNING_THRESHOLD = 50;
|
|
43
|
+
const SEARCH_CAP_WARNING_RATIO = 0.8;
|
|
32
44
|
|
|
33
45
|
function issue(severity, code, path, message) {
|
|
34
46
|
return { severity, code, path, message };
|
|
@@ -125,8 +137,9 @@ export async function runLint(projectRoot, options = {}) {
|
|
|
125
137
|
return lintResult(projectRoot, [], issues);
|
|
126
138
|
}
|
|
127
139
|
|
|
140
|
+
const maxFiles = options.maxFiles || 1000;
|
|
128
141
|
const pages = await collectWikiPages(projectRoot, {
|
|
129
|
-
maxFiles
|
|
142
|
+
maxFiles,
|
|
130
143
|
maxChars: options.maxChars || 75000,
|
|
131
144
|
});
|
|
132
145
|
const byRel = new Map(pages.map((page) => [page.rel, page]));
|
|
@@ -226,9 +239,23 @@ export async function runLint(projectRoot, options = {}) {
|
|
|
226
239
|
}
|
|
227
240
|
|
|
228
241
|
const memoryText = await readText(join(projectRoot, 'llm-wiki', 'wiki', 'memory.md'), '');
|
|
229
|
-
|
|
230
|
-
|
|
242
|
+
const memoryBytes = Buffer.byteLength(memoryText, 'utf8');
|
|
243
|
+
if (memoryBytes > MEMORY_BYTE_LIMIT) {
|
|
244
|
+
issues.push(issue('warning', 'memory-too-large', 'wiki/memory.md', `memory.md is oversized (${memoryBytes} bytes) and larger than the hook excerpt budget`));
|
|
245
|
+
} else if (memoryBytes >= MEMORY_NEAR_BUDGET_BYTES) {
|
|
246
|
+
issues.push(issue('warning', 'memory-near-budget', 'wiki/memory.md', `memory.md is near the hook excerpt budget (${memoryBytes} bytes)`));
|
|
247
|
+
}
|
|
248
|
+
|
|
249
|
+
const searchCap = options.maxFiles || DEFAULT_MAX_WIKI_FILES;
|
|
250
|
+
if (pages.length >= Math.floor(searchCap * SEARCH_CAP_WARNING_RATIO)) {
|
|
251
|
+
issues.push(issue('warning', 'wiki-page-count-near-search-cap', 'llm-wiki/wiki', `wiki page count ${pages.length} is near the default search cap ${searchCap}`));
|
|
231
252
|
}
|
|
253
|
+
|
|
254
|
+
const hiddenEpisodicPages = pages.filter((page) => isEpisodicLayerPage(page) && !isPromotedDurablePage(page));
|
|
255
|
+
if (hiddenEpisodicPages.length >= HIDDEN_PAGE_WARNING_THRESHOLD) {
|
|
256
|
+
issues.push(issue('warning', 'hidden-episodic-growth', 'llm-wiki/wiki', `${hiddenEpisodicPages.length} default-hidden episodic/context/session pages may add wiki growth and search noise`));
|
|
257
|
+
}
|
|
258
|
+
|
|
232
259
|
for (const page of pages.filter(isOrphanCandidate)) {
|
|
233
260
|
const backlinks = graph.backlinks.get(page.rel);
|
|
234
261
|
if (!backlinks || backlinks.size === 0) {
|
|
@@ -236,6 +263,15 @@ export async function runLint(projectRoot, options = {}) {
|
|
|
236
263
|
}
|
|
237
264
|
}
|
|
238
265
|
|
|
266
|
+
for (const page of pages) {
|
|
267
|
+
if (!isArchivedPage(page) && !isStalePage(page)) continue;
|
|
268
|
+
const backlinks = graph.backlinks.get(page.rel);
|
|
269
|
+
const outlinks = graph.outlinks.get(page.rel);
|
|
270
|
+
if (!hasSupersession(page) && (!backlinks || backlinks.size === 0) && (!outlinks || outlinks.size === 0)) {
|
|
271
|
+
issues.push(issue('warning', 'stale-archived-discoverability', page.rel, 'stale/archived page has no supersession metadata or wiki links for discoverability'));
|
|
272
|
+
}
|
|
273
|
+
}
|
|
274
|
+
|
|
239
275
|
const projectState = await inspectProjectState(projectRoot).catch(() => null);
|
|
240
276
|
for (const file of projectState?.managedFiles || []) {
|
|
241
277
|
if (file.needsAttention) {
|
|
@@ -245,12 +281,14 @@ export async function runLint(projectRoot, options = {}) {
|
|
|
245
281
|
}
|
|
246
282
|
}
|
|
247
283
|
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
284
|
+
if (!options.skipMaintenance) {
|
|
285
|
+
const maintenance = await maintenanceSummary(projectRoot, { skipLint: true }).catch(() => null);
|
|
286
|
+
if (maintenance?.tooManyPending) {
|
|
287
|
+
issues.push(issue('warning', 'maintenance-too-many-pending', 'outputs/maintenance/queue.md', `maintenance queue has ${maintenance.pendingCount} pending items; review and merge durable items into wiki pages`));
|
|
288
|
+
}
|
|
289
|
+
if ((maintenance?.stalePending || []).length > 0) {
|
|
290
|
+
issues.push(issue('warning', 'maintenance-stale-pending', 'outputs/maintenance/queue.md', `${maintenance.stalePending.length} pending maintenance item(s) are older than ${maintenance.stalePendingDays} days`));
|
|
291
|
+
}
|
|
254
292
|
}
|
|
255
293
|
|
|
256
294
|
return lintResult(projectRoot, pages, issues);
|
package/src/wiki-search.js
CHANGED
|
@@ -4,10 +4,20 @@ import { redactText } from './redaction.js';
|
|
|
4
4
|
import {
|
|
5
5
|
buildWikiGraph,
|
|
6
6
|
collectWikiPages,
|
|
7
|
+
MEMORY_BYTE_LIMIT,
|
|
8
|
+
MEMORY_LINE_LIMIT,
|
|
7
9
|
pageLookup,
|
|
8
10
|
parseFrontmatter,
|
|
9
11
|
readMemoryExcerpt,
|
|
10
12
|
} from './wiki-model.js';
|
|
13
|
+
import {
|
|
14
|
+
isArchivedPage,
|
|
15
|
+
isDefaultSearchCandidate,
|
|
16
|
+
isEpisodicLayerPage,
|
|
17
|
+
isStalePage,
|
|
18
|
+
isSupersededPage,
|
|
19
|
+
searchWeight,
|
|
20
|
+
} from './wiki-visibility.js';
|
|
11
21
|
|
|
12
22
|
const DEFAULT_LIMIT = 5;
|
|
13
23
|
const SNIPPET_CHARS = 350;
|
|
@@ -65,6 +75,7 @@ function resultRecord(page, score, fields = {}) {
|
|
|
65
75
|
title: page.title,
|
|
66
76
|
type: page.type || '',
|
|
67
77
|
memoryType: page.memoryType || '',
|
|
78
|
+
status: page.status || '',
|
|
68
79
|
score,
|
|
69
80
|
directScore: fields.directScore || 0,
|
|
70
81
|
linkScore: fields.linkScore || 0,
|
|
@@ -81,22 +92,35 @@ function sortHits(a, b) {
|
|
|
81
92
|
return a.path.localeCompare(b.path);
|
|
82
93
|
}
|
|
83
94
|
|
|
84
|
-
function
|
|
85
|
-
|
|
86
|
-
return Number.isFinite(value) ? value : 0;
|
|
95
|
+
export async function searchWiki(projectRoot, query, options = {}) {
|
|
96
|
+
return (await performSearch(projectRoot, query, options)).hits;
|
|
87
97
|
}
|
|
88
98
|
|
|
89
|
-
function
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
99
|
+
function searchVisibility(pages, options = {}) {
|
|
100
|
+
const metadata = {
|
|
101
|
+
totalPages: pages.length,
|
|
102
|
+
searchedPages: 0,
|
|
103
|
+
hiddenEpisodicPages: 0,
|
|
104
|
+
hiddenArchivedPages: 0,
|
|
105
|
+
hiddenSupersededPages: 0,
|
|
106
|
+
stalePagesSearched: 0,
|
|
107
|
+
includeEpisodic: Boolean(options.includeEpisodic),
|
|
108
|
+
includeArchived: Boolean(options.includeArchived),
|
|
109
|
+
};
|
|
110
|
+
const visible = [];
|
|
111
|
+
for (const page of pages) {
|
|
112
|
+
const hiddenArchived = !options.includeArchived && isArchivedPage(page);
|
|
113
|
+
const hiddenSuperseded = !options.includeArchived && isSupersededPage(page);
|
|
114
|
+
const hiddenEpisodic = !options.includeEpisodic && isEpisodicLayerPage(page) && !isDefaultSearchCandidate(page, options);
|
|
115
|
+
if (hiddenArchived) metadata.hiddenArchivedPages += 1;
|
|
116
|
+
if (hiddenSuperseded) metadata.hiddenSupersededPages += 1;
|
|
117
|
+
if (hiddenEpisodic) metadata.hiddenEpisodicPages += 1;
|
|
118
|
+
if (!isDefaultSearchCandidate(page, options)) continue;
|
|
119
|
+
if (isStalePage(page)) metadata.stalePagesSearched += 1;
|
|
120
|
+
visible.push(page);
|
|
94
121
|
}
|
|
95
|
-
|
|
96
|
-
}
|
|
97
|
-
|
|
98
|
-
export async function searchWiki(projectRoot, query, options = {}) {
|
|
99
|
-
return (await performSearch(projectRoot, query, options)).hits;
|
|
122
|
+
metadata.searchedPages = visible.length;
|
|
123
|
+
return { visible, metadata };
|
|
100
124
|
}
|
|
101
125
|
|
|
102
126
|
async function performSearch(projectRoot, query, options = {}) {
|
|
@@ -109,8 +133,21 @@ async function performSearch(projectRoot, query, options = {}) {
|
|
|
109
133
|
}
|
|
110
134
|
|
|
111
135
|
let pages = [];
|
|
136
|
+
let metadata = {
|
|
137
|
+
totalPages: 0,
|
|
138
|
+
searchedPages: 0,
|
|
139
|
+
hiddenEpisodicPages: 0,
|
|
140
|
+
hiddenArchivedPages: 0,
|
|
141
|
+
hiddenSupersededPages: 0,
|
|
142
|
+
stalePagesSearched: 0,
|
|
143
|
+
includeEpisodic: Boolean(opts.includeEpisodic),
|
|
144
|
+
includeArchived: Boolean(opts.includeArchived),
|
|
145
|
+
};
|
|
112
146
|
try {
|
|
113
|
-
|
|
147
|
+
const collected = await collectWikiPages(projectRoot, opts);
|
|
148
|
+
const visibility = searchVisibility(collected, opts);
|
|
149
|
+
pages = visibility.visible;
|
|
150
|
+
metadata = visibility.metadata;
|
|
114
151
|
} catch {
|
|
115
152
|
return { hits: [], search: 'missing-wiki' };
|
|
116
153
|
}
|
|
@@ -144,7 +181,7 @@ async function performSearch(projectRoot, query, options = {}) {
|
|
|
144
181
|
const page = byPath.get(item.id);
|
|
145
182
|
if (!page) continue;
|
|
146
183
|
const subScore = substringScore(page, terms);
|
|
147
|
-
const score = item.score + subScore;
|
|
184
|
+
const score = (item.score + subScore) * searchWeight(page);
|
|
148
185
|
hits.set(page.rel, resultRecord(page, score, {
|
|
149
186
|
directScore: score,
|
|
150
187
|
matchedTerms: item.terms || terms,
|
|
@@ -154,7 +191,7 @@ async function performSearch(projectRoot, query, options = {}) {
|
|
|
154
191
|
}
|
|
155
192
|
|
|
156
193
|
for (const page of pages) {
|
|
157
|
-
const score = substringScore(page, terms);
|
|
194
|
+
const score = substringScore(page, terms) * searchWeight(page);
|
|
158
195
|
if (score <= 0 || hits.has(page.rel)) continue;
|
|
159
196
|
hits.set(page.rel, resultRecord(page, score, {
|
|
160
197
|
directScore: score,
|
|
@@ -175,7 +212,7 @@ async function performSearch(projectRoot, query, options = {}) {
|
|
|
175
212
|
if (hits.has(neighborPath)) continue;
|
|
176
213
|
const page = byPath.get(neighborPath);
|
|
177
214
|
if (!page) continue;
|
|
178
|
-
const linkScore = seed.score * 0.2;
|
|
215
|
+
const linkScore = seed.score * 0.2 * searchWeight(page);
|
|
179
216
|
hits.set(neighborPath, resultRecord(page, linkScore, {
|
|
180
217
|
linkScore,
|
|
181
218
|
source: 'linked',
|
|
@@ -190,6 +227,7 @@ async function performSearch(projectRoot, query, options = {}) {
|
|
|
190
227
|
return {
|
|
191
228
|
hits: [...hits.values()].sort(sortHits).slice(0, limit),
|
|
192
229
|
search,
|
|
230
|
+
metadata,
|
|
193
231
|
};
|
|
194
232
|
}
|
|
195
233
|
|
|
@@ -282,11 +320,12 @@ export async function buildContextPack(projectRoot, query, options = {}) {
|
|
|
282
320
|
const limit = Number(options.limit || DEFAULT_LIMIT);
|
|
283
321
|
const expand = options.expand !== false;
|
|
284
322
|
const memoryExcerpt = await readMemoryExcerpt(projectRoot);
|
|
285
|
-
const
|
|
323
|
+
const rawIndexExcerpt = await readText(join(projectRoot, 'llm-wiki', 'wiki', 'index.md'));
|
|
324
|
+
const indexExcerpt = rawIndexExcerpt.slice(0, 1200).trim();
|
|
286
325
|
const logExcerpt = options.includeLog
|
|
287
326
|
? (await readText(join(projectRoot, 'llm-wiki', 'wiki', 'log.md'))).slice(-1000).trim()
|
|
288
327
|
: '';
|
|
289
|
-
const result = query ? await performSearch(projectRoot, query, { ...options, limit, expand }) : { hits: [], search: 'none' };
|
|
328
|
+
const result = query ? await performSearch(projectRoot, query, { ...options, limit, expand }) : { hits: [], search: 'none', metadata: null };
|
|
290
329
|
return {
|
|
291
330
|
workspace: projectRoot,
|
|
292
331
|
query: redactText(query || '', 1000),
|
|
@@ -297,6 +336,32 @@ export async function buildContextPack(projectRoot, query, options = {}) {
|
|
|
297
336
|
indexExcerpt: redactText(indexExcerpt, 2000),
|
|
298
337
|
logExcerpt: redactText(logExcerpt, 2000),
|
|
299
338
|
hits: result.hits.map(redactHit),
|
|
339
|
+
metadata: {
|
|
340
|
+
search: result.metadata,
|
|
341
|
+
budgets: {
|
|
342
|
+
memory: {
|
|
343
|
+
maxLines: MEMORY_LINE_LIMIT,
|
|
344
|
+
maxBytes: MEMORY_BYTE_LIMIT,
|
|
345
|
+
returnedBytes: Buffer.byteLength(memoryExcerpt.trim(), 'utf8'),
|
|
346
|
+
redactionMaxChars: 30000,
|
|
347
|
+
},
|
|
348
|
+
index: {
|
|
349
|
+
maxChars: 1200,
|
|
350
|
+
sourceChars: rawIndexExcerpt.length,
|
|
351
|
+
returnedChars: indexExcerpt.length,
|
|
352
|
+
redactionMaxChars: 2000,
|
|
353
|
+
},
|
|
354
|
+
hits: {
|
|
355
|
+
requestedLimit: limit,
|
|
356
|
+
returned: result.hits.length,
|
|
357
|
+
snippetChars: SNIPPET_CHARS,
|
|
358
|
+
hookSnippetChars: HOOK_SNIPPET_CHARS,
|
|
359
|
+
expand,
|
|
360
|
+
includeEpisodic: Boolean(options.includeEpisodic),
|
|
361
|
+
includeArchived: Boolean(options.includeArchived),
|
|
362
|
+
},
|
|
363
|
+
},
|
|
364
|
+
},
|
|
300
365
|
};
|
|
301
366
|
}
|
|
302
367
|
|
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
export const DURABLE_MEMORY_TYPES = new Set(['semantic', 'procedural']);
|
|
2
|
+
|
|
3
|
+
export function frontmatterValues(value) {
|
|
4
|
+
if (Array.isArray(value)) return value.map((item) => String(item || '').trim()).filter(Boolean);
|
|
5
|
+
const text = String(value || '').trim();
|
|
6
|
+
if (!text || text === '[]') return [];
|
|
7
|
+
return [text];
|
|
8
|
+
}
|
|
9
|
+
|
|
10
|
+
export function pageImportance(page) {
|
|
11
|
+
const value = Number(page?.frontmatter?.importance || 0);
|
|
12
|
+
return Number.isFinite(value) ? value : 0;
|
|
13
|
+
}
|
|
14
|
+
|
|
15
|
+
export function isArchivedPage(page) {
|
|
16
|
+
return String(page?.status || '').toLowerCase() === 'archived';
|
|
17
|
+
}
|
|
18
|
+
|
|
19
|
+
export function isStalePage(page) {
|
|
20
|
+
return String(page?.status || '').toLowerCase() === 'stale';
|
|
21
|
+
}
|
|
22
|
+
|
|
23
|
+
export function supersededBy(page) {
|
|
24
|
+
return frontmatterValues(page?.frontmatter?.superseded_by);
|
|
25
|
+
}
|
|
26
|
+
|
|
27
|
+
export function supersedes(page) {
|
|
28
|
+
return frontmatterValues(page?.frontmatter?.supersedes);
|
|
29
|
+
}
|
|
30
|
+
|
|
31
|
+
export function isSupersededPage(page) {
|
|
32
|
+
return supersededBy(page).length > 0;
|
|
33
|
+
}
|
|
34
|
+
|
|
35
|
+
export function hasSupersession(page) {
|
|
36
|
+
return supersededBy(page).length > 0 || supersedes(page).length > 0;
|
|
37
|
+
}
|
|
38
|
+
|
|
39
|
+
export function isEpisodicLayerPage(page) {
|
|
40
|
+
const type = String(page?.type || '').toLowerCase();
|
|
41
|
+
const rel = String(page?.rel || '');
|
|
42
|
+
return (
|
|
43
|
+
type === 'query' ||
|
|
44
|
+
type === 'context' ||
|
|
45
|
+
type === 'session-log' ||
|
|
46
|
+
rel.startsWith('wiki/queries/') ||
|
|
47
|
+
rel.startsWith('wiki/context/')
|
|
48
|
+
);
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
export function isPromotedDurablePage(page) {
|
|
52
|
+
return DURABLE_MEMORY_TYPES.has(String(page?.memoryType || '').toLowerCase()) && pageImportance(page) >= 4;
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
export function includeArchivedPages(options = {}) {
|
|
56
|
+
return Boolean(options.includeArchived);
|
|
57
|
+
}
|
|
58
|
+
|
|
59
|
+
export function includeEpisodicPages(options = {}) {
|
|
60
|
+
return Boolean(options.includeEpisodic);
|
|
61
|
+
}
|
|
62
|
+
|
|
63
|
+
export function isDefaultSearchCandidate(page, options = {}) {
|
|
64
|
+
if (!includeArchivedPages(options) && (isArchivedPage(page) || isSupersededPage(page))) return false;
|
|
65
|
+
if (includeEpisodicPages(options)) return true;
|
|
66
|
+
if (isEpisodicLayerPage(page)) return isPromotedDurablePage(page);
|
|
67
|
+
return true;
|
|
68
|
+
}
|
|
69
|
+
|
|
70
|
+
export function searchWeight(page) {
|
|
71
|
+
let weight = 1;
|
|
72
|
+
if (DURABLE_MEMORY_TYPES.has(String(page?.memoryType || '').toLowerCase())) weight *= 1.2;
|
|
73
|
+
if (pageImportance(page) >= 4) weight *= 1.1;
|
|
74
|
+
if (isStalePage(page)) weight *= 0.35;
|
|
75
|
+
return weight;
|
|
76
|
+
}
|